Outside The Matrix: Paul Litvak

LitvakPaul Litvak is currently a Quantitative Researcher working on the Google+Platform team to improve people’s social experiences online. Prior to that he was a Data Analyst at Facebook working on fighting fraud, tracking the flow of money and improving customer service. He also has a PhD in Behavioral Decision Research from Carnegie Mellon and his dissertation was on the impact of money on thought and behavior. During graduate school he co-founded a boutique data science consulting firm, the Farsite Group, which is consulting for some of the largest retailers and private equity firms to improve their data-informed decision-making processes. Through these various activities he’s managed to keep a foot in both the academic decision science and business data science worlds for the last 6 years. 

Tell us about your work: how does decision making psychology fit in it? I work at Google as a quantitative user experience researcher–I use quantitative methods to try and understand how people are (or aren’t using) features of Google products with the hopes of recommending ways to improve upon them. Often times this involves running an experiment but can also often involve correlational analyses instead. Sometimes the sample sizes are so large (millions or even billions!) you don’t need to run any statistics at all–you just count the rate at which some event happened.

Decision-making psychology fits into this work in at least three ways. First, in hypothesis generation and testing, knowing which  effects from psychology are relevant in a situation gives you great product intuition. For example, you might be analyzing how users bid on ad space and remind the engineers and designers of how much the anchor matters. Second, it’s useful in designing and conducting good experiments. In online experiments you are always weighing the pros and cons of different operationalizations of user constructs (e.g. what is “engagement” or “satisfaction” in the context of a particular website?). Being able to operationalize a variable intelligently is the difference between an experiment that convinces a Product Manager to change things accordingly and one that is totally ignored. Third, decision science lets you think clearly about analytic problems that come up a lot in software design. Nowadays it is common to use some machine learning algorithm to classify some otherwise messy data. In doing so, it is crucial to be able to think clearly about false positives and false negatives, and tradeoffs between the various costs of being wrong versus not making predictions for some cases. Fundamental statistical reasoning concepts (e.g. Bayes rule) never go out of style!

Why you decide to go into industry instead of continuing in academia? For me, it was a combination of factors. First, for many reasons (some outside my control), my research hadn’t been as successful as was needed to secure a good tenure track job. In order for me to have continued I would have had to have taken a postdoc for some number of years and continue working hard in the hopes that I could get sufficient papers published. I felt some amount of despair over my floundering career. (In retrospect, I’m not sure how overblown that was.)

Also, I had always had some interest in technology and business. I majored in computer science (and philosophy–I contain multitudes!) and had an interest in technology since I was a 10-year-old programming BASIC in my friend’s basement. Meanwhile I had co-founded a boutique statistics consulting group, Farsite (http://farsitegroup.com), that had had some early successes. Through trying to sell a variety of large businesses on consulting services (which I did in between running lab studies for my dissertation) I learned more and more about the business world. We even won a few contracts! More and more, I was enjoying applying the same scientific thinking I was using in research to solve business problems, like where to put pharmacies.

There were also quality of life issues. I wanted to have a life outside my job, and that seemed close to impossible as an academic. I noticed my advisor, who was a young tenure-track faculty, worked like a madman, seemed very stressed and unhappy. (He seems better now, and might dispute my contention that he was unhappy then.) Consequently, when a job opportunity came along to work for Facebook, pre-IPO, in Austin, Texas, where my best friend was living, it was nigh impossible to turn down.

What do you enjoy the most in your current role? By far the thing I enjoy the most about my role is having a large impact on the world. While I worked for Facebook, my analyses and code affected literally millions of dollars of revenue, and helped keep the site clean of a lot of bad content that would have made people’s daily experience much less pleasant. At Google, my research has launched whole product initiatives, determined whether to keep or get rid of product features, and literally affected what millions of people see across all of Google’s products every day. I have a huge amount of flexibility to work on research projects that interest me, in part because I love working on, and am good at formulating impactful research.

Do you see any challenges to the wider adoption of decision making psychology in your field? Yes, there are at least three challenges:

1) Because of disproportionate incentive to produce positive results and an increasing amount of researchers chasing fewer dollars and jobs, I do think the pressure to cut corners has increased significantly. This is impacting the quality of research that is being produced. Not just in terms of replicability and p-hacking, but also in terms of theoretical comprehensiveness. I read a lot of papers and I can’t help but feel like decision science isn’t very cumulative. Most researchers are chasing individual findings instead of trying to integrate our understanding of decision-making into a cohesive model or theory. It feels like it’s stagnated a bit to me–the best papers I read were written in the 70s, 80s, and 90s. I think the grab-bag nature of our findings makes it difficult to know which findings to apply in a given new context.

2) Another related problem is interactions. Social scientists uncover many many effects, but in real life many different effects could be active at the same time. It’s hard to know if all these effects should be additive, or what will win out when certain psychological antecedents suggest opposite effects. Perhaps more experiments at large scale can help this.

3) A third problem is entrenched attitudes toward experiments. I’ve definitely seen companies and executives resistant to the idea of running experiments. Sometimes they are worried about what will happen if the press finds a weird version of a product or feature. Sometimes they object to a lack of uniformity and vision in a product offering. Sometimes they are just ignorant about statistics, and have basic skepticism about generalizability and research. I’m happy to say that I think this has changed a great deal over the last 5 years. Nate Silver has done some good work in this area.  🙂

How do you see the relationship between academic researchers and practitioners? I see the relationship as fundamentally symbiotic.

Academics help practitioners in at least 4 ways (even setting aside direct collaboration, which is quite common nowadays): creating new methods, discovering findings in the lab that can then be applied, creating new theories from which to base products on (e.g. Goffman’s work on self presentation and different identities could affect the sharing model in social networks), and giving a sense of context and history. The last one is particularly important for various techno-utopists out there who think that they can use technology to fundamentally alter social relations without considering the results of previous attempts to do just that.

Practitioners help academics as well; they provide lots of data and invent useful technology. Have decision scientists and psychologists started thinking yet about what Google Glass will do to transform research? Imagine field studies were you could record what the subject is seeing when they make their choice? Or think about what the second screen could offer in terms of real time experience sampling or extra information to alter a choice. The possibilities are endless. Finally, and most obviously, practitioners often have access to lots of money… which is helpful, I’m told.

What advice would you give to young researchers who might be interested in a career in your field? Three things:

1. Come talk to me. 🙂

2. Learn some programming. R, then SQL, then Python, or some other scripting language. The more programming you learn the higher up the food chain you can go. If you know a lot of programming, you aren’t limited by what data exists, but only by what data you can create. This is hugely empowering, and increases your impact considerably. However, if all you learn is R, that is still incredibly useful,and will still get you into a variety of jobs.

3. Be curious! So many useful insights come from a broader curiosity about the world. This applies to both academic and worldly knowledge. Very random papers have led me to business/product insights. Similarly, keeping curious about what’s going on in the world is what enabled me to get into technology in the first place. Keep learning!

Want to read more? Try these…

Passion – The Converging Theme of the SCP Doctoral Consortium

ImageJim Bettman, Punam Killer, Greeta Menon, Barbara Kahn, and many of the other speakers and round table leaders all touched on one common theme: “one should do work on what you are passionate about.”

This doesn’t mean ignoring the necessary unfun pieces one has to do to have a successful career, but it means making passion the center of one’s career. The faculty also made the point that success comes first and foremost from passion.

Jim Bettman kicked off the day of advice by putting passion as the first principle of his research plan (passion-ownership-impact). Overall, the faculty converged on the idea that passion helps motivate good work and supports a good life.

This advice stands in stark contrast to the advice graduate students often hear about being strategic. One roundtable panelist noted that “One can be over strategic” and another suggested that doing a paper just for an easy “A” publication is bad idea. Other speakers noted that one should follow their passion even if that passion leads them out of academia and into industry.

Jim Bettman summarized that that passion is a way to inspire, drive, and sustain programmatic research.

SCP Doctoral Consortium Advice Highlights: Part 1

IMG_20130228_084125

Greeta Menon and Barbara Kahn recommend the 2 – 2 – 2 pipeline plan.

The professors advised students to have a pipeline of work with a few projects at each of the stages (e.g. review, writing, data collection). As a rule of thumb they offered that one should try to have 2 items at each stage.

Be Your Own Brand

Geeta Menon explained that ““People recognized you for you, not where you go, so you can be your own brand.” She mentioned how no matter where you can go you can use your own work and your own web presence to shape how others see you.

The Punam Keller Goal List

Keller explained that she sets specific academic goals each year (~5). The goals must have actionable steps. Then whenever she considers doing an activity she simply asks “does it fit the goals?” If it does not, she puts the activity and her “say no” list. She says this keeps her focused and her “say no” list allows her to feel okay saying no to things. She also keeps a personal goals list in a similar way and accordingly keeps a great work life balanced. “I have a fabulous life,” Keller told students on Thursday.

“Play around with Facebook Ads”

Zak Tormala recommends students look into the opportunities with Facebook advertising to test hypothesis with cell sizes approaching the millions.  The use of field data was echoed by many throughout the conference. Punam Keller presented a nuanced view of field research. She said she uses the field when it is appropriate and the lab when it is appropriate. She advised against testing hypothesis in the field just because, noting “there needs to a reason” to use the field.

Highlights: Jim Bettman’s SCP Career Talk

ImageJim Bettman brought his blend of insight and humor to his career talk this morning at the Society for Consumer Psychology Doctoral Consortium. Here’s what he advised students.

His guiding principles: “P.O.I.”

Passion – “Love it or leave it.” Only do things you believe in.

Ownership – Do your own ideas, not your advisors’. Or take your advisors ideas and make them your own as his former student Mary Frances Luce did by applying emotion to Bettman, Payne, and Johnson’s established decision research.

Impact – “Go for the gold.” Aim to change the field.

Reading advice: “Make generating study ideas a focus of whatever you’re reading.”

 If you ever get the pleasure of getting a manuscript of yours reviewed by Bettman, you will find that nearly every paragraph has a comment about a potential future direction or connection to another literature. This persistent focus on generating ideas is what has allowed Bettman to be so wildly impactful. Though Bettman often appears very programatic, this should not be taken as sign of him not thinking diversely.

On what to follow up on: “Be ready for lightning bolts.”

Bettman said be open to new ideas and when you find one you are passionate about embrace it and then follow up on it. Inspiration comes first and then the programmatic process begins.

Caveat: “Not a once size fits all.” 

Bettman indicated that even though be believed in his research style, there have been many people in the field who have succeeded in following a different style of research.

Funniest quote: Good research is like “ideas having sex.”

Bettman noted that good ideas come when ideas are pieced together with other ideas, when ideas are allowed to cross-fertilizer and different combinations of idea chromosomes come together.

Second funniest quote: “When talking about how Mary Frances Luce suggested emotion be brought into decision research, “We had never discussed emotion, or been emotional.”