Outside The Matrix: Jolie Martin, Quantitative UX Researcher, Google

jolie martinAfter a long break we return to the Outside the Matrix series with Jolie Martin, a quantitative user experience researcher at Google. She received her PhD in Science, Technology, & Management at Harvard through a joint program between Harvard Business School and Computer Science department, and did post-docs both at Harvard Law School Program on Negotiation as well as the Social and Decision Sciences department at Carnegie Mellon. Prior to joining Google, she was also an Assistant Professor in Strategic Communication at the University of Minnesota.

Tell us about your work: how does decision making psychology fit in it? My title for the last year or so has been Quantitative User Experience Researcher at Google. However cumbersome, all the words are necessary to indicate what I do. Like my colleagues who do “regular” (qualitative) user experience research, my goal is to understand when users successfully satisfy their information needs using Google products. In my case, working on the Search Analysis team, I specifically develop metrics that describe how users interact with features on the Google search results page. The key distinction from other user experience researchers is the data source I draw upon, and as a result the types of analyses I do. Rather than running lab studies or even large online studies through tools like mturk, for the most part I rely on data recorded in logs to tell me how real users behave under natural conditions. The benefit of this approach is massive amounts of data. Nearly everything of interest is significant, sometimes even with very minor tweaks to the product that are imperceptible to the average user. The drawback – although it’s sometimes the fun part – is that I have to draw inferences from behavioral signals about users’ preferences, intentions, and satisfaction.

Judgment and decision making to the rescue! My theoretical background in this field has been extremely helpful in formulating hypotheses about why users search the way they do, from the queries they enter to the sequence of clicks that they take. For example, in considering ways to improve the user experience with exploratory tasks that require large amounts of subjective information (say, choosing where to go on vacation), I need to be mindful of contrasting interpretations of a user’s behavior. If she spends more time and clicks more links, this could be a bad signal that she simply didn’t find the information necessary to make a decision, that she suffered from information overload, or that she was distracted and continued browsing to procrastinate on a more worthwhile task. On the other hand, it could be a good signal that we offered her a rich set of information sources – increasingly tied to her personal characteristics and social networks – that offered insights worth delving into. To tease apart these interpretations requires testing mental and behavioral models of an extremely diverse set of users.

Why you decide to go into industry instead of continuing in academia? Unlike many of my academic colleagues – and even many people I know in industry who jumped ship – I never embarked on a PhD specifically to pursue a career in academia. In fact, I was clueless that this was the expectation of my advisors until several years into my PhD program! I was operating under the assumption that building theoretical knowledge and methodological skills would serve me well in any career. At some point right around my third or fourth year of grad school, I did become somewhat indoctrinated to the notion that academia is the “highest calling” and we should leave the actual implementation of our ideas to others. And of course I realized how difficult it would be to return to academia should I leave, so with this in mind, I gave it the old college + MBA + PhD + 2 post docs + assistant professorship try before finally divesting myself of those sunk costs. I liked each of my academic positions, but often felt as if I was spinning my wheels to achieve an objective (publishing in journals read almost exclusively by other academics) that I didn’t really care about, so when Google contacted me, I figured it couldn’t hurt to interview. During the process, I was surprised to find many other people like me with PhDs and interests in “pure” research. These were very smart people, and all had various personal and professional reasons for leaving academia, but it became clear to me that it was a choice, not necessarily indicating that someone couldn’t make it in academia.

That said, I am a firm believer that people enjoy things that they are good at, and where they can continue improving over time. I thought Google would offer exactly this for me. I have always loved building cool stuff, which is really the core of what we do. At the same time, there would be a lot to learn. When I accepted the offer at Google, I took a one-year leave from my assistant professorship (which was extremely generous of my department chair to offer), and it was nice to have that safety net should I dislike my new job. During the week of orientation with mostly software engineers, I thought more than once that I might need to use it. Just about everything flew over my head. But once I settled in with my teammates, I realized that everyone was willing to help, and no one had all the answers; doing logs analysis from end to end is complex by its very nature, and no one could step into the role as an expert. The expectations of me were that I be persistent and keep asking interesting questions. After a year in my position, the torrent of learning opportunities hasn’t tapered off in the least.

What do you enjoy the most in your current role? The main appeal of my job is the rapid pace that I can have impact on products that improve people’s lives in a tangible way, sometimes just through offering them a whimsical break from a busy life. I love working for a company that takes this mission seriously, and always holds it above monetary factors. Of course, this is not true of every company, so I feel lucky in that regard. I also have a nice variety of projects that result from mutual selection, and work with people in just about every role. There are only about 10 of us across the company in the Quantitative User Experience Researcher position, and our ability to glean insights from large data sets is highly valued by others. There is no prescribed way to perform these analyses, so we have freedom to use novel methods in distributed computing, machine learning, and natural language processing, among others. Last but not least of what makes my work stimulating is the chance to witness the evolution of cutting edge new technologies, such as riding in a self-driving car, wearing Glass, and seeing a prototype of a balloon that may one day provide internet in developing countries. Making these products useful requires not only tech savviness, but also political and legal knowhow.

Do you see any challenges to the wider adoption of decision making psychology in your field? Google and many other large companies are quite receptive to using decision making psychology in some ways. For example, I was involved in a “20% project” (whereby we can spend 20% of our time on something completely unrelated to our job function) running consumer sentiment surveys during the Democratic National Convention and presidential debates. I’m now working on another 20% project that draws upon academic research to test how environmental and informational factors shape food choices in our cafes. Similar studies have been conducted at Google to examine how defaults affect 401K allocations, and programs have been implemented based on the findings, with material effects on employee well-being.

However, for several reasons, there is more resistance to using basic research in the creation of products for end users. First, many companies in the technology industry are comprised mainly of software engineers (at last count, about 75% of Google employees) who may not consider psychology relevant. They often expect that users are “rational” in the sense of taking optimal actions given the set of options and information at their disposal, whereas we know this is rarely the case. Second, what research we do has focused on user response to specific technologies, with little ability to then generalize to a broader set of stimuli or outcome measures. This is related to the fast product development cycle I mentioned previously; we simply don’t have time to test fundamental psychological principles or the product will be launched and onto v2 before we have anything to say about it. This is changing gradually as the value of longer-term focus is realized. Third, while publishing is encouraged, there are not huge incentives to do so, especially given the more rigorous hoops we have to jump through in obtaining approval. Even in cases where we have interesting findings applicable to psychology more broadly, we often can’t disclose them for proprietary or privacy reasons.

How do you see the relationship between academic researchers and practitioners? In my opinion, the ideal relationship between academics and practitioners is one that takes into account the comparative advantages of each. While academics are usually more in touch with trending or provocative research topics that are likely to interest audiences and gain traction, practitioners are more aware of the available data sources and product use cases. Similarly, in terms of resources, academic connections provide legitimacy and wider dissemination of research findings, while those of us in industry can potentially be more useful in supplying funding, a sample population for experiments (be they users or employees), and analysis infrastructure (i.e., computing power). Collaborations would be more synergistic if there was greater engagement in both directions, with academics developing research questions based on real business or social issues, and practitioners making the additional effort to share findings via peer-reviewed conferences and journals.

What advice would you give to young researchers who might be interested in a career in your field? I’d suggest that students contemplating a transition to industry try a temporary or part-time internship; it’s a relatively low risk way to test the waters, and realistically, given the scarcity of professorships at top research universities, your advisors should support your consideration of other options. However, also be aware that one company isn’t going to fully represent all of industry, the same way stepping into a random graduate program or postdoc could be quite different from the one that is the best fit for you. I interned at a hedge fund during grad school and knew pretty quickly that it wasn’t for me, but it was a valuable experience nonetheless.

Perhaps more feasible for faculty members who are dissatisfied with certain aspects of their careers (e.g., working weekends and responding to emails at 3am), consider reaching out to people at companies of interest to you. You will likely find that they are excited to talk to someone with the wherewithal to do in-depth analysis of their users, and may even be open to handing over data or running experiments with you. Ask if you can present at company meetings to get a sense of the culture and style, or invite industry folks to present at your university. And don’t just build your network, but also maintain it by staying in touch with people you’ve worked with in the past. Referrals from a company’s current employees will make a big difference if you decide to apply!

Viewpoint: Why social science grad students make great product managers

A couple of months ago we featured Paul Litvak from Google in our Outside the Matrix series. After his interview, his inbox was inundated with questions from readers and he recently wrote a response on his own blog which we thought was so fantastic we wanted to republished it on InDecision as well. So, this week Paul shares his views on why social science grad students make excellent product managers. Note: even if you’re not a grad student yourself, it’s worth reading Paul’s views in case you’re ever in a position to hire one! 

After my interview with InDecision Blog, a number of graduate students emailed asking me about careers in technology (hey, I asked for it). They were a very impressive lot from top universities, but their programming skills varied quite a bit. Some less technically minded folks were looking at careers in technology aside from data scientist. Enough of them asked specifically about product management, so I thought I would combine my answers for others who might be interested.

What does a product manager do?
Brings the donuts. The nice thing about social science grad students for whom reading about product managers is news is that we can skip over the aggrandized misconceptions about product management that many more familiar with the technology space might harbor. The product manager is the person (or persons) that stands at the interface between an engineering team building a product and the outside world (here includes not only the customers/users of the product, but also the other teams within a given company who might be working on related products). The product manager is in charge of protecting the “vision” of the product. Sometimes they come up with that vision, but more often than not, the scope of what the product should be and what features it needs to have today, next week, or next year is something that emerges out of interactions between the engineers, the engineers’ manager, the product manager, company executives, etc etc. The product manager is really just the locus of where that battle plays out. So obviously there is a great need for politicking at times as well.

But wait, there’s more! Once the product is actually launched, it is typically still worked on and improved (or fixed). So the product manager is also the person that gets to figure out how to prioritize the various additional work that could be done. But how do they figure out what needs to be changed or fixed? This is one of the places where research comes in! So someone like me might do analysis on the data of people’s actual usage of the product (the product manager prioritized getting the recording of people’s actions properly instrumented, right? RIGHT?). Or a qualitative researcher might conduct interviews of users in the field and try and abstract an understanding from that. Either way, the product manager has to make sense of all this incoming information and figure out how to allocate resources accordingly.

Why would social science graduate students be good at that?
Perhaps you can see where I’m going with this. Products are increasing in scope. Even a simple app has potentially tens of thousands of users. Quantitative methods are becoming increasingly important for understanding what customers do. In such an environment, being savvy about data is hugely advantageous. In the same way that many product managers benefit from computer science degrees without coding on a daily basis, product managers will benefit from knowing statistics, along with domain expertise in psychology, sociology, anthropology even if they aren’t the ones collecting and analyzing the data themselves. It will help them ask the right questions and to when to trust results, and when to be more skeptical. It will help them operationalize their measures of success more intelligently.

The soft skills of graduate school also translate more nicely. Replace “crazy advisor” with “manager” (hopefully a good one) and replace “fellow graduate students” with “other product managers” and many of the lessons apply. Many graduate social scientists will have plenty of experience with being part of a lab and engaging in large-scale collaborative projects. Just like in graduate school, a typical product manager will spend hours fine tuning slide decks and giving high stakes presentations meant to convince skeptical elders of the merit of a certain course of research (replace with: feature, product, or strategy).

Finally, building technology products is a kind of applied social science. You start with a hypothesis about a problem that people are having that you can solve. Of course, as a social scientist, the typical grad student understands just how fraught this is! Anthropologist readers of James Scott and Jane Jacobs and economists who love their Hayek will have a keen appreciation for spontaneous order (“look! users are using this feature in a totally unexpected way!”), as well as the difficulties of a priori theories of users’ problems or competencies. In fact, careful reading of social science should make a fledging PM pretty skeptical of grand theories. For instance–should interfaces be simpler or more complicated? How efficient should we make it to do some set of common actions? If everything is easily accessible from one click on the front page, will there be overload of too many buttons? Is that simpler or more complicated? These sorts of debates, much like debates about the function of particular social institutions or legal proscriptions, are not easily solved with simple bromides like “less is always better”, or “more clear rules, less discretion” (I am reading Simpler: The Future of Government by Cass Sunstein right now, and he makes this point very well with respect to regulations). The ethos of the empirical social scientist is to look for incremental improvements bringing all of our particularist knowledge to bear on a problem, not to solve everything with one sweeping gesture. This openness is exactly the right mentality for a product manager, in my opinion.

I hope I have at least partially convinced you that as an empirical social scientist, you would make a great product manager. Now the question is, how do I convince someone in technology of that? The short and most truthful answer is, I’m not 100% certain. It might take some work to break into project management, but I see lots of people with humanities background doing it, so it can’t be that hard (One of my favorite Google PMs is an English PhD). One thing I would suggest is carefully framing your resume to emphasize your PM-pertinent skills–things like, group project management, public speaking experience, making high stakes presentations, etc. You might also consider making a small persuasive deck to show as a portfolio example of a situation where you convinced someone of something (your dissertation proposal could work?). This would be a great start. Another thing is consider more junior PM roles initially–as a PhD coming out of grad school you are still going to make a fine salary as an entry-level product manager. If you apply these principles I have no doubt that you will quickly move up.

Read Paul’s original interview here.

Outside The Matrix: Paul Litvak

LitvakPaul Litvak is currently a Quantitative Researcher working on the Google+Platform team to improve people’s social experiences online. Prior to that he was a Data Analyst at Facebook working on fighting fraud, tracking the flow of money and improving customer service. He also has a PhD in Behavioral Decision Research from Carnegie Mellon and his dissertation was on the impact of money on thought and behavior. During graduate school he co-founded a boutique data science consulting firm, the Farsite Group, which is consulting for some of the largest retailers and private equity firms to improve their data-informed decision-making processes. Through these various activities he’s managed to keep a foot in both the academic decision science and business data science worlds for the last 6 years. 

Tell us about your work: how does decision making psychology fit in it? I work at Google as a quantitative user experience researcher–I use quantitative methods to try and understand how people are (or aren’t using) features of Google products with the hopes of recommending ways to improve upon them. Often times this involves running an experiment but can also often involve correlational analyses instead. Sometimes the sample sizes are so large (millions or even billions!) you don’t need to run any statistics at all–you just count the rate at which some event happened.

Decision-making psychology fits into this work in at least three ways. First, in hypothesis generation and testing, knowing which  effects from psychology are relevant in a situation gives you great product intuition. For example, you might be analyzing how users bid on ad space and remind the engineers and designers of how much the anchor matters. Second, it’s useful in designing and conducting good experiments. In online experiments you are always weighing the pros and cons of different operationalizations of user constructs (e.g. what is “engagement” or “satisfaction” in the context of a particular website?). Being able to operationalize a variable intelligently is the difference between an experiment that convinces a Product Manager to change things accordingly and one that is totally ignored. Third, decision science lets you think clearly about analytic problems that come up a lot in software design. Nowadays it is common to use some machine learning algorithm to classify some otherwise messy data. In doing so, it is crucial to be able to think clearly about false positives and false negatives, and tradeoffs between the various costs of being wrong versus not making predictions for some cases. Fundamental statistical reasoning concepts (e.g. Bayes rule) never go out of style!

Why you decide to go into industry instead of continuing in academia? For me, it was a combination of factors. First, for many reasons (some outside my control), my research hadn’t been as successful as was needed to secure a good tenure track job. In order for me to have continued I would have had to have taken a postdoc for some number of years and continue working hard in the hopes that I could get sufficient papers published. I felt some amount of despair over my floundering career. (In retrospect, I’m not sure how overblown that was.)

Also, I had always had some interest in technology and business. I majored in computer science (and philosophy–I contain multitudes!) and had an interest in technology since I was a 10-year-old programming BASIC in my friend’s basement. Meanwhile I had co-founded a boutique statistics consulting group, Farsite (http://farsitegroup.com), that had had some early successes. Through trying to sell a variety of large businesses on consulting services (which I did in between running lab studies for my dissertation) I learned more and more about the business world. We even won a few contracts! More and more, I was enjoying applying the same scientific thinking I was using in research to solve business problems, like where to put pharmacies.

There were also quality of life issues. I wanted to have a life outside my job, and that seemed close to impossible as an academic. I noticed my advisor, who was a young tenure-track faculty, worked like a madman, seemed very stressed and unhappy. (He seems better now, and might dispute my contention that he was unhappy then.) Consequently, when a job opportunity came along to work for Facebook, pre-IPO, in Austin, Texas, where my best friend was living, it was nigh impossible to turn down.

What do you enjoy the most in your current role? By far the thing I enjoy the most about my role is having a large impact on the world. While I worked for Facebook, my analyses and code affected literally millions of dollars of revenue, and helped keep the site clean of a lot of bad content that would have made people’s daily experience much less pleasant. At Google, my research has launched whole product initiatives, determined whether to keep or get rid of product features, and literally affected what millions of people see across all of Google’s products every day. I have a huge amount of flexibility to work on research projects that interest me, in part because I love working on, and am good at formulating impactful research.

Do you see any challenges to the wider adoption of decision making psychology in your field? Yes, there are at least three challenges:

1) Because of disproportionate incentive to produce positive results and an increasing amount of researchers chasing fewer dollars and jobs, I do think the pressure to cut corners has increased significantly. This is impacting the quality of research that is being produced. Not just in terms of replicability and p-hacking, but also in terms of theoretical comprehensiveness. I read a lot of papers and I can’t help but feel like decision science isn’t very cumulative. Most researchers are chasing individual findings instead of trying to integrate our understanding of decision-making into a cohesive model or theory. It feels like it’s stagnated a bit to me–the best papers I read were written in the 70s, 80s, and 90s. I think the grab-bag nature of our findings makes it difficult to know which findings to apply in a given new context.

2) Another related problem is interactions. Social scientists uncover many many effects, but in real life many different effects could be active at the same time. It’s hard to know if all these effects should be additive, or what will win out when certain psychological antecedents suggest opposite effects. Perhaps more experiments at large scale can help this.

3) A third problem is entrenched attitudes toward experiments. I’ve definitely seen companies and executives resistant to the idea of running experiments. Sometimes they are worried about what will happen if the press finds a weird version of a product or feature. Sometimes they object to a lack of uniformity and vision in a product offering. Sometimes they are just ignorant about statistics, and have basic skepticism about generalizability and research. I’m happy to say that I think this has changed a great deal over the last 5 years. Nate Silver has done some good work in this area.  🙂

How do you see the relationship between academic researchers and practitioners? I see the relationship as fundamentally symbiotic.

Academics help practitioners in at least 4 ways (even setting aside direct collaboration, which is quite common nowadays): creating new methods, discovering findings in the lab that can then be applied, creating new theories from which to base products on (e.g. Goffman’s work on self presentation and different identities could affect the sharing model in social networks), and giving a sense of context and history. The last one is particularly important for various techno-utopists out there who think that they can use technology to fundamentally alter social relations without considering the results of previous attempts to do just that.

Practitioners help academics as well; they provide lots of data and invent useful technology. Have decision scientists and psychologists started thinking yet about what Google Glass will do to transform research? Imagine field studies were you could record what the subject is seeing when they make their choice? Or think about what the second screen could offer in terms of real time experience sampling or extra information to alter a choice. The possibilities are endless. Finally, and most obviously, practitioners often have access to lots of money… which is helpful, I’m told.

What advice would you give to young researchers who might be interested in a career in your field? Three things:

1. Come talk to me. 🙂

2. Learn some programming. R, then SQL, then Python, or some other scripting language. The more programming you learn the higher up the food chain you can go. If you know a lot of programming, you aren’t limited by what data exists, but only by what data you can create. This is hugely empowering, and increases your impact considerably. However, if all you learn is R, that is still incredibly useful,and will still get you into a variety of jobs.

3. Be curious! So many useful insights come from a broader curiosity about the world. This applies to both academic and worldly knowledge. Very random papers have led me to business/product insights. Similarly, keeping curious about what’s going on in the world is what enabled me to get into technology in the first place. Keep learning!

Want to read more? Try these…

New series: Outside The Matrix

As if one new interview series wasn’t enough for March, we’re now kicking off a second one!

As much as you love research, you may feel like academia is not necessarily your place after all, or maybe you want to mix it with some applied work – but what else is out there? As we’ve seen from our first couple of In The Wild interviews, there are plenty of exciting opportunities for PhDs in the ‘real world’. 

But what does it really feel like to make the leap and go outside the parallel universe that is academia? What’s it like there? What skills does one need? And, most importantly, how does it compare to the academic world?

To answer some of those questions we’re speaking to people who changed gear after finishing their PhDs and moved into the commercial sector. First up we have Paul Litvak from Google – buckle up and read on!

Want to read more? Try these…