If, like me, you’re a data scientist in a large startup, then you probably have colleagues who work on things like ‘product,’ ‘growth,’ ‘design,’ or ‘user experience.’ Many of them may not come from a quantitative background — but you’re still likely to find yourself in meetings where you need to find a way to work with them to build great things. In fact, you’re probably not going to build great things unless you are working with them.

In terms of tips for working with non-data scientists, most of the online discussions that I’ve seen have been about bridging the gap between data science and engineering (for example, see “Lesson 7: The Two Faces of Your ML Infrastructure” in this presentation). These conversations are concerned with how to efficiently translate between exploration-oriented data science tasks and production-oriented engineering requirements. Talking with a designer, instead, is about the more abstract task of paving a path together to build a great product. So: How do you build a bridge between data science and a non-quantitative discipline?

An Analogy: From Choice Overload to Recommendation

Let’s step back and consider an analogy: researching decision making vs. building systems that help us overcome choice overload. It’s a good analogy because it is a domain that originated in the social sciences, and is now a classical data science application: it is an area that has been ‘tackled’ from two very different perspectives.

Psychologists and economists have been studying memory and decision-making for years. Their work predates any efforts to build automated systems that aim to support us while making choices — or even make them for us. This research has produced insights ranging from the magical 7 +/- 2 number to how we often make predictably irrational choices. You could argue that all of this research provides us with a conceptual model for reasoning about how we make choices; a framework that we can use to think about this domain.

And yet, none of these insights are fundamentally encoded into the statistical models that we use for recommender systems. Collaborative filtering algorithms are not hard-coded to comply with any of the findings of the psychologists and behavioral economists: if they did, they would hardly be statistical approaches anymore! Statistical models just take our data, crunch the numbers, and spit out some magic — that is what makes them scalable and beautiful.

So, what gives? This analogy is not meant to highlight a mismatch between the two fields. Conceptual models taught us that we are ‘bad’ at making choices — they motivated the need for building recommender systems in the first place. This conceptual model was then translated into a high level question: could data help us navigate choice? That question was then translated into a more specific question: how can I identify the items in a database that a user would be most interested in? Finally, we reach something that can be turned into a statistical model: how can we rank content based on signals of preference?

An Example (and more on conceptual vs statistical models)

Back to cross-disciplinary conversations — here’s an example scenario. I’m going to use it to continue to pick on one of the biggest causes of misunderstanding between disciplines that I have encountered to date: the word model. Let’s say your designer walks in and says something along the lines of:

‘…we’re working on a model of how our customers make choices in the [travel/health/music/e-commerce/your company’s] domain. We ran a workshop where participants used experience mapping to draw their journey from [A] to [B]. We figured out that we can segment our users between those who do a lot of [X] and those who do very little — and the overall trend of going from no [X] to a lot of [X] is mainly influenced by [Z].’

As a data scientist, a number of keywords are probably firing off in your head. Model? Probably supervised learning. We should be able to predict what kind of user this is! Segment? Sounds like clustering — we can find these kinds of users in our data! Trend? Time series forecasting — we can predict what a user will do next based on [Z]!

Actually, this is probably not what your design team is trying to communicate at all. More likely, they have been working on conceptual models, segments, and trends: their goal is to create a framework that we can use to explain (to humans) who the customer is and, based on that, design products for them.

This is a vastly different exercise than creating statistical models, which aims to automate (with machines) a particular, well-defined, task. Most importantly, directly converting a conceptual model into a statistical one (in this case: predicting the kind of user, clustering users based on a single behavioral trait, and time series forecasting of user actions) does not necessarily satisfy the requirements of any of the future products that you are working your way towards.

Pushing the other direction is just as fruitless. Let’s flip the meeting around, and say that you walk in and say:

‘…we’re working on a model of how our customers make choices in the [travel/health/music/e-commerce/your company’s] domain. We looked at 100 million log entries from users in our system who are using it to get from [A] to [B]. We figured out that we can segment the users into [X] groups based by modeling each one as a vector with 500 features, and predict what each group is going to [click/purchase/read/share] with 99% accuracy. The most discriminant feature was [Z].’

That’s pretty dense: you’ve just given a run down of how you could automate a very specific task, such as predicting that a user will click on something. So what? How does it satisfy a customer’s need, and the company’s roadmap? Your statistical model will not actually be helpful until you can translate its implications back up to the product. For example: being able to predict that a user is likely to click on something could be used to rank things in order of interest, which allows us to build a personalized experience for these users.

In summary, some tips for cross-disciplinary work

Over the last few years, I’ve had the pleasure of working on a number of projects that involved a healthy dose of qualitative and quantitative methods. I’ve seen variants of the example above when working with designers on speculative data-driven apps, business mentors brainstorming a startup’s strategy, and behavior-change psychologists who wanted to apply their theories into smartphone apps. There are a number of things to be mindful of:

• Data Science is a ‘young’ discipline. Many of the people on the other side of the table are probably trying to get to grips with what a data scientist is (after all, so are we), what machine learning is (perhaps from a business perspective), and how data scientists can contribute.
• Quantitative and qualitative methods are not adversarial. If you only speak the language of p-values/experimentation and are going to shoot down any argument that is based on design principles, diary studies, interviews, or experience mapping, then you’re not going to get anywhere. The same goes the other way around: if data-driven insights are dismissed as ‘inhuman,’ then progress is unlikely — but part of the onus is on you to present your work in the right way.
• There is no hard-rule on whether to do qualitative or quantitative work first. ‘Big data’ analysis is not an exercise in scaling qualitative research outputs — it is about answering different questions altogether. How can any data analysis contribute to a conceptual model? A useful heuristic to go by is that quantitative work is best suited when characterizing those behaviors that it is easier to log than to recall.
• Everyone in the room is going to be speaking a different language. Specifically, everyone is likely to be using common words, such as ‘data,’ ‘model,’ ‘segment,’ and ‘trend’ — but referring to very different things. If you’re not aware of this, you’re likely to have a very frustrating time.
• Both quantitative and qualitative methods have their own biases: relish that.
• Make sure to clarify the difference between models aimed to explain and models aimed to automate. Don’t try to automate the model that explains; conversely, remember that the model that automates may not provide a useful explanation.
• Most of the conversation is going to revolve around translation. How does their idea, or conceptual model, translate into one that you can design a statistical model around? You’re probably going to have to take a number of steps here. How does your statistical model translate back up to meeting the needs of the customer? There are interesting methods (like design fiction) that healthily intersect the two.