People will answer your questions. Be careful what you ask them.
Imagine volunteering for a psychology experiment. You solve a puzzle and afterward you’re asked how you did it. Did you rely more on the hippocampus proper or the dentate gyrus during that task? It used to be common for psychologists to study mental processes by asking people to describe what happened in their heads.
People were more than happy to answer psychologists’ questions. Unfortunately, we have little to no conscious experience of most mental processes (unlike being able to detect physical pain, a racing heart, or shallow breathing), so their answers were nonsense.
This type of research led the social psychologists Richard Nisbett and Timothy Wilson to write the paper "Telling more than we can know." It has an important message for anyone in the business of asking why and how people do things (i.e., anyone buying or conducting market research, managing employees, or parenting children).
People will answer your questions, even when they don’t know what the hell they’re talking about.
Nisbett and Wilson cite a famous experiment in which participants were asked to find multiple ways to tie together two cords (Maier 1931). The cords hung from the ceiling far enough apart that participants couldn’t reach both at the same time. There were four solutions to the task, but only three were obvious. The fourth solution eluded the participants until the experimenter seemed to casually swing one of the cords. Aha! The participants realized they could swing one cord close enough to the other to grab both and tie them together.
The trouble came when participants were asked how they had solved the puzzle. They reported all kinds of complicated thought processes, including spontaneously imagining monkeys swinging from trees. No one noticed that the lab manager’s action played a role.
The market research industry should read this paper. In 2022, companies spent $62 billion asking people to tell more than they know. This is a big problem because data lacking validity or reliability can lead to incorrect conclusions. Ask the marketers who bought research on Windows Vista, Crystal Pepsi, or the Dove Real Beauty campaign—garbage in means garbage out.
There are some things that people just can’t tell us, but that won’t stop them from trying. Here are some examples:
Why people like something or bought something
Question: What most influenced your decision to buy the product (an ad, word of mouth, internet search, etc.)? Why do you like this brand or individual?
The real answer: My choices and preferences are influenced by a wide variety of factors and I’m unaware of most of them. For example, re-targeted digital advertising and the actions of my family and friends impact me, but I’m not conscious of them.
Predicting the future
Question: How likely are you to recommend this product to a friend? How likely are you to purchase this product in the next 6 months?
The real answer: I’m a very poor predictor of my future preferences and desires, as evidenced by barely used gym memberships, unworn clothes in my closet, and food spoiling in my refrigerator.
How to improve a product
Question: What feature could we add to make this product indispensable to you?
Honest answer: If you don’t know the answer, and you spend all day every day thinking about this product, why would I know?
There are things that participants can tell you with a fairly high degree of accuracy.
Current bodily states
People can report how they feel emotionally (happy, sad, worried) and physically (hot, hungry, tired) in the present moment. If you need to measure this over time, some survey interfaces will periodically text people for a report, which is more accurate than recall.
Attitudes toward things
People can also respond to a stimulus, like food, a political candidate, or a logo. They can tell you how much they like the stimulus and rate its characteristics (how spicy is the food, how credible is the candidate, how much excitement does the logo convey?).
Personal characteristics
People can tell you demographic information about themselves, like education level, marital status, political party identification, and income level. They can also report on their values and beliefs, although these can be shifted by incidental factors, like mood or news events.
How do you get good data?
It’s important to understand the limits of what people can tell you. It’s important to be conservative in expectations about what people can tell you with accuracy. If you truly need to answer questions like “what feature is missing” or “what led you to buy,” a reputable research firm will talk to you about their techniques and the limitations of those techniques. If they handwave or dismiss your concerns, find a different firm.
People don’t always know why or how they do things or like things. That won’t stop them from answering your questions, though. So be careful what you ask them.