Why read this? : Bias in market research makes your results less reliable. We share different ways it appears in each of the 3 types of research approach. Learn who and what causes it, and how to reduce its impact. Read this to learn how to deal with bias in market research.
A bias is an influence or prejudice which leads you to think in a subjective rather than objective way.
In market research, you want objectivity. Objective results are more accurate. More accurate results lead to better marketing decision-making.
Easier said than done though.
Bias is everywhere. It influences how we perceive the world around us. It influences what we like, what we don’t like, how we make decisions and what we do.
To reduce market research bias, you first need to identify where it occurs.
In the market research process, most biases appear when you do the research. That then impacts your analysis and action plan.
Here, we focus mainly on where bias occurs in each main approach to doing market research :-
Bias in secondary research
Secondary research uses existing research. You try to find answers from research originally carried out for some other purpose.
It’s often used to familiarise yourself with a new topic. As it’s often free, it’s also often used where you don’t have much to spend.
For example, there’s often free published reports from national bodies like the Australian Bureau of Statistics. There’s also many specialist publishers. For example, the well-known annual Internet Trends report from Mary Meeker of Bond Capital.
Much of this is available online. See our keyword research article for how helpful Google is, for example.
Secondary research has lots of plusses. There’s a lot of it out there. It’s usually easy to find. And it gives you a broad overview of a new topic for little or no spend.
But, it has limits. Its answers often don’t answer your specific questions. And the temptation to try to make its answers fit your questions, brings in bias to this type of market research.
Self-interest (source) bias
For example, there’s often self-interest bias with the research source.
This is where whoever’s behind the research wants the results to support their own agenda.
That leads to subjective results. Or in other words, biased results.
As an example, we recently looked at this research study from Think TV. Read the research summary, and on the surface, it sounds unbiased.
Done by a “leading independent marketing analytics firm”. It used econometric modelling across 3 years of data on 21 top advertisers in Australia. There’s lots of specific sounding ROI numbers. Specific numbers always sound more believable, right?
But park the methodology for a second.
Consider who paid for this research which shows TV as the most efficient media channel across FMCG, Automotive, Finance and E-Commerce.
An organisation called Think TV. Their stakeholders are all the TV companies who sell TV advertising. It’s hard not to be sceptical. Especially when there are other more independent sources on media and marketing effectiveness. The Ehrenberg Bass Institute, for example.
So always check your sources. Ask why they did the research in the first place. Could the purpose have led to bias in the results? The more the chance of bias, the less reliable the results.
Sampling bias
Also consider if the sample used represents the total market. If it’s been selected to deliver a certain result, that’s sampling bias.
In the Think TV research, for example, it mentions the data from 21 companies. But we don’t know which companies. And why they chose only those 21 companies.
Maybe they only picked companies who used TV advertising successfully? And ignored those whose TV campaigns didn’t work.
We don’t know how they picked the sample. But research should always be done on randomly selected, representative samples. Otherwise, you won’t get the full picture. The research will be biased.
Availability and confirmation bias
Bias also appears when you analyse secondary research. If it’s all you have, you may overvalue its importance. You may stretch answers to fit your questions, even if those answers aren’t correct.
This is called an availability bias. You make decisions on what you have, without doing further research or thinking. This can lead you in the wrong direction. The results could be out of date, inaccurate or incomplete. You make worse decisions, because you’re relying on poorer quality data.
Similarly, confirmation bias can lead to poor decisions. Here you have some idea about what the research answers are (or what you want them to be) before you start researching.
So, you cherry pick results which back up that thinking, rather than staying objective. You ignore anything which contradicts your view. But that’s subjective. Remember, subjectivity creates bias in market research. You need to stay objective.
How to reduce bias in secondary market research
As the research has already been carried out, your only way to reduce bias in secondary market research is in how you analyse and use it.
Gather data from multiple independent sources for example. Look at who’s behind the research and what its purpose is.
Look at the methodology. High quality research shares how it managed bias.
If it doesn’t do this, or you see obvious signs of bias, be wary of using those results.
Be careful as you analyse and interpret it. Are you staying unbiased? Make sure you’re not using it because it’s too hard to find out more. Or just to confirm what you already thought.
Secondary research sounds great, because it’s fast and cheap. But you get what you pay for, and the quality can vary. Be careful of being biased.
Bias in qualitative research
You use qualitative research to speak directly to customers. To find out why they think and do what they do.
Why is an important question. You want to know what influences customer attitudes and behaviours.
The source and purpose of the research is clear. It’s your research questions asked to your target audience. You (or your research company) ask questions, listen to the answers and adjust future questions based on those answers.
It’s a flexible, adaptive process. But that flexibility and adaptiveness brings new market research biases to worry about.
Bias in the interviewer
Market researchers who run one-to-one and focus group interviews are trained to reduce bias.
That includes building up rapport and empathy with the respondents.
People will be more open (and less biased) if they trust the researcher.
Researchers have to take a lot of care to put respondents at ease. Even down to how they dress, how they speak and their body language.
The researcher needs to stay neutral and impartial throughout. They can’t let their own beliefs or opinions creep in. To get the unbiased thoughts of the respondents, the researcher needs to make them feel safe.
Sensitive or emotional topics
Researchers also need to take care if the topic is sensitive or emotional.
They have to keep their own opinions out of research on political or social topics for example.
For topics like taxation, welfare support or abortion, the interviewer has to learn to stay neutral.
No nodding at opinions they agree with. No head-shaking at things they disagree with. These types of reactions bias the results.
It can also creep into more innocuous sounding topics. How much TV someone watches, or what shows they watch, for example. Some respondents might feel the interviewer will judge their answers. They might claim to watch documentaries when actually they mostly watch reality TV. That would bias the results.
Anchoring bias
Then, there’s the anchoring bias. This is where you overemphasise the value of the first piece of information you hear. Everything which follows is anchored to what you heard first.
So, for example, say you’re researching price. If you say the lowest price first, everything after sounds expensive. But if you say the highest price first, everything after sounds like better value. You need to vary the order to reduce bias.
Any time you ask respondents to choose from a list of options, be conscious the first item in the list gets the most attention. Switch the order of lists around to help reduce this bias.
Bias in the respondents
It’s not just the researcher who can bring bias. There can be bias in the respondents too.
For example, there’s the mix of respondents you choose. Like before, this is sampling bias.
You recruit respondents because they’re in the segment you’re researching. But they don’t statistically represent the whole segment.
And though they might fit the demographic (age, gender, income etc) profile, it’s usually attitudes and behaviours you’re more interested in.
There’s no guarantee the people you speak to have the same attitudes and behaviours as the total segment. So, the research results may be subjective to that small group, not an objective view of the total market.
Groupthink and social acceptance bias
In focus groups, you also have to watch for groupthink bias. This is where one person shares an opinion and everyone is too polite to disagree. So, they nod along to keep the group happy, even if they disagree.
Few people like to stand out with different opinions when in a group of strangers. This is known as social acceptance bias. You feel social pressure to go along with the group. You don’t want to express a different opinion and be judged for it. Often the only way to identify these types of beliefs is in one-to-one interviews, where there’s no group influence.
False memory bias
Similarly, there’s false memory bias. In categories where the purchase you’re researching doesn’t happen very often, respondents might not remember how they made a decision. But it can be embarrassing to say you can’t remember. So, sometimes respondents make things up.
It’s less of an issue with regular purchases, but it happens a lot with bigger ticket items like cars or electronics. It could be years since their last purchase. If it’s hard to remember details. you guess or say what you think the research wants to hear. That adds bias to the research results.
Bias in your observations
Often you get the chance to observe the qualitative research as it happens (especially with focus groups). But it’s usually only one or two sessions, not the whole thing.
And that can unfortunately bias how you interpret the research. Because you may well over-value the sessions you saw in your decision-making.
You have an availability bias. You can picture the sessions you observed easily. But the ones you didn’t observe, you can’t really visualise.
So you end up favouring what you observed, and not staying objective. You have to step back and look at the research as a whole. Don’t let your observations bias your interpretation.
There’s also a danger of confirmation bias when you observe research. You listen out for respondents to give the answer you want. You’re biased to listen for answers that validate your opinions. That’s being subjective. Remember, unbiased research means staying objective.
Bias is mainly driven by people
Staying objective to reduce bias is easier said than done though.
It’s especially hard in qualitative research. Biases are mainly driven by people. Qual is full of people-driven interactions. You’re trying to stay objective listening to the subjective thoughts of the respondents. That’s hard to do.
Clear processes can help reduce the bias. Sample selection and the order of the questions for example. But qual depends on open questions and free-flowing responses. It’s difficult to prepare for all eventualities. You never know what respondents will say. If you did, you wouldn’t need to do the research.
Processes to help reduce bias in qual
Work with your market research company on processes to reduce bias. For example :-
- use a team of researchers and rotate them around different roles (e.g interviewer, observer) as the research progresses.
- have multiple people (including you) review the question structure for biased questions before the research starts.
- use counter biasing statements – eg. saying a behaviour is common, then ask the question about it.
- make indirect statements i.e. refer to what “other people” think or do. The assumption is the respondent’s answer reflects what they think or do.
- use labeled response categories for sensitive answers. The answers are given a label, and respondents call out the label, not the actual answer.
Bias in quantitative research
The structure normally revolves around a questionnaire. You review this for bias first which helps screen out biased questions.
Respondents generally fill in the questionnaire themselves online, which removes interviewer bias. (there’s very little on-street or phone interview surveys these days).
That also means the answers come direct and unfiltered from the respondents, so no bias creeps in before the analysis stage.
Biased questions in quantitative research
Let’s start with where bias can come in to the questions you ask. Remember, bias tries to lead you to think in a subjective, rather than objective way. Biased questions don’t get you objective answers.
Self-serving questions
For example, some questions may have a self-serving bias. They try to get the respondent to give an answer which supports your existing opinion. Take this question as an example :-
“on a scale of 1 to 5, how much did you like the award-winning advert from (Product X)?”.
What’s wrong with this? Clearly, it’s that wedged-in “award-winning”. If you mention something’s award-winning, your instant reaction is it must be good. It’s won awards after all. But that’s leading respondents to be more positive about the advert. It’s biased.
False-Consensus bias
Then there’s false-consensus bias. This is where you assume or make out an opinion to be what the majority of people think, when that may not be the case. It can bias respondents to agree with that view, even if they don’t really believe it. Showing “what everyone else thinks” can bias answers and stop you understanding what specific respondents think.
Take this question for example :-
“do you agree with the majority of people that Product X is the best on the market?”.
See how “majority of people” introduces bias? You’re implying the respondent should also think Product X is the best on the market, because everyone else does. But that may not be true. You’re biasing the results.
Question fatigue or confusion
Finally, bias can also creep in if you ask too many questions. Answering questions is tiring. It’s tiring because it takes brainpower, and our brains use a lot of energy.
As per our 5Ws of idea generation article, our brains are 2% of our body weight and 20% of our daily calorie consumption.
If respondents get tired, they’ll stop reading questions properly. They’ll give less well thought out answers. That introduces bias into the result.
In general, attention spans peak at around 20-30 minutes. The longer it takes to answer the questions beyond this, the more the quality of the answers will drop.
Similarly, questions which are confusing also drain people’s energy. If they have to think too hard about it, they’ll end up guessing at answers. Again that brings more bias into the market research.
Keep questions clear, and questionnaires short. The better you do this, the less bias you’ll find in the results.
Biased analysis in quantitative research
Bias can also come into quant research analysis.
For example, there’s the whole area of statistical significance. This should help you stay objective.
But often because it’s not well understood, people misuse it, or make subjective decisions anyway.
For example, say we quant tested two advertising campaigns. A and B. We want to know which one customers prefer. And let’s say 44% prefer A, and 42% prefer B. You go with A, right?
But then the agency tell you the result isn’t statistically significant. It’s inconclusive. It could still be more people prefer B to A. You should really do more research. Ask more questions. But of course, you don’t. Because that would take time, money and effort. So, you go with what you’ve got.
But this is technically biased. It’s what Daniel Kahneman in his book Thinking, Fast and Slow calls WYSIATI. What You See Is All There Is. (see more about Kahneman’s work in our thoughts about thinking article).
Your brain likes likes easy answers. In this case, 44 is bigger than 42. Trying to persuade it to do more research to find a “statistically significant” (and correct) answer is too much hard work. You take the easy (but biased) option.
Biased sampling in quantitive research
You also need to take care to avoid bias in how you do sampling. The sample size and selection is important in quant research.
You need the sample to statistically represent the total segment or market. That statistical back-up is a core part of your decision making.
For some research topics though, this can be a challenge. It can be hard to find a good sample, if the topic is sensitive or emotional. Or the respondent believes their answers might have consequences.
The sample you do get (people willing to answer questions about a difficult subject) may not represent the total market.
You need to find ways to make sure your sample is more representative to reduce the bias. Making respondent answers more anonymous for example, or incentivising a wider range of people to take part.
How to reduce bias in quantitative research
The way quant research works gives you opportunities to reduce bias. It’s usually the market research approach with the least amount of bias, because it’s more structure and process driven. There’s less people bias to manage.
There’s 3 key areas to focus on :-
The questions
Always review the questions before they go out. Look for any you think might be biased. Ask the market research company to share their processes for reducing bias. Do they follow quality control standards such as AS : ISO 20252, the international quality standard for market and social research? (See the Market Research Society website for more on this).
For example, they should make sure several people check the questions for bias. They should also rotate the order of questions. That also helps reduce bias. The less bias in the questions, the more reliable the results.
The analysis
Beware rushing to snap judgments. Don’t assume the answer’s on the first chart you see. Listen to the whole debrief. Stay objective. Ask the research company questions. It’s what you’re paying them for.
Don’t make snap decisions. Give yourself time to process what you hear. Read the debrief again the next day with a fresh mind. Get your team together and listen to their thoughts. You want to make sure your own biases don’t overly influence the decisions you make.
Also, have back-up plan if the research results aren’t definitive, or take you in an unexpected direction. You want to make sure you’re making good decisions.
The sampling
Check the sample size. Is it big enough to statistically represent your target audience? Ask the research company how they calculate the sample size, and how confident you can be in the results.
Quant research costs the most to do, so you want the results to be clear, reliable and bias-free. Make sure you get your money’s worth from the research company.
Bias in the people involved in market research
In all 3 approaches, we’ve mentioned how people bring bias into the market research process. Let’s close with a closer look at the people involved, and why those biases might occur.
Bias in market research companies
Market researchers who sign up to the standards and policies of The Research Society commit to following best practice in eliminating bias from market research.
For example, in their Code of Professional Behaviour, under Data Provision and Reporting, clauses 34-36 spell out their obligations to eliminate bias :-
- 34 : When reporting on a project, Members must make a clear distinction between the findings, the Member’s interpretation of those findings and any conclusions drawn or recommendations made.
- 35 : Members must provide their clients with appropriate methodological details of any project carried out for the clients to enable them to assess the validity of the results and any conclusions drawn.
- 36 : Members must take reasonable steps to ensure that findings from a project, published by themselves or in their company name, are not incorrectly or misleadingly presented.
A distinction between the findings and their interpretation of those findings. That’s about reducing bias. The findings are objective. The researcher’s interpretation is subjective.
Details of the methodology to assess the validity of the results. That’s also about reducing bias.
And steps to ensure findings are not incorrectly or misleadingly presented. Those too are about presenting less biased results.
Bias in marketing agencies
Your marketing agency help you convert research answers into actions. They have an inherent interest in what the research shows. It impacts what and how much you do with them.
For that reason, you should include them in the process. Share the brief, and ask them for feedback. Invite them to observe focus groups. Make sure they’re at the debrief.
But, take care with how they analyse the results, and what they recommend you do next.
They have biased self-interest in the outcomes. They’ll look for signs in the research you need more of the services they provide. So, you’ll get a biased view that you need more advertising, media, digital, or whatever it is they do for you.
You’re the one who needs to stay the most objective. It’s your brand and your marketing mix overall. You need to find the best right answer for you.
Bias in you
Which brings us to the hardest bias to get rid of. That’s the bias in you.
What you think is subjective to you. Hard as you try to stay objective, you need to constantly watch our for your own biases. They’re often hard to spot.
So, ask for feedback from others in the research. Try to be aware of areas where you know you’re biased.
Use some of the techniques we’ve described here to try to reduce the impact of those biases.
Acknowledging they exist is the first step in getting rid of them. The more you can screen out bias, the more objective your decision-making will be.
Conclusion - Managing bias in market research
The best way to reduce bias is to prepare for it.
Look at your market research plan. Where could bias creep in? Look for obvious bias areas first.
Is there self-serving bias from your secondary research source? Confirmation bias from qual interviewers? False-consensus bias in the quant questions you ask?
Think about who’s involved in the research. What biases will the research company bring? What about your marketing agency, what biases will they have? And of course, think about your own biases.
To learn more about bias, there’s a couple of books we really like. First, Richard Shotton’s The Choice Factory. It talks a lot about biases, and we share a very specific example in this article. Also, Daniel Kahneman’s Thinking, Fast and Slow which we cover in this article also has a lot on bias.
You can also check out our behavioural science article or our market research guides for more on this topic. Or contact us if you need help managing bias in market research.
Photo credits
Trash sign : Photo by Gary Chan on Unsplash
Attention sign : Photo by Markus Spiske on Unsplash
Google Tablet : Photo by PhotoMIX Ltd. from Pexels
Google Analytics : Photo by Edho Pratama on Unsplash
Glasses : Photo by Josh Calabrese on Unsplash
Conversation Image : Photo by Joshua Ness on Unsplash
Legal scales : Photo by Tingey Injury Law Firm on Unsplash
Man crying : Photo by Tom Pumford on Unsplash
Frustrated Man (adapted) : Photo by Usman Yousaf on Unsplash
Eye : Photo by Daniil Kuželev on Unsplash
Abacus : Photo by Crissy Jarvis on Unsplash
Question mark sign : Photo by Emily Morter on Unsplash
Bored in front of computer : Photo by Magnet.me on Unsplash
Calculator : Photo by Ray Reyes on Unsplash
Hand / Stop : Photo by Nadine Shaabana on Unsplash
Two men working together at a desk : Photo by Nik MacMillan on Unsplash
Handshake : Cytonn Photography on Pexels
Surprised Monkey : Photo by Jamie Haughton on Unsplash
