Services

Brand and CX trackingKnow what’s changing, and why.Customer satisfaction and NPS surveysThe scores, the reasons behind them and what it means.Focus groups and in-depth interviewsWhat people truly mean and why.Customer segmentationSegments you’ll still be using in three years.Customer journey mappingWhere issues and frustrations arise, and how to fix them.Concept testing and product developmentFind out if it works, if it’s distinctive and how to make it better.Research synthesis and strategyTurn the research you already have into direction.

Work

All 13 case studies →

About

Will AndersonTwenty years on both sides of the tableHow Freeform worksFive things you can count onStandardsRegistered on Victorian and Commonwealth government panelsCapability statementPDF, for procurement

Thinking

Synthetic respondents give you the answer everyone would giveResearch practice · October 2026Who your battery customers actually areEnergy · September 2026Nobody recommends their water companyCustomer metrics · September 2026All articles →

Let’s talk

hello@freeformstrategy.comLinkedInlinkedin.com/in/willanderson2

Every message comes straight to me. I reply personally, usually within a working day.

Or send a note

Your details are used only to reply to you.

← All thinking
Research practice · Will Anderson · October 2026

Synthetic respondents give you the answer everyone would give

Where AI-generated respondents help, where they’ll mislead you, and the newer risk hiding inside real samples.

Synthetic respondents are increasingly becoming a feature in research pitches. Ask a language model to answer as a 45-year-old renter in regional Victoria, run it a thousand times, and you have a “sample” by lunchtime for a fraction of the cost of fieldwork. With research budgets under pressure, it’s easy to see the appeal.

I’m not against it. AI is part of how I work every day. But I’ve read the validation studies, and the pattern in them is consistent enough that anyone commissioning research should understand it before they buy.

It’s built from what people say, not what they do

A language model learns from what people have written: reviews, forums, articles, survey write-ups. That’s the socially available version of people. It’s the reasons they give, not the habits they don’t notice they have.

Most of my work sits in the gap between those two things. People say they bought solar for the environment yet never open the energy monitoring app. People say they’d recycle more if it were easier, then don’t when it is. A synthetic respondent can give you the first half of each of those sentences very convincingly. It can’t give you the second half, because the second half was never written down.

It’s too agreeable, and too alike

The research is fairly consistent on this. A 2026 study in the NIM Marketing Intelligence Review found synthetic respondents matched real people’s brand choices about 79% of the time, which sounds good. But their attitudes were more positive than real people’s, by an average of 1.2 points on a seven-point scale, and much less varied. An earlier study in Political Analysis found synthetic answers landed close to the real averages but with far too little variation, and shifted with small changes to the prompt and between model versions. A 2026 study that generated more than half a million synthetic respondents found they liked almost everything, and exaggerated the links between taste and gender or race.

In other words: roughly the right average, too much enthusiasm, and not enough difference between people.

The variation is usually the finding

That last problem matters most, because the difference between people is usually where the useful answer is. In a value study, the customers who are neutral about you are the group most worth understanding. In the battery work, it was the households who handed the decision to someone else. In a segmentation, the whole point is that groups behave differently. A method that smooths people towards the average removes exactly the thing you were paying to find.

It’s also weakest where you most need help: new categories with little written about them, smaller or less-heard groups, and anything emotional or cultural. Those are the places the model has least to learn from, so it fills the gaps with stereotype.

The newer risk is synthetic answers in your real sample

There’s a second problem that gets less attention. In late 2025, a Dartmouth researcher published a study in PNAS showing an AI agent could complete online surveys and evade standard quality checks 99.8% of the time, at around five cents a response. Online panels have always had a fraud problem. Now the fakes are articulate, consistent and cheap.

So the question isn’t only whether to use synthetic respondents on purpose. It’s whether you’re already getting them by accident. It’s one reason I script and host surveys myself, with the panel supplying sample only, and read the open-ended answers rather than just coding them. We also prefer the only panel in Australia that has introduced identity checks to verify their users, and have seen a dramatic improvement in quality.

Where it does help

Used for the right job, it’s genuinely useful:

  • Drafting and stress-testing a questionnaire before it goes to real people, to catch confusing wording and missing options.
  • Generating hypotheses and stimulus to take into real interviews, rather than treating its answers as the findings.
  • Cutting a long list down at the very earliest stage, when the alternative is a guess, not a survey.

Where it shouldn’t be used on its own: sizing a market, building segments, anything about a new category or a smaller group, and anything going to a board, a regulator or a price submission.

Four questions to ask before you buy

If someone is pitching you synthetic research, ask them: what was it validated against, and for which kinds of question? How did the spread of answers compare, not just the averages? How does it perform for the smallest group you care about? And what would you do differently if the real answer turned out to be different? A good supplier will have answers. If they only talk about the averages, you have your answer.

Sources: Kaiser et al., “Leaving Insight to Digital Twins?”, NIM Marketing Intelligence Review, 2026; Bisbee et al., “Synthetic Replacements for Human Survey Data? The Perils of Large Language Models”, Political Analysis, 2024; Ma et al., “Not-quite-human tastes: the stylized omnivorousness of LLM survey surrogates”, 2026; Westwood, PNAS, November 2025.

Got a decision to make?

Weighing up synthetic research?

If you’re being pitched synthetic data, or wondering what’s in your own sample, I’m happy to talk it through. No deck, just a conversation.

Let’s talk