Hey friend – Rob here.

The other week, i asked this to the Friends of Salmon Theory private group:

By synthetic research, i mean the likes of Synthetic Users, Evidenza, and others.

(By the way, you can hear good and thoughtful interviews with the respective founders here and here.)

And reader, yet again the group did not disappoint.

In particular, thank you to Erika, George, Honza, Jaskaran, Joel, Pollyanna, Steve and Theo for their contributions.

So let's find out the bull and bear case for synthetic research in the Lord's year of 2024.

​

🐂 The bull case

It's faster and cheaper than traditional research methods.

This is where most people's minds immediately go, because ✨ in this economy ✨ yeah?

And in fairness, it's a good point, though it also potentially devalues the research market, so we'll see.

Part of me still thinks 'AI-driven anything' will be a volume play, and 'human-driven most things' will still do value.

It's helpful for performance metrics and A/B testing.

Ah, now we're getting somewhere interesting.

If in doubt of execution A vs execution B, why not test both with separate segments?

Sure beats the "i think" vs "but i think" debates, when none of us represent *checks notes* the customer.

On which note...

It's useful for "quick decisions" about average opinions.

Sometimes you don't need a big research piece, you just need something to state your case.

Or something to clarify your mind, as i've experimented with before by treating Claude as a target customer.

I ask it to give me its response, as a target customer, of this strategy vs that strategy, or this idea vs that idea.

Same thing, just different dataset.

(The AI age version of "same shit, different day?")

It's beneficial for B2B where organic data might be harder to obtain.

If you've ever tried to scope research with high net worth individuals or decision makers... yeah, i feel ya pain.

This rings extremely true, and i imagine synthetic research will take off in more 'professional' environments.

If nothing else, because of time pressures that otherwise would mean you end up with no research at all.

It helps level the playing field for SMEs and startups.

This also feels true given some of the conversations i've had with smaller companies.

The reality is, a lot of the canon of marketing effectiveness targets bigger companies, and SMEs can't always relate.

Of course, the principles of research, mental and physical availability and whatnot still are true no matter your size.

But good luck arguing to the board of your 30-person company you need a 6-month study across 40 markets.

When most of what they'll be wondering is whether they can hit their profitability targets for the year ahead.

And what the opportunity costs of doing such a massive study are, vs say increasing bonuses to the sales force.

​

🐻 The bear case

"95% similar to human responses" is problematic due to human biases.

I loved this because it takes something that's sold as a positive, and challenges it.

Also, you can argue this is based on average response rates, and as we know the averages is not where magic lives.

Then again, you can argue this of any quant (and synthetic research is, weirdly, quant data presented as qual.)

It may not be suitable for evaluating creative ideas, humour, nuance, and emotion.

Related to the previous point, but taking into consideration specific responses to communications ideas.

This becomes even more pressing if you're researching highly localised work, where you need that nuance to win.

But it also becomes a challenge in multi-market work, where the average responses per market create a Frankenstein.

Then again, if humour is a challenge on this, i'd argue emotion is the great universal leveller, but one to discuss.

(A mother's love is a mother's love no matter where you live, etc.)

It has potential for replicating biases in training data ("pale male data").

This is an extremely fair critique of all AI models, but synthetic research promises to perpetuate this even more.

Especially because let's be honest, most people won't know what the source of said data is, so mega ¯\_(ツ)_/¯.

One of the reasons i think combining synthetic with organic (ew) research is where the secret sauce will be.

Accuracy is still questionable.

Very true, because if nothing else none of this stuff is peer-reviewed.

And sure, it may be a matter of time, but the reality is this.

Most businesses might use this because they need to, not so much because they definitely want to.

Not sure yet if this is a problem, but it's worth considering what "good enough" research feels like in a business.

(By the way, i am a massive fan of "good enough" most things, if it helps move things in the right direction.)

Cost concerns: Why pay if it's not real data?

I can just imagine someone is going to do a price elasticity study on synthetic research vs organic (again, ew).

You heard it here first, folks.

It may push us further away from understanding human behaviour.

This is very true, because once you get used to doing things this way, why would you speak to a person again?

Of course, deep down we all know this, and i suspect most businesses will take a generation to ever get there.

But i have no doubt many new companies will imagine this alone will be enough to find product/market fit.

... yeah, good luck with that.

Could be seen as lazy and a shortcut.

Yep, and it probably is in some circles, hence all the debate.

(Plus, the fear of losing our relative position in an industry is often stronger than any potential gains we might get.)

Potentially used for "ass-covering and process theatre" rather than genuine discovery.

But then again, isn't most of research a bit of this?

Not a critique by the way, once you understand the dynamics of ego, buy-in, career paths and company politics.

It's just how things are, but then again i've seen tons of pre-testing done for this exact reason.

And we still keep talking about pre-testing as a valid method to have in your research mix.

​

🐻 So where does that leave us?

Well, consider this a bit like watching those documentaries that show you various sides to a situation.

And then the fun part is... you discuss among yourselves and decide what's right for you!

My take: i am a fan of this as another tool (not the only tool), though i've not experimented with it as it's pricey.

And in a way, i get it, the business model for something like this probably relies on enterprise vs smaller groups.

But if you've worked with it, or know people you have, whatcha think?

Is synthetic research the a real sweet deal?

​