Full disclosure: we haven’t experimented (yet) with an AI-powered virtual audience because it's not the right thing for our clients so these are our initial thoughts. We reserve the right to eat our words if we’re wrong.
Over the last 12 months we’ve watched, and participated in, the emergence of generative AI. We’re revelling in its ability to augment our work, increase efficiencies across our business functions, and even unlock new capabilities. However, we’ve maintained a healthy dose of scepticism when it comes to AI’s ability to meaningfully derive insights from observed human behaviour or, even more scarily, mimic real customer behaviours based on analysing published customer research.
AI output is only as good as its training data. We struggle to understand (although we’re open to exploring) how an AI model can be trained on the nuances of the lived human experience. Recently we’ve been working with a financial service provider and the variety of the stories their customers shared relating to their lived financial experience would be difficult, in fact impossible, to replicate.
Here lies the problem: how are the models trained? We assume quantitative surveys are sent to customer panels and the results, along with other inputs, are used to train the models. We also assume when a researcher interrogates the model it fills the qualitative gaps using its generative capabilities. This is where the proposition is fatally flawed. How can a large language model, at this stage of the technology's development, possibly understand what it means to be human and understand the huge complexities that drive human behaviour - a capability that the world’s smartest humans struggle to do well let alone mathematical models that are based on incomplete and somewhat dubious human inputs. Models are based on the prediction of the next best word, they’re not conscious and they don’t hold a meaningful understanding of the context of the output they are generating.
There’s no doubt AIs are efficient at analysing and reporting on data but without understanding the broader context or having real-world experience how can we possibly rely on their “insights” alone to guide us in our decision making?
Over time we’ve learnt to connect stories, ideas and data in unexpected ways. We look for common threads and test their strength through the development of models and frameworks to see if their application provides tangible benefit. From our perspective the jury is still out as to whether generative AI can have intuition. It’s a predictive model yes, but does it have the ability to join dots, make leaps, test models and continually challenge “its” own assumptions?
Ultimately the proof will be in the pudding. To have a concrete position on AI-powered virtual audiences we’ll need to run a traditional research study and compare the outputs to an AI-generated equivalent. The results will be telling and as we’ll be testing qualitative data they’ll also be subjective. So what does this all mean? It means we should proceed with caution, test the validity of AI-powered audiences and adopt an experimental mindset. We need to understand how to use these new tools and test their real world application, acknowledging their limitations in our Aotearoa context, and then define exactly if and how they’re useful.
