Talking to Hundreds of Users Instead of a Handful
The cost of learning from customers can drop, not just the cost of building. The hard part moves to asking good questions.
What I built
The idea started with a conversation on LinkedIn. The other person pointed out that while engineering is speeding up with AI, product is still limited by how fast it can talk to the market and its customers. I agreed that this is true with existing processes, but I didn't think it had to stay that way. Product could use AI to scale its own work too. That got me thinking about how, specifically.
In earlier product roles, talking to customers always meant choosing between two imperfect options. Qualitative interviews gave rich, open answers that could take us in new and interesting directions, but they took so long to run and process that we could only talk to a few people. Structured surveys could reach many more, but only with predefined questions and answer options. That always worried me: the insights were limited to what we in product thought to ask, and we would miss the unknown unknowns.
So I built a tool to test a third option. A product manager sets up a feedback round with a few open-ended questions and uploads a list of respondents, potentially hundreds. Respondents get an email, sign in with a magic link and answer in free text, with as little friction as possible. When answers come in, AI analyzes them together: it identifies themes, groups responses under them with a sentiment, summarizes the key insights and flags notable outliers.
The analysis of a test round: summary, key insights, notable outliers and themes. Click to view full size.
Because I thought it might become a product, I built it as a multi-tenant application from the start. Organizations are managed centrally, each product manager describes their company and product (which is what respondents see), and a round can be followed and closed as responses come in.
I tested it only myself, using my own email addresses. I wrote deliberately different answers, and I also had an AI generate larger sets of feedback pulling in different directions, with subtle relationships between them, to see whether the analysis would catch them.
What it revealed
The analysis was better than I expected. It consistently found the trends and distinct topics across answers, including the subtle relationships I had hidden in the generated feedback. Turning a pile of free text into structured insight was clearly no longer the hard part.
It noticed things nobody asked about. In one round, it flagged that two responses appeared to come from the same person using different email addresses. That's a small example, but it's exactly the kind of unknown unknown that predefined survey questions never surface.
It doesn't stop at insights. The analysis didn't only summarize. It also suggested what to do, like adjusting positioning or testing pricing. That's useful as a starting point, but it blurs the line between what customers said and what the product team should decide.
The real test is still open. Everything I saw came from my own testing. Whether real respondents write enough in free text, and how well the analysis holds up against genuinely messy answers at scale, remains to be seen.
What it means for a product organization
If engineering gets faster and discovery doesn't, organizations just build the wrong things faster. The bottleneck moves to product discovery, and that's where AI needs to be applied next.
Removing the trade-off between depth and scale changes what discovery can be. Product teams no longer have to choose between talking deeply to a few customers and asking narrow questions to many. That also changes the skill that matters most: instead of designing answer options, product managers need to design open questions that invite answers they couldn't have predicted.
It also shifts the product manager's work from processing input to judging it. When AI produces both insights and recommendations, someone still has to decide which recommendations reflect what customers actually need, and which simply sound reasonable.
What I'd test next
Running a real round with a real customer base, alongside a handful of traditional interviews, to see where the tool finds insights interviews miss, and where it misses what a conversation would catch.
Read more
- The Bottleneck Didn't Disappear. It Moved. (on LinkedIn)
See the other things I've built to test how AI changes product organizations.
Back to Fieldwork