AI QA Assistant
An agent brought QA discipline into the flow of work long before we could afford a QA role. Not as good as a senior QA, but much better than what we had before.
What I set up
At a recruitment technology scale-up, we didn't yet have a dedicated QA role, but we still needed QA discipline in how work moved from idea to release. Our rule was that someone other than the author had to verify each piece of work, and that person varied from card to card. I set up an AI agent to act as a QA sparring partner inside our delivery workflow. It ran as a scheduled workflow in n8n, with an AI agent built on OpenAI's models. The work didn't need deep reasoning, so a smaller, faster model was enough.
I started with the flow of work. The agent reminded engineers to add a colleague for QA when a card reached the QA stage without one, and nudged that colleague if a card had sat in QA for three working days without changes or comments. That part ran in the team's daily work.
Next, I took it upstream to the backlog. The goal was for the agent to review each item before it reached a sprint: ask the product manager about anything unclear, add acceptance criteria when it understood the item, sharpen the problem description and propose test considerations. Our first versions created more noise than value, but working on it with our head of product, we got it to a version that did what we wanted. It was ready, but not yet part of the sprint flow, before more urgent priorities took over.
What it revealed
Before the agent could work, the workflow had to be explicit. Which stage a card was in, who owned it, and what "done" meant had to be clear enough for an agent to act on. Setting it up forced decisions we had been leaving implicit.
Much of its value was in routine, not cleverness. Reminders about missing QA and stalled cards are simple, but they're exactly what slips when everyone is busy. Starting there gave us value quickly, with little risk.
An agent doesn't know what people already understand. Our first backlog experiments created noise instead of value. Not every backlog item is a perfect agile artifact. Sometimes the people involved already understand the work, even if the description isn't ideal. The agent couldn't tell the difference, so it flagged things nobody needed help with.
Fixing it was mostly not about the agent. Together with our head of product, we got better at writing requirements: some had been too implicit, others had been drafted with AI and read well without saying much. That took discipline from product, and help getting the team to apply the same care when they added smaller follow-up items. We also made the difference between larger items and the smaller tasks derived from them explicit in the backlog, and taught the agent to treat them differently.
What it means for a product organization
An agent can hold a discipline's routines in place before you can staff the role. That changes the sequencing: a team doesn't have to choose between hiring for a discipline and doing without it. But it buys time and consistency, not judgment. The role still matters, and what the agent covers should be designed so the role can take it over later.
Agents also tend to multiply whatever process they're added to. If the workflow is clear, an agent makes it more reliable. If it's vague, the agent makes the vagueness more visible, or worse, automates it. So designing the workflow first isn't a technical prerequisite. It's a leadership responsibility.
And agents don't see the understanding that lives between people. A team's shared context, the conversations and the "we all know what this means", is invisible to an agent working only from written artifacts. That's the old agile principle of individuals and interactions over processes and tools, showing up in a new form. An agent that reviews work needs to know when to stay quiet, or it adds noise to exactly the teams that work well together.
What I'd test next
Taking the backlog review into real use, and finding out what an agent needs to know to tell a genuinely unclear item from one the team already understands.
Read more
See the other things I've built to test how AI changes product organizations.
Back to Fieldwork