AI Quality Assurance in Practice: We Cross-Check Every Commit Against Its Ticket

Our developers’ AI reviews their code. That part is standard by now. The question nobody on the team was systematically checking was different: does the shipped code actually match what the ticket asked for?

I am a project manager at VeryCreatives. For the past several months I have run an AI-assisted conformity check on every task that reaches QA, roughly 200 tasks so far on one product. This article is the honest version of how it works. The exact workflow, the real numbers, the rules that keep the team on board, and the parts AI still gets wrong. If the role titles themselves are unclear, start with product manager vs project manager.

Key Takeaways

• Most AI QA content covers test automation. The underused layer is business conformity: checking shipped code against the story, the task, and the acceptance criteria.
• Our PM runs this check on every task: the AI reads the Asana story, the ACs, the comments, and the attached commit or PR, then reports AC-by-AC conformity. The round takes minutes.
• Roughly 200 tasks in: at least 90% of findings are relevant, 95%+ when strictly comparing code against acceptance criteria, and rework cycles dropped from 3-4 rounds to 1-2.
• The rule that makes it work socially: every review is refined by a human and labeled “AI review, checked by PM/QA”. AI doesn’t work instead of us; it works for us, with us.

What is AI quality assurance?

AI quality assurance usually means using AI in testing. Generating test cases, self-healing test scripts, autonomous testing agents, visual regression checks. All of that is real and useful. But it leaves a layer unclaimed, and in my experience it is the layer where projects actually bleed: business conformity.

Does the shipped code do what the story, the task, and the acceptance criteria said it should? That check needs no test infrastructure at all. It needs disciplined tickets, access to the code changes, and an AI that reads both. It is also the check a PM or QA person can now run without reading code fluently. That is exactly what changed.

How does the commit-vs-ticket check actually work?

The workflow is simple enough to describe completely. Our stack is Asana for stories, tasks, and acceptance criteria, git for the code, and Claude or Codex with a maintained set of skills doing the reading.

1. A task reaches QA status in Asana. (The AI can even pull and summarize the day’s reviewable list for me.)
2. Every task descends from a story and carries bullet-point acceptance criteria, plus the linked commit or pull request. That structure is a precondition, not a nice-to-have. More on that below.
3. The AI reads the story, the task, the ACs, the comment thread, and the diff.
4. It returns a structured verdict: acceptability per acceptance criterion, anything definitely missing or divergent, risks to already-shipped work, and its own reasoning and scope.
5. I refine that output over one or two rounds. Then I post it as a task comment, labeled “AI review, checked by PM/QA”, tagging the right colleague with specific code references.
6. Definite gaps become a fix request. Suspicions become a review request. The two are never mixed.

What does it actually catch?

Across roughly 200 tasks, the findings cluster into recognizable types. The most common one surprised me: the backend-to-frontend handoff gap. The frontend needed a function or a field, and the backend simply never sent it.

It also works upstream. It flags tasks whose acceptance criteria are missing or vague before a developer ever picks them up. And it catches scope drift: code that quietly changes things outside the task’s declared boundary.

How accurate is it, honestly?

My working numbers, framed precisely: these are practitioner estimates over roughly 200 tasks, not a lab benchmark. At least 90% of the findings are relevant across conformity checks and suggestions, and 95% or more when the AI strictly compares code against written acceptance criteria. Each round takes minutes. The remaining share is the AI overthinking: flagging things that turn out to be fine.

The number that matters most to the business is rework. Fix cycles on a task used to run 3 to 4 rounds; now it is 1 to 2. Manual testing did not disappear, but it starts from a cleaner state and finishes faster. And there is a reason this layer matters more every quarter. With 42% of code now AI-generated or AI-assisted (Sonar State of Code, 2026), the volume of shipped change is up. Conformity checking is exactly the discipline that keeps that volume honest.

One more thing about the 10% that is “wrong”: it is not waste. When the AI flags a deviation that turns out to be deliberate, the developer explains why, in writing, on the task. That explanation becomes documentation of why the implementation is the way it is. It is the kind of context that otherwise lives in one person’s head. On a team that thinks about bus factor, that side effect alone pays for some of the overthinking.

The rule that makes it work: humans sign the review

This is the part I would keep if I had to drop everything else. Every review the AI produces goes through me for one or two refinement rounds. It gets posted under a visible label: “AI review, checked by PM/QA”. Definite problems arrive as fix requests; suspicions arrive as review requests, phrased as questions.

The reason is not politeness. The fastest way to demotivate a developer in 2026 is to bomb them daily with long, cold, unreviewed AI messages. If the team senses that a machine is generating criticism and a human is just forwarding it, the workflow dies. And it deserves to. The message has to show professional judgment: shortened, prioritized, with the irrelevant findings already removed by someone who knows the project. This is what human in the loop means in practice, not a checkbox but a signature.

“AI doesn’t work instead of us. It works for us, with us, and the message has to carry that.”

The effect on the team ran opposite to what you might fear. Our developers use AI heavily themselves. A PM using it well reads as speaking the same language, not as surveillance. Friction went down. The AI helps me translate between business language and development language in both directions. That translation was always the hardest part of the job.

What has to be in place before this works?

The honest prerequisite list, because this workflow does not bolt onto chaos:

• A standardized work breakdown and a documented store of decisions. The AI can only read what exists.
• Bullet-point acceptance criteria on every task. Every one. I write tasks AI-assisted too, so this costs less than it sounds.
• Commit and PR discipline parallel to the tickets. The diff must be findable from the task.
• The AI kept in frame. Maintained skills that constrain it to the project’s context and stop it from inventing requirements.
• More than one tool. Claude and Codex give me approximately equal results. This work burns through tokens fast, and the skills are portable, so never let the workflow depend on a single vendor.

The punchline is the same one we wrote about in scope creep: an AI reviewer can only check “done” against tickets that define what done means. If your tickets do not say it, no tool can check it, and the discipline of writing it down is where most of the value was hiding all along.

What can’t it do?

Three things, consistently. It cannot hold whole-context judgment at large scope. It evaluates a task brilliantly and a quarter poorly. It cannot judge value: whether a feature matters is not in the diff. And it cannot read human factors: team dynamics, client moods, the difference between a deadline that is real and one that is negotiable. The common warning about agreeable models is real too. An AI reviewer wants to find something, which is precisely why an experienced human filters it before anyone else reads it.

We build this kind of AI-assisted judgment into how we deliver, and into the products we ship (AI features for SaaS). If you want an honest look at where AI fits your delivery process, book a call.

Frequently Asked Questions About AI Quality Assurance

Can AI check code against requirements?

Yes, reliably, if the requirements exist in checkable form. Given a story, bullet-point acceptance criteria, and the linked diff, current models produce an AC-by-AC conformity verdict in minutes. In our practice at least 90% of findings are relevant across roughly 200 tasks. The constraint is the tickets, not the AI.

Does AI replace QA testers?

No. Manual testing stays; it just starts cleaner and ends faster, with rework cycles down from 3-4 rounds to 1-2 in our experience. What changes is the QA role's center of gravity: less mechanical checking, more judgment about risk, value, and whether the right thing was built at all.

How accurate is AI at reviewing development work?

Our practitioner estimate over roughly 200 tasks: at least 90% of findings relevant overall, 95% or more when strictly comparing code against written acceptance criteria. The rest is overthinking rather than nonsense, and flagged-but-deliberate deviations still produce useful written explanations from developers.

What do you need before introducing AI QA?

Four things: a standardized work breakdown with documented decisions, bullet-point acceptance criteria on every task, commit and PR discipline linking code to tickets, and maintained AI skills that keep the model in context. Without the tickets defining what done means, there is nothing for the AI to check.

Which AI tools work for QA reviews?

In our workflow Claude and Codex produce approximately equal results, and we deliberately keep both. This work consumes tokens quickly, and the review skills are written to be portable between tools. The durable asset is the process and the maintained skills, not any single vendor's model.

The bottom line

Two hundred tasks in, the discipline compounds. Better-written tickets, fewer rework loops, clearer communication in both directions, and a growing paper trail of why the code is the way it is. None of that came from buying a tool.

AI quality assurance, the way we run it, is a process decision. Define what done means on every task, let the AI read tirelessly, and put a human signature on everything the team sees. The order of those three matters, and the signature matters most.