There’s a lot of noise right now about “AI adoption” in engineering: vague claims about productivity, thinkpieces about the future of QA, dashboards nobody trusts. What’s harder to find is the practical stuff, so I pulled my friend Nat aside, a senior quality engineer and someone who’s spent the last few months building AI tools into her day-to-day workflow to walk me through three things she’s built and the mistakes she’s seen other engineers and managers make along the way. Her answers were specific enough that I think they’re worth sharing as-is.
1. Turning a Multi-Day Setup Into 30 Minutes
The problem: Her team relies on a test management tool for tracking test plans, requirements, and test cases, partly because it’s required for audits. In practice, though, most of her teammates either avoided using it properly or stopped maintaining it once things were entered. Manually building out a new project’s worth of test cases, linking them to automation, and setting up run tracking was a multi-day task nobody wanted to do.
What she built: An MCP server with a set of custom skills and agents, built organically as she worked rather than architected upfront. It can create, update, and delete test cases; link them to automation; set up test runs across multiple pipelines; and organize everything into the release/folder structure her team needs, compatible with both Kanban and Scrum workflows. It’s usable from the command line or inside VS Code, and shareable with the rest of her team.
The result: A setup that used to take a few days, roughly 1,500 test cases of varying types, linked to automation, organized into runs for two pipelines, and tied to an upcoming release, now takes about 30 minutes.
2. A Code Review Skill That Goes Deeper Than Off-the-Shelf Tools
The problem: Her team has a lot of engineers who are new to the workforce, talented, but without much exposure yet to what “good” code review actually looks for. Her company’s built-in AI code review tool didn’t integrate with their DevOps platform, and nothing she found on the AI tool marketplaces went as deep as she wanted.
What she built: A configurable code review skill with cascading agents that can be run at different intensity levels, low, medium, high, or max, and can focus narrowly (bugs only, security only, language-specific best practices only) or broadly across all of it at once. She also built a “QA narrative” mode that goes deeper on the specific things her team cares about. She rolled it out to both developers and fellow testers.
The result: A noticeable jump in merge request quality across the team, faster reviews, and, maybe most useful for her role, a clean, structured breakdown of what was found and why, so she doesn’t have to walk people through it manually every time.
3. Documentation Tooling That Turned Into Something Bigger
The problem: Her current project runs with loose, shifting requirements, more “rough idea of what we’re building” than firm spec. She needed a way to pull context from proof-of-concept code, team wiki notes, and project management tickets, then turn that into consistent QA documentation (test strategies, implementation docs, tooling docs) across multiple formats: Word, PDF, wiki pages, project management tools, and markdown files in-repo.
What she built: A documentation skill using templates so AI-generated output stays consistent regardless of format or source. That was the original goal.
Where it went instead: The skill evolved into something that scans developers’ in-flight branches, because some devs tend to work in isolation and don’t communicate changes well, anticipates what’s coming, and drafts tests in advance. She pulls code twice a day and adjusts her tests as the branches evolve. The first time a developer apologized for unannounced changes, she was able to say she’d already anticipated the change and had tests ready and adjusted as he pushed updates. Other developers had similar reactions when they realized she’d found issues before code was even merged.
The result: Real time savings for her and her QA teammate, and a QA process that stays ahead of development even when communication breaks down, though she’s the first to point out this isn’t how things should have to work. It’s a workaround for a communication gap, not a replacement for fixing it.
The Mistakes She Sees Most Often
Beyond the builds, I asked what she’s noticed going wrong for engineers and managers alike as teams adopt AI tools. Three things came up repeatedly:
People trust AI output too easily. Her observation, which surprised her: this isn’t limited to less experienced engineers. Even seasoned QA people often assume that if AI-generated output looks right, it is right, without verifying it.
Most people don’t ask AI to check its own work. Prompting AI to self-verify, flag its own assumptions, or double check whether it took the best approach is a step most people skip entirely.
AI guesses when it doesn’t know, and people don’t catch it. Her clearest example: she was using AI to configure a pipeline tool, and it picked a low file-upload limit. When she asked why, the honest answer was that it didn’t actually know the real limit; it assumed a conservative number rather than finding out. She had it run a probe to determine the actual limit and update accordingly. Her point: AI will often make a plausible-sounding choice to fill a gap in its knowledge, and unless you ask how it arrived at that choice, you won’t know the difference between a verified answer and a guess.
Her Framework for Using AI Well
The philosophy underneath all three builds, in her words: she’s not looking for repeatable answers, the same output every time she prompts. She’s looking for repeatable results, consistently strong quality, even if the specific answer varies. She builds in guardrails that keep AI on track while still leaving room for it to explore different approaches, and she requires it to check its own work so it doesn’t repeat the same mistakes going forward.
A few practical guardrails worth stealing from her approach:
- Ask AI to explain how it arrived at a decision, not just what the decision is, especially for anything involving numbers, limits, or configuration.
- Build self-review into your workflow: have AI check its own output before you do.
- Don’t assume “looks right” means “is right.” Verification is still on you.
- Invest time in understanding how different models and effort/reasoning levels actually behave for your use case. It pays off in both quality and token efficiency.
Thanks again, Nat, for the practical lessons. None of this required a huge platform investment, just someone willing to build incrementally, stay skeptical of AI’s first answer, and keep tuning as she went.