For recruiters, vetting Quality Engineering talent can be difficult.
A resume can tell you someone has worked with Playwright, Cypress, Selenium, APIs, CI/CD, or AI tools. But those keywords don’t necessarily tell you how someone thinks about quality, how they work with developers, or whether they understand what is happening underneath the test they’re running.
So I asked senior QE Steven a few questions about how he approaches modern Quality Engineering, particularly around AI, developer collaboration, technical depth, and how recruiters can better evaluate candidates.
Normally, I’d weave these answers into a story, but I think the substance here is too good to lose in my narrative. So I’m rolling this one out Q&A style.
Let’s dive in together!
How do you test an AI agent?
Jaclyn: You’ve built custom Claude agent skills for context-driven test planning. Walk me through how you tested and validated the agent itself. How did you determine whether its output was accurate and complete, and what did your feedback and evaluation loop look like?
Steven:
Steven explained that the process took about a month of iteration.
Before building the skill, he would essentially provide Claude with whatever context he thought was relevant and ask it to generate a test plan. That worked reasonably well for his own testing, but creating something that could produce useful and predictable results for an entire team was a different challenge.
He wasn’t only thinking about whether the LLM was technically accurate. He also had to consider whether the workflow was easy enough that people would actually want to use it, whether the agent consistently received the information it needed, and whether the results were genuinely better than the way he had been working before.
Once the skill was ready for others to use, he asked teammates to provide stories they had already tested and planned. He then compared the test plan generated by Claude against the test plan created by his teammate, looking at the test cases individually and having the QE determine which generated cases were actually useful.
There was also a significant amount of iteration before that beta-testing phase. He eventually introduced an intake form so the agent would collect the necessary context before generating a test plan, reducing the chance that a user would unintentionally leave out information that could affect the result.
And perhaps the simplest form of validation was using the output for actual testing. Steven looked for hallucinations in the generated plans and paid attention to which sections were genuinely helping the testing process versus simply adding more content.
What should recruiters take from this?
If a candidate says they use AI in their QE work, there are better follow-up questions than simply asking which tools they use.
You can ask:
- How did you evaluate the AI’s output?
- How did you test for hallucinations or other failure modes?
- How did you make sure the agent consistently received the context it needed?
- How did you determine that the AI was actually improving the process?
These questions can lead you into concepts such as LLM evaluation, failure-mode testing, structured inputs, and guardrails. You don’t need to be an AI engineer to understand the basic ideas behind them, but knowing the vocabulary can help you recognize when a candidate has actually thought through the reliability of an AI workflow.
What does a strong Dev/QE partnership look like?
Jaclyn: You’ve been pushing testing further left by getting developers more involved in unit, integration, and E2E testing. What does a strong developer/QE partnership actually look like in practice?
Steven:
For Steven, the foundation is mutual trust.
He believes the best partnerships are ones where the QE can contribute test code and the developer can participate in testing. In those environments, the roles can overlap in productive ways without eliminating the distinction between development and QE.
He also believes QEs benefit from understanding basic unit and integration testing patterns and anti-patterns. At the same time, developers may rely on QEs for things like setting up effective test environments or improving automated E2E testing.
The important part is that both groups are collaborating in the codebase and in the application and making decisions together about how testing should work.
One thing Steven deliberately does is become comfortable working in the same repositories as developers. He sees that as a way to build trust and challenge the assumption that QE doesn’t understand or participate in development work.
He also sees AI as potentially lowering the barrier for QEs who don’t have a particularly strong coding background.
What should recruiters take from this?
This is a useful distinction when you’re evaluating candidates who describe themselves as “automation engineers.”
Someone can be very experienced with an automation framework without necessarily understanding how their testing fits into the broader software development lifecycle.
Some questions worth asking:
- How have you collaborated with developers on unit or integration testing?
- Have you contributed to test code in a developer’s repository?
- How do you decide what belongs in a unit test versus an integration or E2E test?
- How involved are you in testing earlier in the development process?
This is where concepts like shift-left testing, unit testing, integration testing, E2E testing, and SDET become useful vocabulary for recruiters.
Which technical skills make a QE stronger?
Jaclyn: Looking back over the last few years, what technical skills have been most important to your ability to operate at a high level today?
Steven:
Interestingly, Steven didn’t start with a technical skill.
He identified communication and social confidence as the most valuable skills because they allowed him to build relationships, expand his influence, and establish trust within an organization.
He described technical skills such as frontend development and understanding unit testing as tools that helped him build that trust with developers. When developers recognize that a QE understands the code and can contribute meaningfully, they become more comfortable collaborating with them on test suites or bug fixes.
But Steven also emphasized the importance of knowing where to stop.
The goal isn’t for QE to take over development. He sees technical skills as a way to become more effective at testing, collaborating, and understanding the software rather than as a reason to take on every development task available.
He also believes that the more a QE understands how full-stack applications are developed, the better equipped they are to test those applications and reason about potential root causes when something goes wrong.
His broader point was that technical skills are most valuable when they increase a QE’s ability to influence the team and build trust.
What should recruiters take from this?
A technically strong QE isn’t necessarily the person with the longest technology list.
When you’re interviewing someone, try asking how a technical skill changed the way they worked, rather than simply whether they’ve used it.
For example:
“How has your development knowledge changed the way you approach testing?”
Or:
“Can you give me an example of something you were able to identify or troubleshoot because you understood how the application was built?”
Those answers can tell you considerably more than a list of frameworks on a resume.
Three questions a recruiter can use to vet modern QE talent
Finally, I asked Steven to give me two or three questions recruiters could ask to determine whether someone actually understands modern QE rather than simply recognizing testing terminology.
His questions were scenario-based.
1. You see a Playwright test failure. What could be causing it?
Imagine a Playwright test fails after a user submits an email and password on a login screen.
Steven’s expectation is that a strong QE won’t immediately assume the problem is with what they see in the UI.
They should consider possible root causes across different layers of the application whether its at the frontend, backend, or database level.
What is this question really testing?
Whether the candidate can think beyond the visible failure and investigate the system more broadly.
For recruiters, this is a good concept to look up: root cause analysis across application layers.
The goal is to understand that a UI failure can have many possible causes.
2. How would you use Claude to help investigate the failure?
Steven would expect the candidate to use the screenshot as part of a larger investigation rather than simply asking an AI tool to explain what it sees.
Ideally, they would provide the screenshot to Claude along with the relevant application code and use the AI to investigate possible causes. If they already had an educated hypothesis, they could also ask Claude to examine or challenge that hypothesis.
What is this question really testing?
Whether the candidate understands AI as an investigation and reasoning aid, rather than simply an answer generator.
A strong answer might involve providing relevant context, forming a hypothesis, asking the AI to investigate it, and then validating the result.
This is another area where recruiters can start looking up concepts such as AI-assisted debugging, context engineering, and human-in-the-loop AI.
3. At what point would you ask a developer for help, and why?
This was probably the most interesting of Steven’s three questions.
His answer wasn’t simply, “When something is too technical.”
He said he would expect a QE to ask for help when they no longer know enough to confidently review Claude’s explanation or suggested fix.
The exact point will be different for every candidate depending on their technical background.
For Steven, the question is less about measuring how much someone knows and more about integrity and judgment.
What is this question really testing?
Whether the candidate knows the limits of their own understanding. Remember, understanding what you don’t know is part of knowing what you’re doing.
That’s particularly important when AI is involved. A QE shouldn’t blindly accept an AI-generated explanation or fix simply because it sounds technically convincing. Knowing when to investigate further, when to validate an answer, and when to bring in someone with deeper expertise is part of being a strong engineer.
The recruiter takeaway
The more I talk with experienced QEs, the more I think the challenge for recruiters isn’t learning every technology on the job description.
It’s learning what good technical thinking sounds like.
Modern QE can involve automation, development, AI, unit and integration testing, debugging, collaboration, and much more. But underneath all of those skills are questions about how someone thinks:
Can they investigate beyond the obvious?
Can they validate what AI gives them?
Can they understand how their testing fits into the application and development process?
Do they know when they have enough information to act and when they need help?
Those are the things I’m trying to get better at recognizing when I talk to QE candidates.
A big thank you to Steven for taking the time to answer all of these questions and, more importantly, for giving me a much better lens into what modern QE looks like in practice.