Years ago, I remember talking to a candidate about a .NET role. I explained the opportunity and his response was basically, “No, no. I’m a Java guy.”
At the time, that didn’t seem particularly unusual.
But I don’t think that mentality flies anymore.
Technology moves too fast. The framework you’re an expert in today might not even be the framework your team is using next year. AI is changing how engineers work. New testing frameworks are emerging. The way we build, test, automate, and deploy software continues to evolve.
So when I’m recruiting today, I’m less interested in finding someone who can tell me, “I’m a Playwright expert.”
I want to know what they’ve actually done with it.
And more importantly, I want to know what happens when the technology changes.
That’s something I’ve been thinking about a lot lately because I’ve watched a lot of recruiters, myself included, struggle with the same problem: How do you actually vet a technical requirement when you aren’t the engineer?
You can read the job description. You can search for the right keywords. You can ask someone if they have experience with Playwright, Selenium, CI/CD, AI, etc.
But that doesn’t necessarily tell you whether someone is actually good at the job.
So I took a role I recently filled and went to one of the engineering managers, Ryan.
I basically said, “If you were recruiting for this role, what would you ask the client? And what would you ask the candidate to figure out if they actually know their shit?”
The role I used as an example was: Looking for a Senior QE to join a team that’s migrating from Selenium to Playwright, modernizing their automation/CI pipeline, and starting to adopt AI for test development using Cursor and Claude.
But the bigger requirement was really the ability to keep learning and adapt as the technology changes.
Because you can find me a Senior QE who knows Selenium inside and out. You can find me someone who knows Playwright really well. You can find someone who has “AI experience” on their resume.
But can they actually navigate change?
That’s a key component to many job requirements.
And my engineering manager’s answers were interesting because his questions weren’t just about whether someone knew the technology. They were about what the organization was actually trying to accomplish, how the team approaches change, and whether the candidate has actually done the work.
So here’s what he’d want a recruiter to ask the hiring manager first.
1. What AI models and tools are currently being used by the company, team, or enterprise?
This sounds like a basic question, but it tells you a lot.
If the answer is Cursor and Claude, I want to know what they’re actually being used for. Are engineers using them to write tests? Generate test cases? Debug failures? Refactor automation? Or does “we use AI” mean someone occasionally opens ChatGPT?
The goal isn’t just to find an organization that says it’s adopting AI. You want to understand how seriously they’re actually using it.
2. How much is actually funded for these tools?
This was one of the questions that made me think a little differently about recruiting.
There’s a big difference between a company saying, “We want our engineers to use AI,” and a company that’s actually willing to spend money on tools that can improve productivity and growth.
If an organization isn’t willing to invest in the technology, that’s important information for the recruiter.
Because it changes what you’re sourcing for.
3. What metrics are you using to determine whether you’re getting the desired result?
This gets back to something I hear a lot in engineering: What are you actually trying to solve?
If a team is migrating from Selenium to Playwright, why?
Is it faster execution? Better reliability? Easier maintenance? More coverage? Less time spent troubleshooting flaky tests?
Same thing with AI.
If you’re adopting Cursor or Claude, what does success look like?
If nobody can answer that, you may have a company that’s buying technology because they feel like they should be, rather than because they know what problem they’re trying to solve. And that answer will change your sourcing and recruitment strategy.
4. What’s the organization’s appetite for process vs. results vs. continuous improvement?
This one is important because not every company actually wants a forward-thinking engineer.
Some organizations want someone to come in, follow the existing process, and execute.
Others want someone to look at that process and say, “Why are we doing it this way? Is there a better way?”
Neither is necessarily wrong. But they’re very different profiles.
If you’re hiring a Senior QE to help migrate frameworks, modernize automation, and introduce AI into the testing process, I’d want to know whether the organization actually wants someone who will challenge the status quo.
5. Is the team encouraged to experiment and fail fast? If so, how?
This is probably one of my favorite questions because “we encourage experimentation” sounds great until you ask what that actually means.
Can engineers try a new tool? Can they test a new approach? Are they given time and budget to experiment? What happens when something doesn’t work?
Because if you’re looking for someone who’s constantly learning and adapting, you need an environment that actually allows them to do that.
And this is where I think recruiters can start to get much better at vetting technical roles.
You don’t have to be an engineer. You just have to understand what you’re actually hiring someone to do.
Once I understood the environment, I asked my engineering manager to flip the question around.
What would you ask the candidate?
And again, the questions weren’t simply about checking whether they knew the right technologies.
They were designed to uncover whether the person had actually used them, understood what they were solving, and was continuing to grow.
1. How often are you currently using AI, what models are you using, and what are you using them for?
This gets past the “AI” keyword on the resume.
If someone tells me they use Claude or Cursor, I want to know what they’re actually doing with them.
Because there’s a big difference between someone who uses AI to write a little boilerplate and someone who’s incorporating it into their test development, debugging, automation, or development workflow.
And there’s another thing to listen for here.
Did it actually work?
My engineering manager made the point that if someone tells you their experience using AI has worked perfectly, that’s probably a red flag. People actually using these tools know they have limitations. They’ve gotten bad outputs. They’ve had to troubleshoot. They’ve had to iterate. They’ve had to figure out where AI is useful and where it isn’t.
And that’s experience.
2. If you’re currently using AI, what gaps, issues, and successes have you had with it?
This is really a continuation of the first question, but now you’re asking them to get specific.
What went wrong? What did you have to fix? Where does AI fall short? What have you figured out that actually works?
Again, you’re looking for someone who is experimenting and learning, not someone who simply knows the names of the latest tools.
3. How do you keep up with the latest best practices in QA, AI, frameworks, and CI/CD?
This is probably one of the most important questions for this particular role.
Because if you’re migrating from Selenium to Playwright today, what happens when the next major shift comes along?
I don’t want the person who says, “I’m a Playwright person.”
I want the person who says, “Here’s how I keep up with what’s changing, here’s what I’ve learned recently, and here’s how I’ve applied it.”
That’s a much more durable skill.
4. What are you currently doing to continuously improve your skillset?
Again, I’m less interested in what someone learned five years ago and more interested in what they’re learning right now.
What are they experimenting with? What are they building? What are they reading? What are they trying to get better at?
Because the technology isn’t going to slow down for us.
5. How do you know when you’re done testing, automating, or implementing something like a CI/CD pipeline?
This one is good because it gets beyond knowing how to use a tool.
It’s about knowing when the job is actually done.
A good QE isn’t just executing tests or writing automation because the task says to. They’re thinking about the outcome.
Is the automation stable? Is it giving us useful information? Is the pipeline actually improving the development process? Are we solving the original problem?
And that’s where we started getting into some more technical questions I could actually learn from as a recruiter.
For example, my engineering manager pointed me toward locator stability when talking about Playwright.
Instead of simply asking, “Do you have Playwright experience?” a recruiter could ask something like, “How would you approach locating elements in Playwright without relying on XPath?”
I don’t necessarily need to know every possible answer. But I can learn enough to understand that there are more stable approaches and then listen to whether the candidate can explain what they’ve actually done.
We also talked about automation flow control.
Imagine a page where the header, sidebar, table, and other components all load at different times. How do you make sure your automation isn’t flaky because you’re assuming everything loads in the same sequence every time?
Again, I don’t need to become a QE.
I need to understand enough to ask a question that forces the candidate to show me how they think. And I think that’s the lesson for recruiters.
The old recruiting model was pretty simple.
Java? Check.
Selenium? Check.
Playwright? Check.
CI/CD? Check.
AI? Check.
But technology is moving too quickly for that to be enough.
The question isn’t just “What do you know?”
It’s “What have you actually done? What are you solving? What happens when it doesn’t work? How do you learn? And what are you learning next?”
Because I don’t think the best engineer for tomorrow is necessarily the person who’s an expert in today’s technology.
I’d rather find the person who says:
“I’ve never done that before, but here’s how I’d figure it out.”
That’s the mindset I’m looking for now.