Here’s a scenario every recruiter knows: you run a clean, professional interview loop. Everyone feels good about it. And at the end, your managers can’t decide or agree which candidate is the better fit and the decision comes down to….a feeling.
That’s what happened here. A friend of mine, an SDET with a strong track record, made it through an entire interview process for a role at a facilities management SaaS company, only to lose the offer to another candidate.
Let’s break down what happened, why it happened, and what we’d do differently.
The Role
The position was a mid-level SDET seat, paying around $130K/year. The job description listed a wide (and honestly unusual) set of tools. But here’s what the listing didn’t say, and what only came out during the interviews:
- The hiring manager had already built the “bones” of the mobile automation framework. They weren’t looking for an architect: they needed someone to maintain and scale it.
- The hire would occasionally need to contribute to the team’s Playwright suite for the web app.
Strip away the jargon, and the actual ask was simple: someone who could write solid automated test code and work independently. Not a mentor. Not a strategist. A steady, technically sharp mid-level engineer.
The Interview Process
To the hiring team’s credit, the loop was lean:
- Phone screen: comp expectations, logistics, general QA background
- Hiring manager conversation: past experience, problem-solving style
- Technical conversation: HM, a software engineer, and an SDET asking about testing philosophy and approach
Notice what’s missing? At no point did anyone ask a candidate to read, write, or debug actual code.
Why He Lost
He made it to the final decision, but lost to another candidate. When the hiring team gave feedback, it was: “the other candidate was slightly stronger.” But my friend was confused as to how they made that decision because he never completed a tech eval. So he asked the friend who’d referred him what really happened. The answer:
The other candidate was seen as more of a planner, someone who always had a game plan. My friend came across as more of an adapter, someone comfortable figuring things out as he went. For a role the team described as “ambiguous,” they leaned toward the planner.
That was how they made the decision. Not a code sample. Not a technical work product. A read on personality style, formed across a few conversations.
The Two Traps Recruiters Should Watch For
1. Vibes-based hiring: A conversational interview is genuinely useful for gauging communication and problem-solving style but it’s a poor substitute for verifying whether someone can actually do the job. Anyone reasonably fluent in the vocabulary of testing can talk convincingly about “increasing coverage” or “cutting CI runtime.” That fluency is easier to fake now than ever, especially with AI tools helping candidates prep talking points.
2. The years-of-experience trap. Two candidates can both show “6 years of experience” and have wildly different depth. Someone who spent two years each across three different stacks (building frameworks from scratch, working across CI/CD tools, adapting to new environments) has a very different skill profile than someone who spent six years doing similar work at one company. Tenure alone tells you almost nothing about density of learning.
What We’d Do Differently
After talking it through together, here’s the framework we came up with, one that gets real signal without turning the interview into a multi-hour take-home:
For tools like Appium (hard to test live remotely): Don’t ask candidates to write code from scratch in a sandbox environment that probably won’t work anyway. Instead, hand them code you’ve already written and verified, with a bug intentionally introduced. Ask them to find it, explain it, and suggest how they’d expand the framework from there (is it running in CI? is there reporting? multi-device coverage?). This reveals real fluency fast, with almost no setup overhead.
For mobile/web testing judgment: Give them a feature description and ask which scenarios they’d automate, which they’d test manually, and which layer (E2E vs. integration vs. unit) different test cases belong at. This shows you how they think about scaling test coverage, not just whether they know the syntax.
For CI/CD skills: Same read-and-critique approach: hand them a GitHub Actions config and have them spot the gaps or risks.
For AI fluency (increasingly relevant): Let the candidate use a tool like Copilot to generate a test for a demo feature, and watch how they prompt and validate the output. Do they blindly accept it? Do they know how to vet generated code critically? This is quickly becoming as important a skill as writing the code themselves.
The common thread across all of these: candidates don’t need to generate everything from scratch anymore. What matters is whether they can critically evaluate code their own, a teammate’s, or an AI’s. That’s the skill that predicts on-the-job success now.
The Takeaway
If, at the end of your interview loop, your hiring team can’t point to a specific rubric or work sample to justify their decision and it comes down to “who felt stronger”, the process wasn’t structured tightly enough from the start.
As recruiters, we own that structure. It’s on us to partner with hiring managers early, define what success actually looks like for the role, and build assessments that test for it directly. Otherwise we’re not hiring…we’re rolling the dice on $130K.
Big thanks to my friend Steven for being generous enough to share this experience. Losing an offer is never fun, but we broke it down together and turned it into a useful hiring lesson