Rethinking how we evaluate engineers.
This is a recap of a conversation I caught in my Promptmates community that felt relevant enough to reshare.
The original question being discussed was around a pattern some hiring teams are noticing in interviews with AI engineers: candidates who appear highly polished and articulate, but give the impression they may be using AI tools or scripted responses during live interviews. While they can clearly describe frameworks and technologies, something feels “off” when the conversation shifts into depth or lived experience. The concern wasn’t about candidates being prepared it was more about how difficult it’s becoming to separate genuine experience from rehearsed or AI-assisted answers in real time interviews. Especially for a recruiter if they aren’t technical.
One of the main concerns raised was whether there are better ways to identify signal during interviews without making the process feel accusatory. For example, some suggested ideas like behavioral or technical signals (eye movement tracking or speech analysis), but there was hesitation around anything that might feel invasive or overly suspicious toward candidates.
The conversation also pointed out a specific pattern being observed: candidates often coming from recent master’s programs in AI-related fields, sometimes tied to top-tier companies in contractor roles rather than full-time positions. In interviews, they tend to use consistent buzzwords and can fluently describe technologies they’ve worked with, but may struggle when pressed beyond surface-level explanations.
What teams are trying
One response in the discussion shared a few practical approaches they’ve been using over the last 6–8 months while hiring AI engineers:
1. Shift interview focus from definitions to decision-making
Rather than asking managers to explain concepts like RAG, fine-tuning, or prompting, the emphasis is on why certain choices were made in real situations. The key observation is that many can recite textbook definitions, but tend to struggle when asked to justify real tradeoffs or explain their reasoning behind specific design decisions.
My personal opinion is let’s first shift the conversation from bodies and tech, to business impact. I said this last year, we will face challenges helping decision makers identify what is required, what their perception of this tech is, and how to best vet and control the process. Not including the massive shifts going on in our line of work. We are now seeing this impact.
2. Use structured, proctored technical assessments
They described using CodeSignal assessments that include system design, incident/failure handling, and core software development skills. The environment includes an IDE with LLM copilots, identity verification, and anti-cheating mechanisms. According to the discussion, this setup has helped reduce noise in the interview process and provided a stronger signal on actual ability, despite the cost.
LoopQA does a fantastic job of creating custom tech evals that assess the role they hire for. Me personally, I typically suggest scenerio based tech evals, none of that leetcode stuff.
3. Add additional verification layers through recruiting tools
They also mentioned using tools like Real Talent / Talent Matching (via Greenhouse) as another signal layer to help validate candidate consistency across profiles and interviews, especially when something feels misaligned.