How to Run Engineering Interviews
Especially in the age of LLM driven coding
I’ve decided that I’ve spent too much time being distracted staring at S&P 500 charts over the past day, so decided to write a bit about engineering interviews / screening in the age of AI.
First Principals Thinking Around Interviews
Interviews are just screening for someone you’re going to work with, so the best way to interview someone is to actually work with them for a bit. At Coinbase (before my time, sub 100 employees), they did work trials for everyone. A week of doing actual work as a 1099 contractor for the company. If you can afford it, this is by far the best interview process. It allows both the potential employee and employeer test what it’s like working at the company, and allows both to set a very high bar - likely the most transparent process you can run. I think up to a certain size every company should do this. Just give candidates a medium size task that is relatively arbitrary and have them present what they build. This also works well for many roles beyond engineering.
What if you don’t have a full week to interview someone?
Second best is a good test of skills, references, and some basic cultural screening to see if the person is a good fit. This post mostly focuses on the skills tests because I think this is what shifts the most for hiring. The thing is, AI changes the process.
For engineers in 2025 you want to test a few key technical skills:
Things AI can do well
Vibe coding
Generating most template code
Things AI can’t do well
Core debugging over very complex issues (critical technical thinking) - AI doesn’t solve this well still and is what most engineers do
System design and taste, especially with existing / legacy systems
Then you also want to test for non-technical soft skills
Ambition and drive
Is this person a self-starter?
Downside protection / reputation
The main key that’s different than before is that AI now writes most of the code, and engineers act as mostly supervisors only meant to intervene when something goes wrong.
Tech Screens
I think the best method is to break tech screens into three parts:
Vibe coding
Debugging
System design
For the vibe coding section, it’s fine to give an engineer anything that a tool like Devin could one-shot. Make it a todo app or something simple, just have the engineer show they can prompt and generate a massive amount of code, quickly. The test here is just are you capable with these tools and can you use them correctly.
Second is debugging. Most non-engineers expect engineers to spend most of their time writing code, but it’s actually quite the opposite. These days most engineers spend most of their time reading code and fixing issues. I think even before AI this was the least tested skill and one that is super critical to a productive engineer - can you take some unknown problem in a very complex codebase, debug the issue and fix it. AI can’t do this today, and it will likely be some time before it can, because it requires a lot of critical thinking and intuition to fix issues like this.
Third is system design, which is often around algorithmic efficiency (ie can you design a scalable or reliable system) and if the engineer has the ability to make reasonable tradeoffs given a current system architecture. For this I like to give them something existing that is poorly implemented (a high speed trading engine implemented in node js) and ask them how they would migrate / rearchitect. There is no real “pass or fail“ for this test, but you can learn a lot about how they think about problems like this.
Soft Skills
For this, if you don’t have a week to get lunch with someone every day, I recommend references, but not the ones they give you. Find people they’ve worked with before - especially try to find the sticky situations folks have been in, jobs where things didn’t work etc. Ask those people via linkedin about the person - it will become very clear if they’ve made enemies or burnt bridges or are hard to work with. It will also become very clear whether the person is ambitious and a self starter, people love to gas up their friends un prompted.
Conclusion
Interviewing in the age of AI isn’t a lot different than it used to be, everyone just moves faster and is more efficient, which actually means you can be more thorough and try to make more accurate hires rather than hiring en-mass. This is true both at startups and at large companies.