Common patterns in technical interviewing

Introduction

Interviewing, both as an interviewee and interviewer, is part of everyday life in the tech industry. Median tenures are 2-4 years, it multiple interview loops to switch, so I estimate that at any given time at least half of all tech workers are doing one or the other. On the hiring side, typical hiring funnel conversion rates are 5-20%. (Of course, on average, the two sides' conversion rate must be exactly equal.) Even when not looking for a job, I believe in ABI - Always Be Interviewing; to be aware of opportunities, market rates and to stay sharp.

Interviewing is also a great learning opportunity: companies tend to compress a lot of what they think is important into interview loops. "They asked X, which I don't know much about; is this a blindspot for me? Why are we not doing X at my current job?"

As a hiring manager, I conduct about 5-10 interviews per month, and I have been doing so for the past 10+ years, so I think about these topics a lot. Although, per the above, this does not make special.

Here, I will attempt to enumerate all the categories of questions commonly asked in technical interview loops, and my experience with them.

Experience

A typical experience question is: "Walk me through a project you worked, what you did specifically on the project, what the end-result and impact of your work was." This is a great way for the candidate to showcase their past work and explain how they went over and beyond their roles to accomplish something awesome. However, there are many problems with experience-based questions:

1. Unfortunately, there is no way to verify what is being said. Even honest people can have selective memory and will bias the story in their own favor in an interview situation. It's also worth realizing that for many people, a job like Software Engineer at Google (in Silicon Valley or London) is a life-changing opportunity; life-changing not just for them, but for their family.
2. Candidates coming from lower tier companies (or coming from countries with a lower presence of tech companies) may have the right skills for the job, but may not have sexy projects under their belt to talk about. But experience questions discriminate against them, and increase pressure to inflate, which can then backfire.

Because of the above reasons, I don't ask any experience-based questions on my interview loops.

Baseline knowledge

The idea here is to test the candidate baseline knowledge on a topic. For example, if the candidate has Python on their CV and the role involves Python programming, implementing 2-3 line Python functions using if/elif/else, return, yield, for loops and basic Python functions like len() or range() demonstrates baseline Python knowledge. A non-coding example is asking the candidate to explain what the difference between pass-by-value and pass-by-reference is, and when does each happen in Python.

Although baseline knowledge questions seem too simple to be useful, they are not. The trick is to ask a lot of them, and cover breadth. I find that the right number of baseline knowledge questions give the strongest signal in technical interview loops.

Deep knowledge

Asking deep knowledge questions makes sense for more senior roles. An example coding task would be to implement a disk-persistent key-value store that support get(), set() and iteration, but is faster than linear. A non-coding example is asking about less common edge cases, tricky decorators, type hints, how Python runtimes differ.

It's worth noting that a senior candidate can be a Senior Generalist (has worked in many environment, can get up to speed quickly, can solve a wide variety of problems) vs Senior Specialist (has been coding in the same environment for 10 years, knows it inside-out). In my experience, big tech companies prefer to hire and grow Senior Specialists, since at 10k+ headcount and deep budgets they can afford to hire the best domain experts, there's less need for Senior Generalists.

As an example, I am a Senior Generalist: I have worked as a C++ programmer, as a Data Engineer, as a Data Scientist, I have managed all 3; but big tech doesn't care. If I interview for a Data Engineering role, I don't get any extra points for the other stuff.

Memory recall

These kind of questions ask the candidate to recall something that can be looked up. For example, a problem is given which can obviously be reduced to finding the shortest path in a directed weighted graph. Now, the question is whether the candidate can recall eg. Floyd's algorithm (and implement it).

I find these questions completely useless. Also, these questions heavily bias towards (i) candidates who have recently gone to school, and/or (ii) candidates who invest time to study for the interview loop. In my opinion, studying for the interview loop is not a problem, as it shows dedication. But the question is not meant to test (i) and (ii), it is meant to evaluate technical competence, which it does not.

Tricky question

Big tech companies have at various times used tricky questions in interview loops. A famous example is this Amazon question: There are two poles 50 meters apart, 30 meter tall. From the tops of the poles hangs a 40 meter rope between the poles. The lowest point of the rope is 10 meters from ground. How far are the poles? The trick is to realize that if the rope is 10 meters from the ground, and the poles are 30 meter high, then it takes just 2x20m=40m to get to the lowest point. Since the rope is 40 meters long, there is no play in the rope at all, it must go straight down, so the 2 poles must be right next to each other, 0 meters apart.

At Facebook, there are a lot of tricky coding questions that are variations of the Knapsack problem. If you know that Facebook interviews often have a question that is a knapsack, and do a few practice examples, it's easier to spot and solve; even though it's irrelevant to actual coding at Facebook.

I find these questions completely useless. Most candidates are nervous and stressed during the interview, potentially with high blood pressure, which impairs their cognitive abilities. Any questions which assumes a clear head will get low performance from people who tend to worry, even though this is unrelated to on-the-job performance (where people in general are not nervous and stressed).

Time challenge

When I was working at Facebook, one summer I and another senior engineer was tasked with building a simple interview loop for Engineering interns. We made the loop a coding loop, where they have to live code relatively simple Python function (2-5 lines each). There were hundreds of applications, which the recruiters narrowed down to 20. So we came up with a list of 10 coding questions for a 60 minute interview. We felt 10 questions is too many, we didn't expect candidates to get to more than 4-5 in the time allotted, but we wanted to make sure we don't run out of questions.

In the end there were multiple candidates who were easily able to complete all 10 questions within the 60 minute limit. It turns out that many of the intern candidates were competitive coders, so they were just able to write out the solutions, without having to think much about it. We the interviewers, although we were senior engineers, have never done competitive coding, so this was a surprise. The fastest candidate was able to solve all 10 in just 25 minutes. I remember thinking there is no way I could solve all 10 in 25 minutes...

In the end, although this was not the plan from the beginning, we ended up ranking them based on completion time, and hired the 2 candidates (fortunately it was a male and a female in the top 2 spots) who got all 10 right and were the fastest. (In reality there were 2 interviews, one for Python and one for SQL, and we combined the rankings.)

I would never do this on purpose (and have never repeated it), since in real life there is no time pressure when doing technical work.

Explicit pressure

In this scenario, explicit pressure is put on the candidate throughout the interview. Note that in the previous scenario, we evaluated candidates based on time, but we didn't put pressure on them during the interview.

Pressure can be created by nudging candidates to move fast(er), or by challenging their responses. When I was interviewing DE and DS candidates at Facebook, I had to follow a script, and as an interviewer I was expected to get throught the entire script in 60 minutes (even if it meant not getting a response for a part). The script, counting sub-items, consisted of 15-20 points. So as part of interview calibration, I was taught that if candidates are slow to get to a milestone in a question by a certain time, they have to be nudged, and eventually helped or somehow pushed along, since the entire script must be finished in the time alloted. The rationale was that different parts of the interview gave us signals on different attributes, and the hiring committee needs all these to make the call. If an interviewer didn't get through the script and didn't get all the signals, sometimes an additional (8th or 9th) interview would be scheduled with the candidate.

Pressure naturally occurs with other type of questions: deep knowledge questions (cannot remember decorator syntax), memory recall questions (cannot remember algorithm) and trick questions (cannot figure it out). For reasons given before, I don't use (or like) this pattern, in fact I do the opposite: I try to minimize pressure on the candidate as much as possible to get a realistic reading.

Bug/feature on existing code

Sometimes companies show the candidate a piece of code that is very close to what they would work on on the job, eg. a piece of Django code for a Python backend position at a company that uses Django. In the real world, most often you are not writing brand new code, but instead modifying existing code. So in this scenario you have to take an existing piece of code, which may have multiple bugs/issues to fix, and/or add new features to it. Although I don't use this type of question, I think this is a good way to gauge candidates for a variety of factors.

An interesting interview detail is whether the candidate is allowed to use Google (this appies to straighforward coding questions too). In my experience, most interview loops ask candidates not to use Google, and instead ask them to ask the interviewer if they have a question. In recent times I have experimented with asking candidates to share their screen when coding, and telling them it's okay to google — after all, we all constantly do it on the job. Interestingly, I found that this gives me valuable signal, but most often it's not in the candidate's favour: seeing how they google something (and chrome/google revealing past queries) communicates a lot. For example, does somebody google [how do I keep the first n characters of a string in python] or [python string prefix]. (To my surprise, many technical people don't know how to google effectively.)

Situational

Situational questions are no-code versions of the previous type: "It's Thursday 7p, you get an alert that the number of logins per minute on your SaaS has dropped significantly. What do you do?"

Situational questions are excellent for eg. senior backend, SRE or production engineer roles. Surprisingly, these questions are also great on Product Manager loops. The tricky thing is to get the amount and direction, guidance and nudging right. The candidate may not know what the interviewer is looking, and needs help to talk about the right topic at the right depth. For example, does the interviewer care in depth how the candidate sets up the crisis response team on Slack? or, is the interviewer want the candidate to talk more about which logs to check on the servers? A candidate may have the right depth in the desired portion, but without good prompting, she may misjudge the interviewer and contentrate on another portion.

Homework

This interview pattern is to ask the candidate to solve a 2-4 hour problem at home. I have been involved (on both sides) in multiple interview loops of this kind. On the hiring side, in all cases we eventually stopped giving homework assignments, because we had too many senior candidates who just didn't want to invest the time. Usually they didn't say so, they just never sent a solution, they silently churned.

The problem is, a good candidate, the kind you want to hire, will never spend 2-4 hours on a 2-4 hour problem. They will go overboard and will (want to) spend 20-40 hours on it. However, senior candidates will have a demanding full-time job and potentially family, so finding 20-40 hours over a week's period is tough.

Work with us for real

This is the best way to gauge a candidate: just have them come to the (remote) office, and work with us for a week as part of the interview loop. This sounds crazy, but at one startup I worked at, we did this for many years. That's also how I got the job, by working at the company for a full week, and presenting what I did Friday afternoon. I was coming off my previous failed startup and was unemployed, so I had the time. Technically, the candidate got payed for the week a competitive salary as a contractor. It goes without saying that this gives the maximum amount of accurate signal on the person, since the future team gets to see the full package, from the time the person arrives at the office, coding, having lunch, presenting, everything. The co-founders of the startup believed strongly in this system, but as the company matured it was first reduced to 3 days, then 1 day, then totally abandoned because of legal risks (and because many senior candidates were not willing to take a week off).

Conclusion

The goal in this first article on interviewing was to lay out (most of) the interview types for technical interviewing. I gave my own opinion, but I'm not looking to convince anyone. My opinion is based on what I've seen work, and what I feel is fair when I'm getting interviewed. Different roles at different companies require different interview loops.