A guide to an inclusive tech interview

The problem with tech interviews today

Let’s be honest. Interviews really suck. They make you anxious and sweaty and a little obsessive. For most people, nothing about walking into a meeting knowing the sole purpose of it is to assess you is a comfortable feeling. It’s worse if you’re broke and really need the job. It’s worse if you’re black, or disabled, or maybe a hijab-wearing Muslim woman. The anxiety is worsened by thoughts of bias, racism, and sexism.

And if you’re a developer, interviews can be Dante’s tenth circle of hell. And I’m not the first to say it; we’ve known for a while just how broken technical interviews can be.

They often involve deeply academic and theoretical algorithmic questions that aren’t relevant to a typical developer’s day-to-day. They often involve white-boarding and thinking on your feet with an audience, which isn’t the most comfortable environment. They often don’t consider skills that can make or break a developer, like clarifying assumptions and edge cases, or resourcefulness, pattern-matching, and the grit and ability to scour 3rd party docs for an answer to your question. Instead, technical interviews often come down to impressive array and string manipulations. But our day-to-day job as developers is so much more than that. It’s our ability to communicate across disciplines, to design systems that interact seamlessly with one another, to understand risks and mitigate them. These are things you cannot assess by asking someone to invert a binary tree on a whiteboard.

Our interviews need an overhaul.

At my last company, I led an effort to overhaul the interview process after consistently feeling like we were saying no to great candidates. I created a process that was objective and fair, promoted diversity and inclusion, and helped us identify our biases. Here are the steps I took to revamp our interviewing:

Step 1: Gather data

If you’re reading this because you think your interview process sucks and you want to make it better, I think you’re awesome. I salute you virtually and the best advice I can give you is to gather data. It’s the first step towards getting buy-in from your team: proving that there is a problem.

Ask yourself: how many marginalized candidates does your organization interview? Are hiring decisions recorded? Who reviews them? How diverse is your interviewing panel? Do you ask candidates for feedback after interviews? Do your interviewers often disagree strongly on hiring decisions? How rigid is the process: does it allow for different personality types?

When it comes down to it, the question you are really aiming to answer is: who aces your interview? And the work you are really aiming to accomplish is to eliminate all aspects of that answer that are not relevant to your job’s requirements.

Step 2: Start a rubric

Candidate rubrics play a crucial role in interview processes. I consider them one of the key building blocks to developing an inclusive tech interview.

There’s a strong case for the rubric:

• Rubrics promote alignment within your departments; they help you have a shared understanding of the criteria you require and the decisions about candidates that you make.
• Rubrics encourage you to be conscious of bias and can safeguard marginalized people from unconscious bias and social barriers. They can help reduce subjective assessments by defining objective criteria.
• Being able to quickly look over a rubric before an interview and after it promotes more productive discussions. It gives your team the language to identify the areas the candidate performed well or poorly in, which in turn helps you make decisions faster and more cohesively.
• Rubrics make it easier to onboard new interviewers into your interviewing process.
• They help your talent team understand how to recruit for your department and what specific qualities you are looking for in a technical role.

Of course, rubrics are always a work in progress; they should evolve as the industry does, as your team scales, and as new technical roles emerge in your organization.

Step 3: Define what you should assess

Every organization’s interview rubric should be different; what you assess for should be tied strongly to your org’s values and mission and should be influenced heavily by the developers on your team. Knowing this, I can offer a general list of categories to start with that have served me well so far:

Code fluency
The degree to which they can comfortably and efficiently read, understand, and write code.

Problem solving
Their capacity to translate a problem statement into a technical solution. How well can they troubleshoot issues? How well can they identify and tackle edge cases?

Systems design
How well can they define the architecture of their solution? Are there clear interfaces between the various components? Do they understand how to divide responsibilities within their system?

Growth vs fixed mindset
Have they demonstrated an openness and eagerness to learn? Do they display signs of a growth mindset?

Resourcefulness
Do they know the right questions to ask and can they easily find answers to their questions?

Self-reflection & awareness
Are they aware of their knowledge gaps? Do they know what they’re good at?

Collaboration & communication
Do they welcome feedback and collaboration? Are they able to clearly explain their approach and why they picked it?

Culture add
Do they bring a unique perspective, identity, or interest to the team that it otherwise does not have?

Step 4: Define what you shouldn’t assess

There are several qualities you might like to see in candidates that are not in fact requirements to succeed as part of your development organization. Assessing for these qualities can often unfairly filter out really great candidates. While these items might help you form a “yes” for a candidate — by providing more information on categories you do assess for — they are not items you should require; therefore, they should never help you form a “no”.

Of course, there are always exceptions but leaving out these items has proven to be a great rule of thumb for me:

Educational background

Developers can come from all walks of professional experience and educational backgrounds. Most development jobs shouldn’t require CS degrees or a certain level of education or GPA. Instead, they should require a certain level of technical skill and experience and we should recognize that there are different avenues of obtaining them.

Side hustle & open-source projects

Side projects are great. A candidate’s personal Github can often offer more information and insight into their technical aptitude, coding style, and interests. Expecting side projects, however, as a requirement or penalizing candidates for not having them is a harmful mistake.

Assessing someone’s passion for technology or their technical aptitude itself through side projects favours single men and disadvantages women, who are statistically the primary caregiver at home (Read more here).

Beyond that, there is nothing wrong with being a developer who doesn’t code outside of work (Read more here). What’s your opinion on a carpenter who doesn’t build houses in their free time? Why is code any different? We spend a significant amount of time at work; work experience can and does demonstrate a sufficient capacity for learning, self-improvement, and technical growth.

Culture fit

• We shouldn’t be looking for candidates who will fit in. Fitting in is not a requirement to succeed at coding; many of us did not fit in at first when we joined our teams. I know I certainly didn’t. As your team grows, so too will your culture!
• Don’t ask yourself “Do I want to grab a beer with this person?” or “Would I want to be stuck in an airport with this person?”. Ask yourself, “Can I write good code with this person?”
• You shouldn’t be assessing your capacity to be friends with a candidate because statistically we gravitate towards people who are like us when choosing our friends. This concept is called homophily and is a common interview bias (read more here and here). When we answer the question of “Do I want to work with this person?” with friendship in mind, we introduce unconscious bias to the interview.
• This is where the Rubric comes in; it’ll help you assess for a fixed criteria as opposed to a gut feeling or your perceived sense of likability.

Technical trivia & gotchas
While we must assess for technical skills given that these are technical roles, we don’t assess for obscure technical or algorithmic trivia. For example,

• A candidate doesn’t need to know every data structure under the sun to succeed, but they should know some common ones and how to pick them.
• They don’t need to know the right algorithm to invert a binary tree, but they should know that tree structures can be used when your data is organized hierarchically.
• They don’t need to know Big O notation or time space complexity jargon, but they should understand performance issues and optimization techniques. Assessing for academic language favours new grads and individuals with a CS educational background.
• A good guiding question to ask: “Do they need this piece of information or knowledge to be successful at this job day-to-day? And is it something they can learn or look up easily on the job?”

Product domain knowledge

We don’t expect developers to understand perfectly the industry domain they operate in. For e.g., a developer being interviewed to work on a fintech application should not be penalized if she lacks a financial background or an an understanding of accounting principles. It should be the product team’s responsibility to help the development team understand the use cases and domain they operate in.

Step 5: come up with technical questions

It’s important to complete defining your rubric before you come up with your technical questions; the rubric should directly inform the questions.

Here’s a healthy list of guidelines for a fair and inclusive technical question:

• It can be solved in any programming language.
• It can be completed in 1–2 hours.
• It represents a real-world problem the candidate might tackle on the job.
• The candidate can google, use 3rd party libraries, etc… to solve it.
• It can be completed on the candidate’s own laptop (if desired).
• It does not involve a necessary white-boarding component.
• It has edge cases to consider.
• It consists of a few technical decisions to discuss.

Since a significant portion of our job as developers is spent reviewing code, I am partial to code review questions as they can test code fluency, debugging, experience, and mentorship skills all at once. They are especially helpful when assessing senior and lead candidates. Try to keep the “bugs” in the code language-agnostic — think of common issues you comment on day-to-day as a reviewer and recreate those in your question.

Another common question type is to ask the candidate to build an app in the language and environment of their choice that does one thing. This could be an inventory management application, a health stats tracker, a blind date matcher, etc…

If your organization hires developers with experience in the particular languages or technologies it employs, then another option is to give the user an app (in that language) with a feature removed and then ask them to add it.

Step 6: Establish interview etiquette

There is so much you can do as an interviewer to make an interview a safe and positive experience for a candidate. From my experience, here’s a list of things I like to keep in mind when interviewing:

💟 Ask the candidate if they’d like a bio break or to grab a glass of water or coffee before you begin the interview.

💟 Begin by introducing yourself, mentioning your pronouns (read more here), what team you’re on, what you do, and what technologies you mostly work with. If you’d like, you can include a personal detail to help create a friendly atmosphere. I like to mention that I’m a Lord of the Rings fan :)

💟 Try not to bring your laptop to the interview. Take notes on paper instead. If you must bring a laptop, be sure to let the candidate know you are taking notes when you’re typing and that you are still paying attention to them. Don’t have Slack open or any other online application.

💟 Remember to practice empathy. Be mindful of the position of power you are in as an interviewer. Interviews are a high-stress and intimidating environment for many; your goal as an interviewer is not to interrogate the candidate or try to push them until they fail.

• As an exercise, recall the worst interview you’ve ever experienced.
• Now recall the last time someone went out of their way to make you feel comfortable, calm, and heard in an interview. Aspire to create that feeling for every candidate you interact with.

💟 Familiarize yourself with legal individual code grounds; these are things you should never ask about in an interview: age, citizenship, race, creed/religion, disability, family status, marital status, record of offences, sex (and pregnancy), sexual orientation. (Learn more from the Ontario Human Rights Commission here)

💟 Read the candidate’s resume before the interview. This indicates during the interview that you are interested in the candidate and eager to learn more about them.

💟 Meet with your co-interviewer(s) before the interview for a quick prep session. During it, check that you have both read the candidate’s resume, read through the rubric quickly, decide on what areas of it you might like to dig deeper into during the interview, and go through the technical questions you will be asking.

💟 Be mindful of time during the interview; don’t be afraid to move it along if a candidate is rambling or stuck on a question. You can simply say “great answer, thanks!” and move on to the next item. It is very important to leave enough time to answer questions the candidate might have for you.

💟 After you present a technical problem to the candidate, make an effort to communicate flexibility. Tell them they can take a few minutes to think to themselves if they’d like to and emphasize the tools available to them. For example: they can google, they can use 3rd party libraries, they can ask you questions, they can visualize the problem on paper or the whiteboard, etc… A good way to consider this is to give them all the tools or as many of them as you can that would be available to them if they were solving this problem on the job and not in an interview setting.

Step 7: Learn the common interview biases

The list of potential biases is endless, but there are a few ones that are particularly relevant to interviewing:

Gender bias

Women are often assessed against a higher standard of qualifications or quick to be labelled “junior”, while men are assessed for potential. (Read more here and here)

Extrovert bias

We often equate extrovertedness to confidence and a passion for technology. Quiet, methodical personalities are often considered “not passionate” despite significant work experience and interest in technology.

Homophily

Homophily (literally “love of sameness”) is the tendency for us to bond and associate with those similar to us. It comes up in interview settings when we are interviewing someone we acknowledge is very different than us.

Anti-black and racial bias

This kind of bias can start as early as reading the candidate’s name. In a study done by the American Economic Association titled “Are Emily and Greg More Employable Than Lakisha and Jamal?”, employers selected candidates with names like Emily Walsh and Greg Baker for callbacks almost 50 percent more often than candidates with names like Lakisha Washington and Jamal Jones. Other ways this manifests is in our unconscious (and sometimes even conscious) reactions to individuals with accents, to different dialects, and to slang and AAVE.

Halo & Horn effect

The Halo & Horn effect is when our first impression of a candidate leads us to some unfair conclusions about them. For example, if we identify they are weak in one area, we might assume they are weak in all areas (Horn effect). And conversely, if we identify they excel in an area, we might assume they excel in all areas (Halo effect).

Step 8: Create a no-hire/hire decision protocol

Once you’ve established a rubric for your interviewing process that reflects the day-to-day roles and responsibilities in your organization, you’re ready to develop a decision protocol.

This means identifying, for each level or position, the basic minimum requirement to fulfill in the rubric. What makes you say no to a candidate? For example, it isn’t essential for an intern or a new grad to demonstrate experience with systems design. Other categories of the rubric, like their self-awareness and whether or not they have a growth mindset are more relevant to the role.

Do this for every position you are hiring for— before you open hiring. Identify the minimum requirements using the rubric. This becomes your no-hire/hire decision protocol.

After your interviewers have concluded an interview, they can refer to this protocol to decide whether or not they will vote 👍🏼 or 👎🏼.

Step 9: Set up an interview training workshop

Set up a presentation that covers common interview biases, the organization’s rubric (what you assess for, what you don’t assess for), the hiring decision protocol, and interview etiquette guidelines.

Invite interviewers and make sure your team is aligned on the interviewing process and understands the ways in which they can contribute to a fair and inclusive experience for candidates.

Step 10: Finally, track how well it’s going

Here’s a quick checklist I put together to help you think about how successful your interview process is:

✅ The efficacy of the interview process is regularly reviewed (at least quarterly) and any necessary changes are made.

✅ Key metrics are collected and reviewed by your recruiting team. These include: how many marginalized individuals applied to postings (broken down by minority), how many were invited to an interview, of those how many received offers, the ratio of interviews to successful hires. Any one of these metrics can signal an opportunity for improvement.

✅ Interviewers receive the organization’s rubric that clearly and concisely outlines what to assess per role.

✅ Interviewers receive interview training that covers the rubric, what you assess for, what you don’t assess for, how to assess, interview etiquette, and common biases to be aware of. This training happens regularly (at least quarterly) and is not a one-time session.

✅ The candidate receives a breakdown of what to expect before the interview. This includes the interview style and if possible, a sample problem, as well as what they’re being assessed for. As often as possible, the candidate is given choices to control the interview. For eg: programming language, communication style, etc…

✅ The interview panel is diverse. Every candidate interacts with at least one marginalized person and with people of varying titles and skill levels.

✅ The interview contains technical components that vary in communication style or the candidate is given a choice. E.g.s: white-boarding, solving a tech problem solo before explaining it, pair programming, take home task, etc…

✅ The interview includes discussion of the candidate’s current and past roles, projects, and what they’re passionate about. It does not consist of just technical coding questions.

✅ There is time allocated in the interview for the candidate to ask the interviewer(s) questions.

✅ The interview questions represent real problems the candidate can anticipate solving in their day-to-day work in the role they applied for. They vary in difficulty, can be solved in more than one way, and involve technical decisions that reflect the rubric.

✅ Interview candidates are provided with a feedback form or another means to provide feedback on the interview. They are specifically asked if the interview was inclusive and fair.

✅ Interviewers are provided with a feedback form or another means to provide feedback on the interview. They are specifically asked if the rubric is easy to understand and apply. Interview training depends heavily on this feedback.

✅ All interview candidates receive meaningful feedback after the interview and a date by which they can expect to hear a decision.

—

Thanks for reading this guide. I wrote it with the hope of making tech interviews just a little friendlier.

I welcome any feedback or inquiries at ✉️ sherine.soliman@gmail.com.

👩🏻‍💻‍Happy interviewing!

📝 Save this story in Journal.