We don’t get to hire new teammates very often at Ei. And because we’re such a highly collaborative and self-managed team, it’s really important that we get it right. We spent the last six weeks hiring Ei’s new Operations Coordinator.

Like many of you who have hired someone in the last few years, we had to grapple with hiring in the age of AI, and everything that comes with it. Here are a few of the questions that arose:

• Knowing that many candidates will turn to AI for writing assistance, how can we engineer the job application to effectively differentiate candidates?

• Knowing that our new hire will need to marshal AI judiciously for some parts of their job, how can we evaluate their skill and judgment in using AI?

• Knowing that human bias can reduce equity in hiring processes, how might we employ AI to expose our biases?

I learned so much from this experience that I thought I’d write about it, in case something we tried can help you in your own hiring processes. Today, I’ll focus on the first question.

How we designed an AI-proof application

We knew that if we asked for a cover letter or a few open-ended responses, as we’ve done in the past, we’d get a bunch of AI-assisted answers that would not tell us much about the candidates.

Instead, we wanted an application that focused on tangible skills and showed us how the candidate’s organizational brain operates. This is an ops position, after all.

We realized that to achieve this, we needed to minimize writing (can be done by AI), explicitly measure key skills, and screen out bots. We decided to go with a three-part format: a bit of writing, several multiple choice questions, and a video.

Keep the written part brief, and don’t use it to judge writing

We asked for a resume or LinkedIn, and used it simply to screen for key experience requirements.

We asked one written question: “What makes you interested in serving as Operations Coordinator at Experience Institute? When you read the full job description, which parts of the role feel already in your sweet spot, and which skills would you be excited to build?”

There is no “right” answer for an AI to guess at in this question. So, even if our candidates used AI to help them write, their answer revealed a lot about them, their motivations, and their sweet spot. I did not penalize anyone for using AI to write, but I did advance a few people if their answer read as particularly human and passionate about the opportunity.

Use multiple choice questions to inquire about key skills

Our new Ops Coordinator will need to quickly onboard to several tech platforms, and use AI, so we wanted to know their starting point with each. For each tech platform, we asked a question with these options:

• No familiarity

• I’ve explored it some

• I’ve used it for work

• I’m a super user

I wondered if people would select “I’m a super user” for all platforms even if it wasn’t true, but guess what? Only one person out of 40 applicants appeared to do that! I think candidates realized they would have to back up their claim later in the application and in subsequent interviews, so they chose to answer honestly.

For AI skill level, here's a photo of what we asked:

Later on, in the first round interview, we followed up with a 'give me an example' question, which let us verify whether their self-description actually held up. I’d say it did about two thirds of the time.

Clear out the bots: Ask for a short video.

In our case, we asked candidates to take two minutes to walk us through a digital system they’ve organized. We wrote, “We're not looking for any rocket science here or for you to create anything new. It can be as simple as your own task list. We just want a glimpse into how your organizational/logistical brain operates. One take is great - no need to polish.”

The video was the single most useful artifact in determining whether someone should move to the next round. Plus, most people showed a system they’d built in one of the platforms listed in the previous section, so it was another way for us to check the accuracy of their self-rating.

Of the 40 applications we received, 13 advanced to first round interviews. That’s a pretty high ratio. One reason for this was a job description with very explicit skill and schedule requirements. But I also think we discouraged a large swath of less qualified folks from applying, simply by creating an application AI could not succeed at on its own.

In the age of AI, job candidates have to both be able to use AI and also show up as a human. As hiring organizations, you can make it easier for them to do just that.

The good news? We found so many phenomenal people through this hiring process, including the one we just hired! Drumroll… Please join us in welcoming our newest member of the team, Colleen Coughlin! We are so excited to get to work with her!

If you found this reflection helpful and you'd like to learn more about our hiring process, follow me on Linkedin where I'll share part two soon.