Stressful, inconsistent, and opaque. These are all words used to describe most engineering interviews. At Magoosh, we want to provide a better experience to engineering candidates, so we’re striving to create a more transparent process from start to finish. As part of that effort, I’d like to share some behind-the-scene details about one important stage of our interview process: the engineering exercise.
Below is an overview our our general hiring process for any position. (You can also read about it in more detail in our full Magoosh hiring process blog post.)
- Application review
- Questionnaire
- Phone interview
- Reference and background checks
- In-person interview
The Engineering Exercise comes at the Questionnaire stage. We’ve sought to create an exercise that’s both representative of the skills a candidate would use at Magoosh and helpful to us in assessing the candidate’s experience.
The questions
We ask candidates to complete two questions: a programming question and an architecture/design question. For the programming question, we give the candidate a file and ask them to write code that is ready for production and code review. For the architecture question, we describe a challenge that we’ve faced and ask the candidate to share how they would approach that challenge.
The first iteration of the exercise was far from perfect—we received a number of questions, and we’d see submissions we didn’t expect. It didn’t take us long to realize our instructions left something to be desired. Over time, we’ve taken candidate feedback and edited our instructions to make the exercise more clear. For example, we previously didn’t explain that we wanted production-ready code, so some candidates solved the programming question by writing a quickly thrown-together script. We plan to continue improving the exercise as we get more feedback.
Timing
Originally, we estimated both questions would take a total of three hours to complete, so we asked candidates to set aside a three-hour window to complete the exercise. We received feedback both from candidates and from people we had hired that the short window created unrealistic stress, which is not what we wanted. Based on that feedback, we now ask candidates to complete the exercise within 24 hours. That way, all candidates have the same amount of time but there isn’t an unrealistic time pressure to complete the exercise exactly within three hours.
Grading
Our most senior engineer, Zack Mayeda, grades the engineering exercises. With the help of others, he’s created a rubric and grades against the rubric so that each exercise is graded on the same criteria. Our hiring team also anonymizes as much of the candidate’s information as possible before handing the exercise off to him, so he can evaluate the work with as little bias as possible.
Testing our test
When designing our exercise, we knew it would be impossible to create a process that was 100% perfect, but we still wanted to run a test to see if it was at least directionally accurate (one of our core values is Data > Intuition). So, I asked two talented senior engineers from another company to complete the exercise and then put their submissions (stripped of their names and information) into the queue along with other candidates’ submissions. If those engineers scored poorly, then we could be fairly certain that there might be something wrong with our exercise or rubric. Fortunately, they both scored very well, which gave us some confidence that our exercise was a useful competency test. (As an aside, Zack was very excited about the candidates, until I told him that they weren’t actually applying…sorry, Zack!) All this said, we still acknowledge that our exercise isn’t perfect and will probably generate some false negatives and false positives. Again, we’re dedicated to continuing to improve the process based on feedback we receive.
Feedback
If a candidate does well enough on the exercise, we move them forward to the Phone stage of the interview process. During the phone interview we may ask questions about why they made certain decisions. If a candidate doesn’t do well on the exercise and doesn’t make it to the phone interview stage, we’ll let them know we’re pursuing other candidates and give them the option to receive feedback on their work. We know feedback makes us all better, so we offer it to all candidates—not just those who make it all the way through to the end of the interview process.
tl;dr
-
- At Magoosh, we believe your Engineering exercise should be representative of the work you’d do at Magoosh
- Our exercise assesses multiple types of engineering work: coding + architecture
- We give everyone the same time constraints and try not to make those time constraints too stressful
- When grading the exercise, we focus on the work not the person
- We give feedback to anyone who wants it
- We’re constantly trying to improve our exercise
Interested in working at Magoosh? Check out our open positions.
If you don’t see an open position that’s perfect for you, please submit your resume via the general application.
Learn more about our philosophy and culture here.
Leave a Reply