By Shikhar Sachdev

Five years ago, I logged into my university’s portal one last time to submit a 2,000 word paper on Greek Philosophy.

These 2,000 words hadn’t come easy. They had cost me dearly the night before: a no-show at Pappy’s Taco Tuesday ($1 tacos, $2 beer). Thanks, Aristotle.

But now it was official. I had forever been released from the purgatory of homework hell.

The one task that causes headaches amongst 8 year olds and graduate students alike was no longer going to be a part of my life. Adios.

You might understand my shock, then, a couple of years later when after applying to a Software Engineering role I was emailed a special concoction called a “takehome assessment”. This concoction didn’t have cute little umbrellas in it.

Instead, it had a link to a Github repository. I would have the pleasure of spending 3 unpaid hours building a fullstack web application for the chance to progress to the next round. A familiar foe had reentered my life.

But what if I told you that after completing a couple of these takehome assessments, I realized that I actually preferred them to the more popular live coding interview style?

Before I tell you why, know that the business of takehome assessments is a polarizing one. Read a couple of Hacker News threads and you’d think they’re the plague. But others are fans.

To settle the score, I scoured the internet for over ~200 public takehome assessments. I compiled them in a spreadsheet and tracked technologies, grading criteria, and scope.

I spoke to anyone and everyone who had a stake in this business. This is what I learnt.

You can download the full spreadsheet here.

The problem engineers have with them

I’m 27 and single. There lies the biggest advantage of a takehome assessment for me: I get to spend more time on it than someone with a wife and two kids. And I get to be the annoying kid in class who does the extra credit.

These two advantages that I have are also the two most common complaints engineers have about takehomes:

1. They fail to define reasonable (if any, at all) time caps.
2. The requirements are open ended.

Take Basecamp. The takehome assessment that they designed for their Security & Infrastructure team was listed to take 3-5 hours, but the engineer they hired spent closer to 7-8 hours on it.

Alex Yang, founder of Tapioca, states that this is a common theme: “I’ve seen that candidates that do well on take-home assessments tend to take a lot longer, around 2x longer as a rule of thumb.”

Let’s take a look at our data and break down what timecaps look like for takehome assessments:

Designated Time

Out of the 203 assessments I inspected, a little less than half - 99 - specified some type of timecap. (note: a particular subset also specified days – I categorized them separately).

Let’s take a look at the distribution of designated time for the ones that did:

Approximately 76% of the assessments were scoped to be between 50 minutes to 4 hours, with a median value of 3 hours. So the middle approximation for how long you’re told to spend on your next takehome will be 3 hours.

Of course, that might mean spending 6 hours to be competitive. Take the messaging below as an example — when I read it, I get the memo that I’m going to have to grind.

But not every startup is secretly interested in figuring out how much free time you get in the evening. Ways to mitigate this include asking candidates to submit their git commit history so that they can see a chronology of tasks completed or asking them to keep a timesheet.

Requirements and Bonus Work

So far we’ve focused on time, but what about scope? Out of 203 assessments, ~130 clearly specified an acceptance criteria, i.e clearly outlining a list of features which if implemented would mean you were “done”. However, this was a bit subjective to judge.

Some of these product specs were very detailed – enough to even make the most diligent product managers blush. See Kin + Carta’s below:

Others, not so much:

Sometimes, though, the lack of requirements is intentional. Companies want to see what features you would prioritize in the absence of clear direction. As the Urban Dynamics team puts it:

Wyeworks takes it even a step further and asks the candidate to determine the scope of the project assigned and to provide a time estimate:

Talk about being in total control of your destiny.

One of the benefits of a take home is that it mirrors your day to day work (unlike a whiteboard interview). But if your competition is also evaluated in the same way, then it doesn’t really matter if you can show them that you can complete a day in their life.

You’re going to need to convince them that your day in the life is superior to all the other candidates. And that’s what turns the 3 hours into 6 hours.

“I, I get money, money I got (I, I get it)” - 50 Cent

Well, not for takehomes you don’t Curtis. Only 10% of companies listed that they would be compensating candidates for completing the assessments.

Some had an interesting structure: DataChef, for example, will donate 400 euros to a charity of your choice if you end up getting hired and 200 euros if you don’t get hired but fulfill most of their “requirements” (which, you know, is totally not ambiguous).

In 2021, the recruiting startup Coderbyte suggested some compensation rates based on the length of a takehome and the seniority level of the developer:

Other companies like Buffer and Basecamp contract developers out at their usual freelance rates for multi-day projects. Overall, though, getting paid for interview work is rare.

Many candidates view unpaid takehome assignments as fundamentally disrespectful. I think context is important.

If the very first thing in your application process is a 6 hour long unpaid assignment, then it’s hard to imagine you’re not wasting someone’s time because at this stage you’ve barely done any filtering.

But if the takehome is offered, say, after 3 rounds of interviews and will be discussed in detail during your final round with a panel of two senior engineers, I think that’s fair and provides a strong signal.

The opportunity companies see

An engineer’s time is expensive. Takehomes (i.e asynchronous technical assessments) can help save time because they are generally more efficient for teams to administer than live coding interviews, whilst also providing a higher signal (when done correctly).

A live coding interview usually takes up ~1 hour of an engineer’s time for every single candidate, plus any extra time they need to write their feedback.

But a take home that takes 3 hours to complete will rarely take 3 hours to grade. There will sometimes be a significant investment in actually generating the takehome, but when you weigh up the extra amount of information you get about a candidate compared to just a leetcode style live interview, it can make sense.

To better understand how these takehomes are generated, let’s first visualize the landscape of technical assessments:

The goal of asynchronous technical assessments is to mimic “on the job” skills. The generic style coding challenges (top left) that check for technical correctness can’t always verify soft skills. These challenges are often also quite generic and pulled from a library of already available questions that the platform offers. Instead of referring to them as takehomes, I’d call them coding tests or quizzes.

Regardless, they can be popular because they are an efficient way to get information from the candidate without wasting engineer time (automated grading and test-cases).

You often see these “top left” challenges at the beginning of a hiring funnel. It’s more common to send longer, more thorough assessments later down in the funnel when candidates are more invested and likely to follow through.

Companies like Tapioca and Woven (top right), on the other hand, specialize in building custom takehome assessments graded by humans. This manual grading process typically allows the company to get a better understanding of communication skills and how candidates assess tradeoffs. The vast majority of the ~200 assessments I compiled would fall under this section.

It’s not always binary though – both when it comes to designing these assessments and when it comes to putting them in practice.

Bumble, for instance, which does all of its iOS hiring through Tapioca, gives candidates a combination of both async and sync work. As Alex Yang describes it "So when they first meet with candidates, they spend around 15min or so as if they’re meeting with a product manager: deciding what you want to build. And then after that, they’re given an async assignment to work on on their own, and then when they submit the follow-up conversation which is live, you can discuss it. It’s an interesting combination where in real life as an engineer you’re not only working asynchronously. Or you’re not only working in real time.”

Problem generation

Oftentimes, a takehome assessment is formulated from a problem that an engineering team already faced in their day to day. I got the chance to speak to Adam Sah, an early employee at Google, who explained to me the advantages of this approach:

“It’s not always just out of laziness – a lot of times if it’s something we’ve already solved, it’s a lot faster for us to evaluate solutions rather than a fresh question.”

I came across an example of this with Rainbow Insurance who even put it as a FAQ within their challenge’s Github repo:

Adam Sah also mentioned that a good takehome assessment or question can be viewed as a competitive advantage to attract the best candidates (just like their larger job application process). People don’t want to reverse linked lists anymore.

As you can probably tell, coming up with a good takehome assessment question can be tricky.

Ken Schumacher, founder of Ropes, confirmed to me that “Creating a high-quality problem to serve to candidates is a deceptively tricky task; the problem needs the correct level of difficulty, the context needs to be limited, you probably need to support the problem in many languages, etc.”

Ropes aims to solve this problem. It leverages AI to generate custom problems specific to the industry that a business operates in. So a hedge fund is able to give candidates a graph theory problem about stock trading instead of some random question.

Even though the work is autograded, the level of insight you get from the candidate's solution is their big differentiator: their software relays the candidate's thinking patterns and aims to go beyond just tracking test cases.

Ropes might be categorized in the top left corner of our matrix, but in practice their technology enables them to function in the top right because of the level of insights they can give hiring managers.

Feedback is an (expensive) gift

The time a takehome assessment takes to grade can also come down to the level of feedback that a company provides. When Alex’s firm Tapioca focused on generating custom hour long takehome assessments for their clients, they promised personalized feedback for each applicant.

But this level of feedback, whilst a great gesture to the candidate, proved to be costly.

First, it is very time consuming to give personalized feedback. Adam Sah states that for many teams the goal from the hiring manager’s perspective is to give a quick green light or red light – you just want to be able to maximize signal and spending too long on this part doesn’t make sense for the hiring team. Alex and his team pivoted away from this model because it was operationally inefficient.

Second, whilst 95% of candidates might take feedback well (and really appreciate it because so many companies don’t give it), there is always going to be a small subset that don’t. At bigger companies especially, where the goal is to minimize risk, you don’t want to open yourself up to a potential lawsuit. So in some cases what a hiring manager might do is hop on a call with the candidate to give them feedback instead of putting it in writing.

The lack of feedback for the above reasons can be frustrating. I thought this frustration was aptly summed up in one of the Github repos that I came across:

“Yet another homework that took seconds for them to assign, hours for me to write, and seconds for them to decide somehow wasn't good enough” - Github user mchannon

***

One final thing.

The majority of the companies I noticed issuing takehomes weren’t your star companies. They weren’t FAANG. They weren’t looking to change the world. They weren’t looking for olympiad champions or kumon alumnis.

Rather, they were the design studios, the lean web development agencies, the mobile dev teams based out of Eastern Europe.

Teams that need builders, not algorithm monkeys.

Over and out - Shikhar

You might also enjoy reading:

• The Underground Economy of Glassdoor Reviews
• The Online Maze of Job Applications
• How I Got My Job Offer Revoked
• I Ran A Background Check On A Startup