Work Simulations: How to Assess Candidates Using Data

Joe Caccavale

11

May

2021

6

min read

|

X
Free Training: How to De-bias Your Hiring
Learn how to build a more ethical, empirical hiring process in a single session.

X

Sign up to our newsletter

Want to see more content like this, every other week?
We're on a mission to change recruitment.

Sign Me Up

What are work simulations?

Work simulations (also known as work simulations) are a form of assessment designed to replicate parts of a role.

They focus on the skills required for the job by asking candidates to essentially perform tasks that they would encounter should they get the job.

Why use work simulations?

Traditional hiring methods are biased and outdated.

Candidates lacking specific experience or academic backgrounds are overlooked, perpetuating systemic inequality.

Someone’s experience can and often does make them a suitable candidate. However, it is no longer ethical or efficient to rely on this assumption.

Take a look at the results of the famous Schmidt-Hunter meta-analysis below…

Hiring methods predictivity chart


50+ years of research tells us that education and experience just aren’t very good at predicting someone’s ability.

Work simulations bypass these flawed proxies and test for skills directly… what could be more predictive than asking candidates to perform parts of the job itself?

Whilst work simulations can be anything from a cognitive ability test to a culture fit assessment, ‘work sample’ are the most predictive form of assessment there is (you’ll notice they’re at the top of the chart above).

Work samples are similar to ‘tell me a time when…’ interview questions. The key difference is that they pose questions hypothetically, so that experience isn’t necessitated.

Instead of guessing at how well someone would perform in a job by looking at their background, work samples test this ability upfront.

This isn’t just a more empirical means of hiring, it’s also fairer.

Taking the emphasis off of education and experience means that every candidate gets a fair chance to show what they can do.

In a world where candidates with non-white sounding names need to send more than 70% more resumes to get the same number of callbacks as someone with a white-sounding name, how can we expect these people to have the sort of profile we’re looking for?

Additional CVs needed to receive a callback by ethnicity


How to create work samples

Decide on skills

Work simulations are most effective when tied to skills.

You’ll need to decide on 6-8 core skills required for the role you’re hiring for.

These can be a mix of soft and technical skills.

By attributing 1-3 of these skills to each work simulation question, you’ll be able to build a clear skills profile for each candidate.

Hiring for a pre-determined skill set also makes it easier to be objective, since candidates are being tested against criteria, rather than using ‘gut instinct’ (which is usually just unconscious bias).

Use realistic scenarios 

The next step is to think of 3-5 scenarios or tasks that would test at least one of the skills you just outlined.

Work simulations of any kind work best when specific to your organization and the role you’re hiring for.

If the role is one that has been vacated, you can use tasks that have cropped up in the past.

And if it’s a brand new role, you can simply present candidates with tasks or challenges that they’ll need to tackle, should they get the job.

Your work simulation questions can consist of both everyday tasks and bigger projects/problems that’ll need to be tackled.

For these larger issues, you can ask candidates to explain how they’d approach them or to draw up a quarterly plan of attack.

Pose questions hypothetically

Once you’ve come up with your scenarios, you can turn them into work simulations by asking candidates what they would do - and if it’s a writing-based task, you can ask them to perform it.


Here’s an example of a work sample we used for an Account Manager role:

Question

You've been given 50 accounts to manage, ranging from newly onboarded users to long term customers of Applied. They vary in terms of size, industry and knowledge of the Applied platform. Your manager asks you to come up with a plan for how you will focus your time to maximise the growth of your accounts over the next 6 months. What things would you consider in putting this plan together? How would you measure your success? Is there any other information you would need?

Skills tested

Accountability, Prioritisation, Data-driven

And for a Community Lead role:

Question:

We have 3000 people in our community. We have grown it by bi-monthly webinars, social content, a LinkedIn group and giving free resources out. The rate of growth using these methods is slowing. 

How do to speed up the rate of community growth to make sure it's primarily in the US? (imagine we are in a purely digital context)

Skills tested

Collaboration, Community Management Knowledge, Data-driven

Asking questions that are forward-looking means that candidates who haven’t encountered the given task before are given an equal chance.

Although experience can make someone the best candidate, it may actually be an outsider perspective that yields the best answer. 

The value of experience is that it forges skills.

And job simulations enable you to test for these skills without making any assumptions.

Create scoring criteria 

To data-proof your work simulations, you’ll need to jot down scoring criteria for each of your questions.

This ensures that candidates are being scored against skills, rather than a hirer’s preferences (and biases).

At Applied, we use a 1-5 star scale.


Below is the review guide we created for the Community Lead work sample above:

1 Star 

- Little or no effort

- Maybe 1 or 2 ideas that don't sound great or expounded with no depth or enthusiasm

3 Star 

- Comes up with a couple of original suggestions that sound interesting 

- Discussed methods to spread for our existing content which show a plausible understanding of different marketing channels

- Refer to how they might collaborate with the rest of the team

5 Star 

- Suggests an original idea not covered in the above examples, which sounds really compelling

- Suggests forms of growth that consider virality, not just our own hard graft or marketing spend

- Talks confidentially about different marketing channels

- Talks confidently about this in a US context 

- Shows awareness of who we want to be in this community, although not explicitly talking about our personas, at least hints that all community members are not equal.

- Has a strong sense of how other members of the team can be utilised for content and/or distribution, ideally across different functions


As you can see, we just added a few bullet points detailed what a good, mediocre and poor answer might include.

How and when to use work simulations 

Job simulations such as work samples are generally used later on in the process, either before or after interviews.

However, this won’t bring dramatic results, since most talent is missed at the screening stage.

That’s why here at Applied, we use work samples as our screening method.

No CVs, no cover letters, no bias.

By using 3-5 work simulation questions to anonymously screen candidates, we’re able to identify who has the skills to actually do the job, without knowing anything about where candidates have come from.

Using this process, we’ve found that 60% of candidates hired would’ve been missed by a traditional CV screening - many of whom are from minority backgrounds.

To ensure bias is mitigated as much as possible, we have three team members review candidates’ answers.

If an individual reviewer has a bias towards or against a candidate, this will be averaged out by the scores of the other reviewers.

We’ve found three to be the ideal number - any more and you’ll see diminishing returns.

Once all answers are scored, we then average out candidates’ scores and bring the highest scorers through to the interview stage.

Applied is the essential platform for debiased hiring. Purpose-built to make hiring empirical and ethical, our platform uses anonymised applications and skill-based assessments to find talent that would otherwise have been overlooked.

Push back against conventional hiring wisdom with a smarter solution: book in a demo