Is Your Assessment Software Predictive? Here’s What Works

Published by:
Joe Caccavale
April 29, 2022
5
 min read

Work Sample Cheatsheet

What is predictive validity?

Predictive validity is a measure of accuracy used in science and psychology. It’s essentially a means of judging how accurately something can predict future outcomes.

When it comes to hiring, predictive validity is used to see how effectively an assessment can predict someone’s job performance.

If you’re going to be spending on assessment software, you’ll want to make sure the assessments they use are actually predictive. 

Using predictive assessments will enable you to reliably identify those most suitable for the job - and the sooner you use them in your hiring process, the more time you’ll save.

Which assessments are the most accurate?

Make no mistake - for the most part, any assessment software is a step in the right direction. 

Why? Because traditional CV screening is prone to bias.

We make snap decisions about people based on their identity and background and once we make our minds up, we tend to seek information that confirms our assumptions.

This is why candidates from underrepresented backgrounds tend to be disproportionately overlooked.

Extra applications needed to receive 1 callback (chart)

So what does work?

Whilst predictive validity studies can take years to complete, we can look at the results of the Schmidt-Hunter meta-analysis for a rough guide to what works and what doesn’t.

This landmark study summarizes 100+ years of research around hiring assessments.

Predictive validity of assessment methods (chart)

As you can see from the chart above, assessments that revolve around background aren’t going to tell us much about someone’s ability.

A step up from this is cognitive ability tests. These are undoubtedly a fairer, more accurate means of testing skills.

Where these kinds of tests fall flat, however, is their relevance to the job itself. 

A cognitive test might be great at identifying people with certain skills or behaviours. 

But there’s no concrete body of evidence that suggests this makes them the perfect person for the role - the results of the test are still just a proxy for on-the-job ability.

Work samples, on the other hand, are designed to simulate the job itself by having candidates perform small parts of it.

Whilst there are many ways to test someone’s skills, work samples (which are sometimes referred to as work simulations), test these skills within the context they’ll actually be used.

In the Schmidt-Hunter study, work samples are actually used to test the skills of manual workers. However, given their predictive power, we adapted them for office-based jobs…

Example work sample question

The idea behind work samples is to have candidates essentially do the job before getting it.

Whilst we can’t have candidates physically join us for a day at the office to see how they handle tasks, we can ask them to do small parts of it.

At the screening stage, writing-based tasks are ideal. If the job itself doesn’t require writing skills, be sure to let candidates know they’re not being tested on their spelling and grammar.

When it comes to interviews, you can have candidates actually role-play tasks to simulate them more closely.

For commercial roles, you could have them give a presentation or simulate a client call. 

And for more strategic roles, you can present candidates with a larger case study to work through.

How to create work samples

The most effective way to implement work samples is by tying them to the skills needed for the job.

Start by listing 6-8 essential skills/behaviours. These can be a mix of technical and softer skills. 

Once you know the skills you’re looking for, you can think of scenarios and tasks that would test them. For example, you could give candidates a list of tasks to prioritize, an email to draft or a tricky situation with a customer to think through.

For the most predictive work samples, use the tasks that candidates would actually be doing should they get the job. Are there any that can be performed via writing? If not, simply ask candidates how they would deal with them or ask them to explain their approach.

Once you have your work samples, you’ll need to give yourself some basic criteria to score them against.

This will allow you to give answers number scores and build a candidate leaderboard.

Applied candidate leaderboard (screenshot)

Your scoring criteria doesn’t need to be very detailed, a few rough and ready bullet points will do - you just want to write down what a good, bad and mediocre answer might include.

Example review guide

Do you need assessment software to find the best talent?

The truth is - you don’t need to use assessment software to start testing candidates with more accuracy.

Work samples can be used completely independently of the Applied Platform.

If you’re not ready to ditch CVs entirely, we’d suggest using 3-5 work samples alongside your usually process. Ideally, you’d score answers before looking at CVs.

You’ll likely find that those who you shortlist with work samples wouldn’t have been chosen based on CVs alone.

So, why use assessment software like Applied at all?

Faster workflow 

Whilst you can (and absolutely should) try using work samples on your own, using software like ours will make creating, distributing and scoring work samples easier.

For high volume roles, we also provide multiple choice tests to narrow the initial pool.

Applied candiate management

Anonymization

We know that unconscious bias leads to minority background candidates being disadvantaged, so we anonymize applications.

Again, although this is possible manually, it can be incredibly time consuming.

Work sample library

One of the most labour intensive steps towards using predictive assessments is actually coming up with the questions.

Thats why we built our work sample question library to help you get started.

Work sample library

Debiased scoring

Once candidates have answered your questions, Applied will anonymize candidates and assign them to your hiring team to review - with the order randomized each time to avoid lesser-known ordering effects.

These scores will be used to generate your candidate leaderboard.

Applied reviewer screen

Analytics 

This is the one thing that would be near-impossible to do manually…

Using assessment software will allow you to see which questions are the most predictive and allow you to see any diversity drop-offs in your process.

Diversity analytics

Applied is the essential platform for debiased hiring. Purpose-built to make hiring empirical and ethical, our platform uses anonymized applications and skill-based assessments to identify talent that would otherwise have been overlooked.

Push back against conventional hiring wisdom with a smarter solution: book in a demo