Skip to content
Assessment Guide
Back to Articles
Analysis4 min read

Norming and bias in tests when weighting the criterion does not deliver more diversity

A new article in Journal of Applied Psychology re-examines the classic advice to give contextual performance more weight in order to reduce adverse impact. According to the abstract, this typically does not improve diversity, is likely to harm validity and is often outperformed by Pareto-optimal predictor weighting. The article explains what that means when HR reads test documentation.

By RedaktionenSource: Journal of Applied Psychology, abstract of 5 November 2026, Jinsoo Choi and Daniel A. Newman (University of Illinois Urbana-Champaign), On criterion weighting to reduce adverse impact

A classic piece of advice gets re-examined

When an organisation discovers that a test hits some groups harder than others, the search for fixes begins. One piece of advice that has sat in textbooks and review articles for decades says that you should give contextual performance more weight in the criterion the test is meant to predict. The reasoning is that if success on the job is measured more broadly than pure task performance, taking in cooperation, helpfulness and initiative, the overall picture becomes fairer, and the test battery can be put together so that differences between groups shrink.

On 5 November 2026, Journal of Applied Psychology published an article in which Jinsoo Choi and Daniel A. Newman of the University of Illinois Urbana-Champaign test exactly that advice. According to the abstract, the analysis is built on an updated meta-analytic correlation matrix among predictors and criteria, in other words the relationships between what the tests measure and what they are supposed to predict, pooled across many studies.

What the researchers find

The conclusion runs against the conventional view. According to the abstract, giving contextual performance more weight is typically unlikely to improve diversity and is also likely to harm criterion validity, meaning how well the selection procedure predicts performance on the job. The researchers also find that criterion weighting is often outperformed by another method, Pareto-optimal predictor weighting.

That last point deserves a pause. When the researchers hold criterion validity constant, the Pareto-optimal solutions show that assigning more criterion weight to contextual performance is likely to exacerbate rather than reduce adverse impact. The advice that was supposed to make selection fairer can, in other words, pull in the opposite direction.

The article also introduces a new approach the authors call a Global Pareto Frontier or outer envelope. Here the weights on the predictors and the weights on the criteria are optimised at the same time, to find the best pairs of adverse impact and criterion validity for a selection system. The researchers also identify the circumstances under which this joint optimisation outperforms Pareto-optimal predictor weighting on its own.

Three terms worth knowing

Adverse impact is the American term for a selection method producing markedly lower hiring rates for one group than for another. The abstract explicitly names racial equality in hiring rates as the frame for the research. That is a US context, but the underlying mechanism, that different tests produce different group differences, applies to Danish employers too.

Criterion weighting is about how success on the job is defined. If cooperation and helpfulness count for more than task performance in the definition of good performance, that changes which tests appear to predict best, and therefore which tests end up in the battery.

Pareto-optimal weighting is a mathematical method that does not look for a single best solution but for the full set of weight combinations where diversity cannot be improved without losing validity, and the other way round. The decision maker can then pick the point on that curve that matches the organisation's priorities.

What this means when you read test documentation

For an HR team evaluating a test or a test battery, the article suggests three concrete questions for the vendor.

The first question is how the criterion was defined in the validation study. If the documentation stresses that the test predicts broad or contextual performance, and at the same time uses that as an argument for lower bias, that is precisely the link Choi and Newman call into question.

The second question is how the weights between the tests in the battery were set. Were they chosen on the basis of a Pareto analysis, by tradition, or by whatever was easiest to implement? The abstract suggests that predictor weighting is where the effort typically pays off most.

The third question is whether group differences and validity are reported together. A vendor that shows only one of the two numbers gives no basis for judging the trade-off between them.

Caveats

It is important to read the article for what it is. The abstract describes an analysis based on meta-analytic correlations, not a new field study in a single company. The results hold typically and probably, as the authors themselves put it, and the full article will contain the assumptions behind them. Danish readers should also keep in mind that the legal framework differs from the one in the United States, even though the statistical relationships are the same. Finally, it is worth asking whether a Pareto analysis is practical at all for a smaller company with few hires, because the method needs data on both group differences and job performance.

Kilde: https://doi.org/10.1037/apl0001426