Validity documentation when AI sits on both sides of the test

When a vendor puts a validity report on the table, it is tempting to read it as a final answer. But validity depends on who takes the test, under what conditions, and what the score is used for. Those assumptions are shifting. The programme of SIOP's Leading Edge Consortium 2026, held in Baltimore on 1 and 2 October under the heading Assessment in a New Era, shows what the profession itself regards as open questions. For buyers, it doubles as a list of what to ask.

SIOP, the American society for industrial and organisational psychologists, describes the theme as the meeting point of new technology, analytics and changing workforce expectations, in support of decisions that can be defended. What matters is the pattern across the titles. A full block on Friday morning was devoted to innovation in scoring and validation. Tianjun Sun of Rice University spoke on why psychometrics matters more than ever when building AI-based assessments people can trust. Mike Fetzer of Aon asked whether AI is better than humans and answered in his title that it depends. Bradley Pitcher of APTMetrics presented work on using large language models to design and score assessment centres.

In the block on assessment design, Beth Bynum of HumRRO asked outright whether the fundamental principles of assessment are still fundamental in the age of AI. Ben Hawkes of TAPnet presented on moving beyond the label bias-free and holding vendors of AI assessments accountable.

We only know the titles, not the content. But they show that the field does not treat validity as a settled matter.

Sara Gutierrez of SHL presented on what an assessment score means when the individual is no longer alone, that is, when candidates complete tasks with AI as a helper. Tariq Shaban of Miraai asked whether unproctored tests can still be trusted, pointing to behavioural telemetry, deterrence and proctoring as possible answers. Jeff Facteau of HackerRank described AI as both a threat to and a protector of assessment integrity.

For a buyer, this has a practical consequence. A validity study tells you how well the test predicted job performance in a particular sample under particular conditions. If that sample was collected before candidates had access to generative language models, it does not necessarily say anything about how the test works today. So ask when the data were collected, whether the test was taken proctored or unproctored, and whether the vendor has examined what AI assistance does to the scores.

Several talks dealt with tests now being designed, scored and reported with AI. MQ Liu of Amazon presented a workflow in which generative AI is used all the way from construct to finished test content. Carolyn Hass of SuccessFinder spoke on validation, guardrails and human oversight when reports for decision makers are written by AI.

That raises a simple question. Does the documentation cover the version of the test you will receive, with the scoring model and the report format that are in the product today? A test whose content or scoring has changed is, in psychometric terms, a new test until someone has shown that the old evidence still holds. Ask for a version history and for documentation of what was done to make sure changes did not shift the measurement.

The consortium had a separate block on legislation, ethics and fairness, in which Kalpana Kotagal, a commissioner at the US Equal Employment Opportunity Commission, spoke on the current regulatory and enforcement landscape, followed by a panel of lawyers and researchers.

The American context is not the Danish one. But the question is the same here. When a vendor writes that a test is bias-free, that is a claim that needs support. Ask which groups were examined, with which method, and what the results were. A report that contains only a conclusion and no numbers is not documentation.

The programme in Baltimore was written by and for professionals who develop tests. When they themselves ask whether the classic principles still hold, buyers should not read a validity report less critically than they do. Read the report with three questions in mind. When and how were the data collected. Is this the same test I am buying. And what exactly was done to examine differences between groups. If the vendor cannot answer, that is an answer in itself.

Source: https://www.siop.org/events/the-leading-edge-consortium/assessment-in-a-new-era/program/schedule/