An AI impact assessment auditors accept: structure and threshold test

AI impact assessments fail in one of two directions. They are so heavy that teams quietly avoid triggering them, or so light that they document a decision already made. Both produce a folder of artefacts and no change in what gets built.

ISO 42001 requires an AI system impact assessment process. The EU AI Act requires a fundamental rights impact assessment for certain deployers. Neither prescribes a format, which is why the design decision matters more than the compliance requirement.

Why the threshold test comes first#

If every AI system requires a full assessment, teams will define their work as not-AI. We have watched a data science team describe a gradient-boosted credit model as "statistical scoring" specifically to avoid a process that took three weeks.

The threshold test solves this. Twelve questions, ten minutes, answerable by the engineer building the thing, producing a defensible yes or no. Roughly 15–20% of systems clear it. The rest are registered and proceed.

The twelve questions we use:

1. Does the system produce an output about an identifiable person?
2. Does that output inform a decision affecting that person's access to a service, opportunity, or entitlement?
3. Could a wrong output cause financial, physical, legal or reputational harm to them?
4. Does the system operate without a human reviewing each individual output?
5. Are the affected people unaware the system is involved?
6. Does the system use data about protected or sensitive characteristics, directly or by proxy?
7. Does it affect children, or people in a position of dependence or vulnerability?
8. Is the training data of unknown or unverifiable provenance?
9. Does the system operate in a domain listed as high-risk under the EU AI Act?
10. Would you be uncomfortable explaining the system's operation publicly?
11. Can an affected person contest or appeal the outcome?
12. Would a wrong output be detected, and how quickly?

Any yes to 2, 3, 7 or 9 triggers a full assessment. Two or more yes answers across the rest triggers one. Question 10 is deliberately subjective, and it is designed to catch what the other eleven miss.

What the full assessment must contain#

Broader than a DPIA. A DPIA asks about risk to data subjects arising from processing. An AI impact assessment asks about consequences for individuals, groups and society arising from the system's behaviour.

System purpose and intended use. What decision it informs, who is affected, at what volume. Also the out-of-scope uses — what the system must not be used for, which becomes a control.

Data provenance and quality. Where training, tuning and evaluation data came from, what lawful basis covers it, whether it is representative of the population the system will be applied to. For third-party foundation models, this is where you record what the provider disclosed and what they did not.

Performance and failure analysis. Accuracy against a defined baseline, and — more importantly — the failure modes. What does a false positive cost, and to whom? A false negative? Are those costs distributed evenly across affected groups?

Fairness across affected groups. Measured disparity across relevant characteristics, with a stated threshold. Where differences exist, whether they are justified by legitimate factors, documented explicitly rather than assumed away.

Human oversight design. Who reviews, with what authority, at what capacity. A reviewer approving four hundred outputs an hour is not oversight, and stating the capacity honestly is what makes this section credible.

Contestability. How an affected person learns the system was involved, and what recourse exists. This is the section most often missing and the one regulators are most interested in.

Monitoring and retirement. What is measured post-deployment, what threshold triggers intervention, who is notified, and what happens when the system is superseded.

Assess before design locks#

The single decision that determines whether the assessment is analysis or paperwork.

An assessment run after the model is built and the integration written can only justify. Every genuine finding at that point implies rework the team will resist, so findings tend not to be genuine.

Run at the point where the objective is defined and the approach is chosen — before training data is assembled — and the assessment can change the design cheaply. In practice that means the trigger is the project intake, not the deployment gate.

Keep the deployment gate as well, but as a review of the earlier assessment against what was actually built. Two touchpoints, different questions.

What an ISO 42001 auditor examines#

Having taken clients through this, the pattern is consistent. Auditors spend little time on whether assessments exist and a great deal on three other things.

Did it change anything? They will sample assessments and look for evidence that a finding altered the system — a feature removed, a threshold adjusted, oversight added, deployment delayed. A set of assessments that all conclude "no significant risk identified" invites the question of whether the process has any teeth.

Who signed, and could they refuse? The residual risk acceptance needs a named individual with the authority to say no. If the accepting person is the project sponsor, that is a conflict and it will be raised.

Is it revisited? Systems change. The auditor will ask when the assessment for a given system was last reviewed, and what triggered the review. "Annually" is acceptable; "when we remember" is not.

That is the kind of evidence that satisfies the first question. A feature list that gets shorter after review is proof the process has consequences.

Third-party systems, where most of the estate now sits#

The template above assumes you built the system. Most organisations mostly did not — AI arrives embedded in SaaS, as API calls to a foundation model, or as a feature switched on by default in a tool already in use.

The assessment still applies, but three sections change materially.

Data provenance becomes a disclosure exercise. You cannot audit a foundation model's training corpus. What you can do is record what the provider disclosed, what they declined to disclose, what you asked for, and what you concluded about the residual risk. An auditor accepts "provenance is not disclosed by the provider; we assessed the consequence for our use case as follows" and does not accept a blank field.

Performance analysis shifts to your context. The provider's benchmark scores describe their evaluation set, not your population. The meaningful measurement is how the system performs on your data, for your decision, across your affected groups — which usually means constructing a small evaluation set of your own. A few hundred labelled examples is enough to detect the failures that matter.

Version control becomes a control. A provider who silently swaps the underlying model changes your output distribution with no change record. For any system informing a consequential decision, ask for version pinning and change notification in the contract. Where the provider cannot offer either, that constraint belongs in the assessment as an accepted risk with a named acceptor.

The recurring mistake is treating a vendor's own compliance claims as a substitute for your assessment. Using a certified provider does not transfer accountability for the decisions their system produces about your customers. Under the EU AI Act a deployer carries obligations distinct from the provider's, and under DPDPA the fiduciary relationship is unaffected by who wrote the model.

Making it stick#

The last point is the one most often skipped. Every template we have written has needed material revision after the first three uses, and running those with the team also builds the internal capability to run the fourth without us.

Our AI Governance & Assurance practice runs these programmes, and the methodology — threshold test, full template, mitigation library and three worked examples — is published as our DPIA & AI Impact Assessment Methodology.