How to Build AI Decision Explanations That Can Be Tested and Challenged

A small-business owner receives a credit decision produced with machine-learning assistance. The notice says only that the application did not meet “internal criteria.” A support agent asks a language model for more detail, and it writes a polished paragraph about cash-flow volatility, business age, and sector risk. The paragraph sounds reasonable. Nobody can show that those factors actually caused the decision.

That is not an explanation problem at the copywriting layer. It is an evidence failure in the decision system.

The reader decision is: which explanation must exist before an AI-assisted outcome may stand, and what evidence must prove that explanation is faithful, useful, and open to correction or challenge? The answer depends on the decision, audience, consequence, governing rules, and explanation method. One universal “Why?” button cannot do all of that work.

The explanation starts before the decision

An explanation is not a story reconstructed from whatever data happens to be available later. It is a decision artifact. The system must preserve the inputs, policy, model, thresholds, intermediate determinations, human actions, and outcome that existed when the decision was made.

This distinction matters when a generative model sits near a predictive model or rules engine. It can turn structured facts into readable language, but it can also build a coherent rationale from correlations, generic policy, or records that differ from the decision-time state. Fluency can hide missing provenance.

The operating rule is simple: no explanation claim without a resolvable decision-time fact; no final decision when the required explanation cannot be produced. If a use case cannot meet that rule, narrow the use, route it to qualified review, choose a more interpretable method, or do not automate the decision.

What the verified sources establish

NISTIR 8312, Four Principles of Explainable Artificial Intelligence, separates four properties: the system supplies an explanation, the explanation is meaningful to its intended audience, it accurately reflects the reason or process that produced the output, and the system recognizes its knowledge limits. Crucially, NIST distinguishes decision accuracy from explanation accuracy. A correct decision can have a false explanation; a wrong decision can have a faithful account of how it went wrong.

The voluntary NIST AI RMF Core asks organizations to explain, validate, and document the model, interpret output in context, document tests, and integrate feedback and appeals into measurement. It is being revised, so treat it as a risk-management reference, not a permanent legal checklist.

The legal duties are narrower and jurisdiction-specific. Since 2 August 2026, Article 86 of the EU AI Act applies within its stated scope: certain affected people subject to decisions based on outputs from listed high-risk systems, with legal or similarly significant adverse effects, have a right to clear and meaningful explanations of the system's role and the main elements of the decision. Exceptions and other Union-law rights matter. This article is general engineering guidance, not legal advice.

In United States credit decisions, the current 12 CFR 1002.9 requires specific principal reasons for adverse action and rejects vague statements about internal standards or a failure to reach a qualifying score. It does not make every AI explanation a credit notice, nor does it define one universal explanation technique.

The UK Information Commissioner's Office and The Alan Turing Institute divide explanations by audience and purpose—rationale, responsibility, data, fairness, safety and performance, and impact—in their guidance on what goes into an explanation. The ICO currently marks this guidance as under review following the Data (Use and Access) Act. Its explanation taxonomy remains useful design input, but teams must verify the current law that applies to their processing.

The architecture and release gate below are ZharfAI analysis derived from these sources.

Decide what the recipient must be able to do

Start with the recipient's task, not the available visualization.

A useful explanation lets its recipient understand, verify, correct data, contest an outcome, approve an action, debug a failure, or assess a control. More detail is not automatically better: an affected person does not need a tensor dump, and an engineer cannot diagnose a feature pipeline from a friendly paragraph.

A nominal human in the loop does not remove this requirement. If a model shaped the evidence, ranking, recommendation, or options, record that role. Our guide to placing human judgment at the right approval boundary distinguishes meaningful review from ceremonial confirmation.

Keep five explanation objects separate

Teams often compress five different objects into one reason field:

1. Outcome: what was decided, when, for whom, and with what effect.
2. Rationale: the factors or evidence that actually supported this outcome.
3. Process: how data, rules, models, tools, and people produced it.
4. System assurance: how the organization tested reliability, fairness, security, and limits.
5. Recourse: how someone can correct input, add evidence, request review, or appeal.

These objects support one another but are not substitutes. “The model passed a fairness test” does not explain one rejection. “Low verified income relative to the requested payment” does not prove the whole system is fair. “Contact support” is not recourse if support cannot access the record or change the outcome.

For systems that should sometimes avoid a decision, connect the explanation contract to an externally enforced abstention policy. A system outside its knowledge or evidence limits should say which requirement failed and what happens next, not fabricate a complete rationale for an outcome it should not issue.

Build a decision receipt before rendering prose

At decision time, write an immutable or append-only receipt. A practical receipt contains four groups:

• identity and effect: decision, subject, request, time, outcome, owner and review route;
• evidence: source identifiers, exact values, effective times, missingness, retrieved spans and contradictions;
• execution: policy and model versions, rules, thresholds, feature pipeline, calibration and disposition;
• accountability: human action, override reason, approved reason codes and explanation-method version.

Do not expose every field to every recipient. The receipt is a protected evidence substrate for authorized views. Apply retention, privacy, security, and legal-hold rules; redact protected data in the explanation without destroying the original basis.

The explanation service should reject unresolved source identifiers, missing policy versions, unknown model routes, stale snapshots, and reason codes that do not map to a tested rule. That turns explainability into an acceptance condition instead of a documentation promise. The same receipt should feed the audit-ready evidence trail, so an explanation and an audit record cannot quietly describe different decisions.

Choose a method that matches the claim

The original SHAP paper defines a family of additive feature-attribution methods and specific desirable properties. It does not turn a feature-importance value into a causal account, a legal notice, or proof that the source data were correct.

Visual plausibility is also a weak test. In Sanity Checks for Saliency Maps, the authors found that some tested saliency methods remained insensitive to the trained model or data-generating process. The durable lesson is not that every saliency method fails; it is that an explanation method needs tests tied to the claim it is supposed to support.

Counterfactuals must survive the real world

A counterfactual can say that a different input would have crossed the model's boundary. The original counterfactual-explanations paper frames them around understanding a decision, contesting it, and identifying changes associated with another modeled result without exposing the model's full internal logic.

Before showing one, test:

• validity: does the archived decision version actually change outcome?
• proximity and sparsity: is the change focused enough to understand?
• feasibility: can the values coexist under domain constraints?
• actionability: can this recipient lawfully and reasonably change them?
• stability: does a tiny model or data update invalidate the guidance?
• non-discrimination: does it hide dependence on protected or proxy attributes?
• causal humility: does the wording say “the model would have returned,” not “this action will cause”?

Never recommend changing an immutable fact or manipulating a measurement merely to satisfy the model. If several feasible paths exist, offer diverse options and disclose the decision snapshot and validity period.

Let language models translate reasons, not invent them

A language model can convert approved reason codes and evidence into plain English or Persian, adjust reading level, explain a term, or produce an accessible format. Constrain the generation to the receipt:

• every factual sentence maps to a field or source span;
• required items cannot be omitted;
• prohibited inference and causal language are blocked;
• numbers, dates, identities, and policy names pass deterministic checks;
• the output states the responsible organization and review route;
• unsupported elaboration causes rejection, not another creative attempt.

Store the rendered explanation and its template, model, prompt, and source receipt. A later correction should create a new version linked to the original; it should not overwrite history. Bilingual editions need fact parity, not sentence-by-sentence translation, and both languages must preserve the same reason, caveat, deadline, and remedy.

Worked example: the polished explanation is wrong

Consider a working-capital application. A risk model estimates probability of repayment difficulty. A separate policy engine denies the request because two controlling conditions are true: the latest tax filing is missing, and verified debt-service coverage is below the approved threshold. The model score influenced queue priority but did not control the denial.

A generic explainer reads all available features and names business age, sector volatility, and recent cash flow as the “top reasons.” That is a plausible summary of model contribution—and the wrong explanation of the final decision.

The decision receipt supports a better explanation:

• outcome: request declined under policy version 7.3 at a named time;
• principal reasons: required filing absent; coverage ratio 0.91 against a 1.10 threshold;
• data: source documents, calculation version, currency, period, and missing-field state;
• AI role: risk score affected review priority, not the controlling rule;
• recourse: correct the filing record, challenge the calculation inputs, or request qualified review through a named channel;
• limits: no promise that changing one value guarantees approval under a later policy or complete assessment.

This engineering pattern is not legal advice.

Test the explanation, not its eloquence

Build an adjudicated test set from real decision classes, languages, recipient needs, edge cases, overrides, missing evidence, policy changes, and successful appeals. Then run six test families:

1. Fidelity: can each stated reason be resolved to the receipt, and does perturbing a controlling factor change the recorded outcome as expected?
2. Completeness: are all mandatory reasons, limits, ownership, and recourse fields present?
3. Data accuracy: do values match the decision-time authoritative record, including units, dates, and effective versions?
4. Meaningfulness: can representative recipients identify what happened and the next valid action without expert coaching?
5. Robustness: do paraphrase, ordering, irrelevant-data, randomization, and model-version tests expose unstable explanations?
6. Safety and rights: does the output avoid protected-data leakage, impossible advice, deceptive certainty, and inaccessible or unequal presentation?

Version the explanation method like any other measuring instrument. Re-run an overlap set when the model, feature pipeline, reason taxonomy, policy, template, translation system, or review process changes. Our frontier-model evaluation guide explains why a score series becomes misleading when the evaluator changes silently.

Failure modes that sound responsible

• Post-hoc rationalization: a generator sees the outcome and writes a likely story without the causal record.
• Nearest checklist item: the system picks the closest standard reason even when it was not used.
• Attribution substitution: top model features replace the principal reason for the final organizational action.
• Current-state reconstruction: an explanation uses corrected or updated data instead of the snapshot that produced the decision.
• Human-washing: the notice calls the result human-made although the person saw only an AI-ranked shortlist or default.
• Impossible recourse: a counterfactual changes immutable, unavailable, or mutually inconsistent features.
• Decorative transparency: a dashboard shows confidence and feature colors but no owner, source, correction path, or review authority.
• Vendor opacity transfer: the buyer promises explanations while the supplier cannot reproduce versions, inputs, or reasons.

When the required evidence is absent, treat the decision as unsupported. Do not solve that condition with more fluent prose.

Measure whether explanations change outcomes

Track explanation coverage, receipt completeness, source-resolution rate, fidelity-test pass rate, missing-principal-reason defects, counterfactual validity, recipient comprehension, correction submissions, appeal reversals, time to qualified review, explanation complaints, privacy incidents, and parity across language and accessibility modes.

Interpret the metrics together. Few appeals may mean trust or a hidden route; many reversals may expose poor decisions or a healthy correction path. Sample accepted and unchallenged outcomes too.

Set stop conditions. Suspend the consequential use when explanations cannot be reproduced, required reasons disappear, fidelity falls below the approved threshold, a model change breaks counterfactuals, one language loses material detail, or reviewers lack authority to repair an error.

The explanation release gate

Before an AI-assisted decision can stand, ask:

1. Is the decision and the AI system's actual role named precisely?
2. Does a protected decision-time receipt exist?
3. Can every reason resolve to a source value, rule, model contribution, or human judgment?
4. Is the chosen explanation method valid for the claim being made?
5. Are decision accuracy and explanation accuracy tested separately?
6. Does the explanation fit the recipient's task and language?
7. Are limits, uncertainty, and omitted information stated honestly?
8. Can the recipient correct data, submit evidence, and reach a reviewer with authority?
9. Are privacy, security, fairness, retention, and accessibility controls applied?
10. Will policy, model, data, or explanation changes trigger revalidation?

If the answer to a required item is no, the system is not ready to issue that outcome. The point of an explanation is not to make an automated decision feel inevitable. It is to make the actual decision inspectable—and repairable when the evidence does not hold.

Source notes — reviewed August 29, 2026

• NISTIR 8312: four principles of explainable AI.
• NIST AI RMF Core: voluntary risk-management outcomes used here.
• EU AI Act, Regulation (EU) 2024/1689: Article 86 scope and application dates.
• 12 CFR 1002.9: current United States credit-notification rule.
• ICO and The Alan Turing Institute guidance: explanation taxonomy and review status.
• Lundberg and Lee: original SHAP framework.
• Adebayo and co-authors: saliency randomization checks.
• Wachter, Mittelstadt, and Russell: counterfactual-explanation framework.