From Demo to Dependable System: An AI Readiness Checklist

A prototype proves that a workflow can work once. Production readiness proves that a team can operate it when inputs are messy, dependencies fail, usage grows, policies change, and a model update shifts behavior without changing the API.

That distinction matters because AI failures rarely stay inside the model. They appear as the wrong customer message, an unauthorized tool call, a missing citation, a duplicated transaction, a silent handoff failure, or a cost spike. A dependable system needs technical controls and a clear operating model.

Use the ten gates below as a release review. They are not a substitute for sector-specific regulation or engineering judgment. NIST explicitly describes its AI RMF Playbook as adaptable guidance rather than a one-size-fits-all checklist.

Gate 1: Name the Purpose, Boundary, and Owner

Write a one-page system charter:

• What decision or task is the system meant to improve?
• Who uses it, and who may be affected by its output?
• What data and tools can it access?
• Which uses are explicitly prohibited?
• What is the maximum consequence of a wrong answer or action?
• Which team owns quality, security, cost, incidents, policy, and retirement?

If ownership is divided, publish a RACI matrix and a single incident lead. “The AI team” is not an actionable owner at 03:00.

Release evidence: approved charter, named service owner, on-call path, risk tier, prohibited-use list.

Gate 2: Define Success and Unacceptable Failure

Model accuracy alone does not describe the product. Define three layers of metrics:

1. Task quality: factual correctness, extraction accuracy, retrieval relevance, citation support, or action success.
2. Business outcome: resolution time, errors avoided, conversion, recovery time, or expert hours saved.
3. Risk and experience: harmful-output rate, permission violations, override success, fairness slices, privacy events, and user trust signals.

For each metric, specify the dataset, calculation, owner, threshold, and response when it breaches. List failures that block release even if the average score is strong—for example, sending data to the wrong customer or executing an action outside the user's authority.

Release evidence: metric contract, launch thresholds, stop conditions, baseline from the current human or software workflow.

Gate 3: Build Representative Evaluations

Create evaluation cases from the real distribution, not a polished demo set. Include:

• common cases and high-value cases;
• rare but severe failures;
• incomplete, contradictory, stale, multilingual, and adversarial inputs;
• important user and customer segments;
• provider errors and tool timeouts;
• tasks where the correct behavior is to abstain or escalate.

Separate the development set from the release gate so prompt changes cannot gradually overfit the test. For nondeterministic systems, run repeated trials and report a distribution rather than one score. Have domain experts review the rubric and a sample of labels.

NIST's AI RMF Core states that AI systems should be tested before deployment and regularly in operation, with uncertainty, comparisons, and documented measurement. See the Measure function for concrete practices.

Release evidence: versioned evaluation set, rubric, slice report, blocked-failure report, human baseline, regression history.

Gate 4: Enforce Data and Security Boundaries

Threat-model the whole workflow: user input, retrieved documents, prompts, memory, models, tools, credentials, output channels, and logs.

Minimum controls include:

• per-user authentication and authorization;
• least-privilege, per-tool scopes;
• secret isolation from prompts and model-visible context;
• explicit trust labels for external or user-controlled content;
• prompt-injection tests for browsing, email, documents, and tool descriptions;
• egress restrictions and allowlists for sensitive systems;
• confirmation for high-impact actions;
• rate, amount, and time limits;
• short-lived credentials and auditable revocation;
• sanitization that preserves meaning and provenance.

The OWASP Top 10 for Agentic Applications 2026 is a useful threat-modeling input for agents that plan or call tools. Our deeper guide to least-privilege tool permissions explains capability design.

Release evidence: threat model, data-flow diagram, permission matrix, red-team results, secret scan, abuse limits.

Gate 5: Make Actions Safe and Reversible

Treat model output as untrusted input to the execution layer. Validate tool name, parameters, business rules, user scope, and current state before each call.

Use:

• idempotency keys for retried operations;
• preview or dry-run modes;
• transaction boundaries where supported;
• compensating actions when rollback is impossible;
• explicit approval for irreversible changes;
• post-condition checks against the system of record;
• immutable records of request, approval, execution, and result.

For a payment workflow, a successful HTTP response is not enough; confirm the intended ledger state and transaction identity. For customer communication, confirm recipient, language, content, and delivery status.

Release evidence: tool contracts, validation tests, rollback plan, post-condition tests, duplicate-action tests.

Gate 6: Design Fallback and Human Handoff

Define what users experience when:

• the model provider is unavailable;
• retrieval is stale or incomplete;
• a required tool fails;
• latency exceeds the product budget;
• the system is uncertain;
• the action needs a higher authority;
• monitoring disables the feature.

A fallback may use a smaller model, cached result, deterministic workflow, manual queue, or clear stop. It must preserve context so a human does not restart the task. Show the reason for handoff and the evidence already collected.

Read designing human approval boundaries for interface and authority patterns.

Release evidence: fallback matrix, queue ownership, context-preservation test, user-facing messages, recovery objective.

Gate 7: Instrument the Decision Path

Traditional uptime metrics are necessary but insufficient. Trace the path from user intent through retrieval, model calls, tool decisions, approval, execution, and outcome.

Capture:

• request and trace IDs;
• model, prompt, retrieval index, policy, and tool versions;
• latency and token or compute cost by step;
• retrieved source identifiers and freshness;
• tool attempts, errors, retries, and results;
• safety and policy decisions;
• approval, override, and correction events;
• privacy-safe quality signals and final business outcomes.

Do not log sensitive prompts by default. Apply access controls, redaction, retention limits, and sampling appropriate to the risk. OpenTelemetry's GenAI semantic conventions can reduce vendor-specific instrumentation, while the NIST 2026 monitoring report explains why post-deployment monitoring must cover more than infrastructure health.

Release evidence: trace example, metric catalogue, log data classification, alerts, monitoring ownership, retention policy.

Gate 8: Control Every Changeable Component

An AI product can change when any of these change:

• model or provider;
• system prompt or template;
• retrieval corpus, chunking, embedding, or ranking;
• tool implementation or schema;
• safety policy or threshold;
• user interface and approval flow;
• dependency, locale, or data transformation.

Version these components together in a release manifest. Run impacted evaluations before rollout. Use canaries or shadow traffic, compare the new and old versions, and retain an immediate rollback path. A provider's silent model alias update should trigger the same review as an internal model change.

Release evidence: release manifest, impact map, regression result, canary plan, rollback owner.

Gate 9: Prove the Economics Under Real Load

Estimate cost per completed outcome, not only cost per model call:

cost per outcome =
  model and infrastructure cost
  + retrieval and tool cost
  + review labor
  + retry and fallback cost
  + expected incident and rework cost

Load-test typical and worst-case paths. Set budgets for tokens, tool loops, wall-clock time, and queued human review. Measure whether the automation moves work faster or merely transfers effort to reviewers.

Release evidence: volume forecast, unit economics, load test, cost alerts, hard execution budgets.

Gate 10: Prepare Incident Response and Retirement

Define incident severity, intake channels, containment actions, evidence preservation, notification requirements, and decision authority. Make it possible to disable a model, prompt, tool, data source, tenant, or feature without taking down unrelated services.

Retirement matters too. Identify stored memories, embeddings, logs, credentials, evaluation data, and downstream integrations that must be deleted, retained, migrated, or revoked.

NIST's Manage guidance explicitly connects post-deployment monitoring with appeal, override, decommissioning, incident response, recovery, and change management.

Release evidence: runbook, kill switches, contact tree, forensic record plan, notification matrix, retirement checklist.

Run a “Bad Day” Exercise

Before launch, conduct a tabletop and a technical drill. Choose scenarios that cross system boundaries:

• the provider returns plausible but systematically wrong outputs;
• a retrieved document contains prompt injection;
• a user's permission is revoked midway through a long-running task;
• the retrieval index is stale after a policy update;
• tool retries create a duplicate external action;
• traffic grows tenfold and pushes all requests onto the expensive fallback;
• a user reports harmful output but the trace contains sensitive data.

Verify who receives the alert, what the user sees, whether new actions stop, how in-flight work is preserved, which records explain the event, and how service is restored. Record gaps as launch blockers or accepted risks with owners and dates.

Use a Simple Launch Scorecard

Do not average a red gate into a green score. Launch only when red conditions are removed or the feature is narrowed so the risk no longer applies.

Launch as Controlled Expansion

Start with a bounded audience, read-only access where possible, and reversible actions. Compare live behavior with the evaluation set. Review corrections and near misses, not only visible incidents. Expand data access, user scope, and autonomy independently so you know which change affected outcomes.

A practical sequence is:

1. Offline evaluation.
2. Shadow mode with no user-visible output.
3. Advisory mode with human judgment.
4. Approval-gated actions.
5. Narrow autonomous actions with hard limits.
6. Broader rollout only after monitored evidence supports it.

The evidence-first automation guide shows what a reviewable decision packet should contain at every stage.

Frequently Asked Questions

Does passing a benchmark mean the system is production-ready?

No. A benchmark measures selected behavior under selected conditions. Production readiness also covers data access, workflow integration, human factors, security, reliability, monitoring, cost, incident response, and the effect of future changes.

How often should evaluations run?

Run impacted tests for every material component change and schedule broader regression runs based on risk and change frequency. Continue sampling production outcomes because real inputs and user behavior differ from controlled evaluation.

What should block launch?

Any unacceptable failure without an enforced control; unclear ownership; unbounded sensitive data or tool access; no safe failure mode; inability to trace or stop consequential actions; or economics that cannot be bounded.

Source Notes

This checklist was substantially reviewed on July 30, 2026 against:

• NIST AI Risk Management Framework Core
• NIST AI RMF Playbook
• NIST AI 800-4: Challenges to the Monitoring of Deployed AI Systems
• OWASP Top 10 for Agentic Applications 2026
• OpenTelemetry semantic conventions for generative AI

Production readiness is not a document signed once. It is the operating discipline that keeps a useful demo useful after launch.