Test AI-Generated Code by Checking What Its Tests Miss

A coding assistant writes a package-size selector and its tests. Everything passes. A reviewer changes one comparison from “less than or equal to” to “less than.” Five-item orders now receive the wrong package, but the tests remain green. The code was exercised; the consequential difference was not detected.

This is a constructed teaching example, not a customer incident. For developers accepting generated patches, the useful question is not simply whether tests exist. It is whether those tests would reject a meaningful defect. ZharfAI proposes checking that sensitivity with small, isolated code changes. Never inject these changes into a live service.

Separate passing tests, execution coverage and defect detection

A passing test says its execution did not contradict its written expectations. That is valuable only to the extent that the expectations express the intended behavior. A test can call a function, receive a plausible value and never ask the question that would distinguish a correct answer from an incorrect one.

Coverage.py's documentation describes tracking executed code and identifying code that was not executed. It supports statement and branch coverage. Those measurements locate exercised paths; reaching a line is not the same as checking its effect. Coverage is useful evidence about where tests went, not a certificate for what they established.

Stryker's mutation-testing introduction describes changing code in small ways, one change at a time, and rerunning tests. A failing test detects a mutation; a change that leaves the tests passing survives. The method probes sensitivity to particular changes. It does not prove every possible behavior correct.

This problem is not exclusive to AI-written code. The specific workflow concern is that an assistant deriving both implementation and expected answers from its own interpretation may reproduce the same misunderstanding in both. That is an analysis of a possible failure mechanism, not a measured claim about the defect rate of coding models.

Establish the expected behavior before reading the implementation

Write a small, reviewable contract for the package example: item counts must be positive integers; orders containing up to and including five items receive a small package; larger counts receive a large package. Five is an invented business boundary for this exercise. A real product owner must establish the actual rule.

The expected answer for five is now available independently of any implementation. By contrast, recording today's function output and requiring tomorrow's output to match can preserve an existing mistake. Such a regression test can detect change, but it cannot by itself establish that the original behavior was correct.

Give each important expectation a short rationale: an approved requirement, a reviewed example or an explicit product decision. If the requirement merely says “small orders,” do not ask the model to invent the threshold. More tests cannot resolve an undefined business term. Return that question to the person responsible for the behavior.

Our AI output-contract guide makes the related distinction between a valid representation and a valid decision. Here, a successful call and the right return type are also insufficient. Test the package choice that affects the user, not merely the existence of a method or the number of times it was invoked.

Use one boundary input to distinguish the two implementations

The following JavaScript deliberately omits input validation. It selects a package only for already-valid positive integer counts; it is not a production-ready service. Run it as a standalone local file with no operational access. The two functions allow a controlled comparison without modifying an application.

const original = n => n <= 5 ? "small" : "large";
const mutant = n => n < 5 ? "small" : "large";

const initial = [[1, "small"], [6, "large"]];
const improved = [...initial, [5, "small"]];
const passes = (fn, cases) =>
  cases.every(([input, expected]) => fn(input) === expected);

console.log(passes(original, initial));  // true
console.log(passes(mutant, initial));    // true
console.log(passes(original, improved)); // true
console.log(passes(mutant, improved));   // false

The initial cases exercise both package choices but not equality at the boundary. The four outputs are true, true, true, false. Adding the requirement-derived expectation for five detects this particular mutation. Three passing examples do not establish that the selector, its validation or its integration is completely correct.

This table covers the wider contract, not just the toy function. The zero case must invoke the real validator; expecting the selector above to reject it would be a mistaken test. Strings, fractional numbers and missing values also belong in validation tests. Do not claim to have tested a boundary that the fixture never calls.

Keep the deliberate fault small, isolated and reproducible

First run the unchanged implementation's tests. If that baseline fails or fluctuates, a failing mutated run does not establish that the injected defect was detected. A broken environment or unavailable dependency is a separate finding. Resolve or characterize the baseline before interpreting the comparison.

Use one change per experiment: alter the boundary operator, remove a check or substitute a constant result. Retain the original revision, exact change, test revision, command and failure reason. Several simultaneous edits make attribution difficult: one may cause the observed failure while another remains completely untested.

Execute mutations in disposable copies without production credentials. Sending mail, charging an account, deleting records or calling a production endpoint must not be a side effect of checking a predicate. Use controlled test dependencies where needed. A narrow, pure-function experiment requires no access to a live agent or customer system.

After improving a test, require it to pass the original implementation and fail the faulty one for the intended assertion. Do not change the expected answer merely to make the faulty version pass. Finally discard the mutated copy and review the release diff: only the actual fix and durable regression tests should remain. The broader DevSecOps and code-review guide places this evidence alongside the other release controls.

Investigate survivors before writing more tests

A surviving change is a question, not an automatic verdict. Does it actually alter observable behavior for permitted inputs? For positive integer counts, changing n <= 5 to n < 6 preserves the package decision. A behavior-oriented test should not be forced to distinguish implementations that satisfy the same contract.

Stryker's equivalent-mutant guide explains why behavior-preserving changes can prevent a perfect score. Establish equivalence against the allowed domain and observable effects. Failing to find a counterexample is not, by itself, proof of equivalence.

The n < 5 change is different: five provides a concrete distinguishing input, so a boundary assertion is justified. If the changed path never ran, improve reachability. If it ran but the different result never reached an assertion, inspect the observation. Adding unrelated tests might improve a headline percentage while leaving the same defect undetected.

Do not let a mock decide the entire answer

Suppose the package test replaces the selector itself with a stub that always returns “small.” An assertion expecting “small” will pass without testing the implementation. A test double is useful for controlling a dependency. Replacing the very behavior under examination removes the evidence you intended to collect.

Python's unittest.mock reference documents replacing components, supplying responses and checking calls. Its autospec feature helps enforce interfaces and call signatures. Interface fidelity is not proof of the real service's behavior; a signature-correct double can still supply an unrealistic answer.

Our proposed arrangement keeps the selector real and substitutes only the outgoing order adapter with a test recorder. Inspect both the selection and the payload passed to that adapter. For invalid input, assert that no outgoing call occurs, as well as checking the error. A call-count assertion alone can miss a wrong payload.

A separate integration test against an appropriate test dependency should exercise request formatting, response mapping and dependency failures. Unit-level mutation testing does not replace that work. Nor does a fabricated response establish compatibility with a provider's latest implementation. Keep each claim attached to the boundary actually tested.

Generate more inputs without losing the meaning of correctness

Once the boundary examples are explicit, property-based tests can explore a wider set of inputs. For this contract, increasing a valid count should not change a large package back to a small one. However, a function that always returns “large” also satisfies that property. A broad invariant needs known-answer anchors.

Keep the examples for one, five and six, and test invalid inputs separately. Derive properties from the contract rather than copying the implementation's conditional into the assertion. Repeating the same expression in two files creates agreement, not independence. Input variety cannot repair a wrong definition of the expected behavior.

Hypothesis distinguishes an input domain from its sampling distribution. Restricting the domain can exclude the values that reveal a bug. Its search for failures is not intended to reproduce production traffic, so the frequency of failures in these tests is not a user-facing defect-rate estimate.

For expensive tests, set a runtime and example budget and disclose the limits. Preserve a discovered failure as a reproducible regression case instead of relying on another fortunate draw. An assistant can suggest inputs, but a reviewer must still establish why the resulting output should be accepted or rejected.

Read a mutation score with its numerator and denominator

Under Stryker's documented metrics, detected mutations include test failures and timeouts. Survivors and uncovered mutations are undetected; compile and runtime errors are excluded from valid mutants. Ignored and pending cases remain separate. Similar percentages can therefore conceal different evidence.

If eight out of ten valid mutations are detected, the score is 80% for that set. It is not an 80% probability that the product is correct. If the two survivors affect the package boundary, detecting many easy changes to error-message text does not compensate. Review behavior risk separately from mutation count.

Record selected files and operators, exclusions, outcome counts and tool versions. A timeout can count as detection in the tool while still requiring engineering investigation: did the mutation cause a loop, or did the environment stall? Do not turn an infrastructure failure into evidence of application correctness.

Start with a small set of consequential behavior: acceptance conditions, boundary calculations and prevention of unintended effects. This is ZharfAI's proposed prioritization, not a universal quality threshold. Whole-repository mutation runs can be expensive. A scoped run is legitimate; presenting its result as assurance for the entire system is not.

Keep acceptance evidence useful after the next change

A compact review packet should identify the product rule, deliberate change, distinguishing input, expected result and outcomes on the original and mutated versions. Give equivalent or unresolved cases a rationale and decision owner. This does not replace reading the patch; it makes the tested failure mechanism visible to its reviewer.

Do not instruct a coding agent to reach 100% at any cost. Ask it to propose a meaningful change, derive a distinguishing input from the contract and write a test that rejects the faulty behavior without rewriting the contract. Review deleted tests, changed expectations, narrowed generators and new tool exclusions as carefully as the application code.

Tests need review when requirements change. If small packages later hold six items, the old case for five remains useful but no longer represents the new boundary. Our change-control and progressive-release guide connects acceptance evidence to a specific revision. Yesterday's result is not permanent authorization for tomorrow's implementation.

Track important survivors, flaky baselines, review time and defects that later escape. The goal is not a larger collection of agreeing tests. At acceptance, a reviewer should be able to say something precise: this behavior is required, this change violates it, and this test detects that violation.

Source notes — reviewed October 6, 2026

The references below are official project documentation. At review, Coverage.py listed version 7.16.2, released September 27, 2026; the Hypothesis documentation identified 6.168.5 and Python identified 3.14.8. The Stryker pages are living references without stated publication dates. The example, decision table and acceptance workflow are ZharfAI engineering analysis, not a customer-product experiment or security guarantee.

• Coverage.py: execution coverage
• Stryker: mutation-testing introduction
• Stryker: mutant states and metrics
• Stryker: equivalent mutants
• Hypothesis: domain and distribution
• Python: unittest.mock reference