Audit AI-generated tests before trusting coverage
A green suite can still prove almost nothing. Review what each test can detect: behavior-specific assertions, awaited failures, meaningful boundaries, controlled mocks, focused snapshots, and at least one change that should make the test fail.
Start from the behavior the test claims to protect. Confirm the system under test actually runs, every asynchronous path is awaited, and the expected value comes from an independent requirement rather than the same fixture or mock that produced the result. Add boundary, invalid-input, and dependency-failure cases. Keep snapshots narrow and review their changes. Finally, change one important branch or operator and prove the relevant test fails. Coverage locates execution; it does not establish that the assertions detect incorrect behavior.
Audit path
1. Write the behavior contract before reading the test
AI-generated tests often mirror the current implementation because the generator saw the same code. Begin with an independent contract: a requirement, public API, bug report, acceptance criterion, or invariant. Then map the smallest set of outcomes that would expose a broken implementation.
| Case | Evidence to require | Weak substitute |
|---|---|---|
| Nominal | Exact observable result or state transition for a representative valid input. | Value exists; function was called; response is truthy. |
| Boundary | Behavior at zero, empty, minimum, maximum, just below, and just above the rule. | Another middle-of-range example. |
| Invalid input | Specific error, rejection, status, or unchanged state. | Any exception; a broad truthy error object. |
| Dependency failure | Timeout, rejection, malformed response, or unavailable dependency produces the declared fallback or error. | Mock always succeeds; dependency is never invoked. |
| Side effect | Persistent state or emitted message is correct and unintended effects are absent. | Only verifies that a mock function was called. |
Minimum inventory
- Changed production files and the tests meant to protect them.
- Public behavior or acceptance criterion for each tested unit.
- Nominal, boundary, invalid, and failure-path expectations.
- Current runner result and coverage report, retained as baseline evidence rather than a quality verdict.
2. Prove the assertion path actually executes
Count real assertions, but do not stop at the count. A Jest test can reach the end before an unreturned promise or callback runs. Jest documents expect.assertions(number) and expect.hasAssertions() for verifying that asynchronous callbacks execute assertions, and its promise matchers must be returned or awaited.
Locate the observation
Identify the actual value, thrown error, emitted event, state change, or call boundary being inspected. A test body with setup and execution but no observation is a smoke run, not behavior evidence.
Check completion semantics
For promises, callbacks, timers, and background work, prove the runner waits for the assertion. An async test that exits early can pass even when the intended assertion never executes.
Reject trivial predicates
toBeDefined(), assert x is not None, and broad truthiness checks may be appropriate for a narrow contract. Flag them when the requirement is more specific than existence or truthiness.
test('rejects an expired token', async () => {
expect.assertions(1)
await expect(validateToken(expiredToken))
.rejects.toThrow('expired')
})
3. Trace where the expected value came from
The most dangerous tautology is not always expect(true).toBe(true). It is an expected value copied from the same fixture, mock, implementation constant, or returned object that produced the actual value. The assertion looks specific while comparing the system with itself.
- Mark the source of every expected literal, object, snapshot, or fixture.
- Prefer an expected value derived from the public contract or a separately calculated invariant.
- Reject referential self-comparisons such as
expect(mockUser).toBe(mockUser)when the service result is the behavior under review. - For transformations, use inputs whose expected output would change if ordering, filtering, rounding, authorization, or boundary logic broke.
- For error cases, assert the relevant class, code, or message fragment rather than merely the existence of an error.
4. Make mocks isolate a boundary, not replace the behavior
Mocks are useful for network calls, time, randomness, and slow or nondeterministic dependencies. They become misleading when the test replaces most of the system and then asserts the behavior of the replacement. Vitest documents that a fully mocked module replaces its exports, while Python and pytest emphasize patching the reference the code under test actually uses.
| Review | Question | Repair |
|---|---|---|
| Patch target | Did the test patch the namespace actually looked up by the system under test? | Patch where the dependency is used; add a failure if the real dependency is unexpectedly called. |
| Mock ratio | Does setup replace more behavior than the test exercises? | Keep the core logic real; isolate only the expensive or nondeterministic boundary. |
| Call proof | Does a call assertion stand in for the output or state the user cares about? | Assert the observable result as well as the interaction when both matter. |
| Negative case | Can the mock reject, time out, return malformed data, or violate the happy path? | Add the dependency failure that the production code promises to handle. |
A high mock count is a review signal, not an automatic defect. Some coordinators exist mainly to call dependencies correctly. Record the intended seam before changing the test.
5. Treat snapshots as reviewed assertions
Jest recommends committing and reviewing snapshots as code and keeping them focused, short, and readable. A snapshot-only test is weak when reviewers routinely accept a large diff without identifying which behavior changed.
- Limit snapshots to a stable, intentionally reviewed interface.
- Use property matchers or explicit assertions for volatile values such as IDs and dates rather than refreshing noise.
- Add direct assertions for the few fields or states that constitute the contract.
- Inspect snapshot updates before accepting them; a newly generated snapshot passes by construction.
- Do not use a broad snapshot to hide missing negative and boundary cases.
6. Make the suite prove it can fail
The final question is counterfactual: if the protected behavior were wrong, would this test fail? Make one small, reversible change to production logic: negate a conditional, alter a boundary, replace a return value, remove a state update, or change an arithmetic operator. Run the focused test, then restore the source.
For a repeatable version, mutation testing automates these changes. Stryker describes a killed mutant as a change that makes at least one test fail; a surviving mutant indicates that covered code changed without the suite detecting it. Mutation score is a separate signal from ordinary line or branch coverage.
Release evidence
- Test file, protected behavior, and independent requirement.
- Finding rule, severity, file, line, and a minimal excerpt or description.
- Assertion and async-completion disposition.
- Mock and snapshot disposition.
- Missing nominal, boundary, invalid, or failure case.
- Focused test command and result.
- Deliberate probe or mutation and the test that failed.
- Known heuristic exceptions, reviewer, date, and commit.
Restore the production file after a manual probe and verify a clean diff. Do not run destructive or broad mutation commands against an uncommitted worktree without a recovery point.
Turn the first-pass review into a queue
Test Quality AuditorRead-only heuristic scan for no-assertion tests, trivial checks, tautologies, mock-only evidence, snapshot-only tests, placeholders, and over-mocking across Jest, Vitest, pytest, and unittest styles. It reports review targets; it does not replace behavior review or mutation testing.See the test audit skill →Common questions
Can a test suite have high coverage and still be weak?
Yes. Coverage records execution. It does not prove that assertions distinguish correct behavior from an incorrect result. Pair coverage with behavior review and a mutation or deliberate-change probe.
How do I find tests with no effective assertions?
Inventory explicit assertions, awaited promise checks, callback assertion counts, snapshots, and mock verifications. Trace the actual and expected values to confirm the comparison is independent and behavior-specific.
Are snapshot tests bad?
No. Keep them small, readable, committed, and reviewed. Add explicit assertions for the important contract and do not accept a generated snapshot merely because it makes the suite green.
Does Test Quality Auditor run mutation testing?
No. It performs a fast static heuristic review of supported test files. Run Stryker or another mutation tool separately when you need counterfactual evidence.
Can an automated audit certify that my tests are good?
No. It can identify suspicious patterns and structure a review, but it cannot infer every business rule or prove that the suite detects every meaningful defect.