AI Writes the Code Now — Test Automation Matters More Than Ever

AI writes a feature, you’re happy. Two weeks later another task edits shared logic and the old feature silently breaks — you only find out from a user bug report. That’s regression: the defect isn’t in the writing, it’s in the changing. Code Review is a point-in-time gate; to guard the future you need a gate that keeps running.

First, a correction: AI makes testing matter more

Test automation was already one of the most important regression defenses long before AI-assisted coding. With AI, a common reaction is the opposite: “I don’t write the code anymore, so why should I worry about tests? The AI writes them.” That conclusion is backwards. Three structural shifts point the other way:

One line: AI drove the cost of writing tests down, and the risk of not having them up.

Approach: a 3-layer regression gate

Treat “add tests for this task” as a first-class quality action and front-load it at the same trigger point as review — after the feature builds and lints, before “done.”

We pick three layers, with e2e kept extremely restrained: unit and integration form the everyday “always-running” body; e2e only covers the 1–3 paths users cannot lose.

• unit — pure functions, logic, edge cases. Fastest, cheapest; cover normal + boundary input (empty, out-of-range, exception branches).
• integration — cross-module and external contracts: DB read/write, API contracts, CLI/GUI parity, message formats. Guards that “the secret handshake between modules hasn’t changed.”
• e2e — only critical user paths (e.g. “login → core action → result”), run with a cheap model in CI, count ≤ number of critical paths.

The key move: humans plan, AI executes

Split “adding tests” into two layers:

• Execution — writing a runnable unit test, adding parametrized cases, wiring fixtures, fixing lint. AI is already excellent at this, often faster and more consistent than a human. Hand it over; don’t do it yourself.
• Planning — which surfaces does this module need covered? Which path is critical? How much e2e is enough? What makes a test valid? The AI doesn’t know by default; it guesses at “looks reasonable” — which produces random coverage, missing boundaries, and a pile of fake-green tests.

So the leverage is: humans freeze the plan into a written guideline; AI executes against it. The guideline buys two things — comprehensiveness and controllability.

Comprehensiveness = a coverage checklist, not AI improvisation

Don’t say “add some tests for this function.” Say “walk these 7 items; if one doesn’t apply, say why in the PR”:

1. Normal input → an explicit expected value (not “it runs”)
2. Boundary input: empty / 0 / 1 / max / out-of-range / very large & very small
3. Illegal input → assert the specific exception type and message
4. Failure paths: dependency timeout, partial failure, retries exhausted
5. Idempotency: repeated calls produce the same result
6. No state leakage: tests are independent, order-agnostic, parallel-safe
7. Contract stability: field names, ordering, error codes, message formats

Controllability = the test must be able to fail

A test that cannot fail isn’t a guarantee, it’s decoration. AI is especially good at producing those, so hold the line:

• Mutation sanity check — before delivery, deliberately break the implementation a little (flip a condition, delete a range check, change a constant). The test must go red. Still green? It guards nothing; rewrite it.
• ❌ No assertion-less tests / bare assert True / print-only tests
• ❌ Never mock the unit under test (only its external dependencies)
• ❌ Never copy the implementation into the assertion (assert result == impl(x))
• ❌ No sleep(n) to wait for async results (inject a clock or wait explicitly)

The guideline: a TESTING_GUIDELINE.md you can drop in your repo

# TESTING_GUIDELINE.md (read before adding tests — humans and AI both)

## Layer quotas
- unit 70% / integration 20% / e2e 10% (by case count, not by importance)
- e2e covers only the 1–3 "users absolutely cannot lose" paths; justify any addition

## Coverage checklist (answer each per unit; note any skip in the PR)
1. Normal input → explicit expected value
2. Boundaries: empty / 0 / 1 / max / out-of-range / very large & very small
3. Illegal input → assert specific exception type and message
4. Failure paths: timeout, partial failure, retries exhausted
5. Idempotent: repeated calls give the same result
6. No state leakage: independent, order-agnostic, parallel-safe
7. Contract stability: field names / order / error codes / message formats

## Validity red lines (violation = PR rejected)
- ❌ No assertions / bare assert True / print-only
- ❌ Mocking the unit under test (external deps only)
- ❌ Assertions copied from the implementation (tautology)
- ❌ sleep(seconds) to wait for async
- ✅ Mutation sanity check before delivery: break the impl → test must go red

Code: assertion-style unit tests cover boundaries

import pytest

def price_after_discount(base: float, pct: float) -> float:
    if pct < 0 or pct > 1:
        raise ValueError("pct must be in [0, 1]")
    return round(base * (1 - pct), 2)

def test_price_boundary():
    assert price_after_discount(100, 0.1) == 90.0
    assert price_after_discount(100, 0.0) == 100.0
    assert price_after_discount(100, 1.0) == 0.0
    for bad in (-0.1, 1.1):
        with pytest.raises(ValueError):  # assert the exception is actually raised
            price_after_discount(100, bad)

The last block isn’t “just run it and see” — it explicitly asserts the exception is raised. A test that only runs green without asserting isn’t holding the boundary.

Mutation sanity check: prove the test can fail

  def price_after_discount(base: float, pct: float) -> float:
-     if pct < 0 or pct > 1:
-         raise ValueError("pct must be in [0, 1]")
+     pass                       # mutation: drop the input validation
      return round(base * (1 - pct), 2)

# Expected: test_price_boundary must FAIL.
# If it still passes → the test is fake-green. Rewrite it; don't just revert the impl.

This is where “controllability” becomes real: it turns “I think it’s tested” into “I proved this test catches that error.”

Enforced rules

Layers & flow:

1. Tests enter the same PR as code — front-load, don’t cram before release.
2. e2e count restrained — as many as there are critical paths, no more.
3. Gate green before review — machine first, then human review on blind spots.
4. Run tests with cheap models / CI (echoes model tiering) — don’t burn premium quota.

Constraints on the AI:

1. Guideline first, tests second — no TESTING_GUIDELINE.md, no unsupervised test generation.
2. No assertion-less tests.
3. Mutation sanity check before delivery — if breaking the impl doesn’t turn it red, it’s fake-green.
4. Never mock the unit under test — external dependencies only.

Outcome (qualitative)

• Regression rate: caught before merge, not in production.
• Control restored: you don’t write every line anymore, but every change gets a machine restating which existing behaviors are still intact.
• Trustworthy tests: the mutation check filters “a pile of green tests” down to “tests proven to catch errors.”
• Review burden: lighter — the machine pre-filters routine regression; review focuses on design/consistency.
• Cost: transparent and near-zero, which is why it can run as often as review.

Reproduction checklist

• Drop a TESTING_GUIDELINE.md at the repo root (layer quotas + coverage checklist + red lines); read it before adding tests.
• Pick a feature you’ll keep changing; list 1–3 “must-not-break” critical paths (e2e scope).
• Extract pure logic into testable functions; cover normal + boundary + exception branches.
• Add integration tests for cross-module / external contracts (DB, API, CLI-GUI parity).
• Add restrained e2e only for critical paths, run with a cheap model in CI.
• Run a mutation sanity check before delivery: break the impl, the test must go red.
• Tests enter the same PR; gate green, then independent review.

Conclusion & further reading

Code Review guards this change; tests guard the next one. And once AI writes the code, the gate matters more: plan with a guideline, let AI execute, keep e2e tiny, and prove the tests can fail. Pair it with Independent Code Review front-loading for the full pre-merge quality loop.

Note: this is a methodology write-up with qualitative observations; quantitative metrics (regressions caught pre-merge, suite runtime per PR, test-related credit share, fake-green hit rate from the mutation check) are best measured in your own setup. The TESTING_GUIDELINE.md above is a template to adapt — the quotas and checklist should match your project’s shape.