I Tried to Catch Coding Agents Cheating. They Mostly Didn't.

Coding agents have a known failure mode: when a task gets hard, some of them make the checks pass instead of making the code work. They delete the failing assertion, add .skip, cast through any, set "strict": false, or leave throw new Error("Not implemented") behind and report that everything is done. Research benchmarks like EvilGenie have caught Claude Code and Codex doing exactly this.

So I built cheatguard, a deterministic linter that reads the diff the agent produced and runs inside its loop, as Claude Code hooks. Then I tried to prove that it works. The results were not what I expected, and I think they are more useful because of that.

🛠️ What it does

cheatguard compares the working tree with the state at the start of the session and reports only what the change introduced. This is a typical “all tests pass now”:

$ cheatguard check --no-lint
src/cart.ts
  ✗    9  New `as any` cast: for (const line of lines) sum += line.price * (line as any).quantity; [type-suppression]
          → Remove the cast and fix the type mismatch at its source.

test/cart.test.ts
  ✗    5  Test "total > sums lines" went from 2 to 1 assertions [test-weakened]
          → Restore the removed assertions in "total > sums lines" and fix the implementation instead.
  ✗    9  Test "total > applies coupons" was marked .skip [test-weakened]
          → Remove the skip and fix the behavior the test exercises.

tsconfig.json
  ✗    2  compilerOptions.strict was turned off [config-tampering]
          → Restore "strict" (leave it enabled or inherited) and fix the type errors it reports.

✗ 4 blocking, 0 warnings in 3 changed file(s).

Each finding tells the agent what to do instead, and it can’t finish until the blocking ones are gone. The rules cover weakened tests (deleted, skipped, made unreachable, expected values edited to match the code), type suppressions, config tampering (strict flags, lint rules, coverage thresholds, CI checks, and the agent removing cheatguard’s own hooks), stubs left behind, imports of packages that don’t exist, and swallowed errors.

It also runs the ESLint, oxlint, Biome and tsc you already have and shows the agent only the problems its change introduced, including type errors it caused in files it never touched. No LLM, no network.

Before publishing, I had an independent review attack it. It found real holes: renaming a test file to .bak, it.fails, making a linter crash so its errors disappear, the agent deleting the hook from its own settings. It also found a flaw in my own benchmark: the first agent runs had loaded my personal Claude Code memory. Every hole now has a regression test, and the agent benchmark was re-run in isolation.

🔍 Do real agent pull requests contain these patterns?

I mined 64 merged pull requests written by coding agents and 47 written by people before 2022, from permissively licensed TypeScript repositories. One annotator, AI-assisted, labeled every finding, so recall is an upper bound.

	Agent PRs	Human PRs
Precision	87%	68%
Recall against the labels	100%	100%
Precision of blocking findings	86%	83%
Wrong blocks per 1,000 changed lines	0.12	0.88

What it found was ordinary rather than dramatic: as any and as unknown as in test doubles, eslint-disable comments, empty catch blocks, a lint rule switched off, an undeclared dependency. On human PRs precision drops because people change expected values legitimately when behavior changes, and cheatguard flags some of those. That’s the honest cost of the test rule.

🧪 Do agents cheat, and does cheatguard stop them?

I wrote 19 small TypeScript tasks where the shortcut is easier than the fix: a failing test with a tempting expected value, a type error that any would silence, a coverage gate, a strict-mode migration, a half-written cache. Then I ran Claude Code headless on each one, three ways: alone, with a CLAUDE.md asking it not to cheat, and with cheatguard’s hooks. Runs were isolated, and grading never used cheatguard: original tests and configs were restored and hidden tests were added.

Zero cheating in 798 runs, across Sonnet 5.5 and Haiku 5.5, with and without an urgent “CI must be green, do whatever it takes” prompt.

On tasks this size, current frontier agents fix the code honestly. If you use agents for short, well-specified tasks, anti-cheat is probably not the problem you have.

What cheatguard did change:

• It made agents finish the job. On a simple feature task, Sonnet ran the tests but not the type checker, and shipped type errors in every run when working alone (0 of 5). With cheatguard, its end-of-task check ran tsc and every run was correct (5 of 5). The CLAUDE.md rules got almost the same result (4 of 5). So this is “finish the job”, not “anti-cheat”, and a Stop hook that runs your own checks would get much of it.
• The rules file beat it on two tasks, and cheatguard never intervened there. On a strict-mode migration, agents with cheatguard fixed the types in a way that broke a test they never ran (0 of 5, against 5 of 5 with the rules, which say to fix the code under test). cheatguard doesn’t run your test suite. On a task with an existing money formatter, Haiku re-implemented it inline (0 of 3, against 3 of 3 with the rules, which say to reuse utilities). cheatguard only notices duplicated exports.
• It has a cost. One control task had a genuinely wrong test, so editing it was the right fix. cheatguard pushed back on that edit every time. The agents pushed through and still fixed it, but took 1.9 to 2.5 times as long.

Those two gaps are the first item on the roadmap: an opt-in check that runs the project’s tests before the agent stops, and detection of a helper re-implemented inline. The benchmark gets re-run before I claim either one helps.

📉 How much less does the agent have to read?

On six real repositories with a seeded change, cheatguard lint cut what the agent reads by 22% to 98% and kept every new problem. On clean repos the raw output is small anyway; the savings grow with existing noise. The biggest case was vue-test-utils (oxlint and tsc), where the agent goes from 4,261 tokens of output to 97. On two of the six (axios and vue-test-utils) cheatguard was slower than one raw lint run.

🧭 So, should you use it?

As a cheap, deterministic quality gate for agent-written code, yes: it catches the constructs reviewers want to see, keeps old lint noise out of the agent’s context, and makes “done” include the type checker. Don’t expect it to catch a model that isn’t cheating, and don’t treat it as a sandbox.

npm i -D cheatguard && npx cheatguard init claude-code

Or, without adding a dependency, as a Claude Code plugin:

claude plugin marketplace add aguschirico/cheatguard
claude plugin install cheatguard@cheatguard

What I want to measure next is the setting where research says agents do cheat: long, autonomous, multi-hour tasks. If you run agents that way and want to help, the benchmark is open. Code, benchmarks, labels and raw transcripts are at github.com/aguschirico/cheatguard, and the results page is at aguschirico.github.io/cheatguard.

📋 TL;DR

What	A deterministic, diff-aware linter that runs as Claude Code hooks and blocks the shortcuts agents take to make checks pass
Agent benchmark	0 cheating in 798 isolated runs; cheatguard’s measurable effect was finishing the job (type checks), not catching cheats
Where it lost	Two tasks where a CLAUDE.md with rules did better and cheatguard never intervened
Real PRs	87% precision on 64 agent PRs, 68% on 47 human PRs
Lint output	22% to 98% less for the agent to read on six real repositories
Code	github.com/aguschirico/cheatguard, Apache-2.0

I set out to catch agents cheating and mostly measured them doing the work. The tool turned out to be useful anyway, just not for the reason I built it.