Your AI Reviewer Says It Read CLAUDE.md. Here’s How to Prove It.

The review comment landed with total confidence: “This violates Rule 4 in CLAUDE.md,” with a file name, a line range, and a link you could click. It was right. The problem came three days earlier, when the same tool looked at a different pull request in the same repository, one that broke a rule written into both CLAUDE.md and AGENTS.md, and said: “No actionable comments were generated. Merge risk: Minimal.”

If that reads like a tool problem, it isn’t. It’s a measurement problem. You cannot tell the difference between a reviewer that read your rules file and one that happens to agree with you, and most teams never try.

If you haven’t met these files yet: CLAUDE.md and AGENTS.md are plain-text rule files that sit in a repository root and get handed to coding agents and AI reviewers as context. They are how you tell a bot “we do it this way here” — naming conventions, security rules, forbidden imports. Almost every AI review tool now claims to read them, and Anthropic documents the format in Claude Code’s memory docs.

Why “does it read CLAUDE.md?” is the wrong question

Ask a vendor and every one says yes. Test it badly and they all look fine, because most of the rules teams write down are rules any decent scanner would flag anyway. Don’t log request bodies. Don’t commit secrets. Escape user input. Those are universal best practices wearing a house-rule costume. A tool that never opened your file can catch every one of them, and you will never know.

The only rule that proves anything is one with zero external backing — a convention that exists nowhere except your repository. If a tool flags that, it had to read the file, because there is nothing else it could be catching.

The two-rule test

In a write-up that circulated on dev.to recently, a developer built exactly this experiment on a small Cloudflare Workers API and ran two review tools, CodeRabbit and Qodo, against it. The setup is worth copying because the two planted rules are deliberately different in kind.

Rule A: the rule with a built-in excuse

Both files said: never log raw request headers or bodies. The test PR logged both, with a commit message calling it temporary debugging. Both tools caught it immediately — which tells you nothing at all. Raw header logging is a well-known bad practice, flagged on instinct by anything with a security ruleset, whether or not it read a single line of your documentation.

Rule B: the rule with no excuse

Both files said: all new route paths must be kebab-case (/feedback-summary), never camelCase — described in the file as an internal convention and explicitly not a general best practice. The test PR added a camelCase route. Nothing about it is a security issue, and nothing about it is a bug.

Qodo caught it right away and cited AGENTS.md[7-10] and CLAUDE.md[7-10]. Then it went further: because that handler falls through to a 404 on unmatched paths, a client calling the correct kebab-case URL would break while the wrong, camelCase one worked. That isn’t pattern-matching against a style guide. That’s tracing the consequence of a naming mismatch once real requests hit it.

CodeRabbit, on its default “Chill” profile, returned nothing. On its stricter “Assertive” profile it did flag the naming — but without naming a rule, a file, or a line. Just “the required kebab-case path,” stated as fact, as if everyone already knows.

And here is the honest part the original author admitted: kebab-case REST routes really are a common convention. CodeRabbit’s catch could be explained by a stricter dial reaching for something it already knew. The test was one notch less clean than intended. Steal the method, then close that hole.

Make the arbitrary rule truly arbitrary

Pick something no model has ever seen in public training data, because it is yours. Rules that work:

• Every new queue name must end in -v2 until the legacy consumer is retired.
• Money is stored as integer minor units, and the variable name must end in _cents.
• No file outside src/legacy/ may import from src/legacy/.
• Every new environment variable must start with your company prefix.

Write it into both files and add one sentence explaining why the rule exists. That sentence matters: a tool that quotes your rationale back is almost certainly reading it.

The 20-minute audit protocol

1. Add one arbitrary rule to CLAUDE.md and AGENTS.md, committed on its own so the diff is a single-file change.
2. Open two PRs. The first breaks Rule A, a universal best practice. The second breaks only your arbitrary rule. Both small, both green in CI, and the second containing no other plausible complaint.
3. Request a full review, not a summary, and don’t hint at what you planted.
4. Check the citation, not the verdict. Does the finding name a file, a line range, and ideally a rule? Click through and confirm the rule says what the tool claims.
5. Repeat under each profile or severity setting the tool offers, and note which dial produced the catch.
6. Re-run the same pair in 90 days. Models and rulesets change under you without a changelog. Your planted rule is the only fixed point.

Score it like an exam, 0 to 3

• 0 — silence. Nothing references the rule. The tool is not reading the file, whatever the dashboard claims.
• 1 — catch without attribution. It flagged the change, but you can’t check its work. This is the most common outcome and the least useful.
• 2 — catch with a citation. Rule, file, line. Verifiable in ten seconds.
• 3 — catch with a consequence. The tool explains what the violation does once real traffic hits it, as Qodo did with the 404 fallthrough.

Only level 2 and above is auditable. Levels 0 and 1 both leave you taking the tool’s word for it — which is the exact thing you were trying to avoid.

Measure the noise, or you’ll turn it off

Precision is the other half, and teams skip it. A reviewer that catches everything because it complains about everything isn’t a reviewer; it’s a notification stream. So run a control: push three clean PRs through the same settings and count the comments. A rough rule of thumb for a tool you’ll actually keep is under two comments per hundred changed lines on clean code, with no more than one in three of those being style-only. If the strict profile gets you the citation but buries it in noise, you’ve learned something real about that profile’s cost — and you now have numbers for the vendor conversation instead of vibes.

Put the test in procurement and in your process

Two cheap changes. First, when you evaluate or renew an AI review tool, refuse the canned demo. Hand the vendor your repository and your rules file, and ask them to review a PR you already seeded. A tool that can’t find a rule you planted in your own repo will not find the rules your team writes next quarter.

Second, treat CLAUDE.md, AGENTS.md, and the reviewer’s tool and permission configuration as production artifacts. They live in git, they change what ships, and in most repos they get less attention than a .prettierrc file. If your team now reviews agents instead of diffs — and on large generated PRs, it does — then the rules file is the spec, and the spec deserves the slow read the diff used to get.

What this doesn’t prove

Be honest about the limits. A citation proves a rule was surfaced, not that it was weighted correctly. A tool can quote your line and still be wrong about whether the change is dangerous, and it can pass your planted test while missing the bug that actually matters.

The audit also isn’t a substitute for behavior-level checks. If the same agent writes the feature, writes the rule file, and grades the result, it can make the same mistaken assumption in all three places and hand you a green checkmark. Planting a rule tells you whether the reviewer reads. It cannot tell you whether the software works — for that you still want independent, executable specifications, the kind that catch the password reset link nobody said had to be single-use.

Key Takeaways

• “Yes, we read CLAUDE.md” is unverifiable. A planted, repo-specific rule is verifiable in twenty minutes.
• Use two rules: one universal best practice (any scanner catches it, so it proves nothing) and one arbitrary house convention (only a file-reader can catch it).
• Score the catch: 0 silence, 1 catch without attribution, 2 citation with file and line, 3 citation plus consequence. Only 2 and 3 are auditable.
• Record which profile produced the catch. A stricter style dial can find the issue without ever reading your file.
• Measure false positives too — under two comments per hundred clean lines, or your team will mute the tool.
• Re-run the test quarterly and at renewal. Models change silently; your planted rule is the fixed point.
• A citation proves a rule was surfaced, not that the verdict is correct. Keep independent behavioral tests for correctness.