Your Gate Thresholds Are Guesses: Grade Code Changes by Percentile, Not Magic Numbers

Open your CI configuration. Find the line that decides whether a change is allowed to merge — something like block_at: 200000. Now try to remember who chose that number. If you cannot, you are in good company. Almost nobody can, because almost nobody chose it. It arrived in a default, survived three refactors, and has been silently grading your team ever since.

That is not a small thing. A quality gate is a policy: it says this change is too expensive to accept. A policy with an unowned number is not a policy. It is a coin that landed heads years ago and stayed that way.

Absolute thresholds are guesses wearing a lab coat

Here is the uncomfortable property of almost every code-quality metric: the typical value varies by orders of magnitude between languages, repositories, and sometimes between folders in the same repository. A change that is loud in a 40-file TypeScript service is background noise in a C++ monorepo with 12,000 files.

So when you hardcode a threshold, you are doing one of two things. Either you tuned it once against your repository as it existed on a Tuesday in 2024, and it has been drifting ever since. Or — much more often — you picked a round number because it looked reasonable in the sprint where somebody had to pick one.

That is the actual failure mode. Not that the number is wrong. That the number has no owner and no derivation. When a pull request gets blocked at 2am, we set it to 200,000 is not an answer anyone can argue with, debug, or fix.

A concrete example: measuring structural decay

To make this less abstract, take a real tool with a published formula. ImpactGate is a CI gate that scores a change by how much structural decay it introduces, where decay means complexity accreting into structures that already exist. Its measure is:

impact = files_changed × Σ max(WMC_other, 1) × CC × Δlines

summed over the changed functions. Translated out of math: WMC_other is the complexity already sitting in the class or file you are editing, measured before your change. CC is cyclomatic complexity — roughly, the number of independent paths through the function you wrote. Δlines is how much you added or removed.

The interesting part is what the formula makes cheap versus expensive:

• A brand-new file or class costs almost nothing, because WMC_other is zero when nothing was there before.
• Adding one more branch to an already-heavy class is expensive, because the existing weight multiplies your new lines.
• The score scales with how many files you touched, so a sprawling change costs more than a focused one of the same size.

That is a defensible definition of decay. A god-class gaining another method is exactly the thing you want to notice, and this scores it higher than a clean new module — which matches what your instincts already tell you.

The number is still the hard part

Now the tool ships with defaults: warn at 50,000, block at 200,000. Those are reasonable. They are also, for your repository, a guess. The people who built the metric knew that, which is why they built a second mode.

Instead of a raw threshold, you can grade a change by its percentile against a distribution, and gate on the grade. You build a baseline from your repository's own merged history, then run with --curve --warn-percentile 90 --block-percentile 98. The gate now asks where a change falls in the distribution instead of comparing it to a constant.

The grade blends two distributions. One is a seed prior shipped with the tool: percentile tables built from a corpus of roughly 20 open-source repositories, with a pooled fallback for languages not in the table. The other is your project baseline, walked from the merged mainline — only landed work, never in-flight branches.

The blend is weighted by w = n / (n + K), where n is the number of landed changes behind your baseline and K defaults to 200. Read that as a trust dial:

• 0 landed changes: you grade entirely on the seed corpus.
• 50 changes: w ≈ 0.20 — one-fifth your history, four-fifths the prior.
• 200 changes: w = 0.50 — an even split.
• 1,000 changes: w ≈ 0.83 — your own history dominates.

K = 200 is the statement that it takes about 200 landed changes before we trust you as much as we trust strangers. You can argue about 200. But notice the difference from before: 200 is a documented prior with a stated purpose, not a magic constant. You can move it and explain why.

What actually changes when you gate on a percentile

The behavioural difference is larger than it sounds. A raw threshold answers is this change expensive in absolute terms? A percentile answers is this change unusual for us?

Set warn at the 90th percentile and block at the 98th, and you have made a claim you can check: roughly one in ten changes should warn, and roughly one in fifty should block. If your block rate is 15%, then either the gate is mis-set or your codebase is genuinely decaying faster than you thought. Both are useful. With a fixed number you never learn anything — you just get a stream of exceptions and an approval culture that rubber-stamps.

There is a second benefit that matters more for long-lived systems: the baseline moves. If your team pays down debt for a quarter, the whole distribution shifts down and the gate keeps its pressure. A fixed threshold quietly loosens every time you improve, because the same absolute number now describes a worse-than-average change.

Why this matters more now, not less

Generation got cheap. Review and maintenance did not. The METR study from 2025 found experienced open-source developers using AI tools took 19% longer, even though they believed they were faster — a reminder that the visible metric (how quickly a draft appeared) and the real one (how expensive the next change will be) can point in opposite directions.

As per-developer output rises, every gate that fires on volume gets noisier, and every gate calibrated against last year's distribution starts blocking ordinary work. That is an argument for percentile grading, not against gating. You need something that re-derives its own expectations from what your team is actually shipping this quarter.

The blind spots, stated plainly

None of this makes the metric true. Here is what it cannot see:

• Split changes. Because impact multiplies by files and by the weight of what you touched, breaking one risky change into five sequential pull requests can lower every individual score. The gate measures per-change decay, not cumulative decay. Per-file rankings — which the tool reports, ordered by each file's share of the total impact — are your defence, because a file quietly growing into a god-class surfaces as a candidate long before it blocks anything.
• Generated and vendored files. A diff larger than a configured limit (200,000 lines by default) is almost always a generated dump or a vendored blob, so it is skipped and explicitly listed as skipped. That is the honest move — it stops a minified bundle from distorting the score — but it also means a huge generated diff riding inside a real change carries no weight.
• Stale baselines. After a major refactor or a monorepo split, your distribution describes a repository that no longer exists. Rebuild it.
• Everything semantic. The formula counts structure, not risk. A one-line change to a payment condition scores near zero and can still be the most dangerous commit of the month.

That last one is why this belongs next to code review, not instead of it. A gate like this is a triage tool. It decides what deserves a slow, careful read. It is not a verdict on correctness, and you should not sell it to your team as one.

A 30-day rollout that will not blow up your CI

1. Week 1 — pick one metric and run it warn-only. No blocking. Report every change's score on the pull request and let people see it.
2. Week 2 — build the baseline from merged history. For an impact-style measure that means walking the mainline and writing a baseline file. Re-run it as the branch moves, not once.
3. Week 3 — read the shape, not the outliers. Where does your median change sit? How heavy is the tail? If you have fewer than about 200 landed changes in the window, know that you are still mostly graded on the seed prior.
4. Week 4 — turn on blocking at the 98th percentile, warning at the 90th. Write the numbers into the repository config with a named owner and a date. The owner's job is not to defend the number. It is to re-derive it quarterly.
5. Ongoing — watch the per-file list, not just the count. The files with the largest share of impact are your refactor queue, and they are usually a better use of a sprint than the next roadmap item.

Start on warn. The failure mode of a gate that is too aggressive is not that people fix their code — it is that people route around the gate, and then you have lost the signal entirely.

Key Takeaways

• If you cannot name who chose a gate threshold, it is not a policy. It is a leftover default.
• Absolute thresholds are unstable because typical change size varies by orders of magnitude across languages and projects.
• Percentile gating answers a better question: is this change unusual for us?
• Blending a prior with your own history using w = n/(n+K) makes the question of how much you trust your own data explicit instead of hiding it in a constant.
• Warn at 90 and block at 98 gives you a checkable prediction: about 1 in 10 changes warns, about 1 in 50 blocks. If reality disagrees, that is information.
• Know the blind spots: split changes evade per-change scores, generated files are skipped, baselines go stale, and nothing structural captures semantic risk.
• Pair the gate with per-file rankings so you see the god-class forming before it blocks anyone.