This page describes how the classifier behind AI Detector Checker was built and tested, and the discipline that decides whether a change is allowed near production. It is deliberately specific about the two parts that determine whether a published number means anything: how the data is split, and what a change has to prove before it ships. The engine currently in production is described on the methodology page.
What this page covers
The document classifier in production today is the V2 fusion core, unchanged since 4 September 2026 and reused without retraining by engine V3. Everything below describes how that core was built and validated. V3 added new frozen length-banded decision thresholds and a frozen 0-100 display map on top of it; those were fixed before the sealed confirmation and are published on the methodology page.
The corpus
A lineage audit of every text corpus in the project, run before any V2 training, counted 527,455 unique human documents and 92,251 unique AI documents after normalisation and de-duplication — against 4.9 million raw rows. That gap is the point of the audit: apparent scale collapses once the same corpus under two names, derived instruments and containment relationships are counted once.
The audit also found the binding constraint. Of those human documents, about 95 in every 100 came from ten source families — so the corpus was large but narrow, and a detector trained on it would be graded mostly on ten kinds of writing. The acquisition work that followed was aimed at source families rather than at document counts: reviews, non-native English, procedural and technical writing, assistant-style replies, parliamentary and congressional records, medical question-and-answer text, radio transcripts and opinionated news, because those are the registers a detector is most likely to misread.
Human text comes from established public research corpora. AI text comes from public machine-generated-text corpora and first-party generations across many model families, including deliberately obfuscated variants — paraphrase, synonym substitution, look-alike characters, zero-width space injection and others — so that evasion is measured rather than assumed.
How the data is split — the part that matters most
The easiest way to publish a flattering, meaningless number is to split evaluation data at random. Passages from the same author, the same article or the same generation run then land on both sides of the split, and the model is graded partly on material it has effectively already seen.
- Human text is separated by source family first, then author or work. Two texts from the same publication or writer never straddle the boundary.
- AI text is separated by generator lineage, so variants of one underlying model stay together and a family used in fitting cannot reappear disguised as an independent test.
- Human data is divided into three non-overlapping roles: one part to fit on, one to choose an operating point on, and one reserved purely for safety validation and not looked at until every parameter is frozen.
That third tier exists because of a specific, expensive lesson. A safety threshold set as “just above the highest score any human in the sample reached” is arithmetically bound to produce zero false positives on that same sample; the number is forced by construction and carries no information. One earlier candidate passed exactly that way and failed on unseen sources. The safety tier is now sealed until it is too late to tune anything against it.
The seed-max rule
V3’s decision thresholds were fitted per length band across several random seeds, and the value taken forward for each band was the maximum across those seeds, not the mean or the best. Taking the maximum makes the threshold harder to cross, which costs recall and buys a lower chance that a favourable seed is what produced a safe-looking false-flag rate. The per-seed and per-fold values behind each published threshold are kept in the frozen thresholds file.
The sealed one-shot confirmation
Before a release is measured, the whole test is written down: the exact decision rule, the evaluation sets, every pass/fail bar and the required effect size. That description is hash-stamped. The confirmation then runs once against the frozen description, and the surfaces it used are marked consumed and never scored again.
What the protocol forbids is the part that matters: no threshold, band, display map or candidate may be changed after the sealed surfaces have been opened, and no bar may be adjusted after seeing the outcome. If a bar is missed, the options are to ship with the miss published or not to ship — never to move the bar. V3’s confirmation missed two of its six bars, and the publisher chose to ship with both published rather than re-tune. The result of that run, exactly as it came out, is on the evidence page.
What a change has to prove
| Requirement | Why |
|---|---|
| No new wrong AI signals on the reserved human data | A wrongly flagged person is the costliest failure this tool has. Measured only on source families never used to fit or tune. |
| A real, pre-declared gain on AI text | Measured on generator families kept disjoint from those used in fitting. |
| No regression | No existing correct detection may be lost. |
| Exact preservation of existing safeguards | Verified byte-for-byte, not argued. |
| Monotonic behaviour | An adjustment meant only to raise scores must be shown never to lower one, across the whole corpus. |
Robustness is tested by leaving a whole human source family out of fitting and then scoring on it, by refitting with an entire generator family removed and then scoring on it, by measuring a signal where it will actually act rather than on a broad average, and against the obfuscation families above. The honest summary of the track record is that most proposals fail: across the eight most recent evaluation rounds exactly one produced a change that reached production, three closed with no candidate at all, and two produced candidates that then failed independent validation on data they had not been tuned against.
Release discipline
Every artefact the running service depends on — the model file, the thresholds, the display map, the pattern module — is hashed, and the service verifies those hashes at start-up and refuses to serve if any of them differs. Each release carries an id that appears on every result card and in the model update history. Rollback is a single command that restores the previous release exactly.
The release harness replays a fixed set of English reference documents and known control cases against the live service after every deployment, and compares the output with the recorded expected result. Non-English text is not part of that harness: it is unsupported, it is not tested for release, and no claim is made about it anywhere on this site.
What this does not mean
None of this makes a result proof. A careful protocol tells you that a published number was measured honestly on data the system had not seen; it does not tell you who wrote the text in front of you. The engine misses much AI-written text, it flags some human writing wrongly at the published rates, and it never identifies which model produced a passage. Read a score as a reason to look at a text more closely, and nothing more.