This page describes how the detector behind Detector Checker is actually built, tuned and tested, with the figures from the evaluation corpus used for the current engine. It is deliberately specific about the parts that decide whether a published result means anything — chiefly how the data is split, and what a change has to prove before it is allowed anywhere near production.
Scale of the evaluation corpus
The corpus assembled for the most recent validation work contains:
| Documents | |
|---|---|
| Total scored documents | 305,234 |
| Human-written | 205,748 |
| Machine-generated | 99,486 |
| Distinct text generators represented | 51 |
| Distinct writing registers | 61 |
| Languages present | 14 |
| Adversarial transformation types | 12 (plus unmodified) |
For the most recent human-safety work a further 457,837 human documents were acquired from public sources and reduced to a working pool of 400,000 by a deterministic, seeded, source-proportional trim. That pool was screened to find the specific documents capable of exercising the behaviour under test.
Where the text comes from
Human writing
Human text is drawn from established public research corpora, chosen to spread across the registers that actually change a detector’s behaviour rather than to maximise volume:
| Register | Source |
|---|---|
| News and journalism | AG News; CNN/DailyMail; XSum |
| Reviews | IMDb reviews |
| Professional correspondence | Enron email corpus |
| Technical Q&A | StackExchange (physics and statistics) |
| Patents and technical writing | BigPatent |
| Poetry | PoetryDB — 2,526 complete public-domain poems by 127 poets |
| Literary prose, oratory, encyclopedic and academic writing | Public-domain and open literary and reference sets |
Poetry, formal oratory and literary prose are included deliberately. They are not common in real traffic, but they are the registers a detector is most likely to misread as machine-written, so leaving them out would flatter the results.
Machine writing
Machine text comes from public machine-generated-text research corpora spanning 51 generator families — open-weight and hosted, base and chat/instruction-tuned, from earlier generations through to current models. Represented registers include news, academic abstracts, arXiv papers, books, wiki articles, recipes, forum answers, medical and financial writing.
Crucially, the corpus contains machine text that has been deliberately manipulated to evade detection. Twelve transformation families are represented, at roughly 2,000 documents each in the most recent instrument:
| Transformation | Documents |
|---|---|
| Unmodified (control) | 7,985 |
| Paraphrase | 2,014 |
| Synonym substitution | 2,009 |
| Numeric perturbation | 2,006 |
| Insert paragraph breaks | 2,002 |
| Homoglyph (look-alike characters) | 2,001 |
| Whitespace manipulation | 2,000 |
| Alternative spelling | 1,999 |
| Perplexity-raising misspelling | 1,997 |
| Article deletion | 1,996 |
| Upper/lower case manipulation | 1,996 |
| Zero-width space injection | 1,995 |
Cleaning: duplicates and prior exposure
Two things quietly inflate detector results, and both are removed before anything is measured.
Duplicates. Public corpora overlap heavily and repeat themselves. In building the most recent human instruments, 33,471 exact duplicates and 2,890 normalised duplicates were dropped from one pool, and a further 18,627 from another. On the machine side, 24,509 duplicates were removed.
Prior exposure. Every candidate document is checked against a firewall of everything the project has previously scored or used. In the most recent build this removed 52,137 machine documents and several thousand human documents that had appeared in earlier work — because a document the system has already been measured on is not evidence.
One entire public corpus is excluded in full, because it is the source distribution of an evaluation set held in reserve. That single exclusion discarded 122,499 otherwise usable documents.
Sampling is also capped per cell rather than taken wholesale — roughly 190 documents per generator/register/transformation combination, with a limit on how many may share a common origin — so no single prolific source can dominate a result.
How the data is split — the part that matters most
The easiest way to publish a flattering and meaningless number is to split evaluation data at random. Passages from the same author, the same article or the same generation run then land on both sides of the split, and the model is graded partly on material it has effectively already seen.
This project splits by group, never by passage:
- Human text is separated by source first, then author or work. Two texts from the same publication or writer never straddle the boundary.
- Machine text is separated by generator lineage, so variants of one underlying model stay together and a family used in fitting cannot reappear disguised as an independent test.
- Human data is divided into three non-overlapping roles: one part to fit on, one to choose an operating point on, and one reserved purely for safety validation — not looked at until every parameter is frozen.
That third tier exists because of a specific, expensive lesson. A safety threshold set as “just above the highest score any human in the sample reached” is arithmetically guaranteed to produce zero false positives on that same sample. The number is forced by construction and carries no information whatsoever. One earlier candidate passed exactly that way, and failed on unseen sources — which is why the safety tier is now sealed until it is too late to tune anything against it.
For the most recent validation, the reserved human tier contained 725 documents that actually exercised the behaviour under test, drawn from six sources across eleven registers, none of which had been used to fit or select anything.
What a change has to prove
A candidate change is written down before it is measured — the mechanism, the exact decision rule, the evaluation sets, the pass/fail conditions and the required effect size — and that description is cryptographically hash-stamped. The result is produced once against the frozen description. Bars are never adjusted after seeing the outcome.
To reach production a change must clear all of the following:
| Requirement | Why |
|---|---|
| No new false positives on the reserved human data | A wrongly flagged person is the costliest failure this tool has. Measured only on sources never used to fit or tune. |
| A real, pre-declared gain on machine text | Measured on generator families kept disjoint from those used in fitting. |
| No regression | No existing correct detection may be lost. |
| Exact preservation of non-English behaviour | Verified byte-for-byte across the full corpus, not argued. |
| Exact preservation of existing safeguards | Same standard. |
| Monotonic behaviour | An adjustment meant to only raise scores must be shown never to lower one, across all 305,234 documents. |
Testing for robustness, not for a headline
- Leave-source-out. The operating point is derived without a given human source, then tested on it. This is the only honest estimate of behaviour on writing the detector has never seen, and it routinely reveals that a threshold which looked perfectly safe in-sample is not.
- Leave-generator-out. The model is refitted with an entire generator family removed, then scored on it. A model that only works when the family appeared in training is rejected.
- Region-specific measurement. A signal is judged where it will actually act, never on a broad average. This matters more than it sounds: one component measured well across the whole corpus turned out to be worse than chance inside the narrow situation it was being used for, and had to be abandoned.
- Adversarial evaluation against all twelve transformation families above.
Track record — how often this actually produces a change
The honest summary is that most proposals fail, and that this is the process working rather than malfunctioning.
Across the eight most recent evaluation rounds, exactly one produced a change that reached production. Three closed with no candidate at all. Two produced candidates that then failed independent validation on data they had not been tuned against. One was stopped because the evidence needed to certify it could not be obtained at any attainable sample size.
Several of those rounds trained no model and cost nothing but analysis time, because the discipline is to stop as soon as the evidence says stop.
Release discipline
- Every deployed release records a cryptographic hash of each artefact the scoring path loads. The service re-verifies all of them at start-up and refuses to report itself ready if anything differs.
- A fixed set of reference documents — covering multiple languages, adversarial transformations and known control cases — is replayed against the live service after every deployment. An unexpected change in any of their results is treated as a release failure.
- Before a change ships, the old and new engines are replayed over the entire evaluation corpus and compared document by document, so the exact number of results that move is known in advance rather than discovered in production.
- The previous release stays immediately restorable, and the rollback procedure is written before deployment, not after.
- An evaluation set is held in reserve, excluded from all development, and has never been opened.
What this does not mean
None of this makes the detector reliable enough to decide anything about a person on its own. Careful methodology narrows the error rate and makes the failure modes predictable and documented; it does not remove them. Formal, technical and non-native writing still attracts false positives, and edited or paraphrased machine text still escapes detection.
The result remains one fallible signal. Read the limitations and false positives page before acting on any score, and see benchmarks for why this site publishes no single accuracy figure.