How accurate are AI detectors?

Why accuracy is the wrong number

Accuracy is the proportion of all cases the tool gets right. It tells you almost nothing about the question that matters when a person is flagged: given that this text was flagged, how likely is it that the flag is correct.

Those are different quantities and they can diverge enormously. The gap is driven by the base rate — how common the thing being detected actually is in the population being tested.

This is the same arithmetic that makes rare-disease screening produce mostly false positives despite excellent test characteristics. It is well understood in medicine and routinely ignored when the same structure appears in content detection.

The arithmetic, worked through

Take a class of 1,000 submitted essays. Suppose 5% were substantially AI-written — 50 essays — and suppose the detector has a 1% false-positive rate, which is better than most vendors claim, and catches 90% of real cases.

It correctly flags 45 of the 50 AI-written essays. It also incorrectly flags 1% of the 950 human-written essays, which is about 10 students.

So 55 essays are flagged, and 10 of them — roughly one in five — are wrong. One in five flagged students is falsely accused, by a tool with characteristics far better than anything on the market.

Now lower the base rate to 1%, which is plausible in a course where most students do their own work. Nine correct flags, about ten false ones. More than half of everyone accused is innocent.

• 5% base rate, 1% false-positive rate: about 1 in 5 flagged people is innocent.
• 1% base rate, 1% false-positive rate: more than half of flagged people are innocent.
• The rarer the behaviour, the worse the flag performs — regardless of how good the tool is.

Who the errors fall on

False positives are not distributed randomly, which is what turns a statistical problem into a fairness problem. Multiple studies have found that text by non-native English speakers is flagged at substantially higher rates than text by native speakers.

The mechanism is straightforward. Detectors key on markers of formal register, lower lexical variety, and predictable sentence construction — the same features characteristic of someone writing carefully in a second language, or of anyone taught to write in a clear, structured way.

So the students most likely to be wrongly accused are those already at a disadvantage, and those who worked hardest on their prose. Autistic writers, students taught rigid essay structures, and heavily-edited professional writing all cluster in the same failure region.

Why vendor numbers do not transfer

Published figures come from evaluations on curated datasets: text from known generators, of consistent length, in a controlled condition. Real submissions are mixed human and machine, edited after generation, of varying length, and sometimes from models the detector never saw.

Every one of those differences degrades performance, and they compound. A tool evaluated on unedited output from three well-known models will not hold its numbers on a paragraph a student generated and then rewrote.

Short text is a separate and severe problem. Statistical tests need volume, and a few hundred words often carries too little signal for any reliable conclusion — which is exactly the length people most often want checked.

Detection versus watermarking

It is worth separating two things that get discussed together. A statistical classifier guesses from style and has the error characteristics described above. A watermark detector tests for a deliberate signal inserted by the provider using a secret key.

Watermark detection is a much stronger method in principle — a genuine statistical test rather than a stylistic guess. But it requires the key, which means only the provider can run it, and it degrades under paraphrasing and disappears under translation.

For images, C2PA provenance is the equivalent that actually works for the public: it is cryptographic rather than probabilistic, and anyone can verify it. There is no text equivalent available today.

What to do with this

If you run a detector institutionally, treat a flag as a prompt to look further, never as a finding. Combine it with process evidence, and never let a score alone initiate a disciplinary consequence.

If you are accused, ask for the specific score, the tool's published false-positive rate, and the base rate in the population you were tested against. Then ask what evidence exists beyond the score. Most processes collapse at that point because the score was the whole case.

If you are designing policy, the arithmetic above should push you toward process-based assessment: in-class writing, drafts and version history, oral discussion of the work. These are harder to administer and dramatically fairer.