How Accurate Are AI Detectors? What Studies and the Vendors’ Own Tests Actually Show

Ask how accurate AI detectors are and you will receive two incompatible answers, both citing real evidence. The vendors report accuracies in the high nineties. Independent researchers who tested the same category of tools report numbers that would fail a school exam — and the company that created ChatGPT quietly withdrew its own detector for inaccuracy. This article lays the evidence side by side, claim by claim, with sources: what Turnitin and GPTZero say about their own tools, what the RAID benchmark and peer-reviewed studies found, and the Vanderbilt calculation that turned a reassuring percentage into 750 wrongly flagged papers. If you need the mechanism background first, read how AI detectors work; for the overall landscape, the pillar is AI Detectors Explained.

First, define “accurate” — because the two sides measure different things

Most arguments about detector accuracy are really arguments about definitions. A detector can be scored on at least three different tasks: separating fully-AI documents from fully-human ones; catching partially AI or lightly edited documents; and doing both without flagging human edge cases — formal writing, second-language writing, short texts. Vendor headlines report performance on the first task, measured on controlled document sets. Studies like Weber-Wulff’s test the second and third, on realistic mixed material. There is also a metric subtlety that dominates real-world outcomes: the false positive rate (the share of human documents flagged) matters far more than raw accuracy once a tool is run over millions of submissions, because the number of genuinely human documents dwarfs everything else. A detector that is “99% accurate” and flags 1% of human work will, in a large school system, produce wrongful flags by the hundred. Keep that arithmetic in mind through every claim below — it is the thread that connects Turnitin’s reassurance, GPTZero’s benchmarks and Vanderbilt’s alarm.

The vendors’ own numbers

Turnitin publishes the most carefully scoped claim in the industry. Its FAQ states that “based on testing data validated in our production environment, our false positive rate is less than 1% for documents with over 20% of AI writing” — glossed as roughly one in every hundred fully human documents — and that before every model release the system is tested on “over 700,000 academic papers written before the release of ChatGPT,” all known to be human-written. Note the construction of that validation: it measures how often the detector flags text that is certainly human. It does not, by itself, tell you how often AI text slips through, and the sub-1% figure is attached to a specific scoring band (documents above the 20% display threshold), not to every judgement the system makes at sentence level. To Turnitin’s credit, the same FAQ warns that the score “should not be used as the sole basis for adverse actions” and that the tool “does not determine misconduct.”

GPTZero reports stronger headline numbers on its explainer: 99% accuracy distinguishing AI-generated from human text, a 1% false positive rate on that task, 96.5% accuracy on mixed documents and document-level human mistakes of 0.24% in some configurations. On the RAID benchmark — a large adversarial test set built to include text designed to trip detectors — GPTZero reports identifying 95.7% of AI text while misclassifying 1% of human writing, and points to a 2024 Penn State validation of its methodology. The same page concedes the category’s limits in its own words: “human writing can be mistakenly flagged,” “short passages are harder to evaluate,” and “no detector is perfect.” Vendor numbers, read whole rather than as headlines, are thus more modest than their marketing: strong on clean separations, hedged everywhere else.

The independent evidence

Weber-Wulff et al. (2023), in the International Journal for Educational Integrity, tested 14 detection tools on human-written, machine-generated, machine-translated and paraphrased documents. Their verdict, quoted directly in the paper, is that the tools are “neither accurate nor reliable”: none achieved accuracy above 80%, most scored lower, and performance degraded further on paraphrased machine text — the realistic case of a user who edits what the model produced. This is the result that should anchor expectations for anyone relying on a detector against a mildly careful user: the tools are decent at catching copy-paste and poor at catching editing.

Liang et al. (2023), published in Patterns, measured the other side of the ledger — whom the errors hit. Seven detectors were run over 91 TOEFL essays by non-native speakers and 88 essays by US eighth-graders. The US essays were handled almost perfectly (average false positive rate about 5.19%); the TOEFL essays were misclassified at an average of 61.22%, every detector flagged at least 97.8% of them at least once across the panel, and all seven unanimously condemned nearly a fifth. The mechanism is documented in the same paper: flagged essays had significantly lower perplexity — predictable vocabulary is what second-language academic writing and machine generation share. An accuracy figure averaged over all writers hides the fact that the tool performs near-perfectly for one population and fails the majority of another.

OpenAI’s classifier supplies the control experiment. The organisation with the most data on its own model’s output launched a detector in January 2023 reporting, on its own challenge set, correct identification of 26% of AI-written text and false positives on 9% of human text. Six months later it withdrew the tool; its announcement page today reads that the classifier is “no longer available due to its low rate of accuracy.” When the maker of the text cannot reliably detect the text, claims of near-perfection from less-resourced vendors deserve the scrutiny the independent studies give them.

The Vanderbilt calculation: why “1%” is not small

The single most instructive document in the accuracy debate is not a study but an administrative decision. When Vanderbilt University disabled Turnitin’s AI detector in August 2023, it published its reasoning: the tool had been enabled with under 24 hours’ notice, could not be turned off by instructors, and — “most importantly” — offered no insight into how it works. Then the arithmetic: 75,000 papers had been submitted through Turnitin the previous year, and at the vendor’s stated 1% false positive rate, “roughly 1%, or 750 papers, could be incorrectly labeled as containing AI-generated text.” Scale that to a national school system processing tens of millions of documents and the sub-1% reassurance inverts: a “99% accurate” tool, run at population scale, reliably manufactures wrongful accusations in industrial quantities — each attached to a real student, as our false positives analysis details. Accuracy, in deployment, is a rate multiplied by a population, and any honest assessment must quote both numbers.

So how accurate are they, finally?

Assemble all the evidence and a consistent picture emerges, narrower than either side’s rhetoric. Against unedited, copy-pasted AI text, the major detectors perform well — the vendor benchmarks and RAID results agree, and even critics concede the tools flag blatant generation. Against edited, paraphrased or human-AI mixed text, accuracy falls steeply (Weber-Wulff) toward coin-flip territory for determined evasion. Against unusual human text — second-language, templated, technical, very short or very formal — the false positive risk is documented to reach majority levels in the worst-measured population (Liang). And against population scale, even the best-stated error rate guarantees large absolute numbers of innocent flags (Vanderbilt). That is why the same tool can be honestly described as 99% accurate and as neither accurate nor reliable: both statements are indexed to different tasks, and institutions err when they import the first task’s number into the third task’s decision. How scores are actually used once adopted — the policies, the quiet deterrent effect, the institutions that opted out — is covered in what schools and employers actually do with AI detectors.

What this means

For US readers, the accuracy question is rarely abstract: it arrives as a flagged essay, a rejected application, a hearing. The evidence assembled here supports a specific, defensible posture for each party. If you operate a detector — a teacher, a department, an HR team — the vendors’ own fine print is your minimum standard: Turnitin says its score must not be the sole basis for action; GPTZero says no detector is perfect; independent science says the tools fail the exact cases (edited text, non-native writers) where you will most want them to be right. Corroborate with drafts and conversation, or do not act. If you are subject to one, know that a flag is a statistical resemblance statement, and that the published record — including the withdrawal of OpenAI’s own classifier and a flagship university’s refusal to use the leading tool — is legitimate context any fair process should hear. The larger significance is about calibration. Americans are building institutional habits around AI percentages the way earlier generations built them around lie detectors: another instrument whose courtroom reputation eventually collapsed under its measured error rates. The studies above are the measurement phase of that story, publicly available and unusually consistent. Institutions that read them now get to skip the scandal phase — by treating detector output as what the evidence says it is: a lead, not a verdict.

One final piece of arithmetic belongs in every accuracy discussion and rarely appears: base rates. Suppose a detector with a genuine 1% false positive rate is run over a class of 200 students, of whom perhaps a handful used AI dishonestly. The tool flags roughly two innocent students for certain, alongside however many of the dishonest few it catches — meaning a flagged name drawn at random may be as likely to be an error as a culprit. Scale that class to a university, and Vanderbilt’s 750 stops looking like an edge case and starts looking like the tool’s normal operating output. This is not an argument that detection is worthless; it is an argument that its output can never ethically be the end of an inquiry, because probability guarantees a steady supply of innocent people inside every flagged set. Accuracy claims that omit the base rate are marketing. Accuracy claims that include it are policy documents — and by that standard, the most honest accuracy statement in this entire article remains the one on OpenAI’s withdrawn classifier page, where the maker of the world’s most-used chatbot simply admitted its detector was not accurate enough to exist.

Bottom line

The evidence supports one posture: a detector score is a prompt for a conversation, never the conclusion of one. The vendors’ own fine print says so — institutions should be held to at least that standard.

Read Next

• AI Detector False Positives: Why Human Writing Gets Flagged and What It Can Cost
• AI Detectors Explained: How AI Text Checkers Work and Why They Sometimes Get It Wrong

Sources

• Turnitin’s AI writing detection capabilities — FAQs (accessed 7 November 2026)
• How AI Detectors Work — GPTZero (accessed 7 November 2026)
• Liang et al. (2023), “GPT detectors are biased against non-native English writers,” Patterns 4(7) (accessed 7 November 2026)
• New AI classifier for indicating AI-written text — OpenAI (accessed 7 November 2026)