How well does Stylome work?

Stylome 1.2 Large cleared every human row across all three EditLens test sections and detected 99.00% of fully generated passages at the 0.01% FPR target.

EditLens contains human-written, AI-edited, and AI-generated text. This evaluation covers 18,192 rows in its standard, Enron email, and Llama test sections after excluding 27 rows due to dataset contamination. The detection tables count any AI-touched decision, whether edited or generated. The separate three-class table measures exact class matches using the dataset’s original labels.

Three-class results

The September 29 run also recorded exact class attribution with the frozen full-embedding KNN and five-neighbor fallback. These are exact-class recalls, not the detection recalls in the tables above. Confidence tiers were not evaluated in this rerun.

Most missed edits were classified as human. These results do not establish performance on unseen sources.

Results by test section, source, and producer

Every section had zero human false positives at both operating points: 0 of 2,035 standard-test human rows, 0 of 1,992 Enron rows, and 0 of 2,035 Llama rows. Each detection cell gives the count, denominator, and recall.

Evaluation details

The 6,062 human rows contain 4,022 distinct whitespace-normalized texts. None was flagged in any occurrence. Related human and AI texts appear across sections, so row counts are not counts of independent documents.

Evaluated September 16–17, 2026, and rescored September 29, 2026. Both thresholds were fixed before scoring. All inputs were scored without truncation or length exclusion. The exact two-sided 95% false-positive-rate interval for the 4,022 unique human texts is 0–0.0917%, under the independent-text assumption; shared authors and prompts limit that assumption.

At the 0.01% calibration FPR target selected for the service default, Stylome 1.2 Large detected 2,266 of 2,312 AI passages and cleared all 1,277 human passages.

The complete PAN 2025 validation set contains 3,589 passages across essays, fiction, and news. Both thresholds were fixed before scoring. The percentages below distinguish the calibration FPR target from the measured false-positive rate on PAN.

Producer names and binary labels follow PAN 2025, including its separately named paraphrase producers.

At the 0.01% FPR target, Llama-3.1-8B accounted for 25 of the 46 missed AI passages, Gemini-1.5-Pro for 9, and Mixtral-8x7B for 6. Fiction accounted for 33 misses.

Evaluation details

All 3,589 responses were verified. Twenty passages were shortened to the first 10,000 normalized characters to meet the API input limit: 17 human and 3 AI passages, all classified correctly at both operating points. The remaining 3,569 passages were scored in full.

Scoring completed in 24.4 minutes at 32 concurrent requests, averaging 1,972 full-request input tokens per second, including the shared prefix. Development compute cost was approximately $0.21. There were no request errors or checked training, reference, or calibration overlaps.

Evaluated September 16, 2026, and rescored September 29, 2026, on the established PAN validation set. The exact two-sided 95% interval for the observed human false-positive rate is 0–0.2885%, using the independent-passage assumption.

Stylome 1.2 Large correctly classified 6,892 of 6,911 eligible StoryScope test outputs as AI-generated at the fixed 0.01% calibration FPR target. Six were labeled AI-edited and 13 human; neither counts as correct here.

Evaluated September 19, 2026, and rescored September 29, 2026. Long stories (6,253) use their first 10,000 normalized characters; 658 shorter outputs were scored in full. Three missing outputs and six under 200 characters were excluded. Human story texts are not supplied, so false-positive rate cannot be measured here.

Stylome 1.2 Large correctly cleared 18,661 of 18,664 essays by non-native English writers, with the same result at both fixed operating points. Before scoring, every essay was checked against our training data and reference bank; none appeared in either. The thresholds were fixed before scoring.

TOEFL11 contains real TOEFL exam essays from writers with 11 different native languages. The ICNALE essays were written by university students in 10 Asian countries and regions between 2007 and 2013, plus a United Arab Emirates module. TOEFL91 is the 91-essay panel from Liang et al. (2023), in which seven earlier GPT detectors labeled an average of 61.3% of the essays as AI-generated. Stylome 1.2 Large flagged 2 of the 91. The panel is small, so each essay changes its rate by about 1.1 percentage points.

Native-speaker comparison. ICNALE also includes 400 essays by native English speakers, written to the same two prompts under the same conditions. Stylome 1.2 Large flagged 0 of them at the 0.01% target and 2 at the 0.1% target, compared with 1 of 5,398 learner essays at both targets. We see no sign that ESL writing is flagged more often than native writing.

Known limitation: professionally proofread essays. ICNALE also contains 795 learner essays revised by professional proofreaders. Stylome 1.2 Large labeled 32 of them (4.0%) as AI-edited at the 0.01% target, and 36 (4.5%) at the 0.1% target. We count these as human-written, so they are false positives. Heavy human editing of learner writing can resemble AI editing to the model.

TOEFL11 and ICNALE evaluated October 19, 2026; TOEFL91 re-evaluated October 20, 2026, with the current reference bank (unchanged: 2 of 91). The TOEFL11 copy is the 13,200-essay NLI Shared Task 2017 release; 25 essays under the 200-character minimum were not scored. ICNALE uses the Written Essays 2.6 and UAE 1.0 learner essays; the Written Essays Plus module was excluded because it was collected after 2022.

All 1,713 essays were classified as human at both fixed operating points. No essays were excluded or truncated, and the thresholds were fixed before scoring.

This evaluation uses the established frozen held half of the ELLIPSE corpus. The same 1,713 human-written essays were evaluated for earlier Stylome versions.

Evaluated September 16, 2026. The exact two-sided 95% binomial interval is 0–0.2151%, using the independent-essay assumption.

All 893 essays were classified as human at both fixed operating points. No essays were excluded or truncated, and the thresholds were fixed before scoring.

This evaluation uses the established frozen held half of the PERSUADE corpus. The same 893 human-written essays were evaluated for earlier Stylome versions.

Evaluated September 16, 2026. The exact two-sided 95% binomial interval is 0–0.4122%, using the independent-essay assumption.

Stylome 1.2 Large correctly cleared 8,556 of 8,557 transcripts at the 0.01% target and 8,555 at the 0.1% target. Transcribed speech is rare in our training data, so this tests how the model handles an unfamiliar kind of text. None of the transcripts appeared in our training data or reference bank.

The monologues were recorded by phone around 2013–2014. The dialogues are the interviewee's answers from interviews recorded around 2017–2019; the interviewer's questions are not included. Both come from the ICNALE project.

Evaluated October 19, 2026. Transcripts under the 200-character minimum were not scored (37 monologues, 36 dialogues). Each speaker contributes several transcripts.