Brief
Telecom-fraud detector scores fall once test calls resemble ordinary service calls
A preprint describes a new audio benchmark for telecom-fraud detection, where classifiers that score perfectly against unrelated negatives drop to 0.65-0.68 once the non-fraud calls come from the same conversational neighbourhood. The numbers so far come from the authors' own experiments, not independent replication.
TeleAntiFraud 2.0 is built with a "Mixed-Tree Anti-Fraud Generation Pipeline" and a monthly frozen evaluation protocol: each set holds 900 Chinese calls, 600 fraud and 300 near-domain non-fraud, with audio, labels, prompts, manifests and provenance frozen so newly observed scam patterns can be added without overwriting earlier test sets.
In controlled text experiments, the authors report, three classifiers achieve perfect macro-averaged F1 against unrelated or ordinary negatives, but fall to 0.65-0.68 with near-domain sibling negatives. Full-set audio and automatic-speech-recognition plus large-language-model evaluations further reveal class-prior shortcuts, prediction collapse and snapshot sensitivity. The authors conclude that near-domain construction and collapse-aware reporting are core requirements for evaluating these models.
This is an arXiv preprint, so the result has not been through journal peer review, and the evidence given to us describes no independent replication. Treat the scores as the authors' own measurements on their own pipeline.
Our reading
Anyone judging an AI scam-detection product should ask what the system is scored against: easy, topic-separated negatives flatter a model that then struggles on lawful calls sitting close to fraud in topic and phrasing. This desk cares because the gap between the two numbers is the difference between a headline accuracy claim and a realistic condition, and readers evaluating such claims benefit f…
What to do or watch
Watch whether teams evaluating fraud-detection audio report near-domain negatives and collapse behaviour rather than a single accuracy figure. The unresolved question is whether these drops reproduce outside the authors' own generation pipeline and monthly frozen sets.
Source details and supporting facts
Each line is stated by the page named above it.
Stated by arXiv
- Each frozen evaluation set contains 900 Chinese calls, comprising 600 fraud and 300 near-domain non-fraud cases.
- Three classifiers achieve perfect macro-averaged F1 against unrelated or ordinary negatives but drop to 0.65-0.68 with near-domain sibling negatives.
- Full-set audio and automatic-speech-recognition plus large-language-model evaluations reveal class-prior shortcuts, prediction collapse, and snapshot sensitivity.
- The benchmark is evaluated under a monthly frozen evaluation protocol that freezes audio, labels, prompts, manifests, and provenance records for each monthly evaluation set.
Sources
- arXivText stored 17 September 2026
How this story was checked. Written from the 1 page listed above, stored 17 September 2026; claims checked against that stored text on 17 September 2026.
What that means
- 4 of 4 reported statements were confirmed against the page that carries them; the rest were removed rather than published.
- Figures in the text were required to appear in the stored source text: yes. Identifiers: yes.
- The check reads stored text only: no claim rests on a fresh look that did not happen.
- Where the reporting was silent, the text says so instead of filling the gap.