Most professional translators cannot reliably tell a ChatGPT-written short story from a human one.
Researchers had 69 Italian-language translators evaluate three anonymized short stories — two produced by ChatGPT-4o, one by a human author. Each participant rated the probability of AI authorship and explained their reasoning. Overall results were inconclusive. The interesting part is the error distribution: 16.2% of participants correctly identified the synthetic texts at a rate statistically above chance, suggesting genuine analytical skill rather than luck. A nearly equal proportion got it backwards, flagging the human story as the AI product — which the researchers suggest may reflect some readers actually preferring the AI's smoother output.
The features that correlated with accurate identification were structural, not stylistic. Grammatical accuracy and emotional tone — things many readers associate with quality writing — actually led participants astray. What worked was noticing low sentence-length variability (a quality sometimes called "burstiness," the tendency of human writers to mix long and short sentences more freely than AI does), internal narrative contradictions, and traces of English syntax bleeding into the Italian text. These are subtle signals that surface-level AI detectors are unlikely to catch.
If trained language professionals miss the machine when they're relying on the wrong cues, automated detection tools built on similar surface features aren't a reliable backstop.