How human and AI translation differ, and what makes readers like both? A cross-corpus empirical re-analysis and LLM-as-a-Judge meta-evaluation
Using four public corpora (LAIT, ParaFarm, PAR3 and WMT25 literary), we measured what separates AI from human translation and what readers and experts reward. AI keeps the information; humans spend it on voice. Readers punish stumbles; experts who read the original punish changed meaning.


On this page
Introduction, or why we can't automate translation quality yet
In a 2026 study of fifteen translated novels, LAIT (Ferstler et al.), avid readers (12–120 books a year) tried to tell human-translated excerpts from AI ones. They guessed right 34 times out of 60: essentially a coin flip. Asked which version they actually preferred, blind, they picked the human translation in 522 of 772 passage comparisons (67.6%). In our guide to translating a book without losing your voice, we traced that gap to four losses: literal idioms, copied source syntax, drifting character voices, and flattened wordplay.
Other studies find the same split between what machines produce and what people value:
- Reviewers rated from-scratch literary translations most creative and raw MT least; post-editing MT constrained translators' creativity (Guerberof-Arenas & Toral 2022).
- Post-edited translations stay up to 38% closer to source sentence lengths than from-scratch ones, with flatter vocabulary and source syntax showing through (Toral 2019).
- In ParaFarm (seven published Ukrainian Animal Farm translations vs. three AIs), AI used up to 2× fewer particles, 2.6× fewer diminutives, and 18% lower lexical diversity, converging at 0.94 similarity vs. 0.71 for humans (Chaplynskyi et al. 2026).
- In LitEval-Corpus, professional raters preferred the human translation 94% of the time; student raters, 42% (Zhang et al. 2025).
- LAIT's three automatic metrics (MetricX-QE, GEMBA-MQM, LiTransProQA) predicted readers at chance (0.48–0.50), and two picked the machine almost every time.
Building our own AI editors and judges, we hit the same gap from the inside. The smoothest drafts stripped the most nuance, source-aware judges forgave calques, and models ironed out deliberate quirks. It appeared that natively, LLM-as-Judge approach cannot measure translation quality.
So we set out to measure what separates AI from human translation, and what separates a good translation from a bad one. We tested two theories: first, that a text carries units of meaning and style and breaks if a translation fails to move all of them; second, that what readers actually feel is reading flow, and the places where it stumbles.
Our data combines LAIT (French, Polish, and Japanese into English, with two readers' choices per passage), three public datasets rated by translators who saw the original (ParaFarm, PAR3, WMT25 literary), and our own texts, including my novel's prologue translated into Ukrainian.
TL;DR: what did we find?
| Question | Answer | Evidence |
|---|---|---|
| What differs most between AI and human translation? | How much of the original's information survives: machines keep 89–98 of 100 units, published translators 73–76. | In LAIT, the published translation kept fewer units than the study's MT in 374 of 386 passages (97%). |
| What do human translators change? | Emphasis, images, and details most; speaker tags, events, and facts least. | Across our LAIT runs, humans rework 30–43% of those top three types; machines rework 1–19% of any type. |
| Do readers reward keeping it all? | No. Within a book, keeping, cutting, changing, or adding units doesn't predict which version readers pick. | Within-book correlations sit near zero (−0.03 to +0.05); only reworked emphasis shows a slight positive link (+0.08). |
| What do readers punish? | Reading stumbles, and likely padding (net expansion). | Stumbles: judge counts match one reader (+0.20) as well as two readers match each other (+0.18). Expansion: −0.36 across 15 books (not significant), though the most expansive translation (Hooked, 18% longer than the machine) lost 90% of its passages. |
| What do experts punish? | Changed meaning, not compression or cuts. | In PAR3, when two versions differed, experts picked the one with fewer changed units 61% of the time (198 of 322 choices); cuts and compression showed no pattern. |
| Can AI judges stand in for readers? | For stumbles, yes. For craft, no: they overrate AI almost entirely because it has fewer stumbles. | Transept Pro beats published translations on 295 of 386 LAIT passages (+0.51); strip stumbles and copy-editing from the ratings and the gap drops to a near coin flip (+0.05; 213 vs. 173). |
| What does a good translation do? | Keeps the information and delivers it in a native voice. | In ParaFarm (Animal Farm), the pros' #1 translation keeps 93 of 100 units (near the machines' 94–98) and still ranks #2 with the source-blind AI judge. |
Theory one: does a translation have to carry every unit of meaning and style?
In short: No. Machines carry almost every unit of information in the original; good human translators don't, and readers don't hold it against them. Readers who never see the original punish expansion. Experts who read the original punish changed meaning.
How much of the original survives is the clearest difference between human and machine translation we measured.
Why did we expect every unit to matter?
A book invents language artifacts (names, coined terms, rhythm, running jokes) that work on several levels at once, while leaning on target-language standards (idioms, register, punctuation, syntax). Translation pulls both ways. We split text into meaning (what a reader learns) and style (how it reads), betting that dropping pieces of either layer makes a translation break.
How did we measure meaning and style?
How to read: one judge per row: what it sees, what it checks, and the number it gives.
| Judge | Sees | Checks | Counts |
|---|---|---|---|
| Meaning judge | The original and the translation | Seven kinds of misreading: reversed meaning, wrong referent, omitted fact, added fact, wrong fact, idiom, word sense. Deliberate liberties (an explained cultural item, an image swapped for one doing the same job) are allowed | Errors that both of two independent runs find, per 1,000 source words |
| Rubric (element maps) | Built from the original alone, before any translation exists | Elements a hurried translator loses: hinge words, logic links, motifs, allusions, double meanings, devices, register, constructions; yes-or-no checks for each | Elements lost per 1,000 words (a bilingual checker); a native reader who sees only the translation says whether each element reads native, foreign, weak or missing |
| Unit tracker | The original and every version of the passage, blind and shuffled, in one call | Splits the original into units of information (event, fact, detail, speaker, feeling, image, logic, emphasis, reference); marks each version kept, compressed, changed or dropped; lists what each version adds | Per 100 units of the original |
The first two work. The meaning judge catches 51 of 61 real errors on our answer key while flagging 10 of 33 deliberate liberties in published translations. The rubric caught all 61 planted errors (57 of 59 on Gemini) and exposed what smooth prose hides: translating through English lost 12–18 elements per 1,000 words vs. 0–1 directly, with only 0–2.4 hard errors.
Neither predicts readers (rubric losses score −0.05 against LAIT). Only the rubric's source-blind native-reader check tracked them (+0.23 on tuning books): our first clue that readers react to how a text reads, not what it kept.
What do translators do with the information?
Machines keep nearly everything. Published translators keep about three units in four and rework the rest.
How to read: a longer "kept" bar means closer to the original; the other segments are what was reworked, and added units sit outside the 100. Neither end is "better" on its own.
Source: 135 LAIT passages; one Gemini 4 Hydrogen call (high thinking) per passage marks all 14 versions blind against one list of units. Compressed and changed are upper bounds (see "How far can the unit tracker be trusted?").
Show the data
| Version | Kept | Compressed | Changed | Dropped | Added |
|---|---|---|---|---|---|
| Published human translation | 73.0 | 11.8 | 11.2 | 4.0 | 9.2 |
| Gemini 3.8 Flash low through Transept | 89.3 | 5.6 | 4.4 | 0.7 | 5.2 |
| Haiku 5.5 through Transept | 91.9 | 4.4 | 2.9 | 0.8 | 2.1 |
| Gemini 3.1 Flash Lite through Transept | 92.8 | 3.5 | 2.9 | 0.8 | 1.7 |
| Gemini 3.5 Flash Lite through Transept | 93.9 | 2.9 | 2.7 | 0.5 | 2.8 |
| Gemini 4 Neon low through Transept | 94.9 | 3.5 | 1.1 | 0.5 | 2.6 |
| Transept Standard | 95.1 | 3.4 | 0.7 | 0.8 | 1.2 |
| Opus 5.5 high, plain API | 95.3 | 3.3 | 0.7 | 0.6 | 1.3 |
| Gemini 4 Argon low through Transept | 95.4 | 2.7 | 1.7 | 0.3 | 4.5 |
| Transept Pro | 96.3 | 2.6 | 0.6 | 0.4 | 1.8 |
| Gemini 4 Neon low, plain API | 96.4 | 2.3 | 0.8 | 0.5 | 1.5 |
| Study MT (the LAIT paper's agentic pipeline) | 96.5 | 2.4 | 0.7 | 0.4 | 0.9 |
| Opus 5.5 low, plain API | 97.0 | 2.1 | 0.5 | 0.4 | 0.7 |
| Gemini 4 Argon high through Transept | 98.1 | 1.4 | 0.4 | 0.1 | 1.8 |
The pattern holds in every corpus we ran:
How to read: units of the original kept per 100. Higher means closer to the original, not a better translation.
| Corpus | Human translators keep (per 100 units) | Machines keep |
|---|---|---|
| LAIT: 15 novels into English, 386 passages | Published translations 76 | Study MT 98, Transept Pro 98 |
| Two official speech translations (Ukrainian, English) | 85–88 | 88–97 |
| Animal Farm, 7 Ukrainian translators (ParaFarm) | Stelmakh 93 down to Okolitenko 55; Dybko's free adaptation 38 | DeepL 98, GPT-5.2 97, Lapa 95 |
| WMT25 literary, English into Ukrainian / Russian | Commissioned reference 81 / 57 | Claude-4 and Gemini 93–96; GPT-4.1 76 |
| PAR3: classics from German, French and Russian into English | 87 | Google Translate 89, GPT-3 85 |
How much survives is the best single tell between a human and a machine translation. On LAIT, the published translation keeps fewer units than the study's MT in 374 of 386 passages (97%) and fewer than Transept Pro in 381 (99%). Among the 14 versions in Chart 1, the human one keeps the fewest units in 115 of 135 passages.
The one machine that compresses like a human is GPT-4.1 in WMT25: it keeps 76 and writes about 12% shorter than the others, a concise style rather than truncation.
For comparison, our reading score puts the published translation above the study MT in 66% of passages (the readers: 67.6%), and the published version is longer than the MT in only 59%.
How to read: one passage, four English versions, and what the tracker marked for each.
«А ще воно відкривало її пружні плечі та жіночний живіт — під яким грали м'язи, коли Агнес сміялася чи затягувалася сигаретою. І те, й інше магеса любила.»
“It also bared her firm shoulders and womanly belly, under which muscles played whenever Agnes laughed or drew on a cigarette. The magess loved both.”
“It also bared her firm shoulders and her womanly belly, where muscles played whenever Agnes laughed or drew on a cigarette. The magess loved doing both.”
Sonnet and Haiku read almost the same. Tracker: Sonnet keeps every unit; Opus too, except that "under which" becoming "where" counts as compressed. A reader judge flagged Opus's "her womanly belly" as hard to follow.
“The outfit bared her toned shoulders and the soft curve of her stomach, where muscle stirred beneath the skin whenever she laughed or drew on her cigarette, two pleasures the magess rarely denied herself.”
Tracker: "womanly belly" changed, "the magess loved both" changed. No reader judge flagged these lines.
the models carry the units over word for word; my version says the same things the English way, and the tracker counts that as rework. Across this chapter my version keeps 71–77 units in 100, the three models 87–100.
Which units do translators rework?
The ones that carry voice. Facts and events stay; emphasis, images and details move.
How to read: the share of each unit type a version reworked. Higher means more reworking, not worse.
| Unit type | Published translation | Study MT | Opus 5.5 high, plain | Neon low, plain | Neon low through Transept | 3.8 Flash through Transept | Transept Standard | Transept Pro |
|---|---|---|---|---|---|---|---|---|
| Emphasis | 37% | 7% | 12% | 9% | 11% | 16% | 10% | 7% |
| Image | 37% | 5% | 4% | 6% | 9% | 19% | 6% | 4% |
| Detail | 32% | 4% | 4% | 5% | 6% | 13% | 6% | 4% |
| Reference | 28% | 4% | 8% | 2% | 5% | 10% | 12% | 7% |
| Feeling | 27% | 3% | 4% | 2% | 3% | 9% | 3% | 3% |
| Fact | 26% | 4% | 4% | 3% | 4% | 10% | 4% | 3% |
| Event | 25% | 3% | 4% | 3% | 4% | 10% | 4% | 3% |
| Logic | 24% | 4% | 8% | 6% | 8% | 10% | 6% | 6% |
| Speaker | 19% | 3% | 4% | 2% | 3% | 6% | 4% | 4% |
| All units | 27% | 3.5% | 4.7% | 3.6% | 5.1% | 10.7% | 4.9% | 3.7% |
"Reworked" means compressed, changed, or dropped across Chart 1's 135 passages (Gemini 4 Hydrogen; reference and emphasis are the smallest groups, so their columns are noisiest).
How to read: further right means more of that unit type was reworked. The gap between the big published dot and the cluster of machine dots is the human–machine difference.
Source: one Gemini 4 Hydrogen call (high thinking) per passage marks all 14 versions blind against one list of units; group sizes 214 emphasis, 247 image, 994 detail, 127 reference, 902 feeling, 1,758 fact, 1,706 event, 555 logic, 205 speaker units. Shares are upper bounds (see "How far can the unit tracker be trusted?"). The cheaper models' higher shares (Haiku, the Flash Lites, 3.8 Flash) are more likely slips than deliberate reworking.
Show the data
| Version | Emphasis | Image | Detail | Reference | Feeling | Fact | Event | Logic | Speaker | All units |
|---|---|---|---|---|---|---|---|---|---|---|
| Published translation | 37% | 37% | 32% | 28% | 27% | 26% | 25% | 24% | 19% | 27% |
| Study MT | 7% | 5% | 4% | 4% | 3% | 4% | 3% | 4% | 3% | 3.5% |
| Opus 5.5 low, plain | 7% | 4% | 3% | 7% | 1% | 3% | 2% | 5% | 3% | 3% |
| Opus 5.5 high, plain | 12% | 4% | 4% | 8% | 4% | 4% | 4% | 8% | 4% | 4.7% |
| Neon low, plain | 9% | 6% | 5% | 2% | 2% | 3% | 3% | 6% | 2% | 3.6% |
| Haiku 5.5 through Transept | 17% | 14% | 9% | 16% | 6% | 8% | 7% | 8% | 6% | 8.1% |
| Gemini 3.1 Flash Lite through Transept | 16% | 11% | 9% | 4% | 6% | 6% | 7% | 5% | 12% | 7.2% |
| Gemini 3.5 Flash Lite through Transept | 9% | 12% | 7% | 10% | 5% | 6% | 6% | 4% | 5% | 6.1% |
| Gemini 3.8 Flash through Transept | 16% | 19% | 13% | 10% | 9% | 10% | 10% | 10% | 6% | 10.7% |
| Gemini 4 Argon low through Transept | 7% | 9% | 6% | 3% | 3% | 4% | 3% | 7% | 2% | 4.6% |
| Gemini 4 Argon high through Transept | 4% | 2% | 3% | 3% | 1% | 1% | 2% | 4% | 2% | 1.9% |
| Gemini 4 Neon low through Transept | 11% | 9% | 6% | 5% | 3% | 4% | 4% | 8% | 3% | 5.1% |
| Transept Standard | 10% | 6% | 6% | 12% | 3% | 4% | 4% | 6% | 4% | 4.9% |
| Transept Pro | 7% | 4% | 4% | 7% | 3% | 3% | 3% | 6% | 4% | 3.7% |
Translators also agree on which units are open. In ParaFarm, units reworked by at least four of the six published translators appear 56 times (against 11 by chance), and units all six left alone appear 378 times (against 244). When AI systems do rework a unit, they usually pick the same ones.
Is information density a property of the language?
Partly. Each source language comes with its own density, and translators carry most of it over.
How to read: units of information per 100 English words. Higher means denser prose.
| Source language | Machine (source density carried over) | Published translation |
|---|---|---|
| French | 11.2 units per 100 English words | 10.7 |
| Polish | 10.4 | 10.0 |
| Japanese | 9.6 | 9.4 |
Across the 15 books, published density tracks the machine's with a slope of 0.75: translators move about a quarter of the way toward a common English level, loosening introspection most (0.95 of machine density) and action least (0.99).
I started with a theory: every language, genre, and author has an informational capacity, and cramming more in breaks rhythm into reader hiccups.
Half true; the other half runs backward. Machines do write denser English than published translators (10.40–10.54 vs. 10.04 units per 100 words), yet within a book density predicts nothing (0.00 [−0.09, +0.08]), and across books readers preferred denser translations (+0.39; +0.25 without Hooked). Writing looser than the machine is what costs translators.
Do readers reward keeping the information?
No. Within a book, passage by passage, nothing in the unit tracker predicts which version LAIT's readers chose:
How to read: above 0, readers preferred the published version more as this grew; below 0, less. An interval that spans 0 means no effect.
| Published minus machine | Correlation with readers choosing the published version (within book, 95% interval) |
|---|---|
| Units dropped | −0.03 [−0.14, +0.08] |
| Units compressed | +0.05 [−0.05, +0.16] |
| Units changed | +0.03 [−0.07, +0.13] |
| Units added | +0.04 [−0.03, +0.11] |
| Density | 0.00 [−0.09, +0.08] |
| Emphasis reworked | +0.08 [+0.01, +0.15] |
Only reworked emphasis clears zero, slightly favouring the translator. Within a book everything is noisy: even our stumble score hits +0.08 (+0.26 pooled).
Across whole books a pattern shows, carried mostly by one book. Each book has one translator, so this is 15 data points:
How to read: higher dots are books where readers preferred the published translation; above 50% it beat the machine. Further right, the translator added more.
Source: LAIT readers' choices; our Claude Opus unit tracker on all 386 passages.
Show the data
| Book | Readers chose the published version | Compressed | Dropped | Changed | Added | Words vs. machine |
|---|---|---|---|---|---|---|
| Hooked (ja) | 10% | 13.8 | 3.3 | 15.2 | 20.7 | 1.18 |
| The Witcher: Crossroads of Ravens (pl) | 30% | 6.2 | 0.3 | 6.9 | 3.7 | 1.00 |
| Mona's Eyes (fr) | 41% | 7.7 | 0.6 | 6.4 | 5.4 | 1.00 |
| The Ark (ja) | 48% | 15.3 | 1.9 | 16.9 | 17.8 | 1.01 |
| Inner Space (pl) | 54% | 5.2 | 2.1 | 6.1 | 12.6 | 1.12 |
| The Story of Marceau Miller (fr) | 62% | 18.3 | 8.6 | 10.7 | 9.3 | 0.92 |
| Make Me Famous (fr) | 67% | 10.5 | 2.2 | 8.8 | 12.5 | 1.10 |
| Watching Over Her (fr) | 68% | 6.2 | 1.4 | 12.0 | 12.2 | 1.03 |
| My Name Is Stramer (pl) | 69% | 6.0 | 0.3 | 7.5 | 6.1 | 1.05 |
| Sisters in Yellow (ja) | 85% | 19.6 | 2.4 | 15.8 | 15.4 | 0.95 |
| The Empusium (pl) | 89% | 8.1 | 1.7 | 8.4 | 5.1 | 0.97 |
| My Grandfather the Master Detective (ja) | 91% | 18.1 | 4.7 | 11.8 | 15.8 | 0.99 |
| Kokun: The Girl from the West (ja) | 92% | 15.2 | 2.9 | 9.6 | 12.6 | 0.96 |
| Needle's Eye (pl) | 96% | 10.6 | 1.7 | 7.1 | 6.5 | 0.99 |
| Symphony of Monsters (fr) | 100% | 9.3 | 1.2 | 9.3 | 12.7 | 1.06 |
How to read: the same signs, across the 15 books. Only values beyond ±0.52 would be significant.
| Book-level measure (Spearman, 15 books) | All books | Without Hooked |
|---|---|---|
| Units compressed | +0.19 | +0.29 |
| Units dropped | +0.05 | +0.23 |
| Units changed | 0.00 | +0.17 |
| Units added | +0.04 | +0.28 |
| Net: added minus compressed minus dropped | −0.36 | −0.26 |
| Length relative to the machine | −0.35 | −0.20 |
| Density relative to the machine | +0.39 | +0.25 |
None clears statistical significance at 15 books, so read these as direction only. Two patterns stand out:
- Expansion loses. Hooked's translator wrote 18% more words than the machine and added 20.7 units per 100; readers chose the machine in 90% of its passages.
- Without Hooked, translators who reworked more did slightly better. The two closest translations (The Witcher, 30% reader share; Mona's Eyes, 41%) lost to the machine, while another close one (My Name Is Stramer, 69%) won, so closeness alone doesn't decide it.
Why do experts who read the original judge differently?
Because they see the source. LAIT's readers didn't; PAR3, ParaFarm, and WMT25 raters did, and penalized changes readers ignored. In ParaFarm, source-aware translators and a source-blind judge ranked the same ten translations almost in reverse.
How to read: what each kind of judge rewards or punishes, by type of change.
| Who judges | Losing or compressing units | Changing meaning | Adding |
|---|---|---|---|
| LAIT readers, original unseen | No effect within a book | No effect within a book | Net expansion loses across books (not significant) |
| PAR3: 450 choices by translators who read the original | No consistent direction (39–59% across pairings) | Where versions differed, the chosen one had fewer changed units in 198 of 322 choices (61%; 58–65% by pairing) | Not held against a version: the chosen one had more additions in 213 of 324 choices (66%) |
| ParaFarm: 4 professional translators, 1,034 pairwise judgments, original visible | Heavy reworking ranked down (−0.53; −0.35 without the adaptation) | Part of reworking | Part of reworking |
| ParaFarm: the same 10 versions ranked by an AI judge (GPT-5.2) without the original | Reworking ranked up (+0.65 without the adaptation) | Part of reworking | Additions ranked up (+0.75) |
| WMT25 literary: professionals with the original | −0.13 between reworking and score, all from GPT-4.1's concise style; +0.04 without it |
How to read: rank 1 is best. A flat line means both judges agree on that translation; a steep line means they disagree.
Source: rankings from Chaplynskyi et al. 2026; units from our unit tracker on 40 passages, all ten versions in one call each.
Show the data
| Version | Professionals' rank | Source-blind judge's rank | Units kept | Units added | Ukrainian words per 100 English |
|---|---|---|---|---|---|
| Stelmakh 2021 | 1 | 2 | 92.7 | 2.7 | 82 |
| DeepL | 2 | 8 | 97.5 | 0.1 | 77 |
| Shevchuk 1991 | 3 | 3 | 86.1 | 2.9 | 70 |
| Nosenok 2020 | 4 | 5 | 74.2 | 6.5 | 71 |
| GPT-5.2 | 5 | 6 | 97.1 | 0.3 | 75 |
| Drozdovskyi 1991 | 6 | 1 | 77.7 | 14.2 | 92 |
| Lapa | 7 | 9 | 94.5 | 1.1 | 80 |
| Cherniatynskyi 1947 | 8 | 7 | 93.8 | 1.8 | 78 |
| Okolitenko 1992 | 9 | 4 | 55.5 | 5.4 | 63 |
| Dybko 1984, a free adaptation | 10 | 10 | 38.0 | 16.7 | 85 |
The professionals' #1 (Stelmakh) keeps nearly as much of the original as the machines (93 of 100 units) and still places second blind; the blind judge's #1 (Drozdovskyi) expands heavily and drops to sixth with pros. Winning both audiences requires keeping the information in a native voice.
How to read: lines the meaning judge, which reads the original, counted as errors, and lines it missed.
My novel's world is called «Плани», "the Plans". Sonnet wrote "make the Planes better", so a reader pictures planes of existence; both runs flagged it.
Haiku turned «Її доля від цього не зміниться» ("Her fate won't change because of it") into "Her share won't change": доля means both "fate" and "share", and the context rules out the second. Both runs flagged it too.
Opus made the same "Planes" error, and neither run caught it.
In the prologue, «правильний бізнес завжди постоїть для вас» is a deliberate double meaning: business will always "stand" for you, unlike a man, which sets up the punchline that follows. Opus and Sonnet wrote "will always stand up for you", which reads as "defend you"; Haiku wrote "always stands by you". One run flagged Sonnet's line for losing the joke; the other run, and both runs on Opus's identical line, did not, so the agreed count missed it.
a judge that reads the original catches misread words more reliably than lost jokes, and on published translations it still mistakes 10 of 33 deliberate liberties for errors.
How far can the unit tracker be trusted?
"Kept" and the human–machine gap are solid; "reworked" is inflated roughly twofold. My hand audit of 78 unit tracker claims:
How to read: share of tracker claims I agreed with. Higher means more trustworthy.
| The tracker says | I agreed |
|---|---|
| Kept | 100% |
| Added | 81% |
| Compressed | 40% |
| Changed | 33% |
Models treat shifts in wording, intensity, or image as changed information where translators see equivalents. Opus and Gemini 4 Carbon, Argon, and Neon agreed with me 69–73%; Hydrogen (high thinking) hit 79% (10/12 "changed" calls right), so Chart 1 uses it. Model voting didn't help: all share the same strict bias.
How to read: one passage of my English version, translated with the book's glossary, and what the tracker made of it.
The book's glossary gives the world its English names: «Академія», literally "the Academy", is CISAME; «Твердь» is "the Ground"; «збирачі» are "the Gathers". My English version follows the glossary. The three models translated without it and wrote "the Academy", "the Firmament" or "Tverd", and "the scavengers".
«Але піти з Академії Теодор дозволити собі більше не міг.»
“But Theodore could no longer afford to leave the Academy.”
“But leaving the Academy was something Theodore could no longer afford.”
“Yet walking away from Cisame was something Theodore could no longer afford.”
the tracker reads a creative decision, made once and applied consistently, as changed information. That is part of why its "changed" claims were right only a third of the time in my audit, and why every "reworked" share here is an upper bound.
Theory two: what breaks a reader's flow?
In short: Stumbles (calques, literal phrasing, foreign syntax, unnatural wording, grammar). AI judges counting them match inter-reader agreement, yet rank AI above published translators almost entirely on stumbles because models prize target-language standards over authorial artifacts.
How did we get there?
By automating LAIT, whose readers judged single passages and highlighted 7,234 good or awkward phrases. Existing metrics failed because reference matching punishes valid alternatives, while error-counting rewards flat correctness.
Our judges mimic readers: source-blind, one passage at a time, flagging wherever flow breaks (designed on 9 books, tested once on 6 held-out books). Hiding the source mattered most: a source-aware judge forgives clumsy phrasing because it knows what was meant.
What do the reading judges check?
How to read: fewer flags means smoother reading; for Reader mirror, a higher rating is better.
| Judge | Runs | What it flags | Number it gives |
|---|---|---|---|
| Editor at a publisher | 2 | Must-fix changes against matters of taste; kinds: calque, word choice, grammar, punctuation, clarity, style | Must-fix style changes per 100 words |
| Demanding reader | 1 | Places the reading was disturbed: unclear, hard to follow, sentence structure, unnatural, too literal, register, wordy, flat, mechanics | Weighted issues per 100 words |
| Avid reader, who reads fast for pleasure | 1 | The same minus "too literal": only what stops the story | Weighted issues per 100 words |
| Reader mirror | 1 | LAIT's own questionnaire: acceptability, smoothness, immersion, would you keep reading | Mean rating |
Each number is standardized against the 772 LAIT versions and the four are averaged with equal weights. 0 is the average LAIT version; higher reads better. Punctuation, grammar and mechanics are counted apart, as copy-editing.
Which judges predict readers?
The ones that count stumbles.
How to read: agreement runs from −1 to +1; higher is closer to real readers. Two readers agree with each other at +0.18.
| Validation on LAIT | Result |
|---|---|
| Agreement with a single reader | +0.22 (two readers agree with each other at +0.18) |
| Passages where both readers were sure | 68% right (90 of 116 human wins, 14 of 24 machine wins) |
| The 6 books held out of all design | +0.19 |
| Ranking whole books | +0.69 |
| GPT-6 instead of Claude as the judges | Same direction; the two families correlate at +0.76 |
Taking the score apart shows where that agreement comes from:
How to read: agreement with one reader; higher is better, and +0.18 is the two-reader ceiling.
| Score built from | Agreement with one reader |
|---|---|
| The full reading score | +0.22 |
| Stumbles only (editor style errors and both readers' issues) | +0.20 |
| The questionnaire ratings only | +0.18 |
| Copy-editing only | +0.13 |
| "Good wording" highlights only | +0.08 |
| The questionnaire with stumbles and copy-editing regressed out | +0.03 |
The questionnaire agrees with readers only because it tracks stumbles. Broken down by kind, the reader signal sits squarely in translationese:
How to read: flags per 100 words, so lower is better. The agreement column shows how much each kind matters to readers: higher matters more.
| Kind | Agreement with one reader | Published translation | Study MT | Opus 5.5 high, plain | Neon low, plain | Neon low through Transept | Transept Standard | Transept Pro |
|---|---|---|---|---|---|---|---|---|
| Sentence structure | +0.15 | 0.66 | 0.78 | 0.57 | 0.72 | 0.60 | 0.56 | 0.53 |
| Unnatural wording | +0.15 | 0.91 | 1.02 | 0.82 | 1.06 | 0.78 | 0.83 | 0.68 |
| Grammar (editor) | +0.14 | 0.16 | 0.15 | 0.04 | 0.15 | 0.08 | 0.06 | 0.03 |
| Too literal | +0.13 | 0.21 | 0.57 | 0.37 | 0.53 | 0.27 | 0.31 | 0.22 |
| Mechanics | +0.13 | 0.26 | 0.35 | 0.19 | 0.30 | 0.21 | 0.20 | 0.15 |
| Calque (editor) | +0.11 | 0.20 | 0.40 | 0.22 | 0.42 | 0.15 | 0.19 | 0.10 |
| Word choice (editor) | +0.10 | 0.22 | 0.21 | 0.10 | 0.19 | 0.10 | 0.10 | 0.07 |
| Register | +0.08 | 0.28 | 0.16 | 0.16 | 0.26 | 0.17 | 0.15 | 0.12 |
| Clarity (editor) | +0.04 | 0.06 | 0.12 | 0.03 | 0.09 | 0.06 | 0.05 | 0.03 |
| Wordy | +0.03 | 0.39 | 0.34 | 0.31 | 0.41 | 0.38 | 0.31 | 0.33 |
| Unclear | +0.02 | 0.37 | 0.54 | 0.46 | 0.51 | 0.46 | 0.51 | 0.42 |
| Punctuation (editor) | +0.02 | 0.12 | 0.27 | 0.29 | 0.15 | 0.12 | 0.07 | 0.06 |
| Style (editor) | −0.01 | 0.04 | 0.05 | 0.03 | 0.05 | 0.02 | 0.02 | 0.02 |
| Hard to follow | −0.02 | 0.31 | 0.44 | 0.35 | 0.34 | 0.37 | 0.37 | 0.34 |
| Flat | −0.05 | 0.24 | 0.15 | 0.19 | 0.19 | 0.19 | 0.17 | 0.17 |
135 shared passages (agreement on all 386). Plain AI trips where readers feel it (calques, literal phrasing). Against the stronger systems, the published translations draw more flags of nearly every kind, the ones readers feel included; the gap is widest for grammar, word choice and register. "Wordy" (the closest proxy for informational capacity) has no reader signal (+0.03).
How to read: darker cells mean more stumbles, which is worse. Rows above the rule are the kinds readers actually feel; below it, the judges flag things readers don't mind.
| Kind | Agreement | Published translation | Study MT | Opus 5.5 low, plain | Opus 5.5 high, plain | Neon low, plain | Neon high, plain | Haiku 5.5 through Transept | Gemini 3.1 Flash Lite through Transept | Gemini 3.5 Flash Lite through Transept | Gemini 3.8 Flash through Transept | Gemini 4 Argon low through Transept | Gemini 4 Argon high through Transept | Gemini 4 Neon low through Transept | Transept Standard | Transept Pro |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Sentence structure | +0.15 | 0.66 | 0.78 | 0.70 | 0.57 | 0.72 | 0.65 | 0.68 | 0.72 | 0.73 | 0.55 | 0.52 | 0.55 | 0.60 | 0.56 | 0.53 |
| Unnatural wording | +0.15 | 0.91 | 1.02 | 1.11 | 0.82 | 1.06 | 0.95 | 1.08 | 1.20 | 1.25 | 0.93 | 0.74 | 0.81 | 0.78 | 0.83 | 0.68 |
| Grammar (editor) | +0.14 | 0.16 | 0.15 | 0.11 | 0.04 | 0.15 | 0.10 | 0.17 | 0.20 | 0.19 | 0.11 | 0.07 | 0.07 | 0.08 | 0.06 | 0.03 |
| Too literal | +0.13 | 0.21 | 0.57 | 0.49 | 0.37 | 0.53 | 0.34 | 0.54 | 0.58 | 0.56 | 0.23 | 0.23 | 0.29 | 0.27 | 0.31 | 0.22 |
| Mechanics | +0.13 | 0.26 | 0.35 | 0.28 | 0.19 | 0.30 | 0.25 | 0.36 | 0.39 | 0.49 | 0.30 | 0.27 | 0.23 | 0.21 | 0.20 | 0.15 |
| Calque (editor) | +0.11 | 0.20 | 0.40 | 0.38 | 0.22 | 0.42 | 0.24 | 0.48 | 0.52 | 0.45 | 0.19 | 0.12 | 0.20 | 0.15 | 0.19 | 0.10 |
| Word choice (editor) | +0.10 | 0.22 | 0.21 | 0.17 | 0.10 | 0.19 | 0.18 | 0.23 | 0.33 | 0.34 | 0.17 | 0.12 | 0.13 | 0.10 | 0.10 | 0.07 |
| Register | +0.08 | 0.28 | 0.16 | 0.15 | 0.16 | 0.26 | 0.22 | 0.18 | 0.32 | 0.26 | 0.20 | 0.17 | 0.20 | 0.17 | 0.15 | 0.12 |
| Clarity (editor) | +0.04 | 0.06 | 0.12 | 0.08 | 0.03 | 0.09 | 0.07 | 0.14 | 0.10 | 0.11 | 0.07 | 0.03 | 0.05 | 0.06 | 0.05 | 0.03 |
| Wordy | +0.03 | 0.39 | 0.34 | 0.26 | 0.31 | 0.41 | 0.40 | 0.36 | 0.40 | 0.44 | 0.41 | 0.43 | 0.43 | 0.38 | 0.31 | 0.33 |
| Unclear | +0.02 | 0.37 | 0.54 | 0.48 | 0.46 | 0.51 | 0.41 | 0.64 | 0.50 | 0.50 | 0.41 | 0.35 | 0.42 | 0.46 | 0.51 | 0.42 |
| Punctuation (editor) | +0.02 | 0.12 | 0.27 | 0.15 | 0.29 | 0.15 | 0.09 | 0.15 | 0.36 | 0.68 | 0.18 | 0.11 | 0.11 | 0.12 | 0.07 | 0.06 |
| Style (editor) | −0.01 | 0.04 | 0.05 | 0.03 | 0.03 | 0.05 | 0.04 | 0.05 | 0.08 | 0.08 | 0.05 | 0.03 | 0.04 | 0.02 | 0.02 | 0.02 |
| Hard to follow | −0.02 | 0.31 | 0.44 | 0.30 | 0.35 | 0.34 | 0.36 | 0.56 | 0.45 | 0.46 | 0.40 | 0.36 | 0.37 | 0.37 | 0.37 | 0.34 |
| Flat | −0.05 | 0.24 | 0.15 | 0.15 | 0.19 | 0.19 | 0.17 | 0.18 | 0.18 | 0.16 | 0.23 | 0.18 | 0.20 | 0.19 | 0.17 | 0.17 |
Above the rule: the kinds that track readers (agreement +0.10 or more). The published translation's column is outlined.
Source: our reading judges (Claude Opus 5.5) on the 135-passage subset; agreement on all 386 LAIT passages.
Show the data
| Kind | Agreement | Published translation | Study MT | Opus 5.5 low, plain | Opus 5.5 high, plain | Neon low, plain | Neon high, plain | Haiku 5.5 through Transept | Gemini 3.1 Flash Lite through Transept | Gemini 3.5 Flash Lite through Transept | Gemini 3.8 Flash through Transept | Gemini 4 Argon low through Transept | Gemini 4 Argon high through Transept | Gemini 4 Neon low through Transept | Transept Standard | Transept Pro |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Sentence structure | +0.15 | 0.66 | 0.78 | 0.70 | 0.57 | 0.72 | 0.65 | 0.68 | 0.72 | 0.73 | 0.55 | 0.52 | 0.55 | 0.60 | 0.56 | 0.53 |
| Unnatural wording | +0.15 | 0.91 | 1.02 | 1.11 | 0.82 | 1.06 | 0.95 | 1.08 | 1.20 | 1.25 | 0.93 | 0.74 | 0.81 | 0.78 | 0.83 | 0.68 |
| Grammar (editor) | +0.14 | 0.16 | 0.15 | 0.11 | 0.04 | 0.15 | 0.10 | 0.17 | 0.20 | 0.19 | 0.11 | 0.07 | 0.07 | 0.08 | 0.06 | 0.03 |
| Too literal | +0.13 | 0.21 | 0.57 | 0.49 | 0.37 | 0.53 | 0.34 | 0.54 | 0.58 | 0.56 | 0.23 | 0.23 | 0.29 | 0.27 | 0.31 | 0.22 |
| Mechanics | +0.13 | 0.26 | 0.35 | 0.28 | 0.19 | 0.30 | 0.25 | 0.36 | 0.39 | 0.49 | 0.30 | 0.27 | 0.23 | 0.21 | 0.20 | 0.15 |
| Calque (editor) | +0.11 | 0.20 | 0.40 | 0.38 | 0.22 | 0.42 | 0.24 | 0.48 | 0.52 | 0.45 | 0.19 | 0.12 | 0.20 | 0.15 | 0.19 | 0.10 |
| Word choice (editor) | +0.10 | 0.22 | 0.21 | 0.17 | 0.10 | 0.19 | 0.18 | 0.23 | 0.33 | 0.34 | 0.17 | 0.12 | 0.13 | 0.10 | 0.10 | 0.07 |
| Register | +0.08 | 0.28 | 0.16 | 0.15 | 0.16 | 0.26 | 0.22 | 0.18 | 0.32 | 0.26 | 0.20 | 0.17 | 0.20 | 0.17 | 0.15 | 0.12 |
| Clarity (editor) | +0.04 | 0.06 | 0.12 | 0.08 | 0.03 | 0.09 | 0.07 | 0.14 | 0.10 | 0.11 | 0.07 | 0.03 | 0.05 | 0.06 | 0.05 | 0.03 |
| Wordy | +0.03 | 0.39 | 0.34 | 0.26 | 0.31 | 0.41 | 0.40 | 0.36 | 0.40 | 0.44 | 0.41 | 0.43 | 0.43 | 0.38 | 0.31 | 0.33 |
| Unclear | +0.02 | 0.37 | 0.54 | 0.48 | 0.46 | 0.51 | 0.41 | 0.64 | 0.50 | 0.50 | 0.41 | 0.35 | 0.42 | 0.46 | 0.51 | 0.42 |
| Punctuation (editor) | +0.02 | 0.12 | 0.27 | 0.15 | 0.29 | 0.15 | 0.09 | 0.15 | 0.36 | 0.68 | 0.18 | 0.11 | 0.11 | 0.12 | 0.07 | 0.06 |
| Style (editor) | −0.01 | 0.04 | 0.05 | 0.03 | 0.03 | 0.05 | 0.04 | 0.05 | 0.08 | 0.08 | 0.05 | 0.03 | 0.04 | 0.02 | 0.02 | 0.02 |
| Hard to follow | −0.02 | 0.31 | 0.44 | 0.30 | 0.35 | 0.34 | 0.36 | 0.56 | 0.45 | 0.46 | 0.40 | 0.36 | 0.37 | 0.37 | 0.37 | 0.34 |
| Flat | −0.05 | 0.24 | 0.15 | 0.15 | 0.19 | 0.19 | 0.17 | 0.18 | 0.18 | 0.16 | 0.23 | 0.18 | 0.20 | 0.19 | 0.17 | 0.17 |
How to read: the same opening translated by Opus and in my English version, with what the reader judges flagged.
«Наступного ранку, перед заняттями, Агнес зустріла Теодора у своєму кабінеті в смарагдовому вбранні із суворим коміром під саму шию…»
“The next morning, before classes, Agnes met Theodore in her office in an emerald outfit with a severe collar right up to the neck…”one long sentence, as Ukrainian allows.
“Agnes met Theodore in her office wearing an emerald outfit with a severe collar buttoned right up to the throat, from which bands fell away…”
Both reader judges flagged the sentence (hard to follow, sentence structure): it keeps the Ukrainian shape, and "wearing" could belong to Theodore. Later, «жіночний живіт» became "her womanly belly", also flagged.
“Agness received Theodore in her office. She wore emerald: a severe collar that closed high at her throat…”
and later "the soft curve of her stomach". Neither spot was flagged.
the kinds that track readers (sentence structure, literal phrasing, calques) are the ones a translation inherits from the shape of the original.
What didn't work, and what did we drop?
How to read: what we tried, what happened, and what we kept.
| Tried | Result | Decision |
|---|---|---|
| LAIT's own metrics (MetricX-QE, GEMBA-MQM, LiTransProQA) | Chance, 0.48–0.50 | Not used |
| Two versions side by side | Agreed less with readers; the verdict flipped with the order in 22% of passages | One version at a time |
| Weights fitted to readers | Lost to equal weights on held-out books | Equal weights |
| Sonnet or Fable as judges | Added nothing | Opus only |
| The previous passage as context | Didn't help | No context |
| Revised prompts (no quota, protect the voice, new kinds) | Agreement fell from +0.29 to +0.21 on the pilot | Kept the first prompts |
| A tense-shift judge | Found 4× more shifts and lowered agreement | A proofreading check, not part of the score |
| Copy-editing inside the reading score | It rewarded polish: AI's lead over the published translations grew from +0.51 to +0.57 | Reported apart |
| "Good wording" highlights | +0.08, and preferred the study MT to published translations | Not scored |
| Neutral framing ("this may not be a translation") | +0.17 against +0.25; 62% against 70% on clear passages | Kept "this is a translation" |
| A clarity note or more thinking, tuned on LAIT | +0.12 while tuning, +0.08 and +0.05 on held-out books | Nothing shipped |
| A slimmer score, 7 stumble kinds picked on the tuning books | +0.17 on held-out books against +0.16 | Kept the full score |
| Rubric losses as part of the reading score | −0.05 against readers | A separate axis |
Do the judges overrate AI?
Yes. On the 116 passages where both readers clearly preferred the published translation to the study MT, the judges agree on 90, yet still rank Transept Pro above the published version on 80. Across all 386 passages, Pro leads by +0.51 [+0.44, +0.58] (winning 295). No reader has compared the two; the judges are validated only on published translations against LAIT's study MT.
How to read: further right reads better; 0 is the average LAIT version, and overlapping intervals mean no real difference. Misreadings: lower is better, and 3 or more per 1,000 words fails the gate.
Source: reading score (Claude Opus 5.5 judges) on 135 passages, the 60 meaning passages plus 5 per book, which reproduce the full 386-passage scores within ±0.05. Misreadings: errors both runs of the meaning judge find, on the 60 meaning passages; the published translation's 1.78 includes deliberate liberties the judge still counts.
Show the data
| Version | Reading score | 95% interval | Misreadings per 1,000 words |
|---|---|---|---|
| Transept Pro | +0.74 | [+0.66, +0.82] | 0.11 |
| Gemini 4 Argon low through Transept | +0.58 | [+0.49, +0.68] | 0.17 |
| Gemini 4 Neon low through Transept | +0.47 | [+0.37, +0.57] | 0.12 |
| Transept Standard | +0.46 | [+0.36, +0.55] | 0.48 |
| Opus 5.5 high, plain API | +0.44 | [+0.34, +0.54] | 0.06 |
| Gemini 4 Argon high through Transept | +0.38 | [+0.28, +0.49] | 0.06 |
| Gemini 3.8 Flash low through Transept | +0.29 | [+0.16, +0.41] | 0.41 |
| Gemini 4 Neon high, plain API | +0.23 | [+0.10, +0.36] | 0.11 |
| Published human translation | +0.21 | [+0.09, +0.32] | 1.78 |
| Opus 5.5 low, plain API | +0.05 | [−0.07, +0.17] | 0.12 |
| Gemini 4 Neon low, plain API | −0.16 | [−0.29, −0.04] | 0.29 |
| Study MT (agentic pipeline) | −0.23 | [−0.38, −0.09] | 0.28 |
| Haiku 5.5 through Transept | −0.46 | [−0.62, −0.31] | 2.02 |
| Gemini 3.5 Flash Lite through Transept | −0.57 | [−0.72, −0.42] | 1.53 |
| Gemini 3.1 Flash Lite through Transept | −0.57 | [−0.70, −0.44] | 0.90 |
Strip stumbles and copy-editing from the questionnaire ratings and Pro's lead collapses from +0.51 to +0.05 (213 passages to 173, near a coin flip).
It isn't length (Pro is 0.99× published length), framing (hiding that it's a translation hurt agreement), or self-preference: GPT-6 puts Pro ahead just as far (+0.59 vs. Claude's +0.56 on 63 passages), pointing to a bias all model judges may share as LLMs.
Both families also under-picked Polish published translations (50% vs. readers' 83% on an earlier score).
Why do models overrate AI?
Because they prize target-language norms over authorial artifacts, and source structure over an artifact's effect. Both AI translators and AI judges make the same trade:
- They read a translator's choices as stumbles. Published translations draw more flags than Pro on register, flatness, and word choice, the kinds that barely track readers. That is plausibly where reworked emphasis and images land, though we haven't measured the link directly.
- They reward polish. Published translations need 0.26 copy-edit fixes per 100 words vs. Transept Pro's 0.08; counting copy-editing inside the score widens Pro's lead from +0.51 to +0.57. Unpolished prose loses to model normalisation whatever else it does well.
- With the original in view, they compare information rather than effect. Our first meaning judge flagged all 33 deliberate liberties in published translations as errors, and the unit tracker calls idiomatic equivalents "changed" so often that I agreed with only a third of those calls.
- As translators, they keep the structure. The study MT and Transept Pro rework 1–6% of each unit type where translators rework 19–43%; across all 13 machine versions the range is 1–19%, highest in the cheapest pipelines. Different AI systems also converge on nearly the same translation.
- They smooth what was deliberate. In our tests, a model made our assistant Literess's smug mood polite in Ukrainian, every system lost a Plans/plan pun, and an AI proofreader reverted coined terms.
The reading judges are a strong stumble meter and a poor judge of craft. Translators' craft (emphasis, images, register) stays mostly invisible to readers, and AI judges partly mark it down.
How to read: one idiom in my Ukrainian text and what each model made of it.
«І тоді Фрея пішла в дамки, як любили казати нью-йоркські ділки»
“And then Freya went to kings, as the New York dealmakers liked to say”a checkers image, the piece that reaches the far row and is crowned, set against Wall Street jargon.
“And that was when Freya went all in, as the New York hustlers liked to say.”
“went all in”
“went for the kill”
Every model replaced the checkers image with a stock English phrase, a poker bet or a hunt. The meaning shifts (a reckless gamble or an attack instead of a well-earned promotion), yet no reading judge flagged it, because the English is idiomatic, and the meaning judge didn't flag it either.
models trade an author's image for the target language's nearest standard phrase, and judges built on the same models read the trade as good English.
So, how do AI and human translation differ?
In short: AI translation keeps the original's information and structure, and polishes toward the norms of the target language. Human translation spends part of the information on voice: it reworks about a quarter of the units, adds detail, loosens density toward the target language, and differs from one translator to the next.
How to read: one trait per row, human and AI side by side, with where the data comes from.
| Human translators | AI translation | Data | |
|---|---|---|---|
| Units of the original kept, per 100 | 73–76 in published literary translation (55–93 across translators) | 89–98; GPT-4.1 76 | LAIT, ParaFarm, WMT25, PAR3 |
| Units added, per 100 | 9–11; the most expansive translator 21 | 1–5 | LAIT |
| Telling them apart | Keeps fewer units than the machine in 97% of passages | LAIT | |
| What gets reworked | Emphasis, images and details most (30–43%); speaker tags, events and facts least | The study MT and Transept Pro 1–6% of any unit type; all 13 machine versions 1–19%, highest in the cheapest pipelines | LAIT, both tracker runs |
| Consistency | Translators agree on which units are open (5× chance) but write very differently (similarity 0.71) | Systems converge on one translation (similarity 0.94) | ParaFarm |
| Texture | More particles, diminutives, varied vocabulary | Up to 2× fewer particles, 2.6× fewer diminutives, 18% lower lexical diversity | ParaFarm (Chaplynskyi et al.) |
| Density | Moves a quarter of the way toward a common English level; loosens introspection most | Carries the source density over | LAIT |
| Length against the machine | Longer in 59% of passages | LAIT | |
| Stumbles readers feel | Against the study MT: more register, flatness and wordiness flags. Against Transept Pro: more of nearly every kind | Study MT: more structure, unnatural, literal and calque flags. Transept Pro: the fewest of almost every kind | LAIT |
| Copy-editing, fixes per 100 words | 0.26 | Study MT 0.40; Transept Pro 0.08 | LAIT |
| Misreadings per 1,000 words | 1.8, including deliberate liberties | 0.06–2.0 | LAIT, 60 passages |
| Readers' choice against a strong MT | 67.6% | 32.4% | LAIT paper |
| Can readers tell which is which? | 34 of 60 guesses right | LAIT paper |
How to read: the study's machine translation against Bill Johnston's published one, for a common Polish phrase about failing legs and eyesight.
“his legs refuse obedience and his eyes no longer come together in seeing”
every unit of the Polish phrase survives, word by word (the legs refusing obedience, the eyes no longer coming together in seeing), and the English reads as a translation.
“when your legs won’t do what they’re told to and your eyes can’t make out the world.”
the same information, compressed into English idiom (legs that won't do what they're told, eyes that can't make out the world).
the machine keeps the information and the structure; the translator keeps the information and spends the structure on voice. Readers chose the published translation of Needle's Eye in 96% of its passages.
Wiesław Myśliwski, Needle’s Eye, Polish into English. Quoted in Ferstler et al. (2026), the same pair as in our voice guide.
…And what makes a translation good?
In short: For a reader who never sees the original: no stumbles and no padding. For an expert who reads it: no changed meaning. The translation that wins with both keeps the information and gives it a native voice.
How to read: what each audience rewards or punishes. "No effect" means the trait doesn't predict their choice.
| Predictor | Readers without the original (LAIT) | Experts with the original (PAR3, ParaFarm, WMT25) |
|---|---|---|
| Stumbles: sentence structure, unnatural or literal phrasing, calques, grammar | The strongest signal: +0.20 alone (+0.22 for the full reading score), as good as a second reader (+0.18) | Not measured |
| Keeping every unit | No effect (−0.03 to +0.05 within a book) | Up to a point: the professionals' first choice keeps 93 in 100, and heavy reworking ranks low (−0.53) |
| Changing meaning | No effect within a book | Punished: where versions differed, the chosen one had fewer changed units in 198 of 322 choices (61%) |
| Compressing or dropping | No effect within a book; slightly positive across books, not significant | No consistent direction |
| Adding | No effect within a book | Not punished |
| Expanding on net | −0.36 across books, not significant; the most expansive translation lost 90% of its passages | The longest Animal Farm translation ranked sixth of ten |
| Reworking emphasis | Slightly positive (+0.08) | Part of reworking |
| Register, flatness, wordiness, unclear passages | No effect | Not measured |
| Copy-editing | Weak (+0.13) | Not measured |
What makes human translators still better?
- Readers prefer them. They beat a strong MT 67.6% of the time; AI's only edge is our judges' stumble count, which collapses to +0.05 once stumbles are stripped out.
- They rework where it pays. Translators agree on which units are open (emphasis, detail, feeling, image) and leave plot, facts and speakers alone. Readers even slightly favour reworked emphasis.
- They protect an artifact's effect over its structure. A pun rebuilt, a register kept, a motif carried across paragraphs: things AI judges either can't see or mark down.
- They vary. Seven translators of Animal Farm wrote seven different books; three AI systems wrote nearly one.
Translators lose to the machine when they pad (Hooked) or break reading flow. The best human work in our data (Stelmakh's Animal Farm) does both jobs at once: it keeps nearly as much of the original as a machine, in a native voice.
How to read: a coined name and the pun built on it, in three versions.
«Щоб творити добро, якщо їй так хочеться робити Плани кращими… Але принаймні в Анни є її план. У нього ж, Теодора, Плану не було. Ані місця в Планах.»
“To do good, if she wants so badly to make the Plans better… But at least Anna has her plan. He, Theodore, had no Plan. Nor a place in the Plans.”«Плани», "the Plans", is the name of the novel's worlds.
“making the Plans a better place… But at least Anna has her plan. He, Theodore, had no Plan. Nor any place in the Plans.”
Every unit kept, and the play on plan, Plan and the Plans survives.
“make the Planes better… Theodore himself had no Plan. Nor any place in the Planes.”
The name became "Planes" and the pun broke. Sonnet switched between "Planes" and "Plans" within two lines.
a good translation keeps the information and the author's invention, and says both in the target language.
What can't we measure yet?
- Voice, and the right amount of information for the moment. The unit tracker counts units; it can't tell whether a cut served the scene. Our judges check information or surface flow, not whether a sentence does to the reader what the original did.
- AI against published translations. No reader has compared them. Our judges are validated only on published translations against a weaker MT.
- Scale. The book-level results rest on 15 books, LAIT translates only into English, and our Ukrainian results come from a handful of texts.
- Model judges. GPT-6 agrees with Claude, which rules out self-preference but not a preference all models share.
How did we use AI in this research?
Claude Code, an AI coding agent, wrote the analysis scripts and orchestrated running the LLM judges. I set the questions and the method, checked 78 of the unit tracker's claims by hand, and made the editorial calls.
Literess from Literess.com, our AI writing assistant, helped edit and tighten the draft.
We don't reproduce any LAIT passages here.
The judge and rubric prompts, the scoring scripts and worked examples are public: Transept-AI/translation-judges on GitHub.
Sources
- Ferstler, Y. et al. (2026). AI translation of literary texts is "fine", but readers still prefer human translations. Findings of EMNLP 2026 (arXiv:2606.26040). The LAIT study; our numbers on its data are published with citation, its passages are not reproduced.
- Chaplynskyi et al. (2026). Semantic Fidelity Versus Literary Quality. UNLP 2026 (ACL Anthology 2026.unlp-1.8). ParaFarm corpus: Zenodo, doi 10.5281/zenodo.20370169 (CC BY-NC 4.0).
- Thai, K. et al. (2022). Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature. EMNLP 2022. PAR3 and its expert annotations.
- WMT25 General Machine Translation shared task: human evaluation (ESA), literary domain, English into Ukrainian and Russian.
- Zhang, R., Zhao, W., & Eger, S. (2025). LitEval-Corpus. NAACL 2025 (arXiv:2410.18697).
- Guerberof-Arenas, A., & Toral, A. (2022). Creativity in translation: machine translation as a constraint for literary texts. Translation Spaces, 11(2), 184–212.
- Toral, A. (2019). Post-editese: an Exacerbated Translationese. Proceedings of MT Summit XVII, 273–281.
- Vlasiuk, V. (2026). How to translate a book without losing voice. Transept Guides.
- Transept (2026). translation-judges: the judge and rubric prompts, scoring scripts and worked examples behind this article. GitHub.
Novels. The 15 LAIT books, each with its published English translation, and the novella behind ParaFarm:
- Asako Yuzuki, Hooked, translated from Japanese by Polly Barton (Fourth Estate, 2026).
- Andrzej Sapkowski, The Witcher: Crossroads of Ravens, translated from Polish by David French (Orbit, 2025).
- Thomas Schlesser, Mona's Eyes, translated from French by Hildegarde Serle (Europa Editions, 2025).
- Haruo Yuki, The Ark, translated from Japanese by Jim Rion (Pushkin Vertigo, 2026).
- Jakub Szamałek, Inner Space, translated from Polish by Kasia Beresford (HarperVia, 2025).
- Marceau Miller, The Story of Marceau Miller, translated from French by Howard Curtis (Blackstone Publishing, 2026).
- Maud Ventura, Make Me Famous, translated from French by Gretchen Schmid (HarperVia, 2025).
- Jean-Baptiste Andrea, Watching Over Her, translated from French by Frank Wynne (Atlantic Books, 2025).
- Mikołaj Łoziński, My Name Is Stramer, translated from Polish by Antonia Lloyd-Jones (Pushkin Press, 2025).
- Mieko Kawakami, Sisters in Yellow, translated from Japanese by Laurel Taylor and Hitomi Yoshio (Knopf, 2026).
- Olga Tokarczuk, The Empusium: A Health Resort Horror Story, translated from Polish by Antonia Lloyd-Jones (Fitzcarraldo Editions, 2024).
- Masateru Konishi, My Grandfather, the Master Detective, translated from Japanese by Louise Heal Kawai (G.P. Putnam's Sons, 2026).
- Nahoko Uehashi, Kokun: The Girl from the West, translated from Japanese by Cathy Hirano (Europa Editions, 2026).
- Wiesław Myśliwski, Needle's Eye, translated from Polish by Bill Johnston (Archipelago Books, 2025).
- Marc Levy, Symphony of Monsters, translated from French by Tina Kover (HarperVia, 2026).
- George Orwell, Animal Farm (1945), in the seven published Ukrainian translations ParaFarm compares.
The author

Co-founder of Transept, writing as “Mevkh.” A Language and Literature degree, then a turn into software: senior AI engineer shipping production LLM features to 50,000+ users — RAG, agentic tools, LLM-as-judge evaluation. A novelist on the slow path, with 120,000 words of satirical romance fantasy in a drawer. The friction between AI translation and his own prose is what set this whole thing in motion.