I translated my novel with ChatGPT, Claude, Gemini and Google Translate. Here’s how it went

I’m writing a novel about Erasmus students in Ostrava. It has a Ukrainian heroine, a Spanish flirt, Turkish roommates, a dorm group chat that never sleeps, and a lot of beer. In chapter 7, Sofia tells her friend Aybars about Claude, and he asks, “Who tf is Claude?” She explains: “Like ChatGPT but different. Better. Smarter…”

So when we wanted to know which AI tool keeps a novel consistent, I knew which book to use. This test made ChatGPT and Claude translate a sentence that ranks them. (Both translated it faithfully. Nobody defended themselves.)

What did we test?

We translated the first nine chapters of my novel from English into Ukrainian with every tool a writer is likely to try first, then counted what changed between chapters.

We ran each chatbot three ways, because writers use them three ways:

1. One long conversation: “I’ll send you my novel chapter by chapter,” then each chapter in the same chat.
2. A fresh chat for every chapter, which is what happens when a conversation gets too long and you start a new one.
3. A fresh chat for every chapter, with a list of names pasted in: “Use these Ukrainian spellings: Nazli → Назлі, Jeff → Джефф…”

Every run got the same short request: translate the chapter in full, keep the tone, the jokes, the slang and the formatting.

Why translate into Ukrainian?

Because Ukrainian makes mistakes countable. Names take case endings (Софія, Софії, Софією), so a translation that writes the same person two ways is easy to tell apart from grammar. And Ukrainian makes every speaker choose between informal “ти” and formal “ви”. Friends who say “ти” in chapter 2 and “ви” in chapter 6 are a slip a native reader feels at once, and a script can find.

How did we count?

We counted three things in every version:

• Names that changed spelling across the nine chapters. Grammar didn’t count: case endings, the possessive “Софіїн” (Sofia’s), the adjective “празький” (from Prague), names inside lines I wrote in Spanish, and my own nickname for Giovanni.
• Dropped lines: sentences, chat messages or dialogue lines whose content is missing from the translation.
• ти/ви slips: one friend addressing another friend with formal “ви”. Meeting a stranger politely didn’t count, and neither did “ви” said to a whole group.

Two AI judges from different companies, Claude Opus 5.5 and Gemini 3.1 Pro, read every chapter of every version against the English. A finding counted only when both flagged it and a script found the exact words in the text; then we read each one in context. For dropped lines we also compared question marks chapter by chapter, which catches a lost line without any AI involved.

What drifted?

What happens to names in a fresh chat?

Meet Nazli, Sofia’s Turkish roommate. In Ukrainian her name can be written Назлі or Назли, and both look fine on their own. So every chapter reads as correct, and the book doesn’t.

Claude, given a fresh chat for every chapter, wrote “Назли” 24 times in chapters 2 and 7 and “Назлі” everywhere else. Gemini split her the other way round, “Назли” in six chapters and “Назлі” in three. ChatGPT made her “Назли” in chapters 5 and 7. Claude also turned Jeff into “Джеф” for two chapters in the middle of three chapters of “Джефф”, and Mariana into “Мар’яна” in the last chapter after four chapters of “Маріана”.

None of this happened in one long conversation, where the model can see how it wrote Nazli in chapter 1. And none of it happened in a fresh chat with a names list pasted in. One page of names fixed every case.

Google Translate needs no chat, and it kept every other name steady. Its one slip was Mariana, who became “Мар’яна” once in chapter 5, between two “Маріана”s.

Did any tool drop lines?

No. In 14,919 words, no version lost a sentence, a chat message or a line of dialogue. Both judges found nothing, and every version kept the source’s question marks chapter by chapter (ChatGPT’s turned three into statements, because a game of Never Have I Ever reads more naturally that way in Ukrainian; the lines are all there).

I expected some. Missing lines are the classic complaint about AI translation of long texts, and on a chapter at a time, in October 2026, we didn’t see one.

Since when are we on “ви”?

This is where Google Translate fell behind. Five times, a friend suddenly turned formal: Diego, mid-flirt, asking Sofia “Зачекайте, і ви думаєте, що це багато?”, and a postcard Sofia writes to Nazli that switches to “ви” halfway through. None of the chatbots did this once.

Our guess at why: a chatbot reads the whole chapter at once, and the friendship is in the chapter. Google’s translator seems to look at much less of the text at a time, so it can’t see that Diego and Sofia have been drinking together for three chapters.

What else did we notice?

• One of my characters is called Ege. In Ukrainian, “еге” means “uh-huh”. Most versions wrote him “Еге”, so his group-chat messages open with “Yeah:”. Two spelled him “Еґе”, with the hard Ukrainian ґ, and dodged the joke.
• Gemini left one line in English in two of its three runs: “Diego! Man, good to see you!”

How did each tool translate “Who tf is Claude?”

Aybars’s question in chapter 7 turned into a ladder of Ukrainian swearing. Here it is from each tool’s fresh-chat run:

Google and Gemini also turned Claude into “Клод”, as if he were a Frenchman. The others kept the name in Latin letters.

Which version read best?

Counting mistakes says nothing about whether the Ukrainian is a pleasure to read, so we asked the two judges that too. For three chapters, the dialogue-heavy “The Devil Arrives”, the multilingual “D7 Intro” and the long “Ginger beer”, each judge got the English and five unlabelled Ukrainian versions, one per tool, and ranked them as a Ukrainian literary editor would. Every chapter went to each judge twice, with the versions in a different order. The chatbots ran with their names lists, so each tool was at its best.

Place 1 is the best of five. Transept’s Standard mode runs on Claude Opus 5.5, the same model as one judge.

Each judge’s favourite runs on its own company’s model: Opus put Transept first five times out of six, and Gemini Pro put Gemini first every time. Read the top of the table with that in mind. The rest the judges agreed on without exception. ChatGPT came out solidly in the middle, Claude (Sonnet, the model its free plan uses) fourth every time, and Google Translate last in every one of the twelve rankings.

The judges’ notes say why. Claude kept “love” in English when Diego says “Thank you, love”, and calqued “small talk” word for word. Gemini’s slang was the liveliest of the five, and sometimes went further than I wrote it. Google got grammar wrong in ways no chatbot did: Sofia, a woman, answers “Sure, I’m free” in the masculine.

What was the worst mistake?

It was Google’s, and it happened in one word. Sofia, laughing, asks Diego: “Are you kidding me? That’s why you ghosted me?” Google Translate wrote “Тому ти мене й зґвалтував?”, which means “So that’s why you raped me?”

A rom-com beat became an accusation of a crime, in fluent, correct Ukrainian, with nothing on the page to warn a reader who doesn’t know the English. If you take one thing from this test, take this: read every line of a machine translation before anyone else does.

What should you do if you translate your book with AI?

• Give the tool your names once and make it keep them. In a chatbot, that means one long conversation or a names list pasted into every new chat. In a translation tool, it means a glossary.
• Read the dialogue aloud in the new language, and listen for a friend who suddenly sounds like a bank.
• Check the chapters where a new chat began. That’s where every name in this test drifted.
• Test the tool on your own chapters, not on a sample paragraph. The trouble starts when the book gets long.

What this test doesn’t show

• One book, one direction. English into Ukrainian, contemporary fiction with a lot of dialogue. A fantasy saga with 200 invented names, or Japanese into English, would stress the tools differently.
• The chatbots ran through their APIs, on the models their free plans use, without the apps’ own instructions and memory. The ChatGPT, Claude and Gemini apps may behave a little differently.
• The judges are AIs. Two of them, from different companies, and every finding was checked against the text by a script and then read in context. There was no judge from OpenAI.
• We make Transept. We ran it the way we tell writers to run it, with a glossary and the novel workflow, and we kept every output.