AI that writes seems to think. What it actually does is simpler — and stranger
A language model — the technology behind ChatGPT, Claude or Gemini — does not think like a person: by reading enormous amounts of text, it learns which word tends to follow another. That is enough to write letters, summarize documents or translate. But it also explains its failures: when it doesn't know something, it makes it up with total confidence, because its job is to continue the text, not to tell the truth. Knowing this protects you more than any manual.
A machine for continuing sentences
Imagine someone who has read almost everything on the internet: books, newspapers, forums, recipes, laws. They didn’t memorize every text, but they did learn the patterns: after “once upon a” almost always comes “time”; after a medical question usually comes an answer with a certain tone. A large language model (LLM) is that: a mathematical system trained to predict the next word.
When you type a question, it doesn’t look up the answer in a database. It generates it, word by word, each time choosing the most likely continuation given everything it has read. That the result sounds intelligent is not magic: human language has an enormous amount of structure, and the machine learned it.
Why it gets so much right
Scale changes things. With enough data and enough computing power, “predicting the next word” produces abilities that surprised even the researchers: translating, summarizing, writing code, explaining a topic in simple words. That is why these tools feel genuinely useful: for drafting a letter, understanding a contract or preparing a class, they save hours.
Why it fails so badly
The same mechanism explains its failures. The model does not distinguish between what it knows and what it doesn’t: ask it about a law that doesn’t exist, and it can “cite” it, article number and all, because its task is to continue the text plausibly. This is called hallucination, and it is not a rare defect: it is part of the design. It also inherits the biases of the texts it was trained on — and most of those texts are in English and come from a small part of the world.
What this means for you
Three practical rules. First, use it as a draft, never as a final source — verify facts, dates, laws and medicines against real sources. Second, don’t hand it private information about yourself or others: what you type may be used to train future models. Third, remember who is answering: not a neutral sage, but a statistical mirror of the internet’s texts, with its accuracies and its prejudices.
AI that writes is neither an oracle nor a fraud: it is a new tool, powerful and sharp-edged. Knowing the mechanism — predicting the next word — is the best protection for using it without being used.