The Paper That Started It All Was About Guessing the Next Word

In 2017 a group of researchers published a paper called Attention Is All You Need. It introduced the transformer — the architecture behind nearly every AI tool you use today. The breakthrough wasn't a machine that understood meaning. It was a machine that got extremely good at one mundane task: guessing what comes next.

The whole trick, stated plainly

A language model is trained on enormous amounts of text. Each time, it sees a sequence of words and tries to predict the next one. Get it wrong, adjust. Get it right, reinforce. Repeat this trillions of times and something unexpected happens: to predict the next word well, the model is forced to internalize grammar, tone, facts, argument structure and the way ideas usually follow one another.

Nobody taught it grammar. Grammar turned out to be the cheapest way to predict the next word.

Why this matters to you

Once you understand that the model is completing a pattern rather than retrieving a fact, a lot of confusing behavior becomes predictable.

• It will continue a confident-sounding sentence with more confidence than it has evidence for.
• It mirrors the register of your prompt — sloppy in, sloppy out.
• It is strongest where patterns are dense, which is why it writes a decent email and a shakier legal opinion.

The practical move

Stop asking 'does the AI know this?' and start asking 'what pattern am I asking it to continue?' Give it a strong, specific pattern — an example of the output you want — and the completion gets dramatically better. That single shift is the difference between people who find AI useful and people who find it overhyped.

The full version of this mental model, with worked examples across writing, analysis and client work, is inside Prompt Foundations — one of the courses in the bundle.