The whole idea in 30 seconds
- An LLM is a prediction engine. It reads your text and guesses the most likely next piece, over and over. That is the entire trick.
- It works because of the transformer, whose key move is attention: every word looks at every other word to figure out what it means in context.
- It does not know facts and has no memory. It produces what is plausible, not what is true, which is exactly why it hallucinates.
- Nearly every practical question, cost, speed, reliability, prompting, has a direct answer once you see the mechanism.
Opening
The autocomplete that learned to reason
Everyone has used the autocomplete on a phone. You type “I’ll be there in a” and it offers “minute,” “few,” “bit.” It is a small, dumb guessing machine, and no one mistakes it for intelligence. Here is the fact that should reframe how you think about the entire AI era: a large language model is the same machine. It predicts the next piece of text. That is the whole trick.
The obvious question is how something so simple can draft a legal summary, debug software, or pass a medical exam. The answer is the most important idea in modern AI, and it is worth sitting with. When you make a next-word predictor large enough, train it on enough of humanity’s written output, and give it the right architecture, something unexpected happens. To predict the next word well across billions of examples, the machine is forced to build internal representations of grammar, facts, relationships, tone, and something that functions a great deal like reasoning. It turns out that predicting the next word, done well enough, requires a working model of the world the words describe.
That is the whole story in miniature. The rest of this guide opens the machine up, one honest step at a time, so you can see how a glorified autocomplete became something you are now considering putting at the center of your enterprise. No mathematics, but no hand-waving either. And at every step, the point that matters to you: what this mechanism means for cost, for risk, and for how you should actually use the thing.
The Map
The big picture, in one diagram
Before the details, hold the shape of the whole thing in your head. When you send a prompt to an LLM, it travels through a short, fixed sequence of stages. Your words are broken into tokens. Each token is turned into a list of numbers, an embedding, that captures its meaning. Those numbers flow through the transformer, a stack of layers whose central operation, attention, lets every token take account of every other. Out the far end comes a single prediction: the most likely next token. Then, crucially, the whole process repeats, adding one token at a time, until the answer is done.
That last point is not a detail; it is the essence. The model does not compose a whole answer the way you plan a sentence. It predicts one token, appends it, and runs the entire prompt-plus-answer-so-far through again to predict the next. An LLM writing a paragraph is a machine taking a few hundred tiny, sequential guesses, each informed by everything before it.
Where It Comes From
How the machine learned in the first place
Before we walk the machine step by step, one question deserves an answer, because every leader eventually asks it: where does the intelligence come from? A model is not programmed with facts and rules. It is trained, in three distinct phases, and understanding them clears up a great deal of confusion, including why the model you use feels so different from the raw technology underneath.
The first phase is pre-training. The model is shown an enormous slice of humanity's written output, a large fraction of the public internet, books, code, and made to play one game billions of times: predict the next token, check against the real one, adjust. Repeated at vast scale, this is where the model absorbs grammar, facts, reasoning patterns, and the structure of the world as language describes it. Pre-training is what makes the model capable. It is also staggeringly expensive, costing tens or hundreds of millions of dollars, and it happens once.
The raw result of pre-training, though, is not the helpful assistant you interact with. It is a brilliant, unfocused text-completer that will happily continue your question with ten more questions. The second phase, fine-tuning, fixes this by showing the model many examples of instructions paired with good responses, teaching it to follow instructions rather than merely continue text. The third phase, alignment, usually through a process called reinforcement learning from human feedback, has people rank competing answers so the model learns which responses are actually preferred, teaching it to be helpful, honest and safe. This last phase is the difference between a raw model and one you would put in front of a customer.
This three-phase picture explains a distinction that matters enormously for your budget, and that vendors do not always make clear.
Training is a one-time build cost, borne by whoever creates the model. What you pay for, almost always, is inference: the cost of running the finished model every time you use it. Training is the factory; inference is the electricity bill that never stops. This is why the earlier point about cost-per-token is the one that governs your economics, and why a cheaper-to-run model can matter more to your bottom line than a more capable one you invoke less often.
Step One
Words become tokens
The model does not read letters or words. It reads tokens: chunks of text that are often whole words, but frequently pieces of words. Common words like “the” are a single token; a longer or rarer word like “unbelievable” is broken into familiar fragments such as “un,” “believ,” and “able.” This is simply a more efficient way to cover all possible text with a manageable vocabulary of pieces.
Why this matters to you
Tokens are the unit you pay for
Every commercial LLM is priced per token, in and out. That single fact drives your API bill. It also has a hidden edge: the same document costs roughly twice as many tokens in some languages (Chinese, Japanese) as in English, so multilingual workloads cost more than a word count suggests. When a vendor quotes you a price, they are quoting you per token, and now you know exactly what that means.
Step Two
Tokens become meaning
A token by itself is just an entry in a dictionary of pieces. To do anything useful, the model converts each token into a long list of numbers called an embedding. The right way to picture this is a vast space of meaning, in which every token is a point. Tokens with similar meanings sit near one another; unrelated ones sit far apart. “King” and “queen” are neighbours; “Paris” and “France” cluster together; “banana” is off in a distant region of its own.
This is the first place the machine stops being a lookup table and starts to look like understanding. Because meaning is now geometry, the model can do arithmetic with concepts. The famous example: the direction from “man” to “woman” is roughly the same as the direction from “king” to “queen.” The model was never taught this. It emerged from the numbers, as a consequence of learning to predict text. But there is a limit worth noting: at this stage, each token’s embedding is still fixed, the same regardless of the sentence it sits in. The word “bank” has one embedding whether you mean a river or a vault. Fixing that is the job of the next, and most important, step.
Step Three · The Core Idea
Attention: every word looks at every other word
Here is the breakthrough that made modern AI possible, introduced in a 2017 paper whose title, “Attention Is All You Need,” turned out to be prophetic. The idea is called attention, and it is worth understanding because it is the single mechanism that separates today’s AI from everything before it.
Consider the sentence: “The trophy did not fit in the suitcase because it was too big.” What does “it” refer to, the trophy or the suitcase? You know instantly that it is the trophy, because a thing that is too big fails to fit. This requires understanding the relationship between distant words in the sentence. Attention is exactly the mechanism by which the model does the same thing: for every word, it looks back at every other word and works out how much each one matters to the current word’s meaning.
This is why the embedding for “bank” stops being fixed. After attention, a token’s meaning is updated by its neighbours: “bank” next to “river” drifts toward one region of meaning, “bank” next to “deposit” toward another. The representation becomes contextual. And because attention compares every token with every other token, it captures relationships across an entire document at once, which is precisely what older approaches, reading strictly left to right, could never do well.
A Closer Look
How attention actually works
You can lead perfectly well knowing only what you have just read. But the next level of detail is genuinely illuminating, and it demystifies a piece of jargon you will hear, query, key and value. The cleanest analogy is searching a library. (And if you or your engineering leaders want the full technical treatment, with the actual mathematics and the reasoning behind each design choice, the companion piece to this guide, Inside the Transformer, goes all the way down.)
Every token generates three things. A query: what it is looking for, as if the word “it” holds up a request that says “I am a pronoun, what physical object do I refer to?” A key: a label advertising what the token offers, so “trophy” holds up a tag that says “I am a physical object.” And a value: the actual meaning it will contribute if it is chosen. The model matches each word’s query against every word’s key, and the words whose keys best fit the query pass their value along most strongly.
Real models run many of these searches in parallel, so-called multi-head attention, with each “head” learning to look for a different kind of relationship. One head might track grammar, another might track which pronoun refers to which noun, another the tone. They run simultaneously and their findings are combined. It is less a single reader and more a committee of specialists, each examining the same sentence through a different lens.
Step Four
Depth: the same trick, many times over
A transformer does not run attention once. It stacks the operation into dozens of layers, each taking the enriched, context-aware output of the one below and refining it further. The effect is a deepening of understanding. Early layers handle the mechanical: grammar, word boundaries, simple associations. Middle layers assemble phrases and track who did what to whom. The deepest layers capture intent, tone, and the kind of meaning that lets the model respond sensibly rather than just grammatically.
This is what people are pointing at, loosely, when they talk about a model’s size, its billions of parameters. The parameters are the adjustable numbers, learned during training, that govern every embedding and every attention step across every layer. More parameters mean more capacity to represent subtle patterns. It also means more computation for every single token, which is the direct reason larger, more capable models cost more and respond more slowly. Capability and cost are two readings of the same dial.
Step Five
The prediction, and the loop
After the text has flowed down through every layer, the model arrives at the only thing it was ever built to do: assign a probability to every possible next token. Not one answer, a ranked list of candidates, each with a likelihood. For “The capital of France is,” the token “Paris” will sit at the top with overwhelming probability, with a long tail of less likely options beneath it.
The model selects a high-probability token, appends it to the text, and then does the entire journey again, tokens, embeddings, attention, layers, prediction, to produce the next one. Word by word, guess by guess, the answer accumulates. A subtle but important knob here is that the model does not always pick the single most likely token; a setting often called temperature controls how much it is allowed to choose lower-ranked options, which is the difference between a predictable, repetitive answer and a more varied, creative one.
The New Generation
The reasoning models everyone is talking about
If everything so far describes a machine that predicts the next token, you may wonder how that squares with the “reasoning” models now dominating the conversation, systems like OpenAI’s o-series, DeepSeek’s R1, and the thinking modes built into the latest models. The reassuring answer: they run on exactly the mechanism you now understand. They simply use it differently.
A standard model, asked a hard question, predicts its answer immediately, the equivalent of blurting out the first thing that comes to mind. A reasoning model is trained to do something more human first: to think on a scratchpad. Before committing to an answer, it generates a long internal chain of tokens, working through the problem step by step, exploring approaches, catching its own contradictions, and backtracking when something does not add up. Only then does it produce the final response. The technical name for this is spending test-time compute: paying for extra thinking at the moment of the question rather than only during training.
Crucially, this is not a different kind of machine. It is the same next-token predictor, taught through reinforcement learning to use its own output as a workspace, the way you would use scratch paper for a hard sum. This is why reasoning models are dramatically better at math, logic and complex code, where a single wrong step ruins the answer, and why they are slower and cost more, since all that thinking is more generated tokens that you pay for. The practical lesson for a leader is simply this: match the model to the task. Routine drafting does not need a reasoning model; a thorny analytical problem may be worth the extra cost and wait.
Why It Matters
Why this one mechanism explains almost everything
Here is the payoff for having sat through the machinery. Nearly every practical, business-relevant behavior of an LLM, the things that determine whether you can trust it, what it costs, and how to get good results, follows directly and predictably from the mechanism you now understand. Once you see it, LLMs stop being magic boxes and become something you can reason about.
Take the most important one: hallucination. An LLM does not look facts up. It generates the most plausible continuation, and a plausible-sounding falsehood scores just as well, by its internal measure, as the truth. This is not a bug to be patched away; it is a direct consequence of being a probability engine rather than a database. It is precisely why techniques that supply the model with verified documents at question time, retrieval-augmented generation, reduce hallucination: they load the context with true material so the most plausible continuation is also the correct one. Understanding the cause tells you the cure.
The same logic explains the rest. The model has no memory between turns; it only ever sees what fits in its current context window, which is why it “forgets” the start of a long conversation, everything outside that window simply does not exist to it. Because attention compares every token with every other, doubling the input roughly quadruples the work, which is why long documents cost more and respond slower, and why stuffing an entire archive into one prompt is often worse, not better, than retrieving the relevant pieces. And because the model has a documented tendency to attend most to the beginning and end of its input, important instructions buried in the middle of a long document can be quietly underweighted, the “lost in the middle” problem. Even prompt engineering is explained: a clear, well-structured prompt with examples steers the probability distribution toward the output you want. You are not persuading an intelligence; you are shaping the statistical field from which it draws.
The single most useful sentence in this guide
It predicts the plausible, not the true
If you remember one thing, remember this. It tells you when to trust an LLM (where plausible and true tend to coincide, and where a human checks the output) and when not to (high-stakes facts with no verification). Every governance decision in your enterprise about AI can start from this single, honest sentence.
Putting It To Work
The decisions this puts in your hands
Understanding the machine is only worth the effort if it sharpens your decisions, so here are the three that matter most, each now easy to reason about.
The first is how to make a general model work for your business. There are three options, and a common, expensive mistake is reaching for the hardest one first. You can prompt: simply instruct the model well, with clear context and examples, which costs nothing to set up and is where you should always start. You can add retrieval, RAG: fetch your own documents and feed them to the model as context at the moment of the question, which grounds its answers in your data and is where most enterprise value actually lives. Or you can fine-tune: further-train the model on your own examples, which is powerful but costly, needs quality data, and requires ongoing upkeep. The right order is usually prompt, then RAG, then fine-tune only if the first two genuinely fall short.
The second decision follows from a development you have surely noticed: models no longer handle only text. The same transformer architecture, with minor adaptations, now processes images, audio and video by turning them into tokens too. A patch of an image or a slice of sound becomes a token, and the identical attention machinery applies. This is why one model can now look at a chart, read a document, and answer a spoken question. For you, it means the mental model in this guide does not expire as AI becomes multimodal; it extends. The engine is the same; only the fuel has diversified.
The third is the one that connects this guide to the strategy editions of this series: how a text predictor becomes an agent that takes action. The bridge is smaller than it appears. If a model can predict text, it can be made to predict a specific kind of text, a decision to use a tool: to run a search, query a database, send an email. The tool runs, the result comes back as more text, and the model predicts the next step, looping until the goal is met. That is the whole of it.
This is why agents are the natural, and consequential, next step in enterprise AI, and why the later editions of this series treat their governance and security so seriously. An agent is not a new intelligence. It is the prediction engine you now understand, pointed at actions instead of prose, and given the freedom to loop. Everything powerful and everything dangerous about agentic AI follows from that single extension.
Conclusion
What a leader should take away
You now understand, genuinely and not superficially, what happens inside the systems reshaping your industry. Text is broken into tokens, tokens become points in a space of meaning, attention lets every piece take account of every other, dozens of layers deepen that understanding, and the whole apparatus resolves into a single, humble act repeated at speed: predicting the next token. That is the machine. It is simpler than the mystique suggests, and more remarkable, because so much apparent intelligence emerges from so plain a foundation.
The reason this is worth a leader’s time is not intellectual tourism. It is that every serious decision you will make about AI rests on this foundation. Whether to let a model act on its own or keep a human in the loop, what to budget, which tasks to trust it with and which to fence off, how to write policy that engineers will respect, all of it becomes clearer, and more defensible, once you can reason from how the thing actually works. The leaders who thrive in this era will not be the ones who can derive the mathematics. They will be the ones who refused to treat the machine as magic, and understood it well enough to deploy it with judgment. You are now one of them.
Key Takeaways
- An LLM is a next-token prediction engine. It reads your text and repeatedly guesses the most likely next piece. That is the entire mechanism.
- Tokens are the unit of everything, including cost. You are billed per token, and some languages cost roughly twice as many for the same content.
- Embeddings turn meaning into geometry, and attention makes it contextual by letting every token weigh every other. Attention is the core breakthrough.
- Depth is understanding: dozens of layers refine meaning, and more parameters mean more capability and more cost, the same dial read two ways.
- It predicts the plausible, not the true. Hallucination, forgetting, cost, the “lost in the middle” problem and why prompting works are all direct consequences.
- Understanding the mechanism is a leadership tool, not a technical indulgence. Nearly every AI decision you face is easier once you can reason from how it works.
The one line to carry into your next AI meeting
It is not a mind that knows things. It is a machine that predicts what comes next, extraordinarily well. Lead accordingly.
Selected Sources & Further Reading
- Vaswani et al., “Attention Is All You Need,” 2017, the paper that introduced the transformer architecture.
- Jurafsky & Martin, Speech and Language Processing (3rd edition drafts, 2024 to 2025), chapters on attention and transformers.
- IBM, “What is a context window?” and related explainer material, 2025 to 2026.
- Stanford CS224N and Elena Voita’s open NLP course, for accessible technical treatments of attention.
- Research and industry writing on next-token prediction, tokenization economics, the “lost in the middle” effect, and retrieval-augmented generation, 2023 to 2026.
- CXO Intelligence Series: Inside the Transformer, the technical Deep Dive companion to this primer, for engineers and CTOs; and Editions 01 to 04, on the strategy, moat, redesign and security of the AI-native enterprise.
This guide deliberately simplifies. The diagrams are conceptual, chosen to build accurate intuition rather than to depict the underlying mathematics exactly. Where a simplification is significant, the text notes it. Technical readers should treat this as a mental model, not a specification.