Inside the Transformer: A Technical Deep Dive for Engineers and CTOs

For the reader in a hurry

A transformer maps a sequence of token embeddings to a next-token distribution through a stack of identical blocks, each built from attention and a feed-forward network, wrapped in residuals and normalization.
The core operation is scaled dot-product attention, softmax(QKᵀ/√dₖ)V, run in parallel across multiple heads.
It beat RNNs for two concrete reasons: parallelism across the sequence, and direct long-range connections between any two positions.
The modern stack has converged: pre-norm RMSNorm, RoPE, SwiGLU, GQA, and often MoE, with FlashAttention and KV-caching making it efficient, all fighting attention’s O(n²) cost.
Every model you buy is this architecture. GPT, Claude, Llama, DeepSeek differ only in scale and efficiency choices, and those choices can move training and inference economics by an order of magnitude.

Opening

The eight-page paper that ate the world

In June 2017, eight researchers at Google published a paper with a title that sounded like a joke: “Attention Is All You Need.” It proposed throwing away the machinery the entire field had spent decades refining, the recurrent networks that read text one word at a time, and replacing all of it with a single mechanism. The paper was written for a machine-translation benchmark. Nobody involved was claiming to have found the architecture of general intelligence.

They had. Every system now reshaping your industry, GPT, Claude, Gemini, Llama, DeepSeek, is that same architecture, scaled. The authors went on to found or lead companies now worth billions, and the eight pages they wrote became arguably the most economically consequential technical document of the century so far. Understanding what those pages actually say is no longer an academic exercise. It is understanding the engine your enterprise is being rebuilt around.

This deep dive walks the architecture honestly: the real mechanism, the actual mathematics, and, at every step, the reasoning behind the design choice, because the choices are where the insight lives. Each technical section is paired with a plain-terms translation, so you can read at full rigor or hand this to someone who needs the intuition. By the end, you will be able to look at any model on the market and see exactly what it is made of.

The Setup

The problem the transformer solves

A language model approximates a probability distribution over sequences of tokens. Concretely, given a sequence of tokens x₁ … xₜ, it models the probability of the next token as a conditional distribution, and the training objective is simply to maximize the likelihood of the true next token across a corpus, equivalently minimizing cross-entropy loss:

L(θ) = − ∑ₜ log P(xₜ | x₁ … xₜ₋₁ ; θ) The model, parameterized by θ, is trained to assign high probability to each actual next token given everything before it. Everything else is architecture in service of computing that conditional well.

In plain terms · the training objectiveThink of how you’d study for an exam by covering the answer key and guessing the next line, then peeking to check. Do that across billions of sentences, nudging yourself each time you’re wrong, and you slowly learn the material. Cross-entropy loss is just the score for how wrong each guess was, and training is the relentless drive to lower it.

Before 2017, the dominant approach to this was the recurrent neural network, which processes tokens one at a time, carrying a hidden state forward. Recurrence has two fatal weaknesses at scale, and the transformer was designed specifically to eliminate both. It cannot be parallelized across the sequence, because step t depends on step t−1, and it struggles to preserve information across long distances, since signal must survive many sequential transformations. The transformer’s central bet, stated in the title of the paper that introduced it, was that a mechanism called attention could replace recurrence entirely. It was right.

The Whole Machine

The architecture, end to end

Here is the entire data flow before we dissect it. Tokens are embedded into vectors, position information is added, and the result passes through a stack of N identical layers. Each layer contains a multi-head attention sublayer and a position-wise feed-forward sublayer, each wrapped in a residual connection followed by normalization. The final layer’s output is projected to the size of the vocabulary and passed through a softmax to yield a probability for every possible next token.

Exhibit 1The Transformer Block, End to End

The full data path. Read bottom to top. The dashed region is one layer, repeated N times (from 6 in the original to 100-plus in frontier models). The teal lines are residual connections; every sublayer is wrapped in Add & Norm. This single diagram is the whole architecture.

Two structural facts are worth holding onto. First, the shape is preserved throughout: every sublayer takes and returns vectors of dimension d_model, which is what makes the layers stackable. Second, the only place tokens exchange information is attention. The feed-forward network processes each position independently. So attention is where context happens, and everything else refines it.

The Input

Embeddings, and why position must be added separately

Each token id indexes into an embedding matrix, a learned lookup table of shape vocab_size × d_model, producing a dense vector per token. These embeddings are learned jointly with the rest of the network, and they arrange themselves so that semantically related tokens occupy nearby regions of the space.

But there is an immediate problem, and it is the first design consequence worth understanding. Attention, as we will see, is permutation-invariant: it treats its input as a set, not a sequence. Compute attention over “dog bites man” and “man bites dog” with no positional information, and you get identical representations, which is catastrophic for language. The transformer therefore adds positional information to the embeddings before the first layer. The original paper used fixed sinusoidal functions of the position; as we will discuss, modern models have largely replaced this with rotary embeddings that encode relative position directly inside the attention computation.

Design reasoning: position is added, not concatenated, so it does not consume extra dimensions, and sinusoids were chosen originally because they let the model generalize to sequence lengths not seen in training, since any position can be expressed as a linear function of others.

In plain terms · embeddings and positionPicture a city where similar shops cluster: all the bakeries in one quarter, all the banks in another. An embedding places each word in exactly such a city of meaning, so related words are neighbors. But a list of shops with no addresses is useless, you need to know where each one sits. Positional encoding is the address system: it tells the model that “dog bites man” and “man bites dog” have the same words in different places.

The Core

Self-attention: the actual mathematics

This is the heart of the machine, and it rewards precision. From each token’s embedding x, the model computes three vectors by multiplying by three learned weight matrices: a query Q = xWᵚ, a key K = xWᵕ, and a value V = xWᵛ. Stack these across all tokens into matrices Q, K and V, and the entire attention operation is a single expression:

Attention(Q, K, V) = softmax( QKᵀ / √dₖ ) V Q, K, V are the query, key and value matrices. dₖ is the dimension of the key vectors. QKᵀ produces an n×n matrix of similarity scores between every pair of tokens; the softmax turns each row into weights that sum to one; multiplying by V produces, for each token, a weighted blend of all tokens’ values.

In plain terms · what attention is doingAttention works like matchmaking. Each word holds up a query (“what am I looking for?”), every other word holds up a key (“here’s what I offer”), and the words whose keys best fit the query hand over their value, their actual meaning. The word “it” asks “which object am I?” and “trophy” is the strongest match, so its meaning flows into “it.”

Exhibit 2Attention as Matchmaking: Query, Key, Value

The matchmaking view. A word’s query is compared to every word’s key; the best match delivers its value. This is exactly what the formula computes, scored, normalized, and blended.

Exhibit 3Scaled Dot-Product Attention, Step by Step

The computation graph of one attention operation. Queries score against keys, the scores are scaled and normalized into weights, and those weights blend the values. Done for every token simultaneously as dense matrix multiplications, which is what makes it fast on a GPU.

The Reasoning

Why each piece of the formula exists

A CTO’s real question is usually not what the formula is but why it takes that shape. Each element is a deliberate choice.

The dot product QKᵀ is a similarity measure: it is large when a query and a key point in similar directions, which is exactly “how relevant is token j to token i.” Splitting into separate Q and K matrices, rather than comparing embeddings directly, lets a token advertise something different in what it “looks for” (query) versus what it “offers” (key), which is what allows asymmetric relationships like a pronoun seeking its antecedent.

The scaling by √dₖ is not cosmetic. For large key dimensions, dot products grow large in magnitude, pushing the softmax into regions where its gradient is vanishingly small, which stalls learning. Dividing by √dₖ keeps the variance of the scores roughly constant regardless of dimension, which keeps the softmax in a well-behaved range. It is a numerical stability fix, and models trained without it degrade.

In plain terms · why divide by √dₖImagine a microphone turned up so loud it clips and distorts, you can no longer tell loud from very loud. Large dot products do the same to the softmax: everything saturates and the model can’t tell which match is best. Dividing by √dₖ is turning the mic down to a clean level, so differences stay audible and learning keeps working.

The softmax converts arbitrary real-valued scores into a probability distribution over tokens: non-negative, summing to one. This is what makes attention a weighted average rather than an arbitrary combination, and it is why we can interpret the weights as “how much each token attends to each other token.” The final multiplication by V is the payoff: each token’s output is the attention-weighted sum of every token’s value vector, so information flows from relevant tokens into the current one, in proportion to relevance.

In plain terms · what softmax doesYou have a fixed budget of attention, say 100 percent, and must split it across everything in the sentence. Softmax is the rule that forces the split to add up to exactly 100 percent: give more to the trophy and, by necessity, less to everything else. That is why attention is a genuine weighing of priorities, not a free-for-all.

Attention is not magic once you see it: it is a differentiable, content-based lookup. Queries search, keys advertise, values are retrieved, and softmax makes the whole thing a smooth weighted average you can train by gradient descent.

Prashant Akhawat

Parallel Perspectives

Multi-head attention

A single attention operation forces the model to average all types of relationship into one set of weights. That is limiting: the relationship that resolves a pronoun is not the one that tracks subject-verb agreement. The solution is multi-head attention: run several attention operations in parallel, each with its own learned Q, K, V projections, so each “head” can specialize.

Mechanically, d_model is split into h heads, each of dimension d_model / h. With d_model = 768 and h = 12, each head operates in 64 dimensions. Each head computes attention independently in its subspace; the outputs are concatenated back to d_model and passed through a final projection matrix W₀.

MultiHead(Q,K,V) = Concat(head₁, …, headₕ) W₀ where headᵢ = Attention(QWᵢᵚ, KWᵢᵕ, VWᵢᵛ) Each head gets its own projections, attends in a lower-dimensional subspace, and the results are concatenated and mixed by W₀. Total computation is similar to single-head attention at full width, because each head is narrower.

Exhibit 4Multi-Head Attention

Specialists in parallel. The representation is split across heads, each learning a different kind of relationship in its own subspace, then recombined. Interpretability research has found heads that track syntax, coreference, and positional patterns.

In plain terms · multi-head attentionHand the same contract to a panel of specialists: a lawyer checks liability, an accountant checks the numbers, an engineer checks the specs. Each reads the same document through a different lens, then they pool their findings. Each attention head is one specialist, one tracks grammar, another tracks which pronoun means what, and combining them gives a far richer read than any single reviewer.

The Rest of the Layer

Feed-forward, residuals, and normalization

Attention moves information between tokens; the feed-forward network (FFN) then processes each position independently, giving the model capacity to transform what attention gathered. It is two linear layers with a non-linearity between them, typically expanding to roughly 4 × d_model in the middle and back down. Most of a transformer’s parameters actually live here, not in attention.

FFN(x) = max(0, xW₁ + b₁) W₂ + b₂ The original used ReLU; modern models use gated variants such as SwiGLU, which empirically train better. The middle dimension is where much of the model’s knowledge is stored.

Two mechanisms make it possible to stack dozens or hundreds of these layers without the network becoming untrainable. Residual connections add each sublayer’s input to its output (x + Sublayer(x)), which gives gradients a direct path backward through the network and prevents the signal from degrading through depth. Layer normalization stabilizes the distribution of activations, keeping training numerically well-behaved. The placement of that normalization, before or after the sublayer, turns out to matter enormously, which is one of the first things the field revised, as we will see.

In plain terms · FFN, residuals and normalizationThink of a factory line. The feed-forward network is a workstation that refines each item. The residual connection is a bypass lane that carries the original item forward untouched, so if a station adds nothing useful, the work isn’t lost, which is what lets you chain a hundred stations safely. Normalization is the quality-control checkpoint that keeps every item within spec before it moves on, so nothing spirals out of range.

The Payoff

Why it beat RNNs, concretely

It is worth being precise about why this architecture displaced recurrence so completely, because the reasons are not aesthetic. They are about computation and gradients.

Exhibit 5Sequential Recurrence vs Parallel Attention

The two decisive advantages. An RNN must process tokens in order, so it cannot be parallelized across the sequence and its long-range signal decays. Attention processes all positions at once, and connects any two directly in a single step.

In plain terms · why it beat RNNsAn RNN reads through a moving slit that shows one word at a time, to recall the first word of a long sentence, it must have carried that memory through every word since, and it fades. A transformer sees the whole page at once, so any word can look directly at any other, near or far, instantly. That is both faster (you read it all in one glance) and better at long-range meaning.

First, parallelism. Because attention computes all pairwise interactions as matrix multiplications, an entire sequence is processed in parallel during training, saturating modern GPU and TPU hardware in a way recurrence never could. This is what made training on internet-scale corpora economically feasible, and it is not a minor speedup; it is the difference between possible and impossible at frontier scale. Second, path length. In an RNN, information between two tokens n positions apart must survive n sequential steps, so gradients vanish and long-range dependencies are lost. In a transformer, any two tokens are connected in a single attention step: the maximum path length is constant, independent of distance. Long-range dependencies, the bane of recurrence, become trivial.

Two Shapes

Encoder-decoder vs decoder-only

The original transformer was an encoder-decoder, built for translation: an encoder reads the full source sentence bidirectionally, and a decoder generates the target while attending to the encoder’s output. Most of today’s large language models, the GPT lineage and its peers, are decoder-only. They drop the encoder entirely and train on a single objective: predict the next token.

The decisive mechanism in the decoder is causal masking. Because the model must not see future tokens when predicting the next one, the attention score matrix is masked so that position i can only attend to positions ≤ i, by setting the disallowed entries to negative infinity before the softmax (which drives their weights to zero). This preserves the autoregressive property, training the model on every position in parallel while ensuring each prediction depends only on the past. The elegance is that a single forward pass computes the loss for every next-token prediction in the sequence at once, which is again why training is efficient.

In plain terms · causal maskingIt’s an exam where each question must be answered before you’re allowed to see the next one, no peeking ahead. Masking enforces exactly that rule inside attention: when predicting word five, the model is blocked from looking at words six onward. This is what keeps the model honest as a genuine next-word predictor rather than a copier of answers it can already see.

Why decoder-only won for LLMs: a single next-token objective scales cleanly, needs only raw text (no paired data), and one architecture handles any task framed as text continuation. Simplicity plus scale beat the more specialized encoder-decoder for general-purpose generation.

The Frontier

What changed since 2017

The transformer you have just read about is the 2017 original. The core is unchanged, but nearly every component has been refined, and the field has converged on a remarkably consistent modern stack. If you are evaluating or building on today’s models, this is the vocabulary that matters.

Exhibit 6The Original vs the Modern Stack

Convergent evolution. Between 2017 and 2026 the field settled on a de facto stack. Each change addresses a specific failure mode of the original at scale or at long context.

The most important shifts, and why each was made: Pre-norm with RMSNorm moved normalization to before each sublayer and simplified it, which stabilizes training of very deep networks. RoPE (rotary positional embeddings) encodes position as a rotation applied inside attention, capturing relative position naturally and extrapolating to far longer contexts than sinusoidal encodings allowed; it is why million-token context windows became feasible. SwiGLU replaced ReLU in the FFN for better empirical performance. GQA and MLA (grouped-query and multi-head latent attention) share or compress the keys and values across heads, cutting the memory the model must hold during generation by several-fold. And Mixture of Experts replaces the dense FFN with many expert FFNs and a router that activates only a few per token, so total capacity grows without a proportional rise in compute per token.

Alongside the architecture, two systems techniques are essential to know because they govern real-world cost and latency. KV-caching stores the keys and values already computed for prior tokens so that generating each new token does not recompute the entire history, the single most important inference optimization. FlashAttention computes exactly the same attention result but reorders the computation to minimize slow memory transfers on the GPU, since attention is bottlenecked by memory bandwidth, not arithmetic. Neither changes the math; both change whether it is affordable.

In plain terms · KV-cache and MoETwo everyday pictures. KV-cache: when you write the next sentence of a letter, you don’t re-read the entire letter from the top each time, you keep the gist in mind. KV-cache lets the model keep the earlier words’ work “in mind” instead of recomputing it for every new word. Mixture of Experts: a hospital doesn’t have every doctor examine every patient, a nurse routes you to the one relevant specialist. MoE routes each token to just a couple of expert sub-networks, so the model can be huge yet only do a little work per word.

The Correlation

The architecture in the wild: the models you actually buy

Everything above stops being abstract the moment you map it onto the models in the market. They are not different species. Every frontier system, GPT, Claude, Gemini, Llama, Qwen, Mistral, DeepSeek, is a decoder-only transformer: the same embedding-attention-FFN stack you have just walked through, differing in scale and in which of the modern efficiency choices each vendor made. When you evaluate a model, you are evaluating a configuration of this one architecture.

Exhibit 7The Models You Know, Decoded

One architecture, many configurations. Every frontier model is this same decoder-only transformer. The differentiators are the efficiency choices, and DeepSeek showed those choices can move training economics by an order of magnitude.

The most instructive case for a leader is DeepSeek. Its V3 model has 671 billion total parameters, but through Mixture of Experts only about 37 billion activate for any given token, and its latent-attention scheme compresses the memory-hungry KV cache dramatically. The result, by the company’s own account, was a frontier-class model trained for roughly 5.6 million dollars of compute, against tens of millions, by public estimates, for comparable dense-era models. Whatever one makes of the exact figures, the direction of the lesson is unambiguous, and it is the business punchline of this entire article: the efficiency techniques in the previous section are not engineering trivia. They are the difference between training economics that differ by an order of magnitude, and inference costs that compound that difference on every single query you pay for.

The same mapping demystifies the reasoning models. OpenAI’s o-series and DeepSeek’s R1 are not new architectures; they are these same transformers, post-trained with reinforcement learning to spend tokens thinking before answering. When a vendor pitches you a “new class of model,” the honest question, which you can now ask precisely, is: same architecture, which choices, at what cost profile?

In plain terms · reading a model spec sheetCar engines are all pistons, valves and crankshafts; what differs is the configuration, and the configuration decides the fuel bill. Models are the same: all of them are this architecture, and the spec-sheet jargon (GQA, MoE, RoPE, 128K context) is just naming which configuration the vendor chose. Now you can read the sheet the way a fleet buyer reads an engine spec: not for the acronyms, but for what they mean for cost, speed and capability.

The Constraint

The quadratic bottleneck, and the war against it

Everything about long-context economics traces back to one property. Because attention compares every token with every other, its cost scales as O(n²) in the sequence length n: double the context and you quadruple the attention computation and memory. This is the single most consequential limitation of the architecture, and a large fraction of research since 2020 has targeted it directly.

Exhibit 8The O(n²) Bottleneck and Its Fixes

Why long context is expensive. Attention cost grows with the square of input length. The frontier fights this on several fronts, from exact-but-efficient kernels to approximations to entirely different architectures.

In plain terms · the quadratic costPicture a party where every guest must shake hands with every other guest. With 4 people that’s 6 handshakes; double it to 8 people and you get 28, more than four times as many. Attention is that party: every token “shakes hands” with every other token, so doubling the input roughly quadruples the work. That single fact is why long documents cost and slow down so sharply, and why so much research fights this exact problem.

Exhibit 9The Handshake Problem: Why Cost Grows With the Square

Every token meets every token. Connections grow far faster than guests. This is quadratic growth made visible.

Exhibit 10Context Length vs Attention Cost

The cost of a longer prompt. Each doubling of context multiplies the pairwise attention work by four. This table is the practical face of O(n²).

The responses fall into tiers. Exact-but-efficient: FlashAttention and KV-cache optimizations reduce the constants and memory traffic without changing the result. Approximate: sparse and linear attention variants attend to only a subset of positions or approximate the softmax, trading some fidelity for sub-quadratic scaling. Architectural: state-space models such as Mamba abandon attention’s quadratic core for linear-time recurrence-like mechanisms, and hybrid designs are now interleaving them with attention layers. The transformer is not the end of the story; it is the current, dominant, and still-evolving answer.

Closing

The whole machine, in one breath

A transformer embeds tokens, injects position, and passes them through a stack of identical layers. In each, multi-head scaled dot-product attention lets every token gather a relevance-weighted blend of every other token’s value, a position-wise feed-forward network transforms the result, and residual connections with normalization keep the deep stack trainable. Causal masking makes it autoregressive; a final projection and softmax yield the next-token distribution. It replaced recurrence because it parallelizes across the sequence and connects any two tokens in a single step. And since 2017 it has been refined, pre-norm, RoPE, SwiGLU, GQA, MoE, FlashAttention, KV-cache, into the efficient, scalable stack running every frontier model today.

That is the architecture, honestly and completely, with the reasoning behind each choice. It is, in the end, a stack of differentiable lookups trained to predict the next token, and the fact that this produces what it produces remains one of the more remarkable results in the history of computing.

The Architecture in Summary

Objective: model the next-token distribution by minimizing cross-entropy; all architecture serves this.
Attention: softmax(QKᵀ/√dₖ)V, a differentiable content-based lookup; the scaling and softmax are deliberate numerical and interpretive choices.
Multi-head: parallel attention in subspaces lets heads specialize; concatenate and project.
The rest of the layer: position-wise FFN holds most parameters; residuals and normalization make depth trainable.
Why it won: full-sequence parallelism and constant-length paths between any two tokens.
Modern stack: pre-norm RMSNorm, RoPE, SwiGLU, GQA/MLA, MoE, with FlashAttention and KV-cache for efficiency, all fighting the O(n²) bottleneck.

The one-line version, for the next architecture review

A transformer is a stack of two alternating operations, attention to move information between tokens and a feed-forward network to transform it, trained end to end to predict the next token. Everything else is efficiency.

Selected Sources & Further Reading

Vaswani et al., “Attention Is All You Need,” NeurIPS 2017, the original transformer.
Su et al., “RoFormer: Enhanced Transformer with Rotary Position Embedding” (RoPE), 2021.
Dao et al., “FlashAttention” and “FlashAttention-2/3,” 2022 to 2024, IO-aware exact attention.
Shazeer, “Fast Transformer Decoding” (MQA) and Ainslie et al. (GQA); DeepSeek, Multi-Head Latent Attention.
Fedus et al., “Switch Transformer,” and the Mixture-of-Experts line of work.
Shazeer, “GLU Variants Improve Transformer” (SwiGLU); Zhang & Sennrich, RMSNorm.
Gu & Dao, “Mamba,” 2023 to 2024, state-space models as a sub-quadratic alternative.
DeepSeek-AI, DeepSeek-V2 and DeepSeek-V3 Technical Report (2024), MLA, DeepSeekMoE, 671B total / 37B active parameters, and reported training cost; public estimates for comparable models vary and are cited as reported.
Jurafsky & Martin, Speech and Language Processing, chapters on attention and transformers; Elena Voita, open NLP course.
CXO Intelligence Series: The Prediction Engine (the executive primer this Deep Dive accompanies), and Editions 01 to 04 on AI strategy, the moat, redesign and security.

Formulas are presented in standard notation and lightly simplified for readability (bias terms and some normalization details are elided where they do not aid understanding). This is an accurate conceptual and mathematical account intended for practitioners, not a substitute for the primary papers or a reference implementation.

Inside the TransformerHow it actually works, and why every design choice was made the way it was