How an LLM works under the hood: from the token to the transformer, without leaving a single detail out

LLMs became a product commodity, but almost nobody who builds an agent, writes a prompt or designs a RAG knows what is happening on the other side of the POST /v1/messages. This post opens everything up. Tokenization, embeddings, transformer architecture, attention mechanism, autoregressive generation, sampling, KV cache, mixture of experts, quantization, RLHF. A dense technical wiki, from the atom (token) to the complete system (a model served in production). No mysticism. Math and engineering.

Whoever understands this prompts better, debugs better, picks a better model, sizes infrastructure better. Whoever does not understand pays 10x for a worse result. The difference between the two curves is this post.

1. Token: the atomic unit

An LLM does not see text. It sees integers. The text coming in is first converted into a sequence of tokens, and each token is an integer between 0 and vocab_size (typically 32k to 256k). The standard algorithm is BPE (Byte Pair Encoding) or variants (OpenAI's Tiktoken, Google's SentencePiece, Anthropic's own tokenizer).

BPE works like this: it starts with each byte as a token. It counts the most frequent token pairs in the training corpus. It merges the most common pair into a new token. It repeats until it reaches the desired vocab_size. Result: common words become 1 token, rare words become 2 to 5 tokens, special characters become their own token.

Practical examples with the GPT-4o tokenizer:

'hello' = 1 token
'antidisestablishmentarianism' = 6 tokens
'São Paulo' = 3 tokens (Portuguese is less efficient than English because the vocab is dominated by English)
'你好' = 2 tokens (languages with their own script are expensive)
1 token in English = ~4 characters = ~0.75 word

Operational consequence: API pricing is per token, not per word. Content in Portuguese costs ~30% more than the same content in English. Compact code (Python, JS) is efficient. Verbose code (Java, COBOL) is expensive. JSON without indentation is efficient. JSON with indentation costs 30% more for nothing.

2. Embedding: from the integer to the vector

A token is an integer. A model operates on vectors. The first layer of any transformer is the embedding table: a matrix of size [vocab_size, d_model], where d_model is the internal dimension (Llama 70B: 8192, GPT-4 class: estimated 12288+).

Each token becomes the vector of the corresponding row in the table. That vector is the initial representation. It will undergo N transformations through the following layers until it becomes a prediction.

A mathematical detail that matters: the embedding table is learned during pre-training. The 12k numbers that represent the token 'hello' were not chosen by a human, they were optimized to minimize loss. Semantically close tokens end up close in vector space. That is why the embeddings of the tokens 'king' and 'queen' have a difference similar to 'man' and 'woman'. Geometry emerges.

LLM embeddings are distinct from the embeddings of a dedicated model (OpenAI's text-embedding-3-large, BGE). An LLM embedding is trained to predict the next token. An embedding model is trained for semantic similarity. Do not use an LLM's intermediate embedding for semantic search, it will do worse than a dedicated model.

3. Positional encoding: how the model knows the order

Attention is invariant to order (this will become clear later). Without positional information, 'dog bites man' and 'man bites dog' turn into the same soup for the model. Solution: add/concatenate position information to the embedding.

Methods in chronological order:

Sinusoidal (original Transformer, 2017): position becomes a combination of sines and cosines at various frequencies. It has no learned parameter. It generalizes to sequences longer than those in training, in theory.
Learned positional embedding (GPT-2): a learned [max_seq, d_model] matrix. Simple, but it does not generalize beyond the max_seq seen in training.
RoPE (Rotary Position Embedding): used in Llama, Mistral, Qwen. It applies a rotation to the query/key vectors proportional to the position. Advantages: it encodes relative position naturally and extrapolates better with techniques like NTK-aware scaling.
ALiBi (Attention with Linear Bias): used in some models. It adds a decreasing bias to the attention scores based on distance. Simpler, good extrapolation.

Why it matters: the form of positional encoding determines how well the model handles long context. Llama 3.1 extended from 8k to 128k with NTK-aware scaling of RoPE, without full retraining. Without that technique, the model degrades catastrophically beyond training.

4. Attention: the heart of the transformer

Attention is the mechanism that lets each token 'look' at other tokens in the sequence and pull relevant information. Without it, the model cannot relate 'she' at the end of a sentence with 'Maria' at the beginning.

For each token at position i, it computes three vectors: Query (Q), Key (K), Value (V). All via multiplication by learned matrices W_Q, W_K, W_V.

Attention score between token i and token j: score(i,j) = (Q_i · K_j) / sqrt(d_k). The division by sqrt(d_k) stabilizes the gradient. A softmax over all the scores of each i normalizes it into a probability distribution.

Output for token i: sum_j (attention(i,j) * V_j). Each token becomes a weighted combination of the values of all tokens, weighted by how relevant they are (attention).

Causal masking: in a generative LLM (decoder-only), attention is masked. Token i can only attend to tokens j ≤ i. It prevents 'seeing the future' during training, which is what makes it possible to train for next-token prediction.

Multi-head attention: instead of one attention, it runs N in parallel (h = 32, 64, 128). Each head learns to focus on different patterns (one handles syntax, another coreference, another semantics). Outputs are concatenated and projected to d_model.

5. Attention variants that change cost

Standard attention (MHA) is O(n²) in memory and compute. For long sequences, it becomes a bottleneck. Optimizations that became standard:

MQA (Multi-Query Attention): all heads share the same K and V, only Q is separate. It drastically reduces KV cache memory with minimal quality loss. Used in PaLM.
GQA (Grouped-Query Attention): a middle ground. Heads grouped into G groups, each group shares K/V. G=8 is common. Used in Llama 2 70B+, Llama 3, Mistral.
FlashAttention: an implementation that reorganizes compute to minimize reads/writes in the GPU's HBM memory. 2x to 4x faster without changing the math. Today it is the default in any serious training/inference.
Sliding Window Attention: each token only attends to a local window. Mistral 7B uses it. It allows long sequences with linear instead of quadratic cost.
Sparse Attention: predefined patterns of which tokens attend to which. Longformer, BigBird.

6. Feed-forward: the other half of the block

Each transformer block has two subcomponents: attention and a feed-forward network (FFN). The FFN is a two-layer MLP applied token-by-token (with no mixing between tokens, that is the attention's job).

Structure: FFN(x) = W_2 * activation(W_1 * x + b_1) + b_2. The intermediate dimension is typically 4x d_model. Classic activation: ReLU. Modern ones use SwiGLU (GLU with Swish), which gives a measurable gain at no relevant additional cost.

The FFN is where most of the model's parameters live. Llama 70B: ~70% of the weights are FFN, ~25% attention, ~5% embeddings. That is why it is the primary target of Mixture of Experts (next section).

7. Mixture of Experts: how to run a large model without activating all of it

MoE replaces the dense FFN with N parallel FFNs ('experts') plus a router that picks the top-K experts per token (usually K=2). Only the K chosen experts run, the others stay dormant.

Mixtral 8x7B: 8 experts, top-2 active. Total ~47B parameters, but only ~13B activated per forward. Inference runs at the cost of 13B, quality gets close to a dense 70B.

Trade-off: it saves compute, but memory stays the same. You need to have all the experts in VRAM even when using only 2. That is why MoE is a favorite in the data center (plenty of VRAM available) and falls flat on the edge (scarce memory).

Notable MoE models: Mixtral 8x7B/8x22B, DeepSeek-V3 (256 experts, 9 active), Qwen 2.5 MoE, GPT-4 (presumed MoE from the inferred architecture).

8. Normalization and residual: the stability that makes deep training possible

Each block has residual connections: output = input + sublayer(input). It lets the gradient flow through N layers without vanishing. Without residuals, a model with more than ~10 layers does not train.

Layer normalization: normalizes activations per sample. RMSNorm (a simplified version, used in Llama) drops the mean and normalizes only by the root mean square. Cheaper, equivalent quality.

Placement: Pre-norm (norm before the sublayer) is the modern standard, it trains more stably than Post-norm (from the original paper).

9. The whole architecture: stacking blocks

A model is N transformer blocks stacked (Llama 70B: 80 blocks, GPT-3 175B: 96 blocks). Each block has attention + FFN + residuals + norms. The complete sequence of a forward pass:

Token IDs → embedding (lookup in the table).
Add positional encoding (or apply RoPE to the Q/K inside the attention).
For each block (1 to N): attention (multi-head, with causal mask) + residual + norm; FFN + residual + norm.
Final norm.
Projection to vocab_size via the 'lm_head' (typically tied with the embedding to save parameters): produces logits of size vocab_size for each position.
Logits → probabilities via softmax.

10. Autoregressive inference: how text is generated

An LLM generates token by token. Each new token depends on all the previous ones.

tokens = encode(prompt)
while not done:
 logits = model.forward(tokens)
 next_logits = logits[-1] # último token
 next_token = sample(next_logits)
 tokens.append(next_token)
 if next_token == EOS: done = True
return decode(tokens)

Without optimization, this is O(n²) per generation: each new token recomputes attention over all the previous ones. Unviable.

11. KV cache: the optimization that makes inference viable

Key observation: with each new token, Q changes but the K and V of the past tokens are the same. Caching the K and V of all already-processed tokens eliminates recomputation.

Memory occupied: 2 * n_layers * n_heads * d_head * seq_len * 2 bytes (FP16). Llama 70B with 8k context: ~40GB of KV cache. That is why a 70B model needs a GPU with 80GB+ of VRAM even though the model itself is smaller.

GQA reduces the KV cache by ~8x (Llama 70B with GQA: ~5GB of KV at 8k context). Practical reason the whole industry adopted GQA: to enable long context.

Quantizing the KV cache (FP8, INT8) reduces it another 2x to 4x. Standard in production serving.

12. Sampling: how the token is chosen among N possibilities

The model produces a probability distribution over the entire vocab (32k to 256k tokens). Sampling decides which one to pick.

Greedy: always the one with the highest probability. Deterministic. Good for tasks with a single correct answer (extraction, classification).
Temperature: divides logits by T before the softmax. T < 1 concentrates on high probability (more deterministic). T > 1 spreads it out (more creative). T = 0 is equivalent to greedy.
Top-k: considers only the K most likely tokens, zeroes the rest. Typical K=40 to 50.
Top-p (nucleus): considers the tokens whose cumulative probability reaches P. Standard P=0.9 to 0.95. It adapts the size of the set to the distribution (more flexible than top-k).
Repetition penalty: reduces the probability of already-generated tokens. Fights loops.
Min-p: a recent variation. Considers only tokens with probability ≥ P * max_prob. More robust than top-p in long-tail cases.

Typical combinations in production: T=0 (extraction), T=0.7+top_p=0.9 (chat), T=1.0+top_p=0.95 (creative).

13. Pre-training: where 99% of the cost lives

Pre-training is where the model learns language. Objective: next-token prediction over a giant corpus (trillions of tokens). Loss: cross-entropy between the predicted distribution and the real token.

Typical data: web (Common Crawl, FineWeb), code (GitHub, The Stack), books, papers, Wikipedia. Curation became a differentiator: dedup, quality filtering, domain balancing. DeepSeek and Llama 3 showed that 'data quality > quantity'.

Compute: GPT-4 class used an estimated 10^25 FLOPs. Llama 3 405B: 3.8 * 10^25. Training a frontier model costs hundreds of millions in GPUs. That is why the ecosystem is dominated by a few labs.

Time: on the order of months, in clusters of thousands of GPUs (Llama 3 405B: 16k H100s for ~54 days).

14. Post-training: what turns a raw model into a useful one

A pre-trained model is a text completer. To answer a question, follow an instruction, refuse dangerous content, it needs post-training. Three typical phases.

SFT (Supervised Fine-Tuning): continues training with (prompt, ideal answer) pairs written by a human. It teaches response format, tone, and the ability to follow instructions. Dataset: 10k to 1M examples.

RLHF (Reinforcement Learning from Human Feedback): humans compare pairs of answers, pick the best. It trains a reward model that learns to score answers. Then the main model is optimized via PPO to maximize the reward. Expensive, unstable, but it brought ChatGPT into existence.

DPO (Direct Preference Optimization): a modern alternative to RLHF. It optimizes directly over the preference pairs, without an intermediate reward model. Simpler, comparable results. Adopted in Llama 3, Mistral, Qwen.

Recent variations: RLAIF (AI generates the feedback), Constitutional AI (Anthropic: the model critiques its own answers based on principles), Online DPO. The field evolves fast.

15. Quantization: running a large model on smaller hardware

The model's weights are floats. Training in BF16 (16 bits). Inference can use lower precision with minimal quality loss.

FP16/BF16: 16 bits. Standard for inference without optimization.
INT8: 8 bits. ~2x less memory, ~2x faster. Loss <1% on most benchmarks.
INT4: 4 bits. ~4x less memory. Perceptible but usable loss (GPTQ, AWQ are standard techniques).
FP8: 8 bits floating point. New, requires compatible hardware (H100, Blackwell). Better quality than INT8.
1.58 bits (BitNet): extreme. Weights in {-1, 0, 1}. Promising research, not production yet.

Quantization is what allows running Llama 70B on a 24GB GPU (INT4) instead of requiring an A100 80GB (FP16). It enables local LLMs on consumer hardware.

16. Context window: limit, cost, and degradation

The context window is the maximum number of tokens the model accepts. Today it ranges from 8k (small models) to 2M (Gemini 1.5 Pro, in preview), with Claude and GPT-4 at 200k to 1M.

Hard limits: the KV cache grows linearly, attention grows quadratically. Even with FlashAttention and GQA, very long context is expensive and slow. TTFT (time to first token) latency can exceed 30s for a 1M-token prompt.

Soft limit: quality degradation along the context. Models forget information from the middle more than from the beginning or end ('lost in the middle'). Benchmarks like needle-in-haystack measure this. Modern models pass the needle, but degrade on tasks that require reasoning over multiple points of the long context.

17. Serving in production: vLLM, TGI, and what matters

Optimized inference frameworks:

vLLM: PagedAttention (KV cache management in pages, avoids fragmentation), continuous batching, prefix caching. The industry default for self-hosted.
TGI (Text Generation Inference): HuggingFace. Mature, but vLLM overtook it in features.
SGLang: focused on structured generation, RadixAttention for cache shared between requests.
TensorRT-LLM: NVIDIA, maximum performance on NVIDIA hardware, more work to configure.
llama.cpp: CPU/Metal/CUDA, focused on quantization and edge. The default for running local.

Optimizations that make it into serious production: continuous batching (mixes requests in different states in the same forward pass, instead of a synchronous batch), speculative decoding (a small model proposes the next tokens, the large model validates in parallel), prefix caching (caches the KV of a common prefix between requests, the basis of Anthropic's prompt caching).

18. The reframe: what this wiki lets you do

Knowing this stack, decisions become technical instead of guesses. Why a prompt in English costs less: tokenization. Why a long system prompt fits in the cache: prefix caching. Why GQA enabled long contexts: KV cache. Why MoE has cheap inference but expensive memory: architecture. Why the model loses track in the middle of a large context: attention degradation in long sequences, not a prompt bug.

Everything that looked like a vendor detail becomes a parameter you can adjust (your own or hold the vendor accountable for). It is the difference between an LLM user and an LLM engineer. This entire post is the border between the two.