How Transformer LLMs Actually Work

I'd been implementing neural networks for years before the Transformer came along. I remember the excitement when LSTMs first clicked for me—finally, a way to model sequences that didn't immediately forget everything! But I also remember the frustration: training was slow, sequences longer than a few hundred tokens were problematic, and there was always this nagging sense that we were fighting the architecture rather than working with it.

Then in June 2017, a paper with possibly the most provocative title in machine learning history dropped: "Attention Is All You Need." The claim was audacious—throw out recurrence entirely and rely solely on attention mechanisms. I was skeptical. How could you model sequences without... sequence processing?

To understand why Transformers were such a breakthrough, we need to understand what they replaced and why that replacement was necessary.

The Sequential Processing Dilemma

Language has order. "Dog bites man" means something very different from "Man bites dog." Any architecture that processes language needs to respect this ordering. The natural approach, which dominated for years, was to process sequences one element at a time.

Recurrent Neural Networks (RNNs) embodied this intuition perfectly. At each timestep t, the network computes a hidden state h_t as a function of the previous hidden state h_{t-1} and the current input x_t:

This is elegant. The hidden state acts as a "memory" that carries context forward through the sequence. Process "The cat sat on the" token by token, and by the time you reach "the," your hidden state hopefully encodes something useful about cats and sitting.

But there's a fundamental problem: sequential computation can't be parallelized. You can't compute h_{10} until you've computed h_9, which requires h_8, and so on. Your expensive GPU with thousands of cores? Most of them sit idle, waiting for the sequential chain to complete.

The Vanishing Gradient Problem

Even worse than the parallelization issue is the vanishing gradient problem. When we train neural networks with backpropagation, gradients flow backward through the network. For RNNs, this means gradients must flow backward through time—through every single timestep.

Here's the mathematical reality: when you backpropagate through a recurrent connection, you multiply by the Jacobian matrix of the hidden state transition. If the eigenvalues of this matrix are less than 1, gradients shrink exponentially. If greater than 1, they explode exponentially.

In practice, this meant RNNs could only effectively learn dependencies spanning maybe 10-20 timesteps. Need to connect a pronoun to its antecedent 50 words back? Good luck.

LSTMs and GRUs: A Partial Solution

Long Short-Term Memory networks were designed specifically to address vanishing gradients. The key innovation was the "cell state"—a highway that allows information to flow through time with minimal transformation. Instead of a single hidden state being multiplied at each step, LSTMs use gating mechanisms:

• Forget gate: decides what to discard from the cell state
• Input gate: decides what new information to store
• Output gate: decides what to output based on the cell state

The cell state updates through addition rather than multiplication, which helps gradients flow more easily. LSTMs could handle dependencies of maybe 100-200 tokens—much better than vanilla RNNs, but still fundamentally limited.

And they were still sequential. Still couldn't parallelize. Still left most of your GPU idle.

The Attention Insight

The breakthrough came from an unexpected direction: machine translation. In 2014, Bahdanau and colleagues introduced attention for sequence-to-sequence models. The idea was simple but profound: instead of forcing all information through a fixed-size hidden state bottleneck, let the decoder "look back" at all encoder states and focus on the relevant ones.

This was attention as an add-on to RNNs. The Transformer paper asked: what if attention was the only mechanism? What if we threw out recurrence entirely?

The visualization above captures the key insight. In an RNN, if position 1 needs to influence position 20, the signal must pass through 19 hidden states—19 chances for information to be corrupted or lost. With attention, position 1 can directly attend to position 20. The path length is constant, regardless of sequence length.

This isn't just theoretically elegant—it's practically transformative. Shorter paths mean better gradient flow. Direct connections mean the model can learn arbitrary dependencies without fighting the architecture.

And crucially: all attention computations for all positions can happen in parallel. Your GPU is finally earning its keep.

Here's what continues to amaze me about modern LLMs: the training objective is almost laughably simple. Next-token prediction. Given the tokens you've seen so far, predict the probability distribution over what comes next.

That's it. No explicit teaching of grammar, no labeled examples of reasoning, no curriculum of skills. Just: predict the next token, billions of times, on trillions of tokens of text.

Why This Works: A Deep Connection

The mathematical justification is elegant. If you model P(x_t | x_1, ..., x_{t-1}) well for all positions in all texts, you're implicitly modeling the entire distribution of human text. Grammar rules? They're just patterns that make certain next tokens more likely. Factual knowledge? It's encoded in which continuations are probable. Reasoning? It's the patterns that connect premises to conclusions.

The loss function is cross-entropy:

Perplexity, the standard metric, is just exp(loss). A perplexity of 10 means the model is, on average, as uncertain as if choosing uniformly among 10 options. Modern LLMs achieve perplexities of 8-15 on standard benchmarks.

Emergent Capabilities: Scale Changes Everything

Emergent capabilities are perhaps the most surprising aspect of LLM scaling. These are abilities that appear suddenly at certain scales but are absent in smaller models.

GPT-2 (1.5B parameters) could generate coherent paragraphs but struggled with few-shot learning. GPT-3 (175B parameters) could solve new tasks from just a few examples in the prompt—a capability that wasn't explicitly trained and that researchers didn't fully anticipate.

Chain-of-thought reasoning emerged similarly. Tell a large enough model to "think step by step," and it produces intermediate reasoning that improves final answers. Smaller models just produce nonsense when prompted the same way.

These aren't linear improvements—they're phase transitions. The capability is essentially zero, then suddenly it's there. We still don't fully understand why.

The original Transformer was an encoder-decoder architecture designed for translation. The encoder processed the source sentence with bidirectional attention (each position attends to all others). The decoder generated the target sentence autoregressively, with causal attention (each position only attends to previous positions) plus cross-attention to the encoder outputs.

Modern LLMs almost universally use decoder-only architectures. Why did encoder-decoder lose?

The Case for Decoder-Only

Unified Training: With decoder-only, everything is next-token prediction. You don't need paired data (source-target). Any text works.

Task Flexibility: Encoder-decoder assumes a clear input/output split. Decoder-only treats everything as a sequence to continue. Want translation? Prompt with "Translate to French: [text]". Want summarization? "Summarize: [text]". The task is implicit in the prompt.

Compute Efficiency: No separate encoder phase. During inference, you're running one forward pass per generated token, not encoder + decoder.

The GPT Progression

GPT-1 (2018, 117M parameters) demonstrated that pre-training on next-token prediction followed by fine-tuning could achieve strong results. The insight: unsupervised pre-training provides a good initialization.

GPT-2 (2019, 1.5B parameters) showed zero-shot capabilities. Without any task-specific training, it could perform basic question answering, summarization, and translation. OpenAI initially declined to release the full model due to concerns about misuse.

GPT-3 (2020, 175B parameters) was the breakthrough that launched the current era. Few-shot learning worked: give the model a few examples in the prompt, and it could perform tasks it was never explicitly trained for. This was genuinely surprising.

GPT-4 (2023) introduced multimodality (images) and likely uses a Mixture of Experts architecture (though OpenAI hasn't confirmed details). Estimated at 1.7T parameters.

Sinusoidal positional encodings worked, but they had limitations. The biggest: extrapolation. Train on sequences up to 2048 tokens, and performance degrades on longer sequences. The model hasn't seen those position values before.

RoPE: Rotary Position Embeddings

RoPE, used by LLaMA and many others, encodes position through rotation. Instead of adding positional information to embeddings, RoPE applies rotation matrices to query and key vectors.

The key insight: when you take the dot product of two rotated vectors, the result depends only on their angle difference—which is proportional to the position difference. Position becomes relative, encoded in the geometry of the operation itself.

ALiBi: Attention with Linear Biases

ALiBi takes a radically different approach: no learned positional encoding at all. Instead, it adds a linear penalty to attention scores based on distance: positions farther apart get lower attention scores.

Different heads use different slopes, allowing some to focus locally and others to attend more globally. ALiBi extrapolates perfectly to any length (with the assumption that recency matters), though it can't learn arbitrary position-dependent patterns.

Here's the scaling dilemma: more parameters generally mean better performance, but more parameters also mean more compute per token. Mixture of Experts (MoE) breaks this link.

Replace the dense FFN with N expert FFNs. A router network selects the top-k experts for each token. Most parameters are inactive for any given token—you get the capacity of a large model with the compute cost of a smaller one.

Mixtral: A Concrete Example

Mixtral 8x7B has 8 experts with top-2 routing. Total parameters: 46.7B. Active parameters per token: ~12.9B. It matches or exceeds LLaMA 2 70B on most benchmarks while using similar compute to a 13B dense model.

The Load Balancing Challenge

Without careful design, the router might learn to always use the same few experts— "expert collapse." An auxiliary load balancing loss encourages even distribution:

The infrastructure implications are significant. All expert parameters must fit in memory (or be distributed with expert parallelism), but compute scales only with active parameters. This creates an unusual memory-compute trade-off.

Training a 70B+ parameter model requires careful orchestration of parallelism strategies. No single GPU can hold the model, and naive approaches waste compute or run out of memory.

The Parallelism Strategies

Data Parallelism: Each GPU holds a full model copy and processes different batches. Gradients are synchronized via all-reduce. Scales compute but not memory.

Tensor Parallelism: Split individual layers horizontally across GPUs. The attention and FFN weight matrices are sharded. Requires communication every layer—needs fast NVLink (400-900 GB/s).

Pipeline Parallelism: Split layers vertically—different GPUs own different layers. Communication only at stage boundaries. InfiniBand (200-400 Gb/s) is sufficient. Uses micro-batching to hide pipeline bubbles.

ZeRO: Memory Efficiency for Data Parallelism

ZeRO (Zero Redundancy Optimizer) eliminates memory redundancy in data parallelism. Standard DDP stores optimizer states, gradients, and parameters on every GPU. ZeRO shards them:

• Stage 1: Shard optimizer states (4× memory reduction for Adam)
• Stage 2: Also shard gradients
• Stage 3: Also shard parameters (requires gather before forward/backward)

Scaling Laws: The Chinchilla Revolution

In 2022, DeepMind's Chinchilla paper upended conventional wisdom. GPT-3 (175B params, 300B tokens) was undertrained. The compute-optimal ratio is roughly 20 tokens per parameter.

Chinchilla (70B params, 1.4T tokens) matched GPT-3's performance with 4× fewer parameters— smaller, faster to run, trained with the same compute budget but allocated differently.

Standard attention is O(n²) in both memory and compute. At 128K tokens, that's 16 billion attention elements per layer. The quadratic wall is the fundamental bottleneck for long-context processing.

Sparse Attention Patterns

Sparse attention reduces complexity by only computing a subset of attention pairs:

• Longformer: Local sliding window + global tokens
• BigBird: Random + window + global patterns
• Theoretical result: sparse patterns can be universal approximators

State Space Models and Alternatives

State Space Models like Mamba offer a different paradigm: O(n) training and inference with O(1) memory during inference. They use structured recurrence that can be computed efficiently via convolution during training.

Hybrid approaches like Jamba combine SSM layers with Transformer layers, getting benefits of both architectures.

RAG: Sidestepping the Problem

Retrieval-Augmented Generation avoids the long-context problem entirely. Instead of fitting everything into context, retrieve relevant chunks from an external knowledge base and include only those.

Trade-offs: latency (retrieval adds time), retrieval quality (wrong chunks = wrong answers), and complexity (another system to maintain).

The field is moving fast. Here's what I'm watching:

Efficiency Focus

Inference costs dominate the economics now. Training is one-time; serving is forever. Expect more aggressive quantization (FP4, even binary for some operations), better speculative decoding, and distillation to smaller, faster models.

Long Context via Architecture

SSM hybrids are becoming practical. The dream: O(n) attention with O(n²) quality. Linear attention variants are improving. The quadratic wall may eventually fall.

Reasoning and Search

Chain-of-thought as explicit search. Tree of Thought explores multiple reasoning paths. Best-of-N sampling trades compute for quality. Inference-time compute scaling—using more compute at inference for harder problems—is an active research area.

The Agent Paradigm

LLMs as reasoning engines embedded in larger systems. Tool use (code execution, search, databases). Multi-step planning. The model becomes one component in an architecture, not the whole system.

Final Thoughts

I started programming on a Commodore 64, typing in BASIC listings from magazines. The idea that I'd one day be explaining how artificial systems can generate coherent text, write code, and reason about problems would have seemed like science fiction.

Yet here we are. The Transformer architecture—matrix multiplications, softmax, layer norm, all pieces that would be familiar to any linear algebra student—combined at scale produces capabilities that still surprise me.

What excites me most is that we're still in early days. The "Attention Is All You Need" paper is less than a decade old. Flash Attention is from 2022. The techniques I've described here will likely seem primitive in another decade.

The fundamentals, though—understanding the memory hierarchy, the parallelism trade-offs, the mathematical foundations—those will remain useful. Learn the principles, not just the current instantiations.

And keep building. That's still the best way to really understand.