Biological Inspiration Introductory

Long before computers existed, scientists observed that the human brain processes information through a vast network of interconnected cells called neurons. Each biological neuron receives chemical signals through branching fibres called dendrites, integrates those signals in its cell body (the soma), and — if the combined signal exceeds an internal threshold — fires an electrical impulse along its axon to downstream cells. This "integrate and fire" mechanism, repeated across roughly 86 billion neurons with trillions of connections, gives rise to everything from reflex actions to abstract reasoning.

In 1943, McCulloch and Pitts created the first mathematical model of a neuron: a binary threshold unit that sums its inputs and outputs 1 if the sum exceeds a fixed threshold, 0 otherwise. The mapping from biology to mathematics is direct: dendrites become numeric inputs, synaptic strengths become weights, the soma becomes a weighted summation, and the axon firing becomes an activation function. This abstraction — inputs → weighted sum → activation — is still the foundation of every neural network today.

The analogy has important limits. Biological neurons communicate via discrete spikes; artificial neurons use continuous real-valued outputs. Biological learning involves complex biochemical processes; artificial networks learn by gradient descent on a loss function. The phrase "inspired by, not modelled after" is exactly right. Deep learning borrowed the high-level architecture of layered computation and discarded almost everything else in favour of mathematical tractability.

The Perceptron In-depth

In 1957, Frank Rosenblatt at the Cornell Aeronautical Laboratory built the Perceptron — the first machine specifically designed to learn from examples. The idea was elegantly simple: represent a decision-making unit as a weighted sum of inputs passed through a step function. If the total weighted input exceeds a threshold, the unit fires (outputs 1); otherwise it stays silent (outputs 0). Crucially, the weights could be adjusted automatically when the unit made a mistake — this was the first learning algorithm for a neural model.

The perceptron structure has four components. First, numeric inputs x₁, x₂, …, xₙ — these could be pixel intensities, sensor readings, or any measurable feature. Second, a weight wᵢ for each input, representing how important that feature is. Third, a bias b that shifts the decision boundary independently of the inputs. Fourth, a step activation function that converts the raw weighted sum into a binary decision.

The perceptron learning rule is the ancestor of gradient descent. After every prediction, if the prediction was correct, do nothing. If the network predicted 0 but the true label was 1, increase each weight by a small fraction of the corresponding input. If the network predicted 1 but should have predicted 0, do the reverse. This simple rule has a remarkable theoretical guarantee: if the training data is linearly separable, the perceptron will converge to a correct solution in a finite number of steps — the Perceptron Convergence Theorem.

Tracing the AND logic gate concretely: AND outputs 1 only when both inputs are 1. Start with all weights at 0. When we show (1,1)→1 and the network predicts 0 (since 0<0), we add the inputs to the weights. After a few cycles the perceptron settles at weights w₁=1, w₂=1, b=−1.5, which correctly separates AND's one positive case from the three negatives by the line x₁ + x₂ = 1.5.

The XOR Problem & MLP Motivation In-depth

In 1969, Marvin Minsky and Seymour Papert published Perceptrons, a rigorous mathematical analysis of what single-layer networks could and could not compute. Their central result was devastating: a single-layer perceptron cannot learn the XOR function. XOR outputs 1 when exactly one of two inputs is 1: (0,0)→0, (0,1)→1, (1,0)→1, (1,1)→0. The proof is geometric — there is no single straight line that can separate the two positive cases from the two negative cases. They form a checkerboard that is impossible to bisect with one hyperplane.

The Minsky-Papert result triggered the first AI winter: funding dried up and neural network research sat dormant for roughly 15 years. The irony is that the paper itself pointed toward the solution — adding hidden layers could overcome these limitations, but they doubted an efficient learning algorithm for such networks could be found. That algorithm — backpropagation, popularised by Rumelhart, Hinton, and Williams in 1986 — became the key that unlocked the field.

The geometric resolution is illuminating. With a hidden layer, the network first learns two intermediate linear boundaries: one that isolates (1,1) and one that isolates (0,0). The hidden layer outputs encode whether the input is in each region. The output layer then combines these hidden representations to produce the XOR decision — a task that is linearly separable in the transformed space. This is the core insight of deep learning: each layer transforms the data into a representation where the next layer's job becomes easier.

Multi-Layer Perceptron (MLP) In-depth

The Multi-Layer Perceptron (MLP) — also called a feedforward network or fully connected network — adds one or more hidden layers between the input and output. Every neuron in each layer connects to every neuron in the next layer. The key architectural decision is that a non-linear activation function is applied after each layer's weighted sum, breaking the linear chain that would otherwise collapse the whole network into a single affine transformation.

Each hidden layer acts as a feature detector. The first hidden layer learns combinations of the raw inputs — in image recognition, this might correspond to edges or local contrasts. The second hidden layer learns combinations of those features — perhaps corners and junctions. Deeper layers learn increasingly abstract concepts. This hierarchical representation learning is why depth is valuable: rather than memorising the training set, the network learns what to look for.

Architecture notation is typically given as a list of layer sizes. [4, 8, 8, 3] means 4 inputs, two hidden layers of 8 neurons, and 3 outputs. The total parameter count: for each layer, (inputs to that layer) × (neurons in that layer) weights plus one bias per neuron. A [4, 5, 4, 3] network has (4×5+5) + (5×4+4) + (4×3+3) = 25 + 24 + 15 = 64 parameters.

The Forward Pass In-depth

The forward pass is the computation that transforms an input vector into a prediction by passing it sequentially through each layer. Understanding the forward pass precisely — including the shapes of every matrix and vector — is essential for debugging, designing architectures, and reasoning about computational cost.

For each layer l, the computation has two steps. First, compute the pre-activation Z by multiplying the previous layer's output A by the weight matrix W and adding a bias b. Second, apply the activation function f element-wise to Z to produce the output A of this layer. The output of the final layer is the network's prediction.

Universal Approximation Theorem Core

In 1989, George Cybenko proved a remarkable result about feedforward networks: a network with a single hidden layer using sigmoid-like activation functions, and sufficiently many neurons, can approximate any continuous function on a compact subset of ℜ² to arbitrary precision. Hornik (1991) extended this to any non-constant, bounded, continuous activation function. This result — the Universal Approximation Theorem — gives MLPs their theoretical power: they are, in principle, general-purpose function approximators.

The intuition is geometric. A single neuron with a step function carves out a half-space. With many neurons, you can approximate arbitrary regions. With smooth activations, you build up smooth functions by summing many "bumps". As you add more neurons, the approximation gets finer — the diagram below shows a coarse 2-neuron step-wise approximation converging to the true function as width increases to 32.

The theorem has important caveats practitioners often overlook. It says a solution exists — it says nothing about whether gradient descent will find it, how many samples are needed, or whether the required network is computationally feasible. In practice, a single very wide layer may require exponentially more neurons than a deeper network to represent the same function. This is the practical motivation for depth: depth enables exponentially more efficient representation. In CNNs, this manifests as hierarchical feature detection: edges in layer 1, textures in layer 2, object parts in layer 3.

Network Anatomy & Hyperparameters Core

Understanding the key design choices of an MLP — and the consequences of setting them poorly — is essential for practitioners. The table below summarises the principal hyperparameters, their typical ranges, and the effects of extreme values. Each will be explored in depth in subsequent chapters; develop an intuition for the tradeoffs now.

Why Non-Linearity Matters Core

Consider two linear layers stacked directly: Z = W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂). The result is another linear function with a combined weight matrix W = W₂W₁ and a combined bias. No matter how many linear layers you stack, the composition remains a single affine (linear + shift) transformation. This means a deep linear network is no more expressive than a logistic regression — it can only learn straight hyperplane decision boundaries.

An activation function f applied between layers breaks this collapse: A₂ = W₂ · f(W₁x + b₁) + b₂. Now the result is genuinely non-linear and the two layers are no longer collapsible into one. A good activation function must satisfy three practical requirements: it must be non-linear (obviously), differentiable almost everywhere (so gradients can flow during backpropagation), and computationally cheap (it is applied millions of times per forward pass).

Sigmoid In-depth

The sigmoid function was the default activation in neural networks throughout the 1980s and 1990s. It takes any real-valued input and squashes it into the range (0, 1), which made it a natural fit for modelling probabilities. The S-shaped curve rises steeply near z = 0 and flattens toward 0 for very negative inputs and toward 1 for very positive inputs. This flattening is the source of its central problem in deep networks.

The derivative of σ(z) is elegantly expressed as σ(z)(1 − σ(z)). This has a maximum of 0.25 at z = 0, and falls to near-zero as |z| grows large. In a deep network, gradients are multiplied together as they propagate backward through layers. If most neurons saturate (i.e., z is large in magnitude), each multiplication by a derivative near 0 shrinks the gradient exponentially — this is the vanishing gradient problem. A network with 10 sigmoid layers loses a gradient factor of 0.25¹⁰ ≈ 0.0000001 before it reaches the first layer.

A second, subtler problem is that sigmoid outputs are never negative — they are always in (0, 1). This means gradients are always the same sign (all positive or all negative), which causes zig-zag updates in weight space. Tanh, which we examine next, solves this by being zero-centred. Today, sigmoid is used almost exclusively at the output layer of binary classifiers (where you genuinely want a probability) and in the gating mechanisms of LSTMs.

Tanh Core

Tanh (hyperbolic tangent) is a scaled and shifted version of sigmoid: tanh(z) = 2σ(2z) − 1. It squashes inputs to (−1, +1) instead of (0, 1). The critical improvement over sigmoid is that tanh is zero-centred — its outputs are balanced around zero. When the activation outputs are always positive (as in sigmoid), the gradient updates to all weights in the next layer always have the same sign. This forces the optimiser into a zig-zag path through weight space. Tanh's zero-centred outputs allow positive and negative gradients, enabling more direct paths toward the minimum.

Tanh still suffers from the vanishing gradient problem for large |z|, where the derivative tanh'(z) = 1 − tanh²(z) approaches zero. The maximum derivative is 1.0 at z = 0 — four times larger than sigmoid's maximum of 0.25 — which makes it somewhat less prone to gradient collapse. Tanh remains the preferred activation inside LSTM and GRU gates, where its zero-centered outputs help regulate cell state updates.

ReLU Family In-depth

The Rectified Linear Unit (ReLU) is disarmingly simple: pass the input through unchanged if it is positive, otherwise output zero. This single function — introduced to deep learning at scale by AlexNet in 2012 — transformed the field. Before ReLU, training deep networks beyond 5–6 layers was nearly impossible due to vanishing gradients from sigmoid and tanh. ReLU's constant gradient of 1 for positive inputs means gradients flow freely through activated neurons, enabling networks of 50, 100, or even 1000 layers.

ReLU also introduces sparse activation: on average, about 50% of neurons output exactly zero for any given input. This sparsity provides implicit regularisation — only the "relevant" neurons participate in each forward pass. However, this same property creates the Dead ReLU problem: if a neuron's pre-activation is always negative (e.g., because the bias drifts negative during training), its gradient is permanently zero and it never recovers. This can kill 10–40% of neurons in poorly initialised or high-learning-rate networks.

Leaky ReLU fixes dead neurons by allowing a small negative slope α (typically 0.01) for z < 0, ensuring the gradient is never exactly zero. ELU (Exponential Linear Unit) goes further with a smooth exponential curve for negative inputs, producing outputs closer to zero-mean — which can improve convergence. PReLU (Parametric ReLU) treats α as a learnable parameter, letting the network decide the optimal negative slope per channel.

GELU & Modern Activations Core

The Gaussian Error Linear Unit (GELU) was introduced by Hendrycks and Gimpel (2016) and quickly became the dominant activation in Transformer-based models. The key motivation: ReLU has a hard kink at z = 0 — the derivative jumps discontinuously from 0 to 1. GELU replaces this with a smooth curve by weighting the input by the probability that it is positive under a standard Gaussian distribution: f(z) = z · Φ(z), where Φ is the Gaussian CDF.

In practice, GELU is computed via a fast approximation: f(z) ≈ 0.5z(1 + tanh[√(2/π)(z + 0.044715z³)]). This smooth transition means GELU has a continuous gradient everywhere, which empirically improves training stability for deep Transformers. GPT-2, GPT-3, BERT, BART, T5, and virtually every large language model published since 2019 uses GELU in its feed-forward sublayers.

Swish (also called SiLU, Sigmoid Linear Unit) is another smooth variant: f(z) = z · σ(z). The input gates itself — neurons with large positive values pass through at full strength, while negative values are softly suppressed. Swish is used in EfficientNet, MobileNetV3, and several LLM variants. Mish extends this idea: f(z) = z · tanh(softplus(z)), and has achieved state-of-the-art performance on some computer vision benchmarks. All three share the property of being smooth, non-monotonic, and having a small negative dip near z ≈ −0.2, which provides a weak self-normalising property.

Softmax In-depth

Softmax is not an activation function in the same sense as ReLU or GELU — it is not applied element-wise independently to each neuron. Instead, it is a normalisation operation over an entire output vector, converting a vector of raw logits (unbounded real numbers) into a valid probability distribution. Every output is positive, and all outputs sum to exactly 1.0, making the output directly interpretable as class probabilities.

Softmax amplifies differences between logits. The largest logit receives a disproportionately high probability — the exponentiation makes differences exponential before normalisation. With logits [3.0, 1.0, 0.5], the first class dominates the probability. Subtracting the maximum logit before computing exponentials — max-trick — prevents numerical overflow without changing the output: softmax(z − max(z)) = softmax(z).

The temperature parameter T controls the sharpness of the distribution: softmax(z/T). With T → 0, the distribution collapses to a one-hot (greedy) selection of the highest logit. With T → ∞, it becomes uniform. In LLM token sampling, temperature is a key knob: T = 0.7 gives creative but coherent text; T = 1.5 gives more random, diverse outputs. At training time T = 1 is almost always used.

In PyTorch, never apply softmax before passing logits to nn.CrossEntropyLoss — this loss already applies log-softmax internally for numerical stability. Applying softmax beforehand causes the loss to compute log(softmax(logits)), introducing numerical errors. Always pass raw logits to CrossEntropyLoss.

Choosing an Activation Function Core

The choice of activation function is one of the most important and most misunderstood hyperparameters. The practical rule is simple: use ReLU as your baseline for CNNs and general MLPs, switch to GELU for anything Transformer-based, use Sigmoid only at binary classification output, and use Softmax only at multi-class output. The table below summarises when and why to use each.

Intuition Core

The central question of training is: "For every weight in the network, how much does the loss change if I nudge that weight by a tiny amount?" This quantity — the partial derivative of the loss with respect to each weight — is the gradient. To reduce the loss, we move each weight in the direction opposite to its gradient.

The naive approach is finite differences: for each weight w, compute loss(w + ε) − loss(w) / ε. This gives an approximate gradient for that weight. The problem is scale. GPT-4 has an estimated 1.8 trillion parameters. Computing one gradient update this way requires 1.8 trillion forward passes — at, say, 1 second per pass on a cluster, that is 57,000 years per update step. Completely impossible.

Backpropagation solves this by computing all gradients simultaneously in a single backward pass through the computational graph. The backward pass is no more expensive than the forward pass — it visits the same operations in reverse. The key ingredient is the chain rule, which tells us how to compose local gradients as they flow backward from the loss to the inputs.

Computational Graph In-depth

Every computation a neural network performs can be represented as a directed acyclic graph (DAG). Each node in the graph is a mathematical operation — addition, multiplication, exp, sigmoid, max. Each directed edge carries a tensor value from one operation to the next. The leaf nodes on the left are the inputs and weights; the single root node on the right is the scalar loss.

The forward pass is data flowing left to right through this graph — compute z₁ = w × x, then z₂ = z₁ − y, then L = z₂². Each intermediate value is stored (this is why training uses more memory than inference). The backward pass is gradients flowing right to left — starting with ∂L/∂L = 1 and applying the chain rule at each node. Every node knows how to compute its local gradient (e.g., the gradient through a multiplication node is the other operand), and backprop just multiplies local gradients together along each path.

PyTorch builds this graph dynamically as you execute Python code — every tensor operation with requires_grad=True records itself into the graph. When you call loss.backward(), PyTorch traverses the graph in reverse topological order and accumulates gradients into each leaf tensor's .grad attribute. JAX uses a slightly different approach (function transformation) but the computational graph concept is identical.

Chain Rule in Neural Networks In-depth

The chain rule is calculus's rule for differentiating composed functions: if L = f(g(x)), then dL/dx = (dL/dg) · (dg/dx). In a neural network, every layer is a composed function. The loss is a composition of all the layer operations stacked together. Backprop is simply the chain rule applied methodically in reverse order through every layer.

For a single layer l with pre-activation Z⁽ˡ⁾ = A⁽ˡ⁻¹⁾ · W⁽ˡ⁾ + b⁽ˡ⁾ and output A⁽ˡ⁾ = f(Z⁽ˡ⁾), the gradient of the loss with respect to the weights W⁽ˡ⁾ decomposes into three factors by the chain rule: how the loss changes with the activation, how the activation changes with the pre-activation (the derivative of the activation function), and how the pre-activation changes with the weights (which is simply A⁽ˡ⁻¹⁾). Multiplied together, these give the weight gradient for that layer.

The error signal δ⁽ˡ⁾ is the gradient of the loss with respect to the pre-activation Z⁽ˡ⁾. It packages the chain rule product up to layer l. To propagate backward one more layer, we multiply δ⁽ˡ⁾ by the weight matrix W⁽ˡ⁾ transposed (to "route" gradients back to the correct inputs), then element-wise multiply by f'(Z⁽ˡ⁻¹⁾) — the local derivative of the activation. This recursion continues all the way to the first layer.

Vanishing Gradients In-depth

Gradients propagate backward by multiplication. If the gradient at each layer is a number less than 1, repeated multiplication makes the product shrink exponentially. Sigmoid's maximum derivative is 0.25. In a 10-layer sigmoid network, the gradient arriving at layer 1 has been multiplied by at most 0.25 per layer — giving 0.25¹⁰ ≈ 9.5 × 10⁻⁷, effectively zero. The first layers receive no gradient signal and learn nothing while the last few layers update normally.

This is why networks deeper than 5–6 layers were impractical before 2012. The symptom is clear in training: the loss decreases at first but then plateaus far above the optimal, and inspecting per-layer gradients shows near-zero values in the early layers. The activations in these layers also collapse — either all outputs are near 0 or near 1 (for sigmoid), with near-zero variance.

The primary solutions in order of importance: (1) ReLU activations — gradient is exactly 1 for positive inputs, breaking the exponential decay. (2) Residual connections (ResNet, Ch 4.5) — add a "skip" path that carries gradients directly from the loss to early layers, bypassing the layer multiplications entirely. (3) Batch Normalisation (Ch 4.4) — normalises activations to prevent saturation. (4) He initialisation — initialises weights to maintain gradient scale across layers.

Exploding Gradients Core

The opposite pathology occurs when the gradient magnitudes grow exponentially as they propagate backward — if the weight matrices have large singular values, each multiplication amplifies rather than shrinks the gradient. This is especially common in Recurrent Neural Networks (RNNs) processing long sequences: the gradient at time step 1 is the product of 100 Jacobian matrices, and if each has norm slightly above 1, the product explodes exponentially.

The symptom is unmistakable: the loss goes to NaN within the first few training steps, and weights become inf. The standard fix is gradient clipping: compute the global norm of all gradients, and if it exceeds a threshold, scale all gradients down proportionally. This preserves the direction of the gradient update but caps its magnitude. A clipping value of 1.0 is a widely used default.

Weight Initialisation In-depth

Before a single training example is shown, every weight must be given a starting value. This choice has enormous consequences. If all weights start at zero, every neuron in a layer computes exactly the same function and receives exactly the same gradient — no matter how many epochs you train, all neurons in a layer remain identical forever. This is the symmetry breaking problem: weights must differ to learn different features.

Initialising with random values breaks symmetry, but the variance of those values is critical. If weights are too small, activations shrink exponentially with depth — by layer 10, inputs have collapsed to near zero and there is no gradient signal. If weights are too large, activations explode exponentially — inputs saturate sigmoid/tanh and gradients vanish, or the network numerically overflows. The goal is to choose a variance that keeps activation magnitudes approximately stable across all layers.

Xavier/Glorot initialisation (Glorot & Bengio, 2010) derives the optimal variance analytically for linear activations and symmetric non-linearities like Tanh. It sets the weight variance to 2/(nᵢₙ + nₒᵤₜ), balancing the signal variance across both forward and backward passes. He/Kaiming initialisation (He et al., 2015) adjusts for the fact that ReLU kills half of all activations (setting them to zero), which halves the effective variance. He init compensates by scaling up by √2, using variance 2/nᵢₙ. For any ReLU-based network, He initialisation is the correct default.

Batch Normalisation In-depth

Ioffe and Szegedy (2015) diagnosed a key training instability they called internal covariate shift: as the parameters of layer l change during training, the distribution of inputs seen by layer l+1 shifts. The later layer must constantly readjust to its changing input distribution, slowing convergence. Their solution — Batch Normalisation — normalises each layer's pre-activation values across the mini-batch, forcing the distribution to approximately N(0,1) regardless of what the previous layer learned.

The normalisation has four steps. First, compute the mean μ_B and variance σ²_B of the current mini-batch. Second, subtract the mean and divide by the standard deviation to get x̂ — a zero-mean, unit-variance vector. Third — and critically — apply a learnable scale γ and shift β: y = γx̂ + β. These learned parameters let the network undo the normalisation if that is optimal; without them, BatchNorm would permanently constrain every layer's activations to N(0,1), which is too restrictive.

At inference time there is no mini-batch, so BatchNorm uses running statistics — exponential moving averages of μ_B and σ²_B accumulated during training — to normalise. This is why you must call model.eval() before inference: it switches BatchNorm from batch statistics to running statistics. Forgetting this is one of the most common and damaging bugs in deep learning practice.

Dropout In-depth

Srivastava et al. (2014) introduced dropout as a computationally cheap approximation to training an ensemble of exponentially many networks. During each forward pass, every neuron is independently deactivated with probability p. The remaining (1−p) fraction of neurons process the input and update normally. At inference, all neurons are active — but since the network was trained with only (1−p) of neurons active on average, the outputs are scaled down by (1−p) to keep the expected activation magnitude consistent. In practice, inverted dropout is used: scale activations up by 1/(1−p) during training so no adjustment is needed at inference.

The theoretical justification has three complementary perspectives. The ensemble view: with N neurons, there are 2^N possible sub-networks; dropout samples a different one each forward pass, and inference approximates their average. The co-adaptation view: neurons cannot rely on specific other neurons being present, so they learn more independent, redundant features. The noise injection view: randomly zeroing neurons adds multiplicative noise, acting like a data augmentation that prevents the network from memorising specific training patterns.

Practical guidance: dropout rates of 0.1–0.2 work well for earlier or convolutional layers; 0.3–0.5 for large fully connected layers. Dropout is rarely applied to convolutional feature maps (DropBlock is preferred there). In Transformer models, dropout is applied after attention and after the feed-forward sublayer with rates of 0.1 being standard. For very large models, lower dropout rates (0.05–0.1) are preferred as the model already has strong regularisation from scale.

Optimisers for Deep Learning In-depth

Stochastic Gradient Descent (SGD) updates each weight by subtracting a fraction of its gradient: θ ← θ − α∇L. Plain SGD oscillates badly in directions with high curvature (the narrow valleys common in deep loss landscapes) and moves too slowly in flat directions. SGD with Momentum adds a velocity term that accumulates gradient history, smoothing oscillations and accelerating through flat regions. It remains the preferred optimiser for training ResNets and CNNs on image classification.

Adam (Kingma & Ba, 2014) computes an adaptive learning rate per parameter: it tracks the first moment (mean of gradients) and second moment (mean of squared gradients) and uses their ratio to scale each parameter's update independently. A parameter whose gradient has been consistently large gets a smaller effective step; a parameter with small, consistent gradients gets a larger step. This makes Adam dramatically faster to converge on most problems and largely insensitive to the global learning rate choice.

AdamW (Loshchilov & Hutter, 2019) fixes a subtle mathematical bug in Adam's weight decay implementation. In Adam, L2 regularisation (weight decay) was applied to the gradient before the adaptive scaling — which means the actual weight penalty is scaled by the adaptive term and varies per parameter. AdamW decouples weight decay from the gradient update, applying it directly to the parameters after the Adam step: θ ← θ − λθ (separately from the Adam gradient term). This is now the mandatory default for training large language models and Transformers.

Learning Rate Schedules Core

A fixed learning rate is rarely optimal throughout training. Early in training, large steps are desirable — the network is far from a good solution and can afford rough updates. Later in training, large steps overshoot the loss minimum and cause oscillation — smaller steps are needed for fine-grained convergence. Learning rate schedules adjust the learning rate automatically over the course of training.

The warmup + cosine annealing schedule has become the dominant approach for Transformer training. For the first few percent of training steps (the "warmup"), the learning rate increases linearly from near-zero to the target learning rate. This protects the model from large, destabilising gradient updates at the start of training, when the parameters are random and gradients are noisy. After warmup, the learning rate follows a cosine curve from the peak down to a small minimum — providing a smooth, continuous decay that typically outperforms staircase schedules.

The Complete Training Loop In-depth

Putting it all together: a production-quality PyTorch training loop that incorporates gradient clipping, the model.train()/eval() switch, a validation loop, and a cosine learning rate schedule. Every line is intentional — understanding why each piece is there is as important as knowing what it does.

Other Regularisation Techniques Reference

Beyond dropout and BatchNorm, a range of regularisation techniques are routinely used in modern deep learning. The right combination depends on the task, architecture, and dataset size. The table below summarises the most important ones with their use cases and PyTorch APIs.

Why Not MLP for Images? Core

A standard 224×224 RGB image contains 224 × 224 × 3 = 150,528 individual pixel values. A single hidden layer of 1,024 neurons in a fully connected MLP requires 150,528 × 1,024 = 154 million weights — for the first layer alone, before any useful representation has been learned. Scale this to the thousands of neurons in a real network and you have a parameter count that dwarfs the available training data, making the network impossible to train effectively.

The parameter count is only the first problem. A deeper issue is that the MLP treats every pixel as equally related to every other pixel — it has no concept of spatial locality. A cat's eye in the top-left corner and a cat's eye in the bottom-right corner are unrelated to the MLP; it must learn to recognise them as independent patterns, requiring separate learned features for every possible position. Images have three structural properties the MLP ignores: local structure (nearby pixels are more related than distant ones), translation invariance (a cat is a cat wherever it is), and compositionality (parts compose into objects).

CNNs address all three with a single idea: weight sharing via convolution. Instead of connecting each pixel to each neuron, a CNN applies a small learned filter (e.g., 3×3) across the entire image. The same 27 weights (3×3×3 channels) are reused at every spatial position. This reduces the first layer's parameters from 154 million to a few hundred, while the filter learns to detect the same feature (an edge, a colour gradient) wherever it appears.

The Convolution Operation In-depth

A convolution applies a small learnable filter — called a kernel — to an input feature map by sliding it across every position and computing an element-wise dot product at each location. At position (i, j), the output value is the sum of all products between the kernel weights and the corresponding input patch. If the kernel has learned to detect horizontal edges, positions where horizontal edges are present produce large activations; other positions produce small ones. With 64 different kernels, you get 64 different feature maps, each detecting a different pattern.

Three hyperparameters control the output size. Kernel size K (typically 3×3 in modern networks): larger kernels see more context but use more parameters. Stride S: how many pixels the kernel jumps between applications. Stride=2 halves the spatial resolution. Padding P: zeros added around the input. "Same" padding (P=(K-1)/2) preserves the input spatial size, which is standard for 3×3 convolutions.

The parameter count scales with kernel size and channel counts, not with image size — this is the core efficiency of CNNs. A 3×3 conv layer with 64 input channels and 128 output channels has 3 × 3 × 64 × 128 + 128 = 73,856 parameters regardless of whether the input is 32×32 or 512×512. This size-invariance is why a single CNN trained on 224×224 images can be applied to any input resolution at inference.

Pooling Core

After convolution extracts local features, the spatial dimensions of the feature maps are often larger than necessary — and carrying large feature maps through many layers is expensive. Pooling layers reduce the spatial size while retaining the most important information. Max pooling (the dominant choice) partitions the feature map into non-overlapping windows and takes the maximum value in each. With a 2×2 window and stride 2, max pooling halves both height and width, reducing the number of activations by 4× while making features more invariant to small spatial shifts.

The intuition behind max pooling: each feature map cell contains an activation measuring "how strongly is this feature present at this position?" The maximum within a 2×2 region answers "was this feature present anywhere in this region?" — a coarser, position-invariant question that is still useful for recognition. An eye is an eye whether it is 2 pixels to the left or right. Max pooling discards that 2-pixel difference.

Global Average Pooling (GAP) is a key modern innovation: instead of pooling 2×2 regions, it averages the entire spatial extent of each channel into a single scalar. Applied after the last convolutional block, GAP converts a [B, C, H, W] tensor into [B, C], replacing the large fully-connected layers that were responsible for most parameters in early CNN architectures. ResNet and subsequent models use GAP as the bridge between convolutional features and the classification head.

CNN Architecture In-depth

A typical CNN follows a regular pattern: alternating convolution blocks and pooling layers, progressively reducing the spatial dimensions while increasing the number of feature channels. The spatial compression concentrates local features into increasingly compact representations. The channel expansion gives the network more "vocabulary" for describing what it sees. The final stage converts the 3D feature tensor into a class prediction via either a series of fully-connected layers or Global Average Pooling.

The hierarchical feature learning in CNNs is perhaps their most important property. Visualisation studies show that early layers (Layer 1-2) learn to detect simple patterns: oriented edges, colour gradients, and textures. Middle layers (Layer 3-4) detect parts: corners, curves, texture patches that resemble scales, fur, or brickwork. Late layers detect whole objects or object parts: faces, wheels, paws. This hierarchy emerges from training alone — it is not hand-crafted. The network discovers it is useful by virtue of gradient descent on classification loss.

ResNet & Skip Connections In-depth

By 2014, the empirical pattern was clear: deeper networks should perform better, because more layers can learn more complex functions. Attempts to train networks with 20–30 layers consistently produced worse results than 10–15 layer networks — not just on validation, but on the training set. This degradation problem was not overfitting. It meant the optimiser was fundamentally unable to train very deep networks, even when additional capacity should have helped.

He et al.'s insight was deceptively simple. If a shallower network achieves some accuracy A, then a deeper network that copies the shallower network's layers and sets all additional layers to identity (f(x) = x) should achieve at least accuracy A. But gradient descent cannot easily learn the identity mapping — pushing all weights in a layer toward zero is hard, because zero weights produce zero outputs (not the input x). The residual block makes this easy by reformulating the learning objective: instead of learning f(x), the block learns the residual r(x) = f(x) − x. The shortcut connection adds the original input directly: output = r(x) + x. Now learning the identity is trivial — just set r(x) = 0.

The practical impact was enormous. ResNet-152 (152 layers, 2015) achieved 3.57% Top-5 error on ImageNet — surpassing human-level performance (~5%). The skip connection also dramatically improves gradient flow: gradients can propagate directly from the loss to any earlier layer through the shortcut path, bypassing the multiplicative chain that causes vanishing gradients. This is why skip connections appear in virtually every modern architecture — Transformers include them as a core component.

Modern CNNs Reference

The history of CNNs on ImageNet is a story of architectural innovations compounding: each milestone introduced one key idea that is now ubiquitous. AlexNet proved GPU-trained deep networks could work. VGG showed depth with only 3×3 convolutions was sufficient and more principled. Inception introduced parallel multi-scale filters. ResNet introduced skip connections enabling extreme depth. EfficientNet used Neural Architecture Search to jointly scale depth, width, and resolution. Vision Transformers (ViT) ultimately replaced convolutions entirely with attention — showing that the convolutional inductive bias, while useful, is not necessary given enough data.

Receptive Field Core

The receptive field of a neuron in layer l is the region of the original input image that can influence that neuron's activation. A neuron in the first conv layer with a 3×3 kernel sees a 3×3 region. A neuron in the second conv layer sees a 5×5 region (each of its 9 input cells saw a 3×3 region, overlapping to cover 5×5). With each additional 3×3 conv layer, the receptive field grows by 2 in each dimension. After k layers of 3×3 convolutions: receptive field = 2k + 1 pixels.

Pooling layers and strided convolutions multiply the receptive field growth. After a 2× pooling layer, subsequent convolutional layers grow the receptive field twice as fast. This is why deep CNNs develop neurons in later layers that respond to large, complex objects: they have receptive fields spanning the entire image. The final convolutional layer in ResNet-50 has a theoretical receptive field of 483×483 — larger than the 224×224 input — ensuring every output cell has seen the full input context.

Why Sequences Need Memory Core

An MLP processes each input independently. Feed it the word "bank" and it produces a prediction — but it has no way to know whether the previous words were "river" or "money". CNNs add local spatial context via convolution windows, but they still process a fixed-size input with no persistent state across positions. Neither architecture is suited to data where order matters and length varies: text, speech, time series, video.

The semantic difference between "The dog bit the man" and "The man bit the dog" lies entirely in word order — same vocabulary, opposite meaning. Processing each word independently destroys this information. A model needs to carry a memory of what it has already seen as it processes each new token. This is the core motivation for the Recurrent Neural Network: maintain a hidden state hǕₜ that accumulates information from all previous time steps.

Basic RNN In-depth

The vanilla RNN cell takes the current input xₜ and previous hidden state hₜ₋₁, applies a weighted sum, and passes through tanh. The same weight matrices Wₕ and Wₓ are used at every single time step — weight sharing across time, analogous to how a CNN shares weights across space. Unrolling through T steps creates an effective computational graph T layers deep: a sequence of 100 words = 100 effective layers = severe vanishing gradient risk.

Backpropagation Through Time (BPTT) In-depth

BPTT applies backprop to the unrolled RNN graph. To update Wₕ, gradients must flow from the loss at step T back through every time step, multiplying by the Jacobian ∂hₜ/∂hₜ₋₁ = Wₕᵀ · diag(tanh’(·)) at each step. Across T steps: product of T such matrices. If ‖Wₕ‖ < 1, the product shrinks exponentially — vanishing gradient. If > 1, it grows — exploding. Practical limit: vanilla RNNs cannot reliably learn dependencies beyond ~10–20 steps.

LSTM — Long Short-Term Memory In-depth

Hochreiter & Schmidhuber (1997) added a cell state cₜ — a horizontal "highway" running through all time steps with only element-wise operations. Gradients flowing through cₜ are multiplied by learned scalar gate values (not weight matrices), dramatically reducing vanishing. Three sigmoid gates ∈ (0,1) control information flow: forget gate fₜ decides what to erase from cₜ₋₁; input gate iₜ decides what new candidate c̃ₜ to write; output gate oₜ decides what to expose as hₜ.

The cell state update is the key: cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ — only addition and element-wise multiply, no matrix multiplication. The forget gate can learn to stay near 1 for important long-range information, creating an almost unimpeded gradient path over hundreds of steps.

GRU — Gated Recurrent Unit Core

Cho et al. (2014) introduced the GRU as a simplified LSTM with only 2 gates: an update gate zₜ (blend old vs new hidden state) and a reset gate rₜ (how much past state to use for candidate). No separate cell state — just one hidden vector. Fewer parameters, faster training, often competitive with LSTM.

Seq2Seq & Attention Preview Core

Seq2Seq (Sutskever et al. 2014) combined two RNNs: an encoder compressing the input into a fixed context vector, and a decoder generating output conditioned on it. The fundamental flaw: the entire input — regardless of length — is squeezed into one fixed-size vector. Bahdanau et al. (2015) fixed this with attention: at each decoder step compute a weighted sum over ALL encoder states cₜ = ∑ αₜᴵ · hᴵ. This is the direct predecessor of Transformer self-attention in Ch 4.7.

Why Transformers Replaced RNNs Core

The RNN’s fundamental flaw is its sequential nature. To process token t, you must first finish token t−1. This means training on a sequence of 10,000 tokens requires 10,000 sequential steps — no amount of hardware parallelism can help. Training GPT-3 (which processes sequences of 2,048 tokens) on an RNN would be computationally impossible at scale. The Transformer abolishes this constraint: all tokens are processed simultaneously, turning a sequential problem into a parallel matrix multiplication problem that GPUs excel at.

The second flaw is information decay. Even with LSTM gating, information from 500 tokens ago is weakly represented in the current hidden state. In contrast, the Transformer’s attention mechanism creates a direct path between any two tokens regardless of distance. The word “it” 300 tokens after “the cat” can attend directly to “cat” with no intermediate steps — the path length is always 1. This is why Transformers handle long documents, code files, and entire books in ways that RNNs fundamentally cannot.

Scaled Dot-Product Self-Attention In-depth

Self-attention is the mechanism that allows each token to gather information from all other tokens in the sequence. For each token, three vectors are computed by applying learned linear projections: a Query (“what am I looking for?”), a Key (“what do I contain?”), and a Value (“what information do I carry?”). The attention weight between token i and token j is computed as the dot product of token i’s Query with token j’s Key, scaled by √dₖ (to prevent the dot products from growing large and saturating the softmax), then normalised via softmax across all tokens. The output for token i is the weighted sum of all Value vectors.

The scaling by √dₖ is critical. For dₖ=64, a random unit vector has dot product with another of approximately 8 in expectation. Without scaling, these large values push the softmax into near-zero gradient regions. Dividing by √64=8 normalises the variance and keeps gradients healthy. This is why the formula specifically includes the √dₖ denominator.

Multi-Head Attention In-depth

A single attention head can only attend to information from one representational subspace at a time. Multi-head attention runs h independent attention functions in parallel, each with its own learned projection matrices Wᴵᵈ, Wᴵᴷ, Wᴵᵝ. Each head learns to attend to a different type of relationship: one head might track syntactic subject-verb agreement, another resolves coreferences (“he” → “John”), another focuses on positional proximity, and yet another captures semantic similarity. All h head outputs are concatenated and projected back to dᵐᵒᵑᵉℹ via Wᵊ — a learned combination of what each head discovered.

The dimension of each head is dₖ = dᵐᵒᵑᵉℹ / h, so the total computation is the same as a single attention with dᵐᵒᵑᵉℹ dimensions. GPT-3 uses 96 attention heads with dᵐᵒᵑᵉℹ=12,288, giving each head a 128-dimensional subspace. This is one of the key scaling choices: more heads = more types of relationships the model can simultaneously track.

Positional Encoding Core

Attention is permutation invariant: swap any two tokens and the attention scores are identical (just reordered). The Transformer has no inherent concept of token order — “cat sat mat” and “mat sat cat” would produce the same attention weights if not corrected. To inject positional information, the original paper adds a fixed sinusoidal positional encoding to each token embedding before the first attention layer.

The sinusoidal encoding uses different frequencies for different embedding dimensions: PE(pos, 2i) = sin(pos/10000^(2i/d)) and PE(pos, 2i+1) = cos(pos/10000^(2i/d)). Each position gets a unique vector. The model can learn to extract relative position from these encodings because PE(pos+k) can be expressed as a linear function of PE(pos) — the Transformer can learn to do relative position arithmetic via attention.

Modern LLMs use Rotary Positional Embeddings (RoPE) instead — used in LLaMA, GPT-NeoX, and most recent architectures. RoPE encodes position by rotating the Q and K vectors by an angle proportional to position before computing dot products. The key advantage: attention scores naturally depend on the relative position between tokens (pos_i − pos_j), not absolute positions — enabling better generalisation to longer sequences than seen during training.

Full Transformer Architecture In-depth

The complete Transformer block repeats the same structure N times: Multi-Head Attention → Add&Norm → Feed-Forward Network → Add&Norm. The Feed-Forward Network is a two-layer MLP applied independently to each token position: FFN(x) = max(0, xW₁+b₁)W₂+b₂, expanding from dᵐᵒᵑᵉℹ to 4·dᵐᵒᵑᵉℹ then back. This 4× expansion and contraction lets the model compute complex non-linear transformations per token. The Add&Norm step adds the input as a residual connection and applies Layer Normalisation — enabling stable training at depth and solving the vanishing gradient problem (Ch 4.3).

The original Transformer had two components. The encoder is bidirectional: every token attends to all other tokens in both directions. It produces contextualised representations of the input sequence. The decoder is causal: each output token attends only to previously generated tokens (enforced by the causal mask), plus cross-attention to all encoder outputs. This asymmetry is fundamental: the encoder understands the full input at once, while the decoder generates output token-by-token, attending to what it has already produced.

BERT vs GPT vs T5 In-depth

The original Transformer (2017) had both an encoder and decoder. Within a year, two research groups realised that you could use just one half and pretrain it on massive text corpora to create a general-purpose language model. Google Brain introduced BERT (Bidirectional Encoder Representations from Transformers, 2018) using the encoder only, pretrained with Masked Language Modelling: randomly mask 15% of tokens and train the model to predict them using bidirectional context. OpenAI introduced GPT (Generative Pre-trained Transformer, 2018) using the decoder only, pretrained with standard next-token prediction. These two approaches define the landscape of modern NLP.

What is Transfer Learning? Core

Training a large neural network from scratch requires two things that most practitioners do not have: millions of labelled examples and millions of GPU-hours. GPT-3 cost approximately $4.6M to pre-train; BERT took 4 days on 64 TPU v3 chips. Transfer learning solves this by splitting the problem into two phases. Pre-training: train on a large, general dataset (entire internet text, 1.2M ImageNet images, all of Wikipedia) until the model learns rich, reusable representations. Fine-tuning (or adaptation): start from those learned weights and update them toward a specific downstream task — with far less data and compute.

The foundational insight is that early layers of neural networks learn general features that transfer across tasks. In CNNs, layer 1 universally detects oriented edges regardless of whether the network was trained for cats, cars, or faces. In language models, early layers build syntactic representations applicable to any NLP task. This hierarchy of generality — general features at the bottom, task-specific at the top — is what makes transfer learning work. The analogy: a radiologist who spent years in medical school (pre-training on general anatomy) can specialise in chest X-ray reading (fine-tuning on specific task data) far faster than someone starting from scratch.

Feature Extraction Core

Feature extraction freezes every parameter of the pre-trained backbone and trains only a small task-specific head attached to the top. Because no gradients need to flow through the frozen backbone, forward passes do not require gradient tracking — making this dramatically faster and less memory-intensive than fine-tuning. For image tasks, the head is typically a linear classifier or small MLP on top of the backbone’s pooled output. For text tasks with BERT, the [CLS] token embedding (position 0 of the last hidden state) serves as a fixed-size sentence representation, since BERT is trained to pack sentence-level information into it during pre-training (via the Next Sentence Prediction objective).

Feature extraction works best when your task is similar to the pre-training distribution and you have limited labelled data. If your task is quite different from pre-training (e.g., medical imaging from a model pre-trained on natural images), the frozen features may not be informative enough — full fine-tuning is needed. The key question: do the features the backbone learned happen to be useful for your task?

Full Fine-Tuning In-depth

Full fine-tuning updates all weights of the pre-trained model on the downstream task. The critical constraint is the learning rate. Pre-trained weights encode years of training signal from massive datasets — a large learning rate will destroy this knowledge within a few gradient steps in a process called catastrophic forgetting. The pre-training knowledge disappears as new task gradients overwrite it. The standard remedy: use a learning rate 10–100× smaller than the original pre-training LR (typically 2e-5 to 5e-5 for BERT/GPT-sized models, vs 3e-4 for pre-training).

Layer-wise learning rate decay (LLRD) refines this further: assign progressively smaller learning rates to earlier layers. The final layer gets the full (small) LR; each preceding layer gets the LR multiplied by a decay factor (typically 0.9 per layer). This preserves the most general representations in early layers while allowing later layers to adapt more aggressively. Google’s ULMFiT (Howard & Ruder, 2018) pioneered this technique and it remains standard practice for fine-tuning large Transformers.

Parameter-Efficient Fine-Tuning (PEFT) In-depth

Full fine-tuning a 7B-parameter model requires the same GPU memory as pre-training it — typically 80GB+ in fp16. For 175B (GPT-3) or 540B (PaLM) models, full fine-tuning is simply impossible on any commercially available hardware. Parameter-Efficient Fine-Tuning (PEFT) methods address this by updating only a tiny fraction of parameters — usually 0.1–5% — while keeping the rest frozen, achieving near-full-fine-tuning quality at a fraction of the cost.

Adapter layers (Houlsby et al., 2019) insert small bottleneck networks between Transformer layers — project down to a small dimension, apply non-linearity, project back up. Only adapter parameters are trained. Prefix tuning (Li & Liang, 2021) prepends learnable virtual tokens to the Key and Value matrices at every layer — the model sees these as additional context but they are just learned parameter vectors. Prompt tuning simply prepends soft tokens to the input embedding — the simplest form, effective only for large models (>10B parameters).

LoRA — Low-Rank Adaptation In-depth

Hu et al. (2021) observed that the weight matrices of large language models have low intrinsic rank — meaning their information content can be well-approximated with far fewer dimensions than the full d×d matrix. The hypothesis is that fine-tuning induces weight updates खW that also have low intrinsic rank: the task adaptation doesn’t require updating all d² values independently, because the update lies in a lower-dimensional subspace.

LoRA exploits this by decomposing खW = A·B, where A is d×r and B is r×d with r ≪ d (typically r=4, 8, or 16). The original weight W is frozen. Only A and B are trained. At the end, the adaptation is merged: W’ = W + (α/r)·A·B — a simple matrix addition — so inference has zero added latency. For a typical d=4096 matrix: full खW = 16.7M parameters; LoRA r=8: A+B = 2×4096×8 = 65,536 parameters (256× smaller).

RLHF — Reinforcement Learning from Human Feedback Core

The final alignment step that transforms a raw language model into a helpful assistant is RLHF — the technique behind ChatGPT, Claude, and Gemini. A pre-trained LLM knows how to predict text distributions but has no concept of “helpful” or “harmful”. RLHF shapes the model’s outputs toward human preferences through a three-stage pipeline: supervised fine-tuning on demonstrations, training a reward model from pairwise human judgements, and then using reinforcement learning to optimise the LLM to maximise the learned reward.

The RL stage uses Proximal Policy Optimisation (PPO) with a KL divergence constraint: the policy (the LLM being trained) cannot stray too far from the SFT baseline. Without this constraint, the model learns to “game” the reward model — generating outputs that score high on the RM but are not actually helpful (reward hacking). The KL term penalises outputs that diverge greatly from the pre-RLHF distribution, maintaining general capability while nudging toward preferred behaviour. Anthropic’s Constitutional AI (CAI) replaces human annotators in Stage 2 with AI-generated critiques, scaling the process to millions of preference pairs.

Generative vs Discriminative Models Core

Most models covered so far are discriminative: given an input x, predict a label y. They learn P(y|x) — the conditional distribution of outputs given inputs. A discriminative model draws a decision boundary in input space but has no model of what the input data actually looks like.

Generative models instead learn P(x) — the distribution of the data itself. Once you have a good model of P(x), you can sample from it: draw a new x that was never in the training set but looks like it could have been. This is qualitatively different from classification: you are not deciding which bucket an input belongs to, you are learning what valid inputs look like and manufacturing new ones. Applications span every domain: faces, voices, molecules, code, music, 3D shapes.

Variational Autoencoders (VAEs) In-depth

Kingma & Welling (2013) introduced the VAE as the first principled neural generative model. A regular autoencoder compresses input x to a latent code z, then reconstructs x — useful for compression but not generation, because the latent space has unpredictable gaps: points between training examples decode to garbage. The VAE fixes this by making the encoder probabilistic: instead of outputting a point z, it outputs a Gaussian distribution N(μ, σ²). The network is then trained to ensure this distribution stays close to a standard normal N(0, I) (via a KL divergence penalty), forcing all latent representations to occupy a continuous, organised neighbourhood around the origin.

The reparameterisation trick is the key engineering insight that makes training possible. You cannot backpropagate through a sampling operation z ~ N(μ, σ²) directly, because sampling is stochastic. The trick: write z = μ + σ·ε where ε ~ N(0,1) is sampled externally. Now μ and σ are deterministic outputs of the encoder, gradients can flow through them, and the stochasticity is isolated in ε which has no parameters to update. This simple algebraic rearrangement is what makes VAE training feasible.

Latent Space Properties Core

The VAE’s KL penalty forces all class clusters in latent space to overlap near the origin, eliminating gaps. This creates a structured, continuous latent space where every point decodes to a plausible output. You can smoothly interpolate between any two encoded examples: a straight line from the latent code of a young face to the code of an old face passes through intermediate codes that decode to faces of intermediate age. This is qualitatively impossible with regular autoencoders, which have large empty voids between training examples.

Generative Adversarial Networks (GANs) In-depth

Goodfellow et al. (2014) proposed a radically different generative approach: instead of maximising a likelihood, frame generation as a two-player game. A Generator G takes random noise z as input and outputs synthetic data G(z). A Discriminator D receives either a real data point x or a fake G(z) and must decide which is which. G is trained to fool D; D is trained to detect fakes. Neither player sees the other’s loss function directly — they only see each other’s outputs. The result of this adversarial dynamic, when it works, is a Generator that produces data indistinguishable from real training examples — because any distinguishable fake will be caught by D and penalised.

The theoretical optimum is a Nash equilibrium: G generates data exactly matching the true distribution P(x), and D can do no better than random guessing (P(real) = 0.5 for all inputs). In practice, reaching this equilibrium is notoriously difficult. The training is unstable, sensitive to hyperparameters, and prone to mode collapse — covered in the next section.

GAN Training Challenges & Mode Collapse Core

Mode collapse is the most notorious GAN failure mode. If G discovers one type of output that consistently fools D, it stops exploring the rest of the distribution and produces the same output (or a small set) regardless of the input noise. The discriminator adapts, G shifts to another single mode, and training cycles without covering the full data distribution. The GAN loss gives no signal that this is happening — the loss values look normal while G has abandoned 90% of the training distribution.

Training instability arises from the balance requirement: D and G must improve at similar rates. If D becomes too powerful early in training, G receives near-zero gradients and cannot improve (D correctly labels everything with high confidence, so the loss for G becomes flat). If G outpaces D early, D provides no meaningful feedback. The WGAN (Wasserstein GAN) addresses both problems by replacing the original loss with the Wasserstein distance, which provides smoother gradients and is more robust to D/G imbalance.

Diffusion Models In-depth

Ho et al. (2020) introduced Denoising Diffusion Probabilistic Models (DDPM), which now power Stable Diffusion, DALL-E 3, Midjourney, and Sora. The key idea is elegant: define a forward process that gradually corrupts data by adding Gaussian noise over T steps until the image becomes pure noise, then train a neural network to learn the reverse process — predicting what noise was added at each step, and thus denoising one step at a time. Generation is simply running the reverse process starting from random noise.

Unlike GANs, diffusion training is stable: the objective is a simple regression (predict the noise added at step t), there is no adversarial game to balance, and the model sees all noise levels during training. Unlike VAEs, there is no explicit latent space to constrain, so the generative quality is not limited by the bottleneck. The tradeoff is sampling speed: generating one image requires T=100–1000 denoising steps, each requiring a full forward pass through the network. Modern techniques like DDIM (deterministic sampling) and SDXL-Turbo (distillation) reduce this to 1–4 steps, largely eliminating the speed disadvantage.

Generative Model Comparison Core