Before any algorithm, you need to understand the three fundamental objects of linear algebra. Think of them as increasing levels of generality:

A 512-dimensional word embedding is just a vector with 512 numbers. When GPT processes a prompt of 100 tokens, it works with a 100 × 512 matrix — each row is one token's embedding. A dataset of 1,000 images (28×28 pixels) becomes a 1,000 × 784 matrix.

Why does matrix multiplication matter? Because a neural network layer is literally: output = W·x + b. Every forward pass through a model is a sequence of matrix multiplications punctuated by nonlinear activation functions. If you can visualise matrix multiplication, you can visualise what a neural network does.

Matrix multiplication works by taking the dot product of each row of the left matrix with each column of the right matrix. For element C[i,j]: you multiply corresponding entries of row i and column j, then sum them up.

Transpose flips rows and columns: if A is (m × n), then A^T is (n × m). You see it constantly — attention computes QK^T, meaning each query vector is dot-producted with each key vector. The transpose makes the shapes line up.

Hadamard (element-wise) product multiplies corresponding entries. Notation: A ⊙ B. It appears in LSTM gating (forget gate × cell state), masked attention, and dropout.

The dot product is the single most important operation in modern AI. It measures how aligned two vectors are — if they point in the same direction, the dot product is large and positive. If they're perpendicular, it's zero. If they point in opposite directions, it's large and negative.

Norms measure the "size" or "length" of a vector. They appear everywhere in regularisation — when you penalise large weights, you're literally penalising the norm of the weight vector.

Intuition first: Most vectors change direction when you multiply them by a matrix. But certain special vectors only get scaled — they keep pointing the same way. These are eigenvectors, and the scale factor is the eigenvalue. Think of them as the "natural axes" of a transformation.

SVD is the generalised eigendecomposition that works for any matrix — not just square ones. It decomposes any m×n matrix into three factors: rotation × scale × rotation.

Low-rank approximation: Keep only the top-k singular values (set the rest to zero). This gives the best rank-k approximation of the original matrix — provably optimal. This is how you compress a matrix while losing the least information.

A tensor is the N-dimensional generalisation of vectors (1D) and matrices (2D). In deep learning, almost all data lives in tensors. Understanding tensor shapes is the single most practical skill for debugging neural networks.

A derivative answers one question: "If I nudge this input by a tiny amount, how much does the output change?" In machine learning, that becomes: "If I nudge this weight by ε, how much does the loss change?" The answer tells you exactly how to adjust the weight to reduce the error.

Partial derivative: when a function has many inputs (like millions of weights), the partial derivative ∂L/∂wᵢ holds all other weights fixed and measures how L changes with respect to wᵢ alone. The gradient stacks all of these into one vector.

A neural network is a chain of composed functions: input → linear → activation → linear → activation → … → loss. The chain rule lets us compute how the loss depends on any weight, no matter how deep the network. It's the mathematical foundation of backpropagation.

When your function takes a vector in and produces a vector out, the derivative is no longer a single number or even a vector — it's a matrix. The Jacobian matrix captures how every output component changes with respect to every input component.

The Taylor series approximates any smooth function locally with a polynomial. The key insight: gradient descent is the first-order Taylor approximation. It assumes the loss surface is locally linear — which is only accurate when the learning rate is small. This is why large learning rates cause training to diverge: the linear approximation breaks down.

Computing gradients for a model with millions of parameters by hand is impossible. Automatic differentiation (autodiff) solves this: the computer applies the chain rule automatically by recording every operation during the forward pass, then replaying them in reverse to compute all gradients simultaneously.

In PyTorch, every tensor with requires_grad=True records operations in a computational graph. When you call .backward(), PyTorch walks the graph in reverse, applying the chain rule at each node. The gradients accumulate in each leaf tensor's .grad attribute. This is the "gradient tape" — record forward, replay backward.

A probability P(A) is a number between 0 and 1 that measures how likely an event A is. The sample space Ω is the set of all possible outcomes; an event is a subset of Ω. Three axioms define the entire theory: P(Ω) = 1, P(A) ≥ 0 for any event, and for disjoint events, P(A ∪ B) = P(A) + P(B).

When GPT-4 generates the next token, it computes a probability for every word in its vocabulary — typically 50,000+ values that must sum to 1. This is a categorical distribution over tokens. The softmax function transforms raw model outputs (logits) into this valid probability distribution. Every chapter in this course connects back to this fundamental mechanism.

A random variable X is a function that maps outcomes to numbers. Flip a coin → X = 1 (heads) or X = 0 (tails). A distribution describes the probabilities of all possible values of X. Distributions come in two flavours:

Expected value E[X] is the weighted average of all possible outcomes. "If I ran the experiment infinitely many times, what's the average?" For discrete X: E[X] = Σ xᵢ P(xᵢ). For continuous X: E[X] = ∫ x f(x) dx. In ML, the loss function is an expectation: L = E[ℓ(ŷ, y)] averaged over the data distribution.

Bayes' theorem is the most important equation in probabilistic reasoning. It tells you how to update your beliefs when you see new data. Start with a prior belief, observe evidence, compute a posterior belief. This is the core logic of Bayesian machine learning, spam filters, medical diagnosis, and A/B testing.

The Gaussian (normal) distribution is the most important continuous distribution in machine learning. It appears in weight initialisation, noise injection, latent spaces, diffusion models, and the Central Limit Theorem guarantees it emerges from the sum of many independent random variables.

Standardisation (z-score normalisation) transforms features to zero mean and unit variance: z = (x − μ) / σ. Almost every ML pipeline starts with this step — it ensures all features contribute equally and speeds up gradient descent convergence.

When an LLM computes P(next_token | context), it still needs to choose a token. The simplest approach — always pick the most probable token (greedy decoding) — produces repetitive, boring text. Instead, we sample from the distribution, with several control knobs.

Law of Large Numbers (LLN): As sample size grows, the sample mean converges to the true mean. This is why test set accuracy is a reliable estimate of true model performance — given enough test samples. It's also why mini-batch gradient is an unbiased estimator of the full-batch gradient: E[∇L_batch] = ∇L_full.

Central Limit Theorem (CLT): The sum (or average) of many independent random variables approaches a Gaussian distribution, regardless of the original distribution. This is why Gaussians appear everywhere — any quantity that arises from the aggregate of many small, independent effects will be approximately normal. Practical implication: larger mini-batches produce lower-variance gradient estimates (variance decreases as 1/n), giving smoother training.

Core idea: Given observed data D = {x₁, x₂, …, xₙ}, find the parameters θ that make the data most probable. The likelihood function L(θ) = P(D | θ) measures how well θ explains the data. MLE finds θ* = argmax L(θ).

In practice we maximise the log-likelihood instead: log L(θ) = Σ log P(xᵢ | θ). This is equivalent (log is monotonic) but easier — products become sums, which are numerically stable and analytically simpler. Minimising the negative log-likelihood = maximising the log-likelihood. This is your loss function.

MLE can overfit — it finds the parameters that best explain the training data, with no constraint on how "reasonable" those parameters are. MAP adds a prior belief: what kind of parameter values do you expect? The prior acts as a regulariser.

Regularisation is not a trick — it's a Bayesian prior on what you believe about your weights. When you add a weight decay term to the loss, you're implicitly saying "I believe weights should be small." The specific penalty shape reveals the prior distribution:

MLE and MAP both produce point estimates — a single "best" θ. Full Bayesian inference goes further: it computes the entire posterior distribution P(θ | D). Instead of saying "the best weight is 0.42", Bayesian inference says "the weight is probably between 0.35 and 0.49, with 95% probability." This is uncertainty quantification.

Two philosophical camps interpret probability differently. Frequentists see probability as long-run frequency of events; parameters are fixed but unknown. Bayesians treat parameters as random variables with distributions that represent degrees of belief. Most deep learning is frequentist MLE in practice, but Bayesian ideas (priors, posterior, uncertainty) are increasingly important.

Probabilistic Graphical Models (PGMs) represent complex joint distributions as graphs. Nodes are random variables, edges encode conditional dependencies. PGMs make the structure of a probabilistic model explicit — you can literally see which variables depend on which.

EM solves MLE when there are latent (hidden) variables that prevent direct optimisation. The classic example: fitting a Gaussian Mixture Model (GMM) when you don't know which Gaussian generated each data point. EM alternates between two steps until convergence:

E-step: Given current parameters, compute the expected value of the latent variables (soft assignments — "this point is 80% from Gaussian 1, 20% from Gaussian 2"). M-step: Given the soft assignments, update parameters (means, variances, mixing weights) to maximise the expected log-likelihood. Repeat until convergence. Guaranteed to increase likelihood at each step (but may converge to local optimum).

Entropy measures uncertainty, surprise, or unpredictability. Claude Shannon defined it in 1948: given a random variable X with possible outcomes, how much "information" does each outcome carry? The less likely an event, the more surprising (informative) it is when it occurs.

Cross-entropy H(P, Q) measures how many bits distribution Q needs to encode samples from distribution P. If Q is a perfect model of P (Q = P), then H(P, Q) = H(P) — the minimum. If Q is wrong, it wastes extra bits. Training a model minimises cross-entropy between the true labels and model predictions.

Kullback-Leibler divergence measures the "extra cost" of using distribution Q when the true distribution is P. It's the gap between cross-entropy and true entropy — the inefficiency penalty for using the wrong model.

Mutual information I(X;Y) measures how much knowing Y reduces your uncertainty about X. Unlike correlation, MI captures any statistical relationship — not just linear ones.

Shannon's source coding theorem (1948): the optimal average code length for messages from a source with entropy H is at least H bits. You cannot compress below the entropy limit. This means a better probability model = better compression. Language models are, in a deep sense, compressors — a model with low perplexity assigns high probability to real text, which means it can encode that text in fewer bits.

Minimum Description Length (MDL) formalises model selection: the best model is the one that gives the shortest total description (model complexity + data encoded by model). This connects directly to the bias-variance tradeoff: a too-complex model has a long description; a too-simple model encodes data poorly. Bits-back coding shows VAEs are theoretically equivalent to lossless compression schemes — the ELBO loss is the expected code length.

The loss landscape is a high-dimensional surface — we want the lowest point. The gradient ∇L tells us which direction is "uphill" at our current position. Gradient descent takes a step in the opposite direction: downhill. Repeat until the gradient is near zero (we've reached a minimum or saddle point).

Computing the gradient over the entire dataset each step is accurate but slow. The solution: estimate the gradient from a random subset (mini-batch). The noise from subsampling is not a bug — it's a feature.

Vanilla SGD has two problems: it oscillates in steep dimensions and moves slowly in flat ones. Momentum fixes the first. Adaptive learning rates fix the second. Adam combines both — and it's the default for almost all deep learning.

A constant learning rate is rarely optimal. The standard practice: warm up from a very small LR, then gradually decay it. Warmup prevents early training instability (large initial gradients). Decay allows fine-grained convergence near the end.

A convex function has a single global minimum — gradient descent is guaranteed to find it. Linear regression (MSE) and logistic regression have convex losses. But deep neural networks are non-convex — the loss surface is filled with local minima, saddle points, and flat plateaus.

Lagrange multipliers solve the problem of optimising f(x) subject to constraints g(x) = 0. The key insight: at the optimal point, the gradient of f must be parallel to the gradient of g. Introduce multiplier λ and solve ∇f = λ∇g. The KKT conditions extend this to inequality constraints (g(x) ≤ 0).

Where constrained optimisation appears in AI: Support Vector Machines maximise the classification margin subject to correct classification constraints (a quadratic program). RLHF constrains the new policy to stay close to the reference policy (KL constraint). Safety-constrained RL limits reward optimisation to stay within safety bounds. Optimal transport problems are constrained linear programs.

Newton's method uses the Hessian (second derivative matrix) to account for curvature. Instead of following the gradient blindly, it finds the optimal step size by solving ∇²L · δ = −∇L. In theory, it converges in far fewer steps. In practice, the Hessian is n×n for n parameters — for GPT-3 with 175B parameters, that's 175B × 175B entries. Completely infeasible.

Quasi-Newton methods (L-BFGS) approximate the Hessian from gradient history, avoiding the O(n²) cost. They work well for small models (up to ~millions of parameters) and are sometimes used in fine-tuning. The natural gradient uses the Fisher information matrix as a metric — accounting for the geometry of probability distributions rather than Euclidean parameter space. K-FAC and Shampoo are practical approximations used in some large-scale training runs.

A graph G = (V, E) consists of vertices (nodes) V and edges (links) E. Edges can be directed (A → B) or undirected (A — B), weighted or unweighted. Graphs are represented in code as adjacency matrices (dense, good for small graphs) or adjacency lists (sparse, good for large graphs).

Standard neural networks expect fixed-size inputs (vectors, grids). Graphs have variable numbers of nodes and edges with no fixed ordering. GNNs solve this with the message passing framework: each node updates its representation by aggregating information from its neighbours.

Knowledge graphs store facts as (subject, predicate, object) triples. For example: (Paris, isCapitalOf, France), (France, isPartOf, Europe). Major KGs include Wikidata (100M+ entities), Google Knowledge Graph, and domain-specific KGs for medicine, law, and science.

KG embeddings like TransE learn vector representations where subject + relation ≈ object in embedding space. This enables link prediction ("if we know Paris is in France and France is in Europe, can we infer Paris is in Europe?"). RAG with KGs retrieves relevant triples and injects them into LLM prompts — grounding generation in verified facts.

Limitations: KGs are brittle, require manual curation, can't handle uncertainty or nuance well, and struggle with temporal facts. Hybrid approaches (KG + LLM) are an active research frontier.

Why symbolic AI died: The search space for reasoning grows exponentially with problem size — the combinatorial explosion. A game tree for Go has ~10¹⁷⁰ positions. Brute-force search is impossible. This is why statistical (learning-based) methods replaced rule-based AI.

For ML practitioners, Big-O notation matters most when reasoning about scaling. Standard Transformer attention is O(n²) in sequence length — the main bottleneck for long-context models. Linear attention variants reduce this to O(n) but often sacrifice quality.