What Is Machine Learning? Core

In 1959, Arthur Samuel coined the term machine learning while building a self-improving checkers program, defining it as "the ability to learn without being explicitly programmed." The most precise formulation came from Tom Mitchell in 1997:

Concrete example — applying Mitchell's definition to a spam filter:

As the filter processes more labelled emails (more E), its accuracy on new mail (P at T) improves — that's the entire definition in one sentence.

The critical insight is the inversion of the programming paradigm. In traditional software you write rules ("if subject contains 'FREE MONEY' → spam"). In ML, you provide examples and the algorithm discovers the rules automatically — rules often far too complex and numerous for any human to encode by hand.

Supervised Learning In-depth

Supervised learning is the most widely deployed form of ML. The word "supervised" means a teacher is present: you provide labelled training examples — pairs of (input X, correct answer y) — and the algorithm learns a mapping f: X → y that generalises to new, unseen inputs. Think of a student studying with an answer key: they infer underlying rules from many (question, answer) pairs, then apply those rules to questions they've never seen.

Tasks split into two types by the nature of y:

The choice of output type determines the loss function, output layer, and evaluation metric.

The learning signal comes from comparing the model's prediction ŷ to the true label y via a loss function. Gradient descent nudges the model's parameters in the direction that reduces the loss. Repeat this across millions of examples and the model converges to a good approximation of f. Supervised learning's key limitation is label dependency: human-annotated examples are expensive, slow, and often scarce in specialised domains.

Unsupervised Learning Core

Unsupervised learning removes the teacher entirely. You provide raw inputs X — no answer key, no rewards. The algorithm must discover hidden structure on its own. Think of giving someone a pile of unlabelled photographs and asking them to organise it: they'll naturally group similar images together even without being told what "similar" means.

Three major tasks fall under unsupervised learning:

The unsupervised challenge is evaluation: without ground-truth labels, how do you know if a clustering is good? Metrics like silhouette score (cohesion vs separation) and reconstruction error (for autoencoders) provide proxies, but ultimately require domain-expert judgment. Despite this, unsupervised learning is invaluable — it can operate on the vast stores of unlabelled data that exist in every organisation.

Semi-Supervised & Self-Supervised Learning Core

Between the extremes of "all data labelled" (supervised) and "no data labelled" (unsupervised) lies a practical middle ground that powers most of modern AI.

Semi-supervised learning combines a small labelled set with a large unlabelled set. Even without labels, the geometric structure of the data (clusters, manifolds) constrains which labels are plausible: two nearby points probably share a label. Algorithms like label propagation, self-training, and co-training exploit this structure to achieve strong performance even when labels cover only 1–5% of data — critical in medical imaging where expert annotation is expensive.

Self-supervised learning is the most important paradigm of the past decade. It creates a training signal directly from raw data, eliminating human annotation entirely. The core trick: mask or corrupt part of the input, then train the model to reconstruct the missing part.

Reinforcement Learning Introductory

Reinforcement learning (RL) is the third major paradigm. An agent takes actions in an environment and receives numerical rewards or penalties based on those actions. No labelled examples exist — only a reward signal over time. The agent must learn, through trial and error, which sequences of actions lead to the highest cumulative reward.

The central challenge is the exploration-exploitation dilemma: should the agent exploit what it already knows works, or explore new actions that might be better? Too much exploitation → stuck in a local optimum. Too much exploration → never commits to a strategy.

📌 Full RL coverage — Q-learning, policy gradients, PPO, actor-critic — is in Domain 7: Reinforcement Learning. Here we establish only where RL fits in the ML landscape.

The Standard ML Pipeline In-depth

Every ML project — from a weekend Kaggle competition to a production system at Google — follows the same general pipeline. Understanding this sequence prevents the most common failure modes: models trained on wrong data, evaluated on contaminated test sets, or deployed without monitoring.

When to Use Which Paradigm Core

The most common mistake early practitioners make is defaulting to neural networks or the most complex algorithm available. The right question is always: what data do I have, and what do I need to output?

Linear Regression In-depth

The goal of regression is to predict a continuous numerical output from one or more inputs. The simplest version — simple linear regression — assumes the relationship between input x and output y is a straight line. We fit that line to the data by finding the optimal slope (weight) and intercept (bias).

Our running example: predicting a house's sale price from its size in square footage. We assume price increases roughly linearly with size, with some noise from other unmodelled factors (location, condition, year built). The model makes predictions using:

The model has just two parameters — w (how much price increases per unit of size) and b (the base price for a zero-size house, which anchors the line). Training means finding the values of w and b that make the line fit the data as closely as possible.

The Cost Function (MSE) In-depth

To fit a line we need to measure how wrong our current w and b are. The residual for one prediction is the gap between the actual value and the predicted value: residual = y − ŷ. If we just summed residuals, positive and negative errors would cancel. Instead we square them.

Mean Squared Error (MSE) averages the squared residuals across all n training examples. Squaring does two things: it makes all errors positive, and it penalises large errors far more than small ones (a 2× larger error contributes 4× the cost). MSE is also smooth everywhere — which means we can take its derivative and use gradient descent.

A worked example — 4 predictions: actual = [200, 250, 300, 350], predicted = [190, 270, 290, 380]. Residuals: [10, −20, 10, −30]. Squared: [100, 400, 100, 900]. MSE = (100+400+100+900)/4 = 375. RMSE = √375 ≈ $19.4k — that's the typical prediction error in the same units as house price.

Gradient Descent for Regression In-depth

We have a cost function J(w, b) — now we need to minimise it. Two approaches exist. The Normal Equation solves for the optimal weights analytically in one shot. Gradient Descent takes many small iterative steps, using the derivative of J to determine which direction to move.

The intuition: the gradient (derivative of J w.r.t. w) tells you the slope of the cost bowl at your current position. Subtract a fraction (α) of that slope from w to move toward the minimum. If the gradient is positive, w is too large — decrease it. If negative, w is too small — increase it. The learning rate α controls how large each step is — too large and you overshoot; too small and training takes forever.

Multiple Linear Regression Core

Real predictions need multiple inputs. House price depends not just on square footage, but on bedrooms, bathrooms, location score, age, and dozens of other features. Multiple linear regression extends the model to handle any number of features:

With multiple features, feature scaling becomes critical. If x₁ (square footage) ranges from 500–3000 and x₂ (bedrooms) ranges from 1–6, the gradient for w₁ is tiny compared to w₂. This creates an elongated cost bowl where gradient descent zig-zags inefficiently. Scaling all features to a comparable range makes the bowl round and descent fast.

Polynomial Regression Core

What if the relationship between x and y is curved, not linear? We can handle this without leaving the linear regression framework — by creating polynomial features: x², x³, and so on. We then fit a standard linear regression to these expanded features. The model is still linear in its parameters (w₀, w₁, w₂...) even though the fit is a curve in the original x space.

The risk: higher polynomial degree → tighter fit on training data → worse generalisation to new data. A degree-12 polynomial can pass through every training point perfectly (zero training error) while oscillating wildly between them — it has memorised the data rather than learned the pattern.

Regularisation: Ridge, Lasso, ElasticNet In-depth

Overfitting happens because the model is too complex for the data — it fits noise rather than signal. Regularisation is the standard fix: add a penalty term to the loss function that discourages large weights. Smaller weights = simpler model = less overfitting. The penalty strength is controlled by a hyperparameter λ (lambda).

The Bias-Variance Tradeoff In-depth

Why does any ML model fail? The expected prediction error can always be decomposed into three components. Understanding them is essential for diagnosing model problems and choosing the right fix.

Logistic Regression In-Depth

Despite its name, logistic regression is a classifier, not a regressor. It predicts P(y=1 | x) — the probability that an input belongs to class 1. The trick is the sigmoid function, which squashes any real number from −∞ to +∞ into the range (0, 1), making the output interpretable as a probability.

Why not use ordinary linear regression for classification? A linear model can predict values well below 0 or well above 1, which are meaningless as probabilities, and the resulting loss surface is non-convex with MSE — gradient descent is not guaranteed to find the global minimum. The sigmoid + binary cross-entropy combination fixes both problems.

Decision rule: predict class 1 if P(y=1|x) ≥ 0.5, else class 0. The threshold 0.5 is the default but can be tuned — lower it to increase recall (catch more positives), raise it to increase precision (fewer false alarms).

# Logistic Regression — sklearn walkthrough
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = make_classification(n_samples=200, n_features=2, n_redundant=0, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

model = LogisticRegression()
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]  # probability of class 1

print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}")
print(f"Probabilities (first 5): {y_prob[:5].round(3)}")

Sigmoid & Decision Boundaries Core

Logistic regression produces linear decision boundaries — a straight line in 2D, a hyperplane in higher dimensions. This is fast and interpretable, but it fails when the true boundary between classes is curved or complex.

Three escape routes from linearity: polynomial features (add x², x·y, ... as new inputs — same model, richer boundary), the kernel trick (implicitly maps to a very high-dimensional space, used in SVMs), or simply switch to a non-linear model (decision trees, neural networks). The softmax function generalises the sigmoid to K classes — each class gets a score zₖ, and softmax converts all K scores into a probability distribution that sums to 1.

Multiclass Classification Core

When there are K > 2 classes, three strategies extend binary classification:

K-Nearest Neighbours In-Depth

KNN is a lazy learner — there is no training phase. The algorithm memorises all training data and performs all computation at prediction time. This makes training instant but prediction potentially slow on large datasets (O(n·d) per query — must compute distance to every training point).

Choosing K: K=1 memorises training data perfectly (zero training error) but is extremely sensitive to noise and outliers — overfitting. K=n (all points) just predicts the majority class — underfitting. The optimal K is found via cross-validation, typically in the range 3–15. Odd K values avoid ties in binary classification.

Naive Bayes Core

Naive Bayes is a probabilistic classifier built on Bayes' theorem. The "naive" assumption is that all features are conditionally independent given the class — in practice this is almost never true, yet the algorithm performs remarkably well, especially on text classification tasks.

Why does it work despite the false assumption? The discriminative signal — the difference between class probabilities — is often large enough that the independence error doesn't flip the argmax. In high-dimensional sparse data (like text), there's also very little co-occurrence signal to exploit anyway.

Worked example — spam filter: Given 5 training emails below, compute the Naive Bayes decision for the new message "free click prize":

# P(spam) = 3/5 = 0.6  |  P(ham) = 2/5 = 0.4
# P(free|spam)=3/3=1.0  P(click|spam)=2/3≈0.67
# P(free|ham)=0/2=0.0   P(click|ham)=0/2=0.0  (add Laplace smoothing: +1)
# With smoothing: P(free|ham)≈0.17  P(click|ham)≈0.17

# Score(spam) = 0.6 × 1.0 × 0.67 ≈ 0.40
# Score(ham)  = 0.4 × 0.17 × 0.17 ≈ 0.012
prediction = "spam"  # argmax wins

Algorithm Comparison Core

Decision Trees In-Depth

Decision trees mirror how humans naturally make decisions: a series of if/else questions on features, arriving at a prediction at the leaves. Each internal node tests one feature, each branch is an outcome of that test, and each leaf holds the class label or numeric prediction. The CART algorithm (Classification And Regression Trees) is the sklearn default and can handle both classification and regression, numerical and categorical features, and naturally captures non-linear relationships.

# Decision Tree — sklearn
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.datasets import load_iris

iris = load_iris()
X, y = iris.data, iris.target

tree = DecisionTreeClassifier(max_depth=3, criterion='gini', random_state=42)
tree.fit(X, y)

# Print tree structure as text
print(export_text(tree, feature_names=list(iris.feature_names)))

# Feature importances
for name, imp in zip(iris.feature_names, tree.feature_importances_):
    print(f"{name}: {imp:.3f}")

Splitting Criteria In-Depth

At each node, CART exhaustively searches all features and all thresholds to find the split that maximises purity in the child nodes. Three measures of impurity are used in practice:

Worked example — which split wins? Parent node: 30 samples (20 blue, 10 red). Gini = 1 − (20/30)² − (10/30)² = 0.444.

Overfitting & Pruning Core

An unconstrained decision tree grows until every leaf contains a single sample — perfect training accuracy, terrible test accuracy. This is the classic overfit problem. Two families of solutions exist: pre-pruning (stop growing early) and post-pruning (grow full, then cut).

Bagging & Random Forests In-Depth

A single decision tree has high variance — small perturbations to the training data produce completely different trees. Bagging (Bootstrap Aggregating) solves this by training N trees on N bootstrapped datasets (sampled with replacement) and averaging their predictions. The individual errors are largely independent, so they cancel out.

Random Forest = Bagging + random feature subset at each split. The critical insight: if you use all features at each node, every tree will pick the same dominant feature at the root, making all trees highly correlated — they'd all make the same mistakes. By restricting each node to a random subset of √d features, the trees become decorrelated, and averaging truly independent errors dramatically reduces variance. About 37% of training samples are never used in each tree — this out-of-bag (OOB) set is a free validation estimate.

Gradient Boosting In-Depth

Gradient Boosting is fundamentally different from bagging. Trees are built sequentially — each new tree specifically targets the mistakes of the current ensemble. The key insight (Friedman, 1999): fit each new tree to the negative gradient of the loss function with respect to the current predictions. This is gradient descent — but in function space rather than parameter space.

The learning rate (shrinkage) controls how much each tree contributes: F(x) += learning_rate × T(x). A small learning rate requires more trees but generalises better. AdaBoost (the precursor) reweighted misclassified examples; modern Gradient Boosting is more general and works with any differentiable loss function.

XGBoost, LightGBM & CatBoost Core

Vanilla gradient boosting is slow — it re-evaluates every possible split at every node. The three production-grade libraries solve this with fundamentally different approaches, making gradient boosting practical on millions of rows.

# XGBoost — binary classification example
import xgboost as xgb
from sklearn.datasets   import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics    import roc_auc_score

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

model = xgb.XGBClassifier(
    n_estimators=200,          # number of trees
    learning_rate=0.1,        # shrinkage — prevents overfitting
    max_depth=4,               # tree depth
    subsample=0.8,             # row sampling per tree
    colsample_bytree=0.8,      # feature sampling per tree
    eval_metric='logloss',
    random_state=42,
)

model.fit(
    X_train, y_train,
    eval_set=[(X_test, y_test)],
    early_stopping_rounds=20,  # stop if no improvement for 20 rounds
    verbose=False,
)

y_prob = model.predict_proba(X_test)[:, 1]   # probability of positive class
print(f"AUC-ROC: {roc_auc_score(y_test, y_prob):.4f}")

Ensemble Comparison Core

Maximum Margin Classifier In-Depth

Many hyperplanes can separate two linearly-separable classes. SVMs choose the one that maximises the margin — the perpendicular distance between the boundary and the nearest training points on each side. A wider margin means more room for error on unseen data, giving better generalisation.

The decision boundary is the hyperplane w·x + b = 0. The two margin planes are w·x + b = +1 and w·x + b = −1. The margin width is 2/||w||, so maximising the margin is equivalent to minimising ||w||². This is a constrained quadratic optimisation problem — convex, so it has a unique global solution.

Support Vectors Core

Support vectors are the training points that lie exactly on the margin boundaries (w·x + b = ±1). They are the only points that matter: if you removed every non-support vector from the training set and retrained, you'd get the exact same model. This is a profound property — the decision boundary is defined by a sparse subset of the data.

The SVM optimisation is solved in its dual formulation using Lagrange multipliers — one multiplier αᵢ per training point. Non-support vectors have αᵢ = 0, so they contribute nothing. The prediction is: ŷ = sign(Σ αᵢ yᵢ K(xᵢ, x) + b), summing only over support vectors.

Soft Margin SVM — The C Parameter In-Depth

The hard-margin SVM requires perfect linear separability — it fails completely if even one point is on the wrong side. Real data is always noisy and often overlapping. Soft-margin SVM introduces slack variables ξᵢ ≥ 0 that allow points to violate the margin. The C hyperparameter controls how heavily violations are penalised.

The Kernel Trick In-Depth

A linear SVM can only draw straight boundaries. But many real datasets are not linearly separable in their original feature space. One solution is to manually engineer polynomial or interaction features — but for high-dimensional data this is computationally prohibitive. The kernel trick solves this elegantly: it allows the SVM to operate in an implicitly high-dimensional feature space without ever computing the transformation explicitly.

The key insight: the SVM dual formulation only requires computing dot products between feature vectors — never the vectors themselves. A kernel function K(xᵢ, xⱼ) computes the dot product that would result from a high-dimensional mapping φ, without actually applying φ. For the RBF kernel, this implicit feature space is infinite-dimensional.

Kernel Types Core

# SVM with RBF kernel + GridSearch
  from sklearn.svm            import SVC
  from sklearn.preprocessing  import StandardScaler
  from sklearn.pipeline       import Pipeline
  from sklearn.model_selection import GridSearchCV

  # SVM requires feature scaling — always use a Pipeline!
  svm_pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('svm',    SVC(kernel='rbf', probability=True)),
  ])

  param_grid = {
    'svm__C':     [0.1, 1, 10, 100],
    'svm__gamma': ['scale', 'auto', 0.001, 0.01, 0.1],
  }

  grid_search = GridSearchCV(
    svm_pipeline, param_grid,
    cv=5, scoring='accuracy', n_jobs=-1
  )
  grid_search.fit(X_train, y_train)

  print(f"Best params:   {grid_search.best_params_}")
  print(f"Best CV score: {grid_search.best_score_:.4f}")

SVR — Support Vector Regression Reference

Support Vector Regression (SVR) extends the SVM idea to continuous outputs. Instead of maximising a margin between classes, SVR fits a tube of width 2ε around the predictions. Points inside the tube incur zero loss — the model doesn't care about small errors. Only points outside the tube (the support vectors) contribute to the optimisation. This gives SVR a natural insensitivity to small noise.

SVR inherits all the power of kernels — with an RBF kernel, SVR can fit complex non-linear regression surfaces while remaining robust to outliers (thanks to the ε-insensitive loss). The tradeoff: SVR can be slow on large datasets (O(n²) to O(n³) training) and requires careful tuning of ε, C, and the kernel parameters.

K-Means Clustering In-Depth

K-Means partitions n points into K clusters by minimising within-cluster variance. The Lloyd algorithm iterates two steps — assignment and update — until centroids stop moving.

K-Means Issues & Choosing K Core

DBSCAN In-Depth

Density-Based Spatial Clustering of Applications with Noise. Clusters are dense regions of points separated by low-density space. Unlike K-Means, DBSCAN does not require specifying K — it discovers the number of clusters from the data.

Hierarchical Clustering Core

Hierarchical clustering builds a full tree (dendrogram) of cluster merges. No need to specify K upfront — pick any cut height and read off the clusters.

Linkage criteria — how "distance between clusters" is measured:

PCA — Principal Component Analysis In-Depth

PCA reduces dimensionality by finding the directions (principal components) of maximum variance. Each PC is orthogonal to all others — no redundancy. The technique is the eigendecomposition of the covariance matrix (equivalently, SVD of the data matrix).

When to use PCA: high-dimensional data (>50 features), visualisation (reduce to 2D/3D), noise reduction, or to remove multicollinearity before regression. Always scale features first (StandardScaler) — PCA is variance-based and sensitive to scale.

t-SNE & UMAP Core

PCA is linear — it cannot preserve complex non-linear structure. t-SNE and UMAP are non-linear manifold methods that excel at visual exploration of high-dimensional data.

Confusion Matrix In-Depth

The confusion matrix is the foundation of all classification metrics. For binary classification it is a 2×2 table of outcomes that decomposes prediction errors into two fundamentally different types — false positives and false negatives — which carry very different costs depending on the domain.

Classification Metrics In-Depth

ROC Curve & AUC In-Depth

The ROC curve (Receiver Operating Characteristic) plots True Positive Rate (Recall) against False Positive Rate at every possible decision threshold. AUC (Area Under the Curve) collapses this to a single number: the probability that the model ranks a random positive example higher than a random negative example.

Regression Metrics Core

For regression problems, evaluation metrics measure how far predictions deviate from true values. The choice of metric determines what kind of errors your model is penalised for — which in turn shapes training and interpretation.

Cross-Validation In-Depth

A single train/test split gives a high-variance performance estimate — one unlucky split can make a great model look bad. Cross-validation rotates the validation set across the full dataset, giving a robust, low-variance estimate of generalisation performance.

Data Leakage In-Depth

Data leakage is the most common and dangerous ML mistake in production. It occurs when information from outside the training set influences the model — artificially inflating validation performance so the model appears to work when it does not.