What Is NLP? Core

Natural Language Processing (NLP) is the branch of AI that enables computers to understand, process, and generate human language. This sounds straightforward until you confront what language actually is: an ambiguous, context-dependent, culturally-loaded communication system that humans have spent millions of years evolving. No sentence carries a single, unambiguous meaning independent of context. A machine must learn to handle every level of this ambiguity simultaneously.

Language ambiguity operates at four distinct levels, each building on the one below. Lexical ambiguity arises when a single word has multiple meanings — "bank" can mean a financial institution or the edge of a river. Syntactic ambiguity occurs when sentence structure is unclear — "I saw the man with the telescope" leaves open whether the speaker or the man possesses the telescope. Semantic ambiguity involves phrases whose meaning is unclear even with structure resolved — "Can you pass the salt?" is literally a question about capability but functions as a request. Pragmatic ambiguity is the deepest: "It's cold in here" is an observation that functions as a request to close the window, but only if you already know the conversational norms.

The history of NLP traces a clear arc: Rule-based systems (1950s–1980s) used hand-crafted grammars and dictionaries — brittle and language-specific. Statistical NLP (1990s–2000s) replaced rules with probabilities learned from corpora. Neural NLP (2013–2017) used word embeddings (Word2Vec, GloVe) and RNNs to learn representations directly from data. Transformer-era (2018–present) introduced BERT, GPT, and their successors — models that learn language representations of staggering generality from massive corpora, making almost all previous approaches obsolete.

Text Preprocessing In-depth

Before any model can process text, it must be transformed from raw characters into a form the model understands. Classical NLP pipelines involve a series of hand-engineered preprocessing steps, each reducing noise and normalising vocabulary. We trace each step using: "The Quick Brown Foxes are JUMPING over lazy dogs! They've been running."

Step 1 — Lowercasing. Convert all characters to lowercase. "The" and "the" are the same word — keeping both wastes vocabulary slots. This alone can reduce vocabulary size by 10–30% for English text.

Step 2 — Punctuation & special character removal. Strip characters that carry no lexical meaning for bag-of-words models. Important caveat: not always appropriate — punctuation carries meaning in some contexts (U.S.A, 3.14, emoticons, code). Remove selectively based on the task.

Step 3 — Tokenisation. Split text into meaningful units (tokens). The naïve approach is whitespace splitting. Better approaches handle contractions ("they've" → ["they", "'ve"]) and punctuation. Chapter 5.2 covers subword tokenisation for neural models in depth.

Step 4 — Stopword removal. Remove high-frequency words ("the", "is", "a") that carry little semantic weight in bag-of-words models. Critical warning: never remove stopwords for neural models or sequence tasks — position and function words are often critical to meaning ("not" changes everything).

Step 5 — Stemming. Reduce words to root form by stripping suffixes using heuristic rules. Porter Stemmer: "jumping" → "jump", "foxes" → "fox". Fast but imprecise — "university" → "univers". Two words with the same stem may not share meaning.

Step 6 — Lemmatisation. Morphologically reduce words to their dictionary form (lemma) using linguistic knowledge. "better" → "good", "ran" → "run", "foxes" → "fox". More accurate than stemming but requires WordNet. "Saw" → "see" (verb) or "saw" (noun) depending on POS tag.

Step 7 — Text normalisation. Expand contractions ("they've" → "they have"), normalise Unicode, standardise numbers, handle abbreviations and acronyms.

Linguistic Structure Core

Even if you never build a classical NLP pipeline, understanding key linguistic concepts will help you reason about what language models are learning, diagnose failure modes, and work effectively with hybrid systems.

Part-of-Speech (POS) Tagging. Label each word with its grammatical role: Noun (NN), Verb (VB), Adjective (JJ), Adverb (RB), Determiner (DT), Preposition (IN). "The cat sat on the mat" → [DT, NN, VBD, IN, DT, NN]. POS tags disambiguate words with multiple roles — "run" is a verb in "I run" but a noun in "a home run". Used in: NER, information extraction, classical feature engineering.

Named Entity Recognition (NER). Identify and classify spans of text as named entities: PERSON, ORG, GPE (geopolitical entity), DATE, MONEY. "Apple's Tim Cook announced on Monday that sales exceeded $100B" → [Apple: ORG], [Tim Cook: PERSON], [Monday: DATE], [$100B: MONEY]. Modern BERT-based NER achieves near-human F1 scores on standard benchmarks.

Dependency Parsing. Identify grammatical relationships between words — subject, object, modifier, etc. "The cat chased the mouse" → "cat" is the nominal subject (nsubj) of "chased"; "mouse" is the direct object (dobj). Dependency parses are directed graphs enabling extraction of who did what to whom.

Coreference Resolution. Determine which mentions refer to the same entity. "When Mary arrived, she said she was tired" — both "she" instances refer to Mary. Crucial for coherent understanding across sentences. SpanBERT achieves state-of-the-art by jointly scoring mention pairs.

Traditional NLP Methods Reference

Before neural networks dominated NLP, three core representation methods powered almost every text application. They remain useful for lightweight tasks, interpretable systems, and benchmarking. Understanding them explains why neural embeddings were such a dramatic improvement.

Bag of Words (BoW) represents a document as a vector of word counts, completely discarding word order. "The cat sat" and "sat cat the" produce identical BoW vectors. Works well for document classification and spam filtering because topic is often determined by which words appear, not their order. Critical weakness: "not good" and "good" look nearly identical.

TF-IDF improves on raw counts by weighting words by how rare they are across the corpus. A word that appears frequently in one document but rarely elsewhere is a likely topic word. "The" appears in every document — TF-IDF assigns it near-zero weight. "Convolutional" in an AI paper is rare and gets high weight. Still discards order, but much more informative than raw counts.

n-grams partially recover word order by treating sequences of n adjacent words as features. Bigrams of "the cat sat": ["the cat", "cat sat"]. Captures local context but explodes vocabulary size exponentially with n. Word2Vec and subsequent neural embeddings made n-gram language models obsolete for most applications.

NLP Task Taxonomy Core

NLP encompasses a wide range of tasks grouped by the type of output they produce. Understanding this taxonomy helps you choose the right architecture (encoder-only, decoder-only, encoder-decoder), the right loss function, and the right evaluation metric for any given problem.

Why Tokenise? Core

Neural language models operate on numbers, not text. Every word, character, or subword must be mapped to an integer ID from a fixed vocabulary before it can be fed into the model. Tokenisation is this mapping — it converts a raw string into a sequence of integers, each representing a "token" from the vocabulary. The choice of what constitutes a token has profound consequences for the model's capabilities.

The vocabulary dilemma has three corners. Word-level tokenisation uses whole words as tokens — intuitive, but English has 170,000+ words and with proper nouns, compounds, and morphological variants, the vocabulary explodes into millions. Words not seen during training become [UNK] (unknown) — the out-of-vocabulary problem. Character-level tokenisation uses individual characters — tiny vocabulary of ~128 ASCII characters, but sequences become very long. "Hello world" is 11 characters; a document of 1,000 words becomes ~6,000 characters. Attention's O(n²) complexity makes this expensive. Subword tokenisation is the sweet spot: split common words into whole tokens, rare or unknown words into subword pieces. "unhappiness" → ["un", "##happy", "##ness"] — known pieces, no OOV, reasonable sequence length.

Modern LLMs universally use subword tokenisation with vocabularies of 32,000–100,000 tokens. GPT-4 uses 100,277 tokens; LLaMA-3 uses 128,256. The vocabulary is fixed at training time and cannot be changed without retraining the model — making the tokeniser one of the most consequential design decisions in LLM development.

Word & Character Tokenisers Core

Understanding why pure word and character tokenisers were abandoned helps clarify the design goals of modern subword tokenisers. Both extremes have fundamental problems that subword methods resolve.

Word tokenisation splits on whitespace and punctuation. Problems pile up quickly. Contractions: "don't" — is that one token or two ("do", "n't")? Hyphenated compounds: "state-of-the-art" — one or four? Morphological variants: "run", "running", "ran", "runs" require four separate vocabulary entries, even though they share meaning. Proper nouns, technical terms, and misspellings not seen during training become [UNK] — the model sees a blank where information should be. The English vocabulary alone exceeds 170,000 words; with all languages and domains, a truly universal word vocabulary would require millions of entries.

Character tokenisation has no OOV problem — the alphabet is fixed. But it fragments language into meaningless units from the model's perspective. The word "hello" becomes 5 separate tokens [h][e][l][l][o]. The model must learn from scratch that these 5 tokens together form a word unit — it cannot start with the useful prior that words are meaningful. More critically, sequence length explodes. A 1,000-word essay becomes ~6,000 character tokens. Transformer attention is O(n²) in sequence length — doubling the sequence length quadruples the compute cost. In practice, character-level models were impractical at scale.

Byte-Pair Encoding (BPE) In-depth

Byte-Pair Encoding was introduced for NLP by Sennrich et al. (2016) as a data compression algorithm adapted for subword vocabulary construction. It is used by GPT-2, GPT-3, GPT-4, LLaMA, Mistral, Falcon, and most modern decoder-based language models. The key insight is elegant: let the training data decide what the vocabulary tokens should be, by iteratively merging the most frequently co-occurring pairs.

The BPE algorithm is a simple loop. Initialise with a character vocabulary (or byte vocabulary for byte-level BPE). Count all adjacent token pairs across the entire training corpus. Merge the most frequent pair into a new single token and add it to the vocabulary. Repeat until the target vocabulary size is reached. The result: common words like "the", "and", "is" become single tokens; common morphological patterns like "-ing", "-tion", "un-" become tokens; rare words are split into recognisable subword pieces.

WordPiece Core

WordPiece is the tokenisation algorithm used by BERT, DistilBERT, ALBERT, and multilingual BERT (mBERT). It is mechanically similar to BPE but uses a different merge criterion: rather than merging the most frequent pair, it merges the pair that most increases the likelihood of the corpus under a language model. In practice, the merge score is: score(A, B) = freq(A+B) / (freq(A) × freq(B)). This prefers pairs where the joint occurrence is disproportionately high relative to how often each appears alone — capturing meaningful linguistic units rather than just common bigrams.

WordPiece uses a distinctive notation: a ## prefix marks a continuation subword — a piece that is attached to the preceding token rather than starting a new word. "playing" → ["play", "##ing"]. The "play" token has no prefix (it starts a word); "##ing" is always a suffix, never a standalone word. This makes the tokenisation reversible and unambiguous: joining tokens without spaces and stripping ## gives back the original word.

The standard BERT vocabulary contains 30,522 tokens. Unknown characters that cannot be represented by any combination of vocabulary tokens are mapped to [UNK] — rare for English but can occur with unusual Unicode characters. The vocabulary also includes special tokens: [CLS] (classification, prepended to all inputs), [SEP] (separator, marks sentence boundaries), [MASK] (for masked language modelling), and [PAD] (padding to fixed length).

SentencePiece Core

SentencePiece (Kudo & Richardson, 2018) is a language-agnostic tokenisation framework used by T5, LLaMA-1/2/3, Mistral, Gemma, XLNet, and others. Its key architectural difference from BPE and WordPiece is that it operates on raw text including whitespace, without any pre-tokenisation step. BPE and WordPiece typically split on whitespace first (giving the language-specific assumption that spaces separate words), then apply subword segmentation within each word. SentencePiece treats spaces as regular characters — the text "Hello world" is tokenised as a single stream of characters including the space character.

To make tokenisation reversible, SentencePiece uses a special ▁ (U+2581, lower one-eighth block) character to mark word boundaries. The space before a word is encoded as ▁: "Hello world" → ["▁Hello", "▁world"]. Decoding is trivial: replace ▁ with a space, concatenate. This approach means the same tokeniser works identically for languages with no spaces (Japanese, Chinese) and languages with regular spacing (English, French) — making it the preferred choice for multilingual models.

SentencePiece supports two underlying algorithms. BPE mode is the same bottom-up merge algorithm as before. Unigram Language Model mode takes the opposite approach: start with a large vocabulary (e.g. all substrings up to length 16), then iteratively remove tokens whose removal least decreases corpus likelihood, until the target size is reached. Unigram produces multiple possible segmentations of a word and assigns probabilities to them — during training, samples are drawn from the distribution, providing a natural form of tokenisation regularisation.

tiktoken & Practical Tokenisation In-depth

OpenAI's tiktoken is a fast BPE tokeniser library used by all GPT models. It supports three encodings: r50k_base (GPT-2/3, 50,257 tokens), p50k_base (Codex), and cl100k_base (GPT-4/GPT-4o, 100,277 tokens). The cl100k vocabulary was specifically designed to handle code and multilingual text more efficiently — common programming patterns like function definitions and import statements are often single tokens.

The rough practical rule is ¾ of a word per token for English text: 1,000 tokens ≈ 750 English words ≈ 4–5 average paragraphs. This ratio degrades significantly for non-English text. Chinese and Japanese characters are typically 1–4 tokens per character (since each character is a complex glyph encoded as multiple UTF-8 bytes). Arabic script runs 2–3 tokens per word. Code is token-efficient for English keywords but indentation and special characters add tokens. Understanding these ratios is essential for prompt engineering and cost estimation at scale.

Token Counting & Context Windows In-depth

The context window is the maximum number of tokens a model can process in a single forward pass — both input prompt and generated output count against this limit. Exceeding the context window truncates input, silently losing information. Context window sizes have grown dramatically: from GPT-3's 2,048 tokens (2020) to GPT-4o's 128,000 (2024), Claude 3.5's 200,000, and Gemini 1.5 Pro's 1,000,000. However, larger context windows don't mean models use all context equally well — empirically, LLMs attend more strongly to the beginning and end of long contexts ("lost in the middle" effect).

Token counting matters for three practical reasons. Cost: commercial LLM APIs price per token — $5–$30 per million tokens for frontier models, multiplied by millions of API calls adds up. Context management: overflow silently truncates your prompt — a bug that is easy to miss and hard to debug. Latency: generation cost is proportional to output tokens; every unnecessary token in the response costs inference time and money. Practitioners routinely count tokens in prompt templates, conversation histories, and retrieved documents before sending API requests.

The Distributional Hypothesis Core

The theoretical foundation of all word embedding methods is the Distributional Hypothesis, stated by linguist J.R. Firth in 1957: "You shall know a word by the company it keeps." The idea is deceptively simple: words that appear in similar linguistic contexts tend to have similar meanings. "Dog" and "cat" both appear near words like "pet", "feed", "vet", "fur", "owner", "breed" — and this co-occurrence pattern reflects their shared semantic category. "Dog" and "quantum" do not share contexts, and they do not share meaning.

This hypothesis transforms the problem of meaning into a problem of statistics. Instead of defining what "happy" means philosophically, we can simply observe that "happy" appears with "smile", "joy", "content", "pleased", "glad" — and "sad" appears with "cry", "grief", "unhappy", "depressed" — and that these two distributional profiles are measurably different. The distributional hypothesis gives us a way to measure semantic similarity without any human annotation: compute the similarity between two words' context distributions.

Every word embedding method — Word2Vec, GloVe, FastText, and even the contextual embeddings of BERT — is an implementation of this hypothesis. They differ in how they model context (local window vs global matrix, character-level vs word-level, static vs contextual), but they all share the core insight: context distribution = meaning.

Word2Vec In-depth

Mikolov et al. (Google Brain, 2013) introduced Word2Vec — a family of shallow neural networks that learn word representations by predicting word context. The key insight was framing representation learning as a self-supervised prediction task: given a word, predict its surrounding words. No labels are needed — the text itself provides the training signal. Train on enough text (Google News, 100 billion words) and the resulting vectors encode semantic structure as geometry.

The architecture is deliberately simple: a single-layer neural network with no non-linearity in the hidden layer. The input is a one-hot vector of vocabulary size V. The single hidden layer projects this to a dense vector of dimension d (typically 300). The output layer projects back to V dimensions and applies softmax to produce a probability distribution over the vocabulary. The weight matrix of the hidden layer — shape V × d — is the embedding matrix. After training, each row is the embedding vector for one word.

Word2Vec uses two architectural variants: Skip-gram and CBOW (Continuous Bag of Words). Skip-gram predicts context words from a centre word and works better on small datasets and rare words. CBOW predicts the centre word from its context and trains faster on large corpora. Both are trained with a practical approximation — negative sampling — rather than full softmax over the entire vocabulary (computing softmax over 50,000+ words every step is prohibitively expensive).

With negative sampling, the objective becomes: for each training pair (centre, context), maximise the probability of the true pair while minimising the probability of k randomly sampled negative pairs. This reduces the per-step computation from O(V) to O(k), where k is typically 5–20. The result is a practical algorithm that can be trained on billions of words in hours on a single machine.

Skip-gram & CBOW Architectures In-depth

Given the sentence "The quick brown fox jumps over the lazy dog" with window size 2: Skip-gram takes the centre word "brown" and tries to predict each context word — ("brown", "quick"), ("brown", "The"), ("brown", "fox"), ("brown", "jumps"). One training pair for each context word in the window. CBOW takes all context words ["The", "quick", "fox", "jumps"] and averages their embeddings, then tries to predict the centre word "brown". CBOW is faster (averages context, one prediction per window); Skip-gram trains on more pairs and handles rare words better.

GloVe — Global Vectors Core

Pennington, Socher, and Manning (Stanford NLP, 2014) pointed out a conceptual limitation of Word2Vec: it trains on individual context windows, one at a time, effectively ignoring the global statistics of word co-occurrence across the entire corpus. If "ice" and "steam" both co-occur with "water" but "ice" co-occurs with "solid" and "steam" with "gas", this distinction should be captured — but Word2Vec only sees local windows, not the global ratio structure.

GloVe directly factorises the global word–word co-occurrence matrix X, where Xᵢⱼ is the count of how many times word j appears in the context of word i across the entire corpus. The objective is to find word vectors such that their dot product approximates the log of the co-occurrence count. The weighting function f(Xᵢⱼ) ensures that very frequent pairs (like "the–the") don't dominate the loss — pairs with Xᵢⱼ above a threshold are capped.

In practice, GloVe and Word2Vec produce embeddings of similar quality. GloVe often edges ahead on analogy tasks (because it explicitly models co-occurrence ratios); Word2Vec can be more efficient to train on very large corpora with negative sampling. Both have been largely superseded for downstream tasks by contextual embeddings (BERT, GPT), but GloVe remains popular as a lightweight baseline and for interpretability research.

FastText — Subword Embeddings Core

Bojanowski et al. (Facebook AI Research, 2016) identified a critical gap in both Word2Vec and GloVe: they treat words as atomic units. "Run", "running", "runner", "runs" each get their own independent vector — the morphological relationship between them is invisible to the model. For languages with rich morphology (Finnish, Turkish, German, Arabic), this is devastating: a model may never see the exact form "Freundschaftsbezeigungen" (German for "demonstrations of friendship") but it shares meaningful subwords with common words.

FastText represents each word as a bag of character n-grams. For "where" with n=3: the word is decomposed as ["<wh", "whe", "her", "ere", "re>", "<where>"] (with boundary markers < and >). The final word vector is the sum of all its n-gram vectors. Each n-gram has its own embedding — these are what get trained. The boundary markers ensure "whe" in "where" and "whe" in "elsewhere" contribute differently because they occur in different boundary contexts.

The payoff: FastText can produce meaningful vectors for words never seen during training, including misspellings, technical jargon, and morphological variants. "Antidisestablishmentarianism" will share n-grams with "establish", "establishment", "disestablish", "ism", etc., and their combined embedding will be semantically meaningful. FastText is still used in production where domain vocabulary is highly variable — scientific text, social media, multilingual pipelines.

Vector Arithmetic & Analogies In-depth

The most celebrated discovery in word embeddings is that semantic relationships are encoded as consistent directions in vector space. The vector from "man" to "woman" is approximately the same as the vector from "king" to "queen", from "uncle" to "aunt", from "actor" to "actress". This "gender direction" is a consistent geometric transformation across the embedding space. Similarly, there is a "capital city direction" (France→Paris ≈ Germany→Berlin ≈ Japan→Tokyo), a "superlative direction" (big→biggest ≈ small→smallest), and a "past tense direction" (run→ran ≈ walk→walked).

This property emerged from training — it was not engineered in. It suggests that the distributional statistics of language contain enough signal to implicitly encode the relational structure of the world. The analogy task became a standard benchmark: given A:B::C:?, find D such that B−A+C ≈ D in vector space. Word2Vec achieves ~65% accuracy on the Google Analogy Dataset (20,000 analogies across semantic and syntactic categories) — far above what was thought possible with shallow models.

Embedding Properties & Limitations Core

Word embeddings inherit — and amplify — the biases present in their training data. Bolukbasi et al. (2016) demonstrated that in Word2Vec trained on Google News: "doctor − nurse ≈ man − woman", "programmer − homemaker ≈ man − woman", "brilliant − dull ≈ man − woman". These gender stereotypes are encoded as geometric structure in the embedding space. When downstream models use these embeddings, the bias propagates: a résumé classifier using Word2Vec may discriminate based on field-specific vocabulary that encodes gender. Debiasing techniques exist (projecting out the gender direction) but are only partially effective — the bias is distributed across the space, not concentrated in one direction.

The most fundamental limitation of all static word embeddings is context independence: every word has a single vector regardless of usage. The word "bank" in "She went to the bank to withdraw money" and "She sat on the river bank" produce the exact same 300-dimensional vector. The model averages the two senses of "bank" into one representation — losing the information needed to distinguish them. This polysemy problem is unresolvable within the static embedding framework, no matter how large the training corpus. It is the primary motivation for contextual embeddings: BERT, GPT, and their successors assign each word occurrence a different vector based on its surrounding context (Chapter 5.4).

The Problem with Static Embeddings Core

Word2Vec, GloVe, and FastText assign each word type exactly one vector, shared across all its occurrences. This is adequate for words with a single dominant sense — "elephant" nearly always means the same thing. But English has thousands of polysemous words: "bank" (financial institution / river edge), "bat" (cricket equipment / flying mammal), "light" (not heavy / illumination / a lamp), "book" (a publication / to reserve), "well" (healthy / a water source / interjection). The Word2Vec vector for "bank" is a weighted average of all its senses — useful for neither.

The polysemy ceiling is not solvable by training on more data. No matter how large the corpus, a single vector must average all contexts. The architecture itself is the limitation: static embeddings compute representations before seeing the sentence. What is needed is a model that reads the full sentence, then assigns each word a representation based on its role in that specific sentence. This is exactly what contextual embedding models provide.

ELMo — Embeddings from Language Models Core

Peters et al. (AllenNLP, 2018) introduced ELMo — Embeddings from Language Models — the first widely adopted contextual word embedding. The architecture is a two-layer bidirectional LSTM language model pre-trained on 1 billion words (1 Billion Word Benchmark). Two passes through the sentence: a forward LM reads left to right and learns to predict the next word; a backward LM reads right to left and learns to predict the previous word. For each token, the forward and backward hidden states from all layers are concatenated, producing a context-sensitive representation.

ELMo representations are used as frozen features — the ELMo model is not fine-tuned on downstream tasks. Instead, the pre-computed ELMo vectors are concatenated to the input of existing task-specific models (NER taggers, QA systems, coreference models). This "feature-based" approach produced large, consistent improvements across NLP benchmarks — the first empirical proof that language model pre-training transfers broadly. ELMo improved the state-of-the-art on 6 NLP tasks simultaneously, which was extraordinary at the time.

ELMo's key limitation: the underlying architecture is an LSTM, which processes sequences sequentially (O(n) depth) and cannot parallelise across token positions. Training is slow and the representations are computed sequentially at inference. The Transformer architecture (Chapter 5.5), with O(1) depth and full parallelism via self-attention, replaced the LSTM backbone in every subsequent contextual embedding model.

The ULMFiT Pre-train / Fine-tune Paradigm In-depth

Howard & Ruder (2018) introduced ULMFiT (Universal Language Model Fine-Tuning) — the paper that established the three-stage paradigm now universal in NLP. Where ELMo froze the language model and used it as a feature extractor, ULMFiT's insight was that the language model itself should be fine-tuned end-to-end on downstream tasks. This shifts the mental model from "use LM features" to "adapt a pre-trained LM for each task". BERT and GPT made this paradigm dominant — but ULMFiT proved it worked first.

ULMFiT introduced two now-standard fine-tuning techniques. Discriminative fine-tuning: assign different learning rates to each layer — earlier layers (which capture general syntax and morphology) are updated very slowly; later layers (which capture task-specific semantics) are updated faster. Gradual unfreezing: start fine-tuning only the last layer, then progressively unfreeze earlier layers one at a time. This prevents catastrophic forgetting — the phenomenon where fine-tuning on a small task dataset destroys the broad language knowledge acquired during pre-training.

The three stages generalise directly to all modern pre-trained language models. Stage 1 (pre-training) is expensive but done once and shared via model hubs. Stage 2 (domain adaptation) is optional but valuable for specialised domains (biomedical, legal, code). Stage 3 (task fine-tuning) is cheap — hours on a single GPU with hundreds to thousands of labelled examples, compared to millions required to train from scratch. This cost asymmetry is the fundamental economic argument for the transformer pre-training paradigm.

Self-Supervised Pre-training Tasks In-depth

Causal Language Modelling (CLM) — used by GPT, GPT-2, GPT-3, LLaMA: predict the next token given all previous tokens. Objective: maximise P(xₜ | x₁,...,xₜ₋₁). The model never sees future tokens during training — it processes left-to-right with a causal (triangular) attention mask. This makes it naturally suited to generation: at inference, repeatedly predict the next token and append it to the sequence.

Masked Language Modelling (MLM) — used by BERT, RoBERTa, DeBERTa: randomly mask 15% of input tokens (replacing them with [MASK]), then predict the original token using the full surrounding context. Objective: maximise P(masked | all other tokens). Because both left and right context is available simultaneously, the model builds bidirectional representations — excellent for understanding tasks (classification, NER, QA) but not for generation.

Next Sentence Prediction (NSP) was used in original BERT alongside MLM: given two text segments, predict whether they appear consecutively in the source document. Later analysis (RoBERTa, 2019) showed NSP adds little benefit and can hurt performance by forcing artificially short segments — it was removed in subsequent models. Replaced Token Detection (RTD) — used by ELECTRA: a small generator network creates plausible but fake token replacements; the main discriminator must identify which tokens were replaced. Every token gets a training signal (not just 15% as in MLM), making ELECTRA 4× more efficient for the same computational budget.

Sentence Embeddings In-depth

Word-level contextual embeddings (one vector per token) are essential for token-level tasks like NER and POS tagging — but many applications require a single vector for an entire sentence, paragraph, or document. How do you pool a variable-length sequence of token vectors into one fixed-size representation? Three approaches have been widely used.

[CLS] token pooling (BERT's approach): prepend a special [CLS] (classification) token to every input. The Transformer processes all tokens together with full self-attention. In theory, the [CLS] token's output representation aggregates information from the entire sequence. In practice, this works well after fine-tuning on a specific task — but out-of-the-box BERT [CLS] vectors perform poorly on semantic similarity benchmarks, because BERT was not trained to produce meaningful sentence-level representations in [CLS].

Mean pooling: average all token embeddings in the final layer. Surprisingly effective as a baseline — often outperforms [CLS] pooling on zero-shot semantic similarity without fine-tuning. Simple to implement and parameter-free. Sentence-BERT (SBERT, Reimers & Gurevych, 2019) addresses both approaches by fine-tuning BERT with a siamese / triplet network objective on sentence pairs — training the model to produce similar vectors for semantically similar sentences. SBERT dramatically outperforms naive pooling on STS benchmarks and is 20–30× faster for pair-wise similarity computation than vanilla BERT (which requires a separate forward pass for every pair).

Embedding Models & Semantic Search Core

Sentence embeddings power semantic search — retrieving documents by meaning rather than keyword overlap. A query like "What city is France's capital?" should retrieve "Paris is the seat of French government" even though it shares no keywords. The approach: embed all documents once into a vector index; at query time, embed the query and retrieve the closest document vectors by cosine similarity. This is the retrieval component of Retrieval-Augmented Generation (RAG) systems.

GPT Architecture In-Depth

GPT = Generative Pre-trained Transformer — a decoder-only Transformer. Unlike BERT (encoder-only, bidirectional), GPT uses causal (masked) self-attention: each token can attend ONLY to previous tokens. This constraint is not a limitation — it's the design that enables generation. You can't look at future tokens while generating them.

The key architectural choices that distinguish GPT from BERT:

Neural Scaling Laws In-Depth

Kaplan et al. (OpenAI, 2020) discovered that language model loss decreases as a smooth power law as you increase model size (N), dataset size (D), or compute (C). This isn't a vague trend — it's a precise mathematical relationship: L(N) ∝ N−α where α ≈ 0.076. Double the parameters and loss drops predictably.

Three factors drive scaling: N (parameters), D (dataset tokens), and C (compute in FLOPs). The breakthrough insight: you must scale all three together. Scaling parameters alone while holding data fixed gives diminishing returns.

GPT-1 to GPT-4 Timeline In-Depth

Emergent Capabilities In-Depth

Wei et al. (2022) documented a surprising phenomenon: certain capabilities appear suddenly at a scale threshold — they are essentially absent in smaller models and then abruptly present in larger ones. These emergent abilities were not explicitly trained. The model was only ever trained to predict the next token. Yet above a certain parameter count, it can perform multi-step arithmetic, chain-of-thought reasoning, translation between unseen language pairs, and code generation.

Inference & Sampling Strategies Core

How does an LLM actually generate text? At each step, the model outputs a probability distribution over the entire vocabulary. The decoding strategy determines which token to pick from that distribution. This choice dramatically affects output quality, diversity, and creativity.

Open-Source LLMs Core

The open-source LLM ecosystem exploded in 2023–2025. Models from Meta, Mistral, Alibaba, Google, Microsoft, and others are approaching closed-source frontier quality. This table captures the major families as of 2024–2025.

BERT Architecture In-Depth

Devlin et al. (Google, 2018): BERT — Bidirectional Encoder Representations from Transformers. BERT uses an encoder-only Transformer stack — no decoder, no causal mask. Every token attends to all other tokens simultaneously (bidirectional attention). This is the key difference from GPT: BERT sees the full context before producing representations.

BERT Special Inputs Core

BERT's input representation is the sum of three embedding types (not concatenated). Every input is prepended with [CLS] and sentence pairs are separated by [SEP].

Segment embeddings tell BERT which sentence each token belongs to (Sentence A vs Sentence B). The three input components are summed element-wise: Token Embedding + Positional Embedding + Segment Embedding.

Fine-tuning BERT In-Depth

BERT's power lies in fine-tuning: take the pre-trained backbone and add a thin task-specific head. All BERT weights are updated during fine-tuning (with a small learning rate). Four canonical task types:

The BERT Family Core

BERT vs GPT

When to Use Encoders vs Decoders Core

What Is Prompt Engineering? Core

LLMs are not search engines — they are conditional probability machines. Given your prompt as the beginning of a document, they predict what comes next. The quality and structure of that beginning determines everything about the continuation.

Compare: "What is the capital of France?" vs "Answer as a geography teacher giving a detailed explanation: What is the capital of France?" — same factual answer but very different style and depth.

Five prompt components:

Zero-Shot & Few-Shot Prompting In-Depth

Zero-shot: ask the model without any examples. Works well for simple, well-defined tasks.

Few-shot (in-context learning): provide 2–5 examples before asking. Brown et al. (GPT-3, 2020) showed that providing examples dramatically improves performance. The model is not fine-tuned — it adapts to the task from examples in its context window.

One-shot: exactly one example — sometimes all you need for well-defined tasks.

Chain-of-Thought Prompting In-Depth

Wei et al. (Google, 2022): "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models". The key insight: prompting LLMs to "think step by step" dramatically improves reasoning accuracy on math, logic, and multi-step problems.

Why it works: the model generates intermediate steps → each step conditions the next → less error accumulation. The chain of reasoning acts as a scratchpad that keeps the model on track.

Structured Prompting In-Depth

Structure your prompts for reliable, parseable output. Three key techniques:

System Prompts

In chat-based APIs (OpenAI, Anthropic, etc.), the system prompt sets the model's behaviour, persona, and constraints before the user speaks. It's the most powerful lever for controlling output quality.

Prompt Patterns Catalogue Core

Common Pitfalls Core

Why RAG? Core

Three fundamental LLM limitations make RAG essential for production systems:

RAG Architecture In-Depth

RAG has two phases: an offline indexing pipeline (run once or periodically) and an online query pipeline (run at every user question). Both share the same embedding model and vector database.

Vector Databases In-Depth

Vector databases are specialised stores for embedding vectors, optimised for similarity search. The core operation is Approximate Nearest Neighbour (ANN) search: "find the k vectors closest to this query vector." Exact search is O(n·d) — ANN algorithms like HNSW make this dramatically faster.

Chunking Strategies In-Depth

Chunk size matters enormously — wrong size degrades RAG quality. Too small: insufficient context in each chunk → incomplete answers. Too large: irrelevant content mixed with relevant → noisy retrieval. Chunks should overlap (50–100 tokens) to avoid splitting context across boundaries.

Advanced RAG Patterns In-Depth

Naive RAG (embed → search → generate) works for many cases. These advanced patterns dramatically improve precision and recall for production systems:

RAG vs Fine-Tuning Core

Hallucination In-Depth

Hallucination: LLMs generate factually incorrect information with apparent confidence. This is not a bug — it's a consequence of the training objective. The model was rewarded for coherent, fluent text, not for verified facts.

Intrinsic vs extrinsic hallucination:

Why LLMs Hallucinate Core

Root cause: LLMs are trained to produce fluent, probable text — not factual text. The model doesn't "know" it doesn't know something. Confidence calibration is poor: models are confidently wrong, which is more dangerous than being uncertainly wrong.

Alignment In-Depth

Alignment problem: ensure AI systems behave according to human values and intentions. A highly capable but misaligned AI is dangerous — capability without alignment amplifies harm.

The specification problem: how do you formally specify "what humans actually want"? Even well-intentioned reward functions can be gamed — the model maximises the metric in unintended ways (reward hacking).

Key alignment challenges:

Helpful, Harmless, Honest (HHH) Core

Anthropic's HHH framework defines the three axes of aligned model behaviour. The tension: being more helpful sometimes means being slightly less cautious (and vice versa). Constitutional AI resolves this by giving the model explicit principles to follow.

NLP Evaluation Metrics In-Depth

Automatic metrics enable scalable evaluation, but each has significant blind spots. Understanding their strengths and weaknesses is essential for trustworthy evaluation.

LLM Benchmarks In-Depth

When to Fine-Tune (vs RAG vs Prompting) In-Depth

Decision framework — try in order (cheapest first):

Fine-tuning IS the right choice when:

Data Preparation In-Depth

The #1 determinant of fine-tuning quality is data quality. Garbage in, garbage out — but amplified by the power of gradient descent. Format: instruction-following datasets use the chat message format (system / user / assistant).

Supervised Fine-Tuning (SFT) In-Depth

SFT objective: minimise cross-entropy loss on the assistant turns only. The user/system turns are masked — the model doesn't compute loss on the prompt tokens, only on the response it should have generated.

Key hyperparameters:

LoRA & QLoRA in Practice In-Depth

QLoRA (Dettmers et al., 2023): LoRA applied on a 4-bit quantised base model. This makes it possible to fine-tune a 70B model on a single 48GB GPU — impossible without quantisation.

DPO: Direct Preference Optimisation Core

Rafailov et al. (2023): "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." RLHF requires three components: SFT model + reward model + PPO training — complex, unstable, expensive.

DPO insight: the optimal RLHF policy has a closed-form solution — no separate reward model needed. DPO uses preference pairs: (prompt, chosen_response, rejected_response). Train to increase likelihood of chosen relative to rejected. Simpler, more stable, often achieves similar or better results.

Full Fine-Tuning Pipeline Core