Image Representation In-depth

Every digital image is, at its core, a grid of numbers. Each cell in this grid is a pixel — the atomic unit of a digital image, representing a single colour sample at a specific grid position. Before any algorithm can process an image, you must understand how those pixels are encoded and what they represent.

Grayscale images use a single channel — each pixel is an integer from 0 (black) to 255 (white), giving 256 possible intensity levels. RGB images use three channels (Red, Green, Blue), each ranging 0–255, producing 256³ = 16.7 million possible colours per pixel. Every pixel in an RGB image is defined by exactly three numbers.

In deep learning, the tensor representation matters enormously. PyTorch uses channels-first ordering: (C, H, W) — so a 224×224 RGB image becomes shape (3, 224, 224). NumPy and OpenCV use height-first: (H, W, C). Confusing these axes is one of the most common bugs in computer vision code.

Data types matter for performance. Raw images are stored as uint8 (0–255) — compact but unsuitable for gradients. Neural networks require float32 in range [0.0, 1.0] or normalised with ImageNet statistics: subtract mean [0.485, 0.456, 0.406] and divide by std [0.229, 0.224, 0.225]. This normalisation centres values around zero, helping gradient-based optimisation converge faster.

Colour Spaces Core

RGB is the natural colour space for displays — your screen uses red, green, and blue LEDs. But RGB mixes luminance (brightness) and chrominance (colour) together, making it poor for many computer vision tasks. Different colour spaces separate these properties, each suited to specific algorithms.

Grayscale conversion is not a simple average of R, G, B. The human eye is most sensitive to green, so the standard formula is: Luminance = 0.299R + 0.587G + 0.114B. Green contributes nearly 60% of perceived brightness. This is why most classical CV algorithms operate on grayscale — it reduces data by 3× while preserving the structural information humans rely on.

Image Filters & Convolutions In-depth

A filter (or kernel) is a small matrix — typically 3×3 or 5×5 — that slides across an image computing a weighted sum at each position. This operation is called convolution, and it's the single most important operation in all of computer vision. Classical filters are hand-designed for specific effects. Deep learning CNNs learn their filters from data — but the underlying mathematical mechanism is identical.

Edge Detection Core

Edges are boundaries between regions of different intensity — they mark where objects begin and end, where surfaces change orientation, and where textures shift. Edges are the most information-dense locations in an image: they encode shape, structure, and object boundaries while discarding uniform regions that carry little useful signal.

The Canny Edge Detector (John Canny, 1986) remains the gold standard for classical edge detection. It's a five-step pipeline, each step carefully designed to balance noise rejection against edge localisation. Nearly every classical CV system used Canny as a preprocessing step — from document scanning to lane detection to augmented reality registration.

The Harris Corner Detector extends edge detection to find corners — points where intensity changes significantly in two directions simultaneously. Corners are more distinctive than edges (an edge looks the same along its length), making them better landmarks for matching between images. Harris corners are still used in camera calibration and simple tracking systems.

Classical Feature Descriptors Core

Before deep learning, the central challenge of computer vision was: how do you represent an image patch as a compact, distinctive vector that's robust to scale, rotation, and illumination changes? Researchers hand-crafted feature descriptors that encode local image structure — these dominated from roughly 2000 to 2012.

Classical CV Pipeline Core

The classical computer vision pipeline dominated the field from approximately 1980 to 2012. Every step was designed and tuned by hand — a brittle, domain-specific process that required deep expertise and did not generalise well across tasks or visual domains.

Why Deep Learning Won In-depth

In 2012, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton entered the ImageNet Large-Scale Visual Recognition Challenge with AlexNet — a deep convolutional neural network trained on two GPUs. It achieved a top-5 error rate of 15.3%, compared to 26.2% for the best classical method. That 41% relative improvement in a single year was the most dramatic result in the competition's history, and it permanently changed computer vision.

Three fundamental advantages explain why deep learning dominates:

CNN Architecture Recap Core

Before diving into specific architectures, recall the three inductive biases that make CNNs ideal for images (covered in Domain 4, Ch 4.5):

The core building block pattern used in almost every CNN: Conv → BatchNorm → ReLU → Pooling. As you go deeper, spatial dimensions shrink (via pooling/stride) while channel depth grows (more filters).

AlexNet & VGG In-depth

AlexNet (Krizhevsky et al., 2012) was the CNN that launched the deep learning revolution. It didn't just beat the competition on ImageNet — it obliterated it, reducing top-5 error from 25.8% to 15.3%. Every key idea it introduced became standard practice.

Inception & GoogLeNet In-depth

Szegedy et al. (Google, 2014) asked: how do you go deeper more efficiently? VGG's approach of stacking 3×3 convolutions worked, but parameters and computation grew together. The Inception module solved this with a radically different idea.

The Inception module applies multiple filter sizes (1×1, 3×3, 5×5) in parallel at each layer, plus a max-pooling branch. Outputs are concatenated along the channel dimension. The network learns which spatial scale to attend to at each layer.

The critical insight: 1×1 convolutions as dimensionality reduction. Before the expensive 3×3 and 5×5 convolutions, a 1×1 "bottleneck" conv reduces channel count dramatically. This made GoogLeNet just 5M parameters — 12× fewer than AlexNet with better accuracy.

ResNet & DenseNet In-depth

ResNet (He et al., 2015) solved the most puzzling problem in deep learning at the time: deeper networks had higher training error. Not overfitting — the networks simply couldn't learn. The solution was deceptively simple.

The residual block learns the change rather than the full mapping: y = F(x) + x. The skip connection allows gradients to flow directly through the identity path, enabling 50, 101, and even 152-layer networks.

DenseNet (Huang et al., 2017) extended this idea: connect every layer to all subsequent layers within a block. With L layers in a dense block, there are L(L+1)/2 direct connections. Each layer receives feature maps from ALL preceding layers, enabling maximum feature reuse with fewer parameters.

MobileNet & Efficient CNNs In-depth

ResNet and VGG are too heavy for mobile phones, embedded systems, and real-time applications. MobileNet (Howard et al., 2017) introduced depthwise separable convolutions — splitting one expensive operation into two cheap ones.

EfficientNet & NAS Core

Tan & Le (Google, 2019) observed that previous architectures scaled only one dimension at a time — depth (more layers), width (more channels), or resolution (larger images). EfficientNet scales all three simultaneously with a fixed compound coefficient.

Neural Architecture Search (NAS) was used to find the optimal base architecture (EfficientNet-B0). NAS automates architecture design by searching over a space of possible layers, connections, and hyperparameters. The EfficientNet family (B0–B7) uses the same base architecture at different compound scales.

Data Augmentation In-depth

Training a CNN requires millions of images — but labelled data is expensive. Data augmentation creates diverse training samples by applying label-preserving transformations. A flipped cat is still a cat. A slightly rotated stop sign is still a stop sign.

Transfer Learning in CV In-depth

Never train from scratch. This is the single most important practical rule in computer vision. ImageNet pre-trained models have already learned universal visual features — edges, textures, shapes — that transfer remarkably well to nearly any image task.

The Detection Task Core

Image classification assigns ONE label to an entire image: "cat." Object detection finds MULTIPLE objects, drawing a bounding box around each and labelling it: "cat at (x,y,w,h) with 97% confidence, dog at (x',y',w',h') with 89% confidence."

The output format is a list of detections, each containing: (class_id, confidence, x_centre, y_centre, width, height). Coordinates are typically normalised to 0–1 relative to image dimensions.

Sliding Window & Anchor Boxes Core

The naive approach: slide a window across the image at multiple scales and aspect ratios, run a classifier on each crop. This is conceptually simple but catastrophically slow — hundreds of thousands of crops per image, each requiring a full forward pass.

Anchor boxes solved this elegantly. Instead of sliding a window, divide the feature map into a grid and place pre-defined bounding box shapes (anchors) at each cell. The model predicts offsets from these anchors: (δx, δy, δw, δh) plus an objectness score and class probabilities. Anchors are designed to cover common shapes — wide rectangles for cars, tall rectangles for people, squares for faces.

R-CNN Family In-depth

The R-CNN family represents the two-stage approach to detection: first propose regions that might contain objects, then classify each region. Three papers over two years went from painfully slow to real-time.

YOLO: You Only Look Once In-depth

Redmon et al. (2015) made a radical move: frame detection as a single regression problem. Divide the image into an S×S grid, and in a single forward pass, predict all bounding boxes and class probabilities simultaneously. No proposals, no second stage — just one neural network, one pass, done.

The result: 45 FPS on 2015 hardware, over 200× faster than R-CNN. The trade-off was accuracy — YOLO struggled with small objects and nearby objects in the same grid cell. But the speed was revolutionary for real-time applications like autonomous driving and robotics.

SSD & Single-Stage Detectors Core

SSD (Liu et al., 2016) combined YOLO's single-pass speed with an elegant multi-scale approach. Instead of predicting from just one feature map, SSD extracts predictions from 6 different feature map scales within the CNN. Early (larger) feature maps detect small objects; later (smaller) feature maps detect large objects.

This solved YOLO's weakness with small objects: 59 FPS at 300×300 input, 74.3% mAP — better small-object detection with comparable speed.

NMS & Detection Metrics In-depth

Three concepts underpin detection evaluation: IoU measures box overlap quality, NMS removes duplicate detections, and mAP quantifies overall detector performance.

Segmentation Types Core

All three segmentation types assign labels at the pixel level — but they differ fundamentally in what they distinguish.

Semantic Segmentation In-depth

The core challenge: classification networks reduce spatial resolution (pooling, striding) to build semantic understanding. Segmentation needs to restore it — predict a class for every single pixel. The solution: encoder-decoder architectures.

Skip connections are critical: they connect encoder layers to decoder layers at matching resolutions. Without them, the decoder must reconstruct fine spatial detail (edges, boundaries) from the bottleneck alone — a lossy process. With skips, fine spatial detail flows directly from encoder to decoder.

U-Net Architecture In-depth

Ronneberger et al. (2015) designed U-Net for biomedical image segmentation — but it became the universal segmentation architecture. The U-shaped design features a symmetric encoder (contracting path) and decoder (expanding path) connected by skip connections at every level.

The critical innovation: skip connections don't just add features (like ResNet) — they concatenate entire feature maps from encoder to decoder. This preserves the fine spatial detail (textures, edges) lost during downsampling. The decoder gets both the upsampled abstract features AND the original high-resolution details.

Mask R-CNN & Instance Segmentation In-depth

Mask R-CNN (He et al., Facebook AI, 2017) extends Faster R-CNN with a simple but powerful addition: a third prediction head that outputs a pixel-level mask for each detected object. Three heads run in parallel after RoI features are extracted:

A critical improvement: RoI Align replaced RoI Pooling. RoI Pooling quantises coordinates (rounding to nearest pixel), causing spatial misalignment that doesn't matter for bounding boxes but is catastrophic for pixel-accurate masks. RoI Align uses bilinear interpolation — no quantisation, precise alignment.

Panoptic Segmentation In-depth

Panoptic segmentation (Kirillov et al., 2019) unifies semantic and instance segmentation into a single coherent output. Every pixel gets a class label. "Things" (countable: cars, people) also get unique instance IDs. "Stuff" (uncountable: sky, road, grass) gets class labels only.

Modern panoptic models like Panoptic FPN and Mask2Former (2022) use a unified architecture that handles both things and stuff with a single decoder. Mask2Former treats all segments (things + stuff) as mask queries processed by a transformer decoder — achieving SOTA on all three segmentation tasks simultaneously.

SAM: Segment Anything Model In-depth

Kirillov et al. (Meta AI, 2023) built a foundation model for segmentation. Trained on SA-1B — 11 million images with 1.1 billion segmentation masks — SAM can segment any object in any image with a simple prompt: click a point, draw a box, or provide text.

GAN Fundamentals In-depth

Goodfellow et al. (2014) introduced one of the most cited frameworks in deep learning: two networks in competition. The Generator G takes random noise z ~ N(0,I) and produces fake images G(z). The Discriminator D receives an image (real or fake) and outputs the probability that it's real.

Training alternates: update D for k steps (improve its detection ability), then update G for 1 step (improve its forgery). At Nash equilibrium, G produces perfect fakes and D outputs 0.5 for everything — it literally cannot tell real from fake.

GAN Loss Functions Core

The original GAN loss has two critical failure modes. Vanishing gradients: when D becomes too good, D(G(z)) ≈ 0, and log(0) = −∞ gives no useful gradient for G. Mode collapse: G finds a few outputs that fool D and keeps generating only those, ignoring the rest of the distribution.

DCGAN Core

Radford et al. (2015) established the first stable recipe for CNN-based GANs. Before DCGAN, most GAN experiments produced noise or collapsed. DCGAN's design rules became gospel for all subsequent work:

Conditional GAN Core

Vanilla GANs generate random images — you have no control over what comes out. Mirza & Osindero (2014) fixed this by feeding a condition label y to both G and D. Now G(z, y) generates an image of class y, and D(x, y) verifies it's a real image of that class.

StyleGAN In-depth

Karras et al. (NVIDIA, 2019) produced the first truly photorealistic synthetic faces. The key insight: separate the latent code into a style that controls appearance at each resolution level, rather than feeding noise directly into the first layer.

CycleGAN Core

pix2pix requires paired training data — the exact same scene in both domains (e.g., the same street in day and night). This is often impossible to collect. Zhu et al. (2017) solved this with CycleGAN: learn translation between domains using only unpaired examples.

The trick: cycle consistency. Two generators (G_AB: A→B, G_BA: B→A) must satisfy G_BA(G_AB(a)) ≈ a. If you translate a horse to a zebra and back, you should get the original horse. This constraint prevents the generators from hallucinating arbitrary outputs.

GAN Training Challenges In-depth

CNN Limitations That Motivated ViT Core

CNNs have two hard-wired inductive biases: locality (convolutions see only a small neighbourhood) and translation equivariance (same filter applied everywhere). These are powerful priors — but they also limit expressiveness.

A 3×3 conv at layer 1 sees only 9 pixels. To see the whole image, information must pass through many layers of pooling — getting progressively diluted. Transformers have no such constraints: self-attention connects every patch to every other patch in a single operation. Global context from layer 1.

ViT: An Image is Worth 16×16 Words In-depth

Dosovitskiy et al. (Google Brain, 2020) applied the standard Transformer encoder — unchanged from NLP — directly to images. The trick: divide the image into fixed-size non-overlapping patches, flatten each patch, linearly project it, and treat the resulting sequence exactly like word tokens.

ViT Training Requirements Core

The original paper's most critical finding: ViT-Large trained on ImageNet-1k ONLY achieved 77.9% — worse than ResNet-50 (76.1%). But pre-trained on JFT-300M (Google's internal 300M-image dataset), it reached 88.55% — crushing every CNN. The conclusion: ViT needs massive data to overcome its lack of inductive biases.

DeiT: Data-Efficient Image Transformers In-depth

Touvron et al. (Facebook, 2021) asked: can we train ViT on ImageNet-1k without Google's proprietary JFT-300M? The answer: yes, with three key tricks — knowledge distillation from a CNN teacher, aggressive data augmentation, and a careful training recipe.

DeiT adds a second special token: [DIST] (distillation token) alongside the standard [CLS]. [DIST] is trained to match the CNN teacher's soft labels, while [CLS] learns from ground-truth hard labels. At inference, both outputs are averaged. Result: DeiT-Base: 81.8% vs original ViT-Base on same data: 77.9%.

Swin Transformer In-depth

Liu et al. (Microsoft, 2021) solved ViT's two biggest problems for dense prediction tasks: fixed single-scale output and quadratic attention cost. Swin Transformer (ICCV 2021 Best Paper) introduced window attention and hierarchical feature maps — making transformers practical for detection and segmentation.

ConvNeXt: A ConvNet for the 2020s Core

Liu et al. (Facebook, 2022) asked: "What if we took ResNet and applied every Transformer design decision?" Starting from ResNet-50, they applied 7 systematic changes. The result: ConvNeXt-Base: 83.8% — matching Swin-Base (83.5%) without any attention mechanism.

Architecture Showdown In-depth

What Is Multimodal AI? Core

Unimodal models process only one type of data — a text-only LLM or an image-only CNN. Multimodal models process and relate multiple data types simultaneously: text + image, image + audio, video + text. Real-world tasks are inherently multimodal — "describe what's in this photo" requires both vision and language.

CLIP: Contrastive Language-Image Pre-training In-depth

Radford et al. (OpenAI, 2021) trained two encoders — one for images, one for text — jointly on 400 million (image, text) pairs scraped from the internet. No manual labels: web captions ARE the supervision. The goal: learn a shared embedding space where matching image-text pairs are close and non-matching pairs are far apart.

What CLIP Enables In-depth

DALL-E 1, 2, and 3 In-depth

Vision-Language Models (VLMs) In-depth

VLMs accept both images and text as input and generate text as output. The core challenge: how do you connect a vision encoder to a language model? Three main approaches have emerged:

GPT-4V, Gemini & Open VLMs Core

Multimodal Benchmarks Core

The Video Challenge Core

Video adds the temporal dimension to images: motion, change, causality, events. Processing frames independently with image models misses all temporal information — you can't tell if a person is walking left or right. Adjacent frames are also ~95% identical pixels, making naive processing extremely redundant.

Optical Flow In-depth

Optical flow computes a dense per-pixel motion vector field between two consecutive frames. For each pixel (x,y) in frame t, it asks: where does this pixel move to in frame t+1? The result is an H×W×2 flow field (Δx, Δy per pixel). Used in action recognition, video stabilisation, compression, and motion segmentation.

Video Understanding Models In-depth

The history of video understanding mirrors the history of image understanding: hand-crafted features → CNNs → Transformers. Each generation solved the temporal modelling problem differently.

Video Generation Core

Video generation is dramatically harder than image generation: objects must stay consistent across hundreds of frames, motion must follow physics, and coherent storylines span many seconds. The field progressed rapidly from 2022–2024.

Depth Estimation In-depth

Depth estimation predicts the distance from camera to each pixel, producing an H×W depth map. Monocular depth (single camera) is ambiguous — a nearby toy car looks like a distant real car. Deep learning now handles this, trained on stereo pairs or synthetic data to learn scale cues like perspective and size.

Point Clouds & LiDAR Core

A point cloud is an unordered set of 3D points, each a (x,y,z) coordinate. LiDAR sensors emit laser pulses and measure return time to build dense 3D point clouds at 1.3M points/second. Unlike images, point clouds have no grid structure, variable density, and only capture visible surfaces.

NeRF & 3D Gaussian Splatting In-depth

Neural Radiance Fields (NeRF) (Mildenhall et al., 2020) learn a 3D scene representation from 2D photos — given any new camera viewpoint, the model synthesises a photorealistic image. A small MLP maps (x, y, z, direction) → (colour, density). Volume rendering integrates these values along camera rays. Given 20–100 posed photos → synthesise any novel angle.

3D Gaussian Splatting (Kerbl et al., 2023) replaces NeRF's implicit MLP with explicit 3D Gaussians — each has a centre, covariance shape, colour, and opacity. Rendering projects Gaussians to 2D and rasterises directly on GPU. Result: near-real-time novel view synthesis at 30fps, better quality, and 30-minute training vs NeRF's hours.