A user types “wireless bluetooth headphones” into your search box and expects to see relevant products in 100 ms. The first instinct is reasonable: stick a LIKE '%query%' on the products table and call it done. It works for the demo. It limps along at a thousand products. At a hundred thousand products it takes seconds. At ten million it takes minutes — and it returns nothing for “wireles bluetoth”, finds “headphones” only when the column literally contains that exact substring, and ranks results by row order rather than relevance. The wall isn't a Postgres tuning problem. The wall is that you're using the wrong tool: relational databases are built to find rows by exact field value, and search systems are built to find documents by what they mean.

A search system flips the data structure inside out. Instead of storing rows and scanning columns, it stores an inverted index — a map from each word to the list of documents containing it. Instead of ranking by primary key, it scores by relevance using statistical signals about how informative each word is. Instead of exact-match, it tokenises and analyses text so “running shoes”, “run shoe”, and “Runners Shoe” can all match “runner's shoes”. The cost: a separate system, a separate index to keep in sync, a different query language, and a separate operational burden. The benefit: sub-100 ms full-text queries over hundreds of millions of documents, with relevance ranking, fuzzy matching, faceting, and aggregations that no relational database can match at any scale.

Open a textbook to its index at the back. You don't find a list of pages numbered 1 to 800; you find an alphabetical list of topics, each pointing to the pages where that topic appears. To find every page that discusses “recursion”, you look up the word “recursion” once and read the page list. To find pages discussing both “recursion” and “memoisation”, you look up both and intersect the lists. That is exactly an inverted index. A relational table is the book's page-by-page contents (find me page 247); an inverted index is the back-of-book index (find me every page that mentions X). The structures are inversions of each other — and once you see this, the rest of search architecture becomes intuitive.

When a document arrives at Elasticsearch, it doesn't go straight into the index. It passes through an analyser that turns text into terms, and only those terms end up in the inverted index. The pipeline has three stages: character filters normalise the raw text (strip HTML, transliterate accents); the tokeniser breaks the stream into tokens (usually on whitespace and punctuation); token filters transform each token (lowercase, remove stopwords like “the”, stem “running” → “run”, expand synonyms). The output — a sequence of normalised terms — is what gets indexed. The same analyser is applied to the query at search time, which is why the choice of analyser is the single most important configuration decision in a search system. Get it wrong and your index is full of unmatchable strings.

Returning matching documents is the easy part. Ranking them by usefulness is what separates a search system from a glorified WHERE clause. The classic algorithm is TF-IDF, on two intuitions: term frequency (a doc that mentions “recursion” ten times is more about recursion than one mentioning it once); inverse document frequency (the word “the” appears everywhere, so its presence tells us nothing — rare words like “recursion” are more informative). The score for a doc is the product: high TF, high IDF → high score. Modern Elasticsearch defaults to BM25, a refinement that prevents term frequency from dominating (the tenth occurrence adds less than the second) and adjusts for document length (a 50-word doc with three matches is more focused than a 5,000-word one with the same three matches). The signals are statistical, not semantic, but they get you remarkably far on real corpora.

Elasticsearch achieves scale by splitting an index into shards (each a self-contained Lucene index). A query fans out to all shards in parallel; each returns its top-K, and the coordinator merges them. Add shards → more parallelism, more capacity. Replicas are full copies of each shard on different nodes — they provide read scale-out and survive node failures. The number of primary shards is fixed when the index is created and can't be changed without reindexing; the number of replicas can be tuned anytime. Inside a shard, data lives in immutable segments — new writes create new segments, and a background merge process consolidates them. Deletions are tombstones; updates are delete-plus-insert. Understanding the shard / replica / segment hierarchy is the difference between a cluster that scales gracefully and one that mysteriously falls over at 80% disk usage.

A search system is not a database upgrade. It is a complement, almost always living alongside your primary store. So the right question is not “ES or Postgres?” — it's “am I trying to answer a question my database is fundamentally bad at, and is that costing me enough to justify a second system?” The honest answer for many teams is no: a Postgres tsvector column with a GIN index will serve a few million rows of full-text data with respectable relevance, no extra infrastructure, and the same transactional guarantees as the rest of your data. Reach for a dedicated search system when you cross thresholds the database can't cross: tens of millions of documents, sub-100 ms query latency requirements, multi-language analysers, faceted navigation across high-cardinality fields, or relevance tuning that goes beyond “ORDER BY ts_rank”.

The right mental model: your relational database is the source of truth. Every product, user, document, log entry — the authoritative copy lives there with referential integrity, transactions, and constraints. The search system is a derived, denormalised view of that data, optimised for one access pattern: full-text + faceted + relevance-ranked queries. Writes go to the database; an indexing pipeline projects them into the search index; reads for search-shaped queries go to the index, while everything else — checkout, billing, account changes — stays on the database. The two systems will be slightly inconsistent at any given moment (typically by milliseconds to seconds); designs that pretend otherwise will hit subtle bugs.

First: consistency lag between database and search index. The index is updated by a pipeline that runs after the write commits in the database — a refresh interval (Elasticsearch default: 1 second) plus pipeline latency. A user updates their profile, immediately searches for themselves, and sees the old value. This is normal. Workarounds — refresh-on-write, query-after-write — exist but cost throughput. Second: relevance versus performance. More analysis (n-grams, synonyms, semantic embeddings) means better recall but bigger indexes and slower queries. The right operating point is a per-product decision and changes over time. Third: operational complexity. Elasticsearch is a stateful distributed system with shard rebalancing, master elections, JVM tuning, segment merges, mapping migrations, and snapshot/restore concerns. Treating it like “just another service” is the fastest way to a Friday-evening cluster red status.

If you take only two patterns into production from this chapter, take these. Outbox-driven indexing over naive dual writes: don't have your application write to the database and the search index in two separate calls — one will eventually fail, the index will drift, and you'll spend a Saturday reconciling. Instead, write the database row and an “outbox” event in the same transaction, and have a downstream worker (or CDC pipeline like Debezium) read that outbox and update the index. The transaction guarantees you can't commit data without queueing the index update. Versioned aliases for zero-downtime reindexing: never write or read against an index name directly — always use an alias. When you need to reindex (mapping change, analyser change, schema evolution), build a new index in parallel, atomically swap the alias once it's caught up, and delete the old one. This turns a downtime-heavy migration into a routine operation.