Search needs more than "which documents match". Lucene keeps several structures per field, and which ones exist depends on the field type (chapter 04):

• Inverted index - term to documents, for matching text and exact values.
• Doc values - a column-oriented, on-disk structure (document to value) used for sorting and aggregations. Most field types except analyzed text have it on by default.
• **Stored fields / _source** - the original JSON, returned in search hits. Elasticsearch stores the original document in the _source field.
• Vectors and BKD trees - used for dense vector search and for numeric/geo range queries respectively.

The _analyze API is the single most useful debugging tool in Elasticsearch: it shows exactly which terms an analyzer produces. Request (Kibana Dev Tools style):

The response lists one entry per token with its position and character offsets. Abridged and shown in a compact layout; this is the documented shape, not a captured run:

Note the hyphen split Brown-Foxes into two tokens, the digit became a token, the punctuation disappeared, and everything was lower-cased. The stop word The was kept: the standard analyzer does not remove stop words unless you configure it.

With this analyzer, <b>Running</b> shoes is indexed as the terms run and shoe, and a query for runs shoe produces the same terms, so they match. Test it with POST /products-v1/_analyze and "analyzer": "english_clean" before you rely on it.

The name field above is indexed as text for searching and as name.raw (keyword) for sorting, exact matching and aggregations. This is the standard answer to "I need both full text and exact": one source value, two indexed fields. You search name, you sort and aggregate on name.raw. Sorting or aggregating on an analysed text field is not allowed by default (it has no doc values).

If you index a document with an unknown field and dynamic is true (the default), Elasticsearch adds the field to the mapping, guessing the type from the first value it sees. Strings typically become a text field with a keyword sub-field; numbers and dates are detected by JSON shape and date-pattern rules. Three risks follow:

Elasticsearch counts every mapper toward index.mapping.total_fields.limit: object levels, leaf fields and each multi-field sub-field all count individually (Elastic documentation, 9.x). The limit exists precisely to stop accidental explosions; treat hitting it as a modelling bug, not as a setting to raise.

Read the example as a sentence: find products whose name matches the words, that are Acme, in stock and at most 200, prefer those with the exact phrase "hiking boots", excluding discontinued ones. Only must and should contribute to the relevance score.

Every clause runs in one of two contexts. In query context the question is "how well does this document match?" and the answer is a score. In filter context the question is "does this document match, yes or no?" and no score is computed. Clauses in bool.filter, bool.must_not, and the filter parameter of some queries run in filter context.

• Use filter for everything that is not relevance: brand, status, price range, date window, tenant id. It is cheaper because no scoring is done.
• Filter results can be cached per segment (a bitset of matching documents) and reused by later queries. Elasticsearch decides what to cache based on how often a filter is used; you do not control it per query.
• Put the text clause in must and the rest in filter: the standard shape of a search request.

Aggregations compute summaries over the documents that match the query: counts per category, histograms over time, averages. They are the engine behind facets and dashboards. They run on doc values, which is why you aggregate on keyword, number and date fields, not on analysed text.

"size": 0 means "no hits, aggregations only". Buckets nest: the revenue sum above runs once per daily bucket. Two accuracy caveats worth knowing: terms on a high-cardinality field across many shards is approximate (each shard returns its top terms, then they are merged), and cardinality is an estimate by design.

A node can have one or more roles, set with node.roles. A small cluster often gives every node several roles; a large one specialises them.

Because every shard must return its top from + size hits, deep pagination is expensive (chapter 09), and searching 1000 tiny shards is slower than searching 10 sized shards (chapter 09).

Segments are immutable, so there is no way to change a field inside an existing document. To update a document Elasticsearch marks the old version as deleted (a bit in a per-segment deleted-documents set) and indexes a new complete document into the buffer. The old version still occupies disk space until a merge rewrites that segment. Consequences:

• An update of one field reindexes the whole document (the _update API fetches _source, applies the change and reindexes it). Frequently updating tiny fields of big documents is expensive.
• Deleting or updating many documents does not shrink the index immediately; space is reclaimed by merges.
• The same document id after an update gets a new internal version; concurrency control uses _seq_no and _primary_term (if_seq_no, if_primary_term) for optimistic locking.

Indexing APIs take a refresh parameter. refresh=true refreshes the affected shards immediately, which is expensive if used per request. refresh=wait_for makes the request wait until the next regular refresh makes the change visible. The Elastic docs advise using neither by default and designing the application to tolerate a short delay; if you need a read-your-own-write, fetch the document by id with the get API, which is real-time.

By default a document goes to shard hash(_id) % number_of_shards, which spreads data evenly but means a search must ask all shards. With custom routing (?routing=customer-42) all documents of one customer land on one shard, and a search that passes the same routing value asks only that shard. It is a strong optimisation for multi-tenant data, at the price of possible uneven shard sizes. In Elasticsearch 9.4 and later the docs describe the routing function for new indices as hash(_routing) % number_of_shards; indices created earlier keep the legacy function.

The Elastic documentation (shard sizing, 9.x) gives these as guidance, not hard limits:

Each shard costs memory and file handles whether it holds data or not. The cluster also enforces cluster.max_shards_per_node; hitting it produces a "maximum shards open" error. Hundreds of tiny shards is the single most common cause of an unhealthy cluster: fewer, bigger shards (within the range above) beat many small ones.

Logs and metrics are append-only and have a retention period. A data stream is a named, append-only abstraction over a series of hidden backing indices: writes go to the newest index, reads search all of them. Index Lifecycle Management (ILM) moves backing indices through phases (hot, warm, cold, frozen, delete) and rolls over to a new backing index when the current one reaches a size or age. Retention then becomes "delete the old index", which is instant and cheap.

The rollover threshold max_primary_shard_size is how you keep shards in the 10-50 GB band automatically. The Elastic docs recommend this over choosing a fixed index per day.

from + size paging works by having every shard return from + size hits, so the work grows with the page number. By default you cannot page past 10,000 hits with from and size: the limit is the index setting index.max_result_window, default 10000. For more, use **search_after with a point in time (PIT). The PIT freezes the view of the index so that pages are consistent even if a refresh happens between requests, and the sort values of the last hit tell Elasticsearch where to continue. The Elastic docs state that the scroll API is no longer recommended** for deep pagination.

Always include a tie-breaker field (unique per document, such as an id) in the sort, otherwise documents with equal sort values can be skipped or repeated. Users almost never need page 500 of a search; offering "next page" and refining filters is a better design.

Elasticsearch can store embeddings in a **dense_vector field and answer k-nearest-neighbour** queries: given a query vector, return the documents with the most similar vectors. This is the basis of semantic search and retrieval-augmented generation (RAG). The Elastic docs (9.x) describe dense_vector with the knn search option and approximate search using the HNSW algorithm. Indexing is on by default for dense vectors, and the docs list automatic quantization defaults that vary by version (for example, int8_hnsw in 9.0; bbq_hnsw for vectors of 384 or more dimensions from 9.1; bbq_disk as default from 9.4 where the licence allows). Quantization trades a little accuracy for much lower memory use.

You can combine knn with a text query for hybrid search. Some vector features are marked preview or have licence requirements in the docs: check the page for the exact version you run before relying on them. Two practical notes: vectors are excluded from _source in search responses by default, and embeddings must come from a model you run or call yourself (or an inference endpoint Elasticsearch hosts).

Elasticsearch stores your documents durably enough for a search index, but it is designed as a derived, rebuildable view. Reasons to keep the authoritative copy elsewhere:

A good test: if the whole cluster vanished, could you rebuild it from another store? If yes, you are using it correctly. If no, you have made it your database.

Because Elasticsearch holds a copy, you need a reliable way to push changes from the system of record. Options, from weakest to strongest:

Both outbox and CDC rely on a durable log between the database and the index; page 07 of the messaging section, Event-Driven Patterns, covers the outbox pattern, idempotent consumers and the dual-write problem in detail. The Elasticsearch side has one nice property: **indexing a document with a fixed _id is idempotent**. Replaying the same event just overwrites the same document. Use the database primary key as _id, and use version_type=external with a version taken from the source row when events can arrive out of order.

• Leading wildcard (*boots, *oo*): the term dictionary is sorted by the first characters, so a leading wildcard cannot seek and must test every term. Index with edge_ngram or the wildcard field type if you need it.
• **script queries and scripted sorting** on large result sets: the script runs per document.
• **Deep from + size** and very large size values (chapter 09).
• Aggregations on high-cardinality fields over huge result sets: heavy on memory.

Both are search engines with a shared ancestry. Facts that are verifiable from the projects' own sites:

• OpenSearch describes itself as a fully open source search and analytics suite; its FAQ states that OpenSearch is derived from Elasticsearch 7.10.2 and OpenSearch Dashboards from Kibana 7.10.2.
• Elastic changed Elasticsearch's licence from Apache 2.0 to a dual SSPL / Elastic License 2.0 in 2021 (which is the background of the fork). Elastic later announced it would add the AGPL as another licence option alongside those two, and described Elasticsearch as open source again (Elastic blog, "Elasticsearch Is Open Source. Again!").
• Since the fork the two have diverged: APIs, query features, clients and vector search implementations are not interchangeable. Pick one, use its own client library and docs, and do not assume a feature or a setting from this page exists in OpenSearch.

The licence terms matter if you host the software for others or modify it; read the current licence text rather than relying on a summary, including this one.

• -m 1GB caps container memory; the docs note this means you do not have to set the JVM heap by hand. Machine-learning features need more memory.
• Security is on by default since Elasticsearch 8.0 (xpack.security.enabled defaults to true). On the first start the node auto-configures security: it generates TLS certificates for the transport and HTTP layers and prints a generated password for the elastic user and an enrollment token for Kibana to the console. These are shown only on first start.
• Copy the password. If you lose it, reset it with docker exec -it es01 /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic.

The response is a small JSON document with the cluster name and the version number. --cacert makes curl trust the self-signed CA that the node generated, so TLS is verified instead of disabled with -k.

Use the mapping from chapter 04 saved as mapping.json (the body of the PUT, without the PUT /products-v1 line) and a two-document bulk file. The Bulk API takes newline-delimited JSON: an action line, then the document line, each ending with a newline.

A single node with the default number_of_replicas: 1 reports yellow cluster health: the replica shard has no other node to live on, so it is unassigned. That is expected in a one-node lab. Set the replicas to 0 with PUT /products-v1/_settings and {"index": {"number_of_replicas": 0}} to see green, and read this as a demonstration of why replicas exist, not as a setting to copy to production.

The official Elasticsearch Java API Client is published as co.elastic.clients:elasticsearch-java on Maven Central (its latest release at the time of writing is 9.5.5; the docs example uses 9.5.0 and requires Java 17 or later). Match the client's major version to the server.

The builder-lambda style (b -> b.host(...)) and the nested query builders mirror the JSON DSL of chapter 05. The API shape for connecting (host, usernameAndPassword, sslContext, TransportUtils.sslContextFromCaFingerprint), index, search with match and range(r -> r.number(...)) follows the Elastic Java client documentation; compile it against the client version you choose, because generated builders can vary slightly between releases. In production prefer an API key (.apiKey(...)) over the elastic superuser.

If your application is built on Spring, Spring Data Elasticsearch offers the familiar repository and template abstractions (ElasticsearchOperations, repository interfaces, entity annotations) on top of the Elasticsearch client. It is released on the Spring Data train (the Spring docs list 6.1.x as the current stable line at the time of writing). Each Spring Data Elasticsearch line supports particular Elasticsearch versions, so check the version compatibility matrix in its reference documentation before choosing a Spring Boot and server pair. See also Spring Data and persistence for the wider Spring picture.

• "Add search to our product catalogue." PostgreSQL stays the system of record; an outbox or CDC stream feeds an products alias on Elasticsearch; text in must, filters in filter; facets via aggregations; a reindex runbook; the page degrades to database queries (or a clear message) if search is down.
• "Store 1 TB of logs per day with 30-day retention." Data stream with ILM rollover at about 50 GB primary shard size, hot nodes for ingest, delete phase at 30 days, Bulk API with batching, and few fields mapped explicitly.
• "Search results are slow after we added a tenant filter." Check query shape (filter context), shard count, and whether routing by tenant id can cut the shards searched.