The model does not know what is relevant, what is correct, or what you intended. It only sees tokens. This single constraint has deep engineering implications:

Every LLM receives a flat sequence of tokens — system prompt, conversation history, retrieved documents, tool results, and user input are all concatenated into a single integer array. The model attends across all of it simultaneously. There is no "memory" separate from this — the context window is the model's entire working memory for a given call.

The 2023 paper "Lost in the Middle" demonstrated experimentally what practitioners already suspected: LLMs pay disproportionate attention to the beginning and end of the context window. Information placed in the middle is recalled less reliably — even when it's clearly the most relevant.

Every production LLM call has a token budget. Blow it and you get truncation errors, silent degradation, or hard failures. Managing this budget is a core engineering discipline.

Most teams treat retrieval and context construction as the same problem. They are not. Retrieval returns candidates. Context construction decides what to include, what to exclude, how to format it, and where to place it in the window.

Each stage affects quality independently. Teams that only focus on retrieval and ignore construction leave significant quality on the table.

Retrieved context always exceeds your budget. Selection decides what makes the cut.

Ordering is the direct answer to the lost-in-the-middle problem. Where you place retrieved chunks determines how well the model can use them.

Formatting determines how clearly the model can distinguish between context chunks and how reliably it can cite them. Poor formatting causes models to conflate sources, miss boundaries, or fail to cite.

Citation anchoring is the practice of instructing the model to explicitly reference which part of the context it used for each claim. It reduces hallucination, improves verifiability, and allows downstream validation.

Most production LLM quality issues trace back to context construction failures — not model limitations. Know these patterns so you can instrument for them.

The instinct is to include more context — more docs, more history, more detail — to give the model "everything it needs." This is wrong. Beyond a quality threshold, adding more context actively degrades performance.

Sentence filtering is the highest-fidelity compression method: score each sentence against the query, keep only sentences above a relevance threshold. Lossless for the kept sentences; zero hallucination risk since no text is generated.

Abstractive compression uses an LLM (typically a small, cheap one) to rewrite chunks into denser summaries targeted at a specific query. It can achieve the highest compression ratios but introduces a quality dependency: the compressor must not lose or distort key facts.

LLMLingua (Microsoft Research, 2023) is a compression method that uses a small LM to compute token-level perplexity. Tokens that are predictable (low perplexity) carry little information and can be dropped. The remaining tokens form a compressed — but still parseable — prompt.

Compression reduces tokens, cost, and latency — but it trades against information fidelity. Applied incorrectly, it loses the details your system depends on.

A 200-page contract is ~150,000 tokens. A typical RAG chunk budget is 8,000–32,000 tokens. Even a "long context" model with 128K capacity can only fit ~85 pages. Windowing is how you navigate this mismatch for both indexing-time chunking and inference-time context assembly.

Fixed windows split documents into equal-size chunks at fixed token boundaries. Simple to implement, but mid-sentence and mid-paragraph cuts destroy semantic coherence.

Sliding windows add overlap between adjacent chunks. Each chunk shares N tokens with the previous chunk, ensuring that information at boundaries exists in at least one complete chunk.

Semantic chunking splits documents at natural topic boundaries rather than fixed token counts. When embedding similarity drops sharply between adjacent sentences, that's a topic transition — a good split point.

Hierarchical windowing indexes small chunks for precise retrieval but retrieves larger parent chunks for fuller context. You get precision from small-chunk search and coherence from large-chunk content.

In multi-turn conversations, history grows unbounded. At some point, it exceeds the context window. Inference-time windowing decides what to keep and what to drop.

Transformer attention is not uniform — the model allocates attention across all tokens, but that allocation is competitive. When your context contains 20% useful signal and 80% boilerplate, the model must "work harder" to attend to the right parts. Dense context = better answers at lower token cost.

Before tuning, measure. Two practical density metrics: relevance density (what fraction of the context is relevant to the query) and entity density (how many unique named entities per 100 tokens).

When the same information can be represented as prose or as a structured list/table, the structured version is almost always higher density — more facts per token, easier for the model to parse.

Research and production experience consistently show a non-monotonic relationship between context length and quality. Adding more context helps up to a point — then it starts to hurt.

Understanding why models degrade at long context requires understanding position embeddings — how models encode token position in the sequence.

Before committing to a long-context model for production, run the "needle in a haystack" test: place a specific fact (the needle) at various positions in a large document (the haystack) and ask questions that require recalling it. This reveals where each model's attention actually degrades.

When an LLM processes a prompt, it computes key-value (KV) pairs for every token in the attention layers. This computation is expensive and proportional to context length. Prefix caching stores these pre-computed KV pairs — so if the same prefix appears in the next request, the model skips recomputing it entirely.

OpenAI caches prompts automatically — no code changes required. Any prompt prefix of 1,024+ tokens that is reused within ~1 hour gets cached at 50% discount. The usage field in the response reports cached_tokens.

Claude's caching is explicit — you mark exactly which parts of the prompt to cache using cache_control breakpoints. The first request writes the cache (slight cost premium); subsequent requests read it at ~90% discount.

Prompt structure determines cache hit rate. Small changes to prompt ordering or dynamic content placement can mean the difference between 90% hit rate and 0%.

Not all context is equally cache-worthy. The value of caching a piece of context depends on how frequently it's reused, how expensive it is to recompute, and how stable it is over time.

When you have multiple retrieved documents, their order in the context window affects which ones the model uses. Rank-aware ordering is different from simple relevance sorting.

With multiple documents, clear formatting is critical. Labels must be unambiguous, metadata must be useful, and delimiters must prevent content bleeding between sources.

Documents often contradict each other — especially across time (policy updated) or authority level (official docs vs user reports). Without explicit conflict handling, models pick arbitrarily.

Some answers require combining information from multiple documents — no single source is complete. Synthesis prompts encourage the model to explicitly integrate rather than just retrieve.

Retrieval metrics measure the quality of what you put into context — before the model sees it. These are fast, cheap, and deterministic.

RAGAS (Retrieval-Augmented Generation Assessment) provides four core metrics that together cover the full RAG quality surface:

For production systems at scale, you can't manually review every retrieval result. LLM-as-judge provides automated, scalable evaluation that correlates well with human judgments.

Faithfulness measures whether the model's answer is grounded in the context. This is your primary hallucination detector for RAG systems.

In production you need continuous metric tracking — not just offline eval. Log the key signals with every request and aggregate them into a live dashboard.

Context quality must be evaluated, not assumed. A system that "seems to work" in manual testing can have systematic failure modes that only show up under controlled evaluation. Build a test harness that isolates context variables.

Each stage has its own latency budget, failure mode, and optimization surface. Treat them as independent services with SLAs — not a single monolithic function.

Every millisecond of context construction latency adds directly to user-perceived response time. Parallelise all retrieval and processing steps that don't depend on each other.

At scale, context size is your primary cost driver. A system consuming 4,000 input tokens per request at $3/M tokens costs $0.012 per request — at 1M daily requests, that's $12,000/day just in input tokens.

Full observability means you can trace any production failure back to its root cause in the context pipeline: was it a bad retrieval, a compression error, a cache miss, or an LLM failure?

Key signals to trace at every request:

In 2024–2026, the capability gap between frontier models has narrowed dramatically. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro are all excellent — and increasingly similar on most benchmarks. The infrastructure for calling them is standardized. What's left as the primary differentiator?