Most of the public argument about retrieval is about which embedding model to use. Most of the failures are about something else: what was indexed, in what pieces, and what happens to the index when something changes. Those decisions are made once, usually early, usually by whoever was building the ingestion script, and they set the ceiling on everything after.
| Decision | Made at | Symptom when it is wrong | Cost to change later |
|---|---|---|---|
| Chunk boundaries | Index time | The answer exists in the corpus and is never retrieved | Re-index everything |
| Embedding model | Index time | Quality falls with no change to the ranker or the prompt | Re-embed the corpus, with a consistency window |
| Filter strategy | Query design | Latency grows with corpus size rather than with the query | Re-partition the store |
| Amount retrieved | Per request | Token cost rises and accuracy does not | A configuration change |
Chunking decides what can be found at all
A retrieval system searches passages, not documents. Chunking is where the passages get made, and a fact split across a boundary is a fact the system cannot retrieve intact — the ranker never sees it whole, so no amount of reranking recovers it.
This survived the growth of context windows, which surprised people. Bigger windows changed how much can be sent, not what gets found. If the chunk containing the answer was never retrieved, the window size is irrelevant; and if the answer to that is to send everything instead, cost and latency rise while precision falls.
It is unglamorous work and it dominates outcomes. A team that spends a week on chunk boundaries and document structure will usually beat one that spends the same week comparing rankers on the same corpus.
Changing the embedding model is a migration
Vectors produced by two different models are not comparable, which means an upgrade is not a configuration change. It is a re-embedding of the entire corpus, and for a large store that is a multi-day job with a consistency window in the middle, during which part of the index answers in one space and part in another.
Teams that version the index and run both during cutover treat this as routine maintenance. Teams that do not discover the problem during an incident, usually one where retrieval quality fell without any change to the ranker or the prompt.
Metadata filters are not free at scale
Pre-filtering and post-filtering look equivalent on a small corpus and diverge sharply once it passes a hundred million vectors. A selective filter — one that eliminates most of the corpus — can turn an approximate index into something close to a scan, depending entirely on how the engine applies it.
The behaviour is documented unevenly across vendors, which makes this a thing to measure rather than to read about. The measurement that matters is on your own selectivity distribution: how often your filters eliminate 90% of the corpus against how often they eliminate 10%, because the engine behaves differently at each end.
How much context to retrieve is a dose
More retrieved material is not uniformly better, and the evidence for that is now quantitative. In an agent setting, injecting a curated set of retrieved guidelines added 5% to the token count — 110K to 116K — while injecting the full set added 51%, reaching 166K; on a stronger model the full set pushed 148K to 263K, a 78% increase. The accuracy effect ranged from 16 points of improvement on one model to none at all on another.
Which means the retrieval budget is a parameter to calibrate per model and per task, not a maximum to fill. A configuration that is clearly correct for one deployment can be pure overhead in the next one.
A checklist for an index you have to live with
- Chunk on document structure rather than on a fixed token count, and check that the facts you expect to retrieve survive the boundaries.
- Version the index from the first day, so that an embedding upgrade is a cutover rather than an outage.
- Measure filter behaviour at your real selectivity, at your real corpus size, before choosing a partitioning scheme.
- Evaluate retrieval separately from generation: a system that answers badly because it retrieved the wrong passage needs a different fix from one that retrieved the right passage and wrote nonsense.
- Treat the amount retrieved as a tunable cost, and measure what each additional passage buys.
None of this is exotic, and that is rather the point. The interesting work in retrieval has moved to the plumbing, because the plumbing is where the failures are.
Questions
- Does a larger context window remove the need for good chunking?
- No. A larger window changes how much can be sent, not what gets found. If the passage containing the answer was never retrieved, the window is irrelevant; and sending everything instead raises cost and latency while lowering precision.
- What happens when I change the embedding model?
- The index becomes invalid, because vectors from two different models are not comparable. The whole corpus has to be re-embedded, which for a large store is a multi-day job with a consistency window in the middle — hence versioning the index and running both during cutover.
- Why do metadata filters slow retrieval down?
- Because a filter that eliminates most of the corpus can push an approximate index towards a scan, depending on whether the engine filters before or after the search. The divergence shows up past roughly a hundred million vectors and is documented unevenly, so it has to be measured on your own selectivity distribution.
- Is more retrieved context always better?
- No. In a measured agent setting the full retrieved set added 51% to 78% more tokens than a curated one, while the accuracy gain ranged from 16 points on one model to nothing on another. The amount retrieved is a dose to calibrate, not a maximum to fill.