Personal Operating System
Jersey · GMTEst. 2026Get in touch →
Practitioner notes · AI applied to finance & operations

Jewel Nguyen

Deep read · June 21, 2026 · source published May 8, 2026

Most retrieval failures start with the chunk, not the retriever

Source: “You're doing RAG wrong — fix the unit, not the retriever” by @akshay_pachaar · May 8, 2026
Almost every retrieval system inherits one unexamined assumption: that a chunk of text is the right thing to embed. The argument here is that this single choice causes most retrieval failures — the ones people then try to patch downstream with rerankers and threshold-tuning. The fix is to embed a question-answer packet — one claim, its validated answer, plus governance fields — instead of a raw text window, and to dedupe near-identical copies so the index holds one canonical version of each fact. The headline number: collapsing 2,042 chunks to 1,200 canonical blocks halved the word count yet improved vector accuracy 13.55% — less data, sharper retrieval.

The assumption nobody checks

Every RAG pipeline starts the same way: take a document, slice it into chunks, embed each chunk. The whole argument is that this first step — treating a chunk of text as the unit of knowledge — is a parsing convenience that quietly became a retrieval assumption, and it is wrong.

A chunk (a fixed-size window of text, cut wherever the token count runs out) is a structurally neutral container. It does not know where an idea begins or ends, which version of a document it came from, or who is allowed to see it. The splitter cuts on length, not on meaning.

So you end up embedding half a table, or a conclusion with its supporting argument chopped off, and the model has no way to know what is missing. The claim: years of RAG tuning — rerankers, hybrid search, threshold-tweaking — are all downstream patches for one upstream mistake.

Three ways the blind chunk fails

The damage breaks into three. First, no idea boundary: a chunk can hold half a thought, so retrieval returns a claim stripped of the context that made it true.

Second, the version problem. Real corpora hold the same document in a dozen near-identical copies across different folders and tools, so top-K retrieval returns five copies of the same paragraph, current and deprecated mixed together, and the model blends them into an answer that is confidently wrong.

Third, no metadata. Because a chunk carries no fields, there is nowhere in the data itself to attach who-can-see-this, so access rules get bolted onto the orchestrator, disconnected from the content they govern. Most stacks have nothing between the document parser and the vector store, and that empty gap is where all three problems compound.

The better unit: a question-answer packet

The fix is to make the unit structurally explicit. Instead of embedding a window of prose, you embed a claim: one question, its one validated answer of two or three sentences, and typed governance fields, all on the same object.

One open-source preprocessing layer, Blockify, calls this unit an IdeaBlock. The key move: your queries are already questions, so when your index stores answers to questions, the match becomes structural, not just semantic. You stop hoping the right paragraph floats to the top and start matching a question to its answer directly.

The geometry backs it. In one benchmark the average cosine distance (how far apart two meanings sit in the embedding space; smaller is a closer match) from query to best block was 0.1585 for these packets against 0.3624 for naive chunks, a 2.29× tighter match.

The counterintuitive part: less data, more accuracy

The result worth sitting up for: shrinking the corpus made retrieval better. After building 2,042 raw blocks, the pipeline ran semantic deduplication — clustering blocks that are 80–85% similar and merging each cluster into one canonical block — across three to five rounds.

That collapsed 2,042 blocks to 1,200 and halved the word count, from 88,877 words to 44,537. The distilled set then beat the undistilled one by 13.55% on vector accuracy.

Why redundancy hurts: fifteen near-duplicates of the same paragraph create fifteen competing vectors crowded into the same region of the embedding space, so retrieval distributes probability mass across all of them, pulling the score of the canonical version down. Collapse them into one and the signal sharpens. The line to keep: your vector index isn't a hard drive you want to fill, it's a retrieval surface, and redundancy degrades it.

A new layer in the stack

This generalises into an architecture prediction: RAG stacks are starting to grow a distillation layer between parsing and vectorization, the same way web stacks grew a CDN layer between origin and browser.

The seven-stage Blockify pipeline (scope, ingest, context-aware chunk, dedupe, auto-tag, human-validate, export) is one implementation, but the principle is portable: you can build it yourself with clustering, LLM-based summarization, and schema enforcement.

Once governance lives on the unit, two things you cannot easily do with raw chunks become free. A single fact updates in one place: change one block, and every query gets the corrected answer on the next request. And access control becomes a typed field on the block rather than orchestrator logic. The closing claim is deliberately narrow: the fix is not a better retrieval algorithm, it's a better unit, and fixing it at the data layer pays off more than tuning retrieval ever will.

Vocabulary

If you're building — what to watch for

Reading it critically

Read the original → ← Back to the Reading Desk