RAG architecture in production: hybrid search, reranking, evals

Retrieval-augmented generation (RAG) is a pattern that fetches passages from your own documents when a question arrives and hands them to a language model, so the answer rests on text you control rather than on what the model remembers from training. Lewis and colleagues named it in 2020. By the end of this guide you will have a production RAG architecture in six steps: chunks that carry their own context, two indexes over the same chunks, hybrid retrieval merged by rank, a reranker, a prompt that cites its sources, and an eval set that tells you when any of it gets worse.
The demo is the easy part. A vector store, an embedding call and a prompt answer the first ten questions well. Production is where retrieval fails quietly, because the model writes a fluent answer from whatever it was given. Anthropic measured the plain version, embeddings over raw chunks, across codebases, fiction, scientific papers and financial filings: in 5.7% of queries the passage that held the answer was missing from the top 20 results. That is more than one question in twenty answered from the wrong page, in the same confident tone as the right ones.
What you need before you start
First, check that you need retrieval at all. Anthropic's own guidance is that a knowledge base under 200,000 tokens, about 500 pages, can go straight into the prompt, with prompt caching keeping the repeat cost down. RAG earns its complexity when the corpus is larger than that, changes every day, or holds documents that some users may not see. We cover the prompt side in our guide to context engineering.
If you are past that line, have these ready:
- 50 real questions, each labelled with the passage that answers it. Take them from support tickets, sales calls or search logs, never from the documents themselves.
- A store that handles vectors and keywords: Postgres with pgvector and full-text search, or a vector database with a keyword index.
- An embedding model and a reranker, chosen together, since these are the two parts you will swap most often.
- An access model for the documents: which user or tenant may read which file, stored as data rather than in someone's head.
Step 1: Chunk by structure and give every chunk its context
Split documents where their authors split them: headings, sections, paragraphs. Keep a table, a code block or a numbered procedure inside one chunk, since half a table answers nothing. A few hundred tokens per chunk is a sound start; Anthropic's cost estimate assumes 800.
A chunk cut out of its document loses what it is about. The sentence "Revenue grew 3% over the previous quarter" retrieves badly because it doesn't say which company or which quarter. Contextual retrieval fixes this before indexing. A model reads the whole document next to each chunk and writes a prefix of 50 to 100 tokens that places it ("This chunk is from ACME's Q2 2023 filing; revenue in the previous quarter was..."). The prefix goes onto the chunk before it is embedded and before it enters the keyword index. With prompt caching, so the document is read once instead of once per chunk, Anthropic put the one-time cost at $1.02 per million document tokens.
Store each chunk with its metadata: document id, title, section path, last-updated date and access list. Every later step reads them.
Step 2: Index every chunk twice
Embeddings match meaning. They find the passage about cancelling a subscription when the user types "how do I stop paying". They are weak on exact strings: error codes, part numbers, product names, clause numbers, the identifiers that technical and legal questions are made of. A keyword index such as BM25, or Postgres full-text search, matches those exactly.
Anthropic's numbers show each index pulling its own weight. Contextual embeddings alone took the failure rate from 5.7% to 3.7%. Adding a keyword index over the same contextualised chunks took it to 2.9%, a 49% cut from the start.
In Postgres both live in one table: a vector column with an HNSW index for similarity and a tsvector column with a GIN index for keywords, as in Supabase's hybrid search guide. Keep the access columns in that same table too. Step 3 depends on it, and the policies follow the patterns in Supabase RLS at scale.
Step 3: Retrieve wide and merge the lists with reciprocal rank fusion
Run both searches for every question and take a generous candidate list from each, somewhere between 50 and 150 chunks. Apply the access filter inside both queries. Filtering after retrieval leaks: a document the user may not see pushes a permitted one out of the list, and a bug in the filter hands it over whole.
The two lists score on different scales, cosine similarity on one side and BM25 on the other, so adding the scores gives nonsense. Reciprocal rank fusion ignores the scores and uses positions:
score(chunk) = 1 / (k + rank in vector list) + 1 / (k + rank in keyword list)A chunk near the top of both lists wins. A chunk only one search found still counts. The constant k flattens the gap between first and tenth place: Cormack, Clarke and Büttcher set it to 60 in the 2009 SIGIR paper that introduced the method, and Supabase's example function defaults to 50. Give one search more weight when your eval set says so, for instance keywords for a corpus of API references.
Step 4: Rerank the candidates to a short list
Embeddings score the question and the chunk separately. A reranker, usually a cross-encoder, reads them together and judges whether this passage answers this question. It's more accurate and slower, so it only sees the fused candidates. Anthropic passed the top 150 to a reranker and kept 20. The failure rate fell from 2.9% to 1.9%, 67% lower than plain embeddings.
The reranker adds a model call to every question, with its own latency and price. Weigh both against the gain on your eval set. And drop near-duplicate chunks before the cut, so five copies of one paragraph don't fill the list.
Step 5: Put the best passages first and make the model cite them
Order matters inside the prompt. The Lost in the Middle study (Liu et al., TACL 2024) found that models use information best when it sits at the start or the end of the context, and noticeably worse when it sits in the middle. So send fewer passages, put the highest-ranked first, and don't fill the window because it's there. Every extra chunk is also tokens paid on every request, a cost we break down in AI feature token cost.
Label each passage with an id, its title and its date. Then ask for three things in the instructions: answer only from the passages, cite the id after each claim, and say plainly when the passages don't contain the answer. In the interface, render the ids as links to the source. A reader who can open the paragraph behind a sentence checks it in seconds, and "the documents don't cover this" beats a guess written up as fact.
Step 6: Measure retrieval and answers separately
A wrong answer has two possible causes, and they need different fixes. Either retrieval missed the passage, or the model had it and wrote something else. Track one number for each.
- Retrieval recall at k: for each labelled question, is the right passage in the top k results? It's the figure Anthropic reports, turned around.
- Faithfulness: is every claim in the answer supported by the passages that were retrieved? The RAGAS framework defines it alongside answer relevance and context relevance, and scores all three without hand-written reference answers.
Run the 50 questions on every change to the chunking, the contextual prompt, the embedding model, the reranker or the system prompt, and keep the results per release. In production, log the question, the retrieved ids with their scores, the answer and its citations. Read a sample every week. That log is where you find the questions your eval set never imagined, and they become its next entries.
How do you know it works?
- A question that contains an exact identifier, an error code or a SKU, retrieves the chunk that holds it.
- A question the documents can't answer gets "not covered", never an invention.
- A test user without access to a document never receives a chunk from it, in the answer or in the citations.
- Recall at 20 and faithfulness on the eval set ship with every release, next to the previous release's numbers.
Common failures and how to fix them
Answers go stale after a document changes. Re-index the document on every write: delete its old chunks and write the new ones with a fresh date. A nightly rebuild leaves a day of wrong answers.
A new embedding model makes results worse overnight. Vectors from two models aren't comparable. Re-embed the whole corpus into a new, versioned index, run the eval set on it, and switch only when it wins.
Tables and procedures come back in pieces. The splitter cut by character count. Move to structure-aware chunking and keep each table or numbered list in one chunk.
The model ignores the passage that holds the answer. It was ranked eighth of twenty, in the middle of the context. Rerank, cut the list shorter and put the best passage first.
Streaming hides slow retrieval. The first token arrives late and nobody can tell why. Time retrieval and reranking on their own, before generation starts; the streaming side is in Anthropic API in Next.js.
Sources
- Anthropic: Introducing Contextual Retrieval
- Liu et al.: Lost in the Middle: How Language Models Use Long Contexts (TACL 2024)
- Cormack, Clarke, Büttcher: Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods (SIGIR 2009)
- Es et al.: RAGAS: Automated Evaluation of Retrieval Augmented Generation
- Supabase: Hybrid search
- Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
Frequently asked questions
For a small, stable corpus, often not. Anthropic's guidance is that a knowledge base under 200,000 tokens, about 500 pages, can go whole into the prompt, with prompt caching keeping the cost of repeat requests down. Long context stops being enough when the corpus is larger, changes daily, or holds documents some users may not see, since a prompt can't enforce permissions. Every question also pays for every token sent. And the Lost in the Middle results show that more context isn't the same as better use of it. The two combine well: retrieval selects, a generous window reads.
If the product already runs on Postgres, start with pgvector and full-text search in the same table. Hybrid search, access filters and joins to your own data then happen in one query, and there's no second system to keep in sync. A dedicated vector database earns its place when the number of vectors or the query load outgrows what one Postgres instance serves within your latency target, which your own load tests will show. Decide by measurement, and keep retrieval behind one function so the store can change later without touching the rest.
Enforce access inside the retrieval query, never in the prompt. Store the tenant or the access list on every chunk, filter on it in both the vector search and the keyword search, and let row-level security back it up. An instruction like "don't reveal confidential files" is text the model can be talked out of. Scope any cache of answers or retrieved chunks per user or tenant as well, since a shared cache serves one person's results to another. Test with a user who lacks access, and check the citations as well as the answer.
Update a document's chunks every time the document changes, so the index never lags behind the source. Rebuild everything only when something global changes: the chunking rules, the contextual prompt or the embedding model. Build the new index next to the old one under a version number, run the eval set against both, and switch when the new one scores at least as well. Keep the old version until the new one has carried real traffic for a few days, so rolling back is a configuration change.
Related articles
July 30, 2026 · 6 min read
October 28, 2026 · 9 min read
July 12, 2026 · 7 min read