Hybrid search: why vectors alone miss the answer

Hybrid search RAG means running two retrievers on every question, a keyword index such as BM25 and a vector index of embeddings, merging their candidates and passing the merged list through a re-ranker that scores each passage against the actual question. Vector search alone is the default in most first RAG builds, and it is the reason those builds miss part numbers, error codes, clause references and people's names. This article explains what each retriever is good at, why neither is enough, how the merge and re-rank work, what it costs in latency and money, and how to prove the improvement on your own documents.

Why vectors alone miss the answer

An embedding model turns a passage into a list of numbers that captures its meaning. Two passages about refund eligibility end up close together even if they share no words, which is exactly what you want when a user asks "can I get my money back" and the policy says "reimbursement". The cost of that abstraction is that specifics blur. "Error E4021" and "Error E4012" embed almost identically. A part number, an invoice ID, a legal clause reference, a product SKU or a person's surname carries almost no semantic content, so the vector retriever returns passages that are about the same kind of thing rather than the thing itself. Users asking those questions know exactly what they want and get something plausibly adjacent.

Why keywords alone miss it too

BM25, the standard keyword ranking function used by Elasticsearch, OpenSearch and Postgres full-text search, scores passages by how often the query terms appear and how rare those terms are across the corpus. It finds "E4021" precisely. It cannot find "reimbursement" when the user typed "money back", cannot handle a question phrased differently from the document, and is fragile to typos and Hinglish. Keyword search was the state of the art for enterprise search for twenty years and its limits are why people were excited about vectors in the first place.

Keyword plus vector search: what each catches

How hybrid retrieval works

Run both retrievers

The question goes to the keyword index and the vector index in parallel, each returning its top candidates, typically twenty to fifty. Both run in tens of milliseconds on a well-sized index. Postgres with pgvector and its built-in full-text search can do both in one database; Elasticsearch and OpenSearch, Qdrant and Weaviate offer hybrid queries natively. The vector database comparison covers the choice.

Merge the candidate lists

The two lists have incompatible scores, so they are merged by rank rather than score. Reciprocal rank fusion is the common method: each passage gets a score based on its position in each list, and passages that appear high in both rise to the top. Fusion alone already beats either retriever on most golden sets, because it rescues the exact matches vectors dropped and the paraphrases keywords dropped.

Re-rank against the question

The merged list of forty or so candidates goes to a re-ranker: a cross-encoder model that reads the question and each passage together and outputs a relevance score. Unlike embeddings, which encode question and passage separately, the cross-encoder sees both at once and can judge whether this passage actually answers this question. It is slower per passage, which is why it runs on a short list rather than the whole corpus. The top five to ten after re-ranking are what the language model receives. Providers such as Cohere offer hosted re-rankers, and open-weight cross-encoders on Hugging Face run well self-hosted.

Re-ranking RAG: what it changes for the answer

Fewer, better passages produce better answers. A model handed fifteen loosely related chunks pads its answer with material from the wrong ones, and citation accuracy drops. A model handed the five most relevant passages, in order, writes a tighter answer that cites correctly. Re-ranking therefore improves groundedness as well as retrieval, and reduces token cost per answer because less context is sent. The measurement is described in How to measure RAG quality.

Latency and cost

Hybrid retrieval adds one parallel index query and one re-ranking pass. In practice the total retrieval step stays within a few hundred milliseconds for a corpus of hundreds of thousands of chunks, which is invisible next to the language model's generation time. Re-ranker cost per query is small relative to generation, and the reduced context it produces often pays for itself. The place to watch is the keyword index's size on very large corpora and the re-ranker's throughput under concurrent load, both of which are sizing questions rather than architecture ones.

Tuning: weights, filters and metadata

Hybrid systems have a few knobs. The relative weight of keyword versus vector results can be tuned per corpus: technical documentation with many identifiers leans keyword; conversational content leans vector. Metadata filters (document type, date, product, language, permissions) apply before ranking and matter more than any weight; a question about a specific product should never retrieve another product's manual however similar the text. Every knob is set against the golden set, not by feel, and the golden set should contain the exact-match questions that motivated hybrid search in the first place.

Query rewriting is a further step worth knowing about. Before retrieval, a small model can expand the user's question with synonyms, correct obvious typos, or split a two-part question into two retrievals. It helps the keyword side most, because it turns "money back" into "refund reimbursement" for the index. It adds a model call, so it is added when the golden set shows paraphrase failures rather than by default.

Multilingual corpora

Hybrid search works across languages if the embedding model is multilingual and the keyword index is configured per language for stemming and tokenisation. Identifiers and product names are usually the same in every language, which is where the keyword side quietly does most of the work for Hindi or Tamil queries against English documentation.

When hybrid search is not enough

Hybrid search finds passages. It does not follow relationships between documents, answer aggregate questions across a corpus, or compute anything. Questions like "which contracts reference this supplier and what are their notice periods?" need entity extraction and a graph, described in GraphRAG explained. Questions over structured data need natural language data querying. Hybrid search is the foundation these sit on, not a replacement for them.

A worked example

A field-service software company built an in-product assistant over its documentation, release notes and resolved support tickets using vector search alone. Users liked it for how-to questions and stopped trusting it after it repeatedly returned the wrong error code's article, which in a technical product is worse than no article. The golden set confirmed it: questions containing an identifier had markedly lower recall than conceptual questions. Adding a keyword index over the same chunks, fusing by rank and re-ranking the top forty raised recall on identifier questions to the level of the conceptual ones without hurting the latter. Precision improved for both, because the re-ranker discarded the loosely related chunks the vector retriever had been padding with. The assistant is described in the in-app copilot case study.

Team and timeline

Adding hybrid search to an existing RAG system is a one to two week task for a retrieval-focused AI engineer, provided a golden set exists; if it does not, building one is the first week. A new build includes it from the start. It is part of the retrieval and knowledge engineering service, from $14,000 / ₹8.8L, and a three-week ProofRun is the right format for proving the improvement on your corpus before committing to a wider build. Every LLM application we deliver with retrieval uses hybrid search by default. Prices are on the pricing page.

Before you start: a checklist

• Build a golden set that includes identifier-heavy questions, not just conceptual ones
• Confirm your store supports keyword and vector indexes together, or plan a second index
• Decide the metadata filters that must apply before ranking
• Choose a re-ranker: hosted or self-hosted, and check its language coverage
• Set candidate counts per retriever and the final passage count
• Measure recall and precision before and after, per question type
• Check retrieval latency under expected concurrent load
• Plan the re-index if chunking changes as a result

Glossary

• BM25: a keyword ranking function based on term frequency and rarity, used by most full-text search engines
• Embedding: a numeric representation of meaning used for vector similarity search
• Reciprocal rank fusion: merging ranked lists by position so items high in both lists rise
• Cross-encoder: a model that reads question and passage together to score relevance; used as a re-ranker
• Recall: whether the right passage appeared in the retrieved set
• Precision: how much of the retrieved set was relevant
• Metadata filter: a hard constraint (product, date, permission) applied before ranking

Related reading

Why basic RAG fails in production, Chunking strategies for RAG and the pricing page.

Run both retrievers, merge by rank, re-rank against the question, and measure; the identifier questions that embarrassed the vector-only system are the ones hybrid search fixes first.