Books

RedLectern topic

Embeddings and search

Learned representations · vector search · keyword search · hybrid retrieval · reranking.

For: Builders · technical learners

After this page you can: Choose and test a retrieval method for a collection.

Page reviewed

Part of: Artificial Intelligence › Retrieval-augmented generation

About Embeddings and search

Search methods represent and rank collection items in different ways. A learned embedding maps an item such as text or an image to a numerical vector. Vector search compares these representations; semantic search uses that comparison to find items related in meaning. Keyword search instead relies on textual terms, which can make it useful when a query contains an exact name, identifier, or phrase.

There is no method to choose by label alone. Build a small set of real queries and the items that should be retrieved, then compare the results. Check whether relevant items appear, where they rank, and whether latency and operating constraints fit your use. Hybrid retrieval combines result lists from methods such as dense vector and lexical search; a reranker can then reorder a shortlist. The sections below offer ways to try these choices and examine the results.

Start here

route

I'm new to embeddings and search

Read the semantic search overview, then try its small-corpus example to see queries and documents compared by similarity.

Start with: Semantic Search

route

Start with keyword search

See how text features and BM25 support lexical matching and ranking.

Start with: Feature extraction from text

route

Compare hybrid search

Learn how dense and sparse search results can be fused, and when an exact term may matter.

Start with: Hybrid Search: Dense and Sparse

Learn the foundations

1.Representations and retrieval methodsAn embedding is a numerical representation, and vector search finds items by comparing their representations.

An embedding is a numerical representation, and vector search finds items by comparing their representations. A vector database can store vectors and support similarity search; a vector index organizes them to speed up candidate search. Approximate nearest neighbor (ANN) methods can trade some precision for faster or smaller searches, so compare their returned candidates against an exact baseline when that matters.

Semantic search uses represented meaning, while keyword search matches textual terms. BM25 is a lexical ranking method with term-frequency normalization and document-length normalization. Hybrid search combines results from different retrieval methods, for example dense vectors with sparse lexical vectors. Exact terms, such as a code or identifier, can be important to test alongside paraphrased queries.

  • Similarity settings

    Documents BM25 and its term-frequency and document-length normalization settings.

  • Faiss documentation

    Introduces vector indexes and similarity search, including a precision and speed trade-off.

Embedding

A learned numerical representation of text, images, or other items.

Vector database

A datastore that stores vector representations and supports similarity search.

Vector index

A data structure that organizes vectors to speed up searches among them.

Retrieval that uses represented meaning rather than matching only textual terms.

Retrieval based on matching textual terms.

Retrieval that combines methods such as keyword and vector search.

BM25

A keyword-ranking method that uses term-frequency and document-length normalization.

Approximate nearest neighbor

A method for efficiently finding vectors likely to be close without checking every candidate.

Try it

For a help collection, compare these queries: “reset a forgotten password” and “ERR-1042.” Which retrieval method or methods would you test first for each, and what would you inspect in the results?

Test semantic search for the paraphrased password query and keyword search for the exact error code. Also test hybrid search on both. Inspect whether the intended help item appears and where it ranks.

What Are EmbeddingsWhat Is A Vector Database

2.Choose a method for your collectionChoose by the failures you observe in your own queries.

Choose by the failures you observe in your own queries. Semantic search can retrieve relevant passages expressed with different words; lexical search can match terms that must be exact. Hybrid search runs more than one method and fuses their result lists, so it is a candidate to test when the query set contains both paraphrases and exact terms. Fusion methods differ: reciprocal rank fusion uses result positions, while score-based combinations depend on scores that may be on different scales.

Make a small labeled set of queries paired with relevant items before changing settings. Compare each method on the same queries, and keep a held-out portion for checking whether a tuning choice generalizes. Decide what counts as improvement for your use: relevant results in the shortlist, better top ranks, or a latency target.

Try it

Create four test queries for a collection: one paraphrase, one exact identifier, one query containing both, and one query that should return no result. For each, name a method to compare and what a useful result would contain.

Compare semantic search on the paraphrase, keyword search on the identifier, and both methods plus hybrid search on the mixed query. For the no-result query, inspect whether each method returns irrelevant candidates. Record the relevant items and their ranks.

3.Rerank candidates and check resultsRetrieval first produces candidates; a reranker can score that shortlist again against the query and change its order.

Retrieval first produces candidates; a reranker can score that shortlist again against the query and change its order. This adds a ranking stage, so compare it with the original results on the same labeled queries. Keep candidate retrieval and reranking distinct in your evaluation: if a relevant item never enters the candidate set, a later reorder cannot surface it.

Before tuning, decide what better retrieval means for the workload and choose a metric and labeled query set that can reveal that change. Check both ranking quality and practical costs such as latency. When you tune fusion weights on labeled queries, use separate tuning and evaluation queries; measuring on the tuning queries alone can inflate the apparent result.

A RAG answer can still be wrong. Inspect five parts separately: retrieval quality, answer faithfulness, factual correctness, citation support, and permissions.

Try it

A relevant item appears at rank 18 in your retrieved candidates. You rerank only the first 10. What should you check before concluding that the reranker failed?

Check whether the item entered the top 10 candidate set. If it did not, the reranker could not reorder it. Measure candidate recall separately from the reranked ordering.

Latest

No recent updates are available.

See the sources list

Books

Glossary

8 terms on this page.

Browse the AI glossary

Questions and comments