kNN search on Elasticsearch

A k-nearest neighbor (kNN) search finds the k nearest vectors to a query vector using a similarity metric such as cosine or L2 norm. In Elasticsearch, kNN is the primary way to query dense_vector fields after you store embeddings.

• Search

  • Semantic text search: Find documents that match the meaning of a query, even when the wording differs.
  • Image and video similarity: Search across text, images, audio, or video to find visually or semantically similar content.

• Recommendations

  • Product recommendations: Surface items similar to what a user is viewing or has interacted with.
  • Collaborative filtering: Match users or items based on shared behavior or preference patterns in vector space.
  • Personalized content discovery: Suggest articles, media, or other content tailored to individual user interests.

• Analysis

  • Anomaly detection: Flag records whose vectors sit unusually far from their nearest neighbors.
  • Pattern matching: Find near-duplicates, suspicious matches, or other patterns that exact matching would miss.

To run a kNN search in Elasticsearch:

• Your data must be vectorized. You can use an NLP model in Elasticsearch or generate vectors outside Elasticsearch.

  • Use the dense_vector field type for dense vectors.
  • Query vectors must have the same dimension and be created with the same model as the document vectors.
  • Already have vectors? Refer to Bring your own dense vectors.

• Required index privileges:

  • create_index or manage to create an index with a dense_vector field
  • create, index, or write to add data
  • read to search the index

Elasticsearch provides several ways to perform kNN search. Which one you use depends on your field type and whether you need to combine kNN with other queries.

• Approximate kNN

  • Best for most production workloads where low latency and scale matter more than perfect recall.
  • Uses graph-based or clustered index structures to find similar vectors quickly without scoring every document in the index.
  • Refer to Approximate kNN search for mapping, indexing, and search examples.

• Exact, brute-force kNN

  • Best for small datasets, pre-filtered subsets, or when you need precise scoring without approximate indexing.
  • Uses a script_score query with a vector function to compute similarity against each matching document.
  • Refer to Exact kNN search for search examples.

Approximate kNN offers low latency and good accuracy, while exact kNN guarantees accurate results but does not scale well for large datasets. With exact kNN, a script_score query must scan each matching document to compute the vector function, which can result in slow search speeds. However, you can improve latency by using a query to limit the number of matching documents passed to the function. If you filter your data to a small subset of documents, you can get good search performance using this approach.

• Approximate kNN search: Learn how to map, index, and query dense_vector fields for fast, scalable approximate kNN search.
• Build search queries: Learn how to construct approximate kNN queries for filtering, hybrid retrieval, semantic search, multiple vector fields, and similarity thresholds.
• Nested kNN search: Learn how to run approximate kNN search on nested vectors for passage retrieval, filtering, inner hits, and chunked content.
• Optimize performance and accuracy: Learn how to tune search speed, recall, vector storage, quantization, and rescoring for approximate kNN search.
• Exact kNN search: Learn how to run exact brute-force kNN search with script_score queries for small datasets or precise scoring.
• Vector search in Elasticsearch: Learn the core concepts and terminology for vector search in Elasticsearch, including embeddings, field types, and how vector retrieval fits with other search strategies.
• Knn query: API reference for the knn query, including parameters, query_vector_builder options, and usage with dense_vector and semantic_text fields.