ES|QL for search tutorial

Serverless Stack

This hands-on tutorial covers full-text search, semantic search, hybrid search, vector search, and AI-powered search capabilities using ES|QL.

In this scenario, we're implementing search for a cooking blog. The blog contains recipes with various attributes including textual content, categorical data, and numerical ratings.

You need a running Elasticsearch cluster, together with Kibana to use the Dev Tools API Console. Refer to choose your deployment type for deployment options.

Want to get started quickly? Run the following command in your terminal to set up a single-node local cluster in Docker:

curl -fsSL https://elastic.co/start-local | sh
		

In this tutorial, ES|QL examples are displayed in the following format:

FROM cooking_blog
| WHERE description:"fluffy pancakes"
| LIMIT 1000
		

If you want to run these queries in the Dev Tools Console, you need to use the following syntax:

				POST /_query?format=txt
					{
  "query": """
    FROM cooking_blog 
    | WHERE description:"fluffy pancakes"  
    | LIMIT 1000 
  """
}
		

If you'd prefer to use your favorite programming language, refer to Client libraries for a list of official and community-supported clients.

Create the cooking_blog index to get started:

				PUT /cooking_blog
		

Now define the mappings for the index:

				PUT /cooking_blog/_mapping
					{
  "properties": {
    "title": {
      "type": "text",
      "analyzer": "standard",
      "fields": {
        "keyword": {
          "type": "keyword",
          "ignore_above": 256
        }
      }
    },
    "description": {
      "type": "text"
    },
    "author": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      }
    },
    "date": {
      "type": "date",
      "format": "yyyy-MM-dd"
    },
    "category": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      }
    },
    "tags": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      }
    },
    "rating": {
      "type": "float"
    }
  }
}
		

1. The standard analyzer is used by default for text fields. It's included here explicitly for demonstration purposes. To know more about analyzers, refer to Anatomy of an analyzer.
2. Multi-fields enable both full-text search and exact matching/filtering on the same field. The title, author, category, and tags fields are declared with both text and keyword data types. The text version allows full-text search, while the keyword version enables exact filtering. If you use dynamic mapping, these multi-fields will be created automatically.
3. The ignore_above parameter prevents indexing values longer than 256 characters in the keyword field. This is the default value and it's included here for demonstration purposes. For more information, refer to the ignore_above parameter.
4. keyword fields store exact values and are used for filtering, sorting, and aggregations. Unlike text fields, they are not analyzed. Use keyword when you need exact matching (e.g., filtering by category or finding a specific author name), but use text when you want to search within longer content like descriptions.

Next, you’ll need to index some example blog posts using the Bulk API. Note that text fields are analyzed and multi-fields are generated at index time.

				POST /cooking_blog/_bulk?refresh=wait_for
					{"index":{"_id":"1"}}
{"title":"Perfect Pancakes: A Fluffy Breakfast Delight","description":"Learn the secrets to making the fluffiest pancakes, so amazing you won't believe your tastebuds. This recipe uses buttermilk and a special folding technique to create light, airy pancakes that are perfect for lazy Sunday mornings.","author":"Maria Rodriguez","date":"2023-05-01","category":"Breakfast","tags":["pancakes","breakfast","easy recipes"],"rating":4.8}
{"index":{"_id":"2"}}
{"title":"Spicy Thai Green Curry: A Vegetarian Adventure","description":"Dive into the flavors of Thailand with this vibrant green curry. Packed with vegetables and aromatic herbs, this dish is both healthy and satisfying. Don't worry about the heat - you can easily adjust the spice level to your liking.","author":"Liam Chen","date":"2023-05-05","category":"Main Course","tags":["thai","vegetarian","curry","spicy"],"rating":4.6}
{"index":{"_id":"3"}}
{"title":"Classic Beef Stroganoff: A Creamy Comfort Food","description":"Indulge in this rich and creamy beef stroganoff. Tender strips of beef in a savory mushroom sauce, served over a bed of egg noodles. It's the ultimate comfort food for chilly evenings.","author":"Emma Watson","date":"2023-05-10","category":"Main Course","tags":["beef","pasta","comfort food"],"rating":4.7}
{"index":{"_id":"4"}}
{"title":"Vegan Chocolate Avocado Mousse","description":"Discover the magic of avocado in this rich, vegan chocolate mousse. Creamy, indulgent, and secretly healthy, it's the perfect guilt-free dessert for chocolate lovers.","author":"Alex Green","date":"2023-05-15","category":"Dessert","tags":["vegan","chocolate","avocado","healthy dessert"],"rating":4.5}
{"index":{"_id":"5"}}
{"title":"Crispy Oven-Fried Chicken","description":"Get that perfect crunch without the deep fryer! This oven-fried chicken recipe delivers crispy, juicy results every time. A healthier take on the classic comfort food.","author":"Maria Rodriguez","date":"2023-05-20","category":"Main Course","tags":["chicken","oven-fried","healthy"],"rating":4.9}
		

Full-text search involves executing text-based queries across one or more document fields. In this section, you'll start with simple text matching and build up to understanding how search results are ranked.

ES|QL provides multiple functions for full-text search, including MATCH, MATCH_PHRASE, and QSTR. For basic text matching, you can use either:

1. Full match function syntax: match(field, "search terms")
2. Compact syntax using the match operator ":": field:"search terms"

Both are equivalent for basic matching and can be used interchangeably. The compact syntax is more concise, while the function syntax allows for more configuration options. We use the compact syntax in most examples for brevity.

Refer to the MATCH function reference docs for advanced parameters available with the function syntax.

Let's start with the simplest possible search - looking for documents that contain specific words:

FROM cooking_blog
| WHERE description:"fluffy pancakes"
| LIMIT 1000
		

This query searches the description field for documents containing either "fluffy" OR "pancakes" (or both). By default, ES|QL uses OR logic between search terms, so it matches documents that contain any of the specified words.

You can specify the exact fields to include in your results using the KEEP command:

FROM cooking_blog
| WHERE description:"fluffy pancakes"
| KEEP title, description, rating
| LIMIT 1000
		

This helps reduce the amount of data returned and focuses on the information you need.

Search results can be ranked based on how well they match your query. To calculate and use relevance scores, you need to explicitly request the _score metadata:

FROM cooking_blog METADATA _score
| WHERE description:"fluffy pancakes"
| KEEP title, description, _score
| SORT _score DESC
| LIMIT 1000
		

Notice two important things:

1. METADATA _score tells ES|QL to include relevance scores in the results
2. SORT _score DESC orders results by relevance (highest scores first)

If you don't include METADATA _score in your query, you won't see relevance scores in your results. This means you won't be able to sort by relevance or filter based on relevance scores.

Without explicit sorting, results aren't ordered by relevance even when scores are calculated. If you want the most relevant results first, you must sort by _score, by explicitly using SORT _score DESC or SORT _score ASC.

Sometimes you need exact matches rather than full-text search. Use the .keyword field for case-sensitive exact matching:

FROM cooking_blog
| WHERE category.keyword == "Breakfast"
| KEEP title, category, rating
| SORT rating DESC
| LIMIT 1000
		

1. Exact match (case-sensitive)

This is fundamentally different from full-text search - it's a binary yes/no filter that doesn't affect relevance scoring.

Now that you understand basic searching, explore how to control the precision of your text matches.

By default, searches with match use OR logic between terms. To require ALL terms to match, use the function syntax with the operator parameter to specify AND logic:

FROM cooking_blog
| WHERE match(description, "fluffy pancakes", {"operator": "AND"})
| LIMIT 1000
		

This stricter search returns zero hits on our sample data, as no document contains both "fluffy" and "pancakes" in the description.

Sometimes requiring all terms is too strict, but the default OR behavior is too lenient. You can specify a minimum number of terms that must match:

FROM cooking_blog
| WHERE match(title, "fluffy pancakes breakfast", {"minimum_should_match": 2})
| LIMIT 1000
		

This query searches the title field to match at least 2 of the 3 terms: "fluffy", "pancakes", or "breakfast".

When you need to find documents containing an exact sequence of words, use the MATCH_PHRASE function:

FROM cooking_blog
| WHERE MATCH_PHRASE(description, "rich and creamy")
| KEEP title, description
| LIMIT 1000
		

This query only matches documents where the words "rich and creamy" appear exactly in that order in the description field.

Once you're comfortable with basic search precision, learn how to search across multiple fields and use query string syntax for complex patterns.

The QSTR function enables powerful search patterns using a compact query language. It's ideal for when you need wildcards, fuzzy matching, and boolean logic in a single expression:

FROM cooking_blog
| WHERE QSTR("description: (fluffy AND pancak*) OR (creamy -vegan)")
| KEEP title, description
| LIMIT 1000
		

Query string syntax lets you:

• Use boolean operators: AND, OR, - (NOT)
• Apply wildcards: pancak* matches "pancake" and "pancakes"
• Enable fuzzy matching: pancake~1 for typo tolerance
• Group terms: (thai AND curry) OR pasta
• Search exact phrases: "fluffy pancakes"
• Search across fields: QSTR("(title:pancake OR description:pancake) AND creamy")

When users enter a search query, they often don't know (or care) whether their search terms appear in a specific field. You can search across multiple fields simultaneously:

FROM cooking_blog
| WHERE title:"vegetarian curry" OR description:"vegetarian curry" OR tags:"vegetarian curry"
| LIMIT 1000
		

This query searches for "vegetarian curry" across the title, description, and tags fields. Each field is treated with equal importance.

In many cases, matches in certain fields (like the title) might be more relevant than others. You can adjust the importance of each field using boost scoring:

FROM cooking_blog METADATA _score
| WHERE MATCH(title, "vegetarian curry", {"boost": 2.0})
    OR MATCH(description, "vegetarian curry")
    OR MATCH(tags, "vegetarian curry")
| KEEP title, description, tags, _score
| SORT _score DESC
| LIMIT 1000
		

1. Title matches are twice as important

Filtering allows you to narrow down your search results based on exact criteria. Unlike full-text searches, filters are binary (yes/no) and do not affect the relevance score. Filters execute faster than queries because excluded results don't need to be scored.

FROM cooking_blog
| WHERE category.keyword == "Breakfast"
| KEEP title, author, rating, tags
| SORT rating DESC
| LIMIT 1000
		

1. Exact match using keyword field

Often users want to find content published within a specific time frame:

FROM cooking_blog
| WHERE date >= "2023-05-01" AND date <= "2023-05-31"
| KEEP title, author, date, rating
| LIMIT 1000
		

1. Inclusive date range filter

Filter by ratings or other numerical values:

FROM cooking_blog
| WHERE rating >= 4.5
| KEEP title, author, rating, tags
| SORT rating DESC
| LIMIT 1000
		

1. Only highly-rated recipes

Find recipes by a specific author:

FROM cooking_blog
| WHERE author.keyword == "Maria Rodriguez"
| KEEP title, author, rating, tags
| SORT rating DESC
| LIMIT 1000
		

1. Exact match on author

Real-world search often requires combining multiple types of criteria. This section shows how to build sophisticated search experiences by combining the techniques you've learned.

Mix filters, full-text search, and custom scoring in a single query:

FROM cooking_blog METADATA _score
| WHERE rating >= 4.5
    AND NOT category.keyword == "Dessert"
    AND (title:"curry spicy" OR description:"curry spicy")
| SORT _score DESC
| KEEP title, author, rating, tags, description
| LIMIT 1000
		

1. Numerical filter
2. Exclusion filter
3. Full-text search in multiple fields

For complex relevance scoring with combined criteria, you can use the EVAL command to calculate custom scores:

FROM cooking_blog METADATA _score
| WHERE NOT category.keyword == "Dessert"
| EVAL tags_concat = MV_CONCAT(tags.keyword, ",")
| WHERE tags_concat LIKE "*vegetarian*" AND rating >= 4.5
| WHERE MATCH(title, "curry spicy", {"boost": 2.0}) OR MATCH(description, "curry spicy")
| EVAL category_boost = CASE(category.keyword == "Main Course", 1.0, 0.0)
| EVAL date_boost = CASE(DATE_DIFF("month", date, NOW()) <= 1, 0.5, 0.0)
| EVAL custom_score = _score + category_boost + date_boost
| WHERE custom_score > 0
| SORT custom_score DESC
| LIMIT 1000
		

1. Convert multi-value field to string
2. Wildcard pattern matching
3. Conditional boost
4. Boost recent content
5. Combine scores
6. Filter based on custom score

Elasticsearch allows you to semantically search for documents based on the meaning of the text, rather than just the presence of specific keywords. This is useful when you want to find documents that are conceptually similar to a given query, even if they don't contain the exact search terms.

ES|QL supports semantic search when your mappings include fields of the semantic_text type. This example mapping update adds a new field called semantic_description with the type semantic_text:

				PUT /cooking_blog/_mapping
					{
  "properties": {
    "semantic_description": {
      "type": "semantic_text"
    }
  }
}
		

Next, index a document with content into the new field:

				POST /cooking_blog/_doc
					{
  "title": "Mediterranean Quinoa Bowl",
  "semantic_description": "A protein-rich bowl with quinoa, chickpeas, fresh vegetables, and herbs. This nutritious Mediterranean-inspired dish is easy to prepare and perfect for a quick, healthy dinner.",
  "author": "Jamie Oliver",
  "date": "2023-06-01",
  "category": "Main Course",
  "tags": ["vegetarian", "healthy", "mediterranean", "quinoa"],
  "rating": 4.7
}
		

Once the document has been processed by the underlying model running on the inference endpoint, you can perform semantic searches. Use the MATCH function (or the ":" operator shorthand) on semantic_text fields to perform semantic queries:

FROM cooking_blog METADATA _score
| WHERE semantic_description:"I need plant-based meals that aren't difficult to make"
| SORT _score DESC
| LIMIT 5
		

When you use MATCH or ":" on a semantic_text field, ES|QL automatically performs a semantic search rather than a lexical search. This means the query finds documents based on semantic similarity, not just keyword matching.

Serverless Preview Stack Preview 9.2.0

Hybrid search combines different search strategies (like lexical and semantic search) to leverage the strengths of each approach. Use FORK and FUSE to execute different search strategies in parallel, then merge and score the combined results.

FUSE supports different merging algorithms, namely Reciprocal Rank Fusion (RRF) and linear combination.

FROM cooking_blog METADATA _id, _index, _score
| FORK (
    WHERE title:"vegetarian curry"
    | SORT _score DESC
    | LIMIT 5
) (
    WHERE semantic_description:"easy vegetarian curry recipes"
    | SORT _score DESC
    | LIMIT 5
)
| FUSE
| KEEP title, description, rating, _score
| SORT _score DESC
| LIMIT 5
		

FROM cooking_blog METADATA _id, _index, _score
| FORK (
    WHERE title:"vegetarian curry"
    | SORT _score DESC
    | LIMIT 5
) (
    WHERE semantic_description:"easy vegetarian curry recipes"
    | SORT _score DESC
    | LIMIT 5
)
| FUSE LINEAR WITH { "weights": { "fork1": 0.7, "fork2": 0.3 } }
| KEEP title, description, rating, _score
| SORT _score DESC
| LIMIT 5
		

Serverless Preview Stack Preview 9.2.0

ES|QL provides commands for AI-powered search enhancements and text generation tasks. You will need to set up an inference endpoint with appropriate models to follow along with these examples.

The RERANK command re-scores search results using inference models for improved relevance. This is particularly useful for improving the ranking of initial search results by applying more sophisticated semantic understanding.

FROM cooking_blog METADATA _score
| WHERE description:"vegetarian recipes"
| SORT _score DESC
| LIMIT 100
| RERANK "healthy quick meals" ON description
| LIMIT 5
| KEEP title, description, _score
		

1. Perform initial lexical search
2. Limit to top 100 results for reranking
3. Re-score using the default reranking model (Elastic Rerank)
4. Return top 5 results after reranking

You can configure RERANK to use a specific reranking model. See the RERANK command documentation for configuration options.

The COMPLETION command sends prompts to a Large Language Model (LLM) for text generation tasks like question answering, summarization, or translation.

FROM cooking_blog
| WHERE category.keyword == "Dessert"
| LIMIT 3
| EVAL prompt = CONCAT("Summarize this recipe: ", title, " - ", description)
| COMPLETION summary = prompt WITH { "inference_id": "my_llm_endpoint" }
| KEEP title, summary
		

This enables you to:

• Generate summaries of search results
• Answer questions about document content
• Translate or transform text using LLMs
• Create dynamic content based on your data

The KNN function finds the k nearest vectors to a query vector through approximate search on indexed dense_vector fields.

First, add a dense_vector field and index a document with a pre-computed vector. This example uses a toy example for readability: the vector has only 3 dimensions.

				PUT /cooking_blog/_mapping
					{
  "properties": {
    "recipe_vector": {
      "type": "dense_vector",
      "dims": 3
    }
  }
}
				POST /cooking_blog/_doc/6
					{
  "title": "Quick Vegan Stir-Fry",
  "description": "A fast and healthy vegan stir-fry with tofu and vegetables",
  "recipe_vector": [0.5, 0.8, 0.3],
  "rating": 4.6
}
		

Then search using a query vector:

FROM cooking_blog METADATA _score
| WHERE KNN(recipe_vector, [0.5, 0.8, 0.3])
| SORT _score DESC
| LIMIT 5
		

Serverless Preview Stack Planned

The TEXT_EMBEDDING function generates dense vector embeddings from text input at query time using an inference model.

FROM cooking_blog METADATA _score
| WHERE KNN(recipe_vector, TEXT_EMBEDDING("vegan recipe", "my_embedding_model"))
| SORT _score DESC
| LIMIT 5
		

This approach gives you full control over vector embeddings, unlike semantic_text which handles embeddings automatically.

This tutorial introduced search, filtering, and AI-powered capabilities in ES|QL. Here are some resources once you're ready to dive deeper:

• Search with ES|QL: Learn about all search capabilities in ES|QL.
• Semantic search: Understand your various options for semantic search in Elasticsearch.
  • The semantic_text workflow: Learn how to use the semantic_text field type for semantic search. This is the recommended approach for most users looking to perform semantic search in Elasticsearch, because it abstracts away the complexity of setting up inference endpoints and models.
• Inference API: Set up inference endpoints for the RERANK, COMPLETION, and TEXT_EMBEDDING commands.

• ES|QL, you know for Search: Introducing scoring and semantic search
• Introducing full text filtering in ES|QL: Overview of ES|QL's text filtering capabilities