Tutorial: Search and filter with ES|QL

This is a hands-on introduction to the basics of full-text search and semantic search, using ES|QL.

In this scenario, we're implementing search for a cooking blog. The blog contains recipes with various attributes including textual content, categorical data, and numerical ratings.

You'll need a running Elasticsearch cluster, together with Kibana to use the Dev Tools API Console. Refer to choose your deployment type for deployment options.

Want to get started quickly? Run the following command in your terminal to set up a single-node local cluster in Docker:

curl -fsSL https://elastic.co/start-local | sh

In this tutorial, you'll see ES|QL examples in the following format:

FROM cooking_blog
| WHERE description:"fluffy pancakes"
| LIMIT 1000

If you want to run these queries in the Dev Tools Console, you'll need to use the following syntax:

 POST /_query?format=txt {
  "query": """
    FROM cooking_blog 
    | WHERE description:"fluffy pancakes"  
    | LIMIT 1000 
  """
}

If you'd prefer to use your favorite programming language, refer to Client libraries for a list of official and community-supported clients.

Create the cooking_blog index to get started:

 PUT /cooking_blog 

Now define the mappings for the index:

 PUT /cooking_blog/_mapping {
  "properties": {
    "title": {
      "type": "text",
      "analyzer": "standard", 1
      "fields": { 2
        "keyword": {
          "type": "keyword",
          "ignore_above": 256 3
        }
      }
    },
    "description": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      }
    },
    "author": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      }
    },
    "date": {
      "type": "date",
      "format": "yyyy-MM-dd"
    },
    "category": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      }
    },
    "tags": {
      "type": "text",
      "fields": {
        "keyword": {
          "type": "keyword"
        }
      }
    },
    "rating": {
      "type": "float"
    }
  }
}

1. The standard analyzer is used by default for text fields if an analyzer isn't specified. It's included here for demonstration purposes.
2. Multi-fields are used here to index text fields as both text and keyword data types. This enables both full-text search and exact matching/filtering on the same field. Note that if you used dynamic mapping, these multi-fields would be created automatically.
3. The ignore_above parameter prevents indexing values longer than 256 characters in the keyword field. Again this is the default value, but it's included here for demonstration purposes. It helps to save disk space and avoid potential issues with Lucene's term byte-length limit.

Now you’ll need to index some example blog posts using the Bulk API. Note that text fields are analyzed and multi-fields are generated at index time.

 POST /cooking_blog/_bulk?refresh=wait_for {"index":{"_id":"1"}}
{"title":"Perfect Pancakes: A Fluffy Breakfast Delight","description":"Learn the secrets to making the fluffiest pancakes, so amazing you won't believe your tastebuds. This recipe uses buttermilk and a special folding technique to create light, airy pancakes that are perfect for lazy Sunday mornings.","author":"Maria Rodriguez","date":"2023-05-01","category":"Breakfast","tags":["pancakes","breakfast","easy recipes"],"rating":4.8}
{"index":{"_id":"2"}}
{"title":"Spicy Thai Green Curry: A Vegetarian Adventure","description":"Dive into the flavors of Thailand with this vibrant green curry. Packed with vegetables and aromatic herbs, this dish is both healthy and satisfying. Don't worry about the heat - you can easily adjust the spice level to your liking.","author":"Liam Chen","date":"2023-05-05","category":"Main Course","tags":["thai","vegetarian","curry","spicy"],"rating":4.6}
{"index":{"_id":"3"}}
{"title":"Classic Beef Stroganoff: A Creamy Comfort Food","description":"Indulge in this rich and creamy beef stroganoff. Tender strips of beef in a savory mushroom sauce, served over a bed of egg noodles. It's the ultimate comfort food for chilly evenings.","author":"Emma Watson","date":"2023-05-10","category":"Main Course","tags":["beef","pasta","comfort food"],"rating":4.7}
{"index":{"_id":"4"}}
{"title":"Vegan Chocolate Avocado Mousse","description":"Discover the magic of avocado in this rich, vegan chocolate mousse. Creamy, indulgent, and secretly healthy, it's the perfect guilt-free dessert for chocolate lovers.","author":"Alex Green","date":"2023-05-15","category":"Dessert","tags":["vegan","chocolate","avocado","healthy dessert"],"rating":4.5}
{"index":{"_id":"5"}}
{"title":"Crispy Oven-Fried Chicken","description":"Get that perfect crunch without the deep fryer! This oven-fried chicken recipe delivers crispy, juicy results every time. A healthier take on the classic comfort food.","author":"Maria Rodriguez","date":"2023-05-20","category":"Main Course","tags":["chicken","oven-fried","healthy"],"rating":4.9}

Full-text search involves executing text-based queries across one or more document fields. These queries calculate a relevance score for each matching document, based on how closely the document's content aligns with the search terms. Elasticsearch offers various query types, each with its own method for matching text and relevance scoring.

Here's how to search the description field for "fluffy pancakes":

FROM cooking_blog 1
| WHERE description:"fluffy pancakes" 2
| LIMIT 1000 3

1. Specify the index to search
2. Full-text search with OR logic by default
3. Return up to 1000 results

By default, like the Query DSL match query, ES|QL uses OR logic between terms. This means it will match documents that contain either "fluffy" or "pancakes", or both, in the description field.

FROM cooking_blog
| WHERE description:"fluffy pancakes"
| KEEP title, description, rating 1
| LIMIT 1000

Sometimes you need to require that all search terms appear in the matching documents. Here's how to do that using the function syntax with the operator parameter:

FROM cooking_blog
| WHERE match(description, "fluffy pancakes", {"operator": "AND"}) 1
| LIMIT 1000

1. Require ALL terms to match

This stricter search returns zero hits on our sample data, as no document contains both "fluffy" and "pancakes" in the description.

Sometimes requiring all terms is too strict, but the default OR behavior is too lenient. You can specify a minimum number of terms that must match:

FROM cooking_blog
| WHERE match(title, "fluffy pancakes breakfast", {"minimum_should_match": 2})
| LIMIT 1000

This query searches the title field to match at least 2 of the 3 terms: "fluffy", "pancakes", or "breakfast".

Elasticsearch allows you to semantically search for documents based on the meaning of the text, rather than just the presence of specific keywords. This is useful when you want to find documents that are conceptually similar to a given query, even if they don't contain the exact search terms.

ES|QL supports semantic search when your mappings include fields of the semantic_text type. This example mapping update adds a new field called semantic_description with the type semantic_text:

 PUT /cooking_blog/_mapping {
  "properties": {
    "semantic_description": {
      "type": "semantic_text"
    }
  }
}

Next, index a document with content into the new field:

 POST /cooking_blog/_doc {
  "title": "Mediterranean Quinoa Bowl",
  "semantic_description": "A protein-rich bowl with quinoa, chickpeas, fresh vegetables, and herbs. This nutritious Mediterranean-inspired dish is easy to prepare and perfect for a quick, healthy dinner.",
  "author": "Jamie Oliver",
  "date": "2023-06-01",
  "category": "Main Course",
  "tags": ["vegetarian", "healthy", "mediterranean", "quinoa"],
  "rating": 4.7
}

Once the document has been processed by the underlying model running on the inference endpoint, you can perform semantic searches. Here's an example natural language query against the semantic_description field:

FROM cooking_blog
| WHERE semantic_description:"What are some easy to prepare but nutritious plant-based meals?"
| LIMIT 5 

You can combine full-text and semantic queries. In this example we combine full-text and semantic search with custom weights:

FROM cooking_blog METADATA _score
| WHERE match(semantic_description, "easy to prepare vegetarian meals", { "boost": 0.75 })
    OR match(tags, "vegetarian", { "boost": 0.25 })
| SORT _score DESC
| LIMIT 5

When users enter a search query, they often don't know (or care) whether their search terms appear in a specific field. ES|QL provides ways to search across multiple fields simultaneously:

FROM cooking_blog
| WHERE title:"vegetarian curry" OR description:"vegetarian curry" OR tags:"vegetarian curry"
| LIMIT 1000

This query searches for "vegetarian curry" across the title, description, and tags fields. Each field is treated with equal importance.

However, in many cases, matches in certain fields (like the title) might be more relevant than others. We can adjust the importance of each field using scoring:

FROM cooking_blog METADATA _score 1
| WHERE match(title, "vegetarian curry", {"boost": 2.0}) 2
    OR match(description, "vegetarian curry")
    OR match(tags, "vegetarian curry")
| KEEP title, description, tags, _score 3
| SORT _score DESC 4
| LIMIT 1000

1. Request _score metadata for relevance-based results
2. Title matches are twice as important
3. Include relevance score in results
4. You must explicitly sort by `_score` to see relevance-based results

Filtering allows you to narrow down your search results based on exact criteria. Unlike full-text searches, filters are binary (yes/no) and do not affect the relevance score. Filters execute faster than queries because excluded results don't need to be scored.

FROM cooking_blog
| WHERE category.keyword == "Breakfast" 1
| KEEP title, author, rating, tags
| SORT rating DESC
| LIMIT 1000

1. Exact match using keyword field(case-sensitive)

Note the use of category.keyword here. This refers to the keyword multi-field of the category field, ensuring an exact, case-sensitive match.

Often users want to find content published within a specific time frame:

FROM cooking_blog
| WHERE date >= "2023-05-01" AND date <= "2023-05-31" 1
| KEEP title, author, date, rating
| LIMIT 1000

1. Inclusive date range filter

Sometimes users want to search for exact terms to eliminate ambiguity in their search results:

FROM cooking_blog
| WHERE author.keyword == "Maria Rodriguez" 1
| KEEP title, author, rating, tags
| SORT rating DESC
| LIMIT 1000

1. Exact match on author

Like the term query in Query DSL, this has zero flexibility and is case-sensitive.

Complex searches often require combining multiple search criteria:

FROM cooking_blog METADATA _score
| WHERE rating >= 4.5 1
    AND NOT category.keyword == "Dessert" 2
    AND (title:"curry spicy" OR description:"curry spicy") 3
| SORT _score DESC
| KEEP title, author, rating, tags, description
| LIMIT 1000

1. Numerical filter
2. Exclusion filter
3. Full-text search in multiple fields

For more complex relevance scoring with combined criteria, you can use the EVAL command to calculate custom scores:

FROM cooking_blog METADATA _score
| WHERE NOT category.keyword == "Dessert"
| EVAL tags_concat = MV_CONCAT(tags.keyword, ",") 1
| WHERE tags_concat LIKE "*vegetarian*" AND rating >= 4.5 2
| WHERE match(title, "curry spicy", {"boost": 2.0}) OR match(description, "curry spicy")3
| EVAL category_boost = CASE(category.keyword == "Main Course", 1.0, 0.0) 4
| EVAL date_boost = CASE(DATE_DIFF("month", date, NOW()) <= 1, 0.5, 0.0) 5
| EVAL custom_score = _score + category_boost + date_boost 6
| WHERE custom_score > 0 7
| SORT custom_score DESC
| LIMIT 1000

1. Convert multi-value field to string
2. Wildcard pattern matching
3. Uses full text functions, will update _score metadata field
4. Conditional boost
5. Boost recent content
6. Combine scores
7. Filter based on custom score

This tutorial introduced the basics of search and filtering in ES|QL. Building a real-world search experience requires understanding many more advanced concepts and techniques. Here are some resources once you're ready to dive deeper:

• ES|QL search functions: Explore the full list of search functions available in ES|QL.
• Semantic search: Understand your various options for semantic search in Elasticsearch.
  • The semantic_text workflow: Learn how to use the semantic_text field type for semantic search. This is the recommended approach for most users looking to perform semantic search in Elasticsearch, because it abstracts away the complexity of setting up inference endpoints and models.

• Introducing full text filtering in ES|QL: Overview of text filtering capabilities