Evaluate ranked search results

  • POST also GET /{index}/_rank_eval
  • POST also GET /_rank_eval
Auth

Evaluate the quality of ranked search results over a set of typical search queries.

More about ranking evaluation

Required authorization

Index privileges read

Path parameters

index union string | string[] optional

A comma-separated list of data streams, indices, and index aliases used to limit the request. Wildcard (*) expressions are supported. To target all data streams and indices in a cluster, omit this parameter or use _all or *.

Leave it out to use /_rank_eval.

Query parameters

allow_no_indices boolean · default: true

A setting that does two separate checks on the index expression. If false, the request returns an error (1) if any wildcard expression (including _all and *) resolves to zero matching indices or (2) if the complete set of resolved indices, aliases or data streams is empty after all expressions are evaluated. If true, index expressions that resolve to no indices are allowed and the request returns an empty result.

expand_wildcards union oneOf · default: open

Whether to expand wildcard expression to concrete indices that are open, closed or both.

Supported values include:

  • all: Match any data stream or index, including hidden ones.
  • open: Match open, non-hidden indices. Also matches any non-hidden data stream.
  • closed: Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.
  • hidden: Match hidden data streams and hidden indices. Must be combined with open, closed, or both.
  • none: Wildcard expressions are not accepted.
Values: all open closed hidden none
ignore_unavailable boolean · default: false

If false, the request returns an error if it targets a concrete (non-wildcarded) index, alias, or data stream that is missing, closed, or otherwise unavailable. If true, unavailable concrete targets are silently ignored.

search_type enum

Search operation type

Supported values include:

  • query_then_fetch: Documents are scored using local term and document frequencies for the shard. This is usually faster but less accurate.
  • dfs_query_then_fetch: Documents are scored using global term and document frequencies across all shards. This is usually slower but more accurate.
Values: query_then_fetch dfs_query_then_fetch

Request body

requests object[] required

A set of typical search requests, together with their provided ratings.

id string required

The search request’s ID, used to group result details later.

request object

The query being evaluated.

query {} Query required
size number
ratings object[] required

List of document ratings

_id string required

The document ID.

_index string required

The document’s index. For data streams, this should be the document’s backing index.

rating number required

The document’s relevance with regard to this search request.

template_id string

The search template Id

params map string to object

The search template parameters.

metric object

Definition of the evaluation metric to calculate.

precision object
k number · default: 10

Sets the maximum number of documents retrieved per query. This value will act in place of the usual size parameter in the query.

relevant_rating_threshold number · default: 1

Sets the rating threshold above which documents are considered to be "relevant".

ignore_unlabeled boolean · default: false

Controls how unlabeled documents in the search results are counted. If set to true, unlabeled documents are ignored and neither count as relevant or irrelevant. Set to false (the default), they are treated as irrelevant.

recall object
k number · default: 10

Sets the maximum number of documents retrieved per query. This value will act in place of the usual size parameter in the query.

relevant_rating_threshold number · default: 1

Sets the rating threshold above which documents are considered to be "relevant".

mean_reciprocal_rank object
k number · default: 10

Sets the maximum number of documents retrieved per query. This value will act in place of the usual size parameter in the query.

relevant_rating_threshold number · default: 1

Sets the rating threshold above which documents are considered to be "relevant".

dcg object
k number · default: 10

Sets the maximum number of documents retrieved per query. This value will act in place of the usual size parameter in the query.

normalize boolean · default: false

If set to true, this metric will calculate the Normalized DCG.

expected_reciprocal_rank object
k number · default: 10

Sets the maximum number of documents retrieved per query. This value will act in place of the usual size parameter in the query.

maximum_relevance number required

The highest relevance grade used in the user-supplied relevance judgments.

templates object[]

Query templates that the rated requests can refer to by their template_id. This way, queries that differ only in their parameters don't have to be repeated in requests.

id string required

The ID of the template. A rated request refers to it by this ID in its template_id.

template string required

The search template, either an inline source or the id of a stored search template. It must be a mustache template (the default lang), and it must render to a search request body. Any params set here are ignored: each rated request provides its own params, which are used to render the template.

Response

application/json
metric_score number required

The overall evaluation quality calculated by the defined metric

details map string to object required

The details section contains one entry for every query in the original requests section, keyed by the search request id

<string> object
metric_score number required

The metric_score in the details section shows the contribution of this query to the global quality metric score

took number required

The time in milliseconds it took to execute the search request for this query

unrated_docs object[] required

The unrated_docs section contains an _index and _id entry for each document in the search result for this query that didn’t have a ratings value. This can be used to ask the user to supply ratings for these documents

_id string required
_index string required
hits object[] required

The hits section shows a grouping of the search results with their supplied ratings

hit object required
_id string required
_index string required
_score number required
rating union number | string nullable
metric_details map string to string to object required

The metric_details give additional information about the calculated quality metric (e.g. how many of the retrieved documents were relevant). The content varies for each metric but allows for better interpretation of the results

failures map string to object required