--- title: Use ES|QL across clusters description: Run a single ES|QL query across multiple Elasticsearch clusters. Covers role and user setup, query syntax, cross-cluster metadata, and enrich and skip-unavailable behavior. url: https://www.elastic.co/elastic/docs-builder/docs/3923/reference/query-languages/esql/esql-cross-clusters products: - Elasticsearch applies_to: - Elastic Cloud Serverless: Unavailable - Elastic Stack: Generally available since 9.1, Preview in 9.0 --- # Use ES|QL across clusters With ES|QL, you can execute a single query across multiple clusters. This page covers remote clusters and cross-cluster search, which are not available in Elastic Cloud Serverless. In Serverless, you can use cross-project search instead. To learn how to query across multiple Serverless projects using CPS, see [Query across Serverless projects with ES|QL](https://www.elastic.co/elastic/docs-builder/docs/3923/reference/query-languages/esql/esql-cross-serverless-projects). ## Prerequisites - ES|QL cross-cluster search requires an [Enterprise subscription](https://www.elastic.co/subscriptions) on both the local (querying) cluster and every remote cluster. - [Remote clusters](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3923/deploy-manage/remote-clusters) must be configured before running ES|QL across clusters. The destination cluster must be configured as a remote cluster on the local cluster, and the remote cluster connection must be correctly established. For setup instructions, refer to [Set up remote clusters](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3923/deploy-manage/remote-clusters#setup). - The local node receiving the query must have the [`remote_cluster_client`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3923/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles#remote-node) node role to connect to the remote clusters. - ES|QL across clusters requires the remote cluster connection to use the [API key-based security model](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3923/deploy-manage/remote-clusters/security-models#api-key). To verify which security model is active, run `GET _remote/info`. When API key authentication is in use, the response includes `"cluster_credentials"`. - For supported version pairings, see [Supported cross-cluster search configurations](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3923/explore-analyze/cross-cluster-search#ccs-supported-configurations). ## Configure roles and users Configure a role with the required local and remote privileges, then assign the role to a user or API key. ### Create a role ES|QL cross-cluster search requires some additional permissions beyond a standard Query DSL search. The following example creates a role that can query remote indices using ES|QL. The final `remote_cluster` privilege is required for remote enrich operations. ```json { "cluster": ["cross_cluster_search"], <1> "indices": [ { "names" : [""], <2> "privileges": ["read"] } ], "remote_indices": [ <3> { "names": [ "logs-*" ], "privileges": [ "read","read_cross_cluster" ], <4> "clusters" : ["my_remote_cluster"] <5> } ], "remote_cluster": [ <6> { "privileges": [ "monitor_enrich" ], "clusters": [ "my_remote_cluster" ] } ] } ``` ### Assign the role to a user You then need a user or API key with the permissions you just created. The following example API call creates a user with the `remote1` role. ```json { "password" : "", "roles" : [ "remote1" ] } ``` For the full reference of cross-cluster role privileges across all deployment types, refer to [Configure privileges for cross-cluster search](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3923/deploy-manage/explore-analyze/cross-cluster-search#configure-privileges-for-ccs-api-key). ### Understand API key privilege limits All cross-cluster requests from the local cluster are bound by the cross-cluster API key's privileges, which are controlled by the remote cluster's administrator. Local roles can only further reduce these permissions; they cannot increase access beyond what the API key allows.For example, if the remote cluster's administrator creates a cross-cluster API key that excludes the indices you need, the role you defined on the local cluster cannot grant access to them. ## Query across multiple clusters In the examples that follow, `cluster_one`, `cluster_two`, and `cluster_three` represent remote clusters that you've already configured on the local cluster where the query runs. The cluster name in each `FROM` clause is the alias you assigned during remote cluster setup. ### Query a single remote cluster In the `FROM` command, specify data streams and indices on remote clusters using the format `:`. For instance, the following ES|QL request queries the `my-index-000001` index on a single remote cluster named `cluster_one`: ```esql FROM cluster_one:my-index-000001 | LIMIT 10 ``` ### Query local and remote clusters This ES|QL request queries the `my-index-000001` index from three clusters: - The local ("querying") cluster - Two remote clusters, `cluster_one` and `cluster_two` ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 ``` ### Query all remote clusters To query the `my-index-000001` index on all remote clusters (`cluster_one`, `cluster_two`, and `cluster_three`), use `*` as the remote cluster name: ```esql FROM *:my-index-000001 | LIMIT 10 ``` #### Include the local cluster The `*:` prefix matches remote clusters only. It does not include the local cluster. To query the index on both the local cluster and all remote clusters, specify the local index separately: ```esql FROM my-index-000001,*:my-index-000001 | LIMIT 10 ``` ## Cross-cluster metadata Use cross-cluster metadata to understand which clusters participated in a query and whether each cluster completed successfully. ### Include metadata in a response Using the `"include_ccs_metadata": true` option, you can request that ES|QL cross-cluster search responses include metadata about the search on each cluster (when the response format is JSON). Here we show an example using the async search endpoint. Cross-cluster search metadata is also present in the synchronous search endpoint response when requested. If the search returns partial results and there are partial shard or remote cluster failures, `_clusters` metadata containing the failures is included in the response regardless of the `include_ccs_metadata` parameter. ```json { "query": """ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index* | STATS COUNT(http.response.status_code) BY user.id | LIMIT 2 """, "include_ccs_metadata": true } ``` Which returns: ```json { "is_running": false, "took": 42, "is_partial": false, "columns" : [ { "name" : "COUNT(http.response.status_code)", "type" : "long" }, { "name" : "user.id", "type" : "keyword" } ], "values" : [ [4, "elkbee"], [1, "kimchy"] ], "_clusters": { "total": 3, "successful": 3, "running": 0, "skipped": 0, "partial": 0, "failed": 0, "details": { "(local)": { "status": "successful", "indices": "blogs", "took": 41, "_shards": { "total": 13, "successful": 13, "skipped": 0, "failed": 0 } }, "cluster_one": { "status": "successful", "indices": "cluster_one:my-index-000001", "took": 38, "_shards": { "total": 4, "successful": 4, "skipped": 0, "failed": 0 } }, "cluster_two": { "status": "successful", "indices": "cluster_two:my-index*", "took": 40, "_shards": { "total": 18, "successful": 18, "skipped": 1, "failed": 0 } } } } } ``` ### Identify skipped clusters You can use the cross-cluster metadata to determine whether any data came back from a cluster. For instance, in the query below, the wildcard expression for `cluster_two` did not resolve to a concrete index (or indices). The cluster is, therefore, marked as *skipped* and the total number of shards searched is set to zero. ```json { "query": """ FROM cluster_one:my-index*,cluster_two:logs* | STATS COUNT(http.response.status_code) BY user.id | LIMIT 2 """, "include_ccs_metadata": true } ``` Which returns: ```json { "is_running": false, "took": 55, "is_partial": true, "columns": [ ... ], "values": [ ... ], "_clusters": { "total": 2, "successful": 1, "running": 0, "skipped": 1, "partial": 0, "failed": 0, "details": { "cluster_one": { "status": "successful", "indices": "cluster_one:my-index*", "took": 38, "_shards": { "total": 4, "successful": 4, "skipped": 0, "failed": 0 } }, "cluster_two": { "status": "skipped", "indices": "cluster_two:logs*", "took": 0, "_shards": { "total": 0, "successful": 0, "skipped": 0, "failed": 0 } } } } } ``` For more on partial results and how cluster status is determined when failures occur, see [Cross-cluster search failures](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3923/explore-analyze/cross-cluster-search#cross-cluster-search-failures). ## Enrich across clusters Choose an enrich mode based on where the enrich policy is available and where you want the command to run. ### Use the default enrich mode Enrich in ES|QL across clusters operates similarly to [local enrich](https://www.elastic.co/elastic/docs-builder/docs/3923/reference/query-languages/esql/commands/enrich). If the enrich policy and its enrich indices are consistent across all clusters, write the enrich command as you would without remote clusters. In this default mode, ES|QL can execute the enrich command on either the local cluster or the remote clusters, aiming to minimize computation or inter-cluster data transfer. Ensuring that the policy exists with consistent data on both the local cluster and the remote clusters is critical for ES|QL to produce a consistent query result. Cross-cluster API keys created in versions prior to 8.15 must be replaced or updated to use the new required permissions for ES|QL cross-cluster enrich with the API key-based security model. Refer to the example in the [Use ES|QL across clusters > Configure roles and users](#esql-ccs-security-model-api-key) section. In the following example, the enrich with `hosts` policy can be executed on either the local cluster or the remote cluster `cluster_one`. ```esql FROM my-index-000001,cluster_one:my-index-000001 | ENRICH hosts ON ip | LIMIT 10 ``` Enrich with an ES|QL query against remote clusters only can also happen on the local cluster. This means the following query requires the `hosts` enrich policy to exist on the local cluster as well. ```esql FROM cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 | ENRICH hosts ON ip ``` ### Enrich with coordinator mode ES|QL provides the enrich `_coordinator` mode to force ES|QL to execute the enrich command on the local cluster. Use this mode when the enrich policy is not available on the remote clusters or maintaining consistency of enrich indices across clusters is challenging. ```esql FROM my-index-000001,cluster_one:my-index-000001 | ENRICH _coordinator:hosts ON ip | SORT host_name | LIMIT 10 ``` Enrich with the `_coordinator` mode usually increases inter-cluster data transfer and workload on the local cluster. ### Enrich with remote mode ES|QL also provides the enrich `_remote` mode to force ES|QL to execute the enrich command independently on each remote cluster where the target indices reside. This mode is useful for managing different enrich data on each cluster, such as detailed information of hosts for each region where the target (main) indices contain log events from these hosts. In the following example, the `hosts` enrich policy is required to exist on all remote clusters: the "querying" cluster (as local indices are included), the remote cluster `cluster_one`, and `cluster_two`. ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | ENRICH _remote:hosts ON ip | SORT host_name | LIMIT 10 ``` A `_remote` enrich cannot be executed after a [`STATS`](https://www.elastic.co/elastic/docs-builder/docs/3923/reference/query-languages/esql/commands/stats-by) command. The following example would result in an error: ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | STATS COUNT(*) BY ip | ENRICH _remote:hosts ON ip | SORT host_name | LIMIT 10 ``` ### Multiple enrich commands You can include multiple enrich commands in the same query with different modes. ES|QL attempts to execute them accordingly. For example, this query performs two enrich commands, first with the `hosts` policy on any cluster and then with the `vendors` policy on the local cluster. ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | ENRICH hosts ON ip | ENRICH _coordinator:vendors ON os | LIMIT 10 ``` A `_remote` enrich command can't be executed after a `_coordinator` enrich command. The following example would result in an error. ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | ENRICH _coordinator:hosts ON ip | ENRICH _remote:vendors ON os | LIMIT 10 ``` ## LOOKUP JOIN across clusters ES|QL [`LOOKUP JOIN`](/elastic/docs-builder/docs/3923/reference/query-languages/esql/esql-lookup-join#cross-cluster-support) is supported in cross-cluster queries. By default, ES|QL resolves the lookup index on every cluster in the query and each cluster joins against its own local index with that name. In this case, the lookup index must exist on every cluster being queried. This follows the same pattern as [remote mode Enrich](#esql-enrich-remote). Elastic Stack: Planned If the lookup index is missing from one or more remote clusters, use [coordinator mode](/elastic/docs-builder/docs/3923/reference/query-languages/esql/esql-lookup-join#coordinator-mode) to join against a local cluster lookup index copy. ## Excluding clusters or indices from ES|QL query Use exclusions in the `FROM` command to omit an entire remote cluster or specific indices from a query. ### Exclude a cluster To exclude an entire cluster, prefix the cluster alias with a minus sign in the `FROM` command, for example: `-my_cluster:*`: ```esql FROM my-index-000001,cluster*:my-index-000001,-cluster_three:* | LIMIT 10 ``` ### Exclude an index To exclude a specific remote index, prefix the index with a minus sign in the `FROM` command, such as `my_cluster:-my_index`: ```esql FROM my-index-000001,cluster*:my-index-*,cluster_three:-my-index-000001 | LIMIT 10 ``` Elastic Stack: Generally available since 9.5 The form `-my_cluster:my_index` is also accepted as an alternative for `my_cluster:-my_index`. For example, the following query is equivalent to the one above: ```esql FROM my-index-000001,cluster*:my-index-*,-cluster_three:my-index-000001 | LIMIT 10 ``` The two forms have different semantics: `-my_cluster:*` is a *cluster-level* exclusion that requires the cluster to have been included by a preceding expression (`-cluster_three:*` on its own is rejected), while `-my_cluster:` is an *index-level* exclusion equivalent to `my_cluster:-` and may appear standalone. ## Skipping problematic remote clusters Cross-cluster search for ES|QL behavior when there are problems connecting to or running a query on remote clusters differs between versions. Remote clusters are configured with the `skip_unavailable: true` setting by default. With this setting, clusters are marked as `skipped` or `partial` rather than causing queries to fail in the following scenarios: - The remote cluster is disconnected from the querying cluster, either before or during the query execution. - The remote cluster does not have the requested index, or it is not accessible due to security settings. - An error happened while processing the query on the remote cluster. The `partial` status means the remote query either has errors or was interrupted by an explicit user action, but some data can be returned.Even when `skip_unavailable` is set to `true`, queries fail if none of the specified indices exist. For example, the following queries fail: ```esql FROM cluster_one:missing-index | LIMIT 10 FROM cluster_one:missing-index* | LIMIT 10 FROM cluster_one:missing-index*,cluster_two:missing-index | LIMIT 10 ``` If a remote cluster disconnects from the querying cluster, cross-cluster search for ES|QL sets it to `skipped` and continues the query with other clusters, unless the remote cluster's `skip_unavailable` setting is set to `false`, in which case the query fails. For broader context on the `skip_unavailable` setting, including how it interacts with `allow_no_indices` and `ignore_unavailable`, see [Optional remote clusters](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3923/explore-analyze/cross-cluster-search#skip-unavailable-clusters). ## Query across clusters during an upgrade You can still search a remote cluster while performing a rolling upgrade on the local cluster. However, the local node receiving the query must have an "upgrade from" and "upgrade to" version that is compatible with the remote cluster's gateway node. Running multiple versions of Elasticsearch in the same cluster beyond the duration of an upgrade is not supported. For more information about upgrades, see [Upgrading Elasticsearch](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3923/deploy-manage/upgrade/deployment-or-cluster). ## Limitations ### Inference endpoints The `RERANK` and `COMPLETION` commands, and the `TEXT_EMBEDDING` and `EMBEDDING` functions, always run on the cluster that receives the query. Their inference endpoint must exist on that same cluster, even when every document comes from a remote. If it does not, the query fails with `Inference endpoint not found`. Elastic Stack: Preview since 9.4 This also applies to `MMR` when its query vector comes from an inference function rather than from a literal or an existing vector field. Querying a [`semantic_text`](https://www.elastic.co/elastic/docs-builder/docs/3923/reference/elasticsearch/mapping-reference/semantic-text) field works the other way around: each cluster generates query embeddings using its own `search_inference_id`, so the endpoint must exist on the cluster that holds the data. A query that does both needs endpoints on both clusters. The following reads no local data at all, and still requires the `semantic_text` field's endpoint on `my_remote_cluster` and the `rerank-1` endpoint on the local cluster: ```esql FROM my_remote_cluster:my-index | WHERE semantic_field : "query text" | RERANK "query text" ON title WITH { "inference_id": "rerank-1" } ``` Querying a `semantic_text` field across clusters requires every participating cluster to run 9.3.1 or later. Against an older remote the query fails with an error naming the required version. ### Relevance scores across clusters Full-text scores are computed per shard, from that shard's own term statistics. ES|QL has no equivalent of the Query DSL `dfs_query_then_fetch` search type, so nothing reconciles those statistics — within a cluster or across them. The shards of one index usually hold similar enough data for this to go unnoticed. Separate clusters often do not. Scores from different clusters are therefore not directly comparable, and the effect grows as the corpora diverge in size or vocabulary. A term that is rare on one cluster and common on another produces systematically higher scores on the first, regardless of how relevant its documents actually are. This affects `SORT _score`, any `WHERE` clause comparing `_score` against a fixed threshold, and the set of candidates that reaches a later `RERANK`. Only lexical scoring is affected by corpus statistics. `KNN` and dense `semantic_text` fields score on vector similarity, and sparse `semantic_text` fields on stored per-token weights. Their scores do not vary with corpus statistics, but they are comparable across clusters only when the fields use the same embedding model and equivalent similarity or scoring configuration. Elastic Stack: Generally available since 9.5, Elastic Stack: Preview from 9.2 to 9.4 This carries into [`FUSE`](https://www.elastic.co/elastic/docs-builder/docs/3923/reference/query-languages/esql/commands/fuse). The default `RRF` method compares ranks rather than score magnitudes, so it is less exposed to the difference, though not immune: those ranks are themselves produced by the scores above. `LINEAR` combines the score values directly, and rescales them only if you set its `normalizer` option, which defaults to `none`. Prefer `RRF` in cross-cluster queries. ### `dense_vector` fields `dense_vector` fields are only available when every cluster in the query runs 9.2.0 or later. If any cluster is older, the fields are treated as unsupported for the whole query, including on clusters that do support them. A query that uses `KNN`, `TO_DENSE_VECTOR`, or one of the `V_*` similarity functions keeps the fields enabled instead, and any cluster that cannot handle the type reports a failure for its own shards. By default the remaining clusters still return their results, marked partial. If `allow_partial_results` is disabled, that failure fails the whole query. ### `LOOKUP JOIN` after other commands A cross-cluster `LOOKUP JOIN` cannot follow a command that runs on the querying cluster. That is any pipeline-breaking command — `STATS`, `INLINE STATS`, `SORT`, `LIMIT`, `TS_INFO`, and `METRICS_INFO` among them — and any command that only ever runs there, such as `CHANGE_POINT`, `FORK`, `FUSE`, `RERANK`, `COMPLETION`, `MMR`, and coordinator-side `ENRICH`. Refer to [`LOOKUP JOIN` coordinator mode](/elastic/docs-builder/docs/3923/reference/query-languages/esql/esql-lookup-join#coordinator-mode) for the workaround and its own constraints. ### Views across clusters ### Datasets across clusters - Elastic Stack: Experimental in 9.5 You can query datasets created through [ES|QL Data Federation](https://www.elastic.co/elastic/docs-builder/docs/3923/reference/query-languages/esql/esql-data-federation) on the local cluster only. Querying a dataset on a remote cluster returns an error. ## Query across Serverless projects - Elastic Cloud Serverless: Preview You can use cross-project search (CPS) to query across multiple linked serverless projects. To learn more, refer to [Query across Serverless projects](https://www.elastic.co/elastic/docs-builder/docs/3923/reference/query-languages/esql/esql-cross-serverless-projects).