--- title: Use ES|QL across clusters description: With ES|QL, you can execute a single query across multiple clusters. Cross-cluster search requires remote clusters. To set up remote clusters, see Remote... url: https://www.elastic.co/elastic/docs-builder/docs/3016/reference/query-languages/esql/esql-cross-clusters products: - Elasticsearch applies_to: - Elastic Cloud Serverless: Unavailable - Elastic Stack: Generally available since 9.1, Preview in 9.0 --- # Use ES|QL across clusters With ES|QL, you can execute a single query across multiple clusters. ## Prerequisites - Cross-cluster search requires remote clusters. To set up remote clusters, see [*Remote clusters*](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/deploy-manage/remote-clusters). To ensure your remote cluster configuration supports cross-cluster search, see [Supported cross-cluster search configurations](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/explore-analyze/cross-cluster-search#ccs-supported-configurations). - For full cross-cluster search capabilities, the local and remote cluster must be on the same [subscription level](https://www.elastic.co/subscriptions). - The local coordinating node must have the [`remote_cluster_client`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles#remote-node) node role. - If you use [sniff mode](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/deploy-manage/remote-clusters/remote-clusters-self-managed#sniff-mode), the local coordinating node must be able to connect to seed and gateway nodes on the remote cluster. We recommend using gateway nodes capable of serving as coordinating nodes. The seed nodes can be a subset of these gateway nodes. - If you use [proxy mode](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/deploy-manage/remote-clusters/remote-clusters-self-managed#proxy-mode), the local coordinating node must be able to connect to the configured `proxy_address`. The proxy at this address must be able to route connections to gateway and coordinating nodes on the remote cluster. - Cross-cluster search requires different security privileges on the local cluster and remote cluster. See [Configure privileges for cross-cluster search](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/deploy-manage/remote-clusters/remote-clusters-cert#remote-clusters-privileges-ccs) and [*Remote clusters*](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/deploy-manage/remote-clusters). ## Security model Elasticsearch supports two security models for cross-cluster search (CCS): - [TLS certificate authentication](#esql-ccs-security-model-certificate) - [API key authentication](#esql-ccs-security-model-api-key) To check which security model is being used to connect your clusters, run `GET _remote/info`. If you’re using the API key authentication method, you’ll see the `"cluster_credentials"` key in the response. ### TLS certificate authentication Use [API key authentication](#esql-ccs-security-model-api-key) instead. TLS certificate authentication secures remote clusters with mutual TLS. This could be the preferred model when a single administrator has full control over both clusters. We generally recommend that roles and their privileges be identical in both clusters. Refer to [TLS certificate authentication](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/deploy-manage/remote-clusters/remote-clusters-cert) for prerequisites and detailed setup instructions. ### API key authentication The following information pertains to using ES|QL across clusters with the [**API key based security model**](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/deploy-manage/remote-clusters/remote-clusters-api-key). You’ll need to follow the steps on that page for the **full setup instructions**. This page only contains additional information specific to ES|QL. API key based cross-cluster search (CCS) enables more granular control over allowed actions between clusters. This may be the preferred model when you have different administrators for different clusters and want more control over who can access what data. In this model, cluster administrators must explicitly define the access given to clusters and users. You will need to: - Create an API key on the **remote cluster** using the [Create cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) API or using the [Kibana API keys UI](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/deploy-manage/api-keys/elasticsearch-api-keys). - Add the API key to the keystore on the **local cluster**, as part of the steps in [configuring the local cluster](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/deploy-manage/remote-clusters/remote-clusters-api-key#remote-clusters-security-api-key-local-actions). All cross-cluster requests from the local cluster are bound by the API key’s privileges. Using ES|QL with the API key based security model requires some additional permissions that may not be needed when using the traditional query DSL based search. The following example API call creates a role that can query remote indices using ES|QL when using the API key based security model. The final privilege, `remote_cluster`, is required to allow remote enrich operations. ```json { "cluster": ["cross_cluster_search"], <1> "indices": [ { "names" : [""], <2> "privileges": ["read"] } ], "remote_indices": [ <3> { "names": [ "logs-*" ], "privileges": [ "read","read_cross_cluster" ], <4> "clusters" : ["my_remote_cluster"] <5> } ], "remote_cluster": [ <6> { "privileges": [ "monitor_enrich" ], "clusters": [ "my_remote_cluster" ] } ] } ``` You will then need a user or API key with the permissions you created above. The following example API call creates a user with the `remote1` role. ```json { "password" : "", "roles" : [ "remote1" ] } ``` Remember that all cross-cluster requests from the local cluster are bound by the cross cluster API key’s privileges, which are controlled by the remote cluster’s administrator. Cross cluster API keys created in versions prior to 8.15.0 will need to replaced or updated to add the new permissions required for ES|QL with ENRICH. ## Remote cluster setup Once the security model is configured, you can add remote clusters. The following [cluster update settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) API request adds three remote clusters: `cluster_one`, `cluster_two`, and `cluster_three`. ```json { "persistent": { "cluster": { "remote": { "cluster_one": { "seeds": [ "35.238.149.1:9300" ], "skip_unavailable": true }, "cluster_two": { "seeds": [ "35.238.149.2:9300" ], "skip_unavailable": false }, "cluster_three": { <1> "seeds": [ "35.238.149.3:9300" ] } } } } } ``` ## Query across multiple clusters In the `FROM` command, specify data streams and indices on remote clusters using the format `:`. For instance, the following ES|QL request queries the `my-index-000001` index on a single remote cluster named `cluster_one`: ```esql FROM cluster_one:my-index-000001 | LIMIT 10 ``` Similarly, this ES|QL request queries the `my-index-000001` index from three clusters: - The local ("querying") cluster - Two remote clusters, `cluster_one` and `cluster_two` ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 ``` Likewise, this ES|QL request queries the `my-index-000001` index from all remote clusters (`cluster_one`, `cluster_two`, and `cluster_three`): ```esql FROM *:my-index-000001 | LIMIT 10 ``` ## Cross-cluster metadata Using the `"include_ccs_metadata": true` option, users can request that ES|QL cross-cluster search responses include metadata about the search on each cluster (when the response format is JSON). Here we show an example using the async search endpoint. Cross-cluster search metadata is also present in the synchronous search endpoint response when requested. If the search returns partial results and there are partial shard or remote cluster failures, `_clusters` metadata containing the failures will be included in the response regardless of the `include_ccs_metadata` parameter. ```json { "query": """ FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index* | STATS COUNT(http.response.status_code) BY user.id | LIMIT 2 """, "include_ccs_metadata": true } ``` Which returns: ```json { "is_running": false, "took": 42, "is_partial": false, "columns" : [ { "name" : "COUNT(http.response.status_code)", "type" : "long" }, { "name" : "user.id", "type" : "keyword" } ], "values" : [ [4, "elkbee"], [1, "kimchy"] ], "_clusters": { "total": 3, "successful": 3, "running": 0, "skipped": 0, "partial": 0, "failed": 0, "details": { "(local)": { "status": "successful", "indices": "blogs", "took": 41, "_shards": { "total": 13, "successful": 13, "skipped": 0, "failed": 0 } }, "cluster_one": { "status": "successful", "indices": "cluster_one:my-index-000001", "took": 38, "_shards": { "total": 4, "successful": 4, "skipped": 0, "failed": 0 } }, "cluster_two": { "status": "successful", "indices": "cluster_two:my-index*", "took": 40, "_shards": { "total": 18, "successful": 18, "skipped": 1, "failed": 0 } } } } } ``` The cross-cluster metadata can be used to determine whether any data came back from a cluster. For instance, in the query below, the wildcard expression for `cluster-two` did not resolve to a concrete index (or indices). The cluster is, therefore, marked as *skipped* and the total number of shards searched is set to zero. ```json { "query": """ FROM cluster_one:my-index*,cluster_two:logs* | STATS COUNT(http.response.status_code) BY user.id | LIMIT 2 """, "include_ccs_metadata": true } ``` Which returns: ```json { "is_running": false, "took": 55, "is_partial": true, "columns": [ ... ], "values": [ ... ], "_clusters": { "total": 2, "successful": 1, "running": 0, "skipped": 1, "partial": 0, "failed": 0, "details": { "cluster_one": { "status": "successful", "indices": "cluster_one:my-index*", "took": 38, "_shards": { "total": 4, "successful": 4, "skipped": 0, "failed": 0 } }, "cluster_two": { "status": "skipped", "indices": "cluster_two:logs*", "took": 0, "_shards": { "total": 0, "successful": 0, "skipped": 0, "failed": 0 } } } } } ``` ## Enrich across clusters Enrich in ES|QL across clusters operates similarly to [local enrich](https://www.elastic.co/elastic/docs-builder/docs/3016/reference/query-languages/esql/commands/enrich). If the enrich policy and its enrich indices are consistent across all clusters, simply write the enrich command as you would without remote clusters. In this default mode, ES|QL can execute the enrich command on either the local cluster or the remote clusters, aiming to minimize computation or inter-cluster data transfer. Ensuring that the policy exists with consistent data on both the local cluster and the remote clusters is critical for ES|QL to produce a consistent query result. Enrich in ES|QL across clusters using the API key based security model was introduced in version **8.15.0**. Cross cluster API keys created in versions prior to 8.15.0 will need to replaced or updated to use the new required permissions. Refer to the example in the [API key authentication](#esql-ccs-security-model-api-key) section. In the following example, the enrich with `hosts` policy can be executed on either the local cluster or the remote cluster `cluster_one`. ```esql FROM my-index-000001,cluster_one:my-index-000001 | ENRICH hosts ON ip | LIMIT 10 ``` Enrich with an ES|QL query against remote clusters only can also happen on the local cluster. This means the below query requires the `hosts` enrich policy to exist on the local cluster as well. ```esql FROM cluster_one:my-index-000001,cluster_two:my-index-000001 | LIMIT 10 | ENRICH hosts ON ip ``` ### Enrich with coordinator mode ES|QL provides the enrich `_coordinator` mode to force ES|QL to execute the enrich command on the local cluster. This mode should be used when the enrich policy is not available on the remote clusters or maintaining consistency of enrich indices across clusters is challenging. ```esql FROM my-index-000001,cluster_one:my-index-000001 | ENRICH _coordinator:hosts ON ip | SORT host_name | LIMIT 10 ``` Enrich with the `_coordinator` mode usually increases inter-cluster data transfer and workload on the local cluster. ### Enrich with remote mode ES|QL also provides the enrich `_remote` mode to force ES|QL to execute the enrich command independently on each remote cluster where the target indices reside. This mode is useful for managing different enrich data on each cluster, such as detailed information of hosts for each region where the target (main) indices contain log events from these hosts. In the below example, the `hosts` enrich policy is required to exist on all remote clusters: the `querying` cluster (as local indices are included), the remote cluster `cluster_one`, and `cluster_two`. ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | ENRICH _remote:hosts ON ip | SORT host_name | LIMIT 10 ``` A `_remote` enrich cannot be executed after a [`STATS`](https://www.elastic.co/elastic/docs-builder/docs/3016/reference/query-languages/esql/commands/stats-by) command. The following example would result in an error: ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | STATS COUNT(*) BY ip | ENRICH _remote:hosts ON ip | SORT host_name | LIMIT 10 ``` ### Multiple enrich commands You can include multiple enrich commands in the same query with different modes. ES|QL will attempt to execute them accordingly. For example, this query performs two enriches, first with the `hosts` policy on any cluster and then with the `vendors` policy on the local cluster. ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | ENRICH hosts ON ip | ENRICH _coordinator:vendors ON os | LIMIT 10 ``` A `_remote` enrich command can’t be executed after a `_coordinator` enrich command. The following example would result in an error. ```esql FROM my-index-000001,cluster_one:my-index-000001,cluster_two:my-index-000001 | ENRICH _coordinator:hosts ON ip | ENRICH _remote:vendors ON os | LIMIT 10 ``` ## Excluding clusters or indices from ES|QL query To exclude an entire cluster, prefix the cluster alias with a minus sign in the `FROM` command, for example: `-my_cluster:*`: ```esql FROM my-index-000001,cluster*:my-index-000001,-cluster_three:* | LIMIT 10 ``` To exclude a specific remote index, prefix the index with a minus sign in the `FROM` command, such as `my_cluster:-my_index`: ```esql FROM my-index-000001,cluster*:my-index-*,cluster_three:-my-index-000001 | LIMIT 10 ``` ## Skipping problematic remote clusters Cross-cluster search for ES|QL behavior when there are problems connecting to or running query on remote clusters differs between versions. Remote clusters are configured with the `skip_unavailable: true` setting by default. With this setting, clusters are marked as `skipped` or `partial` rather than causing queries to fail in the following scenarios: - The remote cluster is disconnected from the querying cluster, either before or during the query execution. - The remote cluster does not have the requested index, or it is not accessible due to security settings. - An error happened while processing the query on the remote cluster. The `partial` status means the remote query either has errors or was interrupted by an explicit user action, but some data may be returned.Queries will still fail when `skip_unavailable` is set `true`, if none of the specified indices exist. For example, the following queries will fail: ```esql FROM cluster_one:missing-index | LIMIT 10 FROM cluster_one:missing-index* | LIMIT 10 FROM cluster_one:missing-index*,cluster_two:missing-index | LIMIT 10 ``` If a remote cluster disconnects from the querying cluster, cross-cluster search for ES|QL will set it to `skipped` and continue the query with other clusters, unless the remote cluster's `skip_unavailable` setting is set to `false`, in which case the query will fail. ## Query across clusters during an upgrade You can still search a remote cluster while performing a rolling upgrade on the local cluster. However, the local coordinating node’s "upgrade from" and "upgrade to" version must be compatible with the remote cluster’s gateway node. Running multiple versions of Elasticsearch in the same cluster beyond the duration of an upgrade is not supported. For more information about upgrades, see [Upgrading Elasticsearch](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3016/deploy-manage/upgrade/deployment-or-cluster).