---
title: Add remote clusters using API key authentication
description: API key authentication enables a local cluster to authenticate itself with a remote cluster via a cross-cluster API key. The API key needs to be created...
url: https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/remote-clusters/remote-clusters-api-key
products:
- Elasticsearch
applies_to:
- Self-managed Elastic deployments: Generally available
---
# Add remote clusters using API key authentication
API key authentication enables a local cluster to authenticate itself with a remote cluster via a [cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key). The API key needs to be created by an administrator of the remote cluster. The local cluster is configured to provide this API key on each request to the remote cluster. The remote cluster verifies the API key and grants access, based on the API key’s privileges.
All cross-cluster requests from the local cluster are bound by the API key’s privileges, regardless of local users associated with the requests. For example, if the API key only allows read access to `my-index` on the remote cluster, even a superuser from the local cluster is limited by this constraint. This mechanism enables the remote cluster’s administrator to have full control over who can access what data with cross-cluster search and/or cross-cluster replication. The remote cluster’s administrator can be confident that no access is possible beyond what is explicitly assigned to the API key.
On the local cluster side, not every local user needs to access every piece of data allowed by the API key. An administrator of the local cluster can further configure additional permission constraints on local users so each user only gets access to the necessary remote data. Note it is only possible to further reduce the permissions allowed by the API key for individual local users. It is impossible to increase the permissions to go beyond what is allowed by the API key.
In this model, cross-cluster operations use [a dedicated server port](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/configuration-reference/networking-settings#remote_cluster.port) (remote cluster interface) for communication between clusters, which defaults to port `9443`. A remote cluster must enable this port for local clusters to connect. Configure Transport Layer Security (TLS) for this port to maximize security (as explained in [Establish trust with a remote cluster](#remote-clusters-security-api-key)).
The local cluster must trust the remote cluster on the remote cluster interface. This means that the local cluster trusts the remote cluster’s certificate authority (CA) that signs the server certificate used by the remote cluster interface. When establishing a connection, all nodes from the local cluster that participate in cross-cluster communication verify certificates from nodes on the other side, based on the TLS trust configuration.
To add a remote cluster using API key authentication:
1. [Review the prerequisites](#remote-clusters-prerequisites-api-key)
2. [Establish trust with a remote cluster](#remote-clusters-security-api-key)
3. [Connect to a remote cluster](#remote-clusters-connect-api-key)
4. (Optional) [Configure strong identity verification](#remote-cluster-strong-verification)
5. [Configure roles and users](#remote-clusters-privileges-api-key)
If you run into any issues, refer to [Troubleshooting](https://www.elastic.co/elastic/docs-builder/docs/3028/troubleshoot/elasticsearch/remote-clusters).
## Prerequisites
- The Elasticsearch security features need to be enabled on both clusters, on every node. Security is enabled by default. If it’s disabled, set `xpack.security.enabled` to `true` in [`elasticsearch.yml`](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/stack-settings). Refer to [General security settings](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/configuration-reference/security-settings#general-security-settings).
- The nodes of the local and remote clusters must be on Elastic Stack 8.14 or later.
- The local and remote clusters must have an appropriate license. For more information, refer to [[https://www.elastic.co/subscriptions](https://www.elastic.co/subscriptions)](https://www.elastic.co/subscriptions).
## Establish trust with a remote cluster
If a remote cluster is part of an Elastic Cloud Hosted (ECH) deployment, the remote cluster server is enabled by default and it uses a publicly trusted certificate provided by the platform proxies. Therefore, you can skip the following steps in these instructions:**On the remote (ECH) cluster:** Skip steps 1-4 (enabling the service, generating certificates, configuring SSL settings, and restarting the cluster), and go directly to [create an API key](#create-api-key).**On the local (self-managed) cluster:** Do not add the `xpack.security.remote_cluster_client.ssl.certificate_authorities` setting to the configuration file because ECH uses publicly trusted certificates that don't require custom CA configuration.
### On the remote cluster
By default, the remote cluster server interface is not enabled on self-managed clusters. Follow the steps below to enable the interface and create an API key on the remote cluster.
#### Enable and secure the remote cluster server interface
1. Enable the remote cluster server on every node of the remote cluster. In [`elasticsearch.yml`](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/stack-settings):
1. Set [`remote_cluster_server.enabled`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/configuration-reference/networking-settings#remote-cluster-network-settings) to `true`.
2. Configure the bind and publish address for remote cluster server traffic, for example using [`remote_cluster.host`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/configuration-reference/networking-settings#remote-cluster-network-settings). Without configuring the address, remote cluster traffic can be bound to the local interface, and remote clusters running on other machines can't connect.
3. Optionally, configure the remote server port using [`remote_cluster.port`](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/configuration-reference/networking-settings#remote_cluster.port) (defaults to `9443`).
2. Generate a certificate authority (CA) and a server certificate/key pair. On one of the nodes of the remote cluster, from the directory where Elasticsearch has been installed:
1. Create a CA, if you don't have a CA already:
```sh
./bin/elasticsearch-certutil ca --pem --out=cross-cluster-ca.zip --pass CA_PASSWORD
```
Replace `CA_PASSWORD` with the password you want to use for the CA. You can remove the `--pass` option and its argument if you are not deploying to a production environment.
2. Unzip the generated `cross-cluster-ca.zip` file. This compressed file contains the following content:
```txt
/ca
|_ ca.crt
|_ ca.key
```
3. Generate a certificate and private key pair for the nodes in the remote cluster:
```sh
./bin/elasticsearch-certutil cert --out=cross-cluster.p12 --pass=CERT_PASSWORD --ca-cert=ca/ca.crt --ca-key=ca/ca.key --ca-pass=CA_PASSWORD --dns= --ip=192.0.2.1
```
- Replace `CA_PASSWORD` with the CA password from the previous step.
- Replace `CERT_PASSWORD` with the password you want to use for the generated private key.
- Use the `--dns` option to specify the relevant DNS name for the certificate. You can specify it multiple times for multiple DNS.
- Use the `--ip` option to specify the relevant IP address for the certificate. You can specify it multiple times for multiple IP addresses.
4. If the remote cluster has multiple nodes, you can do one of the following:
- Create a single wildcard certificate for all nodes.
- Create separate certificates for each node either manually or in batch with the [silent mode](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/command-line-tools/certutil#certutil-silent).
3. On every node of the remote cluster, do the following:
1. Copy the `cross-cluster.p12` file from the earlier step to the `config` directory. If you didn't create a wildcard certificate, make sure you copy the correct node-specific p12 file.
2. Add following configuration to [`elasticsearch.yml`](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/stack-settings):
```yaml
xpack.security.remote_cluster_server.ssl.enabled: true
xpack.security.remote_cluster_server.ssl.keystore.path: cross-cluster.p12
```
3. Add the SSL keystore password to the Elasticsearch keystore:
```sh
./bin/elasticsearch-keystore add xpack.security.remote_cluster_server.ssl.keystore.secure_password
```
When prompted, enter the `CERT_PASSWORD` from the earlier step.
4. Restart the remote cluster.
#### Create an API key
1. On the remote cluster, use the [Elasticsearch API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) or [Kibana](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/api-keys/elasticsearch-api-keys) to create a cross-cluster API key. Configure it to include access to the indices you want to use for cross-cluster search or cross-cluster replication.
2. Copy the encoded key (`encoded` in the response) to a safe location. It is required for the local cluster configuration.
### On the local cluster
1. On every node of the local cluster:
1. Copy the `ca.crt` file generated on the remote cluster earlier into the `config` directory, renaming the file `remote-cluster-ca.crt`.
2. Add following configuration to [`elasticsearch.yml`](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/stack-settings):
```yaml
xpack.security.remote_cluster_client.ssl.enabled: true
xpack.security.remote_cluster_client.ssl.certificate_authorities: [ "remote-cluster-ca.crt" ]
```
If the remote cluster uses a publicly trusted certificate, don't include the `certificate_authorities` setting. This example assumes the remote is using certificates signed by a private certificate authority (CA), which requires the CA to be added.
3. Add the cross-cluster API key, created on the remote cluster earlier, to the keystore:
```sh
./bin/elasticsearch-keystore add cluster.remote..credentials
```
Replace `ALIAS` with the same name that you intend to use to create the remote cluster entry later. When prompted, enter the encoded cross-cluster API key created on the remote cluster earlier.
2. Restart the local cluster to load changes to the keystore and settings.
If you are configuring only the cross-cluster API key, you can use the [Nodes reload secure settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-reload-secure-settings) instead of restarting the cluster. Configuring the `remote_cluster_client` settings in `elasticsearch.yml` still requires a restart.
## Connect to a remote cluster
You must have the `manage` cluster privilege to connect remote clusters.
The local cluster uses the [remote cluster interface](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/configuration-reference/networking-settings) to establish communication with remote clusters. The coordinating nodes in the local cluster establish [long-lived](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/configuration-reference/networking-settings#long-lived-connections) TCP connections with specific nodes in the remote cluster. Elasticsearch requires these connections to remain open, even if the connections are idle for an extended period.
### Using Kibana
To add a remote cluster from Stack Management in Kibana:
1. Go to the **Remote Clusters** management page in the navigation menu or use the [global search field](https://www.elastic.co/elastic/docs-builder/docs/3028/explore-analyze/find-and-organize/find-apps-and-objects).
2. Select **Add a remote cluster**.
3. Select **API keys** as the connection type.
4. Enter a name (*cluster alias*) for the remote cluster.
5. Specify the Elasticsearch endpoint URL, or the IP address or host name of the remote cluster followed by the remote cluster port (defaults to `9443`). For example, `cluster.es.eastus2.staging.azure.foundit.no:9443` or `192.0.2.1:9443`.
Starting with Kibana 9.2, you can also specify IPv6 addresses.
### Using the Elasticsearch API
Alternatively, use the [cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) to add a remote cluster. You can also use this API to dynamically configure remote clusters for *every* node in the local cluster. To configure remote clusters on individual nodes in the local cluster, define static settings in [`elasticsearch.yml`](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/stack-settings) for each node.
If the remote cluster is part of an Elastic Cloud Hosted, Elastic Cloud Enterprise, or Elastic Cloud on Kubernetes deployment, configure the connection to use `proxy`. The default `sniff` mode doesn't work in these environments. Refer to the [connection modes](/elastic/docs-builder/docs/3028/deploy-manage/remote-clusters/remote-clusters-self-managed#sniff-proxy-modes) description for more information.
The following request adds a remote cluster with an alias of `cluster_one`. This *cluster alias* is a unique identifier that represents the connection to the remote cluster and is used to distinguish between local and remote indices.
```json
{
"persistent" : {
"cluster" : {
"remote" : {
"cluster_one" : { <1>
"seeds" : [
":9443" <2>
]
}
}
}
}
}
```
You can use the [remote cluster info API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-remote-info) to verify that the local cluster is successfully connected to the remote cluster:
```json
```
The API response indicates that the local cluster is connected to the remote cluster with the cluster alias `cluster_one`:
```json
{
"cluster_one" : {
"seeds" : [
":9443"
],
"connected" : true,
"num_nodes_connected" : 1,
"max_connections_per_cluster" : 3,
"initial_connect_timeout" : "30s",
"skip_unavailable" : true,
"cluster_credentials": "::es_redacted::",
"mode" : "sniff"
}
}
```
### Dynamically configure remote clusters
Use the [cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) to dynamically configure remote settings on every node in the cluster. The following request adds three remote clusters: `cluster_one`, `cluster_two`, and `cluster_three`.
The `seeds` parameter specifies the hostname and [remote cluster port](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/configuration-reference/networking-settings) (default `9443`) of a seed node in the remote cluster.
The `mode` parameter determines the configured connection mode, which defaults to [`sniff`](/elastic/docs-builder/docs/3028/deploy-manage/remote-clusters/remote-clusters-self-managed#sniff-mode). Because `cluster_one` doesn’t specify a `mode`, it uses the default. Both `cluster_two` and `cluster_three` explicitly use different modes.
```json
{
"persistent": {
"cluster": {
"remote": {
"cluster_one": {
"seeds": [
":9443"
]
},
"cluster_two": {
"mode": "sniff",
"seeds": [
":9443"
],
"transport.compress": true,
"skip_unavailable": true
},
"cluster_three": {
"mode": "proxy",
"proxy_address": ":9443"
}
}
}
}
}
```
You can dynamically update settings for a remote cluster after the initial configuration. The following request updates the compression settings for `cluster_two`, and the compression and ping schedule settings for `cluster_three`.
When the compression or ping schedule settings change, all existing node connections must close and re-open, which can cause in-flight requests to fail.
```json
{
"persistent": {
"cluster": {
"remote": {
"cluster_two": {
"transport.compress": false
},
"cluster_three": {
"transport.compress": true,
"transport.ping_schedule": "60s"
}
}
}
}
}
```
You can delete a remote cluster from the cluster settings by passing `null` values for each remote cluster setting. The following request removes `cluster_two` from the cluster settings, leaving `cluster_one` and `cluster_three` intact:
```json
{
"persistent": {
"cluster": {
"remote": {
"cluster_two": {
"mode": null,
"seeds": null,
"skip_unavailable": null,
"transport.compress": null
}
}
}
}
}
```
### Statically configure remote clusters
If you specify settings in [`elasticsearch.yml`](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/stack-settings), only the nodes with those settings can connect to the remote cluster and serve remote cluster requests.
Remote cluster settings that are specified using the [cluster update settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings) take precedence over settings that you specify in `elasticsearch.yml` for individual nodes.
In the following example, `cluster_one`, `cluster_two`, and `cluster_three` are arbitrary cluster aliases representing the connection to each cluster. These names are subsequently used to distinguish between local and remote indices.
```yaml
cluster:
remote:
cluster_one:
seeds: :9443
cluster_two:
mode: sniff
seeds: :9443
transport.compress: true
skip_unavailable: true
cluster_three:
mode: proxy
proxy_address: :9443
```
## Strong identity verification
- Elastic Stack: Preview since 9.3
Cross-cluster API keys can be configured with strong identity verification to provide an additional layer of security. To enable this feature, a
cross-cluster API key is created on the remote cluster with a certificate identity pattern that specifies which certificates are allowed
to use it. The local cluster must then sign each request with its private key and include a certificate whose subject Distinguished Name
(DN) matches the pattern. The remote cluster validates both that the certificate is trusted by its configured certificate authorities
and that the certificate's subject matches the API key's identity pattern.
Each remote cluster alias on the local cluster can have different remote signing configurations.
### How strong identity verification works
When a local cluster makes a request to a remote cluster using a cross-cluster API key:
1. The local cluster signs the request headers with its configured private key and sends the signature and certificate chain as header
in the request to the remote cluster.
2. The remote cluster verifies that the API key is valid.
3. If the API key has a certificate identity pattern configured, the remote cluster extracts the Distinguished Name (DN) from the
certificate chain's leaf certificate and matches it against the certificate identity pattern.
4. The remote cluster validates that the provided certificate chain is trusted.
5. The remote cluster validates the signature and checks that the certificate is not expired.
If any of these validation steps fail, the request is rejected.
### Configure strong identity verification
To use strong identity verification, the local and remote clusters must be configured to sign request headers and to verify request
headers. This can be done through the cluster settings API or `elasticsearch.yaml`.
The steps in this section describe the configuration for self-managed clusters. The same procedure can be adapted for Elastic Cloud Enterprise and Elastic Cloud on Kubernetes with the appropriate platform-specific configuration steps.For Elastic Cloud Hosted-specific steps to configure strong identity verification, refer to [Strong identity verification on Elastic Cloud Hosted](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/remote-clusters/ec-remote-cluster-strong-identity).
#### On the local cluster
When [adding the remote cluster](#using-the-es-api) to the local cluster, you must configure it to sign cross-cluster requests with a certificate–private key pair. You can generate a signing certificate using [elasticsearch-certutil](#remote-clusters-security-api-key-remote-action) or use an existing certificate. The private key can be encrypted and the password must be stored securely as a secure setting in Elasticsearch keystore. Refer to the [remote cluster settings reference](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/configuration-reference/remote-clusters#remote-cluster-signing-settings) for details.
```yaml
cluster.remote.my_remote_cluster.signing.certificate: "path/to/signing/certificate.crt"
cluster.remote.my_remote_cluster.signing.key: "path/to/signing/key.key"
```
Replace `my_remote_cluster` with your remote cluster alias, and the paths with the paths to your certificate and key files.
#### On the remote cluster
The remote cluster must be configured with a certificate authority that trusts the certificate that was used to sign the request headers.
```yaml
cluster.remote.signing.certificate_authorities: "path/to/signing/certificate_authorities.crt"
```
When creating a cross-cluster API key on the remote cluster, specify a `certificate_identity` pattern that matches the Distinguished
Name (DN) of the local cluster's certificate. Use the [Create cross-cluster API key](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-create-cross-cluster-api-key) API:
```json
{
"name": "my-cross-cluster-api-key",
"access": {
"search": [
{
"names": ["logs-*"]
}
]
},
"certificate_identity": "CN=local-cluster.example.com,O=Example Corp,C=US"
}
```
The `certificate_identity` field supports regular expressions. For example:
- `"CN=.*.example.com,O=Example Corp,C=US"` matches any certificate with a CN ending in "example.com"
- `"CN=local-cluster.*,O=Example Corp,C=US"` matches any certificate with a CN starting with "local-cluster"
- `"CN=.*"` matches any certificate (not recommended for production)
For a full list of available strong identity verification settings for remote clusters, refer to the [remote cluster settings reference](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/elasticsearch/configuration-reference/remote-clusters#remote-cluster-signing-settings).
## Configure roles and users
To use a remote cluster for cross-cluster replication or cross-cluster search, you need to create user roles with [remote indices privileges](/elastic/docs-builder/docs/3028/deploy-manage/users-roles/cluster-or-deployment-auth/role-structure#roles-remote-indices-priv) or [remote cluster privileges](/elastic/docs-builder/docs/3028/deploy-manage/users-roles/cluster-or-deployment-auth/role-structure#roles-remote-cluster-priv) on the local cluster.
To manage users and roles in Kibana, go to the **Roles** management page in the navigation menu or use the [global search field](https://www.elastic.co/elastic/docs-builder/docs/3028/explore-analyze/find-and-organize/find-apps-and-objects). You can also use the [role management APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-security) to add, update, remove, and retrieve roles dynamically.
The following examples use the [Create or update roles](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role) API. You must have at least the `manage_security` cluster privilege to use this API.
The cross-cluster API key used by the local cluster to connect the remote cluster must have sufficient privileges to cover all remote indices privileges required by individual users.
### Configure privileges for cross-cluster replication
Assuming the remote cluster is connected under the name of `my_remote_cluster`, the following request creates a role called `remote-replication` on the local cluster that allows replicating the remote `leader-index` index:
```json
{
"cluster": [
"manage_ccr"
],
"remote_indices": [
{
"clusters": [ "my_remote_cluster" ],
"names": [
"leader-index"
],
"privileges": [
"cross_cluster_replication"
]
}
]
}
```
After creating the local `remote-replication` role, use the [Create or update users](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-user) API to create a user on the local cluster and assign the `remote-replication` role. For example, the following request assigns the `remote-replication` role to a user named `cross-cluster-user`:
```json
{
"password" : "l0ng-r4nd0m-p@ssw0rd",
"roles" : [ "remote-replication" ]
}
```
Note that you only need to create this user on the local cluster.
### Configure privileges for cross-cluster search
Assuming the remote cluster is connected under the name of `my_remote_cluster`, the following request creates a `remote-search` role on the local cluster that allows searching the remote `target-index` index:
```json
{
"remote_indices": [
{
"clusters": [ "my_remote_cluster" ],
"names": [
"target-index"
],
"privileges": [
"read",
"read_cross_cluster",
"view_index_metadata"
]
}
]
}
```
After creating the `remote-search` role, use the [Create or update users](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-user) API to create a user on the local cluster and assign the `remote-search` role. For example, the following request assigns the `remote-search` role to a user named `cross-search-user`:
```json
{
"password" : "l0ng-r4nd0m-p@ssw0rd",
"roles" : [ "remote-search" ]
}
```
Note that you only need to create this user on the local cluster.