Add remote clusters using TLS certificate authentication

To add a remote cluster using TLS certificate authentication:

1. Review the prerequisites
2. Establish trust with a remote cluster
3. Connect to a remote cluster
4. Configure roles and users for remote clusters

If you run into any issues, refer to Troubleshooting.

1. The Elasticsearch security features need to be enabled on both clusters, on every node. Security is enabled by default. If it’s disabled, set xpack.security.enabled to true in elasticsearch.yml. Refer to General security settings.

2. The local and remote clusters versions must be compatible.

   Version compatibility table

   • Any node can communicate with another node on the same major version. For example, 9.0 can talk to any 9.x node.
   • Version compatibility is symmetric, meaning that if 7.16 can communicate with 8.0, 8.0 can also communicate with 7.16. The following table depicts version compatibility between local and remote nodes.

   Note

   Version 8.19 is the final minor release in the 8.x series. Unlike past releases, 8.18 was launched simultaneously with 9.0, allowing cross-version compatibility between them. Hence, as shown in the compatibility table, 8.18 can search 9.0 clusters, but only 8.19 supports searching 9.1 and later.

   	Remote cluster version
   Local cluster version	7.17	8.0	8.1	8.2	8.3	8.4	8.5	8.6	8.7	8.8	8.9	8.10	8.11	8.12	8.13	8.14	8.15	8.16	8.17	8.18	8.19	9.0	9.1	9.2	9.3	9.4
   7.17	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.0	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.1	❌	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.2	❌	❌	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.3	❌	❌	❌	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.4	❌	❌	❌	❌	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.5	❌	❌	❌	❌	❌	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.6	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.7	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.8	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.9	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.10	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.11	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.12	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.13	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.14	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.15	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.16	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.17	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	❌	❌	❌	❌	❌
   8.18	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	❌	❌	❌	❌
   8.19	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	✅	✅	✅
   9.0	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	❌	✅	✅	✅	✅	✅
   9.1	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅	✅
   9.2	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅	✅
   9.3	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅	✅
   9.4	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	❌	✅	✅

To use cross-cluster replication or cross-cluster search safely with remote clusters, enable security on all connected clusters and configure Transport Layer Security (TLS) on every node. Configuring TLS security on the transport interface is minimally required for remote clusters. For additional security, configure TLS on the HTTP interface as well.

All connected clusters must trust one another and be mutually authenticated with TLS on the transport interface. This means that the local cluster trusts the certificate authority (CA) of the remote cluster, and the remote cluster trusts the CA of the local cluster. When establishing a connection, all nodes will verify certificates from nodes on the other side. This mutual trust is required to securely connect a remote cluster, because all connected nodes effectively form a single security domain.

User authentication is performed on the local cluster and the user and user’s roles names are passed to the remote clusters. A remote cluster checks the user’s role names against its local role definitions to determine which indices the user is allowed to access.

Before using cross-cluster replication or cross-cluster search with secured Elasticsearch clusters, complete the following configuration task:

1. Configure Transport Layer Security (TLS) on every node to encrypt internode traffic and authenticate nodes in the local cluster with nodes in all remote clusters. Refer to set up basic security for the Elastic Stack for the required steps to configure security.

   Note

   This procedure uses the same CA to generate certificates for all nodes. Alternatively, you can add the certificates from the local cluster as a trusted CA in each remote cluster. You must also add the certificates from remote clusters as a trusted CA on the local cluster. Using the same CA to generate certificates for all nodes simplifies this task.

The local cluster uses the transport interface to establish communication with remote clusters. The coordinating nodes in the local cluster establish long-lived TCP connections with specific nodes in the remote cluster. Elasticsearch requires these connections to remain open, even if the connections are idle for an extended period.

To add a remote cluster from Stack Management in Kibana:

1. Go to the Remote Clusters management page in the navigation menu or use the global search field.

2. Select Add a remote cluster.

3. Select Certificates as the connection type.

4. Enter a name (cluster alias) for the remote cluster.

5. Specify the Elasticsearch endpoint URL, or the IP address or host name of the remote cluster followed by the transport port (defaults to 9300). For example, cluster.es.eastus2.staging.azure.foundit.no:9300 or 192.0.2.1:9300.

   Starting with Kibana 9.2, you can also specify IPv6 addresses.

Alternatively, use the cluster update settings API to add a remote cluster. You can also use this API to dynamically configure remote clusters for every node in the local cluster. To configure remote clusters on individual nodes in the local cluster, define static settings in elasticsearch.yml for each node.

The following request adds a remote cluster with an alias of cluster_one. This cluster alias is a unique identifier that represents the connection to the remote cluster and is used to distinguish between local and remote indices.

				PUT /_cluster/settings
					{
  "persistent" : {
    "cluster" : {
      "remote" : {
        "cluster_one" : {
          "seeds" : [
            "<MY_REMOTE_CLUSTER_ADDRESS>:9300"
          ]
        }
      }
    }
  }
}
		

1. The cluster alias of this remote cluster is cluster_one.
2. Specifies the hostname and transport port of at least a seed node in the remote cluster.

You can use the remote cluster info API to verify that the local cluster is successfully connected to the remote cluster:

				GET /_remote/info
		

The API response indicates that the local cluster is connected to the remote cluster with the cluster alias cluster_one:

{
  "cluster_one" : {
    "seeds" : [
      "<MY_REMOTE_CLUSTER_ADDRESS>:9300"
    ],
    "connected" : true,
    "num_nodes_connected" : 1,
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : true,
    "mode" : "sniff"
  }
}
		

1. The number of nodes in the remote cluster the local cluster is connected to.
2. Indicates whether to skip the remote cluster if searched through cross-cluster search but no nodes are available.

Use the cluster update settings API to dynamically configure remote settings on every node in the cluster. The following request adds three remote clusters: cluster_one, cluster_two, and cluster_three.

The seeds parameter specifies the hostname and transport port (default 9300) of a seed node in the remote cluster.

The mode parameter determines the configured connection mode, which defaults to sniff. Because cluster_one doesn’t specify a mode, it uses the default. Both cluster_two and cluster_three explicitly use different modes.

				PUT _cluster/settings
					{
  "persistent": {
    "cluster": {
      "remote": {
        "cluster_one": {
          "seeds": [
            "<MY_REMOTE_CLUSTER_ADDRESS>:9300"
          ]
        },
        "cluster_two": {
          "mode": "sniff",
          "seeds": [
            "<MY_SECOND_REMOTE_CLUSTER_ADDRESS>:9300"
          ],
          "transport.compress": true,
          "skip_unavailable": true
        },
        "cluster_three": {
          "mode": "proxy",
          "proxy_address": "<MY_THIRD_REMOTE_CLUSTER_ADDRESS>:9300"
        }
      }
    }
  }
}
		

You can dynamically update settings for a remote cluster after the initial configuration. The following request updates the compression settings for cluster_two, and the compression and ping schedule settings for cluster_three.

				PUT _cluster/settings
					{
  "persistent": {
    "cluster": {
      "remote": {
        "cluster_two": {
          "transport.compress": false
        },
        "cluster_three": {
          "transport.compress": true,
          "transport.ping_schedule": "60s"
        }
      }
    }
  }
}
		

You can delete a remote cluster from the cluster settings by passing null values for each remote cluster setting. The following request removes cluster_two from the cluster settings, leaving cluster_one and cluster_three intact:

				PUT _cluster/settings
					{
  "persistent": {
    "cluster": {
      "remote": {
        "cluster_two": {
          "mode": null,
          "seeds": null,
          "skip_unavailable": null,
          "transport.compress": null
        }
      }
    }
  }
}
		

If you specify settings in elasticsearch.yml, only the nodes with those settings can connect to the remote cluster and serve remote cluster requests.

In the following example, cluster_one, cluster_two, and cluster_three are arbitrary cluster aliases representing the connection to each cluster. These names are subsequently used to distinguish between local and remote indices.

cluster:
    remote:
        cluster_one:
            seeds: <MY_REMOTE_CLUSTER_ADDRESS>:9300
        cluster_two:
            mode: sniff
            seeds: <MY_SECOND_REMOTE_CLUSTER_ADDRESS>:9300
            transport.compress: true
            skip_unavailable: true
        cluster_three:
            mode: proxy
            proxy_address: <MY_THIRD_REMOTE_CLUSTER_ADDRESS>:9300
		

1. Compression is explicitly enabled for requests to cluster_two.
2. Disconnected remote clusters are optional for cluster_two.
3. The address for the proxy endpoint used to connect to cluster_three.

After connecting remote clusters, configure privileges so users can use cross-cluster replication and cross-cluster search:

• Configure privileges for cross-cluster replication
• Configure privileges for cross-cluster search