Remote clusters with Elastic Cloud on Kubernetes

ECK

The remote clusters module in Elasticsearch enables you to establish uni-directional connections to a remote cluster. This functionality is used in cross-cluster replication and cross-cluster search.

When using remote cluster connections with ECK, the setup process depends on where the remote cluster is deployed.

To create a remote cluster connection to another Elasticsearch cluster deployed within the same Kubernetes cluster, specify the remoteClusters attribute in your Elasticsearch spec.

Before you start, consider the security model that you would prefer to use for authenticating remote connections between clusters, and follow the corresponding steps.

API key

For deployments based on Elastic Stack 8.14 or later, you can use an API key to authenticate and authorize cross-cluster operations to a remote cluster. This model offers administrators of both the local and the remote deployment fine-grained access controls.

TLS certificate (deprecated in Elastic Stack 9.0.0)

This model uses mutual TLS authentication for cross-cluster operations. User authentication is performed on the local cluster and a user’s role names are passed to the remote cluster. A superuser on the local deployment gains total read access to the remote deployment, so it is only suitable for deployments that are in the same security domain.

To enable the API key security model you must first enable the remote cluster server on the remote Elasticsearch cluster:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: {{es}}
metadata:
  name: cluster-two
  namespace: ns-two
spec:
  version: 8.16.1
  remoteClusterServer:
    enabled: true
  nodeSets:
    - name: default
      count: 3

Once the remote cluster server is enabled and started on the remote cluster you can configure the Elasticsearch reference on the local cluster to include the desired permissions for cross-cluster search, and cross-cluster replication.

Permissions have to be included under the apiKey field. The API model of the Elasticsearch resource is compatible with the Elasticsearch Cross-Cluster API key API model. Fine-grained permissions can therefore be configured in both the search and replication fields:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: {{es}}
metadata:
  name: cluster-one
  namespace: ns-one
spec:
  nodeSets:
  - count: 3
    name: default
  remoteClusters:
  - name: cluster-two
    elasticsearchRef:
      name: cluster-two
      namespace: ns-two
      apiKey:
        access:
          search:
            names:
              - kibana_sample_data_ecommerce  1
          replication:
            names:
              - kibana_sample_data_ecommerce  1
  version: 8.16.1

1. This requires the sample data: /explore-analyze/index.md#gs-get-data-into-kibana

You can find a complete example in the recipes directory.

The following example describes how to configure cluster-two as a remote cluster in cluster-one using the certificate security model:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: {{es}}
metadata:
  name: cluster-one
  namespace: ns-one
spec:
  nodeSets:
  - count: 3
    name: default
  remoteClusters:
  - name: cluster-two
    elasticsearchRef:
      name: cluster-two
      namespace: ns-two 1
  version: 8.16.1

1. The namespace declaration can be omitted if both clusters reside in the same namespace.

You can configure a remote cluster connection to an ECK-managed Elasticsearch cluster from another cluster running outside the Kubernetes cluster as follows:

1. Make sure that both clusters trust each other’s certificate authority.
2. Configure the remote cluster connection through the Elasticsearch REST API.

Consider the following example:

• cluster-one resides inside Kubernetes and is managed by ECK
• cluster-two is not hosted inside the same Kubernetes cluster as cluster-one and may not even be managed by ECK

To configure cluster-one as a remote cluster in cluster-two:

The certificate authority (CA) used by ECK to issue certificates for the Elasticsearch transport layer is stored in a secret named <cluster_name>-es-transport-certs-public. Extract the certificate for cluster-one as follows:

kubectl get secret cluster-one-es-transport-certs-public \
-o go-template='{{index .data "ca.crt" | base64decode}}' > remote.ca.crt

You then need to configure the CA as one of the trusted CAs in cluster-two. If that cluster is hosted outside of Kubernetes, take the CA certificate that you have just extracted and add it to the list of CAs in xpack.security.transport.ssl.certificate_authorities.

If cluster-two is also managed by an ECK instance, proceed as follows:

1. Create a config map with the CA certificate you just extracted:

   kubectl create configmap remote-certs --from-file=ca.crt=remote.ca.crt
   

2. Use this config map to configure cluster-one's CA as a trusted CA in cluster-two:

   apiVersion: elasticsearch.k8s.elastic.co/v1
   kind: {{es}}
   metadata:
     name: cluster-two
   spec:
     transport:
       tls:
         certificateAuthorities:
           configMapName: remote-certs
     nodeSets:
     - count: 3
       name: default
     version: 8.16.1
   

3. Repeat steps 1 and 2 to add the CA of cluster-two to cluster-one as well.

Expose the transport layer of cluster-one.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: {{es}}
metadata:
  name: cluster-one
spec:
  transport:
    service:
      spec:
        type: LoadBalancer 1

1. On cloud providers which support external load balancers, setting the type field to LoadBalancer provisions a load balancer for your Service. Alternatively, expose the service through one of the Kubernetes Ingress controllers that support TCP services.

Finally, configure cluster-one as a remote cluster in cluster-two using the Elasticsearch REST API:

PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "cluster-one": {
          "mode": "proxy", 1
          "proxy_address": "${LOADBALANCER_IP}:9300" 2
        }
      }
    }
  }
}

1. Use "proxy" mode as cluster-two will be connecting to cluster-one through the Kubernetes service abstraction.
2. Replace ${LOADBALANCER_IP} with the IP address assigned to the LoadBalancer configured in the previous code sample. If you have configured a DNS entry for the service, you can use the DNS name instead of the IP address as well.