---
title: Istio
description: The instructions in this section describe how to connect the operator and managed resources to the Istio service mesh and assume that Istio is already...
url: https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/deploy/cloud-on-k8s/k8s-service-mesh-istio
products:
  - Elastic Cloud on Kubernetes
applies_to:
  - Elastic Cloud on Kubernetes: Generally available
---

# Istio
The instructions in this section describe how to connect the operator and managed resources to the Istio service mesh and assume that Istio is already installed and configured on your Kubernetes cluster. To know more about Istio and how to install it, check the [product documentation](https://istio.io).
These instructions have been tested with Istio 1.24.3. Older or newer versions of Istio might require additional configuration steps not documented here.
<warning>
  Some Elastic Stack features such as [Kibana alerting and actions](https://www.elastic.co/elastic/docs-builder/docs/3028/explore-analyze/alerting) rely on the Elasticsearch API keys feature which requires TLS to be enabled at the application level. If you want to use these features, you should not disable the self-signed certificate on the Elasticsearch resource and enable `PERMISSIVE` mode for the Elasticsearch service through a `DestinationRule` or `PeerAuthentication` resource. Strict mTLS mode is currently not compatible with Elastic Stack features requiring TLS to be enabled for the Elasticsearch HTTP layer.
</warning>

<important>
  If you use a Kubernetes distribution like Minikube, which does not have support for issuing third-party security tokens, you should explicitly set `automountServiceAccountToken` field to `true` in the Pod templates to allow Istio to fallback to default service account tokens. Refer to [Istio security best practices](https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens) for more information.
</important>


## Connect the operator to the Istio service mesh

The operator itself must be connected to the service mesh to deploy and manage Elastic Stack resources that you wish to connect to the service mesh. This is achieved by injecting an Istio sidecar to the ECK operator Pods. The following instructions assume that [automatic sidecar injection](https://istio.io/docs/setup/additional-setup/sidecar-injection/#automatic-sidecar-injection) is enabled on your cluster through a mutating admissions webhook. Refer to [Istio injection documentation](https://istio.io/docs/setup/additional-setup/sidecar-injection/#injection) if you prefer a different method of injection.
1. Create the `elastic-system` namespace and enable sidecar injection:
   ```sh
   kubectl create namespace elastic-system
   kubectl label namespace elastic-system istio-injection=enabled
   ```
2. Install ECK:
   ```sh
   kubectl create -f https://download.elastic.co/downloads/eck/3.3.2/crds.yaml
   kubectl apply -f https://download.elastic.co/downloads/eck/3.3.2/operator.yaml
   ```
3. Check the configuration and make sure the installation has been successful:
   ```sh
   kubectl get pod elastic-operator-0 -n elastic-system -o=jsonpath='{range .spec.containers[*]}{.name}{"\n"}'
   ```

If the output of the above command contains both `manager` and `istio-proxy`, ECK was successfully installed with the Istio sidecar injected.
To make the [validating webhook](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/deploy/cloud-on-k8s/configure-validating-webhook) work under Istio, you need to exclude the inbound port 9443 from being proxied. This can be done by editing the template definition of the `elastic-operator` StatefulSet to add the following annotations to the operator Pod:
```yaml
[...]
spec:
  template:
    metadata:
      annotations:
        traffic.sidecar.istio.io/excludeInboundPorts: "9443"
        traffic.sidecar.istio.io/includeInboundPorts: '*'
[...]
```

As the default `failurePolicy` of the webhook is `Ignore`, the operator continues to function even if the above annotations are not present. The downside is that you are still able to submit an invalid manifest using `kubectl` without receiving any immediate feedback.
ECK has a fallback validation mechanism that reports validation failures as events associated with the relevant resource (Elasticsearch, Kibana, APM Server, Beats, Elastic Agent, Elastic Maps Server, Logstash, AutoOps Agent Policy and Package Registry) that must be manually discovered by running `kubectl describe`. For example, to find the validation errors in an Elasticsearch resource named `quickstart`, you can run `kubectl describe elasticsearch quickstart`.

## Connect Elastic Stack applications to the Istio service mesh

This section assumes that you are deploying ECK custom resources to a namespace that has [automatic sidecar injection](https://istio.io/docs/setup/additional-setup/sidecar-injection/#automatic-sidecar-injection) enabled.
If you have configured Istio in [permissive mode](https://istio.io/docs/concepts/security/#permissive-mode), examples defined elsewhere in the ECK documentation will continue to work without requiring any modifications. However, if you have enabled strict mutual TLS authentication between services either through global (`MeshPolicy`) or namespace-level (`Policy`) configuration, the following modifications to the resource manifests are necessary for correct operation.

### Elasticsearch

```yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elastic-istio
spec:
  version: 9.3.2
  http:
    tls: 
      selfSignedCertificate:
        disabled: true
  nodeSets:
  - name: default
    count: 3
    podTemplate:
      metadata:
        annotations:
          traffic.sidecar.istio.io/includeInboundPorts: "*"
          traffic.sidecar.istio.io/excludeOutboundPorts: "9300" 
          traffic.sidecar.istio.io/excludeInboundPorts: "9300"
      spec:
        automountServiceAccountToken: true 
```

If you do not have [automatic mutual TLS](https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/) enabled, you may need to create a [Destination Rule](https://istio.io/docs/reference/config/networking/destination-rule/) to allow the operator to communicate with the Elasticsearch cluster. A communication issue between the operator and the managed Elasticsearch cluster can be detected by looking at the operator logs to check if there are any errors reported with the text `503 Service Unavailable`.
```sh
kubectl logs -f -n elastic-system -c manager statefulset.apps/elastic-operator
```

If the operator logs indicate a communications problem, create a `DestinationRule` to enable mutual TLS between the operator and the affected Elasticsearch cluster. For example, the following rule enables mutual TLS for a specific Elasticsearch cluster named `elastic-istio` deployed to the `default` namespace.
```yaml
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: elastic-istio
spec:
  host: "elastic-istio-es-http.default.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
```

Refer to the [Istio documentation](https://istio.io/docs/tasks/security/authentication/mtls-migration/) for more information about other configuration options affecting authentication between services.

#### Using init containers with Istio CNI

There are [known issues with init containers](https://istio.io/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers) when Istio CNI is configured. If you use init containers to [install Elasticsearch plugins](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/deploy/cloud-on-k8s/init-containers-for-plugin-downloads) or perform other initialization tasks that require network access, they may fail due to outbound traffic being blocked by the CNI plugin. To work around this issue, explicitly allow the external ports used by the init containers.
To install plugins using an init container, use a manifest similar to the following:
```yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elastic-istio
spec:
  version: 9.3.2
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  nodeSets:
  - name: default
    count: 3
    podTemplate:
      metadata:
        annotations:
          traffic.sidecar.istio.io/includeInboundPorts: "*"
          traffic.sidecar.istio.io/excludeOutboundPorts: "9300,443" 
          traffic.sidecar.istio.io/excludeInboundPorts: "9300"
      spec:
        automountServiceAccountToken: true
        initContainers:
          - name: install-plugins
            command:
              - sh
              - -c
              - |
                bin/elasticsearch-plugin remove --purge analysis-icu
                bin/elasticsearch-plugin install --batch analysis-icu
```


### Kibana

```yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: elastic-istio
spec:
  version: 9.3.2
  count: 1
  elasticsearchRef:
    name: elastic-istio
  http:
    tls: 
      selfSignedCertificate:
        disabled: true
  podTemplate:
    spec:
      automountServiceAccountToken: true 
```


### APM Server

```yaml
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: elastic-istio
spec:
  version: 9.3.2
  count: 1
  elasticsearchRef:
    name: elastic-istio
  http:
    tls: 
      selfSignedCertificate:
        disabled: true
  podTemplate:
    metadata:
      annotations:
        sidecar.istio.io/rewriteAppHTTPProbers: "true" 
    spec:
      automountServiceAccountToken: true 
```