﻿---
title: Configuration for Logstash on Elastic Cloud on Kubernetes
description: You can upgrade the Logstash version or change settings by editing the YAML specification. ECK applies the changes by performing a rolling restart of...
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5231/deploy-manage/deploy/cloud-on-k8s/configuration-logstash
products:
  - Elastic Cloud on Kubernetes
applies_to:
  - Elastic Cloud on Kubernetes: Generally available
---

# Configuration for Logstash on Elastic Cloud on Kubernetes
## Upgrade the Logstash specification

You can upgrade the Logstash version or change settings by editing the YAML specification. ECK applies the changes by performing a rolling restart of Logstash Pods.

## Logstash configuration

Define the Logstash configuration (the ECK equivalent to `logstash.yml`) in the `spec.config` section:
```yaml
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: quickstart
spec:
  version: 9.3.0
  count: 1
  elasticsearchRefs:
  - name: quickstart
    clusterName: qs
  config: 
    pipeline.workers: 4
    log.level: debug
```

Alternatively, you can provide the configuration through a Secret specified in the `spec.configRef` section. The Secret must have a `logstash.yml` entry with your settings:
```yaml
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: quickstart
spec:
  version: 9.3.0
  count: 1
  elasticsearchRefs:
  - name: quickstart
    clusterName: qs
  configRef:
    secretName: quickstart-config
---
apiVersion: v1
kind: Secret
metadata:
  name: quickstart-config
stringData:
  logstash.yml: |-
    pipeline.workers: 4
    log.level: debug
```


## Configuring Logstash pipelines

Define Logstash pipelines in the `spec.pipelines` section (the ECK equivalent to `pipelines.yml`):
```yaml
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: quickstart
spec:
  version: 9.3.0
  count: 1
  elasticsearchRefs:
    - clusterName: qs
      name: quickstart
  pipelines:
    - pipeline.id: main
      config.string: |
        input {
          beats {
            port => 5044
          }
        }
        output {
          elasticsearch {
            hosts => [ "${QS_ES_HOSTS}" ]
            user => "${QS_ES_USER}"
            password => "${QS_ES_PASSWORD}"
            ssl_certificate_authorities => "${QS_ES_SSL_CERTIFICATE_AUTHORITY}"
          }
        }
```

Alternatively, you can provide the pipelines configuration through a Secret specified in the `spec.pipelinesRef` field. The Secret must have a `pipelines.yml` entry with your configuration:
```yaml
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: quickstart
spec:
  version: 9.3.0
  count: 1
  elasticsearchRefs:
    - clusterName: qs
      name: quickstart
  pipelinesRef:
    secretName: quickstart-pipeline
---
apiVersion: v1
kind: Secret
metadata:
  name: quickstart-pipeline
stringData:
  pipelines.yml: |-
    - pipeline.id: main
      config.string: |
        input {
          beats {
            port => 5044
          }
        }
        output {
          elasticsearch {
            hosts => [ "${QS_ES_HOSTS}" ]
            user => "${QS_ES_USER}"
            password => "${QS_ES_PASSWORD}"
            ssl_certificate_authorities => "${QS_ES_SSL_CERTIFICATE_AUTHORITY}"
          }
        }
```

Logstash on ECK supports all options present in `pipelines.yml`, including settings to update the number of workers, and the size of the batch that the pipeline will process. This also includes using `path.config` to point to volumes mounted on the Logstash container:
```yaml
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: quickstart
spec:
  version: 9.3.0
  count: 1
  elasticsearchRefs:
    - clusterName: qs
      name: quickstart
  pipelines:
    - pipeline.id: main
      config.string: |
        input {
          beats {
            port => 5044
          }
        }
        output {
          elasticsearch {
            hosts => [ "${QS_ES_HOSTS}" ]
            user => "${QS_ES_USER}"
            password => "${QS_ES_PASSWORD}"
            ssl_certificate_authorities => "${QS_ES_SSL_CERTIFICATE_AUTHORITY}"
          }
        }
```

<note>
  Logstash persistent queues (PQs) and dead letter queues (DLQs) are not currently managed by the Logstash operator, and using them will require you to create and manage your own Volumes and VolumeMounts
</note>


## Defining data volumes for Logstash

<admonition title="Added in 2.9.0">
  This was added in 2.9.0.
</admonition>

<warning>
  Volume support for Logstash is a breaking change to earlier versions of ECK and requires you to recreate your Logstash resources.
</warning>


## Specifying the volume claim settings

A PersistentVolume called `logstash-data` is created by default. It maps to `/usr/share/logstash/data` for persistent storage, which is typically used for storage from plugins.
By default, the `logstash-data` volume claim is a `1.5Gi` volume, using the standard StorageClass of your Kubernetes cluster. You can override the default by adding a `spec.volumeClaimTemplate` section named `logstash-data`.
For production workloads, you should define your own volume claim template with the desired storage capacity and (optionally) the Kubernetes [storage class](https://kubernetes.io/docs/concepts/storage/storage-classes/) to associate with the persistent volume. To override this volume claim for `data` usages, the name of this volume claim must be `logstash-data`.
This example updates the default data template to increase the storage to `2Gi` for the Logstash data folder:
```yaml
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash
spec:
  # some configuration attributes omitted for brevity here
  volumeClaimTemplates:
    - metadata:
        name: logstash-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 2Gi
```

The default volume size will likely be insufficient for production workloads, especially when you are using:
- the persistent queue (PQ) feature
- dead letter queues (DLQ), or
- Logstash plugins that make heavy use of temporary storage.

Increase the storage capacity, or consider creating separate volumes for these use cases.
You can add separate storage by including an additional `spec.volumeClaimTemplate` along with a corresponding `spec.podTemplate.spec.containers.volumeMount` for each requested volume.
This example shows how to setup separate storage for a PQ:
```yaml
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash
spec:
  # some configuration attributes omitted for brevity here
  volumeClaimTemplates:
    - metadata:
        name: pq 
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
  podTemplate:
    spec:
      containers:
      - name: logstash
        volumeMounts:
        - mountPath: /usr/share/logstash/pq 
          name: pq  
          readOnly: false
  config:
    log.level: info
    queue.type: persisted
    path.queue: /usr/share/logstash/pq 
```

This example shows how to configure Logstash with a Dead Letter Queue setup on the main pipeline, and a separate pipeline to read items from the DLQ.
```yaml
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash
spec:
   # some configuration attributes omitted for brevity here
   podTemplate:
    spec:
      containers:
      - name: logstash
        volumeMounts:
        - mountPath: /usr/share/logstash/dlq 
          name: dlq  
          readOnly: false
  volumeClaimTemplates:
    - metadata:
        name: dlq 
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
  pipelines:
    - pipeline.id: beats
      dead_letter_queue.enable: true
      path.dead_letter_queue: /usr/share/logstash/dlq 
      config.string: |
        input {
          beats {
            port => 5044
          }
        }
        output {
          elasticsearch {
            hosts => [ "${ECK_ES_HOSTS}" ]
            user => "${ECK_ES_USER}"
            password => "${ECK_ES_PASSWORD}"
            ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}"
          }
        }
    - pipeline.id: dlq_read
      dead_letter_queue.enable: false
      config.string: |
        input {
          dead_letter_queue {
            path => "/usr/share/logstash/dlq" 
            commit_offsets => true
            pipeline_id => "beats"
            clean_consumed => true
          }
        }
        filter {
          mutate {
            remove_field => "[geoip][location]"
          }
        }
        output {
          elasticsearch {
            hosts => [ "${ECK_ES_HOSTS}" ]
            user => "${ECK_ES_USER}"
            password => "${ECK_ES_PASSWORD}"
            ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}"
          }
        }
```


## Updating the volume claim settings

If the storage class allows [volume expansion](https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/), you can increase the storage requests size in `spec.volumeClaimTemplates`. ECK updates the existing PersistentVolumeClaims accordingly, and recreates the StatefulSet automatically.
If the volume driver supports `ExpandInUsePersistentVolumes`, the filesystem is resized online. In this case, you do not need to restart the Logstash process or re-create the Pods.
If the volume driver does not support `ExpandInUsePersistentVolumes`, you must manually delete Pods after the resize so that they can be recreated automatically with the expanded filesystem.
Any other changes in the volumeClaimTemplates—such as changing the storage class or decreasing the volume size—are not allowed. To make changes such as these, you must fully delete the Logstash resource, delete and recreate or resize the volume, and create a new Logstash resource.
Before you delete a persistent queue (PQ) volume, ensure that the queue is empty. We recommend setting `queue.drain: true` on the Logstash Pods to ensure that the queue is drained when Pods are shutdown. Note that you should also increase the `terminationGracePeriodSeconds` to a large enough value to allow the queue to drain.
This example shows how to configure a Logstash resource to drain the queue and increase the termination grace period.
```yaml
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash
spec:
  # some configuration attributes omitted for brevity here
  config:
    queue.drain: true
  podTemplate:
    spec:
      terminationGracePeriodSeconds: 604800
```

<note>
  A [Kubernetes known issue](https://github.com/kubernetes/kubernetes/issues/94435): Kubernetes may not honor `terminationGracePeriodSeconds` settings greater than 600. A queue of a terminated Pod may not be fully drained, even when `queue.drain: true` is set and a high `terminationGracePeriodSeconds` is configured.
</note>

<note>
  In this technical preview, there is currently no way to drain a dead letter queue (DLQ) automatically before Logstash shuts down. To manually drain the queue, first stop sending data to it, by either disabling the DLQ feature, or disabling any pipelines that send to a DLQ. Then wait for events to stop flowing through any pipelines reading from the input.
</note>


## EmptyDir

If you are not concerned about data loss, you can use an `emptyDir` volume for Logstash data.
<warning>
  The use of `emptyDir` in a production environment may cause permanent data loss. Do not use with persistent queues (PQs), dead letter queues (DLQs), or with any plugin that requires persistent storage to keep track of state between restarts of Logstash.Plugins that require persistent storage include any plugin that stores state locally. These plugins typically have a configuration parameter that includes the name `path` or `directory`, not including paths to static content, such as certificates or keystores. Examples include the `sincedb_path` setting for the `file`, `dead_letter_queue` and `s3` inputs, the `last_run_metadata_path` for the `JDBC` input, `aggregate_maps_path` for the `aggregate` filter, and `temporary_directory` for the `s3` output, used to aggregate content before uploading to s3.
</warning>

```yaml
spec:
  count: 5
  podTemplate:
    spec:
      volumes:
      - name: logstash-data
        emptyDir: {}
```


## Using Elasticsearch in Logstash pipelines


### `elasticsearchRefs` for establishing a secured connection

The `spec.elasticsearchRefs` section provides a mechanism to help configure Logstash to establish a secured connection to one or more ECK managed Elasticsearch clusters. By default, each `elasticsearchRef` will target all nodes in its referenced Elasticsearch cluster. If you want to direct traffic to specific nodes of your Elasticsearch cluster, refer to [*Traffic Splitting*](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5231/deploy-manage/deploy/cloud-on-k8s/requests-routing-to-elasticsearch-nodes) for more information and examples.
When you use `elasticsearchRefs` in a Logstash pipeline, the Logstash operator creates the necessary resources from the associated Elasticsearch cluster, and provides environment variables to allow these resources to be accessed from the pipeline configuration. Environment variables are replaced at runtime with the appropriate values. The environment variables have a fixed naming convention:
- `NORMALIZED_CLUSTERNAME_ES_HOSTS`
- `NORMALIZED_CLUSTERNAME_ES_USER`
- `NORMALIZED_CLUSTERNAME_ES_PASSWORD`
- `NORMALIZED_CLUSTERNAME_ES_SSL_CERTIFICATE_AUTHORITY`

where NORMALIZED_CLUSTERNAME is the value taken from the `clusterName` field of the `elasticsearchRef` property, capitalized, with `-` transformed to `_`. That is, `prod-es` would become `PROD_ES`.
<note>
  - The `clusterName` value should be unique across all referenced Elasticsearch instances in the same Logstash spec.
  - The Logstash ECK operator creates a user called `eck_logstash_user_role` when an `elasticsearchRef` is specified. This user has the following permissions:
    ```
      "cluster": ["monitor", "manage_ilm", "read_ilm", "manage_logstash_pipelines", "manage_index_templates", "cluster:admin/ingest/pipeline/get",]
      "indices": [
        {
          "names": [ "logstash", "logstash-*", "ecs-logstash", "ecs-logstash-*", "logs-*", "metrics-*", "synthetics-*", "traces-*" ],
          "privileges": ["manage", "write", "create_index", "read", "view_index_metadata"]
        }
    ]
    ```
    You can [update user permissions](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5231/deploy-manage/users-roles/cluster-or-deployment-auth/native) to include more indices if the Elasticsearch plugin is expected to use indices other than the default. Check out [Logstash configuration with a custom index](/elastic/docs-content/pull/5231/deploy-manage/deploy/cloud-on-k8s/configuration-examples-logstash#k8s-logstash-configuration-custom-index) sample configuration that creates a user that writes to a custom index.
</note>

This example demonstrates how to create a Logstash deployment that connects to different Elasticsearch instances, one of which is in a separate namespace:
```yaml
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: quickstart
spec:
  version: 9.3.0
  count: 1
  elasticsearchRefs:        
    - clusterName: prod-es  
      name: prod
    - clusterName: qa-es    
      name: qa
      namespace: qa
  pipelines:
    - pipeline.id: main
      config.string: |
        input {
          beats {
            port => 5044
          }
        }
        output {
          elasticsearch {   
            hosts => [ "${PROD_ES_ES_HOSTS}" ]
            user => "${PROD_ES_ES_USER}"
            password => "${PROD_ES_ES_PASSWORD}"
            ssl_certificate_authorities => "${PROD_ES_ES_SSL_CERTIFICATE_AUTHORITY}"
          }
          elasticsearch {   
            hosts => [ "${QA_ES_ES_HOSTS}" ]
            user => "${QA_ES_ES_USER}"
            password => "${QA_ES_ES_PASSWORD}"
            ssl_certificate_authorities => "${QA_ES_ES_SSL_CERTIFICATE_AUTHORITY}"
          }
        }
```


### Connect to an external Elasticsearch cluster

Logstash can connect to external Elasticsearch cluster that is not managed by ECK. You can reference a Secret instead of an Elasticsearch cluster in the `elasticsearchRefs` section through the `secretName` attribute:
```yaml
apiVersion: v1
kind: Secret
metadata:
  name: external-es-ref
stringData:
  url: https://<example-url>.elastic-cloud.com:443 
  username: logstash_user 
  password: REDACTED 
  ca.crt: REDACTED 
---
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: quickstart
spec:
  version: 9.3.0
  count: 1
  elasticsearchRefs:
    - clusterName: prod-es
      secretName: external-es-ref 
  monitoring:
    metrics:
      elasticsearchRefs:
      - secretName: external-es-ref 
    logs:
      elasticsearchRefs:
      - secretName: external-es-ref 
```

<tip>
  Always specify the port in the URL when Logstash is connecting to an external Elasticsearch cluster.
</tip>


## Expose services

By default, the Logstash operator creates a headless Service for the metrics endpoint to enable metric collection by the Metricbeat sidecar for Stack Monitoring:
```sh
kubectl get service quickstart-ls-api
```

```sh
NAME                TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
quickstart-ls-api   ClusterIP   None         <none>        9600/TCP   48s
```

Additional services can be added in the `spec.services` section of the resource:
```yaml
services:
  - name: beats
    service:
      spec:
        ports:
        - port: 5044
          name: "winlogbeat"
          protocol: TCP
        - port: 5045
          name: "filebeat"
          protocol: TCP
```


## Pod configuration

You can [customize the Logstash Pod](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5231/deploy-manage/deploy/cloud-on-k8s/customize-pods) using a Pod template, defined in the `spec.podTemplate` section of the configuration.
This example demonstrates how to create a Logstash deployment with increased heap size and resource limits.
```yaml
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: logstash-sample
spec:
  version: 9.3.0
  count: 1
  elasticsearchRefs:
    - name: "elasticsearch-sample"
      clusterName: "sample"
  podTemplate:
    spec:
      containers:
      - name: logtash
        env:
        - name: LS_JAVA_OPTS
          value: "-Xmx2g -Xms2g"
        resources:
          requests:
            memory: 1Gi
            cpu: 0.5
          limits:
            memory: 4Gi
            cpu: 2
```

The name of the container in the Pod template must be `logstash`.