﻿---
title: Logstash plugins
description: The power of Logstash is in the plugins--inputs, outputs, filters, and codecs. In Logstash on ECK, you can use the same plugins that you use for other...
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4717/deploy-manage/deploy/cloud-on-k8s/logstash-plugins
products:
  - Elastic Cloud on Kubernetes
applies_to:
  - Elastic Cloud on Kubernetes: Generally available in 3.0+
---

# Logstash plugins
The power of Logstash is in the plugins--[inputs](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/input-plugins), [outputs](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/output-plugins), [filters](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/filter-plugins), and [codecs](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/codec-plugins).
In Logstash on ECK, you can use the same plugins that you use for other Logstash instances—including Elastic-supported, community-supported, and custom plugins. However, you may have other factors to consider, such as how you configure your Kubernetes resources, how you specify additional resources, and how you scale your Logstash installation.
In this section, we’ll cover:
- [Providing additional resources for plugins (read-only and writable storage)](#k8s-plugin-resources)
- [Scaling Logstash on ECK](#k8s-logstash-working-with-plugins-scaling)
- [Plugin-specific considerations](#k8s-logstash-working-with-plugin-considerations)
- [Adding custom plugins](#k8s-logstash-working-with-custom-plugins)


## Providing additional resources for plugins

The plugins in your pipeline can impact how you can configure your Kubernetes resources, including the need to specify additional resources in your manifest. The most common resources you need to allow for are:
- Read-only assets, such as private keys, translate dictionaries, or JDBC drivers
- [Writable storage](#k8s-logstash-working-with-plugins-writable) to save application state


### Read-only assets

Many plugins require or allow read-only assets in order to work correctly. These may be ConfigMaps or Secrets files that have a 1 MiB limit, or larger assets such as JDBC drivers, that need to be stored in a PersistentVolume.

#### ConfigMaps and Secrets (1 MiB max)

Each instance of a `ConfigMap` or `Secret` has a [maximum size](https://kubernetes.io/docs/concepts/configuration/configmap/#:~:text=The%20data%20stored%20in%20a,separate%20database%20or%20file%20service) of 1 MiB (mebibyte). For larger read-only assets, check out [Larger read-only assets (1 MiB+)](#k8s-logstash-working-with-plugins-large-ro).
In the plugin documentation, look for configurations that call for a `path` or an `array` of `paths`.
**Sensitive assets, such as private keys**
Some plugins need access to private keys or certificates in order to access an external resource. Make the keys or certificates available to the Logstash resource in your manifest.
<tip>
  These settings are typically identified by an `ssl_` prefix, such as `ssl_key`, `ssl_keystore_path`, `ssl_certificate`, for example.
</tip>

To use these in your manifest, create a Secret representing the asset, a Volume in your `podTemplate.spec` containing that Secret, and then mount that Volume with a VolumeMount in the `podTemplateSpec.container` section of your Logstash resource.
First, create your secrets.
```bash
kubectl create secret generic logstash-crt --from-file=logstash.crt
kubectl create secret generic logstash-key --from-file=logstash.key
```

Then, create your Logstash resource.
```yaml
spec:
  podTemplate:
    spec:
      volumes:
        - name: logstash-ssl-crt
          secret:
            secretName: logstash-crt
        - name: logstash-ssl-key
          secret:
            secretName: logstash-key
      containers:
        - name: logstash
          volumeMounts:
            - name: logstash-ssl-key
              mountPath: "/usr/share/logstash/data/logstash.key"
              readOnly: true
            - name: logstash-ssl-crt
              mountPath: "/usr/share/logstash/data/logstash.crt"
              readOnly: true
  pipelines:
    - pipeline.id: main
      config.string: |
        input {
          http {
            port => 8443
            ssl_certificate => "/usr/share/logstash/data/logstash.crt"
            ssl_key => "/usr/share/logstash/data/logstash.key"
          }
        }
```

**Static read-only files**
Some plugins require or allow access to small static read-only files. You can use these for a variety of reasons. Examples include adding custom `grok` patterns for [`logstash-filter-grok`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-filters-grok) to use for lookup, source code for [`logstash-filter-ruby`], a dictionary for [`logstash-filter-translate`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-filters-translate) or the location of a SQL statement for [`logstash-input-jdbc`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-jdbc). Make these files available to the Logstash resource in your manifest.
<tip>
  In the plugin documentation, these plugin settings are typically identified by `path` or an `array` of `paths`.
</tip>

To use these in your manifest, create a ConfigMap or Secret representing the asset, a Volume in your `podTemplate.spec` containing the ConfigMap or Secret, and mount that Volume with a VolumeMount in your `podTemplateSpec.container` section of your Logstash resource.
This example illustrates configuring a ConfigMap from a ruby source file, and including it in a [`logstash-filter-ruby`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-filters-ruby) plugin.
First, create the ConfigMap.
```bash
kubectl create configmap ruby --from-file=drop_some.rb
```

Then, create your Logstash resource.
```yaml
spec:
  podTemplate:
    spec:
      volumes:
        - name: ruby_drop
          configMap:
            name: ruby
      containers:
        - name: logstash
          volumeMounts:
            - name: ruby_drop
              mountPath: "/usr/share/logstash/data/drop_percentage.rb"
              readOnly: true
  pipelines:
    - pipeline.id: main
      config.string: |
        input {
          beats {
            port => 5044
          }
        }
        filter {
          ruby {
            path => "/usr/share/logstash/data/drop_percentage.rb"
            script_params => { "percentage" => 0.9 }
          }
        }
```


### Larger read-only assets (1 MiB+)

Some plugins require or allow access to static read-only files that exceed the 1 MiB (mebibyte) limit imposed by ConfigMap and Secret. For example, you may need JAR files to load drivers when using a JDBC or JMS plugin, or a large [`logstash-filter-translate`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-filters-translate) dictionary.
You can add files using:
- **[PersistentVolume populated by an initContainer](#k8s-logstash-ic).** Add a volumeClaimTemplate and a volumeMount to your Logstash resource and upload data to that volume, either using an `initContainer`, or direct upload if your Kubernetes provider supports it. You can use the default `logstash-data` volumeClaimTemplate , or a custom one depending on your storage needs.
- **[Custom Docker image](#k8s-logstash-custom-images).** Use a custom docker image that includes the static content that your Logstash pods will need.

Check out [Custom configuration files and plugins](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4717/deploy-manage/deploy/cloud-on-k8s/custom-configuration-files-plugins) for more details on which option might be most suitable for you.

#### Add files using PersistentVolume populated by an initContainer

This example creates a volumeClaimTemplate called `workdir`, with volumeMounts referring to this mounted to the main container and an initContainer. The initContainer initiates a  download of a PostgreSQL JDBC driver JAR file, and stored it the volumeMount, which is then used in the JDBC input in the pipeline configuration.
```yaml
spec:
  podTemplate:
    spec:
      initContainers:
      - name: download-postgres
        command: ["/bin/sh"]
        args: ["-c", "curl -o /data/postgresql.jar -L https://jdbc.postgresql.org/download/postgresql-42.6.0.jar"]
        volumeMounts:
          - name: workdir
            mountPath: /data
      containers:
        - name: logstash
          volumeMounts:
            - name: workdir
              mountPath: /usr/share/logstash/jars 
  volumeClaimTemplates:
    - metadata:
        name: workdir
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 50Mi
  pipelines:
    - pipeline.id: main
      config.string: |
        input {
          jdbc {
             jdbc_driver_library => "/usr/share/logstash/jars/postgresql.jar"
             jdbc_driver_class => "org.postgresql.Driver"
             <2>
          }
        }
```


#### Add files using a custom Docker image

This example downloads the same `postgres` JDBC driver, and adds it to the Logstash classpath in the Docker image.
First, create a Dockerfile based on the Logstash Docker image. Download the JDBC driver, and save it alongside the other JAR files in the Logstash classpath:
```shell
FROM docker.elastic.co/logstash/logstash:9.2.4
RUN curl -o /usr/share/logstash/logstash-core/lib/jars/postgresql.jar -L https://jdbc.postgresql.org/download/postgresql-42.6.0.jar 
```

After you build and deploy the custom image, include it in the Logstash manifest. Check out [*Create custom images*](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4717/deploy-manage/deploy/cloud-on-k8s/create-custom-images) for more details.
```yaml
  count: 1
  version: {version} 
  image: <CUSTOM_IMAGE>
  pipelines:
    - pipeline.id: main
      config.string: |
        input {
          jdbc {
              <2>
             jdbc_driver_class => "org.postgresql.Driver"
              <3>
          }
        }
```


### Writable storage

Some Logstash plugins need access to writable storage. This could be for checkpointing to keep track of events already processed, a place to temporarily write events before sending a batch of events, or just to actually write events to disk in the case of [`logstash-output-file`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-outputs-file).
Logstash on ECK by default supplies a small 1.5 GiB (gibibyte) default persistent volume to each pod. This volume is called `logstash-data` and is located at `/usr/logstash/data`, and is typically the default location for most plugin use cases. This volume is stable across restarts of Logstash pods and is suitable for many use cases.
<note>
  When plugins use writable storage, each plugin must store its data a dedicated folder or file to avoid overwriting data.
</note>


#### Checkpointing

Some Logstash plugins need to write "checkpoints" to local storage in order to keep track of events that have already been processed. Plugins that retrieve data from external sources need to do this if the external source does not provide any mechanism to track state internally.
Not all external data sources have mechanisms to track state internally, and Logstash checkpoints can help persist data.
In the plugin documentation, look for configurations that call for a `path` with a settings like `sincedb`, `sincedb_path`, `sequence_path`, or `last_run_metadata_path`. Check out specific plugin documentation in the [Logstash Reference](https://docs-v3-preview.elastic.dev/elastic/logstash/tree/main/reference) for details.
```yaml
spec:
  pipelines:
    - pipeline.id: main
      config.string: |
        input {
          jdbc {
             jdbc_driver_library => "/usr/share/logstash/jars/postgresql.jar"
             jdbc_driver_class => "org.postgresql.Driver"
             last_metadata_path => "/usr/share/logstash/data/main/logstash_jdbc_last_run 
          }
        }
```

If the default `logstash-data` volume is insufficient for your needs, see the volume section for details on how to add additional volumes.

#### Writable staging or temporary data

Some Logstash plugins write data to a staging directory or file before processing for input, or outputting to their final destination. Often these staging folders can be persisted across restarts to avoid duplicating processing of data.
In the plugin documentation, look for  names such as `tmp_directory`, `temporary_directory`, `staging_directory`.
To persist data across pod restarts, set this value to point to the default `logstash-data` volume or your own PersistentVolumeClaim.
```yaml
spec:
  pipelines:
    - pipeline.id: main
      config.string: |
        output {
          s3 {
             id => "main_s3_output"
             temporary_directory => "/usr/share/logstash/data/main/main_s3_output<1>
          }
        }
```

1. If you are using more than one plugin of the same type, specify a unique location for each plugin to use.


## Scaling Logstash on ECK

<important>
  The use of autoscalers, such as the HorizontalPodAutoscaler or the VerticalPodAutoscaler, with Logstash on ECK is not yet supported.
</important>

Logstash scalability is highly dependent on the plugins in your pipelines. Some plugins can restrict how you can scale out your Logstash deployment, based on the way that the plugins gather or enrich data.
Plugin categories that require special considerations are:
- [Filter plugins: aggregating filters](#k8s-logstash-agg-filters)
- [Input plugins: events pushed to Logstash](#k8s-logstash-inputs-data-pushed)
- [Input plugins: Logstash maintains state](#k8s-logstash-inputs-local-checkpoints)
- [Input plugins: external source stores state](#k8s-logstash-inputs-external-state)

If the pipeline *does not* contain any plugins from these categories, you can increase the number of Logstash instances by setting the `count` property in the Logstash resource:
```yaml
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
  name: quickstart
spec:
  version: 9.2.4
  count: 3
```

<admonition title="Horizontal scaling for Logstash plugins">
  - Not all Logstash deployments can be scaled horizontally by increasing the number of Logstash Pods defined in the Logstash resource. Depending on the types of plugins in a Logstash installation, increasing the number of pods may cause data duplication, data loss, incorrect data, or may waste resources with pods unable to be utilized correctly.
  - The ability of a Logstash installation to scale horizontally is bound by its most restrictive plugin(s). Even if all pipelines are using [`logstash-input-elastic_agent`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-elastic_agent) or [`logstash-input-beats`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-beats) which should enable full horizontal scaling, introducing a more restrictive input or filter plugin forces the restrictions for pod scaling associated with that plugin.
</admonition>


### Filter plugins: aggregating filters

Logstash installations that use aggregating filters should be treated with particular care:
- They **must** specify `pipeline.workers=1` for any pipelines that use them.
- The number of pods cannot be scaled above 1.

Examples of aggregating filters include [`logstash-filter-aggregate`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-filters-aggregate), [`logstash-filter-csv`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-filters-csv) when `autodetect_column_names` set to `true`, and any [`logstash-filter-ruby`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-filters-ruby) implementations that perform aggregations.

### Input plugins: events pushed to Logstash

Logstash installations with inputs that enable Logstash to receive data should be able to scale freely and have load spread across them horizontally. These plugins include [`logstash-input-beats`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-beats), [`logstash-input-elastic_agent`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-elastic_agent),  [`logstash-input-tcp`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-tcp), and [`logstash-input-http`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-http).

### Input plugins: Logstash maintains state

Logstash installations that use input plugins that retrieve data from an external source, and **maintain local checkpoint state**, or would require some level of co-ordination between nodes to split up work can specify `pipeline.workers` freely, but should keep the pod count at 1 for each Logstash installation.
Note that plugins that retrieve data from external sources, and require some level of coordination between nodes to split up work, are not good candidates for scaling horizontally, and would likely produce some data duplication.
Input plugins that include configuration settings such as  `sincedb`, `checkpoint` or `sql_last_run_metadata` may fall into this category.
Examples of these plugins include [`logstash-input-jdbc`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-jdbc) (which has no automatic way to split queries across Logstash instances), [`logstash-input-s3`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-s3) (which has no way to split which buckets to read across Logstash instances), or [`logstash-input-file`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-file).

### Input plugins: external source stores state

Logstash installations that use input plugins that retrieve data from an external source, and **rely on the external source to store state** can scale based on the parameters of the external source.
For example, a Logstash installation that uses a [`logstash-input-kafka`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-kafka) plugin to retrieve data can scale the number of pods up to the number of partitions used, as a partition can have at most one consumer belonging to the same consumer group. Any pods created beyond that threshold cannot be scheduled to receive data.
Examples of these plugins include [`logstash-input-kafka`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-kafka), [`logstash-input-azure_event_hubs`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-azure_event_hubs), and [`logstash-input-kinesis`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-kinesis).

## Plugin-specific considerations

Some plugins have additional requirements and guidelines for optimal performance in a Logstash ECK environment.
- [Logstash integration plugin](#k8s-logstash-plugin-considerations-ls-integration)
- [Elasticsearch output plugin](#k8s-logstash-plugin-considerations-es-output)
- [Elastic_integration filter plugin](#k8s-logstash-plugin-considerations-integration-filter)
- [Elastic Agent input and Beats input plugins](#k8s-logstash-plugin-considerations-agent-beats)

<tip>
  Use these guidelines *in addition* to the general guidelines provided in [Scaling Logstash on ECK](#k8s-logstash-working-with-plugins-scaling).
</tip>


### Logstash integration plugin

When your pipeline uses the [`Logstash integration`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-integrations-logstash) plugin, add `keepalive=>false` to the [logstash-output](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-outputs-logstash) definition to ensure that load balancing works correctly rather than keeping affinity to the same pod.

### Elasticsearch output plugin

The [`elasticsearch output`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-outputs-elasticsearch) plugin requires certain roles to be configured in order to enable Logstash to communicate with Elasticsearch.
You can customize roles in Elasticsearch. Check out [creating custom roles](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4717/deploy-manage/users-roles/cluster-or-deployment-auth/native)
```yaml
kind: Secret
apiVersion: v1
metadata:
  name: my-roles-secret
stringData:
  roles.yml: |-
    eck_logstash_user_role:
      "cluster": ["monitor", "manage_ilm", "read_ilm", "manage_logstash_pipelines", "manage_index_templates", "cluster:admin/ingest/pipeline/get"],
      "indices": [
        {
          "names": [ "logstash", "logstash-*", "ecs-logstash", "ecs-logstash-*", "logs-*", "metrics-*", "synthetics-*", "traces-*" ],
          "privileges": ["manage", "write", "create_index", "read", "view_index_metadata"]
        }
      ]
```


### Elastic_integration filter plugin

The [`elastic_integration filter`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-filters-elastic_integration) plugin allows the use of [`ElasticsearchRef`](/elastic/docs-content/pull/4717/deploy-manage/deploy/cloud-on-k8s/configuration-logstash#k8s-logstash-esref) and environment variables.
```json
  elastic_integration {
            pipeline_name => "logstash-pipeline"
            hosts => [ "${ECK_ES_HOSTS}" ]
            username => "${ECK_ES_USER}"
            password => "${ECK_ES_PASSWORD}"
            ssl_certificate_authorities => "${ECK_ES_SSL_CERTIFICATE_AUTHORITY}"
          }
```

The Elastic_integration filter requires certain roles to be configured on the Elasticsearch cluster to enable Logstash to read ingest pipelines.
```yaml
# Sample role definition
kind: Secret
apiVersion: v1
metadata:
  name: my-roles-secret
stringData:
  roles.yml: |-
    eck_logstash_user_role:
      cluster: [ "monitor", "manage_index_templates", "read_pipeline"]
```


### Elastic Agent input and Beats input plugins

When you use the [Elastic Agent input](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-elastic_agent) or the [Beats input](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-inputs-beats), set the [`ttl`](https://docs-v3-preview.elastic.dev/elastic/beats/tree/main/reference/filebeat/logstash-output#_ttl) value on the Agent or Beat to ensure that load is distributed appropriately.

## Adding custom plugins

If you need plugins in addition to those included in the standard Logstash distribution, you can add them. Create a custom Docker image that includes the installed plugins, using the `bin/logstash-plugin install` utility to add more plugins to the image so that they can be used by Logstash pods.
This sample Dockerfile installs the [`logstash-filter-tld`](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-filters-tld) plugin to the official Logstash Docker image:
```shell
FROM docker.elastic.co/logstash/logstash:9.2.4
RUN bin/logstash-plugin install logstash-filter-tld
```

Then after building and deploying the custom image (refer to [*Create custom images*](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4717/deploy-manage/deploy/cloud-on-k8s/create-custom-images) for more details), include it in the Logstash manifest:
```shell
spec:
  count: 1
  version: 9.2.4 
  image: <CUSTOM_IMAGE>
```