Configuration examples

This section contains manifests that illustrate common use cases, and can be your starting point in exploring Logstash deployed with ECK. These manifests are self-contained and work out-of-the-box on any non-secured Kubernetes cluster. They all contain a three-node Elasticsearch cluster and a single Kibana instance.

Warning

The examples in this section are for illustration purposes only. They should not be considered production-ready. Some of these examples use the node.store.allow_mmap: false setting on Elasticsearch which has performance implications and should be tuned for production workloads, as described in Virtual memory.

Single pipeline defined in CRD ¶ 

			kubectl apply -f https://raw.githubusercontent.com/elastic/cloud-on-k8s/2.16/config/recipes/logstash/logstash-eck.yaml

Deploys Logstash with a single pipeline defined in the CRD

Single Pipeline defined in Secret ¶ 

			kubectl apply -f https://raw.githubusercontent.com/elastic/cloud-on-k8s/2.16/config/recipes/logstash/logstash-pipeline-as-secret.yaml

Deploys Logstash with a single pipeline defined in a secret, referenced by a pipelineRef

Pipeline configuration in mounted volume ¶ 

			kubectl apply -f https://raw.githubusercontent.com/elastic/cloud-on-k8s/2.16/config/recipes/logstash/logstash-pipeline-as-volume.yaml

Deploys Logstash with a single pipeline defined in a secret, mounted as a volume, and referenced by path.config

Writing to a custom Elasticsearch index ¶ 

			kubectl apply -f https://raw.githubusercontent.com/elastic/cloud-on-k8s/2.16/config/recipes/logstash/logstash-es-role.yaml

Deploys Logstash and Elasticsearch, and creates an updated version of the eck_logstash_user_role to write to a user specified index.

Creating persistent volumes for PQ and DLQ ¶ 

			kubectl apply -f https://raw.githubusercontent.com/elastic/cloud-on-k8s/2.16/config/recipes/logstash/logstash-volumes.yaml

Deploys Logstash, Beats and Elasticsearch. Logstash is configured with two pipelines:

  • a main pipeline for reading from the Beats instance, which will send to the DLQ if it is unable to write to Elasticsearch
  • a second pipeline, that will read from the DLQ. In addition, persistent queues are set up. This example shows how to configure persistent volumes outside of the default logstash-data persistent volume.

Elasticsearch and Kibana Stack Monitoring ¶ 

			kubectl apply -f https://raw.githubusercontent.com/elastic/cloud-on-k8s/2.16/config/recipes/logstash/logstash-monitored.yaml

Deploys an Elasticsearch and Kibana monitoring cluster, and a Logstash that will send its monitoring information to this cluster. You can view the stack monitoring information in the monitoring cluster’s Kibana

Multiple pipelines/multiple Elasticsearch clusters ¶ 

			kubectl apply -f https://raw.githubusercontent.com/elastic/cloud-on-k8s/2.16/config/recipes/logstash/logstash-multi.yaml

Deploys Elasticsearch in prod and qa configurations, running in separate namespaces. Logstash is configured with a multiple pipeline→pipeline configuration, with a source pipeline routing to prod and qa pipelines.