Enable stack monitoring on ECK deployments

ECK

You can enable Stack Monitoring on Elasticsearch, Kibana, Beats and Logstash to collect and ship their metrics and logs to a monitoring cluster. Although self-monitoring is possible, it is advised to use a separate monitoring cluster.

In the background, Metricbeat and Filebeat are deployed as sidecar containers in the same Pod as Elasticsearch and Kibana.

Metricbeat is used to collect monitoring metrics, and Filebeat to monitor the Elasticsearch log files and collect log events.

The two Beats are configured to ship data directly to the monitoring cluster(s) using HTTPS and dedicated Elastic users managed by ECK.

This feature is a good solution if you need to monitor your Elastic applications in restricted Kubernetes environments where you cannot grant the following advanced permissions:

• To Metricbeat, to allow querying the k8s API
• To Filebeat, to deploy a privileged DaemonSet

However, for maximum efficiency and minimizing resource consumption, or advanced use cases that require specific Beats configurations, you can deploy a standalone Metricbeat deployment and a Filebeat DaemonSet. Refer to Beats configuration examples for more information.

To enable stack monitoring, reference the monitoring Elasticsearch cluster in the spec.monitoring section of their specification.

The monitoring cluster must be managed by ECK in the same Kubernetes cluster as the monitored one. To learn how to connect an external monitoring cluster, refer to Connect ot an external monitoring Elasticsearch cluster.

The following example shows how Elastic Stack components can be configured to send their monitoring data to a separate Elasticsearch cluster in the same Kubernetes cluster.

You can also do the following:

• Configure an Elasticsearch cluster to monitor itself. This is not recommended for production deployments.
• Send metrics and logs to two different Elasticsearch monitoring clusters.
• Enable stack monitoring on a single Elastic Stack component only. If Elasticsearch is not monitored, other Elastic Stack components will not be available on the Stack Monitoring Kibana page.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: monitored-sample
  namespace: production
spec:
  version: 8.16.1
  monitoring:
    metrics:
      elasticsearchRefs:
      - name: monitoring 1
        namespace: observability 2
    logs:
      elasticsearchRefs:
      - name: monitoring 1
        namespace: observability 2
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: monitored-sample
spec:
  type: filebeat
  version: 8.16.1
  monitoring:
    metrics:
      elasticsearchRefs:
      - name: monitoring
        namespace: observability 2
    logs:
      elasticsearchRefs:
      - name: monitoring
        namespace: observability 2
---
apiVersion: logstash.k8s.elastic.co/v1alpha1
kind: Logstash
metadata:
    name: monitored-sample
spec:
  version: 8.16.1
  monitoring:
    metrics:
      elasticsearchRefs:
      - name: monitoring
        namespace: observability 2
    logs:
      elasticsearchRefs:
      - name: monitoring
        namespace: observability 2

1. The same monitoring cluster is used for metrics and logs, but separate clusters could be used.
2. The use of namespace is optional if the monitoring Elasticsearch cluster and the monitored Elastic Stack resource are running in the same namespace.

If you want to connect to a monitoring Elasticsearch cluster not managed by ECK, you can reference a Secret instead of an Elasticsearch cluster in the monitoring section through the secretName attribute:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: monitored-sample
  namespace: production
spec:
  version: 8.16.1
  monitoring:
    metrics:
      elasticsearchRefs:
      - secretName: monitoring-metrics-es-ref 1
    logs:
      elasticsearchRefs:
      - name: monitoring-logs
        namespace: observability 2
        serviceName: monitoring-logs-es-coordinating-nodes 2
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false

1. The secretName and name attributes are mutually exclusive, you have to choose one or the other.
2. The namespace and serviceName attributes can only be used in conjunction with name, not with secretName.

The referenced Secret must contain the following connection information:

• url: The URL to reach the Elasticsearch cluster
• username: The username of the user to be authenticated to the Elasticsearch cluster
• password: The password of the user to be authenticated to the Elasticsearch cluster
• ca.crt: The contents of the CA certificate in PEM format to secure communication to the Elasticsearch cluster (optional)

apiVersion: v1
kind: Secret
metadata:
  name: monitoring-metrics-es-ref
stringData:
  url: https://mon1.es.abcd-42.xyz.elastic-cloud.com:9243
  username: monitoring-user
  password: <password>

The user referenced in the Secret must have been created beforehand.

You can customize the Filebeat and Metricbeat containers through the Pod template. Your configuration is merged with the values of the default Pod template that ECK uses.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
spec:
  monitoring:
    metrics:
      elasticsearchRef:
        name: monitoring
        namespace: observability
    logs:
      elasticsearchRef:
        name: monitoring
        namespace: observability
  nodeSets:
  - name: default
    podTemplate:
      spec:
        containers:
        - name: metricbeat
          env:
          - foo: bar
        - name: filebeat
          env:
          - foo: bar