﻿---
title: Tutorial: Observe your Kubernetes deployments
description: Applications running in a containerized environment like Kubernetes pose a unique monitoring challenge: how do you diagnose and resolve issues with hundreds...
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/observability/infra-and-hosts/tutorial-observe-kubernetes-deployments
products:
  - Elastic Observability
---

# Tutorial: Observe your Kubernetes deployments
Applications running in a containerized environment like Kubernetes pose a unique monitoring challenge: how do you diagnose and resolve issues with hundreds of microservices on thousands (or millions) of containers, running in ephemeral and disposable pods?
A successful Kubernetes monitoring solution has a few requirements:
- Monitors all layers of your technology stack, including:
  - The host systems where Kubernetes is running.
- Kubernetes core components, nodes, pods, and containers running within the cluster.
- All of the applications and services are running in Kubernetes containers.
- Automatically detects and monitors services as they appear dynamically.
- Provides a way to correlate related data so that you can group and explore related metrics, logs, and other observability data.


## What you’ll learn

This guide describes how to use Elastic Observability to observe all layers of your application, including the orchestration software itself:
- Collect logs and metrics from Kubernetes and your applications
- Collect trace data from applications deployed with Kubernetes
- Centralize the data in the Elastic Stack
- Explore the data in real-time using tailored dashboards and Observability UIs

This guide describes how to deploy Elastic monitoring agents as DaemonSets using the Elastic Agent manifest files. For other deployment options, see the Kubernetes operator and custom resource definitions from [Elastic Cloud on Kubernetes (ECK)](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/deploy-manage/deploy/cloud-on-k8s).

## Monitoring architecture

The Elastic Stack provides the following components for monitoring Kubernetes:
1. Elastic Agent is a single, unified way to add monitoring for data like logs and metrics to your host.
2. Elastic Agent Kubernetes integration, in tandem with Elastic Agent, collects logs and metrics from Kubernetes clusters.
3. APM (described later) to monitor, detect, and diagnose complex application performance issues.
4. Elasticsearch for storing and searching your data.
5. Observability apps in Kibana for visualizing and managing your observability data.

![Kubernetes monitoring architecture](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/images/observability-k8s-monitoring-architecture.png)

The default installation of Elastic Agent is deployed to Kubernetes as a DaemonSet to ensure an instance is running on each node of the cluster. It collects logs and metrics from pods, containers, and applications running on Kubernetes.

## Metadata

Elastic Agent provides processors for adding metadata to events. The metadata is valuable for grouping and exploring related data. For example, when analyzing container logs, you want to know the host and container name, and be able to correlate logs, metrics, and traces.
The default deployments include processors, when needed, for enriching events with cloud and host metadata.
![Metadata processors for cloud](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/images/observability-metadata-processors.png)

For more on these processors, refer to the [`add_cloud_metadata`](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/reference/fleet/add-cloud-metadata-processor) and [`add_host_metadata`](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/reference/fleet/add_host_metadata-processor) documentation.
By default, the Kubernetes integration enriches logs and metrics with valuable metadata.
All Kubernetes metrics are enriched with metadata by default. The enrichment happens in code, and can be configured with the `add_resource_metadata` block on each dataset. For more information on configuring the `add_resource_metadata` block, refer to [Configure kubelet API metadata enrichment](#monitor-k8s-kubelet-configure-metadata) and [Configure `kube-state-metrics` metadata enrichment](#monitor-k8s-kube-state-configure-metadata).
All Kubernetes logs are enriched with metadata by default. For more on configuring metadata enrichment, refer to [Collect Kubernetes container logs](#monitor-kubernetes-integration-container-logs).
Now that you have a basic understanding of the monitoring architecture, let’s learn how to deploy monitoring to your Kubernetes environment.

## Before you begin

Before you can monitor Kubernetes, you need the following:
- Elasticsearch for storing and searching your observability data and Kibana for visualizing and managing it.
- If you want to collect Kubernetes state metrics, you need to deploy `kube-state-metrics`. For deployment instructions, refer to the Kubernetes [docs](https://github.com/kubernetes/kube-state-metrics#kubernetes-deployment).


## Part 1: Add and Configure the Kubernetes integration

To start collecting logs and metrics from your Kubernetes clusters, first add the [Kubernetes integration](https://docs.elastic.co/en/integrations/kubernetes) to your policy and configure the metrics and logs you want to collect.

### Step 1: Add the Kubernetes integration to your deployment

Follow these steps to add the Kubernetes integration to your policy:
1. Find **Integrations** in the main menu or use the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/explore-analyze/find-and-organize/find-apps-and-objects).
2. Enter "Kubernetes" in the search bar, and select the **Kubernetes** integration.
3. Click **Add Kubernetes** at the top of the Kubernetes integration page.
4. Click **Add integration only (skip agent installation)** at the bottom of the Add integration page.

Continue to [Step 2: Configure your Kubernetes integration](#monitor-k8s-configure-integration).

### Step 2: Configure your Kubernetes integration

The Kubernetes integration can fetch metrics and logs from several components. From the **Add Kubernetes integration** page, configure the options for the metrics and logs you want to collect. The following sections outline the configuration options.
**Collect Metrics**
Collecting metrics about Kubernetes clusters and the workloads running on top of them is a key aspect of Kubernetes observability. You need to collect metrics about the resources running on the underlying infrastructure (for example, the machines of your Kubernetes cluster), metrics for the whole cluster, as well as individual metrics for the containers and pods. Specifically, you need to monitor the health and performance of:
- The hosts where Kubernetes components are running. Each host produces metrics like CPU, memory, disk utilization, and disk and network I/O.
- Kubernetes containers, which produce their own set of metrics.
- The applications running as Kubernetes pods, such as application servers and databases, each producing its own set of metrics.
- Additional Kubernetes resources (like services, deployments, cronjobs) are valuable assets of the whole infrastructure and produce their own set of metrics that need monitoring.

Elastic Agent along with the Kubernetes integration provides a unified solution to monitor all layers of your Kubernetes technology stack, so you don’t need multiple technologies to collect metrics.
You have the following options for collecting metrics about your Kubernetes clusters and the workloads running on top of them:
- [Collect Kubernetes metrics from kubelet API](#monitor-kubernetes-integration-kubelet)
- [Collect Kubernetes metrics from `kube-state-metrics`](#monitor-kubernetes-integration-kube-state)
- [Collect Kubernetes metrics from Kubernetes API server](#monitor-kubernetes-integration-k8s-api-metrics)
- [Collect Kubernetes metrics from Kubernetes proxy](#monitor-kubernetes-integration-k8s-proxy)
- [Collect Kubernetes metrics from Kubernetes scheduler](#monitor-kubernetes-integration-k8s-scheduler)
- [Collect Kubernetes metrics from Kubernetes controller-manager](#monitor-kubernetes-integration-k8s-controller)
- [Collect Kubernetes events from Kubernetes API Server](#monitor-kubernetes-integration-k8s-api-events)

**Collect logs**
Collecting and analyzing logs of both Kubernetes core components and various applications running on top of Kubernetes is a powerful tool for Kubernetes observability. Containers running within Kubernetes pods publish logs to stdout or stderr.
You have the following options for collecting Kubernetes logs:
- [Collect Kubernetes container logs](#monitor-kubernetes-integration-container-logs)
- [Collect Kubernetes audit logs](#monitor-kubernetes-integration-audit-logs)


### Collect Kubernetes metrics from kubelet API

Collecting metrics from the kubelet API is on by default. Kubelet is an agent that runs on each Kubernetes node that is key to managing individual pods and the nodes that host them. For more information on kubelet, refer to the Kubernetes [kubelet docs](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/).
Elastic Agent along with the Kubernetes integration can collect metrics from the kubelet API to gather important information about the state of your Kubernetes nodes, pods, containers, and other resources. paleExpand the following list to see all available metrics from the kubelet API.
<dropdown title="Expand to see available metrics from the kubelet API">
  <definitions>
    <definition term="Container metrics">
      Monitor the overall resource usage, performance, and status at the container level. Learn more at [kubelet container metrics](https://docs.elastic.co/en/integrations/kubernetes/kubelet#container).
    </definition>
    <definition term="Node metrics">
      Monitor the overall resource usage, performance, and status at the node level. Learn more at [kubelet node metrics](https://docs.elastic.co/en/integrations/kubernetes/kubelet#node).
    </definition>
    <definition term="Pod metrics">
      Monitor the overall resource usage, performance, and status at the pod level. Learn more at [kubelet pod metrics](https://docs.elastic.co/en/integrations/kubernetes/kubelet#pod).
    </definition>
    <definition term="System metrics">
      Monitor the overall resource usage, performance, and status of your system containers. Learn more at [kubelet system metrics](https://docs.elastic.co/en/integrations/kubernetes/kubelet#system).
    </definition>
    <definition term="Volume metrics">
      Monitor the storage usage and capacity of your persistent volumes. Learn more at [kubelet volume metrics](https://docs.elastic.co/en/integrations/kubernetes/kubelet#system).
    </definition>
  </definitions>
</dropdown>


#### Configure the kubelet API options

Provide the following information for each of the kubelet API options:
<definitions>
  <definition term="Add Metadata">
    This option adds metadata to events. The metadata is valuable for grouping and exploring related data. This option is on by default.
  </definition>
  <definition term="Bearer Token File">
    The file path to the token used to authenticate with the kubelet API.
  </definition>
  <definition term="Hosts">
    The address of the kubelet API server that Elastic will connect to for collecting metrics. `${env.NODE_NAME}` is an environment variable that represents the name of the Kubernetes node. Port `10250` is the default port where the kubelet API listens for HTTPS connections.
  </definition>
  <definition term="Period">
    How frequently to poll the kubelet API for metrics. The default is every 10 seconds.
  </definition>
  <definition term="SSL Verification Mode">
    Specify how to handle SSL verification. `none` means that SSL verification is disabled.
  </definition>
</definitions>


#### Configure kubelet API metadata enrichment

Under **Kubernetes Container metrics** and **Kubernetes Pod metrics**, you can configure metadata enrichment from **Advanced options → Add node and namespace metadata**.
From here, update the `add_resource_metadata` block to configure enrichment:
```yaml
 add_resource_metadata:
   namespace:
     include_labels: ["namespacelabel1"]
   node:
     include_labels: ["nodelabel2"]
     include_annotations: ["nodeannotation1"]
   deployment: false
```


### Collect Kubernetes metrics from `kube-state-metrics`

<note>
  You need to deploy `kube-state-metrics` before you can use it to collect metric data. To learn how, refer to the Kubernetes [deployment docs](https://github.com/kubernetes/kube-state-metrics#kubernetes-deployment).
</note>

Collecting metrics from `kube-state-metrics` is on by default. The `kube-state-metrics` service provides cluster-wide metrics on the health of Kubernetes objects. Refer to the [kube-state-metrics docs](https://github.com/kubernetes/kube-state-metrics) for more information.
`kube-state-metrics` provides horizontal sharding to support large Kubernetes deployments. Refer to the [`kube-state-metrics` sharding](https://github.com/elastic/elastic-agent/blob/main/docs/elastic-agent-ksm-sharding.md) documentation for more information.
With the Kubernetes integration, you can collect a number of metrics using the `kube-state-metrics`. Expand the following list to see all available metrics from `kube-state-metrics`.
<dropdown title="Expand to see available metrics from kube-state-metrics">
  <definitions>
    <definition term="Container metrics">
      Monitor Container performance to ensure efficiency and stability in pods. Learn more at [`kube-state-metrics` container metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_container).
    </definition>
    <definition term="CronJob metrics">
      Monitor CronJob performance and ensure they’re running reliably and efficiently. Learn more at [`kube-state-metrics` CronJob metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_cronjob).
    </definition>
    <definition term="Kubernetes DaemonSet metrics">
      Monitor DaemonSet health and distribution. Learn more at [`kube-state-metrics` DaemonSet metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_daemonset).
    </definition>
    <definition term="Kubernetes Deployment metrics">
      Monitor deployment status and configuration. Learn more at [`kube-state-metrics` deployment metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_deployment).
    </definition>
    <definition term="Kubernetes Job metrics">
      Monitor job completion statuses and execution. Learn more at [`kube-state-metrics` job metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_job).
    </definition>
    <definition term="Kubernetes Namespace metrics">
      Monitor namespace active and terminating statuses. Learn more at [`kube-state-metrics` namespace metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_namespace).
    </definition>
    <definition term="Kubernetes Node metrics">
      Monitor node health and resource usage. Learn more at [`kube-state-metrics` node metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_node).
    </definition>
    <definition term="Kubernetes PersistentVolume metrics">
      Monitor PersistentVolume size, status, and storage configuration. Learn more at [`kube-state-metrics` PersistentVolume metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_persistentvolume).
    </definition>
    <definition term="Kubernetes PersistentVolumeClaim metrics">
      Monitor PersistentVolumeClaim phases, classes, and storage requests. Learn more at [`kube-state-metrics` PersistentVolumeClaim metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_persistentvolumeclaim).
    </definition>
    <definition term="Kubernetes Pod metrics">
      Monitor pod health and performance. Learn more at [`kube-state-metrics` pod metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_pod).
    </definition>
    <definition term="Kubernetes ReplicaSet metrics">
      Monitor ReplicaSets status and the number of replicas in your ReplicaSets. Learn more at [`kube-state-metrics` ReplicaSet metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_replicaset).
    </definition>
    <definition term="Kubernetes ResourceQuota metrics">
      Monitor resource limits and current usage. Learn more at [`kube-state-metrics` ResourceQuota metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_resourcequota).
    </definition>
    <definition term="Kubernetes Service metrics">
      Monitor service configuration, accessibility, and network integration. Learn more at [`kube-state-metrics` service metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_service).
    </definition>
    <definition term="Kubernetes StatefulSet metrics">
      Monitor StatefulSet configuration, status, and scaling. Learn more at [`kube-state-metrics` StatefulSet metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_statefulset).
    </definition>
    <definition term="Kubernetes StorageClass metrics">
      Monitor how storage is provisioned and allocated. Learn more at [`kube-state-metrics` StorageClass metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-state-metrics#state_storageclass).
    </definition>
  </definitions>
</dropdown>


#### Configure the `kube-state-metrics` options

Provide the following information for each of the `kube-state-metrics` options:
<definitions>
  <definition term="Add Metadata">
    This option adds metadata to events. The metadata is valuable for grouping and exploring related data. This option is on by default.
  </definition>
  <definition term="Hosts">
    The address where `kube-state-metrics` is running. Port 8080 is the default.
  </definition>
  <definition term="Leader Election">
    When on (default behavior), only the Elastic Agent that holds the leadership lock will retrieve metrics from the `kube_state_metrics`. This prevents duplicate data in multi-node Kubernetes clusters.
  </definition>
  <definition term="Period">
    How frequently to poll the `kube-state-metrics` for metrics. The default is every 10 seconds.
  </definition>
</definitions>


#### Configure `kube-state-metrics` metadata enrichment

Under **Kubernetes Container metrics** and **Kubernetes Pod metrics**, you can configure metadata enrichment from **Advanced options → Add node and namespace metadata**.
From here, update the `add_resource_metadata` block to configure enrichment:
```yaml
add_resource_metadata:
  namespace:
   enabled: true
    #use_regex_include: false
    include_labels: ["namespacelabel1"]
    #use_regex_exclude: false
    #exclude_labels: ["namespacelabel2"]
  node:
   enabled: true
    #use_regex_include: false
    include_labels: ["nodelabel2"]
    include_annotations: ["nodeannotation1"]
    #use_regex_exclude: false
    #exclude_labels: ["nodelabel3"]
  #deployment: false
  #cronjob: false
```


### Collect Kubernetes metrics from Kubernetes API server

Collecting metrics from the `kube-apiserver` is on by default. The `kube-apiserver` sets up and validates pods, services, and other API objects. These metrics provide insight into the API server’s performance, workload, and health.
Refer to [`kube-apiserver` metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-apiserver) for more on the metrics collected.

#### Configure Kubernetes API server options

Provide the following information to collect `kube-apiserver` metrics:
<definitions>
  <definition term="Bearer Token File">
    The file path to the token used to authenticate with the `kube-apiserver`.
  </definition>
  <definition term="Hosts">
    The address of the Kubernetes API server that the integration connects to. It uses the `KUBERNETES_SERVICE_HOST` and `KUBERNETES_SERVICE_PORT` environment variables.
  </definition>
  <definition term="Leader Election">
    When on (default behavior), only the Elastic Agent that holds the leadership lock will retrieve metrics from the `kube-apiserver`. This prevents duplicate data in multi-node Kubernetes clusters.
  </definition>
  <definition term="Period">
    How frequently to poll the `kube-state-metrics` for metrics. The default is every 30 seconds.
  </definition>
  <definition term="SSL Certificate Authorities">
    The path to the certificate authority (CA) bundle used to verify the Kubernetes API server’s TLS certificate.
  </definition>
</definitions>


### Collect Kubernetes metrics from Kubernetes proxy

The `kube-proxy` runs on each node and maintains network rules. Collecting metrics from the `kube-proxy` is on by default. These metrics provide insight into the proxy’s networking activity, performance, and resource usage.
Refer to [`kube-proxy` metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-proxy) for more on the metrics collected.

#### Configure Kubernetes proxy options

Provide the following information to collect Kubernetes Proxy metrics:
<definitions>
  <definition term="Hosts">
    The address where `kube-proxy` is running. Port 10249 is the default.
  </definition>
  <definition term="Period">
    How frequently to poll the `kube-state-metrics` for metrics. The default is every 10 seconds.
  </definition>
</definitions>


### Collect Kubernetes metrics from Kubernetes scheduler

The kube-scheduler assigns new pods with no node assignment to the most appropriate node. Turn this option on to get metrics from the kube-scheduler. These metrics provide insight on the performance, resource usage, and health of the `kube-scheduler`.
Refer to [`kube-scheduler` metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-scheduler) for more on the metrics collected.

#### Configure Kubernetes scheduler options

Provide the following information to collect Kubernetes scheduler metrics:
<definitions>
  <definition term="Bearer Token File">
    The file path to the token used to authenticate with the `kube-scheduler`.
  </definition>
  <definition term="Hosts">
    The address and port of the `kube-scheduler` from which the Elastic integration should collect metrics. Port `10259` is the default.
  </definition>
  <definition term="Period">
    How frequently to poll the `kube-scheduler` for metrics. The default is every 10 seconds.
  </definition>
  <definition term="SSL Verification Mode">
    Specify how to handle SSL verification. Defaults to `none` meaning that SSL verification is disabled.
  </definition>
</definitions>


### Collect Kubernetes metrics from Kubernetes controller-manager

The `kube-controller-manager` regulates the state of the clusters. Turn this option on to get metrics from the `kube-controller-manager`. These metrics provide insight on the performance, resource usage, and health of the kube-controller-manager.
Refer to [`kube-controller-manager` metrics](https://docs.elastic.co/en/integrations/kubernetes/kube-controller-manager) for more on the metrics collected.

#### Configure Kubernetes controller-manager options

Provide the following information to collect `kube-controller-manager` metrics:
<definitions>
  <definition term="Bearer Token File">
    The file path to the token used to authenticate with the `kube-controller-manager`.
  </definition>
  <definition term="Hosts">
    The address and port of the `kube-controller-manager` from which the integration should collect metrics. Port 10259 is the default.
  </definition>
  <definition term="Period">
    How frequently to poll the `kube-controller-manager` for metrics. The default is every 10 seconds.
  </definition>
  <definition term="SSL Verification Mode">
    Specify how to handle SSL verification. Defaults to `none` meaning that SSL verification is disabled.
  </definition>
</definitions>


### Collect Kubernetes events from Kubernetes API Server

Event metrics give you an overall view of what’s happening in a cluster. These metrics help you understand what’s happening in your cluster and improve reliability and stability. Collecting Kubernetes events from the Kubernetes API server is on by default.
Refer to [events metrics](https://docs.elastic.co/en/integrations/kubernetes/events) for more on the metrics collected.

#### Configure events from the Kubernetes API server

Provide the following information to collect Kubernetes events metrics:
<definitions>
  <definition term="Period">
    How frequently to poll the `kube-api-server` for events. The default is every 10 seconds.
  </definition>
  <definition term="Add Metadata">
    Turn on to add metadata to events. The metadata is valuable for grouping and exploring related data.
  </definition>
  <definition term="Skip older events">
    Ignores events that occurred before a certain time
  </definition>
  <definition term="Leader Election">
    When on (default behavior), only the Elastic Agent that holds the leadership lock will retrieve metrics from the `kube-apiserver`. This prevents duplicate data in multi-node Kubernetes clusters.
  </definition>
</definitions>


### Collect Kubernetes container logs

Collecting and parsing Kubernetes container logs is on by default. Containers running within Kubernetes pods publish logs to stdout or stderr. These logs are written to a location known to kubelet. The container parser is enabled by default. You can enable additional parsers in **advanced settings**.
Metadata enrichment is also enabled by default, and is based on the Kubernetes provider. Use the `add_resource_metadata` block of the Kubernetes provider to configure it. Refer to the [Kubernetes provider](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/reference/fleet/kubernetes-provider) docs for more on configuring the provider.
Refer to [Kubernetes container logs](https://docs.elastic.co/en/integrations/kubernetes/container-logs) for more on collecting container logs.

#### Configure Kubernetes container logs

Provide the following information to collect container logs:
<definitions>
  <definition term="Use Symlinks">
    A symlink is lightweight and doesn’t contain the data of the log files, but points to their actual location. Symlinks are used by default.
  </definition>
  <definition term="Condition">
    You can specify a condition to control whether a configuration is applied to the running Elastic Agent.
  </definition>
</definitions>


### Collect Kubernetes audit logs

<warning>
  This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.
</warning>

Turn this option on to collect audit logs. Kubernetes audit logs record requests that come to the Kubernetes API from internal and external components. These logs help you understand cluster behavior and debug issues.
Refer to [Kubernetes audit logs](https://docs.elastic.co/en/integrations/kubernetes/audit-logs) for more on collecting audit logs.

## Part 2: Configure and install the standalone Elastic Agent

After configuring your integration, you need to download and update your manifest. First, download the manifest by completing the following steps:
1. At the bottom of the **Add Kubernetes integration** page, click **Save and continue**.
2. Click **Add Elastic Agent to your hosts**.
3. Under **Enroll in Fleet?**, select **Run standalone**.
   ![Select run standalone under Enroll in Fleet](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/images/observability-run-standalone-option.png)
4. Under **Configure the agent**, select **Download Manifest**.

After downloading the manifest, open it and update the `ES_USERNAME` and `ES_PASSWORD` environment variables in the DaemonSet to match your Elasticsearch credentials.
You can also further modify the manifest to fit your needs. For example, you might want to enable autodiscovery to automatically discover container logs. Refer to the [autodiscovery docs](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/reference/fleet/elastic-agent-kubernetes-autodiscovery) in the Fleet guide for more on enabling autodiscovery in your manifest.
Once you are ready to deploy your Elastic Agent:
1. From the directory where you’ve downloaded the manifest, run the following apply command:
   ```sh
   kubectl apply -f elastic-agent-standalone-kubernetes.yml
   ```
2. Check the Elastic Agent status with the following command:
   ```sh
   kubectl -n kube-system get pods -l app=elastic-agent
   ```

Refer to [Debug standalone Elastic Agents](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/reference/fleet/debug-standalone-agents) if you run into any issues with configuring or installing your Elastic Agent.

## Part 3: Explore logs and metrics

Use Kibana to view the metric and log data collected by Elastic Agent. Refer to the following sections for more on viewing your data.
- [View performance and health metrics](#monitor-k8s-explore-metrics).
- [View Kubernetes logs](#monitor-k8s-explore-logs).


### View performance and health metrics

To view the performance and health metrics collected by Elastic Agent, find **Infrastructure** in the main menu or use the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/explore-analyze/find-and-organize/find-apps-and-objects).
On the **Infrastructure inventory** page, you can switch between different views to see an overview of the containers and pods running on Kubernetes:
![Inventory page that shows Kubernetes pods](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/images/observability-metrics-inventory.png)

For more on using the Inventory page, refer to [View infrastructure metrics by resource type](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/observability/infra-and-hosts/view-infrastructure-metrics-by-resource-type).
On the **Metrics Explorer** page, you can group and analyze metrics for the resources that you are monitoring.
![Metrics dashboard that shows CPU usage for Kubernetes pods](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/images/observability-monitor-k8s-metrics-explorer.png)

For more on using the **Metrics Explorer** page, refer to [Explore infrastructure metrics over time](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/observability/infra-and-hosts/explore-infrastructure-metrics-over-time).

### View Kubernetes logs

Find `Discover` in the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/explore-analyze/find-and-organize/find-apps-and-objects).
From the **Data view** menu, select `All logs`. From here, you can quickly search and filter your log data, get information about the structure of log fields, and display your findings in a visualization. Then, you can filter your log data and dive deeper into individual logs to find and troubleshoot issues. For more information, refer to:
- [Explore logs in Discover](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/observability/logs/discover-logs) for an overview of viewing your logs in Discover.
- [Filter logs in Discover](/elastic/docs-content/pull/4431/solutions/observability/logs/filter-aggregate-logs#logs-filter-discover) for more on filtering logs in Discover.


## Part 4: Monitor application performance

Quickly triage and troubleshoot application performance problems with the help of Elastic application performance monitoring (APM).
Think of a latency spike — APM can help you narrow the scope of your investigation to a single service. Because you’ve also ingested and correlated logs and metrics, you can then link the problem to CPU and memory utilization or error log entries of a particular Kubernetes pod.

### Step 1: Set up APM

Application monitoring data is streamed from your applications running in Kubernetes to APM, where it is validated, processed, and transformed into Elasticsearch documents.
There are many ways to deploy APM when working with Kubernetes, but this guide assumes that you’re using an Elastic Cloud Hosted deployment. If you haven’t done so already, enable APM in the [Elastic Cloud Console](https://cloud.elastic.co?page=docs&placement=docs-body).
If you want to manage APM yourself, there are a few alternative options:
<dropdown title="Expand alternatives">
  - [Elastic Cloud on Kubernetes (ECK)](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/deploy-manage/deploy/cloud-on-k8s) — The Elastic recommended approach for managing APM Server deployed with Kubernetes. Built on the Kubernetes Operator pattern, ECK extends basic Kubernetes orchestration capabilities to support the setup and management of APM Server on Kubernetes.
  - Deploy APM Server as a DaemonSet — Ensure a running instance of APM Server on each node in your cluster. Useful when all pods in a node should share a single APM Server instance.
  - Deploy APM Server as a sidecar — For environments that should not share an APM Server, like when directing traces from multiple applications to separate Elasticsearch clusters.
  - [Download and install APM Server](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/observability/apm/get-started) — The classic, non-Kubernetes option.
</dropdown>


### Step 2: Save your secret token

A [secret token](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/observability/apm/secret-token) is used to secure communication between APM agents and APM Server. To create or update your secret token in Kibana:
1. Find **Fleet** in the main menu or use the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/explore-analyze/find-and-organize/find-apps-and-objects).
2. Under the **Agent policies** tab, select the policy you would like to configure.
3. Find the Elastic APM integration and select **Actions → Edit integration**.
4. Navigate to **Agent authorization → Secret token** and set the value of your token.
5. Click Save integration. The APM Server will restart before the change takes effect.

To avoid exposing the secret token, you can store it in a Kubernetes secret. For example:
```cmd
kubectl create secret generic apm-secret --from-literal=ELASTIC_APM_SECRET_TOKEN=asecretpassword --namespace=kube-system 
```

If you’re managing APM Server yourself, see [secret token](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/observability/apm/secret-token) for instructions on how to set up your secret token.
If you are using ECK to set up APM Server, the operator automatically generates an `{{APM-server-name}}-apm-token` secret for you.

### Step 3: Install and configure APM Agents

In most cases, setting up APM agents and thereby instrumenting your applications is as easy as installing a library and adding a few lines of code.
Select your application’s language for details:
<tab-set>
  <tab-item title="Go">
    **Install the agent**Install the APM agent packages for Go.
    ```go
    go get go.elastic.co/apm
    ```
    **Instrument your application**Instrument your Go application by using one of the provided instrumentation modules or by using the tracer API directly.
    ```go
    import (
    	"net/http"

    	"go.elastic.co/apm/module/apmhttp"
    )

    func main() {
    	mux := http.NewServeMux()
    	...
    	http.ListenAndServe(":8080", apmhttp.Wrap(mux))
    }
    ```
    **Configure the agent**Configure the agent using environment variables:
    ```yaml
            # ...
            - name: ELASTIC_APM_SERVER_URL
              value: "apm-server-url-goes-here" 
            - name: ELASTIC_APM_SECRET_TOKEN
              valueFrom:
                secretKeyRef:
                  name: apm-secret
                  key: ELASTIC_APM_SECRET_TOKEN 
            - name: ELASTIC_APM_SERVICE_NAME
              value: "service-name-goes-here" 
    ```
    **Learn more in the agent reference**
    - [Supported technologies](https://docs-v3-preview.elastic.dev/elastic/apm-agent-go/tree/main/reference/supported-technologies)
    - [Advanced configuration](https://docs-v3-preview.elastic.dev/elastic/apm-agent-go/tree/main/reference/configuration)
    - [Detailed guide to instrumenting Go source code](https://docs-v3-preview.elastic.dev/elastic/apm-agent-go/tree/main/reference/set-up-apm-go-agent)
  </tab-item>

  <tab-item title="Java">
    **Attach the agent**The Java agent can instrument supported technologies without any changes to an application image or code. To do this, you’ll need an [init container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) based on an official Elastic APM docker image.Before your application starts, copy the agent from the init container into a shared volume. For example, with the Java agent:
    ```yaml
        # ...
        spec:
          volumes:
          - name: elastic-apm-agent 
            emptyDir: {}
          initContainers:
          - name: elastic-java-agent
            image: docker.elastic.co/observability/apm-agent-java:1.12.0 
            volumeMounts:
            - mountPath: /elastic/apm/agent
              name: elastic-apm-agent
            command: ['cp', '-v', '/usr/agent/elastic-apm-agent.jar', '/elastic/apm/agent'] 
    ```
    The Java command line needs a way to pick up this `javaagent` configuration. You can use the standard JVM TI [JAVA_TOOL_OPTIONS](https://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.md#tooloptions) environment variable to do this. It doesn’t have to be explicitly specified and is picked up automatically by the JVM when it starts.
    <tip>
      For JVMs that don’t support this option, you can use any other environment variable — either one already defined in your startup script, like `JAVA_OPTS` in some servlet container scripts, or add a dedicated empty one that will have no effect if it’s not set.
    </tip>

    ```yaml
          # ...
          containers:
          - name: your-app-container
            env:
            # ...
            - name: JAVA_TOOL_OPTIONS 
              value: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar
    ```
    **Configure the agent**Configure the agent using environment variables:
    ```yaml
            # ...
            - name: ELASTIC_APM_SERVER_URLS
              value: "apm-server-url-goes-here" 
            - name: ELASTIC_APM_SECRET_TOKEN
              valueFrom:
                secretKeyRef:
                  name: apm-secret
                  key: ELASTIC_APM_SECRET_TOKEN 
            - name: ELASTIC_APM_SERVICE_NAME
              value: "service-name-goes-here" 
            - name: ELASTIC_APM_APPLICATION_PACKAGES
              value: "org.springframework.samples.petclinic" 
            - name: JAVA_TOOL_OPTIONS 
              value: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar
    ```
    **Learn more in the agent reference**
    - [Supported technologies](https://docs-v3-preview.elastic.dev/elastic/apm-agent-java/tree/main/reference/supported-technologies)
    - [Advanced configuration](https://docs-v3-preview.elastic.dev/elastic/apm-agent-java/tree/main/reference/configuration)
  </tab-item>

  <tab-item title=".NET">
    <note>
      These instructions are for .NET Core v2.2+. All other use-cases require downloading the agent from NuGet and adding it to your application. See [set up the Agent](https://docs-v3-preview.elastic.dev/elastic/apm-agent-dotnet/tree/main/reference/set-up-apm-net-agent) for full details. Once agent set-up is complete, jump to the **Configure the agent** section on this page.
    </note>
    **Use an init container to download and extract the agent**The .Net agent automatically instruments .NET Core version 2.2 and newer without any application code changes. To do this, you’ll need an [init container](https://kubernetes.io/docs/concepts/workloads/pods/init-containers/) that pulls and unzips the latest agent release:
    ```yaml
        # ...
        spec:
          volumes:
          - name: elastic-apm-agent 
            emptyDir: {}
          initContainers:
          - name: elastic-dotnet-agent
            image: busybox
            command: ["/bin/sh","-c"] 
            args: 
              - wget -qO './elastic-apm-agent/ElasticApmAgent.zip' https://github.com/elastic/apm-agent-dotnet/releases/download/1.7.0/ElasticApmAgent_1.7.0.zip;
                cd elastic-apm-agent;
                cat ElasticApmAgent.zip | busybox unzip -;
            volumeMounts:
            - mountPath: /elastic-apm-agent
              name: elastic-apm-agent
    ```
    To connect the agent to your application, point the `DOTNET_STARTUP_HOOKS` environment variable towards `ElasticApmAgentStartupHook.dll` file that now exists in the `/elastic-apm-agent` directory of the `elastic-apm-agent` volume.
    ```yaml
          # ...
          containers:
          - name: your-app-container
            volumeMounts:
            - mountPath: /elastic-apm-agent
              name: elastic-apm-agent
            env:
            # ...
            - name: DOTNET_STARTUP_HOOKS
              value: "/elastic-apm-agent/ElasticApmAgentStartupHook.dll"
    ```
    **Configure the agent**Configure the agent using environment variables:
    ```yaml
            # ...
            - name: ELASTIC_APM_SERVER_URLS
              value: "apm-server-url-goes-here" 
            - name: ELASTIC_APM_SECRET_TOKEN
              valueFrom:
                secretKeyRef:
                  name: apm-secret
                  key: ELASTIC_APM_SECRET_TOKEN 
            - name: ELASTIC_APM_SERVICE_NAME
              value: "service-name-goes-here" 
            - name: DOTNET_STARTUP_HOOKS 
              value: "/elastic-apm-agent/ElasticApmAgentStartupHook.dll"
    ```
    **Learn more in the agent reference**
    - [Supported technologies](https://docs-v3-preview.elastic.dev/elastic/apm-agent-dotnet/tree/main/reference/supported-technologies)
    - [Advanced configuration](https://docs-v3-preview.elastic.dev/elastic/apm-agent-dotnet/tree/main/reference/configuration)
  </tab-item>

  <tab-item title="Node.js">
    **Install the APM agent**Install the APM agent for Node.js as a dependency to your application.
    ```js
    npm install elastic-apm-node --save
    ```
    **Start the agent**It’s important that the agent is started before you require any other modules in your Node.js application — before `express`, `http`, etc.
    ```js
    var apm = require('elastic-apm-node').start()
    ```
    **Configure the agent**Configure the agent using environment variables:
    ```yaml
            # ...
            - name: ELASTIC_APM_SERVER_URL
              value: "apm-server-url-goes-here" 
            - name: ELASTIC_APM_SECRET_TOKEN
              valueFrom:
                secretKeyRef:
                  name: apm-secret
                  key: ELASTIC_APM_SECRET_TOKEN 
            - name: ELASTIC_APM_SERVICE_NAME
              value: "service-name-goes-here" 
    ```
    **Learn more in the agent reference**
    - [Supported technologies](https://docs-v3-preview.elastic.dev/elastic/apm-agent-nodejs/tree/main/reference/supported-technologies)
    - [Configuring the agent](https://docs-v3-preview.elastic.dev/elastic/apm-agent-nodejs/tree/main/reference/advanced-setup)
  </tab-item>

  <tab-item title="PHP">
    **Install the agent**Install the PHP agent using one of the [published packages](https://github.com/elastic/apm-agent-php/releases).To use the RPM Package (RHEL/CentOS and Fedora):
    ```php
    rpm -ivh <package-file>.rpm
    ```
    To use the DEB package (Debian and Ubuntu):
    ```php
    dpkg -i <package-file>.deb
    ```
    To use the APK package (Alpine):
    ```php
    apk add --allow-untrusted <package-file>.apk
    ```
    If you can’t find your distribution, you can install the agent by [building it from the source](https://docs-v3-preview.elastic.dev/elastic/apm-agent-php/tree/main/reference/set-up-apm-php-agent).**Configure the agent**Configure the agent using environment variables:
    ```yaml
            # ...
            - name: ELASTIC_APM_SERVER_URL
              value: "apm-server-url-goes-here" 
            - name: ELASTIC_APM_SECRET_TOKEN
              valueFrom:
                secretKeyRef:
                  name: apm-secret
                  key: ELASTIC_APM_SECRET_TOKEN 
            - name: ELASTIC_APM_SERVICE_NAME
              value: "service-name-goes-here" 
    ```
    **Learn more in the agent reference**
    - [Supported technologies](https://docs-v3-preview.elastic.dev/elastic/apm-agent-php/tree/main/reference/supported-technologies)
    - [Configuration](https://docs-v3-preview.elastic.dev/elastic/apm-agent-php/tree/main/reference/configuration)
  </tab-item>

  <tab-item title="Python">
    **Install the APM agent**Install the APM agent for Python as a dependency:
    ```python
    # Django
    pip install elastic-apm

    # Flask
    pip install elastic-apm[flask]
    ```
    **Add the agent to your application**For Django, Add `elasticapm.contrib.django` to `INSTALLED_APPS` in your settings:
    ```python
    INSTALLED_APPS = (
       # ...
       'elasticapm.contrib.django',
    )
    ```
    For Flask, initialize the agent for your application using environment variables:
    ```python
    from elasticapm.contrib.flask import ElasticAPM

    app = Flask(__name__)

    apm = ElasticAPM(app)
    ```
    **Configure the agent**Configure the agent using environment variables:
    ```yaml
            # ...
            - name: ELASTIC_APM_SERVER_URL
              value: "apm-server-url-goes-here" 
            - name: ELASTIC_APM_SECRET_TOKEN
              valueFrom:
                secretKeyRef:
                  name: apm-secret
                  key: ELASTIC_APM_SECRET_TOKEN 
            - name: ELASTIC_APM_SERVICE_NAME
              value: "service-name-goes-here" 
    ```
    **Learn more in the agent reference**
    - [Supported technologies](https://docs-v3-preview.elastic.dev/elastic/apm-agent-python/tree/main/reference/supported-technologies)
    - [Advanced configuration](https://docs-v3-preview.elastic.dev/elastic/apm-agent-python/tree/main/reference/configuration)
  </tab-item>

  <tab-item title="Ruby">
    **Install the APM agent**Add the agent to your Gemfile.
    ```ruby
    gem 'elastic-apm'
    ```
    **Start the agent**Rails: APM will automatically start when your app boots.Rack: Include the middleware and start and stop Elastic APM:
    ```ruby
    # config.ru

    app = lambda do |env|
      [200, {'Content-Type' => 'text/plain'}, ['ok']]
    end

    # Wraps all requests in transactions and reports exceptions
    use ElasticAPM::Middleware

    # Start an instance of the Agent
    ElasticAPM.start()

    run app

    # Gracefully stop the agent when process exits.
    # Makes sure any pending transactions have already sent.
    at_exit { ElasticAPM.stop }
    ```
    **Configure the agent**Configure the agent using environment variables:
    ```yaml
            # ...
            - name: ELASTIC_APM_SERVER_URL
              value: "apm-server-url-goes-here" 
            - name: ELASTIC_APM_SECRET_TOKEN
              valueFrom:
                secretKeyRef:
                  name: apm-secret
                  key: ELASTIC_APM_SECRET_TOKEN 
            - name: ELASTIC_APM_SERVICE_NAME
              value: "service-name-goes-here" 
    ```
    **Learn more in the agent reference**
    - [Supported technologies](https://docs-v3-preview.elastic.dev/elastic/apm-agent-ruby/tree/main/reference/supported-technologies)
    - [Advanced configuration](https://docs-v3-preview.elastic.dev/elastic/apm-agent-ruby/tree/main/reference/configuration)
  </tab-item>
</tab-set>


### Step 4: Configure Kubernetes data

In most instances, APM agents automatically read Kubernetes data from inside the container and send it to APM Server. If this is not the case, or if you wish to override this data, you can set environment variables for the agents to read. These environment variable are set via the [Downward API](https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/#the-downward-api) in your Kubernetes pod spec:
```yaml
      # ...
      containers:
      - name: your-app-container
        env:
        # ...
        - name: KUBERNETES_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: KUBERNETES_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: KUBERNETES_POD_UID
          valueFrom:
            fieldRef:
              fieldPath: metadata.uid
```

The table below maps these environment variables to the APM metadata event field:

| Environment variable   | Metadata field name         |
|------------------------|-----------------------------|
| `KUBERNETES_NODE_NAME` | system.kubernetes.node.name |
| `KUBERNETES_POD_NAME`  | system.kubernetes.pod.name  |
| `KUBERNETES_NAMESPACE` | system.kubernetes.namespace |
| `KUBERNETES_POD_UID`   | system.kubernetes.pod.uid   |


### Step 5: Deploy your application

APM agents are deployed with your application.
<dropdown title="Resource configuration file example">
  A complete resource configuration file based on the previous steps.
  ```yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: <<your-app>>
    namespace: kube-system
    labels:
      app: <<your-app>>
      service: <<your-app>>
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: <<your-app>>
    template:
      metadata:
        labels:
          app: <<your-app>>
          service: <<your-app>>
      spec:
        dnsPolicy: ClusterFirstWithHostNet
        volumes:
        - name: elastic-apm-agent
          emptyDir: {}
        initContainers:
        - name: elastic-java-agent
          image: docker.elastic.co/observability/apm-agent-java:1.12.0
          volumeMounts:
          - mountPath: /elastic/apm/agent
            name: elastic-apm-agent
          command: ['cp', '-v', '/usr/agent/elastic-apm-agent.jar', '/elastic/apm/agent']
        containers:
        - name: <<your-app>>
          image: <<your-app>>
          volumeMounts:
          - mountPath: /elastic/apm/agent
            name: elastic-apm-agent
          env:
          - name: ELASTIC_APM_SERVER_URL
            value: "apm-server-url-goes-here"
          - name: ELASTIC_APM_SECRET_TOKEN
            valueFrom:
              secretKeyRef:
                name: apm-secret
                key: ELASTIC_APM_SECRET_TOKEN
          - name: ELASTIC_APM_SERVICE_NAME
            value: "petclinic"
          - name: ELASTIC_APM_APPLICATION_PACKAGES
            value: "org.springframework.samples.petclinic"
          - name: ELASTIC_APM_ENVIRONMENT
            value: test
          - name: JAVA_TOOL_OPTIONS
            value: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar
          - name: KUBERNETES_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: KUBERNETES_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: KUBERNETES_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: KUBERNETES_POD_UID
            valueFrom:
              fieldRef:
                fieldPath: metadata.uid
  ```
</dropdown>

```cmd
kubectl apply -f demo.yml
```


### View your application’s traces in Kibana

Application trace data is available in the **Service Inventory**. To open **Service Inventory**, find **Applications** in the main menu or use the [global search field](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/explore-analyze/find-and-organize/find-apps-and-objects).
The **Applications** app allows you to monitor your software services and applications in real-time: visualize detailed performance information on your services, identify and analyze errors, and monitor host-level and agent-specific metrics like JVM and Go runtime metrics.
![Applications UI Kubernetes](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/images/observability-apm-app-landing.png)

Having access to application-level insights with just a few clicks can drastically decrease the time you spend debugging errors, slow response times, and crashes.
Best of all, because Kubernetes environment variables have been mapped to APM metadata events, you can filter your trace data by Kubernetes `namespace`, `node.name`, `pod.name`, and `pod.uid`.
![Applications UI Kubernetes](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/images/observability-apm-app-kubernetes-filter.png)


## What’s next

- Want to protect your endpoints from security threats? Try [Elastic Security](https://www.elastic.co/security). Adding endpoint protection is just another integration that you add to the agent policy!
- Are your eyes bleary from staring at a wall of screens? [Create alerts](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/observability/incident-management/alerting) and find out about problems while sipping your favorite beverage poolside.
- Want Elastic to do the heavy lifting? Use machine learning to [detect anomalies](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/4431/solutions/observability/logs/inspect-log-anomalies).