Loading

Capture Elastic Agent diagnostics

Elastic's diagnostic tools capture point-in-time snapshots of Fleet and Elastic Agent related statistics and logs. They work against all product versions.

Their information can be used to troubleshoot problems with your setup. You can generate diagnostic information using this tool before escalating to us to minimize turnaround time.

As explained under What is Fleet Server?, for Fleet-managed Elastic Agent, related settings and states can by surfaced by:

Standalone Elastic Agents are not associated to Kibana nor Fleet, but their diagnostics are still accessible from the CLI.

To pull data from the respective applicable locations, you will refer to:

  • Capture Kibana diagnostics
  • Collect Elastic Agent diagnostics (below)

    • Using the CLI

    • Using the Fleet UI

      If you need to pull an Elastic Agent diagnostic for the Fleet Server, as a hosted service you cannot access the CLI directly so will need to grab the diagnostic using the Fleet UI. The Fleet agent will be associated to the managed Agent policy named "Elastic Cloud agent policy". :::

You will need to determine which diagnostic types are needed to investigate your specific issue. The following are common troubleshooting situations and which diagnostics are commonly associated:

Situation Kibana Elastic Agent Fleet
Kibana reports no Fleet server Yes No Yes
Elastic Agent unable to connect to Fleet No Yes Yes
Elastic Agent component or integration errors No Yes No
Elastic Agent update issues or desynced status Yes Yes No

When in doubt, start with the Kibana and Elastic Agent diagnostics.

Tip

Some Elastic Agent configuration issues only appear in their start-up debug logs. This is more common for cloud-connecting Elastic Integrations which maintain checkpoints. This can cause later logs to only document that the subprocess has stopped or that it has not changed state from an earlier error. In these situations, you will want to

  1. Enable debug logging.
  2. Restart Elastic Agent.
  3. Wait 10 minutes for changes to sync from Fleet server to Elastic Agent and for it to restart.
  4. Pull the Elastic Agent diagnostic using prefered method.
  5. Disable debug logging.

Elastic Agent comes bundled with a diagnostics command which generates a zip archive containing troubleshooting diagnostic information. This export is intended only for debugging purposes and its structure can change between releases.

The Fleet UI provides the ability to remotely generate and gather an Elastic Agent's diagnostics bundle if it is online in a Healthy or Unhealthy status. For Elastic Agents in other statuses, you must use the CLI to grab their diagnostic.

Warning

Diagnostics and logs mainly emit product metadata and settings, but they might expose sensitive data which needs to be redacted before being shared outside of your organization. Refer to Sanitize

The diagnostics are sent to Fleet Server which in turn adds it into Elasticsearch. Therefore, this works even with Elastic Agents that are not using the Elasticsearch output. To download the diagnostics bundle for local viewing:

  1. In Fleet, open the Agents tab.

  2. In the Host column, click the agent’s name.

  3. Select the Diagnostics tab and click the Request diagnostics .zip button.

    Collect agent diagnostics under agent details

  4. In the Request Diagnostics pop-up, select Collect additional CPU metrics if you’d like detailed CPU data.

    Collect agent diagnostics confirmation pop-up

  5. Click the Request diagnostics button.

When available, the new diagnostic bundle will be listed on this page, as well as any in-progress or previously collected bundles for the Elastic Agent.

Note that the bundles are stored in Elasticsearch and are removed automatically after 7 days. You can also delete any previously created bundle by clicking the trash can icon.

Run the diagnostics command in the Elastic Agent's install directory. For your convenience, we have outlined pulling the Elastic Agent diagnostic with default install directory for the most common Operating Systems:

  • Linux-based systems

    cd /opt/Elastic/Agent
.elastic-agent diagnostics

  • Windows Powershell

    cd "C:\Program Files\Elastic\Agent"
.\elastic-agent.exe diagnostics

  • Apple MacOS

    sudo -i
cd /Library/Elastic/Agent
./elastic-agent diagnostics

  • Docker

    1. Determine the container ID with Docker ps.
    docker ps | grep "beats/elastic-agent"
    1. Docker exec run the diagnostic, replacing placeholder CONTAINER_ID.
    docker exec -it CONTAINER_ID elastic-agent diagnostics

    Take note of the output diagnostic filename and location.

    1. Docker cp the outputted diagnostic filename to your local machine, replacing placeholders CONTAINER_ID and FILE_NAME.
    docker cp CONTAINER_ID:/usr/share/elastic-agent/FILE_NAME FILE_NAME

  • Kubernetes

    1. Determine the container ID with get.
    kubectl get pods --all-namespaces | grep agent
    1. Run the diagnostic with exec, replacing placeholders NAMESPACE and POD_NAME.
    kubectl exec --stdin --tty POD_NAME -n NAMESPACE -- elastic-agent diagnostics

    Take note of the output diagnostic filename and location.

    1. cp the outputted diagnostic filename to your local machine, replacing placeholders NAMESPACE, POD_NAME. and FILE_NAME.
    kubectl cp NAMESPACE/POD_NAME:FILE_NAME FILE_NAME

Elastic Agent attempts to automatically redact credentials and API keys when creating its diagnostic files. Review the contents of the archive before sharing to ensure that all forms of organizationally-private information is censored as needed. For example, ensure:

  • There are no credentials in plain text in its *.yaml diagnostic files.

  • The raw form of event's failing to output can show under *.ndjson. By default only warn log. When the debug logging level is enabled, all events are logged. If you want to omit the raw events from the diagnostic, add the flag --exclude-events.