Debug standalone Elastic Agents

When you run standalone Elastic Agents, you are responsible for monitoring the status of your deployed Elastic Agents. You cannot view the status or logs in Fleet.

Use the following tips to help identify potential issues.

Also refer to Troubleshoot common problems for guidance on specific problems.

To check the status of the running Elastic Agent daemon and other processes managed by Elastic Agent, run the status command. For example:

elastic-agent status

Returns something like:

State: HEALTHY
Message: Running
Fleet State: STOPPED
Fleet Message: (no message)
Components:
  * log           (HEALTHY)
                  Healthy: communicating with pid '25423'
  * filestream    (HEALTHY)
                  Healthy: communicating with pid '25424'

By default, this command returns the status in human-readable format. Use the --output flag to change it to json or yaml.

For more information about this command, refer to elastic-agent status.

If the Elastic Agent status is unhealthy, or behaving unexpectedly, inspect the logs of the running Elastic Agent.

The log location varies by platform. Elastic Agent logs are in the folders described in Installation layout. Beats and Fleet Server logs are in folders named for the output (for example, default).

Start by investigating any errors you see in the Elastic Agent and related logs. Also look for repeated lines that might indicate problems like connection issues. If the Elastic Agent and related logs look clean, check the host operating system logs for out of memory (OOM) errors related to the Elastic Agent or any of its processes.

The log level of the running agent is set to info by default. At this level, Elastic Agent will log informational messages, including the number of events that are published. It also logs any warnings, errors, or critical errors.

To increase the log level, set it to debug in the elastic-agent.yml file.

The debug setting configures Elastic Agent to log debug messages, including a detailed printout of all flushed events, plus all the information collected at other log levels.

Set other options if you want write logs to a file. For example:

agent.logging.level: debug
agent.logging.to_files: true
agent.logging.files:
  path: /var/log/elastic-agent
  name: elastic-agent
  keepfiles: 7
  permissions: 0600

For other log settings, refer to Logging.

Profiling data produced by the /debug/pprof/ endpoints can be useful for debugging, but presents a security risk. Do not expose these endpoints if the monitoring endpoint is accessible over a network. (By default, the monitoring endpoint is bound to a local Unix socket or Windows npipe and not accessible over a network.)

To expose the /debug/pprof/ endpoints, set agent.monitoring.pprof: true in the elastic-agent.yml file. For more information about monitoring settings, refer to Monitoring.

After exposing the endpoints, you can access the HTTP handler bound to a socket for Beats or the Elastic Agent. For example:

sudo curl --unix-socket /Library/Elastic/Agent/data/tmp/default/filebeat/filebeat.sock http://socket/ | json_pp

Returns something like:

{
   "beat" : "filebeat",
   "binary_arch" : "amd64",
   "build_commit" : "93708bd74e909e57ed5d9bea3cf2065f4cc43af3",
   "build_time" : "2022-01-28T09:53:29.000Z",
   "elastic_licensed" : true,
   "ephemeral_id" : "421e2525-9360-41db-9395-b9e627fbbe6e",
   "gid" : "0",
   "hostname" : "My-MacBook-Pro.local",
   "name" : "My-MacBook-Pro.local",
   "uid" : "0",
   "username" : "root",
   "uuid" : "fc0cc98b-b6d8-4eef-abf5-2d5f26adc7e8",
   "version" : "7.17.0"
}

Likewise, the following request:

sudo curl --unix-socket /Library/Elastic/Agent/data/tmp/elastic-agent.sock http://socket/stats | json_pp

Returns something like:

{
   "beat" : {
      "cpu" : {
         "system" : {
            "ticks" : 16272,
            "time" : {
               "ms" : 16273
            }
         },
         "total" : {
            "ticks" : 42981,
            "time" : {
               "ms" : 42982
            },
            "value" : 42981
         },
         "user" : {
            "ticks" : 26709,
            "time" : {
               "ms" : 26709
            }
         }
      },
      "info" : {
         "ephemeral_id" : "ea8fec0d-f7dd-4577-85d7-a2c38583c9c6",
         "uptime" : {
            "ms" : 5885653
         },
         "version" : "7.17.0"
      },
      "memstats" : {
         "gc_next" : 13027776,
         "memory_alloc" : 7771632,
         "memory_sys" : 39666696,
         "memory_total" : 757970208,
         "rss" : 58990592
      },
      "runtime" : {
         "goroutines" : 101
      }
   },
   "system" : {
      "cpu" : {
         "cores" : 12
      },
      "load" : {
         "1" : 4.8892,
         "15" : 2.6748,
         "5" : 3.0537,
         "norm" : {
            "1" : 0.4074,
            "15" : 0.2229,
            "5" : 0.2545
         }
      }
   }
}

To inspect the running Elastic Agent configuration use the elastic-agent inspect command.

To analyze the current state of the agent, inspect log files, and see the configuration of Elastic Agent and the sub-processes it starts, run the diagnostics command. For example:

elastic-agent diagnostics

For more information about this command, refer to elastic-agent diagnostics.