Use internal collection to send monitoring data

Use internal collectors to send Beats monitoring data directly to your monitoring cluster. Or as an alternative to internal collection, use Use Metricbeat collection. The benefit of using internal collection instead of Metricbeat is that you have fewer pieces of software to install and maintain.

  1. Create an API key or user that has appropriate authority to send system-level monitoring data to Elasticsearch. For example, you can use the built-in apm_system user or assign the built-in apm_system role to another user. For more information on the required privileges, see Create a monitoring role. For more information on how to use API keys, see Grant access using API keys.
  2. Add the monitoring settings in the APM Server configuration file. If you configured the Elasticsearch output and want to send APM Server monitoring events to the same Elasticsearch cluster, specify the following minimal configuration:
    			monitoring:
  enabled: true
  elasticsearch:
    api_key:  id:api_key 1
    username: apm_system
    password: somepassword
    1. Specify one of api_key or username/password.
    If you want to send monitoring events to an Elastic Cloud monitoring cluster, you can use two simpler settings. When defined, these settings overwrite settings from other parts in the configuration. For example:
    			monitoring:
  enabled: true
  cloud.id: 'staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRjZWM2ZjI2MWE3NGJmMjRjZTMzYmI4ODExYjg0Mjk0ZiRjNmMyY2E2ZDA0MjI0OWFmMGNjN2Q3YTllOTYyNTc0Mw=='
  cloud.auth: 'elastic:{pwd}'
    If you configured a different output, such as Logstash or you want to send APM Server monitoring events to a separate Elasticsearch cluster (referred to as the monitoring cluster), you must specify additional configuration options. For example:
    			monitoring:
  enabled: true
  cluster_uuid: PRODUCTION_ES_CLUSTER_UUID 1
  elasticsearch:
    hosts: ["https://example.com:9200", "https:2
    api_key:  id:api_key 3
    username: apm_system
    password: somepassword
    1. This setting identifies the Elasticsearch cluster under which the monitoring data for this APM Server instance will appear in the Stack Monitoring UI. To get a cluster’s cluster_uuid, call the GET / API against that cluster.
    2. This setting identifies the hosts and port numbers of Elasticsearch nodes that are part of the monitoring cluster.
    3. Specify one of api_key or username/password.
    If you want to use PKI authentication to send monitoring events to Elasticsearch, you must specify a different set of configuration options. For example:
    			monitoring:
  enabled: true
  cluster_uuid: PRODUCTION_ES_CLUSTER_UUID
  elasticsearch:
    hosts: ["https://example.com:9200", "https://example2.com:9200"]
    username: ""
    ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
    ssl.certificate: "/etc/pki/client/cert.pem"
    ssl.key: "/etc/pki/client/cert.key"
    You must specify the username as "" explicitly so that the username from the client certificate (CN) is used. See SSL/TLS output settings for more information about SSL settings.
  3. Start APM Server.
  4. View the monitoring data in Kibana.

Settings for internal collection ¶

Use the following settings to configure internal collection when you are not using Metricbeat to collect monitoring data.

You specify these settings in the X-Pack monitoring section of the apm-server.yml config file:

monitoring.enabled ¶

The monitoring.enabled config is a boolean setting to enable or disable X-Pack monitoring. If set to true, monitoring is enabled.

The default value is false.

monitoring.elasticsearch ¶

The Elasticsearch instances that you want to ship your APM Server metrics to. This configuration option contains the following fields:

api_key ¶

The detail of the API key to be used to send monitoring information to Elasticsearch. See Grant access using API keys for more information.

bulk_max_size ¶

The maximum number of metrics to bulk in a single Elasticsearch bulk API index request. The default is 50. For more information, see Elasticsearch.

backoff.init ¶

The number of seconds to wait before trying to reconnect to Elasticsearch after a network error. After waiting backoff.init seconds, APM Server tries to reconnect. If the attempt fails, the backoff timer is increased exponentially up to backoff.max. After a successful connection, the backoff timer is reset. The default is 1s.

backoff.max ¶

The maximum number of seconds to wait before attempting to connect to Elasticsearch after a network error. The default is 60s.

compression_level ¶

The gzip compression level. Setting this value to 0 disables compression. The compression level must be in the range of 1 (best speed) to 9 (best compression). The default value is 0. Increasing the compression level reduces the network usage but increases the CPU usage.

headers ¶

Custom HTTP headers to add to each request. For more information, see Elasticsearch.

hosts ¶

The list of Elasticsearch nodes to connect to. Monitoring metrics are distributed to these nodes in round robin order. For more information, see Elasticsearch.

max_retries ¶

The number of times to retry sending the monitoring metrics after a failure. After the specified number of retries, the metrics are typically dropped. The default value is 3. For more information, see Elasticsearch.

parameters ¶

Dictionary of HTTP parameters to pass within the URL with index operations.

password ¶

The password that APM Server uses to authenticate with the Elasticsearch instances for shipping monitoring data.

metrics.period ¶

The time interval (in seconds) when metrics are sent to the Elasticsearch cluster. A new snapshot of APM Server metrics is generated and scheduled for publishing each period. The default value is 10 * time.Second.

state.period ¶

The time interval (in seconds) when state information are sent to the Elasticsearch cluster. A new snapshot of APM Server state is generated and scheduled for publishing each period. The default value is 60 * time.Second.

protocol ¶

The name of the protocol to use when connecting to the Elasticsearch cluster. The options are: http or https. The default is http. If you specify a URL for hosts, however, the value of protocol is overridden by the scheme you specify in the URL.

proxy_url ¶

The URL of the proxy to use when connecting to the Elasticsearch cluster. For more information, see Elasticsearch.

timeout ¶

The HTTP request timeout in seconds for the Elasticsearch request. The default is 90.

ssl ¶

Configuration options for Transport Layer Security (TLS) or Secure Sockets Layer (SSL) parameters like the certificate authority (CA) to use for HTTPS-based connections. If the ssl section is missing, the host CAs are used for HTTPS connections to Elasticsearch. For more information, see SSL/TLS output settings.

username ¶

The user ID that APM Server uses to authenticate with the Elasticsearch instances for shipping monitoring data.