Loading

Managed intake service event API

Warning

This API is exclusively for APM agent developers. The vast majority of users should have no reason to interact with this API.

The managed intake service exposes endpoints for:

The managed intake service exposes an API endpoint to query general server information. This lightweight endpoint is useful as a server up/down health check.

Send an HTTP GET request to the server information endpoint:

https://{hostname}:{port}/

This endpoint always returns an HTTP 200.

Requests to this endpoint must be authenticated.

Example managed intake service information request:

curl -X POST http://127.0.0.1:8200/ \
  -H "Authorization: ApiKey api_key"

{
  "build_date": "2021-12-18T19:59:06Z",
  "build_sha": "24fe620eeff5a19e2133c940c7e5ce1ceddb1445",
  "publish_ready": true,
  "version": "8.16.1"
}
Note

Most users do not need to interact directly with the events intake API.

The events intake API is what we call the internal protocol that APM agents use to talk to the managed intake service. Agents communicate with the Server by sending events — captured pieces of information — in an HTTP request. Events can be:

  • Transactions
  • Spans
  • Errors
  • Metrics

Each event is sent as its own line in the HTTP request body. This is known as newline delimited JSON (NDJSON).

With NDJSON, agents can open an HTTP POST request and use chunked encoding to stream events to the managed intake service as soon as they are recorded in the agent. This makes it simple for agents to serialize each event to a stream of newline delimited JSON. The managed intake service also treats the HTTP body as a compressed stream and thus reads and handles each event independently.

Refer to Learn about data types to learn more about the different types of events.

The managed intake service exposes the following endpoints for Elastic APM agent data intake:

Name Endpoint
APM agent event intake /intake/v2/events

Send an HTTP POST request to the managed intake service intake/v2/events endpoint:

https://{hostname}:{port}/intake/v2/events

The managed intake service supports asynchronous processing of batches. To request asynchronous processing the async query parameter can be set in the POST request to the intake/v2/events endpoint:

https://{hostname}:{port}/intake/v2/events?async=true
Note

Since asynchronous processing defers some of the event processing to the background and takes place after the client has closed the request, some errors can’t be communicated back to the client and are logged by the managed intake service. Furthermore, asynchronous processing requests will only be scheduled if the managed intake service can service the incoming request, requests that cannot be serviced will receive an internal error 503 "queue is full" error.

On success, the server will respond with a 202 Accepted status code and no body.

Keep in mind that events can succeed and fail independently of each other. Only if all events succeed does the server respond with a 202.

There are two types of errors that the managed intake service may return to an agent:

  • Event related errors (typically validation errors)
  • Non-event related errors

The managed intake service processes events one after the other. If an error is encountered while processing an event, the error encountered as well as the document causing the error are added to an internal array. The managed intake service will only save 5 event related errors. If it encounters more than 5 event related errors, the additional errors will not be returned to agent. Once all events have been processed, the error response is sent.

Some errors, not relating to specific events, may terminate the request immediately. For example: IP rate limit reached, wrong metadata, etc. If at any point one of these errors is encountered, it is added to the internal array and immediately returned.

An example error response might look something like this:

{
  "errors": [
    {
      "message": "<json-schema-err>",   1
      "document": "<ndjson-obj>"   2
    },{
      "message": "<json-schema-err>",
      "document": "<ndjson-obj>"
    },{
      "message": "<json-decoding-err>",
      "document": "<ndjson-obj>"
    },{
      "message": "too many requests"   3
    },
  ],
  "accepted": 2320   4
}
  1. An event related error
  2. The document causing the error
  3. An immediately returning non-event related error
  4. The number of accepted events

If you’re developing an agent, these errors can be useful for debugging.

The managed intake service uses a collection of JSON Schemas for validating requests to the intake API.

Every new connection to the managed intake service starts with a metadata stanza. This provides general metadata concerning the other objects in the stream.

Rather than send this metadata information from the agent multiple times, the managed intake service hangs on to this information and applies it to other objects in the stream as necessary.

Tip

Metadata is stored under context when viewing documents in Elasticsearch.

The managed intake service uses JSON Schema to validate requests. The specification for metadata is defined on GitHub and included below.

APM agents automatically read Kubernetes data and send it to the managed intake service. In most instances, agents are able to read this data from inside the container. If this is not the case, or if you wish to override this data, you can set environment variables for the agents to read. These environment variable are set via the Kubernetes Downward API. Here’s how you would add the environment variables to your Kubernetes pod spec:

- name: KUBERNETES_NODE_NAME
   valueFrom:
     fieldRef:
       fieldPath: spec.nodeName
 - name: KUBERNETES_POD_NAME
   valueFrom:
     fieldRef:
       fieldPath: metadata.name
 - name: KUBERNETES_NAMESPACE
   valueFrom:
     fieldRef:
       fieldPath: metadata.namespace
 - name: KUBERNETES_POD_UID
   valueFrom:
     fieldRef:
       fieldPath: metadata.uid

The table below maps these environment variables to the APM metadata event field:

Environment variable Metadata field name
KUBERNETES_NODE_NAME system.kubernetes.node.name
KUBERNETES_POD_NAME system.kubernetes.pod.name
KUBERNETES_NAMESPACE system.kubernetes.namespace
KUBERNETES_POD_UID system.kubernetes.pod.uid

Transactions are events corresponding to an incoming request or similar task occurring in a monitored service.

The managed intake service uses JSON Schema to validate requests. The specification for transactions is defined on GitHub and included below.

Spans are events captured by an agent occurring in a monitored service.

The managed intake service uses JSON Schema to validate requests. The specification for spans is defined on GitHub and included below.

An error or a logged error message captured by an agent occurring in a monitored service.

The managed intake service uses a JSON Schema to validate requests. The specification for errors is defined on GitHub and included below.

Metrics contain application metric data captured by an APM agent.

The managed intake service uses JSON Schema to validate requests. The specification for metrics is defined on GitHub and included below.

Elastic supports receiving traces, metrics, and logs over the OpenTelemetry Protocol (OTLP). OTLP is the default transfer protocol for OpenTelemetry and is supported natively by the managed intake service.

The managed intake service supports two OTLP communication protocols on the same port:

  • OTLP/HTTP (protobuf)
  • OTLP/gRPC
Name Endpoint
OTLP metrics intake /opentelemetry.proto.collector.metrics.v1.MetricsService/Export
OTLP trace intake /opentelemetry.proto.collector.trace.v1.TraceService/Export
OTLP logs intake /opentelemetry.proto.collector.logs.v1.LogsService/Export
Name Endpoint
OTLP metrics intake /v1/metrics
OTLP trace intake /v1/traces
OTLP logs intake /v1/logs
Tip

See our OpenTelemetry docs to learn how to send data to the managed intake service from an OpenTelemetry agent OpenTelemetry collector.