Loading

Collect Elastic Agent Builder traces

Elastic Agent Builder can collect agent execution traces into your Elasticsearch deployment. Traces record how each agent round runs, including model calls, tool calls, latency, and token usage, so you can monitor agent activity, debug behavior, and build dashboards on the data.

Traces are stored in Elasticsearch, but you do not have to query the data stream to read them. Elastic Agent Builder gives you three ways to work with collected traces:

To do this Use
Debug a single agent response, step by step The View Trace waterfall on a conversation round in Agent Chat
Monitor token usage, latency, and tool errors across all agents The prebuilt overview dashboard
Ask questions about trace data in natural language The built-in traces skill

You can also explore the raw spans yourself in Discover or with ES|QL, as described in Build dashboards on trace data.

When an agent runs, Elastic Agent Builder records the run as OpenTelemetry (OTel) traces. Each trace covers one conversation round. A trace is made up of spans that map to the work the agent did, such as model calls and tool calls.

Trace collection is space-aware. Each Kibana space writes its traces to its own data stream, named with the space id, such as traces-agent_builder.otel-default for the default space. Use the traces-agent_builder.otel-* wildcard to work with every space's traces at once.

Elastic Agent Builder ingests this data into managed OpenTelemetry data streams in your Elasticsearch deployment. Execution spans, such as model calls and tool calls, are stored in traces-agent_builder.otel-*, with their timings, token usage, model, and status.

When you opt in to capturing conversation content, that content is stored as span attributes in traces-agent_builder.otel-*. The chat spans contain the chat history, the model responses, and the system prompt, and the execute_tool spans contain the arguments and results of each tool call. Content is captured only when you enable it in Trace privacy settings.

These data streams are OTel-compatible and use the standard OTel index templates, so they inherit the mappings, settings, and data lifecycle that Elasticsearch maintains for OTel data.

These are regular data streams, not system or hidden indices. You can explore and analyze the data with the same tools you use for any other data in Elasticsearch, including Discover, Dashboards, Lens, and ES|QL.

Each trace is a set of spans that follow a run from the overall conversation round down to its individual steps, including:

  • Each agent execution.
  • Each model call.
  • Each tool call.

Spans follow OpenTelemetry semantic conventions for generative AI (currently experimental) and contain generative AI attributes for the model, the provider, and token usage. Use them to break down usage and latency by model, agent, or tool. For the exact fields and the prebuilt visualizations that use them, refer to Build dashboards on trace data.

By default, traces record structural metadata only. Conversation content such as prompts and responses is excluded unless an administrator opts in. For details, refer to Trace privacy settings.

Trace collection is on by default. To manage it, go to Stack Management → GenAI Settings and open the Agent Builder Traces section.

The Agent Builder Traces section in GenAI Settings, with trace collection enabled and the Install Dashboard button

The Collect conversation traces setting turns collection on and off. When it is on, Elastic Agent Builder collects OpenTelemetry traces for agent conversations and ingests them into Elasticsearch. From the same section, you can install a prebuilt overview dashboard for the current Kibana space.

Note

Any user with index access can read trace data. To restrict access, configure index-level privileges in Stack Management → Roles. For details, refer to Grant access to trace data.

By default, traces record structural metadata only, such as token counts, latency, and model names. Conversation content is not captured unless an administrator opts in.

To change what is captured, expand Advanced privacy settings in the Agent Builder Traces section. Each option is off by default.

The expanded Advanced privacy settings, showing six toggles for including sensitive content in traces, all turned off
Setting Setting ID Effect when enabled
Include user prompts in traces agentBuilder:tracing:includeUserPrompts Captures user messages.
Include LLM responses in traces agentBuilder:tracing:includeLlmResponses Captures agent responses.
Include tool call details in traces agentBuilder:tracing:includeToolDetails Captures tool call arguments and results.
Include system prompt in traces agentBuilder:tracing:includeSystemPrompt Captures agent instructions.
Include real tool and agent names in traces agentBuilder:tracing:includeRealNames Records real tool and agent names instead of anonymized values.
Include real conversation and workflow IDs in traces agentBuilder:tracing:includeRealIds Records real conversation and workflow IDs instead of anonymized values.
Note

Built-in tools and agents always appear under their real names. Anonymized names are replaced with the literal value custom, so every custom tool, agent, and workflow shares one value and you cannot tell them apart by name. Anonymized IDs are different: they are replaced with a stable hash, so you can still group and correlate traces by conversation or agent ID.

Content is stored across different span types:

  • chat spans: Store prompts, responses, and the system prompt in the attributes.gen_ai.input.messages, attributes.gen_ai.output.messages, and attributes.gen_ai.system_instructions attributes.
  • execute_tool spans: Store tool call details in attributes.gen_ai.tool.call.arguments and attributes.gen_ai.tool.call.result.

Anyone who can read the trace data stream can read this content, so review Grant access to trace data before you turn these settings on. For the field-level details, refer to Message content attributes.

Trace data is stored in the traces-agent_builder.otel-* data stream. To read it, a role needs read and view_index_metadata on that pattern.

Access is granted at the index level. Any user who can read these data streams can read all collected traces, so trace access is not scoped per user. To control who can read traces, configure index privileges through roles in Stack Management → Roles.

For the full privilege model, including Kibana feature and cluster privileges, refer to Permissions and access control.

When trace collection is on, Elastic Agent Builder provides a prebuilt overview dashboard for agent activity and token usage. You install or reinstall it per space from the Agent Builder Traces settings section. For what each panel shows and the full span and attribute reference, refer to Agent Builder traces overview dashboard.

Because traces are stored in regular data streams, you can also build your own visualizations with Dashboards and Lens, or query the data with ES|QL. To explore traces in natural language, use the built-in traces skill.