Context Engine concepts
Context Engine has four core building blocks:
Optional agent traces provide feedback about how well that context supports real questions.
Agents and applications use tools and instructions to retrieve and apply the resulting context.
For a hands-on introduction to these building blocks, follow Get started with Context Engine.
An AI index groups the metadata, sources, automations, optional agent traces, and Elasticsearch storage for a particular body of context. Its name and description help an agent determine whether that context is relevant to a question. Its generated Knowledge Indicators (KIs) provide the context that agents and applications query.
Creating an AI index does not ingest source data or generate knowledge. Sources define the data that its automations can analyze, and KIs are stored as the generated context. To plan and maintain one, refer to Build and maintain an AI index. For the complete UI-led example, follow Get started with Context Engine.
A managed AI index is a built-in AI index supplied by an Elastic integration. The owning Elastic feature defines its configuration and supplies its KIs, giving agents access to product-provided context without requiring you to configure its sources or automations.
Managed AI indices have a Managed badge in Context. Agents can discover and query them in the same way as user-created AI indices, subject to the same access checks. You can open a managed AI index and inspect its KIs, but you cannot edit its description, sources, automations, or agent traces, or delete it. To build context for your own use case, create a user-created AI index instead.
A source is the data from which an automation generates Knowledge Indicators (KIs). Context Engine supports two source types:
- An ES|QL source uses a query to select data from one or more Elasticsearch indices or data streams.
- A connector source names a configured connector to an external system. Supported connectors include services such as Google Drive, GitHub, Jira, ServiceNow, and Slack, and cloud object stores. Refer to Connectors in Elastic Agent Builder for more information about configuring connectors.
An ES|QL source can provide a small sample that grounds an automation proposal. The source query is not necessarily the only query the resulting Workflow runs. A Workflow can inspect mappings, take other samples, or calculate full-dataset aggregations. To add an ES|QL source and review the generated Workflow, follow Get started with Context Engine.
Agent traces record how an agent runs, including its model calls and tool calls. An AI index can reference traces from an Elastic Agent Builder agent or from a data stream that contains OpenTelemetry generative AI spans.
Context Engine can analyze these traces to identify query errors, empty retrievals, and cases where an agent queries raw data because its KIs do not cover the question. This feedback can expose gaps in the index's sources, automations, or generated context. Agent traces are feedback about context use. They are not sources from which automations generate KIs.
For information about trace collection, contents, privacy, and access, refer to Collect Elastic Agent Builder traces.
An automation generates or refreshes Knowledge Indicators (KIs) from an AI index's sources. Context Engine implements each automation as an Elastic Workflow, which defines the operations and instructions used to analyze the source and write the result.
The Workflow can retrieve mappings, run ES|QL samples and aggregations, generate structured content with an AI prompt, assemble the KI, verify generated queries, and write the KI when verification passes. You can inspect the Workflow to verify the scope of its queries and the evidence behind its output. Refer to Anatomy of a workflow to understand its triggers, constants, steps, and execution lifecycle.
Passing syntax and runtime checks means that a generated query parses and runs. It does not confirm that the query's fields, grouping, or calculations answer the intended question. To review and run a generated automation, follow Get started with Context Engine.
A Knowledge Indicator (KI) is a document that records reusable context derived from raw source data. Automations generate KIs and store them in an AI index, where agents retrieve them to answer recurring questions without repeatedly finding and interpreting the same information. This can reduce response time and model token use. Depending on the use case, a KI can contain business meaning, derived findings, source limitations, or verified query patterns for retrieving current details.
The generation strategy determines how an automation divides source data into KIs and how much information each KI contains. To examine a generated KI and evaluate its usefulness, refer to Evaluate and improve Knowledge Indicators.
Agents and applications retrieve KIs from an AI index as context. In Elastic Agent Builder, you make an AI index available by adding it to an agent's configuration. Other agent frameworks and applications can use supported Context Engine operations through Model Context Protocol (MCP) tools or APIs.
At retrieval time, an agent or application:
- Lists the AI indices it can access and selects one that is relevant to the question.
- Describes that AI index to discover its fields, KI types, tags, and example queries.
- Queries the AI index for relevant KIs.
To configure retrieval for an agent, refer to Use Context Engine with agents. For the built-in integration, follow Use Context Engine with Elastic Agent Builder.
A tool is an operation an agent can perform. MCP exposes tools to compatible clients, while APIs let applications call supported operations directly or wrap them as tools.
System instructions and skills guide an agent in deciding when and how to use those operations. A skill packages reusable instructions and can associate them with tools and reference content, but it is not itself a tool, connection mechanism, or agent framework. Refer to the Context Engine entries in the built-in skills reference for the skills that support context generation, evaluation, and retrieval.