Knowledge Indicators

Knowledge Indicators (KIs) are structured facts that Elastic extracts from your raw log data automatically without requiring schemas, service catalogs, or manual configuration. When you run extraction against a log stream, Elastic analyzes the raw data and returns facts about your environment: which services are running, the underlying infrastructure they rely on, how they depend on each other, and the log schemas they use.

Rather than a static configuration, this knowledge accumulates over time, automatically expires when a service disappears, and feeds directly into downstream capabilities like the Significant Events detection pipeline, topology maps, AI agent investigations, and dashboards.

To access Knowledge Indicators, open Significant Events from the Streams main page and select the Knowledge Indicators tab.

You can trigger KI extraction on demand or set up continuous extraction at a specific interval.

On demand

From the Significant Events page, select the streams you want to generate KIs for and select Generate.

Continuous extraction

When enabled, continuous extraction runs automatically on managed streams at the interval you configure. Continuous extraction is off by default. To enable it:

1. From the Streams main page, select Significant Events → Settings.
2. Under Continuous KI extraction, turn on Enable continuous KI extraction.
3. Set the Extraction interval in hours, and list any Excluded streams to skip during continuous extraction.

The extraction pipeline samples a small batch of logs from a stream and processes them through a combination of large language model (LLM) analysis and deterministic code generators. It accumulates its findings across multiple iterations, entirely configuration-free.

The pipeline runs up to five iterations. Each iteration fetches a batch of up to 20 documents assembled from three prioritized buckets:

KIs found in one iteration are fed back as exclusions into the next, so each round focuses on what the previous one missed. This ensures quieter, less-represented services are not crowded out by high-volume log categories.

For example, from a single nginx access log line:

192.168.1.45 - - [31/Mar/2026:14:23:01 +0000] "POST /api/v2/claims HTTP/1.1" 200 1247 "-" "claim-intake/1.4.2"
		

The pipeline extracts:

• Entity: claim-intake (identifiable as a service from the User-Agent)
• Version: 1.4.2 (extracted from the User-Agent string)
• Technology: nginx (the web server fielding the request)
• Schema: Combined Log Format

Similarly, from a Java service log:

2026-03-31T14:23:03.412Z INFO fraud-check --- [nio-8080-exec-3] c.e.FraudCheckService : Calling upstream POST http://policy-lookup:8081/v1/policy latency=142ms status=200
		

The pipeline extracts:

• Entity: fraud-check (a Spring Boot service)
• Dependency: fraud-check → policy-lookup (through an outbound HTTP call)
• Technology: Java, Spring Boot

Sampled documents are sent to an LLM that identifies the following feature types:

Every feature must include stable identifying properties and cite direct evidence from the sampled logs. The LLM assigns a confidence score from 0–100 (where 0 is the lowest confidence and 100 is the highest) for each KI. The pipeline also tracks features excluded by users (false positives) and carries them forward to prevent re-identification in future runs.

In parallel with LLM analysis, a set of deterministic code-based generators independently analyze the data to produce statistical summaries, log samples, pattern clusters, and error-specific features. Because these are computed rather than inferred, they always receive a confidence score of 100.

LLM results and computed features are merged and deduplicated. Known KIs reuse their existing UUIDs, new discoveries get fresh ones, and the pipeline drops user-excluded features. Surviving KIs are saved with an active status and an expiration date set 30 days out.

Extraction runs entirely as a background task and never blocks ingestion.

Knowledge Indicators fall into two categories: Feature KIs and Query KIs.

Feature KIs are descriptive and explain the contents of the stream: what services are running, the infrastructure housing them, their dependencies, and the active tech stack.

Feature KIs carry a full data model:

• type / subtype: The category of the fact (Entity, Infrastructure, Technology, Dependency, Schema)
• title / description: A human-readable summary
• properties: Stable key-value pairs used to deduplicate findings across multiple runs
• confidence: 0–100. LLM-identified KIs score based on evidence quality. Deterministic KIs always score 100.
• evidence: 2–5 supporting log excerpts that justify the KI's existence
• filter: An optional StreamLang condition scoping the KI to specific documents

Example dependency KI:

{
  "type": "dependency",
  "subtype": "service_dependency",
  "title": "api_gateway → inference_service",
  "description": "Service-to-service HTTP dependency from api_gateway to inference_service, observed in request logs",
  "properties": {
    "source": "api_gateway",
    "target": "inference_service",
    "protocol": "http"
  },
  "confidence": 85,
  "evidence": [
    "service.name=api_gateway http.url=/v1/inference peer.service=inference_service",
    "upstream=inference_service:8080 request=POST /v1/inference 200"
  ],
  "filter": { "field": "service.name", "eq": "api_gateway" },
  "status": "active",
  "expires_at": "2026-04-09T00:00:00Z"
}
		

The properties field keeps Feature KIs stable across multiple pipeline runs. When extraction runs again, Elastic recognizes an existing relationship and updates the KI's last_seen timestamp rather than creating a duplicate.

Query KIs are actionable. They are ready-to-run ES|QL queries targeting notable conditions like connection exhaustion, out-of-memory errors, or fatal exceptions. Each comes with a severity score from 0 to 100.

Example query KI:

{
  "kind": "query",
  "title": "PostgreSQL connection slot exhaustion",
  "description": "Fires when Postgres runs out of available connection slots",
  "severity_score": 90,
  "esql": {
    "query": "FROM logs-* | WHERE service.name == \"postgres\" AND message : \"remaining connection slots\""
  }
}
		

Query KIs are generated in two ES|QL forms depending on what the LLM determines is most useful for detection:

MATCH queries filter for specific log events, such as errors, exceptions, failure messages:

FROM logs-mystream,logs-mystream.* METADATA _id, _source
| WHERE message : "connection refused" AND service.name == "api_gateway"
		

STATS queries aggregate metrics over time, such as rates, counts, or derived values. These are useful when the signal is a change in volume rather than the presence of a specific event:

FROM logs-mystream,logs-mystream.*
| STATS error_count = COUNT(*) WHERE level == "ERROR" BY service.name, @timestamp
		

When you promote a query KI, it becomes an alerting rule that runs its ES|QL query on a schedule and writes a match count to .rule-events for each time bucket. Only MATCH queries can be promoted; STATS queries cannot be converted to alerting rules yet. The number of promoted query KIs, and how often each one executes, is the main driver of alerting query load on your cluster.

The Significant Events pipeline picks up from there. A Detection workflow runs change point aggregation against those per-rule counts and writes a document to .significant_events-detections when statistically significant shifts are found. A Discovery workflow then uses an LLM to process those documents.

When continuous extraction is enabled, a workflow runs every 35 minutes and processes up to five eligible streams per run. A stream is eligible if:

• It is a wired, classic, or query stream that matches the configured Significant Events index patterns
• No feature identification is currently running for it
• No feature identification task is currently running for it
• Enough time has elapsed since its last extraction (controlled by the Extraction interval setting, default 12 hours)

Streams that have never been processed are always prioritized. Among remaining candidates, streams with the oldest last-completed extraction run are processed first.

The continuous extraction workflow has a 34-minute timeout (one minute shorter than the 35-minute schedule, to prevent overlapping runs). If multiple runs would overlap, excess triggers are silently dropped.

Toggling continuous extraction off cancels any in-flight extraction tasks and disables the workflow. Re-enabling turns it back on. Already-extracted KIs are preserved throughout.

These settings are only modifiable through the Significant Events Settings page.

KIs auto-expire after 30 days if not observed in subsequent extraction runs. KIs for decommissioned services are automatically removed without manual cleanup. If a service comes back online, its KIs are re-extracted automatically.

Users can mark individual Feature KIs as false positives. The system carries those exclusions forward into future runs to prevent re-identification.