Set rule data sources

Every detection rule needs a data source that tells it which Elasticsearch indices to query. By default, rules inherit the index patterns defined in the securitySolution:defaultIndex advanced setting. You can override this default on a per-rule basis to target specific indices, exclude data tiers, or use a data view with runtime fields.

When you create or edit a rule, the Index patterns field (or Data view selector) controls which Elasticsearch indices the rule queries. This field is prepopulated with the space-level defaults, but you can change it for any individual rule.

Common reasons to override the defaults:

• Target a narrower set of indices: If a rule only applies to Windows endpoint data, restricting its index patterns to winlogbeat-* or logs-endpoint.events.process-* reduces the volume of data the rule scans and improves performance.

• Broaden to additional indices: If a rule needs data from a source that isn't in the space-level defaults (for example, a custom integration or a third-party feed), add the relevant index pattern.

• Use a data view: Instead of specifying index patterns directly, you can select a data view from the drop-down. The rule then uses the data view's index patterns and any runtime fields defined on it, which can be useful for enrichment or field normalization.

Cold data tiers store time series data that's accessed infrequently and rarely updated, while frozen data tiers hold time series data that's accessed even less frequently and never updated. Rules may perform slower or time out if they query data stored in these data tiers.

• Retention in hot tier: Keep data in the hot tier (index lifecycle management hot phase) for at least 24 hours. Index lifecycle management policies that move ingested data from the hot phase to another phase (for example, cold or frozen) in less than 24 hours may cause performance issues or rule execution errors.
• Replicas for mission-critical data: Use replicas when data must stay highly available. Frozen tiers don't include replicas by default, so a missing shard can cause a partial rule run failure. The same can happen during or after a Elastic Stack upgrade. When shards are healthy again, manually rerun rules for the time window that was affected.
• Timestamp override: @timestamp holds the event time; event.ingested records when ingest finished. Elasticsearch tries to skip indices outside the rule's lookback. If @timestamp is wrong or in the future, those skips don't work—the rule searches extra indices without changing alerts. Elastic prebuilt rules apply a timestamp override to prefer event.ingested when it exists; do the same for custom rules where it makes sense. For pipeline delays and setting event.ingested, see Troubleshoot ingest pipelines.

• To avoid rule failures and UI loading errors, do not modify index lifecycle management policies for Elastic Security-controlled indices, such as alert indices and list indices.
• Source data must have an index lifecycle management policy that keeps it in the hot or warm tiers for at least 24 hours before moving to cold or frozen data tiers.

You have two options for excluding cold and frozen data from rules:

• Space-level setting (all rules): On Elastic Stack, use advanced settings in the Kibana space to exclude data_cold or data_frozen. Two settings apply in different places; configure one or both:

  • data_views:fields_excluded_data_tiers: For all Data views, excludes those tiers when Elasticsearch resolves field lists for the UI.
  • securitySolution:excludedDataTiersForRuleExecution: For Security rule types, excludes those tiers during rule execution (not used by machine learning rules).

• Per-rule Query DSL filter (individual rules): Add a boolean query in the Query DSL to the rule so it ignores cold or frozen documents at pre-filter query time. This gives you per-rule control and is described below.

  Important

  • Per-rule Query DSL filters are not supported for ES|QL rules or machine learning rules.
  • Even with this filter applied, indicator match and event correlation rules may still fail if a frozen or cold shard that matches the rule's index pattern is unavailable during rule execution. If failures occur, modify the rule's index patterns to only match indices containing hot-tier data.

The Elasticsearch search pre-filter accepts only boolean query DSL. A query string query does not work for this pre-filter.

The snippets below match that requirement: each is a bool query whose must_not clause contains a terms filter on _tier, so cold or frozen-tier documents are excluded during the pre-filter. Change _tier if you need a different mix. This first snippet excludes only the frozen tier:

				
					{
   "bool":{
      "must_not":{
         "terms":{
            "_tier":[
               "data_frozen"
            ]
         }
      }
   }
}
		

This snippet excludes both cold and frozen tiers:

				
					{
   "bool":{
      "must_not":{
         "terms":{
            "_tier":[
               "data_frozen", "data_cold"
            ]
         }
      }
   }
}
		

To apply a filter, paste the Query DSL into the Custom query filter bar when creating or editing a rule.

This page covers per-rule data source settings. For broader configuration options:

• Change the defaults that all rules inherit: Modify the space-level securitySolution:defaultIndex setting (see Per-rule index patterns) to update the index patterns that new rules use by default.
• Configure deployment-level data source settings: Refer to Advanced data source configuration for cross-cluster search setup and logsdb index mode compatibility.