Set rule data sources

Every detection rule needs a data source that tells it which Elasticsearch indices to query. By default, rules inherit the index patterns defined in the securitySolution:defaultIndex advanced setting. You can override this default on a per-rule basis to target specific indices, exclude data tiers, or use a data view with runtime fields.

When you create or edit a rule, the Index patterns field (or Data view selector) controls which Elasticsearch indices the rule queries. This field is prepopulated with the space-level defaults, but you can change it for any individual rule.

Common reasons to override the defaults:

• Target a narrower set of indices: If a rule only applies to Windows endpoint data, restricting its index patterns to winlogbeat-* or logs-endpoint.events.process-* reduces the volume of data the rule scans and improves performance.

• Broaden to additional indices: If a rule needs data from a source that isn't in the space-level defaults (for example, a custom integration or a third-party feed), add the relevant index pattern.

• Use a data view: Instead of specifying index patterns directly, you can select a data view from the drop-down. The rule then uses the data view's index patterns and any runtime fields defined on it, which can be useful for enrichment or field normalization.

Cold data tiers store time series data that's accessed infrequently and rarely updated, while frozen data tiers hold time series data that's accessed even less frequently and never updated. Rules may perform slower or time out if they query data stored in these data tiers.

• Retention in hot tier: Keep data in the hot tier (index lifecycle management hot phase) for at least 24 hours. Index lifecycle management policies that move ingested data from the hot phase to another phase (for example, cold or frozen) in less than 24 hours may cause performance issues or rule execution errors.
• Replicas for mission-critical data: Your data should have replicas if it must be highly available. Since frozen tiers don't have replicas by default, shard unavailability can cause partial rule run failures. Shard unavailability may also be encountered during or after Elastic Stack upgrades. If this happens, you can manually rerun rules over the affected time period once the shards are available.

• To avoid rule failures, do not modify index lifecycle management policies for Elastic Security-controlled indices, such as alert and list indices.
• Source data must have an index lifecycle management policy that keeps it in the hot or warm tiers for at least 24 hours before moving to cold or frozen tiers.

You have two options for excluding cold and frozen data from rules:

• Space-level setting (all rules): Configure the excludedDataTiersForRuleExecution advanced setting to exclude cold or frozen data from all rules in a Kibana space. This does not apply to machine learning rules. Only available on Elastic Stack.

• Per-rule Query DSL filter (individual rules): Add a Query DSL filter to the rule that ignores cold or frozen documents at query time. This gives you per-rule control and is described below.

Exclude frozen-tier documents:

				
					{
   "bool":{
      "must_not":{
         "terms":{
            "_tier":[
               "data_frozen"
            ]
         }
      }
   }
}
		

Exclude cold and frozen-tier documents:

				
					{
   "bool":{
      "must_not":{
         "terms":{
            "_tier":[
               "data_frozen", "data_cold"
            ]
         }
      }
   }
}
		

To apply a filter, paste the Query DSL into the Custom query filter bar when creating or editing a rule.

This page covers per-rule data source settings. For broader configuration options:

• Change the defaults all rules inherit: Modify the space-level securitySolution:defaultIndex setting to update the index patterns that new rules use by default.
• Configure deployment-level data source settings: Refer to Advanced data source configuration for cross-cluster search setup and logsdb index mode compatibility.