Loading

Set rule data sources

Every detection rule needs a data source that tells it which Elasticsearch indices to query. By default, rules inherit the index patterns defined in the securitySolution:defaultIndex advanced setting. You can override this default on a per-rule basis to target specific indices, exclude data tiers, or use a data view with runtime fields.

When you create or edit a rule, the Index patterns field (or Data view selector) controls which Elasticsearch indices the rule queries. This field is prepopulated with the space-level defaults, but you can change it for any individual rule.

Common reasons to override the defaults:

Target a narrower set of indices. If a rule only applies to Windows endpoint data, restricting its index patterns to winlogbeat-* or logs-endpoint.events.process-* reduces the volume of data the rule scans and improves performance.

Broaden to additional indices. If a rule needs data from a source that isn't in the space-level defaults (for example, a custom integration or a third-party feed), add the relevant index pattern.

Use a data view. Instead of specifying index patterns directly, you can select a data view from the drop-down. The rule then uses the data view's index patterns and any runtime fields defined on it, which can be useful for enrichment or field normalization.

Tip

For indicator match rules, the Indicator index patterns field controls which threat intelligence indices the rule queries separately from the main source index patterns. By default, this uses the securitySolution:defaultThreatIndex setting (logs-ti_*).

Note

ES|QL and machine learning rules do not use the index patterns field. ES|QL rules define their data source within the query itself (using the FROM command). machine learning rules rely on the machine learning job's datafeed configuration.

Rules may perform slower or time out if they query data stored in cold or frozen data tiers. You have two options for excluding this data:

Space-level setting (all rules). Configure the excludedDataTiersForRuleExecution advanced setting to exclude cold or frozen data from all rules in a Kibana space. This does not apply to machine learning rules. Only available on Elastic Stack.

Per-rule Query DSL filter (individual rules). Add a Query DSL filter to the rule that ignores cold or frozen documents at query time. This gives you per-rule control and is described below.

Important
  • Per-rule Query DSL filters are not supported for ES|QL and machine learning rules.
  • Even with this filter applied, indicator match and event correlation rules may still fail if a frozen or cold shard that matches the rule's index pattern is unavailable during rule execution. If failures occur, modify the rule's index patterns to only match indices containing hot-tier data.

Exclude frozen-tier documents:

				
					{
   "bool":{
      "must_not":{
         "terms":{
            "_tier":[
               "data_frozen"
            ]
         }
      }
   }
}

Exclude cold and frozen-tier documents:

				
					{
   "bool":{
      "must_not":{
         "terms":{
            "_tier":[
               "data_frozen", "data_cold"
            ]
         }
      }
   }
}

To apply a filter, paste the Query DSL into the Custom query filter bar when creating or editing a rule.