﻿---
title: Set rule data sources
description: Configure which Elasticsearch indices rules query and exclude cold or frozen data from rule execution.
url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5510/solutions/security/detect-and-alert/set-rule-data-sources
products:
  - Elastic Cloud Serverless
  - Elastic Security
applies_to:
  - Serverless Security projects: Generally available
  - Elastic Stack: Generally available
---

# Set rule data sources
Every detection rule needs a data source that tells it which Elasticsearch indices to query. By default, rules inherit the index patterns defined in the [`securitySolution:defaultIndex`](/elastic/docs-content/pull/5510/solutions/security/get-started/configure-advanced-settings#update-sec-indices) advanced setting. You can override this default on a per-rule basis to target specific indices, exclude data tiers, or use a data view with runtime fields.

## Per-rule index patterns

When you create or edit a rule, the **Index patterns** field (or **Data view** selector) controls which Elasticsearch indices the rule queries. This field is prepopulated with the space-level defaults, but you can change it for any individual rule.
Common reasons to override the defaults:
- **Target a narrower set of indices**: If a rule only applies to Windows endpoint data, restricting its index patterns to `winlogbeat-*` or `logs-endpoint.events.process-*` reduces the volume of data the rule scans and improves performance.
- **Broaden to additional indices**: If a rule needs data from a source that isn't in the space-level defaults (for example, a custom integration or a third-party feed), add the relevant index pattern.
- **Use a data view**: Instead of specifying index patterns directly, you can select a data view from the drop-down. The rule then uses the data view's index patterns and any [runtime fields](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5510/solutions/security/get-started/create-runtime-fields-in-elastic-security) defined on it, which can be useful for enrichment or field normalization.

<tip>
  For indicator match rules, the **Indicator index patterns** field controls which threat intelligence indices the rule queries separately from the main source index patterns. By default, this uses the [`securitySolution:defaultThreatIndex`](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5510/solutions/security/get-started/configure-advanced-settings) setting (`logs-ti_*`).
</tip>

<note>
  ES|QL and machine learning rules do not use the index patterns field. ES|QL rules define their data source within the query itself (using the `FROM` command). Machine learning rules rely on the machine learning job's datafeed configuration.
</note>


## Exclude cold and frozen data

Cold data tiers store time series data that's accessed infrequently and rarely updated, while frozen data tiers hold time series data that's accessed even less frequently and never updated. Rules may perform slower or time out if they query data stored in these [data tiers](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5510/manage-data/lifecycle/data-tiers).

### Best practices

- **Retention in hot tier**: Keep data in the hot tier (index lifecycle management hot phase) for at least 24 hours. Index lifecycle management policies that move ingested data from the hot phase to another phase (for example, cold or frozen) in less than 24 hours may cause performance issues or rule execution errors.
- **Replicas for mission-critical data**: Your data should have replicas if it must be highly available. Since frozen tiers don't have replicas by default, shard unavailability can cause partial rule run failures. Shard unavailability may also be encountered during or after Elastic Stack upgrades. If this happens, you can manually rerun rules over the affected time period once the shards are available.


### Limitations

- To avoid rule failures, do not modify index lifecycle management policies for Elastic Security-controlled indices, such as alert and list indices.
- Source data must have an index lifecycle management policy that keeps it in the hot or warm tiers for at least 24 hours before moving to cold or frozen tiers.


### Exclusion options

You have two options for excluding cold and frozen data from rules:
- **Space-level setting (all rules)**: Configure the `excludedDataTiersForRuleExecution` [advanced setting](/elastic/docs-content/pull/5510/solutions/security/get-started/configure-advanced-settings#exclude-cold-frozen-data-rule-executions) to exclude cold or frozen data from all rules in a Kibana space. This does not apply to machine learning rules. Only available on Elastic Stack.
- **Per-rule Query DSL filter (individual rules)**: Add a Query DSL filter to the rule that ignores cold or frozen documents at query time. This gives you per-rule control and is described below.

<important>
  - Per-rule Query DSL filters are not supported for ES|QL and machine learning rules.
  - Even with this filter applied, indicator match and event correlation rules may still fail if a frozen or cold shard that matches the rule's index pattern is unavailable during rule execution. If failures occur, modify the rule's index patterns to only match indices containing hot-tier data.
</important>


### Sample Query DSL filters

Exclude frozen-tier documents:
```json
{
   "bool":{
      "must_not":{
         "terms":{
            "_tier":[
               "data_frozen"
            ]
         }
      }
   }
}
```

Exclude cold and frozen-tier documents:
```json
{
   "bool":{
      "must_not":{
         "terms":{
            "_tier":[
               "data_frozen", "data_cold"
            ]
         }
      }
   }
}
```

To apply a filter, paste the Query DSL into the **Custom query** filter bar when creating or editing a rule.

## More data source configuration options

This page covers per-rule data source settings. For broader configuration options:
- **Change the defaults all rules inherit**: Modify the space-level [`securitySolution:defaultIndex`](/elastic/docs-content/pull/5510/solutions/security/get-started/configure-advanced-settings#update-sec-indices) setting to update the index patterns that new rules use by default.
- **Configure deployment-level data source settings**: Refer to [Advanced data source configuration](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/5510/solutions/security/detect-and-alert/advanced-data-source-configuration) for cross-cluster search setup and logsdb index mode compatibility.