--- title: Troubleshoot timestamp data quality issues description: Detect, investigate, and resolve timestamp data quality issues that can cause unexpected behavior. url: https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/troubleshoot/elasticsearch/fix-date-timestamps products: - Elasticsearch applies_to: - Elastic Stack: Generally available --- # Troubleshoot timestamp data quality issues In Elasticsearch date fields, you can use any valid past, present, or future date as the value, as long as it matches the field's [date format](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/mapping-reference/mapping-date-format). Make sure your stored date fields are valid. When ensuring timestamp accuracy, you might encounter these common client-side data quality issues: - Operating system timezone setting conflicts - Incorrectly formatted date values - Unexpected values for the specific date field, for example: - future dates that don't make sense for the data - default dates, such as the `1970` [Unix epoch](https://en.wikipedia.org/wiki/Unix_time) - negative dates - time-bucketed dates - truncated strings This page summarizes the symptoms of these issues and helps you address them, focusing on unexpected future dates in the `@timestamp` field. When possible, make sure to resolve timestamp data quality issues in the data itself, rather than working around them in Elasticsearch. ## Symptoms Timestamp data quality issues can strain resources and cause unexpected search results. They can especially affect the following features: - [Discover](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/discover) - [Dashboards](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/dashboards) - Elastic Security [detections and alerts](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/solutions/security/detect-and-alert) - Observability [incident management alerts](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/solutions/observability/incident-management/alerting) - [Kibana alerting](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/alerting/alerts) - [Watchers](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/alerting/watcher) Common symptoms of these issues include: - Later [data tiers](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/manage-data/lifecycle/data-tiers) have a [search queue backlog](/elastic/docs-content/pull/6918/troubleshoot/elasticsearch/task-queue-backlog#diagnose-task-queue-thread-pool), but not on `data_hot`. - The `data_frozen` data tier shows ongoing [high CPU usage](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/troubleshoot/elasticsearch/high-cpu-usage). - [Slow logs](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/index-settings/slow-log) report events for `data_cold` or `data_frozen` on short, periodic intervals. - In Kibana, [Inspect](https://www.elastic.co/blog/troubleshooting-guide-common-issues-kibana-discover-load#3.-load-search) reports search results from unexpected indices. - Indices based on Logstash [date math syntax](https://docs-v3-preview.elastic.dev/elastic/logstash-docs-md/tree/main/lsr/plugins-outputs-elasticsearch#plugins-outputs-elasticsearch-index) or the Elasticsearch [Date index name processor](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/enrich-processor/date-index-name-processor) refer to dates far into the past or future. You can [reduce the performance impact](#fix-date-timestamps-recommendations) of these issues even when the underlying timestamp data quality problems still exist. ### Example Here's an example that illustrates how timestamp data quality issues can affect the search performance of your cluster. Suppose a single on-prem host has a misconfigured system clock, causing its `@timestamp` field to log timestamps one year in the future. In this example: - Host data ingests into an Elasticsearch [data stream](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/manage-data/data-store/data-streams) with five [primary shards](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/index-settings/index-modules#index-number-of-shards). - The data stream powers a [dashboard](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/dashboards), which uses a [data view](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/find-and-organize/data-views) with `@timestamp` as the time field. - The data stream has an [index lifecycle management (ILM) policy](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/manage-data/lifecycle/index-lifecycle-management) that guarantees it [rolls over indices](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/elasticsearch/index-lifecycle-actions/ilm-rollover) at least once a day and [migrates data to the frozen tier](/elastic/docs-content/pull/6918/manage-data/lifecycle/index-lifecycle-management/index-lifecycle#ilm-phase-actions) after 15 days. After 30 days, there are 150 shards, half of them hosted in the frozen tier. #### Scenario 1: No data ingested by misconfigured host If the misconfigured host didn't ingest data during the 30-day period, the following issues can occur when a user selects a `now-15m` [time window](/elastic/docs-content/pull/6918/explore-analyze/dashboards/using#_set_a_time_range) in the dashboard: 1. The [search pre-filter](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-msearch) shows only the five shards from the latest backing index. 2. Data is [read](/elastic/docs-content/pull/6918/deploy-manage/distributed-architecture/reading-and-writing-documents#_basic_read_model) from the latest five shards only, and the remaining search [queries](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/query-filter) and [aggregations](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/query-filter/aggregations) run on that subset of data. 3. [Inspect](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/troubleshoot/observability/inspect) reports 5 shards searched and 145 shards `skipped`. #### Scenario 2: Data ingested by misconfigured host If the misconfigured host was ingesting data during the 30-day period, the following issues can occur: 1. The pre-filter can't filter out backing indices, so it allows searches against all 150 shards backing this data stream. 2. Half of the shards are in the `data_frozen` data tier, which is intended for [rarely queried data](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/manage-data/lifecycle/data-tiers). The frozen tier is usually provisioned with low CPU relative to high data volume, so searches run slower. Additionally, in the frozen tier, indices are [partially mounted searchable snapshots](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/deploy-manage/tools/snapshot-and-restore/searchable-snapshots), which slows down searches because data must be fetched from the snapshot repository. 3. The cluster searches all 150 shards for timestamps within the desired window. This resource usage can happen even if no documents end up matching the selected time range. 4. [Inspect](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/troubleshoot/observability/inspect) reports 150 shards searched and 0 `skipped`. In both scenarios, search is more computationally expensive and can return incorrect results because of the host's misconfigured timestamps. The next section explains how to investigate the scope of a timestamp data quality issue. ## Investigate timestamp data quality Timestamp data quality issues can be difficult to notice if they're not actively causing performance strain. For best results, make sure you're familiar with the typical patterns and expected trends in your data (also referred to as "seasonal patterns"), so you can spot anomalies. To check for date values far into the past or future, you can use the following options: - To review [partially mounted searchable snapshots](/elastic/docs-content/pull/6918/deploy-manage/tools/snapshot-and-restore/searchable-snapshots#searchable-snapshot-mount-storage-options) and their `@timestamp` date field only, use a [cluster state request](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-state): ```json ``` To filter and format the results, you can use third-party tools such as [jq](https://jqlang.github.io/jq/manual/). For example, to see a list of indices with a maximum timestamp in the future: ```sh cat cluster_state.json | jq -cMr '.metadata.indices| to_entries| sort_by(.key)| .[]| .value.timestamp_range as $ts| select($ts.min)| {min:($ts.min/1000.0 | todate),max:($ts.max/1000.0 | todate), index:.key}' | jq -r --arg now "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" 'select(.max > $now)' ``` - To list the top 200 [aggregated](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/query-filter/aggregations) indices by the number of documents whose timestamps are in the future, use a search request: ```json { "size": 0, "aggs": { "0": { "terms": { "field": "_index", "order": { "_count": "desc" }, "size": 200 }} }, "query": { "bool": { "filter": [ { "range": { "@timestamp": { "gte": "now" }}}] } } } ``` - To list individual indices' minimum and maximum timestamps, use a search request. This search can be resource-intensive, depending on your search target scope and the hardware profiles of nodes hosting related shards. ```json { "size": 0, "aggs": { "2": { "aggs": { "min": {"min": {"field": "@timestamp"} }, "max": {"max": {"field": "@timestamp"} } }, "terms": { "field": "_index", "order": {"_key": "asc"}, "size": 200 } }} } ``` If you find future dates, check for patterns in the data distribution: ```json { "size": 0, "query": { "bool": { "filter": [ { "range": { "@timestamp": { "gte": "now" }}}] } }, "aggs": { "time_buckets": { "auto_date_histogram": { "field": "@timestamp", "buckets": 30, "format": "yyyy-MM-dd" } } } } ``` The rest of this page explains how to [reduce the performance impact](#fix-date-timestamps-recommendations) of these issues and [clean up](#fix-date-timestamps-cleanup) problematic data. ## Reduce performance impact Even when timestamp data quality issues remain in your data, you can reduce their performance impact by adjusting how scheduled tasks and searches run. ### Default date field [Time series data](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/manage-data/lifecycle) is frequently searched by date field, and the most common date field is [`@timestamp`](https://docs-v3-preview.elastic.dev/elastic/ecs/tree/main/reference/ecs-base). By default, this field's value reflects when the event **originated,** as reported by the source. This is the default date field when creating a [data view](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/find-and-organize/data-views). Discover and Dashboard objects use data views to [resolve](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-resolve-index) and search data. For scheduled tasks that run without user interaction, consider searching on the [`event.ingested`](https://docs-v3-preview.elastic.dev/elastic/ecs/tree/main/reference/ecs-event) date field instead of `@timestamp`. By default, this field's value reflects when the event was **ingested** into the cluster. If `event.ingested` isn't already populated, refer to [Troubleshoot ingest pipelines](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/troubleshoot/elasticsearch/troubleshoot-ingest-pipelines) to add the field to your data with a custom [ingest pipeline](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/manage-data/ingest/transform-enrich/ingest-pipelines). These common scheduled tasks benefit from using `event.ingested`: - [Transforms](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/transforms) - [Machine Learning](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/machine-learning) - [Kibana alerting](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/alerting/alerts) - [Watchers](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/alerting/watcher) The `event.ingested` approach is recommended for [Elastic Security data sources](/elastic/docs-content/pull/6918/solutions/security/detect-and-alert/set-rule-data-sources#best-practices) and is commonly used in [Observability rules](/elastic/docs-content/pull/6918/solutions/observability/incident-management/create-manage-rules#observability-create-manage-rules-observability-rules). For Elastic Security [detection rules](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/solutions/security/detect-and-alert), also consider enabling the advanced setting that ensures `@timestamp` is not used as a [fallback for timestamp overrides](/elastic/docs-content/pull/6918/solutions/security/detect-and-alert/common-rule-settings#rule-ui-advanced-params). ### Kibana data source settings You can use these Kibana [advanced settings](https://docs-v3-preview.elastic.dev/elastic/kibana/tree/main/reference/advanced-settings) to exclude the `data_cold` and `data_frozen` [data tiers](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/manage-data/lifecycle/data-tiers) from searches: - `data_views:fields_excluded_data_tiers` for all [data views](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/explore-analyze/find-and-organize/data-views) - `observability:searchExcludedDataTiers` for [Observability](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/solutions/observability) - For [Security](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/solutions/security): - `securitySolution:excludedDataTiersForRuleExecution` - `securitySolution:excludeColdAndFrozenTiersInPrevalence` - `securitySolution:excludeColdAndFrozenTiersInAnalyzer` For guidance specific to Elastic Security, refer to [Exclude cold and frozen tiers from rule execution](/elastic/docs-content/pull/6918/solutions/security/detect-and-alert/set-rule-data-sources#exclude-cold-frozen-tier). ### Data tier filters You can also use a Query DSL [boolean query](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/query-dsl/query-dsl-bool-query) filter out specific [data tiers](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/manage-data/lifecycle/data-tiers). Filtering with a [query string query](https://docs-v3-preview.elastic.dev/elastic/elasticsearch/tree/main/reference/query-languages/query-dsl/query-dsl-query-string-query) is insufficient. For example, you can filter out `data_cold` and `data_frozen` with the following boolean query: ```json { "bool":{ "must_not":{ "terms":{ "_tier":[ "data_cold", "data_frozen" ] } } } } ``` ## Clean up timestamp data After investigating timestamp data quality and reviewing best practices, clean up any issues by deleting or modifying the problematic data. ### Delete problematic data To remove invalid data, use one of these methods: - [Delete the index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete) or [delete the data stream](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-delete-data-stream). For [searchable snapshot indices](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/deploy-manage/tools/snapshot-and-restore/searchable-snapshots), consider whether to also [delete their associated snapshots](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-snapshot-delete). - Use an index [delete by query](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-delete-by-query) request. For example, to remove documents with future dates: ```json { "query": { "range": { "@timestamp": { "gt": "now" } } } } ``` ### Modify problematic data The following example steps modify invalid data by updating the `@timestamp` field to the current time: 1. Create an [ingest pipeline](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/manage-data/ingest/transform-enrich/ingest-pipelines) that sets the `@timestamp` date field to the current timestamp: ```json { "processors": [ { "rename": { "description": "(Optional) Cache the previous timestamp in a new field", "field": "@timestamp", "target_field": "old_timestamp" } , "set": { "description": "Override the @timestamp value to the ingested time", "field": "_source.@timestamp", "value": "{{_ingest.timestamp}}" } } ] } ``` 2. Run the data through the new pipeline to modify the value, using one of these approaches: - To modify the value across the entire index, [reindex](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-reindex) to a new index. ```json { "source": { "index": ["my-index-000001", "my-index-000002"] }, "dest": { "index": "my-new-index-000001", "pipeline": "update_date" } } ``` - To target specific documents, use an [update by query](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-update-by-query) request within the existing index. ```json { "query": { "range": { "@timestamp": { "gt": "now" } } } } ``` To modify documents in a [searchable snapshot index](https://docs-v3-preview.elastic.dev/elastic/docs-content/pull/6918/deploy-manage/tools/snapshot-and-restore/searchable-snapshots), you must first [restore it to a regular index](/elastic/docs-content/pull/6918/manage-data/lifecycle/data-tiers/manage-data-tiers-ech-ece#searchable-snapshot-data-tier).