Loading

Fix date timestamps

Elasticsearch will accept any valid past, present, or future date for its date fields as long as the value satisfies that field's date format setting.

Users should ensure stored date fields are valid. Elasticsearch accepting a value does not reflect its accuracy. The following are not guaranteed to be problematic for your setup but are common client-side data quality issues that users need to consider to ensure their timestamp's accuracy:

  • Operating system timezones
  • Variances in date formatting
  • Unexpected values such as:
    • future dates
    • default dates, such as the 1970 unix epoch
    • negative dates
    • time-bucketed dates
    • truncated strings

We recommend resolving timestamp data quality issues as close to problem source as possible. The following guide outlines symptoms which can occur due to and how to account for timestamp data quality issues. Since timestamp data quality issues can commonly affect the performance of scheduled tasks which search now-X, our outline will focus examples on checking for unexpected future dates within the @timestamp date field.

Timestamp data quality issues can strain resources and cause unexpected search results.

The following are common locations where users first notice issues:

The following are symptoms that users commonly first notice:

Performance incidents can frequently be avoided by following best practices even if underlying timestamp data quality issues remain.

Here's an example to demonstrate how timestamp data quality issues can affect the search performance of your cluster.

As setup, let's say a single on-prem host has a bad time configuration logging a future date that is one year into the future. This host's @timestamp date field will report data a year ahead of other hosts logging similar data.

For our example, all these hosts' data ingests into a 5 primaries Elasticsearch data stream. This data stream has an Index lifecycle management policy that guarantees it rolls over indices at least once a day and that migrates data to frozen tier after 15 days. After 30 days, there are 150 shards with half hosted in frozen tier.

This data stream is powering a dashboard. The dashboard uses a data view with the @timestamp field as its designated date field.

If this misconfigured host was not ingesting data those 30 days, when a user selects a now-15m time window, you could normally expect:

  1. The search pre-filter could narrow into only the latest backing index's 5 shards.
  2. The data would be read off of only these shards and remaining search queries and aggregations would run off that subset of data.
  3. The Dashboard's Inspect would report 5 shards searched and 145 shards skipped.

However, if this misconfigured host was ingesting data those 30 days, then you could expect:

  1. The pre-filter cannot filter-out any backing indices, so will allow search against all 150 shards backing this data stream.
  2. Half of the shards are in the data_frozen data tier, which is intended for rarely queried data. This is because:
    • Frozen tier hardware profile is usually provisioned to be proportioned for low CPU to high data, so searches run slower.
    • These indices are partially-mounted searchable snapshots so searches can run slower as data needs fetched out of snapshot repository.
  3. The cluster will search all 150 shards for timestamps within the desired window. This resource usage can happen even if no documents end up qualifying for the requested time range.
  4. The Dashboard's Inspect would report 150 shards searched and 0 skipped.

From this you can see that the search is more computationally expensive and can potentially return different results based off the host having misconfigured timestamps. Next, we will discuss how to investigate the scope of the data quality issue.

Timestamp data quality issues can be hard to notice if they're not actively causing performance strain. It can require familiarity with the seasonal patterns of the data.

A common sign to look for is date values far into the past or future. You can check for future dates with the following options:

  • To review partially-mounted searchable snapshots and their @timestamp date field only, you can read the cluster state:

    				GET _cluster/state?filter_path=metadata.indices.*.timestamp_range

    You can use third-party tools such as JQ to filter and format these results. For example to see a list of indices who's maximum timestamp is in the future from the time the command is ran:

    cat cluster_state.json | jq -cMr '.metadata.indices| to_entries| sort_by(.key)| .[]| .value.timestamp_range as $ts| select($ts.min)| {min:($ts.min/1000.0 | todate),max:($ts.max/1000.0 | todate), index:.key}' | jq -r --arg now "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" 'select(.max > $now)'

  • To list the top 200 aggregated indices by document count of having timestamps in the future from the time the command is ran:

    				GET */_search
					{ "size": 0,
  "aggs": { "0": {
    "terms": {
      "field": "_index",
      "order": { "_count": "desc" },
      "size": 200
    }}
  },
  "query": {
    "bool": {
      "filter": [ { "range": { "@timestamp": { "gte": "now" }}}]
    }
  }
}

  • You can list individual indices' minimum and maximum timestamps. This can be expensive, depending on your search target scope and the hardware profiles of nodes hosting related shards.

    				GET my_datastream/_search
					{ "size": 0,
  "aggs": { "2": {
    "aggs": {
      "min": {"min": {"field": "@timestamp"} },
      "max": {"max": {"field": "@timestamp"} }
    },
    "terms": {
      "field": "_index",
      "order": {"_key": "asc"},
      "size": 200
    }
  }}
}

If future dates are found, you will likely want to resolve this data quality concern. If so, it can help to check for potential seasonal patterns in their distributions in order for your team to decide how to proceed:

				GET my_datastream/_search?filter_path=aggregations
					{ "size": 0,
  "query": {
    "bool": {
      "filter": [ { "range": { "@timestamp": { "gte": "now" }}}]
    }
  },
  "aggs": {
    "time_buckets": {
      "auto_date_histogram": {
        "field": "@timestamp",
        "buckets": 30,
        "format": "yyyy-MM-dd"
      }
    }
  }
}

Once you have investigated the scope and source of your timestamp data quality issue, you can determine how your team would like to resolve it. On the rest of the page, we will outline best practices and common cleanup options.

You can consider the following options to minimize the impact of timestamp data quality issues. You should consider these for all scheduled tasks.

Time series data is frequently searched by its date fields. The most common date field used is @timestamp. By default, this field's value reflects when the event originated as reported by the source. This is the most common date view users expect to use to search their data, so is the default date field when creating a data view. Data views are how Discover and Dashboard objects resolve to and search data.

When users are not present, you should instead consider searching off of the event.ingested date field. By default, this field's value reflects when the event ingested into the cluster. Refer to this troubleshooting guide to add it to your data with a custom ingest pipeline if it is not already populating. To avoid complications from ingestion lag, you should first consider event.ingested for all scheduled tasks, such as:

Kibana solutions can encourage this, like in Security's data sources, or automatically choose this on your behalf, like in Observability's rules.

For Security's Detection rules, you might also consider enabling the Do not use @timestamp as a fallback timestamp field advanced setting.

You might consider enabling Kibana Advanced Settings to avoid data_cold,data_frozen data tiers with the following settings:

The Elasticsearch search pre-filter can filter in or out of data tiers with a boolean query DSL. Filtering with query string query is insufficient. For example, you could filter out data_cold,data_frozen like:

{
   "bool":{
      "must_not":{
         "terms":{
            "_tier":[ "data_cold", "data_frozen" ]
         }
      }
   }
}

You can consider the following cleanup options for your timestamp data quality issues.

If your team determines it is best to remove invalid data, then you can delete it either by:

Tip

To modify documents within a searchable snapshot index, you must first restore it to a normal index.

The following is an example flow which users commonly consider to modify invalid data by updating it to current time:

  1. Create an Ingest pipeline which modifies the @timestamp date field to the current timestamp

    				PUT _ingest/pipeline/update_date
					{
  "processors": [
    {
      "rename": {
        "description": "(Optional) Cache the previous timestamp in a new field",
        "field": "@timestamp",
        "target_field": "old_timestamp"
      }
      ,
      "set": {
        "description": "Override the @timestamp value to the ingested time",
        "field": "_source.@timestamp",
        "value": "{{_ingest.timestamp}}"
      }
    }
  ]
}

  2. Run the data over that pipeline to modify the value, either

    • Across the entire index by reindexing into a new index.

      				POST _reindex
					{
  "source": {
    "index": ["my-index-000001", "my-index-000002"]
  },
  "dest": {
    "index": "my-new-index-000001",
    "pipeline": "update_date"
  }
}

    • Targeting specific documents by updating by query within the existing index.

      				POST my_index/_update_by_query?pipeline=update_date
					{
  "query": {
    "range": {
      "@timestamp": {
        "gt": "now"
      }
    }
  }
}