Loading

Using logsdb index mode with Elastic Security

Elastic Stack Serverless Security

Requirements

To use the synthetic _source feature in the Elastic Stack, you must have the appropriate subscription. Refer to the subscription page for Elastic Cloud and Elastic Stack/self-managed for the breakdown of available features and their associated subscription tiers.

This topic explains the impact of using logsdb index mode with Elastic Security and Elastic Security Serverless. You must have the appropriate subscription to use this feature in Elastic Stack. In Serverless, logsdb index mode is enabled by default.

With logsdb index mode, the original _source field is not stored in the index but can be reconstructed using synthetic _source.

When the _source is reconstructed, modifications are possible. Therefore, there could be a mismatch between users' expectations and how fields are formatted.

Continue reading to learn how logsdb index mode affects CPU and storage usage and specific Elastic Security components.

Note

The following statement applies to Elastic Stack users only:

Logsdb index mode is fully supported, and is recommended for all Elastic Security deployments. Users with existing Elastic Security deployments are advised to fully understand and accept the documented changes to detection alert documents, runtime fields, and rule actions (refer to the sections below), and ensure that their deployment has sufficient excess hot data tier CPU capacity to support the logsdb ingest and indexing process. Enabling logsdb index mode without sufficient excess hot data tier CPU capacity may result in data ingestion backups and/or security detection rule timeouts and errors.

Logsdb index mode significantly reduces storage needs by using slightly more CPU during ingest. After enabling logsdb index mode for your data sources, you may need to adjust cluster sizing in response to the new CPU and storage needs. To learn more about how logsdb index mode optimizes CPU and storage usage, check out our blog.

When alerts are generated, the _source event is copied into the alert to retain the original data. When the logsdb index mode is applied, the _source event stored in the alert is reconstructed using synthetic _source.

If you’re switching to use logsdb index mode, the _source field stored in the alert might look different in certain situations:

Alerts generated by the following rule types could be affected:

  • Custom query
  • Event correlation (non-sequence only)
  • Non-aggregate rule types (for example, ES|QL rules that use non-aggregating queries)

Alerts that are generated by threshold, machine learning, and event correlation sequence rules are not affected since they do not contain copies of the original source.

While we do not recommend using _source for actions, in cases where the action relies on the _source, the same limitations and changes apply.

If you send alert notifications by enabling actions to the external systems that have workflows or automations based on fields formatted from the original source, they may be affected. In particular, this can happen when the fields used are arrays of objects.

We recommend checking and adjusting the rule actions using _source before switching to logsdb index mode.

Runtime fields that reference _source may be affected. Some runtime fields might not work and need to be adjusted. For example, if an event was indexed with the value of agent.name in the dot-notation form, it will be returned in the nested form and might not work.

The following is an example of accessing _source that works with the logsdb index mode enabled:

 "source": """  emit(params._source.agent.name + "_____" + doc['agent.name'].value ); """ "source": """  emit(params._source['agent']['name'] + "_____" + doc['agent.name'].value );  """
"source": """  emit(field('agent.name').get(null) + "_____" + doc['agent.name'].value ); """
"source": """  emit($('agent.name', null) + "_____" + doc['agent.name'].value ); """

The following will not work with synthetic source (logsdb index mode enabled):

 "source": """  emit(params._source['agent.name'] + "_____" + doc['agent.name'].value );  """

Also note that runtime fields with scripts that reference params._source may need to be updated. Scripts that currently use dotted field names to access source fields must be converted to use the nested access pattern instead, unless the object being accessed has subobjects set to false. Fields that are not mapped also need to be accessed in scripts using the nested access pattern (for example, params._source['foo']['bar']['baz'] or params._source.foo.bar.baz, not params._source['foo.bar.baz']). To learn more about how synthetic source names fields and changes that you may need to make to your scripts, refer to Fields named as they are mapped.