Autoscaling deciders

ECE ECK Elastic Cloud Hosted

Autoscaling in Elasticsearch enables dynamic resource allocation based on predefined policies. A key component of this mechanism is autoscaling deciders, which independently assess resource requirements and determine when scaling actions are necessary. Deciders analyze various factors, such as storage usage, indexing rates, and machine learning workloads, to ensure clusters maintain optimal performance without manual intervention.

Reactive storage decider

Estimates required storage capacity of current data set. Available for policies governing data nodes.

Proactive storage decider

Estimates required storage capacity based on current ingestion into hot nodes. Available for policies governing hot data nodes.

Frozen shards decider

Estimates required memory capacity based on the number of partially mounted shards. Available for policies governing frozen data nodes.

Frozen storage decider

Estimates required storage capacity as a percentage of the total data set of partially mounted indices. Available for policies governing frozen data nodes.

Frozen existence decider

Estimates a minimum require frozen memory and storage capacity when any index is in the frozen ILM phase.

Machine learning decider

Estimates required memory capacity based on machine learning jobs. Available for policies governing machine learning nodes.

Fixed decider

Responds with a fixed required capacity. This decider is intended for testing only.

The autoscaling reactive storage decider (reactive_storage) calculates the storage required to contain the current data set. It signals that additional storage capacity is necessary when existing capacity has been exceeded (reactively).

The reactive storage decider is enabled for all policies governing data nodes and has no configuration options.

The decider relies partially on using data tier preference allocation rather than node attributes. In particular, scaling a data tier into existence (starting the first node in a tier) will result in starting a node in any data tier that is empty if not using allocation based on data tier preference. Using the ILM migrate action to migrate between tiers is the preferred way of allocating to tiers and fully supports scaling a tier into existence.

The autoscaling proactive storage decider (proactive_storage) calculates the storage required to contain the current data set plus an estimated amount of expected additional data.

The proactive storage decider is enabled for all policies governing nodes with the data_hot role.

The estimation of expected additional data is based on past indexing that occurred within the forecast_window. Only indexing into data streams contributes to the estimate.

forecast_window

(Optional, time value) The window of time to use for forecasting. Defaults to 30 minutes.

This example puts an autoscaling policy named my_autoscaling_policy, overriding the proactive decider’s forecast_window to be 10 minutes.

 PUT /_autoscaling/policy/my_autoscaling_policy {
  "roles" : [ "data_hot" ],
  "deciders": {
    "proactive_storage": {
      "forecast_window": "10m"
    }
  }
}

The API returns the following result:

{
  "acknowledged": true
}

The autoscaling frozen shards decider (frozen_shards) calculates the memory required to search the current set of partially mounted indices in the frozen tier. Based on a required memory amount per shard, it calculates the necessary memory in the frozen tier.

memory_per_shard

(Optional, byte value) The memory needed per shard, in bytes. Defaults to 2000 shards per 64 GB node (roughly 32 MB per shard). Notice that this is total memory, not heap, assuming that the Elasticsearch default heap sizing mechanism is used and that nodes are not bigger than 64 GB.

The autoscaling frozen storage decider (frozen_storage) calculates the local storage required to search the current set of partially mounted indices based on a percentage of the total data set size of such indices. It signals that additional storage capacity is necessary when existing capacity is less than the percentage multiplied by total data set size.

The frozen storage decider is enabled for all policies governing frozen data nodes and has no configuration options.

percentage

(Optional, number value) Percentage of local storage relative to the data set size. Defaults to 5.

The autoscaling frozen existence decider (frozen_existence) ensures that once the first index enters the frozen ILM phase, the frozen tier is scaled into existence.

The frozen existence decider is enabled for all policies governing frozen data nodes and has no configuration options.

The autoscaling machine learning decider (ml) calculates the memory and CPU requirements to run machine learning jobs and trained models.

The machine learning decider is enabled for policies governing ml nodes.

Both num_anomaly_jobs_in_queue and num_analytics_jobs_in_queue are designed to delay a scale-up event. If the cluster is too small, these settings indicate how many jobs of each type can be unassigned from a node. Both settings are only considered for jobs that can be opened given the current scale. If a job is too large for any node size or if a job can’t be assigned without user intervention (for example, a user calling _stop against a real-time anomaly detection job), the numbers are ignored for that particular job.

num_anomaly_jobs_in_queue

(Optional, integer) Specifies the number of queued anomaly detection jobs to allow. Defaults to 0.

num_analytics_jobs_in_queue

(Optional, integer) Specifies the number of queued data frame analytics analytics jobs to allow. Defaults to 0.

down_scale_delay

(Optional, time value) Specifies the time to delay before scaling down. Defaults to 1 hour. If a scale down is possible for the entire time window, then a scale down is requested. If the cluster requires a scale up during the window, the window is reset.

This example creates an autoscaling policy named my_autoscaling_policy that overrides the default configuration of the machine learning decider.

 PUT /_autoscaling/policy/my_autoscaling_policy {
  "roles" : [ "ml" ],
  "deciders": {
    "ml": {
      "num_anomaly_jobs_in_queue": 5,
      "num_analytics_jobs_in_queue": 3,
      "down_scale_delay": "30m"
    }
  }
}

The API returns the following result:

{
  "acknowledged": true
}

The autoscaling fixed decider responds with a fixed required capacity. It is not enabled by default but can be enabled for any policy by explicitly configuring it.

storage

(Optional, byte value) Required amount of node-level storage. Defaults to -1 (disabled).

memory

(Optional, byte value) Required amount of node-level memory. Defaults to -1 (disabled).

processors

(Optional, float) Required number of processors. Defaults to disabled.

nodes

(Optional, integer) Number of nodes to use when calculating capacity. Defaults to 1.

This example puts an autoscaling policy named my_autoscaling_policy, enabling and configuring the fixed decider.

 PUT /_autoscaling/policy/my_autoscaling_policy {
  "roles" : [ "data_hot" ],
  "deciders": {
    "fixed": {
      "storage": "1tb",
      "memory": "32gb",
      "processors": 2.3,
      "nodes": 8
    }
  }
}

The API returns the following result:

{
  "acknowledged": true
}