Get statistics for pipelines
/
- GET /_node/stats/pipelines
Get pipeline performance metrics and plugin details.
Query Parameters
-
prettyboolean If you append
?pretty=trueto the request, the JSON returned will be pretty formatted. Use it for debugging only!
Response
A JSON object containing pipelines statistics.
- the number of events that were input, filtered, or output by each pipeline
- the current and lifetime [flow rates](https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#flow-stats for each pipeline
- stats for each configured filter or output stage
- info about config reload successes and failures (when config reload is enabled)
- info about the persistent queue (when persistent queues are enabled)
- metrics related to processed batch sizes. Includes the size in bytes and the number of events of batches processed in this pipeline. (when setting pipeline.batch.metrics.sampling_mode is not
disabled).
-
pipelinesmap string to EventsStats -
A map where each key is a user-defined pipeline name, and the value contains the metrics for that pipeline.
-
<string>EventsStats -
eventsobject -
ininteger -
filteredinteger -
outinteger -
duration_in_millisinteger -
queue_push_duration_in_millisinteger -
flowoneOf -
input_throughputFlowWindows -
This metric is expressed in events-per-second, and is the rate of events being pushed into the pipeline(s) queue(s) relative to wall-clock time (
events.in/ second). It includes events that are blocked by the queue and have not yet been accepted. -
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
filter_throughputFlowWindows -
This metric is expressed in events-per-second, and is the rate of events flowing through the filter phase of the pipeline(s) relative to wall-clock time (
events.filtered/ second). -
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
output_throughputFlowWindows -
This metric is expressed in events-per-second, and is the rate of events flowing through the output phase of the pipeline(s) relative to wall-clock time (
events.out/ second). -
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
queue_backpressureFlowWindows -
This is a unitless metric representing the cumulative time spent by all inputs blocked pushing events into their pipeline’s queue, relative to wall-clock time (queue_push_duration_in_millis / millisecond). It is typically most useful when looking at the stats for an individual pipeline.
While a "zero" value indicates no back-pressure to the queue, the magnitude of this metric is highly dependent on the shape of the pipelines and their inputs. It cannot be used to compare one pipeline to another or even one process to itself if the quantity or shape of its pipelines changes. A pipeline with only one single-threaded input may contribute up to 1.00, a pipeline whose inputs have hundreds of inbound connections may contribute much higher numbers to this combined value.
Additionally, some amount of back-pressure is both normal and expected for pipelines that are pulling data, as this back-pressure allows them to slow down and pull data at a rate its downstream pipeline can tolerate.
-
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
worker_concurrencyFlowWindows -
This is a unitless metric representing the cumulative time spent by all workers relative to wall-clock time (duration_in_millis / millisecond).
A pipeline is considered "saturated" when its worker_concurrency flow metric approaches its available pipeline.workers, because it indicates that all of its available workers are being kept busy. Tuning a saturated pipeline to have more workers can often work to increase that pipeline’s throughput and decrease back-pressure to its queue, unless the pipeline is experiencing back-pressure from its outputs.
A process is also considered "saturated" when its top-level worker_concurrency flow metric approaches the cumulative pipeline.workers across all pipelines, and similarly can be addressed by tuning the individual pipelines that are saturated.
-
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
worker_utilizationFlowWindows -
This is a unitless metric that indicates the percentage of available worker time being used by this individual plugin (duration / (uptime *
pipeline.workers). It is useful for identifying which plugins in a pipeline are using the available worker resources.A pipeline is considered "saturated" when
worker_utilizationapproaches 100, because it indicates that all of its workers are being kept busy. This is typically an indication of either downstream back-pressure or insufficient resources allocated to the pipeline. Tuning a saturated pipeline to have more workers can often work to increase that pipeline’s throughput and decrease back-pressure to its queue, unless the pipeline is experiencing back-pressure from its outputs.A pipeline is considered "starved" when
worker_utilizationapproaches 0, because it indicates that none of its workers are being kept busy. This is typically an indication that the inputs are not receiving or retrieving enough volume to keep the pipeline workers busy. Tuning a starved pipeline to have fewer workers can help it to consume less memory and CPU, freeing up resources for other pipelines. -
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
input_throughputFlowWindows -
This metric is expressed in events-per-second, and is the rate of events being pushed into the pipeline(s) queue(s) relative to wall-clock time (
events.in/ second). It includes events that are blocked by the queue and have not yet been accepted. -
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
filter_throughputFlowWindows -
This metric is expressed in events-per-second, and is the rate of events flowing through the filter phase of the pipeline(s) relative to wall-clock time (
events.filtered/ second). -
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
output_throughputFlowWindows -
This metric is expressed in events-per-second, and is the rate of events flowing through the output phase of the pipeline(s) relative to wall-clock time (
events.out/ second). -
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
queue_backpressureFlowWindows -
This is a unitless metric representing the cumulative time spent by all inputs blocked pushing events into their pipeline’s queue, relative to wall-clock time (queue_push_duration_in_millis / millisecond). It is typically most useful when looking at the stats for an individual pipeline.
While a "zero" value indicates no back-pressure to the queue, the magnitude of this metric is highly dependent on the shape of the pipelines and their inputs. It cannot be used to compare one pipeline to another or even one process to itself if the quantity or shape of its pipelines changes. A pipeline with only one single-threaded input may contribute up to 1.00, a pipeline whose inputs have hundreds of inbound connections may contribute much higher numbers to this combined value.
Additionally, some amount of back-pressure is both normal and expected for pipelines that are pulling data, as this back-pressure allows them to slow down and pull data at a rate its downstream pipeline can tolerate.
-
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
worker_concurrencyFlowWindows -
This is a unitless metric representing the cumulative time spent by all workers relative to wall-clock time (duration_in_millis / millisecond).
A pipeline is considered "saturated" when its worker_concurrency flow metric approaches its available pipeline.workers, because it indicates that all of its available workers are being kept busy. Tuning a saturated pipeline to have more workers can often work to increase that pipeline’s throughput and decrease back-pressure to its queue, unless the pipeline is experiencing back-pressure from its outputs.
A process is also considered "saturated" when its top-level worker_concurrency flow metric approaches the cumulative pipeline.workers across all pipelines, and similarly can be addressed by tuning the individual pipelines that are saturated.
-
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
worker_utilizationFlowWindows -
This is a unitless metric that indicates the percentage of available worker time being used by this individual plugin (duration / (uptime *
pipeline.workers). It is useful for identifying which plugins in a pipeline are using the available worker resources.A pipeline is considered "saturated" when
worker_utilizationapproaches 100, because it indicates that all of its workers are being kept busy. This is typically an indication of either downstream back-pressure or insufficient resources allocated to the pipeline. Tuning a saturated pipeline to have more workers can often work to increase that pipeline’s throughput and decrease back-pressure to its queue, unless the pipeline is experiencing back-pressure from its outputs.A pipeline is considered "starved" when
worker_utilizationapproaches 0, because it indicates that none of its workers are being kept busy. This is typically an indication that the inputs are not receiving or retrieving enough volume to keep the pipeline workers busy. Tuning a starved pipeline to have fewer workers can help it to consume less memory and CPU, freeing up resources for other pipelines. -
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
queue_persisted_growth_bytesFlowWindows -
This metric is expressed in bytes-per-second, and is the rate of change of the size of the persistent queue on disk, relative to wall-clock time (
queue.queue_size_in_bytes/ second). A positive number indicates that the queue size-on-disk is growing, and a negative number indicates that the queue is shrinking.NOTE: The size of a PQ on disk includes both unacknowledged events and previously-acknowledged events from pages that contain one or more unprocessed events. This means it grows gradually as individual events are added, but shrinks in large chunks each time a whole page of processed events is reclaimed (read more: PQ disk garbage collection).
-
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
queue_persisted_growth_eventsFlowWindows -
This metric is expressed in events-per-second, and is the rate of change of the number of unacknowledged events in the queue, relative to wall-clock time (
queue.events_count/ second). A positive number indicates that the queue’s event-count is growing, and a negative number indicates that the queue is shrinking. -
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
pluginsobject -
inputsInputPluginStats -
idstring -
namestring -
flowThroughputStats -
throughputFlowWindows -
This metric is expressed in events-per-second, and is the rate of events this input plugin is pushing into the pipeline’s queue relative to wall-clock time (
events.in/ second). It includes events that are blocked by the queue and have not yet been accepted. -
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
eventsInputPluginEventsStats -
outinteger -
queue_push_duration_in_millisinteger -
codecsCodecPluginStats -
idstring -
namestring -
decodeobject -
duration_in_millisinteger -
writes_ininteger -
outinteger -
encodeobject -
duration_in_millisinteger -
writes_ininteger -
filtersFilterOutputPluginStats -
idstring -
namestring -
flowWorkerStats -
worker_utilizationFlowWindows -
This is a unitless metric that indicates the percentage of available worker time being used by this individual plugin (
duration/ (uptime*pipeline.workers). It is useful for identifying which plugins in a pipeline are using the available worker resources. -
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
worker_millis_per_eventFlowWindows -
This metric is expressed in worker-millis-spent-per-event (
duration_in_millis/events.in) with higher scores indicating more resources spent per event. It is especially useful for identifying issues with plugins that operate on a small subset of events. An "Infinity" value for a given flow window indicates that worker millis have been spent without any events completing processing; this can indicate a plugin that is either stuck or handling only empty batches. -
last_1_minutenumber | enum -
Values:
Infinity -
last_5_minutesnumber | enum -
Values:
Infinity -
last_15_minutesnumber | enum -
Values:
Infinity -
last_1_hournumber | enum -
Values:
Infinity -
last_24_hoursnumber | enum -
Values:
Infinity -
eventsFilterOutputPluginEventsStats -
ininteger -
outinteger -
duration_in_millisinteger -
outputsFilterOutputPluginStats -
reloadsobject -
last_errorstringnullable -
successesinteger -
last_success_timestampstringnullable -
last_failure_timestampstringnullable -
failuresinteger -
queueoneOf -
type -
Values:
persisted -
capacityobject -
max_unread_eventsinteger -
page_capacity_in_bytesinteger -
max_queue_size_in_bytesinteger -
queue_size_in_bytesinteger -
dataobject -
pathstring -
free_space_in_bytesinteger -
storage_typestring -
eventsinteger -
events_countinteger -
queue_size_in_bytesinteger -
max_queue_size_in_bytesinteger -
compressionobject -
encodeobject -
type -
Values:
memory -
events_countinteger -
queue_size_in_bytesinteger -
max_queue_size_in_bytesinteger -
pipelineobject -
workersinteger -
batch_sizeinteger -
batch_delayinteger -
hashstring -
A unique hash identifier.
-
ephemeral_idstring -
A temporary unique identifier for the instance.
type: persisted
The metrics of persisted queue.
type: memory
The metrics of memory queue.