Hadoop metrics
The Hadoop system records a set of metric counters for each job that it runs. elasticsearch-hadoop extends on that and provides metrics about its activity for each job run by leveraging the Hadoop Counters infrastructure. During each run, elasticsearch-hadoop sends statistics from each task instance, as it is running, which get aggregated by the Map/Reduce infrastructure and are available through the standard Hadoop APIs.
elasticsearch-hadoop provides the following counters, available under org.elasticsearch.hadoop.mr.Counter enum:
| Counter name | Purpose |
|---|---|
| Data focused | |
| BYTES_SENT | Total number of data/communication bytes sent over the network to Elasticsearch |
| BYTES_ACCEPTED | Data/Documents accepted by Elasticsearch in bytes |
| BYTES_RETRIED | Data/Documents rejected by Elasticsearch in bytes |
| BYTES_RECEIVED | Data/Documents received from Elasticsearch in bytes |
| Document focused | |
| DOCS_SENT | Number of docs sent over the network to Elasticsearch |
| DOCS_ACCEPTED | Number of documents sent and accepted by Elasticsearch |
| DOCS_RETRIED | Number of documents sent but rejected by Elasticsearch |
| DOCS_RECEIVED | Number of documents received from Elasticsearch |
| Network focused | |
| BULK_TOTAL | Number of bulk requests made to Elasticsearch |
| BULK_RETRIES | Number of bulk retries (caused by document rejections) |
| SCROLL_TOTAL | Number of scroll pulled from Elasticsearch |
| NODE_RETRIES | Number of node fall backs (caused by network errors) |
| NET_RETRIES | Number of network retries (caused by network errors) |
| Time focused | |
| NET_TOTAL_TIME_MS | Overall time (in ms) spent over the network |
| BULK_TOTAL_TIME_MS | Time (in ms) spent over the network by the bulk requests |
| BULK_RETRIES_TOTAL_TIME_MS | Time (in ms) spent over the network retrying bulk requests |
| SCROLL_TOTAL_TIME_MS | Time (in ms) spent over the network reading the scroll requests |
One can use the counters programatically, depending on the API used, through mapred or mapreduce. Whatever the choice, elasticsearch-hadoop performs automatic reports without any user intervention. In fact, when using elasticsearch-hadoop one will see the stats reported at the end of the job run, for example:
13:55:08,100 INFO main mapreduce.Job - Job job_local127738678_0013 completed successfully
13:55:08,101 INFO main mapreduce.Job - Counters: 35
...
Elasticsearch Hadoop Counters
Bulk Retries=0
Bulk Retries Total Time(ms)=0
Bulk Total=20
Bulk Total Time(ms)=518
Bytes Accepted=159129
Bytes Sent=159129
Bytes Received=79921
Bytes Retried=0
Documents Accepted=993
Documents Sent=993
Documents Received=0
Documents Retried=0
Network Retries=0
Network Total Time(ms)=937
Node Retries=0
Scroll Total=0
Scroll Total Time(ms)=0