ZooKeeper Integration
<div class="condensed-table">
| | |
| --- | --- |
| Version | 1.13.1 (View all) |
| Compatible Kibana version(s) | 8.13.0 or higher |
| Supported Serverless project types
What’s this? | Security
Observability |
| Subscription level
What’s this? | Basic |
| Level of support
What’s this? | Elastic |
</div>
This integration periodically fetches metrics from the ZooKeeper service.
The ZooKeeper integration is tested with ZooKeeper 3.4.8 and is expected to work with all version >= 3.4.0. Versions prior to 3.4 do not support the mntr command.
The connection dataset fetches the data returned by the cons admin keyword.
**Example**
An example event for connection looks as following:
{
"@timestamp": "2020-07-06T16:12:07.612Z",
"agent": {
"ephemeral_id": "4d221f8f-7147-4855-8ea3-b4d2a5b80ae0",
"id": "2ff8a09c-c7f0-42f2-9fe1-65f7fd460651",
"name": "zookeeper-01",
"type": "metricbeat",
"version": "8.0.0"
},
"client": {
"ip": "172.28.0.1",
"port": 44338
},
"ecs": {
"version": "8.11.0"
},
"event": {
"dataset": "zookeeper.connection",
"duration": 3093417,
"module": "zookeeper"
},
"host": {
"name": "zookeeper-01"
},
"metricset": {
"name": "connection",
"period": 10000
},
"service": {
"address": "localhost:2181",
"type": "zookeeper"
},
"zookeeper": {
"connection": {
"interest_ops": 0,
"queued": 0,
"received": 1,
"sent": 0
}
}
}
ECS Field Reference
Please refer to the following document for detailed information on ECS fields.
**Exported fields**
| Field | Description | Type | Metric Type |
|---|---|---|---|
| @timestamp | Event timestamp. | date | |
| agent.id | keyword | ||
| client.ip | IP address of the client (IPv4 or IPv6). | ip | |
| client.port | Port of the client. | long | |
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
| cloud.availability_zone | Availability zone in which this host is running. | keyword | |
| cloud.image.id | Image ID for the cloud instance. | keyword | |
| cloud.instance.id | Instance ID of the host machine. | keyword | |
| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | |
| cloud.region | Region in which this host is running. | keyword | |
| container.id | Unique container id. | keyword | |
| data_stream.dataset | Data stream dataset. | constant_keyword | |
| data_stream.namespace | Data stream namespace. | constant_keyword | |
| data_stream.type | Data stream type. | constant_keyword | |
| event.dataset | Event dataset | constant_keyword | |
| event.module | Event module | constant_keyword | |
| host.containerized | If the host is a container. | boolean | |
| host.name | Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword | |
| host.os.build | OS build information. | keyword | |
| host.os.codename | OS codename, if any. | keyword | |
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | |
| zookeeper.connection.interest_ops | Interest ops | long | |
| zookeeper.connection.queued | Queued connections | long | gauge |
| zookeeper.connection.received | Received connections | long | counter |
| zookeeper.connection.sent | Connections sent | long | counter |
The mntr Metricset fetches the data returned by the mntr admin keyword.
**Example**
An example event for mntr looks as following:
{
"@timestamp": "2020-07-06T16:12:08.494Z",
"agent": {
"ephemeral_id": "4d221f8f-7147-4855-8ea3-b4d2a5b80ae0",
"id": "2ff8a09c-c7f0-42f2-9fe1-65f7fd460651",
"name": "zookeeper-01",
"type": "metricbeat",
"version": "8.0.0"
},
"ecs": {
"version": "8.11.0"
},
"event": {
"dataset": "zookeeper.mntr",
"duration": 15795652,
"module": "zookeeper"
},
"host": {
"name": "zookeeper-01"
},
"metricset": {
"name": "mntr",
"period": 10000
},
"service": {
"address": "localhost:2181",
"type": "zookeeper",
"version": "3.5.5-390fe37ea45dee01bf87dc1c042b5e3dcce88653, built on 05/03/2019 12:07 GMT"
},
"zookeeper": {
"mntr": {
"approximate_data_size": 44,
"ephemerals_count": 0,
"latency": {
"avg": 0,
"max": 0,
"min": 0
},
"max_file_descriptor_count": 1048576,
"num_alive_connections": 1,
"open_file_descriptor_count": 49,
"outstanding_requests": 0,
"packets": {
"received": 152,
"sent": 151
},
"server_state": "standalone",
"watch_count": 0,
"znode_count": 5
}
}
}
ECS Field Reference
Please refer to the following document for detailed information on ECS fields.
**Exported fields**
| Field | Description | Type | Metric Type |
|---|---|---|---|
| @timestamp | Event timestamp. | date | |
| agent.id | keyword | ||
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
| cloud.availability_zone | Availability zone in which this host is running. | keyword | |
| cloud.image.id | Image ID for the cloud instance. | keyword | |
| cloud.instance.id | Instance ID of the host machine. | keyword | |
| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | |
| cloud.region | Region in which this host is running. | keyword | |
| container.id | Unique container id. | keyword | |
| data_stream.dataset | Data stream dataset. | constant_keyword | |
| data_stream.namespace | Data stream namespace. | constant_keyword | |
| data_stream.type | Data stream type. | constant_keyword | |
| event.dataset | Event dataset | constant_keyword | |
| event.module | Event module | constant_keyword | |
| host.containerized | If the host is a container. | boolean | |
| host.name | Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword | |
| host.os.build | OS build information. | keyword | |
| host.os.codename | OS codename, if any. | keyword | |
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | |
| zookeeper.mntr.approximate_data_size | Approximate size of ZooKeeper data. | long | gauge |
| zookeeper.mntr.ephemerals_count | Number of ephemeral znodes. | long | gauge |
| zookeeper.mntr.followers | Number of followers seen by the current host. | long | gauge |
| zookeeper.mntr.latency.avg | Average latency between ensemble hosts in milliseconds. | long | gauge |
| zookeeper.mntr.latency.max | Maximum latency in milliseconds. | long | gauge |
| zookeeper.mntr.latency.min | Minimum latency in milliseconds. | long | gauge |
| zookeeper.mntr.max_file_descriptor_count | Maximum number of file descriptors allowed for the ZooKeeper process. | long | gauge |
| zookeeper.mntr.num_alive_connections | Number of connections to ZooKeeper that are currently alive. | long | gauge |
| zookeeper.mntr.open_file_descriptor_count | Number of file descriptors open by the ZooKeeper process. | long | gauge |
| zookeeper.mntr.outstanding_requests | Number of outstanding requests that need to be processed by the cluster. | long | gauge |
| zookeeper.mntr.packets.received | Number of ZooKeeper network packets received. | long | gauge |
| zookeeper.mntr.packets.sent | Number of ZooKeeper network packets sent. | long | gauge |
| zookeeper.mntr.pending_syncs | Number of pending syncs to carry out to ZooKeeper ensemble followers. | long | gauge |
| zookeeper.mntr.server_state | Role in the ZooKeeper ensemble. | keyword | |
| zookeeper.mntr.synced_followers | Number of synced followers reported when a node server_state is leader. | long | gauge |
| zookeeper.mntr.watch_count | Number of watches currently set on the local ZooKeeper process. | long | gauge |
| zookeeper.mntr.znode_count | Number of znodes reported by the local ZooKeeper process. | long | gauge |
The server Metricset fetches the data returned by the srvr admin keyword.
**Example**
An example event for server looks as following:
{
"@timestamp": "2020-07-06T16:12:12.409Z",
"agent": {
"ephemeral_id": "4d221f8f-7147-4855-8ea3-b4d2a5b80ae0",
"id": "2ff8a09c-c7f0-42f2-9fe1-65f7fd460651",
"name": "zookeeper-01",
"type": "metricbeat",
"version": "8.0.0"
},
"ecs": {
"version": "8.11.0"
},
"event": {
"dataset": "zookeeper.server",
"duration": 3001938,
"module": "zookeeper"
},
"host": {
"name": "zookeeper-01"
},
"metricset": {
"name": "server",
"period": 10000
},
"service": {
"address": "localhost:2181",
"type": "zookeeper",
"version": "3.5.5-390fe37ea45dee01bf87dc1c042b5e3dcce88653"
},
"zookeeper": {
"server": {
"connections": 1,
"count": 0,
"epoch": 0,
"latency": {
"avg": 0,
"max": 0,
"min": 0
},
"mode": "standalone",
"node_count": 5,
"outstanding": 0,
"received": 156,
"sent": 155,
"version_date": "2019-05-03T12:07:00Z",
"zxid": "0x0"
}
}
}
ECS Field Reference
Please refer to the following document for detailed information on ECS fields.
**Exported fields**
| Field | Description | Type | Metric Type |
|---|---|---|---|
| @timestamp | Event timestamp. | date | |
| agent.id | keyword | ||
| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
| cloud.availability_zone | Availability zone in which this host is running. | keyword | |
| cloud.image.id | Image ID for the cloud instance. | keyword | |
| cloud.instance.id | Instance ID of the host machine. | keyword | |
| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | |
| cloud.region | Region in which this host is running. | keyword | |
| container.id | Unique container id. | keyword | |
| data_stream.dataset | Data stream dataset. | constant_keyword | |
| data_stream.namespace | Data stream namespace. | constant_keyword | |
| data_stream.type | Data stream type. | constant_keyword | |
| event.dataset | Event dataset | constant_keyword | |
| event.module | Event module | constant_keyword | |
| host.containerized | If the host is a container. | boolean | |
| host.name | Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. |
keyword | |
| host.os.build | OS build information. | keyword | |
| host.os.codename | OS codename, if any. | keyword | |
| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | |
| zookeeper.server.connections | Number of clients currently connected to the server | long | gauge |
| zookeeper.server.count | Total transactions of the leader in epoch | long | counter |
| zookeeper.server.epoch | Epoch value of the Zookeeper transaction ID. An epoch signifies the period in which a server is a leader | long | |
| zookeeper.server.latency.avg | Average amount of time taken for the server to respond to a client request | long | gauge |
| zookeeper.server.latency.max | Maximum amount of time taken for the server to respond to a client request | long | gauge |
| zookeeper.server.latency.min | Minimum amount of time taken for the server to respond to a client request | long | gauge |
| zookeeper.server.mode | Mode of the server. In an ensemble, this may either be leader or follower. Otherwise, it is standalone | keyword | |
| zookeeper.server.node_count | Total number of nodes | long | gauge |
| zookeeper.server.outstanding | Number of requests queued at the server. This exceeds zero when the server receives more requests than it is able to process | long | gauge |
| zookeeper.server.received | Number of requests received by the server | long | counter |
| zookeeper.server.sent | Number of requests sent by the server | long | counter |
| zookeeper.server.version_date | Date of the Zookeeper release currently in use | date | |
| zookeeper.server.zxid | Unique value of the Zookeeper transaction ID. The zxid consists of an epoch and a counter. It is established by the leader and is used to determine the temporal ordering of changes | keyword |
**Changelog**
| Version | Details | Kibana version(s) |
|---|---|---|
| 1.13.1 | pass:[] Bug fix (View pull request) Remove hostname field from mntr data stream. |
8.13.0 or higher |
| 1.13.0 | pass:[] Enhancement (View pull request) Add processor support for connection, mntr and server data streams. |
8.13.0 or higher |
| 1.12.0 | pass:[] Enhancement (View pull request) ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template. |
8.13.0 or higher |
| 1.11.0 | pass:[] Enhancement (View pull request) Add global filter on data_stream.dataset to improve performance. |
8.8.0 or higher |
| 1.10.0 | pass:[] Enhancement (View pull request) Update the package format_version to 3.0.0. |
8.8.0 or higher |
| 1.9.0 | pass:[] Enhancement (View pull request) Enable time series data streams for the metrics datasets. This improves storage usage and query performance. For more details, see docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md |
8.8.0 or higher |
| 1.8.5 | pass:[] Enhancement (View pull request) Added dimension mapping for server datastream. |
8.2.0 or higher |
| 1.8.4 | pass:[] Enhancement (View pull request) Added dimension mapping for mntr datastream. |
8.2.0 or higher |
| 1.8.3 | pass:[] Enhancement (View pull request) Added dimension mapping for connection datastream. |
8.2.0 or higher |
| 1.8.2 | pass:[] Enhancement (View pull request) Added metric_type mapping for server datastream. |
8.2.0 or higher |
| 1.8.1 | pass:[] Enhancement (View pull request) Added metric_type mapping for mntr datastream. |
8.2.0 or higher |
| 1.8.0 | pass:[] Enhancement (View pull request) Added metric_type mapping for connection datastream |
8.2.0 or higher |
| 1.7.0 | pass:[] Enhancement (View pull request) Rename ownership from obs-service-integrations to obs-infraobs-integrations |
8.2.0 or higher |
| 1.6.0 | pass:[] Enhancement (View pull request) Migrate visualizations to lens. |
8.2.0 or higher |
| 1.5.1 | pass:[] Enhancement (View pull request) Added categories and/or subcategories. |
7.14.0 or higher 8.0.0 or higher |
| 1.5.0 | pass:[] Enhancement (View pull request) Update ECS version to 8.5.1 |
7.14.0 or higher 8.0.0 or higher |
| 1.4.0 | pass:[] Enhancement (View pull request) Added infrastructure category. |
7.14.0 or higher 8.0.0 or higher |
| 1.3.1 | pass:[] Enhancement (View pull request) Add documentation for multi-fields |
7.14.0 or higher 8.0.0 or higher |
| 1.3.0 | pass:[] Enhancement (View pull request) Update to ECS 8.0 |
— |
| 1.2.0 | pass:[] Enhancement (View pull request) Support Kibana 8.0 |
7.14.0 or higher 8.0.0 or higher |
| 1.1.1 | pass:[] Enhancement (View pull request) Uniform with guidelines |
— |
| 1.1.0 | pass:[] Enhancement (View pull request) Update to ECS 1.12.0 |
7.14.0 or higher |
| 1.0.0 | pass:[] Enhancement (View pull request) Release Kafka as GA |
— |
| 0.4.0 | pass:[] Enhancement (View pull request) Update integration description |
— |
| 0.3.1 | pass:[] Bug fix (View pull request) Add missing data_stream fields |
— |
| 0.3.0 | pass:[] Enhancement (View pull request) Set "event.module" and "event.dataset" |
— |
| 0.2.7 | pass:[] Enhancement (View pull request) Updating package owner |
— |
| 0.2.6 | pass:[] Bug fix (View pull request) Correct sample event file. |
— |
| 0.1.0 | pass:[] Enhancement (View pull request) initial release |
— |