AWS CloudWatch

<div class="condensed-table">
| | |
| --- | --- |
| Version | 2.38.2 (View all) |
| Compatible Kibana version(s) | 8.16.2 or higher |
| Supported Serverless project types
What’s this? | Security
Observability |
| Subscription level
What’s this? | Basic |

</div>
The AWS CloudWatch integration allows you to monitor AWS CloudWatch. AWS CloudWatch is a service that provides data and insights for monitoring applications and changes to system performance.

Use the AWS CloudWatch integration to collect metrics and logs on the operational health of your AWS resources, applications, and services running on AWS and on-premises. Then visualize that data in Kibana, create alerts to notify you if something goes wrong, and reference logs and metrics when troubleshooting an issue.

For example, you could use the data from this integration to detect anomalous behavior in your environments. You could also use the data to troubleshoot the underlying issue by looking at additional context in the logs and metrics, such as the source of the behavior, and more.

The AWS CloudWatch integration collects two types of data: logs and metrics.

Logs help you keep a record of different services in AWS, like EC2, RDS, and S3. The log data stream includes the CloudWatch log message along with contextual information. See more details in the Logs reference.

Metrics give you insight into the state of different services in AWS, like EC2, RDS, and S3. The metric data stream includes the metrics that are returned from a CloudWatch API query along with contextual information. See more details in the Metrics reference.

You need Elasticsearch for storing and searching your data and Kibana for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your own hardware.

Before using any AWS integration you will need:

• AWS Credentials to connect with your AWS account.
• AWS Permissions to make sure the user you’re using to connect has permission to share the relevant data.

For more details about these requirements, please take a look at the AWS integration documentation.

Use this integration if you only need to collect data from the AWS CloudWatch service.

If you want to collect data from two or more AWS services, consider using the AWS integration. When you configure the AWS integration, you can collect data from as many AWS services as you’d like.

For step-by-step instructions on how to set up an integration, see the Getting started guide.

The CloudWatch logs input has several advanced options to fit specific use cases.

AWS CloudWatch Logs sometimes takes extra time to make the latest logs available to clients like the Agent.

The CloudWatch integration offers the latency setting to address this scenario. Latency translates the query’s time range to consider the CloudWatch Logs latency. For example, a 5m latency means the integration will query CloudWatch for logs available 5 minutes ago.

If you are collecting log events from multiple log groups using log_group_name_prefix, you should review the value of the number_of_workers.

The number_of_workers setting defines the number of workers assigned to reading from log groups. Each log group matching the log_group_name_prefix requires a worker to keep log ingestion as close to real-time as possible. For example, if log_group_name_prefix matches five log groups, then number_of_workers should be set to 5. The default value is 1.

The cloudwatch data stream collects CloudWatch logs. Users can use Amazon CloudWatch logs to monitor, store, and access log files from different sources.

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.

{
    "@timestamp": "2020-02-20T07:02:37.000Z",
    "aws": {
        "cloudwatch": {
            "message": "ip-172-31-81-156 ec2net: [get_meta] Trying to get http://169.254.169.254/latest/meta-data/network/interfaces/macs/12:e2:a9:95:8b:97/local-ipv4s"
        }
    },
    "data_stream": {
        "dataset": "aws.cloudwatch_logs",
        "namespace": "default",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "event": {
        "ingested": "2021-07-19T21:47:04.696803300Z",
        "original": "2020-02-20T07:02:37.000Z Feb 20 07:02:37 ip-172-31-81-156 ec2net: [get_meta] Trying to get http://169.254.169.254/latest/meta-data/network/interfaces/macs/12:e2:a9:95:8b:97/local-ipv4s"
    },
    "tags": [
        "preserve_original_event"
    ]
}

{
    "@timestamp": "2020-05-28T17:17:02.812Z",
    "agent": {
        "ephemeral_id": "17803f33-b617-4ce9-a9ac-e218c02aeb4b",
        "id": "12f376ef-5186-4e8b-a175-70f1140a8f30",
        "name": "MacBook-Elastic.local",
        "type": "metricbeat",
        "version": "8.0.0"
    },
    "aws": {
        "cloudwatch": {
            "namespace": "AWS/EC2"
        },
        "dimensions": {
            "InstanceId": "i-0830bfecfa7173cbe"
        },
        "ec2": {
            "metrics": {
                "CPUUtilization": {
                    "avg": 0.7661943132361363,
                    "max": 0.833333333333333
                },
                "DiskWriteOps": {
                    "avg": 0,
                    "max": 0
                }
            }
        }
    },
    "cloud": {
        "account": {
            "id": "428152502467",
            "name": "elastic-beats"
        },
        "provider": "aws",
        "region": "us-west-2"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "event": {
        "dataset": "aws.cloudwatch_metrics",
        "duration": 14119105951,
        "module": "aws"
    },
    "metricset": {
        "name": "cloudwatch",
        "period": 300000
    },
    "service": {
        "type": "aws"
    }
}

ECS Field Reference

Please refer to the following document for detailed information on ECS fields.