Quickstart: Collect data with AWS Firehose

Warning

This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features.

In this quickstart guide, you’ll learn how to use AWS Firehose to send logs and metrics to Elastic.

The AWS Firehose streams are created using a CloudFormation template, which can collect all available CloudWatch logs and metrics for your AWS account.

This approach requires minimal configuration as the CloudFormation template creates a Firehose stream, enables CloudWatch metrics collection across all namespaces, and sets up an account-level subscription filter for CloudWatch log groups to send logs to Elastic via Firehose. You can use an AWS CLI command or upload the template to the AWS CloudFormation portal to customize the following parameter values:

Important

Some AWS services need additional manual configuration to properly ingest logs and metrics. For more information, check the AWS integration documentation.

Data collection with AWS Firehose is supported on ESS deployments in AWS, Azure and GCP.

Prerequisites ¶

A deployment using our hosted Elasticsearch Service on Elastic Cloud. The deployment includes an Elasticsearch cluster for storing and searching your data, and Kibana for visualizing and managing your data.
A user with the superuser built-in role or the privileges required to onboard data. ::::{dropdown} Expand to view required privileges
- Cluster: ['monitor', 'manage_own_api_key']
- Index: { names: ['logs-*-*', 'metrics-*-*'], privileges: ['auto_configure', 'create_doc'] }
- Kibana: { spaces: ['*'], feature: { fleet: ['all'], fleetv2: ['all'] } }
::::
An active AWS account and the necessary permissions to create delivery streams.

Note

The default CloudFormation stack is created in the AWS region selected for the user’s account. This region can be modified either through the AWS Console interface or by specifying a --region parameter in the AWS CLI command when creating the stack.

Limitations ¶

The AWS Firehose receiver has the following limitations:

It does not support AWS PrivateLink.
It is not available for on-premise Elastic Stack deployments.
The CloudFormation template detects and ingests logs and metrics within a single AWS region only.

The following table shows the type of data ingested by the supported AWS services:

AWS Service	Data type
VPC Flow Logs	Logs
API Gateway	Logs, Metrics
CloudTrail	Logs
Network Firewall	Logs, Metrics
Route53	Logs
WAF	Logs
DynamoDB	Metrics
EBS	Metrics
EC2	Metrics
ECS	Metrics
ELB	Metrics
EMR	Metrics
MSK	Metrics
Kinesis Data Stream	Metrics
Lambda	Metrics
NAT Gateway	Metrics
RDS	Metrics
S3	Metrics
SNS	Metrics
SQS	Metrics
Transit Gateway	Metrics
AWS Usage	Metrics
VPN	Metrics
Uncategorized Firehose Logs	Logs

Collect your data ¶

In Kibana, go to the Observability UI and click Add Data.
Under What do you want to monitor? select Cloud*, *AWS, and then select AWS Firehose. :::{image} ../../../images/observability-quickstart-aws-firehose-entry-point.png
:alt: AWS Firehose entry point
:class: screenshot
:::
Click Create Firehose Stream in AWS to create a CloudFormation stack from the CloudFormation template.
Go back to the Add Observability Data page.

Visualize your data ¶

After installation is complete and all relevant data is flowing into Elastic, the Visualize your data section allows you to access the different dashboards for the various services.

Here is an example of the VPC Flow logs dashboard:

Refer to What is Elastic Observability? for a description of other useful features.