opentelemetry
Loading

EDOT Cloud Forwarder for AWS

EDOT Cloud Forwarder (CF) for AWS provides the EDOT Collector as a Lambda function that collects and forwards logs to Elastic Cloud Managed OTLP Endpoint.

EDOT Cloud Forwarder for AWS supports the following log sources:

AWS service Telemetry description Availability
Virtual Private Cloud (VPC) VPC Flow Logs to capture information about IP traffic.
Elastic Load Balancer (ELB) Access logs for your Application Load Balancer.
AWS CloudTrail CloudTrail Logs to record account activity.

To collect logs using EDOT Cloud Forwarder for AWS, you need the following:

To collect VPC Flow logs, you need:

  • A Virtual Private Cloud (VPC)
  • An S3 bucket for storing flow logs
  • A flow log configured with the S3 bucket as the destination
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

To collect Elastic Load Balancer (ELB) Access logs, you need:

  • An ELB of any type (ALB, NLB, CLB)
  • An S3 bucket to store the access logs
  • Access logging enabled, with the bucket as the destination
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

To collect AWS CloudTrail logs, you need:

  • A trail that delivers account events as log files to an Amazon S3 bucket
  • An S3 bucket to store the trail logs
  • An Elastic Managed OTLP endpoint and an API key. Refer to Endpoint and API key.

To retrieve your Elastic Cloud Managed OTLP Endpoint endpoint address and API key, follow these steps:

  1. In Elastic Cloud, create an Observability project or open an existing one.
  2. Go to Add data, select Applications and then select OpenTelemetry.
  3. Copy the endpoint and authentication headers values.

Alternatively, you can retrieve the endpoint from the Manage project page and create an API key manually from the API keys page.

You need an Elastic Cloud Hosted deployment version 9.2 or later.

  1. In Elastic Cloud, create an Elastic Cloud Hosted deployment or open an existing one.
  2. Go to Add data, select Applications and then select OpenTelemetry.
  3. Copy the endpoint and authentication headers values.
Important

Trim the API key from Authorization=ApiKey MYKEYVALUE... to just MYKEYVALUE... before using it as the argument to the ElasticAPIKey parameter.

Deploy EDOT Cloud Forwarder for AWS with one click using the AWS CloudFormation console:

Launch Stack

After clicking the button:

  1. Configure the required parameters:

    Parameter Description
    stack-name Name of the CloudFormation stack, for example vpc-edot-cf.
    OTLPEndpoint The OTLP endpoint URL from Elastic Cloud Serverless or Elastic Cloud Hosted.
    ElasticApiKey API key for authentication with Elastic.
    SourceS3BucketARN ARN of the S3 bucket where your logs are stored.
    EdotCloudForwarderS3LogsType The log type: vpcflow, elbaccess, or cloudtrail.

  2. Select Next and check Acknowledge IAM capabilities.

  3. Review your configuration and select Submit to deploy the stack.

  4. Monitor the progress until the stack reaches the CREATE_COMPLETE state.

Tip

The CloudFormation stack deployment region must match the region of the S3 bucket where your logs are stored.

Before deploying EDOT Cloud Forwarder for AWS, consider the following:

  • Deploy a separate CloudFormation stack for each log type, for example VPC Flow Logs or ELB Access Logs. Each CloudFormation stack can only process one log type and format at a time.
  • Logs stored in S3 must be placed in separate buckets. Each log type should reside in its own dedicated bucket.

Logs collected by EDOT Cloud Forwarder for AWS are stored in Elasticsearch datastreams in OpenTelemetry native format. The following table shows which datastreams are used for each log type:

AWS log type Datastream dataset Description
VPC Flow Logs aws.vpcflow.otel VPC Flow Log records
ELB Access Logs aws.elbaccess.otel ELB Access Log records (ALB, NLB, CLB)

The logs are produced in OpenTelemetry native format. For detailed information about the field mappings and structure of each log type, refer to the following documentation:

After EDOT Cloud Forwarder for AWS is successfully running and forwarding logs to Elastic Observability, install the Kibana integrations to visualize your data with out-of-the-box dashboards and visualizations.

To set up data visualization in Kibana:

  1. Log into your Elastic Cloud deployment and open Kibana

  2. Go to Management → Integrations in the Kibana navigation menu.

  3. Search for the appropriate integration based on your log type and install it:

    AWS log type Integration name Description
    ELB Access Logs AWS ELB OpenTelemetry Assets Dashboards and visualizations for Elastic Load Balancer logs
    VPC Flow Logs AWS VPC Flow Logs OpenTelemetry Assets Dashboards and visualizations for VPC flow log data
    CloudTrail Logs AWS CloudTrail Logs OpenTelemetry Assets Dashboards and visualizations for CloudTrail log data

  4. Once installed, navigate to Dashboard to view the pre-built dashboards for your AWS log data.

EDOT Cloud Forwarder for AWS has the following limitations:

Limitation Description
VPC/PrivateLink not supported EDOT Cloud Forwarder cannot be deployed inside a VPC or use AWS PrivateLink endpoints. The Lambda function requires public internet access to forward data to the OTLP endpoint.
Managed OTLP Input only EDOT Cloud Forwarder is tested exclusively with Elastic Cloud Managed OTLP Endpoint. Forwarding to a self-deployed EDOT Collector Gateway is not tested and forwarding to APM Server is not supported.
Single log type per bucket Each S3 bucket can only contain one log type. Mixed log formats in the same bucket are not supported yet.
  • Configure the template: Learn about all configuration options, including optional settings and sizing recommendations.
  • Deployment methods: Explore alternative deployment methods using AWS CLI or AWS Serverless Application Repository.
  • Troubleshooting: Diagnose and resolve issues with log forwarding.