EDOT Cloud Forwarder for AWS

Serverless Observability Preview ECH Preview Self-Managed Unavailable EDOT CF AWS Preview

EDOT Cloud Forwarder (CF) for AWS provides the EDOT Collector as a Lambda function that collects and forwards logs to Elastic Cloud Managed OTLP Endpoint.

EDOT Cloud Forwarder for AWS supports the following log sources:

Read on to learn how to set up EDOT Cloud Forwarder for AWS.

To collect logs using EDOT Cloud Forwarder for AWS, you need:

You also need to know the URL of the managed OTLP endpoint and the API key for authentication.

To retrieve your Elastic Cloud Managed OTLP Endpoint endpoint address and API key, follow these steps:

In the CloudFormation templates, the OTLP endpoint is set as OTLPEndpoint, and the API key is set as ElasticAPIKey.

EDOT Cloud Forwarder for AWS can be deployed using any of the following methods:

Each method achieves the same result and uses CloudFormation templates. Choose the method that best adapts to your workflow.

Before deploying EDOT Cloud Forwarder for AWS, consider the following actions:

• Deploy a separate CloudFormation stack for each log type, for example VPC Flow Logs or ELB Access Logs. Each CloudFormation stack can only process one log type and format at a time.
• Logs stored in S3 must be placed in separate buckets. Each log type should reside in its own dedicated bucket.
• The CloudFormation stack deployment region must match the region of the S3 bucket.

The CloudFormation templates are hosted in a public Amazon S3 bucket and can be accessed through HTTPS URL. You can reference these templates directly during deployment or download them for local use.

For specific versions, edit latest in the URL to the required version in the format vX.Y.Z.

Before deploying EDOT Cloud Forwarder for AWS, configure the CloudFormation template parameters based on your specific requirements. The template uses the following settings to deploy and configure the EDOT Collector Lambda function.

These are the required settings you need:

Set the following settings based on the log source:

These are optional settings you can set in the CloudFormation template:

The default values provided have been determined through extensive load testing across different log types and data volumes. For most use cases, these defaults provide a good balance between cost and performance.

Use the following sizing matrices to select appropriate Lambda memory and concurrency settings for your traffic volume. Proper tuning helps maximize performance and prevent throttling in high‑volume log sources.

Use the AWS CLI to deploy the EDOT Cloud Forwarder with CloudFormation templates. This method is ideal for automation and infrastructure-as-code workflows.

The following examples show how to deploy the ECF Cloud Forwarder using AWS CloudFormation CLI. Copy and paste these commands after replacing the placeholder values with your actual configuration.

• Use the --template-url flag to reference a template hosted on S3.
• Use the --region flag to specify the AWS region where the CloudFormation stack will be deployed. The CloudFormation stack deployment region must match the region of the S3 bucket where your logs are stored.
• To always use the most recent stable templates, use the latest path. For example, v0/latest.
• To pin a specific version, replace latest with the desired version tag. For example, v0/v{{version.edot-cf-aws}}.

Alternatively, if you have downloaded the template file, use the --template-body file://<path> option with a local template file.

aws cloudformation create-stack \
  --stack-name edot-cloud-forwarder-vpc \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-vpc-bucket-arn \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="vpcflow"
		

aws cloudformation create-stack \
  --stack-name edot-cloud-forwarder-alb \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-alb-bucket-arn \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="elbaccess"
		

aws cloudformation create-stack \
  --stack-name edot-cloud-forwarder-cloudtrail \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=SourceS3BucketARN,ParameterValue=your-cloudtrail-bucket-arn \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="cloudtrail_log"
		

To update an existing CloudFormation stack while preserving some parameter values, follow these steps:

For fastest deployment, use the direct link to launch the CloudFormation console with the S3 logs template pre-loaded:

1. Navigate to Launch Stack (S3 logs) to open the AWS CloudFormation console with the template automatically loaded.
2. The stack creation wizard appears. Configure all required parameters using the settings described in Configure the template.
3. Select Next and check Acknowledge IAM capabilities. This is required because the template creates named IAM roles with permissions to access the required resources.
4. Review your configuration and select Submit to deploy the stack.
5. Monitor the progress until the stack reaches the CREATE_COMPLETE state.

To manually specify the template, follow these steps:

1. Navigate to CloudFormation in the AWS Console.
2. Select Create Stack and choose With new resources (standard) to start a fresh deployment.
3. Select one of the following options under Specify template:
   • Amazon S3 URL (Recommended): Paste the CloudFormation template URL from CloudFormation templates.
   • Upload a template file: Download the template from the S3 URL and upload it manually.
4. Select Next and configure all required parameters using the settings described in Configure the template.
5. Select Next again and check Acknowledge IAM capabilities. This is required because the template creates named IAM roles with permissions to access the required resources.
6. Review your configuration and select Submit to deploy the stack.
7. Monitor the progress until the stack reaches the CREATE_COMPLETE state.

To modify parameters of an existing stack through the AWS Console:

1. Navigate to CloudFormation in the AWS Management Console.
2. Select the stack you want to update.
3. Click Update stack and select either Make a direct update or Create a change set.
4. Choose Use existing template.
5. Select Next.
6. Modify the parameter values as needed (refer to Configure the template for parameter descriptions).
7. Select Next and review your changes.
8. Select Submit to apply the updates. In case of a change set, Execute changeset .
9. Monitor the stack update progress in the console.

In addition to deploying through CloudFormation templates, you can deploy the EDOT Cloud Forwarder application directly from the AWS Serverless Application Repository (SAR).

To deploy from SAR, follow these steps:

1. Navigate to AWS Serverless Application Repository in the AWS Management Console.
2. Select Available applications and check the box Show apps that create custom IAM roles or resource policies.
3. Search for edot-cloud-forwarder-s3-logs and select the application.
4. Configure the application settings: Under Application settings, fill in the parameters described in the Configure the template section. Refer to that section for details on each parameter.
5. Acknowledge IAM role creation: At the bottom of the page, check the box to acknowledge that the application will create custom IAM roles. This is required for the forwarder to access your S3 bucket and send data to Elastic Observability.
6. Select Deploy.

The deployment process will start, and a CloudFormation stack will be created with all the necessary resources. You can monitor the progress in the AWS CloudFormation console under Stacks.

The CloudFormation templates create a number of resources to process logs from a specific log source.

This is a list of resources created by the stack when processing S3 logs.

The main Lambda function, LambdaFunction, is the core component for processing S3 logs. S3 event notifications are handled dynamically using CustomNotificationUpdater and NotificationUpdaterLambda.

CloudWatch logs ensure detailed monitoring of Lambda executions. IAM roles and permissions control access between S3 and Lambda functions, while S3FailureBucketARN prevents data loss by capturing unprocessed logs.

Logs collected by EDOT Cloud Forwarder for AWS are stored in Elasticsearch datastreams in OpenTelemetry native format. The following table shows which datastreams are used for each log type:

The logs are produced in OpenTelemetry native format. For detailed information about the field mappings and structure of each log type, refer to the following documentation:

• VPC Flow Logs: See VPC Flow Log record fields for the complete field mapping.
• ELB Access Logs: See ELB Access Log fields for the complete field mapping.

After EDOT Cloud Forwarder for AWS is successfully running and forwarding logs to Elastic Observability, install the Kibana integrations to visualize your data with out-of-the-box dashboards and visualizations.

To set up data visualization in Kibana:

1.Log into your Elastic Cloud deployment and open Kibana 2. Go to Management → Integrations in the Kibana navigation menu. 3. Search for the appropriate integration based on your log type and install it:

4. Once installed, navigate to Dashboard to view the pre-built dashboards for your AWS log data.

EDOT Cloud Forwarder store Lambda invocation events related to retryable errors at the S3 bucket specified by S3FailureBucketARN parameter. Retryable errors here include,

• Network errors when attempting to forward to OTLPEndpoint
• Invalid or expired ElasticApiKey
• Lambda triggered by events that mismatch EdotCloudForwarderS3LogsType selection

These errors can be replayed manually to back-fill any gaps in your data.

To replay errors invoke the Lambda with manual trigger type replayFailedEvents. Replace <LAMBDA_NAME> with the name from your deployment.

aws lambda invoke \
  --function-name <LAMBDA_NAME> \
  --payload '{ "replayFailedEvents": {"replayFailedEvents":{"dryrun":false,"removeOnSuccess":true}}}' \
  --cli-binary-format raw-in-base64-out /dev/null
		

The following settings are available:

When successful, you should get "StatusCode": 200 as the output. Check CloudWatch logs (resource LambdaLogGroup) for detailed logs.

If you no longer need a deployed stack and want to clean up all associated resources, you can remove it using either the AWS CLI or the AWS Console.

Deleting a stack removes all AWS resources created by that stack. However:

• If you allowed the stack to automatically create a dedicated S3 bucket for failed Lambda invocations, that bucket is not removed if it contains objects, because CloudFormation doesn't force-remove non-empty buckets. To remove the bucket entirely, you must empty it manually before deleting it.
• If you specified an existing bucket through the S3FailureBucketARN parameter, that bucket is not removed because it is not managed by the stack.

Use the following command to remove a stack:

aws cloudformation remove-stack \
  --stack-name <stack-name> \
  --region <stack-region>
		

You can monitor the deletion progress through this command:

aws cloudformation describe-stacks \
  --stack-name <stack-name> \
  --region <stack-region>
		

If the stack deletion fails and remains in a DELETE_FAILED state, you can retry the deletion with force mode:

aws cloudformation remove-stack \
  --stack-name <stack-name> \
  --region <stack-region> \
  --deletion-mode FORCE_DELETE_STACK
		

This forcibly removes the stack's resources, except any that cannot be removed, like the failure S3 bucket if it still contains objects. For a complete cleanup, empty the bucket manually before retrying deletion.

aws cloudformation remove-stack \
  --stack-name edot-cloud-forwarder-vpc \
  --region eu-central-1
		

aws cloudformation describe-stacks \
  --stack-name edot-cloud-forwarder-vpc \
  --region eu-central-1
		

To remove a stack using the AWS Management Console:

1. Navigate to CloudFormation in the AWS Management Console.
2. Select the stack you want to remove from the list.
3. Click Remove at the top of the stack details page.
4. Monitor the deletion progress on the Events tab or wait until the stack disappears from the stack list (indicating deletion is complete).

To monitor your EDOT Cloud Forwarder Lambda function performance and troubleshoot issues:

1. CloudWatch Metrics Explorer: View Lambda metrics such as:

   • Duration
   • ConcurrentExecutions
   • Errors
   • Throttles
   • Invocations

2. CloudWatch Logs: Check the Lambda function logs for:

   • Processing errors
   • Configuration issues
   • Data export failures

The LambdaLogGroup resource created by the CloudFormation stack stores all Lambda execution logs. Look for error messages or warnings that indicate configuration or performance issues.