﻿---
title: EDOT Cloud Forwarder for AWS
description: Set up the EDOT Cloud Forwarder for AWS to bring your AWS logs to Elastic Observability.
url: https://www.elastic.co/elastic/docs-builder/assembled-docs/2379/reference/opentelemetry/edot-cloud-forwarder/aws
products:
  - EDOT Cloud Forwarder
  - Elastic Cloud Serverless
  - Elastic Observability
---

# EDOT Cloud Forwarder for AWS
EDOT Cloud Forwarder (CF) for AWS provides the EDOT Collector as a Lambda function that collects and forwards logs to Elastic Cloud Managed OTLP Endpoint.
EDOT Cloud Forwarder for AWS supports the following log sources:

| AWS Service                 | Telemetry Description                                                                                                                                 | Availability                           |
|-----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------|
| Virtual Private Cloud (VPC) | [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) to capture information about IP traffic.                             | `edot_cf_aws: ga 1.0.0, preview 0.2.6` |
| Elastic Load Balancer (ELB) | [Access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html) for your Application Load Balancer. | `edot_cf_aws: ga 1.0.0, preview 0.2.6` |
| AWS CloudTrail              | [CloudTrail Logs](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-examples.html) to record account activity.           | `edot_cf_aws: preview 0.2.6`           |

Read on to learn how to set up EDOT Cloud Forwarder for AWS.
<note>
  We are working to support other popular log types and sources. [Contact us](https://docs-v3-preview.elastic.dev/elastic/docs-builder/assembled-docs/2379/troubleshoot/ingest/opentelemetry/contact-support) to let us know of any specific requirements that could influence our plans.
</note>


## Prerequisites

<important>
  EDOT Cloud Forwarder for AWS requires a Managed OTLP endpoint and an API key. Managed OTLP is available for Elastic Cloud Serverless and Elastic Cloud Hosted.
</important>

To collect logs using EDOT Cloud Forwarder for AWS, you need:
<tab-set>
  <tab-item title="VPC Flow">
    To collect VPC Flow logs, you need:
    - A Virtual Private Cloud (VPC)
    - An S3 bucket for storing flow logs
    - A flow log configured with the S3 bucket as the destination
  </tab-item>

  <tab-item title="ELB Access">
    To collect Elastic Load Balancer (ELB) Access logs, you need:
    - An ELB of any type (ALB, NLB, CLB)
    - An S3 bucket to store the access logs
    - Access logging enabled, with the bucket as the destination
  </tab-item>

  <tab-item title="CloudTrail">
    To collect AWS CloudTrail logs, you need:
    - A trail that delivers account events as log files to an Amazon S3 bucket
    - An S3 bucket to store the trail logs
  </tab-item>
</tab-set>

You also need to know the URL of the managed OTLP endpoint and the API key for authentication.
To retrieve your Elastic Cloud Managed OTLP Endpoint endpoint address and API key, follow these steps:
<applies-switch>
  <applies-item title="serverless:">
    1. In Elastic Cloud, create an Observability project or open an existing one.
    2. Go to **Add data**, select **Applications** and then select **OpenTelemetry**.
    3. Copy the endpoint and authentication headers values.
    Alternatively, you can retrieve the endpoint from the **Manage project** page and create an API key manually from the **API keys** page.
  </applies-item>

  <applies-item title="ess:">
    `stack: preview 9.2`
    1. In Elastic Cloud, create an Elastic Cloud Hosted deployment or open an existing one.
    2. Go to **Add data**, select **Applications** and then select **OpenTelemetry**.
    3. Copy the endpoint and authentication headers values.
  </applies-item>
</applies-switch>

In the CloudFormation templates, the OTLP endpoint is set as `OTLPEndpoint`, and the API key is set as `ElasticAPIKey`.
<important>
  Trim the API key from `Authorization=ApiKey MYKEYVALUE...` to just `MYKEYVALUE...` before using it as the argument to the `ElasticAPIKey` parameter.
</important>


## Deployment methods

EDOT Cloud Forwarder for AWS can be deployed using any of the following methods:

| Deployment Method                           | Description                                                     |
|---------------------------------------------|-----------------------------------------------------------------|
| CloudFormation (AWS CLI)                    | Deploy using AWS CLI commands with CloudFormation templates.    |
| CloudFormation (AWS Console)                | Deploy using the AWS Management Console UI.                     |
| AWS Serverless Application Repository (SAR) | Deploy directly from the AWS Serverless Application Repository. |

Each method achieves the same result and uses CloudFormation templates. Choose the method that best adapts to your workflow.

## Deployment considerations

Before deploying EDOT Cloud Forwarder for AWS, consider the following actions:
- Deploy a separate CloudFormation stack for each log type, for example VPC Flow Logs or ELB Access Logs. Each CloudFormation stack can only process one log type and format at a time.
- Logs stored in S3 must be placed in separate buckets. Each log type should reside in its own dedicated bucket.
- The CloudFormation stack deployment region must match the region of the S3 bucket.


## CloudFormation templates

The CloudFormation templates are hosted in a public Amazon S3 bucket and can be accessed through HTTPS URL. You can reference these templates directly during deployment or download them for local use.

| Log type   | Log source | CloudFormation template                                                                              |
|------------|------------|------------------------------------------------------------------------------------------------------|
| VPC        | S3         | `https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml` |
| ELB        | S3         | `https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml` |
| CloudTrail | S3         | `https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml` |

For specific versions, edit `latest` in the URL to the required version in the format `vX.Y.Z`.

## Configure the template

Before deploying EDOT Cloud Forwarder for AWS, configure the CloudFormation template parameters based on your specific requirements. The template uses the following settings to deploy and configure the EDOT Collector Lambda function.

### Required settings

These are the required settings you need:

| Setting         | Description                                                                                                |
|-----------------|------------------------------------------------------------------------------------------------------------|
| `stack-name`    | Name of the CloudFormation stack, for example, `vpc-edot-cf`Do not use the same name for different stacks. |
| `OTLPEndpoint`  | The OTLP endpoint URL used for data ingestion, obtained from Elastic Cloud Serverless.                     |
| `ElasticApiKey` | API key for authentication with Elastic, obtained from Elastic Cloud Serverless.                           |


### Log source settings

Set the following settings based on the log source:
<tab-set>
  <tab-item title="S3">
    For logs sourced from S3, use the following settings:

    | Setting                        | Description                                                                                                                                              |
    |--------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|
    | `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:- `vpcflow`: VPC Flow Logs- `elbaccess`: ELB Access logs- `cloudtrail`: CloudTrail Logs |
    | `SourceS3BucketARN`            | Amazon Resource Name (ARN) of the S3 bucket where logs are stored. This bucket will trigger the `edot-cloud-forwarder` Lambda function automatically.    |
  </tab-item>
</tab-set>


### Optional settings

These are optional settings you can set in the CloudFormation template:

| Setting                                  | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `EdotCloudForwarderTimeout`              | Maximum execution time for the Lambda function, measured in seconds. The default value is is `300` seconds or 5 minutes. Minimum value is `1` second. Maximum value is `900` seconds. Increase the value if you experience Lambda timeouts. The default timeout is sufficient for most log file sizes.                                                                                                                                                                                                                                          |
| `EdotCloudForwarderVersion`              | Version of the EDOT Cloud Forwarder. Expected format is semantic versioning, for example `1.0.0`. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support.                                                                                                                                                                                                                                                                                                                                    |
| `EdotCloudForwarderMemorySize`           | Sets the allocated memory for the Lambda function, measured in megabytes. The default value is `512` MB. Minimum value is `128` MB. Maximum value is `10240` MB. Increase if memory usage consistently exceeds 80% or if you notice Lambda timeouts and processing times increase significantly. More memory also increases CPU, which improves processing speed and reduces timeouts.                                                                                                                                                          |
| `EdotCloudForwarderConcurrentExecutions` | Sets the maximum number of reserved concurrent executions for the Lambda function. Default value is `5`.  If new log files arrive frequently and you notice Lambda processing times consistently exceeding 1-2 seconds, consider increasing concurrent executions. Concurrent Lambda invocations are only spawned when needed, when new files arrive. This allows multiple log files to be processed in parallel instead of sequentially, avoiding processing delays. Make sure this value doesn't exceed your AWS account's concurrency limit. |
| `EdotCloudForwarderExporterMaxQueueSize` | Sets the internal OTLP exporter queue size, measured in bytes. The default value is `50000000` or 50 MB. Increase if you see "element size too large" errors in your Lambda CloudWatch logs, indicating the queue is full and data is being dropped. This prevents data loss during export.                                                                                                                                                                                                                                                     |

The default values provided have been determined through extensive load testing across different log types and data volumes. For most use cases, these defaults provide a good balance between cost and performance.
<tip>
  Adjust these parameters only if you notice performance issues such as Lambda timeouts, throttling, high memory usage or dropped data. If you need assistance tuning these parameters for your specific workload, refer to [Contact support](https://docs-v3-preview.elastic.dev/elastic/docs-builder/assembled-docs/2379/troubleshoot/ingest/opentelemetry/contact-support).
</tip>


## Sizing and performance tuning

Use the following sizing matrices to select appropriate Lambda memory and concurrency settings for your traffic volume. Proper tuning helps maximize performance and prevent throttling in high‑volume log sources.

### VPC Flow Logs sizing


| Throughput      | Log Rate                  | Recommended Memory | Recommended Concurrency |
|-----------------|---------------------------|--------------------|-------------------------|
| **< 5 MB/s**    | < 50,000 events/s         | 512 MB             | 5                       |
| **5 - 10 MB/s** | 50,000 - 100,000 events/s | 512 MB             | 10                      |
| **> 10 MB/s**   | > 100,000 events/s        | 1024 MB            | 20+                     |


### ELB Access Logs sizing


| Throughput       | Log Rate                  | Recommended Memory | Recommended Concurrency |
|------------------|---------------------------|--------------------|-------------------------|
| **< 10 MB/s**    | < 25,000 events/s         | 512 MB             | 5                       |
| **10 - 40 MB/s** | 25,000 - 100,000 events/s | 1024 MB            | 10                      |
| **> 40 MB/s**    | > 100,000 events/s        | 2048 MB            | 20+                     |


## Deploy using CloudFormation (AWS CLI)

Use the AWS CLI to deploy the EDOT Cloud Forwarder with CloudFormation templates. This method is ideal for automation and infrastructure-as-code workflows.

### Deployment examples

The following examples show how to deploy the ECF Cloud Forwarder using AWS CloudFormation CLI. Copy and paste these commands after replacing the placeholder values with your actual configuration.
- Use the `--template-url` flag to reference a template hosted on S3.
- Use the `--region` flag to specify the AWS region where the CloudFormation stack will be deployed. The CloudFormation stack deployment region must match the region of the S3 bucket where your logs are stored.
- To always use the most recent stable templates, use the `latest` path. For example, `v1/latest`.
- To pin a specific version, replace `latest` with the desired version tag. For example, `v1/v{{version.edot-cf-aws}}`.

Alternatively, if you have downloaded the template file, use the `--template-body file://<path>` option with a local template file.
<tab-set>
  <tab-item title="VPC Flow logs">
    This example deploys a CloudFormation stack to collect VPC Flow logs stored in an S3 bucket.
    ```sh
    aws cloudformation create-stack \
      --stack-name edot-cloud-forwarder-vpc \
      --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml \
      --capabilities CAPABILITY_NAMED_IAM \
      --region eu-central-1 \
      --parameters \
        ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-vpc-bucket-arn \
        ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
        ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
        ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="vpcflow"
    ```
  </tab-item>

  <tab-item title="ELB Access logs">
    This example deploys a CloudFormation stack to collect ALB Access logs stored in an S3 bucket.
    ```sh
    aws cloudformation create-stack \
      --stack-name edot-cloud-forwarder-alb \
      --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml \
      --capabilities CAPABILITY_NAMED_IAM \
      --region eu-central-1 \
      --parameters \
        ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-alb-bucket-arn \
        ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
        ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
        ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="elbaccess"
    ```
  </tab-item>

  <tab-item title="CloudTrail logs">
    This example deploys a CloudFormation stack to collect CloudTrail logs stored in an S3 bucket.
    ```sh
    aws cloudformation create-stack \
      --stack-name edot-cloud-forwarder-cloudtrail \
      --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml \
      --capabilities CAPABILITY_NAMED_IAM \
      --region eu-central-1 \
      --parameters \
        ParameterKey=SourceS3BucketARN,ParameterValue=your-cloudtrail-bucket-arn \
        ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
        ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
        ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="cloudtrail_log"
    ```
  </tab-item>

  <note>
    The `--capabilities CAPABILITY_NAMED_IAM` flag is required because this CloudFormation template creates AWS Identity and Access Management (IAM) resources. More specifically, it creates a named IAM role (`LambdaExecutionRole`) for the Lambda function. To acknowledge that AWS CloudFormation might create or modify IAM resources with custom names, you must specify the `CAPABILITY_NAMED_IAM` capability.
  </note>
</tab-set>


### Update an existing stack

To update an existing CloudFormation stack while preserving some parameter values, follow these steps:
<stepper>
  <step title="Identify the stack to update">
    Determine the name of the CloudFormation stack you want to modify.
  </step>

  <step title="Prepare the update command">
    Use the following structure for your update command:
    - Include all required parameters.
    - Use `UsePreviousValue=true` for parameters that should remain unchanged.
    - Specify `ParameterValue=<new-value>` for parameters that need to be updated.
  </step>

  <step title="Run the `update-stack` command">
    Run the command with the following parameters:
    ```sh
    aws cloudformation update-stack \
      --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/<template-file-name>.yaml \
      --stack-name <stack-name> \
      --capabilities CAPABILITY_NAMED_IAM \
      --region eu-central-1 \
      --parameters \
          ParameterKey=Param1,UsePreviousValue=true \
          ParameterKey=Param2,UsePreviousValue=true \
          ParameterKey=Param3,UsePreviousValue=true \
          ParameterKey=Param4,ParameterValue=<new-value>
    ```

    <dropdown title="Example using S3 logs template">
      For example, to modify the S3 bucket ARN for the `edot-cloud-forwarder-vpc` stack while keeping other parameter values unchanged:
      ```sh
      aws cloudformation update-stack \
        --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml \
        --stack-name edot-cloud-forwarder-vpc \
        --capabilities CAPABILITY_NAMED_IAM \
        --region eu-central-1 \
        --parameters \
            ParameterKey=OTLPEndpoint,UsePreviousValue=true \
            ParameterKey=ElasticAPIKey,UsePreviousValue=true \
            ParameterKey=EdotCloudForwarderS3LogsType,UsePreviousValue=true \
            ParameterKey=SourceS3BucketARN,ParameterValue=your-new-s3-vpc-bucket-arn
      ```
    </dropdown>
  </step>

  <step title="Verify the update">
    After running the command, check the stack status in the AWS Management Console under **CloudFormation** → **Stacks**. Then, run this command to confirm the updated parameter values:
    ```sh
    aws cloudformation describe-stacks --stack-name <stack-name>
    ```
  </step>
</stepper>


## Deploy using CloudFormation (AWS Console)


### Quick deployment (direct link)

Use the direct link to open the CloudFormation console with the S3 logs template preloaded:
[![Launch Stack (S3 logs)](https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png)](https://console.aws.amazon.com/cloudformation/home?#/stacks/new?templateURL=https://edot-cloud-forwarder.s3.amazonaws.com/v1/latest/cloudformation/s3_logs-cloudformation.yaml)
Then follow the steps to deploy the stack:
1. Configure all required parameters using the settings described in [Configure the template](#configure-the-template).
2. Select **Next** and check **Acknowledge IAM capabilities**. This is required because the template creates named IAM roles with permissions to access the required resources.
3. Review your configuration and select **Submit** to deploy the stack.
4. Monitor the progress until the stack reaches the `CREATE_COMPLETE` state.


### Manual deployment

To manually specify the template, follow these steps:
1. Navigate to **CloudFormation** in the AWS Console.
2. Select **Create Stack** and choose **With new resources (standard)** to start a fresh deployment.
3. Select one of the following options under **Specify template**:
   - **Amazon S3 URL (Recommended)**: Paste the CloudFormation template URL from [CloudFormation templates](#cloudformation-templates).
- **Upload a template file**: Download the template from the S3 URL and upload it manually.
4. Select **Next** and configure all required parameters using the settings described in [Configure the template](#configure-the-template).
5. Select **Next** again and check **Acknowledge IAM capabilities**. This is required because the template creates named IAM roles with permissions to access the required resources.
6. Review your configuration and select **Submit** to deploy the stack.
7. Monitor the progress until the stack reaches the `CREATE_COMPLETE` state.


### Update an existing stack

To modify parameters of an existing stack through the AWS Console:
1. Navigate to **CloudFormation** in the AWS Management Console.
2. Select the stack you want to update.
3. Click **Update stack** and select either **Make a direct update** or **Create a change set**.
4. Choose **Use existing template**.
5. Select **Next**.
6. Modify the parameter values as needed (refer to [Configure the template](#configure-the-template) for parameter descriptions).
7. Select **Next** and review your changes.
8. Select **Submit** to apply the updates. In case of a change set, **Execute changeset** .
9. Monitor the stack update progress in the console.


## Deploy using AWS Serverless Application Repository

In addition to deploying through CloudFormation templates, you can deploy the EDOT Cloud Forwarder application directly from the AWS Serverless Application Repository (SAR).

### Deploy from SAR

To deploy from SAR, follow these steps:
1. Navigate to **AWS Serverless Application Repository** in the AWS Management Console.
2. Select **Available applications** and check the box **Show apps that create custom IAM roles or resource policies**.
3. Search for `edot-cloud-forwarder-s3-logs` and select the application.
4. **Configure the application settings**: Under **Application settings**, fill in the parameters described in the [Configure the template](#configure-the-template) section. Refer to that section for details on each parameter.
5. **Acknowledge IAM role creation**: At the bottom of the page, check the box to acknowledge that the application will create custom IAM roles. This is required for the forwarder to access your S3 bucket and send data to Elastic Observability.
6. Select **Deploy**.

The deployment process will start, and a CloudFormation stack will be created with all the necessary resources. You can monitor the progress in the AWS CloudFormation console under **Stacks**.
<note>
  The same [deployment considerations](#deployment-considerations) apply to SAR deployments, including the requirement to deploy separate serverless applications for each log type and ensure the deployment region matches your S3 bucket region.
</note>


## CloudFormation stack resources

The CloudFormation templates create a number of resources to process logs from a specific log source.

### Resources for S3 logs

This is a list of resources created by the stack when processing S3 logs.

| Resource name                       | Type                                  | Description                                                                                                                                                                                                                                           |
|-------------------------------------|---------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `CustomNotificationUpdater`         | `AWS::CloudFormation::CustomResource` | Custom resource used to manage S3 event notifications dynamically.                                                                                                                                                                                    |
| `LambdaExecutionRole`               | `AWS::IAM::Role`                      | IAM role granting permissions needed for the Lambda function to interact with S3 and other AWS services.                                                                                                                                              |
| `LambdaFunction`                    | `AWS::Lambda::Function`               | Core Lambda function responsible for processing incoming logs from S3. This is a key resource in the stack.                                                                                                                                           |
| `LambdaInvokeConfig`                | `AWS::Lambda::EventInvokeConfig`      | Configures error handling and invocation settings for the Lambda function.                                                                                                                                                                            |
| `LambdaLogGroup`                    | `AWS::Logs::LogGroup`                 | CloudWatch log group storing logs for the main Lambda function. Useful for debugging and monitoring.                                                                                                                                                  |
| `LambdaPermissionS3Bucket`          | `AWS::Lambda::Permission`             | Grants permission for S3 to invoke the Lambda function when new logs arrive.                                                                                                                                                                          |
| `LambdaS3TriggerPolicy`             | `AWS::IAM::Policy`                    | IAM policy allowing the Lambda function to process events triggered by S3.                                                                                                                                                                            |
| `NotificationUpdaterLambda`         | `AWS::Lambda::Function`               | Utility Lambda function handling S3 event notification updates dynamically.                                                                                                                                                                           |
| `NotificationUpdaterLambdaLogGroup` | `AWS::Logs::LogGroup`                 | CloudWatch log group storing logs for the `NotificationUpdaterLambda` function.                                                                                                                                                                       |
| `S3FailureBucketARN`                | `AWS::S3::Bucket`                     | ARN of the bucket for storing failed invocations from the `edot-cloud-forwarder` Lambda function, preventing data loss, in the format `arn:aws:s3:::your-bucket-name`. If not defined, the template creates a dedicated failure bucket automatically. |

The main Lambda function, `LambdaFunction`, is the core component for processing S3 logs. S3 event notifications are handled dynamically using `CustomNotificationUpdater` and `NotificationUpdaterLambda`.
CloudWatch logs ensure detailed monitoring of Lambda executions. IAM roles and permissions control access between S3 and Lambda functions, while `S3FailureBucketARN` prevents data loss by capturing unprocessed logs.

## Datastreams

Logs collected by EDOT Cloud Forwarder for AWS are stored in Elasticsearch datastreams in OpenTelemetry native format. The following table shows which datastreams are used for each log type:

| **AWS Log Type** | **Datastream Dataset** | **Description**                        |
|------------------|------------------------|----------------------------------------|
| VPC Flow Logs    | `aws.vpcflow.otel`     | VPC Flow Log records                   |
| ELB Access Logs  | `aws.elbaccess.otel`   | ELB Access Log records (ALB, NLB, CLB) |

The logs are produced in OpenTelemetry native format. For detailed information about the field mappings and structure of each log type, refer to the following documentation:
- **VPC Flow Logs**: See [VPC Flow Log record fields](https://github.com/occamshub-dev/opentelemetry-collector-contrib/blob/main/extension/encoding/awslogsencodingextension/README.md#vpc-flow-log-record-fields) for the complete field mapping.
- **ELB Access Logs**: See [ELB Access Log fields](https://github.com/occamshub-dev/opentelemetry-collector-contrib/blob/main/extension/encoding/awslogsencodingextension/README.md#elb-access-log-fields) for the complete field mapping.


## Kibana integration setup

After EDOT Cloud Forwarder for AWS is successfully running and forwarding logs to Elastic Observability, install the Kibana integrations to visualize your data with out-of-the-box dashboards and visualizations.
To set up data visualization in Kibana:
1.Log into your Elastic Cloud deployment and open Kibana
2. Go to **Management** → **Integrations** in the Kibana navigation menu.
3. Search for the appropriate integration based on your log type and install it:

| **AWS Log Type** | **Integration Name**                         | **Description**                                              |
|------------------|----------------------------------------------|--------------------------------------------------------------|
| ELB Access Logs  | **AWS ELB OpenTelemetry Assets**             | Dashboards and visualizations for Elastic Load Balancer logs |
| VPC Flow Logs    | **AWS VPC Flow Logs OpenTelemetry Assets**   | Dashboards and visualizations for VPC flow log data          |
| CloudTrail Logs  | **AWS CloudTrail Logs OpenTelemetry Assets** | Dashboards and visualizations for CloudTrail log data        |

1. Once installed, navigate to **Dashboard** to view the pre-built dashboards for your AWS log data.


## Error handling and retrying

EDOT Cloud Forwarder store Lambda invocation events related to retryable errors at the S3 bucket specified by `S3FailureBucketARN` parameter.
Retryable errors here include,
- Network errors when attempting to forward to OTLPEndpoint
- Invalid or expired ElasticApiKey
- Lambda triggered by events that mismatch EdotCloudForwarderS3LogsType selection

These errors can be replayed manually to back-fill any gaps in your data.

### Replay failed events

To replay errors invoke the Lambda with manual trigger type `replayFailedEvents`. Replace `<LAMBDA_NAME>` with the name from your deployment.
```sh
aws lambda invoke \
  --function-name <LAMBDA_NAME> \
  --payload '{ "replayFailedEvents": {"replayFailedEvents":{"dryrun":false,"removeOnSuccess":true}}}' \
  --cli-binary-format raw-in-base64-out /dev/null
```

The following settings are available:

| Option          | Description                                                                                                                                    | Default |
|-----------------|------------------------------------------------------------------------------------------------------------------------------------------------|---------|
| dryrun          | Run the command without processing actual backup events. Useful to understand details about replaying error files from Lambda CloudWatch logs. | false   |
| removeOnSuccess | Configure whether to remove error event from S3 error destination, if processing is successful.                                                | true    |

When successful, you should get `"StatusCode": 200` as the output. Check CloudWatch logs (resource `LambdaLogGroup`) for detailed logs.
<note>
  With AWS CLI, you can use `--timeout` to increase currently configured Lambda timeout for custom invocations.
  However, if a timeout occurs, you need to run the custom event multiple times to fully process all error events from the bucket.
</note>


## Remove a CloudFormation stack

If you no longer need a deployed stack and want to clean up all associated resources, you can remove it using either the AWS CLI or the AWS Console.

### Important considerations

Deleting a stack removes all AWS resources created by that stack. However:
- If you allowed the stack to automatically create a dedicated S3 bucket for failed Lambda invocations, that bucket is not removed if it contains objects, because CloudFormation doesn't force-remove non-empty buckets. To remove the bucket entirely, you must empty it manually before deleting it.
- If you specified an existing bucket through the `S3FailureBucketARN` parameter, that bucket is not removed because it is not managed by the stack.


### Remove using AWS CLI

Use the following command to remove a stack:
```sh
aws cloudformation remove-stack \
  --stack-name <stack-name> \
  --region <stack-region>
```

You can monitor the deletion progress through this command:
```sh
aws cloudformation describe-stacks \
  --stack-name <stack-name> \
  --region <stack-region>
```

If the stack deletion fails and remains in a `DELETE_FAILED` state, you can retry the deletion with force mode:
```sh
aws cloudformation remove-stack \
  --stack-name <stack-name> \
  --region <stack-region> \
  --deletion-mode FORCE_DELETE_STACK
```

This forcibly removes the stack's resources, except any that cannot be removed, like the failure S3 bucket if it still contains objects. For a complete cleanup, empty the bucket manually before retrying deletion.
<dropdown title="Example: Deleting a stack using AWS CLI">
  The following command removes the `edot-cloud-forwarder-vpc` stack:
  ```sh
  aws cloudformation remove-stack \
    --stack-name edot-cloud-forwarder-vpc \
    --region eu-central-1
  ```
  Monitor the deletion progress:
  ```sh
  aws cloudformation describe-stacks \
    --stack-name edot-cloud-forwarder-vpc \
    --region eu-central-1
  ```
</dropdown>


### Remove using AWS Console

To remove a stack using the AWS Management Console:
1. Navigate to **CloudFormation** in the AWS Management Console.
2. Select the stack you want to remove from the list.
3. Click **Remove** at the top of the stack details page.
4. Monitor the deletion progress on the **Events** tab or wait until the stack disappears from the stack list (indicating deletion is complete).


## Monitoring and troubleshooting

To monitor your EDOT Cloud Forwarder Lambda function performance and troubleshoot issues:
1. **CloudWatch Metrics Explorer**: View Lambda metrics such as:
   - Duration
- ConcurrentExecutions
- Errors
- Throttles
- Invocations
2. **CloudWatch Logs**: Check the Lambda function logs for:
   - Processing errors
- Configuration issues
- Data export failures

The `LambdaLogGroup` resource created by the CloudFormation stack stores all Lambda execution logs. Look for error messages or warnings that indicate configuration or performance issues.