AWS Fargate Integration (for ECS clusters)
| Version | 1.3.0 (View all) |
| Compatible Kibana version(s) | 8.13.0 or higher 9.0.0 or higher |
| Supported Serverless project types What's this? |
Security Observability |
| Subscription level What's this? |
Basic |
| Level of support What's this? |
Elastic |
The AWS Fargate integration helps to retrieve metadata, network metrics, and Docker stats about your containers and the tasks that are part of an Amazon Elastic Container Service (Amazon ECS) cluster.
The AWS Fargate integration currently supports ECS clusters only. It does not support EKS clusters.
This integration does not require AWS credentials. The ECS task metadata endpoint is accessible only inside the cluster.
To start collecting AWS Fargate metrics, you must run the Elastic Agent as a sidecar container alongside your application container in the same task definition.
Each task definition must run an Agent because task metadata information is only available to containers running in the task.
Here's an example of an Elastic Agent running as a sidecar with an application container:
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Family: !Ref TaskName
Cpu: 256
Memory: 512
NetworkMode: awsvpc
ExecutionRoleArn: !Ref ExecutionRole
ContainerDefinitions:
- Name: <application-container> << ===== Application container
Image: <application-container-image>
<application-container-settings>
- Name: elastic-agent-container << ===== Elastic Agent container
Image: docker.elastic.co/beats/elastic-agent:8.12.0
The Elastic Agent collects metrics using the Amazon ECS task metadata endpoint.
The Amazon ECS task metadata endpoint is an HTTP endpoint available to each container and enabled by default on AWS Fargate platform version 1.4.0 and later. The Elastic Agent uses Task metadata endpoint version 4.
This section shows you how to run the Elastic Agent in a ECS cluster, start collecting Fargate on ECS metrics, and send them to an Elastic Stack.
To quickly deploy on your existing ECS cluster, follow these steps.
Open the AWS Management Console and visit the Amazon ECS page. Here you can select "Task Definitions" and then "Create new Task Definition" to start the wizard.
Step 1:
- Select "Fargate" from the list of available launch types.
Step 2:
- Add your preferred name for the "Task definition name", for example "elastic-agent-fargate-deployment".
- For the "Task role", select "ecsFargateTaskExecutionRole".
- For the "Operating system family", select "Linux".
- Pick a value for "Task memory (GB)" and "Task CPU (vCPU)"; the lowest values are fine for testing purposes.
- Click on "Add container".
As for the container, you can use the following values:
- Container name:
elastic-agent-container - Image:
docker.elastic.co/beats/elastic-agent:8.12.0 - Environment variables:
- FLEET_ENROLL:
yes - FLEET_ENROLLMENT_TOKEN:
<enrollment-token> - FLEET_URL:
<fleet-server-url>
- FLEET_ENROLL:
Tip
use the AWS Secrets Manager to store the Fleet Server enrollment token.
Select an existing ECS cluster and create a new service with launch type "FARGATE". Use the task definition we just created.
As soon as the Elastic Agent is started, open the dashboard "[AWS Fargate] Fargate Overview" and you will see the metrics show up in few minutes.
In this example, we will use the AWS CLI and a CloudFormation template to set up the following resources:
- an ECS cluster,
- a task definition for the Elastic Agent,
- a service to execute the agent task on the cluster.
Prepare you terminal and AWS environment to create the ECS cluster for the testing.
Set default AWS region for this session:
export AWS_DEFAULT_REGION="us-east-1"
Store the enrollment token and the Fleet Server URL in the AWS Secrets Manager:
aws secretsmanager create-secret \
--name FLEET_ENROLLMENT_TOKEN \
--secret-string <your-fleet-enrollment-token-goes-here>
aws secretsmanager create-secret \
--name FLEET_URL \
--secret-string <your-fleet-url>
Take note of the Amazon Resource Name (ARN) of both secrets, we'll use them in a moment.
Tip
if you need to update them during your tests, use the following put-secret-value to do it:
aws secretsmanager put-secret-value \
--secret-id FLEET_ENROLLMENT_TOKEN \
--secret-string <fleet-enrollment-token>
One more thing. You need to pick one subnet where your ECS cluster will be created in. Take note of the subnet ID for the very next step.
Copy the following CloudFormation template and save it on you computer with the name cloudformation.yml:
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
SubnetID:
Type: String
Description: Enter the ID of the subnet you want to create the cluster in.
FleetEnrollmentTokenSecretArn:
Type: String
Description: Enter the Amazon Resource Name (ARN) of the secret holding the enrollment token for the Elastic Agent.
FleetUrlSecretArn:
Type: String
Description: Enter the Amazon Resource Name (ARN) of the secret holding the Fleet Server URL.
ClusterName:
Type: String
Default: elastic-agent-fargate
Description: Enter the name of the Fargate cluster to create.
RoleName:
Type: String
Default: ecsFargateTaskExecutionRole
Description: Enter the Amazon Resource Name (ARN) of the task execution role that grants the Amazon ECS container agent permission to make AWS API calls on your behalf.
TaskName:
Type: String
Default: elastic-agent-fargate-task
Description: Enter the name of the task definition to create.
ServiceName:
Type: String
Default: elastic-agent-fargate-service
Description: Enter the name of the service to create.
LogGroupName:
Type: String
Default: elastic-agent-fargate-log-group
Description: Enter the name of the log group to create.
Resources:
Cluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: !Ref ClusterName
ClusterSettings:
- Name: containerInsights
Value: disabled
LogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Ref LogGroupName
ExecutionRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Ref RoleName
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service: ecs-tasks.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
Policies:
- PolicyName: !Sub 'EcsTaskExecutionRole-${AWS::StackName}'
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- secretsmanager:GetSecretValue
Resource:
- !Ref FleetEnrollmentTokenSecretArn
- !Ref FleetUrlSecretArn
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
Family: !Ref TaskName
Cpu: 256
Memory: 512
NetworkMode: awsvpc
ExecutionRoleArn: !Ref ExecutionRole
ContainerDefinitions:
- Name: elastic-agent-container
Image: docker.elastic.co/beats/elastic-agent:8.12.0
Secrets:
- Name: FLEET_ENROLLMENT_TOKEN
ValueFrom: !Ref FleetEnrollmentTokenSecretArn
- Name: FLEET_URL
ValueFrom: !Ref FleetUrlSecretArn
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-region: !Ref AWS::Region
awslogs-group: !Ref LogGroup
awslogs-stream-prefix: ecs
Environment:
- Name: FLEET_ENROLL
Value: true
# You migh need to set FLEET_INSECURE to true
# if you're connecting to a development
# environment. Use it responsibly.
# - Name: FLEET_INSECURE
# Value: true
RequiresCompatibilities:
- EC2
- FARGATE
Service:
Type: AWS::ECS::Service
Properties:
ServiceName: !Ref ServiceName
Cluster: !Ref Cluster
TaskDefinition: !Ref TaskDefinition
DesiredCount: 1
LaunchType: FARGATE
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: ENABLED
Subnets:
- !Ref SubnetID
We are now finally ready to deploy the ECS cluster with the Elastic Agent running in its own task.
aws cloudformation create-stack \
--stack-name elastic-agent-fargate-deployment \
--template-body file://./cloudformation.yml \
--capabilities CAPABILITY_NAMED_IAM \
--parameters \
ParameterKey=SubnetID,ParameterValue=<subnet-id> \
ParameterKey=FleetEnrollmentTokenSecretArn,ParameterValue=arn:aws:secretsmanager:eu-west-1:000123456789:secret:FLEET_ENROLLMENT_TOKEN-ZxsJGw \
ParameterKey=FleetUrlSecretArn,ParameterValue=arn:aws:secretsmanager:eu-west-1:000123456789:secret:FLEET_URL-mvjF3a \
ParameterKey=ClusterName,ParameterValue=elastic-agent-fargate \
ParameterKey=RoleName,ParameterValue=ecsFargateTaskExecutionRole \
ParameterKey=TaskName,ParameterValue=elastic-agent-fargate-task \
ParameterKey=ServiceName,ParameterValue=elastic-agent-fargate-service \
ParameterKey=LogGroupName,ParameterValue=elastic-agent-fargate-log-group
The AWS CLI will return a StackId:
{
"StackId": "arn:aws:cloudformation:eu-west-1:000123456789:stack/elastic-agent-deployment/fc324160-b0f9-11ec-9c45-0643aa7239c3"
}
Check the stack status until it has reached the CREATE_COMPLETE status. Use the AWS Management Console or the AWS CLI (requires the tool jq):
$ aws cloudformation list-stacks | jq '.StackSummaries[] | .StackName + " " + .StackStatus'
"elastic-agent-fargate-deployment CREATE_COMPLETE"
That's it!
Once you're done with experimenting, you can remove all the resources (ECS cluster, task, service, etc) with the following command:
aws cloudformation delete-stack --stack-name elastic-agent-fargate-deployment
If you want to learn more about Amazon ECS metrics, take a look at the blog post How to monitor Amazon ECS with Elastic Observability.
ECS Field Reference
Please refer to the following document for detailed information on ECS fields.
Exported fields
| Field | Description | Type | Metric Type |
|---|---|---|---|
| @timestamp | Event timestamp. | date | |
| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword | |
| awsfargate.task_stats.cluster_name | Cluster name | keyword | |
| awsfargate.task_stats.cpu.core.*.norm.pct | Percentage of time per CPU core normalized by the number of CPU cores. | scaled_float | gauge |
| awsfargate.task_stats.cpu.core.*.pct | Percentage of time per CPU core. | scaled_float | gauge |
| awsfargate.task_stats.cpu.core.*.ticks | CPU ticks per CPU core. | long | counter |
| awsfargate.task_stats.cpu.kernel.norm.pct | Percentage of time in kernel space normalized by the number of CPU cores. | scaled_float | gauge |
| awsfargate.task_stats.cpu.kernel.pct | Percentage of time in kernel space. | scaled_float | gauge |
| awsfargate.task_stats.cpu.kernel.ticks | CPU ticks in kernel space. | long | counter |
| awsfargate.task_stats.cpu.system.norm.pct | Percentage of total CPU time in the system normalized by the number of CPU cores. | scaled_float | gauge |
| awsfargate.task_stats.cpu.system.pct | Percentage of total CPU time in the system. | scaled_float | gauge |
| awsfargate.task_stats.cpu.system.ticks | CPU system ticks. | long | counter |
| awsfargate.task_stats.cpu.total.norm.pct | Total CPU usage normalized by the number of CPU cores. | scaled_float | gauge |
| awsfargate.task_stats.cpu.total.pct | Total CPU usage. | scaled_float | gauge |
| awsfargate.task_stats.cpu.user.norm.pct | Percentage of time in user space normalized by the number of CPU cores. | scaled_float | gauge |
| awsfargate.task_stats.cpu.user.pct | Percentage of time in user space. | scaled_float | gauge |
| awsfargate.task_stats.cpu.user.ticks | CPU ticks in user space. | long | counter |
| awsfargate.task_stats.diskio.read.bytes | Bytes read during the life of the container | long | counter |
| awsfargate.task_stats.diskio.read.ops | Number of reads during the life of the container | long | counter |
| awsfargate.task_stats.diskio.read.queued | Total number of queued requests | long | counter |
| awsfargate.task_stats.diskio.read.rate | Number of current reads per second | long | gauge |
| awsfargate.task_stats.diskio.read.service_time | Total time to service IO requests, in nanoseconds | long | counter |
| awsfargate.task_stats.diskio.read.wait_time | Total time requests spent waiting in queues for service, in nanoseconds | long | counter |
| awsfargate.task_stats.diskio.reads | Number of current reads per second | scaled_float | gauge |
| awsfargate.task_stats.diskio.summary.bytes | Bytes read and written during the life of the container | long | counter |
| awsfargate.task_stats.diskio.summary.ops | Number of I/O operations during the life of the container | long | counter |
| awsfargate.task_stats.diskio.summary.queued | Total number of queued requests | long | counter |
| awsfargate.task_stats.diskio.summary.rate | Number of current operations per second | long | gauge |
| awsfargate.task_stats.diskio.summary.service_time | Total time to service IO requests, in nanoseconds | long | counter |
| awsfargate.task_stats.diskio.summary.wait_time | Total time requests spent waiting in queues for service, in nanoseconds | long | counter |
| awsfargate.task_stats.diskio.total | Number of reads and writes per second | scaled_float | gauge |
| awsfargate.task_stats.diskio.write.bytes | Bytes written during the life of the container | long | counter |
| awsfargate.task_stats.diskio.write.ops | Number of writes during the life of the container | long | counter |
| awsfargate.task_stats.diskio.write.queued | Total number of queued requests | long | counter |
| awsfargate.task_stats.diskio.write.rate | Number of current writes per second | long | gauge |
| awsfargate.task_stats.diskio.write.service_time | Total time to service IO requests, in nanoseconds | long | counter |
| awsfargate.task_stats.diskio.write.wait_time | Total time requests spent waiting in queues for service, in nanoseconds | long | counter |
| awsfargate.task_stats.diskio.writes | Number of current writes per second | scaled_float | gauge |
| awsfargate.task_stats.identifier | Container identifier across tasks and clusters, which equals to container.name + '/' + container.id. | keyword | |
| awsfargate.task_stats.memory.commit.peak | Peak committed bytes on Windows | long | counter |
| awsfargate.task_stats.memory.commit.total | Total bytes | long | counter |
| awsfargate.task_stats.memory.fail.count | Fail counter. | scaled_float | counter |
| awsfargate.task_stats.memory.limit | Memory limit. | long | gauge |
| awsfargate.task_stats.memory.private_working_set.total | Private working sets on Windows | long | gauge |
| awsfargate.task_stats.memory.rss.pct | Memory resident set size percentage. | scaled_float | gauge |
| awsfargate.task_stats.memory.rss.total | Total memory resident set size. | long | gauge |
| awsfargate.task_stats.memory.rss.usage.max | Max memory usage. | long | counter |
| awsfargate.task_stats.memory.rss.usage.pct | Memory usage percentage. | scaled_float | gauge |
| awsfargate.task_stats.memory.rss.usage.total | Total memory usage. | long | gauge |
| awsfargate.task_stats.memory.stats.* | Raw memory stats from the cgroups memory.stat interface | unsigned_long | |
| awsfargate.task_stats.memory.usage.max | Max memory usage. | long | counter |
| awsfargate.task_stats.memory.usage.total | Total memory usage. | long | gauge |
| awsfargate.task_stats.network.*.inbound.bytes | Total number of incoming bytes. | long | counter |
| awsfargate.task_stats.network.*.inbound.dropped | Total number of dropped incoming packets. | long | counter |
| awsfargate.task_stats.network.*.inbound.errors | Total errors on incoming packets. | long | counter |
| awsfargate.task_stats.network.*.inbound.packets | Total number of incoming packets. | long | counter |
| awsfargate.task_stats.network.*.outbound.bytes | Total number of incoming bytes. | long | counter |
| awsfargate.task_stats.network.*.outbound.dropped | Total number of dropped incoming packets. | long | counter |
| awsfargate.task_stats.network.*.outbound.errors | Total errors on incoming packets. | long | counter |
| awsfargate.task_stats.network.*.outbound.packets | Total number of incoming packets. | long | counter |
| awsfargate.task_stats.task_desired_status | The desired status for the task from Amazon ECS. | keyword | |
| awsfargate.task_stats.task_known_status | The known status for the task from Amazon ECS. | keyword | |
| awsfargate.task_stats.task_name | ECS task name | keyword | |
| container.labels.com_amazonaws_ecs_cluster | ECS Cluster name | keyword | |
| container.labels.com_amazonaws_ecs_container-name | ECS container name | keyword | |
| container.labels.com_amazonaws_ecs_task-arn | ECS task ARN | keyword | |
| container.labels.com_amazonaws_ecs_task-definition-family | ECS task definition family | keyword | |
| container.labels.com_amazonaws_ecs_task-definition-version | ECS task definition version | keyword | |
| container.name | Container name. | keyword | |
| data_stream.dataset | Data stream dataset. | constant_keyword | |
| data_stream.namespace | Data stream namespace. | constant_keyword | |
| data_stream.type | Data stream type. | constant_keyword |
Example
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"awsfargate": {
"task_stats": {
"cluster_name": "default",
"task_known_status": "RUNNING",
"task_desired_status": "RUNNING",
"cpu": {
"core": {
"1": {
"pct": 0,
"norm": {
"pct": 0
},
"ticks": 1520000000
},
"2": {
"pct": 0,
"norm": {
"pct": 0
},
"ticks": 1420180000000
}
},
"kernel": {
"norm": {
"pct": 0
},
"pct": 0,
"ticks": 1520000000
},
"system": {
"norm": {
"pct": 1
},
"pct": 2,
"ticks": 1420180000000
},
"total": {
"norm": {
"pct": 0.2
},
"pct": 0.4
},
"user": {
"norm": {
"pct": 0
},
"pct": 0,
"ticks": 490000000
}
},
"diskio": {
"read": {
"bytes": 3452928,
"ops": 118,
"queued": 0,
"rate": 0,
"service_time": 0,
"wait_time": 0
},
"reads": 0,
"summary": {
"bytes": 3452928,
"ops": 118,
"queued": 0,
"rate": 0,
"service_time": 0,
"wait_time": 0
},
"total": 0,
"write": {
"bytes": 0,
"ops": 0,
"queued": 0,
"rate": 0,
"service_time": 0,
"wait_time": 0
},
"writes": 0
},
"identifier": "query-metadata/1234",
"memory": {
"fail": {
"count": 0
},
"limit": 0,
"rss": {
"pct": 0.0010557805807105247,
"total": 4157440
},
"stats": {
"active_anon": 4157440,
"active_file": 4497408,
"cache": 6000640,
"dirty": 16384,
"hierarchical_memory_limit": 2147483648,
"hierarchical_memsw_limit": 9223372036854772000,
"inactive_anon": 0,
"inactive_file": 1503232,
"mapped_file": 2183168,
"pgfault": 6668,
"pgmajfault": 52,
"pgpgin": 5925,
"pgpgout": 3445,
"rss": 4157440,
"rss_huge": 0,
"total_active_anon": 4157440,
"total_active_file": 4497408,
"total_cache": 600064,
"total_dirty": 16384,
"total_inactive_anon": 0,
"total_inactive_file": 4497408,
"total_mapped_file": 2183168,
"total_pgfault": 6668,
"total_pgmajfault": 52,
"total_pgpgin": 5925,
"total_pgpgout": 3445,
"total_rss": 4157440,
"total_rss_huge": 0,
"total_unevictable": 0,
"total_writeback": 0,
"unevictable": 0,
"writeback": 0
},
"usage": {
"max": 15294464,
"total": 12349440
}
},
"network": {
"eth0": {
"inbound": {
"bytes": 137315578,
"dropped": 0,
"errors": 0,
"packets": 94338
},
"outbound": {
"bytes": 1086811,
"dropped": 0,
"errors": 0,
"packets": 25857
}
}
},
"task_name": "query-metadata"
}
},
"cloud": {
"region": "us-west-2"
},
"container": {
"id": "1234",
"image": {
"name": "mreferre/eksutils"
},
"labels": {
"com_amazonaws_ecs_cluster": "arn:aws:ecs:us-west-2:111122223333:cluster/default",
"com_amazonaws_ecs_container-name": "query-metadata",
"com_amazonaws_ecs_task-arn": "arn:aws:ecs:us-west-2:111122223333:task/default/febee046097849aba589d4435207c04a",
"com_amazonaws_ecs_task-definition-family": "query-metadata",
"com_amazonaws_ecs_task-definition-version": "7"
},
"name": "query-metadata"
},
"service": {
"type": "awsfargate"
}
}
Changelog
| Version | Details | Kibana version(s) |
|---|---|---|
| 1.3.0 | Enhancement (View pull request) Add support for Kibana 9.0.0. |
8.13.0 or higher 9.0.0 or higher |
| 1.2.2 | Enhancement (View pull request) Add missing category. |
8.13.0 or higher |
| 1.2.1 | Enhancement (View pull request) Clarify that the integration supports ECS clusters only. |
8.13.0 or higher |
| 1.2.0 | Enhancement (View pull request) Add processor support for task_stats data stream. |
8.13.0 or higher |
| 1.1.0 | Enhancement (View pull request) ECS version updated to 8.11.0. Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template. |
8.13.0 or higher |
| 1.0.0 | Enhancement (View pull request) Make AWS Fargate GA |
8.12.0 or higher |
| 0.5.1 | Enhancement (View pull request) Improve documentation |
— |
| 0.5.0 | Bug fix (View pull request) Remove memory.usage.pct field and use memory.usage.total instead for plain memory usage. |
— |
| 0.4.0 | Enhancement (View pull request) Update the package format_version to 3.0.0. |
— |
| 0.3.0 | Enhancement (View pull request) Enable TSDB for task stats data stream. This improves storage usage and query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html. |
— |
| 0.2.5 | Enhancement (View pull request) Update DiskIO Write and Read visualizations to use last_value instead of average. |
— |
| 0.2.4 | Enhancement (View pull request) Migrate AWS Fargate input control to new control panel. |
— |
| 0.2.3 | Enhancement (View pull request) Set dimension fields and add agent.id. |
— |
| 0.2.2 | Enhancement (View pull request) Add metric type to fields. |
— |
| 0.2.1 | Enhancement (View pull request) Added categories and/or subcategories. |
— |
| 0.2.0 | Enhancement (View pull request) Improve dashboards by removing individual visualizations from library |
— |
| 0.1.3 | Enhancement (View pull request) Clarify how to run the awsfargate integration as a sidecar container. |
— |
| 0.1.2 | Enhancement (View pull request) Add DesiredStatus and KnownStatus for Fargate Tasks among the collected fields |
— |
| 0.1.1 | Enhancement (View pull request) Improve description and screenshots |
— |
| 0.1.0 | Enhancement (View pull request) initial release |
— |