Elastic
Loading

Parse AWS VPC Flow Log

The parse_aws_vpc_flow_log processor decodes AWS VPC Flow log messages.

Example

The following example configuration decodes the message field using the default version 2 VPC flow log format.

processors:
  - parse_aws_vpc_flow_log:
      format: version account-id interface-id srcaddr dstaddr srcport dstport protocol packets bytes start end action log-status
      field: message

Configuration settings

Note

Elastic Agent processors execute before ingest pipelines, which means that your processor configurations cannot refer to fields that are created by ingest pipelines or Logstash. For more limitations, refer to What are some limitations of using processors?

Name Required Default Description
field No message Source field containing the VPC flow log message.
target_field No aws.vpcflow Target field for the VPC flow log object. This applies only to the original VPC flow log fields. ECS fields are written to the standard location.
format Yes VPC flow log format. This supports VPC flow log fields from versions 2 through 5. It will accept a string or a list of strings. Each format must have a unique number of fields to enable matching it to a flow log message.
mode No ecs Controls which fields are generated. The available options are:

* original: generates the fields specified in the format string.
* ecs: maps the original fields to ECS and removes the original fields that are mapped to ECS.
* ecs_and_original: maps the original fields to ECS and retains all the original fields.

To learn more, refer to Modes.
ignore_missing No false Whether to ignore a missing source field.
ignore_failure No false Whether to ignore failures while parsing and transforming the flow log message.
id No Instance ID for debugging purposes.

This section provides more information about available modes.

Original

This mode returns the same fields found in the format string. It will drop any fields whose value is a dash (-). It converts the strings into the appropriate data types. These are the known field names and their data types.

Note

The AWS VPC flow field names use underscores instead of dashes within Elastic Agent. You may configure the format using field names that contain either.

VPC Flow Log Field Data Type
account_id string
action string
az_id string
bytes long
dstaddr ip
dstport integer
end timestamp
flow_direction string
instance_id string
interface_id string
log_status string
packets long
pkt_dst_aws_service string
pkt_dstaddr ip
pkt_src_aws_service string
pkt_srcaddr ip
protocol integer
region string
srcaddr ip
srcport integer
start timestamp
sublocation_id string
sublocation_type string
subnet_id string
tcp_flags integer
tcp_flags_array* integer
traffic_path integer
type string
version integer
vpc_id string

ECS

This mode maps the original VPC flow log fields into their associated Elastic Common Schema (ECS) fields. It removes the original fields that were mapped to ECS to reduced duplication. These are the field associations. There may be some transformations applied to derive the ECS field.

VPC Flow Log Field ECS Field
account_id cloud.account.id
action event.outcome
action event.action
action event.type
az_id cloud.availability_zone
bytes network.bytes
bytes source.bytes
dstaddr destination.address
dstaddr destination.ip
dstport destination.port
end @timestamp
end event.end
flow_direction network.direction
instance_id cloud.instance.id
packets network.packets
packets source.packets
protocol network.iana_number
protocol network.transport
region cloud.region
srcaddr network.type
srcaddr source.address
srcaddr source.ip
srcport source.port
start event.start

ECS and Original

This mode maps the fields into ECS and retains all the original fields. Below is an example document produced using ecs_and_orignal mode.

{
  "@timestamp": "2021-03-26T03:29:09Z",
  "aws": {
    "vpcflow": {
      "account_id": "64111117617",
      "action": "REJECT",
      "az_id": "use1-az5",
      "bytes": 1,
      "dstaddr": "10.200.0.0",
      "dstport": 33004,
      "end": "2021-03-26T03:29:09Z",
      "flow_direction": "ingress",
      "instance_id": "i-0axxxxxx1ad77",
      "interface_id": "eni-069xxxxxb7a490",
      "log_status": "OK",
      "packets": 52,
      "pkt_dst_aws_service": "CLOUDFRONT",
      "pkt_dstaddr": "10.200.0.80",
      "pkt_src_aws_service": "AMAZON",
      "pkt_srcaddr": "89.160.20.156",
      "protocol": 17,
      "region": "us-east-1",
      "srcaddr": "89.160.20.156",
      "srcport": 50041,
      "start": "2021-03-26T03:28:12Z",
      "sublocation_id": "fake-id",
      "sublocation_type": "wavelength",
      "subnet_id": "subnet-02d645xxxxxxxdbc0",
      "tcp_flags": 1,
      "tcp_flags_array": [
        "fin"
      ],
      "traffic_path": 1,
      "type": "IPv4",
      "version": 5,
      "vpc_id": "vpc-09676f97xxxxxb8a7"
    }
  },
  "cloud": {
    "account": {
      "id": "64111117617"
    },
    "availability_zone": "use1-az5",
    "instance": {
      "id": "i-0axxxxxx1ad77"
    },
    "region": "us-east-1"
  },
  "destination": {
    "address": "10.200.0.0",
    "ip": "10.200.0.0",
    "port": 33004
  },
  "event": {
    "action": "reject",
    "end": "2021-03-26T03:29:09Z",
    "outcome": "failure",
    "start": "2021-03-26T03:28:12Z",
    "type": [
      "connection",
      "denied"
    ]
  },
  "message": "5 64111117617 eni-069xxxxxb7a490 89.160.20.156 10.200.0.0 50041 33004 17 52 1 1616729292 1616729349 REJECT OK vpc-09676f97xxxxxb8a7 subnet-02d645xxxxxxxdbc0 i-0axxxxxx1ad77 1 IPv4 89.160.20.156 10.200.0.80 us-east-1 use1-az5 wavelength fake-id AMAZON CLOUDFRONT ingress 1",
  "network": {
    "bytes": 1,
    "direction": "ingress",
    "iana_number": "17",
    "packets": 52,
    "transport": "udp",
    "type": "ipv4"
  },
  "related": {
    "ip": [
      "89.160.20.156",
      "10.200.0.0",
      "10.200.0.80"
    ]
  },
  "source": {
    "address": "89.160.20.156",
    "bytes": 1,
    "ip": "89.160.20.156",
    "packets": 52,
    "port": 50041
  }
}