F5 BIG-IP

<div class="condensed-table">
| | |
| --- | --- |
| Version | 1.24.0 (View all) |
| Compatible Kibana version(s) | 8.16.2 or higher |
| Supported Serverless project types
What’s this? | Security
Observability |
| Subscription level
What’s this? | Basic |
| Level of support
What’s this? | Elastic |

</div>

The F5 BIG-IP integration allows users to monitor LTM, AFM, APM, ASM, AVR, System Information, iHealth Information, BOT, and DOS activity. F5 BIG-IP covers software and hardware designed around application availability, access control, and security solutions.

Use the F5 BIG-IP integration to collect and parse data from F5 BIG-IP using telemetry streaming and then visualize that data in Kibana. The integration currently only supports JSON-formatted telemetry data generated by following F5 features: LTM, AFM, APM, ASM, AVR, System Information, iHealth Information, BOT, and DOS activity. Ingesting or parsing the syslog events produced at the operating system level is not supported. Use the Custom Logs integration to ingest these system-level log messages instead.

The F5 BIG-IP integration can be used in three different modes to collect data:

• HTTP Endpoint mode - F5 BIG-IP pushes logs directly to an HTTP endpoint hosted by users’ Elastic Agent.
• AWS S3 polling mode - F5 BIG-IP writes data to S3 and Elastic Agent polls the S3 bucket by listing its contents and reading new files.
• AWS S3 SQS mode - F5 BIG-IP writes data to S3, S3 pushes a new object notification to SQS, Elastic Agent receives the notification from SQS, and then reads the S3 object. Multiple Agents can be used in this mode.

For example, users can use the data from this integration to analyze the traffic that passes through their F5 BIG-IP network.

The F5 BIG-IP integration collects one type of data stream: log.

Log help users to keep a record of events happening on the network using telemetry streaming. The log data stream collected by the F5 BIG-IP integration includes events that are related to network traffic. See more details in the logs examples documentation. F5 BIG-IP streams a variety of log types, categorized by the telemetryEventCategory field.

This integration targets the eight types of events as mentioned below:

• LTM provides the platform for creating virtual servers, performance, service, protocol, authentication, and security profiles to define and shape users’ application traffic. For more information, refer to the link here.

• AFM is designed to reduce the hardware and extra hops required when ADC’s are paired with traditional firewalls and helps to protect traffic destined for the user’s data center. For more information, refer to the link here.

• APM provides federation, SSO, application access policies, and secure web tunneling and allows granular access to users' various applications, virtualized desktop environments, or just go full VPN tunnel. For more information, refer to the link here.

• ASM is F5’s web application firewall (WAF) solution. It allows users to tailor acceptable and expected application behavior on a per-application basis. For more information, refer to the link here.

  • BOT and DOS provides details about the BOT events and distributed DOS attacks events related to F5 BIG-IP ASM. For more information on BOT, refer to the knowledge base article here. For more information on DOS, refer to the article here.

• AVR provides detailed charts and graphs to give users more insight into the performance of web applications, with detailed views on HTTP and TCP stats, as well as system performance (CPU, memory, etc.). For more information, refer to the link here.

• System Information provides the system information that F5 BIG-IP Telemetry Streaming collects. For more information, refer to the documentation here.

• iHealth Information highlights vulnerability and importance, potentially leading to memory exhaustion if exploited. It also directs to a solution article for further details and specifies unaffected F5 products. For more information, refer to the documentation here.

Elasticsearch is needed to store and search data, and Kibana is needed for visualizing and managing it. You can use our hosted Elasticsearch Service on Elastic Cloud, which is recommended, or self-manage the Elastic Stack on your hardware.

The reference link for requirements of telemetry streaming is here.

The reference link for requirements of Application Services 3(AS3) Extension is here.

This module has been tested against F5 BIG-IP version 16.1.0, Telemetry Streaming version 1.32.0 and AS3 version 3.40.0.

To set up the F5 BIG-IP environment, users can use the BIG-IP system browser-based Configuration Utility or the command line tools that are provided. For more information related to the configuration of F5 BIG-IP servers, refer to F5 support website here.

For downloading and installing Telemetry Streaming, refer to the link here.

Telemetry Streaming will send logs in the JSON format to the destination. Telemetry Streaming is compatible with BIG-IP versions 13.0 and later. Users have to prepare F5 servers for it and set up the Telemetry Streaming Consumer.

To use telemetry streaming, user have to send POST request on https://<BIG-IP>/mgmt/shared/telemetry/declare for declaration.

Sample declaration to set up Telemetry Streaming Consumer(Generic_HTTP) is as follows:

{
    "class": "Telemetry",
       "My_System": {
        "class": "Telemetry_System",
        "systemPoller": {
            "interval": <INTERVAL>
        }
    },
    "My_Listener": {
        "class": "Telemetry_Listener",
        "port": 6514
    },
    "My_Consumer": {
        "class": "Telemetry_Consumer",
        "type": "Generic_HTTP",
        "host": "<HOST-IP>",
        "protocol": "http",
        "port": <PORT>,
        "path": "/",
        "method": "POST",
        "headers": [
            {
                "name": "content-type",
                "value": "application/json"
            }
        ],
        "outputMode": "processed"
    }
}

For more information related to Generic HTTP consumers, refer to the link here.

To set up TLS client authentication in Generic HTTP consumer, refer to the link here.

Sample declaration to set up Telemetry Streaming Consumer(AWS_S3) is as follows:

{
    "class": "Telemetry",
    "My_System": {
        "class": "Telemetry_System",
        "systemPoller": {
            "interval": <INTERVAL>
        }
    },
    "My_Listener": {
        "class": "Telemetry_Listener",
        "port": 6514
    },
    "My_Consumer": {
        "class": "Telemetry_Consumer",
        "type": "AWS_S3",
        "region": "<AWS_REGION>",
        "bucket": "<BUCKET_NAME>",
        "username": "<ACCESS_KEY_ID>",
        "passphrase": {
            "cipherText": "<SECRET_ACCESS_KEY>"
        }
    }
}

F5 BIG-IP modules named LTM, AFM, ASM, and APM are not configured by Telemetry Streaming, they must be configured with AS3 or another method. Reference link for setup AS3 extension in F5 BIG-IP is here.

To configure logging using AS3, refer to the link here.

• Create an Amazon S3 bucket. Refer to the link here.

• The default value of the "Bucket List Prefix" is listed below. However, the user can set the parameter "Bucket List Prefix" according to the requirement.

  Data Stream Name	Bucket List Prefix
  Log	<Bucket list prefix which has been created in AWS S3>

1. If data forwarding to an AWS S3 Bucket hasn’t been configured, then first set up an AWS S3 Bucket as mentioned in the above documentation.

2. To set up an SQS queue, follow "Step 1: Create an Amazon SQS queue" mentioned in the Documentation.

   • While creating an SQS Queue, please provide the same bucket ARN that has been generated after creating an AWS S3 Bucket.

3. Set up event notifications for an S3 bucket. Follow this link.

   • Users have to set the prefix parameter the same as the S3 Bucket List Prefix as created earlier. (for example, log/ for a log data stream.)
   • Select the event type as s3:ObjectCreated:*, select the destination type SQS Queue, and select the queue that has been created in Step 2.

NOTE:

• Credentials for the above AWS S3 and SQS input types should be configured using the link.
• Data collection via AWS S3 Bucket and AWS SQS are mutually exclusive in this case.

1. In Kibana go to Management > Integrations.
2. In the "Search for integrations" search bar, type F5 BIG-IP.
3. Click on F5 BIG-IP integration from the search results.
4. Click on the Add F5 BIG-IP button to add F5 BIG-IP integration.
5. Enable the Integration to collect logs via AWS S3 or HTTP endpoint input.

This is the log dataset.

{
    "@timestamp": "2018-11-19T22:34:40.000Z",
    "agent": {
        "ephemeral_id": "5783a7ca-031c-49a4-a74c-6bf741bd44a7",
        "id": "7036eed3-e508-4a47-99a4-b144655eb291",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.14.0"
    },
    "client": {
        "ip": "81.2.69.142",
        "port": 49804
    },
    "data_stream": {
        "dataset": "f5_bigip.log",
        "namespace": "45148",
        "type": "logs"
    },
    "destination": {
        "ip": "81.2.69.142",
        "port": 80
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "7036eed3-e508-4a47-99a4-b144655eb291",
        "snapshot": false,
        "version": "8.14.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "network"
        ],
        "dataset": "f5_bigip.log",
        "ingested": "2024-08-06T23:25:39Z",
        "kind": "alert",
        "original": "{\"application\":\"app.app\",\"attack_type\":\"Detection Evasion\",\"blocking_exception_reason\":\"test\",\"captcha_result\":\"not_received\",\"date_time\":\"2018-11-19 22:34:40\",\"dest_ip\":\"81.2.69.142\",\"dest_port\":\"80\",\"device_id\":\"12bdca32\",\"fragment\":\"test_Fragment\",\"geo_location\":\"US\",\"hostname\":\"hostname\",\"http_class_name\":\"/Common/abc/test\",\"ip_address_intelligence\":\"host1\",\"ip_client\":\"81.2.69.142\",\"management_ip_address\":\"81.2.69.142\",\"management_ip_address_2\":\"81.2.69.144\",\"method\":\"GET\",\"policy_apply_date\":\"2018-11-19 22:17:57\",\"policy_name\":\"/Common/abc\",\"protocol\":\"HTTP\",\"query_string\":\"name=abc\",\"request\":\"GET /admin/.\",\"request_status\":\"blocked\",\"response_code\":\"0\",\"route_domain\":\"example.com\",\"session_id\":\"abc123abcd\",\"severity\":\"Critical\",\"sig_ids\":\"abc12bcd\",\"sig_names\":\"Sig_Name\",\"src_port\":\"49804\",\"staged_sig_ids\":\"abc23121bc\",\"staged_sig_names\":\"test_name\",\"staged_threat_campaign_names\":\"test\",\"sub_violations\":\"Evasion technique detected:Directory traversals\",\"support_id\":\"123456789\",\"telemetryEventCategory\":\"ASM\",\"tenant\":\"Common\",\"threat_campaign_names\":\"threat\",\"uri\":\"/directory/file\",\"username\":\"test User\",\"violation_rating\":\"3\",\"violations\":\"Evasion technique detected\",\"virus_name\":\"test Virus\",\"web_application_name\":\"/Common/abc\",\"websocket_direction\":\"test\",\"websocket_message_type\":\"test\",\"x_forwarded_for_header_value\":\"81.2.69.144\"}",
        "type": [
            "info"
        ]
    },
    "f5_bigip": {
        "log": {
            "application": {
                "name": "app.app"
            },
            "attack": {
                "type": "Detection Evasion"
            },
            "blocking_exception_reason": "test",
            "captcha_result": "not_received",
            "client": {
                "ip": "81.2.69.142"
            },
            "date_time": "2018-11-19T22:34:40.000Z",
            "dest": {
                "ip": "81.2.69.142",
                "port": 80
            },
            "device": {
                "id": "12bdca32"
            },
            "fragment": "test_Fragment",
            "geo": {
                "location": "US"
            },
            "hostname": "hostname",
            "http": {
                "class_name": "/Common/abc/test"
            },
            "ip_address_intelligence": "host1",
            "management": {
                "ip_address": "81.2.69.142",
                "ip_address_2": "81.2.69.144"
            },
            "method": "GET",
            "policy": {
                "apply_date": "2018-11-19T22:17:57.000Z",
                "name": "/Common/abc"
            },
            "protocol": "HTTP",
            "query": {
                "string": "name=abc"
            },
            "request": {
                "detail": "GET /admin/.",
                "status": "blocked"
            },
            "response": {
                "code": 0
            },
            "route_domain": "example.com",
            "session": {
                "id": "abc123abcd"
            },
            "severity": {
                "name": "Critical"
            },
            "sig": {
                "ids": "abc12bcd",
                "names": "Sig_Name"
            },
            "src": {
                "port": 49804
            },
            "staged": {
                "sig": {
                    "ids": "abc23121bc",
                    "names": "test_name"
                },
                "threat_campaign_names": "test"
            },
            "sub_violations": "Evasion technique detected:Directory traversals",
            "support": {
                "id": "123456789"
            },
            "telemetry": {
                "event": {
                    "category": "ASM"
                }
            },
            "tenant": "Common",
            "threat_campaign_names": "threat",
            "uri": "/directory/file",
            "username": "test User",
            "violation": {
                "rating": 3
            },
            "violations": "Evasion technique detected",
            "virus_name": "test Virus",
            "web_application_name": "/Common/abc",
            "websocket": {
                "direction": "test",
                "message_type": "test"
            },
            "x_forwarded_for_header_value": [
                "81.2.69.144"
            ]
        }
    },
    "host": {
        "geo": {
            "country_iso_code": "US"
        },
        "id": "12bdca32",
        "name": "hostname"
    },
    "http": {
        "request": {
            "method": "GET"
        }
    },
    "input": {
        "type": "http_endpoint"
    },
    "log": {
        "level": "critical"
    },
    "network": {
        "application": "app.app",
        "protocol": "http"
    },
    "observer": {
        "product": "Application Security Module",
        "vendor": "F5"
    },
    "related": {
        "hosts": [
            "hostname",
            "12bdca32"
        ],
        "ip": [
            "81.2.69.142",
            "81.2.69.144"
        ],
        "user": [
            "test User"
        ]
    },
    "server": {
        "ip": "81.2.69.142",
        "port": 80
    },
    "source": {
        "ip": "81.2.69.142",
        "port": 49804
    },
    "tags": [
        "preserve_original_event",
        "preserve_duplicate_custom_fields",
        "forwarded",
        "f5_bigip-log"
    ],
    "user": {
        "name": "test User"
    }
}