Recorded Future Integration
<div class="condensed-table">
| | |
| --- | --- |
| Version | 1.28.0 (View all) |
| Compatible Kibana version(s) | 8.13.0 or higher |
| Supported Serverless project types
What’s this? | Security
Observability |
| Subscription level
What’s this? | Basic |
| Level of support
What’s this? | Elastic |
</div>
The Recorded Future integration fetches risklists from the Recorded Future API. It supports domain
, hash
, ip
and url
entities.
In order to use it you need to define the entity
and list
to fetch. Check with Recorded Future for the available lists for each entity. To fetch indicators from multiple entities, it’s necessary to define one integration for each.
Alternatively, it’s also possible to use the integration to fetch custom Fusion files by supplying the URL to the CSV file as the Custom URL configuration option.
The ingested IOCs expire after certain duration. An Elastic Transform is created to faciliate only active IOCs be available to the end users. This transform creates a destination index named logs-ti_recordedfuture_latest.threat-1
which only contains active and unexpired IOCs. The destination index also has an alias logs-ti_recordedfuture_latest.threat
. When setting up indicator match rules, use this latest destination index to avoid false positives from expired IOCs. Please read ILM Policy below which is added to avoid unbounded growth on source .ds-logs-ti_recordedfuture.threat-*
indices.
To facilitate IOC expiration, source datastream-backed indices .ds-logs-ti_recordedfuture.threat-*
are allowed to contain duplicates from each polling interval. ILM policy is added to these source indices so it doesn’t lead to unbounded growth. This means data in these source indices will be deleted after 5 days
from ingested date.
For large risklist downloads, adjust the timeout setting so that the Agent has enough time to download and process the risklist.
**Example**
An example event for threat
looks as following:
{
"@timestamp": "2024-08-02T06:24:04.201Z",
"agent": {
"ephemeral_id": "25d7a936-2b7c-4476-9181-82d1296ce9df",
"id": "8299ae35-ee0e-4107-9acb-1b6acfdda1fb",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.13.0"
},
"data_stream": {
"dataset": "ti_recordedfuture.threat",
"namespace": "67234",
"type": "logs"
},
"ecs": {
"version": "8.11.0"
},
"elastic_agent": {
"id": "8299ae35-ee0e-4107-9acb-1b6acfdda1fb",
"snapshot": false,
"version": "8.13.0"
},
"event": {
"agent_id_status": "verified",
"category": [
"threat"
],
"created": "2024-08-02T06:24:04.201Z",
"dataset": "ti_recordedfuture.threat",
"ingested": "2024-08-02T06:24:16Z",
"kind": "enrichment",
"original": "{\"EvidenceDetails\":\"{\\\"EvidenceDetails\\\": [{\\\"Name\\\": \\\"suspectedCncDnsName\\\", \\\"EvidenceString\\\": \\\"1 sighting on 1 source: ThreatFox Infrastructure Analysis. ThreatFox identified ubykou33.top as possible TA0011 (Command and Control) for CryptBot on December 26, 2023. Most recent link (Dec 26, 2023): https://threatfox.abuse.ch/ioc/1223634\\\", \\\"CriticalityLabel\\\": \\\"Unusual\\\", \\\"MitigationString\\\": \\\"\\\", \\\"Rule\\\": \\\"Historical Suspected C\\\\\u0026C DNS Name\\\", \\\"SourcesCount\\\": 1.0, \\\"Sources\\\": [\\\"source:sIoEOQ\\\"], \\\"Timestamp\\\": \\\"2023-12-26T17:06:29.000Z\\\", \\\"SightingsCount\\\": 1.0, \\\"Criticality\\\": 1.0}, {\\\"Name\\\": \\\"malwareSiteDetected\\\", \\\"EvidenceString\\\": \\\"2 sightings on 2 sources: External Sensor Data Analysis, Bitdefender. ubykou33.top is observed to be a malware site domain that navigates to malicious content including executables, drive-by infection sites, malicious scripts, viruses, trojans, or code.\\\", \\\"CriticalityLabel\\\": \\\"Unusual\\\", \\\"MitigationString\\\": \\\"\\\", \\\"Rule\\\": \\\"Historically Detected Malware Operation\\\", \\\"SourcesCount\\\": 2.0, \\\"Sources\\\": [\\\"source:kBB1fk\\\", \\\"source:d3Awkm\\\"], \\\"Timestamp\\\": \\\"2024-01-26T00:00:00.000Z\\\", \\\"SightingsCount\\\": 2.0, \\\"Criticality\\\": 1.0}, {\\\"Name\\\": \\\"malwareSiteSuspected\\\", \\\"EvidenceString\\\": \\\"1 sighting on 1 source: Bitdefender. Detected malicious behavior from an endpoint agent via global telemetry. Last observed on Jan 26, 2024.\\\", \\\"CriticalityLabel\\\": \\\"Unusual\\\", \\\"MitigationString\\\": \\\"\\\", \\\"Rule\\\": \\\"Historically Suspected Malware Operation\\\", \\\"SourcesCount\\\": 1.0, \\\"Sources\\\": [\\\"source:d3Awkm\\\"], \\\"Timestamp\\\": \\\"2024-01-26T00:00:00.000Z\\\", \\\"SightingsCount\\\": 1.0, \\\"Criticality\\\": 1.0}, {\\\"Name\\\": \\\"recentMalwareSiteDetected\\\", \\\"EvidenceString\\\": \\\"1 sighting on 1 source: External Sensor Data Analysis. ubykou33.top is observed to be a malware site domain that navigates to malicious content including executables, drive-by infection sites, malicious scripts, viruses, trojans, or code.\\\", \\\"CriticalityLabel\\\": \\\"Malicious\\\", \\\"MitigationString\\\": \\\"\\\", \\\"Rule\\\": \\\"Recently Detected Malware Operation\\\", \\\"SourcesCount\\\": 1.0, \\\"Sources\\\": [\\\"source:kBB1fk\\\"], \\\"Timestamp\\\": \\\"2024-05-08T23:11:43.601Z\\\", \\\"SightingsCount\\\": 1.0, \\\"Criticality\\\": 3.0}]}\",\"Name\":\"ubykou33.top\",\"Risk\":\"67\",\"RiskString\":\"4/52\"}",
"risk_score": 67,
"type": [
"indicator"
]
},
"input": {
"type": "httpjson"
},
"recordedfuture": {
"evidence_details": [
{
"criticality": 1,
"criticality_label": "Unusual",
"evidence_string": "1 sighting on 1 source: ThreatFox Infrastructure Analysis. ThreatFox identified ubykou33.top as possible TA0011 (Command and Control) for CryptBot on December 26, 2023. Most recent link (Dec 26, 2023): https://threatfox.abuse.ch/ioc/1223634",
"mitigation_string": "",
"name": "suspectedCncDnsName",
"rule": "Historical Suspected C&C DNS Name",
"sightings_count": 1,
"sources": [
"source:sIoEOQ"
],
"sources_count": 1,
"timestamp": "2023-12-26T17:06:29.000Z"
},
{
"criticality": 1,
"criticality_label": "Unusual",
"evidence_string": "2 sightings on 2 sources: External Sensor Data Analysis, Bitdefender. ubykou33.top is observed to be a malware site domain that navigates to malicious content including executables, drive-by infection sites, malicious scripts, viruses, trojans, or code.",
"mitigation_string": "",
"name": "malwareSiteDetected",
"rule": "Historically Detected Malware Operation",
"sightings_count": 2,
"sources": [
"source:kBB1fk",
"source:d3Awkm"
],
"sources_count": 2,
"timestamp": "2024-01-26T00:00:00.000Z"
},
{
"criticality": 1,
"criticality_label": "Unusual",
"evidence_string": "1 sighting on 1 source: Bitdefender. Detected malicious behavior from an endpoint agent via global telemetry. Last observed on Jan 26, 2024.",
"mitigation_string": "",
"name": "malwareSiteSuspected",
"rule": "Historically Suspected Malware Operation",
"sightings_count": 1,
"sources": [
"source:d3Awkm"
],
"sources_count": 1,
"timestamp": "2024-01-26T00:00:00.000Z"
},
{
"criticality": 3,
"criticality_label": "Malicious",
"evidence_string": "1 sighting on 1 source: External Sensor Data Analysis. ubykou33.top is observed to be a malware site domain that navigates to malicious content including executables, drive-by infection sites, malicious scripts, viruses, trojans, or code.",
"mitigation_string": "",
"name": "recentMalwareSiteDetected",
"rule": "Recently Detected Malware Operation",
"sightings_count": 1,
"sources": [
"source:kBB1fk"
],
"sources_count": 1,
"timestamp": "2024-05-08T23:11:43.601Z"
}
],
"list": "test",
"name": "ubykou33.top",
"risk_string": "4/52"
},
"tags": [
"preserve_original_event",
"forwarded",
"recordedfuture"
],
"threat": {
"feed": {
"name": "Recorded Future"
},
"indicator": {
"provider": [
"ThreatFox Infrastructure Analysis",
"External Sensor Data Analysis",
"Bitdefender"
],
"scanner_stats": 5,
"sightings": 5,
"type": "domain-name",
"url": {
"domain": "ubykou33.top"
}
}
}
}
**Exported fields**
Field | Description | Type |
---|---|---|
@timestamp | Event timestamp. | date |
cloud.image.id | Image ID for the cloud instance. | keyword |
data_stream.dataset | Data stream dataset name. | constant_keyword |
data_stream.namespace | Data stream namespace. | constant_keyword |
data_stream.type | Data stream type. | constant_keyword |
event.dataset | Event dataset | constant_keyword |
event.module | Event module | constant_keyword |
host.containerized | If the host is a container. | boolean |
host.os.build | OS build information. | keyword |
host.os.codename | OS codename, if any. | keyword |
input.type | Type of Filebeat input. | keyword |
labels.is_ioc_transform_source | Indicates whether an IOC is in the raw source data stream, or the in latest destination index. | constant_keyword |
log.flags | Flags for the log file. | keyword |
log.offset | Offset of the entry in the log file. | long |
recordedfuture.evidence_details.criticality | double | |
recordedfuture.evidence_details.criticality_label | keyword | |
recordedfuture.evidence_details.evidence_string | keyword | |
recordedfuture.evidence_details.mitigation_string | keyword | |
recordedfuture.evidence_details.name | keyword | |
recordedfuture.evidence_details.rule | keyword | |
recordedfuture.evidence_details.sightings_count | integer | |
recordedfuture.evidence_details.sources | keyword | |
recordedfuture.evidence_details.sources_count | integer | |
recordedfuture.evidence_details.timestamp | date | |
recordedfuture.list | User-configured risklist. | keyword |
recordedfuture.name | Indicator value. | keyword |
recordedfuture.risk_string | Details of risk rules observed. | keyword |
threat.feed.name | Display friendly feed name | constant_keyword |
threat.indicator.first_seen | The date and time when intelligence source first reported sighting this indicator. | date |
threat.indicator.last_seen | The date and time when intelligence source last reported sighting this indicator. | date |
threat.indicator.modified_at | The date and time when intelligence source last modified information for this indicator. | date |
**Changelog**
Version | Details | Kibana version(s) |
---|---|---|
1.28.0 | pass:[] Enhancement (View pull request) Do not remove event.original in main ingest pipeline. |
8.13.0 or higher |
1.27.0 | pass:[] Enhancement (View pull request) Add "preserve_original_event" tag to documents with event.kind set to "pipeline_error". |
8.13.0 or higher |
1.26.3 | pass:[] Bug fix (View pull request) Fix labels.is_ioc_transform_source values |
8.13.0 or higher |
1.26.2 | pass:[] Bug fix (View pull request) Add missing fields in transform |
8.13.0 or higher |
1.26.1 | pass:[] Bug fix (View pull request) Fix ECS date mapping on threat fields. |
8.13.0 or higher |
1.26.0 | pass:[] Enhancement (View pull request) Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template. |
8.13.0 or higher |
1.25.1 | pass:[] Bug fix (View pull request) Adjust field mappings for transform destination index. |
8.12.0 or higher |
1.25.0 | pass:[] Enhancement (View pull request) Decode Evidence_Details field. This is a breaking change since the mapping is changed. |
8.12.0 or higher |
1.24.0 | pass:[] Enhancement (View pull request) Add destination index alias and fix docs. |
8.12.0 or higher |
1.23.0 | pass:[] Enhancement (View pull request) Add dashboards and list field |
8.12.0 or higher |
1.22.0 | pass:[] Enhancement (View pull request) Set sensitive values as secret. |
8.12.0 or higher |
1.21.0 | pass:[] Enhancement (View pull request) Make threat.indicator.url.full available for rule detections. |
8.8.0 or higher |
1.20.2 | pass:[] Enhancement (View pull request) Changed owners |
8.8.0 or higher |
1.20.1 | pass:[] Bug fix (View pull request) Fix exclude_files pattern. |
8.8.0 or higher |
1.20.0 | pass:[] Enhancement (View pull request) Limit request tracer log count to five. |
8.8.0 or higher |
1.19.0 | pass:[] Enhancement (View pull request) ECS version updated to 8.11.0. |
8.8.0 or higher |
1.18.1 | pass:[] Bug fix (View pull request) Fix the parse of providers information. |
8.8.0 or higher |
1.18.0 | pass:[] Enhancement (View pull request) Improve event.original check to avoid errors if set. |
8.8.0 or higher |
1.17.0 | pass:[] Enhancement (View pull request) ECS version updated to 8.10.0. |
8.8.0 or higher |
1.16.0 | pass:[] Enhancement (View pull request) Add DLM policy. Add owner.type to package manifest. Update format_version to 3.0.0 pass:[] Enhancement (View pull request) Add tags.yml file so that integration’s dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI. |
8.8.0 or higher |
1.15.1 | pass:[] Bug fix (View pull request) Replace dotted YAML keys. |
— |
1.15.0 | pass:[] Enhancement (View pull request) Update package-spec to 2.10.0. |
8.8.0 or higher |
1.14.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.9.0. |
8.8.0 or higher |
1.13.0 | pass:[] Enhancement (View pull request) Document duration units. |
8.8.0 or higher |
1.12.0 | pass:[] Enhancement (View pull request) Ensure event.kind is correctly set for pipeline errors. |
8.8.0 or higher |
1.11.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.8.0. |
8.8.0 or higher |
1.10.0 | pass:[] Enhancement (View pull request) Add IOC field to transform source to easily filter destination indices with unexpired indicators |
8.8.0 or higher |
1.9.0 | pass:[] Enhancement (View pull request) Support for IoC Expiration |
8.8.0 or higher |
1.8.0 | pass:[] Enhancement (View pull request) Add a new flag to enable request tracing |
8.7.1 or higher |
1.7.0 | pass:[] Enhancement (View pull request) Scrape provider details from evidence field. |
8.0.0 or higher |
1.6.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.7.0. |
8.0.0 or higher |
1.5.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.6.0. |
8.0.0 or higher |
1.4.1 | pass:[] Bug fix (View pull request) Use ECS definition for threat.indicator.geo.location . |
8.0.0 or higher |
1.4.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.5.0. |
— |
1.3.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.4.0 |
8.0.0 or higher |
1.2.1 | pass:[] Bug fix (View pull request) Expose request timeout setting and increase it to 5m. pass:[] Bug fix (View pull request) Do not fail on invalid URLs. |
8.0.0 or higher |
1.2.0 | pass:[] Enhancement (View pull request) Update categories to include threat_intel . |
8.0.0 or higher |
1.1.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.3.0. |
8.0.0 or higher |
1.0.1 | pass:[] Enhancement (View pull request) update readme added link to recorded future API documentation |
8.0.0 or higher |
1.0.0 | pass:[] Enhancement (View pull request) Make GA |
8.0.0 or higher |
0.1.3 | pass:[] Enhancement (View pull request) Update package descriptions |
— |
0.1.2 | pass:[] Enhancement (View pull request) Add field mapping for event.created |
— |
0.1.1 | pass:[] Enhancement (View pull request) Add documentation for multi-fields |
— |
0.1.0 | pass:[] Enhancement (View pull request) Initial release |
— |