Collective Intelligence Framework v3 Integration
<div class="condensed-table">
| | |
| --- | --- |
| Version | 1.16.0 (View all) |
| Compatible Kibana version(s) | 8.13.0 or higher |
| Supported Serverless project types
What’s this? | Security
Observability |
| Subscription level
What’s this? | Basic |
| Level of support
What’s this? | Community |
</div>
This integration connects with the REST API from the running CIFv3 instance to retrieve indicators.
Indicators are expired after a certain duration. An Elastic Transform is created for a source index to allow only active indicators to be available to the end users. The transform creates a destination index named logs-ti_cif3_latest.dest_feed* which only contains active and unexpired indicators. Destination indices are aliased to logs-ti_cif3_latest.feed. The indicator match rules and dashboards are updated to show only active indicators.
| Indicator Type | Indicator Expiration Duration |
|---|---|
ipv4-addr |
45d |
ipv6-addr |
45d |
domain-name |
90d |
url |
365d |
file |
365d |
| All Other Types | Derived from IOC Expiration Duration setting |
To facilitate IOC expiration, source datastream-backed indices .ds-logs-ti_cif3.feed-* are allowed to contain duplicates. ILM policy logs-ti_cif3.feed-default_policy is added to these source indices so it doesn’t lead to unbounded growth. This means data in these source indices will be deleted after 5 days from ingested date.
The CIFv3 integration collects threat indicators based on user-defined configuration including a polling interval, how far back in time it should look, and other filters like indicator type and tags.
CIFv3 confidence field values (0..10) are converted to ECS confidence (None, Low, Medium, High) in the following way:
| CIFv3 Confidence | ECS Conversion |
|---|---|
| Beyond Range | None |
| 0 - <3 | Low |
| 3 - <7 | Medium |
| 7 - 10 | High |
**Exported fields**
| Field | Description | Type |
|---|---|---|
| @timestamp | Event timestamp. | date |
| cif3.application | The application used by the indicator, such as telnet or ssh. | keyword |
| cif3.asn | AS Number of IP. | integer |
| cif3.asn_desc | AS Number org name. | keyword |
| cif3.cc | Country code of GeoIP. | keyword |
| cif3.city | GeoIP city information. | keyword |
| cif3.confidence | The confidence on a scale of 0-10 that the tags appropriately contextualize the indicator. | float |
| cif3.count | The number of times the same indicator has been reported with the same metadata by the same provider. | integer |
| cif3.deleted_at | The indicator expiration timestamp. | date |
| cif3.description | A description of the indicator. | keyword |
| cif3.expiration_duration | The configured expiration duration. | keyword |
| cif3.indicator | The value of the indicator, for example if the type is fqdn, this would be the value. | keyword |
| cif3.indicator_iprange | IPv4 or IPv6 IP Range. | ip_range |
| cif3.indicator_ipv4 | IPv4 address. | ip |
| cif3.indicator_ipv4_mask | subnet mask of IPv4 CIDR. | integer |
| cif3.indicator_ipv6 | singleton IPv6 address. | keyword |
| cif3.indicator_ipv6_mask | subnet mask of IPv6 CIDR. | integer |
| cif3.indicator_ssdeep_chunk | SSDEEP hash chunk. | text |
| cif3.indicator_ssdeep_chunksize | SSDEEP hash chunk size. | integer |
| cif3.indicator_ssdeep_double_chunk | SSDEEP hash double chunk. | text |
| cif3.itype | The indicator type, can for example be "ipv4, fqdn, email, url, sha256". | keyword |
| cif3.latitude | Latitude of GeoIP. | keyword |
| cif3.location | Lat/Long of GeoIP. | geo_point |
| cif3.longitude | Longitude of GeoIP. | keyword |
| cif3.portlist | The port or range of ports used by the indicator. | text |
| cif3.protocol | The protocol used by the indicator. | text |
| cif3.provider | The source of the indicator information. | keyword |
| cif3.rdata | Extra text or descriptive content related to the indicator such as OS, reverse lookup, etc. | keyword |
| cif3.reference | A reference URL with further info related to the indicator. | keyword |
| cif3.region | GeoIP region information. | keyword |
| cif3.tags | Comma-separated list of words describing the indicator such as "malware,exploit". | keyword |
| cif3.timezone | Timezone of GeoIP. | text |
| cif3.uuid | The ID of the indicator. | keyword |
| data_stream.dataset | Data stream dataset. | constant_keyword |
| data_stream.namespace | Data stream namespace. | constant_keyword |
| data_stream.type | Data stream type. | constant_keyword |
| event.dataset | Event dataset | constant_keyword |
| event.module | Name of the module this data is coming from. | constant_keyword |
| input.type | Type of Filebeat input. | keyword |
| labels.is_ioc_transform_source | Indicates whether an IOC is in the raw source data stream, or the in latest destination index. | constant_keyword |
| log.flags | Flags for the log file. | keyword |
| log.offset | Offset of the entry in the log file. | long |
| threat.feed.name | Display friendly feed name | constant_keyword |
| threat.indicator.first_seen | The date and time when intelligence source first reported sighting this indicator. | date |
| threat.indicator.last_seen | The date and time when intelligence source last reported sighting this indicator. | date |
| threat.indicator.modified_at | The date and time when intelligence source last modified information for this indicator. | date |
| threat.indicator.tls.client.ja3 | An md5 hash that identifies clients based on their TLS handshake. | keyword |
**Example**
An example event for feed looks as following:
{
"@timestamp": "2024-08-01T08:05:14.040Z",
"agent": {
"ephemeral_id": "b351d699-2fd0-49f7-99e1-a7a471a29a62",
"id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.13.0"
},
"cif3": {
"deleted_at": "2022-09-03T20:25:53.000Z",
"expiration_duration": "45d",
"indicator": "20.206.75.106",
"itype": "ipv4",
"portlist": "443",
"uuid": "ac240898-1443-4d7e-a98a-1daed220c162"
},
"data_stream": {
"dataset": "ti_cif3.feed",
"namespace": "26457",
"type": "logs"
},
"ecs": {
"version": "8.11.0"
},
"elastic_agent": {
"id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
"snapshot": false,
"version": "8.13.0"
},
"event": {
"agent_id_status": "verified",
"category": [
"threat"
],
"created": "2024-08-01T08:05:14.040Z",
"dataset": "ti_cif3.feed",
"ingested": "2024-08-01T08:05:26Z",
"kind": "enrichment",
"original": "{\"application\":\"https\",\"asn\":8075,\"asn_desc\":\"microsoft-corp-msn-as-block\",\"cc\":\"br\",\"city\":\"campinas\",\"confidence\":10,\"count\":1,\"firsttime\":\"2022-07-20T20:25:53.000000Z\",\"group\":[\"everyone\"],\"indicator\":\"20.206.75.106\",\"indicator_ipv4\":\"20.206.75.106\",\"itype\":\"ipv4\",\"lasttime\":\"2022-07-20T20:25:53.000000Z\",\"latitude\":-22.9035,\"location\":[-47.0565,-22.9035],\"longitude\":-47.0565,\"portlist\":\"443\",\"protocol\":\"tcp\",\"provider\":\"sslbl.abuse.ch\",\"reference\":\"https://sslbl.abuse.ch/blacklist/sslipblacklist.csv\",\"region\":\"sao paulo\",\"reporttime\":\"2022-07-21T20:33:26.585967Z\",\"tags\":[\"botnet\"],\"timezone\":\"america/sao_paulo\",\"tlp\":\"white\",\"uuid\":\"ac240898-1443-4d7e-a98a-1daed220c162\"}",
"type": [
"indicator"
]
},
"input": {
"type": "httpjson"
},
"network": {
"protocol": "https",
"transport": "tcp"
},
"related": {
"ip": [
"20.206.75.106"
]
},
"tags": [
"preserve_original_event",
"forwarded",
"cif3-indicator",
"botnet"
],
"threat": {
"indicator": {
"as": {
"number": 8075,
"organization": {
"name": "microsoft-corp-msn-as-block"
}
},
"confidence": "High",
"first_seen": "2022-07-20T20:25:53.000Z",
"geo": {
"country_iso_code": "br",
"location": {
"lat": -22.9035,
"lon": -47.0565
},
"region_name": "sao paulo",
"timezone": "america/sao_paulo"
},
"ip": "20.206.75.106",
"last_seen": "2022-07-20T20:25:53.000Z",
"marking": {
"tlp": "WHITE"
},
"modified_at": "2022-07-21T20:33:26.585967Z",
"name": "20.206.75.106",
"provider": "sslbl.abuse.ch",
"reference": "https://sslbl.abuse.ch/blacklist/sslipblacklist.csv",
"sightings": 1,
"type": "ipv4-addr"
}
}
}
**Changelog**
| Version | Details | Kibana version(s) |
|---|---|---|
| 1.16.0 | pass:[] Enhancement (View pull request) Do not remove event.original in main ingest pipeline. |
8.13.0 or higher |
| 1.15.0 | pass:[] Enhancement (View pull request) Add "preserve_original_event" tag to documents with event.kind set to "pipeline_error". |
8.13.0 or higher |
| 1.14.4 | pass:[] Bug fix (View pull request) Use triple-brace Mustache templating when referencing variables in ingest pipelines. |
8.13.0 or higher |
| 1.14.3 | pass:[] Bug fix (View pull request) Fix labels.is_ioc_transform_source values |
8.13.0 or higher |
| 1.14.2 | pass:[] Bug fix (View pull request) Add missing fields in transform |
8.13.0 or higher |
| 1.14.1 | pass:[] Bug fix (View pull request) Fix ECS date mapping on threat fields. |
8.13.0 or higher |
| 1.14.0 | pass:[] Enhancement (View pull request) Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template. |
8.13.0 or higher |
| 1.13.1 | pass:[] Bug fix (View pull request) Adjust field mappings for transform destination index. |
8.12.0 or higher |
| 1.13.0 | pass:[] Enhancement (View pull request) Improve handling of empty responses. |
8.12.0 or higher |
| 1.12.0 | pass:[] Enhancement (View pull request) Support for IOC expiration |
8.12.0 or higher |
| 1.11.0 | pass:[] Enhancement (View pull request) Set sensitive values as secret. |
8.12.0 or higher |
| 1.10.2 | pass:[] Bug fix (View pull request) Clean up null handling |
8.7.1 or higher |
| 1.10.1 | pass:[] Enhancement (View pull request) Changed owners |
8.7.1 or higher |
| 1.10.0 | pass:[] Enhancement (View pull request) Limit request tracer log count to five. |
8.7.1 or higher |
| 1.9.0 | pass:[] Enhancement (View pull request) ECS version updated to 8.11.0. |
8.7.1 or higher |
| 1.8.0 | pass:[] Enhancement (View pull request) Set community owner type. |
8.7.1 or higher |
| 1.7.0 | pass:[] Enhancement (View pull request) ECS version updated to 8.10.0. |
8.7.1 or higher |
| 1.6.0 | pass:[] Enhancement (View pull request) The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added owner.type: elastic to package manifest. |
8.7.1 or higher |
| 1.5.0 | pass:[] Enhancement (View pull request) Add tags.yml file so that integration’s dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI. |
8.7.1 or higher |
| 1.4.0 | pass:[] Enhancement (View pull request) Update package-spec to 2.9.0. |
8.7.1 or higher |
| 1.3.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.9.0. |
8.7.1 or higher |
| 1.2.0 | pass:[] Enhancement (View pull request) Document duration units. |
8.7.1 or higher |
| 1.1.0 | pass:[] Enhancement (View pull request) Document valid duration units. |
8.7.1 or higher |
| 1.0.0 | pass:[] Enhancement (View pull request) Release Collective Intelligence Framework as GA. |
8.7.1 or higher |
| 0.8.0 | pass:[] Enhancement (View pull request) Ensure event.kind is correctly set for pipeline errors. |
— |
| 0.7.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.8.0. |
— |
| 0.6.0 | pass:[] Enhancement (View pull request) Add a new flag to enable request tracing |
— |
| 0.5.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.7.0. |
— |
| 0.4.1 | pass:[] Enhancement (View pull request) Honor preserve_original_event setting. |
— |
| 0.4.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.6.0. |
— |
| 0.3.1 | pass:[] Bug fix (View pull request) Use ECS definition for threat.indicator.geo.location. |
— |
| 0.3.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.5.0. |
— |
| 0.2.2 | pass:[] Bug fix (View pull request) Remove duplicate field. |
— |
| 0.2.1 | pass:[] Enhancement (View pull request) Fix documentation build error |
— |
| 0.2.0 | pass:[] Enhancement (View pull request) Labelling with Threat Intelligence category |
— |
| 0.1.0 | pass:[] Enhancement (View pull request) Initial draft of the package |
— |