Anomali Integration
<div class="condensed-table">
| | |
| --- | --- |
| Version | 1.25.0 (View all) |
| Compatible Kibana version(s) | 8.13.0 or higher |
| Supported Serverless project types
What’s this? | Security
Observability |
| Subscription level
What’s this? | Basic |
| Level of support
What’s this? | Elastic |
</div>
The Anomali integration can fetch indicators from Anomali ThreatStream, a commercial Threat Intelligence service.
It has the following data streams:
intelligenceIndicators retrieved from the Anomali ThreatStream API’s intelligence endpoint.threatstreamIndicators received from the Anomali ThreatStream Elastic Extension, which is additional software. This is deprecated.
An Elastic Transform is created to provide a view of active indicators for end users. The transform creates destination indices that are accessible via the alias of the form logs-ti_anomali_latest.<datastreamname>. When querying for active indicators or setting up indicator match rules, use the alias to avoid false positives from expired indicators. The dashboards show only the latest indicators.
Indicator data from Anomali can contain information about deletion or expiry times. However, some Anomali IOCs may never expire and will continue to stay in the latest destination index. To avoid any false positives from such orphaned IOCs, users are allowed to configure an "IOC Expiration Duration" or "IOC Duration Before Deletion" parameter while setting up a policy. The value set there will limit the time that indicators are retained before deletion, but indicators may be removed earlier based on information from Anomali.
The destination indices created by the transform are versioned with an integer suffix such as -1, -2, for example, logs-ti_anomali_latest.intelligence-1.
Due to schema changes in the destination index, its version number may be incremented.
When this happens, the transform does not have the functionality to auto-delete the old index, so users must delete this old index manually. This is to ensure that duplicates are not present when using wildcard queries such as logs-ti_anomali_latest.intelligence-*. To delete an old index, follow the steps below (either for intelligence as below, or for the older threatstream equivalents):
- After upgrading the integration to the latest version, check the current transform’s destination index version by navigating to:
Stack Management -> Transforms -> logs-ti_anomali.latest_intelligence-default -> Details. Check thedestination_indexvalue. - Run
GET _cat/indices?vand check if any older versions exist. Such aslogs-ti_anomali_latest.intelligence-1 - Run
DELETE logs-ti_anomali_latest.intelligence-<OLDVERSION>to delete the old index.
To prevent unbounded growth of the source data streams logs-ti_opencti.<datastreamname>-*, index lifecycle management (ILM) policies will deletes records 5 days after ingestion.
The Anomali ThreatStream API’s intelligence endpoint is the preferred source of indicators. This data will be be accessible using the alias logs-ti_anomali_latest.intelligence.
**Example**
An example event for intelligence looks as following:
{
"@timestamp": "2024-10-02T16:04:31.789615115Z",
"agent": {
"ephemeral_id": "bfe2b7b4-003a-49a1-b51b-e41ece98943b",
"id": "72cb3ab8-2baa-4ae5-9a62-ee752a56df42",
"name": "elastic-agent-85520",
"type": "filebeat",
"version": "8.14.3"
},
"anomali": {
"threatstream": {
"can_add_public_tags": true,
"confidence": 60,
"deletion_scheduled_at": "2024-10-09T16:04:31.789615115Z",
"expiration_ts": "9999-12-31T00:00:00.000Z",
"feed_id": 0,
"id": "232020126",
"is_anonymous": false,
"is_editable": false,
"is_public": true,
"itype": "apt_domain",
"meta": {
"severity": "very-high"
},
"owner_organization_id": 67,
"retina_confidence": -1,
"source_reported_confidence": 60,
"status": "active",
"threat_type": "apt",
"type": "domain",
"update_id": 100000001,
"uuid": "0921be47-9cc2-4265-b896-c62a7cb91042",
"value": "gen1xyz.com"
}
},
"data_stream": {
"dataset": "ti_anomali.intelligence",
"namespace": "49937",
"type": "logs"
},
"ecs": {
"version": "8.11.0"
},
"elastic_agent": {
"id": "72cb3ab8-2baa-4ae5-9a62-ee752a56df42",
"snapshot": false,
"version": "8.14.3"
},
"event": {
"agent_id_status": "verified",
"category": [
"threat"
],
"created": "2021-04-06T09:56:22.915Z",
"dataset": "ti_anomali.intelligence",
"ingested": "2024-10-02T16:04:31Z",
"kind": "enrichment",
"original": "{\"asn\":\"\",\"can_add_public_tags\":true,\"confidence\":60,\"created_by\":null,\"created_ts\":\"2021-04-06T09:56:22.915Z\",\"description\":null,\"expiration_ts\":\"9999-12-31T00:00:00.000Z\",\"feed_id\":0,\"id\":232020126,\"is_anonymous\":false,\"is_editable\":false,\"is_public\":true,\"itype\":\"apt_domain\",\"locations\":[],\"meta\":{\"detail2\":\"imported by user 136\",\"severity\":\"very-high\"},\"modified_ts\":\"2021-04-06T09:56:22.915Z\",\"org\":\"\",\"owner_organization_id\":67,\"rdns\":null,\"resource_uri\":\"/api/v2/intelligence/232020126/\",\"retina_confidence\":-1,\"sort\":[455403032],\"source\":\"Analyst\",\"source_locations\":[],\"source_reported_confidence\":60,\"status\":\"active\",\"subtype\":null,\"tags\":null,\"target_industry\":[],\"threat_type\":\"apt\",\"threatscore\":54,\"tlp\":null,\"trusted_circle_ids\":null,\"type\":\"domain\",\"update_id\":100000001,\"uuid\":\"0921be47-9cc2-4265-b896-c62a7cb91042\",\"value\":\"gen1xyz.com\",\"workgroups\":[]}",
"severity": 9,
"type": [
"indicator"
]
},
"input": {
"type": "cel"
},
"tags": [
"preserve_original_event",
"forwarded",
"anomali-intelligence"
],
"threat": {
"indicator": {
"confidence": "Medium",
"marking": {
"tlp": "WHITE"
},
"modified_at": "2021-04-06T09:56:22.915Z",
"provider": "Analyst",
"type": "domain-name",
"url": {
"domain": "gen1xyz.com"
}
}
}
}
**Exported fields**
| Field | Description | Type |
|---|---|---|
| @timestamp | Event timestamp. | date |
| anomali.threatstream.can_add_public_tags | Indicates whether a user can add public tags to a Threat Model entity. | boolean |
| anomali.threatstream.confidence | Level of certainty that an observable is of the reported indicator type. Confidence scores range from 0-100, in increasing order of confidence, and is assigned by ThreatStream based on several factors. | long |
| anomali.threatstream.deletion_scheduled_at | At this time the IOC will be deleted by the transform. | date |
| anomali.threatstream.expiration_ts | Time stamp of when intelligence will expire on ThreatStream, in UTC time. Note: expiration_ts can only be specified in an advanced search query. | date |
| anomali.threatstream.feed_id | Numeric ID of the threat feed that generated the indicator. feed_id = 0 for user-created indicators. | long |
| anomali.threatstream.id | Unique ID for the indicator. This identifier is assigned to the indicator when it is first created on ThreatStream. Unlike update_id, this identifier never changes as long as the indicator is available on ThreatStream. | keyword |
| anomali.threatstream.import_session_id | ID of import session in which the indicator was imported. import_session_id=0 if the indicator came in through a threat feed. | long |
| anomali.threatstream.is_anonymous | Whether the organization and user information is anonymized when the observable is accessed by users outside of the owner organization. | boolean |
| anomali.threatstream.is_editable | Indicates whether the imported entity can be updated by an intelligence source. This attribute is reserved for intelligence source providers and can be ignored. | boolean |
| anomali.threatstream.is_public | Visibility of the indicator—public or private. 0/False—if the indicator is private or belongs to a Trusted Circle 1/True—if the indicator is public Default: 0/False | boolean |
| anomali.threatstream.itype | Indicator type. | keyword |
| anomali.threatstream.meta.maltype | Tag that specifies the malware associated with an indicator. | keyword |
| anomali.threatstream.meta.registrant.address | Indicator domain WHOIS registrant address. | keyword |
| anomali.threatstream.meta.registrant.email | Indicator domain WHOIS registrant email. | keyword |
| anomali.threatstream.meta.registrant.name | Indicator domain WHOIS registrant name. | keyword |
| anomali.threatstream.meta.registrant.org | Indicator domain WHOIS registrant org. | keyword |
| anomali.threatstream.meta.registrant.phone | Indicator domain WHOIS registrant phone. | keyword |
| anomali.threatstream.meta.registration_created | Registration created. | date |
| anomali.threatstream.meta.registration_updated | Registration updated. | date |
| anomali.threatstream.meta.severity | Severity assigned to the indicator through machine-learning algorithms ThreatStream deploys. Possible values: low, medium, high, very-high | keyword |
| anomali.threatstream.owner_organization_id | ID of the (ThreatStream) organization that brought in the indicator, either through a threat feed or through the import process. | long |
| anomali.threatstream.rdns | Domain name (obtained through reverse domain name lookup) associated with the IP address that is associated with the indicator. | keyword |
| anomali.threatstream.retina_confidence | Confidence score assigned to the observable by Anomali machine learning algorithms. | long |
| anomali.threatstream.source_created | Time stamp of when the entity was created by its original source. | date |
| anomali.threatstream.source_modified | Time stamp of when the entity was last updated by its original source. | date |
| anomali.threatstream.source_reported_confidence | A risk score from 0 to 100, provided by the source of the indicator. | long |
| anomali.threatstream.status | Status assigned to the indicator. For example, active, inactive, falsepos. | keyword |
| anomali.threatstream.threat_type | Summarized threat type of the indicator. For example, malware, compromised, apt, c2, and so on. | keyword |
| anomali.threatstream.threatscore | Deprecated. | keyword |
| anomali.threatstream.trusted_circle_ids | IDs of the trusted circles with which the indicator is shared. | long |
| anomali.threatstream.type | Type of indicator—domain, email, ip, md5, string, url. | keyword |
| anomali.threatstream.update_id | An incrementing numeric identifier associated with each update to intelligence on ThreatStream. | long |
| anomali.threatstream.uuid | UUID (universally unique identifier) assigned to the observable for STIX compliance. | keyword |
| anomali.threatstream.value | Value of the observable. For example, 192.168.0.10 or http://www.google.com. | keyword |
| cloud.image.id | Image ID for the cloud instance. | keyword |
| data_stream.dataset | Data stream dataset name. | constant_keyword |
| data_stream.namespace | Data stream namespace. | constant_keyword |
| data_stream.type | Data stream type. | constant_keyword |
| event.dataset | Event dataset | constant_keyword |
| event.module | Event module | constant_keyword |
| host.containerized | If the host is a container. | boolean |
| host.os.build | OS build information. | keyword |
| host.os.codename | OS codename, if any. | keyword |
| input.type | Type of Filebeat input. | keyword |
| labels.is_ioc_transform_source | Indicates whether an IOC is in the raw source data stream, or the in latest destination index. | constant_keyword |
| threat.feed.dashboard_id | Dashboard ID used for Kibana CTI UI | constant_keyword |
| threat.feed.name | Display friendly feed name | constant_keyword |
This source of indicators is deprecated. New users should instead use the API source above. This source requires additional software, the Elastic Extension, to connect Anomali ThreatStream to this integration. It’s available on the ThreatStream download page.
Please refer to the documentation included with the extension for a detailed explanation on how to configure Anomali ThreatStream to send indicators to this integration.
Indicators ingested in this way will become accessible using the alias logs-ti_anomali_latest.threatstream.
**Example**
An example event for threatstream looks as following:
{
"@timestamp": "2020-10-08T12:22:11.000Z",
"agent": {
"ephemeral_id": "2f4f6445-5077-4a66-8582-2c74e071b6dd",
"id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.13.0"
},
"anomali": {
"threatstream": {
"added_at": "2020-10-08T12:22:11.000Z",
"classification": "public",
"confidence": 20,
"deleted_at": "2020-10-13T12:22:11.000Z",
"detail2": "imported by user 184",
"id": "3135167627",
"import_session_id": "1400",
"itype": "mal_domain",
"resource_uri": "/api/v1/intelligence/P46279656657/",
"severity": "high",
"source_feed_id": "3143",
"state": "active",
"trusted_circle_ids": [
"122"
],
"update_id": "3786618776",
"value_type": "domain"
}
},
"data_stream": {
"dataset": "ti_anomali.threatstream",
"namespace": "44735",
"type": "logs"
},
"ecs": {
"version": "8.11.0"
},
"elastic_agent": {
"id": "36b03887-7783-4bc4-b8c5-6f8997e4cd1a",
"snapshot": false,
"version": "8.13.0"
},
"event": {
"agent_id_status": "verified",
"category": [
"threat"
],
"dataset": "ti_anomali.threatstream",
"ingested": "2024-08-01T07:49:22Z",
"kind": "enrichment",
"original": "{\"added_at\":\"2020-10-08T12:22:11\",\"classification\":\"public\",\"confidence\":20,\"country\":\"FR\",\"date_first\":\"2020-10-08T12:21:50\",\"date_last\":\"2020-10-08T12:24:42\",\"detail2\":\"imported by user 184\",\"domain\":\"d4xgfj.example.net\",\"id\":3135167627,\"import_session_id\":1400,\"itype\":\"mal_domain\",\"lat\":-49.1,\"lon\":94.4,\"org\":\"OVH Hosting\",\"resource_uri\":\"/api/v1/intelligence/P46279656657/\",\"severity\":\"high\",\"source\":\"Default Organization\",\"source_feed_id\":3143,\"srcip\":\"89.160.20.156\",\"state\":\"active\",\"trusted_circle_ids\":\"122\",\"update_id\":3786618776,\"value_type\":\"domain\"}",
"severity": 7,
"type": [
"indicator"
]
},
"input": {
"type": "http_endpoint"
},
"tags": [
"preserve_original_event",
"forwarded",
"anomali-threatstream"
],
"threat": {
"indicator": {
"as": {
"organization": {
"name": "OVH Hosting"
}
},
"confidence": "Low",
"first_seen": "2020-10-08T12:21:50.000Z",
"geo": {
"country_iso_code": "FR",
"location": {
"lat": -49.1,
"lon": 94.4
}
},
"ip": "89.160.20.156",
"last_seen": "2020-10-08T12:24:42.000Z",
"marking": {
"tlp": [
"WHITE"
]
},
"provider": "Default Organization",
"type": "domain-name",
"url": {
"domain": "d4xgfj.example.net"
}
}
}
}
**Exported fields**
| Field | Description | Type |
|---|---|---|
| @timestamp | Event timestamp. | date |
| anomali.threatstream.added_at | Date when IOC was added. | date |
| anomali.threatstream.classification | Indicates whether an indicator is private or from a public feed and available publicly. Possible values: private, public. | keyword |
| anomali.threatstream.confidence | The measure of the accuracy (from 0 to 100) assigned by ThreatStream’s predictive analytics technology to indicators. | short |
| anomali.threatstream.deleted_at | Date when IOC was deleted/expired. | date |
| anomali.threatstream.detail2 | Detail text for indicator. | text |
| anomali.threatstream.id | The ID of the indicator. | keyword |
| anomali.threatstream.import_session_id | ID of the import session that created the indicator on ThreatStream. | keyword |
| anomali.threatstream.itype | Indicator type. Possible values: "apt_domain", "apt_email", "apt_ip", "apt_url", "bot_ip", "c2_domain", "c2_ip", "c2_url", "i2p_ip", "mal_domain", "mal_email", "mal_ip", "mal_md5", "mal_url", "parked_ip", "phish_email", "phish_ip", "phish_url", "scan_ip", "spam_domain", "ssh_ip", "suspicious_domain", "tor_ip" and "torrent_tracker_url". | keyword |
| anomali.threatstream.maltype | Information regarding a malware family, a CVE ID, or another attack or threat, associated with the indicator. | wildcard |
| anomali.threatstream.md5 | Hash for the indicator. | keyword |
| anomali.threatstream.resource_uri | Relative URI for the indicator details. | keyword |
| anomali.threatstream.severity | Criticality associated with the threat feed that supplied the indicator. Possible values: low, medium, high, very-high. | keyword |
| anomali.threatstream.source | Source for the indicator. | keyword |
| anomali.threatstream.source_feed_id | ID for the integrator source. | keyword |
| anomali.threatstream.state | State for this indicator. | keyword |
| anomali.threatstream.trusted_circle_ids | ID of the trusted circle that imported the indicator. | keyword |
| anomali.threatstream.update_id | Update ID. | keyword |
| anomali.threatstream.url | URL for the indicator. | keyword |
| anomali.threatstream.value_type | Data type of the indicator. Possible values: ip, domain, url, email, md5. | keyword |
| cloud.image.id | Image ID for the cloud instance. | keyword |
| data_stream.dataset | Data stream dataset name. | constant_keyword |
| data_stream.namespace | Data stream namespace. | constant_keyword |
| data_stream.type | Data stream type. | constant_keyword |
| event.dataset | Event dataset | constant_keyword |
| event.module | Event module | constant_keyword |
| host.containerized | If the host is a container. | boolean |
| host.os.build | OS build information. | keyword |
| host.os.codename | OS codename, if any. | keyword |
| input.type | Type of Filebeat input. | keyword |
| labels.is_ioc_transform_source | Indicates whether an IOC is in the raw source data stream, or the in latest destination index. | constant_keyword |
| log.flags | Flags for the log file. | keyword |
| log.offset | Offset of the entry in the log file. | long |
| threat.feed.dashboard_id | Dashboard ID used for Kibana CTI UI | constant_keyword |
| threat.feed.name | Display friendly feed name | constant_keyword |
| threat.indicator.first_seen | The date and time when intelligence source first reported sighting this indicator. | date |
| threat.indicator.last_seen | The date and time when intelligence source last reported sighting this indicator. | date |
| threat.indicator.modified_at | The date and time when intelligence source last modified information for this indicator. | date |
**Changelog**
| Version | Details | Kibana version(s) |
|---|---|---|
| 1.25.0 | pass:[] Enhancement (View pull request) Do not remove event.original in main ingest pipeline. |
8.13.0 or higher |
| 1.24.0 | pass:[] Enhancement (View pull request) Add "preserve_original_event" tag to documents with event.kind set to "pipeline_error". |
8.13.0 or higher |
| 1.23.0 | pass:[] Enhancement (View pull request) Support the ThreatStream API |
8.13.0 or higher |
| 1.22.3 | pass:[] Bug fix (View pull request) Fix labels.is_ioc_transform_source values |
8.13.0 or higher |
| 1.22.2 | pass:[] Bug fix (View pull request) Add missing fields in transform |
8.13.0 or higher |
| 1.22.1 | pass:[] Bug fix (View pull request) Fix ECS date mapping on threat fields. |
8.13.0 or higher |
| 1.22.0 | pass:[] Enhancement (View pull request) Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template. |
8.13.0 or higher |
| 1.21.0 | pass:[] Enhancement (View pull request) Add destination index alias and fix docs. |
8.12.0 or higher |
| 1.20.0 | pass:[] Enhancement (View pull request) Set sensitive values as secret. |
8.12.0 or higher |
| 1.19.2 | pass:[] Enhancement (View pull request) Changed owners |
8.8.0 or higher |
| 1.19.1 | pass:[] Bug fix (View pull request) Fix IOC expiration duration character casting. |
8.8.0 or higher |
| 1.19.0 | pass:[] Enhancement (View pull request) ECS version updated to 8.11.0. |
8.8.0 or higher |
| 1.18.0 | pass:[] Enhancement (View pull request) ECS version updated to 8.10.0. |
8.8.0 or higher |
| 1.17.0 | pass:[] Enhancement (View pull request) Add DLM policy. Add owner.type to package manifest. Update format_version to 3.0.0 pass:[] Enhancement (View pull request) Add tags.yml file so that integration’s dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI. |
8.8.0 or higher |
| 1.16.1 | pass:[] Bug fix (View pull request) Remove dotted YAML keys. |
— |
| 1.16.0 | pass:[] Enhancement (View pull request) Update package-spec to 2.10.0. |
8.8.0 or higher |
| 1.15.1 | pass:[] Bug fix (View pull request) Fix destination fields mapping. Note - Since transform’s destination index version is changed, users must manually delete the old index. See /integration-docs/docs/reference/ingestion-tools/integrations/ti_anomali.md#ti_anomali-logs for details on destination index versioning. |
8.8.0 or higher |
| 1.15.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.9.0. |
8.8.0 or higher |
| 1.14.1 | pass:[] Bug fix (View pull request) Change sort order field in latest transform |
8.8.0 or higher |
| 1.14.0 | pass:[] Enhancement (View pull request) Ensure event.kind is correctly set for pipeline errors. |
8.8.0 or higher |
| 1.13.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.8.0. |
8.8.0 or higher |
| 1.12.0 | pass:[] Enhancement (View pull request) Add IOC field to transform source to easily filter destination indices with unexpired indicators |
8.8.0 or higher |
| 1.11.0 | pass:[] Enhancement (View pull request) Support for IoC Expiration |
8.8.0 or higher |
| 1.10.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.7.0. |
8.0.0 or higher |
| 1.9.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.6.0. |
8.0.0 or higher |
| 1.8.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.5.0. |
8.0.0 or higher |
| 1.7.0 | pass:[] Enhancement (View pull request) Deprecate EOL Limo datastream. See https://www.anomali.com/resources/limo. |
8.0.0 or higher |
| 1.6.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.4.0 |
8.0.0 or higher |
| 1.5.1 | pass:[] Bug fix (View pull request) Fix proxy URL documentation rendering. |
8.0.0 or higher |
| 1.5.0 | pass:[] Enhancement (View pull request) Update categories to include threat_intel. |
8.0.0 or higher |
| 1.4.0 | pass:[] Enhancement (View pull request) Update package to ECS 8.3.0. |
8.0.0 or higher |
| 1.3.3 | pass:[] Enhancement (View pull request) added links to Anomoli documentation in readme |
8.0.0 or higher |
| 1.3.2 | pass:[] Bug fix (View pull request) Fix threatstream |
8.0.0 or higher |
| 1.3.1 | pass:[] Enhancement (View pull request) Update package descriptions |
8.0.0 or higher |
| 1.3.0 | pass:[] Enhancement (View pull request) Update to ECS 8.2 |
8.0.0 or higher |
| 1.2.3 | pass:[] Enhancement (View pull request) Add mapping for event.created |
8.0.0 or higher |
| 1.2.2 | pass:[] Enhancement (View pull request) Add documentation for multi-fields |
8.0.0 or higher |
| 1.2.1 | pass:[] Bug fix (View pull request) Adding first interval to Anomali Limo policy UI |
8.0.0 or higher |
| 1.2.0 | pass:[] Enhancement (View pull request) Update to ECS 8.0 |
8.0.0 or higher |
| 1.1.3 | pass:[] Bug fix (View pull request) Regenerate test files using the new GeoIP database |
8.0.0 or higher |
| 1.1.2 | pass:[] Bug fix (View pull request) Change test public IPs to the supported subset |
— |
| 1.1.1 | pass:[] Enhancement (View pull request) Fixing typo in base-fields.yml |
— |
| 1.1.0 | pass:[] Enhancement (View pull request) Adding dashboards and threat.feed ECS fields |
— |
| 1.0.2 | pass:[] Enhancement (View pull request) Bump minimum version |
8.0.0 or higher |
| 1.0.1 | pass:[] Enhancement (View pull request) Update title and description. |
— |
| 1.0.0 | pass:[] Enhancement (View pull request) Initial release |
— |