Elastic
Start free trial Contact Sales
Loading

Alienvault OTX Integration

<div class="condensed-table">
| | |
| --- | --- |
| Version | 1.27.0 (View all) |
| Compatible Kibana version(s) | 8.13.0 or higher |
| Supported Serverless project types
What’s this? | Security
Observability |
| Subscription level
What’s this? | Basic |
| Level of support
What’s this? | Elastic |

</div>
This integration is for Alienvault OTX. It retrieves indicators for all pulses subscribed to a specific user account on OTX

To use this package, it is required to have an account on Alienvault OTX. Once an account has been created, and at least 1 pulse has been subscribed to, the API key can be retrieved from your user profile dashboard. In the top right corner there should be an OTX KEY.

Retrieves all the related indicators over time, related to your pulse subscriptions on OTX.

**Exported fields**
Field Description Type
@timestamp Event timestamp. date
cloud.image.id Image ID for the cloud instance. keyword
data_stream.dataset Data stream dataset name. constant_keyword
data_stream.namespace Data stream namespace. constant_keyword
data_stream.type Data stream type. constant_keyword
event.dataset Event dataset constant_keyword
event.module Event module constant_keyword
host.containerized If the host is a container. boolean
host.os.build OS build information. keyword
host.os.codename OS codename, if any. keyword
input.type Type of Filebeat input. keyword
log.flags Flags for the log file. keyword
log.offset Offset of the entry in the log file. long
otx.content Extra text or descriptive content related to the indicator. keyword
otx.description A description of the indicator. keyword
otx.id The ID of the indicator. keyword
otx.indicator The value of the indicator, for example if the type is domain, this would be the value. keyword
otx.title Title describing the indicator. keyword
otx.type The indicator type, can for example be "domain, email, FileHash-SHA256". keyword
threat.feed.dashboard_id Dashboard ID used for Kibana CTI UI constant_keyword
threat.feed.name Display friendly feed name constant_keyword
threat.indicator.file.hash.pehash The file’s pehash, if available. keyword
threat.indicator.first_seen The date and time when intelligence source first reported sighting this indicator. date
threat.indicator.last_seen The date and time when intelligence source last reported sighting this indicator. date
threat.indicator.modified_at The date and time when intelligence source last modified information for this indicator. date
**Example**

An example event for threat looks as following:

{
    "@timestamp": "2024-03-08T02:55:33.690Z",
    "agent": {
        "ephemeral_id": "8edc1f21-05cd-4fa5-aadc-66e64f44856a",
        "id": "f29e7d89-991e-4f0a-838f-9c2eb93d876e",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.12.1"
    },
    "data_stream": {
        "dataset": "ti_otx.threat",
        "namespace": "ep",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "f29e7d89-991e-4f0a-838f-9c2eb93d876e",
        "snapshot": false,
        "version": "8.12.1"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "created": "2024-03-08T02:55:33.690Z",
        "dataset": "ti_otx.threat",
        "ingested": "2024-03-08T02:55:45Z",
        "kind": "enrichment",
        "original": "{\"count\":40359,\"next\":\"https://otx.alienvault.com/api/v1/indicators/export?types=domain%2CIPv4%2Chostname%2Curl%2CFileHash-SHA256\\u0026modified_since=2020-11-29T01%3A10%3A00+00%3A00\\u0026page=2\",\"previous\":null,\"results\":{\"content\":\"\",\"description\":null,\"id\":1251,\"indicator\":\"info.3000uc.com\",\"title\":null,\"type\":\"hostname\"}}",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "httpjson"
    },
    "otx": {},
    "tags": [
        "preserve_original_event",
        "forwarded",
        "otx-threat"
    ],
    "threat": {
        "indicator": {
            "type": "domain-name",
            "url": {
                "domain": "info.3000uc.com"
            }
        }
    }
}

Retrieves all indicators from subscribed pulses on OTX from API /api/v1/pulses/subscribed using Filebeat’s CEL input. The following subscriptions are included by this API:

  • All pulses by users you are subscribed to
  • All pulses you are directly subscribed to
  • All pulses you have created yourself
  • All pulses from groups you are a member of

Pulses Subscribed datastream also supports IoC expiration by using latest transform. Below are the steps on how it is handled:

  1. All the indicators are retrieved into source indices named logs-ti_otx.pulses_subscribed-* using CEL input and processed via ingest pipelines. These indicators have a property named expiration which is either a null value or a timestamp such as "2023-09-07T00:00:00". When the value is null or if the timestamp value is less than current timestamp now(), the indicator is not expired, and hence is still active.

  2. A latest transform is continuosly run on source indices. The purpose of this transform is to:

    • Move only the active indicators from source indices into destination indices named logs-ti_otx_latest.pulses_subscribed-<NUMBER> where NUMBER indicates index version.
    • Delete expired indicators based on the expiration timestamp value.

  3. All the active indicators can be retrieved using destination index alias logs-ti_otx_latest.pulses_subscribed which points to the latest destination index version.

    • Note: Do not use the source indices logs-ti_otx.pulses_subscribed-*, because when the indicators expire, the source indices will contain duplicates. Always use the destination index alias: logs-ti_otx_latest.pulses_subscribed to query all active indicators.
**Exported fields**
Field Description Type
@timestamp Event timestamp. date
data_stream.dataset Data stream dataset name. constant_keyword
data_stream.namespace Data stream namespace. constant_keyword
data_stream.type Data stream type. constant_keyword
event.dataset Event dataset constant_keyword
event.module Event module constant_keyword
input.type Type of Filebeat input. keyword
labels.is_ioc_transform_source Indicates whether an IOC is in the raw source data stream, or the in latest destination index. constant_keyword
log.flags Flags for the log file. keyword
log.offset Offset of the entry in the log file. long
otx.content keyword
otx.count integer
otx.created date
otx.description keyword
otx.expiration date
otx.id The ID of the indicator. keyword
otx.indicator keyword
otx.is_active integer
otx.prefetch_pulse_ids boolean
otx.pulse.adversary keyword
otx.pulse.attack_ids keyword
otx.pulse.author_name keyword
otx.pulse.created date
otx.pulse.description keyword
otx.pulse.extract_source keyword
otx.pulse.id keyword
otx.pulse.industries keyword
otx.pulse.malware_families keyword
otx.pulse.modified date
otx.pulse.more_indicators boolean
otx.pulse.name keyword
otx.pulse.public integer
otx.pulse.references keyword
otx.pulse.revision integer
otx.pulse.targeted_countries keyword
otx.pulse.tlp keyword
otx.role keyword
otx.t double
otx.t2 double
otx.t3 double
otx.title keyword
threat.feed.dashboard_id Dashboard ID used for Kibana CTI UI constant_keyword
threat.feed.name Display friendly feed name constant_keyword
threat.indicator.file.hash.pehash The file’s pehash, if available. keyword
threat.indicator.first_seen The date and time when intelligence source first reported sighting this indicator. date
threat.indicator.last_seen The date and time when intelligence source last reported sighting this indicator. date
threat.indicator.modified_at The date and time when intelligence source last modified information for this indicator. date
**Example**

An example event for pulses_subscribed looks as following:

{
    "@timestamp": "2023-08-08T05:05:15.000Z",
    "agent": {
        "ephemeral_id": "c12b4f72-265e-41f0-96e0-103c81de7908",
        "id": "8299ae35-ee0e-4107-9acb-1b6acfdda1fb",
        "name": "docker-fleet-agent",
        "type": "filebeat",
        "version": "8.13.0"
    },
    "data_stream": {
        "dataset": "ti_otx.pulses_subscribed",
        "namespace": "32586",
        "type": "logs"
    },
    "ecs": {
        "version": "8.11.0"
    },
    "elastic_agent": {
        "id": "8299ae35-ee0e-4107-9acb-1b6acfdda1fb",
        "snapshot": false,
        "version": "8.13.0"
    },
    "event": {
        "agent_id_status": "verified",
        "category": [
            "threat"
        ],
        "dataset": "ti_otx.pulses_subscribed",
        "ingested": "2024-08-02T06:03:28Z",
        "kind": "enrichment",
        "original": "{\"content\":\"\",\"count\":2,\"created\":\"2023-08-08T05:05:15\",\"description\":\"\",\"expiration\":null,\"id\":3454375108,\"indicator\":\"pinup-casino-tr.site\",\"is_active\":1,\"prefetch_pulse_ids\":false,\"pulse_raw\":\"{\\\"adversary\\\":\\\"\\\",\\\"attack_ids\\\":[\\\"T1531\\\",\\\"T1059\\\",\\\"T1566\\\"],\\\"author_name\\\":\\\"SampleUser\\\",\\\"created\\\":\\\"2023-08-22T09:43:18.855000\\\",\\\"description\\\":\\\"\\\",\\\"extract_source\\\":[],\\\"id\\\":\\\"64e38336d783f91d6948a7b1\\\",\\\"industries\\\":[],\\\"malware_families\\\":[\\\"WHIRLPOOL\\\"],\\\"modified\\\":\\\"2023-08-22T09:43:18.855000\\\",\\\"more_indicators\\\":false,\\\"name\\\":\\\"Sample Pulse\\\",\\\"public\\\":1,\\\"references\\\":[\\\"https://www.cisa.gov/news-events/analysis-reports/ar23-230a\\\"],\\\"revision\\\":1,\\\"tags\\\":[\\\"cisa\\\",\\\"backdoor\\\",\\\"whirlpool\\\",\\\"malware\\\"],\\\"targeted_countries\\\":[],\\\"tlp\\\":\\\"white\\\"}\",\"role\":null,\"t\":0,\"t2\":0.0050694942474365234,\"t3\":2.7960586547851562,\"title\":\"\",\"type\":\"domain\"}",
        "type": [
            "indicator"
        ]
    },
    "input": {
        "type": "cel"
    },
    "otx": {
        "count": 2,
        "created": "2023-08-08T05:05:15.000Z",
        "expiration": "2023-08-13T05:05:15.000Z",
        "id": "3454375108",
        "is_active": 1,
        "prefetch_pulse_ids": false,
        "pulse": {
            "attack_ids": [
                "T1531",
                "T1059",
                "T1566"
            ],
            "author_name": "SampleUser",
            "created": "2023-08-22T09:43:18.855Z",
            "description": "",
            "extract_source": [],
            "id": "64e38336d783f91d6948a7b1",
            "industries": [],
            "malware_families": [
                "WHIRLPOOL"
            ],
            "modified": "2023-08-22T09:43:18.855Z",
            "more_indicators": false,
            "name": "Sample Pulse",
            "public": 1,
            "references": [
                "https://www.cisa.gov/news-events/analysis-reports/ar23-230a"
            ],
            "revision": 1,
            "targeted_countries": [],
            "tlp": "white"
        },
        "t": 0,
        "t2": 0.0050694942474365234,
        "t3": 2.7960586547851562
    },
    "tags": [
        "preserve_original_event",
        "forwarded",
        "otx-pulses_subscribed",
        "cisa",
        "backdoor",
        "whirlpool",
        "malware"
    ],
    "threat": {
        "indicator": {
            "provider": "OTX",
            "type": "domain-name",
            "url": {
                "domain": "pinup-casino-tr.site"
            }
        }
    }
}
**Changelog**
Version Details Kibana version(s)
1.27.0 pass:[] Enhancement (View pull request)
Do not remove event.original in main ingest pipeline.
 8.13.0 or higher
1.26.0 pass:[] Enhancement (View pull request)
Add "preserve_original_event" tag to documents with event.kind set to "pipeline_error".
 8.13.0 or higher
1.25.3 pass:[] Bug fix (View pull request)
Fix labels.is_ioc_transform_source values
 8.13.0 or higher
1.25.2 pass:[] Bug fix (View pull request)
Add missing fields in transform
 8.13.0 or higher
1.25.1 pass:[] Bug fix (View pull request)
Fix ECS date mapping on threat fields.
 8.13.0 or higher
1.25.0 pass:[] Enhancement (View pull request)
Update the kibana constraint to ^8.13.0. Modified the field definitions to remove ECS fields made redundant by the ecs@mappings component template.
 8.13.0 or higher
1.24.1 pass:[] Bug fix (View pull request)
Fix type-mapping inconsistency for otx.id field.
 8.12.0 or higher
1.24.0 pass:[] Enhancement (View pull request)
Set sensitive values as secret.
 8.12.0 or higher
1.23.2 pass:[] Enhancement (View pull request)
Changed owners
 8.10.3 or higher
1.23.1 pass:[] Bug fix (View pull request)
Fix IOC expiration duration character casting.
 8.10.3 or higher
1.23.0 pass:[] Enhancement (View pull request)
Append hash and IP values to related.* fields
 8.10.3 or higher
1.22.0 pass:[] Enhancement (View pull request)
Limit request tracer log count to five.
 8.10.3 or higher
1.21.0 pass:[] Enhancement (View pull request)
ECS version updated to 8.11.0.
 8.10.3 or higher
1.20.0 pass:[] Enhancement (View pull request)
Improve event.original check to avoid errors if set.
 8.10.3 or higher
1.19.0 pass:[] Enhancement (View pull request)
Add Pulses Subscribed datastream to support IoC expiration

pass:[] Enhancement (View pull request)
Add DLM policy. Add owner.type to package manifest.
 8.10.3 or higher
1.18.0 pass:[] Enhancement (View pull request)
ECS version updated to 8.10.0.
 8.7.1 or higher
1.17.0 pass:[] Enhancement (View pull request)
The format_version in the package manifest changed from 2.11.0 to 3.0.0. Removed dotted YAML keys from package manifest. Added owner.type: elastic to package manifest.
 8.7.1 or higher
1.16.0 pass:[] Enhancement (View pull request)
Add tags.yml file so that integration’s dashboards and saved searches are tagged with "Security Solution" and displayed in the Security Solution UI.
 8.7.1 or higher
1.15.0 pass:[] Enhancement (View pull request)
Update package-spec to 2.10.0.
 8.7.1 or higher
1.14.0 pass:[] Enhancement (View pull request)
Update package to ECS 8.9.0.
 8.7.1 or higher
1.13.0 pass:[] Enhancement (View pull request)
Document duration units.
 8.7.1 or higher
1.12.0 pass:[] Enhancement (View pull request)
Document valid duration units.
 8.7.1 or higher
1.11.0 pass:[] Enhancement (View pull request)
Ensure event.kind is correctly set for pipeline errors.
 8.7.1 or higher
1.10.0 pass:[] Enhancement (View pull request)
Update package to ECS 8.8.0.
 8.7.1 or higher
1.9.0 pass:[] Enhancement (View pull request)
Add a new flag to enable request tracing
 8.7.1 or higher
1.8.0 pass:[] Enhancement (View pull request)
Update package to ECS 8.7.0.
 8.0.0 or higher
1.7.1 pass:[] Enhancement (View pull request)
Honor preserve_original_event setting.
 8.0.0 or higher
1.7.0 pass:[] Enhancement (View pull request)
Update package to ECS 8.6.0.
 8.0.0 or higher
1.6.1 pass:[] Enhancement (View pull request)
Add support to drop empty documents
 8.0.0 or higher
1.6.0 pass:[] Enhancement (View pull request)
Update package to ECS 8.5.0.
 8.0.0 or higher
1.5.0 pass:[] Enhancement (View pull request)
Update package to ECS 8.4.0
 8.0.0 or higher
1.4.2 pass:[] Bug fix (View pull request)
Fix proxy URL documentation rendering.
 8.0.0 or higher
1.4.1 pass:[] Enhancement (View pull request)
Update categories to include threat_intel.
 8.0.0 or higher
1.4.0 pass:[] Enhancement (View pull request)
Update package to ECS 8.3.0.
 8.0.0 or higher
1.3.2 pass:[] Enhancement (View pull request)
Update readme file to add documentation link
 8.0.0 or higher
1.3.1 pass:[] Enhancement (View pull request)
Update package descriptions
 8.0.0 or higher
1.3.0 pass:[] Enhancement (View pull request)
Update to ECS 8.2
 8.0.0 or higher
1.2.2 pass:[] Enhancement (View pull request)
Add field mapping for event.created
 8.0.0 or higher
1.2.1 pass:[] Enhancement (View pull request)
Add documentation for multi-fields
 8.0.0 or higher
1.2.0 pass:[] Enhancement (View pull request)
Update to ECS 8.0
 8.0.0 or higher
1.1.0 pass:[] Enhancement (View pull request)
Adding threat.feed fields and dashboards
 8.0.0 or higher
1.0.3 pass:[] Bug fix (View pull request)
Change test public IPs to the supported subset
 8.0.0 or higher
1.0.2 pass:[] Enhancement (View pull request)
Bump minimum version
 8.0.0 or higher
1.0.1 pass:[] Enhancement (View pull request)
Update title and description.
 —
1.0.0 pass:[] Enhancement (View pull request)
Initial release
 —