---
title: Automatic Import
description: Use Automatic Import with an LLM to build a custom Elastic integration from a data sample when no prebuilt integration exists, for Security, Observability, and other solutions.
url: https://www.elastic.co/elastic/docs-builder/docs/4300/explore-analyze/ai-features/automatic-import
products:
- Elastic Cloud Serverless
- Elastic Documentation
- Elastic Observability
- Elastic Security
- Kibana
applies_to:
- Elastic Cloud Serverless: Generally available
- Elastic Stack: Generally available
---
# Automatic Import
Automatic Import parses, ingests, and maps data to [ECS](https://www.elastic.co/elasticsearch/common-schema) for sources that don’t yet have prebuilt Elastic integrations. It works with Elastic Security, Observability, and other solutions that rely on Elastic Agent and integrations. This lets you onboard custom or niche data sources without building a full integration manually.
Automatic Import uses a large language model (LLM) with specialized instructions to analyze source data and generate a custom integration.
Elastic integrations, including those created by Automatic Import, normalize data to [the Elastic Common Schema (ECS)](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/4300/reference/ecs). This standardization provides consistent use across dashboards, search, alerts, and machine learning features.
Refer to [prebuilt data integrations](https://docs.elastic.co/en/integrations) for a full list of Elastic’s 400+ integrations.
## Requirements
- A working [LLM connector](https://www.elastic.co/elastic/docs-builder/docs/4300/explore-analyze/ai-features/llm-guides/llm-connectors).
- Elastic Stack users: An [Enterprise](https://www.elastic.co/pricing) subscription.
- Serverless **Elastic Security Serverless** projects: the [Security Analytics Complete](/elastic/docs-builder/docs/4300/deploy-manage/deploy/elastic-cloud/project-settings#elastic-sec-project-features) feature tier.
- Serverless **Elastic Observability Serverless** projects: the [Observability Complete](/elastic/docs-builder/docs/4300/deploy-manage/deploy/elastic-cloud/project-settings#obs-serverless-project-features) feature tier.
- A sample of the data you want to import.
## Prepare your sample data
Collect a sample of the data you want to import before you create the integration. Automatic Import sends that sample to the LLM, and the LLM builds an integration from it.
Save that sample in one of the following formats:
- **JSON and NDJSON**: Represent each event as its own object, and keep nesting shallow.
- **CSV**: Include a header row with column names. Automatic Import recognizes the header. Without a header, the LLM attempts to create descriptive field names from the column formats and values.
- **Syslog**: Use a structured or unstructured sample.
Whichever format you use, include a wide range of unique log entries for the event types you want the integration to handle. The more the sample varies, the more accurate the pipeline is.
Start the file with a focused set of the event types you want the integration to handle. Automatic Import sends the first samples in the file to the LLM, and the LLM builds the pipeline from those samples.
### Sample size limits
Automatic Import has limits on what it can send to the LLM. A sample is a log line or a document.
You can upload a file of any size. Automatic Import reads the file from the beginning and sends samples to the LLM until it reaches one of these limits:
| Limit | Value | What happens |
|-----------------------|--------------------|------------------------------------------------------------------------------------------------|
| Maximum samples | 1,000 | Stops after adding 1,000 samples to the request. |
| Maximum sample length | 100,000 characters | Skips the entire sample and continues with later samples. |
| Maximum request size | 10 MB | Stops before the request to the LLM exceeds 10 MB. This limit doesn't cap the file you upload. |
Automatic Import stops at 1,000 samples or 10 MB, whichever comes first. A sample longer than 100,000 characters doesn't count toward either limit. Short log lines reach the 1,000-sample limit first. Long samples, such as verbose JSON, can fill the 10 MB request first.If Automatic Import omits samples, the **Sample log limits applied** warning tells you how many samples it sent to the LLM and how many it left out.
Automatic Import sends the first 100 samples to the LLM.
## Recommended models
You can use Automatic Import with any LLM. Model performance varies. Model performance for Automatic Import is similar to model performance for Attack Discovery: models that perform well for Attack Discovery perform well for Automatic Import. Refer to the [large language model performance matrix for Elastic Security](https://www.elastic.co/elastic/docs-builder/docs/4300/solutions/security/ai/large-language-model-performance-matrix). For Observability AI asistant tasks, refer to the [LLM performance matrix for the Observability AI Assistant](https://www.elastic.co/elastic/docs-builder/docs/4300/solutions/observability/ai/llm-performance-matrix).
Using Automatic Import allows users to create new third-party data integrations through the use of third-party generative AI models (“GAI models”). Any third-party GAI models that you choose to use are owned and operated by their respective providers. Elastic does not own or control these third-party GAI models, nor does it influence their design, training, or data-handling practices. Using third-party GAI models with Elastic solutions, and using your data with third-party GAI models is at your discretion. Elastic bears no responsibility or liability for the content, operation, or use of these third-party GAI models, nor for any potential loss or damage arising from their use. Users are advised to exercise caution when using GAI models with personal, sensitive, or confidential information, as data submitted can be used to train the models or for other purposes. Elastic recommends familiarizing yourself with the development practices and terms of use of any third-party GAI models before use. You are responsible for ensuring that your use of Automatic Import complies with the terms and conditions of any third-party platform you connect with.
## Create a new custom integration
The integration creation flow changed in Elastic Stack 9.4 to support multiple data streams per integration and a new approve-then-install workflow. Serverless already uses the updated flow.
1. In Kibana, open **Integrations**. You can use the main menu, the [global search field](https://www.elastic.co/elastic/docs-builder/docs/4300/explore-analyze/find-and-organize/find-apps-and-objects), or your solution's entry point (for example, **Add integrations** in Elastic Security, or **Add data** in Observability).
2. Under **Can't find an integration?** click **Create integration**.
1. Select an [LLM connector](https://www.elastic.co/elastic/docs-builder/docs/4300/explore-analyze/ai-features/llm-guides/llm-connectors) in the top right.
2. Under **Integration Details**, provide a **Title** (required), **Description** (required), and **Logo** (optional).
1. Under **Define data streams and upload logs**, click **Add Data Stream**. You can add multiple data streams to a single integration.
2. In the **Data Stream** panel, provide a **Data stream title** and **Data stream description**. These fields appear on the integration's configuration page to help identify the data stream it writes to.
1. Select a [**Data collection method**](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/4300/reference/beats/filebeat/configuration-filebeat-options) to determine how the integration ingests the data. Supported methods:
- File Stream
- AWS S3
- AWS Cloudwatch
- Azure Blob Storage
- Azure Event Hub
- GCP Pub/Sub
- Google Cloud Storage
- HTTP Endpoint
- Kafka
- TCP
- UDP
These methods don't call an HTTP API. To build an integration package that calls an HTTP API, use the [Elastic integration skills](https://github.com/elastic/integration-skills). These workflows build the package with an AI coding agent.
2. Under **Logs**, either upload a sample of your data or select an existing index. Only indexes that include the `event.original` field are supported. Make sure your sample includes all the types of events that you want the integration to handle.
3. Click **Analyze logs** and wait for processing to complete. This can take several minutes. The data stream(s) continue to process as shown by the status on the **Manage my integrations** menu, so you can navigate away and come back later.
The **Manage my integrations** menu lists only integrations created with Automatic Import on versions 9.4+.Manually created custom integration `.zip`s get installed from the **Installed integrations** tab and aren't tracked on the **Manage my integrations** menu. For more information about user-created custom integrations, refer to [Upload a new integration](https://www.elastic.co/docs/extend/integrations/upload-new-integration).
1. When all data streams reach a **Success** status, the integration is ready to approve.
1. From the integration's **Actions** menu, click **Review & approve**. Select a category to help identify the integration on the Integrations page, then review the field mappings and the ingest pipeline.
Integration versions automatically increase when more data streams are added after the integration is approved.(Optional) To fine-tune the ingest pipeline, open it from the **Review & approve** panel and make your changes. Refer to the [Elastic Security ECS reference](https://www.elastic.co/elastic/docs-builder/docs/4300/reference/security/fields-and-object-schemas/siem-field-reference) for field-mapping guidance. Click **Save** when you're done.
1. When you approve an integration, by default it's also installed, which makes it available for assignment to a policy. To approve an integration without installing it, turn off the automatic installation option on its approval confirmation popup. Then, to install it later, click **Install** on the **Actions** menu.
2. Once you've installed an integration, you can find it using its category.
1. Click **Add** to start collecting data and assign the integration to an [agent policy](https://www.elastic.co/elastic/docs-builder/docs/4300/reference/fleet/agent-policy).
1. (Optional) After you've added an integration, you can edit the ingest pipeline from the **Ingest Pipelines** page using the navigation menu or the [global search field](https://www.elastic.co/elastic/docs-builder/docs/4300/explore-analyze/find-and-organize/find-apps-and-objects).
1. In Kibana, open **Integrations**. You can use the main menu, the [global search field](https://www.elastic.co/elastic/docs-builder/docs/4300/explore-analyze/find-and-organize/find-apps-and-objects), or your solution's entry point (for example, **Add integrations** in Elastic Security, or **Add data** in Observability).
2. Under **Can't find an integration?** click **Create new integration**.
3. Click **Create integration**.
4. Select an [LLM connector](https://www.elastic.co/elastic/docs-builder/docs/4300/explore-analyze/ai-features/llm-guides/llm-connectors).
5. Define how your new integration appears on the Integrations page by providing a **Title**, **Description**, and **Logo**. Click **Next**.
6. Define your integration's package name, which prefixes the imported event fields.
7. Define your **Data stream title**, **Data stream description**, and **Data stream name**. These fields appear on the integration's configuration page to help identify the data stream it writes to.
8. Select your [**Data collection method**](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/4300/reference/beats/filebeat/configuration-filebeat-options). This determines how your new integration ingests the data (for example, from an S3 bucket, an HTTP endpoint, or a file stream).
CEL generation in Automatic Import is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
If you select **API (CEL input)**, upload an OpenAPI specification (OAS) file in JSON or YAML format. This file is separate from the data sample, and the sample formats described earlier don't apply to it. Automatic Import generates a Common Expression Language (CEL) program from that file. The LLM uses the specification to determine which API endpoints (GET only), query parameters, and data structures to use. Select the endpoints to consume and your authentication method, then upload a sample of the API responses in the next step. For background, refer to the [CEL specification](https://github.com/google/cel-spec) and the [CEL input in Filebeat](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/4300/reference/beats/filebeat/filebeat-input-cel).
9. Upload a sample of your data. Make sure to include all the types of events that you want the new integration to handle.
10. Click **Analyze logs**, then wait for processing to complete. This may take several minutes.
11. After processing is complete, the pipeline's field mappings appear, including ECS and custom fields.
1. (Optional) After reviewing the proposed pipeline, you can fine-tune it by clicking **Edit pipeline**. Refer to the [Elastic Security ECS reference](https://www.elastic.co/elastic/docs-builder/docs/4300/reference/security/fields-and-object-schemas/siem-field-reference) to learn more about formatting field mappings. When you're satisfied with your changes, click **Save**.
If your new integration collects data from an API, you can update the [CEL input](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/4300/reference/beats/filebeat/filebeat-input-cel) configuration (program and API authentication information) from the new integration's integration policy.
1. Click **Add to Elastic**. After the **Success** message appears, your new integration is available on the Integrations page.
1. Click **Add to an agent** to deploy your new integration and start collecting data, or click **View integration** to view detailed information about your new integration.
2. (Optional) After you've added an integration, you can edit the ingest pipeline from the **Ingest Pipelines** page using the navigation menu or the [global search field](https://www.elastic.co/elastic/docs-builder/docs/4300/explore-analyze/find-and-organize/find-apps-and-objects).
If you use Elastic Security, you can use the [Data Quality dashboard](https://www.elastic.co/elastic/docs-builder/docs/4300/solutions/security/dashboards/data-quality-dashboard) to check the health of your data ingest pipelines and field mappings.