--- title: Run Elastic Agents in an air-gapped environment description: When running Elastic Agents in a restricted or closed network, you need to take extra steps to make sure: Kibana is able to reach the Elastic Package... url: https://www.elastic.co/elastic/docs-builder/docs/3028/reference/fleet/air-gapped products: - Elastic Agent - Fleet applies_to: - Elastic Cloud Enterprise: Generally available - Self-managed Elastic deployments: Generally available --- # Run Elastic Agents in an air-gapped environment When running Elastic Agents in a restricted or closed network, you need to take extra steps to make sure: - Kibana is able to reach the Elastic Package Registry to download package metadata and content. - Elastic Agents are able to download binaries during upgrades from the Elastic Artifact Registry. The Elastic Package Registry must therefore be accessible from Kibana through an HTTP Proxy or self-hosted. The Elastic Artifact Registry must therefore be accessible from Kibana through an HTTP Proxy or self-hosted. See the Elastic Security Solution documentation for air-gapped [offline endpoints](https://www.elastic.co/elastic/docs-builder/docs/3028/solutions/security/configure-elastic-defend/configure-offline-endpoints-air-gapped-environments). When upgrading all the components in an air-gapped environment, it is recommended that you upgrade in the following order: 1. Upgrade the Elastic Package Registry. 2. Upgrade the Elastic Stack including Kibana. 3. Upgrade the Elastic Artifact Registry and ensure the latest Elastic Agent binaries are available. 4. Upgrade the on-premise Fleet Server. 5. In Fleet, issue an upgrade for all the Elastic Agents. ## Enable air-gapped mode for Fleet Set the following property in Kibana to enable air-gapped mode in Fleet. This allows Fleet to intelligently skip certain requests or operations that shouldn’t be attempted in air-gapped environments. ```yaml xpack.fleet.isAirGapped: true ``` ## Configure Elastic Agents to download a PGP/GPG key from Fleet Server Starting from version 8.9.0, when Elastic Agent tries to perform an upgrade, it first verifies the binary signature with the key bundled in the agent. This process has a backup mechanism that will use the key coming from `https://artifacts.elastic.co/GPG-KEY-elastic-agent` instead of the one it already has. In an air-gapped environment, an Elastic Agent which doesn’t have access to a PGP/GPG key from `https://artifacts.elastic.co/GPG-KEY-elastic-agent` would fail to be upgraded. For versions 8.9.0 to 8.10.3, you can resolve this problem following the steps described in the associated [known issue](https://www.elastic.co/guide/en/fleet/8.9/release-notes-8.9.0.html#known-issues-8.9.0) documentation. Starting in version 8.10.4, you can resolve this problem by configuring Elastic Agents to download the PGP/GPG key from Fleet Server. Starting in version 8.10.4, Elastic Agent will: 1. Verify the binary signature with the key bundled in the agent. 2. If the verification doesn’t pass, the agent will download the PGP/GPG key from `https://artifacts.elastic.co/GPG-KEY-elastic-agent` and verify it. 3. If that verification doesn’t pass, the agent will download the PGP/GPG key from Fleet Server and verify it. 4. If that verification doesn’t pass, the upgrade is blocked. By default, Fleet Server serves Elastic Agents with the key located in `FLEETSERVER_BINARY_DIR/elastic-agent-upgrade-keys/default.pgp`. The key is served through the Fleet Server endpoint `GET /api/agents/upgrades/{{major}}.{minor}.{{patch}}/pgp-public-key`. If there isn’t a `default.pgp` key in the `FLEETSERVER_BINARY_DIR/elastic-agent-upgrade-keys/default.pgp` directory, Fleet Server instead will attempt to retrieve a PGP/GPG key from the URL that you can specify with the `server.pgp.upstream_url` setting. You can prevent Fleet from downloading the PGP/GPG key from `server.pgp.upstream_url` by manually downloading it from `https://artifacts.elastic.co/GPG-KEY-elastic-agent` and storing it at `FLEETSERVER_BINARY_DIR/elastic-agent-upgrade-keys/default.pgp`. To set a custom URL for Fleet Server to access a PGP/GPG key and make it available to Elastic Agents: 1. In Kibana, go to **Management > Fleet > Agent policies**. 2. Select a policy for the agents that you want to upgrade. 3. On the policy page, in the **Actions** menu for the Fleet Server integration, select **Edit integration**. 4. In the Fleet Server settings section expand **Change defaults** and **Advanced options**. 5. In the **Custom fleet-server configurations** field, add the setting `server.pgp.upstream_url` with the full URL where the PGP/GPG key can be accessed. For example: ```yaml server.pgp.upstream_url: ``` The setting `server.pgp.upstream_url` must point to a web server hosting the PGP/GPG key, which must be reachable by the host where Fleet Server is installed. Note that: - `server.pgp.upstream_url` may be specified as an `http` endpoint (instead of `https`). - For an `https` endpoint, the CA for Fleet Server to connect to `server.pgp.upstream_url` must be trusted by Fleet Server using the `--certificate-authorities` setting that is used globally for Elastic Agent. ## Use a proxy server to access the Elastic Package Registry By default Kibana downloads package metadata and content from the public Elastic Package Registry at [epr.elastic.co](https://epr.elastic.co/). If you can route traffic to the public endpoint of the Elastic Package Registry through a network gateway, set the following property in Kibana to use a proxy server: ```yaml xpack.fleet.registryProxyUrl: your-nat-gateway.corp.net ``` For more information, refer to [Using a proxy server with Elastic Agent and Fleet](https://www.elastic.co/elastic/docs-builder/docs/3028/reference/fleet/fleet-agent-proxy-support). ### Allow network access to the public Elastic Package Registry If you need to configure firewall rules to allow network access to [epr.elastic.co](https://epr.elastic.co/), allowlist the domain rather than specific IP addresses whenever possible, as IP addresses might change without notice. If allowlisting the domain is not an option, these are the current IP addresses used for the Elastic Package Registry: - **IPv4**: `34.120.127.130` - **IPv6**: `2600:1901:0:1d7::` You can verify the current IP addresses using DNS lookup tools like `nslookup` or `dig`. ## Host your own Elastic Package Registry The Elastic Package Registry packages include signatures used in [package verification](https://www.elastic.co/elastic/docs-builder/docs/3028/reference/fleet/package-signatures). By default, Fleet uses the Elastic public GPG key to verify package signatures. If you ever need to change this GPG key, use the `xpack.fleet.packageVerification.gpgKeyPath` setting in [`kibana.yml`](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/stack-settings). For more information, refer to [Fleet settings](https://docs-v3-preview.elastic.dev/elastic/docs-builder/docs/3028/reference/kibana/configuration-reference/fleet-settings). If routing traffic through a proxy server is not an option, you can host your own Elastic Package Registry. If using Elastic Cloud on Kubernetes, you can follow the instructions in [Deploy Elastic Package Registry on Elastic Cloud on Kubernetes](https://www.elastic.co/elastic/docs-builder/docs/3028/deploy-manage/deploy/cloud-on-k8s/package-registry). The Elastic Package Registry can be deployed and hosted onsite using one of the available Docker images. These docker images include the Elastic Package Registry and a selection of packages. There are different distributions available: - 9.3.2 (recommended): *docker.elastic.co/package-registry/distribution:9.3.2* - Selection of packages from the production repository released with Elastic Stack 9.3.2. - lite-9.3.2: *docker.elastic.co/package-registry/distribution:lite-9.3.2* - Subset of the most commonly used packages from the production repository released with Elastic Stack 9.3.2. This image is a good candidate to start using Fleet in air-gapped environments. - production: *docker.elastic.co/package-registry/distribution:production* - Packages available in the production registry ([[https://epr.elastic.co](https://epr.elastic.co)](https://epr.elastic.co)). This image is updated every time a new version of a package gets published. - lite: *docker.elastic.co/package-registry/distribution:lite* - Subset of the most commonly used packages available in the production registry ([[https://epr.elastic.co](https://epr.elastic.co)](https://epr.elastic.co)). This image is updated every time a new version of a package gets published. To update the distribution image, re-pull the image and then restart the docker container. Every distribution contains packages that can be used by different versions of the Elastic Stack. The Elastic Package Registry API exposes a Kibana version constraint that allows for filtering packages that are compatible with a particular version. These steps use the standard Docker CLI, but you can create a Kubernetes manifest based on this information. These images can also be used with other container runtimes compatible with Docker images. 1. Pull the Docker image from the public Docker registry: ```sh docker pull docker.elastic.co/package-registry/distribution:9.3.2 ``` 2. Save the Docker image locally: ```sh docker save -o package-registry-9.3.2.tar docker.elastic.co/package-registry/distribution:9.3.2 ``` Check the image size to ensure that you have enough disk space. 3. Transfer the image to the air-gapped environment and load it: ```sh docker load -i package-registry-9.3.2.tar ``` 4. Run the Elastic Package Registry: ```sh docker run -it -p 8080:8080 docker.elastic.co/package-registry/distribution:9.3.2 ``` 5. (Optional) You can monitor the health of your Elastic Package Registry with requests to the root path: ```sh docker run -it -p 8080:8080 \ --health-cmd "curl -f -L http://127.0.0.1:8080/health" \ docker.elastic.co/package-registry/distribution:9.3.2 ``` You can use the `/health` or `/health?ready=true` API endpoints to check if EPR is ready to serve requests. When either endpoint returns a `200` HTTP status code, the service is ready to handle requests. ### Connect Kibana to your hosted Elastic Package Registry Use the `xpack.fleet.registryUrl` property in the Kibana config to set the URL of your hosted package registry. For example: ```yaml xpack.fleet.registryUrl: "http://package-registry.corp.net:8080" ``` ### TLS configuration of the Elastic Package Registry You can configure the Elastic Package Registry to listen on a secure HTTPS port using TLS. For example, given a key and a certificate pair available in `/etc/ssl`, you can start the Elastic Package Registry listening on the 443 port using the following command: ```sh docker run -it -p 443:443 \ -v /etc/ssl/package-registry.key:/etc/ssl/package-registry.key:ro \ -v /etc/ssl/package-registry.crt:/etc/ssl/package-registry.crt:ro \ -e EPR_ADDRESS=0.0.0.0:443 \ -e EPR_TLS_KEY=/etc/ssl/package-registry.key \ -e EPR_TLS_CERT=/etc/ssl/package-registry.crt \ docker.elastic.co/package-registry/distribution:9.3.2 ``` The Elastic Package Registry supports TLS versions from 1.0 to 1.3. The minimum version accepted can be configured with `EPR_TLS_MIN_VERSION`, it defaults to 1.0. If you want to restrict the supported versions from 1.2 to 1.3, you can use `EPR_TLS_MIN_VERSION=1.2`. ### Using custom CA certificates If you are using self-signed certificates or certificates issued by a custom Certificate Authority (CA), you need to set the file path to your CA in the `NODE_EXTRA_CA_CERTS` environment variable in the Kibana startup files. ```text NODE_EXTRA_CA_CERTS="/etc/kibana/certs/ca-cert.pem" ``` ## Host your own artifact registry for binary downloads Elastic Agents must be able to access the Elastic Artifact Registry to download binaries during upgrades. By default Elastic Agents download artifacts from `https://artifacts.elastic.co/downloads/`. Refer to the available [script and Nginx config](/elastic/docs-builder/docs/3028/deploy-manage/deploy/self-managed/air-gapped-install#air-gapped-elastic-artifact-registry). To make binaries available in an air-gapped environment, you can host your own custom artifact registry, and then configure Elastic Agents to download binaries from it. 1. Create a custom artifact registry in a location accessible to your Elastic Agents: 1. Download the latest release artifacts from the public Elastic Artifact Registry at `https://artifacts.elastic.co/downloads/`. For example, the following cURL commands download all the artifacts that may be needed to upgrade Elastic Agents running on Linux x86_64. You may replace `x86_64` with `arm64` for the ARM64 version. The exact list depends on which integrations you’re using. Make sure to also download the corresponding sha512, and PGP Signature (.asc) files for each binary. These are used for file integrity validations during installations and upgrades. ```shell curl -O https://artifacts.elastic.co/downloads/apm-server/apm-server-9.3.2-linux-x86_64.tar.gz curl -O https://artifacts.elastic.co/downloads/apm-server/apm-server-9.3.2-linux-x86_64.tar.gz.sha512 curl -O https://artifacts.elastic.co/downloads/apm-server/apm-server-9.3.2-linux-x86_64.tar.gz.asc curl -O https://artifacts.elastic.co/downloads/beats/auditbeat/auditbeat-9.3.2-linux-x86_64.tar.gz curl -O https://artifacts.elastic.co/downloads/beats/auditbeat/auditbeat-9.3.2-linux-x86_64.tar.gz.sha512 curl -O https://artifacts.elastic.co/downloads/beats/auditbeat/auditbeat-9.3.2-linux-x86_64.tar.gz.asc curl -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.3.2-linux-x86_64.tar.gz curl -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.3.2-linux-x86_64.tar.gz.sha512 curl -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.3.2-linux-x86_64.tar.gz.asc curl -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.3.2-linux-x86_64.tar.gz curl -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.3.2-linux-x86_64.tar.gz.sha512 curl -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-9.3.2-linux-x86_64.tar.gz.asc curl -O https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-9.3.2-linux-x86_64.tar.gz curl -O https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-9.3.2-linux-x86_64.tar.gz.sha512 curl -O https://artifacts.elastic.co/downloads/beats/heartbeat/heartbeat-9.3.2-linux-x86_64.tar.gz.asc curl -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-9.3.2-linux-x86_64.tar.gz curl -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-9.3.2-linux-x86_64.tar.gz.sha512 curl -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-9.3.2-linux-x86_64.tar.gz.asc curl -O https://artifacts.elastic.co/downloads/beats/osquerybeat/osquerybeat-9.3.2-linux-x86_64.tar.gz curl -O https://artifacts.elastic.co/downloads/beats/osquerybeat/osquerybeat-9.3.2-linux-x86_64.tar.gz.sha512 curl -O https://artifacts.elastic.co/downloads/beats/osquerybeat/osquerybeat-9.3.2-linux-x86_64.tar.gz.asc curl -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-9.3.2-linux-x86_64.tar.gz curl -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-9.3.2-linux-x86_64.tar.gz.sha512 curl -O https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-9.3.2-linux-x86_64.tar.gz.asc curl -O https://artifacts.elastic.co/downloads/cloudbeat/cloudbeat-9.3.2-linux-x86_64.tar.gz curl -O https://artifacts.elastic.co/downloads/cloudbeat/cloudbeat-9.3.2-linux-x86_64.tar.gz.sha512 curl -O https://artifacts.elastic.co/downloads/cloudbeat/cloudbeat-9.3.2-linux-x86_64.tar.gz.asc curl -O https://artifacts.elastic.co/downloads/endpoint-dev/endpoint-security-9.3.2-linux-x86_64.tar.gz curl -O https://artifacts.elastic.co/downloads/endpoint-dev/endpoint-security-9.3.2-linux-x86_64.tar.gz.sha512 curl -O https://artifacts.elastic.co/downloads/endpoint-dev/endpoint-security-9.3.2-linux-x86_64.tar.gz.asc curl -O https://artifacts.elastic.co/downloads/fleet-server/fleet-server-9.3.2-linux-x86_64.tar.gz curl -O https://artifacts.elastic.co/downloads/fleet-server/fleet-server-9.3.2-linux-x86_64.tar.gz.sha512 curl -O https://artifacts.elastic.co/downloads/fleet-server/fleet-server-9.3.2-linux-x86_64.tar.gz.asc curl -O https://artifacts.elastic.co/downloads/prodfiler/pf-host-agent-9.3.2-linux-x86_64.tar.gz curl -O https://artifacts.elastic.co/downloads/prodfiler/pf-host-agent-9.3.2-linux-x86_64.tar.gz.sha512 curl -O https://artifacts.elastic.co/downloads/prodfiler/pf-host-agent-9.3.2-linux-x86_64.tar.gz.asc curl -O https://artifacts.elastic.co/downloads/prodfiler/pf-elastic-collector-9.3.2-linux-x86_64.tar.gz curl -O https://artifacts.elastic.co/downloads/prodfiler/pf-elastic-collector-9.3.2-linux-x86_64.tar.gz.sha512 curl -O https://artifacts.elastic.co/downloads/prodfiler/pf-elastic-collector-9.3.2-linux-x86_64.tar.gz.asc curl -O https://artifacts.elastic.co/downloads/prodfiler/pf-elastic-symbolizer-9.3.2-linux-x86_64.tar.gz curl -O https://artifacts.elastic.co/downloads/prodfiler/pf-elastic-symbolizer-9.3.2-linux-x86_64.tar.gz.sha512 curl -O https://artifacts.elastic.co/downloads/prodfiler/pf-elastic-symbolizer-9.3.2-linux-x86_64.tar.gz.asc ``` 2. On your HTTP file server, group the artifacts into directories and sub-directories that follow the same convention used by the Elastic Artifact Registry: ```shell //--- ``` Where: - `` is in the format `beats/elastic-agent`, `fleet-server`, `endpoint-dev`, and so on. - `` is in the format `elastic-agent`, `endpoint-security`, or `fleet-server` and so on. - `arch-package-type` is in the format `linux-x86_64`, `linux-arm64`, `windows_x86_64`, `darwin_x86_64`, or darwin_aarch64`. - If you’re using the DEB package manager: - The 64bit variant has the format `--amd64.deb`. - The aarch64 variant has the format `--arm64.deb`. - If you’re using the RPM package manager: - The 64bit variant has a format `--x86_64.rpm`. - The aarch64 variant has a format `--aarch64.rpm`. - If you’re ever in doubt, visit the [Elastic Agent download page](https://www.elastic.co/downloads/elastic-agent) to see what URL the various binaries are downloaded from. - Make sure you have a plan or automation in place to update your artifact registry when new versions of Elastic Agent are available. 2. Add the agent binary download location to Fleet settings: 1. Open **Fleet → Settings**. 2. Under **Agent Binary Download**, click **Add agent binary source** to add the location of your artifact registry. For more detail about these settings, refer to [Agent binary download settings](/elastic/docs-builder/docs/3028/reference/fleet/fleet-settings#fleet-agent-binary-download-settings). If you want all Elastic Agents to download binaries from this location, set it as the default. 3. If your artifact registry is not the default, edit your agent policies to override the default: 1. Go to **Fleet → Agent policies** and click the policy name to edit it. 2. Click **Settings**. 3. Under **Agent Binary Download**, select your artifact registry. When you trigger an upgrade for any Elastic Agents enrolled in the policy, the binaries are downloaded from your artifact registry instead of the public repository. **Not using Fleet?** For standalone Elastic Agents, you can set the binary download location under `agent.download.sourceURI` in the [`elastic-agent.yml`](https://www.elastic.co/elastic/docs-builder/docs/3028/reference/fleet/elastic-agent-reference-yaml) file, or run the [`elastic-agent upgrade`](/elastic/docs-builder/docs/3028/reference/fleet/agent-command-reference#elastic-agent-upgrade-command) command with the `--source-uri` flag specified.