--- title: Monitor resources on private networks description: To monitor resources on private networks you can either: Allow Elastic’s global managed infrastructure to access your private endpoints.Use Elastic Agent... url: https://www.elastic.co/elastic/docs-builder/docs/4300/solutions/observability/synthetics/monitor-resources-on-private-networks products: - Elastic Cloud Serverless - Elastic Documentation - Elastic Observability applies_to: - Elastic Cloud Serverless: Generally available - Elastic Stack: Generally available --- # Monitor resources on private networks To monitor resources on private networks you can either: - Allow Elastic’s global managed infrastructure to access your private endpoints. - Use Elastic Agent to create a Private Location. Private Locations using Elastic Agent require only outbound connections from your network, while allowing Elastic’s global managed infrastructure to access a private endpoint requires inbound access, therefore posing an additional risk that you must consider. ## Allow access to your private network To give Elastic’s global managed infrastructure access to a private endpoint, use IP address filtering, HTTP authentication, or both. To grant access using IP, use [this list of egress IPs](https://manifest.synthetics.elastic-cloud.com/v1/ip-ranges.json). The addresses and locations on this list might change, so automating updates to filtering rules is recommended. IP filtering alone will allow all users of Elastic’s global managed infrastructure access to your endpoints. If this is a concern, consider adding additional protection through user/password authentication with a proxy like nginx. ## Monitor using a private agent Private Locations allow you to run monitors from your own premises. Before running a monitor on a Private Location, you’ll need to: - [Set up Fleet Server and Elastic Agent](#synthetics-private-location-fleet-agent). - [Connect Fleet to the Elastic Stack](#synthetics-private-location-connect) and enroll an Elastic Agent in Fleet. - [Add a Private Location](#synthetics-private-location-add) in the Synthetics UI. A Private Location is classic or scalable depending on how many Elastic Agents share its agent policy: - **Classic**: The agent policy runs on a single Elastic Agent, and that agent runs every monitor assigned to the location. - Elastic Stack: Planned Elastic Cloud Serverless: Generally available **Scalable**: With an [Enterprise subscription](https://www.elastic.co/subscriptions) or an active trial, enroll multiple Elastic Agents on the same agent policy. Kibana automatically distributes each monitor to exactly one agent, with failover when an agent becomes unhealthy. You don't need to turn on anything extra when you add the Private Location. Refer to [Scale a Private Location across multiple Elastic Agents](#synthetics-private-location-scalable) for more information. Private Locations running through Elastic Agent must have a direct connection to Elasticsearch. Do not configure any ingest pipelines, or output using Logstash as this will prevent Synthetics from working properly and [is not supported](https://www.elastic.co/elastic/docs-builder/docs/4300/solutions/observability/synthetics/support-matrix). ## Set up Fleet Server and Elastic Agent Start by setting up Fleet Server and Elastic Agent: - **Set up Fleet Server**: If you are using Elastic Cloud, Fleet Server will already be provided and you can skip this step. To learn more, refer to [Set up Fleet Server](https://www.elastic.co/elastic/docs-builder/docs/4300/reference/fleet/fleet-server). - **Create an agent policy**: For more information on agent policies and creating them, refer to [Elastic Agent policy](/elastic/docs-builder/docs/4300/reference/fleet/agent-policy#create-a-policy). The Elastic Agent must be enrolled in Fleet. Private Locations cannot be set up using standalone Elastic Agents. When you create the agent policy: - Decide how many Elastic Agents run the policy: - Run a classic Private Location on a single Elastic Agent. Classic Private Locations do not distribute tests across Elastic Agents, so running the same policy on more than one Elastic Agent can produce duplicate or missing tests. To add capacity on one Elastic Agent, refer to [Scaling Private Locations](#synthetics-private-location-scaling). - Elastic Stack: Planned Elastic Cloud Serverless: Generally available To run tests on multiple Elastic Agents that share one agent policy, use a [scalable Private Location](#synthetics-private-location-scalable). - Elastic Stack: Generally available since 9.4 Elastic Cloud Serverless: Generally available Create the agent policy in the same Kibana space as the Private Location. Cross-space agent policies are not supported. To run monitors in more than one space, create a separate agent policy in each space. ## Connect to the Elastic Stack or your Observability Serverless project After setting up Fleet, you’ll connect Fleet to the Elastic Stack or your Observability Serverless project and enroll an Elastic Agent in Fleet. Elastic provides Docker images that you can use to run Fleet and an Elastic Agent more easily. Additional installation methods can be found on the [Elastic Agent installation guide](https://www.elastic.co/elastic/docs-builder/docs/4300/reference/fleet/install-elastic-agents). For running browser monitors on Private Locations, you *must* use one of the `elastic-agent-complete` Docker image variants in a containerized environment. The standard Elastic Agent variant only supports TCP, ICMP, and HTTP monitors. To pull the Docker image run: ```sh # Supports all monitor types: TCP, ICMP, HTTP and Browser docker pull docker.elastic.co/elastic-agent/elastic-agent-complete:9.5.5 ``` ```shell # Supports TCP, ICMP and HTTP monitors docker pull docker.elastic.co/elastic-agent/elastic-agent:9.5.5 ``` You can download and install a specific version of the Elastic Stack by replacing `{{version.stack}}` with the version number you want. For example, you can replace `{{version.stack}}` with 9.0.0. Then enroll and run an Elastic Agent. You’ll need an enrollment token and the URL of the Fleet Server. You can use the default enrollment token for your policy or create new policies and [enrollment tokens](https://www.elastic.co/elastic/docs-builder/docs/4300/reference/fleet/fleet-enrollment-tokens) as needed. For more information on running Elastic Agent with Docker, refer to [Run Elastic Agent in a container](https://www.elastic.co/elastic/docs-builder/docs/4300/reference/fleet/elastic-agent-container). ```shell docker run \ --env FLEET_ENROLL=1 \ --env FLEET_URL={fleet_server_host_url} \ --env FLEET_ENROLLMENT_TOKEN={enrollment_token} \ --cap-add=NET_RAW \ --cap-add=SETUID \ --rm docker.elastic.co/elastic-agent/elastic-agent-complete:9.5.5 ``` The `elastic-agent-complete` container, when running as Synthetics Private Locations, requires additional capabilities to operate correctly. Ensure `NET_RAW` and `SETUID` are enabled on the container. ```shell docker run \ --env FLEET_ENROLL=1 \ --env FLEET_URL={fleet_server_host_url} \ --env FLEET_ENROLLMENT_TOKEN={enrollment_token} \ --cap-add=NET_RAW \ --cap-add=SETUID \ --rm docker.elastic.co/elastic-agent/elastic-agent:9.5.5 ``` The `elastic-agent` container, when running as Synthetics Private Locations, requires additional capabilities to operate correctly. Ensure `NET_RAW` and `SETUID` are enabled on the container. You may need to set other environment variables. Learn how in [Elastic Agent environment variables guide](https://www.elastic.co/elastic/docs-builder/docs/4300/reference/fleet/agent-environment-variables). ## Add a Private Location When the Elastic Agent is running you can add a new Private Location in the UI: 1. Find `Synthetics` in the [global search field](https://www.elastic.co/elastic/docs-builder/docs/4300/explore-analyze/find-and-organize/find-apps-and-objects). 2. Go to **Settings**. 3. Go to the **Private Locations** tab. 4. Click **Create location**. 5. Give your new location a unique _Location name_. 6. Select the _Agent policy_ you created above. Elastic Stack: Generally available since 9.4 Elastic Cloud Serverless: Generally available Only agent policies that exist in the **current space** are available for selection. 7. (Optional) In _Tags_ select [tags](https://www.elastic.co/elastic/docs-builder/docs/4300/explore-analyze/find-and-organize/tags) to assign to this location. 8. (Optional) In _Spaces_ specify the [spaces](https://www.elastic.co/elastic/docs-builder/docs/4300/deploy-manage/manage-spaces) where this location will be available. 9. Click **Save**. If you upgraded from a version that allowed cross-space agent policy selection, any private location that references an agent policy from a different space will show **Policy not found in the current space** in the UI. To resolve this, reassign the private location to an agent policy in the same space, or create a new agent policy in the current space and re-enroll the Elastic Agent. Using custom CAs for synthetics browser tests in private locations is not currently possible without a workaround. To learn more, refer to the GitHub issue [elastic/synthetics#717](https://github.com/elastic/synthetics/issues/717). ## Monitor integration health Elastic Stack: Generally available since 9.4 Elastic Cloud Serverless: Generally available Synthetics automatically detects when private location monitors have broken Fleet integrations and surfaces health status in the UI with actionable recovery options. This can happen if an agent policy or Fleet package policy is deleted after a monitor is configured, or if the referenced private location no longer exists. ### Failure types Each monitor is evaluated per location. The following failure types can occur, listed in priority order — only the first matching issue per location is reported, since it is the root cause: | Status | Cause | Recovery | |--------------------------|---------------------------------------------------------------------|-------------------| | `missing_location` | The monitor references a private location that no longer exists. | Manual | | `missing_agent_policy` | The agent policy associated with the private location was deleted. | Manual | | `missing_package_policy` | The Fleet package policy for this monitor/location pair is missing. | Automatic (Reset) | ### Health status in the UI Health status surfaces in three places: - **Monitor list**: An orange warning icon appears next to unhealthy monitors. Hover over the icon to see per-location details on why the monitor is affected. - **Monitor details page**: A warning callout lists the affected locations. A **Reset monitor** button is shown when the failure type is `missing_package_policy`, which recreates the missing Fleet resources. - **Private Locations settings**: Each location shows a badge with the count of unhealthy monitors. Clicking the badge opens a popover listing the affected monitors and a **Reset monitors** button that bulk-resets all recoverable monitors at that location. ### Reset behavior Only monitors with a `missing_package_policy` status can be auto-reset. The **Reset** action recreates the missing Fleet package policy, restoring the monitor to a healthy state. Monitors with `missing_agent_policy` or `missing_location` statuses are excluded from auto-reset because the underlying infrastructure must be restored before Fleet resources can be recreated. When you initiate a reset, the confirmation dialog lists which monitors will be reset and which will be skipped and why. To resolve failures that require manual intervention: - **`missing_agent_policy`**: Recreate the agent policy and re-enroll an Elastic Agent, then update the affected private location under **Settings > Private Locations** to reference the new agent policy. - **`missing_location`**: [Re-create the private location](#synthetics-private-location-add) or edit the affected monitors to remove or replace the deleted location. ## Scale a Private Location across multiple Elastic Agents - Elastic Cloud Serverless: Generally available - Elastic Stack: Planned A scalable Private Location runs its monitors on a pool of Elastic Agents that share one agent policy. Each monitor runs on one agent in the pool. When an agent becomes unhealthy, its monitors move to the healthy agents. When you add an agent, some monitors move to it. Enroll a second Elastic Agent on the same agent policy when one Elastic Agent can't run all the monitors in the location, or when monitors must keep running if an agent fails. Keep a single Elastic Agent (a classic Private Location) when it has enough capacity and you don't need failover. ### Requirements - An [Enterprise subscription](https://www.elastic.co/subscriptions) or an active trial. Without one, multiple Elastic Agents on the same agent policy each run every monitor in the location, producing duplicate results. If your subscription level drops below Enterprise, Synthetics clears the agent assignments on its next rebalancing pass and the same duplicate behavior returns until you upgrade again. To avoid duplicate results in the meantime, [remove all but one agent](#synthetics-private-location-scalable-disable). - Elastic Agents enrolled in Fleet on the same agent policy. Failover requires at least two agents. The agent policy must be in the same Kibana space as the Private Location. - For browser monitors, every agent in the pool must use an `elastic-agent-complete` Docker image. Refer to [Connect to the Elastic Stack](#synthetics-private-location-connect) for more information. - Because a monitor can move to any agent in the pool, configure every agent the same way, including network access to monitored hosts, environment variables that your monitors read, and the host timezone. - Each agent needs the CPU and RAM described in [Scaling Private Locations](#synthetics-private-location-scaling). Browser and lightweight concurrency limits apply to each agent separately. Optionally, if you want to view memory and CPU usage for each agent in the Synthetics UI, add the System integration to the agent policy. ### Set up a scalable Private Location To set up a scalable Private Location: 1. [Create an agent policy](#synthetics-private-location-fleet-agent) in the same Kibana space as the Private Location. 2. [Enroll two or more Elastic Agents](#synthetics-private-location-connect) on that agent policy. 3. [Add a Private Location](#synthetics-private-location-add) that uses the agent policy. With an Enterprise subscription or an active trial, Kibana automatically distributes monitors across all enrolled agents. On the **Private Locations** tab, a scalable location shows a **Scalable** badge with the number of enrolled agents. Expand the location's row to view the monitors, health, and resource usage of each agent. Refer to [Private Locations settings](/elastic/docs-builder/docs/4300/solutions/observability/synthetics/configure-settings#synthetics-settings-private-locations) for more information. To add agents to an existing location, enroll additional Elastic Agents on its agent policy. Kibana automatically redistributes monitors to include the new agents. ### Remove agents To reduce a scalable location to a single agent, unenroll all but one Elastic Agent from its agent policy. Kibana reassigns the remaining monitors to the surviving agent when the removed agents are detected as unhealthy. With one agent left, the location becomes a classic Private Location: the remaining agent runs every monitor assigned to the location. You can't move an agent's monitors to other agents before you unenroll it. When you unenroll an agent, its monitors move only after it's detected as unhealthy, and they might miss some scheduled runs in the meantime. ### How monitors are assigned to agents Synthetics assigns each monitor to one agent in the pool. The assignment balances the estimated memory cost of the monitors across agents: a browser monitor counts roughly as much as 50 lightweight monitors, and agents with more total RAM receive a larger share. Because assignment is weighted by memory cost rather than monitor count, the number of monitors per agent can look uneven even when memory is balanced. Current CPU and memory usage appear in the Synthetics UI but aren't used for assignments. About once a minute, Synthetics checks the health of every agent in the pool and adjusts assignments: - **Failover**: When an agent stops checking in to Fleet and stops sending results, Synthetics marks it as unhealthy after about 90 seconds and moves its monitors to healthy agents, usually within 2–3 minutes of the last check-in. During that move, a monitor can briefly run on two agents, which produces duplicate results. - **Recovery and new agents**: After a recovered or newly enrolled agent has been healthy for 3 minutes, Synthetics moves only as many monitors to it as needed to balance the pool. - **No healthy agents**: If no agent in the pool is healthy, monitors keep their assignments and don't run until an agent recovers. To stop these adjustments for every scalable Private Location, turn off **Rebalance private location shards** in [**Settings → Advanced**](/elastic/docs-builder/docs/4300/solutions/observability/synthetics/configure-settings#synthetics-settings-advanced). Turning off **Rebalance private location shards** removes the agent assignment from every monitor in every scalable Private Location. Each monitor then runs on every agent enrolled on its location's agent policy, which duplicates test runs. When you turn the switch back on, Synthetics reassigns the monitors right away. ## Scaling Private Locations By default, each Elastic Agent in a Private Location allows two simultaneous browser tests and an unlimited number of lightweight checks. If more than two browser tests are scheduled to run at the same time on one agent, some of them are delayed. You can change these limits using the environment variables `SYNTHETICS_LIMIT_{{TYPE}}`, where `{{TYPE}}` is one of `BROWSER`, `HTTP`, `TCP`, and `ICMP`, for the container running the Elastic Agent Docker image. Elastic Stack: Planned Elastic Cloud Serverless: Generally available In a scalable Private Location, these limits and the CPU and RAM requirements apply to each agent. To add capacity by adding agents instead of resources, refer to [Scale a Private Location across multiple Elastic Agents](#synthetics-private-location-scalable). ### CPU and RAM requirements It is critical to allocate enough memory and CPU capacity to handle configured limits. Resource requirements will vary depending on simultaneous workload and monitor complexity: **For browser monitors**: Start by allocating at least 2 GiB of memory and two cores _per browser instance_ to ensure consistent performance and avoid out-of-memory errors. Then adjust as needed. **For tcp, http, icmp**: Much less memory is needed, start by allocating at least 512MiB of memory and two cores _globally_. While this will be enough to run many lightweight monitors, it is recommended to track the resource usage and adjust accordingly. Example: For a private location expected to run 2 concurrent browser monitors and 100 HTTP checks, the recommended allocation is 2 * (2 GiB + 2 vCPU) + (512 MiB + 2 vCPU) => 4,5 GiB + 6 vCPU. ### Known limitations on vertical scaling - A single private location will not scale beyond 10,000 monitors. Exceeding this number will result in agent degradation and inconsistent execution, regardless of the resources allocated. Elastic Stack: Planned Elastic Cloud Serverless: Generally available In a scalable Private Location, this limit applies per location, not per agent — adding more agents does not raise it. - Many Synthetics monitors, or monitors with complex configurations, can cause the check-in payload to exceed the default 1 MiB `checkin_limit.max_body_byte_size` limit on Fleet Server. When this happens, check-ins are rejected and agents appear offline or unhealthy in the Fleet UI even though monitors are executing successfully. To resolve this, increase the `server.limits.checkin_limit.max_body_byte_size` setting on your self-managed Fleet Server. Refer to [Advanced Fleet Server options](/elastic/docs-builder/docs/4300/reference/fleet/fleet-server-scalability#fleet-server-configuration) for configuration details and an example. If you're facing one of these scenarios, it is likely that the private location has grown too large and needs to be split into smaller locations, each allotted a portion of the original location monitors. ## Next steps Now you can add monitors to your Private Location in [the Synthetics UI](https://www.elastic.co/elastic/docs-builder/docs/4300/solutions/observability/synthetics/create-monitors-ui) or using the [Elastic Synthetics library’s `push` method](https://www.elastic.co/elastic/docs-builder/docs/4300/solutions/observability/synthetics/create-monitors-with-projects).