Monitor resources on private networks

To monitor resources on private networks you can either:

• Allow Elastic’s global managed infrastructure to access your private endpoints.
• Use Elastic Agent to create a Private Location.

Private Locations using Elastic Agent require only outbound connections from your network, while allowing Elastic’s global managed infrastructure to access a private endpoint requires inbound access, therefore posing an additional risk that you must consider.

To give Elastic’s global managed infrastructure access to a private endpoint, use IP address filtering, HTTP authentication, or both.

To grant access using IP, use this list of egress IPs. The addresses and locations on this list might change, so automating updates to filtering rules is recommended. IP filtering alone will allow all users of Elastic’s global managed infrastructure access to your endpoints. If this is a concern, consider adding additional protection through user/password authentication with a proxy like nginx.

Private Locations allow you to run monitors from your own premises. Before running a monitor on a Private Location, you’ll need to:

• Set up Fleet Server and Elastic Agent.
• Connect Fleet to the Elastic Stack and enroll an Elastic Agent in Fleet.
• Add a Private Location in the Synthetics UI.

A Private Location is classic or scalable depending on how many Elastic Agents share its agent policy:

• Classic: The agent policy runs on a single Elastic Agent, and that agent runs every monitor assigned to the location.
• Scalable: With an Enterprise subscription or an active trial, enroll multiple Elastic Agents on the same agent policy. Kibana automatically distributes each monitor to exactly one agent, with failover when an agent becomes unhealthy. You don't need to turn on anything extra when you add the Private Location. Refer to Scale a Private Location across multiple Elastic Agents for more information.

Start by setting up Fleet Server and Elastic Agent:

• Set up Fleet Server: If you are using Elastic Cloud, Fleet Server will already be provided and you can skip this step. To learn more, refer to Set up Fleet Server.
• Create an agent policy: For more information on agent policies and creating them, refer to Elastic Agent policy.

When you create the agent policy:

• Decide how many Elastic Agents run the policy:
  • Run a classic Private Location on a single Elastic Agent. Classic Private Locations do not distribute tests across Elastic Agents, so running the same policy on more than one Elastic Agent can produce duplicate or missing tests. To add capacity on one Elastic Agent, refer to Scaling Private Locations.
  • To run tests on multiple Elastic Agents that share one agent policy, use a scalable Private Location.
• Create the agent policy in the same Kibana space as the Private Location. Cross-space agent policies are not supported. To run monitors in more than one space, create a separate agent policy in each space.

After setting up Fleet, you’ll connect Fleet to the Elastic Stack or your Observability Serverless project and enroll an Elastic Agent in Fleet.

Elastic provides Docker images that you can use to run Fleet and an Elastic Agent more easily. Additional installation methods can be found on the Elastic Agent installation guide.

To pull the Docker image run:

# Supports all monitor types: TCP, ICMP, HTTP and Browser
docker pull docker.elastic.co/elastic-agent/elastic-agent-complete:9.5.5
		

# Supports TCP, ICMP and HTTP monitors
docker pull docker.elastic.co/elastic-agent/elastic-agent:9.5.5
		

You can download and install a specific version of the Elastic Stack by replacing {{version.stack}} with the version number you want. For example, you can replace {{version.stack}} with 9.0.0.

Then enroll and run an Elastic Agent. You’ll need an enrollment token and the URL of the Fleet Server. You can use the default enrollment token for your policy or create new policies and enrollment tokens as needed.

For more information on running Elastic Agent with Docker, refer to Run Elastic Agent in a container.

docker run \
  --env FLEET_ENROLL=1 \
  --env FLEET_URL={fleet_server_host_url} \
  --env FLEET_ENROLLMENT_TOKEN={enrollment_token} \
  --cap-add=NET_RAW \
  --cap-add=SETUID \
  --rm docker.elastic.co/elastic-agent/elastic-agent-complete:9.5.5
		

docker run \
  --env FLEET_ENROLL=1 \
  --env FLEET_URL={fleet_server_host_url} \
  --env FLEET_ENROLLMENT_TOKEN={enrollment_token} \
  --cap-add=NET_RAW \
  --cap-add=SETUID \
  --rm docker.elastic.co/elastic-agent/elastic-agent:9.5.5
		

When the Elastic Agent is running you can add a new Private Location in the UI:

1. Find Synthetics in the global search field.

2. Go to Settings.

3. Go to the Private Locations tab.

4. Click Create location.

5. Give your new location a unique Location name.

6. Select the Agent policy you created above.

   Only agent policies that exist in the current space are available for selection.

7. (Optional) In Tags select tags to assign to this location.

8. (Optional) In Spaces specify the spaces where this location will be available.

9. Click Save.

Using custom CAs for synthetics browser tests in private locations is not currently possible without a workaround. To learn more, refer to the GitHub issue elastic/synthetics#717.

Synthetics automatically detects when private location monitors have broken Fleet integrations and surfaces health status in the UI with actionable recovery options. This can happen if an agent policy or Fleet package policy is deleted after a monitor is configured, or if the referenced private location no longer exists.

Each monitor is evaluated per location. The following failure types can occur, listed in priority order — only the first matching issue per location is reported, since it is the root cause:

Health status surfaces in three places:

• Monitor list: An orange warning icon appears next to unhealthy monitors. Hover over the icon to see per-location details on why the monitor is affected.
• Monitor details page: A warning callout lists the affected locations. A Reset monitor button is shown when the failure type is missing_package_policy, which recreates the missing Fleet resources.
• Private Locations settings: Each location shows a badge with the count of unhealthy monitors. Clicking the badge opens a popover listing the affected monitors and a Reset monitors button that bulk-resets all recoverable monitors at that location.

Only monitors with a missing_package_policy status can be auto-reset. The Reset action recreates the missing Fleet package policy, restoring the monitor to a healthy state.

Monitors with missing_agent_policy or missing_location statuses are excluded from auto-reset because the underlying infrastructure must be restored before Fleet resources can be recreated. When you initiate a reset, the confirmation dialog lists which monitors will be reset and which will be skipped and why.

To resolve failures that require manual intervention:

• missing_agent_policy: Recreate the agent policy and re-enroll an Elastic Agent, then update the affected private location under Settings > Private Locations to reference the new agent policy.
• missing_location: Re-create the private location or edit the affected monitors to remove or replace the deleted location.

A scalable Private Location runs its monitors on a pool of Elastic Agents that share one agent policy. Each monitor runs on one agent in the pool. When an agent becomes unhealthy, its monitors move to the healthy agents. When you add an agent, some monitors move to it.

Enroll a second Elastic Agent on the same agent policy when one Elastic Agent can't run all the monitors in the location, or when monitors must keep running if an agent fails. Keep a single Elastic Agent (a classic Private Location) when it has enough capacity and you don't need failover.

• An Enterprise subscription or an active trial. Without one, multiple Elastic Agents on the same agent policy each run every monitor in the location, producing duplicate results. If your subscription level drops below Enterprise, Synthetics clears the agent assignments on its next rebalancing pass and the same duplicate behavior returns until you upgrade again. To avoid duplicate results in the meantime, remove all but one agent.
• Elastic Agents enrolled in Fleet on the same agent policy. Failover requires at least two agents. The agent policy must be in the same Kibana space as the Private Location.
• For browser monitors, every agent in the pool must use an elastic-agent-complete Docker image. Refer to Connect to the Elastic Stack for more information.
• Because a monitor can move to any agent in the pool, configure every agent the same way, including network access to monitored hosts, environment variables that your monitors read, and the host timezone.
• Each agent needs the CPU and RAM described in Scaling Private Locations. Browser and lightweight concurrency limits apply to each agent separately.

Optionally, if you want to view memory and CPU usage for each agent in the Synthetics UI, add the System integration to the agent policy.

To set up a scalable Private Location:

1. Create an agent policy in the same Kibana space as the Private Location.
2. Enroll two or more Elastic Agents on that agent policy.
3. Add a Private Location that uses the agent policy. With an Enterprise subscription or an active trial, Kibana automatically distributes monitors across all enrolled agents.

On the Private Locations tab, a scalable location shows a Scalable badge with the number of enrolled agents. Expand the location's row to view the monitors, health, and resource usage of each agent. Refer to Private Locations settings for more information.

To add agents to an existing location, enroll additional Elastic Agents on its agent policy. Kibana automatically redistributes monitors to include the new agents.

To reduce a scalable location to a single agent, unenroll all but one Elastic Agent from its agent policy. Kibana reassigns the remaining monitors to the surviving agent when the removed agents are detected as unhealthy. With one agent left, the location becomes a classic Private Location: the remaining agent runs every monitor assigned to the location.

You can't move an agent's monitors to other agents before you unenroll it. When you unenroll an agent, its monitors move only after it's detected as unhealthy, and they might miss some scheduled runs in the meantime.

Synthetics assigns each monitor to one agent in the pool. The assignment balances the estimated memory cost of the monitors across agents: a browser monitor counts roughly as much as 50 lightweight monitors, and agents with more total RAM receive a larger share. Because assignment is weighted by memory cost rather than monitor count, the number of monitors per agent can look uneven even when memory is balanced. Current CPU and memory usage appear in the Synthetics UI but aren't used for assignments.

About once a minute, Synthetics checks the health of every agent in the pool and adjusts assignments:

• Failover: When an agent stops checking in to Fleet and stops sending results, Synthetics marks it as unhealthy after about 90 seconds and moves its monitors to healthy agents, usually within 2–3 minutes of the last check-in. During that move, a monitor can briefly run on two agents, which produces duplicate results.
• Recovery and new agents: After a recovered or newly enrolled agent has been healthy for 3 minutes, Synthetics moves only as many monitors to it as needed to balance the pool.
• No healthy agents: If no agent in the pool is healthy, monitors keep their assignments and don't run until an agent recovers.

To stop these adjustments for every scalable Private Location, turn off Rebalance private location shards in Settings → Advanced.

By default, each Elastic Agent in a Private Location allows two simultaneous browser tests and an unlimited number of lightweight checks. If more than two browser tests are scheduled to run at the same time on one agent, some of them are delayed. You can change these limits using the environment variables SYNTHETICS_LIMIT_{{TYPE}}, where {{TYPE}} is one of BROWSER, HTTP, TCP, and ICMP, for the container running the Elastic Agent Docker image.

In a scalable Private Location, these limits and the CPU and RAM requirements apply to each agent. To add capacity by adding agents instead of resources, refer to Scale a Private Location across multiple Elastic Agents.

It is critical to allocate enough memory and CPU capacity to handle configured limits. Resource requirements will vary depending on simultaneous workload and monitor complexity:

For browser monitors: Start by allocating at least 2 GiB of memory and two cores per browser instance to ensure consistent performance and avoid out-of-memory errors. Then adjust as needed. For tcp, http, icmp: Much less memory is needed, start by allocating at least 512MiB of memory and two cores globally. While this will be enough to run many lightweight monitors, it is recommended to track the resource usage and adjust accordingly.

Example: For a private location expected to run 2 concurrent browser monitors and 100 HTTP checks, the recommended allocation is 2 * (2 GiB + 2 vCPU) + (512 MiB + 2 vCPU) => 4,5 GiB + 6 vCPU.

• A single private location will not scale beyond 10,000 monitors. Exceeding this number will result in agent degradation and inconsistent execution, regardless of the resources allocated. In a scalable Private Location, this limit applies per location, not per agent — adding more agents does not raise it.

• Many Synthetics monitors, or monitors with complex configurations, can cause the check-in payload to exceed the default 1 MiB checkin_limit.max_body_byte_size limit on Fleet Server. When this happens, check-ins are rejected and agents appear offline or unhealthy in the Fleet UI even though monitors are executing successfully. To resolve this, increase the server.limits.checkin_limit.max_body_byte_size setting on your self-managed Fleet Server. Refer to Advanced Fleet Server options for configuration details and an example.

If you're facing one of these scenarios, it is likely that the private location has grown too large and needs to be split into smaller locations, each allotted a portion of the original location monitors.

Now you can add monitors to your Private Location in the Synthetics UI or using the Elastic Synthetics library’s push method.