Private connectivity with AWS PrivateLink

You can use AWS PrivateLink to establish a secure connection for your Elastic Cloud Hosted deployments and Elastic Cloud Serverless projects to communicate with other AWS services. AWS routes the PrivateLink traffic within the AWS data center and never exposes it to the public internet.

AWS PrivateLink connects your Virtual Private Cloud (VPC) to the AWS-hosted services that you use, treating them as if they were in your VPC. You can create and use VPC endpoints to securely access AWS-hosted services.

You can also optionally filter traffic to your deployments or projects by creating virtual private connection (VPC) filters as part of your private connection policy in Elastic Cloud. This limits traffic to your deployment or project to the VPC specified in the policy, as well as any other policies applied to the deployment or project.

To learn how private connection policies impact your deployment or project, refer to Network security policies in Elastic Cloud.

Before you begin, review the following considerations:

Private connectivity with AWS PrivateLink is supported only in AWS regions.

AWS interface virtual private connection (VPC) endpoints are configured for one or more availability zones (AZ). In some regions, our VPC endpoint service is not present in all the possible AZs that a region offers. You can only choose AZs that are common on both sides. As the names of AZs (for example us-east-1a) differ between AWS accounts, the following list of AWS regions shows the ID (e.g. use1-az4) of each available AZ for the service.

Refer to interface endpoint availability zone considerations for more details.

Transport client is not supported over PrivateLink connections.

Some metadata might differ between Elastic Cloud Hosted and Elastic Cloud Serverless, even if the region is the same.

The process of setting up a private connection with AWS PrivateLink is split between the AWS console and the Elastic Cloud UI. These are the high-level steps:

After you create your private connection policy, you can edit, disassociate, or delete it.

Before you begin, you should ensure your VPC endpoint is in all availability zones supported by Elastic Cloud for the region and resource type. Placing your VPC endpoint in all supported Elastic Cloud availability zones for the region improves throughput and resiliency when connecting over PrivateLink.

For Elastic Cloud Hosted deployments, if your VPC is not in all supported availability zones, traffic can become imbalanced, saturating some coordinating nodes while underutilizing others, which might impact performance.

You can find the zone name to zone ID mapping with AWS CLI:

$ aws ec2 describe-availability-zones --region us-east-1 | jq -c '.AvailabilityZones[] | { id: .ZoneId, name: .ZoneName } ' | sort
{"id":"use1-az1","name":"us-east-1c"}
{"id":"use1-az2","name":"us-east-1e"}
{"id":"use1-az3","name":"us-east-1d"}
{"id":"use1-az4","name":"us-east-1a"}
{"id":"use1-az5","name":"us-east-1f"}
{"id":"use1-az6","name":"us-east-1b"}
		

The mapping will be different for your region. Our production VPC Service for us-east-1 is located in use1-az2, use1-az4, use1-az6. We need to create the VPC Endpoint for the preceding mapping in at least one of us-east-1e, us-east-1a, us-east-1b.

1. Create a VPC endpoint in your VPC using the service name for your region.

   Refer to the AWS documentation for additional details on creating a VPC interface endpoint to an endpoint service.

   Select PrivateLink Ready partner services as the endpoint type. Use the service name for your region as the Service name.

   
   ×

   The security group for the endpoint should, at minimum, allow for inbound connectivity from your instances' CIDR range on ports 443 and 9243. Security groups for the instances should allow for outbound connectivity to the endpoint on ports 443 and 9243.

   Tip

   You can also create a cross-region endpoint. Refer to Setting up an cross-region Private Link connection.

2. Create a DNS record.

   1. Create a Private hosted zone.

      For Elastic Cloud Hosted, refer to the Private hosted zone domain name column in the PrivateLink service names and aliases table for the name of the zone.

      For Elastic Cloud Serverless, to view the service metadata for your selected region, start to create a new private connection policy for the region and find the Domain name in the Service metadata dropdown.

      For example, for Elastic Cloud Hosted deployments in us-east-1, use vpce.us-east-1.aws.elastic-cloud.com as the zone domain name. For Elastic Cloud Serverless deployments in the same region, use private.us-east-1.aws.elastic-cloud.com.

      Don’t forget to associate the zone with your VPC.

      Tip

      Private hosted zone domain names differ between Elastic Cloud Hosted and Elastic Cloud Serverless, even if the region is the same.

      
      ×

   2. Create a DNS CNAME alias pointing to the PrivateLink endpoint. Add the record to a private DNS zone in your VPC. Use * as the record name, and the VPC endpoint DNS name as a value.

      Refer to the AWS documentation for details on creating a CNAME record which points to your VPC endpoint DNS name.

      
      ×

After you create your VPC endpoint and DNS entries, check that you are able to reach your deployment or project over PrivateLink.

Use the following URL structure. This URL is built from endpoint information retrieved from your Elastic deployment or project and the private hosted zone domain name that you registered.

https://{{alias}}.{{product}}.{{private_hosted_zone_domain_name}}
		

For example:

https://my-deployment-d53192.es.vpce.us-east-1.aws.elastic-cloud.com
		

To test the connection:

1. If needed, find the endpoint of an application in your deployment or project:

   1. Log in to the Elastic Cloud Console.

   2. Under Hosted deployments, find your deployment.

   Tip

   If you have many deployments, you can instead go to the Hosted deployments (Elastic Cloud Hosted) page. On that page, you can narrow your deployments by name, ID, or choose from several other filters.

   3. Select Manage.
   4. In the deployment overview, under Applications, find the application that you want to test.
   5. Click Copy endpoint. The value looks something like the following:

   https://my-deployment-d53192.es.us-east-1.aws.elastic-cloud.com
   		

   In this endpoint, my-deployment-d53192 is an alias, and es is the product you want to access within your deployment.

   1. Log in to the Elastic Cloud Console.

   2. Under Serverless projects, find your projects.

      Tip

      If you have many projects, you can instead go to the Serverless projects page. On that page, you can narrow your projects by name, ID, or choose from several other filters.

   3. Select Manage.

   4. In the project overview, under Application endpoints, cluster and component IDs, find the application that you want to test.

   5. Under Public endpoint, you can view your endpoint URL. It looks something like the following:

      https://my-project-d53192.es.us-east-1.aws.elastic-cloud.com
      		

   In this endpoint, my-project-d53192 is an alias, and es is the product you want to access within your project.

2. Test the setup using the following cURL command. Pass the username and password for a user that has access to the cluster. Make sure to replace the URL with your deployment or project's endpoint information and the private hosted zone domain name that you registered.

   Request

   $ curl -v https://my-deployment-d53192.es.vpce.us-east-1.aws.elastic-cloud.com -u {username}:{password}
   		

   $ curl -v https://my-project-d53192.es.private.us-east-1.aws.elastic-cloud.com -u {username}:{password}
   		

   Response

   * Server certificate:
   *  subject: CN=*.us-east-1.aws.elastic-cloud.com
   *  SSL certificate verify ok.
   ..
       < HTTP/1.1 200 OK
   ..
   {
       "name" : "instance-0000000009",
       "cluster_name" : "fb7e805e5cfb4931bdccc4f3cb591f5f",
       "cluster_uuid" : "2cTHeCQYS2a0iH7YnQHrIQ",
       "version" : { ... },
       "tagline" : "You Know, for Search"
   }
   		

The connection is established, and a valid certificate is presented to the client. Elastic responds, in the case of the Elasticsearch endpoint, with basic information about the cluster.

After you test your PrivateLink connection, you can create a private connection policy in Elastic Cloud.

Private connection policies are optional for AWS PrivateLink. After the VPC endpoint and DNS record are created, private connectivity is established.

Creating a private connection policy and associating it with your deployments or projects allows you to filter traffic to your deployments or projects using VPC filters.

Follow these high-level steps to add a private connection policy that can be associated with your deployments or projects.

1. Optional: Find your VPC endpoint ID.
2. Create a private connection policy using the VPC endpoint.
3. Associate the VPC endpoint with your deployment or project.

The VPC endpoint ID is only required if you want to filter traffic to your deployment or project using VPC filters.

You can find your VPC endpoint ID in the AWS console:

×

Create a new private connection policy.

1. Log in to the Elastic Cloud Console.
2. From any deployment or project on the home page, select Manage.
3. From the left navigation menu, select Access and security > Network security.

4. Select Private connection.

5. Select the resource type that the private connection will be applied to.

6. Select the cloud provider and region for the private connection.

   Tip

   Private connection policies are bound to a single resource type and region, and can be assigned only to resources with the same resource type and in the same region. If you want to associate a policy with multiple resource types or resources in multiple regions, then you have to recreate the policy for all applicable resource types and regions.

7. Under Connectivity, select Privatelink.

8. Optional: Under VPC filter, enter your VPC endpoint ID. You should only specify a VPC filter if you want to filter traffic to your deployment or project.

   If you don't specify a VPC filter, then the private connection policy acts only as a record that you've established private connectivity between AWS and Elastic in the applicable region.

   Tip

   You can apply multiple policies to a single deployment or project. The policies can be of different types. In case of multiple policies, traffic can match any associated policy to be forwarded to the resource. If none of the policies match, the request is rejected with 403 Forbidden.

   Learn more about how network security policies affect your deployment or project.

9. Optional: Under Apply to resources, associate the new private connection policy with one or more deployments or projects. If you specified a VPC filter, then after you associate the filter with a resource, it starts filtering traffic.

10. To automatically attach this private connection policy to new resources of this type, select Apply by default.

11. Click Create.

12. (Optional) You can claim your VPC endpoint ID, so that no other organization is able to use it in a private connection policy.

The next step is to associate the policy with your deployment or project.

You can associate a private connection policy with your deployment or project from the policy's settings, or from your deployment's or project's settings.

1. Log in to the Elastic Cloud Console.
2. From any deployment or project on the home page, select Manage.
3. From the left navigation menu, select Access and security > Network security.

5. Find the policy you want to edit.
6. Under Apply to resources, associate the policy with one or more deployments or projects.
7. Click Update to save your changes.

For traffic to connect with the deployment or project over AWS PrivateLink, the client making the request needs to be located within the VPC where you’ve created the VPC endpoint. You can also set up network traffic to flow through the originating VPC from somewhere else, such as another VPC or VPN from your corporate network. This assumes that the VPC endpoint and the DNS record are also available within that context. Check your service provider documentation for setup instructions.

Use the following URL structure. This URL is built from endpoint information retrieved from your Elastic deployment or project and the private hosted zone domain name that you registered.

https://{{alias}}.{{product}}.{{private_hosted_zone_domain_name}}
		

For example:

https://my-deployment-d53192.es.vpce.us-east-1.aws.elastic-cloud.com
		

To access the deployment or project:

1. If needed, find the endpoint of an application in your deployment or project:

   1. Log in to the Elastic Cloud Console.

   2. Under Hosted deployments, find your deployment.

   Tip

   If you have many deployments, you can instead go to the Hosted deployments (Elastic Cloud Hosted) page. On that page, you can narrow your deployments by name, ID, or choose from several other filters.

   3. Select Manage.
   4. In the deployment overview, under Applications, find the application that you want to test.
   5. Click Copy endpoint. The value looks something like the following:

   https://my-deployment-d53192.es.us-east-1.aws.elastic-cloud.com
   		

   In this endpoint, my-deployment-d53192 is an alias, and es is the product you want to access within your deployment.

   1. Log in to the Elastic Cloud Console.

   2. Under Serverless projects, find your projects.

      Tip

      If you have many projects, you can instead go to the Serverless projects page. On that page, you can narrow your projects by name, ID, or choose from several other filters.

   3. Select Manage.

   4. In the project overview, under Application endpoints, cluster and component IDs, find the application that you want to test.

   5. Under Public endpoint, you can view your endpoint URL. It looks something like the following:

      https://my-project-d53192.es.us-east-1.aws.elastic-cloud.com
      		

   In this endpoint, my-project-d53192 is an alias, and es is the product you want to access within your project.

2. Send a request:

   Request

   $ curl -v https://my-deployment-d53192.es.vpce.us-east-1.aws.elastic-cloud.com -u {username}:{password}
   		

   $ curl -v https://my-project-d53192.es.private.us-east-1.aws.elastic-cloud.com -u {username}:{password}
   		

   Response

   * Server certificate:
   *  subject: CN=*.us-east-1.aws.elastic-cloud.com
   *  SSL certificate verify ok.
   ..
       < HTTP/1.1 200 OK
   ..
   {
       "name" : "instance-0000000009",
       "cluster_name" : "fb7e805e5cfb4931bdccc4f3cb591f5f",
       "cluster_uuid" : "2cTHeCQYS2a0iH7YnQHrIQ",
       "version" : { ... },
       "tagline" : "You Know, for Search"
   }
   		

If you are using AWS PrivateLink together with Fleet, and enrolling the Elastic Agent with a private connection URL, you need to configure Fleet Server to use and propagate the AWS PrivateLink URL by updating the Fleet Server hosts field in the Fleet settings section of Kibana. Otherwise, Elastic Agent will reset to use a default address instead of the AWS PrivateLink URL.

The URL needs to follow this pattern:

https://{{fleet_component_ID_or_deployment_alias}}.fleet.{{private_hosted_zone_domain_name}}:443`
		

Similarly, the Elasticsearch host needs to be updated to propagate the private connection URL. The Elasticsearch URL needs to follow this pattern:

https://{{elasticsearch_cluster_ID_or_deployment_alias}}.es.{{private_hosted_zone_domain_name}}:443
		

The settings xpack.fleet.agents.fleet_server.hosts and xpack.fleet.outputs that are needed to enable this configuration in Kibana are not available in the Kibana settings in Elastic Cloud.

AWS supports cross-region PrivateLink as described on the AWS blog.

This means your deployment or project on Elastic Cloud can be in a different region than the PrivateLink endpoints or the clients that consume the deployment or project endpoints.

In this example, region 1 contains your VPC endpoint and region 2 is the region where your deployment or project is hosted.

1. Begin to create your VPC endpoint in region 1, as described in Create your VPC endpoint and DNS entries in AWS. In the service settings, do the following:

   • In the Service name field, enter the VPC service name for region 2.
   • Select Enable Cross Region endpoint and select region 2 from the Select a region drop-down list.

2. Create a private connection policy in the region where your deployment or project is hosted (region 2), and associate it with your deployment or project.

3. Test the connection from a VM or client in region 1 to your Private Link endpoint, and it should be able to connect to your Elasticsearch deployment or project hosted in region 2.

After you create your private connection policy, you can edit it, remove it from your deployment or project, or delete it.

You can edit a policy's name, description, VPC endpoint ID, and more.

1. Log in to the Elastic Cloud Console.
2. From any deployment or project on the home page, select Manage.
3. From the left navigation menu, select Access and security > Network security.

1. Find the policy you want to edit, then click the Edit button.
2. Click Update to save your changes.

If you want to remove a specific policy from a deployment or project, or delete the policy, then you need to disconnect it from any associated deployments or projects first. You can do this from the policy's settings, or from your deployment or project's settings. To remove an association through the UI:

1. Log in to the Elastic Cloud Console.
2. From any deployment or project on the home page, select Manage.
3. From the left navigation menu, select Access and security > Network security.

5. Find the policy you want to edit, then click the Edit button.
6. Under Apply to resources, click the x beside the resource that you want to disconnect.
7. Click Update to save your changes.

If you need to remove a policy, you must first remove any associations with deployments or projects.

To delete a policy:

1. Log in to the Elastic Cloud Console.
2. From any deployment or project on the home page, select Manage.
3. From the left navigation menu, select Access and security > Network security.

4. Find the policy you want to edit, then click the Delete icon. The icon is inactive if there are deployments or projects associated with the policy.