S3 repository for Elasticsearch snapshots

You can use AWS S3 as a repository for Snapshot/Restore.

See this video for a walkthrough of connecting an AWS S3 repository.

To register an S3 repository, specify the type as s3 when creating the repository. The only mandatory repository setting is the bucket name:

				PUT _snapshot/my_s3_repository
					{
  "type": "s3",
  "settings": {
    "bucket": "my-bucket"
  }
}
		

Client settings describe how repositories select an S3 client and how authentication is configured. The PUT request above only specifies the s3 type and bucket, so it implicitly uses the default client, which, by default, also attempts to resolve credentials automatically from the environment if no explicit settings are provided. Client configuration is covered in the following section.

Elasticsearch communicates with S3 through an S3 client. Clients are configured through a combination of secure settings in the Elasticsearch keystore, and standard settings in elasticsearch.yml. Settings use the prefix s3.client.CLIENT_NAME plus a suffix such as access_key. The full set of client settings is listed under S3 repository client settings.

You can define several clients, each with its own settings, using the form s3.client.CLIENT_NAME.SETTING_NAME. When the environment is compatible, Elasticsearch also creates a client named default, and s3 repositories use that client unless the repository setting client selects another name.

default can use automatic credential discovery when available:

• On an EC2 instance, the EC2 Instance Metadata Service (IMDS) can provide temporary credentials for the instance IAM role.
• In an Amazon ECS task, Elasticsearch can use temporary task IAM role credentials.
• On Kubernetes, you can use service account-based authentication with the right setup in the node or pod.

If you do not want to rely on automatic credentials discovery, add explicit keys for a client in the Elasticsearch keystore. A typical choice for the built-in default client is:

• s3.client.default.access_key
• s3.client.default.secret_key
• s3.client.default.session_token (optional)

S3 client settings cover authentication, region and endpoint selection, proxy/network configuration, and connection or retry tuning. For a complete list of all S3 client settings, refer to S3 repository client settings.

The s3 repository type supports a number of settings to customize how data is stored in S3. These can be specified when creating the repository. For example:

				PUT _snapshot/my_s3_repository
					{
  "type": "s3",
  "settings": {
    "bucket": "my-bucket",
    "client": "default"
  }
}
		

1. client must refer to a configured S3 client. If omitted, the default client is used.

Available repository settings define storage placement, snapshot data handling, storage and encryption behavior, throughput limits, and multipart upload tuning. For a complete list of all S3 repository settings, refer to S3 repository settings.

Amazon S3 supports a variety of storage classes, each of which offers different operational characteristics. For instance, some classes cost less per byte stored per month, but cost more per request, and other classes may vary in terms of their availability guarantees.

You can specify the storage class that Elasticsearch uses to store data objects with the storage_class repository setting.

You can also specify the storage class for objects with the data_storage_class and metadata_storage_class settings. For Elasticsearch versions 9.5 and later, we recommended you specify the data_storage_class and metadata_storage_class settings instead of the storage_class setting. To learn about the differences between these settings, refer to the Elasticsearch configuration reference.

Changing the storage_class, data_storage_class, or metadata_storage_class setting on an existing repository only affects the storage class for newly created objects, resulting in a mixed usage of storage classes.

You may use an S3 Lifecycle Policy to adjust the storage class of existing objects in your repository, but you must not transition objects to an unsupported class such as the Glacier classes, and you must not expire objects. If you use a Glacier storage class, or another unsupported storage class, or object expiry, then you may permanently lose access to your repository contents.

You may use the intelligent_tiering storage class to automatically manage the class of objects, but you must not enable the optional Archive Access or Deep Archive Access tiers. If you use these tiers then you may permanently lose access to your repository contents.

For more information about S3 storage classes, see AWS Storage Classes Guide.

In order to restrict the Elasticsearch snapshot process to the minimum required resources, we recommend using Amazon IAM in conjunction with pre-existing S3 buckets. Here is an example policy which will allow the snapshot access to an S3 bucket named "snaps.example.com". This may be configured through the AWS IAM console, by creating a Custom Policy, and using a Policy Document similar to this (changing snaps.example.com to your bucket name).

{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}
		

You may further restrict the permissions by specifying a prefix within the bucket, in this example, named "foo".

{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Condition": {
        "StringLike": {
          "s3:prefix": [
            "foo/*"
          ]
        }
      },
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com/foo/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}
		

The bucket needs to exist to register a repository for snapshots. If you did not create the bucket then the repository registration will fail.

You can also impose similar restrictions using a bucket policy.

Do not impose any additional restrictions on access to your bucket, for instance by using stricter Condition clauses than the ones documented above. Overly-restrictive access policies can prevent Elasticsearch snapshots from working correctly.

If you want to use Kubernetes service accounts for authentication, you need to add a symlink to the $AWS_WEB_IDENTITY_TOKEN_FILE environment variable (which should be automatically set by a Kubernetes pod) in the S3 repository config directory, so the repository can have the read access for the service account (a repository can't read any files outside its config directory). For example:

mkdir -p "${ES_PATH_CONF}/repository-s3"
ln -s $AWS_WEB_IDENTITY_TOKEN_FILE "${ES_PATH_CONF}/repository-s3/aws-web-identity-token-file"
		

If the symlink exists, it will be used by default by all S3 repositories that don't have explicit client credentials.

If you run Elasticsearch on Amazon EKS and want to use EKS Pod Identity for authentication, EKS injects a token file into the pod and sets the AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE environment variable to point at it. However, Elasticsearch is forbidden from reading files at this location for security reasons. To use EKS Pod Identity, add a symlink at ${ES_PATH_CONF}/repository-s3/eks-pod-identity-token which links to the token file, then set the AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE environment variable to the location of this symlink. For example:

mkdir -p "${ES_PATH_CONF}/repository-s3"
ln -s "${AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE}" "${ES_PATH_CONF}/repository-s3/eks-pod-identity-token"
export AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE="${ES_PATH_CONF}/repository-s3/eks-pod-identity-token"
		

The symlink and the file to which it links must be in place before the node starts. Once configured, the token file is used by default by all S3 repositories that don't have explicit client credentials, and it is re-read automatically when EKS rotates it.

AWS instances resolve S3 endpoints to a public IP. If the Elasticsearch instances reside in a private subnet in an AWS VPC then all traffic to S3 will go through the VPC's NAT instance. If your VPC's NAT instance is a smaller instance size (e.g. a t2.micro) or is handling a high volume of network traffic your bandwidth to S3 may be limited by that NAT instance's networking bandwidth limitations. Instead we recommend creating a VPC endpoint that enables connecting to S3 in instances that reside in a private subnet in an AWS VPC. This will eliminate any limitations imposed by the network bandwidth of your VPC's NAT instance.

Instances residing in a public subnet in an AWS VPC will connect to S3 via the VPC's internet gateway and not be bandwidth limited by the VPC's NAT instance.

AWS S3 supports replication of objects, both within a single region and across regions. However, this replication is not compatible with Elasticsearch snapshots.

The objects that Elasticsearch writes to the repository refer to other objects in the repository. Elasticsearch writes objects in a very specific order to ensure that each object only refers to objects which already exist. Likewise, Elasticsearch only deletes an object from the repository after it becomes unreferenced by all other objects. AWS S3 replication will apply operations to the replica repository in a different order from the order in which Elasticsearch applies them to the primary repository, which can cause some objects in replica repositories to refer to other objects that do not exist. This is an invalid state. It may not be possible to recover any data from a repository if it is in this state.

To replicate a repository's contents elsewhere, follow the repository backup process. In particular, you may use the point-in-time restore capability of AWS S3 backups to restore a backup of a snapshot repository to an earlier point in time.

Some storage systems offer an S3-compatible API, and the s3 repository type allows you to use these systems in place of AWS S3. To do so, set the s3.client.CLIENT_NAME.endpoint setting to the storage system's endpoint URL. For example, https://172.17.0.2 or https://172.17.0.2:9000.

By default Elasticsearch communicates with your storage system using HTTPS, and validates the repository's certificate chain using the JVM-wide truststore. Ensure that the JVM-wide truststore includes an entry for your repository. If you wish to use unsecured HTTP communication instead of HTTPS, set s3.client.CLIENT_NAME.endpoint to a URL which starts with http://.

The s3 repository type requires your storage API to be fully compatible with AWS S3. This means it must exactly conform to every aspect of the documented behavior of AWS S3: it must support all the same API endpoints, with all the same parameters, return the same errors in case of failures, and offer consistency, performance, and reliability at least as good as AWS S3 even when accessed concurrently by multiple Elasticsearch nodes. Some storage products, including some from very well-known storage vendors, claim to offer a S3-compatible API but in fact only support a subset of the features of AWS S3. Others offer an API which matches that of AWS S3 but with different behavior, especially when accessed concurrently. Such storage products are not fully S3-compatible. Using a storage product which is not fully S3-compatible puts your snapshot data at risk. If you are using such a system for your snapshots, consider using a shared filesystem repository based on a standardized protocol such as NFS to access your storage system instead.

Ensure that your storage supplier offers a full compatibility guarantee, and work with them to address any incompatibilities you encounter. Don't report Elasticsearch issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using a genuine AWS S3 repository.

You can perform some basic checks of the suitability of your storage system using the repository analysis API. If this API does not complete successfully, or indicates poor performance, then your storage system is not fully compatible with AWS S3 and therefore unsuitable for use as a snapshot repository. However, a successful response from this API does not guarantee full compatibility.

Elastic strives to maintain stability across upgrades. If your storage is fully S3-compatible, upgrades will encounter no storage-related problems. However, if you use the s3 repository type with a storage system which is not fully S3-compatible then you may find that the upgraded version exposes incompatibilities with Amazon S3 which the earlier version did not. Before every release Elasticsearch passes through some checks to detect unexpected changes in its interactions with the Amazon S3 API, but cannot foresee all possible incompatibilities in third-party storage products. Before upgrading your Elasticsearch clusters:

• Contact your storage supplier and confirm that your specific configuration of storage, including its software or firmware version, is fully compatible with the current version of Amazon S3.

• If your storage is known to be incompatible with features of Amazon S3 which were added very recently, consult the Elasticsearch release notes to determine whether those features will be needed after the upgrade. If they are, apply temporary configuration options to work around the incompatibilities as needed until your storage supplier provides fixes.

• In all cases, verify that your storage passes repository analysis in a test cluster running the upgraded version before upgrading any production clusters.

• Don't report Elasticsearch upgrade issues involving storage systems which claim to be S3-compatible unless you can demonstrate that the same issue exists when using Amazon S3.

Amazon adds new features to the S3 API from time to time. Elasticsearch usually waits more than a year before it starts to use new API features, allowing third-party implementations ample time to catch up. In rare cases Elasticsearch adopts a new API feature sooner than a year after its release. This can even happen in a minor or patch release, particularly if the feature strengthens protections of the integrity of your snapshot data. In those cases, whenever possible, Elasticsearch has configuration options to temporarily reinstate the previous behavior. Use of these options can put the integrity of your snapshot data at risk. These options will not be supported beyond the end of the current major series. Use these options only if your storage is not fully compatible with Amazon S3 due to recent changes in the S3 API. Your supplier needs time to release a new fully-compatible version of their product. We therefore recommend you work with your supplier to address all incompatibilities as a matter of urgency. As soon as the incompatibility is fixed by your storage supplier, cease using the temporary configuration options in Elasticsearch.

Amazon also releases new client SDK versions from time to time. Elasticsearch recognizes that client SDK upgrades can inadvertently change how it interacts with the S3 API, and performs such upgrades in a minor or patch release only when they are unavoidable. Usually this occurs when a security vulnerability is discovered in the version of the client SDK in use.

Most storage systems can be configured to log the details of their interaction with Elasticsearch. If you are investigating a suspected incompatibility with AWS S3, it is usually simplest to collect these logs from your storage system and provide them to the supplier of your storage system for further analysis. Contact the supplier of your storage system for advice on how to configure it to log requests sufficiently verbosely for this troubleshooting.

If the incompatibility is not clear from the logs emitted by the storage system, you can enable more granular logging:

				PUT /_cluster/settings
					{
  "persistent": {
    "logger.software.amazon.awssdk.request": "DEBUG"
  }
}
		

				PUT /_cluster/settings
					{
  "persistent": {
    "logger.com.amazonaws.request": "DEBUG"
  }
}
		

When you have finished collecting the logs needed by your supplier, set the logger settings back to null to return to the default logging configuration and deactivate insecure network trace logging again. Refer to Logger and Cluster update settings for more information.

From 9.3.0 onwards the linearizable register implementation for S3 repositories is based on S3's conditional writes using the If-None-Match and If-Match request headers.

If your storage does not support conditional writes then it is not fully S3-compatible. In this case Elasticsearch can fall back to using the multipart upload APIs for its linearizable registers, as long as these APIs have strongly consistent semantics as described below. This lenient behavior will be removed in the next major release of Elasticsearch. Contact the supplier of your storage for further information about conditional writes and the strong consistency of your storage's multipart upload APIs.

In versions before 9.3.0, or if your storage does not support conditional writes, the linearizable register implementation for S3 repositories is based on the strongly consistent semantics of the multipart upload APIs. Elasticsearch first creates a multipart upload to indicate its intention to perform a linearizable register operation. Elasticsearch then lists and cancels all other multipart uploads for the same register. Elasticsearch then attempts to complete the upload. If the upload completes successfully then the compare-and-exchange operation was atomic.