S3 ResourcePath

The basic syntax for using an S3 ResourcePath is:

ResourcePath("s3://bucketname/key")

Configuration

To access files hosted in S3 using ResourcePath, the environment must be configured to choose an S3 service and provide credentials for authentication.

Choosing an S3 service

By default, the library will attempt to use AWS S3. To connect to another S3 service, set the environment variable S3_ENDPOINT_URL to the HTTP URL where the service is hosted. For example, for Google Cloud Storage:

S3_ENDPOINT_URL=https://storage.googleapis.com

Authentication credentials

Authentication for S3 services can be configured in a variety of ways. The simplest and most common is to provide an access key ID and secret. This can be accomplished using environment variables:

AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

or a credentials file. By default, the credentials file is located at ~/.aws/credentials. This path can be changed by setting the environment variable AWS_SHARED_CREDENTIALS_FILE. A basic credentials file looks like this:

[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Using multiple S3 services or sets of credentials

If you need to connect to more than one S3 service, you can configure additional S3 “profiles”. The profile name is added to the S3 URI as a profile_name@ prefix to the bucket name. For example, a ResourcePath URI for an S3 profile called myprofile looks like:

ResourcePath("s3://myprofile@bucket/key")

Each profile must set an environment variable to identify the S3 service it should connect to. The variable name is in the form LSST_RESOURCES_S3_PROFILE_<profile_name>, for example:

LSST_RESOURCES_S3_PROFILE_myprofile=https://private-s3-service.example

The credentials for each profile should be configured by adding additional profile blocks to the credentials file. For example:

# Will be used for S3 URIs without an explicit profile name, e.g.
# s3://bucket/key
[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# Will be used for S3 URIs with a profile name of "myprofile", e.g.
# s3://myprofile@bucket/key
[myprofile]
aws_access_key_id=AKIAIOSFSDAD7EXAMPLE2
aws_secret_access_key=wJakjASDWREMI/FAMDENG/bPxRfiCYEXAMPLEKEY2