S3 ResourcePath#
The basic syntax for using an S3 ResourcePath is:
ResourcePath("s3://bucketname/key")
Configuration#
To access files hosted in S3 using ResourcePath, the environment must be
configured to choose an S3 service and provide credentials for authentication.
Choosing an S3 service#
By default, the library will attempt to use AWS S3. To connect to another S3
service, set the environment variable S3_ENDPOINT_URL to the HTTP URL where
the service is hosted. For example, for Google Cloud Storage:
S3_ENDPOINT_URL=https://storage.googleapis.com
Authentication credentials#
Authentication for S3 services can be configured in a variety of ways. The simplest and most common is to provide an access key ID and secret. This can be accomplished using environment variables:
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
or a credentials file.
By default, the credentials file is located at ~/.aws/credentials. This
path can be changed by setting the environment variable
AWS_SHARED_CREDENTIALS_FILE. A basic credentials file looks like this:
[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Using multiple S3 services or sets of credentials#
If you need to connect to more than one S3 service, you can configure
additional S3 “profiles”. The profile name is added to the S3 URI as a
profile_name@ prefix to the bucket name. For example, a
ResourcePath URI for an S3 profile called myprofile looks
like:
ResourcePath("s3://myprofile@bucket/key")
Each profile must set an environment variable to identify the S3 service it
should connect to. The variable name is in the form
LSST_RESOURCES_S3_PROFILE_<profile_name>, for example:
LSST_RESOURCES_S3_PROFILE_myprofile=https://private-s3-service.example
The credentials for each profile should be configured by adding additional profile blocks to the credentials file. For example:
# Will be used for S3 URIs without an explicit profile name, e.g.
# s3://bucket/key
[default]
aws_access_key_id=AKIAIOSFODNN7EXAMPLE
aws_secret_access_key=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
# Will be used for S3 URIs with a profile name of "myprofile", e.g.
# s3://myprofile@bucket/key
[myprofile]
aws_access_key_id=AKIAIOSFSDAD7EXAMPLE2
aws_secret_access_key=wJakjASDWREMI/FAMDENG/bPxRfiCYEXAMPLEKEY2