webDAV ResourcePath¶
The basic syntax for using a webDAV ResourcePath
is:
ResourcePath("davs://host.example.org:1234/path/to/file")
Configuration¶
Interacting with WebDAV servers may require specific client configuration. That is for instance the case when the server exposes a host certificate which is not issued by a certificate authority trusted by the system where the client runs. Another case is when the server requires the client presents its identity for authentication purposes.
In order to configure ResourcePath
to interact with one or
more WebDAV servers which require specific configuration, you must initialize
the environment variable LSST_RESOURCES_WEBDAV_CONFIG
to point to a
configuration YAML file. This configuration file must contain the settings for
each of the webDAV servers which host any of the
ResourcePath
objects with davs://
or dav://
scheme
you need to instantiate.
The general structure of the configuration file is as follows:
# These settings apply to all ResourcePath objects hosted by
# server 'host.example.org:1234'.
- base_url: "davs://host.example.org:1234/"
user_cert: "${X509_USER_PROXY}"
trusted_authorities: "/etc/grid-security/certificates"
# These settings apply to all ResourcePath objects hosted by
# server 'host.domain.org:5432'.
- base_url: "davs://host.domain.org:5432/"
token: "${HOME}/.token"
# These settings apply to all ResourcePath objects hosted by
# server 'webdav.example.org:9876'.
- base_url: "davs://webdav.example.org:9876/"
timeout_connect: 30.0
timeout_read: 120.0
retries: 5
For each WebDAV endpoint you can specify several settings that all instances
of ResourcePath
hosted at that specific endpoint will use.
If no settings are found in the configuration for a given endpoint, sensible
defaults values are used (see below).
Please note that if your WebDAV endpoint requires client authentication you must have an entry for that endpoint in your configuration file. Currently two client authentication mechanisms are supported: either a client X.509 certificate (and associated private key) or a token. If both are specified, token authentication is preferred.
Similarly, if the WebDAV server exposes a host certificate that is not issued by a certification authority trusted by the operating system where your client runs, you must add an entry for that endpoint to the configuration file which includes the local path to the file or directory where the trusted authorities can be found.
Configuration settings¶
These are the settings you can specify in the configuration file for each WebDAV endpoint:
user_cert
Path to the client certificate the WebDAV client must present to the server for authentication purposes.
If not specified, no client certificate is presented to the server.
This setting is ignored if
token
is specified (see below).user_key
Path to the private key associated to the client certificate the WebDAV client must present to the server for authentication purposes.
If not specified but
user_cert
is specified, it is assumed that the private key and the user certificate are both in the same file specified in the value foruser_cert
.trusted_authorities
Path to a local directory or certificate bundle file where the certificates of the trusted certificate authorities can be found. Those certificates will be used by the client of the WebDAV endpoint to verify the server’s host certificate.
If not specified, the certificates trusted by the system are used. The server’s host certificate is always verified.
token
Value of the token the WebDAV client must sent to the server for authentication purposes. The value of this entry may be the token itself or the path to a local file where the value of the token can be found. It is however discouraged to specify the token value directly in the configuration file. Instead store the value of the token in a protected file and specify the path to that file as the value of this entry.
If
token
is specified that mechanism is preferred for authentication of client requests, even ifuser_cert
is also specified.If you specify a value for
token
which is the path to a local file which contains the actual token, that file must be protected for reading and writing only by its owner. The contents of that file is automatically reloaded when every time it is modified.timeout_connect
Timeout in seconds to establish a network connection with the remote server.
Default: 10 seconds (
float
).timeout_read
Timeout in seconds to read the response to a request sent to the server. This is total time for reading both the headers and the response body. It must be large enough to allow for upload and download of files of typical size.
Default: 300 seconds (
float
).retries
Number of times to retry requests before failing. Retry happens only under certain conditions.
Default: 3 (
int
).retry_backoff_min
Minimal retry backoff (in seconds) for the client to compute the wait time before retrying a request. A value in the interval [
retry_backoff_min
,retry_backoff_max
] is randomly selected as the backoff factor every time a request is retried.Default: 1.0 seconds (
float
).retry_backoff_max
Maximum retry backoff (in seconds) for the client to compute the wait time before retrying a request. A value in the interval [
retry_backoff_min
,retry_backoff_max
] is randomly selected as the backoff factor every time a request is retried.Default: 3.0 seconds (
float
).buffer_size
Size of the buffer (in mebibytes, i.e. 1024*1024 bytes) the WebDAV client of this endpoint will use when sending requests and receiving responses.
Default: 5 mebibytes (
int
).persistent_connections_frontend
Maximum number of network connections to persist against each one of the hosts in the server frontend.
Default: 50 (
int
).persistent_connections_backend
Maximum number of network connections to persist against each one of the hosts in the server backend.
Default: 100 (
int
).enable_fsspec
If specified, expose a fsspec-compatible, read-only file system for accessing a
ResourcePath
object.Default:
true
(boolean
).request_checksum
If specified, the WebDAV client will request the server to compute a checksum of the file contents every time a file is uploaded.
Note that it is the server decision to compute the checksum. Some servers simply ignore that request.
Accepted values:
adler32
,md5
,sha-256
,sha-512
.collect_memory_usage
If specified, memory usage data is collected when running in debug mode. Collecting memory usage data is expensive, so this setting should not be used in production.
Accepted values:
true
,false
.Default:
false
(boolean
).
Configuration Examples¶
These are examples of configuration files for dCache and XRootD WebDAV servers.
Set the environment variable LSST_RESOURCES_WEBDAV_CONFIG
to point to
your configuration file:
export LSST_RESOURCES_WEBDAV_CONFIG="${HOME}/.lsst/dav_conf.yaml"
dCache¶
Typical configuration settings for a dCache endpoint are:
- base_url: "davs://dcache.example.org:2880/"
trusted_authorities: "/etc/grid-security/certificates"
user_cert: "${X509_USER_PROXY}"
request_checksum: "md5"
The example above uses X.509 grid proxy client authentication. If you prefer to use a token use instead:
- base_url: "davs://dcache.example.org:2880/"
trusted_authorities: "/etc/grid-security/certificates"
request_checksum: "md5"
token: "${HOME}/.lsst/dcache_token"
and ensure the file at path ${HOME}/.lsst/dcache_token
contains the
client authentication token and is only readable and writable by you.
In this example we configure ResourcePath
to request the
dCache server to compute and record the MD5 checksum when a file is uploaded,
in addition to the ADLER32 checksum dCache always computes and records for
each file.
XRootD¶
Typical configuration settings for a XRootD endpoint are:
- base_url: "davs://xrootd.example.org:1094/"
trusted_authorities: "/etc/grid-security/certificates"
user_cert: "${X509_USER_PROXY}"
If you prefer to use a token for client authentication use the token
setting instead (see dCache example above).