webDAV ResourcePath

The basic syntax for using a webDAV ResourcePath is:

ResourcePath("davs://host.example.org:1234/path/to/file")

Configuration

Interacting with WebDAV servers may require specific client configuration. That is for instance the case when the server exposes a host certificate which is not issued by a certificate authority trusted by the system where the client runs. Another case is when the server requires the client presents its identity for authentication purposes.

In order to configure ResourcePath to interact with one or more WebDAV servers which require specific configuration, you must initialize the environment variable LSST_RESOURCES_WEBDAV_CONFIG to point to a configuration YAML file. This configuration file must contain the settings for each of the webDAV servers which host any of the ResourcePath objects with davs:// or dav:// scheme you need to instantiate.

The general structure of the configuration file is as follows:

# These settings apply to all ResourcePath objects hosted by
# server 'host.example.org:1234'.
- base_url: "davs://host.example.org:1234/"
  user_cert: "${X509_USER_PROXY}"
  trusted_authorities: "/etc/grid-security/certificates"

# These settings apply to all ResourcePath objects hosted by
# server 'host.domain.org:5432'.
- base_url: "davs://host.domain.org:5432/"
  token: "${HOME}/.token"

# These settings apply to all ResourcePath objects hosted by
# server 'webdav.example.org:9876'.
- base_url: "davs://webdav.example.org:9876/"
  timeout_connect: 30.0
  timeout_read: 120.0
  retries: 5

For each WebDAV endpoint you can specify several settings that all instances of ResourcePath hosted at that specific endpoint will use. If no settings are found in the configuration for a given endpoint, sensible defaults values are used (see below).

Please note that if your WebDAV endpoint requires client authentication you must have an entry for that endpoint in your configuration file. Currently two client authentication mechanisms are supported: either a client X.509 certificate (and associated private key) or a token. If both are specified, token authentication is preferred.

Similarly, if the WebDAV server exposes a host certificate that is not issued by a certification authority trusted by the operating system where your client runs, you must add an entry for that endpoint to the configuration file which includes the local path to the file or directory where the trusted authorities can be found.

Configuration settings

These are the settings you can specify in the configuration file for each WebDAV endpoint:

user_cert

Path to the client certificate the WebDAV client must present to the server for authentication purposes.

If not specified, no client certificate is presented to the server.

This setting is ignored if token is specified (see below).

user_key

Path to the private key associated to the client certificate the WebDAV client must present to the server for authentication purposes.

If not specified but user_cert is specified, it is assumed that the private key and the user certificate are both in the same file specified in the value for user_cert.

user_name

Name of the user the webDAV client must use when the server requires basic access authentication. It must be accompaigned by the user_password setting (see below).

This setting is ignored if token is specified (see below).

user_password

Password the client must use to build the basic authentication header when sending requests to the server. The value of this setting can be the password itself or the path to a local file where the password is stored.

If you specify a value for user_password which is the path to a local file (recommended), that file must be protected for reading and writing only by its owner. The contents of that file is automatically reloaded every time it is modified.

This setting is ignored if token is specified (see below) or if user_name is not specified.

trusted_authorities

Path to a local directory or certificate bundle file where the certificates of the trusted certificate authorities can be found. Those certificates will be used by the client of the WebDAV endpoint to verify the server’s host certificate.

If not specified, the certificates trusted by the system are used. The server’s host certificate is always verified.

token

Value of the token the WebDAV client must sent to the server for authentication purposes. The value of this entry may be the token itself or the path to a local file where the value of the token can be found. It is however discouraged to specify the token value directly in the configuration file. Instead store the value of the token in a protected file and specify the path to that file as the value of this entry.

If token is specified that mechanism is preferred for authentication of client requests, even if user_cert is also specified.

If you specify a value for token which is the path to a local file which contains the actual token, that file must be protected for reading and writing only by its owner. The contents of that file is automatically reloaded every time it is modified.

reuse_connection

Configure this webDAV client to reuse the network connection to the server for submitting subsequent requests. By default, the client attempts to keep the network connection to the server open and to reuse it as long as possible. However, the server may decide to close the connection after serving each request.

Generally speaking, reusing secure HTTP connections to the server is beneficial because of the time necessary to establish the underlying TLS connection.

If this setting is set to false, the client will close the network connection after receiving the response to each request it sends to the server.

Default: true (boolean).

timeout_connect

Timeout in seconds to establish a network connection with the remote server.

Default: 10 seconds (float).

timeout_read

Timeout in seconds to read the response to a request sent to the server. This is total time for reading both the headers and the response body. It must be large enough to allow for upload and download of files of typical size.

Default: 300 seconds (float).

retries

Number of times to retry requests before failing. Retry happens only under certain conditions.

Default: 4 (int).

retry_backoff_min

Minimal retry backoff (in seconds) for the client to compute the wait time before retrying a request. A value in the interval [retry_backoff_min, retry_backoff_max] is randomly selected as the backoff factor every time a request is retried.

Default: 1.0 seconds (float).

retry_backoff_max

Maximum retry backoff (in seconds) for the client to compute the wait time before retrying a request. A value in the interval [retry_backoff_min, retry_backoff_max] is randomly selected as the backoff factor every time a request is retried.

Default: 3.0 seconds (float).

buffer_size

Size of the internal buffer (in mebibytes, i.e. 1024*1024 bytes) the WebDAV client of this endpoint will use when sending requests and receiving responses.

Default: 5 mebibytes (int).

block_size

Size of the block (in mebibytes, i.e. 1024*1024 bytes) the WebDAV client of this endpoint will use when making partial read requests. Each partial read request will attempt to retrieve at least this number of bytes unless the size of the file is lower.

Default: 1 mebibytes (int).

persistent_connections_per_host

Maximum number of network connections to persist against each one of the hosts of this endpoint. If more than this number of network connections are needed they will be established and discarded after use.

Default: 20 (int).

enable_fsspec

If specified, expose a fsspec-compatible, read-only file system for accessing a ResourcePath object.

Default: true (boolean).

request_checksum

If specified, the WebDAV client will request the server to compute a checksum of the file contents every time a file is uploaded.

Note that it is the server decision to compute the checksum. Some servers simply ignore that request.

Accepted values: adler32, md5, sha-256, sha-512.

collect_memory_usage

If specified, memory usage data is collected when running in debug mode. Collecting memory usage data is expensive, so this setting should not be used in production.

Accepted values: true, false.

Default: false (boolean).

frontend_base_urls

If specified, the WebDAV client randomly selects one frontend base URL in this list and uses it to send the request. If not specified, all the requests are sent using the base_url of this endpoint. See the examples section below for the accepted syntax.

Accepted values: list of strings.

Configuration Examples

These are examples of configuration files for dCache and XRootD WebDAV servers.

Set the environment variable LSST_RESOURCES_WEBDAV_CONFIG to point to your configuration file:

export LSST_RESOURCES_WEBDAV_CONFIG="${HOME}/.lsst/dav_conf.yaml"

dCache

Typical configuration settings for a dCache endpoint are:

- base_url: "davs://dcache.example.org:2880/"
  trusted_authorities: "/etc/grid-security/certificates"
  user_cert: "${X509_USER_PROXY}"
  request_checksum: "md5"

The example above uses X.509 grid proxy client authentication. If you prefer to use a token use instead:

- base_url: "davs://dcache.example.org:2880/"
  trusted_authorities: "/etc/grid-security/certificates"
  request_checksum: "md5"
  token: "${HOME}/.lsst/dcache_token"

and ensure the file at path ${HOME}/.lsst/dcache_token contains the client authentication token and is only readable and writable by you.

In this example we configure ResourcePath to request the dCache server to compute and record the MD5 checksum when a file is uploaded, in addition to the ADLER32 checksum dCache always computes and records for each file.

You can also specify a set of frontend servers the client can use to send requests to an endpoint:

- base_url: "dav://dcache.example.org:9090/"
  user_name: "myuser"
  user_password: "${HOME}/path/to/protected/password/file"
  frontend_base_urls:
    - "dav://frontend1.example.org:4242/"
    - "dav://frontend2.example.org:4242/"
    - "dav://frontend3.example.org:4242/"

XRootD

Typical configuration settings for a XRootD endpoint are:

- base_url: "davs://xrootd.example.org:1094/"
  trusted_authorities: "/etc/grid-security/certificates"
  user_cert: "${X509_USER_PROXY}"

If you prefer to use a token for client authentication use the token setting instead (see dCache example above).