ResourcePath¶

Bases: object

Convenience wrapper around URI parsers.

Provides access to URI components and can convert file paths into absolute path URIs. Scheme-less URIs are treated as if they are local file system paths and are converted to absolute URIs.

A specialist subclass is created for each supported URI scheme.

Parameters:

uristr, pathlib.Path, urllib.parse.ParseResult, or ResourcePath: URI in string form. Can be scheme-less if referring to a relative path or an absolute path on the local file system.
rootstr or ResourcePath, optional: When fixing up a relative path in a file scheme or if scheme-less, use this as the root. Must be absolute. If None the current working directory will be used. Can be any supported URI scheme. Not used if forceAbsolute is False.
forceAbsolutebool, optional: If True, scheme-less relative URI will be converted to an absolute path using a file scheme. If False scheme-less URI will remain scheme-less and will not be updated to file or absolute path unless it is already an absolute path, in which case it will be updated to a file scheme.
forceDirectorybool or None, optional: If True forces the URI to end with a separator. If False the URI is interpreted as a file-like entity. Default, None, is that the given URI is interpreted as a directory if there is a trailing / or for some schemes the system will check to see if it is a file or a directory.
isTemporarybool, optional: If True indicates that this URI points to a temporary resource. The default is False, unless uri is already a ResourcePath instance and uri.isTemporary is True.

Notes

A non-standard URI of the form file:dir/file.txt is always converted to an absolute file URI.

Attributes Summary

`fragment`	Return the fragment component of the URI.
`isLocal`	If `True` this URI refers to a local file.
`is_root`	Return whether this URI points to the root of the network location.
`netloc`	Return the URI network location.
`ospath`	Return the path component of the URI localized to current OS.
`params`	Return any parameters included in the URI.
`path`	Return the path component of the URI.
`query`	Return any query strings included in the URI.
`quotePaths`	True if path-like elements modifying a URI should be quoted.
`relativeToPathRoot`	Return path relative to network location.
`scheme`	Return the URI scheme.
`transferDefault`	Default mode to use for transferring if `auto` is specified.
`transferModes`	Transfer modes supported by this implementation.
`unquoted_path`	Return path component of the URI with any URI quoting reversed.

Methods Summary

`abspath`()	Return URI using an absolute path.
`as_local`()	Return the location of the (possibly remote) resource as local file.
`basename`()	Return the base name, last element of path, of the URI.
`dirname`()	Return the directory component of the path as a new `ResourcePath`.
`exists`()	Indicate that the resource is available.
`findFileResources`()	Get all the files from a list of values.
`generate_presigned_get_url`(*, ...)	Return a pre-signed URL that can be used to retrieve this resource using an HTTP GET without supplying any access credentials.
`generate_presigned_put_url`(*, ...)	Return a pre-signed URL that can be used to upload a file to this path using an HTTP PUT without supplying any access credentials.
`getExtension`()	Return the extension(s) associated with this URI path.
`geturl`()	Return the URI in string form.
`isabs`()	Indicate that the resource is fully specified.
`isdir`()	Return True if this URI looks like a directory, else False.
`join`(path[, isTemporary, forceDirectory])	Return new `ResourcePath` with additional path components.
`mexists`(uris)	Check for existence of multiple URIs at once.
`mkdir`()	For a dir-like URI, create the directory resource if needed.
`open`([mode, encoding, prefer_file_temporary])	Return a context manager that wraps an object that behaves like an open file at the location of the URI.
`parent`()	Return a `ResourcePath` of the parent directory.
`read`([size])	Open the resource and return the contents in bytes.
`relative_to`(other)	Return the relative path from this URI to the other URI.
`remove`()	Remove the resource.
`replace`([forceDirectory, isTemporary])	Return new `ResourcePath` with specified components replaced.
`root_uri`()	Return the base root URI.
`size`()	For non-dir-like URI, return the size of the resource.
`split`()	Split URI into head and tail.
`temporary_uri`([prefix, suffix, delete])	Create a temporary file-like URI.
`to_fsspec`()	Return an abstract file system and path that can be used by fsspec.
`transfer_from`(src, transfer[, overwrite, ...])	Transfer to this URI from another.
`updatedExtension`(ext)	Return a new `ResourcePath` with updated file extension.
`updatedFile`(newfile)	Return new URI with an updated final component of the path.
`walk`([file_filter])	Walk the directory tree returning matching files and directories.
`write`(data[, overwrite])	Write the supplied bytes to the new resource.

Attributes Documentation

fragment¶: Return the fragment component of the URI.

isLocal = False¶: If True this URI refers to a local file.

is_root¶

Return whether this URI points to the root of the network location.

This means that the path components refers to the top level.

netloc¶: Return the URI network location.

ospath¶: Return the path component of the URI localized to current OS.

params¶: Return any parameters included in the URI.

path¶: Return the path component of the URI.

query¶: Return any query strings included in the URI.

quotePaths = True¶

True if path-like elements modifying a URI should be quoted.

All non-schemeless URIs have to internally use quoted paths. Therefore if a new file name is given (e.g. to updatedFile or join) a decision must be made whether to quote it to be consistent.

relativeToPathRoot¶

Return path relative to network location.

This is the path property with posix separator stripped from the left hand side of the path.

Always unquotes.

scheme¶

Return the URI scheme.

Notes

(:// is not part of the scheme).

transferDefault: str = 'copy'¶: Default mode to use for transferring if auto is specified.

transferModes: tuple[str, ...] = ('copy', 'auto', 'move')¶

Transfer modes supported by this implementation.

Move is special in that it is generally a copy followed by an unlink. Whether that unlink works depends critically on whether the source URI implements unlink. If it does not the move will be reported as a failure.

unquoted_path¶: Return path component of the URI with any URI quoting reversed.

Methods Documentation

abspath() → ResourcePath¶

Return URI using an absolute path.

Returns:

absResourcePath: Absolute URI. For non-schemeless URIs this always returns itself. Schemeless URIs are upgraded to file URIs.

as_local() → Iterator[ResourcePath]¶

Return the location of the (possibly remote) resource as local file.

Yields:

localResourcePath: If this is a remote resource, it will be a copy of the resource on the local file system, probably in a temporary directory. For a local resource this should be the actual path to the resource.

Notes

The context manager will automatically delete any local temporary file.

Examples

Should be used as a context manager:

with uri.as_local() as local:
    ospath = local.ospath

basename() → str¶

Return the base name, last element of path, of the URI.

Returns:

tailstr: Last part of the path attribute. Trail will be empty if path ends on a separator.

Notes

If URI ends on a slash returns an empty string. This is the second element returned by split().

Equivalent of os.path.basename.

dirname() → ResourcePath¶

Return the directory component of the path as a new ResourcePath.

Returns:

headResourcePath: Everything except the tail of path attribute, expanded and normalized as per ResourcePath rules.

Notes

Equivalent of os.path.dirname. If this is a directory URI it will be returned unchanged. If the parent directory is always required use parent.

exists() → bool¶

Indicate that the resource is available.

Returns:

existsbool: True if the resource exists.

classmethod findFileResources(candidates: Iterable[str | ParseResult | ResourcePath | Path], *, grouped: Literal[True]) → Iterator[Iterator[ResourcePath]]

Get all the files from a list of values.

Parameters:

candidatesiterable [str or ResourcePath]: The files to return and directories in which to look for files to return.
file_filterstr or re.Pattern, optional: The regex to use when searching for files within directories. By default returns all the found files.
groupedbool, optional: If True the results will be grouped by directory and each yielded value will be an iterator over URIs. If False each URI will be returned separately.

Yields:

found_file: ResourcePath: The passed-in URIs and URIs found in passed-in directories. If grouping is enabled, each of the yielded values will be an iterator yielding members of the group. Files given explicitly will be returned as a single group at the end.

Notes

If a value is a file it is yielded immediately without checking that it exists. If a value is a directory, all the files in the directory (recursively) that match the regex will be yielded in turn.

generate_presigned_get_url(*, expiration_time_seconds: int) → str¶

Return a pre-signed URL that can be used to retrieve this resource using an HTTP GET without supplying any access credentials.

Parameters:

expiration_time_secondsint: Number of seconds until the generated URL is no longer valid.

Returns:

urlstr: HTTP URL signed for GET.

generate_presigned_put_url(*, expiration_time_seconds: int) → str¶

Return a pre-signed URL that can be used to upload a file to this path using an HTTP PUT without supplying any access credentials.

Parameters:

expiration_time_secondsint: Number of seconds until the generated URL is no longer valid.

Returns:

urlstr: HTTP URL signed for PUT.

getExtension() → str¶

Return the extension(s) associated with this URI path.

Returns:

extstr: The file extension (including the .). Can be empty string if there is no file extension. Usually returns only the last file extension unless there is a special extension modifier indicating file compression, in which case the combined extension (e.g. .fits.gz) will be returned.

Notes

Does not distinguish between file and directory URIs when determining a suffix. An extension is only determined from the final component of the path.

geturl() → str¶

Return the URI in string form.

Returns:

urlstr: String form of URI.

isabs() → bool¶

Indicate that the resource is fully specified.

For non-schemeless URIs this is always true.

Returns:

isabsbool: True in all cases except schemeless URI.

isdir() → bool¶: Return True if this URI looks like a directory, else False.

join(path: str | ResourcePath, isTemporary: bool | None = None, forceDirectory: bool | None = None) → ResourcePath¶

Return new ResourcePath with additional path components.

Parameters:

pathstr, ResourcePath: Additional file components to append to the current URI. Will be quoted depending on the associated URI scheme. If the path looks like a URI referring to an absolute location, it will be returned directly (matching the behavior of os.path.join). It can also be a ResourcePath.
isTemporarybool, optional: Indicate that the resulting URI represents a temporary resource. Default is self.isTemporary.
forceDirectorybool or None, optional: If True forces the URI to end with a separator. If False the resultant URI is declared to refer to a file. None indicates that the file directory status is unknown.

Returns:

newResourcePath: New URI with the path appended.

Raises:

ValueError: Raised if the given path object refers to a directory but the forceDirectory parameter insists the outcome should be a file, and vice versa. Also raised if the URI being joined with is known to refer to a file.
RuntimeError: Raised if this attempts to join a temporary URI to a non-temporary URI.

Notes

Schemeless URIs assume local path separator but all other URIs assume POSIX separator if the supplied path has directory structure. It may be this never becomes a problem but datastore templates assume POSIX separator is being used.

If an absolute ResourcePath is given for path is is assumed that this should be returned directly. Giving a path of an absolute scheme-less URI is not allowed for safety reasons as it may indicate a mistake in the calling code.

It is an error to attempt to join to something that is known to refer to a file. Use updatedFile if the file is to be replaced.

classmethod mexists(uris: Iterable[ResourcePath]) → dict[lsst.resources._resourcePath.ResourcePath, bool]¶

Check for existence of multiple URIs at once.

Parameters:

urisiterable of ResourcePath: The URIs to test.

Returns:

existencedict of [ResourcePath, bool]: Mapping of original URI to boolean indicating existence.

mkdir() → None¶: For a dir-like URI, create the directory resource if needed.

open(mode: str = 'r', *, encoding: str | None = None, prefer_file_temporary: bool = False) → Iterator[ResourceHandleProtocol]¶

Return a context manager that wraps an object that behaves like an open file at the location of the URI.

Parameters:

modestr: String indicating the mode in which to open the file. Values are the same as those accepted by open, though intrinsically read-only URI types may only support read modes, and io.IOBase.seekable is not guaranteed to be True on the returned object.
encodingstr, optional: Unicode encoding for text IO; ignored for binary IO. Defaults to locale.getpreferredencoding(False), just as open does.
prefer_file_temporarybool, optional: If True, for implementations that require transfers from a remote system to temporary local storage and/or back, use a temporary file instead of an in-memory buffer; this is generally slower, but it may be necessary to avoid excessive memory usage by large files. Ignored by implementations that do not require a temporary.

Yields:

cmAbstractContextManager: A context manager that wraps a ResourceHandleProtocol file-like object.

Notes

The default implementation of this method uses a local temporary buffer (in-memory or file, depending on prefer_file_temporary) with calls to read, write, as_local, and transfer_from as necessary to read and write from/to remote systems. Remote writes thus occur only when the context manager is exited. ResourcePath implementations that can return a more efficient native buffer should do so whenever possible (as is guaranteed for local files). ResourcePath implementations for which as_local does not return a temporary are required to reimplement open, though they may delegate to super when prefer_file_temporary is False.

parent() → ResourcePath¶

Return a ResourcePath of the parent directory.

Returns:

headResourcePath: Everything except the tail of path attribute, expanded and normalized as per ResourcePath rules.

Notes

For a file-like URI this will be the same as calling dirname. For a directory-like URI this will always return the parent directory whereas dirname() will return the original URI. This is consistent with os.path.dirname compared to the pathlib.Path property parent.

read(size: int = -1) → bytes¶

Open the resource and return the contents in bytes.

Parameters:

sizeint, optional: The number of bytes to read. Negative or omitted indicates that all data should be read.

relative_to(other: ResourcePath) → str | None¶

Return the relative path from this URI to the other URI.

Parameters:

otherResourcePath: URI to use to calculate the relative path. Must be a parent of this URI.

Returns:

subpathstr: The sub path of this URI relative to the supplied other URI. Returns None if there is no parent child relationship. Scheme and netloc must match.

remove() → None¶: Remove the resource.

replace(forceDirectory: bool | None = None, isTemporary: bool = False, **kwargs: Any) → ResourcePath¶

Return new ResourcePath with specified components replaced.

Parameters:

forceDirectorybool or None, optional: Parameter passed to ResourcePath constructor to force this new URI to be dir-like or file-like.
isTemporarybool, optional: Indicate that the resulting URI is temporary resource.
**kwargs: Components of a urllib.parse.ParseResult that should be modified for the newly-created ResourcePath.

Returns:

newResourcePath: New ResourcePath object with updated values.

Notes

Does not, for now, allow a change in URI scheme.

root_uri() → ResourcePath¶

Return the base root URI.

Returns:

uriResourcePath: Root URI.

size() → int¶

For non-dir-like URI, return the size of the resource.

Returns:

szint: The size in bytes of the resource associated with this URI. Returns 0 if dir-like.

split() → tuple[lsst.resources._resourcePath.ResourcePath, str]¶

Split URI into head and tail.

Returns:

head: ResourcePath: Everything leading up to tail, expanded and normalized as per ResourcePath rules.
tailstr: Last path component. Tail will be empty if path ends on a separator or if the URI is known to be associated with a directory. Tail will never contain separators. It will be unquoted.

Notes

Equivalent to os.path.split where head preserves the URI components. In some cases this method can result in a file system check to verify whether the URI is a directory or not (only if forceDirectory was None during construction). For a scheme-less URI this can mean that the result might change depending on current working directory.

classmethod temporary_uri(prefix: ResourcePath | None = None, suffix: str | None = None, delete: bool = True) → Iterator[ResourcePath]¶

Create a temporary file-like URI.

Parameters:

prefixResourcePath, optional: Prefix to use. Without this the path will be formed as a local file URI in a temporary directory. Ensuring that the prefix location exists is the responsibility of the caller.
suffixstr, optional: A file suffix to be used. The . should be included in this suffix.
deletebool, optional: By default the resource will be deleted when the context manager is exited. Setting this flag to False will leave the resource alone. False will also retain any directories that may have been created.

Yields:

uriResourcePath: The temporary URI. Will be removed when the context is completed.

to_fsspec() → tuple[fsspec.spec.AbstractFileSystem, str]¶

Return an abstract file system and path that can be used by fsspec.

Returns:

fsfsspec.spec.AbstractFileSystem: A file system object suitable for use with the returned path.
pathstr: A path that can be opened by the file system object.

transfer_from(src: ResourcePath, transfer: str, overwrite: bool = False, transaction: TransactionProtocol | None = None) → None¶

Transfer to this URI from another.

Parameters:

srcResourcePath: Source URI.
transferstr: Mode to use for transferring the resource. Generically there are many standard options: copy, link, symlink, hardlink, relsymlink. Not all URIs support all modes.
overwritebool, optional: Allow an existing file to be overwritten. Defaults to False.
transactionTransactionProtocol, optional: A transaction object that can (depending on implementation) rollback transfers on error. Not guaranteed to be implemented.

Notes

Conceptually this is hard to scale as the number of URI schemes grow. The destination URI is more important than the source URI since that is where all the transfer modes are relevant (with the complication that “move” deletes the source).

Local file to local file is the fundamental use case but every other scheme has to support “copy” to local file (with implicit support for “move”) and copy from local file. All the “link” options tend to be specific to local file systems.

“move” is a “copy” where the remote resource is deleted at the end. Whether this works depends on the source URI rather than the destination URI. Reverting a move on transaction rollback is expected to be problematic if a remote resource was involved.

updatedExtension(ext: str | None) → ResourcePath¶

Return a new ResourcePath with updated file extension.

All file extensions are replaced.

Parameters:

extstr or None: New extension. If an empty string is given any extension will be removed. If None is given there will be no change.

Returns:

updatedResourcePath: URI with the specified extension. Can return itself if no extension was specified.

updatedFile(newfile: str) → ResourcePath¶

Return new URI with an updated final component of the path.

Parameters:

newfilestr: File name with no path component.

Returns:

updatedResourcePath: Updated ResourcePath with new updated final component.

Notes

Forces the ResourcePath.dirLike attribute to be false. The new file path will be quoted if necessary. If the current URI is known to refer to a directory, the new file will be joined to the current file. It is recommended that this behavior no longer be used and a call to isdir by the caller should be used to decide whether to join or replace. In the future this method may be modified to always replace the final element of the path.

walk(file_filter: str | Pattern | None = None) → Iterator[list | tuple[lsst.resources._resourcePath.ResourcePath, list[str], list[str]]]¶

Walk the directory tree returning matching files and directories.

Parameters:

file_filterstr or re.Pattern, optional: Regex to filter out files from the list before it is returned.

Yields:

dirpathResourcePath: Current directory being examined.
dirnameslist of str: Names of subdirectories within dirpath.
filenameslist of str: Names of all the files within dirpath.

write(data: bytes, overwrite: bool = True) → None¶

Write the supplied bytes to the new resource.

Parameters:

databytes: The bytes to write to the resource. The entire contents of the resource will be replaced.
overwritebool, optional: If True the resource will be overwritten if it exists. Otherwise the write will fail.

Navigation

ResourcePath¶