ButlerURI¶
-
class
lsst.daf.butler.
ButlerURI
¶ Bases:
object
Convenience wrapper around URI parsers.
Provides access to URI components and can convert file paths into absolute path URIs. Scheme-less URIs are treated as if they are local file system paths and are converted to absolute URIs.
A specialist subclass is created for each supported URI scheme.
Parameters: - uri :
str
orurllib.parse.ParseResult
URI in string form. Can be scheme-less if referring to a local filesystem path.
- root :
str
orButlerURI
, optional When fixing up a relative path in a
file
scheme or if scheme-less, use this as the root. Must be absolute. IfNone
the current working directory will be used. Can be a file URI.- forceAbsolute :
bool
, optional If
True
, scheme-less relative URI will be converted to an absolute path using afile
scheme. IfFalse
scheme-less URI will remain scheme-less and will not be updated tofile
or absolute path.- forceDirectory: `bool`, optional
If
True
forces the URI to end with a separator, otherwise given URI is interpreted as is.- isTemporary :
bool
, optional If
True
indicates that this URI points to a temporary resource.
Attributes Summary
fragment
The fragment component of the URI. isLocal
If True
this URI refers to a local file.is_root
True
if this URI points to the root of the network location.netloc
The URI network location. ospath
Path component of the URI localized to current OS. params
Any parameters included in the URI. path
The path component of the URI. query
Any query strings included in the URI. quotePaths
True if path-like elements modifying a URI should be quoted. relativeToPathRoot
Returns path relative to network location. scheme
The URI scheme ( ://
is not part of the scheme).transferDefault
Default mode to use for transferring if auto
is specified.transferModes
Transfer modes supported by this implementation. unquoted_path
The path component of the URI with any URI quoting reversed. Methods Summary
as_local
()Return the location of the (possibly remote) resource in the local file system. basename
()Returns the base name, last element of path, of the URI. dirname
()Returns a ButlerURI containing all the directories of the path attribute. exists
()Indicate that the resource is available. getExtension
()Return the file extension(s) associated with this URI path. geturl
()Return the URI in string form. isabs
()Indicate that the resource is fully specified. join
(path, …)Create a new ButlerURI
with additional path components including a file.mkdir
()For a dir-like URI, create the directory resource if it does not already exist. parent
()Returns a ButlerURI containing all the directories of the path attribute, minus the last one. read
(size)Open the resource and return the contents in bytes. relative_to
(other)Return the relative path from this URI to the other URI. remove
()Remove the resource. replace
(**kwargs)Replace components in a URI with new values and return a new instance. size
()For non-dir-like URI, return the size of the resource. split
()Splits URI into head and tail. transfer_from
(src, transfer, overwrite, …)Transfer the current resource to a new location. updateExtension
(ext)Update the file extension associated with this ButlerURI
in place.updateFile
(newfile)Update in place the final component of the path with the supplied file name. write
(data, overwrite)Write the supplied bytes to the new resource. Attributes Documentation
-
fragment
¶ The fragment component of the URI.
-
is_root
¶ True
if this URI points to the root of the network location.This means that the path components refers to the top level.
-
netloc
¶ The URI network location.
-
ospath
¶ Path component of the URI localized to current OS.
-
params
¶ Any parameters included in the URI.
-
path
¶ The path component of the URI.
-
query
¶ Any query strings included in the URI.
-
quotePaths
= True¶ True if path-like elements modifying a URI should be quoted.
All non-schemeless URIs have to internally use quoted paths. Therefore if a new file name is given (e.g. to updateFile or join) a decision must be made whether to quote it to be consistent.
-
relativeToPathRoot
¶ Returns path relative to network location.
Effectively, this is the path property with posix separator stripped from the left hand side of the path.
Always unquotes.
-
scheme
¶ The URI scheme (
://
is not part of the scheme).
-
transferDefault
= 'copy'¶ Default mode to use for transferring if
auto
is specified.
-
transferModes
= ('copy', 'auto', 'move')¶ Transfer modes supported by this implementation.
Move is special in that it is generally a copy followed by an unlink. Whether that unlink works depends critically on whether the source URI implements unlink. If it does not the move will be reported as a failure.
-
unquoted_path
¶ The path component of the URI with any URI quoting reversed.
Methods Documentation
-
as_local
() → Iterator[lsst.daf.butler.core._butlerUri._butlerUri.ButlerURI]¶ Return the location of the (possibly remote) resource in the local file system.
Yields: - local :
ButlerURI
If this is a remote resource, it will be a copy of the resource on the local file system, probably in a temporary directory. For a local resource this should be the actual path to the resource.
Notes
The context manager will automatically delete any local temporary file.
Examples
Should be used as a context manager:
with uri.as_local() as local: ospath = local.ospath
- local :
-
basename
() → str¶ Returns the base name, last element of path, of the URI. If URI ends on a slash returns an empty string. This is the second element returned by split().
Equivalent of os.path.basename().
Returns: - tail :
str
Last part of the path attribute. Trail will be empty if path ends on a separator.
- tail :
-
dirname
() → lsst.daf.butler.core._butlerUri._butlerUri.ButlerURI¶ Returns a ButlerURI containing all the directories of the path attribute.
Equivalent of os.path.dirname()
Returns: - head :
ButlerURI
Everything except the tail of path attribute, expanded and normalized as per ButlerURI rules.
- head :
-
exists
() → bool¶ Indicate that the resource is available.
Returns:
-
getExtension
() → str¶ Return the file extension(s) associated with this URI path.
Returns: - ext :
str
The file extension (including the
.
). Can be empty string if there is no file extension. Usually returns only the last file extension unless there is a special extension modifier indicating file compression, in which case the combined extension (e.g..fits.gz
) will be returned.
- ext :
-
isabs
() → bool¶ Indicate that the resource is fully specified.
For non-schemeless URIs this is always true.
Returns:
-
join
(path: Union[str, lsst.daf.butler.core._butlerUri._butlerUri.ButlerURI]) → lsst.daf.butler.core._butlerUri._butlerUri.ButlerURI¶ Create a new
ButlerURI
with additional path components including a file.Parameters: - path :
str
,ButlerURI
Additional file components to append to the current URI. Assumed to include a file at the end. Will be quoted depending on the associated URI scheme. If the path looks like a URI with a scheme referring to an absolute location, it will be returned directly (matching the behavior of
os.path.join()
). It can also be aButlerURI
.
Returns: - new :
ButlerURI
New URI with any file at the end replaced with the new path components.
Notes
Schemeless URIs assume local path separator but all other URIs assume POSIX separator if the supplied path has directory structure. It may be this never becomes a problem but datastore templates assume POSIX separator is being used.
- path :
-
mkdir
() → None¶ For a dir-like URI, create the directory resource if it does not already exist.
-
parent
() → lsst.daf.butler.core._butlerUri._butlerUri.ButlerURI¶ Returns a ButlerURI containing all the directories of the path attribute, minus the last one.
Returns: - head :
ButlerURI
Everything except the tail of path attribute, expanded and normalized as per ButlerURI rules.
- head :
-
read
(size: int = -1) → bytes¶ Open the resource and return the contents in bytes.
Parameters: - size :
int
, optional The number of bytes to read. Negative or omitted indicates that all data should be read.
- size :
-
relative_to
(other: lsst.daf.butler.core._butlerUri._butlerUri.ButlerURI) → Optional[str]¶ Return the relative path from this URI to the other URI.
Parameters: - other :
ButlerURI
URI to use to calculate the relative path. Must be a parent of this URI.
Returns: - other :
-
remove
() → None¶ Remove the resource.
-
replace
(**kwargs) → lsst.daf.butler.core._butlerUri._butlerUri.ButlerURI¶ Replace components in a URI with new values and return a new instance.
Returns:
-
size
() → int¶ For non-dir-like URI, return the size of the resource.
Returns: - sz :
int
The size in bytes of the resource associated with this URI. Returns 0 if dir-like.
- sz :
-
split
() → Tuple[lsst.daf.butler.core._butlerUri._butlerUri.ButlerURI, str]¶ Splits URI into head and tail. Equivalent to os.path.split where head preserves the URI components.
Returns: - head: `ButlerURI`
Everything leading up to tail, expanded and normalized as per ButlerURI rules.
- tail :
str
Last
self.path
component. Tail will be empty if path ends on a separator. Tail will never contain separators. It will be unquoted.
-
transfer_from
(src: ButlerURI, transfer: str, overwrite: bool = False, transaction: Optional[Union[DatastoreTransaction, NoTransaction]] = None) → None¶ Transfer the current resource to a new location.
Parameters: - src :
ButlerURI
Source URI.
- transfer :
str
Mode to use for transferring the resource. Generically there are many standard options: copy, link, symlink, hardlink, relsymlink. Not all URIs support all modes.
- overwrite :
bool
, optional Allow an existing file to be overwritten. Defaults to
False
.- transaction :
DatastoreTransaction
, optional A transaction object that can (depending on implementation) rollback transfers on error. Not guaranteed to be implemented.
Notes
Conceptually this is hard to scale as the number of URI schemes grow. The destination URI is more important than the source URI since that is where all the transfer modes are relevant (with the complication that “move” deletes the source).
Local file to local file is the fundamental use case but every other scheme has to support “copy” to local file (with implicit support for “move”) and copy from local file. All the “link” options tend to be specific to local file systems.
“move” is a “copy” where the remote resource is deleted at the end. Whether this works depends on the source URI rather than the destination URI. Reverting a move on transaction rollback is expected to be problematic if a remote resource was involved.
- src :
-
updateExtension
(ext: Optional[str]) → None¶ Update the file extension associated with this
ButlerURI
in place.All file extensions are replaced.
Parameters:
-
updateFile
(newfile: str) → None¶ Update in place the final component of the path with the supplied file name.
Parameters: - newfile :
str
File name with no path component.
Notes
Updates the URI in place. Updates the ButlerURI.dirLike attribute. The new file path will be quoted if necessary.
- newfile :
-
write
(data: bytes, overwrite: bool = True) → None¶ Write the supplied bytes to the new resource.
Parameters:
- uri :