NullDatastore

class lsst.daf.butler.datastore.NullDatastore(config: Config | ResourcePathExpression | None, bridgeManager: DatastoreRegistryBridgeManager | None, butlerRoot: ResourcePathExpression | None = None)

Bases: Datastore

A datastore that implements the Datastore API but always fails when it accepts any request.

Parameters:

configConfig or ResourcePathExpression or None

Ignored.

bridgeManagerDatastoreRegistryBridgeManager or None

Ignored.

butlerRootResourcePathExpression or None

Ignored.

Methods Summary

clone(bridgeManager)	Make an independent copy of this Datastore with a different DatastoreRegistryBridgeManager instance.
emptyTrash([ignore_errors])	Remove all datasets from the trash.
exists(datasetRef)	Check if the dataset exists in the datastore.
export(refs, *[, directory, transfer])	Export datasets for transfer to another data repository.
export_records(refs)	Export datastore records and locations to an in-memory data structure.
forget(refs)	Indicate to the Datastore that it should remove all records of the given datasets, without actually deleting them.
get(datasetRef[, parameters, storageClass])	Load an InMemoryDataset from the store.
getLookupKeys()	Return all the lookup keys relevant to this datastore.
getURI(datasetRef[, predict])	URI to the Dataset.
getURIs(datasetRef[, predict])	Return URIs associated with dataset.
get_opaque_table_definitions()	Make definitions of the opaque tables used by this Datastore.
import_records(data)	Import datastore location and record data from an in-memory data structure.
ingest(*datasets[, transfer, ...])	Ingest one or more files into the datastore.
ingest_zip(zip_path, transfer)	Ingest an indexed Zip file and contents.
knows(ref)	Check if the dataset is known to the datastore.
put(inMemoryDataset, datasetRef)	Write a InMemoryDataset with a given DatasetRef to the store.
put_new(in_memory_dataset, ref)	Write a InMemoryDataset with a given DatasetRef to the store.
remove(datasetRef)	Indicate to the Datastore that a Dataset can be removed.
retrieveArtifacts(refs, destination[, ...])	Retrieve the artifacts associated with the supplied refs.
setConfigRoot(root, config, full[, overwrite])	Set filesystem-dependent config options for this datastore.
transfer(inputDatastore, datasetRef)	Transfer a dataset from another datastore to this datastore.
transfer_from(source_datastore, refs[, ...])	Transfer dataset artifacts from another datastore to this one.
trash(ref[, ignore_errors])	Indicate to the Datastore that a Dataset can be moved to the trash.
validateConfiguration(entities[, logFailures])	Validate some of the configuration for this datastore.
validateKey(lookupKey, entity)	Validate a specific look up key with supplied entity.

Methods Documentation

clone(bridgeManager: DatastoreRegistryBridgeManager) → Datastore

Make an independent copy of this Datastore with a different DatastoreRegistryBridgeManager instance.

Parameters:

bridgeManagerDatastoreRegistryBridgeManager

New DatastoreRegistryBridgeManager object to use when instantiating managers.

Returns:

datastoreDatastore

New Datastore instance with the same configuration as the existing instance.

emptyTrash(ignore_errors: bool = True) → None

Remove all datasets from the trash.

Parameters:

ignore_errorsbool, optional

Determine whether errors should be ignored.

Notes

Some Datastores may implement this method as a silent no-op to disable Dataset deletion through standard interfaces.

exists(datasetRef: DatasetRef) → bool

Check if the dataset exists in the datastore.

Parameters:

datasetRefDatasetRef

Reference to the required dataset.

Returns:

existsbool

True if the entity exists in the Datastore.

export(refs: Iterable[DatasetRef], *, directory: ResourcePathExpression | None = None, transfer: str | None = 'auto') → Iterable[FileDataset]

Export datasets for transfer to another data repository.

Parameters:

refsiterable of DatasetRef

Dataset references to be exported.

directorystr, optional

Path to a directory that should contain files corresponding to output datasets. Ignored if transfer is explicitly None.

transferstr, optional

Mode that should be used to move datasets out of the repository. Valid options are the same as those of the transfer argument to ingest, and datastores may similarly signal that a transfer mode is not supported by raising NotImplementedError. If “auto” is given and no directory is specified, None will be implied.

Returns:

datasetiterable of DatasetTransfer

Structs containing information about the exported datasets, in the same order as refs.

Raises:

NotImplementedError

Raised if the given transfer mode is not supported.

export_records(refs: Iterable[DatasetIdRef]) → Mapping[str, DatastoreRecordData]

Export datastore records and locations to an in-memory data structure.

Parameters:

refsIterable [ DatasetIdRef ]

Datasets to save. This may include datasets not known to this datastore, which should be ignored. May not include component datasets.

Returns:

dataMapping [ str, DatastoreRecordData ]

Exported datastore records indexed by datastore name.

forget(refs: Iterable[DatasetRef]) → None

Indicate to the Datastore that it should remove all records of the given datasets, without actually deleting them.

Parameters:

refsIterable [ DatasetRef ]

References to the datasets being forgotten.

Notes

Asking a datastore to forget a DatasetRef it does not hold should be a silent no-op, not an error.

get(datasetRef: DatasetRef, parameters: Mapping[str, Any] | None = None, storageClass: StorageClass | str | None = None) → Any

Load an InMemoryDataset from the store.

Parameters:

datasetRefDatasetRef

Reference to the required Dataset.

parametersdict

StorageClass-specific parameters that specify a slice of the Dataset to be loaded.

storageClassStorageClass or str, optional

The storage class to be used to override the Python type returned by this method. By default the returned type matches the dataset type definition for this dataset. Specifying a read StorageClass can force a different type to be returned. This type must be compatible with the original type.

Returns:

inMemoryDatasetobject

Requested Dataset or slice thereof as an InMemoryDataset.

getLookupKeys() → set[LookupKey]

Return all the lookup keys relevant to this datastore.

Returns:

keysset of LookupKey

The keys stored internally for looking up information based on DatasetType name or StorageClass.

getURI(datasetRef: DatasetRef, predict: bool = False) → ResourcePath

URI to the Dataset.

Parameters:

datasetRefDatasetRef

Reference to the required Dataset.

predictbool

If True attempt to predict the URI for a dataset if it does not exist in datastore.

Returns:

uristr

URI string pointing to the Dataset within the datastore. If the Dataset does not exist in the datastore, the URI may be a guess. If the datastore does not have entities that relate well to the concept of a URI the returned URI string will be descriptive. The returned URI is not guaranteed to be obtainable.

Raises:

FileNotFoundError

A URI has been requested for a dataset that does not exist and guessing is not allowed.

getURIs(datasetRef: DatasetRef, predict: bool = False) → DatasetRefURIs

Return URIs associated with dataset.

Parameters:

datasetRefDatasetRef

Reference to the required dataset.

predictbool, optional

If the datastore does not know about the dataset, controls whether it should return a predicted URI or not.

Returns:

urisDatasetRefURIs

The URI to the primary artifact associated with this dataset (if the dataset was disassembled within the datastore this may be None), and the URIs to any components associated with the dataset artifact. (can be empty if there are no components).

get_opaque_table_definitions() → Mapping[str, DatastoreOpaqueTable]

Make definitions of the opaque tables used by this Datastore.

Returns:

tablesMapping [ str, ddl.TableSpec ]

Mapping of opaque table names to their definitions. This can be an empty mapping if Datastore does not use opaque tables to keep datastore records.

import_records(data: Mapping[str, DatastoreRecordData]) → None

Import datastore location and record data from an in-memory data structure.

Parameters:

dataMapping [ str, DatastoreRecordData ]

Datastore records indexed by datastore name. May contain data for other Datastore instances (generally because they are chained to this one), which should be ignored.

Notes

Implementations should generally not check that any external resources (e.g. files) referred to by these records actually exist, for performance reasons; we expect higher-level code to guarantee that they do.

Implementations are responsible for calling DatastoreRegistryBridge.insert on all datasets in data.locations where the key is in names, as well as loading any opaque table data.

Implementations may assume that datasets are either fully present or not at all (single-component exports are not permitted).

ingest(*datasets: FileDataset, transfer: str | None = None, record_validation_info: bool = True) → None

Ingest one or more files into the datastore.

Parameters:

*datasetsFileDataset

Each positional argument is a struct containing information about a file to be ingested, including its path (either absolute or relative to the datastore root, if applicable), a complete DatasetRef (with dataset_id not None), and optionally a formatter class or its fully-qualified string name. If a formatter is not provided, the one the datastore would use for put on that dataset is assumed.

transferstr, optional

How (and whether) the dataset should be added to the datastore. If None (default), the file must already be in a location appropriate for the datastore (e.g. within its root directory), and will not be modified. Other choices include “move”, “copy”, “link”, “symlink”, “relsymlink”, and “hardlink”. “link” is a special transfer mode that will first try to make a hardlink and if that fails a symlink will be used instead. “relsymlink” creates a relative symlink rather than use an absolute path. Most datastores do not support all transfer modes. “auto” is a special option that will let the data store choose the most natural option for itself.

record_validation_infobool, optional

If True, the default, the datastore can record validation information associated with the file. If False the datastore will not attempt to track any information such as checksums or file sizes. This can be useful if such information is tracked in an external system or if the file is to be compressed in place. It is up to the datastore whether this parameter is relevant.

Raises:

NotImplementedError

Raised if the datastore does not support the given transfer mode (including the case where ingest is not supported at all).

DatasetTypeNotSupportedError

Raised if one or more files to be ingested have a dataset type that is not supported by the datastore.

FileNotFoundError

Raised if one of the given files does not exist.

FileExistsError

Raised if transfer is not None but the (internal) location the file would be moved to is already occupied.

Notes

Subclasses should implement _prepIngest and _finishIngest instead of implementing ingest directly. Datastores that hold and delegate to child datastores may want to call those methods as well.

Subclasses are encouraged to document their supported transfer modes in their class documentation.

ingest_zip(zip_path: ResourcePath, transfer: str | None) → None

Ingest an indexed Zip file and contents.

The Zip file must have an index file as created by retrieveArtifacts.

Parameters:

zip_pathlsst.resources.ResourcePath

Path to the Zip file.

transferstr

Method to use for transferring the Zip file into the datastore.

knows(ref: DatasetRef) → bool

Check if the dataset is known to the datastore.

Does not check for existence of any artifact.

Parameters:

refDatasetRef

Reference to the required dataset.

Returns:

existsbool

True if the dataset is known to the datastore.

put(inMemoryDataset: Any, datasetRef: DatasetRef) → None

Write a InMemoryDataset with a given DatasetRef to the store.

Parameters:

inMemoryDatasetobject

The Dataset to store.

datasetRefDatasetRef

Reference to the associated Dataset.

put_new(in_memory_dataset: Any, ref: DatasetRef) → Mapping[str, DatasetRef]

Write a InMemoryDataset with a given DatasetRef to the store.

Parameters:

in_memory_datasetobject

The Dataset to store.

refDatasetRef

Reference to the associated Dataset.

Returns:

datastore_refsMapping [str, DatasetRef]

Mapping of a datastore name to dataset reference stored in that datastore, reference will include datastore records. Only non-ephemeral datastores will appear in this mapping.

remove(datasetRef: DatasetRef) → None

Indicate to the Datastore that a Dataset can be removed.

Parameters:

datasetRefDatasetRef

Reference to the required Dataset.

Raises:

FileNotFoundError

When Dataset does not exist.

Notes

Some Datastores may implement this method as a silent no-op to disable Dataset deletion through standard interfaces.

retrieveArtifacts(refs: Iterable[DatasetRef], destination: ResourcePath, transfer: str = 'auto', preserve_path: bool = True, overwrite: bool = False, write_index: bool = True, add_prefix: bool = False) → dict[ResourcePath, ArtifactIndexInfo]

Retrieve the artifacts associated with the supplied refs.

Parameters:

refsiterable of DatasetRef

The datasets for which artifacts are to be retrieved. A single ref can result in multiple artifacts. The refs must be resolved.

destinationlsst.resources.ResourcePath

Location to write the artifacts.

transferstr, optional

Method to use to transfer the artifacts. Must be one of the options supported by lsst.resources.ResourcePath.transfer_from(). “move” is not allowed.

preserve_pathbool, optional

If True the full path of the artifact within the datastore is preserved. If False the final file component of the path is used.

overwritebool, optional

If True allow transfers to overwrite existing files at the destination.

write_indexbool, optional

If True write a file at the top level containing a serialization of a ZipIndex for the downloaded datasets.

add_prefixbool, optional

If True and if preserve_path is False, apply a prefix to the filenames corresponding to some part of the dataset ref ID. This can be used to guarantee uniqueness.

Returns:

artifact_mapdict [ lsst.resources.ResourcePath, ArtifactIndexInfo ]

Mapping of retrieved file to associated index information.

Notes

For non-file datastores the artifacts written to the destination may not match the representation inside the datastore. For example a hierarchical data structure in a NoSQL database may well be stored as a JSON file.

classmethod setConfigRoot(root: str, config: Config, full: Config, overwrite: bool = True) → None

Set filesystem-dependent config options for this datastore.

The options will be appropriate for a new empty repository with the given root.

Parameters:

rootstr

Filesystem path to the root of the data repository.

configConfig

A Config to update. Only the subset understood by this component will be updated. Will not expand defaults.

fullConfig

A complete config with all defaults expanded that can be converted to a DatastoreConfig. Read-only and will not be modified by this method. Repository-specific options that should not be obtained from defaults when Butler instances are constructed should be copied from full to config.

overwritebool, optional

If False, do not modify a value in config if the value already exists. Default is always to overwrite with the provided root.

Notes

If a keyword is explicitly defined in the supplied config it will not be overridden by this method if overwrite is False. This allows explicit values set in external configs to be retained.

transfer(inputDatastore: Datastore, datasetRef: DatasetRef) → None

Transfer a dataset from another datastore to this datastore.

Parameters:

inputDatastoreDatastore

The external Datastore from which to retrieve the Dataset.

datasetRefDatasetRef

Reference to the required Dataset.

transfer_from(source_datastore: Datastore, refs: Iterable[DatasetRef], transfer: str = 'auto', artifact_existence: dict[ResourcePath, bool] | None = None, dry_run: bool = False) → tuple[set[DatasetRef], set[DatasetRef]]

Transfer dataset artifacts from another datastore to this one.

Parameters:

source_datastoreDatastore

The datastore from which to transfer artifacts. That datastore must be compatible with this datastore receiving the artifacts.

refsCollection of DatasetRef

The datasets to transfer from the source datastore.

transferstr, optional

How (and whether) the dataset should be added to the datastore. Choices include “move”, “copy”, “link”, “symlink”, “relsymlink”, and “hardlink”. “link” is a special transfer mode that will first try to make a hardlink and if that fails a symlink will be used instead. “relsymlink” creates a relative symlink rather than use an absolute path. Most datastores do not support all transfer modes. “auto” (the default) is a special option that will let the data store choose the most natural option for itself. If the source location and transfer location are identical the transfer mode will be ignored.

artifact_existencedict [lsst.resources.ResourcePath, bool]

Optional mapping of datastore artifact to existence. Updated by this method with details of all artifacts tested. Can be None if the caller is not interested.

dry_runbool, optional

Process the supplied source refs without updating the target datastore.

Returns:

acceptedset [DatasetRef]

The datasets that were transferred.

rejectedset [DatasetRef]

The datasets that were rejected due to a constraints violation.

Raises:

TypeError

Raised if the two datastores are not compatible.

trash(ref: DatasetRef | Iterable[DatasetRef], ignore_errors: bool = True) → None

Indicate to the Datastore that a Dataset can be moved to the trash.

Parameters:

refDatasetRef or iterable thereof

Reference(s) to the required Dataset.

ignore_errorsbool, optional

Determine whether errors should be ignored. When multiple refs are being trashed there will be no per-ref check.

Raises:

FileNotFoundError

When Dataset does not exist and errors are not ignored. Only checked if a single ref is supplied (and not in a list).

Notes

Some Datastores may implement this method as a silent no-op to disable Dataset deletion through standard interfaces.

validateConfiguration(entities: Iterable[DatasetRef | DatasetType | StorageClass], logFailures: bool = False) → None

Validate some of the configuration for this datastore.

Parameters:

entitiesiterable of DatasetRef, DatasetType, or StorageClass

Entities to test against this configuration. Can be differing types.

logFailuresbool, optional

If True, output a log message for every validation error detected.

Raises:

DatastoreValidationError

Raised if there is a validation problem with a configuration.

Notes

Which parts of the configuration are validated is at the discretion of each Datastore implementation.

validateKey(lookupKey: LookupKey, entity: DatasetRef | DatasetType | StorageClass) → None

Validate a specific look up key with supplied entity.

Parameters:

lookupKeyLookupKey

Key to use to retrieve information from the datastore configuration.

entityDatasetRef, DatasetType, or StorageClass

Entity to compare with configuration retrieved using the specified lookup key.

Raises:

DatastoreValidationError

Raised if there is a problem with the combination of entity and lookup key.

Notes

Bypasses the normal selection priorities by allowing a key that would normally not be selected to be validated.