DiaPipelineConnections¶
- class lsst.ap.association.DiaPipelineConnections(*, config=None)¶
Bases:
PipelineTaskConnections
Butler connections for DiaPipelineTask.
Attributes Summary
Class used for declaring PipelineTask input connections
Class used for declaring PipelineTask input connections
Class used for declaring PipelineTask input connections
Class used for declaring PipelineTask input connections
Class used for declaring PipelineTask input connections
Methods Summary
adjustQuantum
(inputs, outputs, label, dataId)Override to make adjustments to
lsst.daf.butler.DatasetRef
objects in thelsst.daf.butler.core.Quantum
during the graph generation stage of the activator.buildDatasetRefs
(quantum)Builds QuantizedConnections corresponding to input Quantum
Attributes Documentation
- allConnections: Dict[str, BaseConnection] = {'apdbMarker': Output(name='apdb_marker', storageClass='Config', doc='Marker dataset storing the configuration of the Apdb for each visit/detector. Used to signal the completion of the pipeline.', multiple=False, dimensions=('instrument', 'visit', 'detector'), isCalibration=False), 'associatedDiaSources': Output(name='{fakesType}{coaddName}Diff_assocDiaSrc', storageClass='DataFrame', doc='Optional output storing the DiaSource catalog after matching, calibration, and standardization for insertation into the Apdb.', multiple=False, dimensions=('instrument', 'visit', 'detector'), isCalibration=False), 'diaSourceTable': Input(name='{fakesType}{coaddName}Diff_diaSrcTable', storageClass='DataFrame', doc='Catalog of calibrated DiaSources.', multiple=False, dimensions=('instrument', 'visit', 'detector'), isCalibration=False, deferLoad=False, minimum=1, deferGraphConstraint=False), 'diffIm': Input(name='{fakesType}{coaddName}Diff_differenceExp', storageClass='ExposureF', doc='Difference image on which the DiaSources were detected.', multiple=False, dimensions=('instrument', 'visit', 'detector'), isCalibration=False, deferLoad=False, minimum=1, deferGraphConstraint=False), 'exposure': Input(name='{fakesType}calexp', storageClass='ExposureF', doc='Calibrated exposure differenced with a template image during image differencing.', multiple=False, dimensions=('instrument', 'visit', 'detector'), isCalibration=False, deferLoad=False, minimum=1, deferGraphConstraint=False), 'solarSystemObjectTable': Input(name='visitSsObjects', storageClass='DataFrame', doc='Catalog of SolarSolarSystem objects expected to be observable in this detectorVisit.', multiple=False, dimensions=('instrument', 'visit'), isCalibration=False, deferLoad=False, minimum=1, deferGraphConstraint=False), 'template': Input(name='{fakesType}{coaddName}Diff_templateExp', storageClass='ExposureF', doc='Warped template used to create `subtractedExposure`. Not PSF matched.', multiple=False, dimensions=('instrument', 'visit', 'detector'), isCalibration=False, deferLoad=False, minimum=1, deferGraphConstraint=False)}¶
- apdbMarker¶
- associatedDiaSources¶
- defaultTemplates = {'coaddName': 'deep', 'fakesType': ''}¶
- diaSourceTable¶
Class used for declaring PipelineTask input connections
- Parameters:
- name
str
The default name used to identify the dataset type
- storageClass
str
The storage class used when (un)/persisting the dataset type
- multiple
bool
Indicates if this connection should expect to contain multiple objects of the given dataset type. Tasks with more than one connection with
multiple=True
with the same dimensions may want to implementPipelineTaskConnections.adjustQuantum
to ensure those datasets are consistent (i.e. zip-iterable) inPipelineTask.runQuantum
and notify the execution system as early as possible of outputs that will not be produced because the corresponding input is missing.- dimensionsiterable of
str
The
lsst.daf.butler.Butler
lsst.daf.butler.Registry
dimensions used to identify the dataset type identified by the specified name- deferLoad
bool
Indicates that this dataset type will be loaded as a
lsst.daf.butler.DeferredDatasetHandle
. PipelineTasks can use this object to load the object at a later time.- minimum
bool
Minimum number of datasets required for this connection, per quantum. This is checked in the base implementation of
PipelineTaskConnections.adjustQuantum
, which raisesNoWorkFound
if the minimum is not met forInput
connections (causing the quantum to be pruned, skipped, or never created, depending on the context), andFileNotFoundError
forPrerequisiteInput
connections (causing QuantumGraph generation to fail).PipelineTask
implementations may provide customadjustQuantum
implementations for more fine-grained or configuration-driven constraints, as long as they are compatible with this minium.- deferGraphConstraint: `bool`, optional
If
True
, do not include this dataset type’s existence in the initial query that starts the QuantumGraph generation process. This can be used to make QuantumGraph generation faster by avoiding redundant datasets, and in certain cases it can (along with careful attention to which tasks are included in the same QuantumGraph) be used to work around the QuantumGraph generation algorithm’s inflexible handling of spatial overlaps. This option has no effect when the connection is not an overall input of the pipeline (or subset thereof) for which a graph is being created, and it never affects the ordering of quanta.
- name
- Raises:
- TypeError
Raised if
minimum
is greater than one butmultiple=False
.- NotImplementedError
Raised if
minimum
is zero for a regularInput
connection; this is not currently supported by our QuantumGraph generation algorithm.
- diffIm¶
Class used for declaring PipelineTask input connections
- Parameters:
- name
str
The default name used to identify the dataset type
- storageClass
str
The storage class used when (un)/persisting the dataset type
- multiple
bool
Indicates if this connection should expect to contain multiple objects of the given dataset type. Tasks with more than one connection with
multiple=True
with the same dimensions may want to implementPipelineTaskConnections.adjustQuantum
to ensure those datasets are consistent (i.e. zip-iterable) inPipelineTask.runQuantum
and notify the execution system as early as possible of outputs that will not be produced because the corresponding input is missing.- dimensionsiterable of
str
The
lsst.daf.butler.Butler
lsst.daf.butler.Registry
dimensions used to identify the dataset type identified by the specified name- deferLoad
bool
Indicates that this dataset type will be loaded as a
lsst.daf.butler.DeferredDatasetHandle
. PipelineTasks can use this object to load the object at a later time.- minimum
bool
Minimum number of datasets required for this connection, per quantum. This is checked in the base implementation of
PipelineTaskConnections.adjustQuantum
, which raisesNoWorkFound
if the minimum is not met forInput
connections (causing the quantum to be pruned, skipped, or never created, depending on the context), andFileNotFoundError
forPrerequisiteInput
connections (causing QuantumGraph generation to fail).PipelineTask
implementations may provide customadjustQuantum
implementations for more fine-grained or configuration-driven constraints, as long as they are compatible with this minium.- deferGraphConstraint: `bool`, optional
If
True
, do not include this dataset type’s existence in the initial query that starts the QuantumGraph generation process. This can be used to make QuantumGraph generation faster by avoiding redundant datasets, and in certain cases it can (along with careful attention to which tasks are included in the same QuantumGraph) be used to work around the QuantumGraph generation algorithm’s inflexible handling of spatial overlaps. This option has no effect when the connection is not an overall input of the pipeline (or subset thereof) for which a graph is being created, and it never affects the ordering of quanta.
- name
- Raises:
- TypeError
Raised if
minimum
is greater than one butmultiple=False
.- NotImplementedError
Raised if
minimum
is zero for a regularInput
connection; this is not currently supported by our QuantumGraph generation algorithm.
- exposure¶
Class used for declaring PipelineTask input connections
- Parameters:
- name
str
The default name used to identify the dataset type
- storageClass
str
The storage class used when (un)/persisting the dataset type
- multiple
bool
Indicates if this connection should expect to contain multiple objects of the given dataset type. Tasks with more than one connection with
multiple=True
with the same dimensions may want to implementPipelineTaskConnections.adjustQuantum
to ensure those datasets are consistent (i.e. zip-iterable) inPipelineTask.runQuantum
and notify the execution system as early as possible of outputs that will not be produced because the corresponding input is missing.- dimensionsiterable of
str
The
lsst.daf.butler.Butler
lsst.daf.butler.Registry
dimensions used to identify the dataset type identified by the specified name- deferLoad
bool
Indicates that this dataset type will be loaded as a
lsst.daf.butler.DeferredDatasetHandle
. PipelineTasks can use this object to load the object at a later time.- minimum
bool
Minimum number of datasets required for this connection, per quantum. This is checked in the base implementation of
PipelineTaskConnections.adjustQuantum
, which raisesNoWorkFound
if the minimum is not met forInput
connections (causing the quantum to be pruned, skipped, or never created, depending on the context), andFileNotFoundError
forPrerequisiteInput
connections (causing QuantumGraph generation to fail).PipelineTask
implementations may provide customadjustQuantum
implementations for more fine-grained or configuration-driven constraints, as long as they are compatible with this minium.- deferGraphConstraint: `bool`, optional
If
True
, do not include this dataset type’s existence in the initial query that starts the QuantumGraph generation process. This can be used to make QuantumGraph generation faster by avoiding redundant datasets, and in certain cases it can (along with careful attention to which tasks are included in the same QuantumGraph) be used to work around the QuantumGraph generation algorithm’s inflexible handling of spatial overlaps. This option has no effect when the connection is not an overall input of the pipeline (or subset thereof) for which a graph is being created, and it never affects the ordering of quanta.
- name
- Raises:
- TypeError
Raised if
minimum
is greater than one butmultiple=False
.- NotImplementedError
Raised if
minimum
is zero for a regularInput
connection; this is not currently supported by our QuantumGraph generation algorithm.
- inputs: Set[str] = frozenset({'diaSourceTable', 'diffIm', 'exposure', 'solarSystemObjectTable', 'template'})¶
- solarSystemObjectTable¶
Class used for declaring PipelineTask input connections
- Parameters:
- name
str
The default name used to identify the dataset type
- storageClass
str
The storage class used when (un)/persisting the dataset type
- multiple
bool
Indicates if this connection should expect to contain multiple objects of the given dataset type. Tasks with more than one connection with
multiple=True
with the same dimensions may want to implementPipelineTaskConnections.adjustQuantum
to ensure those datasets are consistent (i.e. zip-iterable) inPipelineTask.runQuantum
and notify the execution system as early as possible of outputs that will not be produced because the corresponding input is missing.- dimensionsiterable of
str
The
lsst.daf.butler.Butler
lsst.daf.butler.Registry
dimensions used to identify the dataset type identified by the specified name- deferLoad
bool
Indicates that this dataset type will be loaded as a
lsst.daf.butler.DeferredDatasetHandle
. PipelineTasks can use this object to load the object at a later time.- minimum
bool
Minimum number of datasets required for this connection, per quantum. This is checked in the base implementation of
PipelineTaskConnections.adjustQuantum
, which raisesNoWorkFound
if the minimum is not met forInput
connections (causing the quantum to be pruned, skipped, or never created, depending on the context), andFileNotFoundError
forPrerequisiteInput
connections (causing QuantumGraph generation to fail).PipelineTask
implementations may provide customadjustQuantum
implementations for more fine-grained or configuration-driven constraints, as long as they are compatible with this minium.- deferGraphConstraint: `bool`, optional
If
True
, do not include this dataset type’s existence in the initial query that starts the QuantumGraph generation process. This can be used to make QuantumGraph generation faster by avoiding redundant datasets, and in certain cases it can (along with careful attention to which tasks are included in the same QuantumGraph) be used to work around the QuantumGraph generation algorithm’s inflexible handling of spatial overlaps. This option has no effect when the connection is not an overall input of the pipeline (or subset thereof) for which a graph is being created, and it never affects the ordering of quanta.
- name
- Raises:
- TypeError
Raised if
minimum
is greater than one butmultiple=False
.- NotImplementedError
Raised if
minimum
is zero for a regularInput
connection; this is not currently supported by our QuantumGraph generation algorithm.
- template¶
Class used for declaring PipelineTask input connections
- Parameters:
- name
str
The default name used to identify the dataset type
- storageClass
str
The storage class used when (un)/persisting the dataset type
- multiple
bool
Indicates if this connection should expect to contain multiple objects of the given dataset type. Tasks with more than one connection with
multiple=True
with the same dimensions may want to implementPipelineTaskConnections.adjustQuantum
to ensure those datasets are consistent (i.e. zip-iterable) inPipelineTask.runQuantum
and notify the execution system as early as possible of outputs that will not be produced because the corresponding input is missing.- dimensionsiterable of
str
The
lsst.daf.butler.Butler
lsst.daf.butler.Registry
dimensions used to identify the dataset type identified by the specified name- deferLoad
bool
Indicates that this dataset type will be loaded as a
lsst.daf.butler.DeferredDatasetHandle
. PipelineTasks can use this object to load the object at a later time.- minimum
bool
Minimum number of datasets required for this connection, per quantum. This is checked in the base implementation of
PipelineTaskConnections.adjustQuantum
, which raisesNoWorkFound
if the minimum is not met forInput
connections (causing the quantum to be pruned, skipped, or never created, depending on the context), andFileNotFoundError
forPrerequisiteInput
connections (causing QuantumGraph generation to fail).PipelineTask
implementations may provide customadjustQuantum
implementations for more fine-grained or configuration-driven constraints, as long as they are compatible with this minium.- deferGraphConstraint: `bool`, optional
If
True
, do not include this dataset type’s existence in the initial query that starts the QuantumGraph generation process. This can be used to make QuantumGraph generation faster by avoiding redundant datasets, and in certain cases it can (along with careful attention to which tasks are included in the same QuantumGraph) be used to work around the QuantumGraph generation algorithm’s inflexible handling of spatial overlaps. This option has no effect when the connection is not an overall input of the pipeline (or subset thereof) for which a graph is being created, and it never affects the ordering of quanta.
- name
- Raises:
- TypeError
Raised if
minimum
is greater than one butmultiple=False
.- NotImplementedError
Raised if
minimum
is zero for a regularInput
connection; this is not currently supported by our QuantumGraph generation algorithm.
Methods Documentation
- adjustQuantum(inputs, outputs, label, dataId)¶
Override to make adjustments to
lsst.daf.butler.DatasetRef
objects in thelsst.daf.butler.core.Quantum
during the graph generation stage of the activator.This implementation checks to make sure that the filters in the dataset are compatible with AP processing as set by the Apdb/DPDD schema.
- Parameters:
- inputs
dict
Dictionary whose keys are an input (regular or prerequisite) connection name and whose values are a tuple of the connection instance and a collection of associated
DatasetRef
objects. The exact type of the nested collections is unspecified; it can be assumed to be multi-pass iterable and supportlen
andin
, but it should not be mutated in place. In contrast, the outer dictionaries are guaranteed to be temporary copies that are truedict
instances, and hence may be modified and even returned; this is especially useful for delegating tosuper
(see notes below).- outputs
dict
Dict of output datasets, with the same structure as
inputs
.- label
str
Label for this task in the pipeline (should be used in all diagnostic messages).
- data_id
lsst.daf.butler.DataCoordinate
Data ID for this quantum in the pipeline (should be used in all diagnostic messages).
- inputs
- Returns:
- adjusted_inputs
dict
Dict of the same form as
inputs
with updated containers of inputDatasetRef
objects. Connections that are not changed should not be returned at all. Datasets may only be removed, not added. Nested collections may be of any multi-pass iterable type, and the order of iteration will set the order of iteration withinPipelineTask.runQuantum
.- adjusted_outputs
dict
Dict of updated output datasets, with the same structure and interpretation as
adjusted_inputs
.
- adjusted_inputs
- Raises:
- ScalarError
Raised if any
Input
orPrerequisiteInput
connection hasmultiple
set toFalse
, but multiple datasets.- NoWorkFound
Raised to indicate that this quantum should not be run; not enough datasets were found for a regular
Input
connection, and the quantum should be pruned or skipped.- FileNotFoundError
Raised to cause QuantumGraph generation to fail (with the message included in this exception); not enough datasets were found for a
PrerequisiteInput
connection.
- buildDatasetRefs(quantum: Quantum) Tuple[InputQuantizedConnection, OutputQuantizedConnection] ¶
Builds QuantizedConnections corresponding to input Quantum
- Parameters:
- quantum
lsst.daf.butler.Quantum
Quantum object which defines the inputs and outputs for a given unit of processing
- quantum
- Returns:
- retVal
tuple
of (InputQuantizedConnection
, OutputQuantizedConnection
) Namespaces mapping attribute names (identifiers of connections) to butler references defined in the inputlsst.daf.butler.Quantum
- retVal