Pipeline¶

class lsst.pipe.base.Pipeline(description: str)¶

Bases: object

A Pipeline is a representation of a series of tasks to run, and the configuration for those tasks.

Parameters:	description : `str` A description of that this pipeline does.

Methods Summary

`addConfigFile`(label, filename)	Add overrides from a specified file.
`addConfigOverride`(label, key, value)	Apply single config override.
`addConfigPython`(label, pythonString)	Add Overrides by running a snippet of python code against a config.
`addInstrument`(instrument, str])	Add an instrument to the pipeline, or replace an instrument that is already defined.
`addTask`(task, str], label)	Add a new task to the pipeline, or replace a task that is already associated with the supplied label.
`fromFile`(filename)	Load a pipeline defined in a pipeline yaml file.
`fromIR`(deserialized_pipeline)	Create a pipeline from an already created `PipelineIR` object.
`fromPipeline`(pipeline)	Create a new pipeline by copying an already existing `Pipeline`.
`fromString`(pipeline_string)	Create a pipeline from string formatted as a pipeline document.
`from_uri`(uri, urllib.parse.ParseResult, …)	Load a pipeline defined in a pipeline yaml file at a location specified by a URI.
`getInstrument`()	Get the instrument from the pipeline.
`removeTask`(label)	Remove a task from the pipeline.
`subsetFromLabels`(labelSpecifier)	Subset a pipeline to contain only labels specified in labelSpecifier
`toExpandedPipeline`()	Returns a generator of TaskDefs which can be used to create quantum graphs.
`write_to_uri`(uri, urllib.parse.ParseResult, …)	Write the pipeline to a file or directory.

Methods Documentation

addConfigFile(label: str, filename: str) → None¶

Add overrides from a specified file.

Parameters:	label : `str` The label used to identify the task associated with config to modify filename : `str` Path to the override file.

addConfigOverride(label: str, key: str, value: object) → None¶

Apply single config override.

Parameters:	label : `str` Label of the task. key: `str` Fully-qualified field name. value : object Value to be given to a field.

addConfigPython(label: str, pythonString: str) → None¶

Add Overrides by running a snippet of python code against a config.

Parameters:	label : `str` The label used to identity the task associated with config to modify. pythonString: `str` A string which is valid python code to be executed. This is done with config as the only local accessible value.

addInstrument(instrument: Union[Instrument, str]) → None¶

Add an instrument to the pipeline, or replace an instrument that is already defined.

Parameters:	instrument : `Instrument` or `str` Either a derived class object of a `lsst.daf.butler.instrument` or a string corresponding to a fully qualified `lsst.daf.butler.instrument` name.

addTask(task: Union[Type[lsst.pipe.base.pipelineTask.PipelineTask], str], label: str) → None¶

Add a new task to the pipeline, or replace a task that is already associated with the supplied label.

Parameters:	task: `PipelineTask` or `str` Either a derived class object of a `PipelineTask` or a string corresponding to a fully qualified `PipelineTask` name. label: `str` A label that is used to identify the `PipelineTask` being added

classmethod fromFile(filename: str) → lsst.pipe.base.pipeline.Pipeline¶

Load a pipeline defined in a pipeline yaml file.

Parameters:

filename: `str`: A path that points to a pipeline defined in yaml format. This filename may also supply additional labels to be used in subsetting the loaded Pipeline. These labels are separated from the path by a #, and may be specified as a comma separated list, or a range denoted as beginning..end. Beginning or end may be empty, in which case the range will be a half open interval. Unlike python iteration bounds, end bounds are INCLUDED. Note that range based selection is not well defined for pipelines that are not linear in nature, and correct behavior is not guaranteed, or may vary from run to run.

Returns:

pipeline: Pipeline: The pipeline loaded from specified location with appropriate (if any) subsetting

Notes

This method attempts to prune any contracts that contain labels which are not in the declared subset of labels. This pruning is done using a string based matching due to the nature of contracts and may prune more than it should.

classmethod fromIR(deserialized_pipeline: lsst.pipe.base.pipelineIR.PipelineIR) → lsst.pipe.base.pipeline.Pipeline¶

Create a pipeline from an already created PipelineIR object.

Parameters:	deserialized_pipeline: `PipelineIR` An already created pipeline intermediate representation object
Returns:	pipeline: `Pipeline`

classmethod fromPipeline(pipeline: lsst.pipe.base.pipeline.Pipeline) → lsst.pipe.base.pipeline.Pipeline¶

Create a new pipeline by copying an already existing Pipeline.

Parameters:	pipeline: `Pipeline` An already created pipeline intermediate representation object
Returns:	pipeline: `Pipeline`

classmethod fromString(pipeline_string: str) → lsst.pipe.base.pipeline.Pipeline¶

Create a pipeline from string formatted as a pipeline document.

Parameters:	pipeline_string : `str` A string that is formatted according like a pipeline document
Returns:	pipeline: `Pipeline`

classmethod from_uri(uri: Union[str, urllib.parse.ParseResult, lsst.resources._resourcePath.ResourcePath, pathlib.Path]) → lsst.pipe.base.pipeline.Pipeline¶

Load a pipeline defined in a pipeline yaml file at a location specified by a URI.

Parameters:

uri: convertible to `ResourcePath`: If a string is supplied this should be a URI path that points to a pipeline defined in yaml format, either as a direct path to the yaml file, or as a directory containing a “pipeline.yaml” file (the form used by write_to_uri with expand=True). This uri may also supply additional labels to be used in subsetting the loaded Pipeline. These labels are separated from the path by a #, and may be specified as a comma separated list, or a range denoted as beginning..end. Beginning or end may be empty, in which case the range will be a half open interval. Unlike python iteration bounds, end bounds are INCLUDED. Note that range based selection is not well defined for pipelines that are not linear in nature, and correct behavior is not guaranteed, or may vary from run to run. The same specifiers can be used with a ResourcePath object, by being the sole contents in the fragments attribute.

Returns:

pipeline: Pipeline: The pipeline loaded from specified location with appropriate (if any) subsetting

Notes

This method attempts to prune any contracts that contain labels which are not in the declared subset of labels. This pruning is done using a string based matching due to the nature of contracts and may prune more than it should.

getInstrument() → Optional[str, None]¶

Get the instrument from the pipeline.

Returns:	instrument : `str`, or None The fully qualified name of a `lsst.obs.base.Instrument` subclass, name, or None if the pipeline does not have an instrument.

removeTask(label: str) → None¶

Remove a task from the pipeline.

Parameters:	label : `str` The label used to identify the task that is to be removed
Raises:	KeyError If no task with that label exists in the pipeline

subsetFromLabels(labelSpecifier: lsst.pipe.base.pipeline.LabelSpecifier) → lsst.pipe.base.pipeline.Pipeline¶

Subset a pipeline to contain only labels specified in labelSpecifier

Parameters:	labelSpecifier : `labelSpecifier` Object containing labels that describes how to subset a pipeline.
Returns:	pipeline : `Pipeline` A new pipeline object that is a subset of the old pipeline
Raises:	ValueError Raised if there is an issue with specified labels

Notes

This method attempts to prune any contracts that contain labels which are not in the declared subset of labels. This pruning is done using a string based matching due to the nature of contracts and may prune more than it should.

toExpandedPipeline() → Generator[lsst.pipe.base.pipeline.TaskDef, None, None]¶

Returns a generator of TaskDefs which can be used to create quantum graphs.

Returns:	generator : generator of `TaskDef` The generator returned will be the sorted iterator of tasks which are to be used in constructing a quantum graph.
Raises:	NotImplementedError If a dataId is supplied in a config block. This is in place for future use

write_to_uri(uri: Union[str, urllib.parse.ParseResult, lsst.resources._resourcePath.ResourcePath, pathlib.Path]) → None¶

Write the pipeline to a file or directory.

Parameters:	uri : convertible to `ResourcePath` URI to write to; may have any scheme with `ResourcePath` write support or no scheme for a local file/directory. Should have a `.yaml`.

Navigation

Pipeline¶