Note

The butler subcommands documented here are only those defined in the daf_butler package itself; downstream packages can implement additional subcommands via the plugin system described at The Butler Command. The best way to get a complete list of subcommands is to use butler --help.

butler¶

Butler command-line tools.

Log options apply to all subcommands.

butler [OPTIONS] COMMAND [ARGS]...

Options

--log-level <LEVEL|COMPONENT=LEVEL>¶: The logging level. Without an explicit logger name, will only affect the default root loggers (lsst). To modify the root logger use ‘.=LEVEL’. Supported levels are [CRITICAL|ERROR|WARNING|INFO|VERBOSE|DEBUG|TRACE]

--long-log¶: Make log messages appear in long format.

--log-file <log_file>¶: File(s) to write log messages. If the path ends with ‘.json’ then JSON log records will be written, else formatted text log records will be written. This file can exist and records will be appended.

--log-tty, --no-log-tty¶: Log to terminal (default). If false logging to terminal is disabled.

--log-label <log_label>¶: Keyword=value pairs to add to MDC of log records.

--progress, --no-progress¶: Show a progress bar for slow operations when possible.

associate¶

Add existing datasets to a tagged collection; searches for datasets with the options and adds them to the named COLLECTION.

COLLECTION is the collection the datasets should be associated with.

butler associate [OPTIONS] REPO COLLECTION

Options

-d, --dataset-type <dataset_type>¶: One or more glob-style expressions that fully or partially identify the dataset type names to be queried.

--collections <collections>¶: One or more expressions that fully or partially identify the collections to search for datasets. If not provided all datasets are returned.

--where <where>¶: A string expression similar to a SQL WHERE clause. May involve any column of a dimension table or a dimension name as a shortcut for the primary key column of a dimension table.

--find-first, --no-find-first¶: For each result data ID, only yield one DatasetRef of each DatasetType, from the first collection in which a dataset of that dataset type appears (according to the order of ‘collections’ passed in). If used, ‘collections’ must specify at least one expression and must not contain wildcards.

--limit <limit>¶: Limit the number of results that are processed. 0 means no limit. A negative value specifies a cap where a warning will be issued if the cap is hit. Default value is 0.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

COLLECTION¶: Required argument

See ‘butler –help’ for more options.

certify-calibrations¶

Certify calibrations in a repository.

REPO is the URI or path to an existing data repository root or configuration file.

butler certify-calibrations [OPTIONS] REPO INPUT_COLLECTION OUTPUT_COLLECTION
                            DATASET_TYPE_NAME

Options

--begin-date <begin_date>¶: ISO-8601 datetime (TAI) of the beginning of the validity range for the certified calibrations.

--end-date <end_date>¶: ISO-8601 datetime (TAI) of the end of the validity range for the certified calibrations.

--search-all-inputs¶: Search all children of the inputCollection if it is a CHAINED collection, instead of just the most recent one.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

INPUT_COLLECTION¶: Required argument

OUTPUT_COLLECTION¶: Required argument

DATASET_TYPE_NAME¶: Required argument

See ‘butler –help’ for more options.

collection-chain¶

Define a collection chain.

REPO is the URI or path to an existing data repository root or configuration file.

PARENT is the name of the chained collection to create or modify. If the collection already exists the chain associated with it will be updated.

CHILDREN are the collections to be used to modify the chain. The supplied values will be split on comma. The exact usage depends on the MODE option. For example,

$ butler collection-chain REPO PARENT child1,child2 child3

will result in three children being included in the chain.

When the MODE is ‘pop’ the CHILDREN should be integer indices indicating collections to be removed from the current chain. MODE ‘pop’ can take negative integers to indicate removal relative to the end of the chain, but when doing that ‘–’ must be given to indicate the end of the options specification.

$ butler collection-chain REPO –mode=pop PARENT – -1

Will remove the final collection from the chain.

butler collection-chain [OPTIONS] REPO PARENT [CHILDREN]...

Options

--doc <doc>¶: Documentation string associated with this collection. Only relevant if the collection is newly created.

--flatten, --no-flatten¶: If True recursively flatten out any nested chained collections in children first.

--mode <mode>¶

Update mode: ‘redefine’: Create new chain or redefine existing chain with the supplied CHILDREN. ‘remove’: Modify existing chain to remove the supplied CHILDREN. ‘pop’: Pop a numbered element off the chain. Defaults to popping the first element (0). children must be integers if given. ‘prepend’: Modify existing chain to prepend the supplied CHILDREN to the front. ‘extend’: Modify existing chain to extend it with the supplied CHILDREN.

Options:: redefine | extend | remove | prepend | pop

Arguments

REPO¶: Required argument

PARENT¶: Required argument

CHILDREN¶: Optional argument(s)

See ‘butler –help’ for more options.

config-dump¶

Dump either a subset or full Butler configuration to standard output.

REPO is the URI or path to an existing data repository root or configuration file.

butler config-dump [OPTIONS] REPO

Options

-s, --subset <subset>¶: Subset of a configuration to report. This can be any key in the hierarchy such as ‘.datastore.root’ where the leading ‘.’ specified the delimiter for the hierarchy.

-p, --searchpath <TEXT ...>¶: Additional search paths to use for configuration overrides

--file <outfile>¶: Print the (possibly-expanded) configuration for a repository to a file, or to stdout by default.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

See ‘butler –help’ for more options.

config-validate¶

Validate the configuration files for a Gen3 Butler repository.

REPO is the URI or path to an existing data repository root or configuration file.

butler config-validate [OPTIONS] REPO

Options

-q, --quiet¶: Do not report individual failures.

-d, --dataset-type <dataset_type>¶: Specific DatasetType(s) to validate.

-i, --ignore <TEXT ...>¶: DatasetType(s) to ignore for validation.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

See ‘butler –help’ for more options.

convert¶

Convert one or more Butler gen 2 repositories into a gen 3 repository.

REPO is the URI or path to the gen3 repository. Will be created if it does not already exist

This is a highly simplified interface that should only be used to convert suites of gen 2 repositories that contain at most one calibration repo and has no chained reruns. Custom scripts that call ConvertRepoTask should be used on more complex suites of repositories.

butler convert [OPTIONS] REPO

Options

--gen2root <gen2root>¶: Required Root path of the gen 2 repo to be converted.

--skymap-name <skymap_name>¶: Name of the new gen3 skymap (e.g. ‘discrete/ci_hsc’).

--skymap-config <skymap_config>¶: Path to skymap config file defining the new gen3 skymap.

--calibs <calibs>¶: Path to the gen 2 calibration repo. It can be absolute or relative to gen2root.

--reruns <TEXT ...>¶: List of rerun paths to convert. Output collection names will be guessed, which can fail if the Gen2 repository paths do not follow a recognized convention. In this case, the command-line interface cannot be used.

-t, --transfer <transfer>¶

Mode to use to transfer files into the new repository.

Options:: auto | link | symlink | hardlink | copy | move | relsymlink | direct

-j, --processes <processes>¶: Number of processes to use.

-C, --config-file <config_file>¶: Path to a ConvertRepoConfig override to be included after the Instrument config overrides are applied.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

See ‘butler –help’ for more options.

create¶

Create an empty Gen3 Butler repository.

REPO is the URI or path to the new repository. Will be created if it does not exist.

butler create [OPTIONS] REPO

Options

--seed-config <seed_config>¶: Path to an existing YAML config file to apply (on top of defaults).

--dimension-config <dimension_config>¶: Path to an existing YAML config file with dimension configuration.

--standalone¶: Include all defaults in the config file in the repo, insulating the repo from changes in package defaults.

--override¶: Allow values in the supplied config to override all repo settings.

-f, --outfile <outfile>¶: Name of output file to receive repository configuration. Default is to write butler.yaml into the specified repo.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

See ‘butler –help’ for more options.

define-visits¶

Define visits from exposures in the butler registry.

REPO is the URI or path to the gen3 repository. Will be created if it does not already exist

The fully-qualified name of an Instrument subclass.

The calibration collection containing the camera geometry can not be specified.

butler define-visits [OPTIONS] REPO INSTRUMENT

Options

-C, --config-file <config_file>¶: Path to a pex_config override to be included after the Instrument config overrides are applied.

--collections <TEXT ...>¶: The collections to be searched (in order) when reading datasets.

--where <where>¶: A string expression similar to a SQL WHERE clause.

--update-records, --no-update-records¶: Use this option to force updates to the visit definition record. Should only be used if you know that there has been a change to the exposure records, such as a change to the metadata translator.

--incremental, --no-incremental¶: Use this option to force updates to the visit definition record when multi-snap visits are being ingested incrementally and so you might encounter partial visits. Implies –update-records.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

INSTRUMENT¶: Required argument

See ‘butler –help’ for more options.

export-calibs¶

Export calibrations from the butler for import elsewhere.

REPO is the URI or path to an existing data repository root or configuration file.

DIRECTORY is the folder to receive the exported calibrations.

COLLECTIONS are the collection to export calibrations from.

butler export-calibs [OPTIONS] REPO DIRECTORY [COLLECTIONS] ...

Options

-d, --dataset-type <dataset_type>¶: Specific DatasetType(s) to export.

-t, --transfer <transfer>¶

The external data transfer mode.

Options:: auto | link | symlink | hardlink | copy | move | relsymlink | direct

Arguments

REPO¶: Required argument

DIRECTORY¶: Required argument

COLLECTIONS¶: Optional argument(s)

See ‘butler –help’ for more options.

import¶

Import data into a butler repository.

REPO is the URI or path to the new repository. Will be created if it does not exist.

DIRECTORY is the folder containing dataset files.

butler import [OPTIONS] REPO DIRECTORY

Options

-t, --transfer <transfer>¶

The external data transfer mode.

Options:: auto | link | symlink | hardlink | copy | move | relsymlink | direct

--export-file <export_file>¶: Name for the file that contains database information associated with the exported datasets. If this is not an absolute path, does not exist in the current working directory, and –dir is provided, it is assumed to be in that directory. Defaults to “export.yaml”.

-s, --skip-dimensions <TEXT ...>¶: Dimensions that should be skipped during import

--track-file-attrs, --no-track-file-attrs¶: Indicate to the datastore whether file attributes such as file size or checksum should be tracked or not. Whether this parameter is honored depends on the specific datastore implementation.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

DIRECTORY¶: Required argument

See ‘butler –help’ for more options.

ingest-files¶

Ingest files from table file.

REPO is the URI or path to an existing data repository root or configuration file.

DATASET_TYPE is the name of the dataset type to be associated with these files. This dataset type must already exist and will not be created by this command. There can only be one dataset type per invocation of this command.

RUN is the run to use for the file ingest.

TABLE_FILE refers to a file that can be read by astropy.table with columns of:

file URI, dimension1, dimension2, …, dimensionN

where the first column is the URI to the file to be ingested and the remaining columns define the dataId to associate with that file. The column names should match the dimensions for the specified dataset type. Relative file URI by default is assumed to be relative to the current working directory but can be overridden using the --prefix option.

This command does not create dimension records and so any records must be created by other means. This command should not be used to ingest raw camera exposures.

butler ingest-files [OPTIONS] REPO DATASET_TYPE RUN TABLE_FILE

Options

--formatter <formatter>¶: Fully-qualified python class to use as the Formatter. If not specified the formatter will be determined from the dataset type and datastore configuration.

--id-generation-mode <id_generation_mode>¶

Mode to use for generating dataset IDs. The default creates a unique ID. Other options are: ‘DATAID_TYPE’ for creating a reproducible ID from the dataID and dataset type; ‘DATAID_TYPE_RUN’ for creating a reproducible ID from the dataID, dataset type and run. The latter is usually used for ‘raw’-type data that will be ingested in multiple. repositories.

Options:: UNIQUE | DATAID_TYPE | DATAID_TYPE_RUN

--data-id <data_id>¶: Keyword=value string with an additional dataId value that is fixed for all ingested files. This can be used to simplify the table file by removing repeated entries that are fixed for all files to be ingested. Multiple key/values can be given either by using comma separation or multiple command line options.

--prefix <prefix>¶: For relative paths in the table file, specify a prefix to use. The default is to use the current working directory.

--track-file-attrs, --no-track-file-attrs¶: Indicate to the datastore whether file attributes such as file size or checksum should be tracked or not. Whether this parameter is honored depends on the specific datastore implementation.

-t, --transfer <transfer>¶

The external data transfer mode.

Options:: auto | link | symlink | hardlink | copy | move | relsymlink | direct

Arguments

REPO¶: Required argument

DATASET_TYPE¶: Required argument

RUN¶: Required argument

TABLE_FILE¶: Required argument

See ‘butler –help’ for more options.

ingest-photodiode¶

Ingest photodiode data from a directory into the butler registry.

REPO is the URI or path to the gen3 repository. Will be created if it does not already exist

INSTRUMENT is the name of the instrument to use.

LOCATIONS specifies files to ingest and/or locations to search for files.

butler ingest-photodiode [OPTIONS] REPO INSTRUMENT LOCATIONS ...

Options

--regex <regex>¶: Regex string used to find photodiode data in directories listed in LOCATIONS. Defaults to Photodiode_Readings.*txt$|photodiode.ecsv$

-c, --config <TEXT=TEXT>¶: Config override, as a key-value pair.

-C, --config-file <config_file>¶: Path to a pex config override to be included after the Instrument config overrides are applied.

--output-run <output_run>¶: The name of the run datasets should be output to.

-t, --transfer <transfer>¶

The external data transfer mode.

Options:: auto | link | symlink | hardlink | copy | move | relsymlink | direct

--track-file-attrs, --no-track-file-attrs¶: Indicate to the datastore whether file attributes such as file size or checksum should be tracked or not. Whether this parameter is honored depends on the specific datastore implentation.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

INSTRUMENT¶: Required argument

LOCATIONS¶: Required argument(s)

See ‘butler –help’ for more options.

ingest-raws¶

Ingest raw frames into from a directory into the butler registry.

REPO is the URI or path to the gen3 repository. Will be created if it does not already exist

LOCATIONS specifies files to ingest and/or locations to search for files.

butler ingest-raws [OPTIONS] REPO LOCATIONS ...

Options

--regex <regex>¶: Regex string used to find files in directories listed in LOCATIONS. Searches for fits files by default.

-c, --config <TEXT=TEXT>¶: Config override, as a key-value pair.

-C, --config-file <config_file>¶: Path to a pex config override to be included after the Instrument config overrides are applied.

--output-run <output_run>¶: The name of the run datasets should be output to.

-t, --transfer <transfer>¶

The external data transfer mode.

Options:: auto | link | symlink | hardlink | copy | move | relsymlink | direct

-j, --processes <processes>¶: Number of processes to use.

--ingest-task <ingest_task>¶: The fully qualified class name of the ingest task to use.

--track-file-attrs, --no-track-file-attrs¶: Indicate to the datastore whether file attributes such as file size or checksum should be tracked or not. Whether this parameter is honored depends on the specific datastore implentation.

--fail-fast¶: Stop ingest as soon as any problem is encountered with any file. Otherwise problem files will be skipped and logged and a report issued at completion.

--update-records, --no-update-records¶: Use this option to force updates to the exposure records. Should only be used if you know that there has been a change to the exposure records, such as a change to the metadata translator.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

LOCATIONS¶: Required argument(s)

See ‘butler –help’ for more options.

ingest-zip¶

Ingest a Zip file created by retrieve-artifacts.

REPO is the URI or path to an existing data repository root or configuration file.

ZIP is the URI to the Zip file that should be ingested.

This command does not create dimension records and so any records must be created by other means.

butler ingest-zip [OPTIONS] REPO ZIP

Options

-t, --transfer <transfer>¶

The external data transfer mode.

Options:: auto | link | symlink | hardlink | copy | move | relsymlink | direct

Arguments

REPO¶: Required argument

ZIP¶: Required argument

See ‘butler –help’ for more options.

make-discrete-skymap¶

Define a discrete skymap from calibrated exposures in the butler registry.

REPO is the URI or path to the gen3 repository. Will be created if it does not already exist

INSTRUMENT is the name of the instrument to use.

butler make-discrete-skymap [OPTIONS] REPO INSTRUMENT

Options

-C, --config-file <config_file>¶: URI to a pex_config override to be included after the Instrument config overridesare applied.

--collections <collections>¶: Required The collections to be searched (in order) when reading datasets. This includes the seed skymap if –append is specified.

--skymap-id <skymap_id>¶

The identifier of the skymap to write.

Default:: 'discrete'

--old-skymap-id <old_skymap_id>¶: The identifier of the previous skymap to append to, if config.doAppend is True.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

INSTRUMENT¶: Required argument

See ‘butler –help’ for more options.

prune-datasets¶

Query for and remove one or more datasets from a collection and/or storage.

REPO is the URI or path to an existing data repository root or configuration file.

COLLECTIONS is or more expressions that identify the collections to search for datasets. Glob-style expressions may be used but only if the –find-all flag is also passed.

butler prune-datasets [OPTIONS] REPO [COLLECTIONS] ...

Options

--datasets <datasets>¶: One or more glob-style expressions that identify the dataset types to be pruned.

--find-all¶: Purge the dataset results from all of the collections in which a dataset of that dataset type + data id combination appear. (By default only the first found dataset type + data id is purged, according to the order of COLLECTIONS passed in).

--where <where>¶: A string expression similar to a SQL WHERE clause. May involve any column of a dimension table or a dimension name as a shortcut for the primary key column of a dimension table.

--disassociate <TAG>¶: Disassociate pruned datasets from the given tagged collections. May not be used with –purge.

--purge <RUN>¶: Completely remove the dataset from the given RUN in the Registry. May not be used with –disassociate. Implies –unstore. Note, this may remove provenance information from datasets other than those provided, and should be used with extreme care. RUN has to be provided for backward compatibility, but is used only if COLLECTIONS is not provided. Otherwise, datasets will be removed from any RUN-type collections in COLLECTIONS.

--unstore¶: Remove these datasets from all datastores configured with this data repository. If –disassociate and –purge are not used then –unstore will be used by default. Note that –unstore will make it impossible to retrieve these datasets even via other collections. Datasets that are already not stored are ignored by this option.

--dry-run¶

Display the datasets that would be removed but do not remove them.

Note that a dataset can be in collections other than its RUN-type collection, and removing it will remove it from all of them, even though the only one this will show is its RUN collection.

--confirm, --no-confirm¶: Print expected action and a confirmation prompt before executing. Default is –confirm.

--quiet¶: Makes output quiet. Implies –no-confirm. Requires –dry-run not be passed.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

COLLECTIONS¶: Optional argument(s)

See ‘butler –help’ for more options.

query-collections¶

Get the collections whose names match an expression.

REPO is the URI or path to an existing data repository root or configuration file.

GLOB is one or more glob-style expressions that fully or partially identify the collections to return.

butler query-collections [OPTIONS] REPO [GLOB] ...

Options

--collection-type <collection_type>¶

If provided, only list collections of this type.

Options:: RUN | TAGGED | CHAINED | CALIBRATION

--chains <chains>¶

Affects how results are presented:

TABLE lists each dataset in table form, with columns for dataset name and type, and a column that lists children of CHAINED datasets (if any CHAINED datasets are found).

INVERSE-TABLE is like TABLE but instead of a column listing CHAINED dataset children, it lists the parents of the dataset if it is contained in any CHAINED collections.

TREE recursively lists children below each CHAINED dataset in tree form.

INVERSE-TREE recursively lists parent datasets below each dataset in tree form.

FLATTEN lists all datasets, including child datasets, in one list.

[default: TREE]

Options:: TABLE | INVERSE-TABLE | TREE | INVERSE-TREE | FLATTEN

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

GLOB¶: Optional argument(s)

See ‘butler –help’ for more options.

query-data-ids¶

List the data IDs in a repository.

REPO is the URI or path to an existing data repository root or configuration file.

DIMENSIONS are the keys of the data IDs to yield, such as exposure, instrument, or tract. Will be expanded to include any dependencies.

butler query-data-ids [OPTIONS] REPO [DIMENSIONS] ...

Options

--collections <collections>¶: One or more expressions that fully or partially identify the collections to search for datasets. If not provided all datasets are returned. May only be used with –datasets.

--datasets <datasets>¶: An expression that fully or partially identifies dataset types that should constrain the yielded data IDs. For example, including “raw” here would constrain the yielded “instrument”, “exposure”, “detector”, and “physical_filter” values to only those for which at least one “raw” dataset exists in “collections”. Requires –collections.

--where <where>¶: A string expression similar to a SQL WHERE clause. May involve any column of a dimension table or a dimension name as a shortcut for the primary key column of a dimension table.

--order-by <order_by>¶: One or more comma-separated names used to order records. Names can be dimension names, metadata field names, or “timespan.begin” / “timespan.end” for temporal dimensions. In some cases the dimension for a metadata field or timespan bound can be inferred, but usually qualifying these with “<dimension>.<field>” is necessary. To reverse ordering for a name, prefix it with a minus sign.

--limit <limit>¶: Limit the number of results that are processed. 0 means no limit. A negative value specifies a cap where a warning will be issued if the cap is hit. Default value is -20000.

--offset <offset>¶: Skip initial number of records, only used when –limit is specified.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

DIMENSIONS¶: Optional argument(s)

See ‘butler –help’ for more options.

query-dataset-types¶

Get the dataset types in a repository.

REPO is the URI or path to an existing data repository root or configuration file.

GLOB is one or more glob-style expressions that fully or partially identify the dataset types to return.

butler query-dataset-types [OPTIONS] REPO [GLOB] ...

Options

-v, --verbose¶: Include dataset type name, dimensions, and storage class in output.

--collections <collections>¶: Constrain the resulting dataset types by these collections. This constraint does not say that a dataset of this type is definitely present, solely that one may have been present at some point.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

GLOB¶: Optional argument(s)

See ‘butler –help’ for more options.

query-datasets¶

List the datasets in a repository.

REPO is the URI or path to an existing data repository root or configuration file.

GLOB is one or more glob-style expressions that fully or partially identify the dataset type names to be queried.

butler query-datasets [OPTIONS] REPO [GLOB] ...

Options

--collections <collections>¶: One or more expressions that fully or partially identify the collections to search for datasets. If not provided all datasets are returned.

--where <where>¶: A string expression similar to a SQL WHERE clause. May involve any column of a dimension table or a dimension name as a shortcut for the primary key column of a dimension table.

--find-first, --no-find-first¶: For each result data ID, only yield one DatasetRef of each DatasetType, from the first collection in which a dataset of that dataset type appears (according to the order of ‘collections’ passed in). If used, ‘collections’ must specify at least one expression and must not contain wildcards.

--limit <limit>¶: Limit the number of results that are processed. 0 means no limit. A negative value specifies a cap where a warning will be issued if the cap is hit. Default value is -20000.

--order-by <order_by>¶: One or more comma-separated names used to order records. Names can be dimension names, metadata field names, or “timespan.begin” / “timespan.end” for temporal dimensions. In some cases the dimension for a metadata field or timespan bound can be inferred, but usually qualifying these with “<dimension>.<field>” is necessary. To reverse ordering for a name, prefix it with a minus sign.

--show-uri¶: Show the dataset URI in results.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

GLOB¶: Optional argument(s)

See ‘butler –help’ for more options.

query-dimension-records¶

Query for dimension information.

REPO is the URI or path to an existing data repository root or configuration file.

ELEMENT is the dimension element to obtain.

butler query-dimension-records [OPTIONS] REPO ELEMENT

Options

--datasets <datasets>¶: An expression that fully or partially identifies dataset types that should constrain the yielded records. May only be used with –collections.

--collections <collections>¶: One or more expressions that fully or partially identify the collections to search for datasets. If not provided all datasets are returned. May only be used with –datasets.

--where <where>¶: A string expression similar to a SQL WHERE clause. May involve any column of a dimension table or a dimension name as a shortcut for the primary key column of a dimension table.

--order-by <order_by>¶: One or more comma-separated names used to order records. Names can be dimension names, metadata field names, or “timespan.begin” / “timespan.end” for temporal dimensions. In some cases the dimension for a metadata field or timespan bound can be inferred, but usually qualifying these with “<dimension>.<field>” is necessary. To reverse ordering for a name, prefix it with a minus sign.

--limit <limit>¶: Limit the number of results that are processed. 0 means no limit. A negative value specifies a cap where a warning will be issued if the cap is hit. Default value is -20000.

--offset <offset>¶: Skip initial number of records, only used when –limit is specified.

--no-check¶: Don’t check the query before execution. By default the query is checked before it executed, this may reject some valid queries that resemble common mistakes.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

ELEMENT¶: Required argument

See ‘butler –help’ for more options.

register-dataset-type¶

Register a new dataset type with this butler repository.

REPO is the URI or path to an existing data repository root or configuration file.

DATASET_TYPE is the name of the dataset type.

STORAGE_CLASS is the name of the StorageClass to be associated with this dataset type.

DIMENSIONS is a list of all the dimensions relevant to this dataset type. It can be an empty list.

A component dataset type (such as “something.component”) is not a real dataset type and so can not be defined by this command. They are automatically derived from the composite dataset type when a composite storage class is specified.

butler register-dataset-type [OPTIONS] REPO DATASET_TYPE STORAGE_CLASS
                             [DIMENSIONS]...

Options

--is-calibration, --no-is-calibration¶: Indicate that this dataset type can be part of a calibration collection.

Arguments

REPO¶: Required argument

DATASET_TYPE¶: Required argument

STORAGE_CLASS¶: Required argument

DIMENSIONS¶: Optional argument(s)

See ‘butler –help’ for more options.

register-dcr-subfilters¶

Construct a set of subfilters for chromatic modeling and add them to a registry.

REPO is the URI or path to the gen3 repository. Will be created if it does not already exist

NUM_SUBFILTERS is the number of subfilters to be used for chromatic modeling.

BAND_NAMES names of the bands to define chromatic subfilters for in the registry. Each band will have the same number of subfilters defined, for example ‘g0’, ‘g1’, and ‘g2’ for three subfilters and band ‘g’.

butler register-dcr-subfilters [OPTIONS] REPO NUM_SUBFILTERS BAND_NAMES ...

Options

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

NUM_SUBFILTERS¶: Required argument

BAND_NAMES¶: Required argument(s)

See ‘butler –help’ for more options.

register-instrument¶

Add an instrument to the data repository.

REPO is the URI or path to an existing data repository root or configuration file.

The fully-qualified name of an Instrument subclass.

butler register-instrument [OPTIONS] REPO INSTRUMENT ...

Options

--update¶

Arguments

REPO¶: Required argument

INSTRUMENT¶: Required argument(s)

See ‘butler –help’ for more options.

register-skymap¶

Make a SkyMap and add it to a repository.

REPO is the URI or path to the gen3 repository. Will be created if it does not already exist

butler register-skymap [OPTIONS] REPO

Options

-c, --config <TEXT=TEXT>¶: Config override, as a key-value pair.

-C, --config-file <config_file>¶: URI to a config file overrides file.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

See ‘butler –help’ for more options.

remove-collections¶

Remove one or more non-RUN collections.

COLLECTION is a glob-style expression that identifies the collection(s) to remove.

This command can be used to remove only non-RUN collections. If RUN collections are found when searching for collections (and the –no-confirm flag is not used), then they will be shown in a separate table during confirmation, but they will not be removed.

Use the remove-runs subcommand to remove RUN collections.

butler remove-collections [OPTIONS] REPO COLLECTION

Options

--confirm, --no-confirm¶: Print expected action and a confirmation prompt before executing. Default is –confirm.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

COLLECTION¶: Required argument

See ‘butler –help’ for more options.

remove-dataset-type¶

Remove the dataset type definitions from a repository.

REPO is the URI or path to an existing data repository root or configuration file.

butler remove-dataset-type [OPTIONS] REPO [DATASET_TYPE_NAME]...

Arguments

REPO¶: Required argument

DATASET_TYPE_NAME¶: Optional argument(s)

See ‘butler –help’ for more options.

remove-runs¶

Remove one or more RUN collections.

REPO is the URI or path to an existing data repository root or configuration file.

COLLECTION is a glob-style expression that identifies the RUN collection(s) to remove.

This command can be used to remove RUN collections and the datasets within them.

Parameters¶

contextclick.Context: Context provided by Click.
confirmbool: Confirmation for removal of the run.
forcebool: Force removal.
**kwargsdict [str, str]: The parameters to pass to removeRuns.

butler remove-runs [OPTIONS] REPO COLLECTION

Options

--confirm, --no-confirm¶: Print expected action and a confirmation prompt before executing. Default is –confirm.

--force¶: Required to remove RUN collections from parent collections if using –no-confirm.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

COLLECTION¶: Required argument

See ‘butler –help’ for more options.

retrieve-artifacts¶

Retrieve file artifacts associated with datasets in a repository.

REPO is the URI or path to an existing data repository root or configuration file.

Destination URI of folder to receive file artifacts.

butler retrieve-artifacts [OPTIONS] REPO DESTINATION

Options

-d, --dataset-type <dataset_type>¶: One or more glob-style expressions that fully or partially identify the dataset type names to be queried.

--collections <collections>¶: One or more expressions that fully or partially identify the collections to search for datasets. If not provided all datasets are returned.

--where <where>¶: A string expression similar to a SQL WHERE clause. May involve any column of a dimension table or a dimension name as a shortcut for the primary key column of a dimension table.

--find-first, --no-find-first¶: For each result data ID, only yield one DatasetRef of each DatasetType, from the first collection in which a dataset of that dataset type appears (according to the order of ‘collections’ passed in). If used, ‘collections’ must specify at least one expression and must not contain wildcards.

--limit <limit>¶: Limit the number of results that are processed. 0 means no limit. A negative value specifies a cap where a warning will be issued if the cap is hit. Default value is -20000.

--order-by <order_by>¶: One or more comma-separated names used to order records. Names can be dimension names, metadata field names, or “timespan.begin” / “timespan.end” for temporal dimensions. In some cases the dimension for a metadata field or timespan bound can be inferred, but usually qualifying these with “<dimension>.<field>” is necessary. To reverse ordering for a name, prefix it with a minus sign.

-t, --transfer <transfer>¶

The external data transfer mode.

Options:: auto | link | symlink | hardlink | copy | move | relsymlink | direct

-v, --verbose¶: Report destination location of all transferred artifacts.

--preserve-path, --no-preserve-path¶: Preserve the datastore path to the artifact at the destination.

--clobber, --no-clobber¶: If clobber, overwrite files if they exist locally.

--zip, --no-zip¶: Retrieve artifacts and place in a Zip file.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

DESTINATION¶: Required argument

See ‘butler –help’ for more options.

transfer-datasets¶

Transfer datasets from a source butler to a destination butler.

SOURCE is a URI to the Butler repository containing the RUN dataset.

DEST is a URI to the Butler repository that will receive copies of the datasets.

butler transfer-datasets [OPTIONS] SOURCE DEST

Options

-d, --dataset-type <dataset_type>¶: One or more glob-style expressions that fully or partially identify the dataset type names to be queried.

--collections <collections>¶: One or more expressions that fully or partially identify the collections to search for datasets. If not provided all datasets are returned.

--where <where>¶: A string expression similar to a SQL WHERE clause. May involve any column of a dimension table or a dimension name as a shortcut for the primary key column of a dimension table.

--find-first, --no-find-first¶: For each result data ID, only yield one DatasetRef of each DatasetType, from the first collection in which a dataset of that dataset type appears (according to the order of ‘collections’ passed in). If used, ‘collections’ must specify at least one expression and must not contain wildcards.

--limit <limit>¶: Limit the number of results that are processed. 0 means no limit. A negative value specifies a cap where a warning will be issued if the cap is hit. Default value is -20000.

--order-by <order_by>¶: One or more comma-separated names used to order records. Names can be dimension names, metadata field names, or “timespan.begin” / “timespan.end” for temporal dimensions. In some cases the dimension for a metadata field or timespan bound can be inferred, but usually qualifying these with “<dimension>.<field>” is necessary. To reverse ordering for a name, prefix it with a minus sign.

-t, --transfer <transfer>¶

The external data transfer mode.

Options:: auto | link | symlink | hardlink | copy | move | relsymlink | direct

--register-dataset-types¶: Register DatasetTypes that do not already exist in the Registry.

--transfer-dimensions, --no-transfer-dimensions¶: If true, also copy dimension records along with datasets. If the dmensions are already present in the destination butler it can be more efficient to disable this. The default is to transfer dimensions.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

SOURCE¶: Required argument

DEST¶: Required argument

See ‘butler –help’ for more options.

transfer-from-graph¶

Transfer datasets from a quantum graph to a destination butler.

GRAPH is a URI to the source quantum graph file.

DEST is a URI to the Butler repository that will receive copies of the datasets.

butler transfer-from-graph [OPTIONS] GRAPH DEST

Options

--register-dataset-types¶: Register DatasetTypes that do not already exist in the Registry.

--transfer-dimensions, --no-transfer-dimensions¶: If true, also copy dimension records along with datasets. If the dmensions are already present in the destination butler it can be more efficient to disable this. The default is to transfer dimensions.

--update-output-chain¶: If quantum graph metadata includes output run name and output collection which is a chain, update the chain definition to include run name as the first collection in the chain.

--dry-run¶: Run the transfer but do not update the destination butler

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

GRAPH¶: Required argument

DEST¶: Required argument

See ‘butler –help’ for more options.

write-curated-calibrations¶

Add an instrument’s curated calibrations to the data repository.

REPO is the URI or path to the gen3 repository. Will be created if it does not already exist

The fully-qualified name of an Instrument subclass.

LBL: One or more extra strings to include in the collection name (see –label).

butler write-curated-calibrations [OPTIONS] REPO INSTRUMENT  ...

Options

--collection <collection>¶: Name of the calibration collection that associates datasets with validity ranges.

--label <labels>¶: Extra strings to include (with automatic delimiters) in all RUN collection names, as well as the calibration collection name if it is not provided via –collection. May be provided as a positional argument instead of or in addition to this option, as long as at least one label is provided. Positional-argument labels appear before those provided by this option.

--prefix <prefix>¶: Prefix for the collection name. Default is the instrument name. This is ignored if –collection is passed.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO¶: Required argument

INSTRUMENT¶: Required argument

LBL¶: Optional argument(s)

See ‘butler –help’ for more options.

zip-from-graph¶

Transfer datasets from a quantum graph to a Zip archive.

REPO is a URI to a butler configuration that is used to configure the datastore of the quantum-backed butler.

GRAPH is a URI to the source quantum graph file to use when building the Zip archive.

DEST is a directory to write the Zip archive.

butler zip-from-graph [OPTIONS] GRAPH REPO DEST

Options

-d, --dataset-type <dataset_type>¶: Dataset types to include in Zip archive.

-@, --options-file <options_file>¶: URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

GRAPH¶: Required argument

REPO¶: Required argument

DEST¶: Required argument

See ‘butler –help’ for more options.

Navigation

butler¶

associate¶

certify-calibrations¶

collection-chain¶

config-dump¶

config-validate¶

convert¶

create¶

define-visits¶

export-calibs¶

import¶

ingest-files¶

ingest-photodiode¶

ingest-raws¶

ingest-zip¶

make-discrete-skymap¶

prune-datasets¶

query-collections¶

query-data-ids¶

query-dataset-types¶

query-datasets¶

query-dimension-records¶

register-dataset-type¶

register-dcr-subfilters¶

register-instrument¶

register-skymap¶

remove-collections¶

remove-dataset-type¶

remove-runs¶

Parameters¶

retrieve-artifacts¶

transfer-datasets¶

transfer-from-graph¶

write-curated-calibrations¶

zip-from-graph¶