butler

butler [OPTIONS] COMMAND [ARGS]...

Options

--log-level <LEVEL|COMPONENT=LEVEL>

The logging level. Supported levels are [CRITICAL|ERROR|WARNING|INFO|DEBUG]

config-dump

Dump either a subset or full Butler configuration to standard output.

REPO is the URI or path to an existing data repository root or configuration file.

butler config-dump [OPTIONS] REPO

Options

-s, --subset <subset>

Subset of a configuration to report. This can be any key in the hierarchy such as ‘.datastore.root’ where the leading ‘.’ specified the delimiter for the hierarchy.

-p, --searchpath <TEXT ...>

Additional search paths to use for configuration overrides

--file <outfile>

Print the (possibly-expanded) configuration for a repository to a file, or to stdout by default.

-@, --options-file <options_file>

URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO

Required argument

config-validate

Validate the configuration files for a Gen3 Butler repository.

REPO is the URI or path to an existing data repository root or configuration file.

butler config-validate [OPTIONS] REPO

Options

-q, --quiet

Do not report individual failures.

-d, --dataset-type <dataset_type>

Specific DatasetType(s) to validate.

-i, --ignore <TEXT ...>

DatasetType(s) to ignore for validation.

-@, --options-file <options_file>

URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO

Required argument

convert

Convert a Butler gen 2 repository into a gen 3 repository.

REPO is the URI or path to the gen3 repository. Will be created if it does not already exist

butler convert [OPTIONS] REPO

Options

--gen2root <gen2root>

Required Root path of the gen 2 repo to be converted.

--skymap-name <skymap_name>

Name of the new gen3 skymap (e.g. ‘discrete/ci_hsc’).

--skymap-config <skymap_config>

Path to skymap config file defining the new gen3 skymap.

--calibs <calibs>

Path to the gen 2 calibration repo. It can be absolute or relative to gen2root.

--reruns <TEXT ...>

List of gen 2 reruns to convert.

-t, --transfer <transfer>

Mode to use to transfer files into the new repository.

Options:auto|link|symlink|hardlink|copy|move|relsymlink

-j, --processes <processes>

Number of processes to use.

-C, --config-file <config_file>

Path to a ConvertRepoConfig override to be included after the Instrument config overrides are applied.

-@, --options-file <options_file>

URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO

Required argument

create

Create an empty Gen3 Butler repository.

REPO is the URI or path to the new repository. Will be created if it does not exist.

butler create [OPTIONS] REPO

Options

--seed-config <seed_config>

Path to an existing YAML config file to apply (on top of defaults).

--standalone

Include all defaults in the config file in the repo, insulating the repo from changes in package defaults.

--override

Allow values in the supplied config to override all repo settings.

-f, --outfile <outfile>

Name of output file to receive repository configuration. Default is to write butler.yaml into the specified repo.

-@, --options-file <options_file>

URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO

Required argument

define-visits

Define visits from exposures in the butler registry.

REPO is the URI or path to the gen3 repository. Will be created if it does not already exist

The name or fully-qualified class name of an instrument.

butler define-visits [OPTIONS] REPO INSTRUMENT

Options

-C, --config-file <config_file>

Path to a pex_config override to be included after the Instrument config overrides are applied.

--collections <TEXT ...>

The collections to be searched (in order) when reading datasets.

-j, --processes <processes>

Number of processes to use.

-@, --options-file <options_file>

URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO

Required argument

INSTRUMENT

Required argument

import

Import data into a butler repository.

REPO is the URI or path to the new repository. Will be created if it does not exist.

DIRECTORY is the folder containing dataset files.

butler import [OPTIONS] REPO DIRECTORY

Options

-t, --transfer <transfer>

The external data transfer mode.

Options:auto|link|symlink|hardlink|copy|move|relsymlink

--export-file <export_file>

Name for the file that contains database information associated with the exported datasets. If this is not an absolute path, does not exist in the current working directory, and –dir is provided, it is assumed to be in that directory. Defaults to “export.yaml”.

-s, --skip-dimensions <TEXT ...>

Dimensions that should be skipped during import

-@, --options-file <options_file>

URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO

Required argument

DIRECTORY

Required argument

ingest-raws

Ingest raw frames into from a directory into the butler registry

REPO is the URI or path to the gen3 repository. Will be created if it does not already exist

LOCATIONS specifies files to ingest and/or locations to search for files.

butler ingest-raws [OPTIONS] REPO LOCATIONS ...

Options

--regex <regex>

Regex string used to find files in directories listed in LOCATIONS. Searches for fits files by default.

-c, --config <TEXT=TEXT>

Config override, as a key-value pair.

-C, --config-file <config_file>

Path to a pex config override to be included after the Instrument config overrides are applied.

--output-run <output_run>

The name of the run datasets should be output to.

-t, --transfer <transfer>

The external data transfer mode.

Options:auto|link|symlink|hardlink|copy|move|relsymlink

-j, --processes <processes>

Number of processes to use.

--ingest-task <ingest_task>

The fully qualified class name of the ingest task to use.

-@, --options-file <options_file>

URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO

Required argument

LOCATIONS

Required argument(s)

make-discrete-skymap

Define a discrete skymap from calibrated exposures in the butler registry.

REPO is the URI or path to the gen3 repository. Will be created if it does not already exist

The fully-qualified name of an Instrument subclass.

butler make-discrete-skymap [OPTIONS] REPO INSTRUMENT

Options

-C, --config-file <config_file>

Path to a pex_config override to be included after the Instrument config overridesare applied.

-@, --options-file <options_file>

URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

--collections <TEXT ...>

Required The collections to be searched (in order) when reading datasets. This includes the seed skymap if –append is specified.

--out-collection <out_collection>

The collection to write the skymap to.

Default:skymaps

--skymap-id <skymap_id>

The identifier of the skymap to write.

Default:discrete

Arguments

REPO

Required argument

INSTRUMENT

Required argument

prune-collection

Remove a collection and possibly prune datasets within it.

REPO is the URI or path to an existing data repository root or configuration file.

butler prune-collection [OPTIONS] REPO

Options

--collection <collection>

Name of the collection to remove. If this is a TAGGED or CHAINED collection, datasets within the collection are not modified unless –unstore is passed. If this is a RUN collection, –purge and –unstore must be passed, and all datasets in it are fully removed from the data repository.

--purge

Permit RUN collections to be removed, fully removing datasets within them. Requires –unstore as an added precaution against accidental deletion. Must not be passed if the collection is not a RUN.

--unstore

Remove all datasets in the collection from all datastores in which they appear.

-@, --options-file <options_file>

URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO

Required argument

query-collections

Get the collections whose names match an expression.

REPO is the URI or path to an existing data repository root or configuration file.

GLOB is one or more glob-style expressions that fully or partially identify the collections to return.

butler query-collections [OPTIONS] REPO [GLOB] ...

Options

--collection-type <collection_type>

If provided, only list collections of this type.

Options:RUN|TAGGED|CHAINED|CALIBRATION

--flatten-chains, --no-flatten-chains

Recursively get the child collections of matching CHAINED collections. Default is –no-flatten-chains.

--include-chains, --no-include-chains

For –include-chains, return records for matching CHAINED collections. For –no-include-chains do not return records for CHAINED collections. Default is the opposite of –flatten-chains: include either CHAINED collections or their children, but not both.

-@, --options-file <options_file>

URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO

Required argument

GLOB

Optional argument(s)

query-dataset-types

Get the dataset types in a repository.

REPO is the URI or path to an existing data repository root or configuration file.

GLOB is one or more glob-style expressions that fully or partially identify the dataset types to return.

butler query-dataset-types [OPTIONS] REPO [GLOB] ...

Options

-v, --verbose

Include dataset type name, dimensions, and storage class in output.

--components, --no-components

For –components, apply all expression patterns to component dataset type names as well. For –no-components, never apply patterns to components. Default (where neither is specified) is to apply patterns to components only if their parent datasets were not matched by the expression. Fully-specified component datasets (str or DatasetType instances) are always included.

-@, --options-file <options_file>

URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO

Required argument

GLOB

Optional argument(s)

query-datasets

List the datasets in a repository.

REPO is the URI or path to an existing data repository root or configuration file.

GLOB is one or more glob-style expressions that fully or partially identify the dataset types to be queried.

butler query-datasets [OPTIONS] REPO [GLOB] ...

Options

--collections <TEXT ...>

One or more expressions that fully or partially identify the collections to search for datasets.If not provided all datasets are returned.

--where <where>

A string expression similar to a SQL WHERE clause. May involve any column of a dimension table or a dimension name as a shortcut for the primary key column of a dimension table.

--find-first

For each result data ID, only yield one DatasetRef of each DatasetType, from the first collection in which a dataset of that dataset type appears (according to the order of ‘collections’ passed in). If used, ‘collections’ must specify at least one expression and must not contain wildcards.

--show-uri

Show the dataset URI in results.

-@, --options-file <options_file>

URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO

Required argument

GLOB

Optional argument(s)

register-instrument

Add an instrument to the data repository.

REPO is the URI or path to the gen3 repository. Will be created if it does not already exist

The fully-qualified name of an Instrument subclass.

butler register-instrument [OPTIONS] REPO INSTRUMENT ...

Arguments

REPO

Required argument

INSTRUMENT

Required argument(s)

remove-dataset-type

Remove a dataset type definition from a repository.

REPO is the URI or path to an existing data repository root or configuration file.

butler remove-dataset-type [OPTIONS] REPO DATASET_TYPE_NAME

Arguments

REPO

Required argument

DATASET_TYPE_NAME

Required argument

write-curated-calibrations

Add an instrument’s curated calibrations to the data repository.

REPO is the URI or path to the gen3 repository. Will be created if it does not already exist

The fully-qualified name of an Instrument subclass.

butler write-curated-calibrations [OPTIONS] REPO INSTRUMENT

Options

--collection <collection>

Name of the calibration collection that associates datasets with validity ranges.

--suffix <suffix>

Name suffix to append (with an automatic delimiter) to all RUN collection names as well as the calibration collection name if it is not provided via –collection.

-@, --options-file <options_file>

URI to YAML file containing overrides of command line options. The YAML should be organized as a hierarchy with subcommand names at the top level options for that subcommand below.

Arguments

REPO

Required argument

INSTRUMENT

Required argument