Namespace lsst::utils::python¶

namespace python

Functions

template<typename Key, typename Value, typename KeyHash = boost::hash<Key>, typename KeyPred = std::equal_to<Key>> void declareCache(py::module &mod, std::string const &name)

template<typename T, typename PyClass> void addSharedPtrEquality(PyClass &cls)

Add __eq__ and __ne__ methods based on two std::shared_ptr<T> pointing to the same address

Example:

Template Parameters

T: The type to which the std::shared_ptr points.
PyClass: The pybind11 class_ type; this can be automatically deduced.

lsst::afw::table records are considered equal if two std::shared_ptr<record> point to the same record. This is wrapped as follows for lsst::afw::table::BaseRecord, where cls is an instance of pybind11::class_<BaseRecord, std::shared_ptr<BaseRecord>>):

utils::addSharedPtrEquality<BaseRecord>(cls);

Note that all record subclasses inherit this behavior without needing to call this function.

template<class PyClass> void addOutputOp(PyClass &cls, std::string const &method)

Add __str__ or __repr__ method implemented by operator<<.

For flexibility, this method can be used to define one or both of __str__ and __repr__. It can also be used to define any Python method that takes no arguments and returns a string, regardless of name.

Template Parameters

PyClass: The pybind11 class_ type. The wrapped class must support << as a stream output operator.

Parameters

cls: The PyClass object to which to add a wrapper.
method: The name of the method to implement. Should be "__str__" or "__repr__".

template<class PyClass> void addHash(PyClass &cls)

Add __hash__ method implemented by std::hash.

Template Parameters

PyClass: The pybind11 class_ type. The wrapped class must have an enabled specialization of std::hash.

Parameters

cls: The PyClass object to which to add a wrapper.

std::size_t cppIndex(std::ptrdiff_t size, std::ptrdiff_t i)

Compute a C++ index from a Python index (negative values count from the end) and range-check.

Return

index in the range [0, size - 1]

Note

the size argument has type std::ptrdiff_t instead of std::size_t in order to to match the allowed range for the i argument.

Parameters

[in] size: Number of elements in the collection.
[in] i: Index into the collection; negative values count from the end

Exceptions

Python: IndexError if i not in range [-size, size - 1]

std::pair<std::size_t, std::size_t> cppIndex(std::ptrdiff_t size_i, std::ptrdiff_t size_j, std::ptrdiff_t i, std::ptrdiff_t j)

Compute a pair of C++ indices from a pair of Python indices (negative values count from the end) and range-check.

Return

a std::pair of indices, each in the range [0, size - 1]

Parameters

[in] size_i: Number of elements along the first axis.
[in] size_j: Number of elements along the second axis.
[in] i: Index along first axis; negative values count from the end
[in] j: Index along second axis; negative values count from the end

Exceptions

Python: IndexError if either input index not in range [-size, size - 1]

template<typename T> class PySharedPtr

#include <PySharedPtr.h>

A shared pointer that tracks both a C++ object and its associated PyObject.

Each group of PySharedPtr for a given object collectively counts as one reference to that object for the purpose of Python garbage collection.

A PySharedPtr is implicitly convertible to and from a std::shared_ptr to minimize API impact. Any shared_ptr created this way will (I think) keep the Python reference alive, as described above.

class TemplateInvoker

#include <TemplateInvoker.h>

A helper class for wrapping C++ template functions as Python functions with dtype arguments.

TemplateInvoker takes a templated callable object, a pybind11::dtype object, and a sequence of supported C++ types via its nested Tag struct. The callable is invoked with a scalar argument of the type matching the dtype object. If none of the supported C++ types match, a different error callback is invoked instead.

As an example, we’ll wrap this function:

template <typename T>
T doSomething(std::string const & argument);

TemplateInvoker provides a default error callback, which we’ll use here (otherwise you’d need to pass one when constructing the TemplateInvoker).

For the main callback, we’ll define this helper struct:

struct DoSomethingHelper {

    template <typename T>
    T operator()(T) const {
        return doSomething<T>(argument);
    }

    std::string argument;
};

The pybind11 wrapper for doSomething is then another lambda that uses TemplateInvoker::apply to call the helper:

mod.def(
    "doSomething",
    [](std::string const & argument, py::dtype const & dtype) {
        return TemplateInvoker().apply(
            DoSomethingHelper{argument},
            dtype,
            TemplateInvoker::Tag<int, float, double>()
        );
    },
    "argument"_a
);

The type returned by the helper callable’s operator() can be anything pybind11 knows how to convert to Python.

While defining a full struct with a templated operator() makes it more obvious what TemplateInvoker is doing, it’s much more concise to use a universal lambda with the decltype operator. This wrapper is equivalent to the one above, but it doesn’t need DoSomethingHelper:

mod.def(
    "doSomething",
    [](std::string const & argument, py::dtype const & dtype) {
        return TemplateInvoker().apply(
            [&argument](auto t) { return doSomething<decltype(t)>(argument); },
            dtype,
            TemplateInvoker::Tag<int, float, double>()
        );
    },
    "argument"_a
);

Note that the value of t here doesn’t matter; what’s important is that its C++ type corresponds to the type passed in the dtype argument. So instead of using that value, we use the decltype operator to extract that type and use it as a template parameter.

class WrapperCollection

#include <python.h>

A helper class for subdividing pybind11 module across multiple translation units (i.e. source files).

Merging wrappers for different classes into a single compiled module can dramatically decrease the total size of the binary, but putting the source for multiple wrappers into a single file slows down incremental rebuilds and makes editing unwieldy. The right approach is to define wrappers in different source files and link them into a single module at build time. In simple cases, that’s quite straightforward: pybind11 declarations are just regular C++ statements, and you can factor them out into different functions in different source files.

That approach doesn’t work so well when the classes being wrapped are interdependent, because bindings are only guaranteed to work when all types used in a wrapped method signature have been declared to pybind11 before the method using them is itself declared. Naively, then, each source file would thus have to have multiple wrapper-declaring functions, so all type-wrapping functions could be executed before any method-wrapping functions. Of course, each type-wrapping function would also have to pass its type object to at least one method-wrapping function (to wrap the types own methods), and the result is a tangled mess of wrapper-declaring functions that obfuscate the code with a lot of boilerplate.

WrapperCollection provides a way out of that by allowing type wrappers and their associated methods to be declared at a single point, but the method wrappers wrapped in a lambda to defer their execution. A single WrapperCollection instance is typically constructed at the beginning of a PYBIND11_MODULE block, then passed by reference to wrapper-declaring functions defined in other source files. As type and method wrappers are added to the WrapperCollection by those functions, the types are registered immediately, and the method-wrapping lambdas are collected. After all wrapper-declaring functions have been called, finish() is called at the end of the PYBIND11_MODULE block to execute the collecting method-wrapping lambdas.

Typical usage:

// _mypackage.cc

void wrapClassA(WrapperCollection & wrappers);
void wrapClassB(WrapperCollection & wrappers);

PYBIND11_MODULE(_mypackage, module) {
    WrapperCollection wrappers(module, "mypackage");
    wrapClassA(wrappers);
    wrapClassB(wrappers);
    wrappers.finish();
}

// _ClassA.cc

void wrapClassA(WrapperCollection & wrappers) {
    wrappers.wrapType(
        py::class_<ClassA>(wrappers.module, "ClassA"),
        [](auto & mod, auto & cls) {
            cls.def("methodOnClassA", &methodOnClassA);
        }
    );
}

// _ClassB.cc

void wrapClassB(WrapperCollection & wrappers) {
    wrappers.wrapType(
        py::class_<ClassB>(wrappers.module, "ClassB"),
        [](auto & mod, auto & cls) {
            cls.def("methodOnClassB", &methodOnClassB);
            mod.def("freeFunction", &freeFunction);
        }
    );
}

Note that we recommend the use of universal lambdas (i.e. auto & parameters) to reduce verbosity.

Navigation

Namespace lsst::utils::python¶