Class WrapperCollection

Class Documentation

class WrapperCollection

A helper class for subdividing pybind11 module across multiple translation units (i.e. source files).

Merging wrappers for different classes into a single compiled module can dramatically decrease the total size of the binary, but putting the source for multiple wrappers into a single file slows down incremental rebuilds and makes editing unwieldy. The right approach is to define wrappers in different source files and link them into a single module at build time. In simple cases, that’s quite straightforward: pybind11 declarations are just regular C++ statements, and you can factor them out into different functions in different source files.

That approach doesn’t work so well when the classes being wrapped are interdependent, because bindings are only guaranteed to work when all types used in a wrapped method signature have been declared to pybind11 before the method using them is itself declared. Naively, then, each source file would thus have to have multiple wrapper-declaring functions, so all type-wrapping functions could be executed before any method-wrapping functions. Of course, each type-wrapping function would also have to pass its type object to at least one method-wrapping function (to wrap the types own methods), and the result is a tangled mess of wrapper-declaring functions that obfuscate the code with a lot of boilerplate.

WrapperCollection provides a way out of that by allowing type wrappers and their associated methods to be declared at a single point, but the method wrappers wrapped in a lambda to defer their execution. A single WrapperCollection instance is typically constructed at the beginning of a PYBIND11_MODULE block, then passed by reference to wrapper-declaring functions defined in other source files. As type and method wrappers are added to the WrapperCollection by those functions, the types are registered immediately, and the method-wrapping lambdas are collected. After all wrapper-declaring functions have been called, finish() is called at the end of the PYBIND11_MODULE block to execute the collecting method-wrapping lambdas.

Typical usage: 

// _mypackage.cc

void wrapClassA(WrapperCollection & wrappers);
void wrapClassB(WrapperCollection & wrappers);

PYBIND11_MODULE(_mypackage, module) {
    WrapperCollection wrappers(module, "mypackage");
    wrapClassA(wrappers);
    wrapClassB(wrappers);
    wrappers.finish();
}

// _ClassA.cc

void wrapClassA(WrapperCollection & wrappers) {
    wrappers.wrapType(
        py::class_<ClassA>(wrappers.module, "ClassA"),
        [](auto & mod, auto & cls) {
            cls.def("methodOnClassA", &methodOnClassA);
        }
    );
}

// _ClassB.cc

void wrapClassB(WrapperCollection & wrappers) {
    wrappers.wrapType(
        py::class_<ClassB>(wrappers.module, "ClassB"),
        [](auto & mod, auto & cls) {
            cls.def("methodOnClassB", &methodOnClassB);
            mod.def("freeFunction", &freeFunction);
        }
    );
}

Note that we recommend the use of universal lambdas (i.e. auto & parameters) to reduce verbosity.

Public Types

using WrapperCallback = std::function<void(pybind11::module&)>

Function handle type used to hold deferred wrapper declaration functions.

Public Functions

WrapperCollection(pybind11::module module_, std::string const &package)

Construct a new WrapperCollection.

A WrapperCollection should be constructed at or near the top of a PYBIND11_MODULE block.

Parameters

  • [in] module_: Module instance passed to the PYBIND11_MODULE macro.

  • [in] package: String name of the package all wrapped classes should appear to be from (by resetting their __module__ attribute). Note that this can lead to problems if classes are not also lifted into the package namespace in its __init__.py (in addition to confusing users, this will prevent unpickling from working).

WrapperCollection(WrapperCollection &&other)
WrapperCollection(WrapperCollection const&)
WrapperCollection &operator=(WrapperCollection const&)
WrapperCollection &operator=(WrapperCollection&&)
~WrapperCollection()
WrapperCollection makeSubmodule(std::string const &name)

Create a WrapperCollection for a submodule defined in the same binary.

WrapperCollections created with makeSubmodule should generally be destroyed by moving them into a call to collectSubmodule; this will cause all deferred definitions to be executed when the parent WrapperCollection’s finish() method is called.

Attributes added to the returned

WrapperCollection will actually be put in a submodule that adds an underscore prefix to name, with __module__ set with the expectation that they will be lifted into a package without that leading underscore by a line in __init__.py like: 
from ._package import _submodule as submodule
Parameters

  • name: Relative name of the submodule.

This is necessary to make importing _package possible when submodule already exists as a normal (i.e. directory-based) package. Of course, in that case, you’d instead use a submodule/__init__.py with a line like: 

from .._package._submodule import *

Return

a new WrapperCollection instance that sets the __module__ of any classes added to it to {package}.{name}.

void collectSubmodule(WrapperCollection &&submodule)

Merge deferred definitions in the given submodule into the parent WrapperCollection.

Parameters

  • submodule: A WrapperCollection created by makeSubmodule. Will be consumed (and must be an rvalue).

void addInheritanceDependency(std::string const &name)

Indicate an external module that provides a base class for a subsequent addType call.

Dependencies that provide base classes cannot be deferred until after types are declared, and are always imported immediately.

Parameters

  • [in] name: Name of the module to import (absolute).

void addSignatureDependency(std::string const &name)

Indicate an external module that provides a type used in function/method signatures.

Dependencies that provide classes are imported after types in the current module are declared and before any methods and free functions in the current module are declared.

Parameters

  • [in] name: Name of the module to import (absolute).

void wrap(WrapperCallback function)

Add a set of wrappers without defining a class.

Parameters

  • [in] function: A callable object that takes a single pybind11::module argument (by reference) and adds pybind11 wrappers to it, to be called later by finish().

template<typename PyType, typename ClassWrapperCallback>
PyType wrapType(PyType cls, ClassWrapperCallback function, bool setModuleName = true)

Add a type (class or enum) wrapper, deferring method and other attribute definitions until finish() is called.

Return

the cls argument for convenience

Parameters

  • [in] cls: A pybind11::class_ or enum_ instance.

  • [in] function: A callable object that takes a pybind11::module argument and a pybind11::class_ (or enum_) argument (both by reference) and defines wrappers for the class’s methods and other attributes. Will be called with this->module and cls by finish().

  • [in] setModuleName: If true (default), set cls.__module__ to the package string this WrapperCollection was initialized with.

template<typename CxxException, typename CxxBase>
auto wrapException(std::string const &pyName, std::string const &pyBase, bool setModuleName = true)

Wrap a C++ exception as a Python exception.

Return

a pybind11::class_ instance (template parameters unspecified) representing the Python type of the new exception.

Template Parameters

  • CxxException: C++ Exception type to wrap.

  • CxxBase: Base class of CxxException.

Parameters

  • [in] pyName: Python name of the new exception.

  • [in] pyBase: Python name of the pex::exceptions Exception type the new exception inherits from.

  • [in] setModuleName: If true (default), set cls.__module__ to the package string this WrapperCollection was initialized with.

void finish()

Invoke all deferred wrapper-declaring callables.

finish() should be called exactly once, at or near the end of a PYBIND11_MODULE block.

Public Members

pybind11::module module

The module object passed to the PYBIND11_MODULE block that contains this WrapperCollection.