Drake
Python Bindings

Overview

Drake uses pybind11 for binding its C++ API to Python.

At present, a fork of pybind11 is used which permits bindings matrices with dtype=object, passing unique_ptr objects, and prevents aliasing for Python classes derived from pybind11 classes.

pybind11 Tips

Python Types

Throughout the Drake code, Python types provided by pybind11 are used, such as py::handle, py::object, py::module, py::str, py::list, etc. For an overview, see the pybind11 reference.

All of these are effectively thin wrappers around PyObject*, and thus can be cheaply copied.

Mutating the referred-to object also does not require passing by reference, so you can always pass the object by value for functions, but you should document your method if it mutates the object in a non-obvious fashion.

Python Type Conversions

You can implicit convert between py::object and its derived classes (such as py::list, py::class_, etc.), assuming the actual Python types agree. You may also implicitly convert from py::object (and its derived classes) to py::handle.

If you wish to convert a py::handle (or PyObject*) to py::object or a derived class, you should use py::reinterpret_borrow<>.

Conventions

Target Conventions

Names

  • *_py: A Python library (can be pure Python or pybind)
    • File Names: *.py, *_py.cc
  • *_pybind: A C++ library for adding pybind-specific utilities to be consumed by C++.
    • File Names: *_pybind.{h,cc}

File names should follow form with their respective target.

Visibility

  • All Python libraries should generally be private, as pydrake will be consumed as one encapsulated target.
  • All C++ *_pybind libraries for binding utilities should be public to aide downstream Bazel projects. If the API is unstable, consider making it private with a TODO to make public once it stabilizes.

Bazel

Given that libdrake.so relies on static linking for components, any common headers should be robust against ODR violations. This can be normally achieved by using header-only libraries.

For upstream dependencies of these libraries, do NOT depend on the direct targets (e.g. //common:essential), because this will introduce runtime ODR violations for objects that have static storage (UID counters, etc.).

Instead, you must temporarily violate IWYU because it will be satisfied by drake_pybind_library, which will incorporate libdrake.so and the transitive headers.

If singletons are required (e.g. for util/cpp_param_pybind), consider storing the singleton values using Python.

If you are developing bindings for a small portion of Drake and would like to avoid rebuilding a large number of components when testing, consider editing //tools/install/libdrake:build_components.bzl to reduce the number of components being built.

pybind Module Definitions

  • Any Drake pybind module should include this header file, pydrake_pybind.h.
  • PYBIND_MODULE should be used to define modules.
  • Modules should be defined within the namespace drake::pydrake.
  • The alias namespace py = pybind11 is defined as drake::pydrake::py. Drake modules should not re-define this alias at global scope.
  • If a certain namespace is being bound (e.g. drake::systems::sensors), you may use using namespace drake::systems::sensors within functions or anonymous namespaces. Avoid using namespace directives otherwise.

Keep Alive Behavior

py::keep_alive is used heavily throughout this code. For more information, please see the pybind11 documentation.

Terse notes are added to method bindings to indicate the patient (object being kept alive by nurse) and the nurse (object keeping patient alive). To expand on them:

  • "Keep alive, ownership" implies that one argument is owned directly by one of the other arguments (self is included in those arguments, for py::init<> and class methods).
  • "Keep alive, reference" implies a reference that is lifetime-sensitive (something that is not necessarily owned by the other arguments).
  • "Keep alive, transitive" implies a transfer of ownership of owned objects from one container to another (e.g. transfering all Systems from DiagramBuilder to Diagram when calling DiagramBuilder.Build()).

Function Overloads

To bind function overloads, please try the following (in order):

  • py::overload_cast<Args>(func): See the pybind11 documentation. This works about 80% of the time.
  • pydrake::overload_cast_explicit<Return, Args...>(func): When py::overload_cast does not work (not always guaranteed to work).
  • static_cast, as mentioned in the pybind11 documentation.
  • Lambdas, e.g. [](Args... args) -> auto&& { return func(args...); } (using perfect forwarding when appropriate).

Interactive Debugging with Bazel

If you would like to interactively debug binding code (using IPython for general Python behavior, or GDB for C++ behavior), debug C++ behavior from a Python binary, while using Bazel, you may expose Bazel's development environment variables by adding these lines to your Python script:

import subprocess
subprocess.Popen(
    "export -p | sed 's# PWD=# OLD_PWD=#g' | tee /tmp/env.sh",
    shell=True)

Run your target once from Bazel, and then source the generated /tmp/env.sh in your terminal to gain access to the environment variables (e.g. $PYTHONPATH).

Example with GDB

This is a brief recipe for debugging with GDB (note the usage of subshell (...) to keep the variables scoped):

(
    target=//bindings/pydrake/systems:lifetime_test
    target_bin=$(echo ${target} | sed -e 's#//##' -e 's#:#/#')
    bazel run -c dbg ${target}
    workspace=$(bazel info workspace)
    name=$(basename ${workspace})
    cd ${workspace}/bazel-${name}
    source /tmp/env.sh
    gdb --args python ${workspace}/bazel-bin/${target_bin}
)

This allows you to use GDB from the terminal, while being able to inspect the sources in Bazel's symlink forests.

If using CLion, consider using gdbserver.