Using pybind11

The package pybind11 is provides an elegant way to wrap C++ code for Python, including automatic conversions for numpy arrays and the C++ Eigen linear algebra library. Used with the cppimport package, this provides a very nice work flow for integrating C++ and Python:

• Edit C++ code
• Run Python code

pip install pybind11
pip install cppimport

Clone the Eigen library - no installation is required as Eigen is a header only library.

hg clone https://bitbucket.org/eigen/eigen/

A first example of using pybind11

Create a new subdirectory - e.g. example1 and create the following 5 files in it:

• funcs.hpp
• funcs.cpp
• wrap.cpp
• setup.py
• test_funcs.py

First write the C++ header and implementation files

In funcs.hpp

int add(int i, int j);

In funcs.cpp

int add(int i, int j) {
    return i + j;
};

Next write the C++ wrapper code using pybind11 in wrap.cpp. The arguments "i"_a=1, "j"_a=2 in the exported function definition tells pybind11 to generate variables named i with default value 1 and j with default value 2 for the add function.

#include <pybind11/pybind11.h>
#include "funcs.hpp"

namespace py = pybind11;

using namespace pybind11::literals;

PYBIND11_PLUGIN(wrap) {
    py::module m("wrap", "pybind11 example plugin");
    m.def("add", &add, "A function which adds two numbers",
          "i"_a=1, "j"_a=2);
    return m.ptr();
}

Finally, write the setup.py file to compile the extension module. This is mostly boilerplate.

import os, sys

from distutils.core import setup, Extension
from distutils import sysconfig

cpp_args = ['-std=c++11', '-stdlib=libc++', '-mmacosx-version-min=10.7']

ext_modules = [
    Extension(
    'wrap',
        ['funcs.cpp', 'wrap.cpp'],
        include_dirs=['pybind11/include'],
    language='c++',
    extra_compile_args = cpp_args,
    ),
]

setup(
    name='wrap',
    version='0.0.1',
    author='Cliburn Chan',
    author_email='cliburn.chan@duke.edu',
    description='Example',
    ext_modules=ext_modules,
)

Now build the extension module in the subdirectory with these files

python setup.py build_ext -i

And if you are successful, you should now see a new funcs.so extension module. We can write a test_funcs.py file test the extension module:

import wrap

def test_add():
    assert(wrap.add(3, 4) == 7)

if __name__ == '__main__':
    test_add()

And finally, running the test should not generate any error messages:

$ python test_funcs.py

Using cppimport

In the development stage, it can be distracting to have to repeatedly rebuild the extension module by running

python setup.py clean
python setup.py build_ext -i

every single time you modify the C++ code. The cppimport package does this for you.

Create a new sub-directory exaample2 and copy the files func.hpp, funcs.cpp and wrap.cpp from example1 over. For the previous example, we just need to add some annotation (between <% and %> delimiters) to the top of the wrap.cpp file

<%
cfg['compiler_args'] = ['-std=c++11', '-stdlib=libc++', '-mmacosx-version-min=10.7']
cfg['sources'] = ['funcs.cpp']
setup_pybind11(cfg)
%>

#include "funcs.hpp"
#include <pybind11/pybind11.h>

namespace py = pybind11;

PYBIND11_PLUGIN(wrap) {
    py::module m("wrap", "pybind11 example plugin");
    m.def("add", &add, "A function which adds two numbers");
    return m.ptr();
}

and use cpppimport in the test_funcs.py file.

import cppimport
funcs = cppimport.imp("wrap")

def test_add():
    assert(funcs.add(3, 4) == 7)

if __name__ == '__main__':
    test_add()

You can now run

$ python test_funcs.py

without any need to manually build the extension module. Any updates will be detected by cppimport and it will automatically trigger a re-build.

Vectorizing functions for use with numpy arrays

Example showing how to vectorize a square function

<%
cfg['compiler_args'] = ['-std=c++11', '-stdlib=libc++', '-mmacosx-version-min=10.7']
setup_pybind11(cfg)
%>

#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>

namespace py = pybind11;

double square(double x) {
    return x * x;
}

PYBIND11_PLUGIN(code) {
    pybind11::module m("code", "auto-compiled c++ extension");
    m.def("square", py::vectorize(square));
    return m.ptr();
}

and the test file

import cppimport
import numpy as np

code = cppimport.imp("code")

if __name__ == '__main__':
    xs = np.random.rand(10)
    print(xs)
    print(code.square(xs))

    ys = range(10)
    print(code.square(ys))

Using numpy arrays as function arguments and return values

Example showing how to pass numpy arrays in and out of functions. These numpy array arguments can either be generic py:array or typed py:array_t<double>. The properties of the numpy array can be obtained by calling its request method. This returns a struct of the following form:

struct buffer_info {
    void *ptr;
    size_t itemsize;
    std::string format;
    int ndim;
    std::vector<size_t> shape;
    std::vector<size_t> strides;
};

Here is C++ code for two functions - the function twice shows how to change a passed in numpy array in-place using pointers; the function sum shows how to sum the elements of a numpy array. By taking advantage of the information in buffer_info, the code will work for arbitrary n-d arrays.

<%
cfg['compiler_args'] = ['-std=c++11', '-stdlib=libc++', '-mmacosx-version-min=10.7']
setup_pybind11(cfg)
%>

#include <pybind11/pybind11.h>
#include <pybind11/numpy.h>

namespace py = pybind11;

// Passing in an array of doubles
void twice(py::array_t<double> xs) {
    py::buffer_info info = xs.request();
    auto ptr = static_cast<double *>(info.ptr);

    int n = 1;
    for (auto r: info.shape) {
      n *= r;
    }

    for (int i = 0; i <n; i++) {
        *ptr++ *= 2;
    }
}

// Passing in a generic array
double sum(py::array xs) {
    py::buffer_info info = xs.request();
    auto ptr = static_cast<double *>(info.ptr);

    int n = 1;
    for (auto r: info.shape) {
      n *= r;
    }

    double s = 0.0;
    for (int i = 0; i <n; i++) {
        s += *ptr++;
    }

    return s;
}

PYBIND11_PLUGIN(code) {
    pybind11::module m("code", "auto-compiled c++ extension");
    m.def("sum", &sum);
    m.def("twice", &twice);
    return m.ptr();
}

and the test code

import cppimport
import numpy as np

code = cppimport.imp("code")

if __name__ == '__main__':
    xs = np.arange(12).reshape(3,4).astype('float')
    print(xs)
    print("np :", xs.sum())
    print("cpp:", code.sum(xs))

    print()
    code.twice(xs)
    print(xs)

Using the C++ eigen library to calculate matrix inverse and determinant

Example showing how Eigen vectors and matrices can be passed in and out of C++ functions. Note that Eigen arrays are automatically converted to numpy arrays simply by including the pybind/eigen.h header.

<%
cfg['compiler_args'] = ['-std=c++11', '-stdlib=libc++', '-mmacosx-version-min=10.7']
cfg['include_dirs'] = ['/Users/cliburn/hg/eigen']
setup_pybind11(cfg)
%>

#include <pybind11/pybind11.h>
#include <pybind11/eigen.h>

#include <Eigen/LU>

namespace py = pybind11;

Eigen::MatrixXd inv(Eigen::MatrixXd xs) {
    return xs.inverse();
}

double det(Eigen::MatrixXd xs) {
    return xs.determinant();
}

PYBIND11_PLUGIN(code) {
    pybind11::module m("code", "auto-compiled c++ extension");
    m.def("inv", &inv);
    m.def("det", &det);
    return m.ptr();
}

and test code

import cppimport
import numpy as np

code = cppimport.imp("code")

if __name__ == '__main__':
    A = np.array([[1,2,1],
                  [2,1,0],
                  [-1,1,2]])

    print(A)
    print(code.det(A))
    print(code.inv(A))