DMTN-013: Wrapping C++ with Cython

  • Pim Schellart

Latest Revision: 2016-03-31

All LSST code currently uses SWIG to generate Python wrappers around C++ code. This document investigates using Cython as an alternative. The prime motivation for this is that AstroPy uses Cython and a closer collaboration and code sharing with AstroPy is currently being evaluated. To start the investigation Jim Bosch has written a C++/Python Bindings Challenge. It consists of “a small suite of C++ classes designed to highlight any of the most common challenges involved in providing Python bindings ot a C++ library, as well as a set of Python unit tests that attempt to measure the quality of the resulting bindings”. Concretely, this document then describes the (partial) Cython solution to this challenge.

About the scope of this document

Although this document contains some examples of how to wrap C++ with Cython and might help to get you started, it is by no means intended to be a full tutorial. The examples are just there to get some context when describing the various things that I encountered. If you are looking for comprehensive information about Cython and C++ please have a look at some of the documents linked below instead.

About Cython

Unlike most C/C++ Python wrapping solutions, Cython is not just a wrapper tool. Instead, it is a full-featured programming language with a Python-like syntax, but with optional added static typing. Additionally, Cython provides a compiler that translates Cython code into C or C++ code, which can subsequently be compiled to a Python extension module by a standard C/C++ compiler.

In my opinion this is both Cython’s main strength as well as one of its weaknesses. On the plus side having a full-blown programming language available that closely matches Python enables the resulting wrapper modules to be very Pythonic - probably more so than any other wrapper tool allows. On the other hand, it means a lot of wrapping code has to be written manually. Because the wrapping language is closer to Python then it is to C++ it is sometimes a little difficult to figure out what exactly is going on behind the scenes and not everything translates one-to-one.

A quick example

Given some simple C++

#ifndef BASICS_H
#define BASICS_H

#include <string>

namespace basics {

int addOne(int x);

class Doodad {
public:
    Doodad(std::string const & name_, int value_) : name{name_}, value{value_} {};

    std::string name;
    int value;
};

} // namespace basics

#endif

a basic Cython wrapper for this would look like this.

from cython.operator cimport dereference as deref

from libcpp.memory cimport unique_ptr
from libcpp.string cimport string

cdef extern from "basics.hpp" namespace "basics":
    int addOne(int)

    cdef cppclass Doodad:
        Doodad(string, int)

        string name
        int value

def pyAddOne(x):
    return addOne(x)

cdef class PyDoodad:
    cdef unique_ptr[Doodad] thisptr

    def __init__(self, name, value):
        self.thisptr.reset(new Doodad(name, value))

    property name:
        def __get__(self):
            return deref(self.thisptr).name
        def __set__(self, _name):
            deref(self.thisptr).name = _name

    property value:
        def __get__(self):
            return deref(self.thisptr).value
        def __set__(self, _value):
            deref(self.thisptr).value = _value

This can then be compiled into a standard (C)Python extension module with the following “setup.py” file. Note that there are other ways to compile the code, some of which even work on the fly while importing, but for purposes of distribution this is probably best.

import sys
from distutils.core import setup, Extension
from Cython.Build import cythonize

compile_args = ['-g', '-std=c++11', '-stdlib=libc++']

if sys.platform == 'darwin':
    compile_args.append('-mmacosx-version-min=10.7')

basics_module = Extension('example.basics',
                sources=['example/basics.pyx', 'example/basics.cpp'],
                extra_compile_args=compile_args,
                language='c++')

setup(
    name='example',
    packages=['example'],
    ext_modules=cythonize(basics_module)
)

This can be built with:

python setup.py build_ext --inplace

Quick example step-by-step

Now let’s examine the wrapper code step by step.

from cython.operator cimport dereference as deref

This line brings in the dereference “operator”. In Cython a pointer dereference, *p can be written either as p[0] or deref(p). I prefer the latter since it seems to work in more contexts. If it is considered too verbose, we can just change the import line to ... cimport dereference as d.

The next two lines:

from libcpp.memory cimport unique_ptr
from libcpp.string cimport string

bring in unique_ptr and string from the C++ standard library. Cython provides very elegant wrappers around the most frequently used standard library types (e.g. shared_ptr, vector, map, unordered_map, etc.).

The block starting with:

cdef extern from "basics.hpp" namespace "basics":
    ...

declares the C++ types (and functions) to be usable from Cython. The only thing this does is place the declarations in the resulting C/C++ file with an extern modifier. Because of this it is sometimes confusingly the users responsibility of ensuring that the declarations here match those on the C++ side - otherwise this is only discovered at link time. This task is further complicated because C++ declarations cannot always be copied entirely verbatim to Cython, which doesn’t allow *, & or qualifiers such as const in all places. But this is a minor nuisance which decreases with increasing understanding. Note that Cython also supports nested namespaces, but only one namespace can be used per extern block.

Now let’s move on to the class definition.

cdef class PyDoodad:
    cdef unique_ptr[Doodad] thisptr

    def __init__(self, name, value):
        self.thisptr.reset(new Doodad(name, value))

    ...

This now is the class that is going to be available from Python (the Doodad itself is strictly C++). This class structure follows a standard approach with Cython. A thisptr member contains a pointer to an instance of the underlying C++ class. In this case we use a std::unique_ptr<Doodad> (template types use square brackets in Cython) to represent ownership and ensure proper lifetime. Most examples online use raw pointers with new in a __cinit__ constructor and delete in a corresponding __dealoc__ (which are guaranteed by Cython to be called before and after the Python constructor and destructor respectively). However, I would recommend using either unique_ptr` or shared_ptr instead, reserving raw pointers for non-owners, in keeping with modern C++ convention.

The remaining code:

property name:
    def __get__(self):
        return deref(self.thisptr).name
    def __set__(self, _name):
        deref(self.thisptr).name = _name
...

deals with Python attributes and should be obvious.

Inspecting (part of) the wrapper code

From a Cython input file the Cython compiler typically generates a single C++ source file as output (without any additional Python code). This is then directly compiled into a CPython extension module. Unfortunately the whole wrapper is 2697 lines long and not very human-readable. Therefore we restrict ourselves to the wrapper for addOne.

Besides a lot of module initialization code and other boilerplate the wrapper consists of two parts. The outer pyAddOne function:

/* "basics.pyx":4
 *     int addOne(int)
 *
 * def pyAddOne(x):             # <<<<<<<<<<<<<<
 *     return addOne(x)
 *
 */

/* Python wrapper */
static PyObject *__pyx_pw_6basics_1pyAddOne(PyObject *__pyx_self, PyObject *__pyx_v_x); /*proto*/
static PyMethodDef __pyx_mdef_6basics_1pyAddOne = {"pyAddOne", (PyCFunction)__pyx_pw_6basics_1pyAddOne, METH_O, 0};
static PyObject *__pyx_pw_6basics_1pyAddOne(PyObject *__pyx_self, PyObject *__pyx_v_x) {
  PyObject *__pyx_r = 0;
  __Pyx_RefNannyDeclarations
  __Pyx_RefNannySetupContext("pyAddOne (wrapper)", 0);
  __pyx_r = __pyx_pf_6basics_pyAddOne(__pyx_self, ((PyObject *)__pyx_v_x));

  /* function exit code */
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}

and a wrapper around the call to addOne itself.

static PyObject *__pyx_pf_6basics_pyAddOne(CYTHON_UNUSED PyObject *__pyx_self, PyObject *__pyx_v_x) {
  PyObject *__pyx_r = NULL;
  __Pyx_RefNannyDeclarations
  int __pyx_t_1;
  PyObject *__pyx_t_2 = NULL;
  int __pyx_lineno = 0;
  const char *__pyx_filename = NULL;
  int __pyx_clineno = 0;
  __Pyx_RefNannySetupContext("pyAddOne", 0);

  /* "basics.pyx":5
 *
 * def pyAddOne(x):
 *     return addOne(x)             # <<<<<<<<<<<<<<
 *
 */
  __Pyx_XDECREF(__pyx_r);
  __pyx_t_1 = __Pyx_PyInt_As_int(__pyx_v_x); if (unlikely((__pyx_t_1 == (int)-1) && PyErr_Occurred())) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 5; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
  __pyx_t_2 = __Pyx_PyInt_From_int(basics::addOne(__pyx_t_1)); if (unlikely(!__pyx_t_2)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 5; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
  __Pyx_GOTREF(__pyx_t_2);
  __pyx_r = __pyx_t_2;
  __pyx_t_2 = 0;
  goto __pyx_L0;

  /* "basics.pyx":4
 *     int addOne(int)
 *
 * def pyAddOne(x):             # <<<<<<<<<<<<<<
 *     return addOne(x)
 *
 */

  /* function exit code */
  __pyx_L1_error:;
  __Pyx_XDECREF(__pyx_t_2);
  __Pyx_AddTraceback("basics.pyAddOne", __pyx_clineno, __pyx_lineno, __pyx_filename);
  __pyx_r = NULL;
  __pyx_L0:;
  __Pyx_XGIVEREF(__pyx_r);
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}

Please note that all the calls to RefNanny as well as __Pyx_GOTREF and __Pyx_GIVEREF simply check Cython’s own reference counting. They do not reference-count user code. Actually, I am not quite sure why they are there at all, since the Cython docs say that they should only be generated when the code is compiled with -DCYTHON_REFNANNY (which it is not). The rest of the code should be pretty self-explanatory.

Solving the C++/Python bindings challenge with Cython

The previous section gave a quick overview of wrapping a C++ class with Cython. This section describes some of the issues encountered while wrapping the C++/Python bindings challenge code. This code was designed to be more representative of the type of code encountered when porting larger swaths of LSST library code.

It contains four C++ source files which are to be compiled into three different Python modules (with interdependencies).

  • basics contains a class Doodad, a class Secret and a struct WhatsIt. The class WhatsIt should be visible to Python only as a tuple, and Secret can only be constructed by Doodad; it is to be passed around in Python as an opaque object. Doodad is the main class to be wrapped.
  • extensions contains a templated class Thingamajig that inherits from Doodad.
  • containers defines a DoodadSet class (backed by a std::vector) that contains Doodads and Thingamajigs.
  • conversions contains various tests for SWIG compatibility (more on this later).

The current solution passes all unit tests for basics, containers and conversions but does not (yet) wrap extensions. This is only due to time constraints and I do not yet foresee any major problems with it.

Diving in

Class name clashes

The first problem encountered is that the unit tests expect the Python classes to be available under the same name as they have in C++. As can be seen in the example above the extern block brings in the C++ classes under their own name (this is required) and therefore doesn’t allow the Python class to be given the same name when declared in the same file.

One can get around this by placing the extern block in a separate .pxd file and then adding

from _basics cimport Doodad as _Doodad

to the top of the .pxd file. Then _Doodad refers to the C++ class while Doodad can be used for Python.

Dealing with const

The second interesting problem pops up when dealing with const objects. How does one represent them on the Python side? And how can it be stored internally? One solution involves keeping two pointers in the object, one unique_ptr[Doodad] and one unique_ptr[const Doodad] and then raising errors if a write operation is accessed for a const-backed object. This has the advantage of presenting one type to the Python user. But then the behaviour of the object is dependent on the backing object. This doesn’t feel very Pythonic. The solution taken instead was to have two different classes on the Python side. A regular writable Doodad (backed by a unique_ptr[Doodad] and a read-only ImmutableDoodad (backed by a unique_ptr[const Doodad]. The latter simply lacks the methods for writing. This approach is more Pythonic IMHO but does require some duplicate code (although one could probably get away with that with some clever subclassing on the Cython side).

Getting a const object

In the challenge, an instance of the previously mentioned ImmutableDoodad is obtained from by calling the Doodad.get_const() static method. In C++ this returns a shared_ptr<const Doodad>. The easiest way of dealing with this is to simply change the backing smart pointer type in the Python object to a shared_ptr as well. This simply follows the standard C++ rule of using shared_ptr for things that you know are going to be shared.

Note that if methods using shared_ptr didn’t exist on the C++ level we could stick to unique_ptr.

Cloning

The C++ class Doodad also has a method called clone() that returns a unique_ptr to a newly copied object. To pass the unit tests the returned object has to be given back to Python without making any additional copies. This is achieved in Cython using:

def clone(self):
    d = Doodad(init=False)

    d.thisptr = move(deref(self.thisptr).clone())

    return d

which also requires std::move to be declared in the .pxd file.

cdef extern from "<utility>" namespace "std" nogil:
    cdef shared_ptr[Doodad] move(unique_ptr[Doodad])
    cdef shared_ptr[Doodad] move(shared_ptr[Doodad])

There are a few things annoying about this. One is that a separate specialization is to be declared for every type of move and the other is that Cython doesn’t like it if two specializations have the same arguments but different return types. The following is thus not allowed.

cdef extern from "<utility>" namespace "std" nogil:
    cdef unique_ptr[Doodad] move(unique_ptr[Doodad])
    cdef shared_ptr[Doodad] move(unique_ptr[Doodad]) # error!
    cdef shared_ptr[Doodad] move(shared_ptr[Doodad])

Which was fortunately not needed in this case but can be really annoying when it is. That this is necessary at all is the result of Cython not knowing about rvalue references. It is a known bug, with so far no solution.

But hey, in this case it works!

Notice also the init=False. This (rather ugly) thing is needed because:

  • C++ Doodad has no default constructor, and
  • __init__ has two arguments with a default value.

You need some way to tell Cython to make a Python Doodad with an uninitialized shared_ptr. Ideally one would want to use a factory Doodad.__new__(Doodad) here, but for some reason this doesn’t play well with Cython (specifically, when called like that it doesn’t seem to add a thisptr).

Comparison operators

Unlike Python, Cython has only one special method to implement all comparison operators called __richcmp__. The current solution, which has to support custom equality and inequality for Doodad and ImmutableDoodad looks like:

def __richcmp__(self, other, int op):
    if op == Py_EQ and isinstance(other, Doodad):
        return isEqualDD(self, other)
    elif op == Py_EQ and isinstance(other, ImmutableDoodad):
        return isEqualDI(self, other)
    elif op == Py_NE and isinstance(other, Doodad):
        return isNotEqualDD(self, other)
    elif op == Py_NE and isinstance(other, ImmutableDoodad):
        return isNotEqualDI(self, other)
    else:
        raise NotImplementedError

where the functions that are called look like:

cdef isEqualDD(Doodad a, Doodad b):
    return a.thisptr.get() == b.thisptr.get()

those functions are simply for convenience since the type of other is not known (and thus not guaranteed to have a thisptr). An alternative is casting at runtime (see SWIG example below).

Containers

Because of the nice availability of vector and map in Cython writing conversion methods to Python list and dict, given the methods as_vector() and as_map() is easy.

cpdef as_list(self):
    cdef vector[shared_ptr[_Doodad]] v = self.inst.as_vector()

    results = []
    for item in v:
        d = Doodad(init=False)
        d.thisptr = move(item)
        results.append(d)

    return results

cpdef as_dict(self):
    cdef map[string, shared_ptr[_Doodad]] m = self.inst.as_map()

    results = {}
    for k in m:
        d = Doodad(init=False)
        d.thisptr = move(k.second)

        results[k.first] = d

    return results

It would have been even easier if the vector or map only included items that were already known to Cython. In that case we could simply return the result directly, without having to build up a new list or dict. In this case however Cython does not know what Python type to put in for the elements. Perhaps this can be fixed somehow?

Inter-Module dependencies

Note that in the previous example something funny is going on. In the methods as_list() and as_dict() we need both the C++ type _Doodad and the Python type Doodad.

Well we can get those with a simple import right?

from _basics cimport _Doodad
from basics import Doodad

Wrong! The imported Python type doesn’t give access to the thisptr member. To solve this we need to split up basics into three modules.

  • _basics.pxd containing the C++ declarations.
  • basics.pxd containing the Cython class declarations.
  • basics.pyx containing the Cython class definitions.

So why can’t we just stick the Cython class declarations in _basics.pxd? Because we need the classes to be named the same!

So in the end we get _basics.pxd:

cdef extern from "basics.hpp" namespace "basics":
    cdef cppclass Doodad:
        ...

and basics.pxd:

from _basics cimport Doodad as _Doodad

cdef class Doodad:
    cdef shared_ptr[_Doodad] thisptr

    ...

and basics.pyx:

from _basics cimport Doodad as _Doodad
from basics cimport Doodad

cdef class Doodad:
    def __init__(self):
        ...

and finally in containers.pyx:

from basics import Doodad
from basics cimport Doodad
from _basics cimport Doodad as _Doodad

...

Granted, the naming could have been nicer...

Iterators

Implementing iterators in Cython is roughly the same as in Python.

cdef class DoodadSet:
    ...

    def __iter__(self):
        self.it = self.inst.begin()
        return self

    def __next__(self):
        if self.it == self.inst.end():
            raise StopIteration()

        d = Doodad(init=False)
        d.thisptr = deref(self.it)

        incr(self.it)

        return d

Note that in this case we use inst instead of thisptr. If a C++ object has a default constructor you can do this. This also allows the objects to be put on the stack (when used from within Cython). This is just to show how it can be done.

Of course begin() and end() have to be declared as well.

cdef extern from "containers.hpp" namespace "containers":
    cdef cppclass DoodadSet:
        vector[shared_ptr[Doodad]].const_iterator begin() const
        vector[shared_ptr[Doodad]].const_iterator end() const

SWIG interoperability

The final thing that is needed is to pass all unit test for conversions to and from SWIG.

The SWIG wrapped extension module converters contains functions like.

std::shared_ptr<basics::Doodad> make_sptr(std::string const & name, int value) {
    return std::shared_ptr<basics::Doodad>(new basics::Doodad(name, value));
}

These then use typemaps declared in basics_typemaps.i such as.

%typemap(out) std::shared_ptr<basics::Doodad> {
    $result = newDoodadFromSptr($1);
}

%typemap(in) std::shared_ptr<basics::Doodad> {
    if (!sptrFromDoodad($input, &$1)) {
        return nullptr;
    }
}

The key to Cython / SWIG interoperability is of course in the functions newDoodadFromSptr (that should take a shared_ptr<Doodad> and return a Python object) and sptrFromDoodad (which does the opposite).

These are implemented in basics.pyx and look like this.

cdef public newDoodadFromSptr(shared_ptr[_Doodad] _d):
    d = Doodad(init=False)
    d.thisptr = move(_d)

    return d

cdef public bool sptrFromDoodad(object _d, shared_ptr[_Doodad] *ptr) except + :
    d = <Doodad?> _d
    ptr[0] = d.thisptr

    return True # cannot catch exception here

The mysterious <Doodad?> is a Cython style cast. It tries to do a conversion and raises an exception if if fails. The except + is needed to allow this exception to be translated and propagated to Python. Cython knows about some default exception types to map to standard Python ones. In this case a TypeError is raised if _d is not a (subclass of) Doodad.

You may also notice move which is declared as described above (in case you skipped to this section immediately).

Now how is this stuff actually called by the C++ SWIG code? Cython has a nice public keyword as written above. It causes the Cython compiler to generate a C++ header file (basics.h) for these function declarations and places that in the build directory. Then it simply gets included by the SWIG build.

A gotcha here is that when calling these functions from C++, the Python module needs to be initialized (or segfaults and other madness ensue).

Therefore in the SWIG module the following needs to be added

%init %{
    PyImport_ImportModule("challenge.basics");
%}

Another problem is linking. In particular, linking on OSX. Since OSX has the concept of bundles (i.e. .so) and dynamic libraries (i.e. .dylib) things get interesting. By default Cython builds bundles. Which is the sensible thing to do because extension modules is what bundles are meant for. However, now basics is not only an extension module but also a library, that our SWIG module wants to link to. There are two solutions. Either we tell Cython that on OSX we want a dynamic library instead (while still calling the resulting thing basics.so). This is done by adding the following to setup.py.

if sys.platform == 'darwin':
    from distutils import sysconfig
    vars = sysconfig.get_config_vars()
    vars['LDSHARED'] = vars['LDSHARED'].replace('-bundle', '-dynamiclib')
    compile_args.append('-mmacosx-version-min=10.7')

The SWIG module can now be built with.

converters_module = Extension(
    'challenge.converters',
    sources=[
        os.path.join('challenge', 'converters.i'),
    ],
    include_dirs=[
        os.path.join('..', 'include'),
        os.path.join('include')
    ],
    swig_opts = ['-modern', '-c++', '-Iinclude', '-noproxy'],
    extra_compile_args=compile_args,
    extra_link_args=[
        os.path.join('challenge', 'basics.so')
    ]
)

Or, as an alternative we can choose not to link to basics.so and instead tell the linker to resolve all symbols when the modules are imported. This can be done by inserting the following in the packages __init__.py file.

import sys

# This sequence will get a lot cleaner for Python 3, for which the necessary
# flags should all be in the os module.
import ctypes
flags = ctypes.RTLD_GLOBAL
try:
    import DLFCN
    flags |= DLFCN.RTLD_NOW
except ImportError:
    flags |= 0x2  # works for Linux and Mac, only platforms I care about now.
sys.setdlopenflags(flags)

# Ensure basics is loaded first, since we need its
# symbols for anything else.
from . import basics

This approach is better since it does not require separate handling of OSX.

See also

  • The full implementation of the Cython solution to the C++/Python bindings challenge is available in the cython branch of my fork on github.
  • A great book on Cython is “Cython - A guide for Python programmers” by Kurt W. Smith.
  • Another excellent source is the online Cython documentation.

Relevant JIRA tickets

  • DM-5470: Develop C++ code for experimenting with Python binding
  • DM-5471: Wrap example C++ code with Cython