Class name clashes¶

The first problem encountered is that the unit tests expect the Python classes to be available under the same name as they have in C++. As can be seen in the example above the extern block brings in the C++ classes under their own name (this is required) and therefore doesn’t allow the Python class to be given the same name when declared in the same file.

One can get around this by placing the extern block in a separate .pxd file and then adding

from _basics cimport Doodad as _Doodad

to the top of the .pxd file. Then _Doodad refers to the C++ class while Doodad can be used for Python.

Dealing with const¶

The second interesting problem pops up when dealing with const objects. How does one represent them on the Python side? And how can it be stored internally? One solution involves keeping two pointers in the object, one unique_ptr[Doodad] and one unique_ptr[const Doodad] and then raising errors if a write operation is accessed for a const-backed object. This has the advantage of presenting one type to the Python user. But then the behaviour of the object is dependent on the backing object. This doesn’t feel very Pythonic. The solution taken instead was to have two different classes on the Python side. A regular writable Doodad (backed by a unique_ptr[Doodad] and a read-only ImmutableDoodad (backed by a unique_ptr[const Doodad]. The latter simply lacks the methods for writing. This approach is more Pythonic IMHO but does require some duplicate code (although one could probably get away with that with some clever subclassing on the Cython side).

Getting a const object¶

In the challenge, an instance of the previously mentioned ImmutableDoodad is obtained from by calling the Doodad.get_const() static method. In C++ this returns a shared_ptr<const Doodad>. The easiest way of dealing with this is to simply change the backing smart pointer type in the Python object to a shared_ptr as well. This simply follows the standard C++ rule of using shared_ptr for things that you know are going to be shared.

Note that if methods using shared_ptr didn’t exist on the C++ level we could stick to unique_ptr.

Cloning¶

The C++ class Doodad also has a method called clone() that returns a unique_ptr to a newly copied object. To pass the unit tests the returned object has to be given back to Python without making any additional copies. This is achieved in Cython using:

def clone(self):
    d = Doodad(init=False)

    d.thisptr = move(deref(self.thisptr).clone())

    return d

which also requires std::move to be declared in the .pxd file.

cdef extern from "<utility>" namespace "std" nogil:
    cdef shared_ptr[Doodad] move(unique_ptr[Doodad])
    cdef shared_ptr[Doodad] move(shared_ptr[Doodad])

There are a few things annoying about this. One is that a separate specialization is to be declared for every type of move and the other is that Cython doesn’t like it if two specializations have the same arguments but different return types. The following is thus not allowed.

cdef extern from "<utility>" namespace "std" nogil:
    cdef unique_ptr[Doodad] move(unique_ptr[Doodad])
    cdef shared_ptr[Doodad] move(unique_ptr[Doodad]) # error!
    cdef shared_ptr[Doodad] move(shared_ptr[Doodad])

Which was fortunately not needed in this case but can be really annoying when it is. That this is necessary at all is the result of Cython not knowing about rvalue references. It is a known bug, with so far no solution.

But hey, in this case it works!

Notice also the init=False. This (rather ugly) thing is needed because:

• C++ Doodad has no default constructor, and
• __init__ has two arguments with a default value.

You need some way to tell Cython to make a Python Doodad with an uninitialized shared_ptr. Ideally one would want to use a factory Doodad.__new__(Doodad) here, but for some reason this doesn’t play well with Cython (specifically, when called like that it doesn’t seem to add a thisptr).

Comparison operators¶

Unlike Python, Cython has only one special method to implement all comparison operators called __richcmp__. The current solution, which has to support custom equality and inequality for Doodad and ImmutableDoodad looks like:

def __richcmp__(self, other, int op):
    if op == Py_EQ and isinstance(other, Doodad):
        return isEqualDD(self, other)
    elif op == Py_EQ and isinstance(other, ImmutableDoodad):
        return isEqualDI(self, other)
    elif op == Py_NE and isinstance(other, Doodad):
        return isNotEqualDD(self, other)
    elif op == Py_NE and isinstance(other, ImmutableDoodad):
        return isNotEqualDI(self, other)
    else:
        raise NotImplementedError

where the functions that are called look like:

cdef isEqualDD(Doodad a, Doodad b):
    return a.thisptr.get() == b.thisptr.get()

those functions are simply for convenience since the type of other is not known (and thus not guaranteed to have a thisptr). An alternative is casting at runtime (see SWIG example below).

Containers¶

Because of the nice availability of vector and map in Cython writing conversion methods to Python list and dict, given the methods as_vector() and as_map() is easy.

cpdef as_list(self):
    cdef vector[shared_ptr[_Doodad]] v = self.inst.as_vector()

    results = []
    for item in v:
        d = Doodad(init=False)
        d.thisptr = move(item)
        results.append(d)

    return results

cpdef as_dict(self):
    cdef map[string, shared_ptr[_Doodad]] m = self.inst.as_map()

    results = {}
    for k in m:
        d = Doodad(init=False)
        d.thisptr = move(k.second)

        results[k.first] = d

    return results

It would have been even easier if the vector or map only included items that were already known to Cython. In that case we could simply return the result directly, without having to build up a new list or dict. In this case however Cython does not know what Python type to put in for the elements. Perhaps this can be fixed somehow?

Inter-Module dependencies¶

Note that in the previous example something funny is going on. In the methods as_list() and as_dict() we need both the C++ type _Doodad and the Python type Doodad.

Well we can get those with a simple import right?

from _basics cimport _Doodad
from basics import Doodad

Wrong! The imported Python type doesn’t give access to the thisptr member. To solve this we need to split up basics into three modules.

• _basics.pxd containing the C++ declarations.
• basics.pxd containing the Cython class declarations.
• basics.pyx containing the Cython class definitions.

So why can’t we just stick the Cython class declarations in _basics.pxd? Because we need the classes to be named the same!

So in the end we get _basics.pxd:

cdef extern from "basics.hpp" namespace "basics":
    cdef cppclass Doodad:
        ...

and basics.pxd:

from _basics cimport Doodad as _Doodad

cdef class Doodad:
    cdef shared_ptr[_Doodad] thisptr

    ...

and basics.pyx:

from _basics cimport Doodad as _Doodad
from basics cimport Doodad

cdef class Doodad:
    def __init__(self):
        ...

and finally in containers.pyx:

from basics import Doodad
from basics cimport Doodad
from _basics cimport Doodad as _Doodad

...

Granted, the naming could have been nicer...

Iterators¶

Implementing iterators in Cython is roughly the same as in Python.

cdef class DoodadSet:
    ...

    def __iter__(self):
        self.it = self.inst.begin()
        return self

    def __next__(self):
        if self.it == self.inst.end():
            raise StopIteration()

        d = Doodad(init=False)
        d.thisptr = deref(self.it)

        incr(self.it)

        return d

Note that in this case we use inst instead of thisptr. If a C++ object has a default constructor you can do this. This also allows the objects to be put on the stack (when used from within Cython). This is just to show how it can be done.

Of course begin() and end() have to be declared as well.

cdef extern from "containers.hpp" namespace "containers":
    cdef cppclass DoodadSet:
        vector[shared_ptr[Doodad]].const_iterator begin() const
        vector[shared_ptr[Doodad]].const_iterator end() const

SWIG interoperability¶

The final thing that is needed is to pass all unit test for conversions to and from SWIG.

The SWIG wrapped extension module converters contains functions like.

std::shared_ptr<basics::Doodad> make_sptr(std::string const & name, int value) {
    return std::shared_ptr<basics::Doodad>(new basics::Doodad(name, value));
}

These then use typemaps declared in basics_typemaps.i such as.

%typemap(out) std::shared_ptr<basics::Doodad> {
    $result = newDoodadFromSptr($1);
}

%typemap(in) std::shared_ptr<basics::Doodad> {
    if (!sptrFromDoodad($input, &$1)) {
        return nullptr;
    }
}

The key to Cython / SWIG interoperability is of course in the functions newDoodadFromSptr (that should take a shared_ptr<Doodad> and return a Python object) and sptrFromDoodad (which does the opposite).

These are implemented in basics.pyx and look like this.

cdef public newDoodadFromSptr(shared_ptr[_Doodad] _d):
    d = Doodad(init=False)
    d.thisptr = move(_d)

    return d

cdef public bool sptrFromDoodad(object _d, shared_ptr[_Doodad] *ptr) except + :
    d = <Doodad?> _d
    ptr[0] = d.thisptr

    return True # cannot catch exception here

The mysterious <Doodad?> is a Cython style cast. It tries to do a conversion and raises an exception if if fails. The except + is needed to allow this exception to be translated and propagated to Python. Cython knows about some default exception types to map to standard Python ones. In this case a TypeError is raised if _d is not a (subclass of) Doodad.

You may also notice move which is declared as described above (in case you skipped to this section immediately).

Now how is this stuff actually called by the C++ SWIG code? Cython has a nice public keyword as written above. It causes the Cython compiler to generate a C++ header file (basics.h) for these function declarations and places that in the build directory. Then it simply gets included by the SWIG build.

A gotcha here is that when calling these functions from C++, the Python module needs to be initialized (or segfaults and other madness ensue).

Therefore in the SWIG module the following needs to be added

%init %{
    PyImport_ImportModule("challenge.basics");
%}

Another problem is linking. In particular, linking on OSX. Since OSX has the concept of bundles (i.e. .so) and dynamic libraries (i.e. .dylib) things get interesting. By default Cython builds bundles. Which is the sensible thing to do because extension modules is what bundles are meant for. However, now basics is not only an extension module but also a library, that our SWIG module wants to link to. There are two solutions. Either we tell Cython that on OSX we want a dynamic library instead (while still calling the resulting thing basics.so). This is done by adding the following to setup.py.

if sys.platform == 'darwin':
    from distutils import sysconfig
    vars = sysconfig.get_config_vars()
    vars['LDSHARED'] = vars['LDSHARED'].replace('-bundle', '-dynamiclib')
    compile_args.append('-mmacosx-version-min=10.7')

The SWIG module can now be built with.

converters_module = Extension(
    'challenge.converters',
    sources=[
        os.path.join('challenge', 'converters.i'),
    ],
    include_dirs=[
        os.path.join('..', 'include'),
        os.path.join('include')
    ],
    swig_opts = ['-modern', '-c++', '-Iinclude', '-noproxy'],
    extra_compile_args=compile_args,
    extra_link_args=[
        os.path.join('challenge', 'basics.so')
    ]
)

Or, as an alternative we can choose not to link to basics.so and instead tell the linker to resolve all symbols when the modules are imported. This can be done by inserting the following in the packages __init__.py file.

import sys

# This sequence will get a lot cleaner for Python 3, for which the necessary
# flags should all be in the os module.
import ctypes
flags = ctypes.RTLD_GLOBAL
try:
    import DLFCN
    flags |= DLFCN.RTLD_NOW
except ImportError:
    flags |= 0x2  # works for Linux and Mac, only platforms I care about now.
sys.setdlopenflags(flags)

# Ensure basics is loaded first, since we need its
# symbols for anything else.
from . import basics

This approach is better since it does not require separate handling of OSX.