I messed up at work recently. We (a small start-up) have a staging server and a production server. Nothing gets deployed into production until it’s run on staging for a while and seems to be solid. But still I managed to mess this up.

The problem started when I upgraded the deployment from Python 3.4 to 3.5. This can be done without downtime (I just create a fresh VM behind the load balancer and then cut over), but it’s still a little risky. Long story short, I ended up cutting over staging but leaving the production server for a little while to wait for an appropriate maintenance window. Technically staging and production are different, but what does it matter? We’re not using any advanced Python features, or so I thought…

As it happens, while we were in this situation I hit a Pylint warning because one of my modules went over the length limit. It’s a bit of a silly warning, but I decided to attempt to break the file down rather than just ignore it. But of course the two resultant modules (models and query_sets in my case) ended up depending on each other. I ended up doing:

from . import models

in one module and:

from . import models

in the other one. This means that the two modules only see each other’s members in module-qualified form (e.g. models.Foo) rather than in bare form, which is actually an advantage in my opinion, since it starts to emphasise separation between the modules.

Unfortunately, this worked fine in my staging setup (Python 3.5) and then died when it was deployed to production (Python 3.4). I decided to dig into why this is.

Interlude: How circular dependencies (don’t) work in Python

If you already understand the ramifications of circular dependencies in Python you can skip this bit.

Everyone knows there’s a difference between:

from mymodule import Foo

and

import mymodule

What a lot of people don’t realise is that this goes a lot deeper than just how you qualify Foo when you use it in your code.

First thing to understand: Python recurses through imports depth-first. The reason that circular imports don’t cause an infinite loop is that each module is added to sys.modules at the beginning of processing, and if the module is already in sys.modules then the import step is skipped (though namespace modifications are still made to the local module, which will be important later). This means that in a circular import, the recursion is cut off as soon as it hits a module that’s already been visited, even if the module is not yet completely imported (because it’s further up the stack).

Second thing to understand: A python module is (among other things) a namespace, which in Python really means it’s got a dict that maps from names to objects (functions, classes, occasionally bare variables) contained in the module.

Third thing to understand: Definitions in a Python module are executed one at a time, and added to the module’s namespace as they are executed. At the top of the file the namespace is empty. As the definitions are processed it gets updated until by the bottom of the file it contains all that we expect it to.

Now, this sequence doesn’t normally matter because we can usually treat an import statement as a single atomic step from the point of view of the code calling the import. The fact that the module namespace starts out empty and gets filled up step by step doesn’t matter, because we only see the namespace once it’s populated.

Which brings us to the fourth thing to understand: import a and from a import * do completely different things as regards namespaces. The statement:

import mypackage

creates a reference to the module’s namespace (with the name mypackage) in the local module. This is an alias to the real namespace, so as the mypackage namespace gets populated, the changes to that are seen in real time by the code that imported it.

However, if you do:

from mypackage import *

then you copy the objects from the mypackage namespace into the local namespace. This is a one-time operation: once you’ve done the copy once, subsequent changes to the namespace don’t get reflected in the importing module’s namespace. How could they?

Again, you don’t really care about this sequencing until you hit a circular import. Circular imports matter a lot, because if widget imports gadget imports widget, then the second widget import will be skipped, and gadget will get imported with an incomplete copy of widget.

All of this means that if you’re in a situation with circular imports, you should avoid trying to import objects directly into the local namespace. Instead, import whole packages and rely on the fact that by the time your code actually executes, the packages will be populated.

Why this broke my code in production

When I introduced the circular dependency I was aware of all the above issues, and was careful to import whole packages rather than attempting to bring package contents into scope. I thought this would protect me. However, I was still using the import ... from form of the import statement, which turns out to be an issue.

Here’s the C code that implements import ... from in Python 3.4:

static PyObject *
import_from(PyObject *v, PyObject *name)
{
    PyObject *x;
 
    x = PyObject_GetAttr(v, name);
    if (x == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) {
        PyErr_Format(PyExc_ImportError, "cannot import name %R", name);
    }
    return x;
}

This gets called after the module has been loaded from disk and compiled. In our case, since we’re deep in a recursive call when this happens, Python will try to load the module and find it is already (in the process of being) loaded. It then calls import_from to make the namespace alterations.

If you’re not familiar with Python internals you can ignore most of the details, but notice that this is basically just a call to getattr() with some error checking. The getattr() is going to fail because the module has been loaded, but the parent package’s namespace hasn’t been updated (which doesn’t happen until the module loading is complete).

This failure is unnecessary. The module exists in sys.modules, so it’s perfectly possible for the system to load it from there even if it’s not part of the parent package’s namespace yet. In fact, this is what is done in Python 3.5, following this bug report, which discusses the details. The equivalent code is now:

static PyObject *
import_from(PyObject *v, PyObject *name)
{
    PyObject *x;
    _Py_IDENTIFIER(__name__);
    PyObject *fullmodname, *pkgname;
 
    x = PyObject_GetAttr(v, name);
    if (x != NULL || !PyErr_ExceptionMatches(PyExc_AttributeError))
        return x;
    PyErr_Clear();
    pkgname = _PyObject_GetAttrId(v, &PyId___name__);
    if (pkgname == NULL) {
        goto error;
    }
    fullmodname = PyUnicode_FromFormat("%U.%U", pkgname, name);
    Py_DECREF(pkgname);
    if (fullmodname == NULL) {
        return NULL;
    }
    x = PyDict_GetItem(PyImport_GetModuleDict(), fullmodname);
    Py_DECREF(fullmodname);
    if (x == NULL) {
        goto error;
    }
    Py_INCREF(x);
    return x;
 error:
    PyErr_Format(PyExc_ImportError, "cannot import name %R", name);
    return NULL;
}

Again, you can ignore a lot of the details here, but notice the pattern: First it tries getattr(), which will work in most cases. If that doesn’t work, it builds up the full module name in dotted form (by appending .modulename to the parent package name) and looks it up in sys.modules.