Assertion rewriting in Pytest part 2: Simple workarounds

Part 1 looked at why a test framework can’t offer a convenient interface using ordinary functions alone. In this part I’ll look at some ad hoc code that might be used to implement something better. Hopefully this illustrates some of the things that Python can do and sets the boundaries for what a test framework might want to do.

So, to recap: The problem is that if we call a function with a condition that we want to test, the function can’t see out of its scope to figure out the details of the condition. Let’s make this a bit more concrete:

def run_all_tests():
    results = []
    # ...
    results.append(test_something())
    # ...
 
def test_something():
    foo = MyClass()
    bar = foo.do_something()
    assert_with_details(bar == 42)
    return 'pass'
 
def assert_with_details(condition):
    if not condition:
        print("Failed") 
        # We want to print the details of what failed here

When this is executed, we’ll end up with an execution stack like the following (supposing we suspended execution at the moment the assertion fails):

assert_with_details condition = False
test_something foo = <MyClass object>

bar = 43

run_all_tests results = ['pass', 'pass']

The first thing we might try is to pass a string expression into assert_with_details. That way the function can see the expression being executed. We then have to do extra work to actually evaulate the condition, but that’s possible with eval:

def test_something():
    foo = MyClass()
    bar = foo.do_something()
    assert_with_details("bar == 42")
    return 'pass'
 
def assert_with_details(condition):
    if not eval(condition):
        print("Condition {condition} failed".format(condition=condition))

It looks like we’re getting somewhere: This can print diagnostics about the failed condition. However, there are two problems:

  • eval is kind of a nasty habit to get into and tends to lead to code that is hard to follow and insecure. We won’t worry too much about security, because we’re writing a unit test library and so we needn’t defend against malicious users; our users are probably the same people who wrote the code.
  • It doesn’t work: The eval in assert_with_details doesn’t have access to the bar variable.

The second of these is obviously the more serious problem. Our stack looks like this:

assert_with_details condition = "bar == 42"
test_something foo =

bar = 43

run_all_tests results = ['pass', 'pass']

To make the eval work, assert_with_details would have to reach out of its own stack frame and look at the local variables in the caller’s stack frame. A function can’t normally do this, for very sensible reasons: the whole point of a function is to provide code modularity, which requires isolating the called code from the calling code.Is it possible to work around this?

As usual in Python, the answer is “yes”:

import inspect
 
def get_value():
    return 43
 
def run_all_tests():
    results = []
    results.append(test_something())def test_something():
    foo = get_value()
    assert_with_details("foo == 42")
 
def assert_with_details(condition):
    result = eval(condition,
                  globals(),
                  inspect.stack()[1][0].f_locals)
 
    if not result:
        print('Condition "{condition}" failed'.format(condition=condition))

The magic happens here:

result = eval(condition,
              inspect.stack()[1].frame.f_globals,
              inspect.stack()[1].frame.f_locals)

eval allows you to control the context in which it runs, by changing the local and global variables it has access to. inspect.stack() allows you to examine the stack, and therefore to get access to the context that was available to the caller of the current function. By giving eval the globals and locals from the parent frame instead of the current frame, we can make the expression act as if it had been evaluated by the calling function.

This solution isn’t ideal. For one, we have to write:

assert_with_details("foo === 42")

when we’d much rather write:

assert_with_details(foo == 42)

The former doesn’t get proper syntax highlighting and won’t get support from the IDE for autocomplete, nor from Pylint or whatever. This is because the IDE is going to see this as a string rather than Python code.Never the less, I think it’s worth reflecting on what Python has allowed us to do here. This is plain Python using just the standard library, and isn’t relying on undocumented internals. It required virtually no code to be written either.

If this was the best we could do, it might still be better than what other languages offer us. Even so, Python can go one better as I’ll explain in the next part.

Leave a Reply

Your email address will not be published. Required fields are marked *