Pytest assert error

TL;DR

Time is a precious resource so I won’t waste yours. Here’s how you can assert an exception is raised and how to check that in pytest.

Solution: Use pytest.raises

import pytest

def test_raises_exception():
    with pytest.raises(ZeroDivisionError):
        1 / 0

And here’s how you assert no exception is raised.

Solution: Enclose your code in a try/except block and and if the code raises, you can catch it and print a nice message. pytest is smart enough to make the test fail even if you don’t catch it but having a message makes your test cleaner.

def my_division_function(a, b):
    return a / b

def test_code_raises_no_exception():
    """
    Assert your python code raises no exception.    
    """
    try:
        my_division_function(10, 5)
    except ZeroDivisionError as exc:
        assert False, f"'10 / 5' raised an exception {exc}"

And that’s it, if you want to know more, please follow along.

Introduction

In this tutorial, you’ll learn how to use pytest to:

• assert that an exception is raised
• assert the exception message
• assert the exception type
• assert that an exception is not raised

In a nutshell, we’ll see how to use pytest.raises for each of those cases with examples.

Table of Contents

1. How to Assert That an Exception Is Raised
2. How to Assert That NO Exception Is Raised
3. How to Assert the Exception Message — And Type
4. Conclusion

How to Assert That an Exception Is Raised

In this section, I’m going to show you how you can assert that your code raises an exception. This is a frequent use case and can sometimes tricky. The wonderful thing is, if you are using pytest you can do that in an idiomatic and cleaner way.

Let’s imagine that we have a function that checks for some keys in a dictionary. If a key is not present, it should raise a KeyError. As you can see, this is very generic and doesn’t tell the users much about the error. We can make it cleaner by raising custom exceptions, with different messages depending on the field.

import pytest


class MissingCoordException(Exception):
    """Exception raised when X or Y is not present in the data."""


class MissingBothCoordException(Exception):
    """Exception raised when both X and Y are not present in the data."""


def sum_x_y(data: dict) -> str:
    return data["x"] + data["y"]

Now, time to test this. How can we do that with pytest?

This code is deliberately wrong, as you can see we’re not raising anything. In fact, we want to see test failing first, almost like TDD. After seeing the test failing, we can fix our implementation and re-run the test.

def test_sum_x_y_missing_both():
    data = {"irrelevant": 1}
    with pytest.raises(MissingBothCoordException):
        sum_x_y(data)

Then we get the following output:

============================ FAILURES ============================
________________ test_sum_x_y_missing_both _________________

    def test_sum_x_y_missing_both():
        data = {"irrelevant": 1}
        with pytest.raises(MissingBothCoordException):
>           sum_x_y(data)

test_example.py:33: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

data = {'irrelevant': 1}

    def sum_x_y(data: dict) -> str:
>       return data["x"] + data["y"]
E       KeyError: 'x'

test_example.py:27: KeyError
==================== short test summary info =====================
FAILED test_example.py::test_sum_x_y_missing_both - KeyEr...
======================= 1 failed in 0.02s ========================

Ok, this makes sense, now it’s time to fix it. We’ll check if the data dict has both x and y, otherwise we raise a MissingBothCoordException.

def sum_x_y(data: dict) -> str:
    if "x" not in data and "y" not in data:
        raise MissingBothCoordException("Both x and y coord missing.")
    return data["x"] + data["y"]

And when we re-run the test, it passes.

test_example.py .                                          [100%]

======================= 1 passed in 0.01s ========================

Great! And that is pretty much it. This is how you check if an exception is raised withpytest. In the next section, we’re going to improve our function and we’ll need another test.

How to Assert the Exception Message — And Type

In this section, we’ll improve our sum_x_y function and also the tests. I’ll show you how you can make your test more robust by checking the exception message.

With that in mind, let’s expand the sum_x_y function.

def sum_x_y(data: dict) -> str:
    if "x" not in data and "y" not in data and "extra" not in data:
        raise MissingBothCoordException("Both X and Y coord missing.")
    if "x" not in data:
        raise MissingCoordException("The Y coordinate is not present in the data.")
    if "y" not in data:
        raise MissingCoordException("The Y coordinate is not present in the data.")
    return data["x"] + data["y"]

The new test goes like this:

def test_sum_x_y_has_x_missing_coord():
    data = {"extra": 1, "y": 2}
    with pytest.raises(MissingCoordException):
        sum_x_y(data)

And it passes!

$ poetry run pytest -k test_sum_x_y_has_x_missing_coord
====================== test session starts =======================
collected 2 items / 1 deselected / 1 selected                    

test_example.py .                                          [100%]

================ 1 passed, 1 deselected in 0.01s =================

However, it’s a bit fragile… In case you haven’t noticed it, when "x" is missing, the exception message is: "The Y coordinate is not present in the data.". This is a bug, and one way to detect it is by asserting we return the right message. Thankfully, pytest makes it easier to do.

If we refactor the test to take into account the message, we get the following output:

def test_sum_x_y_has_x_missing_coord():
    data = {"extra": 1, "y": 2}
    with pytest.raises(MissingCoordException) as exc:
        sum_x_y(data)
    assert "The X coordinate is not present in the data." in str(exc.value)

============================ FAILURES ============================
_____________ test_sum_x_y_has_x_missing_coord _____________

def test_sum_x_y_has_x_missing_coord():
        data = {"extra": 1, "y": 2}
        with pytest.raises(MissingCoordException) as exc:
            sum_x_y(data)
>       assert "The X coordinate is not present in the data." in str(exc.value)
E       AssertionError: assert 'The X coordinate is not present in the data.' in 'The Y coordinate is not present in the data.'
E        +  where 'The Y coordinate is not present in the data.' = str(MissingCoordException('The Y coordinate is not present in the data.'))
E        +    where MissingCoordException('The Y coordinate is not present in the data.') = <ExceptionInfo MissingCoordException('The Y coordinate is not present in the data.') tblen=2>.value

test_example.py:32: AssertionError
==================== short test summary info =====================
FAILED test_example.py::test_sum_x_y_has_x_missing_coord
======================= 1 failed in 0.02s ========================

That’s exactly what we want. Let’s fix the code and re-run the test.

def sum_x_y(data: dict) -> str:
    if "x" not in data and "y" not in data and "extra" not in data:
        raise MissingBothCoordException("Both X and Y coord missing.")
    if "x" not in data:
        raise MissingCoordException("The X coordinate is not present in the data.")
    if "y" not in data:
        raise MissingCoordException("The Y coordinate is not present in the data.")
    return data["x"] + data["y"]

And the result…

$ poetry run pytest test_example.py::test_sum_x_y_has_x_missing_coord
====================== test session starts =======================
platform linux -- Python 3.8.5, pytest-6.1.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/miguel/projects/tutorials/pytest-raises
collected 1 item                                                 

test_example.py .                                          [100%]

======================= 1 passed in 0.01s ========================

This is possible because pytest.raises returns an ExceptionInfo object that contains fields such as type, value, traceback and many others. If we wanted to assert the type, we could do something along these lines…

def test_sum_x_y_has_x_missing_coord():
    data = {"extra": 1, "y": 2}
    with pytest.raises(MissingCoordException) as exc:
        sum_x_y(data)
    assert "The X coordinate is not present in the data." in str(exc.value)
    assert exc.type == MissingCoordException

However, we are already asserting that by using pytest.raises so I think asserting the type like this a bit redundant. When is this useful then? It’s useful if we are asserting a more generic exception in pytest.raises and we want to check the exact exception raised. For instance:

def test_sum_x_y_has_x_missing_coord():
    data = {"extra": 1, "y": 2}
    with pytest.raises(Exception) as exc:
        sum_x_y(data)
    assert "The X coordinate is not present in the data." in str(exc.value)
    assert exc.type == MissingCoordException

One more way to assert the message is by setting the match argument with the pattern you want to be asserted. The following example was taken from the official pytest docs.

>>> with raises(ValueError, match='must be 0 or None'):
...     raise ValueError("value must be 0 or None")

>>> with raises(ValueError, match=r'must be d+$'):
...     raise ValueError("value must be 42")

As you can see, we can verify if the expected exception is raised but also if the message matches the regex pattern.

How to Assert That NO Exception Is Raised

The last section in this tutorial is about yet another common use case: how to assert that no exception is thrown. One way we can do that is by using a try / except. If it raises an exception, we catch it and assert False.

def test_sum_x_y_works():
    data = {"extra": 1, "y": 2, "x": 1}

    try:
        sum_x_y(data)
    except Exception as exc:
        assert False, f"'sum_x_y' raised an exception {exc}"

When we run this test, it passes.

$ poetry run pytest test_example.py::test_sum_x_y_works
====================== test session starts =======================
collected 1 item                                                 

test_example.py .                                          [100%]

======================= 1 passed in 0.00s ========================

Now, let’s create a deliberate bug so we can see the test failing. We’ll change our function to raise an ValueError before returning the result.

def sum_x_y(data: dict) -> str:
    if "x" not in data and "y" not in data and "extra" not in data:
        raise MissingBothCoordException("'extra field and x / y coord missing.")
    if "x" not in data:
        raise MissingCoordException("The X coordinate is not present in the data.")
    if "y" not in data:
        raise MissingCoordException("The Y coordinate is not present in the data.")
    raise ValueError("Oh no, this shouldn't have happened.")
    return data["x"] + data["y"]

And then we re-run the test…

    def test_sum_x_y_works():
        data = {"extra": 1, "y": 2, "x": 1}

        try:
            sum_x_y(data)
        except Exception as exc:
>           assert False, f"'sum_x_y' raised an exception {exc}"
E           AssertionError: 'sum_x_y' raised an exception Oh no, this shouldn't have happened.
E           assert False

test_example.py:52: AssertionError
==================== short test summary info =====================
FAILED test_example.py::test_sum_x_y_works - AssertionErr...
======================= 1 failed in 0.02s ========================

It works! Our code raised the ValueError and the test failed!

Conclusion

That’s it for today, folks! I hope you’ve learned something new and useful. Knowing how to test exceptions is an important skill to have. The way pytest does that is, IMHO, cleaner than unittest and much less verbose. In this article, I showed how you can not only assert that your code raises the expected exception, but also assert when they’re not supposed to be raised. Finally, we saw how to check if the exception message is what you expect, which makes test cases more reliable.

Other posts you may like:

• Learn how to unit test REST APIs in Python with Pytest by example.

• 7 pytest Features and Plugins That Will Save You Tons of Time

• How to Use Fixtures as Arguments in pytest.mark.parametrize

• How to Check if an Exception Is Raised (or Not) With pytest

• 7 pytest Plugins You Must Definitely Use

• How to Disable Autouse Fixtures in pytest

• How to Unit Test Complex Data Like Numpy Arrays in Python

• The Best Way to Compare Two Dictionaries in Python

See you next time!

This post was originally published at https://miguendes.me

In these examples I’ll keep both the code under test and the testing function in the same file. You’ve already seen how it would look normally if we imported the functions under test from another module. If not check out the getting started with pytest article.

Also, in order to make the results clear, I’ve removed the summary of the test runs and kept only the actual
error reporting.

Comparing numbers for equality in Pytest

Probably the most basic thing to test is whether a function given some input returns an expected number.

examples/python/pt3/test_number_equal.py

def double(n):
    #return 2*n
    return 2+n

def test_string_equal():
    assert double(2) == 4
    assert double(21) == 42

In the above function double someone has mistakenly used + instead of *. The result of the test looks like this

$ pytest test_number_equal.py

    def test_string_equal():
        assert double(2) == 4
>       assert double(21) == 42
E       assert 23 == 42
E        +  where 23 = double(21)

The line starting with the > sign indicates the assert line that failed. The lines starting with E are the details.

Compare numbers relatively

In certain cases we cannot test for equality. For example if we would like to test if some process finishes within a given time, or whether a timeout is triggered at the right time. In such cases we need to compare if a number is less-than or greater-than some other number.

examples/python/pt3/test_number_less_than.py

def get_number():
    return 23

def test_string_equal():
    assert get_number() < 0 

Running the test will provide the following error message:

$ pytest test_number_less_than.py

    def test_string_equal():
>       assert get_number() < 0
E       assert 23 < 0
E        +  where 23 = get_number()

The error-report looks quite similar to what we had above, but in this case too it is clear what was the comparision operation that failed.

Comparing strings

Similar to numbers we might want to know if a string received from some function is the same as we expect it to be.

examples/python/pt3/test_string_equal.py

def get_string():
    return "abc"

def test_string_equal():
    assert get_string() == "abd"

The result looks familiar:

$ pytest test_string_equal.py

    def test_string_equal():
>       assert get_string() == "abd"
E       AssertionError: assert 'abc' == 'abd'
E         - abc
E         + abd

For such short strings seeing both the expected string and the actual string is ok.
We can look at the strings and compare them character by character to see what was the actual difference.

Compare long strings

If the strings are much longer however, it would be really hard for us to pinpoint the specific location of the character (or characters) that differ. Luckily the authors of Pytest have thought about this problem as well:

examples/python/pt3/test_long_strings.py

import string

def get_string(s):
    return string.printable + s + string.printable

def test_long_strings():
    assert get_string('a') == get_string('b')

string.printable is a string containing all the printable ASCII characters:
0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!»#$%&'()*+,-./:;<=>?@[\]^_`{|}~ tnrx0bx0c

Our brilliant get_string function will return it twice with an additional character between them.
We use this to create two nasty and long strings that differ by a single character.

The output looks like this:

$ pytest test_long_strings.py

    def test_long_strings():
>       assert get_string('a') == get_string('b')
E       AssertionError: assert '0123456789ab...tnrx0bx0c' == '0123456789abc...tnrx0bx0c'
E         Skipping 90 identical leading characters in diff, use -v to show
E         Skipping 91 identical trailing characters in diff, use -v to show
E           {|}~
E
E         - a012345678
E         ? ^
E         + b012345678
E         ? ^

I think this explains quite nicely where have the two strings differ and if you really, really want to see the
whole string you can use the -v flag.

Is string in longer string

If you need to check whether a string is part of a larger string we can use the regular in operator.

examples/python/pt3/test_substring.py

import string

def get_string():
    return string.printable * 30

def test_long_strings():
    assert 'hello' in get_string()

In case of failure the result will include only the beginning and the end of the «long string».
This can be very usefule if you need to test whether a certain string appears or not in an HTML page.

examples/python/pt3/test_substring.txt

    def test_long_strings():
>       assert 'hello' in get_string()
E       assert 'hello' in '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~ tnrx0bx0c012345...x0bx0c0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~ tnrx0bx0c'
E        +  where '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~ tnrx0bx0c012345...x0bx0c0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~ tnrx0bx0c' = get_string()

Testing any expression

Instead of calling a function we might have an expression on one side of the equation. (Actually I am not sure how often this would happen in the real world.
Maybe we only see these in examples on how pytest works.)

examples/python/pt3/test_expression_equal.py

def test_expression_equal():
    a = 3
    assert a % 2 == 0

The test result:

$ pytest test_expression_equal.py

    def test_expression_equal():
        a = 3
>       assert a % 2 == 0
E       assert (3 % 2) == 0

Is element in a list?

Besides comparing individual values we might also want to compare more complex data. First, let’s see what happens if our test must ensure that a value can be found in a list?

examples/python/pt3/test_in_list.py

def get_list():
    return ["monkey", "cat"]

def test_in_list():
    assert "dog" in get_list()

We can use the in operator of Python. The result will look like this:

$ pytest test_in_list.py

    def test_in_list():
>       assert "dog" in get_list()
E       AssertionError: assert 'dog' in ['monkey', 'cat']
E        +  where ['monkey', 'cat'] = get_list()

Pytest will conveniently show us the list that did not contain the expected value.

Compare lists in Pytest

A more interesting case might be testing if the returned list is the same as the expected list. Using the == operator can tell us if the two lists are equal or not, but if we need to understand what went wrong, we’d better know where do the lists differ.
Or at least where do they start to differ.

examples/python/pt3/test_lists.py

import string
import re

def get_list(s):
    return list(string.printable + s + string.printable)

def test_long_lists():
    assert get_list('a') == get_list('b')

The result:

$ pytest test_lists.py

    def test_long_lists():
>       assert get_list('a') == get_list('b')
E       AssertionError: assert ['0', '1', '2...'4', '5', ...] == ['0', '1', '2'...'4', '5', ...]
E         At index 100 diff: 'a' != 'b'
E         Use -v to get the full diff

We could further explore the output for the cases when multiple elements differ and when one list is a sublist of the other.

Compare dictionaries in Pytest

Dictionaries can differ in a number of ways. The keys might be identical, but some values might differ.
Some keys might be missing in the actual result or there might be some extra keys.

In this example we test all of these:

Using the string.printable we create a dictionary where the keys are the printable characters and the values
are their respective ASCII value returned by the ord function. Then we add (or replace) one of key-value pair.

examples/python/pt3/test_dictionaries.py

import string
import re

def get_dictionary(k, v):
    d = dict([x, ord(x)] for x in  string.printable)
    d[k] = v
    return d

def test_big_dictionary_different_value():
    assert get_dictionary('a', 'def') == get_dictionary('a', 'abc')

def test_big_dictionary_differnt_keys():
    assert get_dictionary('abc', 1) == get_dictionary('def', 2)

The result looks like this:

$ pytest test_dictionaries.py

______________ test_big_dictionary_different_value _______________

    def test_big_dictionary_different_value():
>       assert get_dictionary('a', 'def') == get_dictionary('a', 'abc')
E       AssertionError: assert {'t': 9, 'n...x0c': 12, ...} == {'t': 9, 'n'...x0c': 12, ...}
E         Omitting 99 identical items, use -v to show
E         Differing items:
E         {'a': 'def'} != {'a': 'abc'}
E         Use -v to get the full diff

_______________ test_big_dictionary_differnt_keys ________________

    def test_big_dictionary_differnt_keys():
>       assert get_dictionary('abc', 1) == get_dictionary('def', 2)
E       AssertionError: assert {'t': 9, 'n...x0c': 12, ...} == {'t': 9, 'n'...x0c': 12, ...}
E         Omitting 100 identical items, use -v to show
E         Left contains more items:
E         {'abc': 1}
E         Right contains more items:
E         {'def': 2}
E         Use -v to get the full diff

The first test function got two dictionaries where the value of a single key differed.

The second test function had an extra key in both dictionaries.

Testing for expected exceptions in Pytest

Finally let’s look at exceptions!

A good test suite will test the expected behaviour both when the input is fine and
also when the input triggers some exception. Without testing the exception we cannot
be sure that they will be really raiesed when necessary. An incorrect refactoring
might eliminate the error checking of our code therby letting through invalid data
and either triggering a different exception as in our example, or not generating any exception
just silently doing the wrong thing.

In this brilliant example the divide function checks if the divider is 0 and raises it
own type of exception instead of letting Python rais its own. If this is the defined behavior
someone using our module will probably wrap our code in some try expression and expect
a ValueError error. If someone changes our divide function and removed our
special exception then we basically have broken the exception-handling of our user.

examples/python/pt3/test_exceptions.py

import pytest

def divide(a, b):
    if b == 0:
        raise ValueError('Cannot divide by Zero')
    return a / b

def test_zero_division():
    with pytest.raises(ValueError) as e:
        divide(1, 0)
    assert str(e.value) == 'Cannot divide by Zero' 

The test ha actually two parts. The first part:

    with pytest.raises(ValueError) as e:
        divide(1, 0)

checks if a ValueError was raised during our call to divide(1, 0)
and will assign the exception object to the arbitrarily named variable e.

The second part is a plain assert that checks if the text of the exception is what
we expect to be.

This is now the expected behaviour. Our test passes:

$ pytest test_exceptions.py

test_exceptions.py .

What if someone changes the error message in our exception from Zero to Null?

examples/python/pt3/test_exceptions_text_changed.py

import pytest

def divide(a, b):
    if b == 0:
        raise ValueError('Cannot divide by Null')
    return a / b

def test_zero_division():
    with pytest.raises(ValueError) as e:
        divide(1, 0)
    assert str(e.value) == 'Cannot divide by Zero' 

The assert in the test will fail indicating the change in the text.
This is actually a plain string comparision.

$ pytest test_exceptions_text_changed.py


    def test_zero_division():
        with pytest.raises(ValueError) as e:
            divide(1, 0)
>       assert str(e.value) == 'Cannot divide by Zero'
E       AssertionError: assert 'Cannot divide by Null' == 'Cannot divide by Zero'
E         - Cannot divide by Null
E         ?                  ^^^^
E         + Cannot divide by Zero
E         ?                  ^^^^

In the second example we show the case when the special exception raising is gone.
Either by mistake or because someone decided that it should not be there.
In this case the first part of our test function will catch the different exception.

examples/python/pt3/test_exceptions_failing.py

import pytest

def divide(a, b):
#    if b == 0:
#        raise ValueError('Cannot divide by Zero')
    return a / b

def test_zero_division():
    with pytest.raises(ValueError) as e:
        divide(1, 0)
    assert str(e.value) == 'Cannot divide by Zero' 

The report will look like this:

$ pytest test_exceptions_failing.py

    def test_zero_division():
        with pytest.raises(ValueError) as e:
>           divide(1, 0)

test_exceptions_failing.py:10:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = 1, b = 0

    def divide(a, b):
    #    if b == 0:
    #        raise ValueError('Cannot divide by Zero')
>       return a / b
E       ZeroDivisionError: division by zero

Exception depositing money to the bank

Another case when checking for proper exceptions might be important is when
we want to avoid silently incorrect behavior.

For example in this code we have a function called deposit that expects
a non-negative number. We added our input validation that will raise an exception protecting
the balance of our bank account. (In our example we only indicated the location of the code
that actually changes the balance.)

examples/python/pt3/test_bank.py

import pytest

def deposit(money):
    if money < 0:
        raise ValueError('Cannot deposit negative sum')

    # balance += money

def test_negative_deposit():
    with pytest.raises(ValueError) as e:
        deposit(-1)
    assert str(e.value) == 'Cannot deposit negative sum' 

We have also created a test-case that will ensure that the protection is there,
or at least that the function raises an exception if -1 was passed to it.

Conclusion

Pytest and its automatic error reporting is awesome.