New Testing Features in Django 3.2

1. setUpTestData() isolation

The release note for this reads:

“Objects assigned to class attributes in TestCase.setUpTestData() are now isolated for each test method.”

setUpTestData() is a very useful hook for making your tests fast, and this change makes its use a lot easier.

It’s common to use the TestCase.setUp() hook from unittest to create model instances that each test uses:

from django.test import TestCase

from example.core.models import Book


class ExampleTests(TestCase):
    def setUp(self):
        self.book = Book.objects.create(title="Meditations")

The test runner calls setUp() before each test. This gives simple isolation, since the data is fresh for each test. The downside of this approach is that setUp() runs many times, which can get quite slow with large amounts of data or tests.

setUpTestData() allows you to create the data at the class-level, so only once per ``TestCase``. Its use is very similar to setUp(), only it’s a class method:

from django.test import TestCase

from example.core.models import Book


class ExampleTests(TestCase):
    @classmethod
    def setUpTestData(cls):
        cls.book = Book.objects.create(title="Meditations")

Django continues to roll back any changes in the database between tests, so they remain isolated there. Unfortunately, until this change in Django 3.2, the rollback did not occur in memory, so any modifications to the model instances would persist. This meant tests weren’t fully isolated.

Take these tests for example:

from django.test import TestCase
from example.core.models import Book


class SetUpTestDataTests(TestCase):
    @classmethod
    def setUpTestData(cls):
        cls.book = Book.objects.create(title="Meditations")

    def test_that_changes_title(self):
        self.book.title = "Antifragile"

    def test_that_reads_title_from_db(self):
        db_title = Book.objects.get().title
        assert db_title == "Meditations"

    def test_that_reads_in_memory_title(self):
        assert self.book.title == "Meditations"

If we run them on Django 3.1, the final test fails:

$ ./manage.py test example.core.tests.test_setuptestdata
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.F.
======================================================================
FAIL: test_that_reads_in_memory_title (example.core.tests.test_setuptestdata.SetUpTestDataTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../example/core/tests/test_setuptestdata.py", line 19, in test_that_reads_in_memory_title
    assert self.book.title == "Meditations"
AssertionError

----------------------------------------------------------------------
Ran 3 tests in 0.002s

FAILED (failures=1)
Destroying test database for alias 'default'...

This is because the in-memory change from test_that_changes_title() persists between the tests.

The change in Django 3.2 changes this by copying the objects on access in each test, so each test uses its own isolated copy of the in-memory model instance. The tests now pass:

$ ./manage.py test example.core.tests.test_setuptestdata
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
...
----------------------------------------------------------------------
Ran 3 tests in 0.002s

OK
Destroying test database for alias 'default'...

Thanks to Simon Charette for creating this functionality, initially in the django-testdata project before merging it into Django core. On older Django versions, you can use django-testdata for the same isolation, by adding its @wrap_testdata decorator on your setUpTestData() methods. It’s great and I have been adding it to every project I work on.

(Older Django versions also document a workaround involving re-querying the database in each test, but that is slower.)

2. faulthandler enabled by default

The release note for this reads:

“DiscoverRunner now enables faulthandler by default.”

This is a small improvement that can help you debug low level failures. Python’s faulthandler module provides a way to dump an “emergency” traceback in response to problems that crash the Python interpreter. Django’s test runner now enables faulthandler. This was copied from pytest which does the same.

The problems faulthandler catches normally occur in C-based libraries, such as database drivers. For example, the OS signal SIGSEGV indicates a segmentation fault, which means an attempt to read memory that does not belong to the current process. We can emulate this in Python by directly sending the signal to ourselves:

import os
import signal

from django.test import SimpleTestCase


class FaulthandlerTests(SimpleTestCase):
    def test_segv(self):
        # Directly trigger the segmentation fault
        # signal, which normally occurs due to
        # unsafe memory access in C
        os.kill(os.getpid(), signal.SIGSEGV)

If we this test on Django 3.1, we see this:

$ ./manage.py test example.core.tests.test_faulthandler
System check identified no issues (0 silenced).
[1]    31127 segmentation fault  ./manage.py test

This is not much help as there’s no clue to what caused the segmentation fault.

On Django 3.2, we instead see a traceback:

$ ./manage.py test example.core.tests.test_faulthandler
System check identified no issues (0 silenced).
Fatal Python error: Segmentation fault

Current thread 0x000000010ed1bdc0 (most recent call first):
  File "/.../example/core/tests/test_faulthandler.py", line 12 in test_segv
  File "/.../python3.9/unittest/case.py", line 550 in _callTestMethod
  ...
  File "/.../django/test/runner.py", line 668 in run_suite
  ...
  File "/..././manage.py", line 17 in main
  File "/..././manage.py", line 21 in <module>
[1]    31509 segmentation fault  ./manage.py test

(Cut down for brevity.)

faulthandler can’t produce exactly the same traceback as a normal Python exception, but it gives us plenty of information to debug the crash.

3. Timing

The release note for this reads:

“DiscoverRunner can now track timings, including database setup and total run time.”

The manage.py test command has gained the --timing option which activates a few lines of output at the end of the test run to summarize database setup and teardown timing:

$ ./manage.py test --timing
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
...
----------------------------------------------------------------------
Ran 3 tests in 0.002s

OK
Destroying test database for alias 'default'...
Total database setup took 0.019s
  Creating 'default' took 0.019s
Total database teardown took 0.000s
Total run took 0.028s

Thanks to Ahmad A. Hussein for contributing this as part of Google Summer of Code 2020.

If you’re using pytest, its --durations N option works similarly. Due the pytest fixture system, the database setup time will appear as “setup” time for just one test, making that test seem slower than it really is.

4. Test transaction.on_commit() callbacks

The release note for this reads:

“The new TestCase.captureOnCommitCallbacks() method captures callback functions passed to transaction.on_commit() in a list.” “This allows you to test such callbacks without using the slower TransactionTestCase.”

This is a contribution I made, and covered previously.

Imagine you’re using Django’s ATOMIC_REQUESTS option to wrap each view in a transaction (and I think you should be!). You then need to use transaction.on_commit() to perform any actions that depend upon data being durably stored in the database. For example, in this simple view for a contact form:

from django.db import transaction
from django.views.decorators.http import require_http_methods

from example.core.models import ContactAttempt


@require_http_methods(("POST",))
def contact(request):
    message = request.POST.get("message", "")
    attempt = ContactAttempt.objects.create(message=message)

    @transaction.on_commit
    def send_email():
        send_contact_form_email(attempt)

    return redirect("/contact/success/")

We don’t send the email unless the transaction commits, as otherwise there won’t be a ContactAttempt in the database.

This is the right way to write a view like this, but it has previously been tricky to test the callback passed to on_commit(). Django won’t run the callback without committing the transaction, and its TestCase avoids commits, instead rolling back the transaction that it wraps around ecah test.

One solution is to use TransactionTestCase, which allows your code’s transactions to commit. But this is much slower than TestCase, since it clears all tables between tests.

(I previously covered a three times speedup from converting tests from TransactionTestCase to TestCase.)

The solution in Django 3.2 is the new captureOnCommitCallbacks(), which we use as a context manager. This captures any callbacks and allows you to make assertions on them or execute them to test their effects. We can use it to test our view like so:

from django.core import mail
from django.test import TestCase

from example.core.models import ContactAttempt


class ContactTests(TestCase):
    def test_post(self):
        with self.captureOnCommitCallbacks(execute=True) as callbacks:
            response = self.client.post(
                "/contact/",
                {"message": "I like your site"},
            )

        assert response.status_code == 302
        assert response["location"] == "/contact/success/"
        assert ContactAttempt.objects.get().message == "I like your site"
        assert len(callbacks) == 1
        assert len(mail.outbox) == 1
        assert mail.outbox[0].subject == "Contact Form"
        assert mail.outbox[0].body == "I like your site"

We use captureOnCommitCallbacks() around our test client post request to the view, passing the execute flag to indicate that a “fake commit” should run all the callbacks. We then assert on the HTTP response and the state of the database, before checking the email sent by the callback. Our test then covers everything in the view whilst remaining fast - nice!

To use captureOnCommitCallbacks() on earlier Django versions, install django-capture-on-commit-callbacks.

5. assertQuerysetEqual() Improved

The release note for this starts:

“TransactionTestCase.assertQuerysetEqual() now supports direct comparison against another queryset…”

If your tests use assertQuerysetEqual(), this change will definitely improve your life.

Prior to Django 3.2, assertQuerysetEqual() requires you to compare a QuerySet after a transformation, which defaults to repr(). So tests using it would typically pass a list of precomputed repr() strings to compare against:

from django.test import TestCase

from example.core.models import Book


class AssertQuerySetEqualTests(TestCase):
    def test_comparison(self):
        Book.objects.create(title="Meditations")
        Book.objects.create(title="Antifragile")

        self.assertQuerysetEqual(
            Book.objects.order_by("title"),
            ["<Book: Antifragile>", "<Book: Meditations>"],
        )

The assertion requires a bit more thought about which model instances are expected. And it requires test churn if the model’s __repr__() method changes.

From Django 3.2, we can pass a QuerySet or list of objects to compare against, allowing us to simplify the test:

from django.test import TestCase

from example.core.models import Book


class AssertQuerySetEqualTests(TestCase):
    def test_comparison(self):
        book1 = Book.objects.create(title="Meditations")
        book2 = Book.objects.create(title="Antifragile")

        self.assertQuerysetEqual(
            Book.objects.order_by("title"),
            [book2, book1],
        )

Thanks to Peter Inglesby and Hasan Ramezani for contributing this change. It has made great improvements to Django’s own test suite.

Fin

Enjoy these changes when Django 3.2 is out, or before via the backport packages,

—Adam