Managing Side Effects and Randomness

All Python tests and doctests are executed by one Python process. All Blender tests and doctests are executed by one Blender process. This means that if tests have side effects, then adding, removing, reordering and even rerunning tests can change the semantics for all tests.

The Problem

Getting random numbers from a random number generator changes its state. Suppose we use the same (deterministically seeded) random number generator for test_f, test_g and test_h (in that order) and then omit test_f. Then test_g would receive the random numbers that test_f previously received and and test_g would receive the random numbers that test_h previously received.
The vast majority of Blender specific functions are supposed to add, mutate and delete Blender data in some way or another. For example, if test_f and test_g are run in that order and test_f adds a Blender object, then test_g will fail if it assumes that bpy.data.objects is empty.

The Solution

We use pytest fixtures to provide a “clean slate” for each test. They are passed to tests via the pytest config files.

For the random number generator fixture, see Random tests. For the Blender fixture, see testing/utils/clear_scene_plugin.py.

Random tests

It can be good for various reasons to use random instead of handpicked data for testing. To handle the randomness well, we use a mechanism based on pytest fixtures:

Basic usage

Every random test should look roughly like this:

def test_something(rng):  # The argument "rng" is a magic keyword
    a = rng.standard_normal(3)
    assert a.size == 3

rng stores a pre-seeded random generator instance returned by np.random.default_rng. For basic usage, you can just treat the argument and variable rng like a magic incantation that gives you such a random generator. There are also two other fixtures seed and rng_seed that work similarly but give you seeds instead of random generators. rng_seed is the exact seed that is also used to seed rng and seed is a different seed you can use for other purposes.

The full mechanism

rng is a pytest fixture, which is really a special kind of function. Writing a test function with an argument called rng “requests” the fixture, in pytest lingo. The return value of the fixture can then be used inside your test function as rng. Fixtures can also request other fixtures. For details, see the pytest docs linked above.

The process works like this: Each test has a unique ID. There is also a global salt which can be set in the test config .ini files or passed to pytest using the --salt command line option. The fixtures use these two strings to generate deterministic seeds that are local and unique to each test and easily refreshable by changing the salt.

The code for all these fixtures can be found in testing/utils/rng_plugin.py. It is a customized version of an existing pytest plugin. See the plugin’s module description for details.