.. _running_tests:

Running Tests
=============

Tests can be roughly grouped into the following:

- The **Python tests** are all the tests that are independent of Blender.
- The **Blender tests** are the tests that use Blender.
- The **doctests** are the tests run on the code snippets in our user guide.
  These are run within the Python and Blender test suites respectively.
- The **style tests** are formatting tests on the code base.
- The **linters** which nitpick your code so that reviewers won't need to.

.. contents:: Table of contents
   :local:
   :backlinks: none


The test command-line interface: test.py
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

All tests can be run using the ``test.py`` cli.
From project root directory, it can be invoked with the commands

.. code-block:: bash

    ./test.py

or

.. code-block:: bash

    python3 test.py

By adding arguments to the above command, you can have more control over which tests to
run and how to run them.
To see all available tests run

.. code-block:: bash

   python3 test.py --help


Passing more arguments
~~~~~~~~~~~~~~~~~~~~~~

You can pass more arguments to the underlying test runners, e.g. ``pytest``, ``sphinx`` and so on.
Refer to the ``test.py`` help for more details.
Some useful flags follow in the sequel, do refer to ``pytest -h``, ``sphinx -h``, ``black -h``, ``flake8 -h`` for many more.

Display test names
++++++++++++++++++
Pytest only shows green and red dots by default, but it can also display the test names with

.. code-block:: bash

    python3 test.py -p -- -v
    python3 test.py -b -- -v
    python3 test.py --blender-examples -- -v

Running tests and building documentation in parallel
++++++++++++++++++++++++++++++++++++++++++++++++++++
This is particularly useful for Python tests (`-p`).
Note that this doesn't work for Blender tests (`--blender`) and Blender example tests (`--blender -examples`), because all of them manipulate global Blender state.

.. code-block:: bash

    python3 test.py -p -- -v -n auto

and 

.. code-block:: bash

    # clean and build the docs
    python3 test.py -d -- -j auto

    # without cleaning, might result in buggy output
    make SPHINXOPTS="-j auto" -C docs html

Don't capture logs and output
+++++++++++++++++++++++++++++
This is particularly useful for the Blender examples, because otherwise pytest will swallow Blender's rendering logs.
It can also be used to see the result of `print` even when the tests succeed
This is incompatible with `-n auto`.

.. code-block:: bash

    python3 test.py -p -- -v -s
    python3 test.py --blender-examples -- -v -s

Stopping at first error (-x)
++++++++++++++++++++++++++++

.. code-block:: bash

   python3 test.py -p -- -x

Checking which tests would be collected (--co)
++++++++++++++++++++++++++++++++++++++++++++++

Use the ``--co`` or ``--collect-only`` flag to get a list of tests that would be run, but
don't run them.

.. code-block:: bash

   python3 test.py -p -- --co

Collect tests from a directory or file
++++++++++++++++++++++++++++++++++++++

Works for both Python and Blender tests.

.. code-block:: bash

    ./test.py -p -- testing/tests/ddg/datastructures

    ./test.py -b -- testing/tests/ddg/blender/mesh_test.py testing/tests/ddg/blender/bmesh_test.py

Note that this overwrites the test paths defined in the Python and Blender test configuration (see :doc:`python_tests` and :doc:`blender_tests`).
That is, you could end up running tests requiring Blender when you try to run Python tests.
To avoid this, either make sure all the tests in you subdirectory correspond to what you intend to test.
Alternatively, use finer control mechanisms such as marks (-m) or substring matching (-k), see below.

Use marks to collect tests (-m)
+++++++++++++++++++++++++++++++

You can add marks to your test functions to select which ones to test or exclude. To do
this import pytest in your Python test file and add the following decorator
above it : ``@pytest.mark.<your-mark>``

You can then control the test collection using the flag ``-m`` followed by a
``MARKEXPR``. For example

.. code-block:: bash

    ./test.py -p -- -m 'mark1 and not mark2'

would run all tests marked ``mark1`` but omit the ones marked ``mark2``.
More detail on ``MARKEXPR`` can be found in ``pytest -h``.

Use a substring to collect tests (-k)
+++++++++++++++++++++++++++++++++++++

The ``-k`` option followed by an expression selects functions and files names matching
the expression. For example

.. code-block:: bash

    ./test.py -p -- -k 'quadric and not sphere'

would only run Python tests that have the substring ``quadric`` in
their name but not the substring ``sphere``.

Test Coverage
~~~~~~~~~~~~~

To create a html code coverage report for the Python API use

.. code-block:: bash

    python3 test.py --coverage
    firefox coverage/index.html