Launching tests
===============

pytest
------

Since we use pytest_ to run the tests, you can also use it on its own.
We do recommend using the tox_ command though::

    pytest pylint -k test_functional

You can use pytest_ directly. If you want to run tests on a specific portion of the
code with pytest_ and your local python version::

    python3 -m pytest


Everything in tests/message with coverage for the relevant code (require `pytest-cov`_)::

    python3 -m pytest tests/message/ --cov=pylint.message
    coverage html

Only the functional test "missing_kwoa_py3"::

    python3 -m pytest "tests/test_functional.py::test_functional[missing_kwoa_py3]"

tox
---

You can also *optionally* install tox_ and run our tests using the tox_ package, as in::

    python -m tox
    python -m tox -epy38 # for Python 3.8 suite only
    python -m tox -epylint # for running Pylint over Pylint's codebase
    python -m tox -eformatting # for running formatting checks over Pylint's codebase

It's usually a good idea to run tox_ with ``--recreate``. This flag tells tox_ to re-download
all dependencies before running the tests. This can be important when a new version of
astroid_ or any of the other dependencies has been published::

    python -m tox --recreate # The entire tox environment will be recreated
    python -m tox --recreate -e py310 # The python 3.10 tox environment will be recreated


To run only a specific test suite, use a pattern for the test filename
(**without** the ``.py`` extension), as in::

    python -m tox -e py310 -- -k test_functional
    python -m tox -e py310 -- -k  \*func\*
    python -m tox --recreate -e py310 -- -k test_functional # With recreation of the environment


.. _primer_tests:

Primer tests
------------

Pylint also uses what we refer to as ``primer`` tests. These are tests that are run automatically
in our Continuous Integration and check whether any changes in Pylint lead to crashes or fatal errors
on the ``stdlib`` and a selection of external repositories.

To run the ``primer`` tests you can add either ``--primer-stdlib`` or ``--primer-external`` to the
pytest_ command. If you want to only run the ``primer`` you can add either of their marks, for example::

    pytest -m primer_stdlib --primer-stdlib

The external ``primer`` can be run with::

    pytest -m primer_external_batch_one --primer-external # Runs batch one

The list of repositories is created on the basis of three criteria: 1) projects need to use a diverse
range of language features, 2) projects need to be well maintained and 3) projects should not have a codebase
that is too repetitive. This guarantees a good balance between speed of our CI and finding potential bugs.

You can find the latest list of repositories and any relevant code for these tests in the ``tests/primer``
directory.

.. _pytest-cov: https://pypi.org/project/pytest-cov/
.. _astroid: https://github.com/pycqa/astroid
