QDYN-pylib Development

The source code for the qdyn Python package (“QDYN-pylib”) that accompanies the QDYN Fortran library is organized in the ./qdynpylib subfolder of the QDYN repository.

While QDYN-pylib is developed as part of QDYN, it is publicly released to the Python Package Index (PyPI) and versioned independently from the Fortran QDYN library. This is so that qdyn may be used without restrictions in Notebooks or scripts to process data generated by QDYN. Therefore, the ./qdynpylib folder is self-sufficient for developing and releasing QDYN-pylib, and contains:

  • The package code in src/qdyn (having a src-directory solves problems with testing)

  • Package installation and metadata files setup.py, setup.cfg, and MANIFEST.in. These determines what goes into the source distribution files on PyPI. Dependencies are defined in setup.py.

  • The PyPI-documentation files README.md and HISTORY.md. Both of these are combined into the “Projet Description” on PyPI.

  • The test environment configuration file tox.ini, wrapped by a Makefile that runs the tests in the tests subfolder with pytest.

Testing

Testing of QDYN-pylib is done with pytest, just like the testing of the QDYN Fortran code (see the notes on The Testing Framework). There are two different context for the tests:

  • Self-contained testing, running the tests in the ./qdynpylib/tests folder under all supported Python versions (using tox)

  • Integration testing within the main QDYN testing framework, running tests in both ./tests and ./qdynpylib/tests within the virtual environment used for testing the QDYN Fortran code.

Self-contained tests

The self-contained tests for QDYN-pylib can be found in the ./qdynpylib/tests folder. These tests do not rely on having access to an installation of the QDYN Fortran library. They are included in the source distribution uploaded to PyPI, such that any user of the QDYN-pylib package can run the tests to verify their installation of the package.

The tests can be run against all supported Python version with tox, as configured in the tox.ini file.

To run them, you type e.g.

tox -e py38-test

from inside the ./qdynpylib folder.

Run

tox -av

to see a list of all tox environments.

Warning

The prerequisites for testing with tox are:

  • You must have all Python versions installed against which you want to test. Python 3.8 is the main version, so a python3.8 executable should be in your $PATH. You must also have python3.7 and python3.9 executables in your $PATH if you want to test against these versions

  • At least one of these Python versions must have the tox package installed (python -m pip install tox) such that the tox executable is in your $PATH

The invocation of tox is also wrapped into a Makefile inside the qdnpylib folder. While being inside the ./qdynpylib folder, execute

make test

To run tests for all available Python versions (via tox)

Run simply

make

to see a list of make targets (or, make help).

Warning

In order to run tox, or use the Makefile for QDYN-pylib’s self-contained tests, you must be inside the ./qdynpylib subdirectory of a checkout of the QDYN repository; it will not work if you are in the root of the repository. Moreover, the Makefile in the repository root is separate from ./qdynpylib/Makefile and will run the full QDYN Test Suite instead.

QDYN-pylib must pass testing with tox when making a release; make release will enforce this automatically.

Testing within the QDYN Test Suite

When running

make test

from the root of a checkout of the QDYN repository, pytest will be used to run all the tests of the QDYN projects. This includes the tests of QDYN-pylib in ./qdynpylib/tests and the tests of the Fortran code in ./tests. All tests will be run within the ./venv environment that the configure script sets up.

The main ./tests folder may also contain “integration tests” where QDYN-pylib is tested in combination with the Fortran code. Moreover, tests of the Fortran code are encouraged to routinely use QDYN-pylib for generating test data and verifying results, implicitly testing the compatibility of the Fortran and Python code.

If you want to just quickly run the isolated QDYN-pylib tests in the context of the main venv (and don’t want to bother with tox as recommended in the previous section), you can run

make test-qdynpylib

Warning

Make sure that you are in the root of the QDYN repository when running make test or make test-qdynpylib: The main QDYN Makefile in the project root is different from the file ./qdynpylib/Makefile use for self-contained testing, see above.

Coverage

When running the full QDYN test suite (make test in the repository root), coverage data for QDYN-pylib will be collected, and an HTML report will be generated n the ./qdynpylib/htmlcov. This report shows which line in each Python file has been run during testing. Ideally, the “coverage” (the percentage of lines run) should be above 90%.

When doing self-contained testing (make test in the ./qdynpylib subdirectory), a coverage report can also be generated by running make coverage). Note that this coverage will only include the self-contained tests; coverage will be lower than that reported by the main QDYN test suite. This is fine: we do not need full coverage for the isolated tests. In particular, some parts of the QDYN-pylib such as qdyn.testing only make sense in the context of the full QDYN repository and cannot be tested in isolation.

Running make clean either from the repository root for the ./qdynpylib directory will remove the coverage report.

Code Style

All code must be compatible with PEP 8. The line length limit is 79 characters, although exceptions are permissible if this improves readability significantly.

Beyond PEP 8, this project adopts the Black code style. You can run make black-check or tox -e run-blackcheck from inside the ./qdynpylib directory to check adherence to the code style, and make black or tox -e run-black to apply it.

Imports within python modules must be sorted according to the isort configuration in ./qdynpylib/setup.cfg. The command make isort-check or tox -e run-isortcheck (run from inside ./qdynpylib) checks whether all imports are sorted correctly, and make isort or tox -e run-isort modifies all Python modules in-place with the proper sorting.

Documentation

The documentation of QDYN-pylib is intermixed with the main QDYN documentation (e.g., this very page). Any how-to, example, or topical overview related to QDYN-pylib should be written according to the Writing Documentation Guide.

There are two “self-contained” parts of the QDYN-pylib documentation:

  • The files README.md and HISTORY.md in the ./qdynpylib directory: These do not show up in the rendered Sphinx documentation, but only on PyPI. They are targeted as a minimal documentation for users who do not have access to the full QDYN, for example someone who just wants to reproduce an analysis for data generated by QDYN. They must not reference any other part of the documentation. The fact that these files are written using Markdown formatting instead of the RestructuredText Syntax used for regular documentation encourages this separation.

  • The API documentation in the form of docstrings in the Python source code, rendered as QDYN-pylib API.

For the API documentation, each function or class must have a docstring; this docstring must be written in the “Google Style” format (as implemented by Sphinx’ napoleon extension). The __init__ method of a Python class should never have a docstring; it’s arguments are described in the class docstring instead.

Python docstrings use the same RestructuredText Syntax as the Fortran docstrings. They can include mathematical formulas in LaTeX syntax (using mathjax). Inline math should be enclosed in dollar signs, see the guidelines for writing math in the main QDYN documentation. Use semantic macros such as \ket{\Psi} instead of \vert\Psi\rangle. Use the same conventions for Bibliographic References as in the Fortran API, by putting Local reference lists in module, class or function docstrings when appropriate.

Warning

Within Python docstrings, the “primary domain” is py, whereas for all other parts of the documnation if it f (for Fortran). Thus, :mod:`qdyn` inside a Python docstring is equivalent to py:mod:`qdyn` and links to qdyn, wheras :mod:`qdyn` in any Fortran docstring or RST document is equivalent to f:mod:`qdyn` and links to qdyn. See also References to Fortran objects and References to Python objects.

Use the :f prefix when referencing Fortran objecs from Python docstrings, and the :py prefix when referencing Python objects from Fortran docstrings!

Versioning and Releases

QDYN-pylib is independently versioned from QDYN as a whole, in order to allow for PyPI releases. The version number is defined in ./qdynpylib/src/qdyn/__init__.py and must be compatible with PEP 440. It is available through the __version__ attribute of the qdyn package.

In short, versions number follow the pattern major.minor.patch, e.g. 0.1.0 for the first release, and 1.0.0 for the first stable release. A major version of 0 indicates that package API has not stabilized and may change in every minor-version release.

Between releases, __version__ on the main branch should either be the version number of the last release, with “+dev” appended (as a “local version identifier”), or the version number of the next planned release, with “-dev” appended (“pre-release identifier” with extra dash). The “+dev” suffix must never be included in a release to PyPI.

The qdyn package is released to PyPI by running

make release

from inside the ./qdynpylib folder and following the prompts.

Releases should be tagged with e.g. qdynpylib-v0.1.0. Running make release will do this automatically.