Continuous Integration in pySDC#

Every commit in pySDC is tested by GitHub continuous integration (CI). You can see the tests for each branch in the action panel. Those tests are currently divided in three main categories : code linting, code testing and code coverage. Finally, the CI also build artifacts that are used to generate the documentation website (see http://parallel-in-time.org/pySDC/), more details given in the documentation generation section.

CI Workflow Triggers#

The CI pipeline is configured to avoid redundant runs. The workflow is triggered:

  • On pull requests: CI runs for all pull request events (opened, synchronized, reopened, etc.)

  • On push to master (and to release branches matching v*): CI runs when code is pushed there directly

  • On schedule: CI runs weekly on Monday at 5:01 AM UTC (via cron schedule)

This configuration ensures that when you push commits to a pull request branch, the CI runs only once for the pull request event, not twice (once for the PR and once for the push). Direct pushes to the master branch will still trigger the CI to ensure the main branch is always tested.

đź”” A branch with no open pull request gets no CI. This is deliberate: a bare push trigger fires alongside pull_request for the same commit and doubles the ~42-job matrix. Run the tests locally while you work, and open a pull request (a draft one is enough) as soon as you want the full matrix to run.

Code linting#

Code style linting is performed using black and ruff for code syntax checking. In particular, black is used to check compliance with (most of) PEP-8 guidelines.

Those tests are conducted for each commit (even for forks), but you can also run it locally in the root folder of pySDC before pushing any commit :

# Install required packages (works also with conda/mamba)
pip install black ruff
# First : test code style linting with black
black pySDC --check --diff --color
# Second : test code syntax with ruff
ruff check pySDC

đź”” To avoid any error about formatting (black), you can simply use this program to reformat your code directly using the command :

black pySDC

Some style rules that are automatically enforced :

  • lines should be not longer than 120 characters

  • arithmetic operators (+, *, …) should be separated with variables by one empty space

You can automate linting somewhat by using git hooks. In order to run black automatically, we want to do a pre-commit hook which adds the modified files to the commit after reformatting. To this end, just add the following to a possibly new file in the path <pySDC-root-directory>/.git/hooks/pre-commit:

#!/bin/sh

# get files that have been staged
export files=$(git diff --staged --name-only HEAD | grep .py | sed -e "s,^,$(git rev-parse --show-toplevel)/,")

# remove any deleted files because black will otherwise fail
for file in $files
do
        if [ ! -f "$file" ]; then
            files=( ${files[@]/$file} )
        fi
done

# apply black and stage the changes that black made
if [[ $files != "" ]]
then
        black $files
        git add $files
fi

You may need to run chmod +x on the file to allow it to be executed. Be aware that the hook will alter files you may have opened in an editor whenever you make a commit, which may confuse you(r editor).

To automate ruff, we want to write a hook that alters the commit message in case any errors are detected. This gives us the choice of aborting the commit and fixing the issues, or we can go ahead and commit them and worry about ruff only when the time comes to do a pull request. To obtain this functionality, add the following to <pySDC-root-directory>/.git/hooks/prepare-commit-msg:

#!/bin/sh

COMMIT_MSG_FILE=$1

export files=$(git diff --staged --name-only HEAD | grep .py | sed -e "s,^,$(git rev-parse --show-toplevel)/,")

if [[ $files != "" ]]
then
        export ruff_output=$(ruff check --quiet $files)
        if [[ "$ruff_output" != "" ]]
        then
                git interpret-trailers --in-place --trailer "$(echo "$ruff_output" | sed -e 's/^/#/')" "$COMMIT_MSG_FILE"
                git interpret-trailers --in-place --trailer "#!!!!!!!!!! WARNING: RUFF FAILED !!!!!!!!!!" "$COMMIT_MSG_FILE"
        fi
fi

Don’t forget to assign execution rights.

Code testing#

This is done using pytest, and runs all the tests written in the pySDC/tests folder. You can run those locally in the root folder of pySDC using :

# Install required packages (works also with conda/mamba)
pip install pytest pytest-benchmark pytest-timeout coverage[toml]
# Run tests
pytest -v pySDC/tests

🔔 Many components are tested (core, implementations, projects, tutorials, etc …) which make the testing quite long. When working on a single part of the code, you can run only the corresponding part of the test by specifying the test path, for instance :

pytest -v pySDC/tests/test_nodes.py  # only test nodes generation

You can also run one specific test only like this:

pytest -v pySDC/tests/test_nodes.py::test_nodesGeneration             # only test_nodesGeneration function
pytest -v pySDC/tests/test_nodes.py::test_nodesGeneration[LEGENDRE]   # only test_nodesGeneration with LEGENDRE nodes

⚠️ CI runs pySDC/tests once per environment with -m <marker>, so every test there needs one of the markers base, fenics, mpi4py, petsc, pytorch, firedrake or cupy (or benchmark, which CI deliberately does not run). A test without one would never run anywhere, so collection stops with an error that names it; see pySDC/tests/conftest.py.

Running CI on HPC from pull requests#

⚠️ Note: The GitLab mirror integration is currently disabled due to technical issues. This section describes functionality that is temporarily unavailable.

By syncing the GitHub repository to a certain Gitlab instance, CI-Jobs can be run on HPC machines. This can be helpful for benchmarks or when running on accelerators that are not available as GitHub runners.

For security and accounting reasons, a few extra steps are needed in order to run the contents of a pull request on HPC:

  • The pull request needs to have the tag “gitlab-mirror” assigned to it.

  • A person with write-permission for the Parallel-in-Time pySDC repository needs to trigger the workflow. Ask for someone with the required permissions to rerun the workflow if needed.

  • The workflow checks if the code can be merged. If this is not the case, the code is not mirrored and the workflow fails. In this case, please merge upstream changes, fix all conflicts, and rerun the workflow.

đź”” Note that direct pushes to Parallel-in-Time/pySDC will always trigger the HPC pipeline on Gitlab

Regardless of why the Gitlab pipeline was triggered, the following holds true:

  • The return-state from Gitlab is transmitted to GitHub (Success/Failure) leading to the same result in GitHub

  • Logs from Gitlab are also transferred. The full logs of all jobs can be read from within GitHub. For better overview, these are folded, so unfolding is needed before reading.

  • Artifacts from Gitlab jobs are also transferred back to GitHub

  • Information, such as coverage is transferred to GitHub, but not yet merged across multiple GitHub workflows. Therefore, there is no complete summary of e.g. coverage-reports across all jobs in all workflows.

⚠️ The coverage report from the HPC tests is not yet merged with other reports. The test coverage will not show up on the respective website or in the badge. We are working on this.

HPC test environments#

In order to run tests on GPUs, please use the pytest marker cupy.

Running the GPU tests#

The gpu_hardware_tests job rents two NVIDIA T4s per run from Modal and runs the same cupy-marked selection on them, through etc/run_mpi_tests.sh as every other leg does. Unlike the stub, this one counts towards the coverage report, because it really executes the lines it reports.

Four GPUs, because the space-time tests ask for four ranks and NCCL wants a GPU per rank. The serial pass – most of the job – is spread over the same four with pytest-xdist rather than leaving three idle, which brings it from 101 seconds to 39 and costs about what two devices did.

Nothing in pySDC selects a device: on a batch system the scheduler gives each task its own, so every process taking device 0 is correct. Inside one container it is not, so etc/gpu_bind.py assigns one, reading whichever of OMPI_COMM_WORLD_LOCAL_RANK or PYTEST_XDIST_WORKER the launcher set.

coverage does not follow xdist workers, which are execnet subprocesses rather than the multiprocessing children pyproject.toml declares. Left alone the split reported 3836 covered lines against 5332 unsplit, with every test passing either way, so the image installs coverage’s .pth hook and the job sets COVERAGE_PROCESS_START.

You can run it yourself against your own Modal account, which is the fastest way to iterate:

modal token new                      # once
modal run etc/modal_gpu_tests.py     # from the repository root, the whole selection

# narrow it while developing -- the full run is around three minutes
modal run etc/modal_gpu_tests.py --tests pySDC/tests/test_sweepers/test_MPI_sweeper.py
modal run etc/modal_gpu_tests.py --k NCCL

In CI it runs on every push, like the rest of the pipeline, with one exception: a pull request from a fork gets no secrets, so the job is skipped there.

It used to be opt-in through a gpu label. That was a mistake worth recording, because a label is a gate and not a trigger: adding one to an open pull request starts nothing, so the job skipped and the pull request went green having never touched a GPU. A skip that reads as a pass is the worst thing a check can do, and it happened four times before the label went away.

Coverage is the other reason to run it unconditionally. NCCL_communicator.py, cupy_mesh.py and log_GPU_timings.py are executed nowhere else, so master’s coverage includes them and a pull request that skipped the job reports a drop that is not real. Fork pull requests still show that drop, which cannot be helped without pull_request_target.

It needs MODAL_TOKEN_ID and MODAL_TOKEN_SECRET as repository secrets. A run costs a few cents and takes about three minutes; the free tier covers several hundred a month.

⚠️ The GPU spectral code needs an mpi4py-fft that no release provides – the cupy and cupyx-scipy FFT backends, DistArrayCuPy and the NCCL comm_backend only exist on the cupy_implementation branch of brownbaerchen/mpi4py-fft, which etc/modal_gpu_tests.py installs at a pinned commit. etc/environment-cupy.yml on its own is not enough to run that code.

If you want to create a new HPC test environment, the following steps need to be completed:

  • Create a new slurm job-script in etc/juwels_*.sh. The name and location of the file is important.

  • Adapt .gitlab-ci.yml to include the new job-script. For this, add a name in the job “test_JUWELS” in the section parallel: matrix: SHELL_SCRIPT. The name there must match the name of the newly created file. As a starting point it is recommended to copy and adapt an existing file (e.g. etc/juwels_cupy.sh).

Code coverage#

This stage allows to checks how much of the pySDC code is tested by the previous stage. It is based on the coverage library and currently applied to the following directories :

  • pySDC/core

  • pySDC/projects

  • pySDC/tutorial

This analysis is done in parallel to the test each time a pull is done on any branch (main repository or fork).

Viewing Coverage Reports#

There are multiple ways to view test coverage for pySDC:

  1. HTML Coverage Report (recommended for detailed analysis)

  2. Codecov Dashboard (for trends)

  3. Coverage Badge (quick status check)

    • Displayed in the README

    • Shows current coverage percentage for master branch

  4. PR Coverage Comments (for contributors)

During developments, you can also run the coverage tests locally, using :

echo "print('Loading sitecustomize.py...');import coverage;coverage.process_startup()" > sitecustomize.py
coverage run -m pytest --continue-on-collection-errors -v --durations=0 pySDC/tests

đź”” Note that this will run all pySDC tests while analyzing coverage, hence requires all packages installed for the code testing stage.

Once the test are finished, you can collect and post-process coverage result :

coverage combine
python -m coverage html

This will generate the coverage report in a htmlcov folder, and you can open the index.html file within using your favorite browser.

⚠️ Coverage can be lower if some tests fails (for instance, if you did not install all required python package to run all the tests).

Coverage exceptions#

The coverage configuration is defined in two places:

  1. Runtime configuration (pyproject.toml) - Controls what coverage measures during test execution

  2. Codecov configuration (codecov.yml) - Controls how Codecov reports and displays coverage

Some types of code lines will be ignored by the coverage analysis (e.g lines starting with raise, …), see the [tool.coverage.report] section in pyproject.toml. Part of code (functions, conditionaly, for loops, etc …) can be ignored by coverage analysis using the # pragma: no cover, for instance

# ...
# code analyzed by coverage
# ...
if condition:  # pragma: no cover
    # code ignored by coverage
# ...
# code analyzed by coverage
# ...
def function():  # pragma: no cover
    # all function code is ignored by coverage

Accepted use of the # pragma: no cover are:

  1. Functions and code used for plotting

  2. Lines in one conditional preceding any raise statement

If you think the pragma should be used in other parts of your pull request, please indicate (and justify) this in your description.

Documentation generation#

Documentation is built using sphinx. To check its generation, you can wait for all the CI tasks to download the docs artifacts, unzip it and open the index.html file there with you favorite browser.

However, when you are working on documentation (of the project, of the code, etc …), you can already build and check the website locally :

# Run all tests, continuing even with errors
pytest --continue-on-collection-errors -v --durations=0 pySDC/tests
# Generate rst files for sphinx
./docs/update_apidocs.sh
# Generate html documentation
sphinx-build -b html docs/source docs/build/html

Then you can open docs/build/html/index.html using you favorite browser and check how your own documentation looks like on the website.

đź”” Important : running all the tests is necessary to generate graphs and images used by the website. But you can still generate the website without it: just all images for the tutorials, projects and playgrounds will be missing. This approach can be considered for local testing of your contribution when it does not concern parts containing images (i.e project or code documentation).

Tutorial pages listed in BROWSER_PAGES in docs/source/conf.py have a “Run in browser” button, which runs their code in Pyodide. CI presses it on every such page after building the site, and fails if a page does not finish or a cell raises. To do the same locally:

python -m pip install playwright
python -m playwright install chromium
python docs/check_run_in_browser.py docs/build/html

⬅️ Back to Pull Request Recommendation — ⬆️ Contributing Summary — ➡️ Next to Naming Conventions