PiT Flow

This work is funded under the embedded CSE programme of the ARCHER UK National Supercomputing Service.

HemeLB [1] is code designed to simulate blood flow in arteries using Lattice-Boltzmann methods (LBM). For complex geometries like the Circle of Willis, it scales up to 25k cores on the Cray XC30 supercomputer ARCHER but simulations still take several days to complete. Building on the work by Randles and Kaxiras [2], this project will integrate parallel-in-time integration capacities into HemeLB to extend scaling and reduce wall-clock times.

The aims of the project are:

1. Implement the Parareal [3] algorithm into HemeLB to provide parallel-in-time integration capacities.
2. Optimise communication and scheduling in time to minimise overheads and improve efficiency and resource utilisation of Parareal.
3. Demonstrate speedup of five or better using the Circle of Willis simulation model, thus bringing simulations times down to a day or less.
4. Simplify and automate the Parallel-In-Time implementation, allowing others in the community to perform these runs with a single-line command using FabSim.
5. Continuously provide information about the employed parallel-in-time approach to the ARCHER community to facilitate adoption for other codes.

1. GroenEtAl2013

   D. Groen, J. Hetherington, H. B. Carver, R. W. Nash, M. O. Bernabeu, and P. V. Coveney, “Analysing and modelling the performance of the HemeLB lattice-Boltzmann simulation environment,” Journal of Computational Science, vol. 4, no. 5, pp. 412–422, 2013 [Online]. Available at: https://dx.doi.org/10.1016/j.jocs.2013.03.002

   BibTeX

   Abstract

   BibTeX entry GroenEtAl2013

   ×

   @article{GroenEtAl2013,
     author = {Groen, Derek and Hetherington, James and Carver, Hywel B and Nash, Rupert W and Bernabeu, Miguel O and Coveney, Peter V},
     url = {https://dx.doi.org/10.1016/j.jocs.2013.03.002},
     journal = {Journal of Computational Science},
     number = {5},
     pages = {412--422},
     title = {Analysing and modelling the performance of the HemeLB lattice-Boltzmann simulation environment},
     volume = {4},
     year = {2013}
   }
   

   Abstract for BibTeX entry GroenEtAl2013

   ×

   We investigate the performance of the HemeLB lattice-Boltzmann simulator for cerebrovascular blood flow, aimed at providing timely and clinically relevant assistance to neurosurgeons. HemeLB is optimised for sparse geometries, supports interactive use, and scales well to 32,768 cores for problems with  81 million lattice sites. We obtain a maximum performance of 29.5 billion site updates per second, with only an 11% slowdown for highly sparse problems (5% fluid fraction). We present steering and visualisation performance measurements and provide a model which allows users to predict the performance, thereby determining how to run simulations with maximum accuracy within time constraints.

1. LionsEtAl2001

   J.-L. Lions, Y. Maday, and G. Turinici, “A ‘parareal’ in time discretization of PDE’s,” Comptes Rendus de l’Académie des Sciences - Series I - Mathematics, vol. 332, pp. 661–668, 2001, doi: 10.1016/S0764-4442(00)01793-6. [Online]. Available at: http://dx.doi.org/10.1016/S0764-4442(00)01793-6

   BibTeX

   Abstract

   BibTeX entry LionsEtAl2001

   ×

   @article{LionsEtAl2001,
     author = {Lions, J.-L. and Maday, Yvon and Turinici, Gabriel},
     doi = {10.1016/S0764-4442(00)01793-6},
     journal = {Comptes Rendus de l'Académie des Sciences - Series I - Mathematics},
     pages = {661--668},
     title = {{A "parareal" in time discretization of {PDE}'s}},
     url = {http://dx.doi.org/10.1016/S0764-4442(00)01793-6},
     volume = {332},
     year = {2001}
   }
   

   Abstract for BibTeX entry LionsEtAl2001

   ×

   The purpose of this Note is to propose a time discretization of a partial differential evolution equation that allows for parallel implementations. The method, based on an Euler scheme, combines coarse resolutions and independent fine resolutions in time in the same spirit as standard spacial approximations. The resulting parallel implementation is done in the non standard time direction. Its main goal concerns real time problems, hence the proposed terminology of “parareal” algorithm.
2. Randles2014

   A. Randles and E. Kaxiras, “Parallel in time approximation of the lattice Boltzmann method for laminar flows,” Journal of Computational Physics, vol. 270, pp. 577–586, 2014, doi: 10.1016/j.jcp.2014.04.006. [Online]. Available at: http://dx.doi.org/10.1016/j.jcp.2014.04.006

   BibTeX

   Abstract

   BibTeX entry Randles2014

   ×

   @article{Randles2014,
     author = {Randles, Amanda and Kaxiras, Efthimios},
     doi = {10.1016/j.jcp.2014.04.006},
     journal = {Journal of Computational Physics},
     pages = {577--586},
     title = {{Parallel in time approximation of the lattice {B}oltzmann method for laminar flows}},
     url = {http://dx.doi.org/10.1016/j.jcp.2014.04.006},
     volume = {270},
     year = {2014}
   }
   

   Abstract for BibTeX entry Randles2014

   ×

   Fluid dynamics simulations using grid-based methods, such as the lattice Boltzmann equation, can benefit from parallel-in-space computation. However, for a fixed-size simulation of this type, the efficiency of larger processor counts will saturate when the number of grid points per core becomes too small. To overcome this fundamental strong scaling limit in space-parallel approaches, we present a novel time-parallel version of the lattice Boltzmann method using the parareal algorithm. This method is based on a predictor–corrector scheme combined with mesh refinement to enable the simulation of larger number of time steps. We present results of up to a 32× increase in speed for a model system consisting of a cylinder with conditions for laminar flow. The parallel gain obtainable is predicted with strong accuracy, providing a quantitative understanding of the potential impact of this method.