This overview poster summarizes the major advances made on the HemeLB code. HemeLB is a parallelized lattice-Boltzmann simulation code, which has been shown to scale near-linearly to up to 32768 cores. Recently, we managed to reach 153 million site updates per seconds using 49152 cores on the ARCHER supercomputer (see http://www.cresta-project.eu). In addition, it highlights two recent project, one about domain decomposition (which has been published in much greater detail in the proceedings of the EASC 2014 conference) and one ongoing project about validating HemeLB against clinical velocity measurements.

1. Derek Groen, Miguel Bernabeu, Rupert Nash, Sebastian Schmieschek and Peter Coveney
Cerebrovascular bloodflow Introducing HemeLB
2008
Evolution of the HemeLB bloodflow simulation environment
First journal publication
on HemeLB.
2010
Development of steering
and visualization client.
2009
HemeLB run across sites
Using MPI-g.
2011
Improved domain
decomposition
with ParMETIS.
2013
Multiscale 3D-1D coupling
with Python Navier Stokes.
2013
Up to 50% faster
Calculations with SSE.
2014
Better load balance using
weighted decomposition.
2012
Support for 2nd order
accurate wall conditions.
2014
First comparison tests with
clinical data.
2014
HemeLB-Chaste coupling
to simulate vascular
remodelling processes.
2013
More stable model
constructionwith CGAL.
2014
Gained support for
implementing immersed
boundary conditions.
2012
Performance prediction
Model.
2012
Improved scalability
With coalesced
comms design pattern.
2011
Code reengineered for
improved accuracy and
stability.
2014
Support for PT-Scotch,
Zoltan and ParMETIS in
domain decomposition.
Performance improvements
Accuracy improvements
Scientific advances
New functionalities
2013
Framework for
convenient property
extraction
Highlight: Comparison against clinical data
We have compared flow predictions from HemeLB with clinical
measurements. We have done this work in collaboration with Fergus
Robertson and Hoskote Chandrashekar from UCL Hospital.
● We obtained rotational angiography images of a middle cerebral
artery, as well as velocity measurements (TCD) in 5 planes within
this artery.
● We imposed one plane as a velocity-based inlet in HemeLB, and
ran the code to predict the velocities in the four other planes (we
used pressure outlets).
● See below for an overview of the geometry, and a comparison
between the HemeLB flow predictions and the TCD measurements
on the plane furthest away from the inlet.
● We are currently working to repeat this exercise with a second
patient, and to improve our comparison techniques.
Highlight: Weighted Decomposition
HemeLB models sparse vasculature geometries as a lattice of fluid
sites. These geometries contain bulk sites, wall sites and in/outlet sites.
Both wall and in/outlet sites are generally more expensive to compute
than bulk sites, leading to load imbalance among different processes.
To reduce this load imbalance, we assign weights to each lattice site
(see right below) before we partition and distribute the domain among
the processes. Using this approach resulted we have managed to
reduce the calculation load imbalance by up to 85%.
The performance of HemeLB has improved considerably over the past 7 years:
Indeed, for smaller geometries we can now do production simulations (e.g., 5M time steps)
in less than an hour. However, simulating a full Circle of Willis model on a relevant time
scale (e.g., 25M time steps) will still take about a day.
Circle of
Willis
run
(first trial)
Contributors Recommended reading
Marco Mazzeo, Steven Manos, Gary Doctors – initial
developers
Rupert Nash, Hywel Carver, James Hetherington, Timm
Krueger – Main developers during 2010-2013.
Miguel Bernabeu, Derek Groen, Sebastian Schmieschek –
current HemeLB developers.
Dan Holmes – Colloids code 2012.
Jiri Jaros, David Abou Chacra – Performance optimizations
during 2013.
Jens Nielsen – CGAL setup tool optimizations 2013.
Gregor Matura, Fang Chen – pre- and post-processing 2014.
Aditya Jitta – Comparison against clinical data, 2013.
2013
Comparison of different
Rheology models.
2012
Comparison of different
wall conditions.
2013
Support for
Colloidal particles
2010
Python model construction
tool developed.
● Stroke is the main cause of about 1.1M
deaths per year in Europe.
● About 15% of these strokes are caused by
bleeding in the brain, e.g. due to the rupture
of brain aneurysms.
● These brain aneurysms frequently reside
in arteries branching from the Circle of
Willis.
● HemeLB is a feature-rich simulation environment for modelling blood flow in
sparse geometries.
● It relies on the lattice-Boltzmann method and is well suited for execution on
large supercomputers.
HemeLB consists of several key components:
● The main simulation code, which can be coupled to other codes.
● The setup tool, for constructing 3 dimensional geometries from segmented
angiography scans.
● A Python-based framework for constructing initial conditions and analyzing
output data.
● An interactive steering and visualization tool.
● A Python-based automation environment for deploying and executing the code
on remote machines.
Circle of Willis with one diminshed artery.
© Nevit Dilmen
Simulating bloodflow
Clinicians can conveniently measure blood
pressure and flow velocities on patients at rest
with limited resolution.
Simulations allow us to estimate and predict
flow properties in other regimes as well. These
include:
● Flow velocity estimates for patients during
exercise and other forms of activity.
● Wall stress estimations under all these
conditions (e.g., wall shear stress).
● Flow properties in specific locations within a
geometry, e.g. velocities and stresses in an
aneurysm sac.
Both very high and very low wall shear stress
have been associated with aneurysm
formation and rupture.
Sample visualization of a HemeLB simulation.
1. Performance: JoCS, DOI: 10.1016/j.jocs.2013.03.002
2. Weighted Decomposition: EASC 2014. Preprint available.
3. Boundary Conditions: Phys. Rev. E 89, 023303 (2014).
4. Clinical Validation: work in progress. Slides available.
5. Multiscale: Interface Focus, DOI: 10.1098/ rsfs.2012.0087
6. Retinal blood flow: Interface (submitted) arXiv:1311.1640.
Access the source code at:
http://ccs.chem.ucl.ac.uk/hemelb
For requesting preprints, please send an e-mail to Derek Groen
(djgroennl@gmail.com).
2012
Tools for automatic
compilation and execution
on remote machines.
inlet @63mm.
v plane @49mm.
2014
Prediction of vascular
development in retinas.
2020 SCIENCE
www.2020science.net
UKCOMES
UK Consortium On Mesoscale
Engineering Sciences
Legend