Presentation by Jarno Verkaik (Deltares) at the iMOD International User Day, during Delft Software Days - Edition 2017. Tuesday, 31 October 2017, Delft.

1. High Performance Parallel Computing with
iMODFLOW-MetaSWAP
Jarno Verkaik (Deltares)
Paul van Walsum (WUR/Alterra)
and
Joseph Hughes (USGS)
Edwin Sutanudjaja (UU)
Raju Ram (TUD)

2. 6 December, 2017
Contents
• Problem statement and solution
• Context
• Parallelization method
• Benchmarks
• Practical usage with iMOD
SURFSARA-Cartesius: the Dutch supercomputer

3. Problem statement and solution
Problem statement:
• In order to support decision makers in solving hydrological problems, detailed high
resolution models are often needed.
• These models typically consist of a large number of computational cells and have
large memory requirements and long run times.
Solution:
• An efficient technique for obtaining realistic run times and memory requirements is
parallel computing, where the problem is divided over multiple processor cores.
6 december 2017

4. Problem statement and solution
Problem statement:
• In order to support decision makers in solving hydrological problems, detailed high
resolution models are often needed.
• These models typically consist of a large number of computational cells and have
large memory requirements and long run times.
Solution:
• An efficient technique for obtaining realistic run times and memory requirements is
parallel computing, where the problem is divided over multiple processor cores.
 Development of the new Parallel Krylov Solver (PKS) package together with
the USGS, Utrecht University, WUR/Alterra and Delft University of Technology
6 december 2017

5. Context
• 2010: start with parallelization of MT3DMS
• 2012: first applied to the National Nutrient Model (H2O, vol18)
• 2016: start parallelization for the Netherlands Hydrological Model (NHM)
 entirely funded by Deltares and Alterra from own investments
• 2017 July: release of iMOD 4.0 with PKS as main new functionality
• 2017 other: PKS added to iMOD-SEAWAT fresh-salt
Measured speedup for Sand Engine Model: ~40
• 2017 other: start of project with USGS to incorporate PKS in core MODFLOW
6 december 2017

6. Netherlands Hydrological Model (NHM)
Components:
• MODFLOW: 3D Groundwater flow
• MetaSWAP: 1D Soil Vegetation
Atmosphere Transfer (“SVAT”)
• TRANSOL : 1 D model of salinity
• MOZART: Surface water
Coupling:
• ID-based N:1 coupling tables
• Seamless MODFLOW-MetaSWAP
coupling with groundwater level as
shared state variable
6 december 2017

7. Parallelization method (1)
6 december 2017
The Parallel Krylov Solver* couples subdomains at
a tight matrix level using domain decomposition:
• The model grid is decomposed into
overlapping subdomains
• Each subdomain is assigned to 1 processor
• The processors exchange data through
Message Passing Interface (MPI)

8. Parallelization method (2)
6 december 2017
• Solving the groundwater flow equation
requires solving the linear system for heads ℎ:
𝐴ℎ = 𝑞
e.g. for two subdomains
𝐴11 𝐴12
𝐴21 𝐴22
ℎ1
ℎ2
=
𝑞1
𝑞2
• Additive Schwarz iteration (block Jacobi):
A11h(k)
1=q1 - A12h(k-1)
2
A22h(k)
2=q2 - A21h(k-1)
1
• PKS uses the block Jacobi preconditioner M in CG
• Parallel solution = serial solution

9. PKS supports for iMODFLOW-MetaSWAP:
• Uniform blocks
• Load weighted blocks (Recursive Coordinate Bisection algorithm)
Parallelization method (3)
6 december 2017
Example “uniform”
128 subdomains
Example “RCB”
128 subdomains

10. Benchmarks: NHM (1)
6 december 2017
• iMODFLOW-MetaSWAP + surface water
• Maximum measured speedup ~5 on
NHM Deltares server (Windows)
• Maximum theoretical speedup is
limited by surface water (< 1/0.06  16.7)
• Exactly the same heads are computed
with PKS as for the serial case
Amdahl’s law

11. Benchmarks: NHM (2)
6 december 2017
• iMODFLOW-MetaSWAP only
• SURFSARA-Cartesius Dutch National supercomputer (Linux)
• Maximum measured speedup ~24.
Further reading:
Deltares R&D Highlights 2016
https://www.deltares.nl/app/uploads/2015/02/RD-Highlights-2016_lowres.pdf
Page. 64

12. Benchmarks: California model
6 december 2017
* Simulated on the SURFSARA-Cartesius Dutch National Super Computer
# On a single code estimated to consume 12 hours
MODFLOW only;
128 core (245 Gb RAM)*;
50 x 50 meter;
23,897 x 27,974 ≈ 335 million active nodes
2,6 Gb file size;
16:34 minutes#
Further reading:
Vermeulen et. al., Large scale high resolution modeling,
MODFLOW and More conference 2017, Golden, USA.

13. Practical usage with iMOD (1)
Easy-to-use in three steps:
1. Install MPI software (MPICH):
www.mpich.org/static/downloads/1.4.1p1/mpich2-1.4.1p1-win-x86-64.msi
2. Modify your run-file*, Dataset 5 (Solver Configuration)
3. Start your parallel job. E.g. starting from the DOS prompt using 4 cores:
mpiexec -localonly 4 iMODFLOW.exe imodflow.run
6 december 2017
Enable PKS with a “minus”
Same options as PCG
Partitioning method, flag for merging IDF output
* The new iMOD project file (.PRJ) is not yet supported for PKS.

14. 6 december 2017
Practical usage with iMOD (2)
Try it yourself with Tutorial 6: Model Simulation!

15. Practical usage with iMOD (3)
Future to-dos for PKS:
• Support more packages: MNW, SFR, LAK, and iPEST
• Update iMOD Project Manager
• Improve line segment input (HFB, ISG) which may slow down speed up
• Improve output:
• Update iMOD batch for PKS for e.g. IDF merging
• Merge BDA-files for MetaSWAP output
• Improve CSV-file naming
• Support MicroSoft MPI for improved scheduling
• Coarse-grid RCB support
• …
6 december 2017

16. Practical usage with iMOD (4)
Take notice:
• Overall speed up strongly depends on hardware
• To get maximum speed-up for a hi-res model tuning is necessary (e.g. load
balancing)
• When considering new iMOD functionality, always check the impact on PKS
6 december 2017

17. !!! Thank you for your attention !!!
6 december 2017