Lutz Gross of the University of Queensland describes running geophysical inversion using e-script, an open source package based on PDEs and python. Other examples of what e-script can do are also shown, such as diffusion calculations, mantle convection, flow in porous media, seismo-electrics and much more!

1. 1 March’18
3D modelling and inversion in escript
Dr. Lutz Gross
School of Earth & Environmental Sciences
The University of Queensland
St Lucia, Australia

2. 2 March’18
esys-escript
from esys.escript import *
import esys.escript.unitsSI as U
Q=10*U.W/U.m**3; K=1.7*U.W/(U.m*U.K); rhocp=1.5*U.Mega*U.J/(U.m**3*U.K)
# … time step size:
dt=1.*U.year; n_end=100
# … 40 x 20 grid on 10km x 5km
mydomain=Rectangle(40, 20,l0=10*U.km, l1=5*U.km)
x=mydomain.getX()
# … create PDE and set coefficients:
mypde=LinearPDE(domain)
mypde.setValue(A=K*kronecker(mydomain), D=rhocp/dt, q=whereZero(x[1]-
5*U.km))
# … initial temperature is a vertical, linear profile:
T=0*U.Celsius+30*U.K/U.km*(5*U.km-x[1])
n=0
while n<n_end :
mypde.setValue(Y=Q+rhocp/dt*T, r=T)
T=myPDE.getSolution()
n+=1
saveVTK(“u%s”%n, temperature=T, flux=-K*grad(T) )
print(“Time “,n*dt,”: min/max temperature =”,inf(T), sup(T))
−∇t
( A ∇ u+Bu)+C ∇ u+Du=−∇t
X+Y
Mathematical Model
esys-escript: scripted Model Implementation
Geometry
Desktop
Parallel Supercomputers

3. 3 March’18
Esys-Escript Software
●
Direct funding for 13+ years: ~$3M
●
Modeling with PDEs
●
Rapid prototyping of new models
●
Ease of Use
●
“Supercomputing for the masses”
●
Programming in python
ρcp
∂T
∂t
−∇
t
K ∇ T=Q

4. 4 March’18
Finite Element Method (FEM)
●
approximative solution PDEs in variational form
●
by continuous, piecewise linear approximation
●
Solve discrete problem with sparse matrix
– With algebraic multi-grid (AMG) (see later)
●
Domain decomposition of mesh on parallel
computers
– Hidden from then user

5. 5 March’18
PDEs in esys-escript
general linear PDE for solution u :
−∇
t
( A ∇ u+Bu)+C ∇ u+D u=−∇
t
X+Y
Identify PDE coefficients!?
+ boundary conditions

6. 6 March’18
How to use esys-escript
Temperature Diffusion:
ρcp
∂T
∂t
−∇
t
K ∇ T=Q
at topT=0
o
C
Q
Domain
10km
5km
Assume homogeneous rock
Initial temperature
T(t=0) = vertical linear profile

7. 7 March’18
∂T
∂t
≈
T
(n)
−T
(n−1)
dt
−∇t
K⏟
= A
∇ T(n)
+
ρcp
dt⏟
=D
T(n)
=Q+
ρcp
dt
T(n−1)
⏟
=Y
Apply Backward Euler Method
How to deal with the time derivative?

8. 8 March’18
esys-escript Implementation
from esys.escript import *
import esys.escript.unitsSI as U
Q=10*U.W/U.m**3; K=1.7*U.W/(U.m*U.K); rhocp=1.5*U.Mega*U.J/(U.m**3*U.K)
# … time step size:
dt=1.*U.year; n_end=100
# … 40 x 20 grid on 10km x 5km
mydomain=Rectangle(40, 20,l0=10*U.km, l1=5*U.km)
x=mydomain.getX()
# … create PDE and set coefficients:
mypde=LinearPDE(domain)
mypde.setValue(A=K*kronecker(mydomain), D=rhocp/dt, q=whereZero(x[1]-
5*U.km))
# … initial temperature is a vertical, linear profile:
T=0*U.Celsius+30*U.K/U.km*(5*U.km-x[1])
n=0
while n<n_end :
mypde.setValue(Y=Q+rhocp/dt*T, r=T)
T=myPDE.getSolution()
n+=1
saveVTK(“u%s”%n, temperature=T, flux=-K*grad(T) )
print(“Time “,n*dt,”: min/max temperature =”,inf(T), sup(T))

9. 9 March’18
Visualization by visit

10. 10 March’18
Some applications
●
Mantel Convection
●
Pores Media Flow
●
Geomechanics
●
Seismo-Electric
●
Geophysical inversion
●
Earthquakes
●
Volcanoes
●
Tsunamis
●
...

11. 11 March’18
Example: Seismo-Electric
●
With Simon Shaw (Msc)

12. 12 March’18
Experiment Set-up
Wet
V

13. 13 March’18
Electro-kinetic Coupling
➢
Ions in pores fluid are
separated by surface
charge on solid
➢
Movement of fluid
creates a current
➢
Induces an electrical field

14. 14 March’18
Mathematical Model
∂2
p
∂t
2
=∇
t
vp
2
∇ p+s(t)
Propagation of pressure wave p in solid
ppore=B⋅p with Skempton coefficient 0≤B<1
Assume saturated media under undrained conditions (Revil et al 2013):
Darcy's law : flux q=−
K
ηf
∇ ppore

15. 15 March’18
Mathematical Model (cont.)
Electro-kinetic Coupling: flux induces current : j=QV⋅q
−∇t
σ ∇ u=∇t
j
Electric potential u from electric current:
electrical field E=−∇ u
Horizontal component of E is measured on surface
electric conductivity σ

16. 16 March’18
Qv vs permeability

17. 17 March’18
Single Layer: Horizontal E
Coupled Spectral Element (SEM) – Finite Element (FEM)

18. 18 March’18
Trace Records: Horizontal E
Dipole offset from source
time
Maximum signal offset
= half of interface depth
(for 2D model)
(~10m spacing)
Travel time in layer → vp

19. 19 March’18
Inversion

20. 20 March’18
Geophysical Inversion
Construct 3D structure of the Earth's subsurface
from 2D data collected above/near the surface.
Measurements on/near surface
physical property in the subsurface
Physical model
Vertical gravity → density

21. 21 March’18
Electric Resistivity Tomography (ERT)
Apply electrical
charge
Measure response
in voltage
Region of increased
electrical conductivity σ
by a pollutant
Initial guess σInitial guess σ
update σupdate σ
Defect
small ?
Defect
small ?
Numerical prediction of
responses from σ
Numerical prediction of
responses from σ
Defect = difference of
measurement and prediction
Defect = difference of
measurement and prediction
Done!Done!

22. 22 March’18
Inversion as Optimization Problem
Find argmin
ρ∈V
J (ρ)
data misfit : D(ϕ)=
1
2
∫Ω
(w⋅∇ ϕ−g)
2
dx
regularization: R(m)=
1
2
∫Ω
‖∇ ρ‖
2
dx
PDE constraint : −Δ ϕ=4 πG⋅(ρref +ρ')
J(ρ)⏟
costfunction
=D(ϕ)⏟
misfit
+μ⋅ R(ρ)⏟
regularization

23. 23 March’18
Regularization
Western Queensland
~ 64 Million Cells
More regularization, larger μ

24. 24 March’18
Quasi-Newton Method
get gradient g
(ν)
=∇ J (m
(ν)
)
find search direction s
(ν)
by solving
~
Ηs
(ν)
=g
(ν)
with
~
Η≈ Hessian ∇ ∇ J
m(ν+1)
=m(ν)
+α⋅s(ν)
with
min
α
J (m(ν)
+α⋅s(ν)
) via line search
ν←ν+1
initial guess m(0)
iteration step ν:

25. 25 March’18
Inversion in escript
●
PDEs: Forward problems → cost function J
●
One for each experiment, frequency
●
PDEs: Adjoint problems → gradient of J
●
One for each experiment, frequency
●
PDE: approx. inverse Hessian → preconditioner
●
Use Hessian of regularization term
●
Joint Inversion: System of coupled PDEs

26. 26 March’18
Remarks
●
Actually used in escript:
●
Broyden–Fletcher–Goldfarb–Shanno (BFGS)
– Iterative Quasi-Newton method
– Self-preconditioning NLCG
●
PDEs solved by FEM
●
Same mesh for forward and adjoint → uniqueness
●
see Lamichhane & Gross 2017

27. 27 March’18
Australia gravity inversion (FEILDS)
●
by A. Aitken (UWA), C. Altinay (now HPE)
● density corrections to the AusREM ρref
●
Implemented in esys-escript
●
1/8o
resolution @ 200Million unknowns
●
Inversion run on >22000 cores on Magnus
●
Hybrid mode
– Shared memory: OpenMP on nodes
– Distributed memory: MPI across nodes

28. 28 March’18
Architecture (conceptional)
Computational nodes with CPU-
cores sharing memory on node
(OpenMP, pthreads).
Message passing between computational
nodes via network (MPI)

29. 29 March’18
Weak Scalability
Constant execution time for doubled problem
size on the double number of compute cores
3D: For ½ resolution and 8x # cores the
compute time remains constant.

30. 30 March’18
Weak Scalability: what to expect?
Computational load ~
#cells/p=const.
log(p)
Execution
Time
Extra communication
costs = const.
Synchronization & global
communications ~ log(p)
# cores =p
#cells per core =const.

31. 31 March’18
Aspects of Scalability
●
Memory Requirements
●
Keep constant per core!
●
Use domain decomposition
●
Communication Costs
●
Sparsity of connectivity in grid/mesh
●
Algorithm
●
Number of iteration steps independent from number
of unknowns → difficult to achieve

32. 32 March’18
Issues for Large Scale Inversion
●
Iteration count matters now; expect
●
Higher costs for more observation stations
●
in the gradient of the cost function
●
Use: adjoint states
●
Dense matrices
●
forward problem for Greens functions → FEM
●
density updates → low rank approximations (BFGS)
count =O(
3
√#ncells)

33. 33 March’18
Inversion Benchmark
●
Joint gravity-magnetic data
●
3D Rectangular Grid
●
Same mesh for all PDEs
●
Synthetic Data with noise
●
Platform: NCI Canberra
●
Intel Xeon Sandy Bridge, 8-core, dual socket, 2.6 GHz
●
Infiniband FDR interconnect
●
MPI + OpenMP
ρ=f (k)

34. 34 March’18
PDE calls vs. Regularization
μ

35. 35 March’18
Weak Scalability
30000 grid cells per core

36. 36
3D ERT
64 electrodes per line

37. 37 June 2016
ERT with Multi-Grid
grid #nodes rel. time rel. size AMG speedup vs
PCG
51 140608 1.00 1.00 5.79
99 1000000 6.57 7.11 13.39
151 3511808 23.57 24.98 20.44
199 8000000 52.29 56.90
219 10648000 71.43 75.73
3 level AMG on four core
Time/size =const!
Codd & Gross 2018

38. 38 June 2016
Structural Joint Inversion
●
Invert for density and
susceptibility
●
spatial change should go
together somehow
Gravity Anomalies QLD:
Magnetic field intensity QLD

39. 39 March’18
Cost Funtion
J (m)=D1(ϕ1)+μ1 R(m1)⏟
grav
+D2(ϕ2)+μ2 R(m2)⏟
mag
+νc⋅∫C(m1 ,m2)⏟
cross gradient
gravity −Δ ϕ1=4 πG⋅ρref m1
magnetic −Δϕ2=∇(kref e
m2
B)

40. 40 March’18
Cross Gradient function
“Where density and susceptibility change they
change in the same direction.”
∇ m1
density contour
normal
susceptibility contour
normal
∇ m2
C(m1, m2)=∣∇ m1×∇ m2∣
2
minimize angle between
contour normals

41. 41 March’18
Example
Measure of correlation: M=
∇ m1⋅∇ m1
|∇ m1||∇ m2|
∈[−1,1]

42. 42 March’18
Example Results (vertical)
m1 , νc=0 m2 , νc=0
m1 , νc=10
8
μ m2 , νc=108
μ
M
2
, νc=0
M2
, νc=108
μ

43. 43 March’18
Can we do better?
∫C(m1 ,m2)∝R(m1)⋅R(m2) small vs. regularization
C(2)
(m1 ,m2)=
( 1
|∇ m1|
2
+
1
|∇ m2|
2
)|∇ m1×∇ m2|2

44. 44 March’18
Example again
m1 , νc=100μ m2 , νc=100μ
M2
, νc=100μ
allways works: νc=100μ

45. 45 March’18
QLD Data (vertical)
m1 , νc=100μ
m1 , νc=100μ
M
2
, νc=100μ
30 Million cells
on ~200 cores

46. 46 March’18
What is next
●
Current focus on ERT-type methods
●
Simulation-Geophysical Signal coupling
●
CCSRDF Sequestration projects
with Uni Melbourne and Uni Savoy Monte Blanc
●
EM/MT problems
●
More on seismic & FWI