1.
WORKING TOGETHER
WITH SURF FOR EARTH
OBSERVATION
Seminar Enabling Copernicus Big Data
Analytics through EOSC (27 October 2021)

2.
THIS IS
SURF

3.
WHAT WE
CAN DO FOR
RESEARCH

4.
High-end
Compute services
Data processing
& analysis
Data storage &
management
Network &
collaboration services

5.
WHAT SOLUTION IS
BEST SUITED FOR MY
PROJECT?

6.
Customised
services and
co-creation
Artificial intelligence
and (high performance)
machine learning
expertise

7.
Customised
services and
co-creation
Scalable data
processing and
analytics

8.
Customised
services and
co-creation
Custom Cloud
Solutions: In-house &
commercial

9.
Access to
commercial
clouds
SURFcumulus
(reduced prices &
in-/egress costs)

10.
ROUTES
TO SURF

11.
1. NWO
Free at the point of use,
including dedicated
consultancy and storage
2. INNOVATION
SURF Labs – collaborations
for the future
3. NATIONAL / EU / GLOBAL PROJECTS
SURF supports national (DCC, Roadmap), EU
(C-SCALE) & global collaborations
4. CONTRACTS
There are (limited) possibilities
to purchase (pre-)paid
resources from SURF

12.
Example:
SRON-collaboration
project via NWO
“Characterization of
aerosols in the
atmosphere (TROPOMI)”

13.
SURF
Infrastructure:
Processing daily
incoming Sentinel-5P
data and additional
reprocessing
Extraction of as
much scientific
information as
possible
Feedback to ESA
proposing changes to
the standardised data-
processing algorithms
Calibration and
further refinement
numerical models
and algorithms
Making high level
data products
available

14.
15
SLIDE BY NATALIE DANEZI (ESA PHI-WEEK 2020)

15.
16
SLIDE BY NATALIE DANEZI (ESA PHI-WEEK 2020)
Public data repository
( https )

16.
Benefits of a dynamic
data-processing
platform
Processing huge
volumes of data

17.
Benefits of a dynamic
data-processing
platform
Analysis-ready data

18.
Benefits of a dynamic
data-processing
platform
Easy scale-up, reliable
infrastructure

19.
Benefits of a dynamic
data-processing
platform
Standardise solutions
with APIs and standards

20.
Benefits of a dynamic
data-processing
platform
Sustainable research
with lower maintance
costs

21.
C-SCALE
AND SURF

22.
Larger areas need
to be analysed in
more detail in less
time
VM / laptop
workflows do not
easily scale and
require interaction
Data access is
fragmented and
difficult
Need to scale up
(HPC) and out
(HTC)
Society requires action
on environmental
topics (e.g., Green Deal)
WHY?

23.
Earth Observation
is truly a global
endeavour C-SCALE brings together at an EU
level and by leveraging the European
Open Science Cloud (EOSC) ,
Data providers
Compute & Network providers
Researchers
at a common table

24.
29
C-SCALE & SURF: A new HPC & HTC federation
C
o
m
p
u
teF
ed
eratio
nH
P
C
/H
TC
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC
Center
1
HPC
Center
2
C
-S
C
A
L
ED
ataF
ed
eratio
nIn
terface
Cop. Data
Archive
(Provider 1)
Cop. Data
Archive
(Provider 2)
EOData
Archive
(Provider 1)
EOData
Archive
(Provider 2)
Remote
Storage
(Provider 3)
C
-S
C
A
L
EH
PC&H
TCA
ccessIn
tegratio
nIn
terface
EOSC Portal
C
-S
C
A
L
E
A
A
I
C
-S
C
A
L
E
E
OA
pp
C
atalo
g
EOSC AAI
Cop. Data
Archive
(on-site)
EOSC Core
Services
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC
Center
3

25.
30
C-SCALE & SURF: A new HPC & HTC federation
C
o
m
p
u
teF
ed
eratio
nH
P
C
/H
TC
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC
Center
1
HPC
Center
2
C
-S
C
A
L
ED
ataF
ed
eratio
nIn
terface
Cop. Data
Archive
(Provider 1)
Cop. Data
Archive
(Provider 2)
EOData
Archive
(Provider 1)
EOData
Archive
(Provider 2)
Remote
Storage
(Provider 3)
C
-S
C
A
L
EH
PC&H
TCA
ccessIn
tegratio
nIn
terface
EOSC Portal
C
-S
C
A
L
E
A
A
I
C
-S
C
A
L
E
E
OA
pp
C
atalo
g
EOSC AAI
Cop. Data
Archive
(on-site)
EOSC Core
Services
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC
Center
3
Goal:
Access Integration

26.
31
C-SCALE & SURF: A new HPC & HTC federation
C
o
m
p
u
teF
ed
eratio
nH
P
C
/H
TC
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC
Center
1
HPC
Center
2
C
-S
C
A
L
ED
ataF
ed
eratio
nIn
terface
Cop. Data
Archive
(Provider 1)
Cop. Data
Archive
(Provider 2)
EOData
Archive
(Provider 1)
EOData
Archive
(Provider 2)
Remote
Storage
(Provider 3)
C
-S
C
A
L
EH
PC&H
TCA
ccessIn
tegratio
nIn
terface
EOSC Portal
C
-S
C
A
L
E
A
A
I
C
-S
C
A
L
E
E
OA
pp
C
atalo
g
EOSC AAI
Cop. Data
Archive
(on-site)
EOSC Core
Services
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC
Center
3
SRAM (ssh key access via LDAP synchronization)

27.
32
C-SCALE & SURF: A new HPC & HTC federation
C
o
m
p
u
teF
ed
eratio
nH
P
C
/H
TC
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC
Center
1
HPC
Center
2
C
-S
C
A
L
ED
ataF
ed
eratio
nIn
terface
Cop. Data
Archive
(Provider 1)
Cop. Data
Archive
(Provider 2)
EOData
Archive
(Provider 1)
EOData
Archive
(Provider 2)
Remote
Storage
(Provider 3)
C
-S
C
A
L
EH
PC&H
TCA
ccessIn
tegratio
nIn
terface
EOSC Portal
C
-S
C
A
L
E
A
A
I
C
-S
C
A
L
E
E
OA
pp
C
atalo
g
EOSC AAI
Cop. Data
Archive
(on-site)
EOSC Core
Services
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC
Center
3
Software Access
- EGI AppDB
- Containers
- Virtual Env.

28.
33
C-SCALE & SURF: A new HPC & HTC federation
C
o
m
p
u
teF
ed
eratio
nH
P
C
/H
TC
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC
Center
1
HPC
Center
2
C
-S
C
A
L
ED
ataF
ed
eratio
nIn
terface
Cop. Data
Archive
(Provider 1)
Cop. Data
Archive
(Provider 2)
EOData
Archive
(Provider 1)
EOData
Archive
(Provider 2)
Remote
Storage
(Provider 3)
C
-S
C
A
L
EH
PC&H
TCA
ccessIn
tegratio
nIn
terface
EOSC Portal
C
-S
C
A
L
E
A
A
I
C
-S
C
A
L
E
E
OA
pp
C
atalo
g
EOSC AAI
Cop. Data
Archive
(on-site)
EOSC Core
Services
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC /
HTC
Scheduler
Account
Manager
Storage
HPC
Center
3
Data access
- Uniform query tool
- STAC compliant
- HTTPs transfer
- Token authentication

29.
Status
HPC & HTC + Cloud
access integration
will be investigated
HPC & HTC access
integration prototype
for EO is available
First use cases onboarded
(e.g., HiSea, RETURN,
LSDA)
EO Software
deployment considered
(e.g., Containers,
Virtual Env.)
OpenEO/Jupyter
implementation considered
(e.g., Slurm+Dask)

30.
Common challenges in
Earth Observation research
database-share.svg
Data management Data processing Collaboration

31.
IMPROVE NATIONAL
SUPPORT FOR EO
RESEARCH

32.
EO is increasingly
more global and reliant on
state-of-the-art IT and
expertise

33.
SURF EO
Focus group
goals Contribute to EO
cloud platforms
Better align national IT
investments with the
needs of EO researchers
Create, together with
researchers, a multi-year IT
roadmap for supporting EO
Share knowledge
(data processing,
HPC, AI/ML )
Build a scientific
EO IT-user
community

34.
We need your
input: Help us
to help you!
- What IT challenges do you
observe for NL EO in the coming
years ?
- What type of IT solutions do
you need in the next 2, 5, 10+
years ?

35.
The SURF
EO Focus groep:
Raymond
Oonk
Thomas
van Himbergen
Annette
Langedijk
Ander
Astudillo
Natalie
Danezi