ESnet6 provides an ultra-fast and reliable network that enables new smart instruments for 21st century science. The network capacity has increased dramatically over time, with 2022 bandwidth being 500,000 times greater than 1993. This network allows rapid data transfer between facilities, such as replicating 7 petabytes of climate data between three labs. It also enables fast assembly and use of new instruments like high energy diffraction microscopy that can perform an analysis in 31 seconds. The integrated research infrastructure provided by Globus further supports use of remote resources and smart instruments that will drive scientific discovery.

1. How ESnet6 enables new
smart instruments for
21st Century science
Ian Foster
Argonne National Laboratory
The University of Chicago
foster@anl.gov

2. 2

3. wikipedia.org
Eratosthenes
measures
earth’s diameter
in 245BC

4. FitzRoy creates
weather forecasts
in 1861 …

5. FitzRoy, The Weather Book: A Manual of Practical Meteorology, 1863
… and uses data to discover extratropical frontal cyclones

6. Science cares about latency, bandwidth, and reliability
Pony express (1860) from St Joseph, Missouri, to Sacramento, California
Latency: 10 days = 864000 seconds High
Bandwidth: 50 letters x 1000 characters/letter = 50 KB / 864000 sec = 0.05 B/s Low
Reliability: Accidents, floods, hostile locals, disease, etc. Poor

7. ESnet has worked since 1986 to improve all three …
ESnet6 network capacity, as of May 2022

8. … as measured also by bisection bandwidth
1993:
0.005 Gbps
Bisection bandwidth
in Gigabit/second
2022:
500,000 Gbps

9. https://dashboard.globus.org/esgf Shown: Status as of March 10, 2022
4 to 6 GB/s
1.5 GB/s
Replicating 7.4 petabytes of climate data from
Livermore to Argonne and Oak Ridge National Labs
17,347,671 directories & 28,907,532 files
From February 12 to May 4, 2022

10. This ultra-fast, stable, and reliable network enables the
rapid assembly of new smart instruments

11. aining of Deep Neural Networks
ources
This ultra-fast, stable, and reliable network enables the
rapid assembly of new smart instruments
High energy diffraction microscopy: https://doi.org/10.48550/arXiv.2105.13967
7 seconds
7 + 19 + 5 = 31 s
5 seconds
19 seconds

12. High energy diffraction microscopy: https://doi.org/10.48550/arXiv.2105.13967
aining of Deep Neural Networks
ources
This ultra-fast, stable, and reliable network enables the
rapid assembly of new smart instruments
7 seconds
7 + 19 + 5 = 31 s
5 seconds
19 seconds
ModelTrain

13. This ultra-fast, stable, and reliable network enables the
rapid assembly of new smart instruments

14. This ultra-fast, stable, and reliable network enables the
rapid assembly of new smart instruments
ModelTrain

15. This ultra-fast, stable, and reliable network enables the
rapid assembly of new smart instruments
Serial synchrotron crystallography: https://doi.org/10.1101/2022.04.06.487333

16. ModelTrain
Mapping of smart instruments to ESnet6-connected resources
requires an integrated research infrastructure

17. For example, Globus, hybrid software-as-a-service for
secure, reliable, performant data orchestration …
https://globus.org

18. … and for task orchestration

19. Polaris
Bebop
Cluster
Argonne
Leadership
Computing
Facility
Laboratory
Computing
Research
Center
Eagle store
APS
Computing
Orthros Cluster
APS DM system
Portal
server
Portal
server
Theta
Advanced
Photon
Source Key: funcX agent
Globus Connect agent
Globus-accessible
storage and
computing
(10,000s of systems)
Endpoints are widely deployed at
DOE facilities

20. Globus is a not-for-profit service operated for the research community
by the University of Chicago, supported by ~200 subscribing institutions

21. ESNet6 ushers in a new era!
An ultra-fast, predictable,
and reliable network …
… plus an integrated
research infrastructure
… enables new smart
instruments that will drive
discovery and innovation
There’s still much to be done:
• Build & apply new instruments
• Reach the white space
• Smart quantum instruments Thanks to ESnet, DOE ASCR, NSF, NIST, NIH,
Argonne, Globus team, and many other partners