Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Pacific Wave and PRP Update
Big News for Big Data
John Hess
Dr. Larry Smarr
WESTNET 2016
FORT LEWIS COLLEGE
JUNE 16, 2016
Six Charter Associates:
• California K-12 System
• California Community Colleges
• California State University System
• St...
Three networks operate simultaneously as
independent layers on a single infrastructure:
 CalREN Digital California (DC) d...
CENIC: California’s Research & Education Network
• 3,800+ miles of optical fiber
• Members in all 58 counties connect via
...
Pacific Wave
and WRN
• Pacific Wave and the Western Region Network provide
for a 100Gbps network spanning the Western Unit...
Pacific Wave
International Exchange
A project of CENIC and PNWGP
John Hess
Network Engineer
w w w . p n w - g i g a p o p ...
Pacific Wave
• Began as first geographically distributed exchange in
2004
• Pacific Wave is an open exchange supporting bo...
R&E Exchanges within R&E
• StarLight (Chicago, IL)
– StarLight Consortium/MREN
• MANLAN (New York, NY)
– NYSERnet
• WIX (W...
National/Global Activities
• NSF provides support of the R&E exchange points
through the competitive IRNC (International R...
GLIF/GOLE
Nx100G Across the Pacific
• CURRENT:
– TransPac/Pacific Wave (Tokyo-Seattle)
– SINGAREN/Internet2 (Singapore-Los Angeles)
...
Pacific Wave and NSF/IRNC
• Pacific Wave has been partially supported
through three separate five-year National
Science Fo...
SDX = SDN + IXP
14
AS A Router
AS C Router
AS B Router
BGP Session
SDN Switch
SDX Controller
SDX
Abstraction Layer (FlowSpace Firewall)
OpenFlow Switches
On-ramp Locations (Ethernet / virtual circuits)
Network Testbed
E...
Next Step: The Pacific Research Platform Creates
a Regional End-to-End Science-Driven “Big Data Freeway System”
NSF CC*DNI...
The Pacific Research Platform (PRP)
• NSF CC-NIE and similar projects represent significant investments in campus
infrastr...
DOE ESnet’s Science DMZ: A Scalable Network
Design Model for Optimizing Science Data Transfers
A Science DMZ integrates fo...
PRPv0 - An experiment including:
Caltech
CENIC / Pacific Wave
ESnet / LBNL
NASA Ames / NREN
San Diego State University
SDS...
21
PRPv0 Experiment
The PRPv0 experiment concentrated on the
regional aspects of the research data movement
challenge.
 H...
FIONA – Flash I/O Network Appliance:
Linux PCs Optimized for Big Data on DMZs
FIONAs Are
Science DMZ Data Transfer Nodes (...
 DTNs loaded with Globus
Connect Server suite to obtain
GridFTP tools.
 cron-scheduled transfers using
globus-url-copy.
...
January 29, 2016 PRPV1 (L3)
PRP Point-to-Point Bandwidth Map
GridFTP File Transfers-Note Huge Improvement in Last Six Mont...
Troubleshooting Unidirectional Performance Issues
Measuring Performance – IPv6
Measuring Performance – IPv4
28
PRP Timeline
• PRPv1
– A routed Layer 3 architecture
– Tested, Measured, Optimized, With Multi-domain Science Data
– Bring...
Resources
w w w . p n w - g i g a p o p . n e t
Pacific Wave
http://www.pacificwave.net/
https://ps-dashboard.pacificwave....
Vision:
Creating a Pacific Research Platform
Use Optical Fiber Networks to Connect
All Data Generators and Consumers,
Crea...
Creating a “Big Data” Freeway on Campus:
NSF-Funded Prism@UCSD and CHeruB Grants
Prism@UCSD, Phil Papadopoulos, SDSC, Cali...
How Prism@UCSD Transforms Big Data Microbiome Science:
Preparing for Knight/Smarr 1 Million Core-Hour Analysis
12 Cores/GP...
For Big Data Science, One Needs Bandwidths Orders of Magnitude Higher
Than the Shared Internet Between Campuses
Bandwidth ...
Invitation-Only PRP Workshop Held in Calit2’s Qualcomm Institute
October 14-16, 2015
• 130 Attendees From 40 organizations...
GPU JupyterHub:
2 x 14-core CPUs
256GB RAM
1.2TB FLASH
3.8TB SSD
Nvidia K80 GPU
Dual 40GbE NICs
And a Trusted Platform
Mod...
Cancer Genomics Hub (UCSC) is Housed in SDSC:
Large Data Flows to End Users at UCSC, UCB, UCSF, …
1G
8G
Data Source: David...
Two Automated Telescope Surveys
Creating Huge Datasets Will Drive PRP
300 images per night.
100MB per raw image
30GB per n...
Global Scientific Instruments Will Produce Ultralarge Datasets Continuously
Requiring Dedicated Optic Fiber and Supercompu...
community resources. This facility depends on a range of common services, support activities, software,
and operational pr...
We are Experimenting with the PRP for Large Hadron Collider Data Analysis
Using The West Coast Open Science Grid on 10-100...
40G FIONAs
20x40G PRP-connected
WAVE@UC San Diego
PRP Links
Creates Distributed Virtual Reality
PRP
CAVE@UC Merced
Dan Cayan
USGS Water Resources Discipline
Scripps Institution of Oceanography, UC San Diego
much support from Mary Tyree, ...
average summer
afternoon temperature
average summer
afternoon temperature
Downscaling Supercomputer Climate Simulations
To...
approximately 50 miles:
Note: locations are approximate
to CI and
PEMEX
Extending PRP/CENIC Optical Backplane
via High Spe...
Real-Time Network Cameras on Mountains
for Environmental Observations
Source: Hans Werner Braun,
HPWREN PI
14 May 2014:
9 Simultaneous Active Fires in San Diego County
San Diego County Red Mountain Fire Cameras
• Southeast (left)...
Interactive Virtual Reality of San Diego County
Includes Live Feeds From 150 Met Stations
TourCAVE at Calit2’s Qualcomm In...
HPWREN Users and Public Safety Clients
Gain Redundancy and Resilience from PRP Upgrade
San Diego Countywide
Sensors and Ca...
NSF Has Funded Over 100 Campuses to Build Local Big Data Freeways:
Imagine Linking All of Them Like the Pacific Research P...
Next Step: Global Research Platform
Building on CENIC/Pacific Wave and GLIF
Current
International
GRP Partners
Pacific Wave and PRP Update Big News for Big Data
Pacific Wave and PRP Update Big News for Big Data
Pacific Wave and PRP Update Big News for Big Data
Prochain SlideShare
Chargement dans…5
×

Pacific Wave and PRP Update Big News for Big Data

327 vues

Publié le

Westnet 2016
Fort Lewis College
June 16, 2016

Publié dans : Données & analyses
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Pacific Wave and PRP Update Big News for Big Data

  1. 1. Pacific Wave and PRP Update Big News for Big Data John Hess Dr. Larry Smarr WESTNET 2016 FORT LEWIS COLLEGE JUNE 16, 2016
  2. 2. Six Charter Associates: • California K-12 System • California Community Colleges • California State University System • Stanford, Caltech, USC • University of California • California Public Libraries • CENIC is a 501(c)3 created to serve California’s K-20 research & education institutions with cost-effective, high-bandwidth networking
  3. 3. Three networks operate simultaneously as independent layers on a single infrastructure:  CalREN Digital California (DC) daily use for e-mail, web browsing, videoconferencing, etc.  CalREN High Performance Research (HPR) high-performance, data-intensive efforts  CalREN eXperimental Developmental (XD) bleeding-edge research on the network itself CENIC: California’s Research & Education NetworkCENIC: California’s Research & Education Network
  4. 4. CENIC: California’s Research & Education Network • 3,800+ miles of optical fiber • Members in all 58 counties connect via fiber-optic cable or leased circuits from telecom carriers • Over 10,000 sites connect to CENIC • 20,000,000 Californians use CENIC • Governed by members on the segmental level • Collaborate with over 500 private sector partners • 88 other peering partners (Google, Microsoft, Amazon …) • Enables worldwide collaboration
  5. 5. Pacific Wave and WRN • Pacific Wave and the Western Region Network provide for a 100Gbps network spanning the Western United States serving PNWGP, CENIC, FRGP, ABQGP and UH. • Pacific Wave and NSF IRNC awardee PIREN (Univ of Hawaii) work together supporting AARNet links to California and Washington and expansion of high- speed service through the Pacific Islands Region w w w . p n w - g i g a p o p . n e t
  6. 6. Pacific Wave International Exchange A project of CENIC and PNWGP John Hess Network Engineer w w w . p n w - g i g a p o p . n e t
  7. 7. Pacific Wave • Began as first geographically distributed exchange in 2004 • Pacific Wave is an open exchange supporting both commercial and R&E peers • Currently serves 29 countries peering across the Pacific and Western United States • With PNWGP and TransPac, announced the first 100Gbps Trans-Pacific link from Tokyo to Seattle in 2015
  8. 8. R&E Exchanges within R&E • StarLight (Chicago, IL) – StarLight Consortium/MREN • MANLAN (New York, NY) – NYSERnet • WIX (Washington, DC) – University of Maryland/MAX GigaPOP • AmLight (Miami, Florida) – Florida International University/Florida LambdaRail • Pacific Wave (Western US) – CENIC and PNWGP
  9. 9. National/Global Activities • NSF provides support of the R&E exchange points through the competitive IRNC (International Research Network Connections) program with funding for backbone, infrastructure and innovation • The Global Lambda Integrated Facility – The GLIF brings together some of the world’s premier networking engineers who are working together to develop and international infrastructure
  10. 10. GLIF/GOLE
  11. 11. Nx100G Across the Pacific • CURRENT: – TransPac/Pacific Wave (Tokyo-Seattle) – SINGAREN/Internet2 (Singapore-Los Angeles) – SINET/SoftBank/Pacific Wave (Tokyo-Los Angeles) – AARNET/PIREN/Pacific Wave (Australia-SEA) • FUTURE: – AARNET/PIREN/Pacific Wave (Australia-LA) – end of June 2016 – UH/PIREN/Pacific Wave (Guam-Hawaii-LA)
  12. 12. Pacific Wave and NSF/IRNC • Pacific Wave has been partially supported through three separate five-year National Science Foundation grants supporting growth, connectivity and innovation • Current award promotes 100G expansion and implementation of SDX capabilities within Pacific Wave (ACI-1451050)
  13. 13. SDX = SDN + IXP 14 AS A Router AS C Router AS B Router BGP Session SDN Switch SDX Controller SDX
  14. 14. Abstraction Layer (FlowSpace Firewall) OpenFlow Switches On-ramp Locations (Ethernet / virtual circuits) Network Testbed Enivironments Circuit Building (NSI) SDX middleware OpenFlow Controllers (plural) Testbed Resources / Other Uses (DTNs) Science Group Applications / Uses Pacific Wave SDX Testbed Control Plane
  15. 15. Next Step: The Pacific Research Platform Creates a Regional End-to-End Science-Driven “Big Data Freeway System” NSF CC*DNI Grant $5M 10/2015-10/2020 PI: Larry Smarr, UC San Diego Calit2 Co-Pis: • Camille Crittenden, UC Berkeley CITRIS, • Tom DeFanti, UC San Diego Calit2, • Philip Papadopoulos, UC San Diego SDSC, • Frank Wuerthwein, UC San Diego Physics and SDSC
  16. 16. The Pacific Research Platform (PRP) • NSF CC-NIE and similar projects represent significant investments in campus infrastructure including SDN, Science DMZ’s (~130 projects) • But the scientists are still struggling with the complexity of using the network and interoperability between different implementations of Science DMZ’s • PRP focuses on enabling the science communities across the Pacific region to make effective use of the high performance infrastructure • Kick-off in December 2014: take advantage of the regional infrastructure; perfSONAR for measurement / analysis and MaDDash for visualization • Include DTN’s: use a common software suite for data movement; reflect disk-to- disk performance on MaDDash • Demonstrated as a proof-of-concept at the CENIC Spring meeting (March 2015)
  17. 17. DOE ESnet’s Science DMZ: A Scalable Network Design Model for Optimizing Science Data Transfers A Science DMZ integrates four key concepts into a unified whole: – A network architecture designed for high-performance applications, with the science network distinct from the general-purpose network – The use of dedicated systems for data transfer – Performance measurement and network testing systems that are regularly used to characterize and troubleshoot the network – Security policies and enforcement mechanisms that are tailored for high performance science environments http://fasterdata.es.net/science-dmz/
  18. 18. PRPv0 - An experiment including: Caltech CENIC / Pacific Wave ESnet / LBNL NASA Ames / NREN San Diego State University SDSC Stanford University University of Washington USC UC Berkeley UC Davis UC Irvine UC Los Angeles UC Riverside UC San Diego UC Santa Cruz 20
  19. 19. 21 PRPv0 Experiment The PRPv0 experiment concentrated on the regional aspects of the research data movement challenge.  High-performance interconnection among campus Science DMZs  A mesh of perfSONAR toolkit instances  perfSONAR MaDDash -- Measurement and Debugging Dashboard  Flash I/O Network Appliances (FIONAs) and Data Transfer Nodes (DTNs)  GridFTP file transfers to quantify throughput, with results reflected on MaDDash  CalREN HPR / AS2153  A partial mesh of bilateral BGP sessions across the Pacific Wave distributed exchange
  20. 20. FIONA – Flash I/O Network Appliance: Linux PCs Optimized for Big Data on DMZs FIONAs Are Science DMZ Data Transfer Nodes (DTNs) & Optical Network Termination Devices UCSD CC-NIE Prism Award & UCOP Phil Papadopoulos & Tom DeFanti Joe Keefe & John Graham Cost $8,000 $20,000 Intel Xeon Haswell E5-1650 v3 6-Core 2x E5-2697 v3 14-Core RAM 128 GB 256 GB SSD SATA 3.8 TB SATA 3.8 TB Network Interface 10/40GbE Mellanox 2x40GbE Chelsi+Mellanox GPU NVIDIA Tesla K80 RAID Drives 0 to 112TB (add ~$100/TB) UCOP Rack-Mount Build: Source: John Graham and Tom DeFanti, Calit2
  21. 21.  DTNs loaded with Globus Connect Server suite to obtain GridFTP tools.  cron-scheduled transfers using globus-url-copy.  ESnet-contributed script parses GridFTP transfer log and loads results in an esmond measurement archive.  FDT – developed by Caltech in collaboration with Polytehnica Bucharest 23 As of 3/9/15, the Pacific Research Platform (PRPv0) as a facility, logs rather good performance: From To Measured Bandwidth Data Transfer Utility San Diego State Univ. UC Los Angeles 5Gb/s out of 10 GridFTP UC Riverside UC Los Angeles 9Gb/s out of 10 GridFTP UC Berkeley UC San Diego 9.6Gb/s out of 10 GridFTP UC Davis UC San Diego 9.6Gb/s out of 10 GridFTP UC Irvine UC Los Angeles 9.6Gb/s out of 10 GridFTP UC Santa Cruz UC San Diego 9.6Gb/s out of 10 FDT Stanford UC San Diego 12Gb/s out of 40 FDT Univ. of Washington UC San Diego 12Gb/s out of 40 FDT UC Los Angeles UC San Diego 36Gb/s out of 40 FDT Caltech UC San Diego 36Gb/s out of 40 FDT Table I.2.1: Bandwidth of flash disk-to-flash disk file transfers shown between several sites for the existing experimental facility “PRPv0.”
  22. 22. January 29, 2016 PRPV1 (L3) PRP Point-to-Point Bandwidth Map GridFTP File Transfers-Note Huge Improvement in Last Six Months June 6, 2016 PRPV1 (L3) Green is Disk-to-Disk In Excess of 5Gbps
  23. 23. Troubleshooting Unidirectional Performance Issues
  24. 24. Measuring Performance – IPv6
  25. 25. Measuring Performance – IPv4
  26. 26. 28
  27. 27. PRP Timeline • PRPv1 – A routed Layer 3 architecture – Tested, Measured, Optimized, With Multi-domain Science Data – Bring Many Of Our Science Teams Up – Each Community Thus Will Have Its Own Certificate-Based Access To its Specific Federated Data Infrastructure. • PRPv2 – Incorporating SDN/SDX, AutoGOLE / NSI – Advanced IPv6-Only Version with Robust Security Features – e.g. Trusted Platform Module Hardware and SDN/SDX Software – Support Rates up to 100Gb/s in Bursts And Streams – Develop Means to Operate a Shared Federation of Caches – Cooperating Research Groups
  28. 28. Resources w w w . p n w - g i g a p o p . n e t Pacific Wave http://www.pacificwave.net/ https://ps-dashboard.pacificwave.net CENIC http://www.cenic.org/ https://ps-dashboard.cenic.net Pacific Research Platform http://prp.ucsd.edu/ http://cenic.org/files/publications/PRP_Overview_%C6%92.pdf http://prp-maddash.calit2.optiputer.net/maddash-webui/ Calit2 http://www.calit2.net/ CITRIS http://citris-uc.org/ ESnet http://www.es.net/ http://fasterdata.es.net/ http://ps-dashboard.es.net/
  29. 29. Vision: Creating a Pacific Research Platform Use Optical Fiber Networks to Connect All Data Generators and Consumers, Creating a “Big Data” Freeway System “The Bisection Bandwidth of a Cluster Interconnect, but Deployed on a 20-Campus Scale.” This Vision Has Been Building for 15 Years
  30. 30. Creating a “Big Data” Freeway on Campus: NSF-Funded Prism@UCSD and CHeruB Grants Prism@UCSD, Phil Papadopoulos, SDSC, Calit2, PI (2013-15) CHERuB, Mike Norman, SDSC PI CHERuB These Are Two of Over 100 NSF Campus Cyberinfrastructure Grants Made in the Last 4 Years
  31. 31. How Prism@UCSD Transforms Big Data Microbiome Science: Preparing for Knight/Smarr 1 Million Core-Hour Analysis 12 Cores/GPU 128 GB RAM 3.5 TB SSD 48TB Disk 10Gbps NIC Knight Lab 10Gbps Gordon Prism@UCSD Data Oasis 7.5PB, 200GB/s Knight 1024 Cluster In SDSC Co-Lo CHERuB 100Gbps Emperor & Other Vis Tools 64Mpixel Data Analysis Wall 120Gbps 40Gbps 1.3Tbps
  32. 32. For Big Data Science, One Needs Bandwidths Orders of Magnitude Higher Than the Shared Internet Between Campuses Bandwidth from My Office in Calit2’s Qualcomm Institute Bandwidth On the Pacific Research Platform: 500 Times the Bandwidth of the Shared Internet!
  33. 33. Invitation-Only PRP Workshop Held in Calit2’s Qualcomm Institute October 14-16, 2015 • 130 Attendees From 40 organizations – Ten UC Campuses, as well as UCOP Plus 11 Additional US Universities – Four International Organizations (from Amsterdam, Canada, Korea, and Japan) – Five Members of Industry Plus NSF
  34. 34. GPU JupyterHub: 2 x 14-core CPUs 256GB RAM 1.2TB FLASH 3.8TB SSD Nvidia K80 GPU Dual 40GbE NICs And a Trusted Platform Module GPU JupyterHub: 1 x 18-core CPUs 128GB RAM 3.8TB SSD Nvidia K80 GPU Dual 40GbE NICs And a Trusted Platform Module PRP UC-JupyterHub Backbone UCB Next Step: Deploy Across PRP UCSD Source: John Graham, Calit2
  35. 35. Cancer Genomics Hub (UCSC) is Housed in SDSC: Large Data Flows to End Users at UCSC, UCB, UCSF, … 1G 8G Data Source: David Haussler, Brad Smith, UCSC 15G Jan 2016 30,000 TB Per Year
  36. 36. Two Automated Telescope Surveys Creating Huge Datasets Will Drive PRP 300 images per night. 100MB per raw image 30GB per night 120GB per night 250 images per night. 530MB per raw image 150 GB per night 800GB per night When processed at NERSC Increased by 4x Source: Peter Nugent, Division Deputy for Scientific Engagement, LBL Professor of Astronomy, UC Berkeley Precursors to LSST and NCSA PRP Allows Researchers to Bring Datasets from NERSC to Their Local Clusters for In-Depth Science Analysis Data Flows Over HPWREN
  37. 37. Global Scientific Instruments Will Produce Ultralarge Datasets Continuously Requiring Dedicated Optic Fiber and Supercomputers Square Kilometer Array Large Synoptic Survey Telescope https://tnc15.terena.org/getfile/1939 www.lsst.org/sites/default/files/documents/DM%20Introduction%20-%20Kantor.pdf Tracks ~40B Objects, Creates 10M Alerts/Night Within 1 Minute of Observing 2x40Gb/s
  38. 38. community resources. This facility depends on a range of common services, support activities, software, and operational principles that coordinate the production of scientific knowledge through the DHTC model. In April 2012, the OSG project was extended until 2017; it is jointly funded by the Department of Energy and the National Science Foundation. OSG Federates Clusters in 40/50 States: Creating a Scientific Compute and Storage “Cloud” Source: Miron Livny, Frank Wuerthwein, OSG
  39. 39. We are Experimenting with the PRP for Large Hadron Collider Data Analysis Using The West Coast Open Science Grid on 10-100Gbps Optical Networks Crossed 100 Million Core-Hours/Month In Dec 2015 Over 1 Billion Data Transfers Moved 200 Petabytes In 2015 Supported Over 200 Million Jobs In 2015 Source: Miron Livny, Frank Wuerthwein, OSG ATLAS CMS
  40. 40. 40G FIONAs 20x40G PRP-connected WAVE@UC San Diego PRP Links Creates Distributed Virtual Reality PRP CAVE@UC Merced
  41. 41. Dan Cayan USGS Water Resources Discipline Scripps Institution of Oceanography, UC San Diego much support from Mary Tyree, Mike Dettinger, Guido Franco and other colleagues NCAR Upgrading to 10Gbps Link Over Westnet from Wyoming and Boulder to CENIC/PRP Sponsors: California Energy Commission NOAA RISA program California DWR, DOE, NSF Planning for climate change in California substantial shifts on top of already high climate variability UCSD Campus Climate Researchers Need to Download Results from NCAR Remote Supercomputer Simulations to Make Regional Climate Change Forecasts
  42. 42. average summer afternoon temperature average summer afternoon temperature Downscaling Supercomputer Climate Simulations To Provide High Res Predictions for California Over Next 50 Years 45 Source: Hugo Hidalgo, Tapash Das, Mike Dettinger
  43. 43. approximately 50 miles: Note: locations are approximate to CI and PEMEX Extending PRP/CENIC Optical Backplane via High Speed Wireless Research and Education Network
  44. 44. Real-Time Network Cameras on Mountains for Environmental Observations Source: Hans Werner Braun, HPWREN PI
  45. 45. 14 May 2014: 9 Simultaneous Active Fires in San Diego County San Diego County Red Mountain Fire Cameras • Southeast (left) “Highway” Fire • Southwest (center rear) “Poinsettia” Fire • West (right) “Tomahawk” Fire
  46. 46. Interactive Virtual Reality of San Diego County Includes Live Feeds From 150 Met Stations TourCAVE at Calit2’s Qualcomm Institute
  47. 47. HPWREN Users and Public Safety Clients Gain Redundancy and Resilience from PRP Upgrade San Diego Countywide Sensors and Camera Resources UCSD & SDSU Data & Compute Resources UCSD UCR SDSU UCI UCI & UCR Data Replication and PRP FIONA Anchors as HPWREN Expands Northward 10X Increase During Wildfires Data From Hans-Werner Braun • PRP CENIC 10G Link UCSD to SDSU – DTN FIONAs Endpoints – Data Redundancy – Disaster Recovery – High Availability – Network Redundancy
  48. 48. NSF Has Funded Over 100 Campuses to Build Local Big Data Freeways: Imagine Linking All of Them Like the Pacific Research Platform Red 2012 CC-NIE Awardees Yellow 2013 CC-NIE Awardees Green 2014 CC*IIE Awardees Blue 2015 CC*DNI Awardees Purple Multiple Time Awardees Source: NSF
  49. 49. Next Step: Global Research Platform Building on CENIC/Pacific Wave and GLIF Current International GRP Partners

×