Campus Bridging with Globus Services

globus online

Campus Bridging Made Easy
via Globus Services
Ian Foster, Rajkumar Kettimuthu, Stuart Martin,
Steve Tuecke: Chicago and Argonne
Thomas Hauser, Daniel Milroy, Jazcek Braden: Colorado
Brock Palen: Michigan www.globusonline.org

Campus bridging

“the seamlessly integrated use of
cyberinfrastructure operated by a scientist or
engineer with other cyberinfrastructure on the
scientist’s campus, at other campuses, and at
the regional, national, and international levels
as if they were proximate to the scientist”

-- NSF Advisory Committee for Cyberinfrastructure
Task Force on Campus Bridging Final Report,
March 2011.
www.globusonline.org

“Use of data resources from campus on
XSEDE, or from XSEDE at a campus”*
•  Researchers often use a range of resources and
must move data among them
•  Desktop, campus clusters, remote instruments, national
computing facilities, commercial clouds, …

•  Researcher desktops and campus clusters often
lack sophisticated data movement tools
•  Transient network and system failures have to be dealt with
•  Each resource has its own security domain
•  Firewalls and other problems often get in the way too
*Campus Bridging Use Cases, XSEDE Project, 2012.

Two distinct groups of stakeholders

1) Individuals (researchers, educators, students)
– Easy installation of access layer interface
– Intuitive GUI for file transfer
– No interruptions for transient failures
– Transfer efficiency

2) System administrators
– Easy integration of a campus resource into
campus and national cyberinfrastructure
– Easy management in terms of adding users,
tracking usage, etc.


as
Globus Transfer: Data movement a
Service
Reliable file transfer.
- Fire-and-forget
- Automatic fault recovery
- High performance
- Across security domains

No IT required.
- Intuitive Web 2.0 interface
- No client software install
- New features available
automatically
- Consolidated support
and troubleshooting Works with existing GridFTP servers; also Globus Connect

XSEDE-aware


Globus Connect

Globus (1) Globus Connect
User
(2) User makes request Online client registers with
to Globus Online: e.g.,"transfer Globus Online
data from MyDesktop to SiteA"
(3) Globus Online
forwards requests
to Globus Connect
GridFTP Globus
server Connect
"SiteA" (4) Globus Connect establishes data channel
connection to SiteA and transfers data
"MyDesktop"



– Easy installation of access layer interface ✔
– Intuitive GUI for file transfer ✔
– No interruptions for transient failures ✔
– Transfer efficiency ✔



GridFTP security configuration, old way

Installation
•  Download, untar, configure, make
Security configuration (server admins)
•  Obtain and install X.509 host certificate from well-known CA
•  Configure trust roots
Security configuration (users)
•  Obtain and install user certificate from well-known CA
•  Configure trust roots
Setup authorization (both users and admins)
•  DN to local username mapping in gridmap file
•  '/DC=org/DC=doegrids/OU=People/CN=Rajkumar Kettimuthu
227852' rajk
Too complex for many users and small labs

www.globustoolkit.org 13 www.globusonline.org

Globus Connect Multi-User
•  What is GCMU?
•  Multi-user version of Globus Connect
•  Packages a GridFTP server and MyProxy CA, pre-configured for
use with Globus Online
•  Why GCMU?
•  Create transfer endpoints in minutes
•  Avoid complex GridFTP install
•  Avoid frequent sources of user and administrator error
•  To download: https://www.globusonline.org/gcmu/
“We used GCMU to form a campus- “As a resource admin, I've
wide GSI authentication service found GCMU an exceedingly
spanning multiple servers. Now my useful tool.... With GCMU,
users have a fast, easy way to get setting up a GridFTP server
their data wherever it needs to go, and handling authentication for
and the setup process was trivial." multiple users is easy."
--University of Michigan --Oak Ridge National Lab
www.globustoolkit.org www.globusonline.org

GCMU makes deploy and config trivial

Make GridFTP deployment trivial
•  GridFTP transfers can be achieved “instantly” even by
non-experts
Automate the process of configuring security
•  Avoid the need for any end-user or system administrator
involvement in security configuration
Reduce burden on both users and
administrators
•  Eliminate frequent sources of errors in GridFTP
configuration and use.


Globus Transfer / GCMU Interaction


OAuth protocol to protect passwords

•  Site passwords flow through Globus Online
•  Globus Online does not store passwords
•  Just pass along to MyProxy servers at site
•  Still a security concern for some sites
•  OAuth
•  Sites run an OAuth
server
•  Users enter username
and password only on
a site’s webpage
•  Globus Online gets an
X.509 credential
via Oauth protocol

Globus Connect Multi User with
OAuth (coming soon)
Step 1
Access Endpoint
Globus Online (Hosted Service)

Step 2 Step 7 Step 8 Transfer
Step 3
Redirect request
Username
password certificate certificate

Step 4 GCMU
Username Step 11 GridFTP
MyProxy
password GridFTP certificate Server
Oauth Online CA Authentication
Server
Server
certificate & Data Transfer Remote Cluster
PAM
Campus Cluster

Step 6
/ User’s PC
Step 5 Step 9 Step 10

Authorization
Username Access files
certifficate
password

Local Authentication System Local
(LDAP, RADIUS, Kerberos etc) Storage



– Easy integration of a campus resource into ✔
– Easy management in terms of adding users, ✔

GCMU deployments (as of April 2012)


GCMU endpoints and users


GCMU – Bytes transferred


GC users


GC – Bytes transferred


Campus bridging at CU-Boulder

•  Janus Supercomputer
- 16,416 Westmere cores, 2GB memory per core
- Four Dell PowerEdge R710s as GridFTP servers
- Dedicated 10Gb ethernet per node
- RC network: “private VLANs”

•  Globus Online endpoints
- colorado#gridftp
122 TB transferred from
22 TB transferred to
- colorado#jila, colorado#nsidc
--data-interface <vlan>

Campus bridging at CU-Boulder (contd)

•  Globus Transfer and “manual tuning”
•  CLI transfer with -cc 4 -p 4 -pp 4
•  In “external” transfers, we noticed 44% increase in
transfer rate for default packets and 26% for MTU 9000

•  Problem with jumbo frames
•  Path MTU discovery and ICMP filtering
•  Probably the issue- reverting to default packets
solved the problem
•  Determined to be the issue with JILA transfers


CU-Boulder

Data transferred from colorado#gridftp 122.5 TB

Data transferred to colorado#gridftp 21.6 TB

Peak transfer rate between distinct endpoints 2.9 Gb/s

Peak transfer rate to/from Janus (disk) 5.9 Gb/s

Peak transfer rate to/from Janus (memory) 9.5 Gb/s


GridFTP at Michigan

Single MyProxy Server for Campus
•  Users: PAM+Kerberos+LDAP
•  Built from GCMU

Multiple GridFTP Servers
•  Not all under umich#
•  Offer documentation and help to setup endpoints
•  Built from GCMU

http://cac.engin.umich.edu/resources/loginnodes/globus.html


Many small users


Campus bridging at UMichigan

•  UMichigan has five Globus Transfer endpoints
•  Two endpoints at College of Engineering HPC systems
•  The other three endpoints at other departments

Data transferred from umich#nyx 9.8 TB
Data transferred to umich#nyx 10.4 TB
Data transferred from umich#flux 20.4 TB

Data transferred to umich#flux 6.5 TB

Globus and Campus Bridging

•  Globus Transfer – simple file transfer service
•  SaaS methods for easy fire-and-forget transfers, high
performance, automatic fault recovery
•  Web 2.0; integrated knowledge of XSEDE resources
•  (Leverages Globus Nexus – identity management; sign
in from federated identity systems such as InCommon
and from OpenID providers such as Google)

•  Globus Connect – one click GridFTP for desktops
•  Globus Connect Multi User (GCMU) – easy-to-
install GridFTP and security package
•  Globus Storage – user-managed storage [soon]

Campus Bridging with Globus Services

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Campus Bridging with Globus Services

Similaire à Campus Bridging with Globus Services (20)

Plus de Ian Foster

Plus de Ian Foster (20)

Dernier

Dernier (20)

Campus Bridging with Globus Services