1.
May 11, 2022
Globus Connect Server v5:
Deployment and Migration

2.
Outline
• GCsv5 concepts
• GCS v4 and v5 comparison
• GCSv5 install demo and walkthrough
• Migration overview
• Migration tools demo and walkthrough
2

3.
GCS v5 Concepts
3

4.
Globus Connect Server version 5
4

5.
5

6.
GCS v4 and v5
Comparison
6

7.
Out with the old, in with the new
• Host endpoints è Mapped collections
– Need local account to access data
• Shared endpoints è Guest collections
– No local account needed for data access, permissions set in Globus
• Use host endpoint to create shared endpoint è
Use mapped collection to create (guest) collections

8.
GCSv4 vs GCSv5
Feature GCSv4 GCSv5
Credentials for
endpoints
• GlobusID was required to
create endpoint
• Any identity that can be used
to log into Globus can be
used.
• Endpoint registered as a
resource server with Globus
Auth, and has client id/secret.
Policy and data access
interfaces
• Static endpoint definition
supporting one collection
• Configuration changes
required re-running setup on
each DTN
• Data access via GridFTP only
• Endpoint supports multiple
storage systems and
collections
• Automated management
across DTNs
• Data access via GridFTP and
HTTPS

9.
GCSv4 vs GCSv5
Feature GCSv4 GCSv5
Server credentials • Default were issued by
Globus CA
• Custom certificates
supported
• Default from Lets Encrypt
to ensure browser based
downloads work
• Custom certificate
supported
Management interface • On the DTN, configuration
files and tools
• GCS Manager service on
the DTNs.
• CLI shipped with the
service

10.
GCSv4 vs GCSv5
Feature GCSv4 GCSv5
Management roles (admin,
activity manager, activity
monitor)
• Supported • Unchanged
High Assurance data
access
• Not supported • A storage gateway can
be configured for high
assurance data handling,
and additional security
constraints are added
User authentication to data
access
• InCommon OAuth,
MyProxy or MyProxy
OAuth
• Any identity used to log
into Globus
• Custom identity provider
via OIDC

11.
GCSv4 vs GCSv5
Feature GCSv4 GCSv5
Mapping to local
account
• Data access identity
independent from Globus
account
• User certificate DN to local
account
• ePPN based, GridMap
files, custom callout
• Policy maps Globus
authentication information
to local account
• Expression based
matching for values in the
authentication context
• Custom external program
Custom domain name • Not applicable • Custom domain name for
collections

12.
GCS management conceptual architecture
12
Data Transfer Node
GCS Command
Line Interface
GridFTP
Server
Globus
Transfer
Service
GCS
management
requests
Globus
Auth
Service
GCS Manager authorize request
using client ID/secret
GCS Manager endpoint:
abc.abc.data.globus.org
Register a Globus Connect Server at
developers.globus.org get GCS
client ID, secret
Define Globus
Transfer
resources
(gateways,
collections, …)

13.
GCS v5 install
walkthrough
13
docs.globus.org/globus-connect-server
docs.globus.org/globus-connect-server/v5.4/quickstart

14.
Requires a Globus subscription
GCSv5 installation/configuration summary
1. Register a Globus Connect Server with Globus Auth
2. Install GCS packages on data transfer node (DTN)
3. Set up the endpoint and add node(s)
4. Create a POSIX storage gateway
5. Create a mapped collection
6. Associate endpoint with a subscription
7. Create a guest collection
8. Enable browser down/upload (HTTPS access)
9. Add other storage systems to the endpoint

15.
Register GCS and get credentials
• Navigate to developers.globus.org and log in
• (Optional) Create a project
• Add a new Globus Connect Server
• Generate a client secret
• Save the client ID and secret
• Add other administrators to this project

16.
1. Register GCS and get credentials
developers.globus.org

17.
2. Install Globus Connect Server v5 packages
$ curl -LOs
http://downloads.globus.org/toolkit/gt6/stable/installers/repo/deb/globus-
toolkit-repo_latest_all.deb
$ dpkg -i globus-toolkit-repo_latest_all.deb
$ sed -i /etc/apt/sources.list.d/globus-toolkit-6-stable*.list
> -e 's/^# deb /deb /'
$ sed -i /etc/apt/sources.list.d/globus-connect-server-stable*.list
> -e 's/^# deb /deb /'
$ apt-key add /usr/share/globus-toolkit-repo/RPM-GPG-KEY-Globus
$ apt-get update
$ apt-get --assume-yes install globus-connect-server54

18.
3. Set up endpoint and add node
$ globus-connect-server endpoint setup
> "My APS Endpoint"
> --organization "Argonne National Laboratory"
> --client-id 4321dddd-af72-4c4b-9533-a0f4055dd321
> --owner me@anl.gov
$ globus-connect-server node setup
> --client-id 4321dddd-af72-4c4b-9533-a0f4055dd321
Note: endpoint setup command generates deployment-key.json
Use this file when setting up additional data transfer nodes

19.
Setup so far
Run globus-connect-server node setup
to set up additional data transfer nodes
Copy deployment-key.json
from original DTN

20.
Storage Gateways define a set of access policies
• Authentication policy for local account-holders
– Which identity domain(s) are acceptable?
– Is MFA required?
– How often must the user authenticate to initiate data access
operations? (authentication timeout)
– Is this storage gateway for access to protected data? (High
Assurance settings)
• Identity Mapping policies
– Which local account does the authenticated user map to?

21.
Storage Gateways define a set of access policies
• Data Access Policies
– Which parts of the storage system are accessible via Globus?
– Which local accounts does this policy allow (or deny)?

22.
Mapping identities to local accounts
• Default: Strip identity domain (everything after “@”)
– e.g. userX@globusdemo.org maps to local account userX
– Best for campus identities w/synchronized local accounts
• Use --identity-mapping option on storage gateway
– Specify expression in a JSON document
– Execute a custom script
docs.globus.org/globus-connect-server/v5.4/identity-mapping-guide

23.
Picking identity domains
• User must present identity from one of the configured
domains
– On access attempts, linked identities will be scanned for a match
– If no identity from the required domain(s), will be asked to link one
• Identity domains may include…
– …any organization in Globus federated list
– …your institution’s identity provider trusted by Globus
– …a local OpenID Connect (OIDC) server integrated via PAM to
authentication of your choice

24.
Setup so far…

25.
5. Create a mapped collection
$ globus-connect-server collection create
> f77ff456-1f18-41d3-94a7-f3fd8858ea4d
> /
> "My APS Mapped Collection"
• Collections are rooted at the specified base path
• Specifying "/" as the base path sets the collection root to the local
user’s home directory
Storage gateway ID
Collection base path

26.
Common Collection configuration options
• Restrict access: local users, local groups
• Allow guest collections à enables sharing
• Restrict sharing: paths, local users, local groups
• Enable HTTPS access
• Force data channel encryption

27.
Users can now
access data via the
mapped collection
27

28.
Walkthrough
28

29.
Our setup so far…

30.
Subscriptions and Endpoint Roles
• Subscription(s) configured for your institution
• Multiple Subscription Managers per subscription
• Subscription Manager ties endpoint to subscription
– Results in a “managed” endpoint
• Assign additional roles for endpoint management
– Administrator, Manager, Monitor

31.
6. Associate endpoint with a subscription
• Subscription managers can enable subscription
features on an endpoint
• If you are not the subscription manager, just send
your endpoint ID to your subscription manager and
ask them to add it.

32.
Make your endpoint “Managed”
• Option A: Put your endpoint ID in the spreadsheet
and we will make it managed
• Option B: Run globus-connect-server endpoint
set-subscription-id
• Confirm: globus-connect-server endpoint show

33.
7. Create a guest collection
• Created by user, not endpoint administrator
• Grants access to specific Globus users without a
mapped local account
• “Guest” users have same (or more limited)
permissions as the guest collection creator
– Access logs show access by the collection creator*
• Guest collection’s root is relative to the mapped
collection’s base path
* High Assurance collections log guest user identities to enable auditing

34.
Sharing restrictions
• Guest collections may be created in any directory accessible by the
collection, by any authorized local account
• You can restrict the authorized accounts…
--sharing-user-allow
--sharing-user-deny
--posix-sharing-group-allow
--posix-sharing-group-deny
• …and sharing paths…
--sharing-restrict-paths (specify JSON PathRestrictions)
• You can also set policies for specific user/path combinations
$ globus-connect-server sharing-policy create ...

35.
8. Enable web browser upload/download
• Authorized users can
upload, download files via
a browser
• Must have permissions to
the collection
– Collection configuration
governs access
– Web server is a different
application (separate
authentication)

36.
Walkthrough
36

37.
9. Add other storage systems to the endpoint
• Update your GCS packages
• Add the appropriate storage gateway
– Non-POSIX systems require add-on connector subscription(s)
• Gateway configuration options vary by connector
– e.g., specify bucket name(s) for AWS S3
• Collection authentication options vary by connector
– e.g., provide user access key and secret key for AWS S3

38.
Using the
management console
38

39.
Things to do with the management console
• Applies to all users because of endpoint
• Monitor current transfers on your endpoints
– See what’s going on at the transfer request level
– Much better than watching individual file transfers
• Pause (and later resume) a transfer in progress
– Sends a notice to the transfer owner
• Set a pause rule for current and future transfers
– Ideal for maintenance mode
– Notifies transfer owners,
– Tasks resume when endpoint is un-paused
docs.globus.org/management-console-guide

40.
When you really need a clean slate…
• Proper clean-up—both on your system and in the
Globus service—is important!
• Execute these commands in the specified order:
o globus-connect-server node cleanup
o globus-connect-server endpoint cleanup
• Delete the GCS registration at developers.globus.org
• Don’t use the same Client ID for another endpoint!

41.
41
Migrating
GCSv4 to GCSv5

42.
Goals
• No user intervention should be required
• Recreate all host and guest endpoints
• Preserve all relevant configuration
• Preserve the UUIDs of the resource
• Minimize downtime
42

43.
Impact
• Downtime after final migration step preserving the UUID
• Active transfers cancelled on the finalize step of
migration; users are notified
• Pause rules NOT preserved; must be recreated
• Applications using a GCSv4 host endpoint (with
activation) must move to GCSv5 model (with consent)
docs.globus.org/globus-connect-server/migrating-to-
v5.4/application-migration
43

44.
Migration tools: approach
• Read v4 configuration to create migration plan
• Allow edits and changes by administrator to plan
• Apply migration plan to a vanilla install of v5
• Test v5 endpoint/collections and validate
• Finalize by assigning UUID of v4 to the new endpoint
44

45.
Migration tools: Steps
45
docs.globus.org/globus-connect-
server/migrating-to-v5.4/migration4-guide/

46.
Walkthrough
Try the first phase of migration on
your GCSv4 deployment
46

47.
Resources
• GCSv5 Installation Guide: docs.globus.org/globus-connect-
server/v5.4/
• Migration Guide home page: docs.globus.org/globus-connect-
server/migrating-to-v5.4/
• Endpoint migration: docs.globus.org/globus-connect-
server/migrating-to-v5.4/migration4-guide/
• Application migration: docs.globus.org/globus-connect-
server/migrating-to-v5.4/application-migration/
• Globus support: support@globus.org
47