Ce diaporama a bien été signalé.
Le téléchargement de votre SlideShare est en cours. ×

Migrating to Globus Connect Server v5


Consultez-les par la suite

1 sur 47 Publicité

Plus De Contenu Connexe

Similaire à Migrating to Globus Connect Server v5 (20)

Plus par Globus (20)


Plus récents (20)

Migrating to Globus Connect Server v5

  1. 1. May 11, 2022 Globus Connect Server v5: Deployment and Migration
  2. 2. Outline • GCsv5 concepts • GCS v4 and v5 comparison • GCSv5 install demo and walkthrough • Migration overview • Migration tools demo and walkthrough 2
  3. 3. GCS v5 Concepts 3
  4. 4. Globus Connect Server version 5 4
  5. 5. 5
  6. 6. GCS v4 and v5 Comparison 6
  7. 7. Out with the old, in with the new • Host endpoints è Mapped collections – Need local account to access data • Shared endpoints è Guest collections – No local account needed for data access, permissions set in Globus • Use host endpoint to create shared endpoint è Use mapped collection to create (guest) collections
  8. 8. GCSv4 vs GCSv5 Feature GCSv4 GCSv5 Credentials for endpoints • GlobusID was required to create endpoint • Any identity that can be used to log into Globus can be used. • Endpoint registered as a resource server with Globus Auth, and has client id/secret. Policy and data access interfaces • Static endpoint definition supporting one collection • Configuration changes required re-running setup on each DTN • Data access via GridFTP only • Endpoint supports multiple storage systems and collections • Automated management across DTNs • Data access via GridFTP and HTTPS
  9. 9. GCSv4 vs GCSv5 Feature GCSv4 GCSv5 Server credentials • Default were issued by Globus CA • Custom certificates supported • Default from Lets Encrypt to ensure browser based downloads work • Custom certificate supported Management interface • On the DTN, configuration files and tools • GCS Manager service on the DTNs. • CLI shipped with the service
  10. 10. GCSv4 vs GCSv5 Feature GCSv4 GCSv5 Management roles (admin, activity manager, activity monitor) • Supported • Unchanged High Assurance data access • Not supported • A storage gateway can be configured for high assurance data handling, and additional security constraints are added User authentication to data access • InCommon OAuth, MyProxy or MyProxy OAuth • Any identity used to log into Globus • Custom identity provider via OIDC
  11. 11. GCSv4 vs GCSv5 Feature GCSv4 GCSv5 Mapping to local account • Data access identity independent from Globus account • User certificate DN to local account • ePPN based, GridMap files, custom callout • Policy maps Globus authentication information to local account • Expression based matching for values in the authentication context • Custom external program Custom domain name • Not applicable • Custom domain name for collections
  12. 12. GCS management conceptual architecture 12 Data Transfer Node GCS Command Line Interface GridFTP Server Globus Transfer Service GCS management requests Globus Auth Service GCS Manager authorize request using client ID/secret GCS Manager endpoint: abc.abc.data.globus.org Register a Globus Connect Server at developers.globus.org get GCS client ID, secret Define Globus Transfer resources (gateways, collections, …)
  13. 13. GCS v5 install walkthrough 13 docs.globus.org/globus-connect-server docs.globus.org/globus-connect-server/v5.4/quickstart
  14. 14. Requires a Globus subscription GCSv5 installation/configuration summary 1. Register a Globus Connect Server with Globus Auth 2. Install GCS packages on data transfer node (DTN) 3. Set up the endpoint and add node(s) 4. Create a POSIX storage gateway 5. Create a mapped collection 6. Associate endpoint with a subscription 7. Create a guest collection 8. Enable browser down/upload (HTTPS access) 9. Add other storage systems to the endpoint
  15. 15. Register GCS and get credentials • Navigate to developers.globus.org and log in • (Optional) Create a project • Add a new Globus Connect Server • Generate a client secret • Save the client ID and secret • Add other administrators to this project
  16. 16. 1. Register GCS and get credentials developers.globus.org
  17. 17. 2. Install Globus Connect Server v5 packages $ curl -LOs http://downloads.globus.org/toolkit/gt6/stable/installers/repo/deb/globus- toolkit-repo_latest_all.deb $ dpkg -i globus-toolkit-repo_latest_all.deb $ sed -i /etc/apt/sources.list.d/globus-toolkit-6-stable*.list > -e 's/^# deb /deb /' $ sed -i /etc/apt/sources.list.d/globus-connect-server-stable*.list > -e 's/^# deb /deb /' $ apt-key add /usr/share/globus-toolkit-repo/RPM-GPG-KEY-Globus $ apt-get update $ apt-get --assume-yes install globus-connect-server54
  18. 18. 3. Set up endpoint and add node $ globus-connect-server endpoint setup > "My APS Endpoint" > --organization "Argonne National Laboratory" > --client-id 4321dddd-af72-4c4b-9533-a0f4055dd321 > --owner me@anl.gov $ globus-connect-server node setup > --client-id 4321dddd-af72-4c4b-9533-a0f4055dd321 Note: endpoint setup command generates deployment-key.json Use this file when setting up additional data transfer nodes
  19. 19. Setup so far Run globus-connect-server node setup to set up additional data transfer nodes Copy deployment-key.json from original DTN
  20. 20. Storage Gateways define a set of access policies • Authentication policy for local account-holders – Which identity domain(s) are acceptable? – Is MFA required? – How often must the user authenticate to initiate data access operations? (authentication timeout) – Is this storage gateway for access to protected data? (High Assurance settings) • Identity Mapping policies – Which local account does the authenticated user map to?
  21. 21. Storage Gateways define a set of access policies • Data Access Policies – Which parts of the storage system are accessible via Globus? – Which local accounts does this policy allow (or deny)?
  22. 22. Mapping identities to local accounts • Default: Strip identity domain (everything after “@”) – e.g. userX@globusdemo.org maps to local account userX – Best for campus identities w/synchronized local accounts • Use --identity-mapping option on storage gateway – Specify expression in a JSON document – Execute a custom script docs.globus.org/globus-connect-server/v5.4/identity-mapping-guide
  23. 23. Picking identity domains • User must present identity from one of the configured domains – On access attempts, linked identities will be scanned for a match – If no identity from the required domain(s), will be asked to link one • Identity domains may include… – …any organization in Globus federated list – …your institution’s identity provider trusted by Globus – …a local OpenID Connect (OIDC) server integrated via PAM to authentication of your choice
  24. 24. Setup so far…
  25. 25. 5. Create a mapped collection $ globus-connect-server collection create > f77ff456-1f18-41d3-94a7-f3fd8858ea4d > / > "My APS Mapped Collection" • Collections are rooted at the specified base path • Specifying "/" as the base path sets the collection root to the local user’s home directory Storage gateway ID Collection base path
  26. 26. Common Collection configuration options • Restrict access: local users, local groups • Allow guest collections à enables sharing • Restrict sharing: paths, local users, local groups • Enable HTTPS access • Force data channel encryption
  27. 27. Users can now access data via the mapped collection 27
  28. 28. Walkthrough 28
  29. 29. Our setup so far…
  30. 30. Subscriptions and Endpoint Roles • Subscription(s) configured for your institution • Multiple Subscription Managers per subscription • Subscription Manager ties endpoint to subscription – Results in a “managed” endpoint • Assign additional roles for endpoint management – Administrator, Manager, Monitor
  31. 31. 6. Associate endpoint with a subscription • Subscription managers can enable subscription features on an endpoint • If you are not the subscription manager, just send your endpoint ID to your subscription manager and ask them to add it.
  32. 32. Make your endpoint “Managed” • Option A: Put your endpoint ID in the spreadsheet and we will make it managed • Option B: Run globus-connect-server endpoint set-subscription-id • Confirm: globus-connect-server endpoint show
  33. 33. 7. Create a guest collection • Created by user, not endpoint administrator • Grants access to specific Globus users without a mapped local account • “Guest” users have same (or more limited) permissions as the guest collection creator – Access logs show access by the collection creator* • Guest collection’s root is relative to the mapped collection’s base path * High Assurance collections log guest user identities to enable auditing
  34. 34. Sharing restrictions • Guest collections may be created in any directory accessible by the collection, by any authorized local account • You can restrict the authorized accounts… --sharing-user-allow --sharing-user-deny --posix-sharing-group-allow --posix-sharing-group-deny • …and sharing paths… --sharing-restrict-paths (specify JSON PathRestrictions) • You can also set policies for specific user/path combinations $ globus-connect-server sharing-policy create ...
  35. 35. 8. Enable web browser upload/download • Authorized users can upload, download files via a browser • Must have permissions to the collection – Collection configuration governs access – Web server is a different application (separate authentication)
  36. 36. Walkthrough 36
  37. 37. 9. Add other storage systems to the endpoint • Update your GCS packages • Add the appropriate storage gateway – Non-POSIX systems require add-on connector subscription(s) • Gateway configuration options vary by connector – e.g., specify bucket name(s) for AWS S3 • Collection authentication options vary by connector – e.g., provide user access key and secret key for AWS S3
  38. 38. Using the management console 38
  39. 39. Things to do with the management console • Applies to all users because of endpoint • Monitor current transfers on your endpoints – See what’s going on at the transfer request level – Much better than watching individual file transfers • Pause (and later resume) a transfer in progress – Sends a notice to the transfer owner • Set a pause rule for current and future transfers – Ideal for maintenance mode – Notifies transfer owners, – Tasks resume when endpoint is un-paused docs.globus.org/management-console-guide
  40. 40. When you really need a clean slate… • Proper clean-up—both on your system and in the Globus service—is important! • Execute these commands in the specified order: o globus-connect-server node cleanup o globus-connect-server endpoint cleanup • Delete the GCS registration at developers.globus.org • Don’t use the same Client ID for another endpoint!
  41. 41. 41 Migrating GCSv4 to GCSv5
  42. 42. Goals • No user intervention should be required • Recreate all host and guest endpoints • Preserve all relevant configuration • Preserve the UUIDs of the resource • Minimize downtime 42
  43. 43. Impact • Downtime after final migration step preserving the UUID • Active transfers cancelled on the finalize step of migration; users are notified • Pause rules NOT preserved; must be recreated • Applications using a GCSv4 host endpoint (with activation) must move to GCSv5 model (with consent) docs.globus.org/globus-connect-server/migrating-to- v5.4/application-migration 43
  44. 44. Migration tools: approach • Read v4 configuration to create migration plan • Allow edits and changes by administrator to plan • Apply migration plan to a vanilla install of v5 • Test v5 endpoint/collections and validate • Finalize by assigning UUID of v4 to the new endpoint 44
  45. 45. Migration tools: Steps 45 docs.globus.org/globus-connect- server/migrating-to-v5.4/migration4-guide/
  46. 46. Walkthrough Try the first phase of migration on your GCSv4 deployment 46
  47. 47. Resources • GCSv5 Installation Guide: docs.globus.org/globus-connect- server/v5.4/ • Migration Guide home page: docs.globus.org/globus-connect- server/migrating-to-v5.4/ • Endpoint migration: docs.globus.org/globus-connect- server/migrating-to-v5.4/migration4-guide/ • Application migration: docs.globus.org/globus-connect- server/migrating-to-v5.4/application-migration/ • Globus support: support@globus.org 47