We describe how developers can use Globus APIs to integrate robust data management capabilities into their research applications. We also demonstrate the new Globus portal framework that can be used in conjunction with the Globus Search service to simplify data search and discovery.

1. Working with Globus
Platform Services and Portals
Vas Vasiliadis
vas@uchicago.edu
September 9, 2022
Sponsored by

2. Globus Platform
Services
…Integrate Globus
capabilities into your
data portal, science
gateway, data commons
or other research app

3. Globus APIs
• Auth
• Groups
• Transfer
• Search
• Timer
• Flows
• GCS Manager
• Globus web app consumes the
same public APIs
• Resources named by URL
(standard REST approach)
• Request/response body is JSON
docs.globus.org/api

4. Globus Python SDK
• Python client library for the Globus REST APIs
• Largely direct mapping to REST API
• globus_sdk.TransferClient class handles
connection management, security, framing,
marshaling
globus-sdk-python.readthedocs.io
github.com/globus/globus-sdk-python
5

5. Globus Auth: Foundational IAM service
• Brokers authentication and authorization among…
– End-users
– Identity providers: enterprise, external (federated identities)
– Services: resource servers with REST APIs
– Apps: web, mobile, desktop, command line clients
– Services acting as clients to other services
• OAuth 2.0 Authorization Framework (a.k.a. OAuth2)
• OpenID Connect Core 1.0 (a.k.a. OIDC)
6

6. Fundamental Concepts
• Scopes
– APIs that client is requesting access to
– Service and resources within that service
• Consents
– Authorizes a client to access a service, within limited scope, on
the resource owner's behalf
• Multiple methods for user to grant consent depending
on the type of application
7

7. Several authentication models supported
• Application acting as user with consent
– Authorization code grant: authenticate as user identity
– Browser redirect; auth code returned automatically; tokens stored securely
– Examples: Globus CLI, Jupyter Hub secured with Globus Auth
• Application authenticating as itself
– Client credentials grant: authenticate as application
– Client ID and Secret stored securely
– Examples: Globus Django portal, custom apps
• Application able to manage tokens for offline/long lived tasks
– Request refresh tokens in addition to access tokens

8. App registration
9

9. Get app credentials at developers.globus.org

10. Getting API access
tokens
11
jupyter.demo.globus.org

11. Getting tokens: jupyter.demo.globus.org
• Sign in with Globus and verify the consents
• Start My Server and open globus-jupyter-notebooks
• Run Platform_Introduction_Native_App_Auth
• If you mess up, just stop and restart the server
• Alternatively, use notebooks on your own machine:
github.com/globus/globus-jupyter-notebooks
12

12. REST APIs
REST APIs
REST APIs
Request
Bearer a45cd...
Configurable HTTP proxy
Authenticator
User DB
Spawner
Notebook
/api/auth
/hub/
/user/[name]/
login
Browser
{"tokens":...
{"tokens":...
Globus Auth integrated with JupyterHub
The world is your
oyster API…
• Globus Transfer
• Globus Search
• Your app
• Data portal
• Analysis engine
• …

13. Working with the
Transfer API
19
jupyter.demo.globus.org

14. TransferClient low-level calls
• Thin wrapper around REST API
– post(), get(), update(), delete()
get(path, params=None, headers=None, auth=None,
response_class=None)
o path – path for the request, with or without leading slash
o params – dict to be encoded as a query string
o headers – dict of HTTP headers to add to the request
o response_class – class response object, overrides the client’s
default_response_class
o Returns: GlobusHTTPResponse object
20

15. TransferClient higher-level calls
• One method for each API resource and HTTP verb
• Largely direct mapping to REST API
endpoint_search(filter_fulltext=None,
filter_scope=None,
num_results=25,
**params)
21

16. API walkthrough: jupyter.demo.globus.org
• Run Platform_Introduction_JupyterHub_Auth
• Note automatic token extraction from JupyterHub
• Run the first few cells
• Experiment with endpoint search/filters
22

17. Making data findable
with Globus Search
23
docs.globus.org/api/search

18. Data description and discovery
• Metadata store with fine-
grained visibility controls
• Schema agnostic
à dynamic schemas
• Simple search using URL
query parameters
• Complex search using
search request document
24
Search
Index

19. Distinct access policies
may be applied to
Data and Metadata
…using permissions on
guest collections
…using permissions on
metadata elements

20. Data ingest with Globus Search
26
Search
Index
POST /index/{index_id}/ingest'
{
"ingest_type": "GMetaList",
"ingest_data": {
"gmeta": [
{
"id": "filetype",
"subject”: "https://search.api.globus.org/abc.txt",
"visible_to": ["public"],
"content": {
"metadata-schema/file#type": "file”
}
},
...
]
}
- Bulk create and update
- Task model for ingest at scale

21. Data ingest with Globus Search
27
Search
Index
POST /index/{index_id}/ingest'
{
"ingest_type": "GMetaList",
"ingest_data": {
"gmeta": [
{
"id": ”weight",
"subject": "https://search.api.globus.org/abc.txt",
"visible_to": ["urn:globus:auth:identity:46bd0f56-
e24f-11e5-a510-131bef46955c"],
"content": {
"metadata-schema/file#size": ”37.6",
"metadata-schema/file#size_human": ”<50lb”
}
},
...
]
}
Visibility limited to Globus Auth identity
- Single user
- Globus Group
- Registered client application

22. Data discovery with Globus Search
28
{
"@datatype": "GSearchResult",
"@version": "2017-09-01",
"count": 1,
"gmeta": [
{
"@datatype": "GMetaResult",
"@version": "2019-08-27",
"entries": [
{ ... }
],
"subject": "https://..."
}
],
"offset": 0,
"total": 1
}
GET /index/{index_id}/search?q=type%3Ahdf5
Search
Index
Simple query

23. Data discovery with Globus Search
29
POST /index/{index_id}/search
Search
Index
Complex query
{
"filters": [
{
"type": "range",
"field_name": ”pubdate",
"values": [
{
"from": "*",
"to": "2020-12-31"
}
]
}
],
"facets": [
{
"name": "Publication Date",
"field_name": "pubdate",
...
}
]
}
Filter
Facets
Boosts
Sort
Limit

24. Why go beyond the standard tools?
• FAIRness: Enable broad audience to access diverse
research data (type, size, metadata, …)
• …and diverse/complicated data sources (beamlines,
electron microscopes, sequencers, …)
• Add curation and cataloguing to make data findable
• Enforce (sometimes complex) access policies
• Enable researchers to customize their experience

25. The Modern Research Data
Portal Design Pattern
docs.globus.org/mrdp

26. MRDP: Key elements
Science DMZ
Fast, clean data path
Data Transfer Nodes
Purpose-built data movers
Globus Platform
Secure, reliable data
orchestration
Globus Connect
Storage system enabler
32
Globus Portal
Framework
Data discovery and access

27. An exemplar:
The ALCF Data Co-op
33
acdc.alcf.anl.gov

28. Accessing data via your portal
• Recall: Data independent of portal logic…
• …ideally, on a guest collection
• Step 1: Create a guest collection; requires authN by
human or by app using client credentials
• Step2: Grant the application Access Manager role
– Allows the application to manage permissions on the collection
– Set for application identity: appclientid@clients.auth.globus.org
• Optional: Grant roles for endpoint/task management
10GE
10GE
10GE
10GE
Border Router
WAN
Science DMZ
Switch/Router
Firewall
Enterprise
perfSONAR
perfSONAR
10GE
10GE
10GE
10GE
DTN
DTN
API DTNs
(data access governed
by portal)
DTN
DTN
perfSONAR
Filesystem
(data store)
10GE
Portal
Server
Browsing path
Query path
Portal server applications:
· web server
· search
· database
· authentication
Data Path
Data Transfer Path
Portal Query/Browse Path

29. Globus Groups simplify permissions management
• Grant group access to
collection(s)
• Restrict search visibility
using group
• Make portal client a group
administrator
• Check authenticated user’s
group membership
• Add/remove user to/from
group

30. Bootstrap a Simple
(but fully functional and extensible)
Research Data Portal
using the Django Globus
Portal Framework

31. Django Globus Portal key features
• Federated login
• Data export using Globus
• Browse datasets via Globus Search service
• Template-driven search results and landing pages
• Django-based framework with extensible templates
• Bootstrap your project using Cookiecutter
37
Source: github.com/globus/django-globus-portal-framework
Docs: django-globus-portal-framework.readthedocs.io

32. Get up and running
with the Globus
portal framework
38
Source: github.com/globus/django-globus-portal-framework
Docs: django-globus-portal-framework.readthedocs.io/en/stable/

33. Step 0: Application registration
• Set redirect URLs
• Get client ID and secret
• Consents implement
least privileges principle
39
developers.globus.org
Redirect URLs
https://tutN.globusdemo.org:8443/
https://tutN.globusdemo.org:8443/complete/globus/

34. Portal deployment
• Deploy a portal instance using cookiecutter
• Configure settings
• Run and use!
• Future: containers
• Note: For production, add robust WSGI/ASGI server

35. Adding a new search
index to your portal
41

36. Update the search index definition
• Edit ~/$PROJECT_SLUG/$PROJECT_SLUG/settings/search.py
• Index ID and name
• Fields
• Facets
• Multiple search indices may be defined

37. Update definition of search result fields
• Edit ~/$PROJECT_SLUG/$PROJECT_SLUG/fields.py
• Type definitions
• Formatting functions
• Data transformations
• Globus collection and data path

38. Modify templates for search results, details
• Fields to be displayed on search item card in list
– Metadata to help help user determine relevance
– Links to actions, e.g., transfer, download, process
– Edit ~/$PROJECT_SLUG/templates/globus-portal-
framework/v2/components/search-results.html
• Fields to be displayed on detail page
– (More) complete metadata
– Preview images (if appropriate)
– Links to actions
– Edit ~/$PROJECT_SLUG/templates/globus-portal-framework/v2/detail-
overview.html
• Uses standard Jinja2 templating language

39. Developer References
• Documentation
– API: docs.globus.org/api
– SDK: globus-sdk-python.readthedocs.io/en/stable/
• Open source repositories: github.com/globus
• Notebooks: github.com/globus/globus-jupyter-
notebooks