understand how Cloud Workflows resolves challenges in connecting services, HTTP based service orchestration and automation. We are going to dive deep how serverless HTTP service automation works to automate step engines. Based on practical examples we will demonstrate the newest features that lets you automate the cloud and integration with any Google Cloud product without worrying about authentication

1. Cloud Workflows: What's new in serverless
orchestration and automation
October 2021 - GDG DevFest 2021 - Doha
Márton Kodok / @martonkodok
Google Developer Expert at REEA.net

2. ● Among the Top 3 romanians on Stackoverflow 195K reputation
● Google Developer Expert on Cloud technologies
● Crafting Web/Mobile backends at REEA.net
● BigQuery + Redis database engine expert
Slideshare: martonkodok
Articles: martonkodok.medium.com
Twitter: @martonkodok
StackOverflow: pentium10
GitHub: pentium10
What's new in serverless orchestration and automation @martonkodok
About me

3. 1. Challenges in connecting services
2. What is Cloud Workflows?
3. Practical use cases
4. What’s new in Workflows?
5. Advanced examples
6. Conclusions
Agenda
What's new in serverless orchestration and automation @martonkodok

4. ● Connectivity - should be easy, but in reality you need to figure out
● Parse the results
● Decisions and conditional step executions
● No built-in error handling, logging
● Struggling with polling, retries, exponential backoff
● Scaling up and down to zero
● Authentication
● Missed capability of capturing callbacks
Challenges in connecting services
What's new in serverless orchestration and automation @martonkodok

5. MeetCloudWorkflows
What's new in serverless orchestration and automation @martonkodok

6. Workflows in Google Cloud portfolio
Introduction
Orchestrate any
1. Google Cloud API
2. SaaS API
3. External or private APIs.
Serverless
Compute
External
API’s
Google
API’s
etc...
Workflows - orchestrate & integrate
SaaS
API’s
Private
API’s
Other
Clouds

7. Step-Automation-as-a-Service - Serverless HTTP service automation
Declarative workflow language (YAML, JSON)
Decent pricing (internal: $1/100K steps, external: $2.5/100K) *Dec 2020
Built-in decision and conditional executions expression formulas, operation on var
Subworkflows similar to routine in a programming language with input/return var
Support for external API calls out of the box support outside of Google Cloud
Integrates with any Google Cloud product without worrying about authentication
Callbacks feature opens solutions for async, human-in-the-loop orchestration tasks
What is GCP Workflows?
What's new in serverless orchestration and automation @martonkodok

8. OAuth, OIDC, Secret Manager integration
Enterprise
Security
Keep your workflows secure
X
Authenticated
Invocations
Authenticated
calls to Google Cloud
services
Integration with
Secret Manager
Encryption at rest and
in transit
External API 1
External API 2 External API

9. Anatomy of a Cloud Workflow!
What's new in serverless orchestration and automation @martonkodok

10. hello.yaml
Developing for Cloud Workflow!
What's new in serverless orchestration and automation @martonkodok
workflowRevisionId: 000001-de1
argument: 'null'
startTime: '2020-10-16T20:19:34.448323739Z'
name:
projects/985596417983/locations/us-central1/workflows/hello/executions/95c99744-c73c-
4065-a696-940bc7658c33
result: '"Hello, Cloud Workflows!"'
endTime: '2020-10-16T20:19:34.845536315Z'
state: SUCCEEDED
Deploying gcloud workflows deploy hello --source=hello.yaml
Executing gcloud workflows execute hello --data={“var”:”value”}
Describing gcloud workflows executions describe-last

11. Sample Workflows
What's new in serverless orchestration and automation @martonkodok

12. IT management automation
Combine automation with
scheduler
Wait for service checks
Orchestrate work across
Compute Engine, PubSub,
Stackdriver and other Google
Cloud Products
9 AM trigger
Start a Compute
Engine VM
Log the event
App
Started?
No
Wait 60 seconds
Notify the team

13. E-commerce invoice generation with Workflows
Steps orchestration
Reliable execution, with error
handling and retries
Orchestration microservices
or other API’s
Create an invoice
Generate PDF
Send PDF via email
Receive Order
Cloud
Run
App
Engine
Cloud
Run

14. Code Examples
What's new in serverless orchestration and automation @martonkodok

15. 1. HTTP calls (GET or POST)
2. Sequence two steps
3. Working with subworkflows
Workflow Basics
What's new in serverless orchestration and automation @martonkodok

16. http_post.yaml
Making an external HTTP POST request
What's new in serverless orchestration and automation @martonkodok
{
"archived":false,
"created_at":"2020-10-16T17:40:17+0000",
"id":"bit.ly/35452TM",
"link":"https://bit.ly/35452TM",
"long_url":"<truncated>",
}

17. wikipedia.yaml
Sequence two steps to get data from Wikipedia
What's new in serverless orchestration and automation @martonkodok

18. subworkflow.yaml
Subworkflow
What's new in serverless orchestration and automation @martonkodok
call
call
def

19. What’s new in Workflows
What's new in serverless orchestration and automation @martonkodok

20. 1. Functions: string, JSON, map, iterating syntax
2. Cloud logging and Call logging
3. Connectors: built-in authentication, retries, long-operation
4. Callbacks: enables async-executions, or human in the loop
What’s new in Workflows
What's new in serverless orchestration and automation @martonkodok

21. String Functions
What's new in serverless orchestration and automation @martonkodok

22. Workflow Connectors
What's new in serverless orchestration and automation @martonkodok

23. They are like client libraries for other services handling auth, retries and waiting.
Connect to other Google Cloud APIs, and to integrate other GCP products.
Connectors
What's new in serverless orchestration and automation @martonkodok

24. Before: Calling Secret Manager traditionally using HTTP Get calls.
Connectors: Replacing GET and POST calls
What's new in serverless orchestration and automation @martonkodok
After: using Connectors to get Secret Manager value

25. 1. Built in Authentication (calling service account needs permissions)
2. Retries (max_retries: 5, max_delay: 60, ability to define custom retry policies)
3. Timeout (default timeout: 30 minutes, can be increased to 1 year)
4. Long-running operations (blocks the workflow execution until the operation completes)
5. Polling (exponential backoff of 1.25 up to 60 seconds, each step is billable)
Connectors details
What's new in serverless orchestration and automation @martonkodok

26. Long running operation type of Workflow Connectors
What's new in serverless orchestration and automation @martonkodok
Product Operation Type Product Operation Type
BigQuery Job Dataflow Job
Cloud Build Operation Firestore Operation
Cloud Functions Operation Document AI Operation
Cloud SQL Operation Cloud Spanner Operation
Compute Engine Operation Workflows Operation
Execution

27. Workflow Callbacks
What's new in serverless orchestration and automation @martonkodok

28. They are webhook endpoints where you signal that external event has occurred.
Callbacks allow workflows executions to wait until the callback endpoint gets a call.
Callbacks
What's new in serverless orchestration and automation @martonkodok

29. 1. Modern way to avoid sys.sleep, wait, tracking tables, polling
2. Offers a webhook endpoint (secured with authentication)
3. Timeout (default timeout: 12 hours, can be increased to 1 year)
4. Event based (pauses the workflow execution until the call is received on endpoint)
5. Full callback details (callback_request holds headers,query,body,raw,json)
Callback details
What's new in serverless orchestration and automation @martonkodok

30. Orchestrate and automate the Cloud
What's new in serverless orchestration and automation @martonkodok

31. Firestore Backups the easy way with Cloud Workflows
- Cloud Scheduler
- Cloud Workflows
- Cloud Firestore
- Cloud Storage
Full guide article on:
martonkodok.medium.com
Cloud
Workflows
Cloud
Storage
Cloud
Scheduler
Firebase
Cloud
Firestore
backup
Roles and
Permissions
Cloud IAM
authorize

32. Using Workflows to load Cloud Storage files into BigQuery
- Cloud Workflows
- Cloud Storage
- BigQuery
Full guide article on:
martonkodok.medium.com
Cloud
Storage
BigQuery
Cloud
Workflows
Authenticated
Invocations
foreach

33. Run shell commands and orchestrate Compute Engine VMs
- Cloud Workflows
- Cloud Build
- Compute Engine
- Identity-Aware Proxy
Full guide article on:
martonkodok.medium.com
Cloud Workflow Steps
Cloud
Workflows
Execution
Authorize
Cloud Build
Roles and
Permissions
Cloud IAM
Shell
Service Account
Start VM
Compute
Engine
Submit Stop VM
Compute
Engine
Firewall
Compute Engine
Cloud
IAP
Tunnel
wait wait

34. Conclusions
What's new in serverless orchestration and automation @martonkodok

35. Reliable workflow execution - execute workflows for enterprise business apps
Low latency of execution - no cold starts
Built-in error handling out of the box error handling with configurable retry policies
Passing variable built-in JSON parsing and expression-based variable manipulation
Rich runtime iterating through an array, embedded steps for readability
Secret Manager integration out of the box
Callbacks feature opens solutions for async, human-in-the-loop orchestration tasks
Benefits of Cloud Workflows
What's new in serverless orchestration and automation @martonkodok

36. Easy to build/operate
Scales out
Does not lose state
Handles errors/timeouts
Out-of-the-box support of Cloud APIs
Auditable
Enables async-workflows with Callbacks
Developer friendly
What's new in serverless orchestration and automation @martonkodok

37. The possibilities are endless
Marketing Retail IndustrialandIoT Developer
Event driven marketing
workflow execution
Relay conversions to
customer profiles in external
services
Workflow based emails,
discounts, promotions
Order management
Inventory chain operations
Data gathering and
processing
Synchronize systems
Generate state machines
Verify equipment lifecycle
Workflow based
maintenance needs
Digitalization of internal
policies
Automate the Cloud
Shell-script replacement
Orchestrate devops
workflows
@martonkodok

38. Thank you. Q&A.
Slides available on:
slideshare.net/martonkodok
Reea.net - Integrated web solutions driven by creativity
to deliver projects.