Contenu connexe Similaire à Building Business Workflows with AWS Step Functions (20) Plus de Amazon Web Services (20) Building Business Workflows with AWS Step Functions1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
Building Business Workflows with AWS Step Functions
Chris Munns
Principal Developer Advocate
AWS Serverless
2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
About me:
Chris Munns - munns@amazon.com, @chrismunns
– Principal Developer Advocate - Serverless
– New Yorker
– Previously:
• AWS Business Development Manager – DevOps, July ’15 - Feb ‘17
• AWS Solutions Architect Nov, 2011- Dec 2014
• Formerly on operations teams @Etsy and @Meetup
• Little time at a hedge fund, Xerox and a few other startups
– Rochester Institute of Technology: Applied Networking and Systems
Administration ’05
– Internet infrastructure geek
3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
https://secure.flickr.com/photos/mgifford/4525333972
Why are we
here today?
4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
DBMSQueue
Modern application
5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Modern application
6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Turning functions into apps
“I want to select functions based on data”
“I want to retry functions”
“I want try/catch/finally”
“I have code that runs for hours”
“I want to run functions in parallel”
“I want to sequence functions”
7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Example Use Case
Suppose I want to
book my vacation to
the Grand Canyon,
…but unwind my reservations if I can’t rent a car.
8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Coordination by method call
9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Coordination by function chaining
10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Coordination by database
11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Coordination by queues
12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
AWS
Step Functions
13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Manages the logic of
your application. This
removes extra code that
may be repeated in your
microservices and
functions.
Easy to connect and
coordinate
distributed components
and microservices to
quickly create apps
AWS Step Functions
manages state,
checkpoints and restarts
to make sure your
application executes in
order and as expected.
Build and update
Apps Quickly
Write Less Code Improve
Resiliency
Benefits of AWS Step Functions
14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
New: Richer workflows
Simplify building workloads such as order
processing, report generation, and data analysis
Write and maintain less code; add services in
minutes
More service integrations:
AWS Step
Functions
Amazon
Simple
Notification
Service
Amazon
Simple Queue
Service
Amazon
SageMaker
AWS Glue AWS Batch Amazon Elastic
Container
Service
AWS Fargate
NEW!
15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Use serverless workflows to build and update apps quickly
Task
Choice
Fail
Parallel
Mountains
People
Snow
16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Define workflows in JSON
{
"Comment": "Image Processing workflow",
"StartAt": "ExtractImageMetadata",
"States": {
"ExtractImageMetadata": {
"Type": "Task",
"Resource": "arn:aws:lambda:::function:photo-backendExtractImageMetadata-
...",
"InputPath": "$",
"ResultPath": "$.extractedMetadata",
"Next": "ImageTypeCheck",
"Catch": [ {
"ErrorEquals": [ "ImageIdentifyError"],
"Next": "NotSupportedImageType"
} ],
"Retry": [ {
"ErrorEquals": [ "States.ALL"],
"IntervalSeconds": 1,
"MaxAttempts": 2,
"BackoffRate": 1.5 }, ...
17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Monitor from the console, CLI or API
18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Run each step in sequence
Vendor A
Vendor B
Vendor C
19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
…and unwind if my plans fail at any point
Vendor A
Vendor B
Vendor C
20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
"HelloWorld": {
"Type": "Task",
…
"Retry": [
{
"ErrorEquals": [”States.TaskFailed"],
"IntervalSeconds": 1,
"MaxAttempts": 2,
"BackoffRate": 2.0
},
… ]
Retry failures
21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Catch Exceptions
"HelloWorld": {
"Type": "Task",
…
"Catch": [
{
"ErrorEquals": [”AllYourBaseAreBelongToUs"],
"Next": "CustomErrorFallback"
},
{
"ErrorEquals": ["States.TaskFailed"],
"Next": "ReservedTypeFallback"
… ]
{
22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Coordination must-haves
• Scales out
• Doesn’t lose state
• Deals with errors/timeouts
• Easy to build & operate
• Auditable
23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Seven state types
Task A single unit of work
Choice Adds branching logic
Parallel Fork and join the data across tasks
Wait Delay for a specified time
Fail Stops an execution and marks it as a failure
Succeed Stops an execution successfully
Pass Passes its input to its output
24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Task states for any compute
Long poll
Traditional server
Request
Worker requests
tasks from Step
Functions
Step Functions
invokes the Task
25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Customer use-cases
• Build modern applications
• Data processing
• Automate tasks
• Break up monoliths
• Automate Machine Learning (ML) Jobs
26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
“I want to sequence functions”
AWS Step Functions, we can easily change and iterate on
the application workflow of our food delivery service in
order to optimize operations and continually improve
delivery times. AWS Step Functions lets us dynamically
scale the steps in our food delivery algorithm so we can
manage spikes in customer orders and meet demand.
Mathias Nitzsche, CTO, foodpanda
“
27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Serverless human approval tasks
28. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
• Challenge
• Transcode ~350 news video clips per day into 14
formats each– as quickly as possible
•
Results
• Able to process video segments in parallel
• Reduced processing time from ~20 min to ~2 minutes
• The bigger the source video, the more segments, the
bigger the savings
Thomson Reuters
Serverless split video transcoding Industry: Media & Entertainment
Solution: Big Data / Machine Learning
Architecture: Serverless
30. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
How they did it
• Derive keyframe locations within the source
• Split the source at the keyframes
• Process segments (typically 0.5 sec per) in
parallel
• Concatenate segments
• Elapsed time: ~20 min down to ~2 minutes
Workflow
31. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
• Challenge
• Ensure that their customers won’t run out of disk space
• Avoid having to over-provision storage allocated to RDS
•
Results
• Reduced storage costs up to 60% per database
• Better customer experience
Outsystems
Saved 60% by creating a database resizing microservice Industry: Internet
Solution: IT Automation
Architecture: Microservices
32. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
How they did it
• Developed an automatic resizing process which they
exposed as a microservice behind Amazon API Gateway
• Takes advantage of the Native Backup and Restore
to/from S3 feature available on RDS SQLServer service
• Use Step Functions as the orchestration engine with all
tasks being Lambda functions
• Reduce default database size from 1024 GB to 400GB –
save 60% right away
• Adjust databases on demand as customer needs grow
Workflow
33. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
• Challenge
• Update nutrition information on over 100 brands to
comply with updated FDA regulations
•
Results
• Processing time shortened from 36 hours to 10 seconds
• Better productivity: reducing the distance between the
business and development teams identified new
enhancements and improvements that could be quickly
implemented
Coca-Cola
Optimizing nutrition syndication processes
Case Study:
https://aws.amazon.com/solutions/case-studies/bigfinite/
Industry: Retail
Solution: Marketing Automation
Architecture: Serverless
34. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Solution architecture
35. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Workflow
36. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Workflow
37. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pricing
• Free Tier
• 4,000 state transitions
• per month
• State Transition, Per 1,000
• $0.025 in most regions
• $0.0271 in Asia Pacific (Seoul)
• $0.0279 in US West (N. California)
• $0.0285 in Asia Pacific (Mumbai)
• $0.0300 in AWS GovCloud (US)
Step Functions counts a state transition each time a step of your workflow is executed.
You are charged for the total number of state transitions across all your state machines,
including retries.
38. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
DEMO!
39. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
aws.amazon.com/serverless
40. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Chris Munns
munns@amazon.com
@chrismunnshttps://www.flickr.com/photos/theredproject/3302110152/
41. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved
?
https://secure.flickr.com/photos/dullhunk/202872717/