Amazon Simple Work Flow Engine (SWF): How Beamr uses SWF for video optimization in the cloud - Pop-up Loft Tel Aviv

Amazon Simple Work Flow Engine (SWF):
How Beamr uses SWF for video optimization in the cloud
Dan Julius, VP R&D, Beamr

There are few times when something new comes along
that makes me think the following:
- Really? Nahh...
- Hmm.. Maybe..
- OMG!! How the f*** does it do that?
- GIVE IT TO ME NOW!!
“
Lens In The Face

- Optimized ±5B Photos in 2015
- Reduces JPEG file size from 20%-80%
- Community of tens of thousands photographers
- Free trial available at jpegmini.com
- Enterprise and consumer products

SOURCE
1080p @ 3.5 mbps
BEAMR VIDEO
1080p @ 2.1 mbps
*Images are courtesy of Universal Studios
REDUCED BY 40%

BEAMR KEY FEATURES
5©2016 Beamr Imaging Ltd. All Rights Reserved
Bitrate reduction Always safe Fully automatic Standard compliant
Significant reduction.
in bitrate or file size
Retains original
image or video
quality
No additional QC
steps required
No modification needed
for existing players

Agenda
6©2016 Beamr Imaging Ltd. All Rights Reserved 55
1. Why workflows?
2. SWF Concepts
3. Tips and Gotchas
4. Q&A
For Python → Ask me later
Java/Ruby → Checkout Flow Framework
No Code

A Simple Workflow

But…
Reality isn’t always simple

The Players
©2016 Beamr Imaging Ltd. All Rights Reserved
Activity Workers
DECIDERS
SWF
11

Activity Workers
Activity Workers
Responsible for doing the “work”
Typically a long-running process
Poll queue task-list → Do work
While True:
Poll an SWF queue task-list
Process the task
Return result to SWF
12

Deciders
DECIDERS
Responsible for making workflow decisions
Typically a long-running process
While True:
Poll task-list for a decision
Analyze execution history
Make a decision on next step(s)
13

SWF Responsibilities
Coordinate system components
Schedule “decision tasks” (a queue?)
Schedule “activity tasks” (a queue?)
Maintain workflow state
Catch errors and timeouts
Provide an API to track workflow-execution progress
Does No work. Makes No decisions.
14

Example of Two-Task Workflow Life Cycle
15©2016 Beamr Imaging Ltd. All Rights Reserved
SWF Decider Worker A Worker B
start execution
new exc. what to do?
Schedule A
starting...
A task for you...
Completed. Result is...
New history. what now?
Schedule B
A task for you...
Completed. Result is...
New history. What now?
Close. Result is...
Are we done?
no
Are we done?
Yes
START ENDTask A Task B

Tips for Activity Workers
Do only one thing
Be Stateless
Catch all exceptions and return failure
Send heartbeats (and check responses)
While True:
Poll an SWF task-list
Process the task
16

Tips for Deciders
Focus on Decision Making. Avoid doing any
work.
Be Stateless
Decide based on entire* execution history
While True:
Poll SWF for an execution
Analyze execution history
Make a decision on next step(s)
Parallelize tasks when appropriate by returning
multiple decisions.
Catch all exceptions and always return a valid
decision
Expect the unexpected... (e.g. a decision may be rejected)
17

The Execution History
[ {
"eventId": 11,
"eventTimestamp": 1326671603.102,
"eventType": "WorkflowExecutionTimedOut",
...
},
...
{
"activityTaskScheduledEventAttributes": {
"activityId": "verification-27",
"activityType": {
"name": "activityVerify",
"version": "1.0"
},
…..
"input": "5634-0056-4367-0923,12/12,437",
...
},
"eventId": 8,
...
"eventType": "ActivityTaskScheduled"
},
...
{
...
"eventId": 2,
"eventType": "DecisionTaskScheduled"
}
]
The workflow-execution “state”
It’s just json
Every event is noted
Decisions are noted
Scheduled tasks are noted
Start and Finish are noted
…
How to make a decision?
- Read the execution history
- Apply some “logic”
- Return one or more decisions
18

CLI Demo
A distributed “Image Processing” workflow
Prerequisites:
- AWS Account, IAM Credentials
- AWSCLI
- bash / jq / ImageMagic / JPEGmini
20

CLI Demo - Registration
Register Domain / Workflow / Activity Types
We use “code” to configure SWF. No CloudFormation available.
LOFT_Domain
demo (Domain)
Process
Media
Ingest
CleanupOptimize
Thumbnail
21

CLI Demo - Process Media Workflow
START ENDIngest Cleanup
Optimize
Thumbnail
Download from Web
to EFS
Process files on EFS
Upload results to S3
Remove files form
EFS
22

Some Advanced Topics
Child Workflows
- Simplify large workflows
- Reuse child-workflows
Signals
- Sent from somewhere (external / activity)
- SWF will schedule a Decision-Task
24
[
...
{
"eventId": 153,
"eventType": "WorkflowExecutionSignaled",
"workflowExecutionSignaledEventAttributes"{
...
}
...
},
...
]

High Availability
REST API
At least two deciders, and two
workers of each type, across
multiple zones
25

The Execution History
Beware of the “LastEvent” samples
Note the previousStartedEventId
When then History gets long
- Note the nextPageToken.
Pages are small (100 items).
Beware of rate limits.
- Cache history pages
- Use ChildWorkflows or ContinueAsNewWorkflow
- Use short names in input and results fields
[ {
"eventId": 11,
"eventType": "WorkflowExecutionTimedOut",
...
},
...
{
"activityTaskScheduledEventAttributes": {
"activityId": "verification-27",
"activityType": {
"name": "activityVerify",
"version": "1.0"
},
…..
"input": "5634-0056-4367-0923,12/12,437",
...
},
"eventId": 8,
...
"eventType": "ActivityTaskScheduled"
},
...
{
...
"eventId": 2,
"eventType": "DecisionTaskScheduled"
}
]
26

Monitoring and Scaling
Processes to Monitoring
- Monitor each decider process
- Monitor each worker process
Metrics to Monitor
- Activity task list sizes
- Decider task list sizes
- DecisionTaskScheduleToStartTime
- ActivityTaskScheduleToStartTime
Scale Workers with Spot instances
- Use short-timeouts
- Heartbeats
27

Things Will Break...
API calls may fail, therefore backoff and retry
- Rate limiting
- Network errors
Expect the unexpected:
- Decisions or Responses may be rejected
- Closing a workflow execution could fail due to pending signals
- An Activity Worker response may fail due to cancelled execution
- Events may be aggregated, or delivered out of order
- Task lists are only mostly FIFO
28

Dan Julius
VP R&D, Beamr
dan@beamr.com

Workflows vs. Messages
A complex workflow requires more than just passing
messages from task to task…
Workflows have “state”
Tasks might need to be synchronized
Tasks can fail or timeout
32

How SWF Works (maybe)
SWF
Worker A Worker B
Decider
task-list
task-list
task-list
Execution
history
33

Poll
The Manual Worker
The worker process:
While True:
Poll an SWF task-list
Send email to manual-worker
The employee:
When an email is received,
Process task, and submit completion form
Web
Server
Email
Respond to SWF
Submit Form
SWFWorker
Process
Manual
Labour
34

More Advanced Topics
Markers
- Added by deciders
- Simplify understanding history
- “Milestones”
Timers
- Set by deciders
- SWF will schedule a Decision-Task when timer fires
Priorities
- SWF will can re-order tasks based on Priority
[
...
{
"eventId": 153,
"eventType": "TimerFired",
"TimerFiredEventAttributes"{
...
}
...
},
...
]
37
[
...
{
"eventId": 153,
"eventType": "MarkerRecorded",
"markerRecordedEventAttributes"{
...
}
...
},
...
]

Some Concepts
Worklow Activity
Task
State
Logical Flow
Activity Workers Asynchronous processing
Distributed processing
Scalability
Domain
Activity Type Activity Version
Activity Timeouts
Results
Decider
Input Data
Events
Workflow execution
Execution History
Polling
workflowId + runId
Workflow Starters
decisions
Activity Task
Lambda Task Decision Task
Task Lists
38

Amazon Simple Work Flow Engine (SWF): How Beamr uses SWF for video optimization in the cloud - Pop-up Loft Tel Aviv

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (19)

En vedette

En vedette (20)

Similaire à Amazon Simple Work Flow Engine (SWF): How Beamr uses SWF for video optimization in the cloud - Pop-up Loft Tel Aviv

Similaire à Amazon Simple Work Flow Engine (SWF): How Beamr uses SWF for video optimization in the cloud - Pop-up Loft Tel Aviv (20)

Plus de Amazon Web Services

Plus de Amazon Web Services (20)

Dernier

Dernier (20)

Amazon Simple Work Flow Engine (SWF): How Beamr uses SWF for video optimization in the cloud - Pop-up Loft Tel Aviv