Presentation about Mistral (Workflow Service) from SF Bay Area OpenStack meetup held at Yahoo! on Feb 6th.

1. Mistral
Workflow as a Service
Renat Akhmerov
Senior Software Engineer
© MIRANTIS 2014
!1

2. Agenda
•
•
•
•
•
•
•
•
Project Mission
Project users
Use Cases
Main Features
DSL, Data Flow concept, Architecture
Current status
Live Demo
Roadmap and Contribution
© MIRANTIS 2014
!2

3. Workflows all Around
Find a job
Stanford,
Harvard,
etc.
Make
money
© MIRANTIS 2014
!3

4. Project Mission
Provide a mechanism to define and execute
tasks and workflows without writing the code
© MIRANTIS 2014
!4

5. Who are Mistral Users?
•
OpenStack developers: Nova, Heat, Murano, Solum etc.
•
Administrators: capabilities like Cloud Cron to
schedule periodic helper cloud tasks (local VM
processes, backups etc.)
•
Enterprise application developers: automate business
processes consisting of multiple distributed
processing steps
© MIRANTIS 2014
!5

6. Use Cases
•
Cloud Cron: a system administrator can schedule cloud
tasks for periodical execution
•
Cloud Environment Deployment: tools like Heat can
represent deployment steps as a workflow and use Mistral
to run it
•
Live Migration: on CPU close to 100% run specific
migration workflow
•
•
Long-Running Business Process
Big Data Analysis & Reporting
© MIRANTIS 2014
!6

7. Cloud Cron
•
•
Start a workflow on event (timer)
•
Wide range of tasks: local
Single point of configuration and
control for helper periodic jobs
processes, OpenStack services
(Nova, Cinder, Swift etc.), email,
SMS, custom actions
•
•
Monitoring & Management
HA & Scalability
© MIRANTIS 2014
!7

8. Main Features
•
•
•
RESTful Web service
•
•
•
•
•
Tasks associated with pluggable actions
Based on Domain Specific Language (DSL)
Runs workflows consisting of synchronous and asynchronous
tasks
Built-in actions: HTTP request, AMQP message
Runs workflows on event (timer, ceilometer alarm)
Provides history and state of currently running workflows
HA and Scalability for workflow executions
© MIRANTIS 2014
!8

9. Mistral Workflow
•
•
•
•
•
•
•
Graph of tasks
T7
Task Actions (HTTP, AMQP,
SSH, email, etc.)
T9
T6
T8
Transitions
depends
Dependencies
T3
Data Flow
T2
T1
T4
Conditions
start
Events (triggers)
© MIRANTIS 2014
T5
!9

10. Mistral DSL Capabilities
•
•
Task dependencies (task3 depends on task2 and task1)
•
Conditional expressions in dependencies and direct
transitions (based on Data Flow)
•
•
•
Task action declarations (signals over HTTP, AMQP)
Direct transitions (on task1 success execute task2, on
error task3)
Standard actions (std:repeat, std:email)
Cron events (run workflow hourly)
© MIRANTIS 2014
!10

11. DSL Snippet
Workflow:
tasks:
runJob:
requires:
createVM: $.vm_ip != null
action: DemoApp:runJob
on-success: deleteVM
on-error: sendJobError
!
!
deleteVM:
action: Nova:deleteVM
on-finish: end
sendJobError:
action: std:email
parameters:
address: admin@mycompany.com
subject: Workflow error
body: Error occurred while running a VM job in execution {$.execution.id}
on-finish: deleteVM
© MIRANTIS 2014
!11

12. Data Flow (concept)
“image_id”: “123”,
“person”: {
“name”: “John”,
“email”: “me@me.com”
},
“vm_ip”: “10.0.0.3”
IsMale
input:
person.name
RunVM
“image_id”: “123”,
“person”: {
“name”: “John”,
“email”: “me@me.com”
},
“vm_ip”: “10.0.0.3”,
“male”: “true”
GoOn
input:
“image_id”: “123”,
image_id
“person”: {
“name”: “John”,
“email”: “me@me.com”
}
SendMsg
“image_id”: “123”,
“person”: {
“name”: “John”,
“email”: “me@me.com”
},
“vm_ip”: “10.0.0.3”
- task input
input:
person.email
“image_id”: “123”,
“person”: {
“name”: “John”,
“email”: “me@me.com”
},
“vm_ip”: “10.0.0.3”,
“sent_msg”: “true”
“image_id”: “123”,
“person”: {
“name”: “John”,
“email”: “me@me.com”
},
“vm_ip”: “10.0.0.3”,
“male”: “true”,
“sent_msg”: “true”
- task result (added into context)
© MIRANTIS 2014
!12

13. Architecture
Workflow Queue
Engine
Scheduler
API Server
workflows
executions
tasks
events
Task Queue
Task
Executor
!
...
Task
Executor
© MIRANTIS 2014
!13

14. Current Status
•
•
•
•
Basic task dependency based model works
Pilot is currently being worked on
Launchpad home page
o
Code in stackforge code repos
o
o
•
•
https://launchpad.net/mistral
o
https://github.com/stackforge/mistral
https://github.com/stackforge/mistral-extra
https://github.com/stackforge/python-mistralclient
Description on OpenStack Wiki
o
wiki.openstack.org/wiki/Mistral
Contributors: Mirantis, StackStorm
© MIRANTIS 2014
!14

15. Live Demo
Demo
App
1. Upload workbook
4)
sk
(ta
4
Put
Service on
hold
w
REST endpoint
rk
flo
3
Mistral
n
Ru
5
Backup
User Data
Backup
Service
Data
2.
HTTP
wo
4
Execute
Backup
© MIRANTIS 2014
!15

16. Mistral Icehouse Roadmap
•
•
End of February, 2014. Pilot.
o Basic DSL capabilities (dependencies, direct transitions, conditionals,
cron events)
o Keystone Integration
o REST API
o Python Client
April, 2014. Release 0.1
o Overall stability
o Refined DSL
o Mistral CLI
o Horizon Dashboard (Workbook CRUD operations, workflow execution
state, history)
© MIRANTIS 2014
!16

17. Contribution Opportunities
•
•
•
•
•
•
•
•
•
Blueprints for additional capabilities
Seed initial dev team
Web UI (history, workflow lifecycle management)
Ceilometer integration
Integration with deployment tools (Puppet, Chef)
Advanced capabilities of Python Client (building workflow
step-by-step locally, callbacks etc.)
Workflow rollback
CLI for Mistral API
CLI for Mistral action handlers
© MIRANTIS 2014
!17

18. How to Become a Contributor?
Install
Python,
git etc.
Learn
Python
Register at
Launchpad
Make
changes
clone
mistral
repos
Sign
OpenStack
CLA
tor
life
-cy
cle
Buy a
computer
Co
ntr
ibu
no
Send
patches
Have
coffee
Tired?
yes
© MIRANTIS 2014
!18

19. Thanks for Attending!
Project: launchpad.net/mistral
IRC: #openstack-mistral
Me: Renat Akhmerov
Email: rakhmerov@mirantis.com
© MIRANTIS 2014
Q&A
!19