Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
DIY: A simple orchestrator built step by step
TIAD PARIS 2015
ChallengesHi, I’m Ahmet Demir
Also known as @ahmet2mir
Opensource Enthusiast @D2SI
Why Orchestration
All IT want
But we’ve got
When we try to
assemble them
http://gothicpug.deviantart.com/art/Wolverine-s-lego-cy-wtf-140295823
Choose the orchestrator
What we orchestrate ?
IT Process: disaster recovery, restart process
Infrastructure: Build a VM, upgrade it
Application De...
State of the art
Eliminate "server management deployment"
FusionDirectory - http://www.fusiondirectory.org
m23 - http://m23.sourceforge.net...
Eliminate config manager
Puppet - https://puppetlabs.com/
Chef - https://chef.io
SaltStack - http://saltstack.com/
Ansible...
Eliminate "server task oriented"
Rundeck - http://rundeck.org/
Eliminate Only Cloud oriented
Eucalyptus - https://github.com/eucalyptus/eucalyptus/wiki
Openstack Heat - https://github.c...
Finally 1/3
Terraform - https://terraform.io/
Terraform provides a flexible abstraction of resources and providers. This
m...
Finally 2/3
Salt Cloud - https://salt-cloud.readthedocs.org
Salt cloud is a public cloud provisioning tool. Salt cloud is ...
Finally 3/3
OpenScore - http://openscore.github.io/
Flow-based orchestration engine to manage deployed applications
Allows...
OpenStack Heat based:
Resource based, CRUD, Graph dependency, describe in Yaml
Terraform:
Execution Plan, Diff on infra an...
API access ?
How to scale ? If I had more than 100 requests in same time ?
Where can I store all my services ?
How can I t...
Do It Yourself
Simple to maintain
Think tomorrow
Infrastructure agnostic
Easy development
Don't forget Windows users :)
Fundamentals
NetworkStorage Compute
Orchestrator
Referential Config Manager
Catalog
HP MP, OpenCart...
Monitoring
Nagios...
CI / CD
Jen...
Engine
API
Resource runner
Resource 1 Resource 2 Resource 3 Resource 4
Run 1 & 2 in parallel Then run 3 Then run 4
Service...
Components
If you think to something, you will find your answer in Python
Both of Devs and Ops can use Python and use it quickly
No c...
YAML is simple to read and write
Many projects use it: Heat, Fig etc...
Service Description: YAML
Many tasks management
Easy to scale
Dedicate queue to some tasks (DMZ, Windows...)
Asynchronous task queue/job
Real-time o...
Messaging broker
Common platform to send and receive messages
Safe place to live until received
Can hits one million messa...
rapid and agile web development
No migrations
Not a relational database
Schemaless
Database
Let’s Get Started
Step 0: Build workers and broker
Rabbit MQ
Celery 1 Celery 2
Flower GUI
orion ~ $ git clone https://github.com/ahmet2mir/tiad-demo.git demo
orion ~ $ cd demo
orion demo $ cd 00_celery_rabbit
# s...
Step 1: Execute a simple resource
Rabbit MQ
Celery 1 Celery 2
AWS Instance
Resource
add in queue
Flower GUI
Celery 1
Wrapper
AWS
Wrapper
Docker
Docker
CRUD
AWS CRUD
Pure Python Package deployed
with pip
Wrapper to simplify executi...
orion demo $ cd 01_aws_resource
# virtual python environment
orion 01_aws_resource $ virtualenv .
orion 01_aws_resource $ ...
Step 2: Store outside
Rabbit MQ
Celery 1 Celery 2
Flower GUI
MongoDBApplication
orion demo $ cd 02_db_provider
# virtual python environment
orion 02_db_provider $ virtualenv .
orion 02_db_provider $ sou...
Application
Step 3: Service level
Rabbit MQ
Celery 1 Celery 2
Service
AWS Instance
Resource
Flower GUI
Docker Resource
1 2...
orion demo $ cd 03_app
# virtual python environment
orion 03_app $ virtualenv .
orion 03_app $ source bin/activate
orion 0...
Commit API
Commit Admin GUI
Commit Referential Management
Commit all modules and wrappers
Build a service catalog
Role bas...
TIAD - DYI: A simple orchestrator built step by step
TIAD - DYI: A simple orchestrator built step by step
TIAD - DYI: A simple orchestrator built step by step
Prochain SlideShare
Chargement dans…5
×

TIAD - DYI: A simple orchestrator built step by step

1 894 vues

Publié le

Use Python, Flask, Docker, Celery and more to start the fun!

  • Soyez le premier à commenter

TIAD - DYI: A simple orchestrator built step by step

  1. 1. DIY: A simple orchestrator built step by step TIAD PARIS 2015
  2. 2. ChallengesHi, I’m Ahmet Demir Also known as @ahmet2mir Opensource Enthusiast @D2SI
  3. 3. Why Orchestration
  4. 4. All IT want
  5. 5. But we’ve got
  6. 6. When we try to assemble them http://gothicpug.deviantart.com/art/Wolverine-s-lego-cy-wtf-140295823
  7. 7. Choose the orchestrator
  8. 8. What we orchestrate ? IT Process: disaster recovery, restart process Infrastructure: Build a VM, upgrade it Application Deployment: Deploy an application on x nodes Developpements: Continuous integration, Continuous Delivery
  9. 9. State of the art
  10. 10. Eliminate "server management deployment" FusionDirectory - http://www.fusiondirectory.org m23 - http://m23.sourceforge.net/PostNuke-0.750/html/index.php Fabric - http://www.fabfile.org/ Archipel - http://archipelproject.org/ AppScale - https://github.com/AppScale/appscale OpenBuild - http://openbuildservice.org/ CompatibleOne - http://compatibleone.org/ Opsi - http://www.opsi.org
  11. 11. Eliminate config manager Puppet - https://puppetlabs.com/ Chef - https://chef.io SaltStack - http://saltstack.com/ Ansible - http://www.ansible.com/home CFengine - http://cfengine.com/
  12. 12. Eliminate "server task oriented" Rundeck - http://rundeck.org/
  13. 13. Eliminate Only Cloud oriented Eucalyptus - https://github.com/eucalyptus/eucalyptus/wiki Openstack Heat - https://github.com/openstack/heat
  14. 14. Finally 1/3 Terraform - https://terraform.io/ Terraform provides a flexible abstraction of resources and providers. This model allows for representing everything from physical hardware, virtual machines, and containers, to email and DNS providers #Go
  15. 15. Finally 2/3 Salt Cloud - https://salt-cloud.readthedocs.org Salt cloud is a public cloud provisioning tool. Salt cloud is made to integrate Salt into cloud providers in a clean way so that minions on public cloud systems can be quickly and easily modeled and provisioned. #Python
  16. 16. Finally 3/3 OpenScore - http://openscore.github.io/ Flow-based orchestration engine to manage deployed applications Allows you to rapidly automate your DevOps use cases using ready-made workflows. Based on Java and HP OO engine. #Java
  17. 17. OpenStack Heat based: Resource based, CRUD, Graph dependency, describe in Yaml Terraform: Execution Plan, Diff on infra and all heat features, describe in “json like” format Salt Cloud: Easy modules integration but no plan, no graph OpenScore: Service in Yaml, no plan but runbook featured Features
  18. 18. API access ? How to scale ? If I had more than 100 requests in same time ? Where can I store all my services ? How can I track ? Is modules integration easy ? What about lifecycle ? Versioning the service ? Multi-user ? Organization > 100 peoples ? High Availability ? But Try to solve or ...
  19. 19. Do It Yourself
  20. 20. Simple to maintain Think tomorrow Infrastructure agnostic Easy development Don't forget Windows users :) Fundamentals
  21. 21. NetworkStorage Compute Orchestrator Referential Config Manager Catalog HP MP, OpenCart... Monitoring Nagios... CI / CD Jenkins, Bamboo ... Service Designer Big Picture
  22. 22. Engine API Resource runner Resource 1 Resource 2 Resource 3 Resource 4 Run 1 & 2 in parallel Then run 3 Then run 4 Service Designer & Orchestrator
  23. 23. Components
  24. 24. If you think to something, you will find your answer in Python Both of Devs and Ops can use Python and use it quickly No compilation and not "magic" Write less and do more Limits the number of times your brain has to do a "context switch." Helps productivity The language
  25. 25. YAML is simple to read and write Many projects use it: Heat, Fig etc... Service Description: YAML
  26. 26. Many tasks management Easy to scale Dedicate queue to some tasks (DMZ, Windows...) Asynchronous task queue/job Real-time operation, but supports scheduling Tasks are executed concurrently Used in production systems to process millions of tasks a day Tasks elery
  27. 27. Messaging broker Common platform to send and receive messages Safe place to live until received Can hits one million messages per second Message Delivery
  28. 28. rapid and agile web development No migrations Not a relational database Schemaless Database
  29. 29. Let’s Get Started
  30. 30. Step 0: Build workers and broker Rabbit MQ Celery 1 Celery 2 Flower GUI
  31. 31. orion ~ $ git clone https://github.com/ahmet2mir/tiad-demo.git demo orion ~ $ cd demo orion demo $ cd 00_celery_rabbit # start celery orion 00_celery_rabbit $ fig -f celery.yml -p demo up -d # start rabbitmq orion 00_celery_rabbit $ fig -f rabbitmq.yml -p demo up -d # create rabbitmq user orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl add_user myuser mypassword orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl add_vhost myvhost orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*" # gui for flower and rabbitmq orion 00_celery_rabbit $ fig -f flower.yml -p demo up -d # Visit http://localhost:5555 Step 0: Demo
  32. 32. Step 1: Execute a simple resource Rabbit MQ Celery 1 Celery 2 AWS Instance Resource add in queue Flower GUI
  33. 33. Celery 1 Wrapper AWS Wrapper Docker Docker CRUD AWS CRUD Pure Python Package deployed with pip Wrapper to simplify execution
  34. 34. orion demo $ cd 01_aws_resource # virtual python environment orion 01_aws_resource $ virtualenv . orion 01_aws_resource $ source bin/activate orion 01_aws_resource $ pip install -r requirements.txt # add credentials in client_provider.py # edit client_provider with some values (optionnal) … # run the resource orion 01_aws_resource $ python client.py Step 1: Demo
  35. 35. Step 2: Store outside Rabbit MQ Celery 1 Celery 2 Flower GUI MongoDBApplication
  36. 36. orion demo $ cd 02_db_provider # virtual python environment orion 02_db_provider $ virtualenv . orion 02_db_provider $ source bin/activate orion 02_db_provider $ pip install -r requirements.txt # start mongodb orion 02_db_provider $ fig -f mongodb.yml -p demo up -d # add credentials in add_*.py orion 02_db_provider $ python add_aws.py orion 02_db_provider $ python add_docker.py # visit http://localhost:8080 Step 2: Demo
  37. 37. Application Step 3: Service level Rabbit MQ Celery 1 Celery 2 Service AWS Instance Resource Flower GUI Docker Resource 1 2 MongoDB
  38. 38. orion demo $ cd 03_app # virtual python environment orion 03_app $ virtualenv . orion 03_app $ source bin/activate orion 03_app $ pip install -r requirements.txt # run the service orion 03_app $ python teecli.py create sample01.yml Step 3: Demo
  39. 39. Commit API Commit Admin GUI Commit Referential Management Commit all modules and wrappers Build a service catalog Role based access to the app http://teerex.io (soon) https://github.com/ahmet2mir/teerex (soon) Next steps

×