Ce diaporama a bien été signalé.

TIAD - DYI: A simple orchestrator built step by step



Prochain SlideShare
Rundeck: The missing tool
Rundeck: The missing tool
Chargement dans…3
1 sur 42
1 sur 42

Plus De Contenu Connexe

Plus par The Incredible Automation Day

Livres associés

Gratuit avec un essai de 14 jours de Scribd

Tout voir

TIAD - DYI: A simple orchestrator built step by step

  1. 1. DIY: A simple orchestrator built step by step TIAD PARIS 2015
  2. 2. ChallengesHi, I’m Ahmet Demir Also known as @ahmet2mir Opensource Enthusiast @D2SI
  3. 3. Why Orchestration
  4. 4. All IT want
  5. 5. But we’ve got
  6. 6. When we try to assemble them http://gothicpug.deviantart.com/art/Wolverine-s-lego-cy-wtf-140295823
  7. 7. Choose the orchestrator
  8. 8. What we orchestrate ? IT Process: disaster recovery, restart process Infrastructure: Build a VM, upgrade it Application Deployment: Deploy an application on x nodes Developpements: Continuous integration, Continuous Delivery
  9. 9. State of the art
  10. 10. Eliminate "server management deployment" FusionDirectory - http://www.fusiondirectory.org m23 - http://m23.sourceforge.net/PostNuke-0.750/html/index.php Fabric - http://www.fabfile.org/ Archipel - http://archipelproject.org/ AppScale - https://github.com/AppScale/appscale OpenBuild - http://openbuildservice.org/ CompatibleOne - http://compatibleone.org/ Opsi - http://www.opsi.org
  11. 11. Eliminate config manager Puppet - https://puppetlabs.com/ Chef - https://chef.io SaltStack - http://saltstack.com/ Ansible - http://www.ansible.com/home CFengine - http://cfengine.com/
  12. 12. Eliminate "server task oriented" Rundeck - http://rundeck.org/
  13. 13. Eliminate Only Cloud oriented Eucalyptus - https://github.com/eucalyptus/eucalyptus/wiki Openstack Heat - https://github.com/openstack/heat
  14. 14. Finally 1/3 Terraform - https://terraform.io/ Terraform provides a flexible abstraction of resources and providers. This model allows for representing everything from physical hardware, virtual machines, and containers, to email and DNS providers #Go
  15. 15. Finally 2/3 Salt Cloud - https://salt-cloud.readthedocs.org Salt cloud is a public cloud provisioning tool. Salt cloud is made to integrate Salt into cloud providers in a clean way so that minions on public cloud systems can be quickly and easily modeled and provisioned. #Python
  16. 16. Finally 3/3 OpenScore - http://openscore.github.io/ Flow-based orchestration engine to manage deployed applications Allows you to rapidly automate your DevOps use cases using ready-made workflows. Based on Java and HP OO engine. #Java
  17. 17. OpenStack Heat based: Resource based, CRUD, Graph dependency, describe in Yaml Terraform: Execution Plan, Diff on infra and all heat features, describe in “json like” format Salt Cloud: Easy modules integration but no plan, no graph OpenScore: Service in Yaml, no plan but runbook featured Features
  18. 18. API access ? How to scale ? If I had more than 100 requests in same time ? Where can I store all my services ? How can I track ? Is modules integration easy ? What about lifecycle ? Versioning the service ? Multi-user ? Organization > 100 peoples ? High Availability ? But Try to solve or ...
  19. 19. Do It Yourself
  20. 20. Simple to maintain Think tomorrow Infrastructure agnostic Easy development Don't forget Windows users :) Fundamentals
  21. 21. NetworkStorage Compute Orchestrator Referential Config Manager Catalog HP MP, OpenCart... Monitoring Nagios... CI / CD Jenkins, Bamboo ... Service Designer Big Picture
  22. 22. Engine API Resource runner Resource 1 Resource 2 Resource 3 Resource 4 Run 1 & 2 in parallel Then run 3 Then run 4 Service Designer & Orchestrator
  23. 23. Components
  24. 24. If you think to something, you will find your answer in Python Both of Devs and Ops can use Python and use it quickly No compilation and not "magic" Write less and do more Limits the number of times your brain has to do a "context switch." Helps productivity The language
  25. 25. YAML is simple to read and write Many projects use it: Heat, Fig etc... Service Description: YAML
  26. 26. Many tasks management Easy to scale Dedicate queue to some tasks (DMZ, Windows...) Asynchronous task queue/job Real-time operation, but supports scheduling Tasks are executed concurrently Used in production systems to process millions of tasks a day Tasks elery
  27. 27. Messaging broker Common platform to send and receive messages Safe place to live until received Can hits one million messages per second Message Delivery
  28. 28. rapid and agile web development No migrations Not a relational database Schemaless Database
  29. 29. Let’s Get Started
  30. 30. Step 0: Build workers and broker Rabbit MQ Celery 1 Celery 2 Flower GUI
  31. 31. orion ~ $ git clone https://github.com/ahmet2mir/tiad-demo.git demo orion ~ $ cd demo orion demo $ cd 00_celery_rabbit # start celery orion 00_celery_rabbit $ fig -f celery.yml -p demo up -d # start rabbitmq orion 00_celery_rabbit $ fig -f rabbitmq.yml -p demo up -d # create rabbitmq user orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl add_user myuser mypassword orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl add_vhost myvhost orion 00_celery_rabbit $ docker exec -t demo_broker_1 rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*" # gui for flower and rabbitmq orion 00_celery_rabbit $ fig -f flower.yml -p demo up -d # Visit http://localhost:5555 Step 0: Demo
  32. 32. Step 1: Execute a simple resource Rabbit MQ Celery 1 Celery 2 AWS Instance Resource add in queue Flower GUI
  33. 33. Celery 1 Wrapper AWS Wrapper Docker Docker CRUD AWS CRUD Pure Python Package deployed with pip Wrapper to simplify execution
  34. 34. orion demo $ cd 01_aws_resource # virtual python environment orion 01_aws_resource $ virtualenv . orion 01_aws_resource $ source bin/activate orion 01_aws_resource $ pip install -r requirements.txt # add credentials in client_provider.py # edit client_provider with some values (optionnal) … # run the resource orion 01_aws_resource $ python client.py Step 1: Demo
  35. 35. Step 2: Store outside Rabbit MQ Celery 1 Celery 2 Flower GUI MongoDBApplication
  36. 36. orion demo $ cd 02_db_provider # virtual python environment orion 02_db_provider $ virtualenv . orion 02_db_provider $ source bin/activate orion 02_db_provider $ pip install -r requirements.txt # start mongodb orion 02_db_provider $ fig -f mongodb.yml -p demo up -d # add credentials in add_*.py orion 02_db_provider $ python add_aws.py orion 02_db_provider $ python add_docker.py # visit http://localhost:8080 Step 2: Demo
  37. 37. Application Step 3: Service level Rabbit MQ Celery 1 Celery 2 Service AWS Instance Resource Flower GUI Docker Resource 1 2 MongoDB
  38. 38. orion demo $ cd 03_app # virtual python environment orion 03_app $ virtualenv . orion 03_app $ source bin/activate orion 03_app $ pip install -r requirements.txt # run the service orion 03_app $ python teecli.py create sample01.yml Step 3: Demo
  39. 39. Commit API Commit Admin GUI Commit Referential Management Commit all modules and wrappers Build a service catalog Role based access to the app http://teerex.io (soon) https://github.com/ahmet2mir/teerex (soon) Next steps