SlideShare une entreprise Scribd logo
1  sur  28
Télécharger pour lire hors ligne
Donnerstag, 19. September 13
“How automated cloud infrastructure setups
can help with Continuous Delivery”
Edmund Haselwanter, eh@cloudbau.de, @ehaselwanter

Donnerstag, 19. September 13
Why Continuous Delivery?
For most companies, IT functions as the nervous system and provides
an increasing amount of the organizational muscle mass. Most critical
business functions are entirely automated within IT, and 95% of all
capital projects depend on IT to get done. Today, nearly every business
decision will result in at least one IT change.
„IT REVOLUTION MANIFESTO“, http://itrevolution.com

Theory of
Constraints

→

→
E.M. Goldratt, 1984

cloudbau
Donnerstag, 19. September 13

Gene Kim, Kevin Behr,
George Spafford, 2013
Continuous Delivery
Wikipedia: http://en.wikipedia.org/wiki/Continuous_delivery

●
●
●
●

Version Control
Continuous Integration
Automated Testing
Continuous Deployment

cloudbau
Donnerstag, 19. September 13
We got the software development part right!
Request

Approve

Develop

Test

WaterScrum

Deploy

- Fall

The core, chronic conflict that every IT
leader faces is the need to
simultaneously enable faster time to
market (i.e., make as many changes as
you can), while providing stable, secure
and reliable IT services (i.e., make as
few changes as you can). 
Gene Kim

„Lowering risk of change through tools and culture“
John Allspaw, Paul Hammond, 2009 (Flickr)

small changes, often
reproducible

Donnerstag, 19. September 13

culture change Dev+Ops

standardized

cloudbau

expect failure
feedback (e.g. tests/metrics/...)
The Last Mile: Dev/Test/Prod Parity
●
●
●
●
●
●
●
●

Hardware configuration (e.g. amount of RAM, HDD vs. SSD)
Software used (e.g. sqlite in dev, mysql in prod)
Software/library versions
Deployment (e.g. automated for test, manual for prod)
System configuration (debug flags, logging)
External dependencies (dev: payment mocked out vs. prod: real gateway)
Database contents (huge prod DB with millions of entries vs. few entries in dev)
Operating system (e.g. dev: Mac OS X, prod: Linux)

See also http://12factor.net/

cloudbau
Donnerstag, 19. September 13
The Last Mile: Dev/Test/Prod Parity
●
●

Log level “normal” to chatty on production

●
●

Changes in external dependencies (Sun Java to Oracle Java)

●
●
●
●

SQL Query no problem in dev environment but too slow for production data

●
●

Snowflake back-port: Live Tuning of Kernel/Mysql parameters, got lost after re-install

CI build broken due to minor version change in upstream mocking framework (0.4.1 to
0.4.2)

Sessions invalid in production due to time difference on multiple servers. No problem in
dev environment (just one app and one db server)

Different hardware: 32 vs. 64 bit. libraries missing
Different hardware: optimized CPU flags on CI system. build not working in prod
Staging In-house, Prod AWS Cloud: Latency ok during staging test, problem on
production

Development on Mac OSX/Windows, deploy on Linux: File not found (case sensitive)

See also http://12factor.net/

cloudbau
Donnerstag, 19. September 13
Getting to the solution...

cloudbau
Donnerstag, 19. September 13
Infrastructure as Code

Application

Environment

Infrastructure

cloudbau
Donnerstag, 19. September 13
Building Blocks: Tools and Culture
Config-DB

CPUVirt.
Binary-Repo

StorageVirt.
SDN

System Description
Orchestration

Prod Env

Programmable Infrastructure (Cloud)

Continuos Delivery

Anti-Fragile Organizations

Cross-Functional Teams

Processes and Culture

cloudbau
Donnerstag, 19. September 13

Dev Env

Executeable System Description

Continuous Build
DevOps

Stage Env
First... automate and equalize all
environments
●
●
●
●
●

Chef/Puppet/Cfengine/.... for config
Dependency management (for programming languages, packages etc.)
Dependency management for config scripts as well (e.g. Berkshelf for chef)
Everything in version control
Infrastructure can still be different...

cloudbau
Donnerstag, 19. September 13
... then use a cloud to equalize
infrastructure!

●
●
●

Encode the infrastructure as well
Describe all the pieces of the infrastructure in code
Now it’s code: it can be dynamic!

cloudbau
Donnerstag, 19. September 13
But ... There be dragons
Infrastructure as Code => We need a process

Source
Code
Test

X

Compiler
Artifact

cloudbau
Donnerstag, 19. September 13
The implementation patterns using OSS

● OpenStack / Eucalyptus / ...
● Chef/Puppet/Cfengine/ ...
● Jenkins (Building and testing the SW but also
infrastructures)

● NetflixOSS (Asgard + Aminate)

cloudbau
Donnerstag, 19. September 13
New Question in the Cloud Era: What is my
deploy Artifact

● The Software Package itself
● The whole VM
● A prepared container
● ...

cloudbau
Donnerstag, 19. September 13
The Software Package is the Artifact

● The build artifact is a binary.
● Use IaaS API to create the environment
● Deployment is after the server creation
● Deploy with automation or any other tool
● The whole VM
● “from source” is just a special case

cloudbau
Donnerstag, 19. September 13
HP Cloud + Jenkins + Puppet

Binary Repo

HP Cloud Servers
(OpenStack)

Jenkins

Puppet Modules

cloudbau
Donnerstag, 19. September 13
The VM itself is the Artifact

● Build artifact is a image (optimal for stateless service)
● Only install needed, there is no upgrade/deploy
● IaaS must support Loadbalancer as a service (or something similar)
● Orchestration for deployment steps needed

cloudbau
Donnerstag, 19. September 13
Jenkins + AWS + Asgard + Aminate + Chef

Jenkins

Aminate

AMIs

Asgard

+ Chef

AWS EC2 (ASG + ELB)

cloudbau
Donnerstag, 19. September 13
A container is the Artifact

●
●
●
●
●

cloudbau
Donnerstag, 19. September 13

Optimization of the VM idea
Transportable lightweight OS containers (e.g. LXC/docker)
Very often back to upgrade/deploy
Needs external orchestration as well
Very new toolchain emerging (docker/maestro)
https://www.docker.io/the_whole_story/

cloudbau
Donnerstag, 19. September 13
Jenkins + docker + cfengine + Aminate +
Chef

Jenkins

Docker

Host VM

Maestro

+ cfengine

AWS EC2 (ASG + ELB)

cloudbau
Donnerstag, 19. September 13
Cloud can help with developer environments
as well

●
●
●
●

cloudbau
Donnerstag, 19. September 13

Have your own environment
Brings the production setup to the developer
One description for local VM and Cloud environment (Dev/Prod)
Needs external orchestration as well
OpenStack + Vagrant + Chef
Chef Repo / Server

Binary Repo
Jenkins

Vagrant

cloudbau
Donnerstag, 19. September 13

OpenStack Servers
One more Thing: Other Areas Cloud can
help

● Persistent Data: Copy/Clone Production Data to test migrations
● Debugging: Developer can stand up whole environment (even with data)
● Backup/Restore (snapshot whole images/servers for rollback)

cloudbau
Donnerstag, 19. September 13
What’s missing in the OpenSource
Ecosystem?

● Better Orchestration/ Pipeline support in Jenkins (Commercial CI-Servers
are advanced here)

● OpenSource Orchestration Engines (like OpenStack Heat)
● NetflixOSS working on OpenStack

cloudbau
Donnerstag, 19. September 13
Don’t forget about Operations

● Cloud/App-Monitoring Solutions (Nagios won’t cut it in dynamic
environments)

● Logfiles
● Metric driven autoscaling

cloudbau
Donnerstag, 19. September 13
Thank you! Questions?

cloudbau
Donnerstag, 19. September 13

Contenu connexe

Tendances

OpenNebulaconf2017US: Rapid scaling of research computing to over 70,000 cor...
OpenNebulaconf2017US:  Rapid scaling of research computing to over 70,000 cor...OpenNebulaconf2017US:  Rapid scaling of research computing to over 70,000 cor...
OpenNebulaconf2017US: Rapid scaling of research computing to over 70,000 cor...
OpenNebula Project
 

Tendances (20)

OpenNebulaconf2017US: Rapid scaling of research computing to over 70,000 cor...
OpenNebulaconf2017US:  Rapid scaling of research computing to over 70,000 cor...OpenNebulaconf2017US:  Rapid scaling of research computing to over 70,000 cor...
OpenNebulaconf2017US: Rapid scaling of research computing to over 70,000 cor...
 
OpenNebulaConf 2016 - Hypervisors and Containers Hands-on Workshop by Jaime M...
OpenNebulaConf 2016 - Hypervisors and Containers Hands-on Workshop by Jaime M...OpenNebulaConf 2016 - Hypervisors and Containers Hands-on Workshop by Jaime M...
OpenNebulaConf 2016 - Hypervisors and Containers Hands-on Workshop by Jaime M...
 
Deploying Foreman in Enterprise Environments
Deploying Foreman in Enterprise EnvironmentsDeploying Foreman in Enterprise Environments
Deploying Foreman in Enterprise Environments
 
OpenNebulaConf 2016 - OpenNebula, a story about flexibility and technological...
OpenNebulaConf 2016 - OpenNebula, a story about flexibility and technological...OpenNebulaConf 2016 - OpenNebula, a story about flexibility and technological...
OpenNebulaConf 2016 - OpenNebula, a story about flexibility and technological...
 
PXEless Discovery with Foreman
PXEless Discovery with ForemanPXEless Discovery with Foreman
PXEless Discovery with Foreman
 
OpenNebula 4.14 Hands-on Tutorial
OpenNebula 4.14 Hands-on TutorialOpenNebula 4.14 Hands-on Tutorial
OpenNebula 4.14 Hands-on Tutorial
 
Foreman in your datacenter
Foreman in your datacenterForeman in your datacenter
Foreman in your datacenter
 
OpenNebulaconf2017US: Multi-Site Hyperconverged OpenNebula with DRBD9
OpenNebulaconf2017US: Multi-Site Hyperconverged OpenNebula with DRBD9OpenNebulaconf2017US: Multi-Site Hyperconverged OpenNebula with DRBD9
OpenNebulaconf2017US: Multi-Site Hyperconverged OpenNebula with DRBD9
 
OpenNebulaConf 2016 - Storage Hands-on Workshop by Javier Fontán, OpenNebula
OpenNebulaConf 2016 - Storage Hands-on Workshop by Javier Fontán, OpenNebulaOpenNebulaConf 2016 - Storage Hands-on Workshop by Javier Fontán, OpenNebula
OpenNebulaConf 2016 - Storage Hands-on Workshop by Javier Fontán, OpenNebula
 
OpenNebulaConf 2016 - Building a GNU/Linux Distribution by Daniel Dehennin, M...
OpenNebulaConf 2016 - Building a GNU/Linux Distribution by Daniel Dehennin, M...OpenNebulaConf 2016 - Building a GNU/Linux Distribution by Daniel Dehennin, M...
OpenNebulaConf 2016 - Building a GNU/Linux Distribution by Daniel Dehennin, M...
 
OpenNebulaConf 2016 - LAB ONE - Vagrant running on OpenNebula? by Florian Heigl
OpenNebulaConf 2016 - LAB ONE - Vagrant running on OpenNebula? by Florian HeiglOpenNebulaConf 2016 - LAB ONE - Vagrant running on OpenNebula? by Florian Heigl
OpenNebulaConf 2016 - LAB ONE - Vagrant running on OpenNebula? by Florian Heigl
 
Deploying RDO OpenStack with a pair of plugins
Deploying RDO OpenStack with a pair of pluginsDeploying RDO OpenStack with a pair of plugins
Deploying RDO OpenStack with a pair of plugins
 
Full Stack Automation with Katello & The Foreman
Full Stack Automation with Katello & The ForemanFull Stack Automation with Katello & The Foreman
Full Stack Automation with Katello & The Foreman
 
Connecting AWS and Katello/The Foreman
Connecting AWS and Katello/The ForemanConnecting AWS and Katello/The Foreman
Connecting AWS and Katello/The Foreman
 
Configuration and lifecycle in Mixed environments
Configuration and lifecycle in Mixed environmentsConfiguration and lifecycle in Mixed environments
Configuration and lifecycle in Mixed environments
 
Build a private cloud – prototype and test with open nebula
Build a private cloud – prototype and test with open nebulaBuild a private cloud – prototype and test with open nebula
Build a private cloud – prototype and test with open nebula
 
OpenNebulaConf2017EU: Alternative Context for Windows by Paul Batchelor, Blac...
OpenNebulaConf2017EU: Alternative Context for Windows by Paul Batchelor, Blac...OpenNebulaConf2017EU: Alternative Context for Windows by Paul Batchelor, Blac...
OpenNebulaConf2017EU: Alternative Context for Windows by Paul Batchelor, Blac...
 
OpenNebulaConf2017EU: One (Windows) Image to Rule them All by Paul Batchelor,...
OpenNebulaConf2017EU: One (Windows) Image to Rule them All by Paul Batchelor,...OpenNebulaConf2017EU: One (Windows) Image to Rule them All by Paul Batchelor,...
OpenNebulaConf2017EU: One (Windows) Image to Rule them All by Paul Batchelor,...
 
Step by Step - Reusing old features to build new ones
Step by Step - Reusing old features to build new onesStep by Step - Reusing old features to build new ones
Step by Step - Reusing old features to build new ones
 
OpenNebula Conf 2014 | Lightning talk: OpenNebula Puppet Module - Norman Mess...
OpenNebula Conf 2014 | Lightning talk: OpenNebula Puppet Module - Norman Mess...OpenNebula Conf 2014 | Lightning talk: OpenNebula Puppet Module - Norman Mess...
OpenNebula Conf 2014 | Lightning talk: OpenNebula Puppet Module - Norman Mess...
 

Similaire à How automated cloud infrastructure setups can help with Continuous Delivery

Similaire à How automated cloud infrastructure setups can help with Continuous Delivery (20)

VASCAN - Docker and Security
VASCAN - Docker and SecurityVASCAN - Docker and Security
VASCAN - Docker and Security
 
Kubernetes 101
Kubernetes 101Kubernetes 101
Kubernetes 101
 
Automated Snap Package build processes without the Build Service
Automated Snap Package build processes without the Build ServiceAutomated Snap Package build processes without the Build Service
Automated Snap Package build processes without the Build Service
 
Adventures in docker compose
Adventures in docker composeAdventures in docker compose
Adventures in docker compose
 
Heroku to Kubernetes & Gihub to Gitlab success story
Heroku to Kubernetes & Gihub to Gitlab success storyHeroku to Kubernetes & Gihub to Gitlab success story
Heroku to Kubernetes & Gihub to Gitlab success story
 
Next in Virtualization Era: Containerization & Docker
Next in Virtualization Era: Containerization & DockerNext in Virtualization Era: Containerization & Docker
Next in Virtualization Era: Containerization & Docker
 
Mobile CICD
Mobile CICD Mobile CICD
Mobile CICD
 
[HKOSCon x COSCUP 2020][20200801][Ansible: From VM to Kubernetes]
[HKOSCon x COSCUP 2020][20200801][Ansible: From VM to Kubernetes][HKOSCon x COSCUP 2020][20200801][Ansible: From VM to Kubernetes]
[HKOSCon x COSCUP 2020][20200801][Ansible: From VM to Kubernetes]
 
Velocity NYC 2017: Building Resilient Microservices with Kubernetes, Docker, ...
Velocity NYC 2017: Building Resilient Microservices with Kubernetes, Docker, ...Velocity NYC 2017: Building Resilient Microservices with Kubernetes, Docker, ...
Velocity NYC 2017: Building Resilient Microservices with Kubernetes, Docker, ...
 
Docker 101
Docker 101 Docker 101
Docker 101
 
Codemotion Rome 2015 IBM Bluemix and Docker
Codemotion Rome 2015 IBM Bluemix and DockerCodemotion Rome 2015 IBM Bluemix and Docker
Codemotion Rome 2015 IBM Bluemix and Docker
 
Cloud-Native Application and Kubernetes
Cloud-Native Application and KubernetesCloud-Native Application and Kubernetes
Cloud-Native Application and Kubernetes
 
HPC on OpenStack
HPC on OpenStackHPC on OpenStack
HPC on OpenStack
 
OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...OSDC 2018 | Three years running containers with Kubernetes in Production by T...
OSDC 2018 | Three years running containers with Kubernetes in Production by T...
 
Exploring Next Generation Buildpacks - Anand Rao & Scott Deeg
Exploring Next Generation Buildpacks - Anand Rao & Scott DeegExploring Next Generation Buildpacks - Anand Rao & Scott Deeg
Exploring Next Generation Buildpacks - Anand Rao & Scott Deeg
 
docker
dockerdocker
docker
 
Docker 0.11 at MaxCDN meetup in Los Angeles
Docker 0.11 at MaxCDN meetup in Los AngelesDocker 0.11 at MaxCDN meetup in Los Angeles
Docker 0.11 at MaxCDN meetup in Los Angeles
 
Making Service Deployments to AWS a breeze with Nova
Making Service Deployments to AWS a breeze with NovaMaking Service Deployments to AWS a breeze with Nova
Making Service Deployments to AWS a breeze with Nova
 
DCSF 19 Building Your Development Pipeline
DCSF 19 Building Your Development Pipeline  DCSF 19 Building Your Development Pipeline
DCSF 19 Building Your Development Pipeline
 
From a cluster to the Cloud
From a cluster to the CloudFrom a cluster to the Cloud
From a cluster to the Cloud
 

Dernier

Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
Joaquim Jorge
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
Enterprise Knowledge
 

Dernier (20)

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfThe Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
The Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptxThe Codex of Business Writing Software for Real-World Solutions 2.pptx
The Codex of Business Writing Software for Real-World Solutions 2.pptx
 
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI SolutionsIAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
 
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptxFactors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
 
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
 
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day PresentationGenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
 

How automated cloud infrastructure setups can help with Continuous Delivery

  • 2. “How automated cloud infrastructure setups can help with Continuous Delivery” Edmund Haselwanter, eh@cloudbau.de, @ehaselwanter Donnerstag, 19. September 13
  • 3. Why Continuous Delivery? For most companies, IT functions as the nervous system and provides an increasing amount of the organizational muscle mass. Most critical business functions are entirely automated within IT, and 95% of all capital projects depend on IT to get done. Today, nearly every business decision will result in at least one IT change. „IT REVOLUTION MANIFESTO“, http://itrevolution.com Theory of Constraints → → E.M. Goldratt, 1984 cloudbau Donnerstag, 19. September 13 Gene Kim, Kevin Behr, George Spafford, 2013
  • 4. Continuous Delivery Wikipedia: http://en.wikipedia.org/wiki/Continuous_delivery ● ● ● ● Version Control Continuous Integration Automated Testing Continuous Deployment cloudbau Donnerstag, 19. September 13
  • 5. We got the software development part right! Request Approve Develop Test WaterScrum Deploy - Fall The core, chronic conflict that every IT leader faces is the need to simultaneously enable faster time to market (i.e., make as many changes as you can), while providing stable, secure and reliable IT services (i.e., make as few changes as you can).  Gene Kim „Lowering risk of change through tools and culture“ John Allspaw, Paul Hammond, 2009 (Flickr) small changes, often reproducible Donnerstag, 19. September 13 culture change Dev+Ops standardized cloudbau expect failure feedback (e.g. tests/metrics/...)
  • 6. The Last Mile: Dev/Test/Prod Parity ● ● ● ● ● ● ● ● Hardware configuration (e.g. amount of RAM, HDD vs. SSD) Software used (e.g. sqlite in dev, mysql in prod) Software/library versions Deployment (e.g. automated for test, manual for prod) System configuration (debug flags, logging) External dependencies (dev: payment mocked out vs. prod: real gateway) Database contents (huge prod DB with millions of entries vs. few entries in dev) Operating system (e.g. dev: Mac OS X, prod: Linux) See also http://12factor.net/ cloudbau Donnerstag, 19. September 13
  • 7. The Last Mile: Dev/Test/Prod Parity ● ● Log level “normal” to chatty on production ● ● Changes in external dependencies (Sun Java to Oracle Java) ● ● ● ● SQL Query no problem in dev environment but too slow for production data ● ● Snowflake back-port: Live Tuning of Kernel/Mysql parameters, got lost after re-install CI build broken due to minor version change in upstream mocking framework (0.4.1 to 0.4.2) Sessions invalid in production due to time difference on multiple servers. No problem in dev environment (just one app and one db server) Different hardware: 32 vs. 64 bit. libraries missing Different hardware: optimized CPU flags on CI system. build not working in prod Staging In-house, Prod AWS Cloud: Latency ok during staging test, problem on production Development on Mac OSX/Windows, deploy on Linux: File not found (case sensitive) See also http://12factor.net/ cloudbau Donnerstag, 19. September 13
  • 8. Getting to the solution... cloudbau Donnerstag, 19. September 13
  • 10. Building Blocks: Tools and Culture Config-DB CPUVirt. Binary-Repo StorageVirt. SDN System Description Orchestration Prod Env Programmable Infrastructure (Cloud) Continuos Delivery Anti-Fragile Organizations Cross-Functional Teams Processes and Culture cloudbau Donnerstag, 19. September 13 Dev Env Executeable System Description Continuous Build DevOps Stage Env
  • 11. First... automate and equalize all environments ● ● ● ● ● Chef/Puppet/Cfengine/.... for config Dependency management (for programming languages, packages etc.) Dependency management for config scripts as well (e.g. Berkshelf for chef) Everything in version control Infrastructure can still be different... cloudbau Donnerstag, 19. September 13
  • 12. ... then use a cloud to equalize infrastructure! ● ● ● Encode the infrastructure as well Describe all the pieces of the infrastructure in code Now it’s code: it can be dynamic! cloudbau Donnerstag, 19. September 13
  • 13. But ... There be dragons Infrastructure as Code => We need a process Source Code Test X Compiler Artifact cloudbau Donnerstag, 19. September 13
  • 14. The implementation patterns using OSS ● OpenStack / Eucalyptus / ... ● Chef/Puppet/Cfengine/ ... ● Jenkins (Building and testing the SW but also infrastructures) ● NetflixOSS (Asgard + Aminate) cloudbau Donnerstag, 19. September 13
  • 15. New Question in the Cloud Era: What is my deploy Artifact ● The Software Package itself ● The whole VM ● A prepared container ● ... cloudbau Donnerstag, 19. September 13
  • 16. The Software Package is the Artifact ● The build artifact is a binary. ● Use IaaS API to create the environment ● Deployment is after the server creation ● Deploy with automation or any other tool ● The whole VM ● “from source” is just a special case cloudbau Donnerstag, 19. September 13
  • 17. HP Cloud + Jenkins + Puppet Binary Repo HP Cloud Servers (OpenStack) Jenkins Puppet Modules cloudbau Donnerstag, 19. September 13
  • 18. The VM itself is the Artifact ● Build artifact is a image (optimal for stateless service) ● Only install needed, there is no upgrade/deploy ● IaaS must support Loadbalancer as a service (or something similar) ● Orchestration for deployment steps needed cloudbau Donnerstag, 19. September 13
  • 19. Jenkins + AWS + Asgard + Aminate + Chef Jenkins Aminate AMIs Asgard + Chef AWS EC2 (ASG + ELB) cloudbau Donnerstag, 19. September 13
  • 20. A container is the Artifact ● ● ● ● ● cloudbau Donnerstag, 19. September 13 Optimization of the VM idea Transportable lightweight OS containers (e.g. LXC/docker) Very often back to upgrade/deploy Needs external orchestration as well Very new toolchain emerging (docker/maestro)
  • 22. Jenkins + docker + cfengine + Aminate + Chef Jenkins Docker Host VM Maestro + cfengine AWS EC2 (ASG + ELB) cloudbau Donnerstag, 19. September 13
  • 23. Cloud can help with developer environments as well ● ● ● ● cloudbau Donnerstag, 19. September 13 Have your own environment Brings the production setup to the developer One description for local VM and Cloud environment (Dev/Prod) Needs external orchestration as well
  • 24. OpenStack + Vagrant + Chef Chef Repo / Server Binary Repo Jenkins Vagrant cloudbau Donnerstag, 19. September 13 OpenStack Servers
  • 25. One more Thing: Other Areas Cloud can help ● Persistent Data: Copy/Clone Production Data to test migrations ● Debugging: Developer can stand up whole environment (even with data) ● Backup/Restore (snapshot whole images/servers for rollback) cloudbau Donnerstag, 19. September 13
  • 26. What’s missing in the OpenSource Ecosystem? ● Better Orchestration/ Pipeline support in Jenkins (Commercial CI-Servers are advanced here) ● OpenSource Orchestration Engines (like OpenStack Heat) ● NetflixOSS working on OpenStack cloudbau Donnerstag, 19. September 13
  • 27. Don’t forget about Operations ● Cloud/App-Monitoring Solutions (Nagios won’t cut it in dynamic environments) ● Logfiles ● Metric driven autoscaling cloudbau Donnerstag, 19. September 13