Continuous Integration
& Continuous Delivery
with OpenSource Tools
Annotated Version
• The grey slides are additional slides added to the
presentation to make it easier to understand on
Sli...
@kaktusmimi
lihsmi.ch
Inspired by…
Jo
Dani
Paddy
Seb
Daniel
Christiane
Inspiration
• This talk was inspired a lot by the work of my
colleagues at punkt.de
• And a presentation of Sebastian Helz...
Motivation
Tests
Version Control
Jenkins
Deployment
Configuration Management
Monitoring
Summary & Further Reading
Server
Developer
Delivery
Motivation
• We want to deliver our Software from Development to
Production
• In small increments
Feedback
Motivation
• As a developer we want to improve constantly
• Build - Measure - Learn is a Mantra for Learning
• Feedback su...
FTP to live Server
Live Server
FTP to test Server
Test Server
FTP to live Server
FTP to test Server
Test Server
FTP to live Server
All developers work on
single Dev Server
instance
FTP to live Server
Dev Server
Live Server
Version Control Build
gitlab triggers
Unit TestsCommit Stage
build succeeds
UI Tests
Acceptance Stage
DB Tests
tests pass
...
Evolution of Software
Delivery
• One developer ships his software to live server using
FTP
• We start to use a Test Server...
① Git
First Step Towards Continuous Integration
if you don’t use it yet

USE GIT
Gitlab Server
git pull / push
Developer
Reviewer
Merge / Decline
Feature Branch
Master Branch
Master Feature
git checkout ...
Merge Request Workflow
• No developer is allowed to directly push to master
• Each developer opens a merge request for a f...
② Tests
Functional Testing
• Test that your code is working
• Not how it is working
• More black-box testing
• Incorporate your Da...
/** @test */

public function createCustomerCreatesExpectedCustomer() {

$this->customerService->createCustomer('4711', 'M...
Use Integration Tests
• Unit tests might be the wrong tool to test, whether
your application is working
• Integration Test...
Feature: Language menu

In order to switch the language on the website

As a website user

I need to be able to select the...
/**
* Given I am in "desktop" layout

*

* @When I am in :layout layout

*/

public function iResizeTheWindowToLayout($lay...
Frontend Tests with Behat
• Behat Tests provide full-stack tests including the
frontend
• Behat uses easy-to-read language...
③ Jenkins
Jenkins
• Java Application
• Can be deployed into Tomcat or Standalone
• Many Plugins available
• Basically a Task Runner
Deployment Stage
* Demo
Deployment
* Production
Deployment
Acceptance Stage
* Functional Tests
* Frontend Tests
Commit Sta...
Jenkins Stages
• Commit Stage
• Fast feedback: project builds, unit tests run?
• Acceptance Stage
• Slower tests: Integrat...
TL;DR
Do not „programm“ your
CI / CD tasks in Jenkins -
use scripts and make
them part of your projects!
④ Surf
TYPO3 Surf
• A Remote Server Automation and Deployment Tool
• Written in PHP
• Based on the TYPO3 Flow Framework
• Can be ...
Deployment Workflow
• Build Locally
• Run Tests
• Ship / Transfer
• Run Smoketests
• Only switch Release if Tests pass
Surf Concepts
$workflow set up HOW to deploy
$node set up WHERE to deploy
$application set up WHAT to deploy
$deployment g...
<?php



$workflow = new TYPO3SurfDomainModelSimpleWorkflow();



$node = new TYPO3SurfDomainModelNode('staging');

$node-...
/var/apache/XXXXXX/staging/XXXXX/releases$ ls
20141105100124
20141105102935
20141112095924
20141118055114
20141118072225
2...
⑤ Chef

&Vagrant
Developer Machine / Laptop
vagrant up
Virtual Machine
Workspace
Project 1
chef-solo
Gitlab Server
git
pull / push
ssh
mysq...
Our Vagrant Approach
* We have VirtualBox and Vagrant running on our laptop
* We start a virtual machine and run Chef insi...
Workspace
Project 1
Workspace
Project 2
Workspace
Project 2
ProvisionProjects
ProvisionJenkinsServer
ConfigureApache
Confi...
Chef & Jenkins
There shall be 2 Chef runs
1. The one that provisions our Jenkins Server [not yet finished…]
2. The one tha...
⑥ Monitoring
Monitoring Stage
• Check for Website to be alive
• Send Notifications
• Email, Jabber, SMS
• Ops: Nagios — Devs: Jenkins
Performance Stage
• Check Realtime Performance of your Website
• Gatling is a great Tool for writing Performance Tests
• V...
Summary
Version Control Build
gitlab triggers
Unit TestsCommit Stage
build succeeds
UI Tests
Acceptance Stage
DB Tests
tests pass
...
Gitlab Server
Jenkins Server
Workspace
Project 1
Workspace
Project 2
Workspace
Project 2
provision
Selenium Server
http / ...
www.codecoon.com
Make it fun to code again
Check it out
Questions
https://about.gitlab.com/
http://jenkins-ci.org/
https://phpunit.de/
https://www.getchef.com/
https://www.vagrantup.com/
h...
http://www.sourcetreeapp.com/download/
http://dashing.io/
http://gatling.io/
http://www.martinfowler.com/articles/
continuousIntegration.html
http://www.thoughtworks.com/insights
http://david.heineme...
https://www.atlassian.com/git/tutorials/comparing-
workflows/forking-workflow
http://nvie.com/posts/a-successful-git-branc...
TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools
TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools
TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools
TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools
TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools
TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools
TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools
TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools
TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools
TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools
TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools
TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools
Prochain SlideShare
Chargement dans…5
×

TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools

1 065 vues

Publié le

In diesem Talk beschreibe ich die Continuous Integartion Pipeline von punkt.de und deren Entstehen. Es wird motiviert, warum es sich lohnt, eine solche Pipeline zu implementieren und welche Tools wir dafür verwendet haben. Neben der Beschreibung von Git, Jenkins, Chef, Vagrant, Behat und Surf geht es auch um Integration der einzelnen Tools in eine Deployment Kette.

Publié dans : Logiciels
0 commentaire
3 j’aime
Statistiques
Remarques
  • Soyez le premier à commenter

Aucun téléchargement
Vues
Nombre de vues
1 065
Sur SlideShare
0
Issues des intégrations
0
Intégrations
125
Actions
Partages
0
Téléchargements
34
Commentaires
0
J’aime
3
Intégrations 0
Aucune incorporation

Aucune remarque pour cette diapositive

TYPO3 Camp Stuttgart 2015 - Continuous Delivery with Open Source Tools

  1. 1. Continuous Integration & Continuous Delivery with OpenSource Tools
  2. 2. Annotated Version • The grey slides are additional slides added to the presentation to make it easier to understand on Slideshare etc.
  3. 3. @kaktusmimi lihsmi.ch
  4. 4. Inspired by… Jo Dani Paddy Seb Daniel Christiane
  5. 5. Inspiration • This talk was inspired a lot by the work of my colleagues at punkt.de • And a presentation of Sebastian Helzle at FOSSASIA 2014 • http://www.slideshare.net/Sebobo/continuous- delivery-with-open-source-tools
  6. 6. Motivation Tests Version Control Jenkins Deployment Configuration Management Monitoring Summary & Further Reading
  7. 7. Server Developer Delivery
  8. 8. Motivation • We want to deliver our Software from Development to Production • In small increments
  9. 9. Feedback
  10. 10. Motivation • As a developer we want to improve constantly • Build - Measure - Learn is a Mantra for Learning • Feedback supports Learning • Continuous Integration and Delivery is a constant source of Feedback • The earlier the Feedback - the better
  11. 11. FTP to live Server Live Server
  12. 12. FTP to test Server Test Server FTP to live Server
  13. 13. FTP to test Server Test Server FTP to live Server
  14. 14. All developers work on single Dev Server instance FTP to live Server Dev Server Live Server
  15. 15. Version Control Build gitlab triggers Unit TestsCommit Stage build succeeds UI Tests Acceptance Stage DB Tests tests pass tests pass Deployment Stage tests pass build succeeds Build locally Demo Stage Production Stage rsync rsync git push Developer Feedback
  16. 16. Evolution of Software Delivery • One developer ships his software to live server using FTP • We start to use a Test Server to protect the production server • Multiple developers make the scenario more complex • Introduction of a developer server • Setup of a sophisticated deployment chain
  17. 17. ① Git
  18. 18. First Step Towards Continuous Integration if you don’t use it yet
 USE GIT
  19. 19. Gitlab Server git pull / push Developer Reviewer Merge / Decline Feature Branch Master Branch Master Feature git checkout -b feature pull !
  20. 20. Merge Request Workflow • No developer is allowed to directly push to master • Each developer opens a merge request for a finished feature • Any other developer can review the request and merge it into master • Provides a first feedback loop!
  21. 21. ② Tests
  22. 22. Functional Testing • Test that your code is working • Not how it is working • More black-box testing • Incorporate your Database • End-to-End testing
  23. 23. /** @test */
 public function createCustomerCreatesExpectedCustomer() {
 $this->customerService->createCustomer('4711', 'MickeyMouse');
 $this->persistenceManager->persistAll();
 $this->assertSame( '4711', $this->customerRepository->findAll()->getFirst()->getId() );
 }
  24. 24. Use Integration Tests • Unit tests might be the wrong tool to test, whether your application is working • Integration Tests provide a full stack test that detect defects easier
  25. 25. Feature: Language menu
 In order to switch the language on the website
 As a website user
 I need to be able to select the language in a menu
 
 Background:
 Given I am in "desktop" layout
 And I am on "/"
 
 Scenario: Switching the language from english to german
 When I follow "Language"
 And I wait for 500 milliseconds
 And I follow "Deutsch"
 Then I should be on "/de.html"
  26. 26. /** * Given I am in "desktop" layout
 *
 * @When I am in :layout layout
 */
 public function iResizeTheWindowToLayout($layout) {
 if (array_key_exists($layout, $this->screenSizes)) {
 $currentLayout = $this->screenSizes[$layout];
 $this->getSession()->getDriver()->resizeWindow(
 $currentLayout['width'], $currentLayout['height'], 'current');
 return TRUE;
 }
 throw new Exception(sprintf('Layout "%s" not defined', $layout));
 }
  27. 27. Frontend Tests with Behat • Behat Tests provide full-stack tests including the frontend • Behat uses easy-to-read language„Gherkin“ for writing tests • Selenium is used to run the tests
  28. 28. ③ Jenkins
  29. 29. Jenkins • Java Application • Can be deployed into Tomcat or Standalone • Many Plugins available • Basically a Task Runner
  30. 30. Deployment Stage * Demo Deployment * Production Deployment Acceptance Stage * Functional Tests * Frontend Tests Commit Stage * Clone Repository * Build Project * Unit Tests Triggered
 by
 Commit green green
  31. 31. Jenkins Stages • Commit Stage • Fast feedback: project builds, unit tests run? • Acceptance Stage • Slower tests: Integration tests and Frontend tests • Deployment Stage • Responsible for the Deployments to Staging and Production
  32. 32. TL;DR Do not „programm“ your CI / CD tasks in Jenkins - use scripts and make them part of your projects!
  33. 33. ④ Surf
  34. 34. TYPO3 Surf • A Remote Server Automation and Deployment Tool • Written in PHP • Based on the TYPO3 Flow Framework • Can be deployed as a Flow Package or Standalone
  35. 35. Deployment Workflow • Build Locally • Run Tests • Ship / Transfer • Run Smoketests • Only switch Release if Tests pass
  36. 36. Surf Concepts $workflow set up HOW to deploy $node set up WHERE to deploy $application set up WHAT to deploy $deployment glue it all together
  37. 37. <?php
 
 $workflow = new TYPO3SurfDomainModelSimpleWorkflow();
 
 $node = new TYPO3SurfDomainModelNode('staging');
 $node->setOptions(array(
 'username' => '<username>',
 'composerCommandPath' => '/usr/local/bin/composer'
 ));
 $node->setHostname($host);
 
 $application = new TYPO3SurfApplicationTYPO3Neos('<project_name>');
 $application->setOptions(array(
 'repositoryUrl' => '<git_remote_url>',
 'keepReleases' => 5,
 'packageMethod' => 'git',
 'transferMethod' => 'rsync',
 'updateMethod' => NULL
 ));
 $application->setDeploymentPath('<deployment_path>');
 $application->addNode($node);
 
 $deployment->addApplication($application);
 
 $deployment->onInitialize(function() use ($workflow, $application, $project) {
 $workflow->afterStage('migrate', 'codecoon:importContent');
 });
  38. 38. /var/apache/XXXXXX/staging/XXXXX/releases$ ls 20141105100124 20141105102935 20141112095924 20141118055114 20141118072225 20141119041835 20141120045634 current -> ./20141119041835 previous -> ./20141118072225 next -> ./20141120045634 /var/apache/XXXXXX$ ls -la htdocs -> staging/XXXXXX/releases/current/htdocs
  39. 39. ⑤ Chef
 &Vagrant
  40. 40. Developer Machine / Laptop vagrant up Virtual Machine Workspace Project 1 chef-solo Gitlab Server git pull / push ssh mysql samba nfs samba nfs git ssh http http chef run ssh
  41. 41. Our Vagrant Approach * We have VirtualBox and Vagrant running on our laptop * We start a virtual machine and run Chef inside this machine * Chef sets up * Our services (Apache, PHP, MySQL, …) * Our projects / webspaces * We can now use our familiar tools to work on our projects * It feels like „working locally“ although we have a Sandbox * We cannot crash the host OS when crashing the Dev-Environment * Think about packages * Think about different software versions for different projects * Think about how long it takes to re-install your laptop…
  42. 42. Workspace Project 1 Workspace Project 2 Workspace Project 2 ProvisionProjects ProvisionJenkinsServer ConfigureApache ConfigurePHP &MySQL We set up our Projects on Jenkins Automatically!!!
  43. 43. Chef & Jenkins There shall be 2 Chef runs 1. The one that provisions our Jenkins Server [not yet finished…] 2. The one that provisions the projects inside our Jenkins Server Those Projects are „publicly“ available * We can use them as „normal“ Website (e.g. for Review and Manual Testing) * We can run UI tests on them In the future we want to change this to a master/slave approach.
  44. 44. ⑥ Monitoring
  45. 45. Monitoring Stage • Check for Website to be alive • Send Notifications • Email, Jabber, SMS • Ops: Nagios — Devs: Jenkins
  46. 46. Performance Stage • Check Realtime Performance of your Website • Gatling is a great Tool for writing Performance Tests • Visualize Results on Dashboard
  47. 47. Summary
  48. 48. Version Control Build gitlab triggers Unit TestsCommit Stage build succeeds UI Tests Acceptance Stage DB Tests tests pass tests pass Deployment Stage tests pass build succeeds Build locally Demo Stage Production Stage rsync rsync git push Developer Feedback
  49. 49. Gitlab Server Jenkins Server Workspace Project 1 Workspace Project 2 Workspace Project 2 provision Selenium Server http / RESTful Services Repository Project 1 Workspace Project 2 Workspace Project 3 git shell / ssh http/UITesting git clone Production 1 Webspace Project 1 Production 2 Webspace Project 2 Production 3 Webspace Project 3 ssh / rsync
  50. 50. www.codecoon.com Make it fun to code again Check it out
  51. 51. Questions
  52. 52. https://about.gitlab.com/ http://jenkins-ci.org/ https://phpunit.de/ https://www.getchef.com/ https://www.vagrantup.com/ http://www.seleniumhq.org/ http://typo3.org/additional-products/surf
  53. 53. http://www.sourcetreeapp.com/download/ http://dashing.io/ http://gatling.io/
  54. 54. http://www.martinfowler.com/articles/ continuousIntegration.html http://www.thoughtworks.com/insights http://david.heinemeierhansson.com/2014/tdd-is-dead- long-live-testing.html http://www.rbcs-us.com/documents/Why-Most-Unit- Testing-is-Waste.pdf http://www.mind-the-seb.de/blog/codereview-made- simple.html
  55. 55. https://www.atlassian.com/git/tutorials/comparing- workflows/forking-workflow http://nvie.com/posts/a-successful-git-branching- model/

×