In diesem Talk beschreibe ich die Continuous Integartion Pipeline von punkt.de und deren Entstehen. Es wird motiviert, warum es sich lohnt, eine solche Pipeline zu implementieren und welche Tools wir dafür verwendet haben. Neben der Beschreibung von Git, Jenkins, Chef, Vagrant, Behat und Surf geht es auch um Integration der einzelnen Tools in eine Deployment Kette.

1. Continuous Integration
& Continuous Delivery
with OpenSource Tools

2. Annotated Version
• The grey slides are additional slides added to the
presentation to make it easier to understand on
Slideshare etc.

3. @kaktusmimi
lihsmi.ch

4. Inspired by…
Jo
Dani
Paddy
Seb
Daniel
Christiane

5. Inspiration
• This talk was inspired a lot by the work of my
colleagues at punkt.de
• And a presentation of Sebastian Helzle at FOSSASIA
2014
• http://www.slideshare.net/Sebobo/continuous-
delivery-with-open-source-tools

6. Motivation
Tests
Version Control
Jenkins
Deployment
Configuration Management
Monitoring
Summary & Further Reading

7. Server
Developer
Delivery

9. Motivation
• We want to deliver our Software from Development to
Production
• In small increments

12. Feedback

13. Motivation
• As a developer we want to improve constantly
• Build - Measure - Learn is a Mantra for Learning
• Feedback supports Learning
• Continuous Integration and Delivery is a
constant source of Feedback
• The earlier the Feedback - the better

14. FTP to live Server
Live Server

15. FTP to test Server
Test Server
FTP to live Server

16. FTP to test Server
Test Server
FTP to live Server

17. All developers work on
single Dev Server
instance
FTP to live Server
Dev Server
Live Server

18. Version Control Build
gitlab triggers
Unit TestsCommit Stage
build succeeds
UI Tests
Acceptance Stage
DB Tests
tests pass
tests pass
Deployment Stage
tests pass
build succeeds
Build locally
Demo Stage
Production Stage
rsync
rsync
git push
Developer
Feedback

19. Evolution of Software
Delivery
• One developer ships his software to live server using
FTP
• We start to use a Test Server to protect the production
server
• Multiple developers make the scenario more complex
• Introduction of a developer server
• Setup of a sophisticated deployment chain

20. ① Git

21. First Step Towards Continuous Integration
if you don’t use it yet
USE GIT

22. Gitlab Server
git pull / push
Developer
Reviewer
Merge / Decline
Feature Branch
Master Branch
Master Feature
git checkout -b feature
pull
!

23. Merge Request Workflow
• No developer is allowed to directly push to master
• Each developer opens a merge request for a finished
feature
• Any other developer can review the request and merge
it into master
• Provides a first feedback loop!

27. ② Tests

28. Functional Testing
• Test that your code is working
• Not how it is working
• More black-box testing
• Incorporate your Database
• End-to-End testing

29. /** @test */
public function createCustomerCreatesExpectedCustomer() {
$this->customerService->createCustomer('4711', 'MickeyMouse');
$this->persistenceManager->persistAll();
$this->assertSame(
'4711',
$this->customerRepository->findAll()->getFirst()->getId()
);
}

30. Use Integration Tests
• Unit tests might be the wrong tool to test, whether
your application is working
• Integration Tests provide a full stack test that detect
defects easier

32. Feature: Language menu
In order to switch the language on the website
As a website user
I need to be able to select the language in a menu
Background:
Given I am in "desktop" layout
And I am on "/"
Scenario: Switching the language from english to
german
When I follow "Language"
And I wait for 500 milliseconds
And I follow "Deutsch"
Then I should be on "/de.html"

33. /**
* Given I am in "desktop" layout
*
* @When I am in :layout layout
*/
public function iResizeTheWindowToLayout($layout) {
if (array_key_exists($layout, $this->screenSizes)) {
$currentLayout = $this->screenSizes[$layout];
$this->getSession()->getDriver()->resizeWindow(
$currentLayout['width'], $currentLayout['height'], 'current');
return TRUE;
}
throw new Exception(sprintf('Layout "%s" not defined', $layout));
}

34. Frontend Tests with Behat
• Behat Tests provide full-stack tests including the
frontend
• Behat uses easy-to-read language„Gherkin“ for writing
tests
• Selenium is used to run the tests

35. ③ Jenkins

36. Jenkins
• Java Application
• Can be deployed into Tomcat or Standalone
• Many Plugins available
• Basically a Task Runner

38. Deployment Stage
* Demo
Deployment
* Production
Deployment
Acceptance Stage
* Functional Tests
* Frontend Tests
Commit Stage
* Clone Repository
* Build Project
* Unit Tests
Triggered
by
Commit
green green

39. Jenkins Stages
• Commit Stage
• Fast feedback: project builds, unit tests run?
• Acceptance Stage
• Slower tests: Integration tests and Frontend tests
• Deployment Stage
• Responsible for the Deployments to Staging and
Production

41. TL;DR
Do not „programm“ your
CI / CD tasks in Jenkins -
use scripts and make
them part of your projects!

42. ④ Surf

43. TYPO3 Surf
• A Remote Server Automation and Deployment Tool
• Written in PHP
• Based on the TYPO3 Flow Framework
• Can be deployed as a Flow Package or Standalone

44. Deployment Workflow
• Build Locally
• Run Tests
• Ship / Transfer
• Run Smoketests
• Only switch Release if Tests pass

45. Surf Concepts
$workflow set up HOW to deploy
$node set up WHERE to deploy
$application set up WHAT to deploy
$deployment glue it all together

46. <?php
$workflow = new TYPO3SurfDomainModelSimpleWorkflow();
$node = new TYPO3SurfDomainModelNode('staging');
$node->setOptions(array(
'username' => '<username>',
'composerCommandPath' => '/usr/local/bin/composer'
));
$node->setHostname($host);
$application = new TYPO3SurfApplicationTYPO3Neos('<project_name>');
$application->setOptions(array(
'repositoryUrl' => '<git_remote_url>',
'keepReleases' => 5,
'packageMethod' => 'git',
'transferMethod' => 'rsync',
'updateMethod' => NULL
));
$application->setDeploymentPath('<deployment_path>');
$application->addNode($node);
$deployment->addApplication($application);
$deployment->onInitialize(function() use ($workflow, $application, $project) {
$workflow->afterStage('migrate', 'codecoon:importContent');
});

47. /var/apache/XXXXXX/staging/XXXXX/releases$ ls
20141105100124
20141105102935
20141112095924
20141118055114
20141118072225
20141119041835
20141120045634
current -> ./20141119041835
previous -> ./20141118072225
next -> ./20141120045634
/var/apache/XXXXXX$ ls -la
htdocs -> staging/XXXXXX/releases/current/htdocs

48. ⑤ Chef
&Vagrant

49. Developer Machine / Laptop
vagrant up
Virtual Machine
Workspace
Project 1
chef-solo
Gitlab Server
git
pull / push
ssh
mysql
samba
nfs
samba
nfs
git
ssh
http
http
chef run
ssh

50. Our Vagrant Approach
* We have VirtualBox and Vagrant running on our laptop
* We start a virtual machine and run Chef inside this machine
* Chef sets up
* Our services (Apache, PHP, MySQL, …)
* Our projects / webspaces
* We can now use our familiar tools to work on our projects
* It feels like „working locally“ although we have a Sandbox
* We cannot crash the host OS when crashing the Dev-Environment
* Think about packages
* Think about different software versions for different projects
* Think about how long it takes to re-install your laptop…

51. Workspace
Project 1
Workspace
Project 2
Workspace
Project 2
ProvisionProjects
ProvisionJenkinsServer
ConfigureApache
ConfigurePHP
&MySQL
We set up our Projects
on Jenkins
Automatically!!!

52. Chef & Jenkins
There shall be 2 Chef runs
1. The one that provisions our Jenkins Server [not yet finished…]
2. The one that provisions the projects inside our Jenkins Server
Those Projects are „publicly“ available
* We can use them as „normal“ Website (e.g. for Review and Manual Testing)
* We can run UI tests on them
In the future we want to change this to a master/slave approach.

53. ⑥ Monitoring

54. Monitoring Stage
• Check for Website to be alive
• Send Notifications
• Email, Jabber, SMS
• Ops: Nagios — Devs: Jenkins

55. Performance Stage
• Check Realtime Performance of your Website
• Gatling is a great Tool for writing Performance Tests
• Visualize Results on Dashboard

58. Summary

59. Version Control Build
gitlab triggers
Unit TestsCommit Stage
build succeeds
UI Tests
Acceptance Stage
DB Tests
tests pass
tests pass
Deployment Stage
tests pass
build succeeds
Build locally
Demo Stage
Production Stage
rsync
rsync
git push
Developer
Feedback

60. Gitlab Server
Jenkins Server
Workspace
Project 1
Workspace
Project 2
Workspace
Project 2
provision
Selenium Server
http / RESTful Services
Repository
Project 1
Workspace
Project 2
Workspace
Project 3
git shell / ssh
http/UITesting
git clone
Production 1
Webspace
Project 1
Production 2
Webspace
Project 2
Production 3
Webspace
Project 3
ssh / rsync

62. www.codecoon.com
Make it fun to code again
Check it out

63. Questions

64. https://about.gitlab.com/
http://jenkins-ci.org/
https://phpunit.de/
https://www.getchef.com/
https://www.vagrantup.com/
http://www.seleniumhq.org/
http://typo3.org/additional-products/surf

65. http://www.sourcetreeapp.com/download/
http://dashing.io/
http://gatling.io/

66. http://www.martinfowler.com/articles/
continuousIntegration.html
http://www.thoughtworks.com/insights
http://david.heinemeierhansson.com/2014/tdd-is-dead-
long-live-testing.html
http://www.rbcs-us.com/documents/Why-Most-Unit-
Testing-is-Waste.pdf
http://www.mind-the-seb.de/blog/codereview-made-
simple.html

67. https://www.atlassian.com/git/tutorials/comparing-
workflows/forking-workflow
http://nvie.com/posts/a-successful-git-branching-
model/