Presentation for PHP Unconference 2011 - unfortunately I did not get enough votes so never did the talk :(

1. Continuous Integration
for Heterogeneous
Environments
Nils Hofmeister, 11.09.2011
PHP Unconference

2. Hello world!
• Nils Hofmeister
• 5+ Years experience in realtime 3D / C++
• 2+ Years experience in web-based games (php, javascript, Java, C++, C#)
• Lead Integration Architect @Bigpoint, Hamburg – “the SVN guy” ;)
2

3. Agenda
• What is continuous integration?
• Why do you want continuous integration for your project?
• How do we implement continuous integration at Bigpoint?
• What are our next steps?
3

4. What is continuous
integration?
4

5. What is continuous integration?
The gossip
• A way to automize certain tasks, e.g.
• code validation
• build automation
• packaging of products
• A system of cooperating hardware- and software components , e.g.
• Source code management system (SCM)
• Staging system + production servers
• Build cluster with all kinds of compilers and frameworks on it
The idea behind this is much more important, so we do the technical stuff later.
5

6. What is continuous integration?
The idea
• Not brand-new: First known example: IBM OS/360, early 60s
• Current ideas and drivers closely related to agile movement and Extreme
Programming
• Martin Fowler says:
•
•
•
•
•
•
•
software development practice
integrate *…+ frequently
multiple integrations per day
verified by an automated build
detect integration errors
significantly reduced integration problems
develop cohesive software more rapidly
Martin Fowler on Continuous Integration - 2006 version
Original article – 2000
6

7. What is continuous integration?
The philosophy
Jez Humble says in "Continuous Delivery":
•
•
•
•
•
•
•
•
Create a repeatable, reliable process [...]
Automate almost everything
Keep everything in version control
If it hurts, do it more freqeuntly and bring the pain forward
Build quality in
Done means released
Everybody is responsible for the delivery process
Continuous improvement
“If something hurts, do it more often”
7

8. What is continuous integration?
The attitude
• Accept that CI is a standard tool. Just as compilers, SCMs or documentation are
• Understand that failing builds must have highest priority - always!
• Keep in mind that errors happen anyway. CI only makes them visible
immediately and reliably.
8

9. Why do you want continuous
integration for your
project?
9

10. Why do you want continuous integration for your project?
Reduce risks
• Defects are detected and fixed sooner
• Health of software is measurable
• Reduce assumptions (environment variables, local setup, etc)
10

11. Why do you want continuous integration for your project?
Reduce repetitive manual processes
• Process runs the same way every time
• The process starts automatically
• This frees resources for higher-value work
11

12. Why do you want continuous integration for your project?
Generate deployable software
• Product is deployable always
• Paradigm-shift from crunch time before release to "always ready to deliver"
12

13. Why do you want continuous integration for your project?
Better project visibility
• Supports effective decisions
• Facilitates recognizing trends
13

14. Why do you want continuous integration for your project?
Greater confidence in your project
• CI provides a safety net to avoid releasing flawed software
• Developers get a better feeling for impact of changes
14

15. How do we implement
continuous integration at
Bigpoint?
15

16. How do we implement continuous integration at Bigpoint?
The setting
• Websites, backend systems, clients
• PHP is one of many technologies required to assemble our games (an important one,
don’t worry ;))
16

17. How do we implement continuous integration at Bigpoint?
The requirements
• Per artifact:
• Check for defects
• Measure quality
• Per project:
• Assemble deliverable packages
• Deploy these
17

18. How do we implement continuous integration at Bigpoint?
The solution space (subjective! and there is more of course)
Microsoft Team Foundation Server
Integrates fine with Microsoft Tools. But with nothing else 
TeamCity
Well-known product by JetBrains. Feature set is fine, but no USP
that justifies to pay for it right now.
CruiseControl
The classic invented by ThoughtWorks. Now an open source
project. Delightfully old-school.
Bamboo
Quite solid, full-featured product by Atlassian. No real USP though
that justifies the high price.
Go
The current CI solution by ThoughtWorks. The theory behind it is
cool but it lacks a lot of features and configuration is no fun.
18

19. How do we implement continuous integration at Bigpoint?
The actual solution for us
• Hudson:
• Created by Kohsuke Kawaguchi
• "Duke's Choice Award" in 2008
• Comes from the Java world, ships with CVS/SVN, Maven, JUnit support
• Jenkins:
• Kohsuke was working for Sun till Oracle acquired Sun in 2010
• Conflict between the Hudson Core developers and Oracle => Fork in 2011
19

20. How do we implement continuous integration at Bigpoint?
The actual solution for us II
• Trends after fork
• Hudson => Focus on quality & stability
• Jenkins => Core developers, faster pace
• Our decision
• choose Jenkins as for us features > stability.
• to compensate we use Jenkins LTS
20

21. How do we implement continuous integration at Bigpoint?
The actual solution for us III
We like the Jenkins community!
(My colleague Marcelo and Kohsuke on their way to get their hands on some beer)
21

22. How do we implement continuous integration at Bigpoint?
Hamburg Jenkins cluster
There are more instances, but this is the biggest one…
• Setup
• 1 fileserver as master (Debian)
• 3 blades (Debian)
• 4 windows workstations
• Toolchains
•
•
•
•
•
Java
Flash/Flex
Php (+ JS, html, CSS)
Unity/Mono
C++
• Highly customized
• Home-grown housekeeping + backup
• Custom plugins / actively develop Jenkins
• All the stuff needed to deal with Kerberos…
22

23. How do we implement continuous integration at Bigpoint?
Hamburg Jenkins cluster II
Under the hood:
• Unity3D
• Unity/Mono + Ant
• Java
• Maven + some Ant
• Artifactory/Nexus (memo: synergies…)
• Flash
• Ant/Maven (Not sure who will win yet)
• C++
• gcc/make via Ant/Shell
• PHP/web apps
• All glory to Sebastian Bergmann – we more or less use his template…
• …plus some mods, e.g. minifiers, CDN stuff,
23

24. How do we implement continuous integration at Bigpoint?
Bigpoint standard game build pipeline
Behold: join plugin, downstream builds, copy artifacts, parameterized builds.
We do not do deployment from within Jenkins but have a custom deployment tool.
Interface is SVN tags.
24

25. How do we implement continuous integration at Bigpoint?
Bigpoint standard build pipeline II
• Oh noes, this is too slow…
• Optimize hardware
• Parallelize wherever possible
• Where not possible, decompose and create additional pipelines
• But… this is a SPOF!
•
•
•
•
Backup _everything_ automatically
Fully automized CI server setup
Find some replication/failover setup (todo)
Remember: everything the CI server does must run on your machine as well!
• Wait a minute… no more Ninja-deployment? But we need that to be agile…
• No you don’t 
25

26. How do we implement continuous integration at Bigpoint?
The obstacles
• The vast amount of knowledge needed
• Understanding the Bigpoint platform (the Release Engineering team is “brand new”)
• The different requirements, setup and skills of about 25 groups of stakeholders
• Linux/IT-driven platform vs. Java toolchain
• Acceptance (everyone claims to like Software Engineering, but some people seem to
have a different understanding of what that is  )
We can handle much of this very well by now.
Our biggest problem remaining:
• Technical debt (already!)
• We need a build framework…
26

27. What are our next steps?
27

28. What are our next steps?
Win the SVN + Kerberos battle
• Custom SVN sub system for Jenkins + custom Ant task
• Allows svn+ssh with Kerberos for our setup. _Massive_ speedup
• At some point we will contribute this to the Jenkins community…
28

29. What are our next steps?
The Build Framework
• Collect learnings and known requirements
• Derive a flexible framework
• Unify maintenance and interfaces to internal customers
Either we automize things further or we need tons of people doing frustrating stuff.
We don’t like frustrating stuff.
29

30. What are our next steps?
The canonical build pipeline in Jenkins
https://wiki.jenkins-ci.org/display/JENKINS/Build+Pipeline+Plugin
• Learn from the ThoughtWorks guys!
• Interesting: build pipeline plugin – Centrum Systems
30

31. “If something hurts, do it
more often”
31

32. Contact us
Bigpoint GmbH
Nils Hofmeister
Lead Integration Architect
Drehbahn 47-48
20354 Hamburg
Germany
Tel +49 40.88 14 13 - 0
Fax +49 40.88 14 13 - 11
info@bigpoint.net
www.bigpoint.net
Bigpoint Inc.
Bigpoint Distribuição de
Entretenimento Online Ltda.
500 Howard Street
Suite 300
San Francisco, CA 94105
Av. Brig. Faria Lima
3729 cj. 528
04538-905 São Paulo
Brazil
Bigpoint GmbH
Bigpoint International Services
Limited
Alexanderstraße 5
10178 Berlin
Germany
1 Villa Zimmermann
Ta’Xbiex Terrace
XBX 1035 Ta’Xbiex
Malta
Find us on
32

33. Bigpoint GmbH
Nils Hofmeister
Lead Integration Architect
Drehbahn 47-48
20354 Hamburg
Germany
Tel +49 40.88 14 13 - 0
Fax +49 40.88 14 13 - 11
info@bigpoint.net
www.bigpoint.net
Find us on
33