Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Scale yourJenkins pipeline         FOSDEM 2013     Anders Nickelsen, PhD   QA engineer @ Tradeshift @anickelsen, ani@trade...
Tradeshift
TradeshiftPlatform for your business interactionsCore services are free~3 years oldOne product – one production environmen...
Slow test is slow
Why scale? Fast feedbackIntegration tests (IT): Selenium 2 from Geb (Java/Groovy)Run by Jenkins on every changeTakes 2 hou...
Release pipeline                                                    Any branch                                            ...
The swarm
© Blizzard 2013
What?12 Jenkins slaves       – New slaves join swarm on boot       – Orchestrated by ec2-collective       – Configured by ...
Fast tests!All sets are put into Jenkins build queuePicked up by any slave       – throttled to one per slave12 slaves => ...
Post processingJoin when all sets complete       – Triggered builds are blockingCollect test results       – JUnit, HTML, ...
Optimizations
OptimizationsOnly on ITCase file-level, file scanOnly on spec level       – min time = longest running specOnly scale test...
Lessons learned
Parallelization overheadSwarm setup and tear down       – node initialization and test result collectionTests break a lot ...
‘Random failures’Failures appear probabilistic / random       – Order dependencies       – Timing issues       – Rebuild ‘...
Cost optimizationAWS EC2 instances:       – 1/2 price gave 3/4 performance       – 4/3 swarm size gave 1/4 price reduction...
Creditspuppetlabs.comgithub.com/andersdyekjaerhansen/ec2_collectiveJenkins and plugins       – Swarm plugin, parameterized...
Scaling your Jenkins CI pipeline
Scaling your Jenkins CI pipeline
Scaling your Jenkins CI pipeline
Scaling your Jenkins CI pipeline
Scaling your Jenkins CI pipeline
Scaling your Jenkins CI pipeline
Prochain SlideShare
Chargement dans…5
×

Scaling your Jenkins CI pipeline

3 400 vues

Publié le

As your product grows the number of tests will grow. At one point, the number of tests overwhelm the number of servers to run those tests, developer feedback becomes slower and bugs can sneak in. To solve this problem, we built parallelization into our Jenkins build pipeline, so we only need to add servers to keep overall test time down. In this talk, I'll cover how we did it and what came out of it in terms of pros and cons.

As your product grows the number of test grows. At Tradeshift, we develop code in a test-driven manner, so every change, be it a new feature or a bug fix, includes a set of functional tests driven by Selenium. At one point, it started taking too long to run all of our tests in sequence, even when we scaled capacity of test-servers to make the individual tests run faster. When tests are slow developers don’t get immediate feedback when they commit a change and more time is wasted switching between tasks and different parts of the code. To solve this problem, we built parallelization into our Jenkins build pipeline, so we only need to add servers to make our test suites run faster and keep overall test time down. In this talk, I'll cover how we did it and what came out of it in terms of pros and cons.

Publié dans : Technologie
  • Soyez le premier à commenter

Scaling your Jenkins CI pipeline

  1. 1. Scale yourJenkins pipeline FOSDEM 2013 Anders Nickelsen, PhD QA engineer @ Tradeshift @anickelsen, ani@tradeshift.com
  2. 2. Tradeshift
  3. 3. TradeshiftPlatform for your business interactionsCore services are free~3 years oldOne product – one production environment~20 developers Copenhagen, DK and San Francisco, CAFeb 2, 2013 FOSDEM 2013
  4. 4. Slow test is slow
  5. 5. Why scale? Fast feedbackIntegration tests (IT): Selenium 2 from Geb (Java/Groovy)Run by Jenkins on every changeTakes 2 hours to run all in sequence10+ team branches, each trigger ITVerified merges to master, also trigger ITPipeline is congestion pointFeedback not fast enoughFeb 2, 2013 FOSDEM 2013
  6. 6. Release pipeline Any branch (unit tested) 10 team branchesProduction Master = 10 projects Integration tests on changesFeb 2, 2013 FOSDEM 2013
  7. 7. The swarm
  8. 8. © Blizzard 2013
  9. 9. What?12 Jenkins slaves – New slaves join swarm on boot – Orchestrated by ec2-collective – Configured by Puppet at bootTests distributed in setsLongest processing timeUses test run-time from last stable build1 set per online slave (dynamic)Feb 2, 2013 FOSDEM 2013
  10. 10. Fast tests!All sets are put into Jenkins build queuePicked up by any slave – throttled to one per slave12 slaves => 20 minutes – Tests: 10 min – Overhead: 10 min24 slaves => 15 minutesFeb 2, 2013 FOSDEM 2013
  11. 11. Post processingJoin when all sets complete – Triggered builds are blockingCollect test results – JUnit, HTML, screenshots, videos, console logs = 500 MB/run – Curl request to Jenkins APIProcess test results – JVM outputs, slice console logs, archive artifacts – Custom groovy scriptFeb 2, 2013 FOSDEM 2013
  12. 12. Optimizations
  13. 13. OptimizationsOnly on ITCase file-level, file scanOnly on spec level – min time = longest running specOnly scale test processing time – 10 minutes todayFeb 2, 2013 FOSDEM 2013
  14. 14. Lessons learned
  15. 15. Parallelization overheadSwarm setup and tear down – node initialization and test result collectionTests break a lot when run in parallelFixing tests and code hurtsFeb 2, 2013 FOSDEM 2013
  16. 16. ‘Random failures’Failures appear probabilistic / random – Order dependencies – Timing issues – Rebuild ‘fixes’ the testsSlave failures – Weakest link breaks the pipeline – 24 slaves => 1 filled up due to slow mount => pipeline brokenFeb 2, 2013 FOSDEM 2013
  17. 17. Cost optimizationAWS EC2 instances: – 1/2 price gave 3/4 performance – 4/3 swarm size gave 1/4 price reduction – Equivalent performanceSwarm is offline when not used – Kept online for 1 hour after useFeb 2, 2013 FOSDEM 2013
  18. 18. Creditspuppetlabs.comgithub.com/andersdyekjaerhansen/ec2_collectiveJenkins and plugins – Swarm plugin, parameterized trigger, envinject, rebuild, join, throttle concurrent build, conditional build stepMore details of our setup – tradeshift.com/blog/just-add-servers/We’re hiring! – tradeshift.com/jobsFeb 2, 2013 FOSDEM 2013

×