SlideShare une entreprise Scribd logo
1  sur  24
Télécharger pour lire hors ligne
Scale your
Jenkins pipeline
         FOSDEM 2013
     Anders Nickelsen, PhD
   QA engineer @ Tradeshift
 @anickelsen, ani@tradeshift.com
Scaling your Jenkins CI pipeline
Scaling your Jenkins CI pipeline
Tradeshift
Tradeshift
Platform for your business interactions
Core services are free

~3 years old

One product – one production environment

~20 developers
       Copenhagen, DK and San Francisco, CA
Feb 2, 2013              FOSDEM 2013
Scaling your Jenkins CI pipeline
Slow test is slow
Why scale? Fast feedback
Integration tests (IT): Selenium 2 from Geb (Java/Groovy)
Run by Jenkins on every change

Takes 2 hours to run all in sequence

10+ team branches, each trigger IT
Verified merges to master, also trigger IT

Pipeline is congestion point
Feedback not fast enough

Feb 2, 2013                FOSDEM 2013
Release pipeline
                                                    Any branch
                                                    (unit tested)
                                 10 team branches



Production    Master




                                                       = 10 projects
                 Integration tests on changes



Feb 2, 2013                        FOSDEM 2013
The swarm
© Blizzard 2013
What?
12 Jenkins slaves
       – New slaves join swarm on boot
       – Orchestrated by ec2-collective
       – Configured by Puppet at boot

Tests distributed in sets
Longest processing time
Uses test run-time from last stable build

1 set per online slave (dynamic)
Feb 2, 2013                FOSDEM 2013
Scaling your Jenkins CI pipeline
Fast tests!
All sets are put into Jenkins build queue
Picked up by any slave
       – throttled to one per slave


12 slaves => 20 minutes
       – Tests: 10 min
       – Overhead: 10 min
24 slaves => 15 minutes
Feb 2, 2013                 FOSDEM 2013
Post processing
Join when all sets complete
       – Triggered builds are blocking

Collect test results
       – JUnit, HTML, screenshots, videos, console logs = 500
         MB/run
       – Curl request to Jenkins API

Process test results
       – JVM outputs, slice console logs, archive artifacts
       – Custom groovy script


Feb 2, 2013                   FOSDEM 2013
Optimizations
Optimizations
Only on ITCase file-level, file scan

Only on spec level
       – min time = longest running spec


Only scale test processing time
       – 10 minutes today


Feb 2, 2013                 FOSDEM 2013
Lessons learned
Parallelization overhead
Swarm setup and tear down
       – node initialization and test result collection


Tests break a lot when run in parallel
Fixing tests and code hurts




Feb 2, 2013                 FOSDEM 2013
‘Random failures’
Failures appear probabilistic / random
       – Order dependencies
       – Timing issues
       – Rebuild ‘fixes’ the tests

Slave failures
       – Weakest link breaks the pipeline
       – 24 slaves => 1 filled up due to slow mount =>
         pipeline broken

Feb 2, 2013                 FOSDEM 2013
Scaling your Jenkins CI pipeline
Cost optimization
AWS EC2 instances:
       – 1/2 price gave 3/4 performance
       – 4/3 swarm size gave 1/4 price reduction
       – Equivalent performance


Swarm is offline when not used
       – Kept online for 1 hour after use


Feb 2, 2013                FOSDEM 2013
Scaling your Jenkins CI pipeline
Credits
puppetlabs.com
github.com/andersdyekjaerhansen/ec2_collective

Jenkins and plugins
       – Swarm plugin, parameterized
         trigger, envinject, rebuild, join, throttle concurrent
         build, conditional build step

More details of our setup
       – tradeshift.com/blog/just-add-servers/


We’re hiring!
       – tradeshift.com/jobs
Feb 2, 2013                          FOSDEM 2013

Contenu connexe

Tendances

SQL Server DevOps Jumpstart
SQL Server DevOps JumpstartSQL Server DevOps Jumpstart
SQL Server DevOps JumpstartOri Donner
 
Selenium lightning-talk
Selenium lightning-talkSelenium lightning-talk
Selenium lightning-talkStephen Donner
 
How Jenkins Builds the Netflix Global Streaming Service
How Jenkins Builds the Netflix Global Streaming ServiceHow Jenkins Builds the Netflix Global Streaming Service
How Jenkins Builds the Netflix Global Streaming ServiceGareth Bowles
 
Hudson at FISL 2009
Hudson at FISL 2009Hudson at FISL 2009
Hudson at FISL 2009Arun Gupta
 
The DynaSlave Plugin
The DynaSlave PluginThe DynaSlave Plugin
The DynaSlave PluginBrian Moyles
 
Improving Engineering Processes using Hudson - Spark IT 2010
Improving Engineering Processes using Hudson - Spark IT 2010Improving Engineering Processes using Hudson - Spark IT 2010
Improving Engineering Processes using Hudson - Spark IT 2010Arun Gupta
 
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWS
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWSAutomated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWS
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWSBamdad Dashtban
 
Getting Started with Dockerization of Selenium Tests Execution - Testwarez 2019
Getting Started with Dockerization of Selenium Tests Execution - Testwarez 2019Getting Started with Dockerization of Selenium Tests Execution - Testwarez 2019
Getting Started with Dockerization of Selenium Tests Execution - Testwarez 2019Sargis Sargsyan
 
DCSF19 Dockerfile Best Practices
DCSF19 Dockerfile Best PracticesDCSF19 Dockerfile Best Practices
DCSF19 Dockerfile Best PracticesDocker, Inc.
 
CI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos
CI and CD at Scale: Scaling Jenkins with Docker and Apache MesosCI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos
CI and CD at Scale: Scaling Jenkins with Docker and Apache MesosCarlos Sanchez
 
Continuous Delivery in Enterprise Environments using Docker, Ansible and Jenkins
Continuous Delivery in Enterprise Environments using Docker, Ansible and JenkinsContinuous Delivery in Enterprise Environments using Docker, Ansible and Jenkins
Continuous Delivery in Enterprise Environments using Docker, Ansible and JenkinsMarcel Birkner
 
제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers
제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers
제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-RanchersTommy Lee
 
Node.js Rocks in Docker for Dev and Ops
Node.js Rocks in Docker for Dev and OpsNode.js Rocks in Docker for Dev and Ops
Node.js Rocks in Docker for Dev and OpsBret Fisher
 
Performance Testing using Real Browsers with JMeter & Webdriver
Performance Testing using Real Browsers with JMeter & WebdriverPerformance Testing using Real Browsers with JMeter & Webdriver
Performance Testing using Real Browsers with JMeter & WebdriverBlazeMeter
 
Un jenkins amélioré avec docker mesos et marathon à Devoxx 2015
Un jenkins amélioré avec docker mesos et marathon à Devoxx 2015Un jenkins amélioré avec docker mesos et marathon à Devoxx 2015
Un jenkins amélioré avec docker mesos et marathon à Devoxx 2015Publicis Sapient Engineering
 
Docker for Ops - Scott Coulton, Puppet
Docker for Ops - Scott Coulton, PuppetDocker for Ops - Scott Coulton, Puppet
Docker for Ops - Scott Coulton, PuppetDocker, Inc.
 
Docker 1.9 Feature Overview
Docker 1.9 Feature OverviewDocker 1.9 Feature Overview
Docker 1.9 Feature OverviewSreenivas Makam
 
Saltconf16 - Salt is Not Configuration Management
Saltconf16 - Salt is Not Configuration ManagementSaltconf16 - Salt is Not Configuration Management
Saltconf16 - Salt is Not Configuration ManagementDrew Malone
 

Tendances (20)

SQL Server DevOps Jumpstart
SQL Server DevOps JumpstartSQL Server DevOps Jumpstart
SQL Server DevOps Jumpstart
 
Selenium lightning-talk
Selenium lightning-talkSelenium lightning-talk
Selenium lightning-talk
 
How Jenkins Builds the Netflix Global Streaming Service
How Jenkins Builds the Netflix Global Streaming ServiceHow Jenkins Builds the Netflix Global Streaming Service
How Jenkins Builds the Netflix Global Streaming Service
 
Hudson at FISL 2009
Hudson at FISL 2009Hudson at FISL 2009
Hudson at FISL 2009
 
The DynaSlave Plugin
The DynaSlave PluginThe DynaSlave Plugin
The DynaSlave Plugin
 
Improving Engineering Processes using Hudson - Spark IT 2010
Improving Engineering Processes using Hudson - Spark IT 2010Improving Engineering Processes using Hudson - Spark IT 2010
Improving Engineering Processes using Hudson - Spark IT 2010
 
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWS
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWSAutomated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWS
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWS
 
Getting Started with Dockerization of Selenium Tests Execution - Testwarez 2019
Getting Started with Dockerization of Selenium Tests Execution - Testwarez 2019Getting Started with Dockerization of Selenium Tests Execution - Testwarez 2019
Getting Started with Dockerization of Selenium Tests Execution - Testwarez 2019
 
DCSF19 Dockerfile Best Practices
DCSF19 Dockerfile Best PracticesDCSF19 Dockerfile Best Practices
DCSF19 Dockerfile Best Practices
 
CI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos
CI and CD at Scale: Scaling Jenkins with Docker and Apache MesosCI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos
CI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos
 
Continuous Delivery in Enterprise Environments using Docker, Ansible and Jenkins
Continuous Delivery in Enterprise Environments using Docker, Ansible and JenkinsContinuous Delivery in Enterprise Environments using Docker, Ansible and Jenkins
Continuous Delivery in Enterprise Environments using Docker, Ansible and Jenkins
 
제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers
제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers
제4회 한국IBM과 함께하는 난공불락 오픈소스 인프라 세미나-Ranchers
 
Node.js Rocks in Docker for Dev and Ops
Node.js Rocks in Docker for Dev and OpsNode.js Rocks in Docker for Dev and Ops
Node.js Rocks in Docker for Dev and Ops
 
Performance Testing using Real Browsers with JMeter & Webdriver
Performance Testing using Real Browsers with JMeter & WebdriverPerformance Testing using Real Browsers with JMeter & Webdriver
Performance Testing using Real Browsers with JMeter & Webdriver
 
Un jenkins amélioré avec docker mesos et marathon à Devoxx 2015
Un jenkins amélioré avec docker mesos et marathon à Devoxx 2015Un jenkins amélioré avec docker mesos et marathon à Devoxx 2015
Un jenkins amélioré avec docker mesos et marathon à Devoxx 2015
 
Docker toolbox
Docker toolboxDocker toolbox
Docker toolbox
 
Jenkins & IaC
Jenkins & IaCJenkins & IaC
Jenkins & IaC
 
Docker for Ops - Scott Coulton, Puppet
Docker for Ops - Scott Coulton, PuppetDocker for Ops - Scott Coulton, Puppet
Docker for Ops - Scott Coulton, Puppet
 
Docker 1.9 Feature Overview
Docker 1.9 Feature OverviewDocker 1.9 Feature Overview
Docker 1.9 Feature Overview
 
Saltconf16 - Salt is Not Configuration Management
Saltconf16 - Salt is Not Configuration ManagementSaltconf16 - Salt is Not Configuration Management
Saltconf16 - Salt is Not Configuration Management
 

En vedette

Jenkins Pipeline - Seville JAM
Jenkins Pipeline - Seville JAMJenkins Pipeline - Seville JAM
Jenkins Pipeline - Seville JAMamunizmartin
 
JCConf2016 Jenkins Pipeline
JCConf2016 Jenkins PipelineJCConf2016 Jenkins Pipeline
JCConf2016 Jenkins PipelineChing Yi Chan
 
Pyramid Slide
Pyramid SlidePyramid Slide
Pyramid Slideaaksland
 
Continuous Delivery Pipeline with Docker and Jenkins
Continuous Delivery Pipeline with Docker and JenkinsContinuous Delivery Pipeline with Docker and Jenkins
Continuous Delivery Pipeline with Docker and JenkinsCamilo Ribeiro
 
3d pyramid stacked shapes chart 4 pieces powerpoint ppt slides.
3d pyramid stacked shapes chart 4 pieces powerpoint ppt slides.3d pyramid stacked shapes chart 4 pieces powerpoint ppt slides.
3d pyramid stacked shapes chart 4 pieces powerpoint ppt slides.SlideTeam.net
 
Delivery Pipeline as Code: using Jenkins 2.0 Pipeline
Delivery Pipeline as Code: using Jenkins 2.0 PipelineDelivery Pipeline as Code: using Jenkins 2.0 Pipeline
Delivery Pipeline as Code: using Jenkins 2.0 PipelineSlawa Giterman
 
3d sales funnel powerpoint ppt slides.
3d sales funnel powerpoint ppt slides.3d sales funnel powerpoint ppt slides.
3d sales funnel powerpoint ppt slides.SlideTeam.net
 
Download editable sales funnel power point slides and ppt diagram templates
Download editable sales funnel power point slides and ppt diagram templates Download editable sales funnel power point slides and ppt diagram templates
Download editable sales funnel power point slides and ppt diagram templates SlideTeam.net
 
7 Habits of Highly Effective Jenkins Users
7 Habits of Highly Effective Jenkins Users7 Habits of Highly Effective Jenkins Users
7 Habits of Highly Effective Jenkins UsersAndrew Bayer
 

En vedette (12)

Jenkins_servers_architecture_ci
Jenkins_servers_architecture_ciJenkins_servers_architecture_ci
Jenkins_servers_architecture_ci
 
Scaling Jenkins
Scaling Jenkins Scaling Jenkins
Scaling Jenkins
 
Jenkins Pipeline - Seville JAM
Jenkins Pipeline - Seville JAMJenkins Pipeline - Seville JAM
Jenkins Pipeline - Seville JAM
 
JCConf2016 Jenkins Pipeline
JCConf2016 Jenkins PipelineJCConf2016 Jenkins Pipeline
JCConf2016 Jenkins Pipeline
 
Pyramid deployment
Pyramid deploymentPyramid deployment
Pyramid deployment
 
Pyramid Slide
Pyramid SlidePyramid Slide
Pyramid Slide
 
Continuous Delivery Pipeline with Docker and Jenkins
Continuous Delivery Pipeline with Docker and JenkinsContinuous Delivery Pipeline with Docker and Jenkins
Continuous Delivery Pipeline with Docker and Jenkins
 
3d pyramid stacked shapes chart 4 pieces powerpoint ppt slides.
3d pyramid stacked shapes chart 4 pieces powerpoint ppt slides.3d pyramid stacked shapes chart 4 pieces powerpoint ppt slides.
3d pyramid stacked shapes chart 4 pieces powerpoint ppt slides.
 
Delivery Pipeline as Code: using Jenkins 2.0 Pipeline
Delivery Pipeline as Code: using Jenkins 2.0 PipelineDelivery Pipeline as Code: using Jenkins 2.0 Pipeline
Delivery Pipeline as Code: using Jenkins 2.0 Pipeline
 
3d sales funnel powerpoint ppt slides.
3d sales funnel powerpoint ppt slides.3d sales funnel powerpoint ppt slides.
3d sales funnel powerpoint ppt slides.
 
Download editable sales funnel power point slides and ppt diagram templates
Download editable sales funnel power point slides and ppt diagram templates Download editable sales funnel power point slides and ppt diagram templates
Download editable sales funnel power point slides and ppt diagram templates
 
7 Habits of Highly Effective Jenkins Users
7 Habits of Highly Effective Jenkins Users7 Habits of Highly Effective Jenkins Users
7 Habits of Highly Effective Jenkins Users
 

Similaire à Scaling your Jenkins CI pipeline

Continuous Integration
Continuous IntegrationContinuous Integration
Continuous IntegrationJoerg Henning
 
Parallel Running Automation Solution with Docker, Jenkins and Zalenium
Parallel Running Automation Solution with Docker, Jenkins and ZaleniumParallel Running Automation Solution with Docker, Jenkins and Zalenium
Parallel Running Automation Solution with Docker, Jenkins and ZaleniumEvozon Test Lab
 
Scaling Puppet Usage to a Global Organization
Scaling Puppet Usage to a Global OrganizationScaling Puppet Usage to a Global Organization
Scaling Puppet Usage to a Global OrganizationPuppet
 
DataStax | Effective Testing in DSE (Lessons Learned) (Predrag Knezevic) | Ca...
DataStax | Effective Testing in DSE (Lessons Learned) (Predrag Knezevic) | Ca...DataStax | Effective Testing in DSE (Lessons Learned) (Predrag Knezevic) | Ca...
DataStax | Effective Testing in DSE (Lessons Learned) (Predrag Knezevic) | Ca...DataStax
 
Effective Testing in DSE
Effective Testing in DSEEffective Testing in DSE
Effective Testing in DSEpedjak
 
developing sysadmin, sysadmining developersGuug devops puppet
developing sysadmin, sysadmining developersGuug devops puppetdeveloping sysadmin, sysadmining developersGuug devops puppet
developing sysadmin, sysadmining developersGuug devops puppetMartin Alfke
 
Om enterprise labs session leader
Om enterprise labs session leaderOm enterprise labs session leader
Om enterprise labs session leaderMark Maclean
 
Be flexible with zalenium, a dockerized selenium grid infrastructure
Be flexible with zalenium, a dockerized selenium grid infrastructureBe flexible with zalenium, a dockerized selenium grid infrastructure
Be flexible with zalenium, a dockerized selenium grid infrastructureGeorgios Romanas
 
Improving Your Heroku App Performance with Asset CDN and Unicorn
Improving Your Heroku App Performance with Asset CDN and UnicornImproving Your Heroku App Performance with Asset CDN and Unicorn
Improving Your Heroku App Performance with Asset CDN and UnicornSimon Bagreev
 
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)STePINForum
 
DX@Scale: Optimizing Salesforce Development and Deployment for large scale pr...
DX@Scale: Optimizing Salesforce Development and Deployment for large scale pr...DX@Scale: Optimizing Salesforce Development and Deployment for large scale pr...
DX@Scale: Optimizing Salesforce Development and Deployment for large scale pr...Paris Salesforce Developer Group
 
TestWorks Conf Scalable QA with docker - Maarten van den Ende and Adé Mochtar
TestWorks Conf Scalable QA with docker - Maarten van den Ende and Adé MochtarTestWorks Conf Scalable QA with docker - Maarten van den Ende and Adé Mochtar
TestWorks Conf Scalable QA with docker - Maarten van den Ende and Adé MochtarXebia Nederland BV
 
Escaping Automated Test Hell - One Year Later
Escaping Automated Test Hell - One Year LaterEscaping Automated Test Hell - One Year Later
Escaping Automated Test Hell - One Year LaterWojciech Seliga
 
Building Docker Containers @ Scale
Building Docker Containers @ ScaleBuilding Docker Containers @ Scale
Building Docker Containers @ Scalelxfontes
 
Spark to Production @Windward
Spark to Production @WindwardSpark to Production @Windward
Spark to Production @WindwardDemi Ben-Ari
 
Load Testing - How to Stress Your Odoo with Locust
Load Testing - How to Stress Your Odoo with LocustLoad Testing - How to Stress Your Odoo with Locust
Load Testing - How to Stress Your Odoo with LocustOdoo
 
Service Configuration Management for Rapid Growth - demo 10 steps to build pi...
Service Configuration Management for Rapid Growth - demo 10 steps to build pi...Service Configuration Management for Rapid Growth - demo 10 steps to build pi...
Service Configuration Management for Rapid Growth - demo 10 steps to build pi...Takashi Someda
 
Parallelizing CI using Docker Swarm-Mode
Parallelizing CI using Docker Swarm-ModeParallelizing CI using Docker Swarm-Mode
Parallelizing CI using Docker Swarm-ModeAkihiro Suda
 

Similaire à Scaling your Jenkins CI pipeline (20)

Continuous Integration
Continuous IntegrationContinuous Integration
Continuous Integration
 
Parallel Running Automation Solution with Docker, Jenkins and Zalenium
Parallel Running Automation Solution with Docker, Jenkins and ZaleniumParallel Running Automation Solution with Docker, Jenkins and Zalenium
Parallel Running Automation Solution with Docker, Jenkins and Zalenium
 
Scaling Puppet Usage to a Global Organization
Scaling Puppet Usage to a Global OrganizationScaling Puppet Usage to a Global Organization
Scaling Puppet Usage to a Global Organization
 
DataStax | Effective Testing in DSE (Lessons Learned) (Predrag Knezevic) | Ca...
DataStax | Effective Testing in DSE (Lessons Learned) (Predrag Knezevic) | Ca...DataStax | Effective Testing in DSE (Lessons Learned) (Predrag Knezevic) | Ca...
DataStax | Effective Testing in DSE (Lessons Learned) (Predrag Knezevic) | Ca...
 
Effective Testing in DSE
Effective Testing in DSEEffective Testing in DSE
Effective Testing in DSE
 
developing sysadmin, sysadmining developersGuug devops puppet
developing sysadmin, sysadmining developersGuug devops puppetdeveloping sysadmin, sysadmining developersGuug devops puppet
developing sysadmin, sysadmining developersGuug devops puppet
 
Om enterprise labs session leader
Om enterprise labs session leaderOm enterprise labs session leader
Om enterprise labs session leader
 
Be flexible with zalenium, a dockerized selenium grid infrastructure
Be flexible with zalenium, a dockerized selenium grid infrastructureBe flexible with zalenium, a dockerized selenium grid infrastructure
Be flexible with zalenium, a dockerized selenium grid infrastructure
 
Improving Your Heroku App Performance with Asset CDN and Unicorn
Improving Your Heroku App Performance with Asset CDN and UnicornImproving Your Heroku App Performance with Asset CDN and Unicorn
Improving Your Heroku App Performance with Asset CDN and Unicorn
 
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)
Docker–Grid (A On demand and Scalable dockerized selenium grid architecture)
 
DX@Scale: Optimizing Salesforce Development and Deployment for large scale pr...
DX@Scale: Optimizing Salesforce Development and Deployment for large scale pr...DX@Scale: Optimizing Salesforce Development and Deployment for large scale pr...
DX@Scale: Optimizing Salesforce Development and Deployment for large scale pr...
 
TestWorks Conf Scalable QA with docker - Maarten van den Ende and Adé Mochtar
TestWorks Conf Scalable QA with docker - Maarten van den Ende and Adé MochtarTestWorks Conf Scalable QA with docker - Maarten van den Ende and Adé Mochtar
TestWorks Conf Scalable QA with docker - Maarten van den Ende and Adé Mochtar
 
Scalable QA With Docker
Scalable QA With DockerScalable QA With Docker
Scalable QA With Docker
 
Escaping Automated Test Hell - One Year Later
Escaping Automated Test Hell - One Year LaterEscaping Automated Test Hell - One Year Later
Escaping Automated Test Hell - One Year Later
 
Tests antipatterns
Tests antipatternsTests antipatterns
Tests antipatterns
 
Building Docker Containers @ Scale
Building Docker Containers @ ScaleBuilding Docker Containers @ Scale
Building Docker Containers @ Scale
 
Spark to Production @Windward
Spark to Production @WindwardSpark to Production @Windward
Spark to Production @Windward
 
Load Testing - How to Stress Your Odoo with Locust
Load Testing - How to Stress Your Odoo with LocustLoad Testing - How to Stress Your Odoo with Locust
Load Testing - How to Stress Your Odoo with Locust
 
Service Configuration Management for Rapid Growth - demo 10 steps to build pi...
Service Configuration Management for Rapid Growth - demo 10 steps to build pi...Service Configuration Management for Rapid Growth - demo 10 steps to build pi...
Service Configuration Management for Rapid Growth - demo 10 steps to build pi...
 
Parallelizing CI using Docker Swarm-Mode
Parallelizing CI using Docker Swarm-ModeParallelizing CI using Docker Swarm-Mode
Parallelizing CI using Docker Swarm-Mode
 

Dernier

The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosErol GIRAUDY
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTopCSSGallery
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfInfopole1
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kitJamie (Taka) Wang
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationKnoldus Inc.
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxSatishbabu Gunukula
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applicationsnooralam814309
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud DataEric D. Schabell
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3DianaGray10
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarThousandEyes
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and businessFrancesco Corti
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)codyslingerland1
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)IES VE
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechProduct School
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInThousandEyes
 

Dernier (20)

The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenarios
 
Top 10 Squarespace Development Companies
Top 10 Squarespace Development CompaniesTop 10 Squarespace Development Companies
Top 10 Squarespace Development Companies
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
Extra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdfExtra-120324-Visite-Entreprise-icare.pdf
Extra-120324-Visite-Entreprise-icare.pdf
 
20140402 - Smart house demo kit
20140402 - Smart house demo kit20140402 - Smart house demo kit
20140402 - Smart house demo kit
 
Introduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its applicationIntroduction to RAG (Retrieval Augmented Generation) and its application
Introduction to RAG (Retrieval Augmented Generation) and its application
 
Oracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptxOracle Database 23c Security New Features.pptx
Oracle Database 23c Security New Features.pptx
 
Graphene Quantum Dots-Based Composites for Biomedical Applications
Graphene Quantum Dots-Based Composites for  Biomedical ApplicationsGraphene Quantum Dots-Based Composites for  Biomedical Applications
Graphene Quantum Dots-Based Composites for Biomedical Applications
 
3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data3 Pitfalls Everyone Should Avoid with Cloud Data
3 Pitfalls Everyone Should Avoid with Cloud Data
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3
 
EMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? WebinarEMEA What is ThousandEyes? Webinar
EMEA What is ThousandEyes? Webinar
 
From the origin to the future of Open Source model and business
From the origin to the future of  Open Source model and businessFrom the origin to the future of  Open Source model and business
From the origin to the future of Open Source model and business
 
The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)The New Cloud World Order Is FinOps (Slideshow)
The New Cloud World Order Is FinOps (Slideshow)
 
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through TokenizationStobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
Stobox 4: Revolutionizing Investment in Real-World Assets Through Tokenization
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
 
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedInOutage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
Outage Analysis: March 5th/6th 2024 Meta, Comcast, and LinkedIn
 

Scaling your Jenkins CI pipeline

  • 1. Scale your Jenkins pipeline FOSDEM 2013 Anders Nickelsen, PhD QA engineer @ Tradeshift @anickelsen, ani@tradeshift.com
  • 5. Tradeshift Platform for your business interactions Core services are free ~3 years old One product – one production environment ~20 developers Copenhagen, DK and San Francisco, CA Feb 2, 2013 FOSDEM 2013
  • 8. Why scale? Fast feedback Integration tests (IT): Selenium 2 from Geb (Java/Groovy) Run by Jenkins on every change Takes 2 hours to run all in sequence 10+ team branches, each trigger IT Verified merges to master, also trigger IT Pipeline is congestion point Feedback not fast enough Feb 2, 2013 FOSDEM 2013
  • 9. Release pipeline Any branch (unit tested) 10 team branches Production Master = 10 projects Integration tests on changes Feb 2, 2013 FOSDEM 2013
  • 12. What? 12 Jenkins slaves – New slaves join swarm on boot – Orchestrated by ec2-collective – Configured by Puppet at boot Tests distributed in sets Longest processing time Uses test run-time from last stable build 1 set per online slave (dynamic) Feb 2, 2013 FOSDEM 2013
  • 14. Fast tests! All sets are put into Jenkins build queue Picked up by any slave – throttled to one per slave 12 slaves => 20 minutes – Tests: 10 min – Overhead: 10 min 24 slaves => 15 minutes Feb 2, 2013 FOSDEM 2013
  • 15. Post processing Join when all sets complete – Triggered builds are blocking Collect test results – JUnit, HTML, screenshots, videos, console logs = 500 MB/run – Curl request to Jenkins API Process test results – JVM outputs, slice console logs, archive artifacts – Custom groovy script Feb 2, 2013 FOSDEM 2013
  • 17. Optimizations Only on ITCase file-level, file scan Only on spec level – min time = longest running spec Only scale test processing time – 10 minutes today Feb 2, 2013 FOSDEM 2013
  • 19. Parallelization overhead Swarm setup and tear down – node initialization and test result collection Tests break a lot when run in parallel Fixing tests and code hurts Feb 2, 2013 FOSDEM 2013
  • 20. ‘Random failures’ Failures appear probabilistic / random – Order dependencies – Timing issues – Rebuild ‘fixes’ the tests Slave failures – Weakest link breaks the pipeline – 24 slaves => 1 filled up due to slow mount => pipeline broken Feb 2, 2013 FOSDEM 2013
  • 22. Cost optimization AWS EC2 instances: – 1/2 price gave 3/4 performance – 4/3 swarm size gave 1/4 price reduction – Equivalent performance Swarm is offline when not used – Kept online for 1 hour after use Feb 2, 2013 FOSDEM 2013
  • 24. Credits puppetlabs.com github.com/andersdyekjaerhansen/ec2_collective Jenkins and plugins – Swarm plugin, parameterized trigger, envinject, rebuild, join, throttle concurrent build, conditional build step More details of our setup – tradeshift.com/blog/just-add-servers/ We’re hiring! – tradeshift.com/jobs Feb 2, 2013 FOSDEM 2013