I would like to share my experience and journey on how we brought down our Jenkins build pipeline time down from over 90 minutes to under 12 minutes. In the process, I would share specific techniques which helped and also some, which logically made sense, but actually did not help. If your team is trying to optimize their build times, then this session might give you some ideas on how to approach the problem.
Development Impact - For one of our build job, below graph shows how the number of builds in a day have increased over a period of time as the build time has reduced. Frequency of code check-in has increased; Wait time has reduced; failed test case faster to isolate and fix.
Details: http://confengine.com/agile-pune-2014/proposal/458/techniques-to-speed-up-your-build-pipeline-for-faster-feedback
Conference: http://pune.agileindia.org/
10. Disk IO –Example
Database operations. Image Source - https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcTPdVawndjUZbU2PDn-oKgjBPqmgDqr3PPZatZh9kxEgNi71AND
http://www.dba-oracle.com/images/large_disk_hot_files.gif
Focus on Bottleneck
11. Disk IO –Alternative
Image Source - http://3.bp.blogspot.com/-bqTjSN7pSpg/UbqyjVojEFI/AAAAAAAADBw/PWe0kiuRHJ4/s200/no+duplicate+content.jpg
•Avoid file operations –e.g. duplicating workspace
Focus on Bottleneck
12. Disk IO –Alternative
•Avoid file operations –e.g. Jar creation.
Focus on Bottleneck
Image Source - http://i1.wp.com/blog.quoteroller.com/wp-content/uploads/2013/04/Dont-start-from-scratch.png?resize=800%2C264
13. Disk IO –Alternative
Image source - http://4.bp.blogspot.com/_4hvqisoH9CE/TSZIs7eiSAI/AAAAAAAAA7E/vanj6bGD8XQ/s1600/big-vs-small-left.jpg
•Test on smaller but apt data set.
Focus on Bottleneck
14. Disk IO –Alternative -SSD
CrystalDiskMark - http://crystalmark.info/software/CrystalDiskMark/index-e.html
•HDD (Toshiba MQ01ACF050 500GB SATA III) vs SSD (Samsung PM851 512GB mSata)
Focus on Bottleneck
16. Disk IO –Alternative -In Memory DB
Memory (Heap) Engine
–had some limitations over myisamengine.
Focus on Bottleneck
17. Disk IO –Alternative -In Memory DB
–was not supporting many MySQL queries so was discarded.
Focus on Bottleneck
18. Disk IO –Alternative -In Memory DB
database
–looked promising as it could support many MySQL queries but still required couple of modifications to our code.
Focus on Bottleneck
19. Disk IO –Alternative -In Memory DB
–looked most promising as it is wire compatible with MySQL, which means without code changes I could just point to memsql and be done with it.
Focus on Bottleneck
25. CPU –Profiling -Insights
Image source - https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcQde6NeSrbuv40CIhKtFa1OuIQXf7F7esMJKp1Ie7zmH2t29l6Z
Scanningresource bundle files from jars.
Focus on Bottleneck
26. CPU –Profiling -Insights
Image source - http://2.bp.blogspot.com/-uKMyLlB3F7o/Tqn_6yqdElI/AAAAAAAAB94/_1FMbHJFQBQ/s1600/weight-lift-cartoon.jpg
Loading Spring Application Context.
Focus on Bottleneck
27. CPU –Profiling -Insights
Image source - http://dilbert.com/dyn/str_strip/000000000/00000000/0000000/000000/00000/7000/000/7029/7029.strip.gif
Avoiding unnecessary activities during build e.g. sending out email.
Focus on Bottleneck
28. CPU –Profiling -Insights
java.util.Calendaris horribly slow.
Total processing time took 20.72minutes out of which Date Arithmetic took 18.15minutes which is about 87.6%of the total processing time!
Focus on Bottleneck
29. CPU –Profiling -Insights
java.util.Calendaris horribly slow. We switched to jodadate library and deprecated java.util.DateAPI.
Now Date Arithmetic takes 1.30minutes; that’s a massive saving of 93.77%
Focus on Bottleneck
30. CPU -ANT 1.7 Junit task options.
Focus on Bottleneck
31. Focus on the Bottlenecks
Divide and Conquer
Key Principles to Speed Up Your Build Pipeline
Fail Fast
32. CPU -Running Tests Concurrently
•Create parallel jobs.
Divide and Conquer
33. CPU -Running Tests Concurrently
•Distribute tasks across multiple slaves.
Divide and Conquer
Image source -https://wiki.jenkins-ci.org/download/attachments/2916393/logo.png?version=1&modificationDate=1302753947000
34. CPU -Running Tests Concurrently
Image source - http://sharpreflections.com/wp-content/uploads/2012/06/multi_core_cpu.png
•Using @RunWith(ConcurrentJunitRunner.class).
–Curtesy -Mathieu Carbou http://java.dzone.com/articles/concurrent- junit-tests
–Maven Surefire plugin has built in mechanism.
Divide and Conquer
35. Focus on the Bottlenecks
Divide and Conquer
Key Principles to Speed Up Your Build Pipeline
Fail Fast
36. Restructure The Build Pipeline
Image Source - http://javapapers.com/wp-content/uploads/2012/11/failfast.jpg
•We want our builds to give us fast feedback. Hence it is very important to prioritise your build tasks based on what is most likely to fail first.
•Push unnecessary stuff to a separate build –Things like JavaDocs can be done nightly.
•Separate out fast and slow running tests.
Fail Fast
37. Incremental Build vs. Clean Build
•Local dev builds are incremental, instead of clean builds, as it helps with faster feedback and fail fast.
Fail Fast
38. Prioritize Test
•We prioritize and group our tests so that the tests which are fast and mostly likely to fail are run first.
–ProTest framework
Fail Fast
39. Summary
•Focus on bottlenecks
–Avoid Disk IO -File operations, file based database operations.
–Use smaller datasets.
–Use in-memory databases, Ram Drives, SSDs.
–Perform CPU profiling, scan logs, to uncover the unknown.
–Verify build tool settings.
•Divide and Conquer
–Create smaller jobs that can run in parallel.
–Distribute jobs across multiple slaves.
–Write tests that can run in isolation and use ConcurrentJunitRunnerto run them in parallel.
•Fail Fast
–Restructure the build pipeline to uncover failures soon.
–Incremental Builds
–Prioritize tests.
40. Build Time Vs No Of Builds
Removed Workspace Duplication
Ant Junit Task –Fork Once
Ram Disk
Caching Resource
Caching Spring Context
Avoided Email
Joda DateTime
Deprecated Date API
Concurrent Junit Runner
41. Impact on life
Image source - http://t3.gstatic.com/images?q=tbn:ANd9GcTCvK8pY5qcp7Gl3ZBjxN1mc1HVHdiy1sQhByKeGgUk_5eJuUk7cA
https://encrypted-tbn3.gstatic.com/images?q=tbn:ANd9GcQpoUXqhEpdGl1cLzn4gQsng_GyxUmOKWxYUH6GfrjN_FRUYPxw-Q
42. Resources
•Jenkins –http://jenkins-ci.org/
•CI –http://en.wikipedia.org/wiki/Continuous_integration
•Mklink –http://technet.microsoft.com/en-us/library/cc753194.aspx
•http://ant.apache.org/manual/Tasks/junit.html
•http://java.dzone.com/articles/javalangoutofmemory-permgen
•SSD –http://en.wikipedia.org/wiki/Solid-state_drive
•Hybrid disk –http://en.wikipedia.org/wiki/Hybrid_drive
•HSQL –http://hsqldb.org/
•H2 –http://www.h2database.com/html/main.html
•Memsql –http://www.memsql.com/
•MySQL is bazillion times faster than MemSQL
•Tmpfs –http://en.wikipedia.org/wiki/Tmpfs
•http://blog.laptopmag.com/faster-than-an-ssd-how-to-turn-extra-memory-into-a-ram-disk
•RAM Disk Software Benchmarked
•http://jvmmonitor.org/
•http://searchvmware.techtarget.com/tip/VMware-snapshot-size-and-other-causes-for-slow- snapshots
•http://blogs.agilefaqs.com/2014/10/03/key-principles-for-reducing-continuous-integration-build- time/
•http://googletesting.blogspot.com/2011/06/testing-at-speed-and-scale-of-google.html
•http://www.infoq.com/presentations/Development-at-Google
•http://crystalmark.info/software/CrystalDiskMark/index-e.html