1. Jenkins Evolutions
Anton Arhipov
ZeroTurnaround
JRebel
@antonarhipov
http://arhipov.blogspot.com

2. Why Evolutions?

3. Why Evolutions?
test automation

4. Why Evolutions?
continuous
integration

5. Why Evolutions?
continuous
deployment

6. Why Evolutions?
cron on steroids

7. Agenda
Background
Stories
• Functional tests
• Servlet tests
• Scaling
Random stuff

8. Background - 2008
First installation
Clicked “Run” when wanted to test
stuff

9. Background
2012
54 215 324 jobs
2 5 8 dedicated machines
Amazon ~7000 instance-hour per month
3 in house plugin developed
Custom dashboard (50 x 300 grid)

11. https://github.com/
jenkinsci/
jenkins-scripts

12. Custom Dashboard

13. story of
functional tests

14. JRebel Functional JDK Tests

15. JRebel Functional JDK Tests

16. JRebel Functional JDK Tests
Per JDK
• Per functional test
• Compile couple of classes
• Run some methods and assert
• Recompile couple of classes
• Run some methods and assert
• …

17. JRebel Functional JDK Tests
Per JDK
• Per functional test
• Compile couple of classes
• Run some methods and assert
• Recompile couple of classes
• Run some methods and assert
• …

18. JRebel Functional JDK Tests
Per JDK
• Per functional test
• Compile couple of classes
• Run some methods and assert
• Recompile couple of classes
• Run some methods and assert
• …

19. JRebel Functional JDK Tests
Per JDK
• Per functional test
• Compile couple of classes
• Run some methods and assert
• Recompile couple of classes
• Run some methods and assert
• …

20. JRebel Functional JDK Tests
Per JDK …
Which Jenkins feature is the
best here?

21. JRebel Functional JDK Tests
Configuration matrix FTW!

22. JRebel Functional JDK Tests
Executed with a Groovy script

23. JRebel Functional JDK Tests

24. JRebel Functional JDK Tests
How to get results? Jenkins
feature…

25. JRebel Functional JDK Tests
Output JUnit XML!

26. JRebel Functional JDK Tests

27. JRebel Functional JDK Tests
Creates ~ 10 000 new files
Compile ~ 1700 java classes
Invoke the JVM ~ 550 times

28. JRebel Functional JDK Tests
Sloooooooow!

29. JRebel Functional JDK Tests
I/O
CPU
Memory

30. JRebel Functional JDK Tests

31. JRebel Functional JDK Tests
Bare metal
Cloud
Bare metal (+ cloud)

32. JRebel Functional JDK Tests
Cloud
• 1 small instance per JDK
• Manual update of AMIs
• Slow
• Scales!

33. JRebel Functional JDK Tests
Which Jenkins plugin to use?

34. JRebel Functional JDK Tests
Amazon EC plugin

35. JRebel Functional JDK Tests
Bare metal
• 8 core & 16GB RAM gaming
machines
• Recycled developer machines
• Marketing demo machines (iMac)
• Developer machines, ad-hoc (?)

36. Jenkins
Master

37. *nix
*nix
*nix
*nix
*nix
Jenkins
Master
Windows
Windows
Windows

38. *nix
*nix
*nix
*nix
*nix
Jenkins
Master
Windows
Windows
Windows

39. *nix
*nix
*nix
*nix
*nix
Jenkins
Master
Windows
Windows
Windows
iMac
My
Laptop

40. Adding new machines
How to manage the
configurations?
How to automate the setup?

41. Adding new machines
Chef, Puppet

42. Taming I/O
SSD: ~30% performance gain

43. story of
servlet tests

44. Servlet Tests
Start a container version
• WebSphere, Weblogic, JBoss, etc
Deploy applications
Test results, reload changes, test
results, …

45. Custom Dashboard

46. Servlet Tests
How to start a server?

47. Servlet Tests
Duh! With a startup script!

48. Servlet Tests
Problem: 1 app server, 2 jobs.
Which plugin to use?

49. Servlet Tests
Throttle Concurrent Builds
plugin

50. Servlet Tests

51. Servlet Tests
How do you deploy an
application?

52. Servlet Tests
Start Tomcat
Copy WAR files to webapps/
GET /some-url
Assert
Update application
GET /some-url
Assert

53. Servlet Tests
Sloooooooow!

54. Servlet Tests
I/O
CPU
Memory

55. Testing Multiple Containers
Cloud
• 1 small instance per container
• Need to update AMIs
• Slow
• Isolation

56. Testing Multiple Containers
Single machine
• Port hell
• Memory limits
• Fast(er)!

57. story of
DVCS

58. Story of DVCS
Problem
• Many servers
• Many containers
• Many tests
• Many JDKs
Slow feedback cycle

59. Story of DVCS
Tests green?
Push to branch/repository
DEV STABLE
N tests on N servers M tests on M servers
(30 minutes) (3 hours)

60. Story of DVCS

61. Story of DVCS

62. scalability

63. Story of Scaling
Problem
• Exponential growths of tests
• (containers) X (configurations) X (tests) + const
• Fight for executors

64. Story of Scaling
Add more machines?
Spin up more cloud instances?
Attach developer machines at
night?
Cut tests scope?

65. Story of Scaling
Jenkins helps with elasticity!

66. Story of Scaling
Jenkins helps with elasticity!
WIN!

67. random stuff

68. Random Stuff
There’s an app plugin for that!

69. Some Cool Plugins
Mercurial plugin
Throttle Concurrent Builds plugin
Matrix Configuration plugin
Matrix Reloaded plugin
SSH slaves plugin
Publish over SSH plugin
Copy Artifact plugin
Copy to slave plugin

70. Some Cool Plugins:
Continuous Deployment
Build Pipeline plugin
Parametrized Trigger plugin
Downstream Buildview plugin
Fingerprints
http://zeroturnaround.com/blog/how-to-use-jenkins-for-job-chaining-and-visualizations/

71. Some Cool Plugins
Chuck Norris plugin
BUILD
SUCCESSFUL

72. Some Cool Plugins
Chuck Norris plugin
BUILD FAILED

73. Build-it-all!
JRebel

74. Build-it-all!
JRebel

75. Build-it-all!
IntelliJ
NetBeans
JRebel
Eclipse
Installer

76. Build-it-all!
IntelliJ
NetBeans
JRebel
Eclipse
Installer

77. Build-it-all!
IntelliJ
NetBeans
JRebel
Eclipse
Installer
Tests

78. UI

83. Questions?