Paychex, a recognized leader in the payroll, human resource, and benefits outsourcing industry, found that the demand for application deployments had increased beyond what could be supported by manual configuration. Keeping up with this demand required a shift from manually providing a service to developing an automated platform for self-service resulting in a culture change with new partnering across their DEV, OPS and Architecture teams. David Jozis, Automation Engineer at Paychex, discusses the challenges they encountered when making these significant changes and how they were able to overcome them to accomplish 5x as many deployments as before.

1. Standardizing and Automating
Deployment Configuration
Paychex: David Jozis

2. Housekeeping
 This webinar is being recorded
 Links to the slides and the recording will
be made available after the presentation
 You can post questions via the
GoToWebinar Control Panel
 Follow the conversation on Twitter

3. About me
 David Jozis
 15+ years experience in coding
 Professionally for about 8 years
 Enjoying the innovation culture at Paychex for
just over 3 years
 1 year as a developer
 2 years in in the EBA group at Paychex
(Enterprise Build Automation)
 Technology/coding is both my job and my
passion
3

4. Agenda
 Introduction and context
 Our build/deploy pipeline
 What was wrong with how we configured it
 What we did to configure it faster
 Standardization
 Automation
 What we learned
 This is not about how we sped up deployments, this is about how we sped up configuring new
deployments
4

5. Introduction and
Context

6. About Paychex
 Industry leader in Human Capital Management (HCM)
services with solutions for payroll, human resources,
insurance, and retirement
 Established 1971
 Headquartered in Rochester, NY; 100+ U.S. locations
 600,000+ small- to medium-sized business clients
 Technology-enabled service company
 24/7/365 days per year support
 13,000+ Employees
 1,500 IT professionals
 Paychex Flex | www.paychex.com
6

7. The Enterprise Build Automation team (EBA)
 A “dev service” team
 Develops/maintains/supports the build/deploy/release pipeline for consistent re-use across Paychex
 We don’t write business application code
 We don’t run the releases
 Configure and write plugins for Bitbucket, Jenkins, Gradle, XL Deploy, XL Release
 Made up of 17 members, 3 agile teams
 Serves ~1,500 IT staff, ~80 agile development teams
 ~90:1 staff ratio
 We interact with lots of different teams
7

8. Our tools
8
Left to right: earlier to later in build/deploy/release pipeline

9. Build/Deploy
Pipeline Challenges
The challenges with manually
configuring your automated platform

10. Previous State: Paychex Build/Deploy Pipeline
Environment
Infrastructure
Git repository
manager/store
Runs build,
publishes to
XL Deploy
Stores artifacts and
deployment metadata
10

11. Challenges around Bitbucket phase
 Application naming inconsistent with Jenkins job or XL Deploy application
 Fragile link between where build artifacts are created and where Jenkins looks for them (maintained in
two places)
 Build/test logic in Jenkins is different from running locally
11

12. Challenges around Jenkins phase
 Definition of XL Deploy metadata done through the Jenkins UI
 No testing outside of Jenkins (locally building XLD packages?)
 No loops or other dynamic configuration
 Copy/paste mistakes
 Not source-controlled – makes change tracking more difficult
 No consistency enforced
 Filling up XL Deploy with build artifacts, not just metadata
 Dev have access to misconfigure things, such as no tags
12

13. Challenges around XL Deploy phase
 Configuration of “Infrastructure” and “Environments” in XL Deploy done manually
 Each team member of our shared services team configured applications slightly differently
 Tag usage inconsistent – mainly based around infrastructure
 Infrastructure based tags problematic for new datacenter with phased onboarding of applications
 Missing information in onboarding requests
 Needed to reconfigure every WebLogic deployment for a phased technology upgrade – too slow
13

14. Challenges around real infrastructure
 Cluster naming inconsistent
 Incorrect information turned over
 WebLogic admin servers down
 The infrastructure does not actually exist
 Uses slightly different configuration due to manual setup
 Deployment failures due to inconsistent/incorrect configuration
14

15. 15
I need my application configured for deployment
Developer EBA member
What sort of application? Does it have properties file, static web content?
It’s a WebLogic application. I think it needs these files.
I only found a cluster “my-exmpl”, is it that? Also, web-server-x doesn’t exist.
Which clusters does it go to? Which webservers do the static content go to?
It’s my-example-cluster and web-server-x.
I don’t know. CC: WebEngineer. (Discussion gets lost between teams.)

16. Summary of struggles
 Lack of consistency
 Lack of source of truth
 Lack of accountability
 Ineffective communication, too much back-and-forth, too many handoffs, etc.
 Too slow
16

17. Standardization and
Automation
What we did and what we learned

18. Inspiration from Netflix
 Nebula is an opinionated set of plugins for the Gradle build system created by Netflix
 Code is built and tested locally using Nebula
 Changes are committed to a central Git repository
 A Jenkins job executes Nebula, which builds, tests, and packages the application for
deployment
 Gradle chosen because:
 easy to write testable plugins
 reducing the size of a project’s build file
 More reading: https://medium.com/netflix-techblog/how-we-build-code-at-netflix-c5d9bd727f15
18

19. Inspiration from Amazon
 Mandate at Amazon, from around 2002
 All teams will henceforth expose their data and functionality through service interfaces
 Teams must communicate with each other through these interfaces
 It doesn’t matter what technology they use
 How to apply this to our manual onboarding process?
 More reading: https://apievangelist.com/2012/01/12/the-secret-to-amazons-success-internal-apis/
19

20. Standardization
 A standard that isn’t enforced by code is a suggestion
 Developed standard with agreement from
 Architecture
 Development
 Infrastructure/Ops teams
 Wrote a Java library that does standard validation
 Reusable set of classes with common methods for validating different standardized elements
 Different classes for different standards – WebLogic, OpenShift, .NET, Java stand-alones,
Pro*C artifacts, etc.
 Throws helpful errors that guide the user to adherence
20

22. Totem
 The Onboarding Tool EBA Made
 Java/GWT (Google Web Toolkit) Web Application
 Exposes scripts that do what we used to do manually
 Consistently
 Automatically
 Instantly
 Has both Web UI and REST endpoints for every script
 Absolutely minimal inputs
 Didn’t use Jenkins Jobs primarily to get finer grained permissions control
 Example: Anyone can run a “validate” on a script, but only LDAP group X can run Execute
22

23. Totem – How it works
 Reads information from canonical source of truth in source control
 Checks the source of truth against the real world for consistency
 Uses the standards library to protect against non-standard configuration and input
 Application naming
 Tags
 Infrastructure configuration
 Permissions
 Detects current state and only does what is necessary – can rerun
23

24. Log in
Select a script
Fill out input
Execute!

25. Wins
 Enabled the customers!
 No more EBA team involvement with onboarding common technologies
 Culture change: Instead of providing a service, develop a platform for self-service
 Much more scalable
 5x more onboardings today than 10 people could perform in the same time
 No deployment failures due to inconsistent XL Deploy configuration
 Application specific tags means we control where things go
25

26. Shift left
 Shift validation to run sooner
 Created a Gradle plugin for publishing to XL Deploy
 Parsed that metadata, applied the same standards library
 Get build failures in Jenkins due to standards violations
 Shift validation to run sooner again
 Run the same Gradle script locally, before check-in
 Get failures before committing configuration
26

27. Package task
Publish task
No duplicate maintenance of artifact paths
Generate/validate application specific tags

28. Additional improvements
 Dynamic/flexible configuration
 Can manipulate artifacts during the publish script to publish slightly different versions
 Can publish in a loop
 Used the above to publish slightly different artifacts for internal/external service deployments
 They can be separately deployed, and be tagged separately
 Created a Gradle plugin to publish the artifacts to Artifactory
 No configuration, apply the plugin and it handles the rest dynamically by reading the XL Deploy
package configuration (and updating it)
 Make the Gradle validation plugin check Jenkins job name and Bitbucket repository name in the future?
28

29. Current State: Paychex Build/Deploy Pipeline
Environment
Infrastructure
Git repository
manager/store
Runs build,
runs Gradle
publish script
Publishes build
artifacts to
Artifactory,
and package to
XL Deploy
Stores binaries
Stores deployment
metadata
29

30. 30
I’ve configured my Application for deployment. Configure the Infrastructure?
Developer Totem
I checked Web Engineering’s files in source control. Because of your
application name, I can see that it goes to servers X/Y, but X doesn’t
exist in reality, and there’s another server Z that is in that cluster. Please
work with them to resolve it and try again.
I’ve configured my Application for deployment. Configure the Infrastructure?
Done. Here’s all the changes I made.

31. Efficiency gains
 One EBA team’s velocity in 2015 was about 15
points per 2 week sprint
 3 onboardings per sprint with nothing else
 Totem puts through ~147 points worth of
onboardings per 2 weeks
 ~10x more throughput
 Equivalent of ~40 EBA members
 Doesn’t need EBA involvement
 EBA team now spends their time on projects that
will advance Paychex CI/CD capabilities
 Velocity has increased 3x or more
 Much less back-and-forth
0
20
40
60
80
100
120
140
160
One (of the two) EBA teams in
2015
Totem in 2017
Story points worth of onboardings per 2
weeks
Story points worth of onboardings per 2 weeks
31

32. Lessons learned
 A standard that isn’t automatically enforced is a suggestion
 Groovy DSLs can offer great flexibility for configuration
 Self-service apps needs both a UI and REST API
 Good automation can result in buy-in and culture change
 Demand scales to meet supply
 Everyone expects everything Just In Time now
 More than 5x output required than 6 months ago
 Assumptions will be made about what it does or doesn’t do
 Fear of the unknown – training and communication are important!
32

33. Lessons learned
 After you improve, get ready to improve again
 Eat your own dogfood
 We build and deploy Totem via the same pipeline
 Good abstractions give you convenient points to extend from
 We’re using the same Gradle scripts now to gather code coverage metrics and associate them with
artifacts
 We were able to trivially add publishing binaries to Artifactory instead of XL Deploy because we put
our configuration for deployments in Gradle
 Take what you do today, write it in code, then iterate
 No need to start with perfection
33

34. Questions?
 Don’t miss our next webinar – Nov. 7th
 Guest speaker: Robert E Stroud CGEIT CRISC –
Principal Analyst, Forrester Research

35. 35
Thank you!
▪ Don’t miss our next webinar – Nov. 7th
▪ Guest speaker: Robert E Stroud CGEIT CRISC – Principal Analyst, Forrester Research