"Increasing Agility by Understanding Risk" given at Puppet Camp London April 2014 by Simon Croome

1. Increasing Agility
by Understanding Risk
Simon Croome
simon@croome.org

2. $ whoami
Simon Croome
Infrastructure Engineer
Financial Industry
Independent Consultant

3. Why Puppet?

4. Velocity

5. Configuration
Management is only part
of the solution

7. Software is deployable
throughout its lifecycle

8. Software is deployable
throughout its lifecycle
!
Priority is keeping the
software deployable over
new features

9. Software is deployable
throughout its lifecycle
!
Priority is keeping the
software deployable over
new features
!
Anybody can get fast,
automated feedback on
the production readiness
of their systems any time
a change is made

10. Software is deployable
throughout its lifecycle
!
Priority is keeping the
software deployable over
new features
!
Anybody can get fast,
automated feedback on
the production readiness
of their systems any time
a change is made
!
Push-button deployments
of any version to any
environment

11. Continuous Delivery

12. Continuous Delivery
Reduces the transaction cost of making change

13. Continuous Delivery
Reduces the transaction cost of making change
Faster ROI

14. Continuous Delivery
Reduces the transaction cost of making change
Faster ROI
Reduces Risk

15. Risk

16. Concept: John Allspaw
http://blog.vincentbrouillet.com/costs-and-risks-benefits-of-
continuous-delivery-in-one-picture/
!

17. If it hurts
Do it more often

18. Controls

19. Controls
Change Management

20. Controls
Change Management
Separation of Duties

21. Controls
Change Management
Separation of Duties
Audit

22. An example workflow

23. Jira Ticket Engineering
Puppet
Data
Code
Branch
Puppet
Modules
Test
Servers
Dev
Puppet
Engineering
UNIX team member is assigned ticket.
Jira creates development sandbox using Stash
integration.
Peer
Review
Code
Format
Tests
Integr’tn
Tests
Unit
Tests
Code Release
Once code is merged into Stash’s Release Branch, a UNIX team
member may create a release containing one or more changes.
Bamboo automates deployment of the release to UAT, further
automated testing, then release to Production.
Note that it will not yet be deployed to client systems.
Create
Release
Deploy to
UAT
Integration
Tests
Deploy to
PROD
New Requirement
Ticket is raised against
UNIX Jira Issue tracker,
e.g. New project, change
to OS build.
Test failures block the release.
Initiate Deployment
Release is selected by
UNIX team member.
Select
Release
Dry-run
Change
Detail
Impacted
Roles
Build
Status
Non-
Impacted
Servers
Impacted
Servers
Manual
Review
Impact Analysis
Change is run across the environment in read-only mode.
Servers that would be changed report back changes.
UNIX Change Manager assesses impact and assigns risk.
Scheduling & Change Management
Each impacted server role enters separate change management
workflow
Maintenance windows and change risk feed into scheduling.
DEV
SIT
DIT
EBF
PERF
EBF
Impacted
Roles
BCP
PROD
CAT
Deployment
Completed approvals trigger deployment during change
window.
.
Approved
CHG Tickets
Apply
Change
Review
Request
Risk
Maintenance
Windows ServiceNow
CHG
Tickets
Reporting
Web console to track deployment of changes across the environment, including summary view of
health by server role and business area.
Un-approved tickets reset
change workflow.
Code Review
UNIX team member creates “Pull Request”, indicating
change is ready to be reviewed.
Bamboo automates code quality review, unit and
integration testing.
If all tests pass, Engineering team alerted to perform
peer review in Stash.

24. Jira Ticket Engineering
Puppet
Data
Code
Branch
Puppet
Modules
Test
Servers
Dev
Puppet
Engineering
UNIX team member is assigned ticket.
Jira creates development sandbox using Stash
integration.
New Requirement
Ticket is raised against
UNIX Jira Issue tracker,
e.g. New project, change
to OS build.
Test fail

25. Jira Ticket Engineering
Puppet
Data
Code
Branch
Puppet
Modules
Test
Servers
Dev
Puppet
Engineering
UNIX team member is assigned ticket.
Jira creates development sandbox using Stash
integration.
New Requirement
Ticket is raised against
UNIX Jira Issue tracker,
e.g. New project, change
to OS build.
Test failures block the release.
Review
Request
Code Review
UNIX team member
change is ready to b
Bamboo automates
integration testing.
If all tests pass, Eng
peer review in Stash

26. Puppet
Data
Puppet
Modules
et.
sing Stash
Peer
Review
Code
Format
Tests
Integr’tn
Tests
Unit
Tests
Code Release
Once code is merged into
member may create a rel
Bamboo automates deplo
automated testing, then r
Note that it will not yet be
Create
Release
Deploy
UAT
Test failures block the release.
Review
Request
Code Review
UNIX team member creates “Pull Request”, indicating
change is ready to be reviewed.
Bamboo automates code quality review, unit and
integration testing.
If all tests pass, Engineering team alerted to perform
peer review in Stash.

27. Peer
Review
Code Release
Once code is merged into Stash’s Release Branch, a UNIX team
member may create a release containing one or more changes.
Bamboo automates deployment of the release to UAT, further
automated testing, then release to Production.
Note that it will not yet be deployed to client systems.
Create
Release
Deploy to
UAT
Integration
Tests
Deploy to
PROD
st”, indicating
unit and
to perform

28. Initiate Deployment
Release is selected by
UNIX team member.
Select
Release
Dry-run
Change
Detail
Impacted
Roles
Build
Status
Non-
Impacted
Servers
Impacted
Servers
Manual
Review
Impact Analysis
Change is run across the environment in read-only mode.
Servers that would be changed report back changes.
UNIX Change Manager assesses impact and assigns risk.

29. nitiate Deployment
Release is selected by
UNIX team member.
Select
Release
Dry-run
Change
Detail
Impacted
Roles
Build
Status
Non-
Impacted
Servers
Impacted
Servers
Manual
Review
Impact Analysis
Change is run across the environment in read-only mode.
Servers that would be changed report back changes.
UNIX Change Manager assesses impact and assigns risk.
Scheduling & Ch
Each impacted serv
workflow
Maintenance window
DE
Impacted
Roles
Reporting

30. nge
ail
ld
tus
Manual
Review
read-only mode.
ack changes.
t and assigns risk.
Scheduling & Change Management
Each impacted server role enters separate change management
workflow
Maintenance windows and change risk feed into scheduling.
DEV
SIT
DIT
EBF
PERF
EBF
Impacted
Roles
BCP
PROD
CAT
Deployment
Completed app
window.
.
Approved
CHG Ticke
Risk
Maintenance
Windows ServiceNow
CHG
Tickets
Reporting
Un-approved tickets r
change workflow.

31. Management
nters separate change management
hange risk feed into scheduling.
T
EBF
PERF
EBF
BCP
PROD
Deployment
Completed approvals trigger deployment during change
window.
.
Approved
CHG Tickets
Apply
Change
Risk
Maintenance
Windows ServiceNow
CHG
Tickets
Un-approved tickets reset
change workflow.

32. DEV
SIT
DIT
EBF
PERF
EBF
Impacted
Roles
BCP
PROD
CAT
Approved
CHG Tickets
Risk
Maintenance
Windows ServiceNow
CHG
Tickets
Reporting
Web console to track deployment of changes across the environment, including summary view of
health by server role and business area.
Un-approved tickets reset
change workflow.

33. Jira Ticket Engineering
Puppet
Data
Code
Branch
Puppet
Modules
Test
Servers
Dev
Puppet
Engineering
UNIX team member is assigned ticket.
Jira creates development sandbox using Stash
integration.
Peer
Review
Code
Format
Tests
Integr’tn
Tests
Unit
Tests
Code Release
Once code is merged into Stash’s Release Branch, a UNIX team
member may create a release containing one or more changes.
Bamboo automates deployment of the release to UAT, further
automated testing, then release to Production.
Note that it will not yet be deployed to client systems.
Create
Release
Deploy to
UAT
Integration
Tests
Deploy to
PROD
New Requirement
Ticket is raised against
UNIX Jira Issue tracker,
e.g. New project, change
to OS build.
Test failures block the release.
Initiate Deployment
Release is selected by
UNIX team member.
Select
Release
Dry-run
Change
Detail
Impacted
Roles
Build
Status
Non-
Impacted
Servers
Impacted
Servers
Manual
Review
Impact Analysis
Change is run across the environment in read-only mode.
Servers that would be changed report back changes.
UNIX Change Manager assesses impact and assigns risk.
Scheduling & Change Management
Each impacted server role enters separate change management
workflow
Maintenance windows and change risk feed into scheduling.
DEV
SIT
DIT
EBF
PERF
EBF
Impacted
Roles
BCP
PROD
CAT
Deployment
Completed approvals trigger deployment during change
window.
.
Approved
CHG Tickets
Apply
Change
Review
Request
Risk
Maintenance
Windows ServiceNow
CHG
Tickets
Reporting
Web console to track deployment of changes across the environment, including summary view of
health by server role and business area.
Un-approved tickets reset
change workflow.
Code Review
UNIX team member creates “Pull Request”, indicating
change is ready to be reviewed.
Bamboo automates code quality review, unit and
integration testing.
If all tests pass, Engineering team alerted to perform
peer review in Stash.

36. Technology Stack
!
Puppet - Dynamic environments
MCollective - Comms
CouchDB - Reports
PouchDB - Replication to browser
AngularJS - Web interface
Rails - API (to be replaced)
Resque / Redis - Job scheduling
!

37. github.com/croomes/gonzo
!
Questions?