1. Transforming Traditional Enterprise
Software Development Processes
“Appling DevOps and Agile Principles at Scale”
Gary Gruver
September 3, 2014

2. - FW no longer a
bottleneck for the
business
- Development costs
reduced from
$100M/yr. to $55M/yr.
- ~140% increase in the
number of products
under development
- Capacity for innovation
increased from ~5% to
~40%

3. Requirements Scheduling Development
Integration &
Qualification
Waterfall Development Model
Resources
ScheduleScope
Waterfall Approach: Do
Enough up front planning
To lock in all three corners

4. Requirements Scheduling Development
Integration &
Qualification
Waterfall Development Model
Resources
ScheduleScope
Waterfall Approach: Do
Enough up front planning
To lock in all three corners
Waterfall Reality: Discoveries
during development and difficulties
with integration/qualification results
in frequent and costly schedule slips

5. Agile Development Model
Drops of working
Complete code on
A fixed interval
Plan
Qual Dev
Plan
Qual Dev
Plan
Qual Dev
Plan
Qual Dev
Drops of working
Complete code on
A fixed interval
• Fully qualified code drops on a fixed schedule
• Frequent small integrations
• Short planning and development cycles
• Delivering to a well prioritized product backlog

6. Scrum Agile≠

7. Water Scrum Fall

8. Transforming Traditional Enterprise Software
Development Processes
Enterprise Level
Continuous
Improvement
Planning &
Prioritized
Backlog
Applying DevOps
Principles at
Scale
Business
Objectives

9. Business Objectives (Don’t “Do Agile”)
Define
your value
proposition
Understand your
cost & cycle-time
drivers
Either automate, eliminate, or engineer out the
drivers that aren’t key to the value prop

10. Mini-milestone
Objectives
Cascading Objectives
to Track Progress
ConversationsLearnings
Agile Adjustments
Having real-time
metrics is essential for
the speed of agile &
aligning the org.
But don’t manage by
metrics.
Use the metrics to
understand where to
have conversations
about what is not
getting done.
Interative Approach to Agile Management

12. Long Term Predictability for SW Schedules
Do we really need the predictability of our current planning processes?
Are our current planning processes really that accurate?
50% of all SW is never used or does not meet the business objectives!
100%
Accuracy
Planning Investment

13. One of the biggest challenges with
Agile Planning at the enterprise
level is getting the organization to
accept the uncertainty in SW
development and appreciate the
flexibility and opportunity.

14. Locking in Capacity over time
100%
Capacity
Time Horizon
Long-range commitments < ~50% of Capacity
Mid-range commitment < ~80%
of Capacity
Delivery

15. Transforming Traditional Enterprise Software
Development Processes
Enterprise Level
Continuous
Improvement
Planning &
Prioritized
Backlog
Applying DevOps
Principles at
Scale
Business
Objectives

16. Business Objectives
Increase the quality and frequency of feedback
Reduce the time and resources between release branch & production
Improve deployment repeatability/env. stability
Testing in an Operational like environment as close to dev. as possible

17. Release testing considerations
Cost of Failure
Ease of Deployment
Architectural
Test Env =Production

18. DevOps unique challenges
Simulators
Emulators
Embedded SW/FW
Upgrade
Compatibility
Patches
Packaged SW
Continuous
Delivery
Web/SaaS

19. Ensuring the architecture/build system is ready
Comp A v1 Comp B v1
Comp D v1 Comp E v1 Comp F v1
Comp C v1Comp B v2Comp B v1Comp A v2
Comp D v2 Comp E v2
Comp C v2
Comp F v2

20. Automated system test architecture
• Component based approach to the automated test architecture
Data Magic
Navigation
Page Objects

21. Finding the offending code
What Code?
When? Are you
sure it wasn’t Bob?

22. Building up a Large SW System
InterfaceTestSimulator
Agile
Comp 1
Agile
Comp 2
Agile
Comp 3
Agile
Comp 4
Agile
Comp 5
Agile
Comp 6
Legacy
Waterfall
IT 1
Legacy
Waterfall
IT 2
Legacy
Waterfall
IT 3
Agile
Comp 1

23. Auto-revert/Gated Commits

24. Continuous Integration and Test System
L1 Sim
(10-14x/day)
L2 Sim
(12x/day)
L3 Emu
(6x/day)
L4 Sim
(1x/day)
STAGE 1
STAGE 2

25. Embedded System Pipeline
Cost of Testing
Drive test escapes upstream
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 2 3 4 5 6
Sim. Quality bar
Emulator
Sim. Regression
Product
ProductEmulatorSimulator

26. Building up the Enterprise
Gating at the Application
or Artifact
InterfaceTestSimulatorInterfaceTestSimulator
Gating at the SCM
Gating at the SCM

27. Developing on Trunk

28. CD - Picking the right tool for the job
Orchistrator
Trigger
Scripte
d
Env
Deploy
ment
EDD
Auto
Testing

29. Scripting Environments - Variances
Dev/QA PRE-PROD PERFORMANCE PRODUCTION
Common Script Common Script Common ScriptCommon Script
Environment descriptors

30. Dev/QA PRE-PROD PERFORMANCE PRODUCTION
Comm
on
Script
Common ScriptCommon ScriptCommon Script
Environment descriptors
Deploy Workflow Complexity
Dev/QA PRE-PROD PERFORMANCE PRODUCTION
Comm
on
Script
Common ScriptCommon ScriptCommon Workflow
Environment descriptors

31. Evolutionary Database
-Add deprecate
-Lazy Instantiation
-Don’t alter or Modify
-It is like crossing the beams

32. Finding the offending component quickly improves production
Configure Servers/Routing Device1:
Deploy Code2:
Run a System Test3: ? ? ? ? ? ? ? ? ? ? ? ?

33. Finding the offending component quickly improves production
Configure Servers/Routing
Device and Validate Data
1:
Deploy Code & Validate
Successful Deployment
2:
Run a System Test3: Code Validated

34. Transforming Traditional Enterprise
Software Development Processes
“Applying DevOps and Agile Principles at Scale
E-mail: gbgruver@gmail.com
Blog: largescaleagile.com
Twitter: @GRUVERGary