Software MTTR: Continuous Delivery Reduces Repair Time

+
Software MTTR: the Path
from Continuous Integration
to Continuous Delivery
Jeff Sussna, Ingineering.IT

Software Mean Time to Repair
Bug-Hours: MTTR for software
Enhancement requests are bug reports
 Gap between actual and desired value
 All software development is “repair”
Continuous Delivery = Continuous Repair = Minimize MTTR
Copyright © 2012 Ingineering.IT, LLC. All Rights Reserved.

Change is Good/Change is Bad
Change is good: how we deliver value
Change is bad: introduces uncertainty and often failure
Continuous Delivery answers its own question
 Reduces uncertainty
 Increases rate of value delivery

Continuous Delivery is Lean
Small batch sizes
Just-in-time production
Andon cord

Batch Size
Batch size: unit of work passing between process stages
Small batch sizes:
 Reduce complexity and risk
 Increase efficiency
 Lower overhead
 Reduce cycle time
 Speed up feedback loops

Just-in-time Production
Only make what’s needed, when and how it’s needed
Avoid wasteful inventory
Applies to:
 Features
 Designs
 Code
 Tests
 Infrastructure

Andon Cord
Any worker can stop the production line
Treat the next step in the pipeline as your customer
Improve quality and efficiency by fixing defects now not later

Agile is Lean
Batch size: one year => one month
JIT production: groom the backlog each iteration
Andon cord: continuous integration

Sources of Software Failures
Unit errors
 Requirements misunderstanding
 Logic errors
 Design errors
Integration errors
 Hard due to inherent complexity
 Delayed/batched due to fear
Irony: large integration batch sizes increase level of difficulty

Continuous Integration
Check into trunk often
Build on every commit
Test on every build
Notify on every error
Fix every broken build
Necessitates build/deploy/test/report automation

Value of Continuous Integration
Reduces integration batch size and latency
Enables Andon cord
Pulls QA into development
Removes waste/variation from manual build/deploy/test/report
Minimizes “merge-hell” due to long-lived branches

Redefining “Done”
Agile says nothing about getting product in customer’s hands
Software product era: cost of delivery is high for all
 Vendor: burn and mail CD
 Customer: upgrade to new version
Cloud changes the development/delivery equation
 Customer’s delivery cost shrinks
 Development org IS the delivery org
Done: “the customer has the change and is satisfied with it”
 Kind of like any other service industry

Continuous Delivery
Finishes what Continuous Integration starts
Minimizes friction in the flow from integration to satisfaction
Forces end-to-end batch size reduction

Doing Continuous Delivery
Small batch sizes all the way to prod
Automation all the way to prod
Decoupled technical and non-technical deployment
 Deployed in prod != visible to customer
Shift your focus from iterations to changes
 Think in terms of bug-hours

Small Batch Sizes
Don’t batch testing ‘til the end of the iteration
Pipeline testing stages
 Acceptance
 Security
 Performance/Scalability/Fault-tolerance
 UAT
 Fail as far upfront as possible
Don’t batch release‘til the end of iteration
 Hotfixes go away!

Automation
Human gates are OK
 When/why to move forward
Manual labor is not OK
 Acceptance testing
 Non-functional testing
 Deployment throughout the pipeline
Deal in atomic releases
 DB + Code + Infrastructure
 Deployment + rollback

Decoupled Deployment
Techniques:
 Feature toggles
 Feature branches
 Application componentization
Business-defined release: the ultimate goal
 Canary
 A-B
 Demographic/Geographical
Multi-tenancy for code

Code Multi-Tenancy as Strategy
Empower the business: release at 12:01 Christmas morning
Free the developers: don’t deploy at 12:01 Christmas morning
1st-class “non-functional” requirement
 Job is to deliver service
 Biz ops needs control over features as part of customer interaction
 Should be fundamental part of application architecture
 Test it!

Continuous Delivery Discipline
Don’t check it in if you don’t want it to go to prod
 Check-in is the middle of the delivery lifecycle, not the beginning
Don’t write code you don’t want to go to prod
Don’t assume technical release equals business release
Strive for release boredom

Boring Releases
Just like integration we avoid release because it’s scary
Do scary things more, not less often
 Practice makes perfect
 Find ways to make it less scary
Making it easy to back out makes it less scary
Holy grail: prod releases are boring

The Importance of Integrated
Quality
Continuously delivering lousy software is not getting to done
QA must be deeply embedded to avoid friction
Deep QA integration can make things faster AND better

Quality Integration Techniques
Testing as software engineering (SET)
Testing pyramid
Test the service not just the software
 In XaaS’the ‘S’ stands for ‘service’ not ‘software’

Software Engineer in Test
Approach test development as software engineering
 Object-oriented design
 TDD
 Performance, maintainability, extensibility
 Use productivity enablers (Groovy, Spock, Geb…)
Do test design spikes, not just code design spikes
Be the advocate for the new definition of done

The Testing Pyramid
Automated tests’ value depends on how often/fast they can run
 System tests: few
 Acceptance tests: some
 Integration tests: many
 Unit tests: lots
The pyramid spans traditional dev/test boundaries
 Developers need to think about testing
 Testers mind developers’ business at the lower pyramid levels
 Test design spikes should involve everyone

Test the Service not the Software
The delivery pipeline is part of the service, so test it
 Inefficiencies are bugs too
What about “non-functional” requirements?
 Performance, resiliency, DB, InfoSec
 Incorporate them into upfront processes
 Test them using a deployment pipeline

The Importance of Configuration
Automation
Beware the sandbox
 Variation creates waste and errors
 Manual developer configuration has opportunity cost
Configuration automation is a solved problem
 Environments: cloud, Vagrant
 Automation: Puppet, Chef, CFEngine
 Deployment: Jenkins, Capistrano, Liquibase, flyway

Keys to Configuration Automation
Do everything to all the things
 Code, database, system, infrastructure
 Version control
 Packaging
Be able to go from any Point A to and Point B
 Including when A = nothing
Testers: don’t be codependent with developers
 Remember, you’re minding their business
 Remember, waste/variation degrades service quality

Solving the Change Conundrum
Smaller changes are easier
Automated processes are less error-prone
More frequent changes require practice => perfection
Change is no longer the enemy
Change is your business

Getting from Here to There
Don’t do “continuous delivery”, do “more continuous delivery”
 CD holy grail requires tremendous tool and process maturity
 Start with something small and simple
 Apply Kaizen
Question assumptions
 Especially about anything that feels “clumsy” or “we have to
because…”
Slow down to go faster
 Find opportunities to reduce batch size
 Think in terms of one-piece flow/new definition of “done”
Find ways to move quality inward and forward

Software MTTR
Repeat after me: our job is to deliver service, not software
 Continual value repair is part of the service
 MTTR is a (the?) key metric
If you grok the logic of CD, the tools/process will flow
 If not, they either frustrate or else represent ‘cargo cult’

My Vitals
www.ingineering.it
blog.ingineering.it
@jeffsussna
www.linkedin.com/in/jeffsussna

Software MTTR: Continuous Delivery Reduces Repair Time

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (11)

Similar to Software MTTR: Continuous Delivery Reduces Repair Time

Similar to Software MTTR: Continuous Delivery Reduces Repair Time (20)

Software MTTR: Continuous Delivery Reduces Repair Time