William Josephson Co-founder at Solano Labs presented at MassTLC's automated testing event on June 12, 2013 at Brightcove, Boston, MA

1. Automated Testing, Continuous Integration & Continuous Deployment
William Josephson, PhD
Co - Founder Solano Labs
@solanolabs
MassTLC Software Development -
Automated Testing

2. Some Facts & Figures*
High performing organizations:
• Deploy code 30 times more frequently.
– 8000 times faster than their peers
– Deploy multiple times a day, versus an average of once a month.
• Have 50 percent fewer failures.
• Restore service 12 times faster than their peers.
– Fewer failures and faster recovery mean less risk to the business when
changes are deployed.
*Source: Puppet Labs 2013 State of DevOps Report

3. How are they doing this?
I don’t have all the answers…
But here’s one: Automation

4. Automated Testing Does More Than Find Bugs
Its Helps Teams Build a Better Product
Bug are Common, Difficult
to Eliminate & Very Costly
Automated Testing:
Using software to test software
• Reduces cost of bugs by
reducing time between
introduction and discovery
• Frees engineers to make
large changes
fearlessly, reducing cycle
time
• Reduces complexity and
risk at feature integration

5. Automation: Reduces the Cognitive Feedback Loop
Automation shortens the
cognitive loop for engineers
• Automation delivers results for a build quickly
• Shorter turn-around times change how
engineers work
• Test-Driven Development (TDD) is one logical
extreme
Waiting for tests to run
is no fun!
Incorporate
Feedback
Run
Tests
Write
Code
Get Results

6. What Makes for a Good Automated Test?
• Eliminate repetitive manual tests
• Yield reproducible results
• Cover common and edge/error cases
• Ideally either pure unit or integration test
• Strive for clarity even when code is gnarly
• Test for business requirements

7. Why Not Automate Everything?
• Writing tests does have a cost
– May not be appropriate for experimental code
– May be fragile – UI/mobile automated testing is evolving
– Difficult or impossible to express design and human factors
• Thorough testing requires commitment and discipline
– Writing tests is a short-term pain for a specific individual
– Test suite as a whole is valuable to the team over time

8. What do you do with all those
automated tests?

9. Continuous Integration: A Discipline for Testing Every
Change
I Strong
Isolation
Detailed
Instrumentation
Consistent
Runtime
Environment
Reproducible
Results
Components that Make for Effective CI:
Typical CI Setup:

10. Continuous Deployment: Automatically Release Validated
Changes
• Goal: make release a non-event
– Popularized by Web 2.0, but much more widely applicable
– Final decision to deploy may still be manual
• Valuable to business and engineering sides of the house:
– Selectively enabled features – feature flags
– Consistent, widespread measurement
• Key Ingredients:
– Small changes, continuously integrated
– Automated infrastructure (DevOps)
– Staging/test environment accurately models production

11. Further Reading
• Martin Fowler from ThoughtWorks: martinfowler.com
• Book: Continuous Delivery: Reliable Software (Jez Humble)
• IMVU: http://timothyfitz.com/2009/02/10/continuous-deployment-at-
imvu-doing-the-impossible-fifty-times-a-day/
• Etsy Blog: codeascraft.com
• Github: teach.github.com/articles/lesson-continuous-delivery/
• http://info.puppetlabs.com/2013-state-of-devops-report.html