William
Josephson
Co-­‐Founder,
Solano
Labs
@SolanoLabs
Automated
Tes,ng
NYC
Automated
Tes,ng,
Con,nuous
Integra,on
&
Con,nuous
Deployment

Solano Labs – Lets you Test More, Faster
Some Metrics on how Fast we have Sped Up our Customers:
•
•
•
•
18 hrs serial, 2 + hrs on Parallel Team City w/ local HW to 13 mins
30 min on Jenkins to 6 mins
5 hrs serial PyPy test suite to 15 mins
5 hour queuing to 5 min max wait time
Solano Labs
© 2013 Confidential
2

Some Facts & Figures*
High performing organizations:
•
Deploy code 30 times more frequently.
– 8000 times faster than their peers
– Deploy multiple times a day, versus an average of once a month.
•
Have 50 percent fewer failures.
•
Restore service 12 times faster than their peers.
– Fewer failures and faster recovery mean less risk to the business when
changes are deployed.
*Source:
Puppet
Labs
2013
State
of
DevOps
Report

How
are
they
doing
this?
I
don’t
have
all
the
answers…
But
here’s
one:
Automa,on

Automated Testing Does More Than Find Bugs
Its Helps Teams Build a Better Product
Bug are Common, Difficult
to Eliminate & Very Costly
Automated Testing:
Using software to test software
• Reduces
cost
of
bugs
by
reducing
,me
between
introduc,on
and
discovery
• Frees
engineers
to
make
large
changes
fearlessly,
reducing
cycle
,me
• Reduces
complexity
and
risk
at
feature
integra,on

Automation: Reduces the Cognitive Feedback Loop
Automa,on
shortens
the
cogni,ve
loop
for
engineers
Waiting for tests to run
is no fun!
•
•
•
Automa,on
delivers
results
for
a
build
quickly
Shorter
turn-­‐around
,mes
change
how
engineers
work
Test-­‐Driven
Development
(TDD)
is
one
logical
extreme
Incorporate
Feedback
Get
Results
Write
Code
Run
Tests

What Makes for a Good Automated Test?
• Eliminate repetitive manual tests
• Yield reproducible results
• Cover common and edge/error cases
• Ideally either pure unit or integration test
• Strive for clarity even when code is gnarly
• Test for business requirements

Why Not Automate Everything?
• Writing tests does have a cost
– May not be appropriate for experimental code
– May be fragile – UI/mobile automated testing is evolving
– Difficult or impossible to express design and human factors
• Thorough testing requires commitment and discipline
– Writing tests is a short-term pain for a specific individual
– Test suite as a whole is valuable to the team over time

What
do
you
do
with
all
those
automated
tests?

Continuous Integration: A Discipline for Testing Every
Change
Typical CI Setup:
Components that Make for Effective CI:
Consistent
Run,me
Environment
I
Strong
Isola,on
Detailed
Reproducible
Instrumenta,on
Results

Continuous Deployment: Automatically Release Validated
Changes
• Goal: make release a non-event
– Popularized by Web 2.0, but much more widely applicable
– Final decision to deploy may still be manual
• Valuable to business and engineering sides of the house:
– Selectively enabled features – feature flags
– Consistent, widespread measurement
• Key Ingredients:
– Small changes, continuously integrated
– Automated infrastructure (DevOps)
– Staging/test environment accurately models production

Further Reading
• Martin Fowler from ThoughtWorks: martinfowler.com
• Book: Continuous Delivery: Reliable Software (Jez Humble)
• IMVU: http://timothyfitz.com/2009/02/10/continuous-deployment-atimvu-doing-the-impossible-fifty-times-a-day/
• Etsy Blog: codeascraft.com
• Github: teach.github.com/articles/lesson-continuous-delivery/
• http://info.puppetlabs.com/2013-state-of-devops-report.html
• http://engineering.quora.com/Continuous-Deployment-at-Quora