The past few years have seen a phenomenon of software organizations abandoning traditional release cycles in favor of daily or even hourly deployments. The emergence of a new, rapid software development workflow has raised questions regarding the role of test and QA in a product’s life cycle. When there is no QA phase, can there still be QA?

1. ContinuousImprovement
HowrapidreleasecyclesalterQAandtesting.
NoahSussman
SeleniumConf2013
@noahsussman
#seconf

2. TheCanonicalAgileReleaseCycle
Sprintsoftwoweeksormoreinlength.
Startdeploymentprocessattheendofthesprint.
QAispartofthedeploymentprocess.
QAmustbecompletebeforenewcodegoeslive.

3. TheContinuousDeliveryEnvironment
Minimumviablefeatureset.
Deploymentisdecoupledfromrelease.
Real-timedataonhowreleasesimpactrevenue.
Constanttweakstolivefeatures.

4. HowContinousDeliveryDiffersFrom
“Canonical”AgileProcess
Largefeaturesaredeployedpiecemealovertime
ConfigFlags
Darklaunches
Wire-Offs
EveryfeatureispartofanA/Bcampaign
Thereisno“DoneDone”

5. Releasingafeature
isdecoupledfrom
deployingcode.
David E. Smith http://www.flickr.com/photos/david_e_smith/3566697122

6. Anairportwithout
anairtrafficcontroller.
—ChadDickerson

7. ObservedBehaviorOfComplexSystems
Emergentbehaviorsrequireunplannedresponses.
Improvements,tooarediscoverednotdesigned.
Usersofthesystemhavecomplexexpectations.
Suchsystemsarenever“complete.”

8. QAHappensWhen?
Firstofall,whatis“QualityAssurance?”
Authoritativelyassuringthattherearenodefects?
No.Becausecompletetestingisimpossible.

9. Wearenotcustodians
ofinherentlysafe
systems.
—SidneyDekker

10. Testingiseveryone’sjob.
Library of Congress

11. FalsehoodsAboutQualityAssurance
Thereareafinitenumberofbugs.
Thereareafinitenumberofdetectablebugs.
Allseverityonebugscanbefoundbeforerelease.
Softwareisbuilttospecifications.
Atsomepoint,softwareisfinished.

12. AndTheBiggestMythAboutQA…
FALSE:Allbugshavelarge,complex,unpredictablecauses.
Infact,manybugs—evenseverity1productionissues—
resultfromerrorsthatatfirstglanceappeartrivial.

13. John Allspaw

14. ManySmallAnomaliesCombined
The“SwissCheese”modelofriskpresentsuswitha
strongcaseforprioritizingtheeliminationofsmall
errorsratherthanfocusingonthemitigationoflarge
catastrophicfailures.
Unittestingisgreatateliminatingsmallerrors.

15. Safetyisanemergentproperty…its
erosionisnotaboutthebreakageor
lackofqualityofsinglecomponents…
managingqualityisaboutsingle
components,aboutseeinghowthey
meetparticularspecifications…
—SidneyDekker

16. …thebestsoftwareengineering
principles…qualityassurance,
testing…thehigheststandards—
it'snotgoingtobeenough
—NancyLeveson

17. Leveson’s3PracticesForLimitingRisk
1.Oversight.
2.Limitingcomplexity.
3.Systemsthinking.

18. CharacteristicsofResilientSoftware
Readablecode.
Reasonableautomatedtestcoverage.
Sanearchitecture.
Gooddebuggingtools.
Anengineeringculturethatvaluesrefactoring.

19. QAHappensWhen???
Exploratorytestingcanbeperformedanytime.
Rigorous,scientificapproach.
Focusoncustomersatisfactionratherthanaspec.
Equallyusefulbeforeorafterarelease.

20. …exploratorytestingistestdesignandtest
executionatthesametime.Thisisthe
oppositeofscriptedtesting(predefined
testprocedures,whethermanualor
automated).Exploratorytests,unlike
scriptedtests,arenotdefinedinadvance
andcarriedoutpreciselyaccordingtoplan.
—JamesMarcusBach

21. ExploratoryTesting
AddressesareasthatDeveloperTestingcan’t.
Unit&Integrationtestsvalidateassumptions.
There’salsoaneedtoinvalidateassumptions.

22. ManualTesting
Itdoesn’talwayslooklikeyouthinkitlooks.
Real-TimeMonitoringisthenewfaceofmanualtesting.

23. Sufficientlyadvanced
monitoringis
indistinguishable
fromtesting.
—EdKeyes

24. CaseStudy:WatchingTheGraphs
Asof2012,Etsycollected>250,000real-timemetrics.
Decidingwhichmetricsmatterisahumanproblem.
Humanvisionexcelsatanomalydetection.
Thus,atEtsy,everyonewatchesasubsetofthegraphs.

31. Anomalydetectionishard.

32. Mostbugs,mostofthetime,are
easilynailedgivenevenan
incompletebutsuggestive
characterizationoftheirerror
conditionsatsource-codelevel.
—EricS.Raymond

33. Sourcecode,diffsandlogs.
IfyourQAAnalystsdon’tlookatthese,teachthem!

34. Keepthefeedbackloopshort.

35. PayCloseAttentionToYour
Effeciency-ToThoroughness-Trade-Offs
Continuousreleasecycleshavedifferentrisksthan
monthlyorquarterlyreleasecycles.
ButnoaspectofContinuousDeliveryislessrisky.

36. ThereIsNoSpecification
Watchyourgraphs.
Testyourwholesystem.
Incrementallyimproveyourproduct.

37. Questions?
ns@noahsussman.com
@noahsussman
falsehoodsabouttime.com

38. FurtherReading
“How Google Tests Software,” James Whittaker (especially chapter 5)
“Look At Your Data,” John Rausser
“Optimizing For Developer Happiness,” Chad Dickerson
“Outages, Postmortems and Human Error,” John Allspaw
“Prevention versus Governance versus Adaptive Capacities”, John Allspaw
The Swiss Cheese Model of Risk (Wikipedia)
The ETTO Principle (Wikipedia)
“What Is Exploratory Testing?,” James Marcus Bach
“How Many Eyeballs Tame Complexity,” Eric S. Raymond
“The Timeless Way of Building,” Christopher Alexander
“Ten Questions about Human Error”, Sidney Dekker
“Sufficiently Advanced Monitoring is Indistinguishable From Testing”, Ed Keyes
“SEC Technology Roundtable 10/2/2012”, Nancy Leveson
“Reliable Software Systems Design,” Gerard J. Holzmann
“The Power of Ten: Rules for Developing Safety Critical Code,” Gerard J. Holzmann