For the last four years myself and my colleagues at RainStor have been evolving a process for testing a structure data archiving system in an Agile development environment. In this talk I will discuss the evolution of a team from a rudimentary Agile implementation on an unreleased product, to our current process which uses the fundamental elements of Specification By Example to successfully deliver software functionality across 30 different platform/backend configurations to a series of high profile and demanding customers. Last year our company was used as a case study for successful implementation in Gojko Adzic's book on Specification By Example. My report will discuss the lessons learned during the early implementation and the challenges faced in moving away from a compressed waterfall approach. Through a process of incremental change we have identified and tackled the fundamental issues that undermined the development effort as a team. I’ll describe some of the mistakes made in attempting to implement a more formal process of requirements documentation into an Agile implementation and the benefits we uncovered on moving to a more flexible user story based approach. I’ll also discuss some of the issues around trying to implement user stories in a server system with no GUI and very technical and performance based requirements. Raising the importance of quality and the status of testing both within the development team and the organisation as a whole has allowed the challenges facing the team to be recognised and respected. The result has been a more collaborative approach taken between developers and testers both through “collaborative specification” of user stories and tackling the problems that impact the delivery of value to the customers. I also plan to discuss how we’ve expanded from documenting acceptance criteria for each user story such that we now document Criteria, Assumptions and Risks for each feature and, rather than a ‘Done/Not Done’ approach how we identify the confidence in each of these categories to measure the confidence we have in each new feature being implemented. Having the test team as an involved and influential team through the entire development process has also allowed us to implement a number of testability features to help to make the product more testable. I will discuss the benefits of having development understand and prioritise testability issues with some illustrative examples. I will discuss the challenges and benefits of developing our own metadata driven test harnesses as opposed to an off the shelf solution. I’ll detail how having control over these harnesses has allowed us to work towards a self documenting test system using realistic customer examples as “Automated Specifications” of the RainStor system allowing us to explain current behaviour to Product Management in terms of well understood customer scenarios.

1. An Evolution Into SpecificationAn Evolution Into Specification
By ExampleBy Example
Adam KnightAdam Knight
Twitter: adampknightTwitter: adampknight

2. Two ConversationsTwo Conversations
• Zachary KnightZachary Knight
 ““Can we seeCan we see
the picture ofthe picture of
the Monkeythe Monkey
turning into aturning into a
Man?”Man?”
• Gojko AdzicGojko Adzic
 ““Although theirAlthough their
process has almostprocess has almost
all the elements ofall the elements of
SBE, they just thinkSBE, they just think
of it as theirof it as their
homegrown way ofhomegrown way of
developingdeveloping
software”software”

3. Evolution - ContextEvolution - Context
• Successful StrategySuccessful Strategy
Depends on ContextDepends on Context
• ProductProduct
 API/CLI interfacesAPI/CLI interfaces
 Scalable ParallelScalable Parallel
ProcessingProcessing
 SQL ParserSQL Parser
• MarketMarket
 OEM PartnersOEM Partners
 Varied implementationsVaried implementations
 Traditional DeliveryTraditional Delivery
ModelsModels

4. Evolutionary CyclesEvolutionary Cycles
• It takes time to adapt to changeIt takes time to adapt to change
• Methodology is an environmental factorMethodology is an environmental factor
• Faster feedback cycles encourage evolutionFaster feedback cycles encourage evolution

5. Requirements and SpecificationsRequirements and Specifications

6. • Included Functional DesignIncluded Functional Design
• Abstracted out Customer DetailAbstracted out Customer Detail
• Not maintainedNot maintained
• Not completed in sprintNot completed in sprint
Requirements DocumentsRequirements Documents

7. User StoriesUser Stories
• Developer ReservationsDeveloper Reservations
• Hard to break downHard to break down
• Still Solution BasedStill Solution Based
• Value drivenValue driven
• IndependentIndependent
• RecursiveRecursive

8. Challenging RequirementsChallenging Requirements
• Rejecting SolutionsRejecting Solutions
• Requesting ExamplesRequesting Examples
• Raising Testing ProfileRaising Testing Profile

9. Collaborative SpecificationCollaborative Specification
• Defining SuccessDefining Success
 Acceptance CriteriaAcceptance Criteria
 RisksRisks
 AssumptionsAssumptions
• Discussing ExamplesDiscussing Examples
• Breaking the ModelBreaking the Model

10. Test ApproachTest Approach

11. Early Test ManagementEarly Test Management
• Poorly selected TCM toolPoorly selected TCM tool
• Excel with Test CaseExcel with Test Case
EstimatesEstimates
Counting test cases is inappropriateCounting test cases is inappropriate

12. Exploratory TestingExploratory Testing
• Session Based ETSession Based ET
 Run times too longRun times too long
 Can’t parallelise sessionsCan’t parallelise sessions
• Thread Based ETThread Based ET
 Estimate chartersEstimate charters
• Reporting ConfidenceReporting Confidence

13. Test AutomationTest Automation

14. Automation PrinciplesAutomation Principles
• IncrementalIncremental
GrowthGrowth
• DRYDRY
• User InputUser Input
DrivenDriven
• TestabilityTestability

15. Towards Living DocumentationTowards Living Documentation
• Self DocumentingSelf Documenting
• Reporting in Business TermsReporting in Business Terms
• Specification for RefactoringSpecification for Refactoring

16. Current PositionCurrent Position
• Around100 End Customer installationsAround100 End Customer installations
• Highly Rated Testing/Support TeamHighly Rated Testing/Support Team
• ““The product was installed seamlessly and the customerThe product was installed seamlessly and the customer
was delighted.”was delighted.”

17. RecapRecap
It takes time to adapt to changeIt takes time to adapt to change
FROMFROM TOTO
Formal RequirementsFormal Requirements User Stories, CollaborativelyUser Stories, Collaboratively
Specified using ExamplesSpecified using Examples
Test Case ManagementTest Case Management Exploratory ThreadsExploratory Threads
Bespoke AutomationBespoke Automation
ScriptsScripts
Self Documenting File DrivenSelf Documenting File Driven
Test AutomationTest Automation

18. ““Improbably excellent teamsImprobably excellent teams
become so through a sustainedbecome so through a sustained
series of entirely probable smallseries of entirely probable small
improvements ”improvements ”
Jason GormanJason Gorman
Bletchley Park :Bletchley Park :
www.bletchleypark.org/content/contact/donation/support.rhtmwww.bletchleypark.org/content/contact/donation/support.rhtm

19. ThanksThanks
• Email:Email: adampknight@a-sisyphean-task.comadampknight@a-sisyphean-task.com
• Twitter: adampknightTwitter: adampknight
• Blog: http://www.a-sisyphean-task.comBlog: http://www.a-sisyphean-task.com
• Email: adam.knight@rainstor.comEmail: adam.knight@rainstor.com
• WebSite: http://www.rainstor.comWebSite: http://www.rainstor.com

20. ReferencesReferences
• Further ReadingFurther Reading
 Specification By Example – Gojko Adzic ; ISBN 9781617290084Specification By Example – Gojko Adzic ; ISBN 9781617290084
 Thread BasedTest Management – James BachThread BasedTest Management – James Bach
http://www.satisfice.com/blog/archives/503http://www.satisfice.com/blog/archives/503
• ImagesImages
 All images in public domain or underWikimedia or Creative CommonsAll images in public domain or underWikimedia or Creative Commons
LicenseLicense
 1&14 © BIODIDAC 2004 ; 2 Steveoc 86 ; 3 – Pangolin Hubert Ludwig ;1&14 © BIODIDAC 2004 ; 2 Steveoc 86 ; 3 – Pangolin Hubert Ludwig ;
4 - Rafael Brix ; 5 - www.palaeocritti.com ; 6 Brachiosaurus - Богданов4 - Rafael Brix ; 5 - www.palaeocritti.com ; 6 Brachiosaurus - Богданов
dmitrchel@mail.ru ; 7 - Matt Martyniuk ; 8 – Public Domain ; 9 -dmitrchel@mail.ru ; 7 - Matt Martyniuk ; 8 – Public Domain ; 9 -
Original uploader was Killdevil at en.wikipedia ; 10 –WikmediaOriginal uploader was Killdevil at en.wikipedia ; 10 –Wikmedia
commons no name ; 11&12 (unkown, extensively used in Publiccommons no name ; 11&12 (unkown, extensively used in Public
Domain); 13 –Yewenyi ; 15 - Richard Bartz, Munich Makro Freak &Domain); 13 –Yewenyi ; 15 - Richard Bartz, Munich Makro Freak &
Beemaster Hubert Seibring ; 16 - Coniac PublishingBeemaster Hubert Seibring ; 16 - Coniac Publishing