Describing how shift left in Agile and Devops is creating a need for TestOps as part of ITOPs to maintain testing infrastructure and data

1. 1
Prepared By:
Gervais Johnson
601 Montgomery St, Suite 650, San Francisco, CA 94111 | 415-678-1350 | www.MatrixRes.com
TestOps
Test Driven Development
and Shift Left

2. 2
Today’s Presenter
Gervais (Jay) Johnson, Agile Delivery Manager
 Leads, trains, coaches teams and companies in Agile development and adoption
 Architect, Application and Software Engineer, and Agile Leader for cross sector and
industry including NASA and DoD organizations
 IBM - 16 Year Veteran and Thought Leader
 30 Years developing applications and changing the world
 Education and certifications
 Certified Scrum Master and Scrum Professional
 Scrum Master, Scrum Developer, Agile Expert Certified
 PMI Project Management Professional
 Scaled Agile Framework Program Consultant
 IBM Certified Consultant, PM, SPSS, Agile Expert
 Six Sigma Black Belt

3. 3
Outline
 What is TestOps
 TestOps Ecosystem
 Shift Left Movement
 Test Driven Development (TDD)
 Behavior Driven Development (BDD)
 Acceptance Test Driven Development (ATDD)
 Continuous Testing (CT)
 TDD Details
 Open Discussion

4. 4
Introductions
 What is your name?
 What is your role?
 What prior Testing and Agile experience do you
have?
 0 to 10
 0 = Brand New
 10 = Testing Expert

5. 5 MATRIX Confidential
The Agile Manifesto- “BE AGILE”
We are uncovering better ways of developing software by doing it and helping others do it.
“Responding to change over
following a plan”
“Customer collaboration
over contract negotiation”
“Working software over
comprehensive
documentation”
“Individuals and interactions
over process and tools”
Values
“The most efficient and
effective method of
conveying information to
and within a development
team is face-to-face
conversation.”
“Welcome changing
requirements, even late in
development. Agile
processes harness change
for the customer's
competitive advantage.”
“Working software is the
primary measure of
progress.”
“Agile processes promote
sustainable development.
The sponsors, developers,
and users should be able to
maintain a constant pace
indefinitely.”
“Our highest priority is to
satisfy the customer
through early and
continuous delivery of
valuable software.”
“Deliver working software
frequently, from a couple of
weeks to a couple of
months, with a preference
to the shorter timescale.”
“At regular intervals, the
team reflects on how to
become more effective,
then tunes and adjusts its
behavior accordingly.”
“Continuous attention to
technical excellence and
good design enhances
agility.”
“Business people and
developers must work
together daily throughout
the project.”
“Build projects around
motivated individuals, give
them the environment and
support they need, and trust
them to get the job done,
“Simplicity -- the art of
maximizing the amount of
work not done – is
essential.”
“The best architectures,
requirements, and designs
emerge from self-organizing
teams”
Principles

6. 6
What is TestOps?
Testing never stops, it is continuous and is the fabric of enterprise operations and
the quality glue for Agile + DevOps

7. 7
TestOps Includes Various Testing
Are we building the right
product?
Are we building the product
right?
Business Facing
Technology Facing
Functional, Story &
Prototype Testing
Unit & Component
Testing
Business Facing
Technology Facing
Exploratory &
Usability Testing
Performance, Load
& Security Testing
CritiqueProduct
SupportingtheTeam

8. 8
An Agile TestOps Ecosystem Example A

9. 9
An Agile TestOps Ecosystem Example B
‘Shift Left’ – Operational Concerns
Also BizDevOps and DevOpsSec is becoming an important part of the fabric or glue

10. 10
What is Shift Left Testing?
 "Shift Left" is a QA practice being implemented in high-performninh Agile/DevOps teams
that provides an effective means to perform testing during and in parallel of software
development activities.
– Software Development, test and operations work together to plan, manage and execute automated
and continuous testing to accelerate feedback todevelopers
– The rate of the feedback is determined by an organization’s desiredvelocity
– Technology is used to automate processes and virtualizecomponents
 QA/Testing is performed as a part of the development process (TDD) or as a service
running (BDD or DDD) in parallel to development activities, shifting left accelerates
feedback to developers and improves the quality of code delivered to QA
 Shift left is an essential capability for Agile/DevOps teams and IT Operations teams

11. 11
 Perspectives
– Testing: Begin test executionearlier
– Behavior Design Testing
– A/B Testing
– Feature Driven Testing
– Development: Act on feedback to achieve value
– PM: Resource allocation to address feedback
– Infrastructure: Shift left environments available
early, capacity drives availability
– Deployment: Accelerated capacity to meet test
periodicity
 Activities:
– plan, create and automate integrationtest
cases in advance
– support a periodic cycle of integrationtesting
for the components/applications under
change
– prioritize, process, and resolve feedbackby
development teams within the feedback
period
– plan, automate and deploy components,
applications and virtual services
– plan, create and virtualizecomponent
services relevant to the development
activities
– execute delivery, integration, build, deploy,
testing, feedback and correction during the
defined periodicity.
Whatever testing you can perform shifts left - Integration testing could be your
first step as a means to provide a means to prepare for a formal QA and
reduce bottlenecks associated with Integration testing post development.
What is Left Shifting?

12. 12
Shift Left Planning Workshop – Part of Agile Adoption
Roadmap
Goals and
Assessment
•Client Capabilities: CurrentStatus
•Business Goals for Improvement
•Solution Capability Oriented
•People, Practices, Technology,information
Capability
Priorities
•Best Practices for SolutionCapabilities
•Discussions to refine ObjectiveCapability
•Prioritize Incremental Capabilities
•Vision, User Value, Pain and Complexity
Executive
Sponsor
Review
•Review Outcomes
•Assumptions & Risks
•Gain concurrence
•Identify next steps
Solution
Improvement
Roadmap
•Best Practices for AdoptingSolution
•Identify Key Milestones
•Roadmap activity to define actionableplan
•Define Quick Win Pilot
Day1Day2Day3Day4
Current State
Assessment
Objective & Prioritized
Capabilities
Agile Adoption Roadmap
DraftResults
Detailed 1-3 month roadmap
for initial win. High level
roadmap for remainder of
initiative. Executable week
following the workshop
Theme Activities Objective

13. 13
TDD and BDD Place
• Unit test (Unit)
• Integration test (Collaboration)
• User interface test (Frontend, ex. WatiN)
• Regression test (Continuous Integration)
• …, System, Performance, Stress, Usability, …
Test types:
• Black-box test
• White-box test
The only tests relevant to TDD and BDD is Black-box Unit Testing

14. 14
Feature Driven Development (FDD)
● Is a client-centric, architecture-centric, and pragmatic software
process
● Feature Breakdown Structure (FBS) instead of WBS
● A feature is a small, client-valued function expressed in the form:
<action> the <result> by/for/of/to a <object>

15. 15
Test Driven Development (TDD)
Is a rapid cycle of testing, coding, and refactoring
● Relies on a very short development
cycle
● Developer writes automated test case
first
● Test case defines desired improvement or
new function
● Next develops minimum amount of
code to pass test
● Lastly refactors the new code
to acceptable standards
Code refactoring is a "disciplined technique for restructuring
an existing body of code, altering its internal structure without
changing its external behavior"

16. 16
Behavior Driven Development (BDD)
● Outside-in and pull-based - Wire-frame > Test Cases > Coding
● Multiple-stakeholder, multiple-scale, high-automation
● Describes a cycle of interactions with well-defined outputs
● Results in the delivery of working, tested software that matters

17. 17
BDD and TDD

18. 18
Agile Story and BDD
ID: 27
STORY NAME: Save a list of potential cars for later review
As a: Buyer
I can: Add a car to my wish list
So that: I can review my top choices at a later time.
Acceptance Criteria
Scenario 27.1: Add a car to my wish list
Given: A potential buyer is logged in
And: The car is available for sale
When: The ‘Add to Favorites’ option appears
And: A buyer flags the car for the wish list
Then: The car details are displayed in the wish list
Scenario 27.2: Review list of favorite cars
Given: A potential buyer is logged in
Given: A potential buyer has previously picked some favorite
cars’
And: A buyer clicks on “view my favorites’
When: The buyer views the wish list
And: The car is still available for sale
Then: The buyer can view the car summary in the wish list
Story
As a <role>,
I can <activity>,
so that <business value>
Scenarios
Given <context>
When <event>
Then <outcome>

19. 19
Acceptance Test Driven Development

20. 20
Technical Debt and Testing
TDD
BDD
Source: http://www.slideshare.net/nashjain/inverting-the-testing-pyramid
FDD

21. 21
Automated Testing Across Platforms
Loadster
LogiGear
TestArchitect
Microsoft test tools
QASymphony
Telerik Test Studio
Sauce Labs
Selenium Appium
Synopsys
Xunit SDK
Apache JMeter
Grinder
Gatling
Tsung
LoadRunner
WebLOAD
Appvance
NeoLoad
LoadUI
WAPT
LoadImpact
Rational Performance Tester
Testing Anywhere

22. 22
PVCS
SVN
Artifactory
Splunk
Jenkins Maven
Ant
Puppet
chef
HP ALM
Policy Tester
Selenium
Smart Cloud Control
Desk
PMG
ServiceNow
Microsoft Team Foundation Server
Serena Dimensions
Example Testing Across Platforms

23. 23
TDD History
• Test-driven development (TDD) is a software development technique that uses short
development iterations based on pre-written test cases that define desired
improvements or new functions. Each iteration produces code necessary to pass that
iteration's tests. Finally, the programmer or team refactors the code to accommodate
changes. A key TDD concept is that preparing tests before coding facilitates rapid
feedback changes. Note that test-driven development is a software design method,
not merely a method of testing.

24. 24
TDD Method and Cycle
New
require-
ment
Write
new test
Run
tests
Write
new
code
Run
tests
Refactor
Run
tests
• Write Test Code
– Guarantees that every functional code is
testable
– Provides a specification for the functional code
– Helps to think about design
– Ensure the functional code is tangible
• Write Functional Code
– Fulfill the requirement (test code)
– Write the simplest solution that works
– Leave Improvements for a later step
– The code written is only designed to pass the
test
• no further (and therefore untested code is
not created).
• Refactor
– Clean-up the code (test and functional)
– Make sure the code expresses intent
– Remove code smells
– Re-think the design
– Delete unnecessary code

25. 25
TDD Benefits
0
1
2
3
4
5
6
7
8
9
10
0%
20%
40%
60%
80%
100%
Requirements Coding Integration Testing Support
Thousand$s
%defectscreated
% of Defects Introduced Cost to Fix a Defect
The pain is here! This is too late…

26. 26
TDD Benefits
0%
20%
40%
60%
80%
100%
120%
140%
160%
MS: VS MS: MSN MS:
Windows
IBM: Drivers
Time To Code Feature
Time To Code Feature
using TDD
Defect density of team
Defect density of team
using TDD
Major quality improvement for minor time investment

27. 27
TDD Fundamental Mind Exercise
• When you first start at doing TDD you "know"
what the design should be. You "know" what
code you want to write. So you write a test that
will let you write that bit of code.
• When you do this you aren't really doing TDD –
since you are writing the code first in your mind,
mind change to write test first
• It takes some time and mentoring to realize that
you need to focus on the test first.
• Write the test for the behavior you want - then
write the minimal code needed to make it pass -
then let the design emerge through refactoring.
• Repeat until done.
• TDD and Paired Programming are a great
combination to improve quality and increase
delivery time

28. 28
TDD Adoption Hurdles
• Why, when TDD was born a
decade ago?
– The one and only reason: #1:
Learning curve…
– Infrastructure and willingness
• Why do most developers not
write unit tests, still?
http://weblogs.asp.net/rosherove/archive/2008/09/20/goodbye-mocks-
farewell-stubs.aspx
mocks, stubs, dependency injection, IoC containers, Extract & override
technique, RecordReplay, AAA and more

29. 29
TDD Fundamental Infrastructure
• Interfaces
• Dependency Injection
• Test Doubles (Mock Objects)
• Mocking Framework.
• The builder and fluent
interface patterns.
Either (the old way)
“Mock”
Using Interfaces, Dependency Injection and
Test Doubles (Mock Objects)
Or (the new way)
“Isolate” and “Fake”
Using Typemock Isolator
and AAA - The "Arrange-act-assert" pattern
• Refactoring

30. 30
Why No Unit Testing and no TDD
1. I don’t have time to unit test.
2. The client pays me to develop
code, not write unit test.
3. I am supporting a legacy
application without unit tests.
4. QA and User Acceptance Testing
is far more effective in finding
bugs.
5. I don’t know how to unit test, or I
don’t know how to write good
unit tests.

31. 31
TDD Maturity
• Classicists
– The classical TDD style is to use real objects if possible and a double if
it's awkward to use the real thing. So a classical TDDer would use a real
warehouse and a double for the mail service. The kind of double doesn't
really matter that much.
• Mockist
– A mockist TDD practitioner, however, will always use a mock for any
object with interesting behavior. In this case for both the warehouse and
the mail service.
• Isolator and Fake
– An isolator TDD practitioner, will always “isolate” or “fake” any object
with interesting behavior. In this case for both the warehouse and the
mail service.
The Classicists does White-box Integration testing
First generation TDD
The Mockist does White-box Unit testing
Sencond generation TDD
The Isolator and Fake does Black-box Unit testing
Third generation TDD

32. 32
TDD Maturity, adding BDD
• Behaviour-Driven Development (BDD)
– By Dan North
– BDD is another variation on TDD that tends to use mockist testing
• Test method names should be sentences
• A simple sentence template keeps test methods focused
– This sentence template – The class should do something – means you can only
define a test for the current class. This keeps you focused. If you find yourself
writing a test whose name doesn’t fit this template, it suggests the behavior may
belong elsewhere.
• An expressive test name is helpful when a test fails
– “Behavior” is a more useful word than “test”.
– Determine the next most important behaviour.
– Requirements are behavior, too.
– BDD provides a “ubiquitous language” for analysis.
– Acceptance criteria should be executable.

33. 33
How to start TDD/BDD
• You don’t have to start big
• Start new tasks with TDD
• Add Tests to code that you need to change or maintain – but
only to small parts
• Proof of concept

34. 34
References
• Books
– Test-Driven Development in Microsoft® .NET (http://www.amazon.co.uk/gp/product/0735619484)
– Test-Driven Development by Kent Beck (C++) (http://www.amazon.co.uk/gp/product/0321146530)
– Clean Code by Martin Fowler
– Refactoring by Martin Fowler
– Refactoring of Patterns by Kerievsky
• Blog
– The Typemock Insider Blog (http://blog.typemock.com/)
• Unit test study from Microsoft Research:
– http://research.microsoft.com/en-us/projects/esm/nagappan_tdd.pdf
– https://en.wikipedia.org/wiki/XUnit and https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks
• Typemock
– http://www.typemock.com/
• Unit testing Silverlight:
– http://www.codeplex.com/CThru
• Other links:
– http://en.wikipedia.org/wiki/Test-driven_development
– http://www.testdriven.com/
– http://www.mockobjects.com/ - Online TDD book: http://www.mockobjects.com/book/index.html
– http://www.martinfowler.com/articles/mocksArentStubs.html
– http://dannorth.net/introducing-bdd
– http://behaviour-driven.org/Introduction

35. 35
TesOps Future Topics for Discussion
 Testing Value Chain and Tools Chain
 Test Automation Tools Demo
 TDD and BDD Demo
 Test Environment Management
 Security Testing
 Cloud Testing
 Mobile Testing
 Performance Testing
 Test Data Management

36. 36
Open Forum
Discussion