Are you embarking on a large-scale, globally distributed, multi-team scrum project? Have you already identified the potential testing challenges that lie ahead? Or have you belatedly encountered them and are now working on them in real-time? Five years and more than 200 projects into its agile journey, Dell Enterprise Solutions (ESG) has empirically determined that once a project extends beyond three scrum teams, interesting testing challenges arise—inconsistent “done” criteria, integration testing underscored by epic/story interdependencies across teams, test automation inconsistency, and uncoordinated regression testing. Worse yet, the more teams involved, the less likely it is that a single scrum team has the visibility to validate the overall product from a customer usage perspective as the product evolves through sprints. Geoff Meyer serves up some lessons learned from within the Dell ESG Validation organization as it evolved its agile testing and automation strategies from a waterfall-based environment to one that fully embraced agile Scrum across its entire software product portfolio.

1. W16
Agile Testing
5/7/2014 3:00:00 PM
Meet Big Agile: Testing on
Large-Scale Projects
Presented by:
Geoff Meyer
Dell, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Geoff Meyer
Dell, Inc.
A test architect in the Dell Enterprise Solutions Group, Geoff Meyer has more than twenty-eight
years of experience as a software developer, manager, business analyst and test architect.
Geoff helps serve the software development practices of more than 800 development, test, and
UX engineers across four Global Design Centers. Geoff is an active member in and contributor
to the Agile Austin community.

3. 4/26/2014
1
Agile Testing @ Large
Geoff Meyer, geoffrey_meyer@dell.com
May 2014
Last updated: 25 February, 2014
Courtesy: Gulliver's Travels (2010)
Session Objectives
• Challenges to expect with Agile practices in
large-scale, multi-team projects
• Brief overview of Industry Models
• Real-world examples − from a large organization
perspective
• Organizational strategies & considerations
• Test automation considerations
• An In-depth Tools Discussion
• Focused on how to develop Test Automation
IS
IS
NOT

4. 4/26/2014
2
Geoff Meyer
• Dell Inc, 1998 – present
– Test Architect
› Agile Test & Automation Strategy
– Agile Steering Committee Co-Chair
– Global Projects
› 15 Scrum teams (2 regions)
› 11 Scrum teams (4 regions)
› 9 scrum teams (2 regions)
› 7 Scrum teams (4 regions)
• NCR Corp. 1984 – 1998
– SW developer, Project Lead, SW Manager
• B.S. Computer Science, San Diego State University
• Masters Engineering Management - NTU
3
4
Agenda
• The Dell Context and
• Early Agile Testing Challenges
• In Search of… Solutions for Scale
• Adaptations for Testing at Scale
• Automating in an Agile world
• Lessons of Large
Agile @

5. 4/26/2014
3
5
The Context at
Dell ESG
http://amazngwallpapers.blogspot.com
Dell Enterprise Solutions Group
Global Design and Development using Agile Scrum
6
Noida Design
Center
Bangalore
Design Center
Austin
Design Center
Silicon Valley
Design Center

6. 4/26/2014
4
Dell Enterprise Solutions
Software Products
• Server Systems Management
• Converged Infrastructure Systems Management
• Private Cloud Systems Management
• Console Plug-ins (i.e. for SCCM, vCenter…)
7
Common Product Characteristics:
• Embedded Software
• Large hardware support test matrix
• Software is installed in the Data Center
• Enterprise update cycles ~6 months
• Products must function even as underlying HW, FW,
BIOS, and Drivers are upgraded - SUSTAINING
Storage
Network
Compute
The Early
Challenges
8
?
http://cleaneatingchelsey.com/

7. 4/26/2014
5
Culture Transition
• Early commitment on Requirements
• Insufficient Detail in Requirements
• Development vs. Test
• A different “School” of Test1
• Fail Early, Inspect & Adapt
91 – Scott Barber “Approaches to Software Testing: An Introduction “
“ “Everybody, every team, every platform, every division,
every component is there not for individual competitive
profit or recognition, but for contribution to the system
as a whole”
W. Edwards Deming
Signs of Trouble
• Thursday’s Build
• Planning more stories into
sprint than team’s velocity
• High % of carry-overs
• Inability to Automate
Acceptance Tests within
Sprint
• “Engineering” Stories
Large-Scale
• Cross-team interdependencies block sprint
progress
• Large HW configuration validation consumes
test resources
• No single team had visibility to the entire product
• Insufficient Product Owner bandwidth
http://www.huffingtonpost.com/2012/03/21/truck-stuck-thomas-
circle_n_1369545.html?p=truck+and+underpass&vm=r

8. 4/26/2014
6
Test Automation Pitfalls
• Minimal guidance to Test beyond “Go forth and automate”
• Black Box Test Mentality
• Automation was interpreted by many as “Automate the UI”
11
• Insufficient SW engineering
skills across Test teams
• Architecture(s) not optimized
for Automatability
12
In Search of…
Solutions For
Scale

9. 4/26/2014
7
Leverage the Community
13
Books
Conferences
Professional Groups
Consultants
Dallas Agile
Leaders’
roundtable
Scaled Agile Framework™ Big Picture
© 2008- 2013 Scaled Agile, Inc and Leffingwell, All rights reserved

10. 4/26/2014
8
Agile Testing Quadrants
15
4 - http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/
4
ALM Tooling
16
• Multi-team, geographically distributed projects
create a necessity for ALM Tooling
–Requirements Management
–User Story Management
–Test Case Management
–Defect Management
–Test Script Repository

11. 4/26/2014
9
Automation Tooling
17
• IDE
• UI Tools
• Scripting Languages
• Services
• Frameworks
18
Agile @ Dell
Adaptations

12. 4/26/2014
10
Agile Testing @ PG Enterprise Solutions Group
How did we go about it?
19
Process/Project
Culture/Organization
X-Large
project (15)
2011 2014
UI Automation
“Automate-First”
Service-level
automation ALM using
TFS
In-sprint
Automation
Accelerate
Automation
Skillset
Maximize
Utilization
Automation
Architects
Automated
Setup & Config
4 Geography
project (11)
2012
TDD &
Automated
UT
Non-Functional
Scale, Longevity
2010 2013
Test into
Scrum
2009
Collaborative
Seating
Rally
Agile @ Dell
Core Activities
• Pre-Sprint activities:
– Staffing
– Training - Project tools and processes
– CI/Build environment
– Automation Framework and BVT
• Establish Project-wide ‘Done’ criteria
• End-to-end, short duration User Stories
• Test Automation is included User Story acceptance
criteria
• Refresher Workshops for new projects
20

13. 4/26/2014
11
Week 1 Week 2
2 week Sprint Lifecycle Example
Manual Regression
Automated
Regression
Automated
Regression
Automated
Regression
Sprint
Planning
Automated
Regression
Automated
Regression
Sprint
Review
Retro
spective
Demo
Accept
User
Story
Demo
Accept
User
Story
Demo
Accept
User
Story
Demo
Accept
User
Story
Demo
Accept
User
Story
Product Owner
(or Proxy)
Scrum Master
Development
Test
Tech Pubs
UI
22
In-Sprint: Testing
• Design/Develop Test cases and Review with team
• Develop test automation and maintain in TFS
• Execute test cases (manual and automated)
• Immediate Feedback (IM, email, in-person) with team of
sightings… do not automatically enter defects
• Verify fixes for defects; re-test if needed
• Maintain test case status (in MTM)
• Cross-team coordination including Weekly SoS
Confidential

14. 4/26/2014
12
Agile @ Dell Roles/Responsibilities
23
Product
Owner
Product Owner Proxy
1 per Scrum
Scrum Master
1 per Scrum
Development
4-5 per Scrum
Test
3:1
Tech Pubs
1 per 3 scrum team
UI
1 per Scrum
Scrum teams
• Small teams
• Co-located
• Multi-disciplined
• Feature-based
• Distributed Projects
• Scrum of Scrums
• Teams formed from
functional silos
…
Release Manager
SW Architect
Test Architect
Automation Architect
Cross-team Test roles
24
• Test Architect
– Defines Test Strategy across all Scrum
teams
– Establishes non-Functional Test
approach
– Influences consistent Application
Architectures
• Automation Architect
– Leads Analysis and POC activities for
new automation needs
– Ensures automation consistency for
BVT and nightly regression suites
– Safeguards against automation tool
proliferation

15. 4/26/2014
13
EXAMPLE - Project-wide “Done” criteria
25
Code
– Junit Tests
– SONAR
– Code Reviews completed
– All User Story tasks completed
– Demo and accepted by Product
Owner
– All known error conditions caught
and handled appropriately
– Code is Globalized
Test
– Acceptance Tests Reviewed with
scrum team
– Acceptance Tests documented in
Tool
– All planned Acceptance Tests for
User Story has been attempted
– Acceptance Tests executed
against Test system configuration
– Defects that are “must fix” before
releasing must be fixed prior to
story acceptance
– Service-level automated tests
– Service-level automated Tests are
globalized
Large-Scale Complexities
26
Characteristic Description
Large-Scale Project consists of more
than 4 Scrum teams
Interdependency Requirements
implemented across
multiple Scrum teams
Extensive
Configuration
Matrix
Extensive HW or SW
configurations
Waterfall
Intersection
One or more
components are
managed using Waterfall
Geography Project members are
Geographically dispersed

16. 4/26/2014
14
Development
Usability
Test
Wireframes
Architecture
Automation FW/Tools
Release Planning
Pre-sprints
Pre-sprints
HW resources
HW resources
Pre-sprints
: Agile with Adaptations
27
Hardening
1 2 3 N-1 N
1 2 3 N-1 N
1 2 3 N-1 N
… Sprints
Define Plan Develop Launch
OLP
…
Software System Test
Extended Sprint Test
Stability
Agile @
Extended Sprint Test
Extended Hardware Configurations
• Optimizes HW utilization and cost in
support of Scrum teams
• Owns Test Execution against extended
configurations
• Scrum team:
– Establishes Acceptance criteria
– Identifies all Test scenarios
– Owns Test Execution against reference
configuration(s)
– Develops in-sprint automation tests
28
Configurations
Positive X X X X
Negative X X X
Boundary X X X
Stress X X
Scalability X X X
Concurrency X X X
Globalization X X
Candidate for
Extended Sprint Test

17. 4/26/2014
15
Pre-SST
Software System Test
“Alpha Customer”
29
Hardening
• Workflow Design
• Test Analysis
• Test Design
• Test Development
• Test Automation
Development
• Test Execution
• Regression Test
• Test Execution
• Regression Test
SST
IS IS NOT
Requirements-based Functional testing
Customer-usage based First time integration test of workflows
Interoperability coverage of HCL in
specific customer environments
Exhaustive interoperability matrix
coverage of the HCL
Three phased approach Performance baseline testing
SAFe1 &
30
Agile @
1- Courtesy of Dean Leffingwell and Al Goerner of Valtech
Scrum teams
System team

18. 4/26/2014
16
PG Enterprise Validation
Agile & Waterfall Converged Model
Dell Confidential31
SST
Hardware Product Test
EST
Final
EST NEST 1
Sprint1 Sprint2
Sprint
N-1
SprintN
Sprint1 Sprint2
Sprint
N-1
SprintN
Sprint1 Sprint2
Sprint
N-1
SprintN
Hardening
Stability
Sprint
ExtendedSprints
SW
Regression
Test
Solution stack with
software focus SST
EST Exit
CCB Review
Code Freeze
A-Rev
Final
Extended Regression
F
V
PT
Stability
SST
A-Rev
Candidate
SSI
Stability
Sprint
RC-0
Regression
Regression
HW-Specific User Stories Complete
HW Enablement Features Complete
Vendor Deliverables Required
100% PT Attempted
All Stories Accepted
Feature Complete
X-on-A Stories
Accepted
PG Enterprise Validation
Agile @ Dell
Geography
32
• Co-location of a Scrum
team is always preferred
• Limit geographical
distribution of Scrum
team to no more than
two time-zones
• Scrum teams within a
project can be distributed

19. 4/26/2014
17
33
Automating in
an Agile World
Opportunities for Automation
• It’s not just about automating test cases
• Can also provide efficiencies to:
–Test Preparation, Setup and Configuration
–Non-Functional Testing (“ility”)
› Longevity, Scale and Performance Characterization
–Compatibility Testing (Solution and Device)
34

20. 4/26/2014
18
Test Preparation
• Bare-metal Deployment
• Setup & Configuration
– OS Provisioning and
Configuration
• Test case staging
• Environment
Cleanup/Baseline
• Virtualization-based test
environments
35
Test Content Development
• UI automation on Customer
Usage workflows
36
CAUTION: Application Architecture can be an enabler or inhibitor
• Unit Test Automation
• In-Sprint, Automated
Acceptance Tests
• Web-services (or CLI) Test
Automation for Functional
Testing
• Automated CI (UT, Build,
BVT)
2 - Agile Testing: A Practical Guide for Testers and Agile teams
2

21. 4/26/2014
19
Application Architecture Matters
• Does it facilitate testing at the API/Services level?
• Does the Business/Error Logic reside below the services
level?
• Does the UI architecture support test automation other
than record-playback?
37
Large Scale Organizations:
• Standardize UI architectures across the product
portfolio
• HTML5, Flex, Silverlight…
• Standardize Service architectures across the
product portfolio
• SOAP, REST, CLI, API
Non-Functional Testing
• Performance Characterization
• Longevity
• Stress
• Scale
• Concurrency
38
Often times analysis of
these areas are simply
too cost-prohibitive to
be done manually
3- http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/
3

22. 4/26/2014
20
Confidential39
Functional
Validation
Performance-
Characterization and
non-Functional Validation
Longevity
11 M2 1.0
SST
Baseline
M1
Scale
Concurrency
Simulation
L
a
r
g
e
-
S
c
a
l
e
L
a
r
g
e
-
S
c
a
l
e
12 13 17 18 19
Sprint 15
Performance Characterization Timeline
14 16
Sprint 21
Key Dependency:
• 3rd party HW available
• 3rd party user stories
completed
The Lessons of
‘Large’
40
Courtesy: Gulliver's Travels (2010)

23. 4/26/2014
21
The Lessons of ‘Large’
1. Co-located Teams
2. Project-wide ‘Done’
Criteria
3. Culture of Automation
4. System Test Team
5. Test Architect
6. ALM Tooling
Questions?
42

24. 4/26/2014
22
Resources
• Agile Manifesto - http://agilemanifesto.org/
• Articles/Blogs:
– http://support.smartbear.com/articles/testcomplete/automated-testing-agile-environment/ -
SmartBear
– http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ - Lisa Crispin
– http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=177
93&tth=DYN&tt=siteemail&iDyn=2 – Rajini Padmanaban
– Beyond Agile Testing: http://www.utest.com/int-v1/a/beyond-agile-testing - uTest
– Testing and Checking Refined: http://www.satisfice.com/blog/archives/856 - James Bach
and Michael Bolton
• Books:
– Scaling Software Agility: Best Practices for Large Enterprises – Dean Leffingwell
– How Google Tests Software – James Whittaker, Jason Arbon, Jeff Carollo
– Agile Testing: A Practical Guide for Testers and Agile teams – Lisa Crispin, Janet Gregory
– http://www.dorothygraham.co.uk/automationExperiences/index.html - Dorothy Graham
• Presentations:
– Approaches to Software Testing: An Introduction – Scott Barber
– Scaled Agile Framework Big Picture – Dean Leffingwell
43
Backup
44