Capital One has a highly integrated environment, which creates many dependencies for its agile teams. Because these dependencies are often not completed until late in their sprints, Capital One faced prolonged integration and regression testing phases, and did not realize the expected improvements in quality or time to market. As technology leaders pushed for continuous delivery, testing needed to “shift left” and execute test in real time concurrently with development. Adam Auerbach shares Capital One’s experience implementing continuous testing. He explains the core principles of continuous testing, service virtualization, and the continuous integration/continuous delivery pipeline—and why testers need to understand and leverage these important concepts. Adam believes that testers need to learn basic development skills, including Ruby and Java, so they can take advantage of advanced automation practices. Because continuous testing is not easy and many companies have large populations of manual testers, Adam will provide a learning map to help you plan your personal and team’s transition.

1. Presented by Adam Auerbach
Putting Quality first through Continuous
Testing

2. 2
About Me
Senior Director of Technology at Capital One
• Tweet me @Bugman31 | Adam.Auerbach@capitalone.com
Responsible for Capital One’s Advanced Testing and Release Services Teams,
which include:
• Performance Testing
• Service Virtualization
• Automated Testing
• Release Management
• Test Data
• Quality Engineering Community of Practice
Before joining Capital One in 2013 Mr. Auerbach was with Chase and other
Financial and Insurance companies, in various leadership positions, with a
focus on Quality and Agile practices.

3. 3
We are more than a credit card company

4. 4
Three years ago, we had a robust Hybrid Automation
Framework
QTP
Adaptor
EnterpriseTestAutomationPlatform
QTP Automation Components
…
VB Script
Adaptor
Services Automation Components
…
Mainframe Automation Components
…
SQL DB
SQL
Interface
XML
Interface
eTAFReusableComponents
DOC
Interface
. . .
Doc
temp
TE
Adaptor
eTAFAutomationDriver
QC
Interface
Main Frame Apps
Auto
Reports
eTAF Automation framework Artifacts
Version
Control
Web Services
Excel
files
Web App 1 Web App 2 MF App
XML
files
QC/ALM
Selenium Automation Components
…
Selenium
Adaptor
Web Apps
Not$Complete$
Complete$

5. 5
Excel based Keyword scripts enabled Manual testers to
automate regression tests

6. 6
At the same time, we transitioned from Waterfall to Agile

7. 7
Early on, we tried to reuse what worked well in Waterfall
Design
Build
Test
Design
Build
Test
Sprint 1
Sprint 2
Hardening Test
Defect
Fix
Plan
Design
Build
System Test
Integration Test
Regression Test
Security Test
Performance Test
Deploy

8. 8
We adopted Agile with the hopes of delivering high quality
features to production, early and often.

9. 9
Continuous Integration and Continuous Testing provide
the foundation for Continuous Delivery

10. 10
Key Enablers

11. 11
Automate Everything
https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcRQZ7jklHi78z0zBEiu0xTfihEouErN3_KZHBKl-n5esoduNMQtSQ

12. 12
Fast Feedback

13. 13
Simulation and Virtualization
http://upload.wikimedia.org/wikipedia/commons/6/6e/Crash-test_simulation_(3467719123).jpghttp://upload.wikimedia.org/wikipedia/commons/d/d6/IIHS_crash_test_dummy_in_Hyundai_Tucson.jpg

14. 14
Why can’t this support Continuous Testing?
QTP
Adaptor
EnterpriseTestAutomationPlatform
QTP Automation Components
…
VB Script
Adaptor
Services Automation Components
…
Mainframe Automation Components
…
SQL DB
SQL
Interface
XML
Interface
eTAFReusableComponents
DOC
Interface
. . .
Doc
temp
TE
Adaptor
eTAFAutomationDriver
QC
Interface
Main Frame Apps
Auto
Reports
eTAF Automation framework Artifacts
Version
Control
Web Services
Excel
files
Web App 1 Web App 2 MF App
XML
files
QC/ALM
Selenium Automation Components
…
Selenium
Adaptor
Web Apps
Not$Complete$
Complete$

15. 15
How we got to Continuous Testing

16. 16
Real Time Automation
16
ACCEPTANCE TEST-DRIVEN DEVELOPMENT
• Team Focused
• Stress on Acceptance Criteria
• Add a Test
• Run all Tests
• Write Code
• Refactor Until Added Test Passes
TEST-DRIVEN DEVELOPMENT (TDD)
• Developer Focused
• Closer to Unit Level
• Add a Test
• Run all Tests
• Write Code
• Refactor Until Added Test Passes
BEHAVIOR-DRIVEN DEVELOPMENT (BDD)
• Team Focused
• Stress on Stories
• Add a Test
• Run all Tests
• Write Code
• Refactor Until Added Test Passes

17. 17
Everything tied into the Pipeline
17
• Unit Testing (TDD) – Tests written by
developers, which run as part of all code
builds
• Automation / ATDD – Tests written by the
team, either before or along side of
development. They run as part of the build
pipeline
• Continuous Performance Testing – Tests
to validate the system performs as
expected during peak loads, periods of
extended load, etc.
17

18. 18
Service Virtualization

19. 19
Robust Test Environments
https://upload.wikimedia.org/wikipedia/commons/e/e2/Responsive_Web_Design.png

20. 20
Self Service Tools

21. 21
Real Time Dashboards

22. 22
What we learned along the way

23. 23
Enterprise support groups help minimize the upfront costs to getting
started

24. 24
Provide training and off the shelf tools and frameworks to jumpstart
adoption

25. 25
Some lessons learned about adopting Continuous Testing
• Reuse proven Frameworks
• Scaled Agile
• Continuous Delivery
• ATDD
• Open Source
• Evangelists at every level and discipline
• Be ready for the need to educate everyone on everything
• Leverage dedicated resources who support and nurture the community
• Be prepared for additional costs

26. 26
What does this mean for me?
Developer
• Accountable for writing
“automatable” code
• Responsible for passing tests
(all tests)
• Become fungible in all aspects
of testing
Tester
• Must learn a programing
language
• Understand how to use CI tools
• Be able to apply enablers based
on current constraints
Everyone
• Work in Pairs or at least communicate like a pair.
• Take accountability for “Done” being working code and passing tests
• No more “throwing it over the fence”

27. 27
Questions?

28. Presented by Adam Auerbach
Putting Quality first through Continuous
Testing