Arthur Hicken Chief Evangelist of Parasoft @ PSQT 2016 discusses: • What the shift from automated to continuous means • How disruption requires changes to how we test software • Addressing gaps between Dev and Ops • Technologies that enable Continuous

1. No DevOps Without
Continuous Testing
Arthur Hicken - Parasoft
1

2. Your Presenter
Arthur Hicken is Chief Evangelist at
Parasoft where he has been involved in
automating various software development
and testing practices for over 20 years.
He has worked on projects including
cybersecurity, database development, the
software development lifecycle, web
publishing and monitoring, and integration
with legacy systems and maintains the IoT
Hall-of-Shame http://bit.ly/iotshame and
SQLi Hall-of-Shame http://bit.ly/sqlishame
Follow him @codecurmudgeon
Blog: http://codecurmudgeon.com
Web: http://parasoft.com

3. Agenda
• What the shift from automated to
continuous means
• How disruption requires changes to how
we test software
• Addressing gaps between Dev and Ops
• Technologies that enable Continuous
www.iist.org 3
www.psqtconference.com

4. Continuous
4
www.psqtconference.com
• Build, Integration, Testing, Release,
Delivery, Deployment
• Builds on DevOps movement
• Enables DevOps reality
• High dependency on automation:
– processes
– assessment
– decisions

5. DevOps
“DevOps is the practice of operations and
development engineers participating
together in the entire service lifecycle, from
design through the development
process to production support.”
– The Agile Admin
• No more throwing code over the wall
• Design for deployment
5
www.psqtconference.com

6. DevOps Pipeline
6
www.psqtconference.com

7. Automated vs Continuous
• Just being automated isn’t continuous
• Scheduling / frequency
• Intermittent manual dependencies
• Decisions points
– Is it done?
– Is it good?
– Is it safe?
– Can I push it out the door?
7
www.psqtconference.com

8. Complexity a Barrier to Testing
3rd Party System
Evolving Component
Mainframe
Scheduled Access

9. API Economy
Pieces/Parts come from everywhere
Definitions changing (upgrades, new versions)
Not always available
Can’t test with real data without causing problems
9
www.psqtconference.com

10. We must Re-invent Test
• The penalty for faulty software is
increasing
– Brand erosion
– All industries at risk for total transformation
• Technology is faster and more distributed
– Containers
– Microservices

11. Fixed Fixed
Impact of Test in the Evolving
SDLC
Plan Develop Build Test Release
Test as a time-boxed event is the prevailing method
“Are we done testing?”

12. Impact of Test in Evolving SDLC
Plan Develop
Build
Test
Release
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
Build
Test
“Does the release candidate have an
acceptable level of risk?”

13. Gaps between Dev and Ops
What is “done”?
Fully testing with external
dependencies
Full environments and all their
variants
13
www.psqtconference.com

14. Speed of Deployments
Rapid deployments
Deliver features and bug fixes
Deployments outpace ability to test
Too many changes
Since we didn’t test enough, Devs are mired in bug fixes
Glitches and outages can happen
14
www.psqtconference.com

15. QA and Testing
• Cannot keep up with the rate of change
• Quality lags behind
• Do we even know what to test?
• Complex technologies including message
protocols and services
15
www.psqtconference.com

16. Enabling Technologies
• Stubs
• Service Virtualization
• API testing
• Test data management
• Environment management
• Self-service test environments
16
www.psqtconference.com

17. How to do it better ?
• Give absolute priority to automated tests
• Invest time into designing interfaces (API)
• Use API (service) testing tools to cover
interfaces
• Measure the quality of the test

18. How to do it better ?
• Use stubs and mocks to improve:
– Facilitate test automation
– Increase code penetration

19. How to do it better ?
• Use service virtualization to improve
testing automation
– Isolate at the message layer
– Simulate functional scenarios and
performance conditions

20. How to do it better?
Test management
•Environments
•Data
•Dependencies
20
www.psqtconference.com

21. • The Challenge
– Performance issues found late in the SDLC resulting in large rework
and cost
– Middleware component able to handle large load but available
downstream dependent systems were unable to provide throughput
needed
Reduced testing time by 27-days for a major
commercial bank
Capacity Constrained staged environments
Database
Service
Mainframe
Application
1000 tps

22. Database
Servic
Mainfram
e
Applicatio
n
Reduced testing time by 27-days for a major
commercial bank
1000 tps
Virtual
Asset
Virtual
Asset
Virtual
Asset
Virtual
Asset
• The Solution
– Identified key use-cases and emulate the behavior of dependent
systems. Asset performance setup to match production response values
– API and Load testing tools used to validate system under test. Virtual
assets changed to model different scenarios
• The Business Benefit
– Able to predict performances issues and fix before full Perf testing.
– Able to run different scenarios not easily achieved with production
environment (excessive response times).
– Setup for ‘performance test’ reduced to near zero

23. – The Challenge
• Multiple teams using the same test database
• Teams not respecting data integrity & others test data records
• Regression tests consistently failing. Takes >1 hour to determine
that it was due to “data changes”.
• “Real problems” were getting lost in the noise
Shared
Databas
e
Eliminated 83% of the configuration time for a major
telecom company
Test data management for complex transactions
✔
✖
✖
Shared
Database

24. – The Solution
• Setup Virtual Assets to model the SQL queries and use API
testing tool to manage automated nightly regressions against
both virtual assets and live systems
– The Business Benefit
• Test teams able to focus on ‘real regressions’ and separate out
data integrity issues from functional test failure
Eliminated 83% of the configuration time for a major
telecom company
Virtual
Asset
Virtual
Asset
Virtual
Asset
✔
✔
✔

25. Reduced wait time for test team by 60% for a major media
conglomerate
• The Challenge
– Large agile development effort to adopt Service Oriented Architecture
(SOA)
– High risk project but the Test team “stuck waiting for the first build”
– Development of functionality was not easy to coordinate as different
teams had different schedules; not all finished at the same time
Agile/Parallel development limited by system dependencies
Iteration
Iteration
Iteration
Team A
Team B
Team C
Iteration Iteration
Iteration Iteration Iteration
Iteration Iteration
Current
Development/Testing
dependencies

26. Reduced wait time for test team by 60% for a
major media conglomerate
• The Solution
– Use descriptions of the new services (WSDL, XSD, example JSON
payloads) to build Virtual assets prototyping the new functionality.
– Test team builds tests with against the prototypes with API testing tool
and the independent development tests use the prototypes to perform
early stage Integration Testing
• The Business Benefits
– Met business goals and timelines, were able to test functionality “as
soon as” it was available. Practiced TDD against prototypes to get a
head-start on ‘full system testing’
Iteration
Iteration
Iteration
Team A
Team B
Team C
Iteration Iteration
Iteration Iteration Iteration
Iteration Iteration
✔✖✔✖✔✖

27. Extended testing hours for a stock trading broker
• The Challenge
– Leverages 3rd party (out sourced) system for trade exchanges
– Many teams need access, so time is scheduled into 4 hour time
windows, and could only test during market hours
Capacity Constrained staged environments
Database
Servic
e
Mainfram
e
Applicatio
n

28. Extended testing hours for a stock trading broker
• The Solution
– Use Test environment manager tool to setup ‘Test Silos’ where each
team can do 24x7 testing against virtual assets
Database
Mainfram
e
Applicatio
nEnvironment
Manager

29. Conclusions
Emulates dependencies for the Test Environment
• Reduces complexity for early stage testing
• Increases predictability
On-Demand “Test Anytime, Anywhere”
• Automated Provisioning for different use-cases
• Automated Test Data Management/Simulation
Reduce time to configure for test
Realistic reusable assets
Reduce hardware costs and system access fees

30. Q&A
• Web
– http://www.parasoft.com
• Blog
– http://alm.parasoft.com
– http://codecurmudgeon.com
•Social
–Facebook: https://www.facebook.com/parasoftcorporation
–Twitter: @Parasoft @CodeCurmudgeon
–LinkedIn: http://www.linkedin.com/company/parasoft