As more organizations move towards continuous integration (CI) and continuous delivery (CD) with DevOps pipelines becoming the norm, where is the right place to do different kinds and levels of testing? In this presentation, I will provide a blueprint for test managers on how to think about shifting left and shifting right while keeping the overall QA picture and goals in mind.

1. ®
The DevOps Dance:
Shift Left, Shift Right
Get It Right
Presented by
Adam Sandman

2. ®
2 | 5/23/2020 © Copyright 2006-2020 Inflectra Corporation
®
About Me
 Adam Sandman was a programmer
from the age of 10 and has been
working in the IT industry for the past
20 years.
 Currently Adam is a Director of
Technology at Inflectra Corporation,
where he is interested in technology,
business and innovation.
 Adam lives in Washington, DC, USA

3. ®
3 | 5/23/2020 © Copyright 2006-2020 Inflectra Corporation
Agenda
The DevOps Dance
 What is Continuous Testing?
Shifting “Left”
Shifting “Right”
Stuck in the Middle with You
 Assessing risk and the decision to release
Questions?

4. ®
4 | 5/23/2020 © Copyright 2006-2020 Inflectra Corporation
®
Takeaways
1. How you can shift left by adding more testing to your
DevOps Pipeline
2. Ways to shift right by adding monitoring and tracing for
applications in production
3. Putting it all together into an integrated end to end test
management process
4. Some tools both open source and commercial that can
help along the way

5. ®
The DevOps Dance
What is Continuous Testing?

6. ®®
What is DevOps Anyway?
(By Devops.png: Rajiv.Pant derivative work: Wylve - derived from Devops.png: , Link, originally by Gary Stevens)

7. ®®
The DevOps “Lifecycle”
(By Devops.png: Rajiv.Pant derivative work: Wylve - derived from Devops.png: , Link, originally by Gary Stevens)

8. ®®
What is Continuous Testing?
 Continuous testing is the process of executing tests as part
of the software delivery pipeline to obtain immediate
feedback on the business risks associated with a software
release candidate.
 Originally limited to automated tests in the CI portion
 Should include all kinds of testing, throughout CI/CD pipeline
 For Continuous testing, the scope of testing extends from
validating bottom-up requirements or user stories to
assessing the system requirements associated with
overarching business goals.
1. https://en.wikipedia.org/wiki/Continuous_testing

9. ®®
Adding - Continuous Testing

10. Adding - Continuous Testing
TESTING
TESTING
TESTING
TESTING

11. ®®
Key Facts About Continuous Testing
 Without Continuous Testing, you won’t realize significant
value from DevOps.
 Pushing low quality code into production faster = bad
 Lack of visibility and feedback to assess risk = bad
 Chaos engineering without testing = simply chaos!
 Continuous Testing Myths
 Continuous Testing ≠ 100% test automation
 All testing should be done earlier in the lifecycle
 No testing is needed, just have production monitoring

12. ®®
Some Best Practices
 Continuous means testing is happening before, during, and
after each software change is made.
 Collaborate on requirements
 Validate changes constantly (e.g.: regression testing)
 Dev/Test Pairing (ex: exploratory testing, reviewing test
cases)
 Automated testing in CI
 Continuous improvement of test approach
 Review of customer feedback and business metrics

13. ®®
Introducing The DevOps Dance
TESTING
TESTING
TESTING
TESTING

14. ®®
Shift-Left
TESTING
TESTING
TESTING
TESTING
SHIFT LEFT

15. ®®
Shift-Right
TESTING
TESTING
TESTING
TESTING
SHIFT RIGHT

16. ®
Shifting Left
Testing Early, in the CI Pipeline

17. ®®
Shift-Left
TESTING
TESTING
TESTING
TESTING
SHIFT LEFT

18. ®®
What Does “Shift Left” Mean?
 Shifting Left is about removing downstream blockers and
finding and fixing defects closer to where they are
introduced.
 Rapid, automated testing to prevent known issues reappearing
 Identifying areas of risk earlier to know where to focus later:
 Performance
 Security
 Integration
 Infrastructure
 Usability
 Early exploration of features to understand business issues/risks

19. ®®
Continuous Integration (CI)
 Ensure that you have a robust set of “smoke”
tests that can run after every CI build
 Unit, API and Automated UI tests

20. Unit Testing
Plugins Available for:

21. API & Integration Testing

22. ®®
User Interface Testing
• Automated Tests • Manual Tests
You need both to really test and verify!

23. ®®
Non-Functional Testing
• Don’t leave these to the end of the process, before deployment
• Do just enough of each type of testing early in the pipeline to
determine if further testing will be needed later.

24. ®®
Exploratory Testing: Where & Why
Good testers find issues that no one knows about
Exploratory testing finds those edge cases
It’s the unknown ‘unknowns’ that trip you up
It’s particularly useful when the functionality is still
being formed in early sprints

25. ®®
Some Tools
 Dependency checkers – DependencyCheck
 Static Code Analysis – Parasoft, SonarQube
 Unit Tests – xUnit (jUnit, Nunit, PyTest, etc.)
 Performance Tests – JMeter, Gatling, NeoLoad
 API Testing – Postman, Rapise, SoapUI
 Security Testing – OWASP ZAP
 UI Testing – Rapise, Selenium
 Exploratory Testing – SpiraCapture

26. ®
Shifting Right
Monitoring in Production, after CD

27. ®®
Shift-Right
TESTING
TESTING
TESTING
TESTING
SHIFT RIGHT

28. ®®
Shifting Right
 Shifting Right is about leveraging customers and access
to production data to test effectively and support
feedback loops
 Enabling testing activities
 Customer feedback on features (ex: A/B and Canary Testing)
 Monitoring and healing production issues
 Usability and experience (ex: usability testing, UI/UX)
 Customer satisfaction (ex: NPS ratings, engagement, adoption)
 Monitoring business metrics to “test” value of changes

29. ®®
Continuous Deployment (CD)
Automating the process from release ready, tests passed, to
production, post-release checks.

30. Business Monitoring
Business
metrics can
be a useful
indicator of
a problem
that system
monitoring
may have
missed!

31. User Monitoring

32. ®®
Some Tools
 System Monitoring – Dynatrace, Solarwinds, Cloudwatch
 Customer Helpdesk – ZenDesk, KronoDesk, Remedy, etc.
 Customer Behavior – Google Analytics, Splunk, Matomo
 Business Monitoring
 Traditional BI Platforms – PowerBI, Splunk, Cognos, Domo, etc.
 AI-Driven Data Analytics – Qlik, Tellius, etc.
 Data Visualzation – Tableau, etc
 Customer Satisfaction – Delighted, Promoter, AskNicely, …

33. ®
Stuck in the Middle with You
Assessing Risk and Deciding Whether to Go Live

34. ®®
There Is Never Enough Time for Testing
• How Do We Know if We Can Release into Production?
• How Do We Know We Have Tested the Right Areas?
• We Don’t Have Enough Time to Test Everything!

35. ®
Testing = Manage Technical Risk

36. Focus on Areas Identified Earlier in Pipeline
Unit Tests
API Tests
Security Tests
Performance Tests
Exploratory Sessions
Shift Left Tests Deeper Testing
Deeper Functional
Testing
Deeper Performance
Testing
Focused Exploratory
Sessions

37. ®®
Example: Performance Testing
Early Load Testing
• Development hardware
• No isolation
• 10 VUs, 10,000 requests
• Analyze trends
• Is this CI build faster or
slower than before?
• Should we spend more time
testing it later?
Full-Blown Load Testing
• Replica of production
hardware
• Dedicated environment
• Isolated networks
• 10,000 VUs for 4 hours
• What is the maximum load,
throughput, bottlenecks
• Do we meet our SLAs?

38. Hierarchy of Testing Risk
Happy Path, Safe Environment
Realistic Paths, Safe Environment
Edge Cases, Safe Environment
Edge Cases, All Environments
If it breaks here, then
no point trying further
levels in the hierarchy

39. ®
All The Automated Tests
Pass
Should We Ship?

40. ®®
User Stories
Requirements
Automated Testing
Exploratory Testing: Navigating Edges

41. ®®
Exploratory Testing: Navigating Edges
The Actual System

42. ®®
Exploratory Testing: Navigating Edges
The Actual System
Edge Cases

43. ®
Evaluating Risk & Deciding
The decision to release is a
business not technical decision

44. ®
44 | 5/23/2020 © Copyright 2006-2020 Inflectra Corporation
Wrap Up & Final Thoughts
1. Integrate Shift-Left techniques into your CI Pipeline
 Address risk areas identified early to focus deeper testing
2. Add Shift-Right techniques into your post-deployment
infrastructure, so that testing doesn’t end after live
3. Have a risk-based approach to deciding when to release
functionality into production

45. ®
45 | 5/23/2020 © Copyright 2006-2020 Inflectra Corporation
Questions?

46. ®
46 | 5/23/2020 © Copyright 2006-2020 Inflectra Corporation
Thank you for attending this
workshop/session.
Please fill out an evaluation form.