SeleniumCamp 2020 presentation about the rise of observability, risk management, and weaving testing concepts throughout the SDLC. Examples of using analytics to increase confidence in your release cycle

1. CLOSING THE LOOP: SHIFTING RIGHT
& OBSERVABILITY (& RISK)
MARCUS MERRELL, SR DIRECTOR OF FIELD SERVICES
FEBRUARY 22, 2020

2. WHAT WE’RE TALKING ABOUT
• Shift Left
• Shift Right
• The Feedback Loop
• Observability
• User Behavior
• Risk
© Sauce Labs, Inc. 2
@mmerrell
FEBRUARY 22, 2020

3. WHO AM I?
FEBRUARY 22, 2020
• Sr Director of Field Services, Sauce Labs
• Chair, Selenium Conference Organizing Committee
• Selenium Contributor
• 19 years in QA/Testing role
• Lurker on Social Media
• Family, Music, Robots
© Sauce Labs, Inc. 3
@mmerrell

4. WHAT IS SHIFT RIGHT?
• What is it that’s shifting?
• To the right of what?
• What is our ultimate goal?
© Sauce Labs, Inc. 4
@mmerrell
1 0 0 1 0 1 1 1
1 1 0 0 1 0 1 1
MSB
LSB
7 6 5 4 3 2 1 0
FEBRUARY 22, 2020

5. THE TRADITIONAL MODEL
© Sauce Labs, Inc. 5FEBRUARY 22, 2020

6. SHIFT LEFT, A BRIEF REVIEW
© Sauce Labs, Inc. 6FEBRUARY 22, 2020

7. WHAT WE’RE AFTER
© Sauce Labs, Inc. 7FEBRUARY 22, 2020

8. WHAT’S THE REAL QUESTION?
• What are the users doing with our software?
• We know what we thought they’d do with it
• We’ve studied their behavior...sort of
• (someone has)
• But have we closed the loop? Have we fed that back
into the product development cycle?
• Why do we still do so many things by gut feel?
© Sauce Labs, Inc. 8
@mmerrell
FEBRUARY 22, 2020

9. BUT WHY??
No matter what risks we define, we are still in an
information vacuum before we release the
software.
© Sauce Labs, Inc. 9
@mmerrell
FEBRUARY 22, 2020

10. “Without requirements or design, programming
is the art of adding bugs to an empty text file”
—Louis Srygley
© Sauce Labs, Inc. 10
@mmerrell
FEBRUARY 22, 2020

11. THE JOURNEY
Planning UAT MaintainingMonitoringReleasingCoding Testing
© Sauce Labs, Inc. 11
@mmerrell
SHIFT LEFT SHIFT RIGHT
FEBRUARY 22, 2020

12. THE JOURNEY
© Sauce Labs, Inc. 12
Requirements
Real-world user Behavior is now
available before requirements are
gathered, better informing focus group
and research
Monitor Analytics
Look at both overall traffic (compared to
Model) and qualitative user behavior
patterns
Development
Development understands User Traffic
Model, and has more trust & confidence
in Requirements
Testing
New tests are added to automation suite
based on analytics gathered after previous
release cycle
Release
Feature flags, quick-rollback, and New User Traffic
Model is in-place before release
FEBRUARY 22, 2020

13. We must use data, driven by user signals,
to generate and validate requirements
@mmerrell
© Sauce Labs, Inc. 13FEBRUARY 22, 2020

14. @mmerrell
© Sauce Labs, Inc. 14
THOUGHT EXERCISE
t.ly/XBzMw
FEBRUARY 22, 2020

15. @mmerrell
© Sauce Labs, Inc. 15
THOUGHT EXERCISE
t.ly/XBzMw
FEBRUARY 22, 2020

16. FOR MONDAY: CLOSE THE LOOP
16
Have a cross functional retrospective—Not just cross-engineering
• Business Intelligence
• Customer Success
• Marketing
• even an Exec
• Keep it brief
• Keep it action-oriented

17. 17
• Ask how the user behaviors changed after release
• Identify opportunities and risks based on these inputs!
• Story time!
FOR MONDAY: CLOSE THE LOOP

18. OBSERVABILITY (O11y)
• Instrumentation
• Resilience
• Team Culture
• Release/Rollback Confidence
• Predictability
• Managed Tech Debt
• User Behavior, Well Understood
© Sauce Labs, Inc. 18
@mmerrell
FEBRUARY 22, 2020

19. WHERE DOES OBSERVABILITY FIT?
• Everywhere and nowhere—it’s not just about software
• Different areas of the system (and culture) give off signals
• These signals are studied by different teams at different times
• Silos don’t help us here
• Tooling, but also observation itself
© Sauce Labs, Inc. 19
@mmerrell
FEBRUARY 22, 2020

20. WHERE DOES OBSERVABILITY FIT?
© Sauce Labs, Inc. 20
@mmerrellt.ly/XBZDw
So much talk about
Observability is about
INFRASTRUCTURE
It needs to be about the
USER JOURNEY
FEBRUARY 22, 2020

21. WHERE DOES OBSERVABILITY FIT?
© Sauce Labs, Inc. 21
@mmerrellt.ly/XBZDw
Monitoring is an action you
PERFORM, to mitigate
risk
Observability is a
CHARACTERISTIC your
system possesses, which
lowers risk
FEBRUARY 22, 2020

22. FOR MONDAY: OBSERVABILITY
• For UI/Selenium Testers:
• Find an API that your UI depends on
• Find its Swagger spec
• Learn about the calls the UI is making
• Replace one or more Selenium steps with an API call
• For API/Back-end Testers:
• Learn about logging/tracing in your app
• How do developers diagnose production issues?
• For Exploratory Testers:
• What alerts will fire when errors occur in the app?
• What can those alerts tell you about your users’ actions?
@mmerrell
© Sauce Labs, Inc. 22FEBRUARY 22, 2020

23. RISK
Everything we do can be quantified in terms of risk
• The risk of doing something (or not doing something)
• The risk of the something being late
• The risk of the something being low quality
• The risk of the something being the wrong thing
• The risk of the market shifting
@mmerrell
© Sauce Labs, Inc. 23FEBRUARY 22, 2020

24. FOR MONDAY: RISK
The next time you report a bug, talk about it in terms of risk
• “Inserting a non-printable character into this field displays an
unattractive .NET stack trace to the user”
• “Inserting a non-printable character into this field surfaces a
security risk: the user sees a .NET stack trace, which allows
nefarious users to fingerprint our system”
@mmerrell
© Sauce Labs, Inc. 24FEBRUARY 22, 2020

25. WHAT DO WE PROPOSE?
~8 Stages (so far) of Shifting Right:
1. Smoke Tests in Production
2. Functional/Performance/Security Testing
3. Integrate with monitoring
4. Testing Analytics post-release
5. Testing full Analytics pipeline
6. Correlating UI tests with monitoring
7. Using Analytics to inform requirements
8. Using Analytics to generate tests
@mmerrell
© Sauce Labs, Inc. 25FEBRUARY 22, 2020

26. 26
There’s actually one more stage:
Stage 0: Give people real insights into their tests
WHAT DO WE PROPOSE?
@mmerrell

27. STAGE 0
• Test failure pattern identification
• Functionality “clusters”
• “Ideal Suite” generation
• Proxy-generated HAR files
• For analytics testing
© Sauce Labs, Inc. 27
@mmerrell
FEBRUARY 22, 2020

28. WHAT WE’VE TALKED ABOUT
• Shift Left
• Shift Right
• The Feedback Loop
• Observability
• User Behavior
• Risk
© Sauce Labs, Inc. 28
@mmerrell
FEBRUARY 22, 2020

29. WHERE DO I FIT?
• This is your responsibility
• (This is everyone’s responsibility)
• You have a unique perspective
• We have a unique opportunity
© Sauce Labs, Inc. 29
@mmerrell
FEBRUARY 22, 2020

30. WHERE DO I FIT?
What does the $$ want?
© Sauce Labs, Inc. 30
@mmerrell
FEBRUARY 22, 2020

31. WHERE DO I FIT?
The more you talk about
Observability, the more
Engineers will hear you
© Sauce Labs, Inc. 31
@mmerrell
FEBRUARY 22, 2020

32. WHERE DO I FIT?
The more you talk about Risk, the
more Executives will hear you
© Sauce Labs, Inc. 32
@mmerrell
FEBRUARY 22, 2020

33. REFERENCES
• O11y
• Charity Majors - @mipsytipsy
• Liz Fong-Jones - @lizthegrey
• O11y Maturity Model - t.ly/lKPWB
• https://opentracing.io/
• Analytics
• Anand Bagmar - @BagmarAnand
• t.ly/YOwLD
© Sauce Labs, Inc. 33
@mmerrell
FEBRUARY 22, 2020

34. THANK YOU

35. WHAT WE’RE ULTIMATELY AFTER
© Sauce Labs, Inc. 35FEBRUARY 22, 2020

36. THE JOURNEY
PLANNING UAT MAINTAININGMONITORINGRELEASINGCODING TESTING
© Sauce Labs, Inc. 36FEBRUARY 22, 2020