Presentation for our paper at NEXTA2020 workshop at ICST conference.

1. @maaretp @yuqingwang5
Test Automation Process
Improvement in a DevOpsTeam:
Experience Report
by Maaret Pyhäjärvi byYuqingWang

2. @maaretp @yuqingwang5
Change of jobs 5 months ago, and this
published story is from…
Windows Endpoint Protection,
Protection for Businesses
Not aTEST team
but a PRODUCT team
Unit testing considered inseparable
From application programing

3. @maaretp @yuqingwang5
(Note:Testomat Self-Assessment: 39/80)

4. @maaretp @yuqingwang5
Open coding ->
success factors
A final set of
critical success
factors
The evolution of test automation
practices from 2005 to 2019
Online
meetings
among co-
authors
Research method
Maaret wrote
experience notes
Telemetry
statistics
Release and test
automation
change log
excerpts

5. @maaretp @yuqingwang5
High satisfaction and maturity (self-reported 2019)
- Increasing speed to release
(from 2-4 releases in 2008 to 9 releases in 2019)
- Decreasing time to making release
(from 5 days in 2018 to 4 hours 2019)
- Improving team productivity
(11 team members, 2917 code changes out of 6583 for 3 teams)
- Finding relevant issues
(new product in days, MANY timing issues and regressions)
- High satisfaction of customers
(167 support issues, 1 escalation for millions of customers)
- High test efficiency
(213 708 tests in single working day)

6. @maaretp @yuqingwang5

7. @maaretp @yuqingwang5
Perceived Success Factors
Dimension Factor
Human Whole team effort
Expert team members
Self-motivated team members
Organizing Allow time for learning curve
Internal open source community mindset
Technical Test tool choice and architecture
Testlab infrastructure
Product testability
Telemetry
Process Incremental approach
Process observation and optimization

8. @maaretp @yuqingwang5
Whole Team Effort
• 2005: two-personTA team
• 2009: architecture with testability
• 2016: new product from codebase, team starts
• 2018: whole teamTA ‘organically’
The team understood testing.

9. @maaretp @yuqingwang5
Internal Open Source Community
• 2009: anyone can change any code
• 2016: our ownTA repo
• 2018: combinedTA repos
You can’t control change,
have tests.

10. @maaretp @yuqingwang5
Product and TA telemetry
• Product telemetry in test environment tells
coverage
• TA telemetry enables analyzing 200 000 tests
a day effectively
Numbers make sense when
there is mass of things

11. @maaretp @yuqingwang5
Incremental approach
• What we had was a result of lessons from 15
years
• Every day is a chance of making things better
• Every day you learn
Importance of experimentingImportance of experimenting
and respecting what we had
+ stretching it

12. @maaretp @yuqingwang5
Conflict with previous research!
• No explicit test automation strategy
• No careful tool selection
• No quality measurement of test automation
• No explicit guidelines
“…relaxed, verbally communicated strategy
without strict rules and processes relying on
developers’ voluntary participation...”

13. @maaretp @yuqingwang5