In the presentation, Ashley will review how Blackboard revived a dead automated test suite and turned it into a gold standard, transforming the way the company thinks about software quality along the way. Topics covered will include: - QA is taboo! (we are now Engineering Productivity – but what does that mean?) – establishing goals as a department goes beyond ensuring quality. How did Blackboard build the teams needed to insure success? - Documentation is like pulling teeth, but so very critical – how the documentation that was created laid the groundwork to improve. She will show examples that have helped teams get on the right track. - Getting the right environment and pieces in place – and slashing execution times (going from our own PD Lab to the world of Mesos). - Project Guardrails. Feature by feature, risk analysis, defining/redistributing/removing tests, and getting developers to trust the tests. Pairing our team with scrum teams fix their tests. It wasn’t all fun and games – what were some of the hiccups that you can learn from Blackboard? Ashley Hunsberger – Test Automation Architect, Blackboard

1. Transforma)ve Culture
The shi' from QA to Engineering Produc6vity

2. @aahunsberger
/ashleyhunsberger

3. About a year ago…

4. 1 hr.
370 Unreliable FUTURE?
33
It wasn’t all bad…

6. 1 hr.
370 Unreliable FUTURE?
33
It wasn’t all bad…

7. So what to do?

8. “Produc6vity is our job; tes6ng and
quality are the job of everyone
involved in development. This means
that developers own tes6ng and
developers own quality. The
produc6vity team is responsible for
enabling development to nail those
two things.”
- Patrick Copeland, How Google Tests
So9ware

9. QA is Taboo!
The shi' to Engineering Produc6vity
begins…

10. Reduce the 6me from concept to
deliverable by providing our product
development teams with
the tools, prac2ces and support to
increase their produc6vity while
maintaining high quality standards.
Mission

11. Provide an easily maintainable and
extensible framework that enables
scrum teams to add and remove tests
Goal #1

12. Enable the automa2c and early
detec2on of failures within the
so'ware under development
Goal #2

13. Prevent the source of detected failures
from moving any further downstream Goal #3

14. Accommodate all of this without
impac2ng the feature engineer’s 6me. Goal #4

15. The Team

16. Product
Owner
The Team

17. Product
Owner
Scrum
Master
The Team

18. Product
Owner
Scrum
Master
SETs
The Team

19. Product
Owner
Scrum
Master
SETs Engineer
The Team

20. Product
Owner
Scrum
Master
SETs Engineer
The Team CI/CD

21. Documenta)on

22. All code delivered during the
project to provide beZer tooling,
prac6ces, and support has been
reviewed, tested, documented
and released.
All documenta6on created to
beZer support development
teams has been reviewed, tested,
communicated and published in a
convenient loca6on.
Defini)on of Done

23. Goals Trigger
Gate Requirements
Test Suite Defini)on

24. Goals Trigger
Gate Requirements
Test Suite Defini)on

25. Goals Trigger
Gate Requirements
Test Suite Defini)on

26. Goals Trigger
Gate Requirements
Test Suite Defini)on

27. Goals Trigger
Gate Requirements
Test Suite Defini)on

30. Environments
Insert picture of environment here.

31. Single Slave
(VM)

34. Jenkins
Pipeline
(Build Per
Branch)

35. Project Guardrails

36. Guardrails
• Three phases – (1) Quality review; (2) Clean up exis6ng tests (3) add or remove other tests.
• Defini6on of Done
• Risk Analysis
• Impact
• Likelihood
• Risk priority number
• Sedng a threshold
• At Bb – our ‘general guidance’ is 6.
• hZps://docs.google.com/spreadsheets/d/1j8RV0wJsAgAVZqOZyXfFBugtNEuTzfIVE7bA3s_G5bg/edit?usp=sharing
hZps://docs.google.com/spreadsheets/d/1j8RV0wJsAgAVZqOZyXfFBugtNEuTzfIVE7bA3s_G5bg/edit?usp=sharing
• Quaran6ne Strategy
• Point is to further ins6ll confidence in the tests. We want reproducible failures.
• Figuring out where to focus energy… allows us to focus on flaky tests while not impeding dev workflow in the pipeline.
• E2E Retry strategy – reruns test if it fails up to three 6mes; if it passes, we do not fail the build. Test gets inves6gated. Make
call to take test out and into quaran6ne.
• We are consultants now! Representa6ve from our team that works with the scrum teams (risk analysis AND
code review);
• One of the first 6mes I’ve seen devs and QA really working together on what to test.

38. Risk Analysis Guidelines
Stability Gate
Test Suite Defini)on

39. # Quality Risk Likelihood Impact Risk
Priority #
Extent of
Tes2ng
Tracing
Instructor can grade a
test 2 1
2
0
0
0

40. Don’t repeat our mistakes…

41. Lessons Learned
• Have a quaran6ne strategy in place! (keeping things reliable and building
trust with developers)
• Checking failures is hard work! No one likes to do it.
• Everyone has a different idea of what’s cri6cal.
• Risk Analysis got the FULL team talking, and became easier to decide as a team as
they set their threshold for a feature.
• Not everything can have impact or likelihood of 1
• Everyone wants to test everything – but you can’t.
• Regardless of threshold, cannot test everything in your list or you will go bankrupt.
• Scalable environment is key to our success
• Dockerize or containerize repeatable processes (like our E2E cache – all the
dependent node modules)

42. 1 hr.
370 Unreliable FUTURE?
33
It wasn’t all bad…

43. 30 min
Quaran6ne More…
165

44. Thank you!