We build software that runs on 21+ platform with millions of users and billions of deployments. Our software engineers have to keep the high pace of quality. More than 500k tests are executed daily. In this talk I would like to share my experience of dealing with unstable and long running tests, collecting, aggregating and analysis test execution data.

1. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
ESCAPE FROM THE
AUTOMATION HELL
• By Yan Drugalya

2. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
KID DREAMS

3. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
WHO I BECAME

4. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
HOW OTHERS SEE ME

5. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
NOW

6. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
NOW

7. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
AUTOMATION

8. COPYRIGHT 2014 @ UNITY TECHNOLOGIES

9. COPYRIGHT 2014 @ UNITY TECHNOLOGIES

10. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
CHALLENGES
• Slow tests
• Unstable tests
• Hard to run tests locally
• Hard to reproduce failure locally

11. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
FIRST THERE WERE THE SCOUTS

12. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
AFTER ½ YEARS OF SCOUTING

13. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
THE HOPE

14. COPYRIGHT 2014 @ UNITY TECHNOLOGIES

15. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
UNIFIED TEST RUNNER
• A Simple command line that changed a lot
test -suite=<runtime|integration…>

16. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
TDD IN PERL?

17. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
ISSUES
• Slow tests
• Unstable tests
• Hard to run tests locally
• Hard to reproduce failure locally

18. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
NEXT STEP
• Build farm started to use UTR

19. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
ISSUES
• Slow tests
• Unstable tests
• Hard to run tests locally
• Hard to reproduce failure locally

20. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
HOARDER

21. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
BuildAgents&Humans Hoarder WS

22. COPYRIGHT 2014 @ UNITY TECHNOLOGIES

23. COPYRIGHT 2014 @ UNITY TECHNOLOGIES

24. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
NICE DISCOVERIES
• 16 559 749 tests run in Aug 2015!
• Never failed tests
• Compare performance regressions
• How people use UTR to run tests locally
• Automatic discovery of unstable test
• ....

25. COPYRIGHT 2014 @ UNITY TECHNOLOGIES
FUTURE
• Immediately get an overview of the state of our test
automation
• Automatically find and assign test failures
• Extend the REST API interface to make it easier to
integrate with other tools we use at Unity