Test Automation - A Right Way!

2. Inverting Test Pyramid
Experience Report
Sachin Natu @natusachin
Naresh Jain @nashjain

3. 25YEARS 6000+CLIENTS Nr 1M ROOMS 94COUNTRIES 95%+
ago we started
later we’re still finding
around the world are
helping clients
the revenue opportunity
priced every single day
manage their
in every situation
using IDeaS
revenue
on 6 continents have
hotels that now run on
IDeaS Revenue
Management
Systems
More than 95
percent of all
our clients are
still working
with us today.
About IDeaS

4. Our Worldwide Clients :
Hospitality + Travel + Distribution

5. Agenda
● Where were
we?
● Problem
● How we tried to
solve?
● How we
optimized?

6. Status of regression time
Page

7. No Automation !

8. Problems
● Thousands of
regression tests
● Cross cutting defects
found were more.
● Repeat Exeuction

9. Moving to shorter release cycle - a dream?

10. Regression period is incompressible

11. Manual and repetitive tasks are error prone.

12. Issue leakages
Page
Releases

13. Let’s automate regression
Page

14. UI Test
Page

15. After Automation Introduced

16. Production feedback
Page

17. Duration…
2
Yrs
Team of 2 full time Automation Testers

18. State of the Product
Manual
Checking 45%
End- to End UI
Tests 40%
Integration Tests
10%
Unit Tests
5%

19. Are we happy now?
Page

20. Always Catching Up

21. No pinpointed feedback

22. Silos

23. Tests are slow

24. Costly to maintain

25. Is there way out?
Page

26. Analysis of Bugs Report

27. Why test from far

28. The Test Pyramid
Typical test strategies lead to an inverted testing
pyramid
Manual
Checking
End- to End Selenium
Tests 70%
Integration Tests
20%
Unit Tests
UI
1
%
End to
End Flow
Tests 4%
Workflow API
Tests 6%
Integration Tests 9%
Domain Logic Acceptance
Tests 10%
10% Unit Tests 70%
This is the need of the hour…. Courtesy: Naresh Jain @AgileFAQs.com

29. Our transition story…
• We decided to "Test things right , at
right place on right time" which would
result in "right test pyramid“
UI
1
%
End to
End Flow
Tests 4%
Workflow API
Tests 6%
Integration Tests 9%
Domain Logic Acceptance
Tests 10%
Unit Tests 70%

30. Business logic test

31. Current Status

32. Benefits

33. Automation is part of development
• TDD
• BDD
• UI Tests only worry for UI

34. Improved collaboration
Developer QA

35. Grouping Tests - ROI
80 Scenarios - 300 + Min!

36. Grouping Tests - ROI - Pyramid
UI Tests - 4 Min
End to End - 4 Min
80 Scenarios < Min!

37. Regression timelines
Months
Page

38. Feels like Agile now...
Page

39. Challenges
• Legacy code - “Unit test
unfriendly”
• Mapping acceptance tests
with various slices of
pyramid
• Building team
competencies
• Pairing/ collaboration is
backbone

40. Key Learnings
• Do not overdo UI
regression automation
• Keep it
simple/pinpointed
• Design for Testability
UI
1
%
End to
End Flow
Tests 4%
Workflow API
Tests 6%
Integration Tests 9%
Domain Logic Acceptance
Tests 10%
Unit Tests 70%

41. Naresh Jain @nashjain
Sachin Natu @natusachin
Page

42. Some Statistics...