3. 3@shwetasharma84
Fully-distributed, 6 Time Zones
10 Years Of Growth
120+ Enthusiastic, Kind,
Open Professionals
100+ Enterprise Engagements
Qualitative Drupal
Contributor
Axelerant from above
4. 4@shwetasharma84
01
What is Automated Visual
testing?
02 How does it work?
03
Level - 1 (Objectives, Tools,
demo, challenges-resolutions)
04
Level - 2 (Objectives, Tools,
demo, challenges-resolutions)
05
Level - 3 (Objectives, Tools,
demo, challenges-resolutions)
Agenda
06 Comparison between tools
07 Testing Strategy
08 Good practices
09 Key results achieved
10 Facts
11 Where to go from here?
12 References
6. 6@shwetasharma84
How does it work?
Test Runner
Capture screenshots at run time
Test passes
Update baselineBug
Compare
Capture baseline images
First run
Subsequent runs
unintended intended
Fails and report differences
7. 7@shwetasharma84
● Site development in
progress/maintenance
● Optimise testing efforts
as an individual
● Resolve challenges
faced at individual level
LEVEL 1 -
Beginner
20. 20@shwetasharma84
GET STARTED - TOOLS AVAILABLE
Open-source BackstopJS, Wraith, Spectre
Commercial Applitools SIDE (Selenium IDE) extension, Percy, Diffy
21. 21@shwetasharma84
LEVEL 2 - OBJECTIVES and CHALLENGES
OBJECTIVES CHALLENGES
Involve more than 1 individual so that
more people benefit from automation
Baseline images created from one
machine fail on others due to anti-aliasing
Simulating user actions like filling up
forms, clicking, hovering etc.
Seamless integration with existing testing
frameworks
Optimise testing efforts for a small
development/testing team
Harmonious integration with your CI setup
22. 22@shwetasharma84
Resolution for Level 2 challenges
● Anti-aliasing
○ Use of Docker so that the environment is the same every time
○ Run visual tests on cloud
■ BrowserStack
■ SauceLabs
■ AWS Device Farms
■ Cross Browser testing
■ Select a tool which handles it implicitly
● Integration with existing testing frameworks
○ Open source and commercial tools available
● Integration with the CI setup
○ Commercial tools available
29. 29@shwetasharma84
LEVEL 3 - OBJECTIVES and CHALLENGES
CHALLENGES CAUSE OBJECTIVES
Running into false positives ● Small vertical/horizontal
shifts
● Browser updates
Avoid pixel-to-pixel matching
Increased time test on several
browsers & devices
Execution happens on all
configured browsers/devices
Better Test coverage for visual
tests at a faster pace
Frequently/Major change in
elements
Change requests Better baseline management
Scattered tests, poor readability Inability to group Better Test Suite management
32. 32@shwetasharma84
BROWSERS/DEVICES & SCATTERED TESTS - Resolution
● APPLITOOLS
○ Supports all browsers &
devices
○ Executes once and
captures everywhere in
background in parallel
○ Create relevant Batches
and group tests
● PERCY
○ Supports only Chrome &
FF
○ Supports only viewports
33. 33@shwetasharma84
Comparison between tools (Must have)
PARAMETER PERCY APPLITOOLS
Pixel-to-pixel matching ✅ ✅
AI-based matching ❌ ✅
Browsers Chrome and Firefox All
Devices Only Viewports All
Dynamic content
handling
✅ ✅
Baseline management ❌ ✅
Team management ✅ ✅
34. 34@shwetasharma84
Comparison between tools (Need basis)
PARAMETER PERCY APPLITOOLS
Root cause analysis ❌ ✅
Integration tools Source control, build
servers, collaboration
Source control, build
servers, collaboration
SDK support Good Excellent
On-premise support ❌ ✅
Collaborative
dashboard
❌ ✅
Desktop, mobile app
support
❌ ✅
36. 36@shwetasharma84
Testing Strategy
● Capturing screenshots
○ Element level
■ Component based libraries
● Storybook
● Pattern Lab
■ Development in progress
● Group the components logically (For e.g. Header, Footer etc.)
○ Full page
■ Comparing two similar environments
■ Entire page is developed
37. 37@shwetasharma84
Testing Strategy contd...
● Plan the level of visual coverage needed
○ Both for browsers and devices
■ Every project is different as our customers are different
■ Identify patterns from previously learned lessons
○ Application wide
■ Don’t capture all pages
■ Identify appropriate samples
38. 38@shwetasharma84
Good practices to follow
● Organize Test Suites
○ Browser wise
○ Feature wise
○ Device wise
● Speed up test execution by running tests in parallel
● Identify the frequency to run visual tests
41. 41@shwetasharma84
Key results achieved at Axelerant
● Test data
○ Time saved to handle dynamic content scenarios because of data consistency
○ Minimal efforts for Visual tests
● Seamless integration of Automation tools involved
● Visual tests validated for the site which is a replica of the wireframe
● Less number of assertions required for the Acceptance test Automation suite
42. 42@shwetasharma84
Facts
● Efforts needed to maintain baseline images
● Respect the Test Pyramid
○ Include Unit tests, Service tests, Acceptance tests and Visual tests
● Best practices in Automating Visual tests:
○ Ensure full page validation
○ Do not have too many element specific tests
■ Avoid element locators maintenance
○ General rules for automating Acceptance tests apply here too:
■ Pick up the right candidates for Automation
■ Logical division of tests
■ Usage of Seams
● Do not expect overnight success
43. 43@shwetasharma84
Where to go from here?
● No coding knowledge?
○ Use configuration based tools like BackstopJS, Wraith, Applitools SIDE browser
extension
○ Pair up with developers, SDETs to have the structure ready
● Good with coding?
○ Look for harmonious integration with your automated functional suite
■ Reduce the unnecessary functional assertions
● Maintainable suite
● Better execution time
○ Be brave and bring the tests in your CI pipeline
■ Let the whole team benefit
44. 44@shwetasharma84
References
● BackstopJS
● WebdriverIO - Visual Regression
Service
● Applitools for Selenium IDE Tutorial
● Visual Regression Testing Using Wraith
● Run Selenium tests with WebDriverIO
● Cypress visual testing plugins
● Diffy
● Spectre
● Applitools Eyes
● Applitools Ultrafast Test Cloud
● Percy - Getting started