As a Front End Web Developer, experimenting with new tools to add to your workflow (and going down the rabbit hole with them!) is all part and parcel of refining your craft. Chris Eccles, Technical Manager at Building Blocks has been doing just this and has some invaluable insight into CSS Visual Regression using Backstop.JS. CSS Visual Regression testing is the process of running automated visual test comparisons on pages or elements in your projects. Using Backstop.JS, Chris has discovered that this tool is intuitive, allowing quick configuration to allow you to get up and rolling quickly. Backstop.JS serves your tests via a webpage which gives you the visual feedback needed for targeting bugs caused from CSS related issues. These comparisons can uncover bugs you’d otherwise not learn about until it’s too late. A very useful tool to have in your Front End arsenal, wouldn’t you agree? Chris has been sharing his insights with the BB team and wanted to share with our blog readers also. So, sit back and enjoy the ride through the wonderful world of Backstop.JS.

1. Backstop JS
Visual Testing

2. 1
I would define visual testing as using a headless browser to capture and
compare images of components of a website.
This can be used to implement unit tests for how the CSS is rendering the HTML
in a browser.
Not intended to test how the HTML is generated or replace Unit tests in a back
end service layer.
Not intended to replace cross browser testing.
Allows comparison between sites or comparing against a reference image
captured form the same site before changes were made.
What do you mean by Visual
Testing ?

3. 1. Chrome Headless
2. Setup and Examples
3. What to test
4. Setup in your project
Contents :

4. Chrome Headless
3
What has changed ?
Why is this a big deal ?

5. 4
 Headless Chrome is shipping in Chrome 59. It's a way to run the Chrome
browser in a headless environment. Essentially, running Chrome without
chrome! It brings all modern web platform features provided by Chromium
and the Blink rendering engine to the command line.
Why is that useful?
A headless browser is a great tool for automated testing and server
environments where you don't need a visible UI shell. For example, you may
want to run some tests against a real web page, create a PDF of it, or just
inspect how the browser renders an URL.
Announcement blog :
https://developers.google.com/web/updates/2017/04/headless-chrome
How to Get it:
As long as you have a version of chrome installed greater than Chrome 59 –
( Chrome 61 on Windows) you already have it 
Chrome Headless

6. 5
 BackstopJS automates visual regression testing of your responsive web UI by
comparing DOM screenshots over time.
Backstop is a open source project to run visual tests using headless browsers to
capture screenshots.
Originally ran using either PhantomJS or SlimerJS headless browser libraries.
These were implementations of WebKit (Chrome) or Gecko (Firefox).
From release v3.0.19 this now supports Chrome headless.
The latest release has a lot of stability improvements v3.0.27
Git Hub URL : https://github.com/garris/BackstopJS
Features :
• Allows capture and comparison of screenshots form headless browser
• Excellent comparison tool to identify and highlight differences
• Allows setup of multiple breakpoints to test responsive sites
• Uses simple CSS selectors to identify what to capture
• Pre capture scripts can be enabled to be ran
Backstop JS

7. 6
Chrome is based on WebKit but using PhantomJS as a headless browser
caused issues making it unusable.
There would be styling issues that would be thrown that would not be present on
the site when viewed through chrome.
As visual testing is about how the site renders this causes the tests to be
unreliable.
When I investigated implementing visual testing this was the stumbling block
each time.
Chrome != Webkit

8. Setup and Examples
7

9. 8
• backstop init : Set up a new BackstopJS instance -- specify URLs, cookies,
screen sizes, DOM selectors, interactions etc. (see examples directory)
• backstop test : BackstopJS creates a set of test screenshots and compares
them with your reference screenshots. Any changes show up in a visual
report. (Run this after making CSS changes as many times as needed.)
• backstop approve : If the test you ran looks good, then go ahead and
approve it. Approving changes will update your reference files with the
results from your last test. Future tests are compared against your most
recent approved test screenshots.
The BackstopJS workflow

10. 9
backstop init

11. 10
backstop test (initial)

12. 11
backstop approve

13. 12
backstop test

14. 13
{
"id": "backstop_default",
"viewports": [
{
"label": "phone",
"width": 320,
"height": 480
},
{
"label": "tablet",
"width": 1024,
"height": 768
}
],
"onBeforeScript": "chromy/onBefore.js",
"onReadyScript": "chromy/onReady.js",
"scenarios": [
…………
],
"paths": {
"bitmaps_reference": "backstop_data/bitmaps_reference",
"bitmaps_test": "backstop_data/bitmaps_test",
"engine_scripts": "backstop_data/engine_scripts",
"html_report": "backstop_data/html_report",
"ci_report": "backstop_data/ci_report"
},
"report": ["browser"],
"engine": "chrome",
"engineFlags": [],
"asyncCaptureLimit": 5,
"asyncCompareLimit": 50,
"debug": false,
"debugWindow": false
}
Default Configuration
{
"label": "BackstopJS Homepage",
"cookiePath": "backstop_data/engine_scripts/cookies.json",
"url": "https://garris.github.io/BackstopJS/",
"referenceUrl": "",
"readyEvent": "",
"readySelector": "",
"delay": 0,
"hideSelectors": [],
"removeSelectors": [],
"hoverSelector": "",
"clickSelector": "",
"postInteractionWait": "",
"selectors": [],
"selectorExpansion": true,
"misMatchThreshold" : 0.1,
"requireSameDimensions": true
}

15. 14
On the CIPD PM build we have incorporated it in to the HTML dev process.
When components are created a new test scenario is added to the backstop
tests.
This captures the components from the locally setup demo site.
This creates a reference image for that component that is then checked in to
source control.
Project Setup

16. What to Test
15

17. 16
Visual regression testing is the process of creating reference images for each
component as they are created. This allows a comparison over time to monitor
changes.
Ran by the developer on their local development environment after changes
made to ensure no regressions issues have occurred due to their changes.
Why is that useful?
• This allows the developer greater test coverage than they could do manually
when ensuring their changes have not had regression impact on other
components.
• Allows greater detail comparison than possible reviewing the site manually.
E.g. If the font size is increased by 1px then it will be caught which could
easily be missed.
• Allows developers without a full understanding of the project have a safety
net about what the site should look like before their changes.
Visual Regression Testing

18. 17
Using visual testing to compare different environments involves comparing one
site against another . The reference images are not stored beyond the test.
A reference image set is generated form one site and then another site is
compared against them
Why is that useful?
• To ensure that the HTML created has been integrated within the website
correctly and is rendering out in the same way. This highlights mistakes with
the integration or conflicting integrations that are not present on the HTML
Demo site.
• Running regression tests against the Staging and Live instances of a site to
see where the changes have occurred.
• When setting up comparison between a demo and dev site powered by a
CMS this can make the tests brittle so the tests pages need to be isolated
from changes in content as possible.
Environment Comparison

19. 18
Within a site their are hover states and interactive components. These need the
mouse or user interactions to trigger them.
How to test these in a headless browser?
Backstop allows scripts to be ran before a screenshot is captured. Using the
chromy API you can move the mouse and manipulate the page through script.
This allows testing of :
• Field entry
• Hover states
• Interactive items
• Video loading – Playing
• Ajax content loading
Chromy Git Hub URL : https://github.com/OnetapInc/chromy
Interacting with the site

20. 19
When capturing items it is much quicker to capture components on a page than
the whole page to compare them
• It is much quicker to compare the smaller images
• It is more relevant the items that are being tested.
This allows the components themselves to be bested in isolation and most of our
development work is done using component based development.
Smaller is better

21. Setup in your project
20

22. Process to include in you
project
21
Install
Backstop
init within you
project Create
scenarios
Run Baseline
generation
Keep running
and
monitoring

23. 22
Prerequisites
• Node – (If you don’t already have it)
• Chrome - (61 needed but latest is 62.0.3202.62 at the moment)
• Update node to v8+
• Update npm to v5+
On Windows you will also need to have python installed
> PhantomJS needs Python -- please make sure you have Python installed...
see https://github.com/garris/BackstopJS/issues/185
Installation
• npm install -g backstopjs
• Installs backstop globally
This allows the backstop command to be ran from anywhere on the PC.
Install Backstop - Globally

24. 23
Backstop can also be installed as a grunt dependency and called through grunt.
var backstopjs = require('backstopjs');
grunt.registerTask('backstop', 'BackstopJS integration', function(cmd) {
// cmd is either 'reference', 'test', or 'openReport'
var done = this.async();
backstopjs(cmd).then(function() {
done(true);
}).catch(function() {
done(false);
});
});
This allows it to be integrated in to the existing grunt tasks.
Including it in grunt also allows the dynamic creation of the scenarios using
variables instead of them being held in a JSON file.
Installing Backtop in Grunt

25. 24
There is a basic backstop.json file that is created when you run backstop init.
This can be edited to add in your own scenarios.
This is an example of the setup we are using.
Key Points:
• The Viewports are defined for use in all tests
• resembleOutputOptions define the comparison colours and display style
• The asyncCaptureLimit and asyncCompareLimit are used to control the
number of threads and memory the tests use.
• "engine": "chrome“ defines that the chrome headless browser will be used.
Configuration Setup
{
"id": "pm",
"viewports": [
{
"label": "phone",
"width": 320,
"height": 1024
},
{
"label": "tablet",
"width": 768,
"height": 1024
},
{
"label": "desktop",
"width": 1268,
"height": 1024
}
],
"onBeforeScript": "chromy/onBefore.js",
"onReadyScript": "chromy/onReady.js",
"scenarios": [
{
……..
}
],
"resembleOutputOptions": {
"errorColor": {
"red": 255,
"green": 0,
"blue": 255
},
"errorType": "movement",
"transparency": 0.3,
"ignoreAntialiasing" : true
},
"asyncCaptureLimit": 25,
"asyncCompareLimit": 400,
"engineFlags": [],
"engine": "chrome",
"report": ["browser","CI"],
"debug": false,
"debugWindow": false
}

26. 25
For scenarios setup there are two key parameters :
• url
• selectors
These define the page the what will be captured on the page. The selectors
used simple CSS selectors to identify elements on the page to capture.
To capture the contents of the <div id=“header”> then the #header is used as a
selector.
If a referenceUrl is specified then this will be used instead of the url when
generating the reference images. This is used when comparing sites.
The hideSelectors parameter is used to define the elements on the page that
should not be shown. This is useful to hide cookie notifications for example that
could obscure capture of components.
Scenario Setup
{
"label": "header",
"url": "http://people-management.local/bb-prototype-
website_example_grid.html",
"selectors": [
"#header"
],
"hideSelectors": [
"#monitor_mq",
".grid",
".visible-grid-btn"
],
"readySelector": ".page-loaded“
},
{
"label": "podcast",
"url": "http://beta.pmdev.co.uk/test/podcast",
"referenceUrl": "http://internal.people-
management.demo.building-blocks.com/bb-prototype-
website_components-podcast.html",
"selectors": [
".podcast-container"
],
"hideSelectors": [
"#monitor_mq",
"#SEStartButton",
"#cookieMsgWrapper2",
".grid",
".visible-grid-btn"
]
}

27. 26
Scenario Setup :
"onReadyScript": "burgerMenuTest.js",
Chromy Script :
module.exports = function (chromy, scenario) {
var clickSelector = scenario.clickSelector;
var postInteractionWait = scenario.postInteractionWait; // selector [str] | ms [int]
if (clickSelector) {
chromy
.wait(clickSelector)
.click(clickSelector)
}
if (postInteractionWait) {
chromy.wait(postInteractionWait);
}
};
Chromy

28. Backstop JS the future …..
27
• Catch Regression
• Allow comparison between DEMO HTML and Website
• Quick to setup
• Easily integrate in to existing HTML build process