Fullstack End-to-end test automation with Node.js, one year later

Fullstack End-to-End Test
Automation with Node.js
One year later
SF Selenium Meetup @ Saucelabs
Chris Clayman
Mek Srunyu Stittri

2
Agenda
● Background
○ What we presented last year
● Async / Await - an alternative to Webdriver’s built-in control flow.
○ Limitations with control flow
○ Use Babel to write the latest ES6 JavaScript syntax.
ES6 Pageobjects
● Extending MochaJS
○ Custom reporter with screenshots from Sauce Labs
○ Parallel tests and accurate reporting
● Type-safe JavaScript with Facebook’s Flow-type library.
● Robust visual diffs
● What’s next

3
Background
Back in 2015, we started looking at node.js
for end-to-end functional test framework.
● Kept Node.js adoption in mind
○ More and more company moving to node.js
○ Share code with fullstack developers
● Team presented at San Francisco Selenium Meetup
in Nov 2015
○ Event - http://www.meetup.com/seleniumsanfrancisco/events/226089563/
○ Recording - https://www.youtube.com/watch?v=CqeCUyoIEo8
○ Slides - http://www.slideshare.net/MekSrunyuStittri/nodejs-and-selenium-webdriver-a-journey-from-the-java-side

4
Why we chose node.js
QA, Automation engineers
Frontend engineers
Backend engineers
Java
Javascript
Java
Python
Javascript
Java
Ruby
Javascript
Node.js
Company A Company B Company C
Node.js
Javascript
Node.js
Go, Python
Airware

5
What we presented last year
Input / update
data
Get data
Input / update
data
Get data
UI Framework : node.js
● selenium-webdriver
● mocha + wrapper
● Applitools
● co-wrap for webclient
● chai (asserts)
Rest API Framework :
node.js
● co-requests
● mocha
● co-mocha
● chai (asserts)
● json, jayschema
WebClients
Pageobjects
Webclient
adaptor
Database
Backend : Go, Python
Browser
Frontend : Javascript
Microservice
#1
Microservice
#2 ... #n
Rest APIs

8
Heavily dependent on selenium-webdriver control flows
http://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/lib/promise.html
What is the promise manager / control flow
● Implicitly synchronizes asynchronous actions
● Coordinate the scheduling and execution of all commands.
Maintains a queue of scheduled tasks and executing them.
driver.get('http://www.google.com/ncr');
driver.findElement({name: 'q'}).sendKeys('webdriver');
driver.findElement({name: 'btnGn'}).click();
driver.get('http://www.google.com/ncr')
.then(function() {
return driver.findElement({name: 'q'});
})
.then(function(q) {
return q.sendKeys('webdriver');
})
.then(function() {
return driver.findElement({name: 'btnG'});
})
.then(function(btnG) {
return btnG.click();
});
The core Webdriver API is built on top of the control
flow, allowing users to write the below.
Instead of thatThis

9
Achieving Sync-like Code
Code written using Webdriver Promise Manager
JavaScript selenium tests using Promise Manager
driver.get("http://www.google.com");
driver.findElement(webdriver.By.name('q')).sendKeys('webdriver');
driver.findElement(webdriver.By.name('btnG')).click();
driver.getTitle().then(function(title) {
console.log(title);
});
Equivalent Java code
driver.get("http://www.google.com");
driver.findElement(By.name("q")).sendKeys("webdriver");
driver.findElement(By.name("btnG")).click();
assertEquals("webdriver - Google Search", driver.getTitle());
Hey, we look similar now!

10
MochaJS with Webdriver Wrapper
Provided Mocha test wrapper with Promise Manager
Selenium-webdriver’s wrapper for mocha methods that automatically handles all calls into the
promise manager which makes the code very sync like.
var test = require('selenium-webdriver/testing');
var webdriver = require('selenium-webdriver');
var By = require('selenium-webdriver').By;
var Until = require('selenium-webdriver').until;
test.it('Login and make sure the job menu is there', function() {
driver.get(url, 5000);
driver.findElement(By.css('input#email')).sendKeys('useremail@email.com');
driver.findElement(By.css('input#password')).sendKeys(password);
driver.findElement(By.css('button[type="submit"]')).click();
driver.wait(Until.elementLocated(By.css('li.active > a.jobs')));
var job = driver.findElement(By.css('li.active a.jobs'));
job.getText().then(function (text) {
assert.equal(text, 'Jobs', 'Job link title is correct');
});
});
Mocha wrapper makes
the code very
“synchronous” like.

11
test.it('Verify data from both frontend and backend', function() {
var webClient = new WebClient();
var projectFromBackend;
// API Portion of the test
var flow = webdriver.promise.controlFlow();
flow.execute(function *(){
yield webClient.login(Constants.USER001_EMAIL, Constants.USER_PASSWORD);
var projects = yield webClient.getProjects();
projectFromBackend = projectutil.getProjectByName(projects, Constants.QE_PROJECT);
});
// UI Portion of the test
var login = new LoginPage(driver);
login.enterUserInfo(Constants.USER001_EMAIL, Constants.USER_PASSWORD);
var topNav = new TopNav(driver);
topNav.getProjects().then(function (projects){
Logger.debug('Projects from backend:', projectsFromBackend);
Logger.debug('Projects from frontend:', projects);
assert.equal(projectsFromBackend.size, projects.size);
});
Heavily dependent on the promise manager / control flow
Here we handle execution order including a generator call
What We Presented Last Year
Context switching
to REST API calls

Alternatives to Webdriver’s Built-in
Control Flow
Async / Await

JS and Webdriver: the Good, the Bad...
Good Stuff
● Functional programming
● More Collaboration with
front-end development teams
● JavaScript Developers writing
Selenium tests
● Fast paced open source
community
● Able to build things really quick
● JavaScript is fun!
Bad Stuff
● Webdriver’s Control Flow & Promise
Manager
○ Not agnostic
○ Parent variable declarations
○ Iteration can be hacky
● Context switching between
Webdriver and non-Webdriver
asynchronous function calls

14
...and the Ugly
const promise = require('selenium-webdriver').promise
PageObject.prototype.getMessages = function() {
const els = this.driver.findElements(By.css('.classname');
const defer = promise.defer();
const flow = promise.controlFlow();
flow.execute(function* () {
const textArray = yield els.map((el) => {
return el.getText();
});
defer.fulfill(textArray);
});
return defer.promise;
}
We want our tests to be:
● Readable / Flat structure ✔
● Agnostic / Context-free ❌
● De-asynchronous ✔
● In line with ECMA standards ❌
Context Switch

15
...and the (kind of) Ugly
test.it('Archives job', function () {
let job;
// Create a job through API
job = yield webClient.createJob();
driverutil.goToJob(driver, job.id);
});
const jobInfo = new ViewJob(driver);
jobInfo.clickArchive();
jobInfo.isModalDisplayed().then((displayed) => {
assert.isTrue(displayed, 'Modal should be displayed');
});
yield webClient.deleteJob(job.id);
});
});
● Readable / Flat structure
Context Switch
Context Switch

16
JobsPage.prototype.getJobList = function () {
this.waitForDisplayed(By.css('.job-table'));
const jobNames = [];
const jobList = this.driver.findElement(By.css('.job-table'));
// get entries
flow.execute(() => {
jobList.findElements(By.css('.show-pointer')).then((jobs) => {
// Get text on all the elements
jobs.forEach((job) => {
let jobName;
flow.execute(function () {
job.findElement(By.css('.job-table-row-name')).then((element) => {
element.getText().then((text) => {
jobName = text;
});
// look up more table cells...
});
}).then(() => {
jobNames.push({
jobName: jobName
});
});
});
});
}).then(() => {
// fulfill results
defer.fulfill(jobNames);
});
};
...and the Really Ugly
Not planning for complexity + promise chaining =
● Readable / Flat structure ❌
● ‘De-asynchronous’ ✔

17
As if we had trainer
wheels on...
Doing complex things
with selenium-webdriver
promise manager ended
up taking more time and
being more cumbersome.
What It Felt Like

19
Async/Await
Introducing Async/Await
● ES2016 language specification grabbed from C#
● Async functions awaits and returns a promise
● No callbacks, no control flow libraries, no promise
chaining, nothing but simple syntax.
● ‘De-asynchronous’ done easy

20
function notAsync () {
foo().then((bar) => {
console.log(bar)
});
}
Async/Await
async function isAsync() {
const bar = await foo();
console.log(bar);
}
Given function foo() that returns a promise...
ES5 Javascript ES2016 Latest Javascript

21
PageObject.prototype.getMessages = function() {
const els = this.driver.findElements(By.css('.classname');
const textArray = yield* els.map((el) => {
});
defer.fulfill(textArray);
});
}
async getMessages() {
const els = await this.driver.findElements(By.css('.classname');
return Promise.all(els.map((el) => {
}));
}
We want our tests to be
● Portable / Context-free ✔
● In line with ECMA standards ✔
Promise Manager vs. Async/Await

22
test.it('Archives job', function () {
let job;
// Create a job through API
job = yield webClient.createJob();
driverutil.goToJob(driver, job.id);
});
jobInfo.clickArchive();
jobInfo.isModalDisplayed().then((displayed) => {
});
yield webClient.deleteJob(job.id);
});
});
Promise Manager vs. Async/Await
it('Archives job', async function () {
const job = await webClient.createJob();
await driverutil.goToJob(driver, job.id);
await jobInfo.clickArchive();
const displayed = await jobInfo.isModalDisplayed();
await webClient.deleteJob(job.id);
});
We want our tests to be
● Portable / Context-free ✔
● In line with ECMA standards ✔

23
How to use Async / Await
How did we get Async / Await?
● Babel compiles latest JS syntax to ES5
compatible code
● Babel-register can be a pre-runtime compiler in
Mocha.
● See the repo!

24
'use strict';
var BasePage = require('./BasePage');
var By = require('selenium-webdriver').By;
//Constructor for the Top Navigation Bar
function TopNav(webdriver) {
BasePage.call(this, webdriver);
this.isLoaded();
}
//BasePage and Constructor wiring
TopNav.prototype = Object.create(BasePage.prototype);
TopNav.prototype.constructor = TopNav;
TopNav.prototype.isLoaded = function () {
this.waitForDisplayed(By.css('.options'));
return this;
};
TopNav.prototype.openProjectDropdown = function () {
this.waitForDisplayed(By.css('.options'));
this.waitForEnabled(By.css('.options
ul:nth-of-type(1)'));
this.click(By.css('.options ul:nth-of-type(1)'));
return this;
};
'use strict';
import BasePage from './BasePage';
import { By } from 'selenium-webdriver';
import ui from './../util/ui-util';
export default class TopNav extends BasePage {
//Constructor for the Top Navigation Bar
constructor(webdriver: WebDriverClass) {
super(webdriver);
}
async isLoaded(): Promise<this> {
await ui.waitForDisplayed(this.driver, By.css('.options'));
return this;
}
async openProjectDropdown(): Promise<this> {
await ui.waitForDisplayed(this.driver, By.css('.options'));
await ui.waitForEnabled(this.driver, By.css('.options ul:nth-of-type(1)'));
await ui.click(this.driver, By.css('.options ul:nth-of-type(1)'));
return this;
}
}
ES6 Pageobjects with flow annotation

25
Input / update
data
Get data
Input / update
data
Get data
UI Framework : node.js
● mocha + wrapper
● Applitools
● co-wrap for webclient
● chai (asserts)
Rest API Framework :
node.js
● co-requests
● mocha
● co-mocha
● chai (asserts)
● json, jayschema
WebClients
Pageobjects
Webclient
adaptor
Database
Browser
Microservice
#1
Microservice
#2 ... #n
Rest APIs

26
Current
Database
Browser
Read & Write
- Click, Drag
- Enter text
- Get text
Microservice
#1
Microservice
#2 ... #n
Rest APIs
Read & Write
API Calls
Get, Post,
Put, Delete
UI tests
● requests
Rest API tests
● requests
● Json, jayschema
WebClients
- Job Client
- Project Client
- File Client
- etc..
UI Pageobjects
- Project List
- Job List
- Job info
- Maps
- Annotations
Node.js common tooling
● Mocha
● Babel (ES6)
● Bluebird
● Asserts (Chai)
● Flow type (Facebook)
Visual tests
● Applitools (visual diffs)

27
Async / Await Cons
● Error handling / stack tracing
● Forces devs to actually understand promises
● Async or bust - difficult to incrementally refactor
● Promise Wrapper optimizations gone
● Chewier syntax than control flow - `await`
everywhere

28
● Async / await was almost designed for browser
automation ‘desync’ing. The glove fits
● Refactoring out of Promise Manager is
cumbersome
● Simplifying test syntax -> less dependencies
and opinion -> happy and efficient devs
The Bottom Line...

Custom reporter with screenshots from
Saucelabs
Extending MochaJS

30
Why Roll Your Own Reporter?
● Start a conversation with developers:
○ What is the most important data you need to see in
order to be efficient and successful?
● 5 things important to Airware cloud team:
○ Failures First
○ Flakiness vs. Failure
○ Assertion messaging
○ Screenshots
○ Video

32
Parallelization Try mocha-parallel-tests. But be careful!

33
● Take advantage of scalability to deliver what
other devs want
● Keep an eye on the OS-community for new
solutions
The Bottom Line...

Type-safe JavaScript
Facebook FlowType

35
Flowtype and Eslint
Don’t like the willy-nilly-ness of JavaScript? Lint! Type check!
● Static type analysis is available with Flow and TypeScript
● Both have awesome IDE / editor plugins
● We picked Flow in order to start annotating our code piece by
piece.
● We added Webdriver, Mocha, and Chai type definitions
● ESlint has some great plugins related to test structuring. We’ve
also written our own
● The bottom line: the best time to capture test errors is as you write
them

And best practices
Robust visual diffs

37
Robust automated visual tests
Powered by

38
Size of test
The test pyramid*
Cost
Time
Coverage
Martin Fowler - Test Pyramid, Google Testing Blog
2014 Google Test Automation Conference
Big
E2E tests
Medium
Integration tests
Small
Unit tests
As you move up the pyramid,
your tests gets bigger. At the
same time the number of tests
(pyramid width) gets smaller.
● Cost: Test execution,
setup time and
maintenance is less as
you come down
● Feedback: Detection
cycle (time) is less as you
come down
● Stability: Smaller tests are
less flaky
● Coverage: E2E workflow
tests have better
coverage but with
tradeoffs; longer time,
flakiness
# Number of tests

39
Google suggests a 70/20/10 ratio for test
amount allocation. (Of a total 100 tests)
○ 70% unit tests,
○ 20% integration tests
○ 10% end-to-end tests
The exact mix will be different for each team,
but we should try to retain that pyramid shape.
The ideal ratio
Google Testing Blog
2014 Google Test Automation Conference
10
20
70

40
Visual diff test pyramid
Visual
tests
UI Selenium
tests
REST API
tests
QA
test pyramid
Big
E2E tests
Medium
Integration tests
Small
Unit tests
Engineering
test pyramid Browser
Screenshots
Backend
image diffs
Visual diff
test pyramid
Apply the test pyramid concept to
automated visual test suite
● Browser screenshot tests are
big E2E tests
● Add smaller visual testing with
commandline image diffs. No
browsers involved.

41
Applitools commandline image diff
Powered by
Using Applitools eyes.images SDK
var Eyes = require('eyes.images').Eyes;
await image = getImage("store.applitools.com","/download/contact_us.png/" + version);
// Visual validation point #1
await eyes.checkImage(img, 'Contact-us page');
● Downloads or streams the images as PNG format
● Uploads images to validate against Applitools service
More info - https://eyes.applitools.com/app/tutorial.html?accountId=m989aQAuq8e107L5sKPP9tWCCPU10JcYV8FtXpBk1pRrlE110
Thanks Liran Barokas !!

42
Visual diff test pyramid
Browser
Screenshots
Backend
image diffs
Powered by

Test trend analysis and etc.
Whats next ?

44
Important attributes of CI/CD systems
● Trustworthy results
● Tests that do not add value
are removed
● Tests have the privilege
(not the right) to run in CI
● Metadata from build and test
results
● Trend analysis
● Is the feature ready to ship
● Cutting edge technology
○ Visual diff
■ Applitools
○ Kubernetes
○ Containers
○ Unikernels
Denali Lumma (Uber), testing in 2020

45
Test trend analytics
Work in Progress!

46
Testability as a product requirement
Collaboration between Frontend and Automation teams
Surface data attributes in the UI DOM
● uuids - test can cross validate by making API calls in UI tests
● Image group, image names, photo clustering attributes
● etc.. Testability as a
product requirement! :)

47
Github links
The Airware Github repos https://github.com/airware
Our Async / Await Example
https://github.com/airware/webdriver-mocha-async-await-example
Flowtype interfaces for Webdriver, Chai, Mocha
https://github.com/airware/forseti-flow-interfaces

Fullstack End-to-end test automation with Node.js, one year later

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Fullstack End-to-end test automation with Node.js, one year later

Similaire à Fullstack End-to-end test automation with Node.js, one year later (20)

Dernier

Dernier (20)

Fullstack End-to-end test automation with Node.js, one year later