How to create a stable automated test suite in DevOps.

1. Automated
Acceptance Test
~ How to build it stable & maintainable

3. Bryan
QA
Programmer
Automation Engineer
DevOps Evangelist &
Facilitator
Big Data
Ad Exchange (RTB)
Low Latency
High Concurrency
(We are hiring)

4. Automated Acceptance Test
In Continuous Delivery
• “Are we good to go live?”
- Test jobs are your “eyes and ears”
- Optimize for them!
• Test code equals production code
Common problems:
• Costly to maintain test buckets
• Spent much time running and debugging false alarms

5. Acceptance Test
“ An Executable Specification of
System Behaviour ”

6. Culture
~ Responsibilities?

7. < 100 ms
What’s AD
Exchange

9. “ Don’t have a separate Testing/QA team!
Quality is down to everyone - Developer owns Acceptance Tests!! ”
― David Farley, Pipeline Conf. 2015

11. “ Not enough to have acceptance test and acceptance test got to be
created and maintained by developer
will make code more testable and maintainable. ”
― Jez Humble, Pipeline Conf. 2016

12. don’t hire too many dedicated testers
developer relies on them, lazier and write more bugs
hire people who can development and test functions

13. code review occupies a central position
no separate group of tester or QA people
relies on automated testing

14. “Testing is essentially the responsibility of the person
who develops a given feature”
(tens of thousands of regression tests)
developer responsible for writing unit tests, regression
tests and performance tests

15. regulated FX exchange
London & Tokyo
launched 2010
~ $2 trillion traded in 2015
one of UK’s fastest growing technology companies

16. high quality - very low rate of production issues
(order of magnitude below industry average)
“lowest bug count we’ve ever found”
- independent analysis by a well-known tool vendor

17. 2 million lines of code
(50:50 test/production)
half the codebase < 18 months old

18. Process / Discipline
~ Definition of done

19. Definition of Done
• Automation task and sprint end demo
• Code review
• Pipeline dashboard
- pipeline always in deliverable status
- statistic analysis report

20. • Build and unit tests
• Full acceptance against the change sets
• Only delivered if above passed
Definition of Done

21. • Code review of automated test scripts
• Loop test suite 20 times before code merge
Loop of Your Tests
stage concurrency: 1, name: ‘Test Looping before merge'
docker.image('qa/chrome-slave:2.53.0').inside('-v /dev/shm:/dev/shm --privileged=true') {
wrap([$class: 'Xvfb', additionalOptions: '-fbdir /tmp']) {
git branch: 'develop', url: 'http://abc.com/qa/accept-test.git'
withEnv(["baseUrl=${baseUrl}","mySQLUrl=${mySQLUrl}"]) {
for(int i = 0; i < count; i++) {
println(“Loop of: ${i}”)
sh 'mvn -B test -fae —DsuiteXmlFile=testng.xml’
}
}
}
}

22. Jenkins Scriptable Build
& Multibranch Pipeline

23. Immutable Environment
• Dockerize environments for build & test
- Always fresh
- Quick
- Immutable
- Scalable & parallel
• Dockerize testing infrastructure

24. Strategies
~ How to make good one

25. Test Automation Pyramid
src: http://www.ontestautomation.com/tag/mike-cohn/

26. Test Isolation
• Reliability/Repeatability
- Provide consistent, repeatable results.
• Isolation
- Tests should not depend, or be affected by, the results of
other tests: testContext
- users, accounts, advertiser names etc: alias name
- external services, 3rd-party integrations: stub, simulator
account: ‘Johnny’
==> ‘Johnny_4534031’
book: ‘DevOps 101’
==> ‘DevOps 101_1234567’

27. Parallelisation
• Commit stage < 15 mins
• Acceptance Test stage < 45 mins
• Fail FASTer!
• Radical parallelisation
- throw-away environments (e.g. containers)

28. Separation of Concern
• Use Domain Specific Language (DSL)
- Focus on “What” not “How”
• Ease of Development
- Hide details of how the tests talk to the SUT
• Ease of Maintenance
- When tests break, we Identify the problem and fix it quickly.

29. ‘What’ Not
‘How’

30. ‘What’ Not
‘How’

31. ‘What’ Not
‘How’

32. Focus on
‘What’
Not
‘How’

33. API Example
@Given("^I create a advertiser with those fields fill : name '(.+)', state '(.+)', currency '(.+)'$")
public void createAdvertiser(final String name, final String state, final String currency) {
advertiserAPI.createDefaultAdvertiser(name, state, currency);
}
package com.vpon.dsp.driver.web.rest.service.adv;
public class AdvertiserServiceImpl {
static { AdvertiserService advService = ServiceGenerator.createService(AdvertiserService.class); }
public static String createAdvertiser(Advertiser obj) throws IOException{
Call<Advertiser> advExec = advService.createAdvertiser(obj);
Response<Advertiser> resp = advExec.execute();
// id should not be null if created successfully
if(resp.isSuccessful() && null != resp.body().id) {
...
}}}
package com.vpon.dsp.dsl.web;
public String createDefaultAdvertiser(String name) {
Advertiser obj = defaultAdvertiser(name);
return AdvertiserServiceImpl.createAdvertiser(obj);
}
DSL layer:
Driver layer:
Test case layer:

34. Implementation
~ Right tools

35. API Test
“ API is more than just a status code,
Verify every part of it.”

36. API Test

37. Concerns?
• Support DevOps - high frequent delivery & BDD
• Maintainability - UI, API are volatile, avoid boilerplate code
• Tools: SoapUI, REST-assured, POSTMAN, Unirest

38. Test Case Example

39. Goal
@end2end
Feature: LineItem end-to-end system targeting test
Background: An Advertiser, LineItem with different System targeting conditions created
Given There is a 'advTargeting' advertiser and following 'System' targeting lineItems:
| name | category | targeting |
| LI_OS_iOS | OS Family | Apple iOS |
| LI_LANG_Eng | Language | English |
Scenario Outline: Bid accordingly with lineItem targeting setting
Given I enable '<lineItemName>' lineItem
When I send a bid request with '<criteria>' targeting only
And I send a 'default' bid request with no targeting criteria
Then I should receive '1' successful bid response
Examples:
| lineItemName | criteria |
| "LI_OS_iOS" | "iOS" |
| "LI_LANG_Eng" | "English" |

40. Retrofit
A type-safe
HTTP client
for
Android
and
Java

41. GUI (Flaky) Test
Stability is super important for automation in CD
• 1% failure rate with 100 test scripts ( 0.99¹⁰⁰) = 63% chance of failure
• You can’t get away with flaky tests in CI/CD
• Spend much time debugging false alarms

42. Selenium + jQuery Selector
• Implement JQueryBy class
- Tells Selenium how to find element with jQuery locator
• If no jQuery in application page
- Inject one for testing
• jQuery
- Powerful API for navigating and selecting content
- Manipulate UI for easy assertion
- CSS based, a whole lot better than XPath
- Has “qualified” feature (ex: has, contains) for filtering
• No xPath locator
- Slow & tight to DOM structure
- Easily broken with html changes
- Bad readability and hard to maintain

43. Selenium PageObject
• Separate test logic from web implementation
- No html stuff (locator, Selenium code, …) in test case
• Increase maintainability
- No need to update all hundreds of TCs when UI changes
- Object classes are the only place to modify
• Increase stability
- More tuning on the framework, it gets more stable
src: Martin

44. Handling Wait in UI Test
User Perform
Action
if_Async
CheckPageReady()
Check Ajax Request
Complete
Wait Dynamic
Element Presence
Make Assertion
Check DB / Queue
Status Match
Thread.sleep()

45. Handling Wait in UI Test
• Each page loading is following the same ‘wait’ checking procedure
• Each page can define it’s own ‘getPageLoadCondition()’
public T initPage(Class<T> clazz) {
T page = PageFactory.initElements(getDriver(), clazz);
checkPageReady(); //<== check for DOM, JS lib
ExpectedCondition pageLoadCondition =
((AbstractPage) page).getPageLoadCondition();
// for extended page does not want to check loading condition ...
if (null != pageLoadCondition) {
waitForCondition(pageLoadCondition); //<== check for page’s load condition
}
return page;
}

46. Handling Wait in UI Test
private static void checkPageReady() {
/* for generic loading, but not all browsers compatible...
final String chk1 =
“(document.readyState==='complete'|| document.readyState===‘interactive')";
*/
final String chk1 =
"(typeof jQuery != 'undefined') && ($(window).load(function() {return true;}))";
//for jQuery loaded and no ajax requests pending
final String chk2 = "($.active === 0)";
final String condition = "return " + chk1 + " && " + chk2 + ";";
ExpectedCondition<Boolean> response = new ExpectedCondition<Boolean>() {
public Boolean apply(WebDriver d) {
return (Boolean)((JavascriptExecutor) d).executeScript(condition);
}
};
new WebDriverWait(getDriver(), LOAD_TIMEOUT).until(response);
}

47. //for those with data grid loading
public static ExpectedCondition getDataLoadingCondition() {
LogUtils.printDebugMessage(logger, "Before returning _loading wait condition...");
return new ExpectedCondition<Boolean>() {
public Boolean apply(WebDriver d) {
boolean result = (Boolean)((JavascriptExecutor) d).executeScript(
"return ($("div[id$='_loading'][style*='display: none;']").length===2" +
" && $.active===0 “ +
“ && $("div[id$='_loading'][style$='display: block;']").length===0)"
);
return result;
}
};
}
Wait Ajax Data Loading
(1) $.active === 0
=> No ajax reqs pending
(2) $(”div[id$=‘_loading’]…..”).length===0)
=> loading icon disappear

48. Wait Dynamic Web Element
public String getToolTipText() {
SeleniumDriver.onPage(ByJQuery.jQuerySelector("div#content"));
. . . . .
//mouseover
new Actions(getDriver()).moveToElement(e).build().perform();
String toolTipLocator = "div.tooltip[style$='display: block;']";
LogUtils.printMessage(logger, "trying to get tooltips of a warning ...");
return SeleniumDriver.waitUntilElementPresence(
ByJQuery.jQuerySelector(toolTipLocator)).getText();
}

49. Capacity Testing
• Performance testing for components
• Long run and stress
• Use production traffic - boost confidence level
• Tools: Gor, Gatling

50. Reference
[How google Test Software]
[Continuous Delivery: Reliable Software Releases]
[From research paper “Moving Fast with Software Verification - 2015”]
Others:
https://dzone.com/articles/dev-centric-culture-breaking-down-the-walls
http://blog.xebialabs.com/2015/06/22/guidelines-for-a-successful-test-strategy/
http://www.androidwarriors.com/2015/12/retrofit-20-android-example-web.html
https://gortool.com/
http://gatling.io/#/
[About Vpon]
[About Me]

51. Q & A