Arik Lerner, Automation Team Leader, and Waseem Hamshawi, Automation Infra Developer, present how to build a large scale automated testing platform by leveraging containers orchestration over GCP, with the ability to scale out and provide fast feedback while maintaining a highly reliable test infrastructure. The presentation includes new approach of managing a scalable testing platform of distributed automated tests with Kubernetes and Docker over Google Cloud Platform. Topics: • GCP and Kubernetes introduction for automated testing • Traditional Selenium Grid vs Selenium Standalone with Kubernetes and Docker for Web and Mobile tests • Distributed and containerized testing environment over container cluster - different use cases Ephemerals - "Short-lived Testing Endpoints". An Open Source by LivePerson which makes automation testing at large scale like a "Walk in the park".

1. By Arik Lerner & Waseem Hamshawi
Kubernetes your Tests!
Testing with Docker on Google Cloud Platform
The JourneyBased On True Story!

2. {
“Name”: “Arik Lerner”,
“Role”: “Automation Infra Team Leader at LivePerson”,
“Data”: [
“3 years as Team Leader in Liveperson” ,
“2 years Java developer Reporting Platform Liveperson”
],
“Hobbies”: “Private pilot”
}
Active Prevention Department

3. {
“Name”: “Waseem Hamshawi”,
“Role”: “Senior Automation Infra Developer at Liveperson”,
“Data”: [
“2.5 years at Liveperson”,
“github: waseemh.github.io”,
],
“Hobbies”: “Running”
}
Waseem is Here!
Active Prevention Department

4. Short Story
From Manual Post Deployment Verification
TO
Automation

5. ❏ Fast feedback
❏ Stable Environment
❏ Isolation
❏ Investigation tools - Hard to support (Video/HAR)
❏ Scale
❏ Efficiency
❏ Browser diversity
❏ Mobile Support
Requirements

6. Selenium Hub
Running Test
Static VMs
Supported Requirements
Browser diversity
Mobile Support
Parallelism
Traditional Selenium Grid

7. Ranking:
Tooling
Deployment/
Configuration
Feedback
Isolation
Efficiency
Scalability
❏ Less stable
❏ Crashes - require restarts
❏ Cancel build in progress leave open sessions
❏ Investigation tools - Hard to support (Video/HAR)
❏ Manual grid node Configuration/deployment
❏ Not scalable
❏ Slow feedback
❏ Not efficient - idle grid machines
Facing Issues

9. Cattlevs.Pets
VM
Container
● When they get ill you nurse them back
to health
● You name them! And take of them!
● When they get ill you get another one!
● They are almost identical to one another
● Don't have specific names. Given numbers
● Can be simply replaced if they die

10. What Docker gives us ?
● Build it once run it everywhere!
● Resource Isolation
● Test Isolation
● Package format
● Implementation of a container
Docker

11. Chrome FirefoxVNC Server
Selenium Server
Hub/Node JAR
Mobile Android
What is in a Selenium Container ?

12. VM
VM
VM
VM
Nodes
HUB
N Hub’s
Running Tests
Selenium Grid Architecture Makeover

13. Selenium Docker Images
hub.docker.com/r/selenium

14. DEMO

15. We need more than just Packaging and
Isolation:
● Orchestration - Distribute containers
● Health Checks
● Resilience and Self-healing
● Scaling - making jobs bigger or smaller
Kubernetes Orchestrator

16. VM
VM
VM
VM
Pods
HUB
N Hub’s
Running Tests
Selenium Grid Architecture Makeover

17. Testing on the Cloud!

18. Benefits of Cloud Platform
● Kubernetes cluster in a click of a button (GCP)
● No idle machines - Use when needed
● Costs per build/test - measurable
● Not part of Liveperson network
● Geolocation simulation

19. DEMO

20. Build Life Cycle

21. GCP - Selenium Grid lifecycle
Run tests
Kill Selenium Grid
Scale Down GCP Instances
Scale Out GCP Instances
Create Selenium Grid container cluster

22. Requirements

23. Requirements
❏ Fast feedback
❏ Stable Environment
❏ Isolation
❏ Investigation tools - Hard to support (Video/HAR)
❏ Scale
❏ Efficiency
❏ Browser diversity
❏ Mobile Support

24. Ranking:
Tooling
Deployment/
Configuration
Feedback
Isolation
Efficiency
Scalability
❏ Investigation tools - Video tools not reliable and not
stable.
❏ Feedback
❏ Improved BUT still needs to wait for nodes to
connect into Hub
❏ Point of failures
❏ Hub/Nodes is more failure points in the
architecture that not really necessary
Facing Issues

26. waseemh-mac$ java -jar selenium-server-standalone-2.53.1.jar -role
Selenium server can run in one of the following roles:
hub as a hub of a Selenium grid
node as a node of a Selenium grid
standalone as a standalone server not being a part of a grid
Test Hub
Selenium Server
(Node)
Selenium Standalone

27. Non-Grid Solution
● Fast → Deploy Selenium Standalone containers on-demand
● Short-lived → Selenium containers are destroyed once test ends
● Distribute and manage containers with Kubernetes
Test
createRemoteWebDriverSession(caps)
VM
VM
VM
VM
new RemoteWebDriver(External Endpoint)

28. Test Cluster
?

29. github.com/LivePersonInc/ephemerals
Ephemerals

30. Ephemerals
API
Deploy
Endpoint IP:Portnew RemoteWebDriver(Endpoint IP:Port)
createRemoteWebDriver
(desiredCapabilities)
TestTestTest
TestTestTestTestTest
Test

31. JUnit Integration
@Rule
public EphemeralResource<RemoteWebDriver> seleniumResource =
new EphemeralResource(
new SeleniumEphemeral.Builder(deploymentContext)
.withBrowser(FIREFOX_42)
.withVideoRecording(true)
.build());
@Test
public void test() {
RemoteWebDriver remoteWebDriver = seleniumResource.get();
.....
}

32. DEMO

33. Challenges
● Appium crashes/unresponsive, adb server restarts
● Android SDKs environment installation/maintenance
● Appium↔WebDriver API Versions Incompatibility
● Difficult to parallelise mobile tests
● Scale mobile tests ?!
Mobile Testing
Emulator

34. Ephemeral’s Way
Emulator
Appium
WebDriver
Test
Install
(APK URL)
Ephemerals
API
createMobileDriver
(capabilities) Deploy
Appium Endpoint
IP:Port
new AndroidDriver(Appium IP:Port)

35. DEMO

36. Requirements

37. Ranking:
Tooling
Deployment/
Configuration
Feedback
Isolation
Efficiency
Scalability
Requirements
❏ Fast feedback
❏ Stable Environment
❏ Isolation
❏ Investigation tools - Hard to support (Video/HAR)
❏ Scale
❏ Efficiency
❏ Browser diversity
❏ Mobile Support

38. Beyond Selenium...
Deploy any test dependency inside a container
● Web Servers (/modules/nginx)
● Temporary Databases (/modules/mysql)
● Mock Servers (/modules/wiremock)
● Any Docker Image (/modules/custom)

39. Talking Together...
Web Application
Nginx Ephemeral Selenium Ephemeral
EP
Test Script
EP
WireMock Ephemeral
EP
REST
API
HTTP

40. Ephemerals - Future Plans
● REST API (for non-Java tests)
● Support more cluster management systems:
○ Apache Mesos
○ Docker Swarm
● More modules

41. Takeaways
● Enhance traditional Selenium tools
● Containerization - Short lived, throwaway instances
● Scalable tests with Containers Orchestration
● Benefits of Testing on Cloud Platforms

43. THANK YOU!