Implementing Quality Automation at the scale of Walmart

1. Pranav Verma
Sr. Engineering Manager
Developer Experience Tools
EMAIL:
pverma@walmartlabs.com
Quality Automation at Walmart Scale
October 23, 2018
E X P E R I E N C E T O O L S
CUSTO M ER
E X P E R I E N C E T O O L S
DEVELOPER

2. Agenda
• Scope Across Walmart
• Test Armada Infrastructure
• Continuous Integration
• Automation Cloud
• Data Insights
• Q&A

3. Scope Across Walmart
TEAMS WE SUPPORT
85 projects in 2 years
200 projects by 2020
1.2M Development Hours Saved
19.5M Functional Tests Executed
196M Mocked API Calls
LOCATIONS WE SUPPORT NUMBERS AT SCALE
Bay Area
San Diego
Bangalore
Canada
Omaha Hoboken
Bentonville
UK
Mexico
Point of Sales RCT ASDA Growth
USGM Jet.com Pharmacy GIF Store Systems OmniChannel
Customer Promise Hayneedle
USGR
EXPO
Portland

4. Infrastructure at Scale
AUTOMATION
CLOUD
CONTINUOUS
INTEGRATION
DATA
INSIGHTS

5. How to run automation tests fast
at a massive scale reliably.
CONTINUOUS INTEGRATION
PROBLEM
Tests need to run against multiple OS/Browsers, which multiplies the numbers of tests.
Need a solution to run massive number of tests without compromising execution time.
GOAL

6. SOLUTION
Shard
Master
Homepage
64 TESTS
Item Page
65 TESTS
Cart
228 TESTS
64 TestsShard 1
64 TestsShard 2
64 TestsShard 3
65 TestsShard 1
65 TestsShard 2
228 TestsShard 1
228 TestsShard 2
228 TestsShard 3
228 TestsShard 4
Thread 2
Thread 1
Thread 3
Thread 4
Thread n

7. BENEFITS
CART PR Verify
(228 Tests on 2 Browsers)
6
HOURSAverage Serial
Test Execution Time
25
MINUTESAverage Parallel
Test Execution Time
to 93%
Savings
=

8. How to test Walmart websites and native apps on all
supported browsers and devices.
AUTOMATION CLOUD
PROBLEM
Need a solution to support automation reliably at Walmart scale, and also allows team to
configure as per their testing needs.
GOAL

9. SOLUTION
Browser Plugin
Device Plugin
Test Agent Plugin
Resource
Manager
Reconciliation Engine
Quota Management
Priority Management
ResourcesTest LabsPlugin Architecture
Internal
Cloud
External
Cloud
Internal
Cloud
External
Cloud
Internal
Cloud
External
Cloud

10. BENEFITS
1,000+
Unique combinations
1,150
Available
Browser/OS
combinations
Devices in a
private cloud
81
250KAutomated Developer
Hours Per Day
Unlimited
Devices
30KEngineers Working
Per Day

11. How to see health of your project in one place.
DATA INSIGHTS
PROBLEM
The application quality information is scattered between JIRA, CI, Test frameworks and
Automation clouds. Need a solution to visualize all information in one place to make
informed decisions.
GOAL

12. Data Insights Platform
Managed Services
SOLUTION
Functional Test Agent
Perf Test Agent
CI
…
DB
Functional Report Client Perf Report Backend Perf Report
Self-Service Portal
API
H
X
HYGIEIA
XRAY
CLIENT

13. BENEFITS
Messages per Second
1Million
SCALABLE PROCESSING
Report Retention
6Months
TRENDING REPORTS
Latency
1Minute
REAL TIME REPORTS

14. Common Test Armada Questions
Can Test Armada support
Java Tests?
Can I use Test Armada for
FE testing if I am not using
Nightwatch? How can I use Test
Armada for BE
Testing?
Can I use Test Armada Data
Insights with my custom test
runner?

15. r
Open Test Armada Platform
Resource Manager Self ServiceData Insights
TEST ARMADA FEATURES
PROGRAMMING LANGUAGE
Magellan
TEST RUNNER
Test Armada TDK
TEST ARMADA LIBRERIES
OUT OF THE BOX SOLUTION
Any Language
PROGRAMMING LANGUAGE
Any Runner
TEST RUNNER
Test Armada Service
TEST ARMADAAPI
Resource Manager Self ServiceData Insights
TEST ARMADA FEATURES
CUSTOMIZED SOLUTION

16. Q&A

17. Contact Us
@testarmada
github.com/testarmada
http://www.testarmada.io