Load testing execution produces a huge amount of data. Investigation and analysis are time-consuming, and numbers tend to hide important information about issues and trends. using machine learning is a good way to solve data issues by giving meaningful insights about what happened during test execution. Julio Cesar de Lima Costa will show you how to use K-means clustering, a machine learning algorithm, to reduce almost 300,000 records to fewer than 1,000 and still get good insights into load testing results. He will explain K-means clustering, detail what use cases and applications this method can be used in, and give the steps to help you reproduce a K-means clustering experiment in your own projects. You'll learn how to use this machine learning algorithm to reduce the scope of your load testing and getting meaningful analysis from your data faster.

1. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 1
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
USING MACHINE LEARNING TO SUPPORT TESTING

2. Júlio de Lima
Principal QA Engineer @ Capco
julio.lima@capco.com
twitter.com/juliodelimas
linkedin.com/in/juliodelimas/

3. THE PROBLEM

4. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 4
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
Response Time Graph
Accessing the Main Page Logging in
Milliseconds

5. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 5
Response Time Graph
Accessing the Main Page Logging in
Milliseconds
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
Response Time Graph
1-hour load testing execution
50 virtual users
300k lines of results from JMeter
Two transactions being executed
About the Test

6. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 6
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA Where are the flaws?

7. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 7
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
Where should I investigate?
Application
Logs
Responses from
the Server
Infrastructure
Logs

8. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 8
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
Where should I investigate?
Application
Logs
Responses from
the Server
Infrastructure
Logs
Here, on the JMeter Test Results

9. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 9
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
Access the Main Page
Access the Main Page
Access the Main Page
Access the Main Page
Access the Main Page
Access the Main Page
Logging in
Logging in
Logging in
Logging in
Logging in
Logging in
Reading the
Results Data
Table
300k samples, It’s
difficult to find flaws
here!

10. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 10
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
Response Time Graph
Reading the Results Graph
It's too dense!
Response Time Graph
Accessing the Main Page Logging in
Milliseconds
It’s impossible to read it!

11. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 11
Response Time Graph
Accessing the Main Page Logging in
Milliseconds
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
Gotcha, 72 seconds!

12. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 12
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
Are you sure?

13. THE SOLUTION APPROACH

14. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 14
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
1 Pre-process
the data
2 Use K-means
clustering algorithm
3 Analyze a cluster and
its central tendency
measures
4 Be happy
by reducing the
scope analysis
300k → 1k

15. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 15
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
Tell me more about it!

16. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 16
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
1 Pre-process
the data
a) Redundant records were removed, causing the
number of instances, 294,554, to be reduced to
196,335;
b) The identification attributes were ignored and
only the following were used in the experiments:
elapsed, bytes, sentBytes, Latency and Connect;
c) The data mentioned were normalized within the
range of 0 to 1.

17. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 17
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
2 Use K-means
clustering algorithm
A clustering algorithm that aims at identifying
groups of similar observations based on the mean.

18. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 18
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
3 Analyze a cluster and
its central tendency
measures
I define, empirically, that I want to know what it
would be like if there was a separation of the
dataset into 6 clusters of similar information
Cluster Observations (ms) Distribution
0 5472 3%
1 19119 10%
2 79035 40%
3 871 0%
4 48549 25%
5 43309 22%

19. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 19
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
4 Be happy
by reducing the
scope analysis
300k → 1k
Response Message Samples
OK 50
Non HTTP response message:
157.230.233.185:80 failed to respond
761
Non HTTP response message: Connect to
157.230.233.185:80 [/157.230.233.185]
failed: operation timed out (connection timed
out)
9
Non HTTP response message: operation timed
out (read failed)
1
Non HTTP response message: socket closed 50
A quick mode analysis of Cluster 3 response
message attribute

20. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 20
The response time of most samples was 10%
shorter than the peak initially identified (72
seconds)
MAXIMUM
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA

21. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 21
REDUCING THE SCOPE OF
LOAD TEST ANALYSIS
BY JÚLIO DE LIMA
Smaller than you thought ;)

22. HANDS-ON!

23. Júlio de Lima
Principal QA Engineer @ Capco
julio.lima@capco.com
t.me/juliodelimas-en
twitter.com/juliodelimas
linkedin.com/in/juliodelimas/
Questions?

24. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 24
1998FOUNDED
21EXPERIENCE
INDUSTRY
YEARS
5000EMPLOYEES
27 WORLDWIDE OFFICES
ACROSS 4 CONTINENTS
GLOBAL
90
+
CLIENTS
INDUSTRIES
BANKING & PAYMENTS
WEALTH & ASSET MANAGEMENT
ENERGY MARKETS
CAPITAL MARKETS
INSURANCE
INNOVATION
AND THOUGHT
LEADERSHIP
DIGITAL
TRANSFORMATION
EXECUTION
EXCELLENCE
UNIQUE
CULTURE
INDUSTRY
EXPERIENCE
FAST FACTS
Capco is a global business and technology consultancy focused on financial services.
INNOVATORS: We are innovators who combine a disruptive, design-thinking approach with
unrivalled, first-hand industry knowledge and technology acumen.
FACILITATORS: We facilitate the handshake between the business and technology organizations to
deliver end-to-end consulting services and solutions.
TRANSFORMATION SPECIALISTS: We help our clients transform and advance their businesses,
increase revenue, manage risk and regulatory change.
We consistently deliver these core tenets as part of our brand:
ABOUT CAPCO
24

25. © 2019 The Capital Markets Company. Capco Confidential. All rights reserved. 25
ABOUT CAPCO
WORLDWIDE OFFICES
Bangalore  Bangkok  Bratislava  Brussels  Charlotte  Chicago  Dallas  Dusseldorf  Edinburgh  Frankfurt  Geneva  Hong Kong  Houston
Kuala Lumpur  London  New York  Orlando  Paris  Pune  Sao Paulo  Singapore  Toronto  Tysons Corner  Vienna  Warsaw
Washington, D.C.  Zurich
Capco is a global technology and management consultancy dedicated to the financial services industry. Our professionals combine innovative
thinking with unrivalled industry knowledge to offer our clients consulting expertise, complex technology and package integration,
transformation delivery, and managed services, to move their organizations forward.
Through our collaborative and efficient approach, we help our clients successfully innovate, increase revenue, manage risk and regulatory
change, reduce costs, and enhance controls. We specialize primarily in banking, capital markets, wealth and investment management, finance,
risk & compliance and insurance. We also have an energy consulting practice in the US. We serve our clients from offices in leading financial
centers across the Americas, Europe, and Asia Pacific.
To learn more, visit our website at www.capco.com or follow us on Twitter, Facebook, YouTube, LinkedIn, Instagram and Xing.