Presented by King Chun (Derek) Foo at MESOCA 2011

1. Modeling the Performance of
Ultra-Large-Scale Systems
1
Using
Layered Simulations

2. [Slide courtesy of Randy Shoup, eBay Distinguished Architect] 2
ULS are Everywhere, and they’re HUGE!

3. 1. Performance Testing should happen …
4
development
process
here?
here!

4. 2. One Model to Rule Them All?
1. How do response time differ
between these two mechanism?
5

5. 3. Different Stakeholders, Different Interests
End User Developer
System
Engineer
System
Integrator
Sales
Rep
6

6. An Analogy: The “4+1” View Model
Scenario
Logical View
Process
View
Development
View
Physical
View
8

7. The “3+1” View Model for
Performance Modeling
Scenario
World
View
Physical
View
Component
View
9

8. Component
Layer
A Layered Simulation Model!
10
World
View
Layer
Physical
Layer

9. Case Study 1
Bottleneck Detection in RSS Cloud
• RSS blogging service providers (e.g., Wordpress)
• Evaluation of two notification delivery mechanisms:
– Periodic Pull
– Real-time Push
Q1. How does response time differ?
Q2. How does resource usage
(e.g., CPU, memory) look like?
11

10. Component
Layer
The RSS Cloud Model
12
World
View
Layer
Physical
Layer

11. CPU is the Bottleneck!
13

12. Case Study 2
Evaluating a Distributed Monitor
1. How do response time differ
between these two mechanism?
14

13. Configuration Score
16
DISCRETIZE
fuzzy values
LESS is MORE

14. Distributed vs. Centralized
17
0.1Hz
0.2Hz
0.3Hz
0.1Hz
0.2Hz
0.3Hz
faster “cheaper”
lower resource usage

15. Keeping the Model Up-to-Date
• (What-if) Experimenting with different
configuration/deployment scenarios
• (Digging deeper) Adding new “layers”, as system is
understood better
• (Design evolution) Updating the simulation models as
the system evolves
18

16. GH'@&+/) +A 12(&'I &-829'-3) *#%'3144&2'J '
6!
1) +) 789: ) 02!
9(8; ) <<!
3&+&K'
3&+&F'
LH'7$M&+&2,'",1N&3) #%&+-5'7$M&+&2,'O2,&+&-,-'
?2%'= -&+' 7&B&#) 4&+'
". -,&A ''
?29$2&&+'
". -,&A ''
O2,&9+1,) +'
"1#&-''
0&4'
=!
Component
Layer'
: 'Z1. &+&%'"$A *#18) 2'P ) %&#F'
"I !
World
View
Layer'
Physical
Layer'
7$-,+$S*,&%'B-H'&2,+1#$d&%!
"F!
eHG<d'
eHL<d'
eHQ<d'
TABLE IX
SIMULATION RESULTS FOR THE ORIGINAL DESIGN OF THE PERFORMANCE MONITOR.
DataCollection Response time(s) Cost per Central Monitor Central Monitor Central Monitor
Frequency (Hz) transmission ($) Thread Util. (%) CPU Util. (%) RAM Util. (%)
0.1 0.2 2.2 31.0 42.6 37.2
0.2 0.3 2.7 43.7 68.2 47.6
0.3 0.4 3.1 60.2 86.6 59.2
the CPU of the central monitor is close to running at its full
capacity, which will likely result in system instability. More-
over, whiletheoriginal designhaslower cost per transmission,
be validated as information becomes available throug
development. Due to the lack of access to production
we could only estimate a list of resources and proce
eHG<d'
eHL<d'
eHQ<d'
/1-,&+! U(3&14&+X!
#) D&+'+&-) *+(&'*-19&!
QUESTIONS!

17. 20

18. Backup Slides
21

19. FOUR CHALLENGES
The Current Practice of Performance Verification
22

20. 1. Too Late in the Development Lifecycle
• Design changes are not evaluated until after
code is written
– Happens at the last stage of a delayed schedule
23
Requirements
Design
Implementation
Verification
Maintenance

21. 2. Lots of Data
• Industrial case studies
have 2000> counters
• Time consuming to
analyze
• Hard to compare more
than 2 tests at once
24

22. 3. No Documented Behavior
25
• Analysts have different perceptions of
performance regressions
• Analysis may be influenced by
– Analyst’s knowledge
– Deadline

23. 4. Heterogeneous Environments
• Multiple labs to parallelize test executions
– Hardware and software may differ
– Tests from one lab may not be used to analyze
tests from another lab
26

24. Categorize Each Challenge
27

25. AT THE DESIGN LEVEL
Performance Verification
28

26. Evaluate Design Changes through
Performance Modeling
• Analytical models are often not suitable for all
stakeholders
– Abstract mathematical and statistical concepts
• Simulation models can be implemented with
support of existing framework
– Visualization
– No systematic approach to construct models that
can be used by different stakeholders
29

27. Layered Simulation Model
30
Physical layerComponent layer
World view layer
Can the current
infrastructure support
the projected growth of
users?
Investigate
threading model
Hardware resource
utilization

28. Case Studies
• We conducted two case studies
– RSS Cloud
• Show the process of constructed the model
• Derive the bottleneck of the application
– Performance monitor for ULS systems
• Evaluate whether or not an organization should re-
architect the software
• Our model can be used to extract important
information and aid in decision making
31