Thomas Friebel: Preventing Guests from Spinning Around - Video

1. Preventing Guests from Spinning Around
How to Deal with Lock-Holder Preemption
Thomas Friebel
June 23, 2008

2. Spinlock Basics
Spinlocks wait actively as opposed to sleeping locks
Used for short critical sections
2 How to Deal with Lock-Holder Preemption

3. Spinlock Wait Times – Kernbench
3 How to Deal with Lock-Holder Preemption

4. Spinlock Wait Times – Kernbench
4 How to Deal with Lock-Holder Preemption

5. Spinlocks and Virtualization
5 How to Deal with Lock-Holder Preemption

6. Spinlocks and Virtualization
6 How to Deal with Lock-Holder Preemption

7. Spinlocks and Virtualization
7 How to Deal with Lock-Holder Preemption

8. Spinlocks and Virtualization
8 How to Deal with Lock-Holder Preemption

9. Spinlocks and Virtualization
9 How to Deal with Lock-Holder Preemption

10. Spinlocks and Virtualization
10 How to Deal with Lock-Holder Preemption

11. Is lock-holder preemption problematic?
11 How to Deal with Lock-Holder Preemption

12. Kernbench in a Guest
12 How to Deal with Lock-Holder Preemption

13. Kernbench vs. 'while(true)'
13 How to Deal with Lock-Holder Preemption

14. Time, not Times
14 How to Deal with Lock-Holder Preemption

15. And in Numbers?
guest time time spent spinning
[s] [s] [%]
single kernbench 109.0 0.2 0.2%
kernbench vs while(1) 117.3 9.0 7.6%
difference 7.6%
15 How to Deal with Lock-Holder Preemption

16. What can we do about it?
16 How to Deal with Lock-Holder Preemption

17. Dealing with lock-holder preemption
LHP avoidance
 No spinlock held in userspace
 Idea: Avoid preempting guest in kernel space
 Postpone guest switch to kernel exit
 Problem: extraordinary long critical sections, e.g.
Apache using sendfile()
Helping locks
 Instead of busy waiting, switch to preempted lock-
holder
 Problem: finding the preempted lock-holder
17 How to Deal with Lock-Holder Preemption

18. Helping locks: Ingredients
1) Guest kernel: new 'yield' hypercall when waiting
unusually long
Modify spinlock loop
●
2) Reasonable threshold for 'unusually long'
Histograms help
●
3) Selecting which VCPU to switch to
18 How to Deal with Lock-Holder Preemption

19. Threshold: Upper boundary
19 How to Deal with Lock-Holder Preemption

20. Threshold: Lower boundary
20 How to Deal with Lock-Holder Preemption

21. Scheduling Strategy
Good choices:
 VCPUs of the same VM to make progress locally
 (Potential) preempted lock-holders
 Cache-„near“ VCPUs
Neither/nor:
 VCPUs in user space
Bad choices:
 VCPUs which yielded recently
21 How to Deal with Lock-Holder Preemption

22. What about performance?
22 How to Deal with Lock-Holder Preemption

23. Histogram with 'yield' hypercall
23 How to Deal with Lock-Holder Preemption

24. Performance
wall clock guest time time spent spinning
[s] [s] [s] [%]
LHP 34.8 117.3 9.0 7.6%
yield 33.5 108.4 0.0 0.0%
difference -3.9% -7.6% -7.6%
24 How to Deal with Lock-Holder Preemption

25. Efficiency
25 How to Deal with Lock-Holder Preemption

26. Efficiency
26 How to Deal with Lock-Holder Preemption

27. Efficiency
27 How to Deal with Lock-Holder Preemption

28. Efficiency
28 How to Deal with Lock-Holder Preemption

29. Efficiency
29 How to Deal with Lock-Holder Preemption

30. Efficiency
30 How to Deal with Lock-Holder Preemption

31. Efficiency
31 How to Deal with Lock-Holder Preemption

32. Efficiency
117 sec
×7.6 % = 3.7 %
117 sec 126 sec
Real result of 3.9% is reasonable
➔
Highly efficient
➔
32 How to Deal with Lock-Holder Preemption

33. FIFO ticket spinlocks
33 How to Deal with Lock-Holder Preemption

34. FIFO ticket spinlocks
Next ticket in dispenser: queue tail
„Now serving“ display at counter: queue head
Lock: atomic( ticket = tail++ ); while ( head != ticket );
Unlock: atomic( head++ );
34 How to Deal with Lock-Holder Preemption

35. FIFO ticket spinlocks
35 How to Deal with Lock-Holder Preemption

36. Ticket locks and virtualization
wall clock guest time time spent spinning
[s] [s] [s] [%]
LHP 2825.1 22434.2 22270.4 99.3%
36 How to Deal with Lock-Holder Preemption

37. Ticket locks and virtualization
wall clock guest time time spent spinning
[s] [s] [s] [%]
LHP 2825.1 22434.2 22270.4 99.3%
yield 34.1 123.6 6.6 5.4%
37 How to Deal with Lock-Holder Preemption

38. Conclusion
Lock-holder preemption quite serious:
7.6% guest time wasted
Helping locks:
3.9% system performance improvement!
(Amdahl's law explains why)
New ticket spinlocks:
30 secs kernbench takes 45 minutes
Helping locks help here, too
38 How to Deal with Lock-Holder Preemption

39. Trademark Attribution
AMD, the AMD Arrow logo and combinations thereof are trademarks of Advanced Micro Devices, Inc. in the United States and/or
other jurisdictions. Other names used in this presentation are for identification purposes only and may be trademarks
of their respective owners.
©2008 Advanced Micro Devices, Inc. All rights reserved.
39 How to Deal with Lock-Holder Preemption