SlideShare une entreprise Scribd logo
1  sur  39
Télécharger pour lire hors ligne
Preventing Guests from Spinning Around


     How to Deal with Lock-Holder Preemption
                              Thomas Friebel

                              June 23, 2008
Spinlock Basics

Spinlocks wait actively as opposed to sleeping locks

Used for short critical sections




 2              How to Deal with Lock-Holder Preemption
Spinlock Wait Times – Kernbench




3           How to Deal with Lock-Holder Preemption
Spinlock Wait Times – Kernbench




4           How to Deal with Lock-Holder Preemption
Spinlocks and Virtualization




5   How to Deal with Lock-Holder Preemption
Spinlocks and Virtualization




6            How to Deal with Lock-Holder Preemption
Spinlocks and Virtualization




7            How to Deal with Lock-Holder Preemption
Spinlocks and Virtualization




8            How to Deal with Lock-Holder Preemption
Spinlocks and Virtualization




9            How to Deal with Lock-Holder Preemption
Spinlocks and Virtualization




10           How to Deal with Lock-Holder Preemption
Is lock-holder preemption problematic?




11    How to Deal with Lock-Holder Preemption
Kernbench in a Guest




12          How to Deal with Lock-Holder Preemption
Kernbench vs. 'while(true)'




13          How to Deal with Lock-Holder Preemption
Time, not Times




14          How to Deal with Lock-Holder Preemption
And in Numbers?




                      guest time                  time spent spinning
                              [s]                      [s]       [%]
single kernbench          109.0                        0.2      0.2%
kernbench vs while(1)     117.3                        9.0      7.6%
difference                 7.6%




15             How to Deal with Lock-Holder Preemption
What can we do about it?




16   How to Deal with Lock-Holder Preemption
Dealing with lock-holder preemption

LHP avoidance
 No spinlock held in userspace
 Idea: Avoid preempting guest in kernel space
 Postpone guest switch to kernel exit
 Problem: extraordinary long critical sections, e.g.
  Apache using sendfile()

Helping locks
 Instead of busy waiting, switch to preempted lock-
  holder
 Problem: finding the preempted lock-holder



17              How to Deal with Lock-Holder Preemption
Helping locks: Ingredients

1) Guest kernel: new 'yield' hypercall when waiting
   unusually long
         Modify spinlock loop
     ●




2) Reasonable threshold for 'unusually long'
         Histograms help
     ●




3) Selecting which VCPU to switch to




18                  How to Deal with Lock-Holder Preemption
Threshold: Upper boundary




19         How to Deal with Lock-Holder Preemption
Threshold: Lower boundary




20         How to Deal with Lock-Holder Preemption
Scheduling Strategy

Good choices:
 VCPUs of the same VM to make progress locally
 (Potential) preempted lock-holders
 Cache-„near“ VCPUs

Neither/nor:
 VCPUs in user space

Bad choices:
 VCPUs which yielded recently




21              How to Deal with Lock-Holder Preemption
What about performance?




22   How to Deal with Lock-Holder Preemption
Histogram with 'yield' hypercall




23           How to Deal with Lock-Holder Preemption
Performance




             wall clock guest time time spent spinning
                    [s]         [s]      [s]      [%]
LHP                34.8      117.3       9.0     7.6%
yield              33.5      108.4       0.0     0.0%
difference      -3.9%       -7.6%               -7.6%




24            How to Deal with Lock-Holder Preemption
Efficiency




25           How to Deal with Lock-Holder Preemption
Efficiency




26           How to Deal with Lock-Holder Preemption
Efficiency




27           How to Deal with Lock-Holder Preemption
Efficiency




28           How to Deal with Lock-Holder Preemption
Efficiency




29           How to Deal with Lock-Holder Preemption
Efficiency




30           How to Deal with Lock-Holder Preemption
Efficiency




31           How to Deal with Lock-Holder Preemption
Efficiency




                      117 sec
                                  ×7.6 % = 3.7 %
                 117 sec 126 sec

                 Real result of 3.9% is reasonable
             ➔


                 Highly efficient
             ➔




32           How to Deal with Lock-Holder Preemption
FIFO ticket spinlocks




33   How to Deal with Lock-Holder Preemption
FIFO ticket spinlocks

Next ticket in dispenser: queue tail

„Now serving“ display at counter: queue head




Lock: atomic( ticket = tail++ ); while ( head != ticket );

Unlock: atomic( head++ );



34              How to Deal with Lock-Holder Preemption
FIFO ticket spinlocks




35           How to Deal with Lock-Holder Preemption
Ticket locks and virtualization




           wall clock guest time                 time spent spinning
                  [s]         [s]                      [s]      [%]
LHP          2825.1     22434.2                   22270.4     99.3%




36           How to Deal with Lock-Holder Preemption
Ticket locks and virtualization




           wall clock guest time                 time spent spinning
                  [s]         [s]                      [s]      [%]
LHP          2825.1     22434.2                   22270.4     99.3%
yield            34.1     123.6                       6.6      5.4%




37           How to Deal with Lock-Holder Preemption
Conclusion


       Lock-holder preemption quite serious:
             7.6% guest time wasted

                  Helping locks:
      3.9% system performance improvement!
           (Amdahl's law explains why)

               New ticket spinlocks:
        30 secs kernbench takes 45 minutes

             Helping locks help here, too




38           How to Deal with Lock-Holder Preemption
Trademark Attribution


AMD, the AMD Arrow logo and combinations thereof are trademarks of Advanced Micro Devices, Inc. in the United States and/or
other jurisdictions. Other names used in this presentation are for identification purposes only and may be trademarks
of their respective owners.

©2008 Advanced Micro Devices, Inc. All rights reserved.




 39                               How to Deal with Lock-Holder Preemption

Contenu connexe

Similaire à XS Boston 2008 Guest Spinning

Preemptable ticket spinlocks: improving consolidated performance in the cloud
Preemptable ticket spinlocks: improving consolidated performance in the cloudPreemptable ticket spinlocks: improving consolidated performance in the cloud
Preemptable ticket spinlocks: improving consolidated performance in the cloud
Jiannan Ouyang, PhD
 
Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
Aggregating Ad Events with Kafka Streams and Interactive Queries at InvidiAggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
HostedbyConfluent
 

Similaire à XS Boston 2008 Guest Spinning (19)

Preemptable ticket spinlocks: improving consolidated performance in the cloud
Preemptable ticket spinlocks: improving consolidated performance in the cloudPreemptable ticket spinlocks: improving consolidated performance in the cloud
Preemptable ticket spinlocks: improving consolidated performance in the cloud
 
Meltdown & spectre
Meltdown & spectreMeltdown & spectre
Meltdown & spectre
 
Meltdown & Spectre
Meltdown & Spectre Meltdown & Spectre
Meltdown & Spectre
 
BlueHat v18 || A mitigation for kernel toctou vulnerabilities
BlueHat v18 || A mitigation for kernel toctou vulnerabilitiesBlueHat v18 || A mitigation for kernel toctou vulnerabilities
BlueHat v18 || A mitigation for kernel toctou vulnerabilities
 
Distributed Lock Manager
Distributed Lock ManagerDistributed Lock Manager
Distributed Lock Manager
 
Sangam 18 - Database Development: Return of the SQL Jedi
Sangam 18 - Database Development: Return of the SQL JediSangam 18 - Database Development: Return of the SQL Jedi
Sangam 18 - Database Development: Return of the SQL Jedi
 
Need for Async: Hot pursuit for scalable applications
Need for Async: Hot pursuit for scalable applicationsNeed for Async: Hot pursuit for scalable applications
Need for Async: Hot pursuit for scalable applications
 
An Introduction to Locks in Go
An Introduction to Locks in GoAn Introduction to Locks in Go
An Introduction to Locks in Go
 
Ensuring Kubernetes Cost Efficiency across (many) Clusters - DevOps Gathering...
Ensuring Kubernetes Cost Efficiency across (many) Clusters - DevOps Gathering...Ensuring Kubernetes Cost Efficiency across (many) Clusters - DevOps Gathering...
Ensuring Kubernetes Cost Efficiency across (many) Clusters - DevOps Gathering...
 
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and LatencyOptimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latency
 
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
 
opt-mem-trx
opt-mem-trxopt-mem-trx
opt-mem-trx
 
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
 
ioDrive de benchmarking 2011 1209_zem_distribution
ioDrive de benchmarking 2011 1209_zem_distributionioDrive de benchmarking 2011 1209_zem_distribution
ioDrive de benchmarking 2011 1209_zem_distribution
 
MSc_thesis_defence
MSc_thesis_defenceMSc_thesis_defence
MSc_thesis_defence
 
Smallsat 2021
Smallsat 2021Smallsat 2021
Smallsat 2021
 
Psychtoolbox (PTB) practical course by Volodymyr B. Bogdanov, Lyon/Kyiv 2018...
Psychtoolbox (PTB) practical course  by Volodymyr B. Bogdanov, Lyon/Kyiv 2018...Psychtoolbox (PTB) practical course  by Volodymyr B. Bogdanov, Lyon/Kyiv 2018...
Psychtoolbox (PTB) practical course by Volodymyr B. Bogdanov, Lyon/Kyiv 2018...
 
On the way to low latency (2nd edition)
On the way to low latency (2nd edition)On the way to low latency (2nd edition)
On the way to low latency (2nd edition)
 
Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
Aggregating Ad Events with Kafka Streams and Interactive Queries at InvidiAggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
Aggregating Ad Events with Kafka Streams and Interactive Queries at Invidi
 

Plus de The Linux Foundation

Plus de The Linux Foundation (20)

ELC2019: Static Partitioning Made Simple
ELC2019: Static Partitioning Made SimpleELC2019: Static Partitioning Made Simple
ELC2019: Static Partitioning Made Simple
 
XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...
XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...
XPDDS19: How TrenchBoot is Enabling Measured Launch for Open-Source Platform ...
 
XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...
XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...
XPDDS19 Keynote: Xen in Automotive - Artem Mygaiev, Director, Technology Solu...
 
XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...
XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...
XPDDS19 Keynote: Xen Project Weather Report 2019 - Lars Kurth, Director of Op...
 
XPDDS19 Keynote: Unikraft Weather Report
XPDDS19 Keynote:  Unikraft Weather ReportXPDDS19 Keynote:  Unikraft Weather Report
XPDDS19 Keynote: Unikraft Weather Report
 
XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...
XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...
XPDDS19 Keynote: Secret-free Hypervisor: Now and Future - Wei Liu, Software E...
 
XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx
XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, XilinxXPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx
XPDDS19 Keynote: Xen Dom0-less - Stefano Stabellini, Principal Engineer, Xilinx
 
XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...
XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...
XPDDS19 Keynote: Patch Review for Non-maintainers - George Dunlap, Citrix Sys...
 
XPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender
XPDDS19: Memories of a VM Funk - Mihai Donțu, BitdefenderXPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender
XPDDS19: Memories of a VM Funk - Mihai Donțu, Bitdefender
 
OSSJP/ALS19: The Road to Safety Certification: Overcoming Community Challeng...
OSSJP/ALS19:  The Road to Safety Certification: Overcoming Community Challeng...OSSJP/ALS19:  The Road to Safety Certification: Overcoming Community Challeng...
OSSJP/ALS19: The Road to Safety Certification: Overcoming Community Challeng...
 
OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...
 OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making... OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...
OSSJP/ALS19: The Road to Safety Certification: How the Xen Project is Making...
 
XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix
XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, CitrixXPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix
XPDDS19: Speculative Sidechannels and Mitigations - Andrew Cooper, Citrix
 
XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd
XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltdXPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd
XPDDS19: Keeping Coherency on Arm: Reborn - Julien Grall, Arm ltd
 
XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...
XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...
XPDDS19: QEMU PV Backend 'qdevification'... What Does it Mean? - Paul Durrant...
 
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&DXPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
 
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM SystemsXPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
XPDDS19: [ARM] OP-TEE Mediator in Xen - Volodymyr Babchuk, EPAM Systems
 
XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...
XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...
XPDDS19: Bringing Xen to the Masses: The Story of Building a Community-driven...
 
XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...
XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...
XPDDS19: Will Robots Automate Your Job Away? Streamlining Xen Project Contrib...
 
XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...
XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...
XPDDS19: Client Virtualization Toolstack in Go - Nick Rosbrook & Brendan Kerr...
 
XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE
XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSEXPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE
XPDDS19: Core Scheduling in Xen - Jürgen Groß, SUSE
 

Dernier

Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
 

Dernier (20)

Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelNavi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
Apidays Singapore 2024 - Scalable LLM APIs for AI and Generative AI Applicati...
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 

XS Boston 2008 Guest Spinning

  • 1. Preventing Guests from Spinning Around How to Deal with Lock-Holder Preemption Thomas Friebel June 23, 2008
  • 2. Spinlock Basics Spinlocks wait actively as opposed to sleeping locks Used for short critical sections 2 How to Deal with Lock-Holder Preemption
  • 3. Spinlock Wait Times – Kernbench 3 How to Deal with Lock-Holder Preemption
  • 4. Spinlock Wait Times – Kernbench 4 How to Deal with Lock-Holder Preemption
  • 5. Spinlocks and Virtualization 5 How to Deal with Lock-Holder Preemption
  • 6. Spinlocks and Virtualization 6 How to Deal with Lock-Holder Preemption
  • 7. Spinlocks and Virtualization 7 How to Deal with Lock-Holder Preemption
  • 8. Spinlocks and Virtualization 8 How to Deal with Lock-Holder Preemption
  • 9. Spinlocks and Virtualization 9 How to Deal with Lock-Holder Preemption
  • 10. Spinlocks and Virtualization 10 How to Deal with Lock-Holder Preemption
  • 11. Is lock-holder preemption problematic? 11 How to Deal with Lock-Holder Preemption
  • 12. Kernbench in a Guest 12 How to Deal with Lock-Holder Preemption
  • 13. Kernbench vs. 'while(true)' 13 How to Deal with Lock-Holder Preemption
  • 14. Time, not Times 14 How to Deal with Lock-Holder Preemption
  • 15. And in Numbers? guest time time spent spinning [s] [s] [%] single kernbench 109.0 0.2 0.2% kernbench vs while(1) 117.3 9.0 7.6% difference 7.6% 15 How to Deal with Lock-Holder Preemption
  • 16. What can we do about it? 16 How to Deal with Lock-Holder Preemption
  • 17. Dealing with lock-holder preemption LHP avoidance  No spinlock held in userspace  Idea: Avoid preempting guest in kernel space  Postpone guest switch to kernel exit  Problem: extraordinary long critical sections, e.g. Apache using sendfile() Helping locks  Instead of busy waiting, switch to preempted lock- holder  Problem: finding the preempted lock-holder 17 How to Deal with Lock-Holder Preemption
  • 18. Helping locks: Ingredients 1) Guest kernel: new 'yield' hypercall when waiting unusually long Modify spinlock loop ● 2) Reasonable threshold for 'unusually long' Histograms help ● 3) Selecting which VCPU to switch to 18 How to Deal with Lock-Holder Preemption
  • 19. Threshold: Upper boundary 19 How to Deal with Lock-Holder Preemption
  • 20. Threshold: Lower boundary 20 How to Deal with Lock-Holder Preemption
  • 21. Scheduling Strategy Good choices:  VCPUs of the same VM to make progress locally  (Potential) preempted lock-holders  Cache-„near“ VCPUs Neither/nor:  VCPUs in user space Bad choices:  VCPUs which yielded recently 21 How to Deal with Lock-Holder Preemption
  • 22. What about performance? 22 How to Deal with Lock-Holder Preemption
  • 23. Histogram with 'yield' hypercall 23 How to Deal with Lock-Holder Preemption
  • 24. Performance wall clock guest time time spent spinning [s] [s] [s] [%] LHP 34.8 117.3 9.0 7.6% yield 33.5 108.4 0.0 0.0% difference -3.9% -7.6% -7.6% 24 How to Deal with Lock-Holder Preemption
  • 25. Efficiency 25 How to Deal with Lock-Holder Preemption
  • 26. Efficiency 26 How to Deal with Lock-Holder Preemption
  • 27. Efficiency 27 How to Deal with Lock-Holder Preemption
  • 28. Efficiency 28 How to Deal with Lock-Holder Preemption
  • 29. Efficiency 29 How to Deal with Lock-Holder Preemption
  • 30. Efficiency 30 How to Deal with Lock-Holder Preemption
  • 31. Efficiency 31 How to Deal with Lock-Holder Preemption
  • 32. Efficiency 117 sec ×7.6 % = 3.7 % 117 sec 126 sec Real result of 3.9% is reasonable ➔ Highly efficient ➔ 32 How to Deal with Lock-Holder Preemption
  • 33. FIFO ticket spinlocks 33 How to Deal with Lock-Holder Preemption
  • 34. FIFO ticket spinlocks Next ticket in dispenser: queue tail „Now serving“ display at counter: queue head Lock: atomic( ticket = tail++ ); while ( head != ticket ); Unlock: atomic( head++ ); 34 How to Deal with Lock-Holder Preemption
  • 35. FIFO ticket spinlocks 35 How to Deal with Lock-Holder Preemption
  • 36. Ticket locks and virtualization wall clock guest time time spent spinning [s] [s] [s] [%] LHP 2825.1 22434.2 22270.4 99.3% 36 How to Deal with Lock-Holder Preemption
  • 37. Ticket locks and virtualization wall clock guest time time spent spinning [s] [s] [s] [%] LHP 2825.1 22434.2 22270.4 99.3% yield 34.1 123.6 6.6 5.4% 37 How to Deal with Lock-Holder Preemption
  • 38. Conclusion Lock-holder preemption quite serious: 7.6% guest time wasted Helping locks: 3.9% system performance improvement! (Amdahl's law explains why) New ticket spinlocks: 30 secs kernbench takes 45 minutes Helping locks help here, too 38 How to Deal with Lock-Holder Preemption
  • 39. Trademark Attribution AMD, the AMD Arrow logo and combinations thereof are trademarks of Advanced Micro Devices, Inc. in the United States and/or other jurisdictions. Other names used in this presentation are for identification purposes only and may be trademarks of their respective owners. ©2008 Advanced Micro Devices, Inc. All rights reserved. 39 How to Deal with Lock-Holder Preemption