Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

OpenNebulaConf 2016 - Measuring and tuning VM performance by Boyan Krosnov, StorPool

318 vues

Publié le

In this session we'll explore measuring VM performance and evaluating changes to settings or infrastructure which can affect performance positively. We'll also share the best current practice for architecture for high performance clouds from our experience.

Publié dans : Technologie
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

OpenNebulaConf 2016 - Measuring and tuning VM performance by Boyan Krosnov, StorPool

  1. 1. Measuring and tuning VM performance Boyan Krosnov StorPool Storage
  2. 2. 1. Measuring performance and evaluating changes to settings or infrastructure which can affect performance positively 2. Best current practice for architecture for high performance clouds Agenda
  3. 3. ● Better application performance -- e.g. time to load a page, time to rebuild, time to execute specific query ● Happier customers (in cloud / multi-tenant environments) ● Lower cost per delivered resource (per VM) ○ through higher density Why
  4. 4. “For every fact there is an infinity of hypotheses.” “The real purpose of the scientific method is to make sure nature hasn’t misled you into thinking you know something you actually don’t know.” Robert M. Pirsig, Zen and the Art of Motorcycle Maintenance Mandatory inspirational quote
  5. 5. What to measure Application performance Proxies for application performance Common synthetic workloads Throughput vs latency 2000kg of X per day (cost/efficiency) vs 10 seconds to first X (what user cares about)
  6. 6. Application performance Latency: Time to load a page Time to execute search q. Throughput: How many visitors can we sustain at up to X ms load time?
  7. 7. Application performance Latency: Time to load a page Time to execute search q. Throughput: How many visitors can we sustain at up to X ms load time?
  8. 8. Proxies for app performance Latency: Break down total latency to components Throughput: Identify bottlenecks Measure just the largest contributors. E.g. database insert.
  9. 9. Common synthetic workloads Throughput: - 4k random read, iodepth 32 or ∞ - 4k random write, iodepth 32 or ∞ - 4k random read/write 50/50, iodepth 32 or ∞ - Sequential read - Sequential write Word of advice: real workloads don’t look like this at all!
  10. 10. Common synthetic workloads Latency: - random read 4k, iodepth 1 - random write 4k, iodepth 1 Latency under load: - Same as throughput but look at latency
  11. 11. fio --name=a --ioengine=aio --direct=1 --randrepeat=0 --norandommap --rw=randwrite --bs=4k --iodepth=256 --filename=/dev/storpool/test a: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=256 Fio-2.1.3 Starting 1 process ^Cbs: 1 (f=1): [w] [6.8% done] [0KB/376.5MB/0KB /s] [0/96.4K/0 iops] [eta 04m:34s] fio: terminating on signal 2 a: (groupid=0, jobs=1): err= 0: pid=6604: Wed Oct 26 09:48:37 2016 write: io=6655.8MB, bw=358993KB/s, iops=89748, runt= 18985msec slat (usec): min=0, max=5316, avg= 1.65, stdev= 7.71 clat (usec): min=109, max=635066, avg=2850.26, stdev=6644.01 lat (usec): min=112, max=635067, avg=2851.98, stdev=6644.02 (StorPool, 3x HC nodes, 12x Micron M500DC 800GB, 3 copies)
  12. 12. fio --name=a --ioengine=aio --direct=1 --randrepeat=0 --norandommap --rw=randwrite --bs=4k --iodepth=1 --filename=/dev/storpool/test a: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1 Fio-2.1.3 Starting 1 process ^Cbs: 1 (f=1): [w] [0.1% done] [0KB/25224KB/0KB /s] [0/6306/0 iops] [eta 01h:08m:38s] fio: terminating on signal 2 a: (groupid=0, jobs=1): err= 0: pid=9141: Wed Oct 26 09:52:05 2016 write: io=119580KB, bw=26086KB/s, iops=6521, runt= 4584msec slat (usec): min=1, max=42, avg= 2.61, stdev= 1.30 clat (usec): min=105, max=12543, avg=150.07, stdev=190.67 lat (usec): min=107, max=12545, avg=152.76, stdev=190.69 (StorPool, 3x HC node, 12x Micron M500DC 800GB, 3 copies)
  13. 13. ● Compute platform ○ Hardware selection ○ HW tuning ○ OS / hypervisor tuning ● Network ● Storage system Best practices
  14. 14. Typically - 2x E5-2690v4 -- 28 cores, 56 threads, @3.2 GHz all-cores turbo - 256-384-512 GB RAM - 10/40 GbE NICs, optionally with RDMA - firmware versions and BIOS settings - Understand power management -- esp. C-states, P-states and “bias” - Think of rack level optimization - how do we get the lowest total cost per delivered resource. Host hardware
  15. 15. RHEL7 Virtualization_Tuning_and_Optimization_Guide Also https://pve.proxmox.com/wiki/Performance_Tweaks http://events.linuxfoundation.org/sites/events/files/slides/CloudOpen2013_Khoa_Huynh_v3.pdf http://www.linux-kvm.org/images/f/f9/2012-forum-virtio-blk-performance-improvement.pdf http://www.slideshare.net/janghoonsim/kvm-performance-optimization-for-ubuntu … but don’t trust everything you read. Perform your own benchmarking! Good references
  16. 16. Recent Linux kernel, KVM and QEMU … but beware of the bleeding edge E.g. qemu-kvm-ev from RHEV (repackaged by CentOS) tuned-adm virtual-host tuned-adm virtual-guest Host OS, guest OS
  17. 17. ● Use virtio-net driver ● regular virtio vs vhost_net ● SR-IOV (PCIe pass-through) Networking
  18. 18. ● cache=none -- direct IO, bypass host buffer cache ● io=native -- use Linux Native AIO, not POSIX AIO (threads) ● virtio-blk -> dataplane ● virtio-scsi -> multiqueue ● in guest virtio_blk.queue_depth 128 -> 256 Block I/O
  19. 19. - balloon - KSM (RAM dedup) - huge pages, THP - NUMA - use local-node memory if you can - route IRQs of network and storage adapters to a core on the node they are on Compute - Memory
  20. 20. (heavy) oversubscription, because VMs are mostly idling CPU Pinning HT NUMA Compute - CPU
  21. 21. Typically 4x 10GE per hypervisor, 2 for storage, 2 for inter-VM/internet Typical cluster has just 2 switches. Up to 128x 10GE ports at low cost. 40/56 GbE and 25G VLANs, Jumbo frames, flow control. RDMA Networks
  22. 22. ● Lots of snake oil out there! ● performance numbers from hardware configurations totally unlike what you’d use in production ● synthetic tests with high iodepth - 10 nodes, 10 workloads * iodepth 256 each. (because why not) ● testing with ramdisk backend ● synthetic workloads don’t work to approximate real world (example) Storage
  23. 23. Performance matters for your users. Work with partners who understand this and help you with it. Conclustion
  24. 24. Boyan Krosnov b k @ storool.com @bkrosnov https://storpool.com/

×