Benchmarking (JAXLondon 2015)
•
1 like•765 views
The document discusses common mistakes made when benchmarking systems and provides recommendations for improving benchmarking methodology. Some key issues covered include failing to account for hardware caches, using too few samples which can miss outliers, and testing with unrealistic workloads not representative of real-world usage. The document recommends approaches like profiling tools to better understand system behavior, ensuring workloads are representative, eliminating sources of noise, and reproducing results.
Recommended
Recommended
More Related Content
Similar to Benchmarking (JAXLondon 2015)
Similar to Benchmarking (JAXLondon 2015) (18)
More from Aysylu Greenberg
More from Aysylu Greenberg (20)
Recently uploaded
Recently uploaded (20)
Benchmarking (JAXLondon 2015)
- 1. Benchmarking:- You’re Doing It Wrong Aysylu-Greenberg- @aysylu22- October-2015-
- 6. Today- • How-Not-to-Write-Benchmarks- • Benchmark-Setup-&-Results:- - -You’re-wrong-about-machines- - -You’re-wrong-about-stats- - -You’re-wrong-about-what-maOers- • Becoming-Less-Wrong-
- 8. Website-Serving-Images- • Access-1-image-1000-Fmes- • Latency-measured-for-each-access- • Start-measuring-immediately- • 3-runs- • Find-mean- • Dev-environment- Web-Request- Server- S3-Cache-
- 24. Website-Serving-Images- • Access-1-image-1000-Fmes- • Latency-measured-for-each-access- • Start-measuring-immediately- • 3-runs- • Find-mean- • Dev-environment- Web-Request- Server- S3-Cache-
- 26. Website-Serving-Images- • Access-1-image-1000-Fmes- • Latency-measured-for-each-access- • Start-measuring-immediately- • 3-runs- • Find-mean- • Dev-environment- Web-Request- Server- S3-Cache-
- 27. Wrong-About-the-Machine- • Cache,-cache,-cache,-cache!- • Warmup-&-Fming- • Periodic-interference-
- 30. Website-Serving-Images- • Access-1-image-1000-Fmes- • Latency-measured-for-each-access- • Start-measuring-immediately- • 3-runs- • Find-mean- • Dev-environment- Web-Request- Server- S3-Cache-
- 31. Wrong-About-the-Machine- • Cache,-cache,-cache,-cache!- • Warmup-&-Fming- • Periodic-interference- • Test-!=-Prod-
- 32. Website-Serving-Images- • Access-1-image-1000-Fmes- • Latency-measured-for-each-access- • Start-measuring-immediately- • 3-runs- • Find-mean- • Dev-environment- Web-Request- Server- S3-Cache-
- 33. Wrong-About-the-Machine- • Cache,-cache,-cache,-cache!- • Warmup-&-Fming- • Periodic-interference- • Test-!=-Prod- • Power-mode-changes-
- 37. 0- 20- 40- 60- 80- 100- 120- 0- 10- 20- 30- 40- 50- 60- Latency$ #$Runs$ Convergence$of$Median$on$Samples$ Stable-Samples- Stable-Median- Decaying-Samples- Decaying-Median-
- 38. Website-Serving-Images- • Access-1-image-1000-Fmes- • Latency-measured-for-each-access- • Start-measuring-immediately- • 3-runs- • Find-mean- • Dev-machine- Web-Request- Server- S3-Cache-
- 40. Website-Serving-Images- • Access-1-image-1000-Fmes- • Latency-measured-for-each-access- • Start-measuring-immediately- • 3-runs- • Find-mean- • Dev-machine- Web-Request- Server- S3-Cache-
- 41. Wrong-About-Stats- • Too-few-samples- • Gaussian-(not)- • MulFmodal-distribuFon-
- 44. Wrong-About-Stats- • Too-few-samples- • Gaussian-(not)- • MulFmodal-distribuFon- • Outliers-
- 45. Coordinated-Omission- 0- request- response- request- response- 10- request- 20- 30- 40- 50- 60- 70- 80- response- Fme- request- response- request-
- 46. Wrong-About-Stats- • Too-few-samples- • Gaussian-(not)- • MulFmodal-distribuFon- • Outliers-
- 51. Wrong-About-What-MaOers- • Premature-opFmizaFon- • UnrepresentaFve-workloads- • Memory-pressure-
- 52. Wrong-About-What-MaOers- • Premature-opFmizaFon- • UnrepresentaFve-workloads- • Memory-pressure- • Hidden-components-
- 53. Wrong-About-What-MaOers- • Premature-opFmizaFon- • UnrepresentaFve-workloads- • Memory-pressure- • Hidden-components- • Reproducibility-of-measurements-
- 55. User-AcFons-MaOer- - X->-Y-for-workload-Z- with-trade-offs-A,-B,-and-C- p-hOp://www.toomuchcode.org/-
- 56. Profiling- -
- 62. gprof:-Where-Does-It-Spend-Its-Time?- • Compile-with-profiling- - • Execute-the-code- - • Run-the-gprof- hOp://www.thegeekstuff.com/2012/08/gprofptutorial/-
- 75. Microbenchmarking:-Blessing-&-Curse- • Choose-your-N-wisely- • Measure-side-effects- • Beware-of-clock-resoluFon-
- 76. Microbenchmarking:-Blessing-&-Curse- • Choose-your-N-wisely- • Measure-side-effects- • Beware-of-clock-resoluFon- • Dead-Code-EliminaFon-
- 77. Microbenchmarking:-Blessing-&-Curse- • Choose-your-N-wisely- • Measure-side-effects- • Beware-of-clock-resoluFon- • Dead-Code-EliminaFon- • Constant-work-per-iteraFon-
- 80. Followpup-Material- • How$NOT$to$Measure$Latency$by-Gil-Tene- – hOp://www.infoq.com/presentaFons/latencyppi}alls- • Taming$the$Long$Latency$Tail-on-highscalability.com- – hOp://highscalability.com/blog/2012/3/12/googleptamingptheplongplatencyp tailpwhenpmorepmachinespequal.html- • Performance$Analysis$Methodology$by-Brendan-Gregg- – hOp://www.brendangregg.com/methodology.html- • Silverman’s$Mode$Detec@on$Method-by-MaO-Adereth- – hOp://adereth.github.io/blog/2014/10/12/silvermanspmodepdetecFonp methodpexplained/- • How$Not$To$Measure$System$Performance-by-James-Bornholt$ – hOps://homes.cs.washington.edu/~bornholt/post/performancep evaluaFon.html- • Trust$No$One,$Not$Even$Performance$Counters-by-Paul-Khuong$ – hDp://www.pvk.ca/Blog/2014/10/19/performancePop@misa@onP~Pwri@ngPanP essay/#trustPnoPone$
- 86. Benchmarking:- You’re Doing It Wrong Aysylu-Greenberg- @aysylu22-