SlideShare a Scribd company logo
1 of 17
3/5/15
Challenge the future
Delft
University of
Technology
Understanding Software Performance
Regressions Using Differential Flame
Graphs
Cor-Paul Bezemer, Johan Pouwelse, Brendan Gregg
2Understanding Software Performance Regressions Using DFGs
Regression Testing
3Understanding Software Performance Regressions Using DFGs
Regression Testing Cycle
1)Write code
2)Deploy to continuous integration system
3)Test
4)Deploy to production code or return to 1)
4Understanding Software Performance Regressions Using DFGs
Performance Regression Testing
• Functional testing is difficult, but:
• Functionality is either OK or broken
• Performance testing is even more difficult:
• Depends on requirements of application
• A library may be fast enough for one application, while it's too slow for
another
• How much performance regression is acceptable?
5Understanding Software Performance Regressions Using DFGs
6Understanding Software Performance Regressions Using DFGs
What to Measure
• Built-in performance metrics (or counters)
• Response time, throughput, queue lengths, utilization
• Usually system-wide
• Profilers
• Per-function info on execution time, bytes written, etc.
• Higher granularity == more expensive
• After measuring: tons of data! :) / :(
7Understanding Software Performance Regressions Using DFGs
8Understanding Software Performance Regressions Using DFGs
Detecting I/O Regression
• Previous work:
• “Detecting and analyzing I/O performance regressions”,
C. Bezemer, E. Milon, A. Zaidman, J. Pouwelse (Journal of Software:
Evolution and Process 26 (12), 1193-1212)
• Approach for finding out which function exhibits I/O write
regression over two software versions
• Results promising, but sometimes difficult to interpret
9Understanding Software Performance Regressions Using DFGs
Work in Progress: Visualization
10Understanding Software Performance Regressions Using DFGs
Flame Graphs (1)
def a ( ) :
if some condition :
b()
…
def b ( ) :
if some condition :
c ()
...
def c( ):
...
def d ( ) :
if some condition :
c ()
...
Stack trace, CPU time
a(), 100
a()→b(), 25
a()→b()→c(), 10
d(), 50
d()→c(), 10
11Understanding Software Performance Regressions Using DFGs
Flame Graphs (2)
12Understanding Software Performance Regressions Using DFGs
Flame Graphs (3)
13Understanding Software Performance Regressions Using DFGs
Differential Flame Graphs (1)
14Understanding Software Performance Regressions Using DFGs
Demo
15Understanding Software Performance Regressions Using DFGs
Differential Flame Graphs (2)
• You can do this with performance profiles..
• But also with other data?
• Ideas:
• Website clickpaths
• Parallel/distributed computing
• ...
• Let us know! http://corpaul.github.io/flamegraphdiff/
• (they use this at Netflix, so it must be pretty cool!)
16Understanding Software Performance Regressions Using DFGs
Discussion
• Data collection is difficult
• No out of the box solutions
• Expensive
• Data variation between executions can be considerable
• Especially for 'non-stable' metrics (CPU, memory)
17Understanding Software Performance Regressions Using DFGs
c.bezemer@tudelft.nl
http://corpaul.github.io/flamegraphdiff/

More Related Content

What's hot

Coverage Solutions on Emulators
Coverage Solutions on EmulatorsCoverage Solutions on Emulators
Coverage Solutions on Emulators
DVClub
 
Testing using load runner performance testing
Testing using load runner  performance testingTesting using load runner  performance testing
Testing using load runner performance testing
SivaprasanthRentala1975
 

What's hot (20)

Coverage Solutions on Emulators
Coverage Solutions on EmulatorsCoverage Solutions on Emulators
Coverage Solutions on Emulators
 
Regression Testing with Symfony
Regression Testing with SymfonyRegression Testing with Symfony
Regression Testing with Symfony
 
Test Automation for Embedded Devices
Test Automation for Embedded DevicesTest Automation for Embedded Devices
Test Automation for Embedded Devices
 
Practical unit testing in c & c++
Practical unit testing in c & c++Practical unit testing in c & c++
Practical unit testing in c & c++
 
JavaCro'14 - Profile any environment with Java Flight Recorder – Johan Janssen
JavaCro'14 - Profile any environment with Java Flight Recorder – Johan JanssenJavaCro'14 - Profile any environment with Java Flight Recorder – Johan Janssen
JavaCro'14 - Profile any environment with Java Flight Recorder – Johan Janssen
 
Linux binary analysis and exploitation
Linux binary analysis and exploitationLinux binary analysis and exploitation
Linux binary analysis and exploitation
 
Chris brown ti
Chris brown tiChris brown ti
Chris brown ti
 
.NET Debugging Workshop
.NET Debugging Workshop.NET Debugging Workshop
.NET Debugging Workshop
 
Test execution may_04_2006
Test execution may_04_2006Test execution may_04_2006
Test execution may_04_2006
 
Overview of FreeBSD PMC Tools
Overview of FreeBSD PMC ToolsOverview of FreeBSD PMC Tools
Overview of FreeBSD PMC Tools
 
STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017STAMP at Open Cloud Forum by OW2 2017
STAMP at Open Cloud Forum by OW2 2017
 
How We Test Linux
How We Test LinuxHow We Test Linux
How We Test Linux
 
Mesa and Its Debugging
Mesa and Its DebuggingMesa and Its Debugging
Mesa and Its Debugging
 
Benchmarking PyCon AU 2011 v0
Benchmarking PyCon AU 2011 v0Benchmarking PyCon AU 2011 v0
Benchmarking PyCon AU 2011 v0
 
Testing using load runner performance testing
Testing using load runner  performance testingTesting using load runner  performance testing
Testing using load runner performance testing
 
Fantastic performance and where to find it
Fantastic performance and where to find itFantastic performance and where to find it
Fantastic performance and where to find it
 
Mixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting exampleMixing d ps building architecture on the cross cutting example
Mixing d ps building architecture on the cross cutting example
 
Continuous Performance Testing for Microservices
Continuous Performance Testing for MicroservicesContinuous Performance Testing for Microservices
Continuous Performance Testing for Microservices
 
Linux Server Deep Dives (DrupalCon Amsterdam)
Linux Server Deep Dives (DrupalCon Amsterdam)Linux Server Deep Dives (DrupalCon Amsterdam)
Linux Server Deep Dives (DrupalCon Amsterdam)
 
Symfony 2 under control
Symfony 2 under controlSymfony 2 under control
Symfony 2 under control
 

Viewers also liked

Logging library migrations - A case study for the Apache Software Foundation ...
Logging library migrations - A case study for the Apache Software Foundation ...Logging library migrations - A case study for the Apache Software Foundation ...
Logging library migrations - A case study for the Apache Software Foundation ...
corpaulbezemer
 
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
Chakkrit (Kla) Tantithamthavorn
 
Optimizing the Performance-Related Configurations of Object-Relational Mappin...
Optimizing the Performance-Related Configurations of Object-Relational Mappin...Optimizing the Performance-Related Configurations of Object-Relational Mappin...
Optimizing the Performance-Related Configurations of Object-Relational Mappin...
corpaulbezemer
 

Viewers also liked (7)

Logging library migrations - A case study for the Apache Software Foundation ...
Logging library migrations - A case study for the Apache Software Foundation ...Logging library migrations - A case study for the Apache Software Foundation ...
Logging library migrations - A case study for the Apache Software Foundation ...
 
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
The Impact of Mislabelling on the Performance and Interpretation of Defect Pr...
 
An exploratory study of the state of practice of performance testing in Java-...
An exploratory study of the state of practice of performance testing in Java-...An exploratory study of the state of practice of performance testing in Java-...
An exploratory study of the state of practice of performance testing in Java-...
 
Optimizing the Performance-Related Configurations of Object-Relational Mappin...
Optimizing the Performance-Related Configurations of Object-Relational Mappin...Optimizing the Performance-Related Configurations of Object-Relational Mappin...
Optimizing the Performance-Related Configurations of Object-Relational Mappin...
 
Blazing Performance with Flame Graphs
Blazing Performance with Flame GraphsBlazing Performance with Flame Graphs
Blazing Performance with Flame Graphs
 
Hype vs. Reality: The AI Explainer
Hype vs. Reality: The AI ExplainerHype vs. Reality: The AI Explainer
Hype vs. Reality: The AI Explainer
 
TEDx Manchester: AI & The Future of Work
TEDx Manchester: AI & The Future of WorkTEDx Manchester: AI & The Future of Work
TEDx Manchester: AI & The Future of Work
 

Similar to SANER 2015 ERA track: Differential Flame Graphs

Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...
Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...
Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...
Databricks
 
Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)
Babul Mirdha
 

Similar to SANER 2015 ERA track: Differential Flame Graphs (20)

PERFORMANCE COMPARISON ON JAVA TECHNOLOGIES - A PRACTICAL APPROACH
PERFORMANCE COMPARISON ON JAVA TECHNOLOGIES - A PRACTICAL APPROACHPERFORMANCE COMPARISON ON JAVA TECHNOLOGIES - A PRACTICAL APPROACH
PERFORMANCE COMPARISON ON JAVA TECHNOLOGIES - A PRACTICAL APPROACH
 
Performance comparison on java technologies a practical approach
Performance comparison on java technologies   a practical approachPerformance comparison on java technologies   a practical approach
Performance comparison on java technologies a practical approach
 
Debuggers in system software
Debuggers in system softwareDebuggers in system software
Debuggers in system software
 
Groovy In the Cloud
Groovy In the CloudGroovy In the Cloud
Groovy In the Cloud
 
ICPE 2022 - Data Challenge
ICPE 2022 - Data ChallengeICPE 2022 - Data Challenge
ICPE 2022 - Data Challenge
 
C# Production Debugging Made Easy
 C# Production Debugging Made Easy C# Production Debugging Made Easy
C# Production Debugging Made Easy
 
Real time debugging: using non-intrusive tracepoints to debug live systems
Real time debugging: using non-intrusive tracepoints to debug live systemsReal time debugging: using non-intrusive tracepoints to debug live systems
Real time debugging: using non-intrusive tracepoints to debug live systems
 
Real time debugging: using non-intrusive tracepoints to debug live systems
Real time debugging: using non-intrusive tracepoints to debug live systemsReal time debugging: using non-intrusive tracepoints to debug live systems
Real time debugging: using non-intrusive tracepoints to debug live systems
 
Debugging Modern C++ Application with Gdb
Debugging Modern C++ Application with GdbDebugging Modern C++ Application with Gdb
Debugging Modern C++ Application with Gdb
 
程序员实践之路
程序员实践之路程序员实践之路
程序员实践之路
 
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
[ CNCF Q1 2024 ] Intro to Continuous Profiling and Grafana Pyroscope.pdf
 
Deep into your applications, performance & profiling
Deep into your applications, performance & profilingDeep into your applications, performance & profiling
Deep into your applications, performance & profiling
 
GCC Compiler as a Performance Testing tool for C programs
GCC Compiler as a Performance Testing tool for C programsGCC Compiler as a Performance Testing tool for C programs
GCC Compiler as a Performance Testing tool for C programs
 
Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...
Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...
Accelerating Deep Learning Training with BigDL and Drizzle on Apache Spark wi...
 
Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)
 
Apache Spark Performance is too hard. Let's make it easier
Apache Spark Performance is too hard. Let's make it easierApache Spark Performance is too hard. Let's make it easier
Apache Spark Performance is too hard. Let's make it easier
 
STATISTICAL ANALYSIS FOR PERFORMANCE COMPARISON
STATISTICAL ANALYSIS FOR PERFORMANCE COMPARISONSTATISTICAL ANALYSIS FOR PERFORMANCE COMPARISON
STATISTICAL ANALYSIS FOR PERFORMANCE COMPARISON
 
REDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTS
REDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTSREDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTS
REDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTS
 
REDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTS
REDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTSREDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTS
REDUSHARPTOR: A TOOL TO SIMPLIFY DEVELOPER-WRITTEN C# UNIT TESTS
 
Oleksii Moskalenko "Continuous Delivery of ML Pipelines to Production"
Oleksii Moskalenko "Continuous Delivery of ML Pipelines to Production"Oleksii Moskalenko "Continuous Delivery of ML Pipelines to Production"
Oleksii Moskalenko "Continuous Delivery of ML Pipelines to Production"
 

Recently uploaded

AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM TechniquesAI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
VictorSzoltysek
 
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdfintroduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
VishalKumarJha10
 
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
9953056974 Low Rate Call Girls In Saket, Delhi NCR
 
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
masabamasaba
 

Recently uploaded (20)

AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM TechniquesAI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
AI Mastery 201: Elevating Your Workflow with Advanced LLM Techniques
 
Chinsurah Escorts ☎️8617697112 Starting From 5K to 15K High Profile Escorts ...
Chinsurah Escorts ☎️8617697112  Starting From 5K to 15K High Profile Escorts ...Chinsurah Escorts ☎️8617697112  Starting From 5K to 15K High Profile Escorts ...
Chinsurah Escorts ☎️8617697112 Starting From 5K to 15K High Profile Escorts ...
 
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
 
Architecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the pastArchitecture decision records - How not to get lost in the past
Architecture decision records - How not to get lost in the past
 
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
 
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
%in Hazyview+277-882-255-28 abortion pills for sale in Hazyview
 
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
Crypto Cloud Review - How To Earn Up To $500 Per DAY Of Bitcoin 100% On AutoP...
 
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park %in ivory park+277-882-255-28 abortion pills for sale in ivory park
%in ivory park+277-882-255-28 abortion pills for sale in ivory park
 
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
 
VTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learnVTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learn
 
%+27788225528 love spells in Vancouver Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Vancouver Psychic Readings, Attraction spells,Br...%+27788225528 love spells in Vancouver Psychic Readings, Attraction spells,Br...
%+27788225528 love spells in Vancouver Psychic Readings, Attraction spells,Br...
 
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdfintroduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
introduction-to-automotive Andoid os-csimmonds-ndctechtown-2021.pdf
 
Generic or specific? Making sensible software design decisions
Generic or specific? Making sensible software design decisionsGeneric or specific? Making sensible software design decisions
Generic or specific? Making sensible software design decisions
 
10 Trends Likely to Shape Enterprise Technology in 2024
10 Trends Likely to Shape Enterprise Technology in 202410 Trends Likely to Shape Enterprise Technology in 2024
10 Trends Likely to Shape Enterprise Technology in 2024
 
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa%in tembisa+277-882-255-28 abortion pills for sale in tembisa
%in tembisa+277-882-255-28 abortion pills for sale in tembisa
 
Announcing Codolex 2.0 from GDK Software
Announcing Codolex 2.0 from GDK SoftwareAnnouncing Codolex 2.0 from GDK Software
Announcing Codolex 2.0 from GDK Software
 
%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand
 
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICECHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
CHEAP Call Girls in Pushp Vihar (-DELHI )🔝 9953056974🔝(=)/CALL GIRLS SERVICE
 
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
%+27788225528 love spells in Colorado Springs Psychic Readings, Attraction sp...
 
8257 interfacing 2 in microprocessor for btech students
8257 interfacing 2 in microprocessor for btech students8257 interfacing 2 in microprocessor for btech students
8257 interfacing 2 in microprocessor for btech students
 

SANER 2015 ERA track: Differential Flame Graphs

  • 1. 3/5/15 Challenge the future Delft University of Technology Understanding Software Performance Regressions Using Differential Flame Graphs Cor-Paul Bezemer, Johan Pouwelse, Brendan Gregg
  • 2. 2Understanding Software Performance Regressions Using DFGs Regression Testing
  • 3. 3Understanding Software Performance Regressions Using DFGs Regression Testing Cycle 1)Write code 2)Deploy to continuous integration system 3)Test 4)Deploy to production code or return to 1)
  • 4. 4Understanding Software Performance Regressions Using DFGs Performance Regression Testing • Functional testing is difficult, but: • Functionality is either OK or broken • Performance testing is even more difficult: • Depends on requirements of application • A library may be fast enough for one application, while it's too slow for another • How much performance regression is acceptable?
  • 5. 5Understanding Software Performance Regressions Using DFGs
  • 6. 6Understanding Software Performance Regressions Using DFGs What to Measure • Built-in performance metrics (or counters) • Response time, throughput, queue lengths, utilization • Usually system-wide • Profilers • Per-function info on execution time, bytes written, etc. • Higher granularity == more expensive • After measuring: tons of data! :) / :(
  • 7. 7Understanding Software Performance Regressions Using DFGs
  • 8. 8Understanding Software Performance Regressions Using DFGs Detecting I/O Regression • Previous work: • “Detecting and analyzing I/O performance regressions”, C. Bezemer, E. Milon, A. Zaidman, J. Pouwelse (Journal of Software: Evolution and Process 26 (12), 1193-1212) • Approach for finding out which function exhibits I/O write regression over two software versions • Results promising, but sometimes difficult to interpret
  • 9. 9Understanding Software Performance Regressions Using DFGs Work in Progress: Visualization
  • 10. 10Understanding Software Performance Regressions Using DFGs Flame Graphs (1) def a ( ) : if some condition : b() … def b ( ) : if some condition : c () ... def c( ): ... def d ( ) : if some condition : c () ... Stack trace, CPU time a(), 100 a()→b(), 25 a()→b()→c(), 10 d(), 50 d()→c(), 10
  • 11. 11Understanding Software Performance Regressions Using DFGs Flame Graphs (2)
  • 12. 12Understanding Software Performance Regressions Using DFGs Flame Graphs (3)
  • 13. 13Understanding Software Performance Regressions Using DFGs Differential Flame Graphs (1)
  • 14. 14Understanding Software Performance Regressions Using DFGs Demo
  • 15. 15Understanding Software Performance Regressions Using DFGs Differential Flame Graphs (2) • You can do this with performance profiles.. • But also with other data? • Ideas: • Website clickpaths • Parallel/distributed computing • ... • Let us know! http://corpaul.github.io/flamegraphdiff/ • (they use this at Netflix, so it must be pretty cool!)
  • 16. 16Understanding Software Performance Regressions Using DFGs Discussion • Data collection is difficult • No out of the box solutions • Expensive • Data variation between executions can be considerable • Especially for 'non-stable' metrics (CPU, memory)
  • 17. 17Understanding Software Performance Regressions Using DFGs c.bezemer@tudelft.nl http://corpaul.github.io/flamegraphdiff/

Editor's Notes

  1. Regression testing is done to make sure that old errors are not reintroduced Functional regression testing is fairly straightforward to interpret: did we break existing functionality after doing an update?