Slides of my talk about the DashProfiler perl module, which enables lightweight always-on performance monitoring for critical sections of code. See
http://search.cpan.org/perldoc?DashProfiler
2. A Problem
A web application ~100K lines of code
Using many external services
If response time goes up... what’s causing it?
3. A Problem
A web application ~100K lines of code
Using many external services
If response time goes up... what’s causing it?
Continuous monitoring in production
Must have very low CPU and I/O cost
Minimal code changes
4. A Typical Approach
package MyNetIO;
sub send_request {
my ($hostname, $request) = @_
...send to $hostname...
}
How much time was spent sending the request?
5. A Typical Approach
package MyNetIO;
use Time::Hires qw(time);
sub send_request {
my ($hostname, $request) = @_
my $start = time();
...send to $hostname...
$durations->{MyNetIO}{$hostname} = time() - $start;
}
6. A Typical Approach
package MyNetIO;
use Time::Hires qw(time);
sub send_request {
my ($hostname, $request) = @_
my $start = time();
...send to $hostname...
$durations->{MyNetIO}{$hostname} = time() - $start;
}
• Doesn’t record count so can’t produce averages.
7. A Typical Approach
package MyNetIO;
use Time::Hires qw(time);
sub send_request {
my ($hostname, $request) = @_
my $start = time();
...send to $hostname...
$durations->{MyNetIO}{$hostname} = time() - $start;
}
• Doesn’t record count so can’t produce averages.
• Two lines of code. Worse if multiple return statements.
8. A Typical Approach
package MyNetIO;
use Time::Hires qw(time);
sub send_request {
my ($hostname, $request) = @_
my $start = time();
...send to $hostname...
$durations->{MyNetIO}{$hostname} = time() - $start;
}
• Doesn’t record count so can’t produce averages.
• Two lines of code. Worse if multiple return statements.
• Doesn’t record time if function exits via an exception.
10. DashProfiler
• Can group samples into granular time units
• Can measure exclusive time in a period
• Can flush to disk at intervals
• Just needs one line of code per sample
11. DashProfiler Internals
Built on DBI::Profile, part of the DBI
Aggregates measurements into a data tree
Two-level tree by default:
$root->{ $key1 }->{ $key2 }->[ ...leaf node... ]
$root->{ ‘MyNetIO’ }->{ $hostname }->[ ...leaf node... ]
12. DashProfiler Data
Each leaf node in the tree is a reference to an array:
$root->{ $key1 }->{ $key2 } = [
106, # 0: count of samples at this node
0.0312958955764771, # 1: total duration
0.000490069389343262, # 2: first duration
0.000176072120666504, # 3: shortest duration
0.00140702724456787, # 4: longest duration
1023115819.83019, # 5: time of first sample
1023115819.86576, # 6: time of last sample
]
13. DashProfiler By-Time
Optional extra time level in the data tree
$time = int(time() / $granularity) * $granularity;
$root->{ $time }->{ ‘MyNetIO’ }->{ $hostname }->[ ... ]
So a new sub-tree is grown each granularity seconds
15. Without DashProfiler
package MyNetIO;
use Time::Hires qw(time);
sub send_request {
my ($hostname, $request) = @_
my $start = time();
...send to $hostname...
$durations->{MyNetIO}{$hostname} = time() - $start;
}
16. Without DashProfiler
package MyNetIO;
use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];
sub send_request {
my ($hostname, $request) = @_
my $sample = foo_profiler( $hostname );
...send to $hostname...
}
17. With DashProfiler
package MyNetIO;
use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];
sub send_request {
my ($hostname, $request) = @_
my $sample = foo_profiler( $hostname );
...send to $hostname...
}
Duration is measured when
$sample goes out of scope
18. With DashProfiler
Name of profile created with add_profile()
package MyNetIO;
use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];
sub send_request {
my ($hostname, $request) = @_
my $sample = foo_profiler( $hostname );
...send to $hostname...
}
Duration is measured when
$sample goes out of scope
19. With DashProfiler
Name of profile created with add_profile()
Value to use for ‘key1’
package MyNetIO;
use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];
sub send_request {
my ($hostname, $request) = @_
my $sample = foo_profiler( $hostname );
...send to $hostname...
}
Duration is measured when
$sample goes out of scope
20. With DashProfiler
Name of profile created with add_profile()
Value to use for ‘key1’
package MyNetIO;
use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];
sub send_request {
Value to use for ‘key2’
my ($hostname, $request) = @_
my $sample = foo_profiler( $hostname );
...send to $hostname...
}
Duration is measured when
$sample goes out of scope
21. With DashProfiler
package MyNetIO;
use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];
sub send_request {
my ($hostname, $request) = @_
my $sample = foo_profiler( $hostname )
if foo_profiler_enabled();
...send to $hostname...
}
22. With DashProfiler
package MyNetIO;
use DashProfiler::Import foo_profiler => [ ‘MyNetIO’ ];
sub send_request {
my ($hostname, $request) = @_
my $sample = foo_profiler( $hostname )
if foo_profiler_enabled();
...send to $hostname...
Automatically imported
compile-time constant
} reduces cost to zero
if profile is disabled
23. DashProfiler Flush
Data is written to STDERR on exit, by default
Regular flushing is enabled by specifying a flush_interval
The dbi_profile_class handles the flush. Choices include:
DBI::Profile
DBI::ProfileData
DBI::ProfileData::Apache
DashProfiler->add_profile( foo => {
...,
flush_interval => 600,
dbi_profile_class => ‘DBI::ProfileData’,
flush_hook => sub { ... },
...
});
24. DashProfiler Periods
• Group samples into periods
- e.g. http request to response
- start_sample_period() and end_sample_period()
- counted, to enable averages and totals per period
- can output period counts instead of sample counts
• Measure ‘exclusive’ time
- time from period start to end that’s not been
accounted for by other samples
- enabled via period_exclusive option
25. Example Data
Average response times over 24 hours
DashProfiler doesn’t generate graphs itself, but the
data can be used to create graphs like these
28. DashProfiler Perspectives
• Each DashProfiler can have multiple DBI
Profile objects attached
• Samples accumulate in all attached profiles
• Each profile can have a different Path
• giving different ‘perspectives’ or level of detail
- key1 + key2
- key1 + country + browser type
- key2 + browser type
- ... etc.
29.
30. DashProfiler Per-Period
• Optional extra ‘per-period’ DBI profile
• Enabled via period_summary option
• Automatically attached and reset by
start_sample_period()
• Gives current totals for this period
• Great for ‘debug footers’ on web page showing
how much time was spent generating this page