SlideShare une entreprise Scribd logo
1  sur  45
Télécharger pour lire hors ligne
Open Source
Systems
Performance
Brendan Gregg
Lead Performance Engineer
Joyent
Jul, 2013
A Play in Three Acts
 A tale of operating systems, performance, and open source
 Dramatis Personae
- Solaris, an Operating System
- Brendan Gregg, a Performance Engineer
- Linux, a Kernel
 Acts
- 1. Before open source (traditional tools)
- 2. Open source (source code-based tracing)
- 3. Closed source
Setting the Scene: Why Performance?
 Reduce IT Spend
- price/performance
 Choose performing components
- evaluation (benchmarking) of software and hardware
 Develop scalable architectures
- understand system limits and develop around them
 Solve issues
Setting the Scene: What is Systems Performance?
 Analysis of:
- A) the kernel
- 2-20% wins: tuning TCP, NUMA, etc
- 2-200x wins: latency outliers, bugs, etc
- B) applications from system context
- 2-2000x wins: eliminating
unnecessary work
 The basis is the system
 The target is everything, down to metal
 Think LAMP not AMP
Applications
Block Device Interface
Volume Managers
File Systems
VFS
System Libraries
Device Drivers
System Call Interface
Metal
Systems Performance Analysis
Kernel
Scheduler
Virtual
Memory
Ethernet
IP
TCP/UDP
Sockets
Firmware
Part 1. Before Open Source
Part 1. Before Open Source
 The year is 2002
 Enter Solaris 9, stage left
 Solaris 9 is not open source
Solaris 9
 Numerous performance observability tools
 Performance, including resource controls and observability, were main features
Scope Type Tools
system counters vmstat(1M), iostat(1M), netstat(1M), kstat(1M), sar(1)
system tracing snoop(1M), prex(1M), tnfdump(1)
process counters ps(1), prstat(1M), ptime(1)
process tracing truss(1), sotruss(1), apptrace(1)
both profiling lockstat(1M), cpustat(1M), cputrack(1)
Systems Performance
 Typified by Unix tools like vmstat(1M) (from BSD):
 Some drill-down were possible with options; eg, the Solaris -p:
 Despite many tools, options, and metrics, the extent of observability was limited.
This can be illustrated using a functional diagram
$ vmstat 1
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr cd cd s0 s5 in sy cs us sy id
0 0 0 8475356 565176 2 8 0 0 0 0 1 0 0 -0 13 378 101 142 0 0 99
1 0 0 7983772 119164 0 0 0 0 0 0 0 224 0 0 0 1175 5654 1196 1 15 84
0 0 0 8046208 181600 0 0 0 0 0 0 0 322 0 0 0 1473 6931 1360 1 7 92
[...]
$ vmstat -p 1
memory page executable anonymous filesystem
swap free re mf fr de sr epi epo epf api apo apf fpi fpo fpf
8475336 565160 2 8 0 0 1 0 0 0 0 0 0 0 0 0
7972332 107648 1 29 0 0 0 0 0 0 0 0 0 0 0 0
7966188 101504 0 0 0 0 0 0 0 0 0 0 0 0 0 0
[...]
Operating System Functional Diagram
Applications
DBs, all server types, ...
Block Device Interface Ethernet
Volume Managers IP
File Systems TCP/UDP
VFS Sockets
Disk Disk Port Port
Expander Interconnect
I/O Bus
Interface Transports
I/O Controller Network Controller
I/O Bridge
System Libraries
Device Drivers
Scheduler
Virtual
Memory
System Call Interface CPU
Interconnect
Memory
Bus
CPU
1
DRAM
Operating System Hardware
Kernel
Solaris 9 Observability Coverage
Applications
DBs, all server types, ...
Block Device Interface Ethernet
Volume Managers IP
File Systems TCP/UDP
VFS Sockets
Disk Disk Port Port
Expander Interconnect
I/O Bus
Interface Transports
I/O Controller Network Controller
I/O Bridge
System Libraries
Device Drivers
Scheduler
Virtual
Memory
System Call Interface CPU
Interconnect
Memory
Bus
CPU
1
DRAM
Operating System Hardware
SolarisKernel
truss
apptrace
sotruss
iostat
prex
vmstat
mpstat
prstat
ps
cpustat
cputrack
snoop
netstat
lockstat
kstat
sar
kstat
Various:
netstat
kstat
Problems
 Below the syscall interface was dark, if not pitch black
 Many components either had:
- No metrics at all
- Undocumented metrics (kstat)
 Certain performance issues could not be analyzed
- Time from asking Sun for a new performance metric to having it in production could
be months or years or never
- You solve what the current tools let you: the “tools method” of iterating over existing
tools and metrics
 Situation largely accepted as a better way wasn’t known
 Much systems performance literature was written in this era, and is still around
High Performance Tuning
 Performance experts were skilled in the art of inference and experimentation
- Study Solaris Internals for background
- Determine kernel behavior based on indirect metrics
- Create known workloads to test undocumented metrics,
and to explore system behavior
- Heavy use of the Scientific method
 Science is good, source is better
... If the Universe was Open Source
vi universe/particles.c:
photon_t *
spontaneous_emission(electron_t *e) {
photon_t *p;
if (e->e_orbit > 1) {
p = palloc(e);
e->e_orbit--;
} else {
electron_capture(e->e_nucleusp);
return (NULL)
}
return (p);
}
vi universe/include/electron.h:
struct electron {
mass_t e_mass; /* electron mass */
charge_t e_charge; /* electron charge */
uint64_t e_flags; /* 0x01 particle; 0x10 wave */
int e_orbit; /* current orbit level */
boolean_t e_matter; /* 1 = matter; 0 = antimatter */
[...]
} electron_t;
Part 2. Open Source
Part 2. Open Source
 The year is 2005
 Solaris 10, as OpenSolaris, becomes open source
- In response to Linux, which always was
Open Source Metrics
 Undocumented kstats could now be understood from source
- it was like being handed the source code to the Universe
- I wasn’t a Sun badged employee; I’d been working without source access
 Tool metrics could also be better understood, and exact behavior of the kernel
 For example, where does “r” come from?
$ vmstat 1
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr cd cd s0 s5 in sy cs us sy id
0 0 0 8475356 565176 2 8 0 0 0 0 1 0 0 -0 13 378 101 142 0 0 99
1 0 0 7983772 119164 0 0 0 0 0 0 0 224 0 0 0 1175 5654 1196 1 15 84
Understanding “r”
 Starting with vmstat(1M)’s source, and drilling down:
usr/src/cmd/stat/vmstat/vmstat.c:
static void
printhdr(int sig)
{
[...]
if (swflag)
(void) printf(" r b w swap free si so pi po fr de sr ");
else
(void) printf(" r b w swap free re mf pi po fr de sr ");
[...]
static void
dovmstats(struct snapshot *old, struct snapshot *new)
{
[...]
adjprintf(" %*lu", 1, DELTA(s_sys.ss_sysinfo.runque) / sys_updates);
Understanding “r”, cont.
 Searching on ss_sysinfo:
usr/src/cmd/stat/common/statcommon.h:
struct sys_snapshot {
sysinfo_t ss_sysinfo;
[...]
usr/src/uts/common/sys/sysinfo.h:
typedef struct sysinfo { /* (update freq) update action */
uint_t updates; /* (1 sec) ++ */
uint_t runque; /* (1 sec) += num runnable procs */
uint_t runocc; /* (1 sec) ++ if num runnable procs > 0 */
uint_t swpque; /* (1 sec) += num swapped procs */
uint_t swpocc; /* (1 sec) ++ if num swapped procs > 0 */
uint_t waiting; /* (1 sec) += jobs waiting for I/O */
} sysinfo_t;
Understanding “r”, cont.
 ss_sysinfo is populated from kstat:
usr/src/cmd/stat/common/acquire.c:
int
acquire_sys(struct snapshot *ss, kstat_ctl_t *kc)
{
size_t i;
kstat_named_t *knp;
kstat_t *ksp;
if ((ksp = kstat_lookup(kc, "unix", 0, "sysinfo")) == NULL)
return (errno);
if (kstat_read(kc, ksp, &ss->s_sys.ss_sysinfo) == -1)
return (errno);
[...]
Understanding “r”, cont.
 Searching on runque population, in the kernel:
usr/src/uts/common/os/clock.c:
static void
clock(void)
{
* There is additional processing which happens every time
* the nanosecond counter rolls over which is described
* below - see the section which begins with : if (one_sec)
[...]
do {
uint_t cpu_nrunnable = cp->cpu_disp->disp_nrunnable;
nrunnable += cpu_nrunnable;
[...]
} while ((cp = cp->cpu_next) != cpu_list);
[...]
if (one_sec) {
[...]
if (nrunnable) {
sysinfo.runque += nrunnable;
sysinfo.runocc++;
}
Once-a-second snapshots?
That’s good to know!
Statistic Spelunking
 A matter of browsing and reading source code
- I use cscope, a text-based source code browser:
 Doesn’t require expertise to
begin with: keep reading code
until it makes sense
 Might take hours or days if
you are new to a complex
code base
 You may only do this three
times in your career, but
each time was worth it!
C symbol: runque
File Function Line
0 sa.h <global> 188 uint64_t runque;
1 sysinfo.h <global> 132 uint_t runque;
2 sar.c prt_q_opt 919 (float )xx->si.runque / (float )xx->si.runocc,
3 kstat.c save_sysinfo 1066 SAVE_UINT32(ksi, sysinfo, runque);
4 vmstat.c dovmstats 316 adjprintf(" %*lu", 1,
DELTA(s_sys.ss_sysinfo.runque) / sys_updates);
5 clock.c clock 862 sysinfo.runque += nrunnable;
Find this C symbol:
Find this global definition:
Find functions called by this function:
Find functions calling this function:
Find this text string:
Change this text string:
Find this egrep pattern:
Find this file:
Find files #including this file:
Open Source Dynamic Tracing
 Solaris 10 also provided Dynamic Tracing (DTrace), which can observe virtually
everything
 Core feature of all later OpenSolaris derivatives, including SmartOS and OmniOS
 Observability gaps now filled
Solaris 10/SmartOS/OmniOS Observability Coverage
Applications
DBs, all server types, ...
Block Device Interface Ethernet
Volume Managers IP
File Systems TCP/UDP
VFS Sockets
Disk Disk Port Port
Expander Interconnect
I/O Bus
Interface Transports
I/O Controller Network Controller
I/O Bridge
System Libraries
Device Drivers
Scheduler
Virtual
Memory
System Call Interface CPU
Interconnect
Memory
Bus
CPU
1
DRAM
Operating System Hardware
Solaris/illumos
Kernel
truss
iostat
vmstat
mpstat
prstat
ps
cpustat
cputrack
snoop
netstat
sar
kstat
lockstat
Various:
kstat
netstat
kstat
dtrace
plockstat
intrstat
Open Source Dynamic Tracing: Example
 Given the kernel source code, eg, ZFS SPA sync:
 Trace and time it using the DTrace function boundary tracing (fbt) provider:
usr/src/uts/common/fs/zfs/spa.c:
/*
* Sync the specified transaction group. New blocks may be dirtied as
* part of the process, so we iterate until it converges.
*/
void
spa_sync(spa_t *spa, uint64_t txg)
{
dsl_pool_t *dp = spa->spa_dsl_pool;
[...]
# dtrace -n 'fbt::spa_sync:entry { self->ts = timestamp; } fbt::spa_sync:return /self->ts/
{ printf("%Y %d ms", walltimestamp, (timestamp - self->ts) / 1000000); self->ts = 0; }'
dtrace: description 'fbt::spa_sync:entry ' matched 2 probes
CPU ID FUNCTION:NAME
0 53625 spa_sync:return 2013 Jul 26 17:37:02 12 ms
0 53625 spa_sync:return 2013 Jul 26 17:37:08 726 ms
6 53625 spa_sync:return 2013 Jul 26 17:37:17 6913 ms
6 53625 spa_sync:return 2013 Jul 26 17:37:17 59 ms
Awesome!
Dynamic Tracing Scripts
Applications
DBs, all server types, ...
Block Device Interface Ethernet
Volume Managers IP
File Systems TCP/UDP
VFS Sockets
System Libraries
Device Drivers
Scheduler
Virtual
Memory
System Call Interface
iosnoop, iotop
disklatency.d
satacmds.d
satalatency.d
scsicmds.d
scsilatency.d
sdretry.d, sdqueue.d
ide*.d, mpt*.d
priclass.d, pridist.d
cv_wakeup_slow.d
displat.d, capslat.d
opensnoop, statsnoop
errinfo, dtruss, rwtop
rwsnoop, mmap.d, kill.d
shellsnoop, zonecalls.d
weblatency.d, fddist
dnlcsnoop.d
zfsslower.d
ziowait.d
ziostacks.d
spasync.d
metaslab_free.d
fswho.d, fssnoop.d
sollife.d
solvfssnoop.d
hotuser, umutexmax.d, lib*.d
node*.d, erlang*.d, j*.d, js*.d
php*.d, pl*.d, py*.d, rb*.d, sh*.d
mysql*.d, postgres*.d, redis*.d, riak*.d
Language Providers:
Databases:
sotop.d, socketio.d, so1stbyte.d, soconnect.d, soaccept.d
ipio.d, ipproto.d, ipstat.d, ipfbtsnoop.d, icmpsnoop.d
tcp1stbyte.d, tcpaccept.d, tcpconnect.d, tcpconnlat.d, tcpio.d
tcpbytes.d, tcpsize.d, tcpnmap.d, udpio.d, udpstat.d
cifs*.d, iscsi*.d
nfsv3*.d, nfsv4*.d
ssh*.d, httpd*.d
:Services
minfbypid.d
pgpginbypid.d
macops.d
ngesnoop.d, ngelink.d
These are some of my scripts from the DTraceToolkit, the DTrace book, and other collections. I’d add more but I ran out of room.
Modern Systems Performance
 Typified by an abundance of high
resolution useful metrics (latency)
 No longer a problem of missing
metrics, but how to visualize many
metrics, and across clouds
- eg, latency heat maps:
 Prior tools are useful as starting
points, with tracing to dig deeper
 In the following sections, I’ll
describe modern Linux Systems Performance, summarizing how traditional and new
tools can be used together
- I’ll group dtrace/systemtap/perf/lttng/ktap/etc as “dynamic tracing”, which is a
simplification: some needs may not be met by all those tools
Linux CPU Analysis
 Traditional tools:
- 1. system wide usage vmstat
- 2. per-processor usage mpstat
- 3. per-process usage top, ps
- 4. user- or kernel-stack profiling perf record -agF
- 5. cycle analysis perf stat
 Modern tools:
- 6. tracing scheduler latency perf sched
- 7. tracing CPU usage of functions dynamic/static tracing
- 8. tracing CPU consumption of spin locks dynamic/static tracing
- 9. CPU cross call tracing dynamic/static tracing
- 10. interrupt tracing dynamic/static tracing
Linux Memory Analysis
 Traditional tools:
- 1. system wide usage vmstat
- 2. per-process usage top
- 3. kernel usage /proc/meminfo, slaptop
- 4. swapping activity sar
- 5. leak detection valgrind
 Modern tools:
- 6. tracing allocations dynamic tracing
- 7. tracing page faults dynamic/static tracing
- 8. tracing kswapd activity dynamic/static tracing
Linux File System Analysis
 Traditional tools:
- 1. cache usage free, /proc/meminfo
- 2. syscall access strace (expensive)
 Modern tools:
- 3. tracing VFS accesses dynamic/static tracing
- 4. tracing file system latency latencytop, dynamic/static tracing
- 5. tracing file system internals dynamic/static tracing
Linux Disk Analysis
 Traditional tools:
- 1. per-disk statistics iostat
- 2. per-process usage pidstat -d, iotop
 Modern tools:
- 3. disk I/O latency tracing blktrace, static tracing
- 4. lower I/O stack tracing dynamic/static tracing
- 5. SCSI command tracing dynamic/static tracing
- 6. device driver tracing dynamic/static tracing
Linux Network Analysis
 Traditional tools:
- 1. system wide usage netstat -s
- 2. per-interface usage netstat -i, sar -n DEV, ip
- 3. TCP statistics netstat -s, sar -n TCP
- 4. packet tracing tcpdump
- 5. socket call tracing strace (expensive)
- 6. experimental tests ping, traceroute
 Modern tools:
- 6. tracing socket-level latency dynamic/static tracing
- 7. TCP retransmit (only) tracing dynamic tracing
- 8. tracing TCP kernel internals dynamic tracing
Linux Network Analysis, Example
 TCP retransmits: given tcp_retransmit_skb(), show the dest IP addr. Source code:
net/ipv4/tcp_output.c:
int tcp_retransmit_skb(struct sock *sk, struct sk_buff *skb)
{
struct tcp_sock *tp = tcp_sk(sk);
int err = __tcp_retransmit_skb(sk, skb);
[...]
include/linux/tcp.h:
struct tcp_sock {
/* inet_connection_sock has to be the first member of tcp_sock */
struct inet_connection_sock inet_conn;
[...]
include/net/inet_connection_sock.h:
struct inet_connection_sock {
/* inet_sock has to be the first member! */
struct inet_sock icsk_inet;
[...]
Linux Network Analysis, Example
 ... More spelunking, like earlier. Not trivial, but doable.
include/net/inet_sock.h:
struct inet_sock {
/* sk and pinet6 has to be the first two members of inet_sock */
struct sock sk;
#if IS_ENABLED(CONFIG_IPV6)
struct ipv6_pinfo *pinet6;
#endif
/* Socket demultiplex comparisons on incoming packets. */
#define inet_daddr sk.__sk_common.skc_daddr
[...]
include/net/sock.h
struct sock {
/*
* Now struct inet_timewait_sock also uses sock_common, so please just
* don't add nothing before this first member (__sk_common) --acme
*/
struct sock_common __sk_common;
[...]
Here it is
Linux Network Analysis, Example Script
 TCP retransmit tracing script, using DTrace4Linux (prototype):
#!/usr/sbin/dtrace -s
#pragma D option quiet
dtrace:::BEGIN { trace("Tracing TCP retransmits... Ctrl-C to end.n"); }
fbt::tcp_retransmit_skb:entry {
this->so = (struct sock *)arg0;
this->d = (unsigned char *)&this->so->__sk_common.skc_daddr;
printf("%Y: retransmit to %d.%d.%d.%d, by:", walltimestamp,
this->d[0], this->d[1], this->d[2], this->d[3]);
stack(99);
}
Linux Network Analysis, Example Output
 TCP retransmit tracing script, using DTrace4Linux (prototype):
# ./tcpretransmit.d
Tracing TCP retransmits... Ctrl-C to end.
2013 Feb 23 18:24:11: retransmit to 10.2.124.2, by:
kernel`tcp_retransmit_timer+0x1bd
kernel`tcp_write_timer+0x188
kernel`run_timer_softirq+0x12b
kernel`tcp_write_timer
kernel`__do_softirq+0xb8
kernel`read_tsc+0x9
kernel`sched_clock+0x9
kernel`sched_clock_local+0x25
kernel`call_softirq+0x1c
kernel`do_softirq+0x65
kernel`irq_exit+0x9e
kernel`smp_apic_timer_interrupt+0x6e
kernel`apic_timer_interrupt+0x6e
[...]
Linux Network Example, cont.
 I created a custom performance tool on the fly, without kernel changes
 Would it be possible if the kernel wasn’t open source?
Opportunities
 Open source allows dynamic tracing: otherwise you are tracing blind
 Dynamic tracing allows custom metrics and scripts (tools) to be written
- fill in all observability gaps; can solve most performance issues
 Many people will use dynamic tracing: eg, DTraceToolkit, DTrace book, company tools;
only some may author the tools: the OS or perf engineer on your team (which is ok)
 Dynamic tracing also allows new methodologies
- prior methodologies constrained by existing tools and metrics
- new methodologies can be explored, as any question posed
can be answered
 Examples of new methodologies
- USE method
- Thread State Analysis method
Challenges
 Systems performance literature written for the pre-open source Unix days
- Gives the impression that performance ends with older static tools
 DTrace not on Linux yet
- Two ports are in progress:
- DTrace4Linux: https://github.com/dtrace4linux/linux
- Oracle Linux DTrace
 Instead of waiting, you can try an illumos-kernel based distro like SmartOS
- illumos is the surviving fork of OpenSolaris. Which brings us to Act 3.
Act 3. Closed Source
Act 3. Closed Source
 The year is 2010
 Oracle stops releasing updates for OpenSolaris
 Oracle Solaris 11 is released a year later, closed source
 Provides us with a unique additional perspective for open source systems performance
Closed Source Metrics
 This closed the only documentation for many metrics and kernel internals
- Back to inference and experimentation by the end user
- Will get harder over time as documentation ages: without a Solaris Internals 3rd
Edition, kernel internals may become as opaque as it was in the 90’s
Closed Source Dynamic Tracing
 Makes using the DTrace fbt provider much harder
- Hypothetical example to show how this could manifest:
- Dynamic tracing of ZFS SPA sync during a performance investigation:
Where’d spa_sync() go? Did it get renamed or removed?
- Could be worse if tracing succeeds, but produces misleading metrics due to
unknown changes
 Note that the capabilities are still there, and can be used by Oracle support
# dtrace -n 'fbt::spa_sync:entry { printf("%Y", walltimestamp); }'
dtrace: invalid probe specifier fbt::spa_sync:entry { printf("%Y", walltimestamp); }:
probe description fbt::spa_sync:entry does not match any probes
Elsewhere at Oracle
 Their DTrace port for Oracle Linux won’t have this handicap
- although, the fbt provider hasn’t been included yet
Epilog: The New Systems Performance
 An era of:
- Open source
- Dynamic tracing
- Methodologies
- Distributed systems (cloud)
 Covered in my book, out this year:
Thank you!
 email: brendan@joyent.com
 twitter: @brendangregg
 blog: http://dtrace.org/blogs/brendan

Contenu connexe

Tendances

DTrace Topics: Introduction
DTrace Topics: IntroductionDTrace Topics: Introduction
DTrace Topics: IntroductionBrendan Gregg
 
Systems Performance: Enterprise and the Cloud
Systems Performance: Enterprise and the CloudSystems Performance: Enterprise and the Cloud
Systems Performance: Enterprise and the CloudBrendan Gregg
 
EuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis MethodologiesEuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis MethodologiesBrendan Gregg
 
Linux Performance Tools
Linux Performance ToolsLinux Performance Tools
Linux Performance ToolsBrendan Gregg
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at NetflixBrendan Gregg
 
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPFOSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPFBrendan Gregg
 
Linux Performance Profiling and Monitoring
Linux Performance Profiling and MonitoringLinux Performance Profiling and Monitoring
Linux Performance Profiling and MonitoringGeorg Schönberger
 
Performance Wins with eBPF: Getting Started (2021)
Performance Wins with eBPF: Getting Started (2021)Performance Wins with eBPF: Getting Started (2021)
Performance Wins with eBPF: Getting Started (2021)Brendan Gregg
 
Container Performance Analysis
Container Performance AnalysisContainer Performance Analysis
Container Performance AnalysisBrendan Gregg
 
Velocity 2015 linux perf tools
Velocity 2015 linux perf toolsVelocity 2015 linux perf tools
Velocity 2015 linux perf toolsBrendan Gregg
 
YOW2020 Linux Systems Performance
YOW2020 Linux Systems PerformanceYOW2020 Linux Systems Performance
YOW2020 Linux Systems PerformanceBrendan Gregg
 
Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)Brendan Gregg
 
SREcon 2016 Performance Checklists for SREs
SREcon 2016 Performance Checklists for SREsSREcon 2016 Performance Checklists for SREs
SREcon 2016 Performance Checklists for SREsBrendan Gregg
 
Designing Tracing Tools
Designing Tracing ToolsDesigning Tracing Tools
Designing Tracing ToolsBrendan Gregg
 
ACM Applicative System Methodology 2016
ACM Applicative System Methodology 2016ACM Applicative System Methodology 2016
ACM Applicative System Methodology 2016Brendan Gregg
 
Linux Systems Performance 2016
Linux Systems Performance 2016Linux Systems Performance 2016
Linux Systems Performance 2016Brendan Gregg
 
The New Systems Performance
The New Systems PerformanceThe New Systems Performance
The New Systems PerformanceBrendan Gregg
 
Performance Wins with BPF: Getting Started
Performance Wins with BPF: Getting StartedPerformance Wins with BPF: Getting Started
Performance Wins with BPF: Getting StartedBrendan Gregg
 
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven RostedtKernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven RostedtAnne Nicolas
 
Performance Tuning EC2 Instances
Performance Tuning EC2 InstancesPerformance Tuning EC2 Instances
Performance Tuning EC2 InstancesBrendan Gregg
 

Tendances (20)

DTrace Topics: Introduction
DTrace Topics: IntroductionDTrace Topics: Introduction
DTrace Topics: Introduction
 
Systems Performance: Enterprise and the Cloud
Systems Performance: Enterprise and the CloudSystems Performance: Enterprise and the Cloud
Systems Performance: Enterprise and the Cloud
 
EuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis MethodologiesEuroBSDcon 2017 System Performance Analysis Methodologies
EuroBSDcon 2017 System Performance Analysis Methodologies
 
Linux Performance Tools
Linux Performance ToolsLinux Performance Tools
Linux Performance Tools
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at Netflix
 
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPFOSSNA 2017 Performance Analysis Superpowers with Linux BPF
OSSNA 2017 Performance Analysis Superpowers with Linux BPF
 
Linux Performance Profiling and Monitoring
Linux Performance Profiling and MonitoringLinux Performance Profiling and Monitoring
Linux Performance Profiling and Monitoring
 
Performance Wins with eBPF: Getting Started (2021)
Performance Wins with eBPF: Getting Started (2021)Performance Wins with eBPF: Getting Started (2021)
Performance Wins with eBPF: Getting Started (2021)
 
Container Performance Analysis
Container Performance AnalysisContainer Performance Analysis
Container Performance Analysis
 
Velocity 2015 linux perf tools
Velocity 2015 linux perf toolsVelocity 2015 linux perf tools
Velocity 2015 linux perf tools
 
YOW2020 Linux Systems Performance
YOW2020 Linux Systems PerformanceYOW2020 Linux Systems Performance
YOW2020 Linux Systems Performance
 
Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)Linux Performance 2018 (PerconaLive keynote)
Linux Performance 2018 (PerconaLive keynote)
 
SREcon 2016 Performance Checklists for SREs
SREcon 2016 Performance Checklists for SREsSREcon 2016 Performance Checklists for SREs
SREcon 2016 Performance Checklists for SREs
 
Designing Tracing Tools
Designing Tracing ToolsDesigning Tracing Tools
Designing Tracing Tools
 
ACM Applicative System Methodology 2016
ACM Applicative System Methodology 2016ACM Applicative System Methodology 2016
ACM Applicative System Methodology 2016
 
Linux Systems Performance 2016
Linux Systems Performance 2016Linux Systems Performance 2016
Linux Systems Performance 2016
 
The New Systems Performance
The New Systems PerformanceThe New Systems Performance
The New Systems Performance
 
Performance Wins with BPF: Getting Started
Performance Wins with BPF: Getting StartedPerformance Wins with BPF: Getting Started
Performance Wins with BPF: Getting Started
 
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven RostedtKernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
Kernel Recipes 2017 - Understanding the Linux kernel via ftrace - Steven Rostedt
 
Performance Tuning EC2 Instances
Performance Tuning EC2 InstancesPerformance Tuning EC2 Instances
Performance Tuning EC2 Instances
 

En vedette

Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsLinux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
 
Broken Linux Performance Tools 2016
Broken Linux Performance Tools 2016Broken Linux Performance Tools 2016
Broken Linux Performance Tools 2016Brendan Gregg
 
BPF: Tracing and more
BPF: Tracing and moreBPF: Tracing and more
BPF: Tracing and moreBrendan Gregg
 
From DTrace to Linux
From DTrace to LinuxFrom DTrace to Linux
From DTrace to LinuxBrendan Gregg
 
Linux BPF Superpowers
Linux BPF SuperpowersLinux BPF Superpowers
Linux BPF SuperpowersBrendan Gregg
 
Linux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPFLinux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPFBrendan Gregg
 
Lisa12 methodologies
Lisa12 methodologiesLisa12 methodologies
Lisa12 methodologiesBrendan Gregg
 
Stop the Guessing: Performance Methodologies for Production Systems
Stop the Guessing: Performance Methodologies for Production SystemsStop the Guessing: Performance Methodologies for Production Systems
Stop the Guessing: Performance Methodologies for Production SystemsBrendan Gregg
 
Performance Analysis: The USE Method
Performance Analysis: The USE MethodPerformance Analysis: The USE Method
Performance Analysis: The USE MethodBrendan Gregg
 
Linux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF SuperpowersLinux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF SuperpowersBrendan Gregg
 
Linux Performance Analysis and Tools
Linux Performance Analysis and ToolsLinux Performance Analysis and Tools
Linux Performance Analysis and ToolsBrendan Gregg
 
Capturando pacotes de rede no kernelspace
Capturando pacotes de rede no kernelspaceCapturando pacotes de rede no kernelspace
Capturando pacotes de rede no kernelspaceCampus Party Brasil
 
Linux下Poll和Epoll内核源码剖析
Linux下Poll和Epoll内核源码剖析Linux下Poll和Epoll内核源码剖析
Linux下Poll和Epoll内核源码剖析Hao(Robin) Dong
 
Data Structures used in Linux kernel
Data Structures used in Linux kernel Data Structures used in Linux kernel
Data Structures used in Linux kernel assinha
 
Uma visão do mundo rails campus party 2011 - fabio akita
Uma visão do mundo rails   campus party 2011 - fabio akitaUma visão do mundo rails   campus party 2011 - fabio akita
Uma visão do mundo rails campus party 2011 - fabio akitaCampus Party Brasil
 
DevConf 2014 Kernel Networking Walkthrough
DevConf 2014   Kernel Networking WalkthroughDevConf 2014   Kernel Networking Walkthrough
DevConf 2014 Kernel Networking WalkthroughThomas Graf
 
Epoll - from the kernel side
Epoll -  from the kernel sideEpoll -  from the kernel side
Epoll - from the kernel sidellj098
 
Open stack + Cloud Foundry: Palo Alto Meetup February 2015
Open stack + Cloud Foundry: Palo Alto Meetup February 2015Open stack + Cloud Foundry: Palo Alto Meetup February 2015
Open stack + Cloud Foundry: Palo Alto Meetup February 2015Joshua McKenty
 

En vedette (18)

Linux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old SecretsLinux Performance Analysis: New Tools and Old Secrets
Linux Performance Analysis: New Tools and Old Secrets
 
Broken Linux Performance Tools 2016
Broken Linux Performance Tools 2016Broken Linux Performance Tools 2016
Broken Linux Performance Tools 2016
 
BPF: Tracing and more
BPF: Tracing and moreBPF: Tracing and more
BPF: Tracing and more
 
From DTrace to Linux
From DTrace to LinuxFrom DTrace to Linux
From DTrace to Linux
 
Linux BPF Superpowers
Linux BPF SuperpowersLinux BPF Superpowers
Linux BPF Superpowers
 
Linux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPFLinux 4.x Tracing: Performance Analysis with bcc/BPF
Linux 4.x Tracing: Performance Analysis with bcc/BPF
 
Lisa12 methodologies
Lisa12 methodologiesLisa12 methodologies
Lisa12 methodologies
 
Stop the Guessing: Performance Methodologies for Production Systems
Stop the Guessing: Performance Methodologies for Production SystemsStop the Guessing: Performance Methodologies for Production Systems
Stop the Guessing: Performance Methodologies for Production Systems
 
Performance Analysis: The USE Method
Performance Analysis: The USE MethodPerformance Analysis: The USE Method
Performance Analysis: The USE Method
 
Linux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF SuperpowersLinux 4.x Tracing Tools: Using BPF Superpowers
Linux 4.x Tracing Tools: Using BPF Superpowers
 
Linux Performance Analysis and Tools
Linux Performance Analysis and ToolsLinux Performance Analysis and Tools
Linux Performance Analysis and Tools
 
Capturando pacotes de rede no kernelspace
Capturando pacotes de rede no kernelspaceCapturando pacotes de rede no kernelspace
Capturando pacotes de rede no kernelspace
 
Linux下Poll和Epoll内核源码剖析
Linux下Poll和Epoll内核源码剖析Linux下Poll和Epoll内核源码剖析
Linux下Poll和Epoll内核源码剖析
 
Data Structures used in Linux kernel
Data Structures used in Linux kernel Data Structures used in Linux kernel
Data Structures used in Linux kernel
 
Uma visão do mundo rails campus party 2011 - fabio akita
Uma visão do mundo rails   campus party 2011 - fabio akitaUma visão do mundo rails   campus party 2011 - fabio akita
Uma visão do mundo rails campus party 2011 - fabio akita
 
DevConf 2014 Kernel Networking Walkthrough
DevConf 2014   Kernel Networking WalkthroughDevConf 2014   Kernel Networking Walkthrough
DevConf 2014 Kernel Networking Walkthrough
 
Epoll - from the kernel side
Epoll -  from the kernel sideEpoll -  from the kernel side
Epoll - from the kernel side
 
Open stack + Cloud Foundry: Palo Alto Meetup February 2015
Open stack + Cloud Foundry: Palo Alto Meetup February 2015Open stack + Cloud Foundry: Palo Alto Meetup February 2015
Open stack + Cloud Foundry: Palo Alto Meetup February 2015
 

Similaire à Open Source Systems Performance

Week1 Electronic System-level ESL Design and SystemC Begin
Week1 Electronic System-level ESL Design and SystemC BeginWeek1 Electronic System-level ESL Design and SystemC Begin
Week1 Electronic System-level ESL Design and SystemC Begin敬倫 林
 
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak   CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak PROIDEA
 
Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1
Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1
Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1Jagadisha Maiya
 
Troubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device DriversTroubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device DriversSatpal Parmar
 
Debugging Ruby Systems
Debugging Ruby SystemsDebugging Ruby Systems
Debugging Ruby SystemsEngine Yard
 
Designing Tracing Tools
Designing Tracing ToolsDesigning Tracing Tools
Designing Tracing ToolsSysdig
 
Debugging Ruby
Debugging RubyDebugging Ruby
Debugging RubyAman Gupta
 
Linux Performance Tools 2014
Linux Performance Tools 2014Linux Performance Tools 2014
Linux Performance Tools 2014Brendan Gregg
 
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...Orgad Kimchi
 
Auditing the Opensource Kernels
Auditing the Opensource KernelsAuditing the Opensource Kernels
Auditing the Opensource KernelsSilvio Cesare
 
breed_python_tx_redacted
breed_python_tx_redactedbreed_python_tx_redacted
breed_python_tx_redactedRyan Breed
 
[CCC-28c3] Post Memory Corruption Memory Analysis
[CCC-28c3] Post Memory Corruption Memory Analysis[CCC-28c3] Post Memory Corruption Memory Analysis
[CCC-28c3] Post Memory Corruption Memory AnalysisMoabi.com
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLCommand Prompt., Inc
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLMark Wong
 
Introduction to Java Profiling
Introduction to Java ProfilingIntroduction to Java Profiling
Introduction to Java ProfilingJerry Yoakum
 
Virtual platform
Virtual platformVirtual platform
Virtual platformsean chen
 
Nodejs性能分析优化和分布式设计探讨
Nodejs性能分析优化和分布式设计探讨Nodejs性能分析优化和分布式设计探讨
Nodejs性能分析优化和分布式设计探讨flyinweb
 

Similaire à Open Source Systems Performance (20)

Debug generic process
Debug generic processDebug generic process
Debug generic process
 
Week1 Electronic System-level ESL Design and SystemC Begin
Week1 Electronic System-level ESL Design and SystemC BeginWeek1 Electronic System-level ESL Design and SystemC Begin
Week1 Electronic System-level ESL Design and SystemC Begin
 
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak   CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
 
Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1
Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1
Troubleshooting linux-kernel-modules-and-device-drivers-1233050713693744-1
 
Troubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device DriversTroubleshooting Linux Kernel Modules And Device Drivers
Troubleshooting Linux Kernel Modules And Device Drivers
 
Debugging Ruby Systems
Debugging Ruby SystemsDebugging Ruby Systems
Debugging Ruby Systems
 
Designing Tracing Tools
Designing Tracing ToolsDesigning Tracing Tools
Designing Tracing Tools
 
Debugging Ruby
Debugging RubyDebugging Ruby
Debugging Ruby
 
Linux Performance Tools 2014
Linux Performance Tools 2014Linux Performance Tools 2014
Linux Performance Tools 2014
 
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
Performance analysis in a multitenant cloud environment Using Hadoop Cluster ...
 
Auditing the Opensource Kernels
Auditing the Opensource KernelsAuditing the Opensource Kernels
Auditing the Opensource Kernels
 
breed_python_tx_redacted
breed_python_tx_redactedbreed_python_tx_redacted
breed_python_tx_redacted
 
[CCC-28c3] Post Memory Corruption Memory Analysis
[CCC-28c3] Post Memory Corruption Memory Analysis[CCC-28c3] Post Memory Corruption Memory Analysis
[CCC-28c3] Post Memory Corruption Memory Analysis
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 
pg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQLpg_proctab: Accessing System Stats in PostgreSQL
pg_proctab: Accessing System Stats in PostgreSQL
 
Dpdk applications
Dpdk applicationsDpdk applications
Dpdk applications
 
Introduction to Java Profiling
Introduction to Java ProfilingIntroduction to Java Profiling
Introduction to Java Profiling
 
Sge
SgeSge
Sge
 
Virtual platform
Virtual platformVirtual platform
Virtual platform
 
Nodejs性能分析优化和分布式设计探讨
Nodejs性能分析优化和分布式设计探讨Nodejs性能分析优化和分布式设计探讨
Nodejs性能分析优化和分布式设计探讨
 

Plus de Brendan Gregg

YOW2021 Computing Performance
YOW2021 Computing PerformanceYOW2021 Computing Performance
YOW2021 Computing PerformanceBrendan Gregg
 
IntelON 2021 Processor Benchmarking
IntelON 2021 Processor BenchmarkingIntelON 2021 Processor Benchmarking
IntelON 2021 Processor BenchmarkingBrendan Gregg
 
Systems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting StartedSystems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting StartedBrendan Gregg
 
Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)Brendan Gregg
 
BPF Internals (eBPF)
BPF Internals (eBPF)BPF Internals (eBPF)
BPF Internals (eBPF)Brendan Gregg
 
re:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflixre:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at NetflixBrendan Gregg
 
UM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of SoftwareUM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of SoftwareBrendan Gregg
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceLISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceBrendan Gregg
 
LPC2019 BPF Tracing Tools
LPC2019 BPF Tracing ToolsLPC2019 BPF Tracing Tools
LPC2019 BPF Tracing ToolsBrendan Gregg
 
LSFMM 2019 BPF Observability
LSFMM 2019 BPF ObservabilityLSFMM 2019 BPF Observability
LSFMM 2019 BPF ObservabilityBrendan Gregg
 
YOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflixYOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflixBrendan Gregg
 
eBPF Perf Tools 2019
eBPF Perf Tools 2019eBPF Perf Tools 2019
eBPF Perf Tools 2019Brendan Gregg
 
YOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at NetflixYOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at NetflixBrendan Gregg
 
NetConf 2018 BPF Observability
NetConf 2018 BPF ObservabilityNetConf 2018 BPF Observability
NetConf 2018 BPF ObservabilityBrendan Gregg
 
ATO Linux Performance 2018
ATO Linux Performance 2018ATO Linux Performance 2018
ATO Linux Performance 2018Brendan Gregg
 
How Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for PerformanceHow Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for PerformanceBrendan Gregg
 
LISA17 Container Performance Analysis
LISA17 Container Performance AnalysisLISA17 Container Performance Analysis
LISA17 Container Performance AnalysisBrendan Gregg
 
Kernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at NetflixKernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at NetflixBrendan Gregg
 

Plus de Brendan Gregg (20)

YOW2021 Computing Performance
YOW2021 Computing PerformanceYOW2021 Computing Performance
YOW2021 Computing Performance
 
IntelON 2021 Processor Benchmarking
IntelON 2021 Processor BenchmarkingIntelON 2021 Processor Benchmarking
IntelON 2021 Processor Benchmarking
 
Systems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting StartedSystems@Scale 2021 BPF Performance Getting Started
Systems@Scale 2021 BPF Performance Getting Started
 
Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)Computing Performance: On the Horizon (2021)
Computing Performance: On the Horizon (2021)
 
BPF Internals (eBPF)
BPF Internals (eBPF)BPF Internals (eBPF)
BPF Internals (eBPF)
 
re:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflixre:Invent 2019 BPF Performance Analysis at Netflix
re:Invent 2019 BPF Performance Analysis at Netflix
 
UM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of SoftwareUM2019 Extended BPF: A New Type of Software
UM2019 Extended BPF: A New Type of Software
 
LISA2019 Linux Systems Performance
LISA2019 Linux Systems PerformanceLISA2019 Linux Systems Performance
LISA2019 Linux Systems Performance
 
LPC2019 BPF Tracing Tools
LPC2019 BPF Tracing ToolsLPC2019 BPF Tracing Tools
LPC2019 BPF Tracing Tools
 
LSFMM 2019 BPF Observability
LSFMM 2019 BPF ObservabilityLSFMM 2019 BPF Observability
LSFMM 2019 BPF Observability
 
YOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflixYOW2018 CTO Summit: Working at netflix
YOW2018 CTO Summit: Working at netflix
 
eBPF Perf Tools 2019
eBPF Perf Tools 2019eBPF Perf Tools 2019
eBPF Perf Tools 2019
 
YOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at NetflixYOW2018 Cloud Performance Root Cause Analysis at Netflix
YOW2018 Cloud Performance Root Cause Analysis at Netflix
 
BPF Tools 2017
BPF Tools 2017BPF Tools 2017
BPF Tools 2017
 
NetConf 2018 BPF Observability
NetConf 2018 BPF ObservabilityNetConf 2018 BPF Observability
NetConf 2018 BPF Observability
 
FlameScope 2018
FlameScope 2018FlameScope 2018
FlameScope 2018
 
ATO Linux Performance 2018
ATO Linux Performance 2018ATO Linux Performance 2018
ATO Linux Performance 2018
 
How Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for PerformanceHow Netflix Tunes EC2 Instances for Performance
How Netflix Tunes EC2 Instances for Performance
 
LISA17 Container Performance Analysis
LISA17 Container Performance AnalysisLISA17 Container Performance Analysis
LISA17 Container Performance Analysis
 
Kernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at NetflixKernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at Netflix
 

Dernier

Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 

Dernier (20)

Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
20230104 - machine vision
20230104 - machine vision20230104 - machine vision
20230104 - machine vision
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 

Open Source Systems Performance

  • 1. Open Source Systems Performance Brendan Gregg Lead Performance Engineer Joyent Jul, 2013
  • 2. A Play in Three Acts  A tale of operating systems, performance, and open source  Dramatis Personae - Solaris, an Operating System - Brendan Gregg, a Performance Engineer - Linux, a Kernel  Acts - 1. Before open source (traditional tools) - 2. Open source (source code-based tracing) - 3. Closed source
  • 3. Setting the Scene: Why Performance?  Reduce IT Spend - price/performance  Choose performing components - evaluation (benchmarking) of software and hardware  Develop scalable architectures - understand system limits and develop around them  Solve issues
  • 4. Setting the Scene: What is Systems Performance?  Analysis of: - A) the kernel - 2-20% wins: tuning TCP, NUMA, etc - 2-200x wins: latency outliers, bugs, etc - B) applications from system context - 2-2000x wins: eliminating unnecessary work  The basis is the system  The target is everything, down to metal  Think LAMP not AMP Applications Block Device Interface Volume Managers File Systems VFS System Libraries Device Drivers System Call Interface Metal Systems Performance Analysis Kernel Scheduler Virtual Memory Ethernet IP TCP/UDP Sockets Firmware
  • 5. Part 1. Before Open Source
  • 6. Part 1. Before Open Source  The year is 2002  Enter Solaris 9, stage left  Solaris 9 is not open source
  • 7. Solaris 9  Numerous performance observability tools  Performance, including resource controls and observability, were main features Scope Type Tools system counters vmstat(1M), iostat(1M), netstat(1M), kstat(1M), sar(1) system tracing snoop(1M), prex(1M), tnfdump(1) process counters ps(1), prstat(1M), ptime(1) process tracing truss(1), sotruss(1), apptrace(1) both profiling lockstat(1M), cpustat(1M), cputrack(1)
  • 8. Systems Performance  Typified by Unix tools like vmstat(1M) (from BSD):  Some drill-down were possible with options; eg, the Solaris -p:  Despite many tools, options, and metrics, the extent of observability was limited. This can be illustrated using a functional diagram $ vmstat 1 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr cd cd s0 s5 in sy cs us sy id 0 0 0 8475356 565176 2 8 0 0 0 0 1 0 0 -0 13 378 101 142 0 0 99 1 0 0 7983772 119164 0 0 0 0 0 0 0 224 0 0 0 1175 5654 1196 1 15 84 0 0 0 8046208 181600 0 0 0 0 0 0 0 322 0 0 0 1473 6931 1360 1 7 92 [...] $ vmstat -p 1 memory page executable anonymous filesystem swap free re mf fr de sr epi epo epf api apo apf fpi fpo fpf 8475336 565160 2 8 0 0 1 0 0 0 0 0 0 0 0 0 7972332 107648 1 29 0 0 0 0 0 0 0 0 0 0 0 0 7966188 101504 0 0 0 0 0 0 0 0 0 0 0 0 0 0 [...]
  • 9. Operating System Functional Diagram Applications DBs, all server types, ... Block Device Interface Ethernet Volume Managers IP File Systems TCP/UDP VFS Sockets Disk Disk Port Port Expander Interconnect I/O Bus Interface Transports I/O Controller Network Controller I/O Bridge System Libraries Device Drivers Scheduler Virtual Memory System Call Interface CPU Interconnect Memory Bus CPU 1 DRAM Operating System Hardware Kernel
  • 10. Solaris 9 Observability Coverage Applications DBs, all server types, ... Block Device Interface Ethernet Volume Managers IP File Systems TCP/UDP VFS Sockets Disk Disk Port Port Expander Interconnect I/O Bus Interface Transports I/O Controller Network Controller I/O Bridge System Libraries Device Drivers Scheduler Virtual Memory System Call Interface CPU Interconnect Memory Bus CPU 1 DRAM Operating System Hardware SolarisKernel truss apptrace sotruss iostat prex vmstat mpstat prstat ps cpustat cputrack snoop netstat lockstat kstat sar kstat Various: netstat kstat
  • 11. Problems  Below the syscall interface was dark, if not pitch black  Many components either had: - No metrics at all - Undocumented metrics (kstat)  Certain performance issues could not be analyzed - Time from asking Sun for a new performance metric to having it in production could be months or years or never - You solve what the current tools let you: the “tools method” of iterating over existing tools and metrics  Situation largely accepted as a better way wasn’t known  Much systems performance literature was written in this era, and is still around
  • 12. High Performance Tuning  Performance experts were skilled in the art of inference and experimentation - Study Solaris Internals for background - Determine kernel behavior based on indirect metrics - Create known workloads to test undocumented metrics, and to explore system behavior - Heavy use of the Scientific method  Science is good, source is better
  • 13. ... If the Universe was Open Source vi universe/particles.c: photon_t * spontaneous_emission(electron_t *e) { photon_t *p; if (e->e_orbit > 1) { p = palloc(e); e->e_orbit--; } else { electron_capture(e->e_nucleusp); return (NULL) } return (p); } vi universe/include/electron.h: struct electron { mass_t e_mass; /* electron mass */ charge_t e_charge; /* electron charge */ uint64_t e_flags; /* 0x01 particle; 0x10 wave */ int e_orbit; /* current orbit level */ boolean_t e_matter; /* 1 = matter; 0 = antimatter */ [...] } electron_t;
  • 14. Part 2. Open Source
  • 15. Part 2. Open Source  The year is 2005  Solaris 10, as OpenSolaris, becomes open source - In response to Linux, which always was
  • 16. Open Source Metrics  Undocumented kstats could now be understood from source - it was like being handed the source code to the Universe - I wasn’t a Sun badged employee; I’d been working without source access  Tool metrics could also be better understood, and exact behavior of the kernel  For example, where does “r” come from? $ vmstat 1 kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr cd cd s0 s5 in sy cs us sy id 0 0 0 8475356 565176 2 8 0 0 0 0 1 0 0 -0 13 378 101 142 0 0 99 1 0 0 7983772 119164 0 0 0 0 0 0 0 224 0 0 0 1175 5654 1196 1 15 84
  • 17. Understanding “r”  Starting with vmstat(1M)’s source, and drilling down: usr/src/cmd/stat/vmstat/vmstat.c: static void printhdr(int sig) { [...] if (swflag) (void) printf(" r b w swap free si so pi po fr de sr "); else (void) printf(" r b w swap free re mf pi po fr de sr "); [...] static void dovmstats(struct snapshot *old, struct snapshot *new) { [...] adjprintf(" %*lu", 1, DELTA(s_sys.ss_sysinfo.runque) / sys_updates);
  • 18. Understanding “r”, cont.  Searching on ss_sysinfo: usr/src/cmd/stat/common/statcommon.h: struct sys_snapshot { sysinfo_t ss_sysinfo; [...] usr/src/uts/common/sys/sysinfo.h: typedef struct sysinfo { /* (update freq) update action */ uint_t updates; /* (1 sec) ++ */ uint_t runque; /* (1 sec) += num runnable procs */ uint_t runocc; /* (1 sec) ++ if num runnable procs > 0 */ uint_t swpque; /* (1 sec) += num swapped procs */ uint_t swpocc; /* (1 sec) ++ if num swapped procs > 0 */ uint_t waiting; /* (1 sec) += jobs waiting for I/O */ } sysinfo_t;
  • 19. Understanding “r”, cont.  ss_sysinfo is populated from kstat: usr/src/cmd/stat/common/acquire.c: int acquire_sys(struct snapshot *ss, kstat_ctl_t *kc) { size_t i; kstat_named_t *knp; kstat_t *ksp; if ((ksp = kstat_lookup(kc, "unix", 0, "sysinfo")) == NULL) return (errno); if (kstat_read(kc, ksp, &ss->s_sys.ss_sysinfo) == -1) return (errno); [...]
  • 20. Understanding “r”, cont.  Searching on runque population, in the kernel: usr/src/uts/common/os/clock.c: static void clock(void) { * There is additional processing which happens every time * the nanosecond counter rolls over which is described * below - see the section which begins with : if (one_sec) [...] do { uint_t cpu_nrunnable = cp->cpu_disp->disp_nrunnable; nrunnable += cpu_nrunnable; [...] } while ((cp = cp->cpu_next) != cpu_list); [...] if (one_sec) { [...] if (nrunnable) { sysinfo.runque += nrunnable; sysinfo.runocc++; } Once-a-second snapshots? That’s good to know!
  • 21. Statistic Spelunking  A matter of browsing and reading source code - I use cscope, a text-based source code browser:  Doesn’t require expertise to begin with: keep reading code until it makes sense  Might take hours or days if you are new to a complex code base  You may only do this three times in your career, but each time was worth it! C symbol: runque File Function Line 0 sa.h <global> 188 uint64_t runque; 1 sysinfo.h <global> 132 uint_t runque; 2 sar.c prt_q_opt 919 (float )xx->si.runque / (float )xx->si.runocc, 3 kstat.c save_sysinfo 1066 SAVE_UINT32(ksi, sysinfo, runque); 4 vmstat.c dovmstats 316 adjprintf(" %*lu", 1, DELTA(s_sys.ss_sysinfo.runque) / sys_updates); 5 clock.c clock 862 sysinfo.runque += nrunnable; Find this C symbol: Find this global definition: Find functions called by this function: Find functions calling this function: Find this text string: Change this text string: Find this egrep pattern: Find this file: Find files #including this file:
  • 22. Open Source Dynamic Tracing  Solaris 10 also provided Dynamic Tracing (DTrace), which can observe virtually everything  Core feature of all later OpenSolaris derivatives, including SmartOS and OmniOS  Observability gaps now filled
  • 23. Solaris 10/SmartOS/OmniOS Observability Coverage Applications DBs, all server types, ... Block Device Interface Ethernet Volume Managers IP File Systems TCP/UDP VFS Sockets Disk Disk Port Port Expander Interconnect I/O Bus Interface Transports I/O Controller Network Controller I/O Bridge System Libraries Device Drivers Scheduler Virtual Memory System Call Interface CPU Interconnect Memory Bus CPU 1 DRAM Operating System Hardware Solaris/illumos Kernel truss iostat vmstat mpstat prstat ps cpustat cputrack snoop netstat sar kstat lockstat Various: kstat netstat kstat dtrace plockstat intrstat
  • 24. Open Source Dynamic Tracing: Example  Given the kernel source code, eg, ZFS SPA sync:  Trace and time it using the DTrace function boundary tracing (fbt) provider: usr/src/uts/common/fs/zfs/spa.c: /* * Sync the specified transaction group. New blocks may be dirtied as * part of the process, so we iterate until it converges. */ void spa_sync(spa_t *spa, uint64_t txg) { dsl_pool_t *dp = spa->spa_dsl_pool; [...] # dtrace -n 'fbt::spa_sync:entry { self->ts = timestamp; } fbt::spa_sync:return /self->ts/ { printf("%Y %d ms", walltimestamp, (timestamp - self->ts) / 1000000); self->ts = 0; }' dtrace: description 'fbt::spa_sync:entry ' matched 2 probes CPU ID FUNCTION:NAME 0 53625 spa_sync:return 2013 Jul 26 17:37:02 12 ms 0 53625 spa_sync:return 2013 Jul 26 17:37:08 726 ms 6 53625 spa_sync:return 2013 Jul 26 17:37:17 6913 ms 6 53625 spa_sync:return 2013 Jul 26 17:37:17 59 ms Awesome!
  • 25. Dynamic Tracing Scripts Applications DBs, all server types, ... Block Device Interface Ethernet Volume Managers IP File Systems TCP/UDP VFS Sockets System Libraries Device Drivers Scheduler Virtual Memory System Call Interface iosnoop, iotop disklatency.d satacmds.d satalatency.d scsicmds.d scsilatency.d sdretry.d, sdqueue.d ide*.d, mpt*.d priclass.d, pridist.d cv_wakeup_slow.d displat.d, capslat.d opensnoop, statsnoop errinfo, dtruss, rwtop rwsnoop, mmap.d, kill.d shellsnoop, zonecalls.d weblatency.d, fddist dnlcsnoop.d zfsslower.d ziowait.d ziostacks.d spasync.d metaslab_free.d fswho.d, fssnoop.d sollife.d solvfssnoop.d hotuser, umutexmax.d, lib*.d node*.d, erlang*.d, j*.d, js*.d php*.d, pl*.d, py*.d, rb*.d, sh*.d mysql*.d, postgres*.d, redis*.d, riak*.d Language Providers: Databases: sotop.d, socketio.d, so1stbyte.d, soconnect.d, soaccept.d ipio.d, ipproto.d, ipstat.d, ipfbtsnoop.d, icmpsnoop.d tcp1stbyte.d, tcpaccept.d, tcpconnect.d, tcpconnlat.d, tcpio.d tcpbytes.d, tcpsize.d, tcpnmap.d, udpio.d, udpstat.d cifs*.d, iscsi*.d nfsv3*.d, nfsv4*.d ssh*.d, httpd*.d :Services minfbypid.d pgpginbypid.d macops.d ngesnoop.d, ngelink.d These are some of my scripts from the DTraceToolkit, the DTrace book, and other collections. I’d add more but I ran out of room.
  • 26. Modern Systems Performance  Typified by an abundance of high resolution useful metrics (latency)  No longer a problem of missing metrics, but how to visualize many metrics, and across clouds - eg, latency heat maps:  Prior tools are useful as starting points, with tracing to dig deeper  In the following sections, I’ll describe modern Linux Systems Performance, summarizing how traditional and new tools can be used together - I’ll group dtrace/systemtap/perf/lttng/ktap/etc as “dynamic tracing”, which is a simplification: some needs may not be met by all those tools
  • 27. Linux CPU Analysis  Traditional tools: - 1. system wide usage vmstat - 2. per-processor usage mpstat - 3. per-process usage top, ps - 4. user- or kernel-stack profiling perf record -agF - 5. cycle analysis perf stat  Modern tools: - 6. tracing scheduler latency perf sched - 7. tracing CPU usage of functions dynamic/static tracing - 8. tracing CPU consumption of spin locks dynamic/static tracing - 9. CPU cross call tracing dynamic/static tracing - 10. interrupt tracing dynamic/static tracing
  • 28. Linux Memory Analysis  Traditional tools: - 1. system wide usage vmstat - 2. per-process usage top - 3. kernel usage /proc/meminfo, slaptop - 4. swapping activity sar - 5. leak detection valgrind  Modern tools: - 6. tracing allocations dynamic tracing - 7. tracing page faults dynamic/static tracing - 8. tracing kswapd activity dynamic/static tracing
  • 29. Linux File System Analysis  Traditional tools: - 1. cache usage free, /proc/meminfo - 2. syscall access strace (expensive)  Modern tools: - 3. tracing VFS accesses dynamic/static tracing - 4. tracing file system latency latencytop, dynamic/static tracing - 5. tracing file system internals dynamic/static tracing
  • 30. Linux Disk Analysis  Traditional tools: - 1. per-disk statistics iostat - 2. per-process usage pidstat -d, iotop  Modern tools: - 3. disk I/O latency tracing blktrace, static tracing - 4. lower I/O stack tracing dynamic/static tracing - 5. SCSI command tracing dynamic/static tracing - 6. device driver tracing dynamic/static tracing
  • 31. Linux Network Analysis  Traditional tools: - 1. system wide usage netstat -s - 2. per-interface usage netstat -i, sar -n DEV, ip - 3. TCP statistics netstat -s, sar -n TCP - 4. packet tracing tcpdump - 5. socket call tracing strace (expensive) - 6. experimental tests ping, traceroute  Modern tools: - 6. tracing socket-level latency dynamic/static tracing - 7. TCP retransmit (only) tracing dynamic tracing - 8. tracing TCP kernel internals dynamic tracing
  • 32. Linux Network Analysis, Example  TCP retransmits: given tcp_retransmit_skb(), show the dest IP addr. Source code: net/ipv4/tcp_output.c: int tcp_retransmit_skb(struct sock *sk, struct sk_buff *skb) { struct tcp_sock *tp = tcp_sk(sk); int err = __tcp_retransmit_skb(sk, skb); [...] include/linux/tcp.h: struct tcp_sock { /* inet_connection_sock has to be the first member of tcp_sock */ struct inet_connection_sock inet_conn; [...] include/net/inet_connection_sock.h: struct inet_connection_sock { /* inet_sock has to be the first member! */ struct inet_sock icsk_inet; [...]
  • 33. Linux Network Analysis, Example  ... More spelunking, like earlier. Not trivial, but doable. include/net/inet_sock.h: struct inet_sock { /* sk and pinet6 has to be the first two members of inet_sock */ struct sock sk; #if IS_ENABLED(CONFIG_IPV6) struct ipv6_pinfo *pinet6; #endif /* Socket demultiplex comparisons on incoming packets. */ #define inet_daddr sk.__sk_common.skc_daddr [...] include/net/sock.h struct sock { /* * Now struct inet_timewait_sock also uses sock_common, so please just * don't add nothing before this first member (__sk_common) --acme */ struct sock_common __sk_common; [...] Here it is
  • 34. Linux Network Analysis, Example Script  TCP retransmit tracing script, using DTrace4Linux (prototype): #!/usr/sbin/dtrace -s #pragma D option quiet dtrace:::BEGIN { trace("Tracing TCP retransmits... Ctrl-C to end.n"); } fbt::tcp_retransmit_skb:entry { this->so = (struct sock *)arg0; this->d = (unsigned char *)&this->so->__sk_common.skc_daddr; printf("%Y: retransmit to %d.%d.%d.%d, by:", walltimestamp, this->d[0], this->d[1], this->d[2], this->d[3]); stack(99); }
  • 35. Linux Network Analysis, Example Output  TCP retransmit tracing script, using DTrace4Linux (prototype): # ./tcpretransmit.d Tracing TCP retransmits... Ctrl-C to end. 2013 Feb 23 18:24:11: retransmit to 10.2.124.2, by: kernel`tcp_retransmit_timer+0x1bd kernel`tcp_write_timer+0x188 kernel`run_timer_softirq+0x12b kernel`tcp_write_timer kernel`__do_softirq+0xb8 kernel`read_tsc+0x9 kernel`sched_clock+0x9 kernel`sched_clock_local+0x25 kernel`call_softirq+0x1c kernel`do_softirq+0x65 kernel`irq_exit+0x9e kernel`smp_apic_timer_interrupt+0x6e kernel`apic_timer_interrupt+0x6e [...]
  • 36. Linux Network Example, cont.  I created a custom performance tool on the fly, without kernel changes  Would it be possible if the kernel wasn’t open source?
  • 37. Opportunities  Open source allows dynamic tracing: otherwise you are tracing blind  Dynamic tracing allows custom metrics and scripts (tools) to be written - fill in all observability gaps; can solve most performance issues  Many people will use dynamic tracing: eg, DTraceToolkit, DTrace book, company tools; only some may author the tools: the OS or perf engineer on your team (which is ok)  Dynamic tracing also allows new methodologies - prior methodologies constrained by existing tools and metrics - new methodologies can be explored, as any question posed can be answered  Examples of new methodologies - USE method - Thread State Analysis method
  • 38. Challenges  Systems performance literature written for the pre-open source Unix days - Gives the impression that performance ends with older static tools  DTrace not on Linux yet - Two ports are in progress: - DTrace4Linux: https://github.com/dtrace4linux/linux - Oracle Linux DTrace  Instead of waiting, you can try an illumos-kernel based distro like SmartOS - illumos is the surviving fork of OpenSolaris. Which brings us to Act 3.
  • 39. Act 3. Closed Source
  • 40. Act 3. Closed Source  The year is 2010  Oracle stops releasing updates for OpenSolaris  Oracle Solaris 11 is released a year later, closed source  Provides us with a unique additional perspective for open source systems performance
  • 41. Closed Source Metrics  This closed the only documentation for many metrics and kernel internals - Back to inference and experimentation by the end user - Will get harder over time as documentation ages: without a Solaris Internals 3rd Edition, kernel internals may become as opaque as it was in the 90’s
  • 42. Closed Source Dynamic Tracing  Makes using the DTrace fbt provider much harder - Hypothetical example to show how this could manifest: - Dynamic tracing of ZFS SPA sync during a performance investigation: Where’d spa_sync() go? Did it get renamed or removed? - Could be worse if tracing succeeds, but produces misleading metrics due to unknown changes  Note that the capabilities are still there, and can be used by Oracle support # dtrace -n 'fbt::spa_sync:entry { printf("%Y", walltimestamp); }' dtrace: invalid probe specifier fbt::spa_sync:entry { printf("%Y", walltimestamp); }: probe description fbt::spa_sync:entry does not match any probes
  • 43. Elsewhere at Oracle  Their DTrace port for Oracle Linux won’t have this handicap - although, the fbt provider hasn’t been included yet
  • 44. Epilog: The New Systems Performance  An era of: - Open source - Dynamic tracing - Methodologies - Distributed systems (cloud)  Covered in my book, out this year:
  • 45. Thank you!  email: brendan@joyent.com  twitter: @brendangregg  blog: http://dtrace.org/blogs/brendan