2. JVM diagnostic interfaces
• JMX
• JVMTI – native API only
• Attach API
Ad hoc instrumentation
and more
• Perf counters
• Heap dump
• Flight recorder
3. MBeans: threading
CPU usage per thread (user / sys)
Memory allocation per thread
Block / wait times
Should be enabled
Stack traces
Invaluable
5. MBeans: memory
• Memory geometry information
• Collection count
• Last collection details
for each collector
• GC events available as notifications
since Java 7
8. JVM Attach API
• List JVM processes
• Attach to JVM by PID
• Send control commands
heap dump / histogram
stack dump
• Inspect system properties and VM options
• Launch instrumentation agents
https://github.com/gridkit/jvm-attach
9. SJK: hh --dead
Dead object histogram
Similar to jmap –histo
Invoke jmap –histo two time
all heap objects
live heap object
calculates difference
Can show top N rows
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#hh-command
11. SJK: jps
JDK’s jps on steroid
Uses attach API
Lists VMs
Filtering by JVM system properties
Prints property values
Prints effective –XX options
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#jps-command
13. Perf counters
Based on shared memory
safe for target JVM
Flat data model
misc JVM counters
true GC CPU usage data
you can add own counter programmatically
14. Stack Trace Sampling
Capture
• Dump stack traces via local connection
• Store in highly compressed dump
10-30 bytes per trace
Analysis
• Frame frequency
• Conditional frame frequency
• Traces classification histogram
https://github.com/aragozin/jvm-tools/blob/master/sjk-core/COMMANDS.md#stcap-command
16. Working with heap dumps
Java API to traverse heap dump object graph
Available at https://github.com/aragozin/jvm-tools/tree/master/hprof-heap
Based on NetBeans profiler library
No temporary files used
Fixed generic method signatures
Improved performance
Useful for
In-place processing of large heap dumps
150 GiB is my personal record
Write domain specific heap usage reports
17. Working with heap dumps
HeapPath
Convenient way to extract value from dump
Error proof
Handles String, primitives/boxed and arrays
myfield1.myfield2.myfield3
myarrayfield[0].myfield
myarrayfield[*].myfield
myarrayfield[*][*]
myfield1.*.myfield3
[*].value(MyClass)
myhashmap?entrySet[key=description].value
18. Working with heap dumps
See also
https://github.com/vlsi/mat-calcite-plugin
Heap dump meets SQL
19. SJK Summary
Visit https://github.com/aragozin/jvm-tools
Single executable JAR
Command line interface
Exploits JMX / Attach API / PerfCounters
Sampling profiler included
Extensible commands
Write commands for your own application
20. Sigar
System Information Gatherer And Reporter
https://github.com/hyperic/sigar
• Cross platform
• Common system metrics
CPU, Context switches, IO, etc
• Java bindings
Self extracting JAR: org.gridkit.lab:sigar-lib:1.6.4
26. Self profiling benchmarks
Memory allocation regression tests
assert for memory consumption using thread’s
allocation counter
Microbenchmarks
MonitorGCeventstoexcluderesultaffectedbyGCpause
Track CPU usage
Thread sampling during benchmark
27. Performance tests
Nimble – framework for automated distributed performance testing
https://github.com/gridkit/nimble
https://code.google.com/p/gridkit/source/browse/grid-lab/trunk/examples/zk-benchmark-sample/
End – to – End automation
Setup environment
Run test scenarios
Capture application KPI
Capture metrics for OS and Java processes
Integrated profiling with BTrace
Single output data file
28. Profiling in production
In-house continuous query engine
20+ applications
Different environments, support teams, etc
A lot of performance challenges
Built-in self profiling – currently in pilot
Thread sampling
CPU / allocation tracing
Application specific diagnostics
Fully encapsulated in application itself
Threading MBean performance
depredate for multithreaded access
29. Heap analyzer
In-house continuous query engine
Relational graph up to few thousand nodes
High memory consumption
Heap dump reporter automatically generates
Memory consumption by operators
Row count per operator
Graph topology
In Excel friendly format
32. Heap dump API + scripting
Coding in Java works well, but
Reports are not interactive
Slow trial and error turn around
Howaboutinteractive consoleforheapanalysis?
bean shell / groovy ?
how to do code completion in console?
33. Big Brother 4 J
We have already done it for performance testing
Project detection
Metrics capturing
Integrated profiling sampling + instrumentation
All metric in one place
How about ?
Standalone tool for production
Detecting specific Java processing
Creating flight recording automatically
In hyper dense file format