In order to scale AdGear's real-time bidding (RTB) platform, we've been optimizing system and application performance. In this talk, I'll share all my findings, from basic tips to advanced topics. I'll cover coding patterns, metric collection, tracing and much more!
Talk objectives:
- Share basic tips, common pitfalls, efficient coding patterns
- Introduce tooling for metric collection (statsderl, vmstats, system-stats, riak_sysmon)
- Highlight erlang trace (fprof, flame graphs)
- Expose advanced topics (VM tuning, lock-counter, systemtap, cpuset)
Target audience:
- Anyone looking to improve the performance of their Erlang application.
14. HTTP API server
parsing request
•
binary matching is very powerful!
•
working with binaries is more memory efficient
•
binaries over 64 bytes are shared (not copied)
•
faster than the built-in http parser (BIF) when
running on many cores and using hipe
•
keep state in a record
•
O(1) lookups
20. HTTP API server
building response
•
ETS is your friend!
•
cache time date in ETS public table
•
{read_concurrency, true}
•
if you store a binary over 64 bytes, it won’t get
copied!
•
have a gen_server update the cache
•
every second for the time
•
every day for the date
21. HTTP API server
building response
•
do not try to rewrite everything
•
use community projects and contribute back!
•
often your application will spend most of its time
talking to external services
•
premature optimization is usually bad
22. Gotchas
slow functions / modules
•
erlang:now/0 vs os:timestamp/0
•
proplists:get_value() vs lists:keyfind()
•
timer:send_after() vs erlang:send_after()
•
gen_udp:send() vs erlang:port_command()
•
avoid erlang:controlling_process() if you can
•
avoid base64, string, unicode modules
24. Profiling
info
•
useful to find slow code paths
•
fprof
•
•
output is really hard to understand
•
•
uses erlang:trace/3
erlgrind to read in kcachegrind
eflame
•
also uses erlang:trace/3
•
nice graphical output
28. Micro benchmarks
info
•
start with profiling
•
useful for experimentation and to validate
hypothesis
•
small benchmarking library called timing
•
uses the excellent bear (statistics) library
33. NIFs
info
•
function that is implemented in C instead of Erlang
•
can be dangerous…
•
•
OOM (memory leak)
•
•
crash VM (segfault)
must return < 500 us (to be safe…)
ideally should yield and use enif_consume_timeslice
•
•
what is a reduction?
dirty schedulers (R17)
•
finally!
37. Statsderl
info
•
statsd client
•
very cheap to call (async)
•
offers 3 kinds of metrics:
•
counters - for counting (e.g QPS)
•
gauges - for absolute values (e.g. system memory)
•
timers - similar to gauges but with extra statistics
41. System Stats
info
•
load1, load5, load15
•
cpu percent
•
can be misleading because of spinning
schedulers
•
virtual memory size
•
resident memory size
•
very useful to track those OOM crashes
!
46. Entop
info
•
top(1)-like tool for the Erlang VM
•
can be used remotely
•
gives per process:
•
•
reductions
•
message queue length
•
!
pid / name
heap size
48. VM Tuning
info
•
+K true (kernel polling)
•
+sct db (scheduler bind)
•
+scl false (disable load distribution)
•
+sfwi 500 (force sheduler wakeup NIFs)
•
+spp true (port parallelism)
•
+zdbbl (distribution buffer busy limit)
•
test with production load (synthetic benchmarks
can be misleading)
49. Cset
info
•
tool to help create cpusets
•
reduces non voluntary
context-switches
•
reserve first two CPUs for
interrupts and background
jobs
•
reserve rest of CPUs for the
Erlang VM
•
linux only
