More Related Content Similar to Bring the Noise (20) Bring the Noise2. Ninety minutes is a
long time.
- motivations
- skyline
- oculus
- demo!
- questions
This talk:
~10
~25
~30
~10
~15
Tuesday, 9 July 13
3. Ninety minutes is a
long time.
- motivations
- skyline
- oculus
- demo!
- questions
This talk:
~10
~25
~30
~10
~15
But we have some
sweet stuff to show
you.
Tuesday, 9 July 13
6. 1.5 billion page views
$117 million of goods sold
950 thousand users
Tuesday, 9 July 13
7. 1.5 billion page views
$117 million of goods sold
950 thousand users
(in december ‘12)
Tuesday, 9 July 13
13. 30+ DEPLOYS A DAY
(~8 commits per deploy!)
Tuesday, 9 July 13
15. We optimize for quick recovery
by anticipating problems...
Tuesday, 9 July 13
17. Can’t fix what you
don’t measure!
- W. Edwards Deming
Tuesday, 9 July 13
20. “Not all things that
break throw errors.”
- Oscar Wilde
Tuesday, 9 July 13
25. If it doesn’t move,
graph it anyway
(it might make a run for it)
Tuesday, 9 July 13
27. [1358731200,20]
[1358731200,20]
[1358731200,20]
[1358731200,20]
[1358731200,20]
[1358731200,20]
[1358731200,20]
[1358731200,20]
[1358731200,60]
[1358731200,20]
[1358731200,20]
Tuesday, 9 July 13
31. “...but there are also
unknown unknowns -
there are things we do
not know we don’t
know.”
Tuesday, 9 July 13
37. Q). How do you analyze a
timeseries for anomalies
in real time?
Tuesday, 9 July 13
38. A). Lots of HTTP requests
to Graphite’s API!
Tuesday, 9 July 13
39. Q). How do you analyze a
quarter million timeseries
for anomalies in real time?
Tuesday, 9 July 13
47. ~ 10s
( ~ 1minBest case:
Tuesday, 9 July 13
52. Q). How do you get a
quarter million timeseries
into Redis on time?
Tuesday, 9 July 13
55. Graphite’s relay agent
original
graphite backup graphite
[statsd.numStats, [1365603422, 82345]]
pickles
[statsd.numStats, [1365603432, 80611]]
[statsd.numStats, [1365603412, 73421]]
Tuesday, 9 July 13
64. “[1358711400, 51],
=> get statsD.numStats
----------------------------
[1358711410, 23],
[1358711420, 45],”
Tuesday, 9 July 13
67. [ 1 , 2 ]
Stuff we care about
Extra junk
Tuesday, 9 July 13
76. Assign Redis keys to each process
Process decodes and analyzes
The Analyzer
Tuesday, 9 July 13
82. Basic algorithm:
“A metric is anomalous if its
latest datapoint is over three
standard deviations above
its moving average.”
Tuesday, 9 July 13
90. Real world data doesn’t
necessarily follow a perfect
normal distribution.
Tuesday, 9 July 13
92. A robust set of algorithms is the
current focus of this project.
Tuesday, 9 July 13
93. Q). How do you analyze a
quarter million timeseries
for correlations?
Tuesday, 9 July 13
96. “[[975, 1365528530],
[643, 1365528540],
[750, 1365528550],
[992, 1365528560],
[580, 1365528570],
[586, 1365528580],
[649, 1365528590],
[548, 1365528600],
[901, 1365528610],
[633, 1365528620]]”
Use raw timeseries instead of raw graphs
Tuesday, 9 July 13
106. No need to run DTW on all 250k.
Tuesday, 9 July 13
108. “975 643 643 750 992 992 992 580”
“sharpdecrement flat increment
sharpincrement flat flat
shapdecrement”
Shape Description Alphabet
Tuesday, 9 July 13
109. “975 643 643 750 992 992 992 580”
“sharpdecrement flat increment
sharpincrement flat flat
shapdecrement”
Shape Description Alphabet
“24 4 4 11 25 25 25 0 1”
(normalization step)
Tuesday, 9 July 13
111. Search for shape description
fingerprint in Elasticsearch
Tuesday, 9 July 13
112. Run DTW on results
as final polish
Tuesday, 9 July 13
115. Fast DTW - O(N)
coarsen
project
refine
Tuesday, 9 July 13
117. Elasticsearch Details
Phrase search for first pass scores
across shape description fingerprints
Custom FastDTW and euclidian
distance plugins to score across the
remaining filtered timeseries
Tuesday, 9 July 13
120. First pass query
:match => {
:fingerprint => {
:query => “sdec inc sinc sdec inc”,
:type => "phrase",
:slop => 20
}
}
shape description
fingerprint
Tuesday, 9 July 13
121. Refinement query{:custom_score
=>
{
:query
=>
<first_pass_query>,
:script
=>
"oculus_dtw",
:params
=>
{
:query_value
=>
“10
20
20
10
30”,
:query_field
=>
"values.untouched",
},
}
raw timeseries
Tuesday, 9 July 13
131. Special thanks to:
Dr. Neil Gunther, PerfDynamics
Dr. Brian Whitman, Echonest
Burc Arpat, Facebook
Seth Walker, Etsy
Rafe Colburn, Etsy
Mike Rembetsy, Etsy
John Allspaw, Etsy
Tuesday, 9 July 13