Slides from a talk delivered during 4Developers conference in Warsaw covering concepts of time-series databases.

1. @SrcMinistry @MariuszGil
Time series databases
Data processing

3. We are developers

4. Code

5. Write it

6. Test it

7. Release it

8. Where does our
responsibility end?

9. Devs vs SysAdmins

10. No man’s land

12. Context…

13. The story

14. One app
10+ servers
Millions of users

15. Infra monitoring

16. Pro blems

17. What’s going on?

18. You can't manage what
you can't measure
W. Edwards Deming

19. Gather

20. 2016-04-11 02:00:00 45
2016-04-11 03:00:00 42
2016-04-11 04:00:00 30
2016-04-11 05:00:00 46
2016-04-11 06:00:00 70
2016-04-11 07:00:00 120

21. Store

22. Present

24. React

25. Low resolution
No context

26. Metrics

27. Metrics, values that
change over time
Matches everything

28. What should be
measured?

29. Everything what is
important for you

30. Tech
CPU / PV / reqs / resp_time

31. Business
KPIs / PV / UU

32. Why it should be
measured?

33. Insights

34. Time series
databases

35. Software system that’s
optimized for handling arrays
of numbers indexed by time
Wikipedia definition

36. Many solutions…

42. Hello world

43. CREATE DATABASE test

44. SHOW DATABASES
name: databases
---------------
name
_internal
test

45. USE test
Using database test

46. INSERT clicks,type=cpc,source=ad-partner-1,paid=yes,widget=promo-box value=0.43
INSERT clicks,type=cpc,source=ad-partner-2,paid=yes,widget=promo-box value=0.50
INSERT clicks,type=cpc,source=ad-partner-1,paid=yes,widget=top value=0.62
INSERT clicks,type=cpm,source=ad-partner-1,paid=yes,widget=top value=0.80

47. SELECT * FROM clicks
name: clicks
------------
time paid source type value widget
1459850636770890539 yes ad-partner-1 cpc 0.43 promo-box
1459850643866389522 yes ad-partner-2 cpc 0.50 promo-box
1459850656407067481 yes ad-partner-1 cpc 0.62 top
1459850668781668282 yes ad-partner-1 cpm 0.80 top

48. SELECT COUNT(value), MEAN(value), MEDIAN(value) FROM clicks
WHERE time > NOW() - 2m GROUP BY time(5s)
name: clicks
------------
time count mean median
1459851275000000000 0
1459851280000000000 0
1459851285000000000 3 0.330 0.33
1459851290000000000 0
1459851295000000000 2 0.420 0.420
1459851300000000000 2 0.365 0.365
1459851305000000000 3 0.446 0.43
1459851310000000000 0
1459851315000000000 0

49. SELECT COUNT(value), MEAN(value), MEDIAN(value) FROM clicks
WHERE time > NOW() - 2m GROUP BY time(5s) FILL(0)
name: clicks
------------
time count mean median
1459851275000000000 0 0 0
1459851280000000000 0 0 0
1459851285000000000 3 0.330 0.33
1459851290000000000 0 0 0
1459851295000000000 2 0.420 0.420
1459851300000000000 2 0.365 0.365
1459851305000000000 3 0.446 0.43
1459851310000000000 0 0 0
1459851315000000000 0 0 0

50. Functions
Aggregrations
COUNT
DISTINCT
INTEGRAL
MEAN
MEDIAN
SPREAD
SUM
Selectors
BOTTOM
FIRST
LAST
MAX
PERCENTILE
TOP
Transformations
CEILING
DERIVATIVE
DIFFERENCE
FLOOR
HISTOGRAM
MOVING_AVERAGE
STDDEV

52. Real world

53. Writing data

54. App
HTTP API
Graphite protocol
Collectd protocol
JSON+UDP

55. curl -i -XPOST 'http://localhost:8086/write?db=test' --data-binary
'cpu_load_short,host=server01,region=us-west value=0.64 1434055562000000000’
HTTP/1.1 204 No Content
Request-Id: 02e7757f-fb1e-11e5-809d-000000000000
X-Influxdb-Version: 0.11.0
Date: Tue, 05 Apr 2016 11:03:22 GMT

56. curl -G 'http://localhost:8086/query'
--data-urlencode "db=test"
--data-urlencode "q=SELECT value FROM cpu_load_short WHERE region='us-west'"
{"results":[{"series":[{"name":"cpu_load_short","columns":["time","value"],"values":
[["2015-06-11T20:46:02Z",0.64]]}]}]}

57. // create an array of points
$points = array(
new Point(
'test_metric', // name of the measurement
0.64, // the measurement value
['host' => 'server01', 'region' => 'us-west'], // optional tags
['cpucount' => 10], // optional additional fields
1435255849 // Time precision has to be set to seconds!
),
new Point(
'test_metric', // name of the measurement
0.84, // the measurement value
['host' => 'server01', 'region' => 'us-west'], // optional tags
['cpucount' => 10], // optional additional fields
1435255849 // Time precision has to be set to seconds!
)
);
// we are writing unix timestamps, which have a second precision
$result = $database->writePoints($points, Database::PRECISION_SECONDS);

58. Data model

59. $statsd = new LeagueStatsDClient();
$statsd->increment('web.pageview');
$statsd->decrement('storage.remaining');
$statsd->increment(array(
'first.metric',
'second.metric'
), 2);
$statsd->increment('web.clicks', 1, 0.5);
Counters

60. $statsd = new LeagueStatsDClient();
$statsd->gauge('api.logged_in_users', 123456);
Gauges

61. $statsd = new LeagueStatsDClient();
$statsd->timing('api.response_time', 23.1);
Timers

62. $statsd = new LeagueStatsDClient();
$userID = 23;
$statsd->set('api.unique_logins', $userID);
Sets

63. Data volume

65. CREATE CONTINUOUS QUERY <cq_name>
ON <database_name>
[RESAMPLE [EVERY <interval>] [FOR <interval>]]
BEGIN
SELECT <function>(<stuff>)[,<function>(<stuff>)]
INTO <different_measurement>
FROM <current_measurement>
[WHERE <stuff>]
GROUP BY time(<interval>)[,<stuff>]
END

67. Rsyslog

68. Rsyslog

69. Rsyslog

70. Possibilities!

71. App heart beat

72. Anomaly detection

74. Monitoring with
existing stack

76. Alternatives

77. Gather. Store.
Present. React.

78. No rocket science

79. Only rocket fuel

80. @SrcMinistry
Thanks!
@MariuszGil