InfluxDB Time Series Database Overview
•
5 j'aime•1,727 vues
InfluxDB is an open-source, distributed, time series database with no external dependencies. Slides displayed during Devoxx FR 2015
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à InfluxDB Time Series Database Overview
Similaire à InfluxDB Time Series Database Overview (20)
Dernier
Dernier (20)
InfluxDB Time Series Database Overview
- 2. @zepouet#InfluxDB :: InfluxDB :: Time Series :: • About Me • What is a time serie ? • State of the Art in 2015 • Why yet another product for time series ? • Live Demo • Q/A
- 4. What is a time series ? Things happening in times…
- 6. @zepouet#InfluxDB Events, events… events • Measurements (physical sensors…) • Exceptions (applications) • Page views • User actions • Commits Git • Webapp Deployment • Things appening in time
- 7. State of the Art :: 2015
- 8. @zepouet#InfluxDB What we have to store ? • At the moment, we have : • Graphite • OpenTSDB (events, Hadoop, HBase…) • Kairos (events, rewrite from OpenTSBD) • Ganglia (more present in BigData/Hadoop) • And others…
- 9. @zepouet#InfluxDB What we have to collect ? • At the moment, we have : • CollectD • Sensu • DropWizard/Metrics • JMXTrans • Jolokia
- 11. @zepouet#InfluxDB Because in 2015, we need • Simple product to install and manage • To store millions of points (IoT is here) • HTTP native support (JSON) • Build with API • Automatically clear out old data • Easy scalable : cloud is a buzzword
- 14. @zepouet#InfluxDB Feedback •Data volume : •1 event / sensor / minute •1 * 60 * 24 = 1440 events per day •42.300 events per month •518.400 events per year •First error : use MYSQL •Second error : bad pattern with InfluxDB
- 16. @zepouet#InfluxDB About InfluxDB •An opensource distributed time series database • ErrPlane • MIT License • Written in GO • Young but awesome project
- 17. @zepouet#InfluxDB InfluxDB :: design goals • Simple to install and manage thank to Go. • No external dependencies like Zookeeper and Hadoop. • HTTP(s) interface for reading and writing data. • Horizontally scalable. • On disk and in memory. Most data is cold. • Compute percentiles and others functions on the fly. • Downsample data on different windows of time.
- 18. @zepouet#InfluxDB InfluxDB :: installing • MacOS : $ brew install influxdb • Debian : $ sudo dpkg -i influxdb_latest_amd64.deb • CentOS : $ sudo rpm -ivh influxdb-latest-1.x86_64.rpm • Docker : $ docker run tutum/influxdb • Soon ARM and Windows
- 19. @zepouet#InfluxDB InfluxDB :: running • $ influxdb -config=/usr/local/etc/influxdb.conf • Ports • 8083 : UI • 8086 :API • 8090 : Cluster management raft • 8099 : Cluster management protobuf
- 20. @zepouet#InfluxDB InfluxDB :: design • Database (like in Mysql, Postgres…) • Time Series (kind of like tables with time, sequence number and columns) • A timeserie is composed by points or events (kinds of like rows) • Primary index is always time • Null values are not stored • You can have millions of series
- 21. @zepouet#InfluxDB InfluxDB :: security • Cluster admins • Database admins • Database users • Read permissions • only certains series • only queries with a column having a specific value (e.g. customer_id = 32) • Write permissions • only certains series • only columns having a specific value
- 22. @zepouet#InfluxDB InfluxDB :: create points curl -X POST -d '[{"name":"temp","columns": ["celsius"],"points":[[23]]}]' ‘http://localhost:8086/db/ mydb/series?u=root&p=root curl -G 'http://localhost:8086/db/mydb/series? u=root&p=root' --data-urlencode "q=select * from temp"
- 23. @zepouet#InfluxDB InfluxDB :: Pitfalls • Schemaless Warning • Data partinioning with one serie Time Name Host Metrics 3236765 cpu web0 78 3236765 disk_io web0 98344 3236765 load db1 5 3236765 eth_0 ldap0 8755
- 24. @zepouet#InfluxDB Time Name Host Metrics 3236765 disk_io web0 98344 3236766 disk_io web0 98354 3236767 disk_io web0 98224 3236768 disk_io web0 98994 Time Name Host Metrics 3236765 eth_0 ldap0 8755 3236766 eth_0 ldap0 8721 3236767 eth_0 ldap0 8734 3236768 eth_0 ldap0 8723 Time Name Host Metrics 3236765 cpu web0 78 3236766 cpu web0 77 3236767 cpu web0 79 3236768 cpu web0 76 Time Name Host Metrics 3236765 load db1 5 3236766 load db1 6 3236767 load db1 5 3236768 load db1 7
- 25. @zepouet#InfluxDB InfluxDB :: Why so many series? • To take advantage of the Storage engines • Points are indexed by time, not by any other columns • Tricks : easily work with grafana InfluxDB works best with large number of series with fewer columns in each one
- 26. @zepouet#InfluxDB :: Query Langage • select * from /.*/ limit 1 • select val1, val2 from serverA • select cpu from /server.*/ • select * from /.*/ where time > now() - 1h • select * from /.*/ where time > ‘2013-08-12 23:32:00’ • select * from /.*/ group by time(10m) • select count(val) from /.*/ group by time(10m) • select percentile(val, 95) from /.*/ group by time(10m) • select count(distinct(val)) from /.*/
- 27. @zepouet#InfluxDB :: Query Langage • DELETE • delete from response_times where time < now() - 1h • delete from /^stats.*/ where time < now() - 7d • drop series response_times • GROUP BY • select count(type) from events group by time(10m); • select count(type),type from events group by time(10m), type;
- 28. @zepouet#InfluxDB :: Visualize and summarize • Graphs • Last 10 minutes • Last 4 hours • Last 24 hours • Past week • Past month • All time
- 29. @zepouet#InfluxDB :: Merging :: Series • select count(type) from user_events merge admin_events group by time(10m) • select mean(value) from merge(/.*az.1.*.cpu/) group by time(1h)
- 30. @zepouet#InfluxDB :: Joining :: Series • select hosta.value + hostb.value from cpu_load as hosta inner join cpu_load as hostb where hosta.host = 'hosta.influxdb.orb' and hostb.host = ‘hostb.influxdb.org’; • select errors_per_minute.value / page_views_per_minute.value from errors_per_minute inner join page_views_per_minute
- 31. @zepouet#InfluxDB :: Naming Strategy :: 0.8 • Tag versus Value • Rule : <tagName>.<tagValue>.serieName • Examples : arduino.uno.shield.ethernet.sensor.dht11.temperature arduino.uno.shield.ethernet.sensor.dht11.temperature arduino.uno.shield.wifi.sensor.dht22.humidity arduino.uno.shield.wifi.sensor.dht22.humidity
- 32. @zepouet#InfluxDB :: Naming Strategy :: 0.9+ • Migration processus • Rule : serieName = serieName • Tag are defined into JSON and indexed
- 33. { "database" : "domotic", "points": [ { "name": "temperature_x", "tags": { "arduino": "uno", "shield": "wifi", "position": "indoor", "sensor": "dht22", }, "timestamp": "2015-03-28T14:50:00Z", "fields": { "celsius": 23.2, "farenheit": 192 } } ] }
- 35. @zepouet#InfluxDB :: Continuous Queries • select count(type) from events group by time(10m), type into events.count_per_type.10m DOWNSAMPLING
- 36. Next release
- 37. @zepouet#InfluxDB Soon in april 2015 • New model Clustering • Influx shell • Tags indexed • Backup
- 38. For Java Dev and Devops
- 39. @zepouet#InfluxDB Libraries • https://github.com/influxdb/influxdb-java Official java client • https://github.com/davidB/metrics-influxdb A reporter for metrics which announces measurements to an InfluxDB server. • https://github.com/vietj/vertx-influxdb-metrics Proof of concept of reporting to InfluxDB
- 40. @zepouet#InfluxDB davidb/metrics-influxdb Non official plugin from https://github.com/dropwizard/metrics