In this presentation Maikel shares and discusses the design and implementation considerations Jexia faced when creating Jexia’s real time API monitoring solution. http://www.jexia.com

2. Our Story

3. User Initiated
RESTful
DataApps & Services
Jexia Middleware layer
Jexia Services layer
PM2
Jexia High Level Topology
Jexia Cloud Infrastructure (CentOS/ GlusterFS/Cisco)
MongDB Cluster
Data
APP
Data
APP
Data
APP
Jexia Manager
n
Jexia User Management Jexia Real Time Services
RedisPostgreSQLNATS
messaging
NGINX (proxy / web servers layer)
NodeJS middleware
Jexia UI
(Angular)
Jexia Security Services Jexia
APP
Services
Number of
created
DataApps will be
large!
Jexia Infrastructure
Layer
Your RESTful APIs

4. Our Requirements
● No programming
● Real Time Insights
● Superior support for us when helping our users
● Flexible, scalable and secure
● Easy to use

5. In search of?

6. To
good
to be
true?

7. Beats Platform
Elastic
Libbeats : The libbeat library, written entirely in Golang,
offers the API that all Beats use to ship data to Elasticsearch.
Elastic : NoSQL DB Storage
Logstash : Collect, Enrich, and Transport logs
Kibana : Explore, Visualize data stored in Elastic
Logstash
All Logs
Jexia
Systems
Packet Beat
Top Beat
File Beat
Libbeats
Kibana

8. GROK and
Filters
Logstash Instance
Syslog
Elasticsearch
Input
Plugin
Filter
Plugin
Output
Plugin
Example Filters:
Csv, xml, Date, IP and GROK
Files

9. Install
Use
and
Evaluation

10. Our
learning
points
● Meets our requirements and by default already
powerful (Real time works!)
● Power of GROK
● ELK uses Java
● Very flexible architecture (e.g. try Grafana on top!)
But is it really good enough for us? (and for you!)

11. New
=
Better
?!

12. Time-
Series
Databases
● Data is accessed in sequence by time and can
be processed and analyzed in chronological
order.
● Time-Series Databases are optimized for time
series data (storing and queries).
● Typical use-cases: Events! (e.g. IoT, sensor
data, trading data, GPS, logistics, data center
monitoring, network (API calls!) etc)

13. What
Is
InfluxDB
● Written in GOLang
● SQL query language with great aggregation functions like:
Percentile, Distinct, Median
● Data can be tagged (so very flexible querying possible)
● Serving queries in real-time (continuous queries). Incoming
data is processed immediately at arrival.
● Clustering is built in. Nothing else is needed to make data
highly available.
● HTTP API for putting data into InfluxDB. But the POST is NO
JSON but a text based format (line protocol - [key] [fields]
[timestamp])
●

14. Beats Platform
InfluxDBLogstash
Logging
Some
RESTful
API traffic
Packet Beat
Top Beat
File Beat
Libbeats
Grafana
Jexia Docker
image

15. Jexia= Solutions && ((Elastic) || (Influx))
Challenge your thinking

16. Questions
Discussion

17. Is it fast?
● Nightly benchmarks: https://benchmarks.elastic.co/index.html
● Define what speed you need! (Indexing, Searching, Analytics etc)
● StackExange: 3 elasticsearch servers ,RAM: 196GB , 7 % CPU usage(Peak:
20%), 34M searches /day ,440 gb index size (see: http://stackexchange.
com/performance )