How we use the ELK stack for aggregating logs at Zerynth (2019).

1. Monitorare microservizi
con Elasticsearch
Antonio Pitasi
Software Developer @ Zerynth
a.pitasi@zerynth.com
@pitasiantonio
#MeetupElastic
2019

2. Before microservices
API Gateway
Cart
Database
E-Commerce App
Product Catalog Search engine

3. API Gateway
E-Commerce App
Microservices!
Cart
Database
Product Catalog Search engine
Database Database

4. Pros
- Fault isolation
- Technology commitment
- Application scaling
- Continuous Development and Deployment

5. Unix philosophy
Write programs that do one thing and do it well.
Write programs to work together.
“

6. Zerynth Introduction - Confidential | © 2018 www.zerynth.com
Zerynth is the middleware for IoT and Industry 4.0
Zerynth is a software toolkit that allows programming in Python or hybrid C/Python the most popular
32-bit microcontrollers, and connect them to the leading Cloud Infrastructures.

7. Zerynth Introduction - Confidential | © 2018 www.zerynth.com
Zerynth speeds up the IoT development
Zerynth provides developers with an ecosystem of software tools to program microcontrollers in
Python or C and connect them to the Cloud.

8. Zerynth Introduction - Confidential | © 2018 www.zerynth.com
Connecting ESP32 to AWS using Zerynth
MQTT client connecting to AWS IoT
service, sending data and receiving
Shadow updates.
Only 40 lines of easy and readable
Python.

9. www.thingsoninternet.it
TOI Introduction - Confidential | © 2018
TOI provides 4ZeroPlatform: a plug-and-play data gathering, processing, and reporting solution
for small and large enterprises who need to achieve full visibility and optimization of Industrial Processes.
4ZEROPLATFORM
A machine-to cloud-interface that can be plugged
into old and modern machines.
A cloud-based device management service for organizing,
monitoring, and remotely updating connected devices at scale.
More info at: www.thingsoninternet.it/4zeroplatform/

10. www.thingsoninternet.it
TOI Introduction - Confidential | © 2018
USE CASES
New and legacy machines in different
manufacturing sectors.
• Real-time sensor data acquisition and
processing of anomalies and deviations
• Integrated in cloud or on-premises
solutions
4.0 OUTCOMES
• Efficiency: real-time production data and
OEE measuring
• Maintenance: failures detection
improvement
• Productivity: integration with ERP-MES
tools
Manufacturing Industry Process Industry
Biogas production monitoring for efficiency
improvement.
• Pressure, Temperature, Humidity, CH4
and
O2 data acquisition and local computing
• LoRa connection to local servers
4.0 OUTCOMES
• Efficiency: less time and more precision
for
wells regulation
• Visibility: awareness of unknown biogas
cyclic dynamics
Warehousing
Industrial refrigerators monitoring for
efficiency and regulatory purposes.
• Temperature, humidity, doors and energy
data
for real-time monitoring, recording and
COP
• Wi-fi and GSM connectivity
4.0 OUTCOMES
• Efficiency: reduction of Maintenance
costs
and energy
• Quality: cold chain monitoring meeting
HACCP regulation

11. www.thingsoninternet.it
TOI Introduction - Confidential | © 2018
Wi-Ref - Features
Monitoring, Reporting, alarms configuration

12. Tons of microservices == tons of microproblems
API Gateway
E-Commerce App
Cart
Database
Product Catalog Search engine
Database Database

13. Tons of microservices == tons of microproblems
API Gateway
E-Commerce App
Cart
Database
Product Catalog Search engine
Database Database
X

14. import structlog
log = wrap_logger(
logging.getLogger(__name__),
processors=[
filter_by_level,
add_timestamp,
JSONRenderer(indent=1)
]
)
log.warning("Oops :(", code=42)
{
"event": "Oops :(",
"level": "warning",
"code": 42,
"timestamp": "2019-04-30T10:24:54.001346Z"
}

15. Elastic to the rescue
Data
Collection
Data
Aggregation
& Processing
Indexing
& Storage
Analysis
& Visualization
beats logstash elasticsearch kibana

16. Extensions
- SearchGuard
- Logtrail
- Elastalert
- Curator

17. SearchGuard

18. Logtrail

20. Curator
actions:
1:
action: delete_indices
filters:
- filtertype: pattern
kind: prefix
value: filebeat-
- filtertype: age
source: creation_date
direction: older
unit: days
unit_count: 30

21. Antonio Pitasi
Software Developer @ Zerynth
a.pitasi@zerynth.com
@pitasiantonio
#MeetupElastic
2019