Slides from my presentation given at 4developers 2020 conference

1. ^
KEEP
CALM
AND
SERILOG
ELASTICSEARCH
KIBANA
ON .NET CORE

2. Maciej Szymczyk
• Big Data Engineer / Software Developer
• Professional Soldier / National Cyber Security Centre
• Teacher at Military University of Technology
• Blogger – Wiadro Danych – https://wiadrodanych.pl
• Amateur Cyclist & Triathlete ( ½ Ironman Finisher)

3. My first webapp
• MVC – All logic in Controllers
• Zero logging
• Zero metrics
• Asking customers what to do to reproduce bugs

4. Complex apps
• Microservices
• Many instances
• Polyglot persistence
• Web servers Apache/Nginx/IIS
• Many types of clients (Android / iOS / Web)
• Docker/Docker Swarm/Kubernetes
• External dependencies (GCM / APNS)

5. What can we log?
Usage
Audit
Statistics
Performance Errors Debug

6. When/What can we log?
• Messages
• Requests
• Body
• Query String Values
• Headers
• Methods
• Parameters
• User Id/IP
• Client type/version
• Service/Machine Name
• Correlation/Trace Id (Guid)

7. How much should I log?

8. What should not be logged?

9. Where should I put my logs?
• Console
• Log Files
• Relational Database (MSSQL)
• Elasticsearch / Splunk / Loki
• HDFS / S3 / ADLS
LET’S USE RDB
OK, YOU WILL
BUILD
DASHBOARDS

10. Let’s use RDB and build graphs and dashboards!

11. When your only tool is a hammer,
everything looks like a nail

12. Elasticsearch
• Full text search engine based on Apache Lucene
• REST API
• Open Source
• Scalable
• Multiple ways of injestion
• REST
• Beats (Filebeat, Metricbeat, Heartbeat, etc)
• Logstash

13. Elastic Stack usecase
• Logs/metrics aggregation
• Search engine
• Realtime Business Analytics
• SIEM – Security Information and Event Management

14. RBD Elasticsearch
Database Index
Table Type* / Index
Row Document
Column Field
Schema Mapping
* Deprecated since 7.x

15. Elastic Stack Architecture
Beats
Logstash
Elasticsearch
Kibana
HTTP

16. Kibana
• Visualisation
• Dashboards
• Analytics
• Uptime
• Infrastructure
• SIEM

17. Logstash

18. Beats
• Filebeat – logs
• Metricbeat – metrics
• Packetbeat – netflow
• Winlogbeat – Windows events/logs
• Auditbeat – security (auditd, files integrity, logins,
proceses)
• Heartbeat – uptime
• Functionbeat – cloud data collector (FaaS)

19. Easy way
HTTP

20. A little bit more complex
HTTP

22. Data Temperature + Retention Policy
APP HOT
Logs
WARM
COLD
Metrics

23. Scalability
• Master nodes
• Data nodes
• Coordinator nodes
• Ingest nodes
• ML nodes

24. Minimal recommended cluster
C C
MM
D D
M

25. Shard
• Part of Elasticsearch index
• Shards are distributed across cluster
• You can’t change numer of shards after creating index
• Each shard can has 0 - n replicas
Index
SHARD 1
SHARD N
REPLICA N
REPLICA 1
...
...

26. Indices, shards, replicas
26
Posts
SHARD 1
Node 2Node 1 Node 3 Node 4
SHARD 2
REPLICA 2
REPLICA 1SHARD 3
REPLICA 3

27. Indices, shards, replicas
27
Posts
SHARD 1
Node 2Node 1 Node 3 Node 4
SHARD 2
REPLICA 2
REPLICA 1SHARD 3
REPLICA 3

28. Indices, shards, replicas
28
Posts
SHARD 1
Node 2Node 1 Node 3 Node 4
SHARD 2 REPLICA 2
REPLICA 1SHARD 3
REPLICA 3

29. Elasticsearch licence

30. How to try Elasticsearch?
Visit
https://wiadrodanych.pl/docker/docker-na-pomoc/
for Elasticsearch + Kibana Docker Compose YAML

31. Demo 1

32. Why Serilog?
• Structured logging
• Easy to use
• Multiple sinks
• Integration with .NET Core

33. Demo 2

34. Too lazy to…
• Implement wrapper…
• …and config for it…
• …and convince other to use it.
• Replace existing ILogger<> usages
• Try catch every piece of code for errors
• Use Stopwatch all the time
• Look for username in logs from controller

35. .NET Core Middlewares

36. TraceId
https://devblogs.microsoft.com/aspnet/improvements-in-net-core-3-0-for-troubleshooting-and-monitoring-distributed-apps/

37. Demo 3

38. Application Performance Monitor (APM)
• Distributed tracing
• APM agent streams
application
performance metrics
to APM server
• Detect anomalous
response Times
• Java, .NET, Node.js,
Django, Flask, Rails,
Rack, RUM – JS, Go

39. Elasticsearch takeaways
• Do not exceed 64 GB of RAM per data node (31 GB JVM)
• SSD > HDD
• avoid NAS/SAN
• Use ECS (Elastic Common Schema)
• Use ILM (Index Lifecycle Management)
• Hot/Warm/Cold Architecture
• define rules over when to perform certain actions

40. Thx!
https://wiadrodanych.pl/4dev <= code
https://wiadrodanych.pl/elastic <= course