Learn how to run and manage several hundred databases across a company using two administrators and OVH services.

2. PRESENTEDBY
REX: BUILD EFFICIENT DBA
MONITORING
DASHBOARDS WITH OVH
WILFRIED ROSET
DB TEAM LEAD, OVH
ROOM 2
5.30 PM

3. IS THIS HAPPENING TO YOU?
"The website is slow
because of the DB!"
"The API is slow
because of the DB!"
ME
"We need a bigger DB
cluster, ASAP!"

4. LET’S STOP IT: USAGE TARGET
Allow tech people to be autonomous
Monitor our databases and set up alerts…
… using standard OVH solutions!
For any open-source RDBMS (PostgreSQL, MySQL, MariaDB…)
For any provider (OVH, on-premises servers, other cloud providers…)

5. LET’S STOP IT: INFRA TARGET
Metrics Logs
Grafana Graylog
Or on-premises servers,
other cloud providers…
DB DB DB
WWW
Customers
OVH DB team

6. BAD MONITO: THE OSTRICH

7. UGLY MONITO: UNEXPLOITABLE

8. GOOD MONITORING!

9. OUR INTERNAL DB PERIMETER
Total Volumetry: 20TB
# Clusters: ~60
# Applications: ~3k
# Users: ~700
# Databases: ~400
• 24 hours a day
• 2 DevOps & DBAs

10. HOW TO ISOLATE A SLOW QUERY
SELECT username
FROM customer
WHERE username LIKE '_wilfried%';

11. OVH INTERNAL DB INFRA
Master
ReplicaBackup
Prod
Dev Replica
Dev
Write
Read
Replication
Read/Write
24-
hour
restore
point
Archive

12. OVH’S "OLD" REQUEST PROCESS
Team DB
"Why is my DB
slow?"
"Can I have an
extract from my
logs?"

13. EASIER, BETTER, FASTER, STRONGER
Self-service, for complete autonomy
Meaningful KPIs
Capacity planning
Ownership and proactivity

14. PREREQUISITES
As-a-service
 OVH Logs
 OVH Metrics
 UI
Open source
 Data collection
 Reporting

15. DBMS CONFIGURATION
PostgreSQL
 Everything happens in postgresql.conf
 Format log to produce report
▪ log_line_prefix = '%t [%p]: [%l-1] db=%d,user=%u,app=%a,client=%h '
 Log slow queries
▪ log_min_duration_statement = '1000'
MySQL / MariaDB
 Everything happens in my.cnf
 Log slow queries
▪ slow_query_log = 1
▪ slow_query_log_file = /var/log/mysql/slow.log

16. GIVE MEANING TO LOGS
Pgbadger and pt-query-digest
 Process log files and produce fully readable reports
website
PostgreSQL MySQL

17. MONITORING: STEP 1/4
Team DB
Reporting
"Why is my DB
slow?"
"Can I have an
extract from my
logs?"

18. OVH LOGS (AKA LDP)
Everything starts with an OVH Logs account
 Control Panel > Cloud > Logs Data Platform
Add one stream per log file
 mysql-slow-queries
 pgsql-logs
LDP quickstart

19. LOGS’ DATA-GATHERING TOOLS
Open-source tools
 Logstash: data collection and log-parsing engine from Elastic
 Grok: pattern matching filter
Create and configure one logstash per log file
Grok is the trickiest funniest part
 Now available in the OVH Control Panel
OVH Logstash
for MySQL

20. EXAMPLE
# User@Host: app_user @ [1.2.3.4]
^# User@Host: %{WORD:query_user}s*@s*[%{IP:query_ip}?].*
{
"query_user": "app_user",
"query_ip": "1.2.3.4"
}

21. SENDING DATA
Useful information is available in the OVH Control Panel
 IP
 Port
 SSL cert
Filebeat
 /etc/filebeat /filebeat.yml
 Define which file to parse, multiline, and where to send it
Syslog
 Define a destination

22. CONFIGURATION FILEBEAT
# /etc/filebeat /filebeat.yml
output.logstash:
hosts: ["graX.logs.ovh.com:6514"]
ssl.certificate_authorities: ["/etc/filebeat/cert.pem"]
filebeat.prospectors:
- type: log
enabled: true
paths:
- /var/log/postgresql/postgresql.log

23. CONFIGURATION SYSLOG
destination d_pg_ldp {
tcp( "graX.logs.ovh.com",
port(6514),
ts_format("iso"),
keep-alive(yes),
so_keepalive(yes),
log-fifo-size(10000),
);
};

24. MONITORING: STEP 2/4
Team DB
Reporting
Logs
Syslog/Logstash
"Why is my DB
slow?"
"Can I have an
extract from my
logs?"

25. GRAYLOG

26. OVH METRICS
Order your Metrics account
 Control Panel > Cloud > Metrics Platform
Get your write token

27. COLLECT YOUR METRICS
There are several options for doing this:
 Homemade
 Collectl, Collectd, Statsd...
 Telegraf

28. TELEGRAF’S PLUG-INS
We’re using the following plug-ins:
 System-related
▪ cpu, disk, diskio, system, network ...
 Dbms
▪ PostgreSQL, MySQL
 Homemade
▪ Exec
There are a lot more plug-ins in telegraf
Telegraf @ Github

29. LET’S PUSH TO METRICS
# /etc/telegraf/telegraf.conf
[agent]
interval = "30s"
flush_interval = "30s"
[[outputs.influxdb]]
urls = ["https://influxdb.graXXX.metrics.ovh.net"]
timeout = "15s"
username = "telegraf"
password = "write.token.from.metrics.manager"

30. MONITORING: STEP 3/4
Team DB
Reporting
Logs
Metrics
Syslog/Logstash
Telegraf "Why is my DB
slow?"
"Can I have an
extract from my
logs?"

31. SYSTEM DASHBOARD

32. I/O DASHBOARD

33. DBMS DASHBOARD

34. EXPLOIT
Reports are fully readable
Graylog is really efficient for quick searches
 Document URLs can be shared
Grafana is a powerful tool for visualisation

35. MONITORING: STEP 4/4
Reporting
LogsSyslog/Logstash
Telegraf
"Why is my DB
slow?"
"Can I have an
extract from my
logs?"
Metrics

36. SLOW QUERIES COUNT/CUMUL. TIME

37. QUERY TYPES AND SLOWNESS

38. SPOT BUGS

39. REMEMBER THIS SLOW QUERY
SELECT username
FROM customer
WHERE username LIKE '_wilfried%';

40. WHAT WE LEARNED INTERNALLY
Choose the right tools for the job
 Grafana for dashboard (mix and match sources)
 Graylog for search
 Moving to Kibana
An infrastructure can be set up in less than a week
Three months down the road, we can expect…
 A widely-adopted gamechanger
 -30% slow queries
KPIs

41. A FEW WORDS ABOUT OUR OFFERS
Right now:
- OVH Cloud Databases: managed DB, up to 4GB RAM, no cluster
- … New Lab starting! For your high-end needs.
 Bare metal performance
 Clustered (minimum three hosts)
 Horizontal scale
 Will include LOGS and Metrics
 Starting with PgSQL
OVHLabs

42. PS: WE ARE HIRING !
Sysadmin Super Hero and DBA enthusiast
☀️• Bordeaux/Toulouse/Lyon
• ⛅ Rennes/Nantes
🍺• Roubaix
Job offer!