ROMA NOVIKOV, BAQ, "Prometheus + grafana based monitoring"

1. © 2017 Percona1
Roma Novikov
Prometheus + Grafana based Monitoring
Director of Platform Engineering
BAQ CONFERENCE 2019
24 March, 2019

2. © 2017 Percona2
About me
Roma Novikov
▪Percona - Director of Platform Engineering (2+ years)
▪Since 2001: Web developer -> Lead/Architect -> Manager -> CTO
▪Interests
• web, highload, monitoring and observability.

3. © 2017 Percona3
Agenda

4. © 2017 Percona4
Agenda
▪Why do you need Monitoring

5. © 2017 Percona5
Agenda
▪Why do you need Monitoring

6. © 2017 Percona6
Agenda
▪Why do you need Monitoring
▪ Tools

7. © 2017 Percona7
Agenda
▪Why do you need Monitoring
▪ Tools
▪Methods

8. © 2017 Percona8
Agenda
▪Why do you need Monitoring
▪ Tools
▪Methods
▪Prometheus + Grafana

9. © 2017 Percona9
Agenda
▪Why do you need Monitoring
▪ Tools
▪Methods
▪Prometheus + Grafana
▪Percona’s solution

10. © 2017 Percona10
Monitoring

11. © 2017 Percona11
Why do you need The Monitoring?

12. © 2017 Percona12
Why do you need The Monitoring?
Before: Get to know stakeholders and their needs

13. © 2017 Percona13
Performance Stakeholders

14. © 2017 Percona14
Performance Stakeholders
▪Internal

15. © 2017 Percona15
Performance Stakeholders
▪Internal
• DEV

16. © 2017 Percona16
Performance Stakeholders
▪Internal
• DEV
• QA

17. © 2017 Percona17
Performance Stakeholders
▪Internal
• DEV
• QA
• OPS

18. © 2017 Percona18
Performance Stakeholders
▪Internal
• DEV
• QA
• OPS
▪External

19. © 2017 Percona19
Performance Stakeholders
▪Internal
• DEV
• QA
• OPS
▪External
• Managers

20. © 2017 Percona20
Performance Stakeholders
▪Internal
• DEV
• QA
• OPS
▪External
• Managers
• C-Level

21. © 2017 Percona21
Monitoring tools

22. © 2017 Percona22
Monitoring tools

23. © 2017 Percona23
Monitoring tools
● top

24. © 2017 Percona24
Monitoring tools
● top
● ps aux

25. © 2017 Percona25
Monitoring tools
● top
● ps aux
● SHOW PROCESSLIST

26. © 2017 Percona26
Monitoring tools
● top
● ps aux
● SHOW PROCESSLIST
● Why I need something new?

27. © 2017 Percona27
Monitoring tools
● top
● ps aux
● SHOW PROCESSLIST
● Why I need something new?
● Is everything “нормально/fine”

28. © 2017 Percona28
Monitoring tools
● top
● ps aux
● SHOW PROCESSLIST
● Why I need something new?
● Is everything “нормально/fine”

29. © 2017 Percona29
Monitoring tools
● top
● ps aux
● SHOW PROCESSLIST
● Why I need something new?
● Is everything “нормально/fine”
● What about data from “yesterday”?

30. © 2017 Percona30
Monitoring tools
● top
● ps aux
● SHOW PROCESSLIST
● Why I need something new?
● Is everything “нормально/fine”
● What about data from “yesterday”?
● What's going on now with another part of the system? How it's
affected?

31. © 2017 Percona31
Methods to use

32. © 2017 Percona32
Methods to use - U.S.E.?
USE
▪ For every resource, check utilization, saturation, and errors.
• utilization: the average time that the resource was busy servicing work
• saturation: the degree to which the resource has extra work which it can't service,
often queued
• errors: the count of error events
Brendan D. Gregg : http://www.brendangregg.com/usemethod.html

33. © 2017 Percona33
What to use - R.E.D.?
RED
▪ The RED Method defines the three key metrics you should measure for every
microservice in your architecture. Those metrics are:
• (Request) Rate - the number of requests, per second, you services are serving.
• (Request) Errors - the number of failed requests per second.
• (Request) Duration - distributions of the amount of time each request takes.
Tom Wilkie https://www.weave.works/blog/the-red-method-key-metrics-for-microservices-architecture/

34. © 2017 Percona34
Where to apply?

35. © 2017 Percona35
Where to apply?
● U.S.E. - Resource (OS, DB … )

36. © 2017 Percona36
Where to apply?
● U.S.E. - Resource (OS, DB … )
● R.E.D. - Application / Service

37. © 2017 Percona37
What should I use?

38. © 2017 Percona38
What/Why should I use? - Commercial
● “embedded” From your provider
○ AWS
○ DigitalOcean monitoring
○ ...

39. © 2017 Percona39
What/Why should I use? - Commercial
● “embedded” From your provider
○ AWS
○ DigitalOcean monitoring
○ ...
● Limitations
○ features
○ coverage

40. © 2017 Percona40
What/Why should I use? - Commercial
● SAAS
○ VividCortex
○ DataDog
○ NewRelic
○ ….

41. © 2017 Percona41
What/Why should I use? - Commercial
● SAAS
○ VividCortex
○ DataDog
○ NewRelic
○ ….
● Limitations
○ $
○ you send data outside your network

42. © 2017 Percona42
What/Why should I use? - Open Source
● OpenSource
○ Zabbix
○ Cacti
○ Nagios
○ ...

43. © 2017 Percona43
What/Why should I use? - Open Source
● OpenSource
○ Zabbix
○ Cacti
○ Nagios
○ ...
● Limitations
○ Open Source VS Free
○ Visualization

44. © 2017 Percona44
Information System architecture

45. © 2017 Percona45
Information System architecture
● (Old style VS Modern )* Hybrid Cloud

46. © 2017 Percona46
Information System architecture
● (Old style VS Modern )* Hybrid Cloud

47. © 2017 Percona47
What/Why should I use? - Cloud Native
● Prometheus
● Grafana*

48. © 2017 Percona48
Prometheus

49. © 2017 Percona49
Prometheus - Architecture

50. © 2017 Percona50
Prometheus - Architecture

51. © 2017 Percona51
Prometheus - Advantages

52. © 2017 Percona52
Prometheus - Advantages
● Multi-dimensional data model

53. © 2017 Percona53
Prometheus - Advantages
● Multi-dimensional data model
○ Stores everything

54. © 2017 Percona54
Prometheus - Advantages
● Multi-dimensional data model
○ Stores everything
○ Good compression

55. © 2017 Percona55
Prometheus - Advantages
● Multi-dimensional data model
○ Stores everything
○ Good compression
● PromQL

56. © 2017 Percona56
Prometheus - Advantages
● Multi-dimensional data model
○ Stores everything
○ Good compression
● PromQL
● Single Server Node

57. © 2017 Percona57
Prometheus - Advantages
● Multi-dimensional data model
○ Stores everything
○ Good compression
● PromQL
● Single Server Node
● Collection via pull over HTTP

58. © 2017 Percona58
Prometheus - Advantages
● Multi-dimensional data model
○ Stores everything
○ Good compression
● PromQL
● Single Server Node
● Collection via pull over HTTP
● Simple targets discovery (or static configuration)

59. © 2017 Percona59
Prometheus - Exporter

60. © 2017 Percona60
Prometheus - Exporter

61. © 2017 Percona61
Prometheus - Exporter
EXPORTER

62. © 2017 Percona62
Prometheus - Exporter
EXPORTER

63. © 2017 Percona63
Prometheus - Exporter

64. © 2017 Percona64
Prometheus - Exporter
● Usage
○ 150 + exporters mentioned on the website
○ reality - much more

65. © 2017 Percona65
Prometheus - Exporter
● Usage
○ 150 + exporters mentioned on the website
○ reality - much more
● How to Extend?
○ just include instrumentation lib into your project
■ Go
■ Java or Scala
■ Python
■ Ruby…..

66. © 2017 Percona66
Prometheus - Exposition Format

67. © 2017 Percona67
Prometheus - Exposition Format
● Exporter format
○ Simple
○ Text-based

68. © 2017 Percona68
Prometheus - Exposition Format
● Exporter format
○ Simple
○ Text-based
● Example : http://localhost:port/metrics
○ # HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1027 1395066363000
http_requests_total{method="post",code="400"} 3 1395066363000

69. © 2017 Percona69
Grafana

70. © 2017 Percona70
Grafana
● The analytics platform for all your metrics
○ Grafana allows you to query, visualize, alert on and understand your
metrics no matter where they are stored. Create, explore, and share
dashboards with your team and foster a data driven culture.
https://grafana.com/grafana

71. © 2017 Percona71
Grafana - Advantages

72. © 2017 Percona72
Grafana - Advantages
● 30+ Data Sources

73. © 2017 Percona73
Grafana - Advantages
● 30+ Data Sources
● 50+ Panel Types

74. © 2017 Percona74
Grafana - Advantages
● 30+ Data Sources
● 50+ Panel Types
● XXX Dashboards to use

75. © 2017 Percona75
Grafana - Advantages
● 30+ Data Sources
● 50+ Panel Types
● XXX Dashboards to use
● DIY Dashboards

76. © 2017 Percona76
Grafana - Advantages
● 30+ Data Sources
● 50+ Panel Types
● XXX Dashboards to use
● DIY Dashboards
● Alerts / Notifications

77. © 2017 Percona77
Grafana - Advantages
● 30+ Data Sources
● 50+ Panel Types
● XXX Dashboards to use
● DIY Dashboards
● Alerts / Notifications
● Annotations

78. © 2017 Percona78
Grafana - Advantages
● 30+ Data Sources
● 50+ Panel Types
● XXX Dashboards to use
● DIY Dashboards
● Alerts / Notifications
● Annotations
● You can show it to Manager. There is no shame

79. © 2017 Percona79
Prometheus + Grafana

80. © 2017 Percona80
Make them work together - Default

81. © 2017 Percona81
Make them work together - Default
1. Start Exporter
2. Configure Exporter
3. Start Prometheus
4. Configure Prometheus
5. Edit config and Add Exporter details
6. Reload Prometheus Config
7. Start Grafana
8. Set up Grafana and Data source
9. Create Dashboard

82. © 2017 Percona82
Make them work together - simplified

83. © 2017 Percona83
Make them work together - simplified
1. install PMM Client
2. Install PMM Server (docker)
3. pmm-admin add mysql <MYSQL_PARETERS>
4. Done

84. © 2017 Percona84
Percona Monitoring and Management

85. © 2017 Percona85
About Percona
Unbiased Open Source Database Experts

86. © 2017 Percona86
What is PMM and Why it was created

87. © 2017 Percona87
What is PMM and Why it was created
▪Our own

88. © 2017 Percona88
What is PMM and Why it was created
▪Our own
▪On premise / host your own

89. © 2017 Percona89
What is PMM and Why it was created
▪Our own
▪On premise / host your own
▪Free and Open source

90. © 2017 Percona90
Architecture

91. © 2017 Percona91
Architecture
Main Components
● Prometheus
● Grafana
● Percona Dashboards
● Consul
● Query Analytics
● CLI tool
● prometheus exporters

92. © 2017 Percona92

93. © 2017 Percona93
PMM Dashboards

94. © 2017 Percona94
PMM Dashboards
● 40+ Dashboards (DB and Host metrics)
● Created and used by Percona’s Experts

95. © 2017 Percona95
Technical details

96. © 2017 Percona96
Technical details
Distribution methods
• Docker (server)
• AMI (server)
• OVA (server)
• Linux packages (client)

97. © 2017 Percona97
Technical details
Distribution methods
• Docker (server)
• AMI (server)
• OVA (server)
• Linux packages (client)
What is supported
▪MySQL Databases (MySQL, MariaDB, Percona Server, AWS/Aurora)
▪NoSQL Databases(MongoDB, Percona Server for MongoDB)
▪OS (Linux,..)

98. © 2017 Percona98
Product Demo
https://pmmdemo.percona.com

99. © 2017 Percona99
Crosserver

100. © 2017 Percona100
System Overview

101. © 2017 Percona101
MySQL Overview

102. © 2017 Percona102
MySQL Overview VS Query Analytics

103. © 2017 Percona103
How to Extend?

104. © 2017 Percona104
How to Extend?

105. © 2017 Percona105
How to Extend?
● Create your Dashboards

106. © 2017 Percona106
How to Extend?
● Create your Dashboards
● Add existed exporters or write your own
○ Add them as External exporter with one simple command
○ pmm-admin add external:service job_name [instance]
--service-port=PORT_NUMBER

107. © 2017 Percona107
Join Us
● DevOps/QA Engineer
● Database QA Engineer
● Golang and K8s SE
● Junior MySQL DBA
// All Remote

108. © 2017 Percona108
Thanks!
QA

109. DATABASE PERFORMANCE
MATTERS
Database Performance MattersDatabase Performance MattersDatabase Performance MattersDatabase Performance MattersDatabase Performance Matters