1. Setting Up a TIG Stack
for Your Testing

2. Me
 Jet Liu
Super Fund Of The Year 2019
Super Fund Of The Year 2019

3. Why Monitor?
• Speed up identification of issues
• Assist with repair of issues
• Reduce incidents
• Measure changes in performance &
availability
• Capacity planning
• Communicate changes

4. Monitoring

5. What to Monitor?
 Platform
 Databases
 Servers
 Infrastructure (physical or virtual)
 Storage etc.
 APP
 Logs
 Errors
 Metrics
 Services etc.
 User Experience
 Response time
 Latency
 Availability etc.

8. TIG (Telegraf, InfluxDB, Grafana)
 Open source monitoring & visualisation stack
 Collector -> Time Series Database -> Visualisation
 User friendly
 Very easy to set up locally or in a cluster
 IAC friendly
 CICD friendly
 Gaining popularity
“We start each exercise by reviewing the plan and projecting/sharing dashboards
in Grafana” – Slack’s process for approachable chaos engineering

9. Sample

10. How to Build TIG a Stack?
 Step 1: Install & Test InfluxDB
 Step 2: Install & Test Telegraf
 Step 3: Install Grafana
 Step 4: Build a dashboard
 Step 5: Install a custom plugin
 Step 6: Send custom data to influxdb
 Step 7: Configure & test new dashboard

11. Influxdb cli commands
 show databases
 create database tests
 drop database tests
 show measurements
 select count(*) from httpd limit 5
 select time, authFail from httpd where bind = ':8086' limit 1

12. REST API

13. One Way to Build a Dashboard

14. Building a dashboard – InfluxDB internals

15. Performance Testing

17. Custom

18. Export Dashboards
#!/usr/bin/env bash
Tnow=$(date +%Y-%m-%d-%S)
for dash in $(curl -sS -k -H "Authorization: Bearer $KEY" $HOST/api/search?query=& | jq -r '.[] | .uri'; do
curl -sS -k -H "Authorization: Bearer ${KEY}" "${HOST}/api/dashboards/${dash}" | jq 'del(.meta)' | jq
'.dashboard' > dashboards/$(echo ${dash}|sed 's,db/,,g')-$Tnow.json
done

19. kubernetes

20. Beyond a POC
 InfluxDb clustering – only available for InfluxDB Enterprise
 InfluxDb – backup & restore (natively supported)
 Set data retention policy for influxdb
 Capacity planning
 AD integration
 Authorisation & permission control
 Grafana – clustering is easy
 Grafana – backup & restore (rest api support)
 Alerting & notification

21. Questions?