From the talk Chris Chandler, Principal Engineer at T-Mobile, gave at Sensu Summit 2018: We all know and love sensu-client, but there are some times where deploying the client is either sub-optimal (e.g.: a box you don’t control) or impossible (e.g.: appliances, 3rd party services). I opted to tackle these kind of challenges Serverless-style, running my own code, then shipping the check outcome to the /results API. Some reasons for going this route include: - Sometimes you simply need to do more than the community plugins offer, which means you’re already writing custom code anyway. All you need to do is do a POST to the /results API, and boom. - I didn’t want to manage “bastion” clients that do proxy client style monitors, because I don’t live in a world where I can run Configuration Management tools against boxes at-will. Running in a PaaS/Serverless mode gave me more autonomy to iterate. - Checks can be more dynamic vs having to re-run Configuration Management to change sensu-client configs. For example, I can pull a list of service endpoints from Service Registry in real-time, then iterate that list in your code. Having to re-trigger Ansible/Chef/Puppet when endpoints light up/die would cause a lot of churn in highly dynamic environments.

1. T-Mobile Public
Clientless Monitoring
with Sensu
Chris Chandler
Principal Engineer
@cjchand

2. AKA: Look, ma! No Clients!

3. Part 2

4. So, wait… NO clients? How does that
work?

5. Simple: Proxy clients

6. Yes, technically still a “client”, but there’s
no client process/daemon
Come on, work with me here...

7. Proxy clients allow check results to be
shipped on behalf of anything where you
can’t or don’t want to run a client

8. Appliances,
Network gear,
SaaS / PaaS / FaaS,
etc.

9. Super High-Level Architecture

10. Clientless monitor is 100% technology
agnostic
Just needs to ship a result to Sensu’s API

11. Example Clientless API call
curl -s POST 'http://sensu.example.com:4567/results'
-H 'Content-Type: application/json'
-d '{ "source": "really-important-database",
"name": "check-my-db",
"output": "ZOMG! The DB is dead!",
"status": 2,
"occurrences": 1,
"refresh": 3600,
"handlers": [ "pagerduty", "email" ]
}'
Required
Optional

12. Just because we can do it,
why should we?

13. Unable/Unwilling to deploy a client
(You did see prior slides, right?)

14. Reuse tests as monitors
(e.g.: Postman Collections,
unit/functional tests)

15. Leverage Service Registry to
discover and monitor endpoints

16. Getting the “twofer” of monitoring +
service registry

17. Demo Overview
• docker-compose based
• 3x web servers
• Consul
• Sensu and Co.
• Python-based monitor

18. Containers and Communications

19. High-Level Monitoring Logic
1. Get all services in Consul via REST API
2. For each service, inspect nodes for IP and
ServicePort
3. If URI in metadata, add that to IP + Port
4. Poll HTTP endpoint
5. Send results to Sensu’s /results API
6. Delete any Sensu proxy clients that are no longer
registered in Consul

21. Consul Metadata
{ "service": {
"name": "web-hello-world",
<... content omitted …>
"meta": {
"environment": "production",
"handler": "pagerduty",
"pdkey": "1234567890",
"uri": "/health"
} <...>

22. Self-Service via Metadata
• Devs can use Consul’s “ServiceMeta” to specify:
• Dependencies
• Subdue
• Handlers
• TTL
• Routing/contact info (e.g.: PagerDuty API key)
• URI to check
• $your_idea_here

23. Processing Consul Metadata
try:
for k,v in node['ServiceMeta'].items():
payload[k] = v
except:
# ServiceMeta section is empty, ignore
pass

24. Some caveats...
• No keepalives with proxy clients
• Use TTL in check payload
• Best if run in PaaS/FaaS/Container platform
• Reachability of servers

26. Other Possibilities
• By passing a “type” k/v pair in the metadata,
you can specify what type of service this is:
• SNMP
• DBs
• $your_service_here

27. And now, on to the shameless self-
promotion!

28. Twitter: @cjchand
(Rarely updated) Blog:
https://cjchand.wordpress.com
Clone the Demo Code:
https://github.com/cjchand/sensu-clientless-demo