4. Life Without Splunk Service Desk Application Support Systems Administrator Application Developer Application Developer Database Administrator Log call. The console says everything is green. App monitoring tools don’t show anything either. Call the developer. Stop working on new code to troubleshoot. Need production logs! Stop what they’re doing to identify and gather production logs for developer. Manual investigation establishes not application problem. DBA analyzes the logs which points to corrupted database files. Escalate. Escalate. Escalate. Respond. Escalate. Now what?
5. Life With Splunk Service Desk Trouble Ticket Search on IP address shows related Web session and User ID “ 192.168.169.100” Last 60 minutes 192.168.169.100 Search at same time reveals database error due to corrupted files Search for failure or error across entire IT Last 2 minutes failure OR error Search on corruption in the db logs shows that an index file has been corrupted Search for corruption in db logs Last 1 minute host=db.domain.com source=*db.log corrupt* Setup monitoring and alerting for db file corruption Set up Monitoring and Alerting Last hour host=db.domain.com source=*db.log corrupt*
7. Monitor and Alert in Real Time 2. Evaluate alerting condition 1. Get data Scheduled search Real-time search Alert Condition 3. Execute actions RSS Email SNMP Script Yes Noop No
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21. August 15, 2011 Questions? Ledion Bitincka, Search and Alerting Team