2. Hello
my name is:
Lee Melvin
Mentor Graphics
Technical Architect - Storage
Hello
my name is:
Shawn Roberts
Mentor Graphics
Technical Architect - Clients
3. Mentor Graphics® provides
software and hardware design
solutions that enable companies
to develop better electronic
products faster and more cost-
effectively.
4. How I got started with Splunk
• Dual vendor network-attached storage strategy
• 100+ filers across 30+ sites, used by 10+ divisions
• Needed ways to
• Provide per-division summary reporting, accessible to engineers on demand
• Implement performance service level objectives
• Provide storage administrators with actionable issues, not just graphs and
metrics
• Provide reporting that spans vendors
• Vendor reporting tools expensive and inadequate for these needs
• Previously, layered a fragile and manual scripting process on top
• Initial focus on capacity and performance analysis
Lee
6. How I got started with Splunk
• We had ~8,500 systems around the world that were not being
monitored
• No visibility into what could be causing occasional crashes
• No ability to:
• Add another agent
• “increase the IT tax”
• Negatively impact performance
• Track changes over time
• People were asking questions…
Shawn
16. Doesn’t add another agent
Doesn’t “increase the IT tax”
No impact to performance
Track changes over time
Shawn
17. Tough client reports now possible
Boot Time
Average aka What’s
normal?
Last 30 vs 365
Top/Bottom 10
Error Events
Top 10
Errors by category
Startup impacting
Application crashes
!
Shawn
19. Storage reporting enhanced!
• Tagging filesystems by division, product group, performance tier,
or other abstract (via Splunk lookup table)
• Custom technology add-on to ingest data from Hitachi HNAS
• Capacity/Performance reports that span both NetApp and Hitachi
• Division summary and per-filesystem views
• Actionable items for storage administrators
• Outliers view – slow, busy, near-full
• Stale data reporting – big chunks of unused data
Lee
24. No good deed, etc…what’s next?
• Lookup/compare by:
• Location
• AD OU
• Manager
• Type
• Model
• Manufacturer
Shawn
25. Client dashboard
• Measurements
• Make, Model, CPU, Memory, Disks,
Encryption status, etc.
• Age of Windows installation
• Most common errors
• Last N errors
• SCCM Data
• Installed software
• Update status
• Changes!
• Hardware
• Software
• AV Data
• Versions
• Activity
Shawn
26. Client dashboard with Splunk
• Measurements
• Make, Model, CPU, Memory, Disks,
Encryption status, etc.
• Age of Windows installation
• Most common errors
• Last N errors
• SCCM Data
• Installed software
• Update status
• Changes!
• Hardware
• Software
• AV Data
• Versions
• ActivityX vs Normal
X over Time
Shawn
27. Storage reporting
• Reports
• Filer configuration auditing
• Geographic visualization
• Long term (5-year) reports leveraging
summary indexes
• Matching workloads to appropriate
performance tier
• Alerting
• Alert filesystem owners on near-
capacity using lookup table
• Infrastructure
• Web frontend to edit tags in
lookup table
• Division-specific view roles to
limit access/confusion
• Ingest additional HDS metrics
• Ingest Symantec NetBackup logs
• Round the learning curve on
data models, simplify
Lee
28. Lessons learned
• You have data in more places than you might think.
• You don’t always need a huge license purchase to answer hard
questions.
• Use the provided apps for inspiration (and for cut/paste).
Don’t treat them as an end state.
Shawn
Lee
I went shopping for a “Client Monitoring” tool. There aren’t any options in this space.
We have a great working relationship with Microsoft. I asked them for guidance.
Here’s our conversation.
Now that I could get the logs in one place, it was much easier to go shopping.
I happened to be attending OSCON around the same time…
I talked with Splunk.
With Windows Event Forwarding and Splunk, I’m able to put together a fairly simple solution.
All green.
Also, we are able to perform on demand reporting to solve tough client problems like this one.
We’re a little bruised from the journey, but we’re in a good place.
The next project I’ve been tasked with is creating a Client Dashboard to help our help desk and desktop administrators. Without Splunk, I can do a lot with DB queries and HTML.
With Splunk I can call out what’s important in all that data. This is really useful to our admins who don’t have a lot of familiarity with each system they work with.