The document discusses best practices for building a large-scale SolarWinds installation. It emphasizes the importance of having an accurate device inventory, ensuring proper network connectivity and permissions, designing for scalability including separating database and polling functions, and establishing processes for device lifecycle management and troubleshooting issues. Common challenges include event correlation, determining scalability limits, lacking a database administrator, and inconsistent time zones.

1. Building a Large-Scale SolarWinds® Installation
With Leon Adato, Monitoring Architect at Cardinal Health®
© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

2. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Enterprise – Bigger Than a Breadbox
Leon’s not-so-humble-opinion:
» Over 2,000 devices
» More than 20 locations
» 10,000 elements

3. Most Projects Fail at the Top
The Real OSI Model
Politics
Legal/Compliance
Finance
Application
Presentation
Session
Transport
Network
Data© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

4. REPLACE TITLE CANT USE SONG LYRICS
» A complete, accurate, meaningful inventory.
» Network connectivity to all devices part 1: ping
» Network connectivity part 2: Firewall
» A domain or local account that has permission to pull WMI/RPC
» SNMP enabled with a known RO string on all devices
– And your polling engines PERMITTED to SNMP poll
© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

5. SNMP WTF?
© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

6. ALERTS: The Four Questions
» What will you do when you get this alert?
» Do you have a documented procedure?
» Can you make this problem happen on purpose (for testing)
» What will happen (what is the cost) if you don’t get this alert?
© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

7. Standard Enterprise Design
© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

8. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Kick It Up a Notch

9. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Portocalypse Now*
For SNMP and RPC, you will need:
» ICMP bi-directional
» TCP 135
» TCP 139
» TCP 445
» TCP 161
» UDP 137
» UDP 138
» For WMI? All ports over 1024, or go read “WMI Portocalypse” on thwack
*Tip of the Thwack-cap to aLTeReGo

10. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
DMZ? Make sure these are open
» ICMP bi-directionally
– And that DNS works in both directions
» TCP 17777
» TCP 17778
» TCP 80 (or whatever your SolarWinds portal runs on)
» TCP 1433 (or whatever your database listens on)
– Named instances will not work here.

11. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Account Permissions
» RPC – local or domain user that is a member of:
– Performance Monitor Users group
– Distributed COM users group
– And has the “Allow logon locally” permission
– (NOT a member of allow login through remote desktop)
» WMI – local or domain user that is a member of:
– Performance Monitor Users group
– Distributed COM users group
– Remote desktop users group
– Has “Allow logon locally” permission
– Has been added to the “root” and “CIMv2” branches of the WMI settings.
» Or just give it local admin already!

12. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Security Hates Me*
So you can’t get them to permit WMI
» Most SAM items that use WMI can be obtained via RPC
– Most processes can be monitored via SNMP with CPU and RAM stats
– Windows eventlog can be gathered via RPC
– Windows process and/or services
– Perfmon has always used RPC
What you are giving up (ie: How To Make Your Case):
» Virtual Mount Points
» Restarting a service as an alert action
» Realtime process/service explorer
» Windows® script monitoring (VBScript™ or PowerShell®)
» WMI Monitors (duh!)
*Or: “I have to monitor through a NAT-ed VPN”

13. Server Spec’s
» Separate the Database and primary poller.
» Database is physical. “Need More Power” should be your request. Mine has 12 CPU
and 128Gb RAM.
» Pollers can be virtual. I tend to run with 8 CPU, 12Gb ram to start.
» Disk is a big deal. You want lots of spindles. You want RAID 10 or you want the SAN
team to tell you they can load your entire database into the memory of the storage
array.
» Everything has to be in the same time zone.
» Primary poller and database must be in the same location
© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

14. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Element Counts – The Real story
» Sure, you have elements (~10,000)*
– Ping-able IP’s
– Interfaces
– Disks
» But you also have UnDP’s
» And SAM components
» And the number of DOWN items is going to affect how much you can handle
» Upshot: There is no formula.
*Ultimate limit is ~110,000 per Solarwinds instance

15. Know Thy Pollers!
© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

16. Processes You Have to Nail Down
» Device Lifecycle
– Who, how, where, when devices are added
– Ditto for elements
– Ditto for SAM items
» Devices (and volumes, and interfaces) missing key custom property information
» Decom devices
» Down Devices
» Devices not SNMP polling
» Applications in “unknown” status
» Duplicate Nodes
» Bad application report
© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

17. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Device Down Report

18. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Applications with Problems

19. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Handy Documents to Have on Hand
SolarWinds Run Book
» Overall design
» Accounts used for installation, connectivity (Poller to db) etc.
» Additional software installed (Perl®, ImageMagik®, PuTTY™, etc)
» How to install Windows patches
» How to install SolarWinds patches
» Common troubleshooting techniques
» Common validation/error-checking techniques
» How to open a ticket with SolarWinds (with your account ID)

20. Gotchas
» Event correlation
» Where does scalability end?
» No DBA? That’s gonna hurt.
» Time zone
© 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.

21. © 2013 SOLARWINDS WORLDWIDE, LLC. ALL RIGHTS RESERVED.
Thank You!
The SOLARWINDS and SOLARWINDS & Design marks are the exclusive property of SolarWinds
Worldwide, LLC, are registered with the U.S. Patent and Trademark Office, and may be registered or
pending registration in other countries. All other SolarWinds trademarks, service marks, and logos
may be common law marks, registered or pending registration in the United States or in other
countries. All other trademarks mentioned herein are used for identification purposes only and
may be or are trademarks or registered trademarks of their respective companies.