1. “Operationalizing”
MongoDB@AOL
Michael DelNegro
Principal Database Administrator
AOL

2. About Me
• DBA at AOL (Dulles) for 6+ years
• Original DBMS Background is in Sybase
• Now MySQL, MongoDB, PostgreSQL and
NoSQL (Neo4J, Redis, etc)
• Check out www.meetup.com/NOVA-
MySQL

3. The First Mongo at AOL
• A MapQuest Developer in the Summer of
2010 Replaced a MySQL Datastore with
Mongo
• That Developer left in early 2011 :-/
• Was Handed Over to Operations and
Yours Truly

4. Never Forget Your First
• Not Small: 40+ Hosts
• Upgrade Challenges
• Connection Floods
• Support to the Rescue
• Magnified the Reasons to Operationalize

5. Operation “Operationalize”

6. Operation “Operationalize”
• Take Advantage of 10Gen Support
• Establish Standards
• Monitoring/Alerting
• Backups
• Information Sharing
• Challenges

7. 10Gen Support
• JIRA Documents Cases Enabling
Information Sharing and Referencing
• Serves Us Well in Both “Emergencies” and
“Hey Quick Question”
• Training Classes: Administration &
Development

8. Standard Host Setup
• CentOS 6
• XFS/EXT4
• Deadline I/O Scheduler
• dirty_ratio = 10 (down from 40)
• dirty_background_ratio = 5 (down from 10)
• noatime

9. Standard Build Scripts
• Use 64bit Linux Binaries Stored in AOL
Repository
• mongodb_install.sh
– Parameters Include:
• MongoDB Version
• Replica Set Name
• Sharded? (Y/N)
• Port (27017 is standard)
• Type (DB, Arbiter, Config, Mongos)

10. Standard Directory/Layouts
• Based on our standard MySQL layout
• Binaries in /opt/local/pkg
• /db/mongodb/<port> base directory
– Ex. /db/mongodb/27017
– Can Override
• /db/mongodb/27017-repset1
• Under base directory
– data
– etc (config files)
– logs
– scripts (ex stop/start)

11. ORB
• AOL Technologies’ Configuration
Management Database (CMDB)
• Integrated with many authoritative data
repositories
• Unique namespace for many operations data
points
• Data model for operations management
• Projects, Assets, People, Applications,
Network Data, HCM
• SQL Interface

12. Monitoring/Alerting
• Argus
• Nagios
• Component Test Tool
• MMS (MongoDB Monitoring Service)

13. Argus
• Metric and Event
– Collector
– Thresholds
– Management
– Data Viewing

14. Argus
• Capture serverStatus Metrics (once a minute)
• Capture Replication Lag
• Great for Trending
• Great for Capacity Planning
• Great for Troubleshooting
• We Also Use for Host Metrics (CPU, I/O, etc)
• Administrated By a Small and very Busy
Group
– Requested Additions/Changes Can Be Slow

15. Nagios
• Great For Fault-detection/Alerting
• Great For Show Me What Is Currently
Broken
• Great For Service Availability Metrics
• Flexible
• Reduces Pressure On NOC
• Integrates With Netcool, Ignore Tool
• We Write Our Own Plug-Ins

16. Component Test Tool
• Runs Parallel HTTP GETs Across a Group
of Hosts
• Matches Regular Expressions On the
Page Content (or HTTP Headers)
• Displays Number (or Content) of Matches
in a HTML Table

17. MMS

18. Backups
• Scripts Delivered With Install
• Choice Between LVM (default) and
Standard (mongodump)
• Ability to Backup Sharded Setup

19. Mongo @ AOL Currently

20. Mongo @ AOL Currently
• 30+ Active MongoDB Projects
• 500+ Servers in Production
• Developed in:
– Java
– PHP
– Ruby
– C#
– Erlang
– Scala
– Perl
– C++

21. Information Sharing
• Internal Wikis/Websites
• UnUniversity (Organized Brown Bags)
• Architecture Reviews
• 10Gen JIRA Access

22. Challenges
• Developers Love It
• BUT Requires More Upfront Planning
Than They May Realize
• Should NOT Be Our Default Datastore
• Better Developer OnBoarding
• Aggressive Upgrading Is Required
• Coordination of Driver Upgrades

23. The Future
• “MongoDB As a Service”
• Puppet or Chef
• Internal Administration Console
– Already has Visual Topologies
• Internal MongoDB Operations Class
– First One is in July

24. Resources
• Recommended Presentation: Todd
Dampier’s Rock-Solid Mongo Ops
(mongolab)
• Check out Jon Reed’s MongoDB for
Online Advertising at AOL (MongoDB UK
2012)
• www.meetup.com/Washington-DC-
MongoDB-Users-Group/

25. Thank You!
• www.slideshare.net/radiocats
• @radiocats
• www.linkedin.com/in/mdelnegro
• Thank you, SeniorGif