Deploying any software can be a challenge if you don't understand how resources are used or how to plan for the capacity of your systems. Whether you need to deploy or grow a single MongoDB instance, replica set, or tens of sharded clusters then you probably share the same challenges in trying to size that deployment.
This webinar will cover what resources MongoDB uses, and how to plan for their use in your deployment. Topics covered will include understanding how to model and plan capacity needs for new and growing deployments. The goal of this webinar will be to provide you with the tools needed to be successful in managing your MongoDB capacity planning tasks.
7. Preparing for Launch
• Developers are about to finish final Sprint
• Code is good (so they say )
• You feeling comfortable to launch soon
• How to deploy?
8. Requirements
• Availability
– Uptime requirements: RPO and RTO
• Throughput
– Average read/writes/users
– Peek throughput
– Operations per second ? per day? per month?
• Responsiveness
– What's the acceptable latency?
• Higher during peek time?
RTO=recovery time objective RPO=recovery point objective
14. Why?
• Once we launch, we don't want to have avoidable down
time due to poorly selected HW
• As our success grows we want to stay in front of the
demand curve
• We want to meet business and users expectations
• We want to keep our jobs!
• Don't be the "goat"
29. Storage Capability
Type IOPS
7200 rpm SATA ~ 75 – 100
15000 rpm SAS ~ 175 – 210
SSD Intel X25-E (SLC) ~ 5000
SSD Intel X25-M G2 (MLC) ~ 8000
Amazon EBS ~ 100
Amazon EBS Provisioned Up to ~10,000
Amazon EBS Provisioned IOPS (SSD) Up to ~20,000
http://en.wikipedia.org/wiki/IOPS
30. Storage Capability
Type IOPS
7200 rpm SATA ~ 75 – 100
15000 rpm SAS ~ 175 – 210
SSD Intel X25-E (SLC) ~ 5000
SSD Intel X25-M G2 (MLC) ~ 8000
Amazon EBS ~ 100
Amazon EBS Provisioned Up to ~10,000
Amazon EBS Provisioned IOPS (SSD) Up to ~20,000
FusionIO ~135,000
Violin Memory 6000 ~ 1,000,000
http://en.wikipedia.org/wiki/IOPS
Higher IOPS higher the Cost!!!
31. Storage Considerations
• Work out how much data you need to write per unit of
time!
• Databases will use storage to persist data
– More data = Bigger indexes = More Storage
• MongoDB Stores Information into Documents
• BSON Format
– http://bsonspec.org/
32. Memory
• Working Set
– Active Data in Memory
– Measured Over Periods
• And other operations
– Sorting
– Aggregation
– Connections
• WiredTiger Storage Engine Cache
35. Basic Rules
• Determine data size, working set, query throughput
requirements
• Use good measuring and monitoring practices
• Plan ahead but be flexible!
• Iterate
– Review Requirements
– Review Capacity
37. • Generate a sample document set
– Write some code?
• Use db.stats() to measure
– Disk space
– Compression
• Do the math
– Estimate production disk requirements
• Sum of disk space across shards > greater than required
storage size
Disk Space
38. • Sum of disk space across shards > greater than required
storage size
Disk Space: How Many Shards Do I Need?
Example
Data Size = 9 TB
WiredTiger Compression
Ratio: .33
Storage size = 3 TB
Server disk capacity = 2 TB
2 Shards Required
39. RAM Requirements
• Working Set < RAM
• WorkSet = Indexes plus the set of documents accessed
frequently
• WorkSet in RAM
– Shorter latency
– Higher Throughput
40. Estimating Working Set
• Using your sample data set
– Create required indexes based upon queries
– Use db.coll.stats() to get index size
• Do the math to get production index size
• Estimate the working set
– Given the queries
– What are the frequently accessed docs?
– Examples:
• Last x days of data
• Most queried devices
41. RAM: How Many Shards Do I Need?
Example
Working Set = 428 GB
Server RAM = 128 GB
428/128 = 3.34
4 Shards Required
42. • Measure max sustained query rate of a single server
(with replication)
– Use prototype/development version
– Use application queries and data
– Measure max sustained performance
• Assume sharding overhead of 20-30%
Query Rate
43. • Measure max sustained query rate of a single server (with replication)
– build a prototype and measure
• Assume sharding overhead of 20-30%
Query Rate: How Many Shards Do I Need?
Example
Require: 50K ops/sec
Prototype performance: 20
ops/sec (1 replica set)
4 Shards Required: 80
ops/sec * .7 = 56K ops/sec
44. Measuring & Monitoring
• What to measure
– IOPS
– Page Faults
– Resident Memory (Working Set)
– Connections
– Lock %
• How to measure and monitor Ops/Cloud Manager
• Command Line Tools
– iostat
– vmstat
– mongostat
45. The Best Way to Run MongoDB
Ops Manager allows you leverage and automate the best
practices we’ve learned from thousands of deployments in
a comprehensive application that helps you run MongoDB
safely and reliably.
Benefits include:
10x-20x more efficient operations
Complete performance visibility
Protection from data loss
Assisted performance optimization
47. Monitoring and Alerting
Over 100+ database metrics
Dozens of optimized charts
Custom alerts so incidents don’t
become emergencies
48. APM lntegration
Monitor MongoDB alongside the rest of
your app infrastructure using our
RESTful API
Leverage packaged integrations with
leading APM platforms
52. Database Automation
Automate tasks that you would have
otherwise performed manually, such
as…
• Deploying a new cluster
• Upgrades
• Adding capacity
• Database restores
53. Backup with Point-in-time Recovery
Restore to precisely the moment you
need, quickly and safely.
Ops Manager is the only MongoDB
backup solution that offers point-in-time
backups of replica sets and cluster-
wide snapshots of sharded clusters.
55. Capacity Planning is …
• Needed
– Involves resource allocation
– Hardware specification and sizing
– Cost!
• Vital
– Translate Requirements and Expectations into Experience
and Functionality
• And meeting those
• Requires understanding your application
– Measuring resource needs
– Monitoring
– Iterating
– Repeating process
56. For More Information
Resource Location
Case Studies mongodb.com/customers
Presentations mongodb.com/presentations
Free Online Training education.mongodb.com
Webinars and Events mongodb.com/events
Documentation docs.mongodb.org
MongoDB Downloads mongodb.com/download
Additional Info info@mongodb.com
Fine art of translating Requirements of the application into resources needed to support the application that implements those requirements
Fine art of translating Requirements of the application into resources needed to support the application that implements those requirements
Understanding the requirements and resources implication
Collection scans
These numbers are already old and not relevant anymore.
These numbers are already old and not relevant anymore.
These numbers are already old and not relevant anymore.
These numbers are already old and not relevant anymore.
MongoDB Ops Manager can do a lot for [ops teams].
Best Practices, Automated. Ops Manager takes best practices for running MongoDB and automates them. So you run ops the way MongoDB engineers would do it. This not only makes it more fool-proof, but it also helps you…
Cut Management Overhead. No custom scripting or special setup needed. You can spend less time running and managing manual tasks because Ops Manager takes care of a lot of the work for you, letting you focus on other tasks.
Meet SLAs. Automating critical management tasks makes it easier to meet uptime SLAs. This includes managing failover as well as doing rolling upgrades with no downtime.
Scale Easily. Provision new nodes and systems with a single click.
Ops Manager agents are installed on servers (where MongoDB will be deployed), either through configuration tools such as Chef or Puppet, or by an administrator.
The administrator creates a new design goal for the system, either as a modification to an existing deployment (e.g., upgrade, oplog resize, new shard), or as a new system.
The agents periodically check in with the Ops Manager central server and receive the new design instructions.
Agents create and follow a plan for implementing the design. Using a sophisticated rules engine, agents continuously adjust their individual plans as conditions change. In the face of many failure scenarios – such as server failures and network partitions – agents will revise their plans to reach a safe state.
Minutes later, the system is deployed – safely and reliably.
Ops Manager monitors 100+ metrics that could impact the performance of your database
Track key performance indicators across dozens of optimized charts
Build customized alerts that trigger when metrics are out of range; have them delivered how you want
RESTful API to monitor MongoDB alongside the rest of your application infrastructure, from a single “pane of glass”
Packaged integrations with leading APM platforms such as New Relic now available with MongoDB Cloud Manager
Quickly identify your slow-running queries.
Part of MongoDB Ops Manager, the Visual Query Profiler displays how query and write latency vary over time
With the click of a button, the Visual Query Profiler consolidates and displays metrics from all your nodes on a single screen
The Visual Query Profiler analyzes the data it displays and presents recommendations for new indexes that can be created to improve the performance of your deployment.
The best practice for adding new indexes to your deployment is a rolling index build – starting with each of the secondaries and finally applying changes to the original primary, after swapping its role with one of the secondaries.
Ops Manager can automate this process across your replica sets, reducing your operational overhead and the risk of failovers caused by incorrectly sequencing management processes.
Ops Manager coordinates and orchestrates critical operational tasks across the servers in a MongoDB system. Tasks that you would have otherwise performed manually, such as…
Deploying a new cluster
Upgrades
Adding capacity
Database restores
Use the Ops Manager UI directly, or invoke the Ops Manager RESTful API from your existing enterprise orchestration frameworks, such as Chef, Puppet, etc.
Restore to precisely the moment you need, quickly and safely.
Ops Manager is the only MongoDB backup solution that offers point-in-time backups of replica sets and cluster-wide snapshots of sharded clusters.
Integrates with standard network-mountable file systems
Backups can be configured against specific collections, rather than the entire database