MongoDB 3.2 introduces a host of new features and benefits, including encryption at rest, document validation, MongoDB Compass, numerous improvements to queries and the aggregation framework, and more. To take advantage of these features, your team needs an upgrade plan.
In this session, we’ll walk you through how to build an upgrade plan. We’ll show you how to validate your existing deployment, build a test environment with a representative workload, and detail how to carry out the upgrade. By the end, you should be prepared to start developing an upgrade plan for your deployment.
2. • Upgrading the database itself is pretty easy.
• Upgrading all the bits around the database is…
less easy.
• The effort is worth it.
• We can help you.
4. More use cases. Pluggable storage engines enables you to use MongoDB in more
projects with the same core database.
Mission-critical apps. MongoDB delivers major advances in the critical areas of
governance, high availability, and disaster recovery.
New tools for new users. Now MongoDB is an integral part of the tooling and
workflows of Data Analysts, DBAs, and Operations teams.
3.2 Overview
5. Document Validation
What you get
• Implement data governance
• Enforce data quality
• Use familiar MongoDB queries to specify validation rules
More power to the DBAs
• The DBA can specify which documents in a collection should be validated
– Pre-existing documents can be left unvalidated, optionally.
• Validation failure consequences are tunable
• Hard error
• Just a warning
6. Document Validation Example
The example on the left adds a rule to the
contacts collection that validates:
• The year of birth is no later than 1994
• The document contains a phone number and / or
an email address
• When present, the phone number and email
addresses are strings
7. Partial Indexes
{
"_id" : 1234,
"archived" : "true"
... }
A partial index can
ignore all documents
where ignored
is true.
8. $lookup Join
What you get
• Equivalent to left outer join
• Built into aggregation framework
$lookup: {
localField : "stock",
foreignField : "stockQuote",
from : "other_collection",
as : "joined_stock" }
10. Faster Failover for Replica Sets
• Enhanced algorithm detects failure and isolation of primary.
• Clusters more resilient to overloaded or unreliable networks
What you get
• <2 seconds failover (tunable with the electionTimeoutMillis)
How to upgrade:
• Conceptually:
– cfg=rs.conf()
– cfg.protocolVersion=1
– rs.reconfig(cfg)
11. Config Servers as Replica Sets
Overview
• Config server replica sets can span more than 3 data centers with up to 50 replica set
members supported
What you get
• Simplified sharded deployments
– Config servers are deployed as replica sets
• Improved metadata consistency
• Easily scale to many data centers
How to upgrade
• New clusters only, or a cluster reboot (for now)
13. Storage Engine Architecture in 3.2
Content
Repo
IoT Sensor
Backend
Ad Service
Customer
Analytics
Archive
MongoDB Query Language (MQL) + Native Drivers
MongoDB Document Data Model
WT MMAP
Available for MongoDB 3.2
Management
Security
In-memory
(beta)
Encrypted 3rd party
14. WiredTiger is the Default
What you get
• Best general-purpose storage engine
• 7-10x better throughput
• Up to 80% compression
“Out of the box”
• No special configuration needed
15. 7x-10x Performance, 50%-80% Less Storage
How: WiredTiger Storage Engine
• Same data model, same query
language, same ops
• Write performance gains driven
by document-level concurrency
control
• Storage savings driven by native
compression
• 100% backwards compatible
• Non-disruptive upgrade
MongoDB
3.0
MongoDB
2.6
Performance
16. 50%-80% Less Storage via Compression
• Better storage utilization
• Higher I/O scalability
• Multiple compression options
– Snappy
– zlib
– None
• Data and journal compressed on disk
• Indexes compressed on disk and in memory
18. Encrypted Storage
What you get
• Encryption of sensitive data for regulated industries
• ~ 15% overhead
• Better than many 3rd party encryption tools
Note
• Based on the WiredTiger Storage Engine
• Available in MongoDB Enterprise Advanced
How to upgrade
• Rolling upgrade
19. In-Memory Storage
• Predictable throughput and latency
• In-memory computing without trading away guarantees of disk-based
databases like
– Rich query flexibility
– Real-time analytics
– Scalable capacity
– Durability
• Durability via replica set secondaries (using mmap/wt)
26. Single-click provisioning, scaling &
upgrades, admin tasks
Monitoring, with charts, dashboards and
alerts on 100+ metrics
Backup and restore, with point-in-time
recovery, support for sharded clusters
MongoDB Ops Manager / Cloud Manager
The Best Way to Manage MongoDB In Your Data Center
Up to 95% Reduction in Operational Overhead
27. How Ops Manager / Cloud Manager Helps You
Scale EasilyMeet SLAs
Best Practices,
Automated
Cut
Management
Overhead
28. Query Perf. Visualizations & Optimization
Fast and simple query optimization with the
new Visual Query Profiler
• Query and write latency are consolidated and
displayed visually; your ops teams can easily
identify slower queries and latency spikes
• Visual query profiler analyzes the data it displays
and provides recommendations for new indexes
that can be created to improve query
performance
• Ops Manager and Cloud Manager can automate
the rollout of new indexes, reducing risk and your
team’s operational overhead
29. Ops Manager Backup Enhancements
3.2 includes Ops Manager enhancements to
improve the productivity of your ops teams
and further simplify installation and
management
• MongoDB backup on standard network-mountable
filesystems; integrates with your existing storage
infrastructure
• Automated database restores; Build clusters from backup in a
few clicks
• Faster time to first database snapshot
• Support for maintenance windows
• Centralized UI for installation and config of all application and
backup components
33. Upgrading a cluster manually
• Follow the steps in the fine documentation:
– http://docs.mongodb.org/manual/release-notes/3.2-upgrade/
– First result if Googling for “MongoDB Upgrade”
• It's critical to note that upgrade order matters:
– For replica sets, first upgrade the secondaries, then the primaries.
– For a sharded cluster, follow the 7 steps in the documentation.
35. Upgrading a cluster withAutomation
• Ops Manager / Cloud Manager is the easiest, fastest, and best way to
upgrade a MongoDB deployment
– One-click upgrades.
– Rolling upgrades, so zero downtime.
37. As you can see, upgrading the
database software itself is pretty easy.
38. The fine details
• Backward compatibility & your application
• Backward compatibility & your operations
• Database behavior & performance
• Enabling 3.2 Features
40. Backward Compatibility: App
• Comprehensive documentation about incompatible changes
– https://docs.mongodb.org/manual/release-notes/3.2-compatibility
• Highlights
– You may need to upgrade your applications' drivers before upgrading
MongoDB!
– Some commands/methods deprecated, removed, changed.
• Audit your codebase for uses of deprecated functionality.
41. Backward Compatibility: Ops
• Comprehensive documentation about incompatible changes
– https://docs.mongodb.org/manual/release-notes/3.2-compatibility
• Highlights
– Some settings deprecated, others mmapv1-only.
– Stricter replica set configuration validation
– mongo* tools' options changed
– Legacy (pre-2.6) user account model removed
• Audit scripts & other infrastructure you've got around.
43. Database Behavior & Performance
• Always test your application with the new database version before upgrading
production
– Apps sometimes rely on unspecified behaviors (e.g. stability of results
from unsorted queries)
– Query optimizer improvements may affect index selection (check
important queries with explain)
44. Good practice: Test against Staging
• Run all your performance correctness tests against a staging environment
for some time before upgrading production.
• Best practice: run a real application workload against a 3.2 staging
environment, too.
– Replicate the workload manually
– Use Facebook’s Flashback tool
45. In our experience, upgrading
everything around the database
software is where the work is.
48. Plan Your Upgrade
• Review the compatibility notes
– Audit apps, ops for deprecated details
• Upgrade drivers, scripts, if necessary
– Might require some recoding, in edge cases
• Make sure you've got good tests
– App behavior, performance, etc.
• Write a checklist of upgrade steps for your environments
49. Practice Your Upgrade
• Upgrade apps & database in staging environment
• Test, test, test.
– If no problems appear within (say) a week of continuous testing, probably
a decent candidate.
50. Productionize Your Upgrade
• Upgrade apps & database in production environment
– Ideally, during minimum load/traffic conditions, just for good measure
52. How we help users upgrade
• MongoDB's Professional Services team offers consulting engagements
specifically for upgrading:
– To ensure upward compatibility for your apps, tools
– To assist with performance & behavior testing around database upgrades
– To prepare & execute upgrade plans with your team in your environments.
Contact us for more info: http://mongodb.com/upgrade
Presentation at 5, so I'll do a quick overview of the features
Validation is optional, and can be as simple as a single field, all the way to every field, including existence, data types, and regular expressions.
Failed validation tests can result in a hard error or just a warning
Validation is optional, and can be as simple as a single field, all the way to every field, including existence, data types, and regular expressions.
Failed validation tests can result in a hard error or just a warning
Raft like protocol, with election terms
Timeout was previously for all
Fast Failure - in many scenarios failover will be well under 2 seconds including sub-second, but we want to set reliable expectations
The exact failover time is dependent on the system’s configuration (for example the network latency between data centers), but for a typical configuration it would be no more than 2 seconds – the user can tune this using the electionTimeoutMillis parameter.
Config server setup provides higher levels of availability and lower cross-region latency.
Now let’s take a deep dive into the Storage Engines and how they can help you use MongoDB for a wide array of new projects. With this release you get a lot more choice with your storage engines. You can mix and match multiple storage engines within a deployment to bring a new level of optimization to your applications.
Let’s take a look so you can see which is right for you.
WiredTiger was available as an alternative to the MMAP Storage Engine, which was the default storage engine in previous to MongoDB 3.0.
WiredTiger offers a lot of benefits to MongoDB users. WiredTiger has granular concurrency control and native compression, giving you better storage efficiency and higher performance for a broad range of apps.
Now it is the default storage engine in MongoDB 3.2, so no need to do any special configuration to get these benefits.
Upgrade can be made transparently, the application will not know.
Different performance operational profiles.
WT: Excellent write performance
Encrypted Storage engine is the name of the feature but is NOT separate from WT, rather an option -- want to avoid any chance of confusion!
KMIP is a popular standard that we plan to support; SafeNet and Vormetric are good examples, as is Amazon’s KMS
In-memory computing is known to be much faster, and it enables data access and analytics at speeds never before possible. However, in the past, memory costs were prohibitive. Now, that the cost of memory has gone down, and will continue to tumble, you can take advantage of these performance gains.
As illustrated by the ecommerce example above, user data is managed by the In-Memory engine to provide the throughput and bounded latency essential for great customer experience. However, the product catalog’s data storage requirements exceed server memory capacity, so is provisioned to another MongoDB replica set configured with the disk-based WiredTiger storage engine.
In this example, MongoDB’s flexible storage architecture means developers are freed from the complexity of having to use different in-memory and disk-based databases to support the e-commerce application. Administrators are freed from the complexity of having to configure and manage separate data layers. Instead, the application uses the same MongoDB database with each service powered by the storage engine best optimized for the use case.
BI Connector requires Enterprise Advanced subscription
SQL supported is read-only and covers what is required for most BI tools (we can’t simply say SQL92 b/c it only supports reads)
MongoDB Compass requires Professional of Enterprise Advanced subscription
“Determine validator rules”: you can use the tool to figure out what you want to set as validation rules. A future version could integrate with the database to set document validation rules for a collection.
MongoDB Ops Manager and Cloud Manager is the best way to run MongoDB, reducing tasks such as deployment, scaling, upgrades and backups to just a few clicks or an API call. Operations teams can be 10-20x more productive using the Ops or Cloud Manager platforms.
With these enhancements to Ops and Cloud manager, administrators can
Integrate MongoDB alongside existing Application Performance Monitoring platforms for global health visibility over the entire IT estate, all from a single pane of glass
Drill down into any MongoDB-specific issues using Ops Manager’s monitoring of key database telemetry, including new query profiler visualizations
Create a point-in-time backups and consistent snapshots of the database on standard network-mountable filesystems
Use Ops Manager automation to initiate zero-downtime maintenance and upgrade activities, such as rolling out new indexes across a sharded cluster
Profiler Visualization: Enabling Fast and Simple Query Optimization
Automated Index Builds
New Indexing Option: Partial Indexes
Ops Manager can do a lot for [ops teams].
Best Practices, Automated. Ops Manager takes best practices for running MongoDB and automates them. So you run ops the way MongoDB engineers would do it. This not only makes it more fool-proof, but it also helps you…
Cut Management Overhead. No custom scripting or special setup needed. You can spend less time running and managing manual tasks because Ops Manager takes care of a lot of the work for you, letting you focus on other tasks.
Meet SLAs. Automating critical management tasks makes it easier to meet uptime SLAs. This includes managing failover as well as doing rolling upgrades with no downtime.
Scale Easily. Provision new nodes and systems with a single click.
Order Matters
xxx
If upgrading to 3.0, votes
User Roles, of the SuperUser/ReadOnly
Audit
Be careful, MongoDB will have some changes in the behavriou or performance
Large complicated piece of software. Some things we intend to do, and we document them, and there are behavrious that we don’t follow.
From time to time, changes in implementaion.
Your application might inadvertenly rely on this.
Always test your application against the new version.Most of you might way of course.
Optimizer, look at the explain plain
When you throw a query at MongoDB when you don’t specify sort order, you’ll get one order, and newer version don’t make that promeise.
They never made that promise
Always test your application against the new version.Most of you might way of course.
Optimizer, look at the explain plain
When you throw a query at MongoDB when you don’t specify sort order, you’ll get one order, and newer version don’t make that promeise.
They never made that promise
enable 3.2.0 features
A few days, perhaps a week.
Rolling upgrade
Burn in test
We’ll construct these plans, we’ll help write test scripts to simulate the workload or maybe even record the workload and then replay it.