Mongo db operations_v2

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
mongoDB
NoSQL Team
Administration
And Operations

2
Agenda
MongoDB
• Part I--Review (Q&A)
• Mongodb install in Windows
• Question oriented (gage where we need to start)
• Basics and Review
• Architecture
• Interactive Lab I – Starting mongo service, Mongo shell, connecting to mongod
• Review of MongoDB Terminology
• Common Methods to use
• CRUD
• Interactive Lab 2 – Creating a Collection and Inserting Documents

3
Agenda
MongoDB
• Part II
• Indexes
• Backup and Recovery
• Replication
• Security Management
• Administration
• Support Process
• Configuration, Maintenance, Analysis
• Performance (Profiler/Current Ops)
• Installation – Linux - Siva

4
Installation
MongoDB
http://www.mongodb.org/downloads

5
Installation
MongoDB
Student Lab 1

6
MongoDB

7
Comparison
MongoDB - Review

8
DB Landscape-Review (Source InfoChimps)

9
Scalability & Performance vs Functionality
MongoDB-Review
Depth of functionality
Scalability
&
Performance
memcached
Key / value
RDBMS
mongoDB

10
Job Trends
MongoDB-Review

11
Document Oriented
MongoDB-Review
NoSQL Databases
Document Oriented
Databases
MongoDB

12
Document Oriented
MongoDB-Review
MongoDB is a scalable, high-performance, open source and document-oriented
database written in C++.
To understand what we mean when we talk about document-oriented we need
some basic MongoDB terminology. Let’s take a look at terminology as viewed
in a JSON file :
{
"terminology" :
{ "Database" : "Database",
"Table" : "Collection",
"Row" : "Document“
}
}

13
Document Oriented
MongoDB-Review
A document-oriented database is a database where each document in a same
collection may have a totally different structure. With this orientation, any
number of fields of any length can be added to a document even after its
creation.
Because all the information of the same entity can be dynamically stored within
a single document, joins operations are no longer needed in this type of
database.
Joins operations are really expensive, they require a strong consistency and a
fixed schema. Avoiding them results in a great capacity of horizontal scaling.

14
Data Types
MongoDB-Review
MongoDB uses BSON as the data storage and network transfer format. BSON
stands for "Binary JSON" and is a binary-encoded serialization of JSON-like
documents. According to the bsonspec.org, it was designed to be lightweight,
traversable and efficient.
It’s key advantage over XML and JSON is efficiency in term of space and
compute time.
BSON documents can be used to store several data types like ''string, integer,
boolean, double, null, array, object, date, binary data, regular expression and
source code''. You can find more by exploring the specification.

15
Document Size
MongoDB-Review
The maximum BSON document size in MongoDB is 16MB. Users should
avoid certain application patterns that would allow documents to grow
unbounded. For
instance, applications should not typically update documents in a way that
causes them to grow significantly after they have been created, as this can lead
to inefficient use of storage. If the document size exceeds its allocated space,
MongoDB will relocate the document on disk. This automatic process can be
resource intensive and time consuming, and can unnecessarily slow down other
operations in the database.

16
Document Size
MongoDB-Review
Use Case Example:
How would we tackle this problem in Mongodb:
Use Case Description: Product catalogs must have capacity to store many
different types of objects with different site attributes.
Check all use-case documentation under mongodb.org!

17
Document Size
MongoDB-Review
Use Case Example:
How would we tackle this problem in Mongodb:
Use Case Description: We have an ecommerce application that has 1000
products. In this application, each product can have n number of reviews by
customers. It is not possible to know how many reviews a product will receive.
Further, we may only display a subset of the reviews maybe by popularity or the
most recent reviews.

18
Data Store
MongoDB-Review
• JSON-style data structures composed of field-and-value pairs:
Field/value
{“FirstName”:”Kim”}
{“FirstName”:”Siva”,
“LastName”:”Padidam”,
“Hobbies”:[”soccer”,”movies”,”databases”]}

19
Schemaless
MongoDB-Review
• Schemaless means that documents don’t need to have the same schema.
{“FirstName”:”Robin”,
“LastName”:”Thicke”}
{“FirstName”:”Pharrell”,
“LastName”:”Harris”,
“Hobbies”:[”music”,”singing”,”collaborating”,”song writing”]}

20
Main Features
MongoDB-Review
• Replication
• Sharding/Load balancing
• File storage
• Aggregation
• Capped collections

21
Comparing RDBMS to MongoDB
MongoDB-Review
RDBMS
Database
Table
Row
Column
Index
MongoDB
Database
Collection
Document
Field/KeyName
Index

22
Use cases and production deployments
MongoDB-Review
• Archiving & Event Logging
• Document & Content Management System.
• E-commerce (in combination with a RDBMS)
• High Volume Problems
• Data store of a web site
• Real-Time stats/analytics

23
MongoDB Differences to RDBMS
MongoDB - Review
• Joins
• Complex transactions
• SQL
• No constraints support

24
Scalability
• Scales like all NOSQL and other modern solutions
- Many servers
- MongoDB provides scale out solution (horizontal scalability)
• Local Storage spread across multiple servers
• Memory and IO drives speed of Mongo Cluster
• Network optimization can be key
- Lack of it could deterrent scaling

25
A View of Mongo Cluster – High Availability
Mongo
Server
- Mongod
hosts
Mongo
Storage
Local Data
Local Data
Mongo
Server
- Mongod
Mongo
Server
- Mongod
…….
Local Data
Local Data
Local Data
Local Data
Local Data
Local Data
Local Data
Local Data
Local Data
Local Data
Local Data
-Mongos
Application Server
-Mongos
Application Server
…………
..
Local Data
Primary Secondary Secondary
RSE
T
-
1
Reads
Reads +
Writes
Reads
OpLog OpLog OpLog

26
A View of Mongo Cluster – High Availability
• One Primary
• Read + Write
• Multiple – Secondary’s
• Only Reads
• Generally in different data centers
• Switchover happens automatically when primary becomes unavailable
• MongoS process knows the state of each mongod instance

27
Architecture -GridFS
GridFS is a specification for storing and retrieving files that exceed the
BSON-document size limit of 16MB.
Instead of storing a file in a single document, GridFS divides a file into parts, or
chunks, [1] and stores each of those chunks as a separate document. By default
GridFS limits chunk size to 256k. GridFS uses two collections to store files. One
collection stores the file chunks, and the other stores file metadata.
When you query a GridFS store for a file, the driver or client will reassemble the
chunks as needed. You can perform range queries on files stored through
GridFS. You also can access information from arbitrary sections of files, which
allows you to “skip” into the middle of a video or audio file.

28
Architecture -GridFS
GridFS is useful not only for storing files that exceed 16MB but also for storing
any files for which you want access without having to load the entire file into
memory.

29
Architecture –GridFS Implementation
To store and retrieve files using GridFS, use either of the following:
• A MongoDB driver. See the drivers documentation for information on
using GridFS with your driver.
• The mongofiles command-line tool in the mongo shell. See mongofiles.
To return a list of all files in a GridFS collection in the records database
$mongofiles -d records list
To upload a file named bill.lp to the GridFS collection in the records database,
you can use the following command:
$mongofiles -d records put bill.lp

30
Architecture –GridFS Collection
GridFS stores files in two collections:
• chunks stores the binary chunks. For details, see The chunks
Collection.
• files stores the file’s metadata. For details, see The files Collection.
GridFS places the collections in a common bucket by prefixing each with the
bucket name. By default, GridFS uses two collections with names prefixed by fs
bucket:
• fs.files
• fs.chunks
You can choose a different bucket name than fs, and create multiple
buckets in a single database.

31
Architecture –GridFS Collection
GridFS Index
GridFS uses a unique, compound index on the chunks collection for the
files_id and n fields. The files_id field contains the _id of the chunk’s
“parent” document. The n field contains the sequence number of the
chunk. GridFS numbers all chunks, starting with 0.

32
Configuration and Operation
MongoDB - Review
• Files Location on GDS Implementation (DEV, ITG and PRO)
/opt/mogodb/mongod.conf (mongo configuration file)
/log/mongod.log (mongodb log)
/opt/mongodb/data (db path)
/opt/mongodb/config/mongod.pid (pidfilepath / process ID of the mongod process)
/opt/mongodb/mongodb-ent/bin (binary files path) 2.4.5 Version
/etc/rc.d/init.d/mongod (init script)

33
MongoDB - Review
• Start the mongod process by issuing the following command (using root permission)
$service mongod start
• You can verify that the mongod process has started successfully by checking the
contents of the log file at log/mongod.log.
• Another way to verify mongoDB process is running is by executing the following Linux
command:
$ ps –ef | grep mongod
• You may also use the –list option with chkconfig as follows on linux:
$chkconfig --list mongod

34
Configuration and Operation—Stopping Mongodb
MongoDB - Review
• Stopping Mongod
$service mongod stop
Alternatively you could also issue the following command:
>/mongodb/bin/mongod –shutdown –config <config file path>
Or, from mongo shell:
>db.shutdownServer();

35
MongoDB - Review
• You can restart the mongod process by issuing the following command (using root
privilege):
>service mongod restart
• MongoDB will start automatically after a system reboot since chkconfig option is set to
ON
>chkconfig mongod on
[root@g9t3417 ~]# chkconfig | grep mongo
mongod 0:off 1:off 2:off 3:on 4:off 5:on
6:off

36
Mongo Process
MongoDB
• Additionally you can search for information about the mongodb instance and errors on
the mongod.log file by executing more or tail Linux commands.
• > more /log/mongod.log (filter for paging through text)
• > tail /log/mongod.log -n 30 (get the last 30 lines in the file)
• >db.adminCommand({getLog:"*"}) (Display available log filters)

37
mongodb.log
MongoDB - Review
You can get useful information from the mongodb.log such as:
• Mongodb Binary Version 64 or 32 bits.
• Mongodb version – 2.4.5
• Options used to start the mongodb instance
• Port Number - 20001
• DB Path
• Config Path
• Replset Name
• Warning/Error messages

38
Defaults turned on when starting mongo
MongoDB
journal Default: (on 64-bit systems) true
Default: (on 32-bit systems) false
Set to true to enable operation journaling to ensure write durability and data
consistency. Set to false to prevent the overhead of journaling in situations
where durability is not required.
To reduce the impact of the journaling on disk usage, you can leave journal
enabled, and set smallfiles to true to reduce the size of the data and journal
files.
Note: You must use nojournal to disable journaling on 64-bit systems.

39
Defaults turned on when starting mongo
MongoDB
With journaling enabled, MongoDB creates a journal directory within the
directory defined by dbpath, which is /data/db by default.
The journal directory holds journal files, which contain write-ahead redo logs.
The directory also holds a last-sequence-number file. A clean shutdown
removes all the files in the journal directory.
Journal files are append-only files and have file names prefixed with j._. When a
journal file holds 1 gigabyte of data, MongoDB creates a new journal file. Once
MongoDB applies all the write operations in the journal files, it deletes these
files. Unless you write many bytes of data per-second, the journal directory
should contain only two or three journal files.

40
Terminology
MongoDB-Review
SQL Terms/Concepts MongoDB Terms/Concepts
database database
table collection
row document or BSON document
column field
index index
table joins embedded documents and linking
primary key
Specify any unique column or column combination as primary key.
primary key
In MongoDB, the primary key is automatically set to the _id field.
aggregation (e.g. group by)
aggregation framework
See the SQL to Aggregation Framework Mapping Chart.

41
Terminology
MongoDB-Review
MongoDB stores data in the form of documents, which are JSON-like field
and value pairs. Documents are analogous to structures in programming
languages that associate keys with values, where keys may nest other pairs
of keys and values (e.g. dictionaries, hashes, maps, and associative arrays).
Formally, MongoDB documents are BSON documents, which is a binary
representation of JSON with additional type information

42
Terminology
MongoDB-Review
MongoDB stores all documents in collections. A collection is a group of related
documents that have a set of shared common indexes. Collections are
analogous to a table in relational databases. A collection of mongodb
documents is shown below:

43
Hierarchy
MongoDB-Review
Mongo Cluster
Database
Collections
Documents(BSON<
JSON)

44
Student Lab 2
MongoDB-Review
Student Lab 2

45
MongoDB-Review

46
CRUD – MongoDB vs. SQL
MongoDB - Review
CRUD CREATE READ UDATE DELETE
SQL INSERT SELECT UPDATE DELETE
MongoDB INSERT FIND UPDATE REMOVE

47
Insert
MongoDB - CRUD
Summary:
In MongoDB, the db.collection.insert() method adds new documents into a collection. In
addition, both the db.collection.update() method and the db.collection.save() method can
also add new documents through an operation called an upsert. An upsert is an operation
that performs either an update of an existing document or an insert of a new document if
the document to modify does not exist.
Format:
>db.<collection name>.insert(<document>)

48
Insert
MongoDB - Review
Example 1:
• >db.products.insert(“Name”: “Kim”)
Example 2 Using a Variable:
• >x = { name : "mongo" }
• >db.ServiceOfferings.insert(x)

49
Insert Document
MongoDB
• db.collection.insert(document)
document - A document or array of
documents to insert into the collection.
db.Person.insert({"FirstName":"Peter","LastName":"Parker"})
/* JSON text */
{
"_id" : ObjectId("52132cd2e54e8870e6372d4f"),
"FirstName" : "Peter",
"LastName" : "Parker"
}

50
Insert Document
MongoDB
db.Person.insert(
{
"FirstName":"Bruce","LastName":"Wayne",
"Hobbies":["Being Batman","Buy
Restaurants","Buy cars"]
})
{
"_id" : ObjectId("52132f12e54e8870e6372d50"),
"FirstName" : "Bruce",
"LastName" : "Wayne",
"Hobbies" : ["Being Batman", "Buy
Restaurants", "Buy cars"]
}

51
Insert Embedded data
MongoDB
db.Person.save(
{
"FirstName":"Michael","LastName":"Jordan"
, "Address": {"street":"1000 somestreet",
"city":"Chicago", "zip":60808}
})

52
Remove a document
MongoDB
db.collection.remove(query, justOne)
query - Specifies deletion criteria using query operators
justOne - Optional. To limit the deletion to just one document, set to true. The
default value is false.
db.Person.remove({ "FirstName" : "Peter" });

53
Update document
MongoDB
db.collection.update(query, update, options)
Query – The selection criteria for the update
Update – The modifications to apply
Options:
upsert (Optional. If set to true, creates a new document when no document
matches the query criteria)
multi (Optional. If set to true, updates multiple documents that meet the
query criteria)

54
Update document
MongoDB
db.Person.update(
{ 'Address.street': "1000 somestreet" },
{
$set: { 'Address.street': "1000 Page Mill Rd" },
}
)

55
MongoImport
MongoDB
Summary:
The mongoimport tool provides a mechanism to import content from a JSON,
CSV, or TSV export created by mongoexport, or other tools.
Syntax:
$mongoimport –db <dbname> -collection <collectionname> <
FILENAME to import
Many options exit such as username, port, type:
$mongoimport - -help

56
MongoImport
MongoDB
Example 3 using mongoimport with a JSON type file:
$mongoimport –db <dbname> -collection <collectionname> <
<path:jsonfilename>
$mongoimport –db product –collection widget < widget.json

57
Oracle vs. SQLSever vs. MongoDB
MongoDB - Review
Oracle SQLSever MongoDB Oracle SQLSever MongoDB
oracle server sqlservr monogd where Where $match
sqlplus sqlcmd Mongo group by Group by $group
database database database having having $match
table Table collection select select $project
row row document order by order by $sort
column column field limit top $limit
index Index index sum() sum() $sum
join join n/a count() count() $sum
primary key (user
defined)
Primary key (user
defined) primary key (_id) partition
Federations (SQL
Azure) shard

58
SQL query Vs. Mongo Query
MongoDB - Review
create table users (name varchar(30), age number) db.createCollection(“users”)
insert into users values (‘Bob’,32) db.users.insert({name:”Bob”,age:32})
select name, age from users where age=33 db.users.find({age:32},{name:1,age:1,_id:0})
select * from users db.users.find()
select * from users where age=33 order by name ASC db.user.find({age:33}).sort({name:1})
select * from users where name like 'B%' db.users.find({name:/^B/})
create index on users (name ASC, age DESC) db.users.ensureIndex({name:1,age:-1})

59
Querying, Finding Results
MongoDB
Summary:
MongoDB has a rich query system that allows you to select and filter the
documents in a collection along specific fields and values. Use the find()
method to display results.
>db.<collection>.find() – Finds all results
>db.<collection>.find().pretty() – More readable display

60
MongoDB
Summary:
findOne Method returns one document that satisfies the specified query criteria. If multiple
documents satisfy the query, this method returns the first document according to the
natural order which reflects the order of documents on the disk. In capped collections,
natural order is the same as insertion order.
The following operation returns a single document from the parts collection:
>db.parts.findOne()

61
MongoDB
In MongoDB a query targets a specific collection of documents. Queries specify
criteria, or conditions, that identify the documents that MongoDB returns to the
clients. A query may include a projection that specifies the fields from the matching
documents to return. You can optionally modify queries to impose limits, skips, and
sort orders.
Query Operators:
http://docs.mongodb.org/manual/reference/operator/nav-query/
To find documents that match a set of selection criteria, call find() with the
<criteria> parameter.
>db.products.find( { qty: { $gt: 25 } } )

62
MongoDB
Query for Ranges
Combine comparison operators to specify ranges. The following operation returns
documents with field between value1 and value2:
 db.collection.find( { field: { $gt: value1, $lt: value2 } }
);

63
MongoDB
Name Description
$gt
Matches values that are greater than the value specified in the
query.
$gte
Matches values that are equal to or greater than the value
specified in the query.
$in
Matches any of the values that exist in an array specified in the
query.
$lt Matches values that are less than the value specified in the query.
$lte
Matches values that are less than or equal to the value specified
in the query.
$ne
Matches all values that are not equal to the value specified in the
query.
$nin
Matches values that do not exist in an array specified to the
query.
Comparison
Logical

64
MongoDB
Name Description
$or
Joins query clauses with a logical OR returns all documents that
match the conditions of either clause.
$and
Joins query clauses with a logical AND returns all documents that
match the conditions of both clauses.
$not
Inverts the effect of a query expression and returns documents that
do not match the query expression.
$nor
Joins query clauses with a logical NOR returns all documents that
fail to match both clauses.
Logical
Operators

65
MongoDB
Name Description
$exists Matches documents that have the specified field.
$type Selects documents if a field is of the specified type.
Element

66
MongoDB
Name Description
$mod
Performs a modulo operation on the value of a field and selects
documents with a specified result.
$regex
Selects documents where values match a specified regular
expression.
$where Matches documents that satisfy a JavaScript expression.
Evaluation

67
MongoDB
To increase performance, you can constrain the size of the result by limiting the amount of
data your application must receive over the network.
To specify the maximum number of documents in the result set, call the limit() method on
a cursor, as in the following command:
>db.<collection>.find().limit(5)
This will limit the result set to 5.

68
MongoDB
Finding specific results
>db.<collection>.find({ foo : 1});
Find the results in a collection where foo = 1;
Example: Below example, shows how to find documents in the
Names Collection where first_name equals ‘Latasha’.
db.Names.find( { first_name : ‘Latasha’ } )
NOTE: To pretty up the display append pretty method at the
end: db.Names.find({first_name: ‘Latasha’}).pretty()

69
Student Lab
MongoDB-Review
Student Lab 3

70
Query Optimization

71
Indexing - Review
MongoDB
An index is a data structure that allows you to quickly locate documents
based on the values stored in certain specified fields. Fundamentally,
indexes in MongoDB are similar to indexes in other database systems.
MongoDB supports indexes on any field or sub-field contained in
documents within a MongoDB collection.

72
Query Optimization
MongoDB
Indexes improve the efficiency of read operations by reducing the amount of
data that query operations need to process. This simplifies the work associated
with fulfilling queries within MongoDB.
To create an index in MongoDB use:
db.collection.ensureIndex() method

73
Query Optimization
MongoDB
Example
An application queries the books collection on the title field. The value of
the title field is user-driven.
>var typeValue = <someUserInput>; db.books.find( { title: typeValue } );
To improve the performance of this query, add an ascending, or a
descending, index to the book collection on the title field.
>db.books.ensureIndex( { title: 1 } )

74
Query Optimization
MongoDB
To create a compound index use an operation that resembles the
following prototype:
db.collection.ensureIndex( { a: 1, b: 1, c: 1 } )

75
Query Optimization—Sparse Indexes
MongoDB
Sparse indexes only contain entries for documents that have the indexed
field. Any document that is missing the field is not indexed. The index is
“sparse” because of the missing documents when values are missing.
By contrast, non-sparse indexes contain all documents in a collection,
and store null values for documents that do not contain the indexed field.
To create a sparse index on the comedy field, of the movie collection,
using the following operation in the mongo shell:
db.movie.ensureIndex( { “comedy": 1 }, { sparse: true } )
By default, sparse is false on MongoDB indexes.

76
Query Optimization
MongoDB
MongoDB indexes have the following core features:
• MongoDB defines indexes on a per-collection level.
• You can create indexes on a single field or on multiple fields using a compound index.
• All MongoDB indexes use a B-tree data structure.
• Indexes enhance query performance, often dramatically. However, each index also
incurs some overhead for every write operation.
• Every query, including update operations, uses one and only one index. The query
optimizer selects the index empirically by occasionally running alternate query plans
and by selecting the plan with the best response time for each query type.
You can override the query optimizer using the hint() method.

77
Query Plans
MongoDB
The MongoDB query optimizer processes queries and chooses the most
efficient query plan for a query given the available indexes. The query
system then uses this query plan each time the query runs. The query
optimizer occasionally reevaluates query plans as the content of the
collection changes to ensure optimal query plans.
You can use the explain() method to view statistics about the query plan
for a given query.

78
Query Plans
MongoDB
The $explain operator provides information on the query plan. It returns a
document that describes the process and indexes used to return the
query. This may provide useful insight when attempting to optimize a
query.
db.collection.find()._addSpecial( "$explain", 1 )
db.collection.find( { $query: {}, $explain: 1 } )
You also can specify $explain through the explain() method in the mongo
shell:
db.collection.find().explain()
db.collection.find({customer: ‘ACME’).explain()

79
Query Plans-When do the Query Plans Get Re-Evaluated?
MongoDB
As collections change over time, the query optimizer deletes the query
plan and re-evaluates after any of the following events:
• The collection receives 1,000 write operations.
• The reIndex rebuilds the index.
• You add or drop an index.
• The mongod process restarts.
To display names, and status of collections and indexes of a database:
db.printCollectionStats()

80
Query Coverage
MongoDB
An index “covers” a query if:
• all the fields in the query are part of that index, and
• all the fields returned in the documents that match the query are in the same
index.
• When an index covers a query, the server can both match the query
conditions and return the results using only the index; MongoDB does not
need to look at the documents, only the index, to fulfill the query. Querying
the index can be faster than querying the documents outside of the index.

81
How to find the indexes of a collection
MongoDB
When performing maintenance you may want to check which indexes exist on a
collection. Every index on a collection has a corresponding document in the
system.indexes collection, and you can use standard queries (i.e. find()) to list the
indexes, or in the mongo shell, the getIndexes() method to return a list of the
indexes on a collection, as in the following examples.
To see the indexes for the entire database use:
>db.system.indexes.find()
To see the indexes for a specific collection use:
>db.<collection>.getIndexes()

82
Sharding

83
A View of Mongo Cluster – Sharded Design
A shard is a container that holds a subset of a collection’s data; generally
deployed for write scaling
Uses of sharding
• Data set approaches or exceeds the storage capacity of a single node in your
system.
• The size of your system’s active Working set will soon exceed the capacity of
the maximum amount of RAM for your system.
• System has a large amount of write activity, a single MongoDB instance
cannot write data fast enough to meet demand, and all other approaches have
not reduced contention.
• Balancer balances across shards behind the scenes

84
Read Operations to Sharded Clusters:
Sharded clusters allow you to partition a data set among a cluster of
mongod instances in a way that is nearly transparent to the application.
For a sharded cluster, applications issue operations to one of the mongos
instances associated with the cluster:

85

86
A View of Mongo Cluster – Sharded Cluster
Mongo
Server
- Mongod
hosts
Mongo
Storage
Mongo
Server
-Mongod
Mongo
Server
-Mongod
…….
Secondary
Secondary
Primary
Secondary
Primary
Primary
Secondary
Secondary
Application Server
- Mongos
Application Server
- Mongos
…………
..
Secondary
RS1
RS2
RS3
ConfigS ConfigS ConfigS Keeps
metadata
about
chunk
distribution
s

87
Sharded key selection
A shard key is s key in a collection that is used to distribute
data across different shards.
• { shardCollection: "<database>.<collection>", key: <shardkey> }
• Must have an index on this key.
• Cardinality is the key, higher the cardinality, thinner the
data gets distributed across cluster
• EmpNo, Phone Number etc are good examples.
• Gender is bad key
• Hash key, where good natural key is not available

88
A shard is a container that holds a subset of a collection’s
data
• Need additional servers called - config servers
• A Config Server is a mongodb instance that holds
metadata about the cluster. The metadata
maps chunks to shards.
• Mongo automatically balances shards
• With 2.4 Hash Sharding available
- Hash sharded keys use a hashed index of a single field as
the sharded key to partition data across your sharded cluster.

89
Backup and Restore

90
MongoDB

91
Mongodump
MongoDB
The mongodump utility can back up data by either:
• connecting to a running mongod or mongos instance, or
• accessing data files without an active instance.
The utility can create a backup for an entire server, database or collection,
or can use a query to backup just part of a collection.
When you run mongodump without any arguments, the command
connects to the MongoDB instance on the local system (e.g. 127.0.0.1 or
localhost) on port 27017 and creates a database backup named dump/ in
the current directory.

92
Mongodump
MongoDB
To limit the amount of data included in the database dump, you can
specify --db and --collection as options to the mongodump command. For
example:
mongodump --dbpath /data/db/ --out /data/backup/

93
Point in Time Operation Using Oplogs
MongoDB
Use the --oplog option with mongodump to collect the oplog entries to
build a point-in-time snapshot of a database within a replica set.
With --oplog, mongodump copies all the data from the source database as
well as all of the oplog entries from the beginning of the backup procedure
to until the backup procedure completes.
This backup procedure, in conjunction with mongorestore --oplogReplay,
allows you to restore a backup that reflects a consistent and specific
moment in time.

94
Mongorestore
MongoDB
The mongorestore utility restores a binary backup created by
mongodump. By default, mongorestore looks for a database backup in the
dump/ directory.
The mongorestore utility can restore data either by:
• connecting to a running mongod or mongos directly, or
• writing to a set of MongoDB data files without use of a running
mongod.
The mongorestore utility can restore either an entire database backup or a
subset of the backup.

95
Mongorestore
MongoDB
If you restore to an existing database, mongorestore will only insert into
the existing database, and does not perform updates of any kind. If
existing documents have the same value _id field in the target database
and collection, mongorestore will not overwrite those documents.
Remember the following properties of mongorestore behavior:
• mongorestore recreates indexes recorded by mongodump.
• all operations are inserts, not updates.
• mongorestore does not wait for a response from a mongod to ensure
that the MongoDB process has received or recorded the operation.

96
Mongorestore – Point in Time Oplog Backup
MongoDB
If you created your database dump using the --oplog option to ensure a
point-in-time snapshot, call mongorestore with the --oplogReplay option,
as in the following example:
mongorestore --oplogReplay
You may also consider using the mongorestore --objcheck option to check
the integrity of objects while inserting them into the database, or you may
consider the mongorestore --drop option to drop each collection from the
database before restoring from backups.

97
Mongorestore – Restore a subset of data from a binary database dump
MongoDB
mongorestore also includes the ability to a filter to all input before
inserting it into the new database. Consider the following example:
mongorestore --filter '{"field": 1}'
Here, mongorestore only adds documents to the database from the dump
located in the dump/ folder if the documents have a field name field that
holds a value of 1. Enclose the filter in single quotes (e.g. ') to prevent the
filter from interacting with your shell environment.

98
Student Lab
MongoDB-Review
Student Lab 4

99
Administration and Operations

100
Mongo administration commands examples
MongoDB-Administration
List all administration commands db.listCommands()
Get a list of databases (including size) db.runCommand({listDatabases:1})
Get status of current database db.runCommand({dbStats:1})
mongoDB server build information db.runCommand({buildinfo:1})
server status db.runCommand({serverStatus:1})
usage by collection db.runCommand({top:1})
status of replica set db.runCommand({ReplSetGetStatus:1})
Get the current version of MongoDB
db.version()
mongod --version
Obtain the Operating system, CPU, memory,
OS version used by mongod command db.hostInfo()
Display Names and status of collections and
indexes of a database db.printCollectionstats()

101
Mongo administration commands examples
MongoDB-Administration
Process Default MongoDB TCP Port
mongod 27017
mongos 27017
shard server (mongod --shardsvr) 27018
config server (mongod --configsvr) 27019
web stats page for mongod
add 1000 to mongod port number
(28017, by default)

102
Admin Interface
MongoDB
MongoDB provides an Admin interface that can be used to see basic information about
the MongoDB server such as:
• DB Version
• Sys Info
• UpTime
• ReplicationInfo
• Most recent client requests
• # databases
• Master/Slave
• Log

103
Admin Interface
MongoDB
To see the admin interface, start the database and go
to http://g9t3418.houston.hp.com:21001/ in a web browser. Please note, that this is one
of the GDS DEV servers. (See the infrastructure diagram for all the hosts).
Because of the nature of the information that is provided through this interface, only the
first page is accessible on our secure systems as REST API is disabled.
The mongod process includes a simple REST interface, with no support for
insert/update/remove operations, as a convenience – it is generally used for
monitoring/alerting scripts or administrative tasks. v1.4+: The REST interface is disabled
by default. Use --rest on the mongod command line to enable.
To view this from you own installation you can use:
http://localhost:28017

104
Replication
MongoDB

105
Replication
MongoDB
Why Use Replication?
How many have faced an instance failure?
How many have been woken up from sleep to do a fail-over?
How successful have the failovers been?
How many have experienced issues due to network latency?
Different uses are possible with the use of replicas:
--Normal Processing
--Analytics
--Backup only replica

106
Replication
MongoDB
Why Use Replication?
Operations and Maintenance
- No downtime
- Rolling upgrade/maintenance
- Start with Secondary
- Primary Last

107
Replication
MongoDB
Node 1
(Primary
)
Node 2
(Secondary
)
Node 3
(Secondary
)
Heartbea
t
Replication

108
Replica Set - Failure
MongoDB
Node 1
(Primary
)
Node 2
(Secondary
)
Node 3
(Secondary
)
X
Primary
Election

109
Replica Set - Recovered
MongoDB
Node 1
(Secondary
)
Node 2
(Primary)
Node 3
(Secondary
)
Heartbea
t
Replication

110
Managing Replication
MongoDB
To see the replica set configuration object you can run the following command in mongo
shell
 rs.conf() or db.system.replset.find()
This will display the id for the replica set and the members including their IDs, host name
and ports.

111
MongoDB
You can check last heartbeat between nodes, priority and state by clicking on Replica Set
Status option. As well, recent replset log activity will be shown here.

112
MongoDB
To check replication status you can connect to one of the nodes and raise the following
command
>rs.status()
>db.printSlaveReplicationInfo()
The output reflects the current status of the replica set, using data derived from the
heartbeat packets sent by the other members of the replica set

113
MongoDB
Removing Nodes
To remove a node from the replica set you will
have to raise the following command
rs.remove(hostname)
Note: Before running the rs.remove() operation,
it’s a good practice that you shutdown the
replica set member that you are removing.

114
MongoDB
Adding nodes
To add a new node you must first ensure it is
part of the replset in the configuration file in the
#Replication Options tag.
# Replication Options
replSet = rs0
After that from primary we will just add the new
node to the configuration file by executing the
following command
rs.add("g9t3417:20001")

115
MongoDB
Accessing the replica set:
To access a specific host on a specific port:
Mongo –port 27017
To always connect to the primary Mongo DB in a replicat set named rs0 that contains 3
members, host 1, host2, and host 3 on the same port:

116
Student Lab
MongoDB
Student Lab 5

117
Student Lab
MongoDB
Student Lab 6

118
Topics
MongoDB
 Security
 Security Management
 Vulnerability Notification to 10gen
 Analyze Performance of Database Operations
 MongoDB profiler
 Current operation method

119
Security Management
MongoDB
Strategies for Reduce Security Risk
 Limit accessibility to MongoDB servers
• Firewall - Network Filter, control incoming traffic on specific port to specific
system
• Limits the network interfaces on which MongoDB programs will listen for
incoming connections.
 Change default port numbers of the mongoDB instances (20001)
 Disable REST API (default), restrict access for HTTP admin Console
 Authentication for access to MongoDB instances
 Application development best practices including validating input, managing
sessions, and application-level access control

120
Authentication for access to MongoDB instances
MongoDB
 Authentication is disabled by default
 Parameter to enable authentication
• single MongoDB instance : auth=true
• multiple servers (replica sets, sharded clusters) :
keyFile=“/path/of/the/keyFile”
 MongoDB supports role-based access to databases
• Roles: R,R/W,UserAdmin,dbAdmin, ClusertAdmin, ‘Any’
Database roles
 User credentials stored in a database’s system.users collection

121
db.addUser() Definition
Field Type Description
user string
The username for a new database
user.
roles array An array of user roles.
pwd hash
The pwd field and the userSource field are
mutually exclusive.
userSource (new to 2.4) string
Optional:The database that contains he
credentials for the user. the userSource
field are mutually exclusive.
otherDBRoles (new to 2.4) document
Optional. Roles this user has on other
databases. Only support in Admin
database

122
Authentication for access to MongoDB instances
MongoDB
Procedure to enable MongoDB authentication
 Start the mongod or mongos instance without the auth or keyFile setting
 Create the administrator user with “userAdminAnyDatabase” role in admin database
 Generate keyFile on one of the replica servers
• openssl rand -base64 741 -out keyFile ; chmod 400 keyFile
 Copy to rest of the replica servers; ensure it is owner read only permission
 Add keyFile parameter (auth=true) in the mongod.config file
 Re-start the mongod or mongos instance with the auth or keyFile setting

123
Demonstrations of the delegated authentication credential and otherDBRoles
MongoDB
use appadmindb
db.addUser( { user:"appadmin", pwd:"D2xzigasAb", roles: [ "userAdmin" ,"dbAdmin"]})
use admin
db.addUser(
{
user: "appadmin",
userSource: "appadmindb",
roles: [ "read"],
otherDBRoles: {
app1db: [ "userAdmin","readWrite" ],
app2db: [ "userAdmin","readWrite" ]
}
}
)

124
MongoDB Security
MongoDB
Create an MongoDB administrator
Login to the local host running the mongod
Add user in the admin DB
>user admin db.addUser("admin", "some_secret")
To enable MongoDB security, restart mongod with
C:mongodbbinmongod --auth
To authenticate yourself in MongoDB
>db.auth("admin", "some_secret")
If --auth is on, you can get into a mongo session but all operations will fail unless
you authenticate yourself

125
MongoDB Security
MongoDB
Security information for a specific DB is stored in the collection: system.users
To find users of a MongoDB DB
>use myDB db.system.users.find()
Add a new user to myDB
>user myDB db.addUser("some_user", "some_secret")
To create a read only MongoDB user
>db.addUser("some_read_only_user", "some_secret", true)
Change MongoDB user password
>db.addUser("some_user", "some_new_secret")
Remove a MongoDB user
>db.removeUser("some_user")

126
Vulnerability Notification to 10gen
MongoDB
To report MongoDB Security or any issue to 10gen
• https://jira.mongodb.org/secure/Dashboard.jspa
• Send email to security@10gen.com
• 10gen responds to vulnerability notification within 48 hours

127
Analyze Performance of Database Operations – Profiler
MongoDB
 Database profiler captures data about read/write operation, cursors, database
commands
>db.setProfilingLevel(level, slowms)

128
Analyze Performance of Database Operations – Profiler
MongoDB
 Set threshold in milliseconds for profiling using the slowms option
 System.profile collection – capped collection (1MBytes)
• No index; write fast
 Enable profile per database or per-instance basis

129
Analyze Performance of Database Operations – currentOp()
MongoDB
 Enable profle in the instance level : mongod --profile=1 --slowms=15
 Example of setting profile for slow operation by threshold with 20 millsecs
db.setProfilingLevel(1,20)
 Check profiling level: db.getProfilingLevel() , db.getProfilingStatus()
 Check system.profile collection size: db.system.profile.stats() or
db.system.namespaces.find()

130
MongoDB
 Enable profle in the instance level : mongod --profile=1 --slowms=15
 Example of setting profile for slow operation by threshold with 20 millsecs
db.setProfilingLevel(1,20)
 Check profiling level: db.getProfilingLevel() , db.getProfilingStatus()
 Check system.profile collection size: db.system.profile.stats() or
db.system.namespaces.find()

131
MongoDB
By default, mongod records all “slow” queries to its log, as defined by
slowms. Unlike log data, the data in system.profile does not persist
between mongod restarts.
Note:
Because the database profiler can negatively impact performance, only
enable profiling for strategic intervals and as minimally as possible on
production systems.
You may enable profiling on a per-mongod basis. This setting will not
propagate across a replica set or sharded cluster.

132
Resizing system.profile size
MongoDB
1. Disable profiling
>db.setProfilingLevel(0)
2. Drop system.profile collection
>db.system.profile.drop()
3. Create a new system.profile collection
>db.createCollection(“system.profile”, {capped: true,
size:4000000} )
4. Re-enable profiling
>db.setProfilingLevel(1)
5. Check the profiling Status
>db.getProfilingStatus()

133
MongoDB
{
"ts" : ISODate("2012-12-
10T19:31:28.977Z"),
"op" : "update",
"ns" : "social.users",
"query" : {
"name" : "jane"
},
"updateobj" : {
"$set" : {
"likes" : [
"basketball",
"trekking"
]
}
},
"nscanned" : 8,
"moved" : true,
"nmoved" : 1,
"nupdated" : 1,
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(0),
"w" : NumberLong(258)
},
"timeAcquiringMicros" : {
"r" : NumberLong(0),
"w" : NumberLong(7)
}
},
"millis" : 0,
"client" : "127.0.0.1",
"user" : “Appuser"
}
Profiling output

134
View Profiler Data
MongoDB
 Display the most recent 10 log entries in the system.profile collection
• db.system.profile.find().limit(10).sort( { ts : -1 } ).pretty()
 Display all operations except command operations
• db.system.profile.find( { op: { $ne : 'command' } } ).pretty()
 Display data for a specific collection
• db.system.profile.find( { ns : ‘/mydb.test/' } ).pretty()
 Disaply operatons slower than 10 milliseconds
• db.system.profile.find( { millis : { $gt : 10 } } ).pretty()
 information from a certain time range
• db.system.profile.find( { ts : { $gt : new ISODate("2012-12-09T03:00:00Z") , $lt : new
ISODate("2012-12-09T03:40:00Z") } } ).pretty()
 Display the 5 most recent operation that took at least 1 millisecond
• Show profile

135
MongoDB
 Report in-progress operations for the database instance
 db.currentOp() vs. db.currentOp(true)
 Only for users with administrative privileges
 db.killOp() in conjunction with the opid filed to terminate a current running
operation.

136
MongoDB
{
"inprog": [
{
"opid" : 3434473,
"active" : <boolean>,
"secs_running" : 0,
"op" : "<operation>",
"ns" : "<database>.<collection>",
"query" : {
},
"client" : "<host>:<outgoing>",
"desc" : "conn57683",
"threadId" : "0x7f04a637b700",
"connectionId" : 57683,
“locks" : {
"^" : "w",
"^local" : "W",
"^<database>" : "W"
},
"waitingForLock" : false,
"msg": "<string>"
"numYields" : 0,
"progress" : {
"done" : <number>,
"total" : <number>
}
"lockStats" : {
"timeLockedMicros" : {
"R" : NumberLong(),
"W" : NumberLong(),
"r" : NumberLong(),
"w" : NumberLong()
},
"timeAcquiringMicros" : {
"R" : NumberLong(),
"W" : NumberLong(),
"r" : NumberLong(),
"w" : NumberLong()
}
}
},
]
}

137
MongoDB
• All pending write operations
db.currentOp().inprog.forEach(
function(d)
{ if(d.waitingForLock &&
d.lockType != "read")
printjson(d)
})
• Active write operations
db.currentOp().inprog.forEach
(
function(d)
{ if(d.active &&
d.lockType = “write")
printjson(d)
})
• All active read operations
db.currentOp().inprog.forEach
(
function(d)
{ if(d.active &&
d.lockType != “read")
printjson(d)
})

138
Monitoring utilities and tools
Mongo OS tools
 Mongotop
 mongostat
tools in mongodb
 profiler
(system.profile)
 currentOp()
 db.serverStatus()
 db.stats()
 db.coll.stats()
 rs.status()
 sh.status()
Web Tools
 REST admin interface
 MMS
 Applications
Manager
Statistics and introspection tools

139
Student Lab
Student Lab 7

140
mongotop
mongotop tracks and reports the current read and write activity of a MongoDB
instance, and reports these statistics on a per collection basis. Use mongotop to
check if your database activity and use match your expectations.
mongostat
mongostat captures and returns the counts of database operations by type (e.g.
insert, query, update, delete, etc.). These counts report on the load distribution
on the server.
Use mongostat to understand the distribution of operation types and to inform
capacity planning
Administration Monitoring tools

141
Support – Entering Cases
http://ent141.sharepoint.hp.com/teams/simit-gds-
engineering/NoSQL/Shared%20Documents/Vendor%20Support/10Gen/G
DS%2010Gen%20Support%20Process%20Flow.pdf

142
Support – Obtaining mongod Version Information
From mongoshell:
>db.version()
Command Line:
>mongod –version
mongo shell version:
>mongo --version

143
Student Lab
Student Lab 8

144
NoSQL-Humor

Thank You
Q & A

Mongo db operations_v2

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Mongo db operations_v2

Similaire à Mongo db operations_v2 (20)

Dernier

Dernier (20)

Mongo db operations_v2

Notes de l'éditeur