1. 1 2 APR IL , 2018
# M D B l o c a l
How to leverage
what’s new in MongoDB
3.6
2. # M D B l o c a l
Maxime Beugnet
Developer Advocate EMEA @MongoDB Paris
Twitter : @MBeugnet
Github : MaBeuLux88
Email : maxime@mongodb.com
Meetups
Hackathons
Workshops
Conferences
7. # M D B l o c a l# M D B l o c a l
updatereplaceinsertdeleteMongoDB Application Action
8. # M D B l o c a l# M D B l o c a l
db.coll.watch()
db.coll.watch([{$match: {operationType: “insert”}}])
db.coll.watch([], {fullDocument:“updateLookup”})
db.coll.watch([], {resumeAfter:<cachedResumeToken>})
Change streams
• $match
• $project
• $addFields
• $replaceRoot
• $redact
9. # M D B l o c a l# M D B l o c a l
cursor = db.users.watch([ ], {"fullDocument":"updateLookup"});
while ( !cursor.isExhausted() ) {
if ( cursor.hasNext() ) {
print(tojson(cursor.next()));
}
}
Change streams => cursors
10. # M D B l o c a l# M D B l o c a l
•Resumable
•Targeted
Changes
•Total ordering
•Durability
•Security
•Ease of use
•Idempotence
Characteristics of Change Streams
11. # M D B l o c a l
EXPRESSIBILITY
More expressive query language
Array updates
12. # M D B l o c a l# M D B l o c a l
db.products.find( {
$expr: {
$gt: [ "$currRating" , "$prevRating" ]
}
} )
Comparing Fields within a document
13. # M D B l o c a l# M D B l o c a l
db.products.find( {
$expr: {
$gt: [ {$subtract:
["$currRating" ,"$prevRating"] }, 2]
}
} )
Comparing Fields within a document
14. # M D B l o c a l# M D B l o c a l
Comparing Fields within a document
db.supplies.find( {
$expr: {
$lt:[ {
$cond: {
if: { $gte: ["$qty", 100] },
then: { $divide: ["$price", 2] },
else: { $divide: ["$price", 4] }
}
}, 5 ]
}
})
15. # M D B l o c a l# M D B l o c a l
Expressive Array updates
Update all matching
items in an array
Match Nested Arrays
16. # M D B l o c a l# M D B l o c a l
Expressive Array updates
MongoDB V3.4
• $
• $addToSet
• $pop
• $pull
• $push
• $pullAll
{ _id: 1,
grades: [
{ exam: 80, quizz: 75},
{ exam: 85, quizz: 90}
]
}
db.students.update(
{ _id: 1, "grades.exam": 85 },
{ $set: { "grades.$.quizz" : 95 } }
)
17. # M D B l o c a l# M D B l o c a l
Expressive Array updates
MongoDB V3.6
• $[<id>]
• $[]
{ _id: 1,
grades: [
{ exam: 80, quizz: 75},
{ exam: 85, quizz: 90}
]
}
db.students.update( { },
{ $set: { "grades.$[elem].quizz" : 100 } },
{ multi: true, arrayFilters: [ { "elem.exam": { $gte: 80 } } ] }
)
db.students.update( { },
{ $set: { "grades.$[].quizz" : 100 } },
{ multi: true }
)
18. # M D B l o c a l# M D B l o c a l
Expressive Array updates
{ _id: 1, name: "X",
Medications: [
{ id: 23, name: "DrugName99", Sched:"I", Rx: [
{ id:13, Qty: 60, started: "2009-01-01" },
{ id:77, Qty: 30, started: "2011-02-01", current:true }
]},
{ id: 41, name: "OtherDrugName", Sched: "II, Rx: […] },
{ id: 59, name: "ThirdDrug", Sched:"I", Rx:[
{ id:994, Qty: 60, started: "2012-01-01", current:true },
{ id:1034, Qty: 90, started: "2007-02-01" }
]},
"lastVisit": ISODate("2017-01-22T13:01:13.000Z")
}
19. # M D B l o c a l# M D B l o c a l
Expressive Array updates
{ _id: 1, name: "X",
Medications: [
{ id: 23, Sched:"I", Rx: [
{ id:13, Qty: 60},
{ id:77, Qty: 30, current:true}
]},
{ id: 41, Sched: "II" },
{ id: 59, Sched:"I", Rx:[
{ id:994, Qty: 60, current:true},
{ id:1034, Qty: 90}
]}
]
}
db.patientRx.update(
{},
{"$set": {"Medications.$[med].Rx.$[rx].Qty": 20}},
{ "multi": true,
"arrayFilters": [
{"med.Sched": "I"},
{"rx.current":true, "rx.Qty":{$gt:30}}
]}
)
20. # M D B l o c a l# M D B l o c a l
Expressive Array updates
{ _id: 1, name: "X",
Medications: [
{ id: 23, Sched:"I", Rx: [
{ id:13, Qty: 60},
{ id:77, Qty: 30, current:true}
]},
{ id: 41, Sched: "II" },
{ id: 59, Sched:"I", Rx:[
{ id:994, Qty: 60, current:true},
{ id:1034, Qty: 90}
]}
}
db.patientRx.update(
{"Medications": {"$elemMatch": {
"Sched": "II",
"Rx": {"$elemMatch": {
"current": true,
"Qty": {"$gt": 30}
}}
}}},
{"$set": {"Medications.$[med].Rx.$[rx].Qty": 20}},
{ "multi": true,
"arrayFilters": [
{"med.Sched": "I"},
{"rx.current":true, "rx.Qty":{$gt:30}}
]}
)
21. # M D B l o c a l
ANALYTICS
New operators
Timezone support
Expressive $lookup
R Driver
BI Connector
22. # M D B l o c a l# M D B l o c a l
New operators
23. # M D B l o c a l# M D B l o c a l
New operators : $arrayToObject
{ "_id" : 1, dimensions: [ { "k": "l", "v": 25} , { "k": "w", "v": 10 }, { "k": "uom", "v": "cm" } ] }
{ "_id" : 2, dimensions: [ [ "l", 50 ], [ "w", 25 ], [ "uom", "cm" ] ] }
{ "_id" : 3, dimensions: [ [ "l", 50 ], [ "l", 25 ], [ "l", "cm" ] ] }
db.inventory.aggregate( [ { $project: { dimensions: { $arrayToObject: "$dimensions" } } } ] )
{ "_id" : 1, "dimensions" : { "l" : 25, "w" : 10, "uom" : "cm" } }
{ "_id" : 2, "dimensions" : { "l" : 50, "w" : 25, "uom" : "cm" } }
{ "_id" : 3, "dimensions" : { "l" : 50 } }
24. # M D B l o c a l# M D B l o c a l
New operators : $objectToArray
{ $objectToArray: { item: "foo",
qty: 25,
size: { len: 25, w: 10, uom: "cm" }
} }
[ { "k" : "item", "v" : "foo" },
{ "k" : "qty", "v" : 25 },
{ "k" : "size", "v" : { "len" : 25, "w" : 10, "uom" : "cm" } } ]
25. # M D B l o c a l# M D B l o c a l
New operators : $mergeObjects
{ $mergeObjects: [ { a: 1 },
{ a: 2, b: 2 },
{ a: 3, b: null, c: 3 } ] }
{ a: 3, b: null, c: 3 }
26. # M D B l o c a l# M D B l o c a l
New operators : $dateFromString
{ _id: 1, date: "2017-02-08T12:10:40.787, timezone: "America/New_York" },
{ _id: 2, date: "2017-02-08" , timezone: "-05:00" },
{ _id: 3 }
db.logmessages.aggregate( [
{ $project: {
date: {
$dateFromString: { dateString: '$date', timezone: '$timezone’ }
}
}
} ] )
{ "_id" : 1, "date" : ISODate("2017-02-08T17:10:40.787Z") }
{ "_id" : 2, "date" : ISODate("2017-02-08T05:00:00Z") }
{ "_id" : 3, "date" : null }
27. # M D B l o c a l# M D B l o c a l
New operators : $dateFromParts
db.sales.aggregate([ {
$project: {
date: { $dateFromParts: { 'year' : 2017, 'month' : 2, 'day': 8, 'hour' : 12 } },
date_iso: { $dateFromParts: { 'isoWeekYear' : 2017, 'isoWeek' : 6, 'isoDayOfWeek' : 3, 'hour' : 12 } },
date_timezone: { $dateFromParts: { 'year' : 2016, 'month' : 12, 'day' : 31, 'hour' : 23, 'minute' : 46,
'second' : 12, 'timezone' : 'America/New_York’ } }
}
}])
{
"_id" : 1,
"date" : ISODate("2017-02-08T12:00:00Z"),
"date_iso" : ISODate("2017-02-08T12:00:00Z"),
"date_timezone" : ISODate("2017-01-01T04:46:12Z")
}
28. # M D B l o c a l# M D B l o c a l
New operators : $dateToParts
db.sales.aggregate([ {
$project: {
date: { $dateToParts: { date: "$date" } },
date_iso: { $dateToParts: { date: "$date", iso8601: true } },
date_timezone: { $dateToParts: { date: "$date", timezone: "America/New_York" } }
}
}])
{
"_id" : 2,
"date" : { "year" : 2017, "month" : 1, "day" : 1, "hour" : 1,
"minute" : 29, "second" : 9, "millisecond" : 123 },
"date_iso" : { "isoWeekYear" : 2016, "isoWeek" : 52, "isoDayOfWeek" : 7, "hour" : 1,
"minute" : 29, "second" : 9, "millisecond" : 123 },
"date_timezone" : { "year" : 2016, "month" : 12, "day" : 31, "hour" : 20,
"minute" : 29, "second" : 9, "millisecond" : 123 }
}
29. # M D B l o c a l# M D B l o c a l
Timezone support
$dayOfYear, $dayOfMonth, $dayOfWeek, $year, $month, $week, $hour,
$minute, $second, $millisecond, $isoDayOfWeek, $isoWeek, $isoWeekYear
{ $operator: { date: <isoDateExpression>, timezone: <tzExpression> } }
30. # M D B l o c a l# M D B l o c a l
>db.collection.aggregate({$lookup:{
from:"coll2",
localField:"x",
foreignField:"y",
as:"coll2details"
}}])
More expressive $lookup
>db.collection.aggregate({$lookup:{
from:"coll2",
let: {x: "$x"},
pipeline: [
{$match: {$expr:
{$eq: [ "$y", "$$x"] }
} }
]
as:"coll2details"
}}])
Prior to 3.6 New in 3.6
31. # M D B l o c a l# M D B l o c a l
$lookup example
orders:
{
line_items : [
{ id: 123,
title : “USB Battery”,
price: 15.0 },
{ id: 512,
title : “Hip T-shirt”,
price : 45.0 }
]
}
db.orders.aggregate([
{$unwind: "$line_items"},
{$lookup:{
from: "reviews",
let: {p_id: "$line_items.id"},
pipeline: [
{$match: {$expr: {$eq: ["$p_id","$$p_id"]}}},
{$group: { _id: 1, rating: {$avg:"$rating"}}}
], as: "avgRating" }
}
])
32. # M D B l o c a l# M D B l o c a l
Recommended MongoDB R driver for data scientists, developers & statisticians
• MongoDB read & write concerns to control data consistency & durability
• Idiomatic, native language access to the database
• Data security with enterprise authentication mechanisms
• BSON data type support, e.g., Decimal 128 for high precision scientific &
financial analysis
R Driver for MongoDB
33. # M D B l o c a l# M D B l o c a l
MongoDB Connector for BI
+ many more
34. # M D B l o c a l# M D B l o c a l
MongoDB Connector for BI
• Faster
• Takes advantage of 3.6 expressive Lookup to push more
computation into the database
• Supports Show Status function to enable deeper performance
optimization
• Simpler
• Lifecycle management with Ops Manager
• Schema sampling and mapping now managed by the mongosqld
process, rather than separate utility (mongodrdl)
• Authenticate via client-side plugins, rather than managing TLS
certificates. Kerberos support added
35. # M D B l o c a l
APPLICATIONS AVAILABILITY
Retryable writes
DNS seed list
Read Concern: Available & Tunable
consistency
36. # M D B l o c a l# M D B l o c a l
Retryable writes
PS
S
Application P
write successful :D
write unsuccessful :O
37. # M D B l o c a l# M D B l o c a l
Retryable writes
Application MongoDB
{ _id: 1,
team : "Manchester United",
gameID: "game123",
coach : "José Mourinho",
score : 2,
league: "Premier League"
}
{ _id: 2,
team : "Chelsea",
gameID: "game456",
coach : "Antonio Conte",
score : 2,
league: "Premier League"
}
db.games.update(
{gameID:"game123"},
{$inc: {score:1}}
)
db.games.update(
{team: "Chelsea"},
{$set: {coach:"Zidane"}}
)
38. # M D B l o c a l# M D B l o c a l
• Automatic Drivers logic
• Network errors
• Elections
• NOT for logic errors
• Safe
• For both non-idempotent and idempotent writes
• NOT for multi: true
Characteristics of Retryable Writes
39. # M D B l o c a l# M D B l o c a l
Retryable writes
uri = "mongodb://example.com:27017/?retryWrites=true"
client = MongoClient(uri)
database = client.database
collection = database.collection
40. # M D B l o c a l# M D B l o c a l
DNS seed list
"mongodb://example1.com:27017,example2.com:27017,example3.com:27017
"mongodb+srv://my-dns-server.mongodb.com
"mongodb://example1.com:27017
41. # M D B l o c a l# M D B l o c a l
Tunable consistency
Availability Consistencymagic
42. # M D B l o c a l# M D B l o c a l
• readConcern (Read Isolation)
• Local
• Majority
• Linearizable
• writeConcern (write acknowledgement)
• <number> (i.e. 1)
• Majority
• Tag
What is readConcern and writeConcern?
43. # M D B l o c a l# M D B l o c a l
Tunable consistency
mongos
3 1
2
Shard 1 Shard 2 Shard 3
readConcern: available
44. # M D B l o c a l# M D B l o c a l
Tunable consistency
readConcern: available is equivalent to readConcern: local on replica sets
you can pass readConcern: available to a primary in a sharded cluster
readConcern: available is default for secondaries in a sharded cluster
Secondaries in sharded clusters will now respect readConcern : local for safe reads
45. # M D B l o c a l# M D B l o c a l
Tunable consistency
mongos
3 1
2
Shard 1 Shard 2 Shard 3
Global Logical Clock
wait until cluster time has moved past the last time you saw
Causal consistency:
guarantees monotonic,
logically consistent reads
from any replica node in
the same user session
46. # M D B l o c a l# M D B l o c a l
Tunable consistency
//start client session, which is causally consistent by default
try (ClientSession session =
client.startSession(ClientSessionOptions.builder().build())) {
//Run causally related operations within the session
collection.insertOne(session, ... );
collection.updateOne(session, ...);
try (MongoCursor<Document> cursor =
collection.find(session).filter(...).iterator()) {
while (cursor.hasNext()) {
Document cur = cursor.next();
}
}
47. # M D B l o c a l
OPERATIONS
JSON Schema
Network security
Session management
End-to-end Compression
48. # M D B l o c a l# M D B l o c a l
Network Security
Bind to localhost by Default
IP Whitelisting
• Associate IP addresses or ranges with roles in auth
• If a the IP restrictions are not met, fail to authenticate
• Able to restrict __system user to authenticate from only cluster
nodes
49. # M D B l o c a l# M D B l o c a l
Network Security
192.168.1.25
Application
Application
System Administrator
192.168.1.48
172.16.4.13
172.16.4.88
172.33.20.62
172.33.20.11
Restrict each user’s
authentication based on:
• Client IP Address Range
and/or
• Server IP Listen Address
50. # M D B l o c a l# M D B l o c a l
JSON Schema
Enforces strict schema structure over a complete collection
for data governance & quality
• Builds on document validation introduced by restricting new content that
can be added to a document
• Enforces presence, type, and values for document content, including
nested array
• Simplifies application logic
Tunable: enforce document structure, log warnings, or allow
complete schema flexibility
Queryable: identify all existing documents that do not comply
51. # M D B l o c a l# M D B l o c a l
JSON Schema
db.createCollection( "orders",
{validator: {$jsonSchema:
{properties:
{line_items:
{type: "array",
items:
{properties:
{title: {type: "string"},
price: {type: "number", minimum: 0.0} },
required: ["_id", "title", "price"],
additionalProperties: false}}},
required: ["line_items"]}}}
)
http://json-schema.org/
52. # M D B l o c a l# M D B l o c a l
Session Management
Server sessions
• Every Operation is wrapped in a server session by default in 3.6
• killSessions by user
Client sessions
• Every operation within a defined client session have causal consistency
• Not by default, must be explicitly defined
53. # M D B l o c a l# M D B l o c a l
MongoDB 3.6 adds compression
of wire protocol traffic between
client and database
• Up to 80% bandwidth savings
MongoDB End to End
Compression
• Wire protocol
• Intra-cluster
• Indexes in memory
• Storage
Application
MongoDB Primary
Replica
Wire Protocol
Compression
MongoDB Secondary Replica
Single ViewMongoDB Secondary Replica
Single ViewMongoDB Secondary Replica
Single ViewMongoDB Secondary Replica
Single ViewMongoDB Secondary Replica
MongoDB Secondary Replica
Intra-Cluster
Compression
Compression of
Data on Disk
Compression of
Indexes in Memory
End to End
Compression