Does exploring data excite you? Do you use Python or R as your language of choice for data analysis? Does your job title include the term Data Analyst? If you answered yes to any of those questions, then the Exploring Your MongoDB Data with Pirates and Snakes is the session for you! MongoDB Developer Advocate Ken Alger will show the suggested method for using dataframe structures in R and Python with your MongoDB data. He’ll show the code for best practices in both languages to move your array based MongoDB data into the popular fast, flexible, and expressive dataframes used for data analysis in these prominent programming languages.

1. Ken W. Alger – Developer Advocate, MongoDB
Exploring your MongoDB Data with Pirates (R) and
Snakes (Python)
@kenwalger

2. Ken W. Alger
Developer Advocate

3. Overview
§ The Document Model
§ Data Frames
§ R vs. Python
§ MongoDB to Data Frames
§ Array Consumption
§ The Power of MongoDB

4. The Document Model

5. Document Model Features
Naturally maps objects to
code using JSON
Represent data of any
structure. Our data model is
very flexible.
Strongly typed for ease of
processing. We support
over twenty binary
encoded JSON data types.

6. Document Model for Analytics
Flexibility helps with
feature engineering by
allowing for
experimentation and the
picking of features
iteratively.
For Deep Learning the
flexibility allows for faster
iteration.
Pre-filtering of data with
aggregation framework.

7. Flexibility is Great if You're a Python or a Pirate…

8. Pirate Ships

9. Island Packet 31

10. Götheborg & Batavia

11. Sail Data
{
"Name": "Götheborg",
"Year Completed": 1738,
"Sail Area": {
"Lateen mizzen": 160,
"Mizzen topsail": 2500,
…
},
…
}
{
"Name": "Batavia",
"Year Completed": 1628,
"Sail Area": {
"Lateen mizzen": 250,
”Main topsail": 3500,
…
},
…
}

12. …but some tasks require data that's rigidly structured.

13. Data Frames

14. Data Frame
A data frame is a list of vectors, factors,
and/or matrices all having the same length
(number of rows in the case of matrices).
Used for storing data tables.

15. Data Frame Data
Length (ft) Year Completed Displacement Sail Area (sq. ft)
Batavia 186 1628 1200 33000
Cutty Sark 280 1869 2100 32000
Götheborg 190 1738 NaN 21140
HMS Endeavor 97.75 1764 NaN 29889
Kruzenshtern 375 1926 3064 NaN
HMS Victory 227.5 1765 3500 58556

16. Distributed Data Frames
Big data distributed across clusters.
Use tools like:

17. R vs. Python

18. R Data Frame
R dataframe is more or less built into the
language.
More functional than Python.
More statistical support in general.

19. Python Data Frame
More object-oriented.
Relies on packages (pandas, numpy, scikit-
learn)
As a language it’s great for additional tasks
along side of analytics.

20. Language Usage

21. MongoDB to Data Frames

22. Sail Data
{
"Lateen mizzen": 120,
"Mizzen topsail": 400,
"Mainsail": 2500,
…
}
{
"Lateen mizzen": 120,
"Main topgallant": 600,
"Mainsail": 3500,
…
}

23. library(mongolite)
connection <-
mongo(collection = "sails",
db="ships",
url="mongodb://localhost”)
sails <- data.frame(
connection$find()
)
MongoDB Data to Data Frames
import pandas as pd
from pymongo import
MongoClient
client =
MongoClient('localhost',
27017)
db = client.ships
df = pd.DataFrame(list
(db.sails.find())
)

24. Data Frame
Lateen mizzen Main topgallant Mainsail Mizzen topsail _id
0 120.0 NaN 2500 400.0 5cf53129984ae2b600701611
1 120.0 600.0 3500 NaN 5cf53141984ae2b600701612
Lateen mizzen Mizzen topsail Mainsail Main topgallant
1 120 400 2500 NA
2 120 NA 3500 600

25. Array Consumption

26. Array Data
Pattern 1 – Arrays of Arrays
sail_area = [
[160, 2500, 1600, 450],
[250, 3500, 2800, 575],
[120, 3500, 295]
]
0 1 2 3
0 160 2500 1600 450
1 250 3500 2800 575
2 120 3500 295 NaN
Resulting Data Frame
pd.DataFrame(sail_area)
-or-
data.frame(sail_area)
pd.DataFrame(list(db.sail_area.find()))
-or-
data.frame(connection$find())

27. Array Data
Pattern 2
[
{"lateen mizzen": 160, "mail topsail": 2500, "mainsail": 1600, "main topgallant": 450},
{"lateen mizzen": 250, "mail topsail": 3500, "mainsail": 2800, "main topgallant": 575},
{"lateen mizzen": 120, "mailsail": 3500, "jib": 450},
]
_id jib lateen mizzen mail topsail mainsail main topgallant
0 5cf5607b984ae2b600701613 NaN 160.0 2500.0 1600.0 450.0
1 5cf5609f984ae2b600701614 NaN 250.0 3500.0 2800.0 575.0
2 5cf560d4984ae2b600701615 450.0 120.0 NaN 3500.0 NaN
Resulting Data Frame

28. Array Data
Pattern 3
[
{"area": [160, 2500, 1600, 450]},
{"area": [250, 3500, 2800, 575]},
{"area": [120, 3500, 295]}
]
_id area
0 5cf57574984ae2b600701623 [160.0, 2500.0, 1600.0, 450.0]
1 5cf57584984ae2b600701624 [250.0, 3500.0, 2800.0, 575.0]
2 5cf5758f984ae2b600701625 [120.0, 3500.0, 295.0]
Resulting Data Frame

29. Are our hopes lost?

30. Moving Data from MongoDB Arrays

31. Array Data
[
{"name": "Batavia", "area": [160, 2500, 1600, 450]},
{"name": "Götheberg", "area": [250, 3500, 2800, 575]},
{"name": "HMS Endeavor", "area": [120, 3500, 295]}
]

32. library(mongolite)
connection <- mongo(collection =
"sails", db="ships",
url="mongodb://localhost”)
sails <-
data.frame(connection$find())
Working with MongoDB Arrays
import pandas as pd
from pymongo import MongoClient
client = MongoClient('localhost',
27017)
db = client.ships
values = []
for ship in db.sailareas.find():
values.append(ship['area'])
print(pd.DataFrame(values))

33. Array Data
[
{"name": "Batavia", "area": [160, 2500, 1600, 450]},
{"name": "Götheberg", "area": [250, 3500, 2800, 575]},
{"name": "HMS Endeavor", "area": [120, 3500, 295]}
]
Resulting Data Frame
0 1 2 3
0 160.0 2500.0 1600.0 450.0
1 250.0 3500.0 2800.0 575.0
2 120.0 3500.0 95.0 NaN

34. library(mongolite)
connection <- mongo(collection =
"sails", db="ships",
url="mongodb://localhost”)
sails <-
data.frame(connection$find())
Working with MongoDB Arrays
import pandas as pd
from pymongo import MongoClient
client = MongoClient('localhost',
27017)
db = client.ships
values = []
seriesLabels = []
for ship in db.sailareas.find():
values.append(ship['area'])
seriesLabels.append(ship['name'])
print(pd.DataFrame(values,
index=seriesLabels))

35. Array Data
[
{"name": "Batavia", "area": [160, 2500, 1600, 450]},
{"name": "Götheberg", "area": [250, 3500, 2800, 575]},
{"name": "HMS Endeavor", "area": [120, 3500, 295]}
]
Resulting Data Frame
0 1 2 3
Batavia 160.0 2500.0 1600.0 450.0
Götheborg 250.0 3500.0 2800.0 575.0
HMS Endeavor 120.0 3500.0 95.0 NaN

36. The Power of
MongoDB
Aggregation Framework

37. Aggregation Framework
• Pre-filter and/or pre-aggregate data on the server before
moving it across the network.
• Reduces the amount of data in the data frame.
• Improves performance.

38. Sample Data
Country
Year
Completed
Displacement
Individual Sail Areas
(sq. ft)
Batavia NLD 1628 1200 [292, 2012, 990, 550, 403, 642, 1056, ...]
Cutty Sark GBR 1869 2100 [2408, 866, 155, 2041, 518, 1675, …]
Götheborg SWE 1738 NaN [315, 614, 314, 2451, 2096, 2477, …]
HMS
Endeavor
GBR 1764 NaN [1060, 2089, 1101, 420, 2320, 2245]
Kruzenshtern DEU 1926 3064 [1476, 1352, 2383, 1100, 1807, 448, 2415]
HMS Victory GBR 1765 3500 [1310, 2445, 1327, 1668, 2098, 2179, …]

39. from datetime import datetime, timezone
values = []
seriesLabels = []
for ship in db.ships.aggregate [
{
'$match': {
'year_completed': {
'$gte': datetime(1571, 1, 1, 0, 0, 0, tzinfo=timezone.utc)
}
}
},
Aggregation Pipeline 1

40. {
'$match': {
'year_completed': {
'$lt': datetime(1862, 1, 1, 0, 0, 0, tzinfo=timezone.utc)
}
}
}, {
'$match': {
'country_of_origin': {
'$ne': 'USA'
}
}
},
Aggregation Pipeline 2

41. {
'$project': {
'name': 1,
'country_of_origin': 1,
'sail_areas': 1,
'total_sails': {
'$cond': {
'if': {
'$isArray': '$sail_areas'
},
'then': {
'$size': '$sail_areas'
},
'else': 'NA'
}
}}},
Aggregation Pipeline 3

42. {
'$project': {
'name': 1,
'country_of_origin': 1,
'total_sails': 1,
'total_area': {
'$sum': '$sail_area'
}
}
}
]):
values.append(ship['sail_area'])
seriesLabels.append(ship['name'])
dataframe = pd.DataFrame(values, index=seriesLabels)
Aggregation Pipeline 4

43. Results
0
La Amistad 19335
Batavia 31246
Götheborg 18464
HMS Endeavour 9235
Golden Hind 11749
Grand Turk 8405
Kalmar Nyckel 3710
Lady Nelson 24938
Pallada 20785
Shtandart 9363
HMS Sultana 35061
HMS Surprise 26272
HMS Trincomalee 21744
HMS Victory 14740

44. Other Sessions
Today
1:00pm Real-time Clinical Decision Support System – Prem Timisina & Arash Kia
2:00 Analytics with MongoDB – Stuart Shiell & Mark Clancy
2:00 A Complete Methodology to Data Modeling for MongoDB –Daniel Coupal
3:15 Unleash the Power of the MongoDB Aggregation Framework – Abhishek Bagga
Tomorrow
9:00am Best Practices for Working with IoT and Time-series Data – Robert Walters
3:00pm MongoDB in Data Science – Vigen Sahakyan

45. Takeaways
MongoDB's flexible data model is very powerful for data analytics.
Some analytic tools require a more structured approach.
When forming your data the schema design used can make a huge
impact on analytics.
Use MongoDB's Aggregation Framework to improve performance.

46. Thank You!
Ken W. Alger - @kenwalger