Combining the awesomeness of Apache Spark and Azure DocumentDB together for real-time machine learning on globally-distributed data.

1. Spark to DocumentDB
Connector
Denny Lee,
Principal Program Manager, Azure DocumentDB

2. Denny Lee
• Principal Program Manager for Azure DocumentDB
• 20+ years of experience in databases, distributed systems, data
sciences, and software development at Microsoft, Concur, and
Databricks
• Noteable Projects:
• Project Isotope: Incubation team for HDInsight
• Yahoo! 24TB cube: Largest SSAS cube in production
@dennylee

3. A Brief Overview...

4. Elastically Scalable Throughput + Storage

5. Guaranteed low latency
Reads <10ms @ P99
Writes <15ms @ P99

6. Globally Distributed

7. Speaks your language

8. DocumentDB
REST over HTTPS/TCP
MongoDB wire protocol
drivers for MongoDB
Java .NET
Java .NET
Ruby
…

9. Aggregations
Demo

10. Running Aggregations from Portal
Supports SUM, COUNT,
MIN, MAX, AVG
Working on DISTINCT and
GROUP BY

11. Data Sciences:
Apache Spark + DocumentDB
Demo
Notebook
View: https://aka.ms/docdb-spark-graph
pyView: https://aka.ms/pydocdb-spark-graph
Code: https://aka.ms/docdb-spark-graph-code

12. Advantages
Data Science Scenarios
• Distributed Aggregations and Analytics
• Blazing Fast IoT Scenarios
• Updateable columns
• Push-down predicate filtering

13. Advantages
Distributed Aggregations and Analytics

14. Advantages
Blazing Fast IoT Scenarios
Flight
information
global safety
alerts
weather
Data Science Scenarios
Device
Notifications
Web / REST API

15. Advantages
Updateable Columns
Flight
information
Data Science Scenarios
Device
Notifications
Web / REST API
{
tripid: “100100”,
delay: -5,
time: “01:00:01”
}
{
tripid: “100100”,
delay: -30,
time: “01:00:01”
}
{delay:-30}
{delay:-30}
{delay:-30}

16. Advantages
Pushdown Predicate Filtering Data Science Scenarios
{city:SEA}
locations headquarter exports
0 1
country
Germany
city
Seattle
country
France
city
Paris
city
Moscow
city
Athens
Belgium 0 1
{city:SEA, dst: POR, ...},
{city:SEA, dst: JFK, ...},
{city:SEA, dst: SFO, ...},
{city:SEA, dst: YVR, ...},
{city:SEA, dst: YUL, ...},
...

17. gateway
node data
nodes
master
node
worker nodes
pyDocumentDB
1
2
3
pyDocumentDB
1. Connection is between Spark
master node and DocumentDB
gateway node.
2. Query is submitted from
DocumentDB gateway node to
data nodes. Results are sent back
to the gateway node and then
transmitted back to the Spark
master node.
3. Spark master node converts the
dictionary to a DataFrame and
distributed out to the worker
nodes.

18. gateway
node data
nodes
master
node
worker nodes
Spark-DocumentDB
Connector (Java)
1
3
2
4
Spark to DocumentDB Connector
1. Connection is between Spark
master node and
2. map data is transmitted back to
DocumentDB gateway node
3. Query is submitted from Spark
worker nodes to
4. DocumentDB data nodes and the
data is transmitted back to Spark
worker nodes for further
processing

19. Query Test Results
Query pyDocumentDB Azure-DocumentDB-Spark
LIMIT 100 0:00:00.774820 00:00:01.286
All Seattle flights (23K rows) 0:00:05.146107 00:00:01.582
All flights (~1.39M rows) 0:02:36.335267 00:00:08.899
More info at: https://github.com/Azure/azure-documentdb-spark/wiki/Query-Test-Runs

20. Query Test Results
Issue # Issue Description
7 Improve push down predicates (e.g. take advantage of TOP/LIMIT, aggregations,
etc.)
6 Schema-less query bug
5 Optimize computation push to partitions
3 Add Python wrapper / examples
2 Add Azure-DocumentDB-Spark connector as Spark package
More info at: https://github.com/Azure/azure-documentdb-spark/issues

21. Asks
Go to https://github.com/Azure/azure-documentdb-spark/ and try it out!
References:
• Real-time machine learning on globally-distributed data with Apache
Spark and DocumentDB
• Accelerate real-time big-data analytics with the Spark to DocumentDB
connector
Any questions?
• We’re on StackOverflow #azure-documentdb
• Email askdocdb@ or denny.lee@

22. Data Sciences:
Apache Spark + DocumentDB

23. Example: Graph Structures

24. Example: Graph Structures

25. Graph Calculations: Degrees, PageRank
What is the most important
airport (most flights in / out)
tripGraph.inDegrees
.sort(desc("inDegree"))
.limit(10))

26. Classic Graph Scenario: Flights
vertex = airports
edges = flights