We present a novel deep learning approach to create a robust object detection network for use in an infra-red, UAV-based, poacher recognition system. More specifically we have used Microsoft AirSim to generate thousands of hours of simulated drone footage in the African Savannah. We then used deep domain adaptation to translate our simulation into a form that is adversarially indistinguishable from real infrared drone footage. This yields a programmable data generator that can be used to dramatically improve the accuracy of algorithms without requiring expensive human curated annotations. Furthermore, we extend this work and contribute a photorealism extension to AirSim, automating much of the domain specific expertise needed for computer graphics work, and enabling the generation of limitless quantities of photorealistic data for use in reinforcement learning and autonomous vehicles.
7. Deep Learning
• Brain-like algorithms trained with gradient
descent
• Has become a recent favorite because:
– Spectacular performance in many
domains
– Quick training
– Low memory
– Large space of possible model
architectures
– Automatic differentiation software makes
it very easy
Hamilton and Raman, #SAISExp4 7
8. 8Hamilton and Raman, #SAISExp4
Tire
Dog
Scotty Gu
Input Images
Low Level
Features
Mid Level
Features
High Level
Features
Output
Classes
Filters from Zeiler + Fergus 2013
9. 9Hamilton and Raman, #SAISExp4
Input Images
Low Level
Features
Mid Level
Features
Logistic
Regression
Output
Classes
Filters from Zeiler + Fergus 2013
Any
SparkML
Model
Other
Leopard
Predictions
10. ML
• High level library for distributed
machine learning
• More general than SciKit-Learn
• All models have a uniform
interface
– Can compose models into
complex pipelines
– Can save, load, and
transport models
Load Data
Tokenizer
Term Hashing
Logistic Reg
Evaluate
Pipeline
Evaluate
Load Data
● Estimator
● Transformer
Hamilton and Raman, #SAISExp4 10
11. GPU/CPU acceleration
Automated gradient calculations
Flexible language for defining
huge space of models
State of the art performance
and accuracy
ONNX Runtime
Large scale parallelism
Fault tolerance
Auto-scaling / elasticity
High throughput streaming
Data source + orchestrator
agnostic
Distributed Computing
Hamilton and Raman, #SAISExp4 11
Deep Learning
12. Microsoft ML for
Apache Spark
• Open Source Distributed ML
library
• Unifies major computing
paradigms and Microsoft tools
into a single library and API
– Python, Java, Scala, R
– Batch, Streaming, Serving
– Local, Static Cluster,
Dynamic Cluster,
Serverless
– Databricks, HDI, AZTK,
Kubernetes, …
www.aka.ms/spark
Hamilton and Raman, #SAISExp4 12
Scalable Deep
Learning
Stress Free
Serving
Distributed
Microservices
Fast Gradient
Boosting
CNTK on Spark Spark Serving
HTTP on Spark LightGBM on Spark
MMLSpark
13. Training a Deep Transfer
Learner:
Hamilton and Raman, #SAISExp4 13
15. …
Time
1 2 3 4 5 6
Time
1 2 3 4 5 6
Deep Network
…
Hamilton and Raman, #SAISExp4 15
16. Real Time monitoring
with PowerBI
Stream from Kafka /
Cosmos
OpenCV on Spark
CNTK on Spark
SparkML Logistic Reg
Stream Results to
PowerBI
Camera Image Upload in
Kyrgyzstan
MMLSpark
Contribution
27 Lines of
Code
~1k imgs/s
throughput
on CPU
~3 second
latencies
Hamilton and Raman, #SAISExp4 16
18. 18Hamilton and Raman, #SAISExp4
Lightning Fast Web Services on Any Spark Cluster
• Sub-millisecond latencies
• Fully Distributed
• Spins up in seconds
• Same API as Batch and
Streaming
• Scala, Python, R and Java
• Fully Open Source
www.aka.ms/spark
JIRA: SPARK-25350
Serving
113
0.9
0
50
100
150
Latency (ms)
Spark Serving Spark Continuous Serving
Announcing: 100x Latency Reduction
with MMLSpark v0.14
20. • 3 main modes:
– server: 1 service on the head node
– distributedServer: 1 service per
executor
– continuousServer: 1 service per
partition
• Built on top of Spark Streaming
• Each worker keeps a running
service, and a routing table
• Use HTTP on Spark objects to
represent requests and
responses in Spark SQL
Spark Serving Architecture
Hamilton and Raman, #SAISExp4 20
21. Going Further: No Labels, No Problem
Hamilton and Raman, #SAISExp4 21
22. Bing Powered Snow Leopard
Detection Pipeline
Bing Photo Search of
“Snow Leopard” +
Related terms
Logistic
Regression
Add Class +
Union +
Distinct
Truncated
ResNet50
Cloud Service
CNTK on Spark
Key
Bing Image Search of
random nouns Spark ML
Hamilton and Raman, #SAISExp4 22
24. Announcing: Intelligent and Distributed
Microservices in SparkML
www.aka.ms/spark
Microsoft
Cognitive Services
+
Bing Text Vision Face
Hamilton and Raman, #SAISExp4 24
26. HTTP on Spark
• Allows Spark users to call into any
HTTP service
• Leverage parallel networking of
cluster
• Allows Spark to work with
microservice architectures
• Mini-batching, asynchrony, and
buffering supported
• Statically typed and usable from
Spark, PySpark, and SparklyR
Hamilton and Raman, #SAISExp4 26
29. Announcing: LIME on
• Elastic and Distributed
• Works with any image classification
model
• Example notebook instructions at:
– www.aka.ms/spark
Hamilton and Raman, #SAISExp4 29
30. Using LIME to generate
bounding boxes
30Hamilton and Raman, #SAISExp4
31. Transferring LIME’s Knowledge
to FasterRCNN
31Hamilton and Raman, #SAISExp4
𝑚𝑎𝑠𝑘 = 𝑢𝑛𝑖𝑜𝑛 𝑠𝑢𝑝𝑒𝑟𝑝𝑖𝑥𝑒𝑙𝑠 𝑡𝑜𝑝 70% 𝑜𝑓 𝑤𝑒𝑖𝑔ℎ𝑡𝑠)
𝑥1, 𝑦1 = min 𝑚𝑎𝑠𝑘 𝑥 , min 𝑚𝑎𝑠𝑘 𝑦
𝑥2, 𝑦2 = max 𝑚𝑎𝑠𝑘 𝑥 , max(𝑚𝑎𝑠𝑘 𝑦)
32. Results
32Hamilton and Raman, #SAISExp4
Human Labels
Unsupervised
FRCNN Outputs Human Labels
Unsupervised
FRCNN Outputs
35. In Conclusion
www.aka.ms/spark
Contributions Welcome!
github.com/Azure/mmlspark
Hamilton and Raman, #SAISExp4 35
• Big Releases in v0.14:
– Cognitive Services on Spark
– Sub-Millisecond latency
distributed web services
– Distributed Model
Interpretability
• Easy batch, streaming apps,
PowerBI dashboards, or RESTful
web services
• We aim to give all work back to the
community!
• Easy to Get Started on Databricks:
– 16 Jupiter notebook guide
MMLSpark v0.14
36. Thanks to
• You all!
• Microsoft AI Development Acceleration Program: Abhiram Eswaran, Ari
Green, Courtney Cochrane, Janhavi Suresh Mahajan, Karthik Rajendran,
Minsoo Thigpen, Casey Hong, Soundar Srinivasan
• MMLSpark Team: Sudarshan Raghunathan, Ilya Matiach, Eli Barzilay, Tong
Wen, Ben Brodsky
• The Snow Leopard Trust: Rhetick Sengupta, Koustubh Sharma, Jeff Brown,
Michael Despines
• Microsoft: Joseph Sirosh, Lucas Joppa, WeeHyong Tok
Get in touch: marhamil@microsoft.com
MMLSpark Website: aka.ms/spark
Hamilton and Raman, #SAISExp4 36