Live demo and lessons learned building and publishing an advanced video analytics solution in the Azure Marketplace. This is a deep technical dive into the engineering and data science employed throughout, with all challenges encountered by combining Deep Learning and Computer Vision for object detection and tracking, the operational management and tool building efforts for scaling the video processing and insights extraction to large GPU/CPU Databricks clusters and the machine learning required to detect behavioral patterns, anomalies and scene similarities across processed video tracks. The entire solution was build using open source scala, python, spark 3.0, mxnet, pytorch, scikit-learn as well as Databricks Connect.

1. Video analytics at
scale: DL, CV, ML on
Databricks platform
Claudiu Barbura
Director of Engineering
Blueprint Technologies

2. Agenda
§ Live demo of Video
Analytics solution
§ Solution Architecture
§ Lessons learned
▪ Infrastructure (gpu vs cpu)
▪ Object detection & tracking
▪ Behavioral patterns, anomalies
§ Q&A

3. Live Demo of Nash from Azure Marketplace

5. Lessons learned: Infrastructure

6. Infrastructure
• 1 Linux VM + ADB Cluster + BLOB
• Shift ALL compute from VM to ADB (Res Mgr is key)
• Video generation (ffmpeg) bottleneck
• Custom docker image (+opencv, pytorch, sklearn …) pushed to ADB cluster (DCS)
• CPU arch optimization: torch.set_num_threads(1) (2-5x)
• GPU arch optimization: mxnet->pytorch, cuda.benchmark=true, torch.Tensor(value, device='cuda')
• Object Detection model as Spark Broadcast variable (2-4x)
• Spark 3.0 GPU-aware scheduling to avoid GPU OOM
• ADB cluster: GPU vs CPU perf comparison at price parity (3.77x)
• 3 x GPU vs 10 x CPU (80 cores)
• GPU: 1x Tesla V100 16GB + 6c CPU/112GB(too much!)

7. Video processing time at price parity

8. Video processing time at price parity

9. Lessons learned: Object detection and tracking

10. Object detection and tracking
• From fasterrcnn_resnet_50 (mAP 37, 21fps) to faster and more accurate efficientdet-d3 (mAP 46.8,
22.7 fps)
• efficientdet-d3 detector trained on the MS-COCO dataset, classes used are [bus, car, truck]
• Detection confidence threshold 40% (configuration)
• mxnet->pytorch due to GPU architecture requirements
• Batch frames for detection (CPU/GPU) vs tracking (CPU-only) to avoid context switching
• Tracking strategies: FairMot vs Our Own
• FairMot: default tracker, up to 10x faster … when it tracks correctly
• Custom tracker fallback: JDE (Joint Detection and Embedding) + Kalman Filter + TemplateMatch
(openCV)

12. Lessons learned: Behavioral patterns, anomalies

13. Anomaly Detection
• Vehicle Trajectory Anomaly Detection uses tracking for common, average and rare paths
• 'cluster representatives' as pseudo-centroids in non-uniform vector length feature space
• Rarity is computed as distance from cluster centroid
• DBSCAN for short videos
• AgglomerativeClustering for long videos

14. Behavioral patterns
• Time-scale based smoothing algorithms
• Low-pass filter/Fourier Series/Numpy.hanning

15. Feedback
Your feedback is important to us.
Don’t forget to rate and review the sessions.