This document discusses using surveillance analytics to automatically detect anomalous behavior and potential crimes without manual monitoring. It outlines challenges with traditional surveillance methods and proposes a solution using object detection models to identify abandoned objects, loitering individuals, and unauthorized vehicles in real-time video feeds. Key aspects of the proposed architecture include preprocessing video data for analytics, scaling the system using Databricks Delta storage, and experiment management with MLflow. A demonstration of the live video analytics pipeline showed it could process frames at 25-30 FPS on a GPU system while updating activity logs and video summaries to a central monitoring system.

1. Detecting
Anomalous Behavior
with Surveillance
Analytics
Automating security protocols &
detecting potential crimes

2. Speakers
§ Vinamre Dhar, Data Scientist
(vinamre.dhar@kmati.in)
§ Rishan Sanjay, Data Scientist
(rishan.sanjay@kmati.in)

3. Agenda
• Popular analysis utilizing surveillance data
• Challenges in existing surveillance solutions
• Our proposed solution and demonstration
• Architecting for scale

4. Popular use-cases for Surveillance Analytics
▪ Physical role-based
access management
▪ Violent behavior
detection in public
spaces
▪ Crime and intrusion
alerts
• Flag and log activity:
create searchable
events
• Trend forecasting
• Covid protocols in public
spaces
• Fire and other safety
protocols
Monitoring
Security
• License plate
recognition
• Infrastructure
bottlenecks and unused
spaces
• Automation and
customer intelligence
solutions
Miscellaneous

5. Challenges in traditional surveillance
Error prone, Slow & not Scalable
• Manual monitoring is error-prone, slow, expensive to
scale and often an unfeasible endeavor
• Human monitoring introduces individual biases
• Limited integration with security alerting tools like burglar
alarms

6. Use cases tackled
To detect and trigger anomalies without manual intervention
• Abandoned object detection
• Loitering and unauthorized individual detection
• Unauthorized vehicle entry or location detection

7. Object Detection
Ingest Store Cleansing and Pre-
Processing
Video Data
Video Feed
(unstructured) Real Time
Streaming
Protocol
Architecture Diagram
• Background
Subtraction
• Erosion
• Dilation
• Time Avg
Frame
Source
1. Abandoned object
2. Loitering Individuals
3. Unauthorized vehicle
Activity log and incident video
updated to Central Monitoring
System

8. Accessing the live video feed
Communication protocol used
• Real Time Streaming Protocol (RTSP) : communicating with DVR
• The VideoCapture() object gathers live video from device using RTSP
• A general format of the RTSP protocol looks like:
rtsp://username:password@IP:port_no/streaming/channels/camera_no/

9. Preprocessing and cleaning of the video feed
• Live video feed require a low
latency fluent frame rate
• VideoCapture object is set to skip
frame(s) if they aren't received
within threshold, to avoid
overloading the protocol thread
• Gray scaling conversion is
important to obtain a binary
format(scaled) frame
• Pre-processing steps:
§ Gray scaling
§ Gaussian blur
§ Background subtraction
§ Erosion
§ Dilation

10. Video Processing
Key modules
• Background subtraction: Extrapolate foreground objects from the background
• Erosion and dilation: obtaining object contours clearly
• Time averaged frame: accurately extract foreground objects, accounts for
changing lighting conditions, transient objects in frame and noise
• Object recognition: OpenCV for simple shaped objects like boxes
• Pretrained model on the COCO dataset for complex objects like people
• Threshold prediction scores are used to reduce false positive detections

12. Metrics
• 25 – 30 FPS on a GPU enabled system
• Business Impact:
• Video feed summarized by activity log of searchable events
• Cost reduction for scalable deployment
• Using AI significantly increases accuracy and improves responsiveness

13. Key considerations to note
• Compute: For real time analytics, GPU based computing will be critical
• Network latency: Its presence may cause a delay in real-time analytics
• Databricks Delta facilitates efficient probing of summarized video data
• Object detection and tracking models becomes significantly easy using
MLflow experiment management

14. Serve
Ingest Store Prep and Train
Data Lake Storage
Activity Logs
(unstructured)
Data Factory
MLFlow
Experiment
Management
Video Feed
(unstructured)
Sensors and IoT
(unstructured)
Scaling with Databricks
Delta
Format
Raw
Format
1. Abandoned object
2. Loitering Individuals
3. Unauthorized vehicle
Activity log and video
summary updated to Central
Monitoring System

15. Conclusions
Demonstrated video analytics on a live surveillance feed
Open-source packages and frameworks used: OpenCV,
Numpy, Tensorflow and Keras
Developed a real time surveillance analytics pipeline

16. Feedback
Your feedback is important to us.
Don’t forget to rate and review the sessions.
Reach out to us on sales@kmati.in