Tracking crime as it occurs with apache phoenix, apache hbase and apache nifi. Ingesting JSON Crime Feeds, XML Feeds, Twitter feeds, Traffic Camera Images.

1. Tracking Crime as It Occurs with Apache
Phoenix, Apache HBase and Apache NiFi
HENRY SOWELL
Technical Director
Cloudera Government Solutions
TIMOTHY SPANN
Field Engineer, Data in Motion
Cloudera

2. © 2019 Cloudera, Inc. All rights reserved. 2
Introduction - Henry Sowell
Henry Sowell
Technical Director,
Cloudera Government Solutions Inc
hsowell@cloudera.com

3. © 2019 Cloudera, Inc. All rights reserved. 3
Introduction - Tim Spann
• Tim has been running meetups in Princeton on Big Data technologies since
2015.
• Tim has spoken at many international conferences on Apache NiFi, Deep
Learning and Streaming.
• Tim is the Field Engineer for the Data in Motion team for the East Coast.
https://www.meetup.com/futureofdata-princeton/
https://community.hortonworks.com/users/9304/tspann.html
https://dzone.com/users/297029/bunkertor.html
https://dzone.com/articles/integrating-keras-tensorflow-yolov3-into-apache-ni

4. © 2019 Cloudera, Inc. All rights reserved. 4
Introduction
• Using Apache NiFi we can ingest various sources of crime related data in
real-time as activities happen. Simultaneously, we can monitor live traffic
cameras for infractions and correlation to other crime related activities
(Source: TrafficLand).
• We can do a lot of alerting, routing and reacting to crime data as it arrives,
but that’s only the beginning.
• Apache HBase and Apache Phoenix are great tools to tackle this problem.
• We’ll populate tables with ease and speed!
https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.html
https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html
https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html
https://github.com/tspannhw/nifi-cdsw

5. © 2019 Cloudera, Inc. All rights reserved. 5
DATAFLOW

6. © 2019 Cloudera, Inc. All rights reserved. 6

7. © 2019 Cloudera, Inc. All rights reserved. 7
CONTROL DATA-IN-MOTION FROM EDGE-TO-ENTERPRISE
Cloudera DataFlow - Collect, Curate and Analyze Data-in-Motion
● Edge-to-enterprise streaming data platform for management, security
and governance of real-time streaming data
● Edge data collection, processing and content routing of sensor data
from edge devices
● Continuous data ingestion from any streaming source or IoT device
● Ease-of-use in building sophisticated data flows with drag-and-drop
user interface
● Real-time stream processing and content syndication at the scale of
millions of messages per second
● Predictive and prescriptive analytics from streaming analytics engines
to gain actionable intelligence
DataFlow &
Steaming

8. © 2019 Cloudera, Inc. All rights reserved. 8
CLOUDERA FLOW MANAGEMENT
● Web-based user interface
● Highly configurable
● Out-of-the-box data provenance
● Designed for extensibility
● Secure
● NiFi Registry
○ DevOps support
○ FDLC
○ Versioning
○ Deployment

9. © 2019 Cloudera, Inc. All rights reserved. 9
300+ PROCESSORS FOR DEEPER ECOSYSTEM INTEGRATION
Hash
Extract
Merge
Duplicate
Scan
GeoEnrich
Replace
ConvertSplit
Translate
Route Content
Route Context
Route Text
Control Rate
Distribute Load
Generate Table Fetch
Jolt Transform JSON
Prioritized Delivery
Encrypt
Tail
Evaluate
Execute
Fetch
HTTP
Syslog
Email
HTML
Image
HL7
FTP
UDP
XML
SFTP
AMQP
WebSocket

10. © 2019 Cloudera, Inc. All rights reserved. 10
ARCHITECTURE

11. © 2019 Cloudera, Inc. All rights reserved. 11

12. © 2019 Cloudera, Inc. All rights reserved. 12
Crime JSON
{ "cartodb_id" : "2513323",
"the_geom" : "0101000020E61000009B2FFECF04CA52C0D3EE8080EFF94340",
"the_geom_webmercator" : "0101000020110F0000672C610B4DEA5FC1556E4BF02C895241",
"objectid" : "33869482",
"dc_dist" : "06",
"psa" : "2",
"dispatch_date_time" : "2019-05-19T00:00:00Z",
"dispatch_date" : "2019-05-19",
"dispatch_time" : "00:16:00",
"hour_" : "0",
"dc_key" : "201906023125",
"location_block" : "1000 BLOCK FILBERT ST",
"ucr_general" : "600",
"text_general_code" : "Thefts",
"point_x" : "-75.15654373",
"point_y" : "39.95262152",
"shape" : {
"type" : "Point",
"coordinates" : [ -75.15654373, 39.95262152 ]}}

13. © 2019 Cloudera, Inc. All rights reserved. 13
DEMO NiFi

14. © 2019 Cloudera, Inc. All rights reserved. 14

15. © 2019 Cloudera, Inc. All rights reserved. 15

16. © 2019 Cloudera, Inc. All rights reserved. 16

17. © 2019 Cloudera, Inc. All rights reserved. 17
HBase

18. © 2019 Cloudera, Inc. All rights reserved. 18
SPRING BOOT APPLICATION TO PHOENIX TABLE
CREATE TABLE PHILLYCRIME (dc_dist varchar,
dc_key varchar not null primary key,dispatch_date
varchar,dispatch_date_time varchar,dispatch_time varchar,hour
varchar,location_block varchar,psa varchar,
text_general_code varchar,ucr_general varchar);
java -Xms512m -Xmx2048m -Dhdp.version=3.1
-Djava.net.preferIPv4Stack=true -jar target/phoenix-0.0.1-SNAPSHOT.jar
@RequestMapping("/query/{query}")

19. © 2019 Cloudera, Inc. All rights reserved. 19
SPRING BOOT APPLICATION TO PHOENIX
https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html
https://github.com/tspannhw/phillycrime-springboot-phoenix

21. © 2019 Cloudera, Inc. All rights reserved. 21
DEMO HBase

22. © 2019 Cloudera, Inc. All rights reserved. 22
Beyond Collection: Gaining Actionable Intelligence
● Visualization and Reporting
● Optimizing the Allocation of Police
Resources
○ Analyze trends and activity to make
timely resource assignments
● Data Driven Policing
○ Holistic allsource analysis: Criminals
are not defined by a single data source
(e.g., social media account, etc.)
● Develop machine learning approaches
to enable law enforcement missions
● Add additional datasets

23. © 2019 Cloudera, Inc. All rights reserved. 23
THANK YOU!