Architects from DataTorrent talk about Malhar framework for processing streaming events in real

1. Hadoop’s Most Powerful Platform for Real-Time Stream
Computations
Prepared for Big Data Gurus
February 27th, 2014

3. DataTorrent Big Data Platform
• Vision: Ubiquitize Real-Time Big Data Computations
– Enterprise quality: Highly Available, Linearly Scalable, Operable and
Easy to Use
– Big data dimensional computations in real time with linear scalability
• Real-Time ETL: De-dup, Staging, Cleanup, Transformations, Load …
• Real-Time Computation Apps and Feed Ingestion (Games, Mobile, Set-top
Boxes, Devices, …)
– Multi-Feed Sources
– Run business logic in real-time with HA
• Real-Time Monitoring, and Security: Capacity, DDOS, …
• Real-Time Predictive Analytics: Web Analytics, Business Analytics, …

4. DataTorrent in Hadoop Ecosystem

5. © DataTorrent, 2014 - Confidential
DataTorrent in the Modern Data Architecture
APPLICATIONSDATASYSTEMSOURCES
RDBMS EDW
Emerging Sources
(Sensor, Sentiment, Geo, Unstructured)
HANA
BusinessObjects BI
OPERATIONAL TOOLS
DEV & DATA TOOLS
Existing Sources
(CRM, ERP, Clickstream, Logs)
INFRASTRUCTURE
Business Analytics Business Intelligence Tools OLAP Clients
Real-time Stream Analytics
DATAIN
MOTION

6. StrAM (Stream Application Master)
Security
SLA
Scalability
Alerts
Fault
Tolerance
Tools
Partitioning
Web Services
Dynamic
Modifications
State
Snapshot
Malhar – Open Source Operators and Apps Library
(Apache v2 License)
DataTorrent Technology Stack

7. © DataTorrent Inc. 2014 - Confidential
DataTorrent in Hadoop Reference Architecture
DATA IN MOTION
REAL TIME STREAMING APPLICATIONS
SOURCE DATA
MS Queue’s
Events
Files
Databases
Sensor data
Social
APPLICATIONS
BusinessObjects BI
Query/Visualization/ Reporting/Analytical Tools and Apps
Enterprise Repositories
RDBMS
EDW
NoSQL
Real Time Ingestion
DATA AT REST
BATCH APPLICATIONS
Hive
Pig
HBase
Custom
Message QueueData In Motion
YARN
HDFS
YARN
Map
Reduce
HDFS
OPERATIONAL INTELLEGENCE
BUSINESS ACTIONS
PREDICTIVE ANALYTICS
STREAM ETL
REALTIME ALERTS

8. © DataTorrent, 2014 - Confidential
Stream Processing
• A Stream is a sequence of data events with schema
• An Operator takes input streams and compute output streams
• An Application is a Directed Acyclic Graph (DAG)
• In-memory asynchronous distributed computations
• A Streaming Window is an atomic batch of sequential data events

9. DataTorrent Hadoop GRID
1
2
4
3
6
NM NM NM NM
Resource
Manager
StrAM
3
5
5
64
2
1
DT
Gateway
dtCLI
DT
Console
MapReduce
MapReduce
MapReduce
MapReduceMapReduce
MapReduce

10. Malhar Open Source Project
• Apache 2.0
• Operators (over 400 operators)
– Algorithms
– Ingestion, ETL
– Input and Output Adapters
• UI Widgets (over 50 widgets)
– Console widgets for stats
– Application widgets for app data
• Application Templates
– LogStream
– Map Reduce Debugger
– Shuffle less MapReduce
• Demo Apps (15 demo apps)

11. Malhar Open Source Project
• Continuous Integration: Unit tests
• Performance tests for operators
• Daily tests of Demos and Apps for memory usage
• More operators and UI widgets added as per new use cases/user requests
• Fully supported: Documentation, Certification
• Input and Output Adapters
– HBase, MongoDB, CouchDB, Redis, Memcache
– Flume, Kakfa, RabbitMQ, ActiveMQ, ZeroMQ
– JBDC, MySql, DerbyDB, TimesTen
– MQTT, Twitter, RSS, HTTP, WebSockets, Socket
– Logs: Apache, SMTP
– DFS, Local cache (Guava)
• Languages: Java, Python, JavaScript, Script, R

12. DataTorrent’s Platform Differentiators
.
Extreme Scalability Mission Critical Hadoop-Native
• Automatically scales to
changing loads. Massive
performance per node.
Billions of events/sec
• Sub-second latency with
linear scalability.
• Complex monitoring
applications with massive
computations.
• Built-in Stateful Fault-
tolerance. 24/7 uptime -
Highly Available.
• Predictive Analysis, Root
cause. Real-time ETL.
• Update your application
while it's running! A/B
testing (2h2014).
• Develop faster and
implement any business
logic with our open-source
framework.
• Runs on your existing
Apache Hadoop cluster.
• Integrate seamlessly with
your existing data flow and
monitoring stack.

13. Live Demonstration

14. Thank You!
• DataTorrent
• Try Sandbox (https://datatorrent.com)
• Free for
• Startup Program (Contact us for more details)
• Up to 25GB memory usage in production
• Non-production clusters
• Malhar Open Source (Apache 2.0) project
• https://github.com/DataTorrent/Malhar
• malhar-users@googlegroups.com
• Applications available Jan 2014
• LogStream Application
• Map-Reduce Monitor
DataTorrent Inc.
3200 Partrick Henry, 2nd Fl
Santa Clara, CA 95054
info@datatorrent.com
www.datatorrent.com
Twitter.com/DataTorrent
Facebook.com/DataTorrent