This document discusses building a real-time analytics platform on big data in the cloud. It outlines the need to move from batch processing to real-time analytics using approaches like streaming analytics and in-memory analytics. It provides examples of software that can be used for these approaches. The document also presents a reference architecture for a real-time analytics strategy, including components for data ingestion, processing, storage and business rules management. Impetus is introduced as a company that provides consulting and services for big data analytics.

1. 1
Building a Real Time Analytics Platform on
Big Data in the Cloud
Sanjay Sharma
Principal Architect
June 12 2013

2. Impetus
©
2013
Impetus
Technologies
• Software Solutions and Services Company
• Leader in Innovation led Technology services
• 17 years of customer success, 1500 people
• Big Data, Enterprise Mobility,Test and
Performance Engineering, Carrier Grade Large
Systems

3. ©
2013
Impetus
Technologies
?
Why
s9ll
Batch?
$$$
Credits
*
hAp://thenextweb.com/insider/2013/02/07/mailboxs-­‐user-­‐access-­‐queue-­‐is-­‐a-­‐work-­‐of-­‐frustra9ng-­‐genius/
**
hAp://en.wikipedia.org/wiki/Moore's_law
Batch
to
Real
Time
Now
is
the
Time

4. ©
2013
Impetus
Technologies
OLTP
Online
Transac'ons
Processing
NOW
OLAP/
DW
Online
Analy'cal
Processing/DW
ZZZZZZZZZ…
Transac9ons
&
Analy9cal
Processing
Now
is
the
Time

5. ©
2013
Impetus
Technologies
Age
of
Horizontal
Scalability
-­‐Divide
and
Conquer
Online
Transac'ons
Processing
Online
Analy'cal
Processing/
DW
OLTP
OLAP

6. ©
2013
Impetus
Technologies
Online
Transac9ons
Processing
Online
Analy9cal
Processing/
DW
On
Demand
Scalability
-­‐CLOUD
OLTP
OLAP

7. ©
2013
Impetus
Technologies
State
of
Play:
Analy9cs/
BI
Database
RESULTS
Time
Time
Ingest/ETL
SQL
Machine
Data

8. ©
2013
Impetus
Technologies
Faster
Analy9cs:
Streaming
Analy9cs/
Complex
Event
Processing
Database
RESULTS
Time
Time
RESULTS
Ingest/ETL
SQL
Approach
1

9. ©
2013
Impetus
Technologies
Faster
Analy9cs:
In-­‐memory
Analy9cs
Database
RESULTS
Time
Time
RESULTS
Ingest/ETL
SQL
Approach
2

10. • Streaming/CEP Analytics
– Storm by Twitter
– Spark Streaming by Berkley Amplabs
– IBM Streams
– SQLstream
– MS StreamInsight
– StreamBase
– Oracle CEP/Streams
– Esper
©
2013
Impetus
Technologies
• In-Memory Analytics
– SAP Hana
– Oracle Exalytics
– SAS in-memory analytics
– MS Powerpivot
Some
Real
Time
So_ware

11. ©
2013
Impetus
Technologies
• Others
– NoSQL databases
• Cassandra, MongoDB, HBase
– Grid solutions
– MPP and NewSQL
– Druid, Kognitio
– Solr/Elastic Search/Marklogic
– Splunk
– BI products with OLAP
• Hadoop based
– Summarization/ cube
preparation engine
– Impala/Stinger/Hawq/
Drill
– Hstreaming
Some
Real
Time
So_ware
• Specialized
– Hardware accelerators / GPU
cards
• Sisense

12. ©
2013
Impetus
Technologies
Real
Time
Analy9cs
Strategy
-­‐Key
Components
RESULT
S
Ingest
SQL
Machine
Data
OLTP
OLAP/
DW
ETL
Discovery
& Search
Deeper
Analytics
Streaming
Analytics
Real time
Analytics

13. ©
2013
Impetus
Technologies
NoSQL
+
Search
Machine
Data
Real
Time
Analy9cs
Strategy
-­‐Big
Data
Reference
Architecture
Example

14. ©
2013
Impetus
Technologies
Real
Time
Analy9cs
Strategy
-­‐Data
Inges9on/
Streaming
–
Customer
Example
Machine
Data
Inges9on
Engine
(Real
9me
+
Batch
components)
Real
Time
Processing
Engine
(CEP/Analy9cs/
Rule
Engine)
Real
Time
Data
Storage
Engine
(Store
+
Indexing/
Search)
Business
Process
Engine
(Business
Process/
Rule
management)
Kafa
Storm
+
Esper
HBase+
Solr
JBoss
Drools/jBPM

15. ©
2013
Impetus
Technologies
NoSQL
+
Search
Machine
Data
Real
Time
Analy9cs
Strategy
-­‐Big
Data
Reference
Architecture
Example

16. Impetus provides vendor-neutral Consulting &
Services for Big Data Analytics
We take enterprises through the journey of
successful Big Data adoption
We deliver: Low-risk acceleration to
Business value at moderate cost
©
2013
Impetus
Technologies
Impetus

17. Q&A
impetus.com
bigdata@impetus.com
Booth# 120
©
2013
Impetus
Technologies