Real time analytics on big data applications, what is real time, what are the challenges to achieve real time complex calculation on big data and how they can be solved.
2. We’re Living in a Real Time World…
Social User Tracking & Homeland Security
Engagement
eCommerce Financial Services Real Time Search
2 ® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
3. The Flavors of Big Data Analytics
Counting Correlating Research
3 ® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
4. Twitter in Numbers (March 2011)
It takes a week for users to
send 1 billion tweets
Source: http://blog.twitter.com/2011/03/numbers.html
4 ® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
5. Twitter in Numbers (March 2011)
On average,
140 million
tweets get sent every day
Source: http://blog.twitter.com/2011/03/numbers.html
5 ® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
6. Twitter in Numbers (March 2011)
The highest
throughput to date is
6,939 tweets/sec.
Source: http://blog.twitter.com/2011/03/numbers.html
6 ® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
7. Twitter in Numbers (March 2011)
460,000 new
accounts
are created daily
Source: http://blog.twitter.com/2011/03/numbers.html
7 ® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
8. Challenge – Word Count
Tweets
8
?
® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
Count
Count
Word:Count
9. Analyze the Problem
Thousands of tweets per second to process
Aggregate counters for each word
Latency – less than a second
System needs to linearly scale
System needs to be fault tolerant
Querying & Persisting Data
Managing the system
9 ® Copyright 2011 Gigaspaces Ltd. All Rights Reserved
