I describe where I think (big) data is going. This presentation alternates between the business view of things and the technical.

1. © 2016 MapR Technologies 1© 2014 MapR Technologies
Where is Data Going?
Ted Dunning Chief Application Architect

2. © 2016 MapR Technologies 2
My Background
• University, Startups
– Aptex, MusicMatch, ID Analytics, Veoh
– big data since before it was big
• Open source
– even before the internet
– Calcite, Datafu, Drill, Kylin, Mahout, Myriad, Samoa, Singa, Storm,
Zookeeper
– bought the beer at first HUG
• VP Incubator at Apache Software Foundation
• MapR Chief Application Architect

3. © 2016 MapR Technologies 3
When you
see this
Real tech is
coming shortly

4. © 2016 MapR Technologies 4
IT Spending at an Inflection Point
Source: IDC, Gartner; Analysis & Estimates: MapR
Next-gen consists of cloud, big data, software and hardware related expenses
(100,000)
(80,000)
(60,000)
(40,000)
(20,000)
-
20,000
40,000
60,000
80,000
100,000
120,000
2013 2014 2015 2016 2017 2018 2019 2020
Next Gen vs. Legacy Shrinkage ($M)
$120
100
80
60
40
20
(20)
(40)
(60)
(80)
(100)
In Billions
Total $ Growth of IT Market Next-Gen Growth Legacy Market Growth/Shrink in $

5. © 2016 MapR Technologies 5
You Need to Make a Decision Now
In 4 years 90% of Data will be on next-gen technology

6. © 2016 MapR Technologies 6
MapR Converged Platform
Enterprise Hadoop + Storage2011-2012
2013
2014
2015
2016
+ NoSQL
+ Interactive SQL
+ Global Event Streaming
+ …
Top Ranked
Hadoop
Top Ranked
NoSQL
Top Ranked
SQL

7. © 2016 MapR Technologies 7
$
$
$

8. © 2016 MapR Technologies 8
Wait a minute!
What’s really going on?

9. © 2016 MapR Technologies 9
Evolution of Data Storage
Functionality
Compatibility
Scalability
Linux
POSIX
Over decades of progress,
Unix-based systems have set
the standard for compatibility
and functionality

10. © 2016 MapR Technologies 10
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
Hadoop achieves much higher
scalability by trading away
essentially all of this
compatibility
Evolution of Data Storage

11. © 2016 MapR Technologies 11
Evolution of Data Storage
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
MapR enhanced Apache Hadoop by
restoring the compatibility while
increasing scalability and
performance
Functionality
Compatibility
Scalability
POSIX

12. © 2016 MapR Technologies 12
Functionality
Compatibility
Scalability
Linux
POSIX
Hadoop
Evolution of Data Storage
Adding tables and streams enhances
the functionality of the base file
system

13. © 2016 MapR Technologies 13
What is Convergence?
Files
tokyo
Streams
User
profiles
Tables

14. © 2016 MapR Technologies 14

15. © 2016 MapR Technologies 15
What is Convergence?
Files
tokyo
Streams
User
profiles
Tables

16. © 2016 MapR Technologies 16
What is Convergence?
Files
tokyo
Streams
User
profiles
Tables

17. © 2016 MapR Technologies 17
Basic System Design
FTP
landing
Retail
data
Query pre-
processing Files
Web
access

18. © 2016 MapR Technologies 18
Basic System Design
FTP
landing
Retail
data
Query pre-
processing Files
Web
access
Convergence
helps here

19. © 2016 MapR Technologies 19

20. © 2016 MapR Technologies 20
Consumer Behavior
Log files
Consumer
electronics
Pre-
aggregation
Columnar
files
Consumer
electronicsConsumer
electronicsConsumer
electronicsConsumer
electronicsConsumer
electronics
Consumer
electronics
Dashboards
Ad hoc
query

21. © 2016 MapR Technologies 21
Consumer Behavior
Log files
Consumer
electronics
Pre-
aggregation
Columnar
files
Convergence
helps here
Consumer
electronicsConsumer
electronicsConsumer
electronicsConsumer
electronicsConsumer
electronics
Consumer
electronics
Dashboards
Ad hoc
query

22. © 2016 MapR Technologies 22

23. © 2016 MapR Technologies 23
$
$
$

24. © 2016 MapR Technologies 24PEOPLE
1.2B
PEOPLE
Largest Biometric Database in the World

25. © 2016 MapR Technologies 25
4 data centers
Synchronous replication EW
Asynchronous replication NS

26. © 2016 MapR Technologies 26
EW distance is short

27. © 2016 MapR Technologies 27
NS separation is large

28. © 2016 MapR Technologies 28
Summary
• Updates can happen anywhere, anytime
• Consistency maintained by insertion timestamp
• System is robust to link, power, data-center failure

29. © 2016 MapR Technologies 29

30. © 2016 MapR Technologies 30
Tenants Share Access to Data
External
dataset
Columnar
files
Internal
dataset
Pre-
processing Tables
Legacy
databases
Pre-
processing
Internal
dataset
Pre-
processing
Tenant 1
Tenant 2
Tenant 3

31. © 2016 MapR Technologies 31
Tenants Share Access to Data
External
dataset
Columnar
files
Internal
dataset
Pre-
processing Tables
Legacy
databases
Pre-
processing
Internal
dataset
Pre-
processing
Tenant 1
Tenant 2
Tenant 3
Convergence
helps here

32. © 2016 MapR Technologies 32

33. © 2016 MapR Technologies 33

34. © 2016 MapR Technologies 34
Streaming and Micro-services
Incoming
lab results
Archival
files
Order
reconciliation
Fraud
detection
uploads
anomaly
updates
Medical
records
Anomaly
detection

35. © 2016 MapR Technologies 35
Streaming and Micro-services
Incoming
lab results
Archival
files
Order
reconciliation
Fraud
detection
uploads
anomaly
updates
Medical
records
Anomaly
detection

36. © 2016 MapR Technologies 36
And just a slight bit of
future tense here

37. © 2016 MapR Technologies 37
Financial Services Use Case
• For reference assume:
– 1000 - 10,000 unique senders and receivers
– each bid or offer includes 10 recipients on average
– bids and offers arrive at a rate of 300k messages / second
• What would you do?

38. © 2016 MapR Technologies 38
Financial Services: Our Solution

39. © 2016 MapR Technologies 39
Massive IoT
• 100 million cars
• 2kB / second
• Roaming across data centers

40. © 2016 MapR Technologies 40
Gateway uploads-1
Gateway
uploads-2
Main ECU
in car
Gateway
Gateway
forward-1
forward-1
Forwarding
Car home
db
Car home
db
https

41. © 2016 MapR Technologies 41
Streaming First for Micro-Services
• How do applications need to change?
• How can we get micro-service benfits (this time around)?
• How does our development need to change?

42. © 2016 MapR Technologies 42
Traditional Solution
POS
1..n
Fraud
detector
Last card
use

43. © 2016 MapR Technologies 43
What Happens Next?
POS
1..n
Fraud
detector
Last card
use
POS
1..n
Fraud
detector
POS
1..n
Fraud
detector

44. © 2016 MapR Technologies 44
What Happens Next?
POS
1..n
Fraud
detector
Last card
use
POS
1..n
Fraud
detector
POS
1..n
Fraud
detector

45. © 2016 MapR Technologies 45
How to Get Service Isolation
POS
1..n
Fraud
detector
Last card
use
Updater
card activity

46. © 2016 MapR Technologies 46
New Uses of Data
POS
1..n
Fraud
detector
Last card
use
Updater
Card
location
history
Other
card activity

47. © 2016 MapR Technologies 47
Data Platform
Integrated Analytics
Enterprise Grade
Scale
Legacy Support
Transformational
Applications
Real Time

48. © 2016 MapR Technologies 48
Let’s Review the Options
S T O R A G E
Storag
e
Icon D A T A

49. © 2016 MapR Technologies 49
H A D O O P
Let’s Review the Options
D T A
C O M P U T E
D A T A

50. © 2016 MapR Technologies 50
Let’s Review the Options
C O M P U T E
S P A R K
D A T A

51. © 2016 MapR Technologies 51
Let’s Review the Options
C A S S A N D R A
D T A
C O M P U T E
D A T A

52. © 2016 MapR Technologies 52
Let’s Review the Options
C O M P U T E
D T AD A T A
Write In:
–– –– –– –
O T H E R

53. © 2016 MapR Technologies 53
C O N V E R G E D D A T A P L A T F O R M
Let’s Review the Options
Data in MotionData at Rest
C O M P U T E
D T AD A T A

54. © 2016 MapR Technologies 54
C O N V E R G E D D A T A P L A T F O R M
Let’s Review the Options
Data in MotionData at Rest

55. © 2016 MapR Technologies 55
C O N V E R G E D D A T A P L A T F O R M
Let’s Review the Options
CloudOn Premise

56. © 2016 MapR Technologies 56
C O N V E R G E D D A T A P L A T F O R M
Let’s Review the Options
Legacy Next Generation

57. © 2016 MapR Technologies 57
C O N V E R G E D D A T A P L A T F O R M
Let’s Review the Options
Real TimeBatch

58. © 2016 MapR Technologies 58
Platform Appeal
Reduce Costs
Drive Innovation

59. © 2016 MapR Technologies 59
Platform Appeal for Architects and Administrators
1. Simplifies administration
2. Avoids cluster sprawl
3. Unifies security, data protection, disaster recovery

60. © 2016 MapR Technologies 60
Platform Appeal for Developers
Flexibility
Agility
Simplicity
Real Time

61. © 2016 MapR Technologies 61
Integration of data-in-motion and data-at-rest supports
continuous, low latency processing
Converged Data Platform
mapr.com/appblueprint

62. © 2016 MapR Technologies 62
Q&A
@mapr maprtech
tdunning@mapr.tech.com
Engage with us!
MapR
maprtech
mapr-technologies