Slides of my talk at Liferay Berlin Dev Con 2014 about building an analytics platform in Liferay

1. Liferay & Big Data
Getting value from your data
!
Miguel Ángel Pastor Olivar
miguel.pastor@liferay.com

2. Who am I?
!
• Some random guy
!
• Member of the Liferay core infrastructure
team
!
•Disclaimer: Not a computer scientist
!
• @miguelinlas3

3. What are we going to talk about?
!
• Big Data: what is this about?
!
• Simple architecture proposal
!
• Use cases
!
• Questions (and hopefully answers)

4. Big Data?

5. • Data is so big that regular solutions are:
!
–Extremely slow
!
–Too small
!
–Really expensive
!
• How we use all the data we already own

6. !
• Volume
–Transactions, data streaming from social media, …
!
• Velocity
–Torrents of data in real time
!
• Variety
–Numerical data, text, email, video, audio, …

7. Popular usages

8. • Recommender systems
!
• Predicting the future:
– Netflix does autoscaling based on past
network data traffic
!
• Churn models
– Big telco companies build social networks
to reduce the churn

9. • Sentiment analysis
–Are talking about you in the Internet?
!
• Real Time Bidding
–Optimise advertising
!
• Health care
–Improve patients health while reducing costs
–Improve quality of life of multiple sclerosis patients

10. Terminology

11. • Storage models
• How to store relevant information
!
• Computation models
• Process and transform all the information
!
• Analytics
• How we can take actions based on the
previous steps

12. Big Data
Architectures

13. Data storage

14. Hadoop Distributed File System (HDFS)
!
• Java based file system
!
• Scalable, fault-tolerant, distributed storage
!
• Designed to run on commodity hardware
!
• Closely related to MapReduce

15. Source: http://hortonworks.com/

16. NoSQL storage

17. • Semistructured data
!
• Focused on
!
• Horizontal scalability
!
• Availability
!
• Different trade-offs: CAP, BASE, …
!

18. NewSQL
storage

19. • Modern relational databases
!
• Same scalable performance than NoSQL for
OLTP
!
• Maintain ACID guarantees
!
• A few alternatives: VoltDB, Google Spanner,
FoundationDB, …

20. Computation
and analytics

21. Apache Hadoop

22. Apache Hadoop Map Reduce
!
• Distributed processing
!
• Large datasets
!
•Clusters of computers
#LRNAS2014
!
• Simple programming model
!
• Verbose and hard to use API

23. Liferay
projects
is
the
best
Open
Source
project
best: 1
is: 1
Liferay: 1
Open: 1
project: 2
Source: 1
the: 1
(index, “…”)
(index, “…”)
(index, “…”)
(index, “…”)
(index, “…”)
Sort
and
shuffle
(best, [1])
(is, [1])
(Liferay: 1)
(Open, [1])
(project, [1,1])
(Source, [1])
(the, [1])

24. • Batch model data crunching
!
• Not so good event stream processing
!
• But …
!
• Many algorithms hard to implement using
MapReduce
!
• Cascading, Scalding, Cascalog, Impala, …

25. Apache Storm

26. • Distributed realtime computation system
!
• Easy to reliably process unbounded streams of data
!
• Multi language support
!
• Realtime analytics, online machine learning, continuous
computation, distributed RPC, ETL, …

27. Spout
Spout
Bolt Bolt
Bolt

28. Apache Spark

29. • Fast and general-purpose cluster computing
• Developed by Berkeley AMP
!
• High level APIs (not MapReduce)
!
• Optimised engine:
• supports general execution graphs
!
• Higher-level tools:
• Spark SQL, MLib, Spark Streaming, Graphx

30. Apache Mahout

31. !
• Scalable machine learning library
#LRNAS2014
!
• Built on top of Hadoop
!
• Some algorithms don’t require Hadoop at all
#LRNAS2014

32. R language

33. • Focused on:
• Data visualisation
• Statistical computations
• Analysis of data
!
• Tons of built-in packages
!
• Connect to Hadoop through Hadoop Streaming
!
• Not a fast language

34. Reference
Architecture

35. RDBMS
Event Broker
Hadoop
User
Tracking
NoSQL
Storage
System
Events
Search
Data
Logs
Monitoring Dataware
House
Streaming Social
Graph

36. Datasources

37. RDBMS
Event Broker
Hadoop
User
Tracking
NoSQL
Storage
System
Events
Search
Data
Logs
Monitoring Dataware
House
Streaming Social
Graph

38. • System events
!
• User tracking (client side)
• Clicks, navigation, activities, …
!
• Monitoring (transactions, load page times, …)
!
• Models (message boards, blogs, wiki …)
!
• Custom developments …

39. Event broker

40. RDBMS
Event Broker
Hadoop
User
Tracking
NoSQL
Storage
System
Events
Search
Data
Logs
Monitoring Dataware
House
Streaming Social
Graph

41. Data Source
0 1 2 3 4 5 6 7 8
Writes
9
Reads Reads
System A System B

42. Apache Kafka
!
• Publish-subscribe as distributed commit log
!
• Fast
!
• Scalable
!
• Durable
!
• Distributed by design

43. Broker A
Broker B
Producer Consumer
Broker C
ZooKeeper

44. Computation
and analytics

45. RDBMS
Event Broker
Hadoop
User
Tracking
NoSQL
Storage
System
Events
Search
Data
Logs
Monitoring Dataware
House
Streaming Social
Graph

46. Batch processing?
!
Real time processing?
!
Machine learning algorithms?
!
Graph analysis?
!
Unified programming model?

48. !
• Fast and general engine for large-scale data
processing
!
• Write your apps in Java, Scala or Python
!
• Run on YARN cluster manager
!
• Can read any existing Hadoop data (HDFS)
!
• In memory or disk

49. Apache Spark Main Components
Apache Spark
Spark SQL
Spark
Streaming MLib GraphX

50. Spark Core

51. • Driver main function and executes various
parallel operations on a cluster
!
• Resilient Distributed Datasets (RDD)
• HDFS (or any Hadoop file system)
!
• Scala collection
!
• Second abstraction: shared variables

52. Spark SQL

53. • Mix SQL queries with Spark programs
!
• Unified Data Access
!
• Hive compatibility
!
• Standard JDBC or ODBC connectivity
!
• Same engine for both interactive and long running
queries

54. Spark Streaming

55. • Build your apps using high-level operators
!
• Fault tolerance: exactly-once semantics out of the box
!
• Combine streaming with batch and interactive queries
!
• Can read from HDFS, Flume, Kafka, Twitter and ZeroMQ
!
• Define your own custom data sources

56. Spark MLib

57. !
• Basic statistics
• Summary statistics
• Correlations
• ….
!
• Classification and regression
• Linear models
• Decision tress
• Naive Bayes

58. !
• Clustering
• K-Means
!
• Collaborative filtering
• Alternate least squares
!
• Dimensionality reduction
• Singular value decomposition
!
• Principal component analysis

59. Spark GraphX

60. !
• Graphs API and graph-parallel computation
!
• Growing scale and importance
• From social networks to language modelling
!
• Directed multigraph with properties attached to each
vertex and edge
!
• Growing collection of graph algorithms and builders

61. Live demo!
Building a messages
classifier

62. Takeaways

63. • Not about data size, but how you use it
!
• You already own tons of data, you just need to take get
value from it
!
• There is no silver bullet: you’ve plenty of alternatives
!
• JVM Big data related techs are usually a great choice
!
• Try it yourself!!

64. References

65. !•
Apache Kafka
!
• Apache Spark
!
• Apache Storm
!
• Apache Hadoop
!
• Big Data definition at Wikipedia
!
• Liferay Kafka Bridge
!
• What every software engineer should know about a log

66. Thank you!!

67. Questions
(and hopefully answers)