This is the presentation of my session at the IoT Tech Day on 14 April 2016.

1. Fast Data Intelligence in the IoT
Real-time Data Analytics with Spark Streaming and MLlib
Bas Geerdink
#iottechday

2. ABOUT ME
• Chapter Lead in Analytics area at ING
• Academic background in Artificial
Intelligence and Informatics
• Working in IT since 2004, previously as
developer and software architect
• Spark Certified Developer
• Twitter: @bgeerdink
• Github: geerdink

4. WHAT’S NEW IN THE IOT?
• More data
– Streaming data from multiple sources
• New use cases
– Combining data streams
• New technology
– Fast processing and scalability
Front
End
Back
End
Data

5. PATTERNS & PRACTICES
FOR FAST DATA ANALYTICS
• Lambda Architecture
• Reactive Principles
• Pipes & filters
• Event Sourcing
• REST, HATEOAS
• …

6. LAMBDA ARCHITECTURE
Source: Nathan Marz & James Warren (2015)

7. REACTIVE PRINCIPLES
Source: Reactive Manifesto (2014)

8. USE CASE
nest
WWW

9. FAST DATA ARCHITECTURE
Products
Users
API
App
Web
…
Batch
(Machine Learning)
Social
Media
Search
History
GPS
Data
…
Message
Broker
Events
Streaming
(Business Logic)
VisualizeProcessing Database

10. A SHIFT IN TECHNOLOGY PARADIGMS
Disk  In-memory
Database  Stream
Objects  Functions
Centralized  Distributed
Shared Memory/CPU/Disk  Shared Nothing

11. TOOLS FOR THE JOB
• Apache Kafka
• Apache Cassandra
• Apache Spark
• Apache Zeppelin
• Akka
• Scala

12. FAST DATA ARCHITECTURE
Products
Users
API
App
Web
…
Batch
Machine Learning
Social
Media
Search
History
GPS
Data
GPS
Data
Message
Broker
Streaming
Business Logic
Events VisualizeProcessing Database

13. KAFKA
• Distributed Message broker
• Built for speed, scalability, fault-tolerance
• Works with topics, producers, consumers
• Created at LinkedIn, now open source
• Written in Scala

14. CODE: KAFKA
• build.sbt:
"org.apache.kafka" %% "kafka" % kafkaVersion
• Application.conf:
kafka { producer … consumer }
• KafkaConnection.scala:
def producer, def consumer
• KafkaProducerActor.scala:
producer.send(msg)
• KafkaConsumerActor.scala:
val kafkaStream =
connection.createMessageStreams(Map(topic -> 1))(topic)(0)

15. CASSANDRA
• NoSQL database
• Built for speed, scalability, fault-tolerance
• Works with CQL, consistency levels, replication factors
• Created at Facebook, now open source
• Written in Java

16. CODE: CASSANDRA
CREATE TABLE products (user_name text, product_category text, product_name text,
score int, insertion_time timeuuid, PRIMARY KEY (user_name, product_category,
product_name));
val cluster = new Cluster.Builder().
addContactPoints(uri.hosts.toArray: _*).
withPort(uri.port).
withQueryOptions(new
QueryOptions().setConsistencyLevel(defaultConsistencyLevel)).build
val session = cluster.connect
session.execute(s"USE ${uri.keyspace}")
def insertScore(productScore: ProductScore): Unit = {
val query = s”INSERT INTO products (user_name, product_category, product_name,
score, insertion_time) VALUES ('${productScore.userName}',
'${productScore.productCategory}', '${productScore.productName}',
${productScore.score}, now())"
session.execute(query)
}

17. SPARK
• Fast, parallel, in-memory, general-purpose data
processing engine
• Winner of Daytona Gray Sort benchmark 2014
• Runs on Hadoop YARN, Mesos, cloud, or standalone
• Created at AMPLab UC Berkeley, now open source
• Written in Scala

18. CODE: SPARK BASICS
val l = List(1,2,3,4,5)
val p = sc.parallelize(l) // create RDD
p.count() // action
def fun1(x: Int): Int = x * 2
p.map(fun1).collect() // transformation
p.map(i => i * 2).filter(_ < 6).collect() // lambda

19. SPARK

20. SPARK STREAMING

21. CODE: SPARK STREAMING
val conf = new SparkConf().setAppName("fast-data-search-history").setMaster("local[2]")
val ssc = new StreamingContext(conf, Seconds(2)) // batch interval = 2 sec
val kafkaParams = Map[String, String]("metadata.broker.list" -> "localhost:9092")
val kafkaDirectStream = KafkaUtils.createDirectStream[String, String, StringDecoder,
StringDecoder](ssc, kafkaParams, Set("search_history"))
kafkaDirectStream
.map(rdd => ProductScoreHelper.createProductScore(rdd._2))
.filter(_.productCategory != "Sneakers")
.foreachRDD(rdd => rdd.foreach(CassandraHelper.insertScore))
ssc.start() // it's necessary to explicitly tell the StreamingContext to start receiving data
ssc.awaitTermination() // wait for the job to finish

22. CODE: SPARK MLLIB
// initialize Spark MLlib
val conf = new SparkConf().setAppName("fast-data-social-media").setMaster("local[2]")
val sc = new SparkContext(conf)
// load machine learning model from disk
val model = LinearRegressionModel.load(sc, "/home/social_media.model")
def processEvent(sme: SocialMediaEvent): Unit = {
// feature vector extraction
val vector = new DenseVector(Array(sme.userName, sme.message))
// get a new prediction for the top user category
val value = model.predict(vector)
// store the predicted category value
val user = new User(sme.userName, UserHelper.getCategory(value))
CassandraHelper.updateUserCategory(user)
}

23. THREE KEY TAKEAWAYS
• The IoT comes with new architecture: reactive and
scalable are the new normal
• Be aware of the paradigm shift: in-memory,
streaming, distributed, shared nothing
• Open source tooling such as Kafka, Cassandra, and
Spark can help to process the fast data flows

24. Thank You!
“please rate my talk in the offical IoT Tech Day app”
@bgeerdink
#iottechday