7. Data Model
• Familiar syntax
• Collections
• PRIMARY KEY for uniqueness
CREATE TABLE videos (
videoid uuid,
userid uuid,
name varchar,
description varchar,
location text,
location_type int,
preview_thumbnails map<text,text>,
tags set<varchar>,
added_date timestamp,
PRIMARY KEY (videoid)
);
8. Data Model - User Defined Types
• Complex data in one place
• No multi-gets (multi-partitions)
• Nesting! CREATE TYPE address (
street text,
city text,
zip_code int,
country text,
cross_streets set<text>
);
9. Data Model - Updated
• Now video_metadata is
embedded in videos
CREATE TYPE video_metadata (
height int,
width int,
video_bit_rate set<text>,
encoding text
);
CREATE TABLE videos (
videoid uuid,
userid uuid,
name varchar,
description varchar,
location text,
location_type int,
preview_thumbnails map<text,text>,
tags set<varchar>,
metadata set <frozen<video_metadata>>,
added_date timestamp,
PRIMARY KEY (videoid)
);
12. Example 1: Weather Station
• Weather station collects data
• Cassandra stores in sequence
• Application reads in sequence
13. Use case
• Get all data for one weather station
• Get data for a single date and time
• Get data for a range of dates and times
• Store data per weather station
• Store time series in order: first to last
Needed Queries
Data Model to support queries
14. Data Model
• Weather Station Id and Time
are unique
• Store as many as needed
CREATE TABLE temperature (
weather_station text,
year int,
month int,
day int,
hour int,
temperature double,
PRIMARY KEY ((weather_station),year,month,day,hour)
);
INSERT INTO temperature(weather_station,year,month,day,hour,temperature)
VALUES (‘10010:99999’,2005,12,1,7,-5.6);
INSERT INTO temperature(weather_station,year,month,day,hour,temperature)
VALUES (‘10010:99999’,2005,12,1,8,-5.1);
INSERT INTO temperature(weather_station,year,month,day,hour,temperature)
VALUES (‘10010:99999’,2005,12,1,9,-4.9);
INSERT INTO temperature(weather_station,year,month,day,hour,temperature)
VALUES (‘10010:99999’,2005,12,1,10,-5.3);
15. Storage Model - Logical View
weather_station hour temperature
2005:12:1:7
-5.6
2005:12:1:8
-5.1
2005:12:1:9
-4.9
SELECT weather_station,hour,temperature
FROM temperature
WHERE weatherstation_id=‘10010:99999’
AND year = 2005 AND month = 12 AND day = 1;
10010:99999
10010:99999
10010:99999
2005:12:1:10
-5.3
10010:99999
16. 2005:12:1:12
-5.4
2005:12:1:11
Storage Model - Disk Layout
SELECT weather_station,hour,temperature
FROM temperature
WHERE weatherstation_id=‘10010:99999’
AND year = 2005 AND month = 12 AND day = 1;
-5.1 -4.9 -5.3 -4.9
2005:12:1:7
-5.6
2005:12:1:8 2005:12:1:9
10010:99999
2005:12:1:10
Merged, Sorted and Stored Sequentially
23. Query patterns
• Range queries
• “Slice” operation on disk
SELECT weatherstation,hour,temperature
FROM temperature
WHERE weatherstation_id=‘10010:99999'
AND year = 2005 AND month = 12 AND day = 1
AND hour >= 7 AND hour <= 10;
Single seek on disk
2005:12:1:12
-5.4
2005:12:1:11
-5.1 -4.9 -5.3 -4.9
2005:12:1:7
-5.6
2005:12:1:8 2005:12:1:9
10010:99999
2005:12:1:10
Partition key for locality
24. Query patterns
• Range queries
• “Slice” operation on disk
Sorted by event_time
Programmers like this
SELECT weatherstation,hour,temperature
FROM temperature
WHERE weatherstation_id=‘10010:99999'
AND year = 2005 AND month = 12 AND day = 1
AND hour >= 7 AND hour <= 10;
weather_station hour temperature
2005:12:1:7
-5.6
2005:12:1:8
-5.1
2005:12:1:9
-4.9
10010:99999
10010:99999
10010:99999
2005:12:1:10
-5.3
10010:99999
26. Apache Spark
• 10x faster on disk,100x faster in memory than Hadoop MR
• Works out of the box on EMR
• Fault Tolerant Distributed Datasets
• Batch, iterative and streaming analysis
• In Memory Storage and Disk
• Integrates with Most File and Storage Options
Up to 100× faster
(2-10× on disk)
2-5× less code
30. RDD Operations
•Transformations - Similar to scala collections API
•Produce new RDDs
•filter, flatmap, map, distinct, groupBy, union, zip,
reduceByKey, subtract
•Actions
•Require materialization of the records to generate a value
•collect: Array[T], count, fold, reduce..
32. Collections and Files To RDD
scala> val distData = sc.parallelize(Seq(1,2,3,4,5)
distData: spark.RDD[Int] = spark.ParallelCollection@10d13e3e
val distFile: RDD[String] = sc.textFile(“directory/*.txt”)
val distFile = sc.textFile(“hdfs://namenode:9000/path/file”)
val distFile = sc.sequenceFile(“hdfs://namenode:9000/path/file”)
34. Spark on Cassandra
• Server-Side filters (where clauses)
• Cross-table operations (JOIN, UNION, etc.)
• Data locality-aware (speed)
• Data transformation, aggregation, etc.
• Natural Time Series Integration
35. Spark Cassandra Connector
• Loads data from Cassandra to Spark
• Writes data from Spark to Cassandra
• Implicit Type Conversions and Object Mapping
• Implemented in Scala (offers a Java API)
• Open Source
• Exposes Cassandra Tables as Spark RDDs + Spark DStreams
37. Spark Cassandra Example
val conf = new SparkConf(loadDefaults = true)
.set("spark.cassandra.connection.host", "127.0.0.1")
.setMaster("spark://127.0.0.1:7077")
val sc = new SparkContext(conf)
val table: CassandraRDD[CassandraRow] = sc.cassandraTable("keyspace", "tweets")
val ssc = new StreamingContext(sc, Seconds(30))
val stream = KafkaUtils.createStream[String, String, StringDecoder,
StringDecoder](
ssc, kafka.kafkaParams, Map(topic -> 1), StorageLevel.MEMORY_ONLY)
stream.map(_._2).countByValue().saveToCassandra("demo", "wordcount")
ssc.start()
ssc.awaitTermination()
Initialization
CassandraRDD
Stream Initialization
Transformations
and Action
38. Weather Station Analysis
• Weather station collects data
• Cassandra stores in sequence
• Spark rolls up data into new
tables
Windsor California
July 1, 2014
High: 73.4F
Low : 51.4F
39. Roll-up table
CREATE TABLE daily_aggregate_temperature (
wsid text,
year int,
month int,
day int,
high double,
low double,
PRIMARY KEY ((wsid), year, month, day)
);
• Weather Station Id(wsid) is unique
• High and low temp for each day
40. Setup connection
def main(args: Array[String]): Unit = {
// the setMaster("local") lets us run & test the job right in our IDE
val conf = new SparkConf(true).set("spark.cassandra.connection.host", "127.0.0.1").setMaster("local")
// "local" here is the master, meaning we don't explicitly have a spark master set up
val sc = new SparkContext("local", "weather", conf)
val connector = CassandraConnector(conf)
val cc = new CassandraSQLContext(sc)
cc.setKeyspace("isd_weather_data")
41. Get data and aggregate
// Case class to store row data
case class daily_aggregate_temperature (wsid: String, year: Int, month: Int, day: Int, high:Double, low:Double)
// Create SparkSQL statement
val aggregationSql = "SELECT wsid, year, month, day, max(temperature) high, min(temperature) low " +
"FROM raw_weather_data " +
"WHERE month = 6 " +
"GROUP BY wsid, year, month, day;"
val srdd: SchemaRDD = cc.sql(aggregationSql);
val resultSet = srdd.map(row => (
new daily_aggregate_temperature(
row.getString(0), row.getInt(1), row.getInt(2), row.getInt(3), row.getDouble(4), row.getDouble(5))))
.collect()
42. Store back into Cassandra
connector.withSessionDo(session => {
// Create a single prepared statement
val prepared = session.prepare(insertStatement)
val bound = prepared.bind
// Iterate over result set and bind variables
for (row <- resultSet) {
bound.setString("wsid", row.wsid)
bound.setInt("year", row.year)
bound.setInt("month", row.month)
bound.setInt("day", row.day)
bound.setDouble("high", row.high)
bound.setDouble("low", row.low)
// Insert new row in database
session.execute(bound)
}
})
44. What just happened?
• Data is read from raw_weather_data table
• Transformed
• Inserted into the daily_aggregate_temperature table
Table:
raw_weather_data
Table:
daily_aggregate_tem
perature
Read data
from table Transform Insert data
into table
45. Weather Station Stream Analysis
• Weather station collects data
• Data processed in stream
• Data stored in Cassandra
Windsor California
Today
Rainfall total: 1.2cm
High: 73.4F
Low : 51.4F
48. DStream - Micro Batches
• Continuous sequence of micro batches
• More complex processing models are possible with less effort
• Streaming computations as a series of deterministic batch
computations on small time intervals
DStream
μBatch (ordinary RDD) μBatch (ordinary RDD) μBatch (ordinary RDD)
Processing of DStream = Processing of μBatches, RDDs
49. Spark Streaming Reduce Example
val sc = new SparkContext(..)
val ssc = new StreamingContext(sc, Seconds(5))
val stream = TwitterUtils.createStream(ssc, auth, filters, StorageLevel.MEMORY_ONLY_SER_2)
val transform = (cruft: String) =>
Pattern.findAllIn(cruft).flatMap(_.stripPrefix("#"))
/** Note that Cassandra is doing the sorting for you here. */
stream.flatMap(_.getText.toLowerCase.split("""s+"""))
.map(transform)
.countByValueAndWindow(Seconds(5), Seconds(5))
.transform((rdd, time) => rdd.map { case (term, count) => (term, count, now(time))})
.saveToCassandra(keyspace, suspicious, SomeColumns(“suspicious", "count", “timestamp"))
Even Machine Learning!
50. Temperature High/Low Stream
Weather
Stations
Receive API
Apache Kafka
Producer
TemperatureActor
TemperatureActor
TemperatureActor
Consumer
51. You can do this at home!
https://github.com/killrweather/killrweather
52. Databricks & Datastax
Apache Spark is packaged as part of Datastax
Enterprise Analytics 4.5
Databricks & Datastax Have Partnered for
Apache Spark Engineering and Support
http://www.datastax.com/
55. FREE tickets to our Annual Cassandra Summit Europe taking place in London in early December (3rd
and 4th). The 4th is a full conference day with free admission to all attendees and will feature
presentations by companies like ING, Credit Suisse, Target, UBS, The Noble Group as well as other top
Cassandra experts in the world.
There will be content for those entirely new to Cassandra all the way to the most seasoned Cassandra
veteran, spanning development, architecture, and operations as well as how to integrate Cassandra with
analytics and search technologies like Apache Spark and Apache Solr.
December 3rd is a paid training day. If you are interested in getting a discount on paid training, please
speak with Diego - dferreira@datastax.com
