5. New features
• 2.2
- JSON
- User defined functions
- User defined aggregates
- Other useful features
- http://docs.datastax.com/en/cassandra/2.2/cassandra/features.html
- http://www.datastax.com/dev/blog/cassandra-2-2
• 3.0
- New storage engine (8099)
- A new way to denormalise/duplicate : Materialized View
6. So who’s taken some data out of C* and
serialised it as JSON?
7. Hello JSON
• create TABLE user (username text primary key,
first_name text , last_name text , emails set<text> ,
country text);
• INSERT INTO user JSON '{"username": "chbatey",
"first_name":"Christopher", "last_name": "Batey",
“emails":["christopher.batey@datastax.com"]}';
9. JSON + User Defined Types
• CREATE TYPE movie (title text, time timestamp,
description text);
• ALTER TABLE user ADD movies set<frozen<movie>>;
• UPDATE user SET movies = {{ title:'Batman',
time:'2011-02-03T04:05:00+0000', description:
'This film rocks' }} where username = 'chbatey';
11. • Run code on the server !Dangerous!
- Disabled by default
• Java + Java Script supported out of the box
• Any language that supports the Java Scripting API
(Java, Javascript, Ruby, Python …)
User Defined Functions
12. UDF example
CREATE TABLE user (
username text primary key,
first_name text ,
last_name text ,
emails set<text> ,
country text);
13. Concat function
CREATE FUNCTION name ( first_name text, last_name text )
CALLED ON NULL INPUT
RETURNS text LANGUAGE java
AS ‘return first_name + " " + last_name;’;
cqlsh:twotwo> select name(first_name, last_name) FROM user;
twotwo.name(first_name, last_name)
------------------------------------
Victor Coustenoble
14. User Defined Aggregates
CREATE AGGREGATE average ( int )
SFUNC averageState
STYPE tuple<int,bigint>
FINALFUNC averageFinal
INITCOND (0, 0);
Called for every row
state passed between
Initial state
Return type (CQL)
Optional function called on
final state
15. State function (like a UDF)
CREATE FUNCTION averageState ( state tuple<int,bigint>, value int )
CALLED ON NULL INPUT
RETURNS tuple<int,bigint>
LANGUAGE java
AS '
if (value != null) {
state.setInt(0, state.getInt(0)+1);
state.setLong(1, state.getLong(1)+val.intValue());
}
return state;
';
Type Columns
16. Final function
CREATE FUNCTION averageFinal ( state tuple<int,bigint> )
CALLED ON NULL INPUT
RETURNS double
LANGUAGE java
AS '
if (state.getInt(0) == 0) return null;
double r = state.getLong(1) / state.getInt(0);
return Double.valueOf(r);
';
State typeOverall return type
20. Built in aggregates
• count
• max
• min
• avg
• sum
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cql3/functions/AggregateFcts.java
21. Built in time functions
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cql3/functions/TimeFcts.java
28. Other bits and pieces…
• Compressed commit log
• Resumable bootstrapping
• New types
- smallint - short
- tinyint - byte
- date
- time
• Warnings now sent back to client
- batch too large
30. New Storage Engine
• CASSANDRA-8099
• More efficient storage
• Aware of CQL structure
• Reduce sstable size
• Reduce memory used
• …
31. Customer events table
CREATE TABLE if NOT EXISTS customer_events (
customer_id text,
staff_id text,
store_type text,
time timeuuid ,
event_type text,
PRIMARY KEY (customer_id, time))
create INDEX on customer_events (staff_id) ;
32. Indexes to the rescue?
customer_id time staff_id
chbatey 2015-03-03 08:52:45 trevor
chbatey 2015-03-03 08:52:54 trevor
chbatey 2015-03-03 08:53:11 bill
chbatey 2015-03-03 08:53:18 bill
rusty 2015-03-03 08:56:57
bill
rusty 2015-03-03 08:57:02
bill
rusty 2015-03-03 08:57:20 trevor
staff_id customer_id
trevor chbatey
trevor chbatey
bill chbatey
bill chbatey
bill rusty
bill rusty
trevor rusty
33. Secondary index are local
• The staff_id partition in the secondary index is not
distributed like a normal table
• The secondary index entries are only stored on the node
that contains the customer_id partition
34. Indexes to the rescue?
staff_id customer_id
trevor chbatey
trevor chbatey
bill chbatey
bill chbatey
staff_id customer_id
bill rusty
bill
rusty
trevor rusty
A B
chbatey rusty
customer_id time staff_id
chbatey 2015-03-03 08:52:45 trevor
chbatey 2015-03-03 08:52:54 trevor
chbatey 2015-03-03 08:53:11 bill
chbatey 2015-03-03 08:53:18 bill
rusty 2015-03-03 08:56:57
bill
rusty 2015-03-03 08:57:02
bill
rusty 2015-03-03 08:57:20 trevor
customer_events table
staff_id customer_id
trevor chbatey
trevor chbatey
bill chbatey
bill chbatey
bill rusty
bill
rusty
trevor rusty
staff_id index
35. Do it yourself index ?
CREATE TABLE if NOT EXISTS customer_events (
customer_id text,
staff_id text,
store_type text,
time timeuuid ,
event_type text,
PRIMARY KEY (customer_id, time))
CREATE TABLE if NOT EXISTS customer_events_by_staff (
customer_id text,
staff_id text,
store_type text,
time timeuuid ,
event_type text,
PRIMARY KEY (staff_id, time))
37. Pattern
• Write only:
- Duplicate with a different primary key
- (Optional) Logged batch for eventual consistency
• Full updates:
- No real difference
• Partial updates:
- No staff id in update?
38. Score Data Model
CREATE TABLE scores
(
user TEXT,
game TEXT,
year INT,
month INT,
day INT,
score INT,
PRIMARY KEY (user, game, year, month, day)
)
39. Materialized Views
CREATE MATERIALIZED VIEW alltimehigh AS
SELECT user FROM scores
WHERE game IS NOT NULL AND
score IS NOT NULL AND
user IS NOT NULL AND
year IS NOT NULL AND
month IS NOT NULL AND
day IS NOT NULL
PRIMARY KEY (game, score, user, year, month, day)
WITH CLUSTERING ORDER BY (score desc)
44. Fine print
• All Primary Key columns must be present in your view
• If the part of your primary key is NULL then it won't
appear in the materialised view
• Performance will be a factor!
- More operations to complete (read-before-write,
consistency check …)
- Batch writes for MV
• Bad for low cardinality data (hot spot)
45. Conclusions
• We still denormalise and duplicate to achieve scalability
and performance
• We just let C* do it for us :)
46. Find Out More
• Documentation: http://www.datastax.com/docs
• Developer Blog: http://www.datastax.com/dev/blog
• Academy: https://academy.datastax.com
• Community Site: http://planetcassandra.org
Editor's Notes
also a toJson and fromJson if you want individual fields
User defined types??
time is modelled as a long, nanoseconds since midnight
time is modelled as a long, nanoseconds since midnight
Looks good so far.
The first problem however is that a single query can result in many partitions being queries. We know why this is bad.
Each of the segments of the index table
Start by writing it out to a batch log on 2 other replicas
Downside:
Look at the extra round trips
Extra complexity
Serial reads