Big Data Camp LA 2014, Don't re-invent the Big-Data Wheel, Building real-time, Big Data applications on Cassandra with the open-source Kiji project by Clint Kelly of Wibidata

1. Don’t Reinvent the
Big-Data Wheel!
Clint Kelly - @clintwkelly
WibiData
Building real-time, Big Data applications on
Cassandra with the open-source Kiji project
Big Data Camp LA
14 June 2014

2. Agenda

3. Agenda
The problem

4. Agenda
The problem
How Kiji works

5. Agenda
The problem
How Kiji works
Kiji in production

6. Agenda
The problem
How Kiji works
Kiji in production
Kiji on Cassandra

7. The problem.

8. !

16. !

17. !
Open source
software

18. !

19. !

20. !

21. !

22. !

23. !
?

24. Data in

25. Data in

26. Data in
REST

27. Inspect

28. Inspect

29. Inspect

30. Inspect

31. Inspect

32. Train

33. Train

34. Train
“Trained
model”

35. Train
“Trained
model”

36. Train
“Trained
model”

37. Train
“Trained
model”

38. Train
“Trained
model”

39. Model

40. Model
AaBb

41. Model
AaBb

42. Score

43. Score

44. Score
AaBb
AaBb
AaBb
AaBb
AaBb
AaBb
AaBb
AaBb
AaBb

45. Score
AaBb
AaBb
AaBb
AaBb
AaBb
AaBb
AaBb
AaBb
AaBb

46. Score
Batch
AaBb
AaBb
AaBb
AaBb
AaBb
AaBb
AaBb
AaBb
AaBb

47. Data out

48. Data out

49. Data out
REST

50. Data out
REST

56. REST

57. REST

58. REST

65. AaBb

66. AaBb

67. AaBb

68. AaBb

69. Experiments / Deployment

70. Experiments / Deployment

71. Experiments / Deployment
c
d
c
d

72. Experiments / Deployment
c
d
c
d

73. 3

74. Data in / out

75. Data in / out
(REST)

76. Inspect and train

77. Score

78. Score
(real-time)

79. !
?

80. !!
Kiji

81. How Kiji works

82. Kiji History

83. Kiji History

84. Kiji History

85. How does it work?
Kiji

86. How does it work?
Kiji
Engineering
Data
Science

87. How does it work?
Kiji
Data
Science
Write
Engineering

88. How does it work?
Kiji
Data
Science
Write
Channels Engineering

89. How does it work?
Kiji
Data
Science
Write
Logs
DBs
EngineeringChannels

90. How does it work?
Kiji
Data
Science
Write
Logs
DBs
KijiMR
EngineeringChannels

91. How does it work?
Kiji
Data
Science
Write
KijiREST
Stream
EngineeringChannels

92. How does it work?
Kiji
Data
Science
Write
Read
KijiREST
Stream
EngineeringChannels

93. How does it work?
KijiSchema
(Cassandra)
Data
Science
Write
Read
KijiREST
Stream
EngineeringChannels

94. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
EngineeringChannels

95. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
C
C
C
EngineeringChannels

96. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
C
C
C
EngineeringChannels

97. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
C
C
C
EngineeringChannels

98. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
C
C
C
EngineeringChannels

99. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
Data
C
C
C
EngineeringChannels

100. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
Data
C
C
C
EngineeringChannels

101. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
Data
C
C
C
EngineeringChannels

102. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
Data
C
C
C
EngineeringChannels

103. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiMR
C
C
C
EngineeringChannels
Data

104. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
C
C
C
EngineeringChannels
Data

105. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
Scorer
C
C
C
EngineeringChannels
Data

106. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
Scorer
C
C
C
EngineeringChannels
Data

107. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
Scorer
C
C
C
R
EngineeringChannels
Data

108. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
Scorer
C
C
C
EngineeringChannels
Data

109. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
Scorer
C
C
C
EngineeringChannels
Data

110. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
Scorer
C
C
C
R
R
R
EngineeringChannels
Data

111. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
KijiScoring
C
C
C
R
Kiji Model
Repository
EngineeringChannels
Data
Scorer

112. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
KijiScoring
C
C
C
R
Kiji Model
Repository
EngineeringChannels
Data
Scorer

113. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
KijiScoring
C
C
C
R
Kiji Model
Repository
EngineeringChannels
Data
Scorer

114. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
KijiScoring
C
C
C
R
Kiji Model
Repository
EngineeringChannels
Data
Scorer

115. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
KijiScoring
C
C
C
R
Kiji Model
Repository
EngineeringChannels
Data
Scorer

116. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
KijiScoring
C
C
C
R
Kiji Model
Repository
EngineeringChannels
Data
Scorer

117. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
KijiScoring
C
C
C
R
Kiji Model
Repository
EngineeringChannels
Data
Scorer

118. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
KijiScoring
C
C
C
R
Kiji Model
Repository
EngineeringChannels
Data
Scorer

119. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
KijiScoring
C
C
C
R
Kiji Model
Repository
EngineeringChannels
Data
Scorer
R

120. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
KijiScoring
C
C
C
R
Kiji Model
Repository
EngineeringChannels
Data
Scorer
R
R

121. KijiSchema
(Cassandra)
How does it work?
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
Query
KijiHive
KijiExpress
KijiMR
KijiScoring
C
C
C
R
Kiji Model
Repository
EngineeringChannels
Data
Scorer
R
R
R

122. 3

123. Data in / out
KijiREST
KijiMR

124. Inspect and train
KijiHive
KijiMR
KijiExpress

125. Score
(real-time)
KijiModelRepository
KijiScoring

126. Modular

127. Kiji in production

128. In production now
Fortune 500 retailer: Personalized recommendations
Opower: Energy usage and analytics reporting

129. Fortune 500 retailer
Serving personalized recommendations

130. Kiji
Write
Logs
DBs
KijiMR
EngineeringChannels
Bulk load

131. KijiSchema
(Cassandra)
Data
Science
User 1
User 2
User 3
KijiExpress
KijiMR
C
C
C
Data
Train

132. KijiSchema
(Cassandra)
Data
Science
Write
Read
KijiREST
Stream
User 1
User 2
User 3
KijiScoring
C
C
C
R
Kiji Model
Repository
EngineeringChannels
Scorer
Score

133. Kiji on Cassandra

134. KijiSchema

135. KijiSchema

136. KijiSchema
Cassandra

137. KijiSchema
Cassandra

138. KijiSchema
HBase

139. Kiji ~ BigTable

140. table

141. table
row
row
row
row
row
row
row
row
row
row
row
row

142. row

143. Row key = entity ID
entity ID data

144. Composite entity IDs
data0xfa “bob”

145. Column families
payment0xfa “bob” interactions recommendations

146. inter:
clicks
inter:
search0xfa “bob”
payment:
cardnum
payment:
address
rec:
scorer1
rec:
scorer2
Columns

147. Timestamped versions
songs:
let it be
inter:
search0xfa “bob” songs:
let it besongs:
let it besongs:
let it be
inter:
clicks
1396560123
payment:
cardnum
payment:
address
rec:
scorer2
rec:
scorer3rec:
scorer3rec:
scorer3
rec:
scorer1
1395650231

148. Complex data types
record Search {
string search_term;
long session_id;
device_type device;
}
songs:
let it be
inter:
search0xfa “bob” songs:
let it besongs:
let it besongs:
let it be
inter:
clicks
1396560123
payment:
cardnum
payment:
address
rec:
scorer2
rec:
scorer3rec:
scorer3rec:
scorer3
rec:
scorer1
1395650231

149. Locality group

150. Locality group
Column families

151. Locality group

152. Locality group
Batch Batch Batch

153. Locality group
Batch Batch Batch
Real-
time
Real-
time
Real-
time

154. Locality group
Batch Batch
Real-
time
Real-
time
Real-
time
Batch

155. locality_group_real_timelocality_group_batch
Locality group
Batch Batch
Real-
time
Real-
time
Real-
time
Batch

156. locality_group_real_timelocality_group_batch
Locality group
Batch Batch
Real-
time
Real-
time
Real-
time
Batch

157. locality_group_real_timelocality_group_batch
Locality group
Batch Batch
Real-
time
Real-
time
Real-
time
Batch

158. locality_group_real_timelocality_group_batch
Locality group
Batch Batch
Real-
time
Real-
time
Real-
time
Batch
On disk.
Compressed.

159. locality_group_real_timelocality_group_batch
Locality group
Batch Batch
Real-
time
Real-
time
Real-
time
Batch
On disk.
Compressed.
In memory.

160. Row ➔ transactional consistency

161. Locality group ➔ Column family
CREATE TABLE loc_grp
songs:
let it be
inter:
search0xfa “bob” songs:
let it besongs:
let it besongs:
let it be
inter:
clicks
1396560123
payment:
cardnum
payment:
address
rec:
scorer2
rec:
scorer3rec:
scorer3rec:
scorer3
rec:
scorer1
1395650231

162. Entity ID ➔ Primary key
CREATE TABLE loc_grp (city text, user text,
PRIMARY KEY (city, user) )
WITH CLUSTERING ORDER BY (user ASC);
songs:
let it be
inter:
search0xfa “bob” songs:
let it besongs:
let it besongs:
let it be
inter:
clicks
1396560123
payment:
cardnum
payment:
address
rec:
scorer2
rec:
scorer3rec:
scorer3rec:
scorer3
rec:
scorer1
1395650231

163. Family, Qualifier,Version ➔ Clustering Columns
CREATE TABLE loc_grp (city text, user text,
family text, qualifier text, version bigint,
PRIMARY KEY (city, user, family, qualifier, version) )
WITH CLUSTERING ORDER BY (user ASC, family ASC, qualifier ASC,
version DESC);
songs:
let it be
inter:
search0xfa “bob” songs:
let it besongs:
let it besongs:
let it be
inter:
clicks
1396560123
payment:
cardnum
payment:
address
rec:
scorer2
rec:
scorer3rec:
scorer3rec:
scorer3
rec:
scorer1
1395650231

164. Column values ➔ Blobs
CREATE TABLE loc_grp (city text, user text,
family text, qualifier text, version bigint, value blob,
PRIMARY KEY (city, user, family, qualifier, version) )
WITH CLUSTERING ORDER BY (user ASC, family ASC, qualifier ASC,
version DESC);
songs:
let it be
inter:
search0xfa “bob” songs:
let it besongs:
let it besongs:
let it be
inter:
clicks
1396560123
payment:
cardnum
payment:
address
rec:
scorer2
rec:
scorer3rec:
scorer3rec:
scorer3
rec:
scorer1
1395650231

165. Implementation notes

166. Implementation notes
DataStax Java driver

167. Implementation notes
DataStax Java driver
Cassandra 2.0.6

168. Implementation notes
DataStax Java driver
Cassandra 2.0.6
Async API

169. Implementation notes
DataStax Java driver
Cassandra 2.0.6
Async API
New MapReduce InputFormat

170. Issues

171. Operations across locality groups

172. Operations across locality groups
Kiji locality group ➔ C* column family

173. Operations across locality groups
Kiji locality group ➔ C* column family

174. Operations across locality groups
Kiji locality group ➔ C* column family
Read across locality groups

175. Operations across locality groups
Kiji locality group ➔ C* column family
Read across locality groups
➔ multiple C* reads (async API!)

176. Operations across locality groups
Kiji locality group ➔ C* column family
Read across locality groups
➔ multiple C* reads (async API!)

177. Operations across locality groups
Kiji locality group ➔ C* column family
Read across locality groups
➔ multiple C* reads (async API!)
Compare-and-set across locality groups

178. Operations across locality groups
Kiji locality group ➔ C* column family
Read across locality groups
➔ multiple C* reads (async API!)
Compare-and-set across locality groups
➔ not allowed in C* Kiji

179. Operations across locality groups
Kiji locality group ➔ C* column family
Read across locality groups
➔ multiple C* reads (async API!)
Compare-and-set across locality groups
➔ not allowed in C* Kiji

180. Operations across locality groups
Kiji locality group ➔ C* column family
Read across locality groups
➔ multiple C* reads (async API!)
Compare-and-set across locality groups
➔ not allowed in C* Kiji
Lose transactional consistency

181. Filters
HBase ➔ Rich server-side filters
Cassandra ➔ WHERE clauses

182. Filters
HBase ➔ Rich server-side filters
Cassandra ➔ WHERE clauses
Client-side filtering

183. Project status

184. Components working with
Cassandra
KijiSchema
KijiMR
KijiREST
KijiExpress

185. KijiSchema available for
download / tutorial
https://github.com/kijiproject/kiji-
schema/blob/cassandra/
cassandra_tutorial.md
(tinyurl.com/mmubg5o)

186. All code available with tutorial
within 1-2 months

187. Summary

188. 3

189. Data in / out
KijiREST
KijiMR

190. Inspect and train
KijiHive
KijiMR
KijiExpress

191. Score
(real-time)
KijiModelRepository
KijiScoring

192. Thanks to Cassandra community
Mailing lists
Meetups, webinars, conferences

193. Try it now!
www.kiji.org
tinyurl.com/mmubg5o
@clintwkelly