The realtime processing tool based on Storm and Kafka.

1. genn.ai (源内)
Realtime processing for everyone
Takeshi Nakano
@tf0054
1

2. Who am I?
Takeshi NAKANO
 Senior Researcher / Architect.
 Co-authered Getting started with Solr in Japanese.
 Co-authered Hadoop Hacks in Japanese.
 Writing Getting started with Kafka in Japanese.
http://genn.ai/
2

3. Agenda
 What is genn.ai?
 Let s use genn.ai
 Realtime processing
 Download & get your key
 Usecases
 Define a schema and a filter
 Make it real!
http://genn.ai/
3

4. A usecase of genn.ai
Customer
A
Web Sites
じゃらん
Customer
B
Customer
C
Marketer
Analyst
Realtime Analysis
Platform
Client
(Hotels)
4

5. Visual editor for genn.ai

6. Structure of genn.ai
MongoDB
MetaStore
genn.ai UI
User account
Tuple schema
Query design
view
Query
Gungnir
Client
on finagle
Gungnir
Cosole
Query
Gungnir
Thrift Server
Gungnir
on finagle
Thrift Server
Storm
nimbus
Topology
Kafka
User account
Tuple schema
JSON Tuple
Gungnir
Rest Server
Gungnir
Gungnir
on finagle
Rest Server
Rest Server
Tuple
Kafka
Kafka
broker
Kafka
broker
broker
Tuple
Storm
Storm
supervisor
Storm
supervisor
supervisor
MongoD
B
HDFS
Tuple
Topic
http://genn.ai/
6

7. Preparing for starting.
http://dev.genn.ai/

8. Preparing for starting.
http://dev.genn.ai/
> cd gungnir-client
> bin/gungnir -u tf0054 -p 2e2232073109
Picked up _JAVA_OPTIONS: -Dfile.encoding=UTF-8
2013/10/14 22:34:58 com.twitter.finagle.Init$ apply
情報: Finagle version 6.5.1 (rev=59..9cde41) built at 20130626-111057
Gungnir server connected... ec2-xxx.amazonaws.com/54.238.99.212:9290
Welcome tf0054
gungnir>

9. Define your schema!
CREATE TUPLE userAction (
userId STRING,
hotelId STRING
);
CREATE TUPLE commitAction (
userId STRING,
hotelId STRING,
checkin TIMESTAMP('yyyy-MM-dd'),
nights INT
);
STRING Java type string
TINYINT Java type byte
SMALLINT Java type short
INT Java type int
BIGINT Java type long
FLOAT Java type float
DOUBLE Java type double
TIMESTAMP long value
LIST
MAP
STRUCT

10. Define your filter! (1/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()
EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user')
;
From
EMIT

11. Define your filter! (2/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()
JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image
USING mongo_fetch('getusers', 'userInfo')
JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo
USING mongo_fetch('hotelscom', 'hotel')
EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user')
;
From
JOIN
(user)
JOIN
(hotel)
EMIT

12. Define your filter! (3/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()
JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image
USING mongo_fetch('getusers', 'userInfo')
JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo
USING mongo_fetch('hotelscom', 'hotel')
INTO stream_0
;
FROM stream_0
EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user')
;
From
JOIN
(user)
JOIN
(hotel)
stream_0
EMIT
(user)

13. Define your filter! (4/4)
FROM userAction AS ua, commitAction AS ca USING kafka_spout()
JOIN userInfo ON userInfo.uid = userId TO userInfo.name AS name, userInfo.image AS image
USING mongo_fetch('getusers', 'userInfo')
JOIN hotel ON hotel.hotelId = hotelId TO hotel.name AS hotelname, hotel.photo AS photo
USING mongo_fetch('hotelscom', 'hotel')
INTO stream_0
;
FROM stream_0
EMIT userId, hotelId, name, image USING kafka_emit('${TOPOLOGY_ID}_user')
;
FROM stream_0
BEGIN GROUP BY hotelId
EACH count() AS count, *
EMIT hotelId, hotelname, photo, count USING kafka_emit('${TOPOLOGY_ID}_hotel')
END GROUP
;
EMIT
(user)
From
JOIN
JOIN
EACH sum(nights)
EACH avg(nights)
stream_0
GROUP
BY
EMIT
(hotel)

14. Register your filter! and ...
SUBMIT TOPOLOGY;
DESC TOPOLOGY;
SHOW TOPOLOGIES;
STOP TOPOLOGY xxx;
gungnir> SUBMIT TOPOLOGY;
OK
gungnir> DESC TOPOLOGY;
{
"id":"52373363e4b0a525adadb4f2",
"explain":" ... ",
"status":"RUNNING",
"owner":"tf0054@gmail.com",
"createTime":"2013-09-16T16:35:47.432Z",
"summary":{
...
}
}
gungnir> STOP TOPOLOGY 52373363e4b0a525adadb4f2 ;
DROP TOPOLOGY xxx;
gungnir> DELETE TOPOLOGY 52373363e4b0a525adadb4f2 ;

15. Testing it!
TRACK userAction {"userId":"siniida","hotelId":"226979"};
TRACK commitAction {"userId":"siniida","hotelId":"226979","checkin":"2013-10-18","nights":1};
*** We are changing this name because “TRACK” isn’t fit to its function! (advised in GOTO berlin 2013. Thx.)
*** The function of this command is to make and post a event to gennai.
gungnir> DESC TOPOLOGY;
{
"id":"52373363e4b0a525adadb4f2",
"explain":" ... ",
"status":"RUNNING",
"owner":"tf0054@gmail.com",
...
"hotelId" : "128767"
"hotelId" : "356564"
"hotelId" : "226979"
"hotelId" : "325073"
"hotelId" : "357090"
"hotelId" : "126942"
"hotelId" : "310294"
... USING kafka_emit('${TOPOLOGY_ID}_user')
"hotelId" : "367746"
"hotelId" : "232896"
> bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_user ;
"hotelId" : "220231"
or
"hotelId" : "163175"
> bin/kafka-consumer.sh 52373363e4b0a525adadb4f2_hotel;
"hotelId" : "367856"
"hotelId" : "360863"
"hotelId" : "119747"

16. Knowing http tuple receiver
gungnir> DESC USER;
{
"id":"525e67f8e4b0e7cfca98d040",
"name":"siniida",
"password":"CmGOdN2UU/Xtmk0wRCZKzDy/BN65LAHPCp2+1KMDOOM=",
"createTime":"2013-10-16T10:18:32.849Z"
}
gungnir>
> curl -v -H "Content-type: application/json" -X POST
-d '{"userId":"tf0054","hotelId":"226979"}'
http://dev.genn.ai:9191/gungnir/v1.0/track/525e67f8e4b0e7cfca98d040/userAction

17. Design of genn.ai
Built on Storm
 Converts
 The
 ..
properly (e.g. building AST)
Good Parts (e.g. Supporting parallelism setting)

18. The future of genn.ai
genn.ai wants friends!
 genn.ai
codes will be open-sourced in the future.
 We
are looking for co-challenging partners for
improving genn.ai.
 You
can use genn.ai on your web sites completely for
free (infrastructure etc..) if we can share the dream
together.
Please feel free to contact us!
 http://genn.ai/
 tf0054@gmail.com

19. Thanks!
http://genn.ai/
19

20. 20