We are making a realtime marketing system for general marketplace type web sites with Kafka + Strom. http://genn.ai/

1. Genn.ai (源内)
Attached realtime marketing platform
1

2. http://genn.ai/
Who am I?
Takeshi NAKANO
Senior Researcher / Architect.
Co-authered Getting started with Solr in Japanese.
Co-authered Hadoop Hacks in Japanese.
2
Writing Getting started with Kafka in Japanese.

3. http://genn.ai/
3
Company Name Recruit Holdings Co., Ltd
Founded March 31, 1960 (incorporated: August 26, 1963）
※governance changed from Recruit Co., Ltd since October 1st, 2012
Headquarters Tokyo, Japan
Number of Group Employees 22,368
（as of the end of March,2012）
Number of Group Companies 102
（Consolidated Subsidiaries/as of the end of June,2012）
Capital 3,002.64 million yen
（since March 1st, 1995）
Consolidated Sales 806,661 million yen
（from April 1st 2011 to March 31st 2012）
Consolidated Ordinary
Income
117,617 million yen
（ from April 1st 2011 to March 31st 2012)

4. http://genn.ai/
4
Recruit
Holdings
Headquarters
Function
Global Business
Business R&D
Recruit Career Co., Ltd
Recruit Jobs Co., Ltd
Recruit Sumai Company Ltd
Recruit Marketing Partners Co., Ltd
Recruit Lifestyle Co., Ltd
Recruit Administration Co., Ltd
Recruit Technologies Co., Ltd
Recruit Communications Co., Ltd
Operating Companies Domain
Recruit Staffing Co., Ltd
STAFF SERVICE HOLDINGS CO., LTD

5. http://genn.ai/
What is our challenge?
Clients can get the information about their customers
checking their items on our web sites.
Providing a printed report monthly.
▽
Send them more detailed report in real time.
In real time fashion, the value of the information will be
dramatically increased!
5

6. http://genn.ai/
Agenda
 What is genn.ai?
 Overview
 How to gather clicks
 0) Injecting our javascripts
6
 Genn.ai core details
 Structure
 1) Setting up filters
 2) Filtering customers
 3) Visualizing them
 Future
 Dealing with historical streams
 Genn.ai wants friends.

7. http://genn.ai/
Overview of using Genn.ai
7
Customer
A
Customer
B
Customer
C
Realtime Analysis
Platform
Genn.ai
じゃらん
Web Sites
Client
(Hotels)
Marketer
Analyst

8. http://genn.ai/
Demo videos!
•They are based on a hotel reservation site like
Expedia.com.
•An animation and a screencapture what the
marketting staff see using Genn.ai.
At this moment ..
•What prefectures are all my users searching for? (0:26)
•Comparing the number of searches in today s and
yesterday s regarding places. (2:50)
8

9. http://genn.ai/
Overview of using Genn.ai
9
Customer
A
Customer
B
Customer
C
Realtime Analysis
Platform
Genn.ai
じゃらん
Web Sites
Client
(Hotels)
Marketer
Analyst

10. http://genn.ai/
Overview of using Genn.ai
10
Customer
A
Customer
B
Customer
C
じゃらん
Web Sites
Realtime Analysis
Platform
Genn.ai
Client
(Hotels)
Marketer
Analyst

11. http://genn.ai/
How to gain access to Genn.ai
11
Customer
A
Customer
B
Customer
C
App
(ex.Tomcat)
Apache
Genn.ai will be able to be attached!

12. http://genn.ai/
How to gain access to Genn.ai
A custom apache module for embeding our
code on returning html sources from App.
12
Customer
B
Apache
Genn.ai will be able to be attached!
mod
Genn.ai
App
(ex.Tomcat)

13. http://genn.ai/
How to gain access to Genn.ai
13
The embedded values and js on a html.

14. http://genn.ai/
Structure
14

15. http://genn.ai/
Steps for getting results
There are 3 steps! (after getting clicks)
15
1 2 3
Pouring clicks
into Kafka
Recording them
as behaviors
Visualizing them
0
gather clicks

16. Backend for Genn.ai
http://genn.ai/
Structure (Pouring into Kafka)
16
Storm
Topology
(Filter)
Kafka
kakan.puTomcat
iPad
(APP)
MongoDB
321
Genn.ai Front
(Based on Finagle)
Query
genn.ai/web)Click Streams
Frontend for Genn.ai
Visualizer for Genn.ai
Zoomdata

17. Backend for Genn.ai
http://genn.ai/
Structure (Recording as behaviors)
17
Storm
Topology
(Filter)
Kafka
kakan.puTomcat
iPad
(APP)
MongoDB
Genn.ai Front
(Based on Finagle)
Query
genn.ai/web)Click Streams
321
Visualizer for Genn.ai
Frontend for Genn.ai
Zoomdata

18. Backend for Genn.ai
Visualizer for Genn.ai
http://genn.ai/
Structure (Visualizing customers)
18
Storm
Topology
(Filter)
Kafka
kakan.puTomcat
iPad
(APP)
MongoDB
Genn.ai Front
(Based on Finagle)
Query
genn.ai/web)Click Streams
321
Frontend for Genn.ai
Zoomdata

19. http://genn.ai/
How is Genn.ai composed?
Genn.ai Frontend! : Gathering clicks.
Genn.ai Backend! : Recording behaviors.
Genn.ai Visualizer! : Visualizing customers’ attrs.
↑
Kafka : Used for message bus
MongoDB : Storing Additional tables
Storm : Processing click streams
19

20. http://genn.ai/
Details of recording behaviors.
20

21. Backend for Genn.ai
http://genn.ai/
Structure (Recording behaviors)
21
Storm
Topology
(Filter)
Kafka
kakan.puTomcat
iPad
(APP)
MongoDB
Genn.ai Front
(Based on Finagle)
Query
genn.ai/web)Click Streams
321
Visualizer for Genn.ai
Zoomdata
Frontend for Genn.ai

22. http://genn.ai/
Calculated values
Recording behaviors
Useful values for each sites
 Ex.)Hotel reservation site,
22
{
"_id": 154,
"largeAreaCntAcl": {
"Tokyo": 3,
"Berlin": 1
},
"largeAreaCntPre": {
"Tokyo": 4,
"Berlin": 3
},
"rateAvgAcl": 4200,
"rateAvgPre": 6200,
"rateCntAcl": [
4, 1, 0, 0, 0
],
"rateCntPre": [
4, 1, 2, 0, 0
],
"lastUpdate": "2013-04-16T04:04:37.432Z"
}

23. http://genn.ai/
Calculated values
Recording behaviors
Useful values for each sites
 Ex.)Hotel reservation site,
 Where is the favorite place?
23
{
"_id": 154,
"largeAreaCntAcl": {
"Tokyo": 3,
"Berlin": 1
},
"largeAreaCntPre": {
"Tokyo": 4,
"Berlin": 3
},
"rateAvgAcl": 4200,
"rateAvgPre": 6200,
"rateCntAcl": [
4, 1, 0, 0, 0
],
"rateCntPre": [
4, 1, 2, 0, 0
],
"lastUpdate": "2013-04-16T04:04:37.432Z"
}

24. http://genn.ai/
Calculated values
Recording behaviors
Useful values for each sites
 Ex.)Hotel reservation site,
 Where is the favorite place?
 Where is the this searching place?
24
{
"_id": 154,
"largeAreaCntAcl": {
"Tokyo": 3,
"Berlin": 1
},
"largeAreaCntPre": {
"Tokyo": 4,
"Berlin": 3
},
"rateAvgAcl": 4200,
"rateAvgPre": 6200,
"rateCntAcl": [
4, 1, 0, 0, 0
],
"rateCntPre": [
4, 1, 2, 0, 0
],
"lastUpdate": "2013-04-16T04:04:37.432Z"
}

25. http://genn.ai/
Calculated values
Recording behaviors
Useful values for each sites
 Ex.)Hotel reservation site,
 Where is the favorite place?
 Where is the this searching place?
 What price range has he booked?
25
{
"_id": 154,
"largeAreaCntAcl": {
"Tokyo": 3,
"Berlin": 1
},
"largeAreaCntPre": {
"Tokyo": 4,
"Berlin": 3
},
"rateAvgAcl": 4200,
"rateAvgPre": 6200,
"rateCntAcl": [
4, 1, 0, 0, 0
],
"rateCntPre": [
4, 1, 2, 0, 0
],
"lastUpdate": "2013-04-16T04:04:37.432Z"
}
 etc..

26. http://genn.ai/
Functions for recording behaviors
Time shift function
Time sensitive counter (old item will be expired and deleted)
Facet ( like Solr )
Categorized counter
Range Facet ( like Solr )
Dynamic categorized counter
26
321

27. http://genn.ai/
Functions for recording behaviors
Time shift function
Time sensitive counter (old item will be expired and deleted)
Facet ( like Solr )
Categorized counter
Range Facet ( like Solr )
Dynamic categorized counter
27
321

28. http://genn.ai/
Time Shift function?
We make 3 min time window and now start.
( no clicks here )
28
t
Now

29. http://genn.ai/
Time Shift function?
In our 3 min time window, 1.5 min are captured.
2 clicks have been recorded.
29
1.5 mins
Now
Time window
▲
Click
▲
Click
2 clicks are captured.
t

30. http://genn.ai/
Time Shift function?
There are 3 clicks in the past.
The length of the capturing window becomes
max.
30
3 mins
Now
Time window
▲
Click
▲
Click
▲
Click
t

31. http://genn.ai/
Time Shift function?
There are 4 clicks in the past.
Ignore the first 2 clicks
31
▲
Click
▲
Click
Now
Time window
▲
Click
▲
Click
1.5 min 3 mins
t

32. http://genn.ai/
Time Shift function?
There are 4 clicks in the past.
Ignore the first 3 clicks
32
▲
Click
▲
Click
Now
Time window
▲
Click
▲
Click
3 mins3 mins
t

33. http://genn.ai/
Functions for recording behaviors
Time shift function
Time sensitive counter (old item will be expired and deleted)
Facet ( like Solr )
Categorized counter
Range Facet ( like Solr )
Dynamic categorized counter
33
321

34. http://genn.ai/
Functions for recording behaviors
Time shift function
Time sensitive counter (old item will be expired and deleted)
Facet ( like Solr )
Categorized counter
Range Facet ( like Solr )
Dynamic categorized counter
34
321

35. http://genn.ai/
Counting the frequency
This function is under the effect of the Time
shift function.
35
{
"_id": 154,
"largeAreaCntAcl": {
"Tokyo": 3,
"Berlin": 1
},
"largeAreaCntPre": {
"Tokyo": 4,
"Berlin": 3
},
"rateAvgAcl": 4200,
"rateAvgPre": 6200,
"rateCntAcl": [
4, 1, 0, 0, 0
],
"rateCntPre": [
4, 1, 2, 0, 0
],
"lastUpdate": "2013-04-16T04:04:37.432Z"
}

36. http://genn.ai/
Counting the frequency
This function is under the effect of the Time
shift function.
You have booked 1 stay in Berlin and 3 stays
in Tokyo in this 3 weeks.
36
▲
L.A.
▲
S.F.
t
Now
Time window
▲
Berlin
▲
Tokyo
1.5 week 3 weeks
▲
Tokyo
"largeAreaCntAcl": {
"Tokyo": 3,
"Berlin": 1
}
▲
Tokyo

37. http://genn.ai/
Details of visualization.
37

38. Backend for Genn.ai
http://genn.ai/
Visualizing the customers
38
Storm
Topology
(Filter)
Kafka
kakan.puTomcat
iPad
(APP)
MongoDB
Genn.ai Front
(Based on Finagle)
Query
genn.ai/web)Click Streams
321
Visualizer for Genn.ai

39. Backend for Genn.ai
http://genn.ai/
Visualizing the customers
39
Storm
Topology
(Filter)
Kafka
kakan.puTomcat
iPad
(APP)
MongoDB
Genn.ai Front
(Based on Finagle)
Query
genn.ai/web)Click Streams
321
Visualizer for Genn.ai

40. http://genn.ai/
Kafka consumer
Kakan.pu
The program for relaying clicks from Kafka to the
other systems.
The enriching function (joining the MongoDB data)
The converting function (for following systems)
40

41. Backend for Genn.ai
http://genn.ai/
Visualizing the customers
41
Storm
Topology
(Filter)
Kafka
kakan.puTomcat
iPad
(APP)
MongoDB
Genn.ai Front
(Based on Finagle)
Query
genn.ai/web)Click Streams
321
Visualizer for Genn.ai
Zoomdata

42. http://genn.ai/
Visualizing tool
Zoomdata
A product containing a set of server app and ipad
app for visualizing event data.
42
321

43. http://genn.ai/
Zoomdata
Zoomdata has the functions like..
WebAPI for streaming data.
Own event processor for vizualizing.
43
321
The bar and bubble graph on default setting.
Customizable visualizing templates.

44. http://genn.ai/
Feature plan.
44

45. http://genn.ai/
Genn.ai Evolution
Adding functions on Genn.ai core.
Genn.ai should have filtering / aggrigation functions
for federating with other systems.
Historical rewinding function will be implemented with
the Hadoop cluster.
45

46. http://genn.ai/
Future plan
Genn.ai wants friends!
 We are looking for co-challenging partners for tuning
and improving Genn.ai.
 You can use Genn.ai on your web sites for totally free
(infrastructure etc..) if we could share the dream.
 Genn.ai codes will be opensourced in the future.
Please feel free to contact us!
 http://genn.ai/
46

47. http://genn.ai/
Thanks!
47

48. 48