Contenu connexe
Similaire à Kafkaによるリアルタイム処理
Similaire à Kafkaによるリアルタイム処理 (7)
Kafkaによるリアルタイム処理
- 1. Kafka
Twitter: yanaoki
2011/11/27
16 +WEB
http://www.flickr.com/photos/devnull/19765635/
2011 11 27
- 2. •
•
• Kafka
• Kafka
2011 11 27
- 3. • Twitter: @yanaoki
•
•
• Java Ruby Hadoop/Mahout Cassandra
2011 11 27
- 9. • Facebook Insight
• PUMA Scribe / HDFS / pTail / HBase /Thrift
• http://slidesha.re/ijWfPh
• Twitter Promoted Tweets Reporting
• Rainbird ZooKeeper / Cassandra
• http://slidesha.re/dRxtIp
• Twitter Web Analytics
• Storm Zookieper
• http://slidesha.re/qbpKbY
• Google Analytics
•
• LinktedIn
• Kafka ZooKeeper
2011 11 27
- 10. Kafka
• Kafka
•
• Linkedin
• http://incubator.apache.org
•
2011 11 27
- 11. Linkedin
•
• SNS
•
• 2011 11
• Kafka
• 2010 11
• 2011 07 Apache incubator project
• a Distributed Messaging System for Log Processing
• http://research.microsoft.com/en-us/um/people/
srikanth/netdb11/netdb11papers/netdb11-final12.pdf
2011 11 27
- 12. Kafka
•
• Kafka
•
•
•
• Hadoop/HDFS
• DWH
•
2011 11 27
- 14. •
• ZooKeeper
2011 11 27
- 15. •
• SPOF
• ZooKeeper
•
2011 11 27
- 16. Push or Pull
• Push
• scribe flume
• Pull
• Kafka
•
2011 11 27
- 17. • Pub/Sub
•
• Publish
•
• Subscribe
• subscribe
•
Kafka
2011 11 27
- 18. •
•
•
•
• Kafka
• Zookeeper
•
2011 11 27
- 19. •
•
•
•
• ZooKeeper
•
•
•
2011 11 27
- 25. ● ●
― ―
● ● ●
―
●
● ● ● ●
● ZK
● ●
●
ZooKeeper ― ●
2011 11 27
- 26. •
• Kafka O(log n) O(1)
• OS
• Java GC
• BTree
2011 11 27
- 27. •
•
•
• Java
NIO (※ Linux sendfile )
2011 11 27
- 28. HUG January 2011 Kafka Presentation
http://www.slideshare.net/ydn/hug-january-2011-kafka-presentation
2011 11 27
- 30. Facebook
• MapReduce(Not HadoopMR)
• Scribe/PTail/Puma Map
• HBase Reduce
http://www.slideshare.net/tatsuya6502/tokyo-hbase-meetup-realtime-big-data-at-facebook-ja
Map Reduce
2011 11 27
- 31. Kafka HBase
• Twitter
• (en ja
lang
client
lang HBaseImport
Comsumer
TwitterStreaming
Producer
ja→10
en→32
Kafka Broker
2011 11 27
- 32. Kafka HBase
• Twitter
lang
client
lang
HBaseImport
Comsumer
TwitterStreaming
Producer ja→10
en→32
client web → 100
iPhoone→10
Kafka Broker Android→10
2011 11 27
- 33. Hadoop
• Hadoop
• Hadoop Map Kafka
• KafkaETLJob / KafkaETLInputFormat / KafkaETLRecordReader
• MapReuduce API
2011 11 27
- 34. Hadoop
Offset
Offset
HDFS
Map
Offset
Offset Limit
map
→
Kafka Mapper Kafka
Reducer
HDFS
2011 11 27