Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Kafka AWS Kinesis
/ @laclefyoshi / ysaeki@r.recruit.co.jp
•
• Kafka Kinesis
•
•
•
•
2
• 2011/04
• 2015/09
•
• Druid (2015; Kindle Direct Publishing)
• RDB NoSQL (2016; ; HBase )
•
• (WebDB Forum 2014)
• Spark...
Kafka Kinesis
• Apache
• Publish-Subscribe messaging rethought as a
distributed commit log
• AWS Kinesis
• Stream
• Fireho...
Kafka Kinesis
• Apache
• Publish-Subscribe messaging rethought as a
distributed commit log
• AWS Kinesis
• Stream
• Fireho...
:
5
: <100ms
: ~8000msgs/sec
Kinesis
[Kafka] [Kinesis]
6
[Kafka] [Kinesis]
6
[Kafka] [Kinesis]
Kafka Connect
Kafka-rest
Kafka-Pixy
Kastle
AWS
API Gateway
HTTP API
ETL ETL 7
[Kafka] [Kinesis]
Kafka Connect
Kafka-rest
Kafka-Pixy
Kastle
AWS
API Gateway
HTTP API
ETL ETL
OSS
•Kafka Streams
•Pipeline...
[Kafka] [Kinesis]
JMX Reporter
Kerberos
Cloudwatch
AWS
7
← →
← →
← →
8
[Kafka] [Kinesis]
JMX Reporter
Kerberos
Cloudwatch
AWS
7
← →
← →
← →
8
[ ]
http://insightdataengineering.com/blog/ingestion-comparison/
X
Kafka Kinesis
• Kafka or Kinesis
• Kafka OSS
Kinesis
•
9
Kafka Kinesis
• (< 5000msgs/sec) Kinesis
• Kafka
• Kinesis
• Hadoop Kafka
• Kinesis
• Kafka
10
• OSS → Kafka
• → Kinesis
• API
•
• SNS Bot IoT
•
11
Prochain SlideShare
Chargement dans…5
×

KafkaとAWS Kinesisの比較

8 381 vues

Publié le

Apache Kafka Meetup Japan #1 http://connpass.com/event/32293/ LT発表資料

Publié dans : Technologie
  • Soyez le premier à commenter

KafkaとAWS Kinesisの比較

  1. 1. Kafka AWS Kinesis / @laclefyoshi / ysaeki@r.recruit.co.jp
  2. 2. • • Kafka Kinesis • • • • 2
  3. 3. • 2011/04 • 2015/09 • • Druid (2015; Kindle Direct Publishing) • RDB NoSQL (2016; ; HBase ) • • (WebDB Forum 2014) • Spark Streaming (Spark Meetup December 2015) 3
  4. 4. Kafka Kinesis • Apache • Publish-Subscribe messaging rethought as a distributed commit log • AWS Kinesis • Stream • Firehose • Analytics : Pub/Sub : : (SQL) 4
  5. 5. Kafka Kinesis • Apache • Publish-Subscribe messaging rethought as a distributed commit log • AWS Kinesis • Stream • Firehose • Analytics : Pub/Sub : : (SQL) 4
  6. 6. : 5 : <100ms : ~8000msgs/sec Kinesis
  7. 7. [Kafka] [Kinesis] 6
  8. 8. [Kafka] [Kinesis] 6
  9. 9. [Kafka] [Kinesis] Kafka Connect Kafka-rest Kafka-Pixy Kastle AWS API Gateway HTTP API ETL ETL 7
  10. 10. [Kafka] [Kinesis] Kafka Connect Kafka-rest Kafka-Pixy Kastle AWS API Gateway HTTP API ETL ETL OSS •Kafka Streams •PipelineDB AWS •Kinesis Analytics 7
  11. 11. [Kafka] [Kinesis] JMX Reporter Kerberos Cloudwatch AWS 7 ← → ← → ← → 8
  12. 12. [Kafka] [Kinesis] JMX Reporter Kerberos Cloudwatch AWS 7 ← → ← → ← → 8
  13. 13. [ ] http://insightdataengineering.com/blog/ingestion-comparison/ X
  14. 14. Kafka Kinesis • Kafka or Kinesis • Kafka OSS Kinesis • 9
  15. 15. Kafka Kinesis • (< 5000msgs/sec) Kinesis • Kafka • Kinesis • Hadoop Kafka • Kinesis • Kafka 10
  16. 16. • OSS → Kafka • → Kinesis • API • • SNS Bot IoT • 11

×