Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Better Together: Apache
Cassandra and Apache Kafka
1
2 © DataStax, All Rights Reserved.
Confidential
Agenda
1 Apache Cassandra and Apache Kafka
2 Better Together – Common Patt...
Your Presenters
3 © DataStax, All Rights Reserved.
4 © DataStax, All Rights Reserved.
Confidential
Agenda
1 Apache Cassandra and Apache Kafka
2 Better Together – Common Patt...
Apache Cassandra Overview
• First developed by Facebook
• Top-level Apache project since 2010
• Partitioned row store
• Di...
Apache Kafka Overview
• First developed by LinkedIn
• Top-level Apache Project since 2012
• Distributed streaming platform...
Kafka Concepts
• Topics
– Collection of key/value pairs
– Append-only
– Can be partitioned
• Producers
• Consumers
– Separ...
Kafka Concepts
• Streams applications
– Combined Producer/Consumer
• KSQL
– Query language used by stream
applications
8 ©...
Kafka Concepts
• Brokers
• Clusters
• Connect Framework
– Sources
– Sinks
9 © DataStax, All Rights Reserved.
Cassandra + Kafka – Similarities and Distinctives
• Concepts in common
– Distributed Systems
– Partitioning / Hashing
– Re...
+
Better Together – using the best of both
12 © DataStax, All Rights Reserved.
Confidential
Agenda
1 Apache Cassandra and Apache Kafka
2 Better Together – Common Pat...
Pattern 1: Cassandra + Kafka in Microservices
13 © DataStax, All Rights Reserved.
Some
Producer
My
microservice
DataStax E...
KillrVideo Services Suggested
Videos
Service
DataStax Enterprise
DSE Graph
• UserCreated
• YouTubeVideoAdded
• UserRatedVi...
Confidential© DataStax, All Rights Reserved.
Pattern 2: Kafka into Cassandra
15
16 © DataStax, All Rights Reserved.
Confidential
Agenda
1 Apache Cassandra and Apache Kafka
2 Better Together – Common Pat...
Why a Kafka Connector ?
Account &
Product Usage
Geolocation
ClickStream
Mobile Device
System of records
Event Sources
…
To...
Why a Kafka Connector?
• Spark Streaming = PULL
➢ Enable advanced transformations and
computations
➢ Mode Pull with a dedi...
What is the Kafka Connector ?
Account &
Product Usage
Geolocation
ClickStream
Mobile Device
System of records
Event Source...
What is the Kafka Connector?
• Automatically ingest from Kafka to DSE
– Simple, Fast, Flexible, Secure
• Deployed in the K...
What is the Kafka Connector?
• Built by DataStax drivers team
– Best practices for writing to DSE
– Resiliency of DS drive...
What is the Kafka Connector?
22
Kafka Connect
Data Sinks
DataStax Connector
workers
<standalone-worker>.properties
<distri...
23 © DataStax, All Rights Reserved.
Confidential
Agenda
1 Apache Cassandra and Apache Kafka
2 Better Together – Common Pat...
Academy.datastax.com/downloads
24
What versions does this work with?
25
DSE 5.0+
Supported Versions
- DSE 5.0+
- Confluent 3.2+
- Apache Kafka 0.10.2+
Suppo...
Docker and Datastax
26 Confidential
• WHERE
– https://hub.docker.com/u/datastax/
– https://github.com/datastax/docker-
ima...
https://github.com/clun/kafka-dse/tree/driver2
27 © DataStax, All Rights Reserved. Confidential
Demonstration Overview
© DataStax, All Rights Reserved.
Kafka-dse-producer
TickGenerator
StockTickProducer
timer
Alpha Van...
29 © DataStax, All Rights Reserved.
Confidential
Agenda
1 Apache Cassandra and Apache Kafka
2 Better Together – Common Pat...
Documentation and Examples
• Blog
– https://www.datastax.com/2018/12/introducing-the-datastax-apache-kafka-connector
• Dow...
DataStax Academy
• Free self-paced courses
• DS201: Apache Cassandra™
• DS210: Operations
• DS220: Data Modeling
• DS310: ...
Live Coding on Twitch
• Live coding sessions with advocates and
guests
• Working through the challenges of
building distri...
Join us at Accelerate!
33 © DataStax, All Rights Reserved.
http://www.datastax.com/accelerate
Discount Code: ADVOCATE20
Cassandra + Kafka at Accelerate!
34 © DataStax, All Rights Reserved.
…and additional sessions
Thank you
35 © DataStax, All Rights Reserved.
Confidential
Prochain SlideShare
Chargement dans…5
×

Webinar | Better Together: Apache Cassandra and Apache Kafka

278 vues

Publié le

In this webinar, you’ll also be introduced to DataStax Apache Kafka Connector, and get a brief demonstration of this groundbreaking technology. You’ll directly experience how this tool can help you stream data from Kafka topics into DataStax Enterprise versions of Cassandra. The future of your organization won’t wait. Register now to reserve your spot in this exciting new webinar.

Youtube: https://youtu.be/HmkNb8twUNk

Publié dans : Technologie
  • Soyez le premier à commenter

Webinar | Better Together: Apache Cassandra and Apache Kafka

  1. 1. Better Together: Apache Cassandra and Apache Kafka 1
  2. 2. 2 © DataStax, All Rights Reserved. Confidential Agenda 1 Apache Cassandra and Apache Kafka 2 Better Together – Common Patterns 3 DataStax Kafka Connector 4 Demonstration 5 Resources
  3. 3. Your Presenters 3 © DataStax, All Rights Reserved.
  4. 4. 4 © DataStax, All Rights Reserved. Confidential Agenda 1 Apache Cassandra and Apache Kafka 2 Better Together – Common Patterns 3 DataStax Kafka Connector 4 Demonstration 5 Resources
  5. 5. Apache Cassandra Overview • First developed by Facebook • Top-level Apache project since 2010 • Partitioned row store • Distributed, decentralized • Elastic scalability / high performance • High availability / fault tolerant • Tuneable consistency • Cassandra Query Language (CQL) © DataStax, All Rights Reserved.5 Apache Cassandra ® Apache Software Foundation
  6. 6. Apache Kafka Overview • First developed by LinkedIn • Top-level Apache Project since 2012 • Distributed streaming platform • Used for real-time data pipelines and streaming applications • Horizontal scalability / high performance • High availability / Fault tolerance • Stream persistence and querying (KSQL) • Connect framework 6 © DataStax, All Rights Reserved. Apache Kafka ® Apache Software Foundation
  7. 7. Kafka Concepts • Topics – Collection of key/value pairs – Append-only – Can be partitioned • Producers • Consumers – Separate offsets 7 © DataStax, All Rights Reserved.
  8. 8. Kafka Concepts • Streams applications – Combined Producer/Consumer • KSQL – Query language used by stream applications 8 © DataStax, All Rights Reserved.
  9. 9. Kafka Concepts • Brokers • Clusters • Connect Framework – Sources – Sinks 9 © DataStax, All Rights Reserved.
  10. 10. Cassandra + Kafka – Similarities and Distinctives • Concepts in common – Distributed Systems – Partitioning / Hashing – Replication • Slight differences in implementation – Multi-DC – Log-structured – TTL / retention • Cassandra excels at… – High volume, write intensive data storage workloads at scale – Suitable as a system of record – High performance searching via DSE • Kafka excels at… – Streaming data to/from services and legacy data sources – Acting upon changes in data from multiple sources (aka pipelines) 10 © DataStax, All Rights Reserved.
  11. 11. + Better Together – using the best of both
  12. 12. 12 © DataStax, All Rights Reserved. Confidential Agenda 1 Apache Cassandra and Apache Kafka 2 Better Together – Common Patterns 3 DataStax Kafka Connector 4 Demonstration 5 Resources
  13. 13. Pattern 1: Cassandra + Kafka in Microservices 13 © DataStax, All Rights Reserved. Some Producer My microservice DataStax Enterprise • Consume topic(s) Other consumers • Read / write data • Publish to topic(s)
  14. 14. KillrVideo Services Suggested Videos Service DataStax Enterprise DSE Graph • UserCreated • YouTubeVideoAdded • UserRatedVideo • Populate graph • Graph recommender traversal • Read and write data User Management, Video Catalog, Ratings Cassandra + Kafka – KillrVideo Example
  15. 15. Confidential© DataStax, All Rights Reserved. Pattern 2: Kafka into Cassandra 15
  16. 16. 16 © DataStax, All Rights Reserved. Confidential Agenda 1 Apache Cassandra and Apache Kafka 2 Better Together – Common Patterns 3 DataStax Kafka Connector 4 Demonstration 5 Resources
  17. 17. Why a Kafka Connector ? Account & Product Usage Geolocation ClickStream Mobile Device System of records Event Sources … Topic: stocks-ticks
  18. 18. Why a Kafka Connector? • Spark Streaming = PULL ➢ Enable advanced transformations and computations ➢ Mode Pull with a dedicated runtime (poll) 18 • Kakfa Connector = PUSH ➢ No extra runtime
  19. 19. What is the Kafka Connector ? Account & Product Usage Geolocation ClickStream Mobile Device System of records Event Sources … Topic: stocks-ticks Kafka Connect Sources Sinks HERE
  20. 20. What is the Kafka Connector? • Automatically ingest from Kafka to DSE – Simple, Fast, Flexible, Secure • Deployed in the Kafka Connect framework – Managed through the built-in REST API • Visibility into running connectors and tasks • Endpoints for operator tasks – Automatic rebalancing • Useful for availability and scaling 20 Kafka Connect Sources Sinks HERE
  21. 21. What is the Kafka Connector? • Built by DataStax drivers team – Best practices for writing to DSE – Resiliency of DS drivers 21 Kafka Connect Sources Sinks HERE
  22. 22. What is the Kafka Connector? 22 Kafka Connect Data Sinks DataStax Connector workers <standalone-worker>.properties <distributed-worker>.properties or reads Connectors list <connector>.properties or <connector>.json read Config (mapping) instantiate start
  23. 23. 23 © DataStax, All Rights Reserved. Confidential Agenda 1 Apache Cassandra and Apache Kafka 2 Better Together – Common Patterns 3 DataStax Kafka Connector 4 Demonstration 5 Resources
  24. 24. Academy.datastax.com/downloads 24
  25. 25. What versions does this work with? 25 DSE 5.0+ Supported Versions - DSE 5.0+ - Confluent 3.2+ - Apache Kafka 0.10.2+ Supported offerings - DS Enterprise - DS Basic - DDAC Confluent Apache Kafka 3.2.x+ 0.10.2.x+ 3.3.x+ 0.11.0.x+ 4.0.x+ 1.0.x+ 4.1.x+ 1.1.x+ 5.0.x+ 2.0.x+
  26. 26. Docker and Datastax 26 Confidential • WHERE – https://hub.docker.com/u/datastax/ – https://github.com/datastax/docker- images/tree/master/datastax-docker-image- examples • We provide – Dockers images for DSE, studio, Opscenter – Docker-compose configuration files – Sample Deployments • We support – Installation on dev before 6.7 – Installation on prod from 6.7 (December 2018)
  27. 27. https://github.com/clun/kafka-dse/tree/driver2 27 © DataStax, All Rights Reserved. Confidential
  28. 28. Demonstration Overview © DataStax, All Rights Reserved. Kafka-dse-producer TickGenerator StockTickProducer timer Alpha Vantage 2181 9092 KAFKA-CONNECT 8083 Sources Sinks DSE Kafka-dse-webui Java-dse-driver Spring Web Flux
  29. 29. 29 © DataStax, All Rights Reserved. Confidential Agenda 1 Apache Cassandra and Apache Kafka 2 Better Together – Common Patterns 3 DataStax Kafka Connector 4 Demonstration 5 Resources
  30. 30. Documentation and Examples • Blog – https://www.datastax.com/2018/12/introducing-the-datastax-apache-kafka-connector • Download – https://academy.datastax.com/downloads#connectors • Docs – https://docs.datastax.com/en/kafka/doc/index.html • Demonstration – https://github.com/clun/kafka-dse/tree/driver2 • Examples – https://github.com/datastax/kafka-examples 30 © DataStax, All Rights Reserved.
  31. 31. DataStax Academy • Free self-paced courses • DS201: Apache Cassandra™ • DS210: Operations • DS220: Data Modeling • DS310: Search • DS320: Analytics • DS330: Graph • Kafka Connector Getting Started 31 © DataStax, All Rights Reserved. https://academy.datastax.com
  32. 32. Live Coding on Twitch • Live coding sessions with advocates and guests • Working through the challenges of building distributed systems • Join the conversation and ask questions • Twitch Rewind: Kafka Connector – https://www.youtube.com/watch?v=2_BidD K5zGE https://www.twitch.tv/datastaxacademy 32 © DataStax, All Rights Reserved.
  33. 33. Join us at Accelerate! 33 © DataStax, All Rights Reserved. http://www.datastax.com/accelerate Discount Code: ADVOCATE20
  34. 34. Cassandra + Kafka at Accelerate! 34 © DataStax, All Rights Reserved. …and additional sessions
  35. 35. Thank you 35 © DataStax, All Rights Reserved. Confidential

×