Ce diaporama a bien été signalé.
Le téléchargement de votre SlideShare est en cours. ×

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement Database

Publicité
Publicité
Publicité
Publicité
Publicité
Publicité
Publicité
Publicité
Publicité
Publicité
Publicité
Publicité

Consultez-les par la suite

1 sur 22 Publicité

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement Database

Télécharger pour lire hors ligne

(Bruno Simic, Solutions Engineer, Couchbase)

Breakout during Confluent’s streaming event in Munich. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.

(Bruno Simic, Solutions Engineer, Couchbase)

Breakout during Confluent’s streaming event in Munich. This three-day hands-on course focused on how to build, manage, and monitor clusters using industry best-practices developed by the world’s foremost Apache Kafka™ experts. The sessions focused on how Kafka and the Confluent Platform work, how their main subsystems interact, and how to set up, manage, monitor, and tune your cluster.

Publicité
Publicité

Plus De Contenu Connexe

Diaporamas pour vous (20)

Similaire à Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement Database (20)

Publicité

Plus par confluent (20)

Plus récents (20)

Publicité

Billions of Messages in Real Time: Why Paypal & LinkedIn Trust an Engagement Database

  1. 1. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. Milliarden von Messages in Echtzeit: Warum Paypal & LinkedIn auf eine Engagement-Datenbank vertrauen 09.10.2018 Bruno Šimić – Solutions Engineer
  2. 2. ATTRIBUTES OF AN ENGAGEMENT DATABASE Always on, always fast Secure, secure, secure Built-in smarts Seamlessly mobile Hello cloud, hello world Built for change - at scale
  3. 3. Why Customers Choose Couchbase? Memory-first Architecture Full SQL Query Language Active-Active Global Data Replication Multi-dimensional scaling Mobile Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
  4. 4. Core Principles of Couchbase Platform Evolution Started with Core Principles – True auto sharding – JSON-based flexible data model – Memory-first Architecture – Asynchronous approach to everything – Scale workloads independently Managed Cache Key-Value Store Document Database Mobile N1QL Query Full Text Search Analytics Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
  5. 5. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 6Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. Couchbase Hybrid / Multi-Cloud Data Platform • Service-Centric Clustered Data System - Multi-process Architecture - Dynamic Distribution of Facilities - Cluster Map Distribution - Automatic Failover • Offline Mobile Data Integration • N1QL - SQL-like Query Engine for JSON • Global Secondary Indexes • Lowest Latency Key-Value API • Active-Active Inter-DC Replication • Full-Text Search • Eventing • Operational Analytics On-premPrivate Cloud AWSAzure GCP Eventing Mobile Query Key Value Analytics Indexing Full Text Search
  6. 6. Couchbase designed for containerized applications
  7. 7. Introducing Couchbase Autonomous Operator Couchbase Autonomous Operator is an application-specific controller that extends the Kubernetes API to create, configure and manage instances of complex stateful applications on behalf of a Kubernetes user. It builds upon the basic Kubernetes resource and controller concepts, but also includes domain or application-specific knowledge to automate common tasks better managed by computers.
  8. 8. Couchbase K8 operator Architecture
  9. 9. Next Generation Data Management Architecture Data Sources Enterprise Applications Social Media Web & Ecommerce Mobile AppsSensor Data MainframeExternal DataSystem Logs Data Integration Speed Layer Batch Layer Data Management Data Lake Data Warehouse Business Intelligence Dashboards Query ToolsData Access In Memory Cache Data Serving Layer
  10. 10. Next Generation Data Management Architecture Enterprise Applications Social Media Web & Ecommerce Mobile AppsSensor Data MainframeExternal DataSystem Logs Speed Layer Batch Layer Data Lake Business Intelligence Dashboards Query Tools In Memory Cache Data Warehouse N1QL Query Workbench Data Sources Data Integration Data Management Data Access Data Serving Layer Real Time Data Ingestion
  11. 11. Ingest, Process, Load and Serve Data at global scale Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
  12. 12. Kafka Connect Couchbase Connector 3.4 Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. • Use Couchbase as either a consumer or producer with Kafka message queues • Continuous streaming, filter, and transformation of events to and from Couchbase with Source and Sink connectors. • Fast, reliable and fault tolerant: Based on DCP (Couchbase replication protocol). • Efficient: Only load new or modified documents. • Real-time: Every mutation to Couchbase generates an event which is published to a Kafka topic. • Compression and IPv6 support • Support for rollback mitigation Couchbase cluster … Kafka cluster Kafka Connect (Connectors to Extract and Load data)
  13. 13. Confluent and Couchbase - Synergies Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. • Distributed and fault tolerant • Horizontally scalable • Geographically replicated • Low latency • Open source
  14. 14. Customer Spotlight 1.1 trillion hits a day 75% of flight bookings worldwide are made through Amadeus 50M Unique monthly visitors 2.5B monthly page views Replaced MongoDB 2821 nodes, 100+ clusters 16M entries every 5 min 2.5 millon ops/sec. on a single cluster 1 billion+ documents 10TB+ data Sub-2oo ms response time E-Commerce Travel CommunicationsGaming Financial Services Health Industrial IoTDigital Media
  15. 15. 17 Objectives & Challenges Provide business users with real time reports and visualizations of user interaction data • Collect web and mobile clickstream in real time • Integrate with other big data technologies (Hadoop and Storm) • Provide views of data across multiple dimensions (e.g., time, location, browser and device types) Solution Deploy Couchbase Server to capture, store, and process real time web data • Ingests data (via Storm) from multiple inputs, including mobile, web, and other services, storing data as JSON documents • Integrates with Hadoop to pass data for additional offline analytics • 130m+ active accounts, in 190+ countries, 25 currencies • 10TB data • 1B documents The Couchbase Advantage Real time performance, easy integration with Kafka, Storm and Hadoop Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. User Activity Tracking and real-time analytics @ Paypal Results • Consistent low latency (sub 10-msec response) • High availability enabled by distributed caching and XDCR • Views for business users are generated in under 1 minute, based on 10-minute data collection intervals
  16. 16. 17 Objectives & Challenges Read scaling and TCO becomes important • Constant growth of user profiles, groups, jobs and publications • Difficult to move data across data centers • Very complex administration of the system • Risk of security breach by using many different components Solution Deploy Couchbase Server to cache, store and replicate data in real time with low latency • One singe solution for in memory caching, ephemeral counter store, temporary de-duping store and SoT for internal tooling • Center of excellence for Couchbase within Linkedin created • LinkedIn build ist own NoSQL databas (Voldemort), Couchbase is still used as the main database for this use cases • 560M users, 26M companies, 15M active jobs listings • 1.5B documents in cluster • 10M queries/sec • 30 Offices in 24 countries • 24 languages Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. Read scaling & data replication across DCs worldwide Results • Consistent low latency • High availability enabled by distributed caching and XDCR • Higher acceptance at users • Scalable and performant system with zero downtime • Lower TCO
  17. 17. Additional information Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
  18. 18. Couchbase Mobile: The Full-Stack Mobile Data Platform Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. Lightweight embedded NoSQL database with full CRUD and query functionality. Secure web gateway with synchronization, data access, and data integration APIs for accessing, integrating, and synchronizing data over the web. Highly scalable, highly available, high performance NoSQL database server. Built-in enterprise level security throughout the entire stack includes user authentication, user and role based data access control (RBAC), secure transport (TLS), and 256-bit AES full database encryption. Couchbase Lite Sync Gateway Couchbase Cluster SECURITY EMBEDDED DATABASE SYNCHRONIZATION DATABASE SERVERCLIENT WEB TIER DATABASEinternet intranet
  19. 19. Developing with Couchbase Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. Couchbase supports a wide range of frameworks, languages, platforms and infrastructure choices. FRAMEWORKS LANGUAGES PLATFORMS INFRASTRUCTURE MOBILEMOBILE
  20. 20. Services & Training geared to customer success 2 3 Professional Services • Center of Excellence with product experts • Packaged Services • Workshops • Custom Consulting • Comprehensive NoSQL & Couchbase training • Hands-on, intensive labs • Best practices based on 100s of real use cases • Developer and Admin courses • Global Couchbase facilities, or at customer site Learning Services Email inquiries: Services@couchbase.com Visit us: http://www.couchbase.com/couchbase-services Email inquiries: Training@couchbase.com Visit us: http://training.couchbase.com
  21. 21. Additional Resources • Kafka connector: https://docs.couchbase.com/kafka- connector/3.4/index.html • Blog: http://blog.couchbase.com • Forum: http://forums.couchbase.com • General Docs: http://docs.couchbase.com • Developer Portal: http://developer.couchbase.com • Couchbase Labs: https://github.com/couchbaselabs • Query Portal: http://query.couchbase.com Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved.
  22. 22. Thank you 22

×