Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Distributed Tests on Pulsar with
Fallout
Enrico Olivelli
DataStax - Luna Streaming Team
Apache Pulsar Committer
Member of ...
Agenda
● Testing Distributed Messaging Systems
● Introduction to Fallout
● Fallout architecture
● NoSQLBench
● Anatomy of ...
Testing Distributed Messaging Systems
3
Classic types of tests: Unit tests, Integration tests, System tests
Distributed Sy...
Fallout
4
OpenSource project (ASLv2 licensed), created at DataStax
https://github.com/datastax/fallout
Initially started f...
NoSQLBench
5
OpenSource project (ASLv2 licensed)
https://github.com/nosqlbench/nosqlbench
Allows you to exercise your syst...
Fallout Architecture
6
Key components:
- Provisioners: where to run the test
- Configuration Managers: what to run
- Provi...
Anatomy of a Fallout test - Provisioner and ConfigurationManager
7
# Parameters
image:
name: datastax/pulsar
version: 2.6....
Anatomy of a Fallout test - Workload and Checkers
8
workload:
phases:
- create-topic:
module: kubernetes_job
properties:
m...
Testing Pulsar with Fallout
9
Release stability validation:
- Test cluster wide features, in k8s environments (like k8s fu...
Simulating Bookie failure with ChaosMesh
10
Sample scenario:
- Start a 6 nodes cluster on GKE
- Deploy Apache Pulsar 2.7.2...
Simulating Bookie failure with ChaosMesh
11
Live demo
Key points:
- Fallout UI
- Template
- Parameters
- System wide log a...
Pulsar Release Validation toolkit
12
Repository with sample files for basic release validation and NoSQLBench based testin...
Future works
13
At DataStax we are already using Fallout for Apache Cassandra and Apache Pulsar.
Useful follow ups for the...
Wrapping up
14
Fallout:
- Distributed system tests are hard to design and to deploy
- Testing manually a complex project i...
References
15
LinkedIn - https://www.linkedin.com/in/enrico-olivelli-984b7874/
Twitter: @eolivelli
Apache Pulsar Community...
Thank you !
16
Prochain SlideShare
Chargement dans…5
×
Prochain SlideShare
What to Upload to SlideShare
Suivant
Télécharger pour lire hors ligne et voir en mode plein écran

0

Partager

Télécharger pour lire hors ligne

Distributed Tests on Pulsar with Fallout - Pulsar Summit NA 2021

Télécharger pour lire hors ligne

Fallout is an open source testing framework based on Jepsen. In this talk we will see how distributed testing works and how to use these tools to verify Pulsar quality. We will see how we can easily deploy a reproducible Pulsar cluster on K8S and how to use ChaosMesh to inject failures. We will also cover integrated metrics reporting tools, very useful to verify the behaviour of the system for any Pulsar version, system environment and especially during maintenance operations (rollout restarts/upgrades) and unexpected failures.

  • Soyez le premier à aimer ceci

Distributed Tests on Pulsar with Fallout - Pulsar Summit NA 2021

  1. 1. Distributed Tests on Pulsar with Fallout Enrico Olivelli DataStax - Luna Streaming Team Apache Pulsar Committer Member of Apache BookKeeper and Apache ZooKeeper PMC Apache Curator VP
  2. 2. Agenda ● Testing Distributed Messaging Systems ● Introduction to Fallout ● Fallout architecture ● NoSQLBench ● Anatomy of a Fallout test ● Fallout and Pulsar ● Live Demo ● Future works 2
  3. 3. Testing Distributed Messaging Systems 3 Classic types of tests: Unit tests, Integration tests, System tests Distributed System Tests: ● Launch N machines (or clusters!) ● Deploy applications (Helm, Unzip tarballs…) ● Run clients ● Inject failures ● Perform system wide assertions ● Create reports (performance, failures…) ● Compare reports (regression tests)
  4. 4. Fallout 4 OpenSource project (ASLv2 licensed), created at DataStax https://github.com/datastax/fallout Initially started for Apache Cassandra®, now it is a general purpose tool. A layer on top of Jepsen.io https://jepsen.io/ Design Repeatable Experiments with real clusters: - Declarative language (YAML) - Deterministic Setup-Run-Check loop - Supports k8s natively (helm, kubectl, k8s jobs...) - Integrated with GKE - Monitor longevity tests - Aggregate logs from all nodes/pods
  5. 5. NoSQLBench 5 OpenSource project (ASLv2 licensed) https://github.com/nosqlbench/nosqlbench Allows you to exercise your system: - Load generator - Performance measurement - Distributed execution Supports many drivers: - Apache Cassandra, MongoDB... - JDBC - Generic HTTP based services - Messaging: Kafka, Pulsar, JMS 2.0 For every driver it tracks basic metrics: throughput/latency but also driver specific metrics (like transaction commit time for Pulsar) Integrated with Dropwizard metrics and with Graphite
  6. 6. Fallout Architecture 6 Key components: - Provisioners: where to run the test - Configuration Managers: what to run - Providers: access to the services and information Workload: - Modules: actions - Phases: execution model: concurrent, sequential - Artifact checkers: summarize metrics, produce charts, verify logs
  7. 7. Anatomy of a Fallout test - Provisioner and ConfigurationManager 7 # Parameters image: name: datastax/pulsar version: 2.6.2_1.0.0 ........... --- ensemble: server: node.count: {{cluster.numNodes}} provisioner: name: gke configuration_manager: - name: helm properties: helm.install.values.file: <<file:pulsar-values.yaml>> helm.repo.name: {{helmchart.reponame}}
  8. 8. Anatomy of a Fallout test - Workload and Checkers 8 workload: phases: - create-topic: module: kubernetes_job properties: manifest: <<file:createtopic.yaml>> - produce_messages: module: nosqlbench properties: cycles: {{producer.nummessages}} consume_messages: module: nosqlbench properties: checkers: nofail: checker: nofail artifact_checkers: generate_chart: artifact_checker: hdrtool
  9. 9. Testing Pulsar with Fallout 9 Release stability validation: - Test cluster wide features, in k8s environments (like k8s functions) - Longevity tests - Simulate failures: BookKeeper, ZooKeeper, Broker, Proxy - Simulate rollout restarts - Simulate upgrades Benchmarks: - Hunt for performance regressions, running tests against current ‘master’ branch - Compare different releases (Apache Pulsar, Luna Streaming …) - Measure a given setup (configuration + cluster size + machines), in a reproducible way - Reproduce performance issues
  10. 10. Simulating Bookie failure with ChaosMesh 10 Sample scenario: - Start a 6 nodes cluster on GKE - Deploy Apache Pulsar 2.7.2 using Helm - 1 broker - 3 bookies - 1 proxy - Replication parameters: 2-2-2 (2 copies) - Deploy a NoSQLBench pod - Deploy ChaosMesh (using Helm) - Create a partitioned topic - Produce and Consume messages - Simulate Bookie pod failure (one bookie at a time) - Track time series for latency - No error must be reported by Producers and Consumers Template: https://github.com/datastax/pulsar-fallout/blob/master/benchmarks/template.yaml
  11. 11. Simulating Bookie failure with ChaosMesh 11 Live demo Key points: - Fallout UI - Template - Parameters - System wide log aggregation - Verify Bookie failures in the logs of the Broker - Show latency generated graph
  12. 12. Pulsar Release Validation toolkit 12 Repository with sample files for basic release validation and NoSQLBench based testing: https://github.com/datastax/pulsar-fallout Examples for: - Deploy Pulsar, from 2.7.0 up to your custom docker image - Using Apache Pulsar Helm Chart and Luna Streaming Helm Chart - Running NoSQLBench - Using ChaosMesh for failure injection - Creating custom configurations of Pulsar - Run client tools (pulsar-perf, pulsar-client, pulsar-admin)
  13. 13. Future works 13 At DataStax we are already using Fallout for Apache Cassandra and Apache Pulsar. Useful follow ups for the community: - Contribute the corpus of tests to the Apache repo - Give to the community an easy to test Apache Pulsar with real distributed system tests - Integrate Fallout based validation for pre-release validation or PR validation - Use Fallout Docker images to run tests on GitHub actions Fallout and NoSQLBench are public Open Source projects, everyone can contribute and enhance these powerful tools
  14. 14. Wrapping up 14 Fallout: - Distributed system tests are hard to design and to deploy - Testing manually a complex project is error-prone - Fallout is a brand new framework to easily write distributed system tests - Reproducible - Easy to use (YAML based, declarative style) - NoSQLBench is the perfect companion for Fallout (but you are not required to use it) Filling in the gaps in Pulsar testing: - Systematically test and verify performances - Ensure that Pulsar runs well on real world clusters (k8s as first class citizen) - Be able to reproduce real world workloads in lab
  15. 15. References 15 LinkedIn - https://www.linkedin.com/in/enrico-olivelli-984b7874/ Twitter: @eolivelli Apache Pulsar Community: http://pulsar.apache.org/en/contact/ (Slack, ML…) References: Fallout - https://github.com/datastax/fallout Pulsar Templates - https://github.com/datastax/pulsar-fallout NoSQLBench - https://github.com/nosqlbench/noslqbench Great tutorial about Fallout - https://www.youtube.com/watch?v=45iTmTBjU0M DataStax Fallout - Testing Scaleable Distributed System with Sean McCarthy
  16. 16. Thank you ! 16

Fallout is an open source testing framework based on Jepsen. In this talk we will see how distributed testing works and how to use these tools to verify Pulsar quality. We will see how we can easily deploy a reproducible Pulsar cluster on K8S and how to use ChaosMesh to inject failures. We will also cover integrated metrics reporting tools, very useful to verify the behaviour of the system for any Pulsar version, system environment and especially during maintenance operations (rollout restarts/upgrades) and unexpected failures.

Vues

Nombre de vues

121

Sur Slideshare

0

À partir des intégrations

0

Nombre d'intégrations

2

Actions

Téléchargements

4

Partages

0

Commentaires

0

Mentions J'aime

0

×