This document discusses isolation in Apache Pulsar. It introduces the presenters as experts in distributed systems and the Pulsar open source project. It then outlines ways to isolate resources in Pulsar like brokers, bookies, and clusters to separate namespaces and tenants. The key methods covered are namespace isolation policies, failure domains, anti-affinity groups, and bookie affinity groups. It provides examples of how these are configured and allows scaling resources up and down independently per namespace. Finally, it invites questions and provides contact details.

1. 1

2. Who are We?
● Leading Cloud data Platform Team
● Loves distributed Systems, open Source
● Data geek (data stores, stream, analytics etc)
● Pulsar & MySql contributor
● Developer by passion
● Love Distributed Systems, Streaming Platform
● Pulsar Enthusiast
Shivji Jha
SME for Pulsar,
Nutanix
Sourabh Agrawal
Pulsar Ninja,
Nutanix
2

3. Catalogue
● Quick Intro: Apache Pulsar
● Pulsar Isolation
○ What
○ Why
○ How
● Isolating by clusters
● Isolating brokers
● Isolating bookies
● Demo
3

4. About Pulsar
Apache Pulsar is a cloud-native, distributed, open-source pub-sub messaging and streaming
platform. Originally developed by Yahoo and contributed to the Apache Software Foundation in
2016
Provides :
● Stateless Brokers
● Horizontally Scalable
● Isolate Read and Writes
● Multi-tenant System
● Geo-Replication
● Active Community Support
4

5. Multi-tenancy
5

6. What is Pulsar Isolation ?
● Scale storage and Serving layer independently.
● Can separate resources (brokers & bookies) for specific use cases.
● Scale up/down resources as per need for specific namespaces by keeping rest of the
cluster untouched.
● Localized resource allocation
6

7. Why Pulsar Isolation ?
● Avoid Sharing resources within teams
○ Prevent unexpected consequences.
● Dedicated resources for a namespace.
● Allocate Localized resource in multi-region setup.
● Scale up/down nodes as per load on a namespace.
● More Secure (Permissions managed at namespace level)
● Easy setup
7

8. Isolation Categories
● Hard Isolation
○ Different physical Pulsar clusters (or nodes) for your isolation units
■ Failure Domains
■ Anti affinity Groups
■ Broker Isolation using Namespace Isolation Policy
■ Bookie Isolation using Affinity Groups
● Soft Isolation
○ Same Physical clusters but restrictions on tenants / namespaces by configuration
■ Disk Quotas
■ Throttling
8

9. Isolating by Configuration
9

10. Soft isolation
● Limit by configurations
○ Namespace:
● encryptionRequired
● Auth_policies
● Backlog_quota
● retention
■ Topic:
● maxProducersPerTopic
● maxConsumersPerTopic
● maxSubscriptionsPerTopic
● Limits by resources (capped per broker):
■ loadBalancerBrokerMaxTopics
■ loadBalancerBrokerOverloadedThresholdPercentage
10

11. Isolating by Cluster
11

12. Geo Replication
eu-west-2 eu-west-3
Geo
Replication
On
12
New Hotel company starts operating in London
Expand business to Paris
Enable HA

13. Isolating Brokers
13

14. 14
https://github.com/apache/pulsar/wiki/PIP-7:-Pulsar-Failure-domain-and-Anti-affinity-namespaces
● Failure Domain
○ Group brokers into failure domains
○ Reduces blast radius
○ Example
■ Fix a bug / patch
■ Upgrade brokers by failure domain
■ Anti affinity of namespaces
Failure Domains

15. 15
https://github.com/apache/pulsar/wiki/PIP-7:-Pulsar-Failure-domain-and-Anti-affinity-namespaces
● Anti affinity Group
○ Group of namespaces that should stay far.
○ Best effort
■ Prefer different failure domain
■ Prefer different broker
Anti affinity Groups

16. 16
What is “Namespace Isolation Policy”?

17. Namespace Isolation Policy: Internals
min_availability=1
Primary = Broker 1,2
17

18. Namespace Isolation Policy: Internals
min_availability=1
Primary = Broker 1,2
18

19. Namespace Isolation Policy: Internals
min_availability=1
Primary = Broker 1,2
19

20. Namespace Isolation Policy: Internals
min_availability=1
Secondary = Broker 5
Primary = Broker 1,2
20

21. Namespace Isolation Policy: Setup
Configure Isolation Policy
Unload the namespace to take effect for existing topics in namespaces or restart brokers.
./pulsar-admin ns-isolation-policy set --auto-failover-policy-type min_available
--auto-failover-policy-params min_limit=?,usage_threshold=80 --namespaces <tenant/namespace>
--primary broker-node-1,broker-node2 my-cluster <policy-name> --secondary broker-node-5
./pulsar-admin namespaces unload <tenant/namespace>
./pulsar-admin topics lookup <tenant/namespace/topic>
21

22. Isolating Bookies
22

23. Bookie Affinity Groups
23
● Isolate storage at namespace
● Leverage data durability configs in
isolated bookies
● Leverage
ZkIsolatedBookieEnsemblePlacement
Policy
● Scale bookies for namespace
https://streamnative.io/uploads/images/blogs/isolation-3.png

24. Bookie Isolation using
Affinity Group
Setup Bookie Rack
Bookie Affinity Group
./pulsar-admin bookies set-bookie-rack --bookie 127.0.0.1:3181 --hostname 127.0.0.1:3181
--group group1 --rack rack1
./pulsar-admin namespaces set-bookie-affinity-group <tenant/namespace> --primary-group
<anti-affinity-group-name> --secondary-group <anti-affinity-group-name>
./pulsar-admin namespaces set-bookie-affinity-group public/default --primary-group group1
24
Configure Bookie Group
Configure Bookie Group with secondary

25. Scaling Up/Down
Broker
● When scaling up brokers, change the isolation groups and add newly added broker as
primary or secondary broker for required namespaces.
● When scaling down, make sure existing isolation groups have enough brokers.
Bookie
● When scaling up bookies, change the bookie affinity group and add newly added bookies as
primary or secondary group for required namespaces.
● When scaling down, make sure existing affinity groups have enough bookies.
25

26. Demo : Namespace Isolation Policy
26

27. References
Namespace Isolation: https://pulsar.apache.org/docs/en/administration-isolation/#broker-isolation
RackAware Ensemble Policy:
https://bookkeeper.apache.org/docs/latest/api/javadoc/org/apache/bookkeeper/client/EnsemblePl
acementPolicy.html
Bookie Affinity Group:
https://pulsar.apache.org/docs/en/administration-isolation/#bookie-isolation,
https://github.com/apache/pulsar/pull/3933
Crash course on pulsar :
https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works
Soft Isolation:
https://streamnative.io/en/blog/tech/2021-03-02-taking-an-in-depth-look-at-how-to-achieve-isolat
ion-in-pulsar/
27

28. Questions?
Pulsar Slack
https://apache-pulsar.slack.com/
Pulsar Mail Group:
users@pulsar.apache.org
dev@pulsar.apache.org
LinkedIn :
https://www.linkedin.com/in/sourabhaggrawal/
https://www.linkedin.com/in/shivjijha/
28