Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or
dist...
Speakers
2
Rajan Dhabalia
Principal Software Engineer, Verizon Media
Ludwig Pummer
Principal Production Engineer, Verizon ...
Agenda
● Pulsar in Yahoo/Verizon Media
● Multi tenancy
● Security
● SNI routing and proxy support
● Future
● QA
3
Pulsar journey in Yahoo
● Developed as a hosted pub-sub service within Yahoo/VMG
○ open-sourced in 2016
● Global deploymen...
● Pulsar scale and storage evolution talk
https://pulsar-summit.org/en/event/virtual-conference-2020/sessions/pulsar-stora...
6
Secured multi-tenant system with Apache Pulsar
Multi-tenancy & Security Requirement
7
Multi-tenancy
Tenant and Namespace
IO isolation
Quota and Throttling
Broker and Boo...
Multi-tenancy
8
Tenant
● Highest level of provisioning
● Unit of administration
● Managed by Pulsar
administrators
● Usually one team
9
Te...
1. Portal find User to Team mapping
2. User creates or modifies tenant
○ Tenant name, Admin Authorization Principals
○ Clu...
11
IO Isolation
Writer Reader
Journal Data File
Data Device
Journal Device
Write Reads (cold)
Storage Quota
● Tenant-controlled
● Namespace-level and
Topic-level
● Storage Limit
● Policy
Throttling
● Pulsar Administr...
Broker Isolation
● Regex of Namespaces to
Regex of Brokers/IP Range
● Primary and Secondary broker
Regexes
13
Broker Isola...
Bookie Isolation
● Bookies to “Affinity Group”
● Namespace(s) to
Primary/Secondary Affinity
Group
● Rack-Aware within group
14...
● Common unit of failure
for multiple brokers
15
Failure Domain
bin/pulsar-admin clusters 
create-failure-domain 
cluster-...
● Assign Namespaces to
Anti-Affinity Group
● Changes Load Balancer
Behavior
16
Anti-affinity group
bin/pulsar-admin namespaces...
Security
17
● Authentication
○ TLS Authentication
○ Athenz
○ Kerberos
○ JSON Web Token Authentication
○ Pluggable authentication provi...
19
Encryption over the wire
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar+ssl://pulsar-broler:6651/")
....
Producer creation
Producer producer = pulsarClient.newProducer()
.topic(
"persistent://my-tenant/my-ns/my-topic"
)
.addEnc...
● Proxy for hybrid could application
● Gateway in a cloud environment or on
Kubernetes
21
Pulsar Proxy: Public cloud acces...
● Proxy server creates a TLS tunnel between remote client and server
● The goal is to enable external clients to connect t...
23
Pulsar client: SNI Routing
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar+ssl://pulsar-broker:6651/")...
24
Cross Organization geo-replication
pulsar-admin clusters create orgB-cluster 
--broker-url-secure pulsar+ssl:// orgB-br...
Future Roadmap
● Tenant based broker virtualization
○ Container based brokers on BookKeeper service
● Hybrid cloud deploym...
Questions?
26
Thank you
Rajan Dhabalia rdhabalia@verizonmedia.com
Ludwig Pummer ludwig@verizonmedia.com
Prochain SlideShare
Chargement dans…5
×

1

Partager

Télécharger pour lire hors ligne

Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pulsar Summit NA 2021

Télécharger pour lire hors ligne

With the rise of the number of tenants and traffic in the cluster, we are always striving for a system that is both multi-tenant and secure enough to onboard applications having different use cases and those applications can access pulsar from different cloud providers or even from cross-organization for enterprise integration.
Large organizations use TLS proxy servers which act as a gateway between a local network and a large-scale network, such as the internet. Aside from traffic forwarding, proxy servers provide security by hiding the actual IP address of a server. Organizational policies often require systems to stay behind enterprise proxy/gateway servers such as HAProxy, ATS, Nginx and follow standard security regulations to protect systems against known vulnerabilities. Apache Pulsar provides various solutions for TLS proxy and Pulsar is the only messaging system that supports SNI proxy to leverage various enterprise proxy solutions.

In this talk, we will discuss security and proxy solutions for Apache Pulsar which enables users in multi-tenant environments to access Pulsar instances securely from the on-prem, public cloud, and cross-enterprise. We will also talk about different multi-tenancy dimensions of Apache Pulsar which we use in Verizon Media to serve different use cases and applications on a shared pulsar cluster.

Livres associés

Gratuit avec un essai de 30 jours de Scribd

Tout voir

Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pulsar Summit NA 2021

  1. 1. Confidential and proprietary materials for authorized Verizon personnel and outside agencies only. Use, disclosure or distribution of this material is not permitted to any unauthorized persons or third parties except by written agreement. Apache Pulsar Multi-tenancy and Security June 17, 2021 Rajan Dhabalia rdhabalia@verizonmedia.com Ludwig Pummer ludwig@verizonmedia.com 1
  2. 2. Speakers 2 Rajan Dhabalia Principal Software Engineer, Verizon Media Ludwig Pummer Principal Production Engineer, Verizon Media
  3. 3. Agenda ● Pulsar in Yahoo/Verizon Media ● Multi tenancy ● Security ● SNI routing and proxy support ● Future ● QA 3
  4. 4. Pulsar journey in Yahoo ● Developed as a hosted pub-sub service within Yahoo/VMG ○ open-sourced in 2016 ● Global deployment ○ 6 DC (Asia, Europe, US) ○ full mesh replication ● Mission critical use cases ○ Serving applications ○ Lower latency bus for use by other low latency services ○ Write availability 4
  5. 5. ● Pulsar scale and storage evolution talk https://pulsar-summit.org/en/event/virtual-conference-2020/sessions/pulsar-storage-on-bookkeepe r-seamless-evolution ● Pulsar growth since 2015 ○ 120+ tenants and 15M rps ○ Storage evolution : HDD, SDD, NVMe, PMEM ○ On-prem, public-cloud and cross org integration ● Scale but what about multi-tenancy? Scale & Multi-Tenancy 5
  6. 6. 6 Secured multi-tenant system with Apache Pulsar
  7. 7. Multi-tenancy & Security Requirement 7 Multi-tenancy Tenant and Namespace IO isolation Quota and Throttling Broker and Bookie isolation Anti-affinity group Security Authentication & Authorization Encryption in transit Encryption at rest Pulsar proxy Support ATS, HAProxy, Nginx
  8. 8. Multi-tenancy 8
  9. 9. Tenant ● Highest level of provisioning ● Unit of administration ● Managed by Pulsar administrators ● Usually one team 9 Tenant and Namespace Namespace ● Middle level of provisioning ● Unit of data policy ● Managed by Pulsar administrators and/or Tenants ● Usually one application/use case persistent://tenant/namespace/topic
  10. 10. 1. Portal find User to Team mapping 2. User creates or modifies tenant ○ Tenant name, Admin Authorization Principals ○ Clusters, WPS & RPS Estimates ○ Jira project, Contact Info, Documentation Link 3. Portal reviews capacity & calls Admin API to manage tenant ○ Jira ticket for Pulsar operator if needed 10 Self-Service Tenant Management
  11. 11. 11 IO Isolation Writer Reader Journal Data File Data Device Journal Device Write Reads (cold)
  12. 12. Storage Quota ● Tenant-controlled ● Namespace-level and Topic-level ● Storage Limit ● Policy Throttling ● Pulsar Administrator-controlled ● Namespace-level ● Publish Rate (broker) ● Dispatch Rate ● Replicator Dispatch Rate ● Max ○ Producers ○ Subscriptions ○ Consumers ○ Unacked Messages 12 Quota & Throttling
  13. 13. Broker Isolation ● Regex of Namespaces to Regex of Brokers/IP Range ● Primary and Secondary broker Regexes 13 Broker Isolation Why ● High Profile/Reserved capacity ● Misbehaving tenants ● Debugging bin/pulsar-admin ns-isolation-policy set --auto-failover-policy-type min_available --auto-failover-policy-params min_limit=5,usage_threshold=80 --namespaces ‘my-tenant/.*’ --primary ‘broker-mytenant[0-9]+.mydomain’ --secondary ‘spare[0-9]+.mydomain’ my-cluster policy-name
  14. 14. Bookie Isolation ● Bookies to “Affinity Group” ● Namespace(s) to Primary/Secondary Affinity Group ● Rack-Aware within group 14 Bookie Isolation Why ● SLA ● High Profile/Reserved capacity bin/pulsar-admin bookies set-bookie-rack -b 1.1.1.1:3181 -g group-bookie1 --hostname bookie1.mydomain -r /default-rack ... bin/pulsar-admin namespaces set-bookie-affinity-group my-tenant/my-namespace1 --primary-group group-bookie1
  15. 15. ● Common unit of failure for multiple brokers 15 Failure Domain bin/pulsar-admin clusters create-failure-domain cluster-name --domain-name domain-1 --broker-list broker-1,broker-2 Broker-1 Broker-2 Domain-1 Broker-3 Broker-4 Domain-2 Namespace-1 Namespace-2 Namespace-3 Namespace-4 1 2 3 4 Loadbalancer: Namespace assignment sequence Anti-affinity-namespaces: “Namespace-X”
  16. 16. ● Assign Namespaces to Anti-Affinity Group ● Changes Load Balancer Behavior 16 Anti-affinity group bin/pulsar-admin namespaces set-anti-affinity-group tenant/namespace1 --group tenant-aag-a bin/pulsar-admin namespaces set-anti-affinity-group tenant/namespace2 --group tenant-aag-a Broker-1 Broker-2 Domain-1 Broker-3 Broker-4 Domain-2 Namespace-1 Namespace-2 Namespace-3 Namespace-4 1 2 3 4 Loadbalancer: Namespace assignment sequence Anti-affinity-namespaces: “Namespace-X”
  17. 17. Security 17
  18. 18. ● Authentication ○ TLS Authentication ○ Athenz ○ Kerberos ○ JSON Web Token Authentication ○ Pluggable authentication provider ● Authorization ○ Pluggable authorization provider ○ Default authorization provider on metadata service 18 Authentication & Authorization
  19. 19. 19 Encryption over the wire PulsarClient client = PulsarClient.builder() .serviceUrl("pulsar+ssl://pulsar-broler:6651/") .tlsTrustCertsFilePath("/ca.cert.pem") .authentication(AUTH, "tlsCertFile:/cert.pem,"+"tlsKeyFile:/key.pem") .enableTlsHostnameVerification(true) .build();
  20. 20. Producer creation Producer producer = pulsarClient.newProducer() .topic( "persistent://my-tenant/my-ns/my-topic" ) .addEncryptionKey("myappkey") .cryptoKeyReader(new MyCryptoKeyReader()) .create(); 20 Encryption at rest Consumer creation Consumer consumer = pulsarClient.newConsumer() .topic( "persistent://my-tenant/my-ns/my-topic" ) .subscriptionName( "my-subscriber-name" ) .cryptoKeyReader(new MyCryptoKeyReader()) .subscribe();
  21. 21. ● Proxy for hybrid could application ● Gateway in a cloud environment or on Kubernetes 21 Pulsar Proxy: Public cloud access Proxy Configuration brokerServiceURLTls=pulsar+ssl://brokers.example.com:6651 brokerWebServiceURLTls=https://brokers.example.com:8443
  22. 22. ● Proxy server creates a TLS tunnel between remote client and server ● The goal is to enable external clients to connect to internal services and do their own client certificate verification, possibly because distribution of private keys to the edge Traffic Server instances is too difficult or too risky. 22 Support Layer-4 SNI Routing
  23. 23. 23 Pulsar client: SNI Routing PulsarClient client = PulsarClient.builder() .serviceUrl("pulsar+ssl://pulsar-broker:6651/") .enableTls(true).tlsTrustCertsFilePath("/ca.cert.pem") .proxyServiceUrl(proxyUrl, ProxyProtocol.SNI) .authentication(AUTH, "tlsCertFile:/cert.pem,"+"tlsKeyFile:/key.pem") .build();
  24. 24. 24 Cross Organization geo-replication pulsar-admin clusters create orgB-cluster --broker-url-secure pulsar+ssl:// orgB-broker-vip:6651 --proxy-protocol SNI --proxy-url pulsar+ssl:// orgA-proxy:443 pulsar-admin clusters create orgA-cluster --broker-url-secure pulsar+ssl:// orgA-broker-vip:6651 --proxy-protocol SNI --proxy-url pulsar+ssl:// orgB-proxy:443 For more info: PIP-60: https://github.com/apache/pulsar/wiki/PIP-60%3A-Support-Proxy-server-with-SNI-routing
  25. 25. Future Roadmap ● Tenant based broker virtualization ○ Container based brokers on BookKeeper service ● Hybrid cloud deployment with geo-replication 25
  26. 26. Questions? 26
  27. 27. Thank you Rajan Dhabalia rdhabalia@verizonmedia.com Ludwig Pummer ludwig@verizonmedia.com
  • jitabc

    Jul. 27, 2021

With the rise of the number of tenants and traffic in the cluster, we are always striving for a system that is both multi-tenant and secure enough to onboard applications having different use cases and those applications can access pulsar from different cloud providers or even from cross-organization for enterprise integration. Large organizations use TLS proxy servers which act as a gateway between a local network and a large-scale network, such as the internet. Aside from traffic forwarding, proxy servers provide security by hiding the actual IP address of a server. Organizational policies often require systems to stay behind enterprise proxy/gateway servers such as HAProxy, ATS, Nginx and follow standard security regulations to protect systems against known vulnerabilities. Apache Pulsar provides various solutions for TLS proxy and Pulsar is the only messaging system that supports SNI proxy to leverage various enterprise proxy solutions. In this talk, we will discuss security and proxy solutions for Apache Pulsar which enables users in multi-tenant environments to access Pulsar instances securely from the on-prem, public cloud, and cross-enterprise. We will also talk about different multi-tenancy dimensions of Apache Pulsar which we use in Verizon Media to serve different use cases and applications on a shared pulsar cluster.

Vues

Nombre de vues

124

Sur Slideshare

0

À partir des intégrations

0

Nombre d'intégrations

0

Actions

Téléchargements

7

Partages

0

Commentaires

0

Mentions J'aime

1

×