Securing kafka with 500 billion messages a day

Securing Kafka at PayPal with
500 Billion Messages a Day

Agenda
© 2019 PayPal Inc. Confidential and proprietary.
Kafka @ PayPal Today
TLS on Kafka Overview
Enabling mTLS
Authentication
Future Work
Conclusion

Overview
500+
Billion messages
per day
50+
Clusters
~7PB
Disk
5000+
Topics
0.9 0.10 1.10.8Kafka Journey 2.2

Tech stack
Language
s
Gimel
Application Frameworks Multi-Tenant
Multiple Security &
Availability Zones

Data Pipelines
Use Cases
User behavioral tracking
Experimental
Merchant monitoring
Risk & compliance
Business Events
Application logs
Application metrics
Gimel
Frameworks
& Platforms
Real-Time Streaming
Batch Processing
Kafka

SSL & TLS
Terminology
• SSL Key
• SSL Certificate
• CA
• Trusted CAs

TLS Introduction
Encryption and Authenticating using SSL in Open Source Kafka
• Generate SSL Key and Certs
• Create CA
• Sign the certificates
• Configure Kafka Properties

keytool -keystore server.keystore.jks -alias localhost -validity {validity} -genkey -keyalg RSA
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days {validity} -CAcreateserial -passin pass:{ca-
password}
keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed
CLI Commands

KeyStore & TrustStore
KeyStore TrustStore

Configuration Sample
Broker Configurations
listeners=PLAINTEXT://host.name:port,SSL://host.name:port
ssl.keystore.location=/var/private/ssl/server.keystore.jks
ssl.keystore.password=test1234 [Optional]
ssl.key.password=test1234
ssl.truststore.location=/var/private/ssl/server.truststore.jks
ssl.truststore.password=test1234 [Optional]

Configuration Sample
Client Configuration
security.protocol=SSL
ssl.truststore.location=/var/private/ssl/client.truststore.jks
ssl.truststore.password=test1234 [Optional]
[Optional with mTLS]
ssl.keystore.location=/var/private/ssl/client.keystore.jks
ssl.keystore.password=test1234 [Optional]
ssl.key.password=test1234

Kafka Broker
File-Based
KeyStore
Client Application
Server Authentication
File-Based
TrustStore
TLS with One-way Auth

Kafka Broker
File-Based
KeyStore
Client Application
Server Authentication
File-Based
TrustStore
File-Based
TrustStore
File-Based
KeyStore
Client Authentication
TLS with Mutual Auth

Challenges to enable Kafka TLS @ PayPal
InfoSec/AppSec Restrictions
• File-based security material/credentials are not allowed
• Regular key rotations
Scalability
• Deploying the key/trust store files to the thousands of brokers
• Deploying the key/trust store files to 10x/100x Kafka Client boxes
• Keep all those key/trust store files and their passwords secure
• Lack of unified way to distribute KeyStore and TrustStore for different languages

PayPal’s Key Management Service
In-house Key Management Service [Like Vault / AWS KMS]
• A company internal CA authority issuing the certificates
• Rotate the Keys

Work Flow without SSL in PayPal
Kafka Client
Kafka Client
Config Service
Config Service
Send Topic Config Request
Return Topic Config:
{Topic1: Bootstrap Servers,
client configs
Topic 2: Bootstrap Servers,
client configs }
Connect to Kafka Broker

Solutions: Kafka Source Code Change – Client & Broker
Change Kafka Source Code to allow KeyStore/TrustStore plug-in
• Provide KeyStoreLoader & TrustStoreLoader Interface for customized KeyStore and TrustStore
Loading

Work Flow with SSL in PayPal
Kafka Client
Kafka Client
Config Service
Config Service
Send Topic Config Request
Return Topic Config
Connect to Kafka Broker
KMS
KMS
Send Key/Trust Store Request
Get KeyStore and TrustStore

Configuration Sample with Custom Key/Trust Store
Broker Configurations
listeners=SSL://host.name:port
ssl.keystore.loader=com.paypal.kafka.broker.KeyStoreLoaderImpl
ssl.keystore.location=
ssl.keystore.password=
ssl.key.password=
ssl.truststore.loader=com.paypal.kafka.broker.TrustStoreLoaderImpl
ssl.truststore.location=
ssl.truststore.password=

Client Configurations
security.protocol=SSL
ssl.truststore.loader= com.paypal.kafka.client.TrustStoreLoaderImpl
ssl.keystore.location=
ssl.keystore.password=
ssl.key.password=
[Optional with mTLS]
ssl.keystore.loader=com.paypal.kafka.client.KeyStoreLoaderImpl
ssl.truststore.location=
ssl.truststore.password=
Configuration Sample with Custom Key/Trust Store

Kafka Improvement Proposal
KIP-486 – Support custom way to load key and trust stores (Work-In-Progress)
KIP-519 – Make SSL context/engine configuration extensible (Work-In-Progress)

Authentication
MTLS
SASL
• GSSAPI (Kerberos)
• SCRAM
• Delegation Tokens
• PLAIN
• OAUTHBEARER
Authentication Overview

Authentication
KIP-12 - Kafka Sasl/Kerberos and SSL implementation
• 0.9.0.0
KIP-85: Dynamic JAAS configuration for Kafka clients
• 0.10.2.0
KIP-86: Configurable SASL callback handlers
• 2.0.0
KIP-255: OAuth Authentication via SASL/OAUTHBEARER
• 2.0.0
KIP 368: Allow SASL Connections to Periodically Re-Authenticate
• 2.2.0
Authentication Overview – evolution

Authentication
KafkaClient (Procuder/Consumer)
KafkaChannel
Authenticator
Authentication Overview – Auth Components
Kafka (Server)
KafkaServer
Processor
Authenticator
KafkaRequestHandler
KafkaApis
Authorizer

Authentication
sasl.login.class
LoginModule
sasl.login.callback.handler.class
sasl.client.callback.handler.class
sasl.server.callback.handler.class
principal.builder.class
Authentication Overview - Pluggable Components

Authentication
KafkaClient
LoginManager.acquireLoginManager
DefaultLogin.login()
LoginContext.login()
LoginModule
.configure()
.login()
AuthenticateCallbackHandler.handle(Callback)
.commit()
Set Credentials to Subject
KafkaChannel.prepare()
Authenticator.authenticate
sendSaslClientToken
SaslClient.evaluateChallenge
retrieve Credentials from Subject
Authentication Overview – Client Auth Pluggable Components
<- sasl.login.class
<- KafkaClient Jaas config
<- sasl.login.callback.handler.class
<- sasl.client.callback.handler.class

Authentication
Kafka Server
LoginModule
.configure()
.login()
.commit()
Processor(1).run()
poll()
SaslServerAuthenticator.authenticate
…
validate()
Authentication Overview - Server Auth Pluggable Components
<- sasl.login.class
<- KafkaServer jaas config
<- sasl.server.callback.handler.class

Authentication
…
LoginModule
.configure()
.login()
.commit()
…
Authentication Overview – Pluggable Components
<- sasl.login.class

Authentication
sasl.login.class
• An implementation of org.apache.kafka.common.security.auth.Login that performs login for each
LoginModule specified through login context.
• LoginManager initializes Login class by passing appropriate login handler configured through
• In most cases out of box Login Implementations are sufficient.
DefaultLogin, KerberosLogin, OAuthBearerRefreshingLogin
• Applicable to client and server

Authentication
…
LoginModule
.configure()
.login()
.commit()
…
<- sasl.login.class
<- KafkaClient/KafkaServer Jaas config

Authentication
LoginModule
• Interface to implement SASL specific Authentication
• Java security LoginContext invokes LoginModule to perform implementation specific Authentication and
initializes public and private credentials for a Subject
• LoginModule should be provided in jaas config format
• KIP-86 enables clients to load jaas config dynamically,
As a jvm parameter export KAFKA_OPTS="-Djava.security.auth.login.config=$KAFKA_HOME/config/kafka_server_jaas.conf”
Through Configuration property sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required
• In most cases out of box LoginModule implementations are sufficient for each type of authentication.
OAuthBearerLoginModule, PlainLoginModule, ScramLoginModule, Krb5LoginModule

Authentication
• An implementation of AuthenticateCallbackHandler that should retrieve credentials from external
services and set it to Callback, these credentials are used by LoginModule to set private and public
credentials on the Subject.
• For Production use cases DON’T use default handlers, it is recommended to write custom implementation
of AuthenticateCallbackHandler compatible to respective SASL mechanisms LoginModule.
• Out of box Handlers are
• OAuthBearerUnsecuredLoginCallbackHandler, DefaultLoginCallbackHandler

Authentication
KafkaClient
LoginModule
.configure()
.login()
.commit()
Authenticator.authenticate
sendSaslClientToken
SaslClient.evaluateChallenge
<- sasl.login.class
<- sasl.client.callback.handler.class

Authentication
sasl.client.callback.handler.class
• An implementation of AuthenticateCallbackHandler that should retrieve credentials from Subject that were
created during LoginModule.login() and return it to Authenticator.
• In most cases out of box Handlers are sufficient
• SaslClientCallbackHandler, OAuthBearerSaslClientCallbackHandler, KerberosClientCallbackHandler
• Applicable to client

Authentication
Kafka Server
LoginModule
.configure()
.login()
.commit()
Processor(1).run()
poll()
SaslServerAuthenticator.authenticate
…
Authentication Overview - Server Auth Pluggable Components
<- sasl.login.class
<- KafkaServer jaas config
<- sasl.server.callback.handler.class

Authentication
sasl.server.callback.handler.class
• An implementation of AuthenticateCallbackHandler that should validate credentials passed by client
through external service or by some means.
• If you are using Plan or Oauth SASL mechanisum it is recommended to write custom implementation
of AuthenticateCallbackHandler to validate credentials.
• Out of box Handlers are
• SaslServerCallbackHandler, OAuthBearerUnsecuredValidatorCallbackHandler, PlainServerCallbackHandler,
ScramServerCallbackHandler

Kafka Security at PayPal - OAUTHBEARER

OAuth Authentication at PayPal
Kafka Client
Request Access to
Kafka
KMS Kafka
KafkaKMSKafka Client
Creates a JWS token with
Kafka Public Key
Get Access Token
Validate User
Encrypted JWS
token
Authenticate(JWS token)
Decrypt JWS token
and extract Principal
and Scopes
OfflineOnline

Authentication
1. Broker Loading Keymaterial
(PayPalServerLoginCallbackHandler)
1. On startup it authenticates with KMS
2. Loads required key material from KMS that is required to decrypt jws
token provided by client.
2. Get Client Access Token (PayPalClientLoginCallbackHandler)
1. On startup it authenticates with KMS
2. Loads jws token
3. Set the token to Channel Subject Public and Private Credentials
3. Authenticate Client
(PayPalServerTokenValidatorCallbackHandler)
1. SASL Authenticator uses Client Callback handler to retrieve credentials
from Channel Subject
2. Pass it to Server for Authentication,
3. Server Callback handler receives the request for validation
4. It uses the private key to decrypt the token and extract the principal &
scopes
5. Update this information to Channel Subject public and private
credentials for later use by Principal extractor to Authorizing the
resource.

Authentication
sasl.login.class: OAuthBearerRefreshingLogin
LoginModule: OAuthBearerLoginModule
sasl.login.callback.handler.class: PayPalClientLoginCallbackHandler, PayPalServerLoginCallbackHandler
sasl.client.callback.handler.class: OAuthBearerSaslClientCallbackHandler
sasl.server.callback.handler.class: PayPalServerTokenValidatorCallbackHandler
principal.builder.class: DefaultKafkaPrincipalBuilder

Authentication
Kafka Security at PayPal
Connectio
n
Service Definition
1,4 Kafka
SSL + SASL
OAuth
2 Kafka SSL
3 ZK
SASL:DIGEST
-MD5
6 ZK
Admin ports
for follower if
this instance is
Leader
7 ZK
Admin ports
for Leader
election

Future Work
• Automating ACL management

Conclusion

Securing kafka with 500 billion messages a day

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Securing kafka with 500 billion messages a day

Similaire à Securing kafka with 500 billion messages a day (20)

Dernier

Dernier (20)

Securing kafka with 500 billion messages a day

Notes de l'éditeur