How to Lock Down Apache Kafka and Keep Your Streams Safe

How to Lock Down Apache Kafka
and Keep Your Streams Safe
Rajini Sivaram

About me
• Principal Software Engineer at Pivotal UK
• Apache Kafka Committer
• Project Lead: Reactor Kafka
– https://github.com/reactor/reactor-kafka
• Previously at IBM
– Message Hub developer: Kafka-as-a-Service on Bluemix

Outline
• Kafka Cluster Overview
• Securing Kafka Clusters
– Authentication
– Authorization
– Quotas
– Encryption
• Lock Down Kafka and ZooKeeper
• New security features

Kafka Cluster
Kafka BrokerKafka BrokerKafka Broker
Kafka Cluster
Kafka BrokerKafka BrokerZookeeper Server
Zookeeper Cluster
Kafka Clients
Kafka Producer Kafka Consumer Kafka Connect Kafka Streams Kafka Admin
Admin/ConfigTools

External client
Internal client
Security Protocol
security.protocol=SASL_SSL
bootstrap.servers=kafka01.a.com:9094
listeners=PLAINTEXT://10.0.0.1:9092,
SSL://192.168.1.1:9093,
SASL_SSL://192.168.1.1:9094
advertised.listeners=PLAINTEXT://10.0.0.1:9092,
SSL://kafka01.a.com:9093,
SASL_SSL://kafka01.a.com:9094
security.inter.broker.protocol=PLAINTEXT
External client
Internal client
Kafka Broker
Kafka Broker
security.protocol=SSL
bootstrap.servers=kafka01.a.com:9093
 PLAINTEXT
 SSL
 SASL_SSL
 SASL_PLAINTEXT

Authentication
• Client authentication
– Server verifies the identity (user principal) of the client
• Server authentication
– Client verifies that connection is to a genuine server
• Authentication mechanisms in Kafka
– TLS
– SASL

Authentication using TLS or SASL
Kafka Cluster
Zookeeper Cluster
Kafka Clients
TLS/SASL TLS/SASL TLS/SASL TLS/SASL
SASL
TLS/SASL
SASL
Admin/ConfigTools
SASL
TLS/SASL

TLS Handshake
Client
ClientHello
Server
ServerHello
Certificate
[ServerKeyExchange]
[CertificateRequest]
ServerHelloDone
[Certificate]
ClientKeyExchange
[CertificateVerify]
ChangeCipherSpec
Client Finished
ChangeCipherSpec
Server Finished
Server
cert
Client
cert

Client trust store
Server key store
Issuer’s certificate
TLS authentication
ssl.keystore.location=/path/ks.jks
ssl.keystore.password=ks-secret
ssl.key.password=key-secret
ssl.truststore.location=/path/trust.jks
ssl.truststore.password=ts-secret
ssl.endpoint.identification.algorithm=https
Server’s certificate
Distinguished Name(DN)
Server hostname (SAN)
Valid from: to:
Issuer DN
Issuer Digital Signature
Server Public Key
Issuer’s certificate
Issuer Public Key
Issuer Digital Signature
Issuer DN
Server
Private Key
✔
✔
✔

TLS Security Considerations
Threat Mitigation
Security vulnerability in older
protocols
• Use latest TLS version: TLSv1.2
Cryptographic attacks • Only strong cipher suites (e.g. 256-bit encryption key size)
• Minimum 2048-bit RSA key size
Man-in-the-middle attack • Disable anonymous key exchange using Diffie-Hellman
ciphers
• Enable hostname verification
Private key compromised • Certificate revocation using CRL
• Use short-lived keys to reduce exposure
Man-in-the-middle attack during
renegotiation
• Disable insecure renegotiation
• Note: TLS renegotiation is disabled in Kafka
Tampering with data during transit • Use ciphers with secure message digest to guarantee
integrity
DDoS attack • Enable quotas and connection rate throttling

Why TLS?
• Authentication
– Server
– Client
• Confidentiality
– Guarantees privacy of data in motion
• Integrity
– Message digest included with many ciphers
• Horizontally scalable

TLS drawbacks
• Performance impact
– latency and throughput
• 20-30% degradation
• High CPU cost of encryption
– Lose zero-copy transfer
• TLS-renegotiation is disabled
– Authenticate only once
• Vulnerable to DDoS attacks
• PKI infrastructure required
Throughput
Message Size
CA
VA
RA
CRL
RA
VA

SASL
• Simple Authentication and Security Layer
– Extensible authentication framework for
connection-oriented protocols
• Standard protocol for different mechanisms
– GSSAPI (since 0.9.0)
– PLAIN (since 0.10.0)
– SCRAM (since 0.10.2)
• Can negotiate security layer, but this feature
is not used in Kafka
– SASL_SSL/SASL_PLAINTEXT

SASL Handshake
Client
Kafka SaslHandshake request
(mechanism=GSSAPI)
Server
Establish connection
Kafka SaslHandshake response
Enabled mechanisms=GSSAPI,PLAIN
SASL handshake for selected mechanism
Challenge
Transport Layer
(eg. TLS handshake)
Kafka SASL
Handshake request
SASL authentication
using selected
mechanism
Kafka requests and
responses
Response
Authenticated

Kafka SASL configuration
JAAS configuration
listeners=SASL_SSL://host:port1,
SASL_PLAINTEXT://host:port2
security.inter.broker.protocol=SASL_PLAINTEXT
sasl.mechanism.inter.broker.protocol=GSSAPI
sasl.enabled.mechanisms=GSSAPI,SCRAM-SHA-256
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true storeKey=true
keyTab="/etc/security/keytabs/kafka_server.keytab“
principal="kafka/kafka1.host.com@EXAMPLE.COM";
o.a.k.c.s.s.ScramLoginModule required;
};
KafkaClient {
o.a.k.c.s.s.ScramLoginModule required
username="alice”
password="alice-secret";
};
http://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/LoginConfigFile.html
Broker config: server.properties
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=o.a.k.c.s.s.ScramLoginModule required
username="alice"
password="alice-secret”;
producer/consumer.properties
sasl.jaas.config (since 0.10.2)
JAAS configuration

KDC
SASL/GSSAPI
Key Distribution Centre
Kafka BrokerKafka
Client
Authentication
Service
Ticket
Granting
Service
KafkaServer {
useKeyTab=true storeKey=true keyTab=“/server.keytab"
principal="kafka/kafka1.a.com@EXAMPLE.COM";};
sasl.jaas.config=
useKeyTab=true storeKey=true keyTab=/client.keytab”
principal=“kafka-client-1@EXAMPLE.COM”;
• Kerberos V5 (RFC 475https://tools.ietf.org/html/rfc4752)
• Principal: <primary>[/<instance>]@<REALM>
TGT
TGT
ticket
ticket

SASL/GSSAPI Security Considerations
Threat Mitigation
Dictionary attack • Enforce strong password policies
Keytab file compromised • Restrict access to keytab files and directory
• If user compromised, revoke access using ACLs. Restart
processes to force reconnections if required.
Eavesdropping, tampering with
data (after authentication
completes)
• Kafka does not use Kerberos encryption
• SASL_SSL should be used to guarantee confidentiality and
integrity if the traffic is not on a secure network
Hostname resolution issues • Secure correctly configured DNS
KDC failure • Set up multiple slave KDCs alongside a master KDC to
avoid single-point-of-failure

SASL/PLAIN
sasl.jaas.config=
org.apache.kafka.common.security.plain.PlainLoginModule required
username="alice” password="alice-secret";
Kafka Broker
Kafka
Client
alice
alice-secret
• Simple username/password authentication
RFC 4616: https://tools.ietf.org/html/rfc4616
• Basic support in Kafka brokers, replace for production use
KafkaServer {
o.a.k.c.security.plain.PlainLoginModule required
user_alice=“alice-secret”; };

SASL/PLAIN customization
• Integrate with external authentication server
• SASL/PLAIN security provider
Kafka Broker
MyPlainProviderMyPlainLoginModule
KafkaServer {
com.pivotal.MyPlainLoginModule required
authentication.server=“https://my.server";
};
Authentication
Server

SASL/PLAIN Security Considerations
Threat Mitigation
Eavesdropping and replay attack • PLAIN must only be used with TLS
• Connection between Kafka and authentication
server/database must also be secure
User compromised • Revoke all access using ACLs
• Restart brokers if required to break connections
Password database compromised • Update authentication server
• Re-authentication of existing connections is not
supported, restart brokers.

SASL/SCRAM
• Salted Challenge Response Authentication Mechanism
– RFC 5802: https://tools.ietf.org/html/rfc5802
– Secure username/password authentication
• SCRAM-SHA-256 and SCRAM-SHA-512
• Default implementation in Kafka stores salted keys in Zookeeper
bin/kafka-configs.sh --zookeeper localhost:2181 –alter
--add-config 'SCRAM-SHA-256=[iterations=8192,password=alice-secret]
--entity-type users --entity-name alice
Create user:

SASL/SCRAM protocol
sasl.jaas.config=
org.apache.kafka.common.security.scram.ScramLoginModule required
username="alice” password="alice-secret”;
Kafka Broker
Kafka
Client
Zookeeper
• Client proves to the broker that client possesses the password for user
• Broker proves to the client that broker once possessed the password for user
alice, c-nonce /config/users/alice
salt,iterations,
salted keys
c-s-nonce, salt,
iterations
c-s-nonce,
client-proof
c-s-nonce,
server-proof
✔
✔
KafkaServer {
o.a.k.c.s.scram.ScramLoginModule required;
};
Cache

SASL/SCRAM Security Considerations
Threat Mitigation
Offline brute force attack • Use high iteration count, strong hash function
User compromised • Revoke all access for user
• Restart broker to disconnect if required
Zookeeper compromised • SCRAM is safe against replay attack
• Use with TLS to avoid interception of messages for use in
dictionary/brute force attacks
• Use strong hash function like SHA-256 or SHA-512
• Use high iteration count
Insecure Zookeeper installation • Use alternative secure password store for SCRAM

Custom SASL mechanisms
• Integrate with existing authentication servers
– e.g sasl.mechanism=EXTERNAL
Kafka Broker
MyServerProvider
MyServerLoginModule
KafkaServer {
MyServerLoginModule required
authentication.server=“https://my.server";
};
Authentication
Server
KafkaClient {
MyClientLoginModule required
identity=“alice“;
};
Kafka Client
MyClientProvider
MyClientLoginModule

Choosing an authentication protocol
Authentication protocol Use if:
TLS • On insecure network and require encryption
• Server authentication and hostname verification required
• Already have PKI infrastructure for client auth
SASL/GSSAPI • Already have Kerberos infrastructure
• Insecure ZooKeeper installation, don’t want to integrate
with custom password database for SCRAM
SASL/PLAIN • Integrating with existing password server/database
SASL/SCRAM • Require username/password authentication without
external server
• Secure ZooKeeper installation
Custom SASL mechanism • Integrating with existing authentication server

Authorization
• User Principal
– ANONYMOUS for unauthenticated clients
– Configurable PrincipalBuilder for TLS
– Mechanism-specific user name for SASL
• Access Control Lists (ACL)
• Pluggable Authorizer
– Default out-of-the-box authorizer: SimpleAclAuthorizer
bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal
User:alice --allow-host 198.51.100.0 --operation Read --operation Write --topic test-topic
✗

Access Control
alice Allow Read Topic Host
Deny Cluster
Operation Resource From hostPermissionUser Principal
Consumer
Group
Create
Delete
Alter
Describe
Write
ClusterAction
bob
✔
✗
Super user

Kafka authorization sequence
Client
Request
Broker Authorizer ZooKeeper
Initialize
Load all ACLs
Authorize Check ACL
cache
ACL CLI
Update ACL
Alter ACL
Update ACL
cache
Response
Process Request
Cache

Quotas
• Quota types
– Replication quota
– Bandwidth quota (Produce/Fetch)
– Request quotas (from 0.11.0)
• Per-broker quotas
– If usage exceeds quota, response is delayed
– Throttle time returned to clients, exposed as metrics
• Quota configuration in ZooKeeper
– Can be dynamically updated
bin/kafka-configs.sh --zookeeper localhost:2181 --alter --add-config
'producer_byte_rate=1024,consumer_byte_rate=2048' --entity-name alice --entity-type users
Kafka
Broker
Client

Quota Configuration
• Multi-level quotas: <client-id>, <user> or <user, client-id> levels
• The most specific quota configuration is applied to any connection
<user>
<client-id>
users
clients
<default>
<default>
<client-id>
<client-id>
clients
<default>clients
<default>
config

Encryption
• TLS
– Encrypt data during transit to prevent
eavesdropping
• Disk encryption
– Encrypt data at rest to protect sensitive data
• End-to-end encryption
– Clients send encrypted data (eg.
serialize/deserialize)
– Different keys to encrypt data to different topics
– Combine with TLS/SASL for authentication, TLS
to avoid man-in-the-middle

Rolling upgrade to enable security
Kafka Client Kafka Broker
Kafka Broker
listeners=PLAINTEXT://host:9092
listeners=PLAINTEXT://host:9092,SSL://host:9093
listeners=PLAINTEXT://host:9092,SSL://host:9093
security.inter.broker.protocol=SSL
listeners=SSL://host:9093
security.inter.broker.protocol=SSL
Dynamic configs
• ACL
• Quotas

Zookeeper Server
Securing ZooKeeper
• ZooKeeper stores critical metadata for Kafka
• Lock down updates to Zookeeper
– SASL
• GSSAPI (Kerberos)
• Digest-MD5
– Set zookeeper.set.acl=true on Kafka brokers
• TLS is currently not supported for ZooKeeper
– Use network segmentation to limit access
SASL

Secure Kafka Cluster
Kafka Cluster
Zookeeper Cluster
Kafka Clients
Admin/ConfigTools

Secure Kafka on the Cloud
Private Network
Public Network
TLS ProxyTLS ProxyTLS Proxy
Kafka Clients
Admin/ConfigTools

New features in 0.10.2
• Broker
– Multiple endpoints with the same security protocol
• Client
– Dynamic JAAS configuration without a file
– Multiple credentials within a JVM
• SASL mechanisms
– SCRAM-SHA-256, SCRAM-SHA-512
Kafka
Broker
Kafka
Broker

Future work
• KIP-48: Delegation tokens
• KIP-124: CPU utilization quota for requests
• KIP-117: Add a public AdminClient API for Kafka
• KIP-86: Configurable SASL callbacks
• KIP-111: Improve custom
PrincipalBuilder/Authorizer integration

Summary
• Authentication
– TLS
– SASL: GSSAPI, PLAIN, SCRAM
• Authorization
– User principal
– IP address
• Quotas
– <client-id>, <user>, <user, client-id>
• Encryption
– TLS
– End-to-end encryption

Want to find out more?
• References
– https://kafka.apache.org/documentation/
– https://kafka.apache.org/documentation/#security
– https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-
encryption/
– http://zookeeper.apache.org/doc/r3.4.9/zookeeperProgrammers.html#sc_ZooKeeperA
ccessControl
– https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
• Mailing lists
– users@kafka.apache.org, dev@kafka.apache.org
• Report security issues
– security@kafka.apache.org

Thank you for listening.
Questions?
Stay connected.
rsivaram@pivotal.io

How to Lock Down Apache Kafka and Keep Your Streams Safe

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à How to Lock Down Apache Kafka and Keep Your Streams Safe

Similaire à How to Lock Down Apache Kafka and Keep Your Streams Safe (20)

Plus de confluent

Plus de confluent (20)

Dernier

Dernier (20)

How to Lock Down Apache Kafka and Keep Your Streams Safe