The document discusses securing connections between KSQL and Kafka. It covers enabling encryption using TLS for the KSQL-Kafka connection. It also covers enabling authentication using SASL and authorization using Kafka ACLs. It provides configuration examples for securing each part of the connection and recommends configuring the KSQL output topic name prefix to more easily manage ACLs for output topics.

1. 1
KSQL and Security:
The current state of affairs,
and where it’s headed
Victoria Xia

2. 2
A Little about… You?

3. 3
?
A Little about… You?

4. 4
Outline
● Background
● Securing KSQL’s connections
○ Encryption
○ Authentication
○ Authorization
○ Quotas
● KSQL-specific considerations
● Limitations and Futures

5. 5
KSQL 101

6. 6
KSQL 101
KSQL
Server
KSQL
Server

7. 7
KSQL 101
KSQL
Server
KSQL
Server
KSQL
Server
KSQL
Server

8. 8
KSQL 101
KSQL
Server
KSQL
Server

9. 9
KSQL 101
KSQL
Server
KSQL
Server
CREATE STREAM purchases (
productID BIGINT,
quantity INT,
storeLocation VARCHAR)
WITH (
KAFKA_TOPIC=’purchases’,
VALUE_FORMAT=’JSON’);

10. 10
KSQL 101
KSQL
Server
KSQL
Server
CREATE STREAM purchases (
productID BIGINT,
quantity INT,
storeLocation VARCHAR)
WITH (...);
SELECT
productID, quantity * 10
FROM purchases;

11. 11
KSQL 101
KSQL
Server
KSQL
Server
CREATE STREAM purchases (
productID BIGINT,
quantity INT,
storeLocation VARCHAR)
WITH (...);
SELECT
productID, SUM(quantity)
FROM purchases
GROUP BY productID;

12. 12
KSQL 101
KSQL
Server
KSQL
Server
CREATE STREAM purchases (
productID BIGINT,
quantity INT,
storeLocation VARCHAR)
WITH (...);
SELECT
productID, SUM(quantity)
FROM purchases
WHERE storeLocation=’NYC’
GROUP BY productID;

13. 13
KSQL 101
KSQL
Server
KSQL
Server
CREATE STREAM purchases (
productID BIGINT,
quantity INT,
storeLocation VARCHAR)
WITH (...);
CREATE TABLE NYC_totals
AS SELECT
productID, SUM(quantity)
FROM purchases
WHERE storeLocation=’NYC’
GROUP BY productID;

14. 14
KSQL 101
CREATE TABLE NYC_totals
AS SELECT
productID, SUM(quantity)
FROM purchases
WHERE storeLocation=’NYC’
GROUP BY productID;
kafka
Streams
purchases NYC_totalsintermediary
topic
intermediary
topic

15. 15
CREATE STREAM purchases (
productID BIGINT,
quantity INT,
storeLocation VARCHAR)
WITH (
KAFKA_TOPIC=’purchases’,
VALUE_FORMAT=’Avro’);
KSQL 101
Schema
Registry
KSQL
Server
KSQL
Server

16. 16
KSQL 101
Schema
Registry
KSQL
Server
KSQL
Server
CREATE STREAM purchases (
productID BIGINT,
quantity INT,
storeLocation VARCHAR)
WITH (
KAFKA_TOPIC=’purchases’,
VALUE_FORMAT=’Avro’);

17. 17
Interactive Use
Schema
Registry
KSQL
Server
KSQL
Server

18. 18
Interactive Use
Schema
Registry
KSQL
Server
KSQL
Server
REST
REST

19. 19
Interactive Use
Schema
Registry
REST
KSQL
Server
KSQL
Server
REST
REST

20. 20
Interactive Use
Schema
Registry
CLI
REST
KSQL
Server
KSQL
Server
REST
REST

21. 21
Interactive Use
Schema
Registry
CLI
REST
UI
KSQL
Server
KSQL
Server
REST
REST

22. 22
Non-interactive (Headless) Use
Schema
Registry
KSQL
Server
KSQL
Server

23. 23
KSQL’s Connections
Schema
Registry
CLI
REST
UI
KSQL
Server
KSQL
Server
REST
REST

24. 24
Motivation: Encryption

25. 25
Motivation: Authentication

26. 26
Motivation: Authentication

27. 27
Solution: TLS

28. 28
Solution: TLS

29. 29
KSQL <-> Kafka: TLS
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-kafka-encrypted-communication
https://kafka.apache.org/documentation/#security_ssl
listeners=
PLAINTEXT://host.name:port
bootstrap.servers=
http://host.name:port

30. 30
bootstrap.servers=
https://host.name:port
security.protocol=SSL
ssl.truststore.location=
/path/to/truststore.jks
ssl.truststore.password=zzz
listeners=
SSL://host.name:port
ssl.keystore.location=
/path/to/keystore.jks
ssl.keystore.password=xxxx
ssl.key.password=yyyy
KSQL <-> Kafka: TLS
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-kafka-encrypted-communication
https://kafka.apache.org/documentation/#security_ssl

31. 31
listeners=
SSL://host.name:port
ssl.keystore.location=
/path/to/keystore.jks
ssl.keystore.password=xxxx
ssl.key.password=yyyy
bootstrap.servers=
https://host.name:port
security.protocol=SSL
ssl.truststore.location=
/path/to/truststore.jks
ssl.truststore.password=zzz
KSQL <-> Kafka: TLS
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-kafka-encrypted-communication
https://kafka.apache.org/documentation/#security_ssl

32. 32
listeners=
SSL://host.name:port
ssl.keystore.location=
/path/to/keystore.jks
ssl.keystore.password=xxxx
ssl.key.password=yyyy
ssl.client.auth=required
ssl.truststore.location=
/path/to/truststore.jks
ssl.truststore.password=zzzz
bootstrap.servers=
https://host.name:port
security.protocol=SSL
ssl.truststore.location=
/path/to/truststore.jks
ssl.truststore.password=zzz
ssl.keystore.location=
/path/to/keystore.jks
ssl.keystore.password=xxx
ssl.key.password=yyy
KSQL <-> Kafka: TLS
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-kafka-encrypted-communication
https://kafka.apache.org/documentation/#security_ssl

33. 33
KSQL <-> Kafka: SASL
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-kafka-authentication
https://kafka.apache.org/documentation/#security_sasl
● GSSAPI (Kerberos)
● OAUTHBEARER
● SCRAM
● PLAIN

34. 34
KSQL <-> Kafka: SASL
listeners=
SASL_SSL://host.name:port
security.protocol=SASL_SSL
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-kafka-authentication
https://kafka.apache.org/documentation/#security_sasl

35. 35
KSQL <-> Kafka: SASL
listeners=
SASL_SSL://host.name:port
sasl.enabled.mechanisms=PLAIN
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-kafka-authentication
https://kafka.apache.org/documentation/#security_sasl

36. 36
KSQL_OPTS=
-Djava.security.auth.login.config=
/path/to/jaas_config.file
KSQL <-> Kafka: SASL
listeners=
SASL_SSL://host.name:port
sasl.enabled.mechanisms=PLAIN
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=<jaas_contents>
KAFKA_OPTS=
-Djava.security.auth.login.config=
/path/to/jaas_config.file
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-kafka-authentication
https://kafka.apache.org/documentation/#security_sasl
OR

37. 37
Motivation: Authorization

38. 38
Motivation: Authorization

39. 39
Motivation: Authorization
Read Write Delete
alices_topic ? ? ?
bobs_topic ? ? ?
secrets_topic ? ? ?

40. 40
Motivation: Authorization
Read Write Delete
alices_topic ✔ ✔ ✔
bobs_topic ✔
secrets_topic

41. 41
OperationPrincipal
KSQL <-> Kafka: ACLs
Permission Type Pattern Name
Resource
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Host

42. 42
*
12.1.1.0ReadAllowUser:Alice
OperationPrincipal
KSQL <-> Kafka: ACLs
Permission Type Pattern Name
Resource
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Host
Topic Literal foo
WriteDenyUser:Bob Topic Prefixed prod-

43. 43
[ksql.host]?Allow[ksql-user]
OperationPrincipal
KSQL <-> Kafka: ACLs
Permission Type Pattern Name
Resource
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Host
? ? ?

44. 44
TypeOperation Pattern
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Resource Name
KSQL <-> Kafka: ACLs

45. 45
kafka-clusterLiteralClusterDescribeConfigs
TypeOperation Pattern
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Resource Name
KSQL <-> Kafka: ACLs

46. 46
kafka-clusterLiteralClusterDescribeConfigs
TypeOperation Pattern
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Resource Name
CREATE STREAM output_stream AS SELECT ... FROM input_stream;
KSQL <-> Kafka: ACLs

47. 47
kafka-clusterLiteralClusterDescribeConfigs
Read Topic Literal [ input topics ]
[ output topics ]LiteralTopicWrite
TypeOperation Pattern
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Resource Name
CREATE STREAM output_stream AS SELECT ... FROM input_stream;
KSQL <-> Kafka: ACLs

48. 48
kafka-clusterLiteralClusterDescribeConfigs
Read Topic Literal [ input topics ]
[ output topics ]LiteralTopicWrite
Create Topic Literal [ output topics (that don’t exist) ]
TypeOperation Pattern
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Resource Name
CREATE STREAM output_stream AS SELECT ... FROM input_stream;
KSQL <-> Kafka: ACLs

49. 49
CREATE STREAM output_stream AS SELECT ... FROM input_stream;
kafka-clusterLiteralClusterDescribeConfigs
Read Topic Literal [ input topics ]
[ output topics ]LiteralTopicWrite
Create Topic Literal [ output topics (that don’t exist) ]
_confluent-ksql-<ksql.service.id>PrefixedGroupAll
TypeOperation Pattern
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Resource Name
KSQL <-> Kafka: ACLs

50. 50
kafka-clusterLiteralClusterDescribeConfigs
Read Topic Literal [ input topics ]
[ output topics ]LiteralTopicWrite
Create Topic Literal [ output topics (that don’t exist) ]
_confluent-ksql-<ksql.service.id>PrefixedGroupAll
All Topic Prefixed _confluent-ksql-<ksql.service.id>
TypeOperation Pattern
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Resource Name
KSQL <-> Kafka: ACLs

51. 51
kafka-clusterLiteralClusterDescribeConfigs
Read Topic Literal [ input topics ]
[ output topics ]LiteralTopicWrite
Create Topic Literal [ output topics (that don’t exist) ]
_confluent-ksql-<ksql.service.id>PrefixedGroupAll
All Topic Prefixed _confluent-ksql-<ksql.service.id>
<ksql.logging.processing.topic.name>LiteralTopicAll
TypeOperation Pattern
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Resource Name
KSQL <-> Kafka: ACLs

52. 52
[ input topics ]LiteralTopicRead
kafka-clusterLiteralClusterDescribeConfigs
[ output topics ]LiteralTopicWrite
Create Topic Literal [ output topics (that don’t exist) ]
_confluent-ksql-<ksql.service.id>PrefixedGroupAll
All Topic Prefixed _confluent-ksql-<ksql.service.id>
<ksql.logging.processing.topic.name>LiteralTopicAll
TypeOperation Pattern
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Resource Name
KSQL <-> Kafka: ACLs

53. 53
[ output topics (that don’t exist) ]
[ output topics ]Literal
LiteralTopic
Topic
Create
Write
kafka-clusterLiteralClusterDescribeConfigs
Read Topic Literal [ input topics ]
_confluent-ksql-<ksql.service.id>PrefixedGroupAll
All Topic Prefixed _confluent-ksql-<ksql.service.id>
<ksql.logging.processing.topic.name>LiteralTopicAll
TypeOperation Pattern
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Resource Name
KSQL <-> Kafka: ACLs

54. 54
Configure ksql.output.topic.name.prefix
KSQL <-> Kafka: ACLs
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz

55. 55
CREATE TABLE results
AS SELECT …
FROM events;
Configure ksql.output.topic.name.prefix
KSQL <-> Kafka: ACLs
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Output topic:
<ksql.output.topic.name.prefix>RESULTS

56. 56
[ output topics (that don’t exist) ]
[ output topics ]
Literal
Literal
kafka-clusterLiteralClusterDescribeConfigs
Read Topic Literal [ input topics ]
TopicWrite
Create Topic
_confluent-ksql-<ksql.service.id>PrefixedGroupAll
All Topic Prefixed _confluent-ksql-<ksql.service.id>
<ksql.logging.processing.topic.name>LiteralTopicAll
TypeOperation Pattern
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Resource Name
KSQL <-> Kafka: ACLs

57. 57
Prefixed
Prefixed
<ksql.output.topic.name.prefix>
<ksql.output.topic.name.prefix>
kafka-clusterLiteralClusterDescribeConfigs
Read Topic Literal [ input topics ]
TopicWrite
Create Topic
_confluent-ksql-<ksql.service.id>PrefixedGroupAll
All Topic Prefixed _confluent-ksql-<ksql.service.id>
<ksql.logging.processing.topic.name>LiteralTopicAll
TypeOperation Pattern
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Resource Name
KSQL <-> Kafka: ACLs

58. 58
CREATE TABLE results
AS SELECT …
FROM events;
Configure ksql.output.topic.name.prefix
KSQL <-> Kafka: ACLs
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
Output topic:
<ksql.output.topic.name.prefix>RESULTS

59. 59
CREATE TABLE results
AS SELECT …
FROM events;
Configure ksql.output.topic.name.prefix
KSQL <-> Kafka: ACLs
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-authorization-of-ksql-with-kafka-acls
https://kafka.apache.org/documentation/#security_authz
CREATE TABLE results
WITH (KAFKA_TOPIC=‘foo’)
AS SELECT …
FROM events;
Output topic:
<ksql.output.topic.name.prefix>RESULTS
Output topic:
foo

60. 60
Motivation: Quotas

61. 61
Motivation: Quotas

62. 62
Motivation: Quotas

63. 63
KSQL <-> Kafka: Quotas
● Network bandwidth quotas
Learn more:
https://kafka.apache.org/documentation/#design_quotas
https://kafka.apache.org/documentation/#quotas
https://docs.confluent.io/current/ksql/docs/capacity-planning.html#kafka
producer_byte_rate=1024
consumer_byte_rate=2048

64. 64
KSQL <-> Kafka: Quotas
● Network bandwidth quotas
● Request rate quotas
Learn more:
https://kafka.apache.org/documentation/#design_quotas
https://kafka.apache.org/documentation/#quotas
https://docs.confluent.io/current/ksql/docs/capacity-planning.html#kafka
producer_byte_rate=1024
consumer_byte_rate=2048
request_percentage=200

65. 65
KSQL <-> Kafka: Quotas
● Network bandwidth quotas
● Request rate quotas
● By user and/or client-id
Learn more:
https://kafka.apache.org/documentation/#design_quotas
https://kafka.apache.org/documentation/#quotas
https://docs.confluent.io/current/ksql/docs/capacity-planning.html#kafka
user=user1, client-id=clientA:
producer_byte_rate=1024
consumer_byte_rate=2048
request_percentage=200

66. 66
KSQL <-> Kafka: Quotas
● Network bandwidth quotas
● Request rate quotas
● By user and/or client-id
○ Configure via client.id in server properties
Learn more:
https://kafka.apache.org/documentation/#design_quotas
https://kafka.apache.org/documentation/#quotas
https://docs.confluent.io/current/ksql/docs/capacity-planning.html#kafka
user=user1, client-id=clientA:
producer_byte_rate=1024
consumer_byte_rate=2048
request_percentage=200

67. 67
KSQL’s Connections
Schema
Registry
CLI
REST
UI
KSQL
Server
KSQL
Server
REST
REST

68. 68
KSQL <-> Schema Registry: TLS
listeners=
http://host.name:port
ksql.schema.registry.url=
http://host.name:port
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-secured-sr-long
https://docs.confluent.io/current/schema-registry/docs/security.html#schema-registry-http-https

69. 69
ksql.schema.registry.url=
https://host.name:port
ksql.schema.registry.ssl.truststore
.location=/path/to/truststore
ksql.schema.registry.ssl.truststore
.password=xxx
listeners=
https://host.name:port
ssl.keystore.location=
/path/to/keystore
ssl.keystore.password=xxxx
ssl.key.password=yyyy
KSQL <-> Schema Registry: TLS
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-secured-sr-long
https://docs.confluent.io/current/schema-registry/docs/security.html#schema-registry-http-https

70. 70
ksql.schema.registry.url=
https://host.name:port
ksql.schema.registry.ssl.truststore
.location=/path/to/truststore
ksql.schema.registry.ssl.truststore
.password=xxx
listeners=
https://host.name:port
ssl.keystore.location=
/path/to/keystore
ssl.keystore.password=xxxx
ssl.key.password=yyyy
KSQL <-> Schema Registry: TLS
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-secured-sr-long
https://docs.confluent.io/current/schema-registry/docs/security.html#schema-registry-http-https

71. 71
ksql.schema.registry.url=
https://host.name:port
ksql.schema.registry.ssl.truststore
.location=/path/to/truststore
ksql.schema.registry.ssl.truststore
.password=xxx
ksql.schema.registry.ssl.keystore
.location=/path/to/keystore
ksql.schema.registry.ssl.keystore
.password=yyy
ksql.schema.registry.ssl.keypass
word=zzz
listeners=
https://host.name:port
ssl.keystore.location=
/path/to/keystore
ssl.keystore.password=xxxx
ssl.key.password=yyyy
ssl.client.auth=true
ssl.truststore.location=
/path/to/truststore
ssl.truststore.password=zzzz
KSQL <-> Schema Registry: TLS
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-secured-sr-long
https://docs.confluent.io/current/schema-registry/docs/security.html#schema-registry-http-https

72. 72
KSQL <-> Schema Registry: Basic HTTP Auth
authentication.method=BASIC
authentication.roles=user
authentication.realm=SR-Props
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-secured-sr-long
authentication.method=BASIC
authentication.roles=user
authentication.realm=
SchemaRegistry-Props

73. 73
KSQL <-> Schema Registry: Basic HTTP Auth
authentication.method=BASIC
authentication.roles=user
authentication.realm=SR-Props
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-secured-sr-long
authentication.method=BASIC
authentication.roles=user
authentication.realm=
SchemaRegistry-Props
SCHEMA_REGISTRY_OPTS=
-Djava.security.auth.login.config=
/path/to/jaas_config.file
SchemaRegistry-Props {
...
};

74. 74
authentication.method=BASIC
authentication.roles=user
authentication.realm=
SchemaRegistry-Props
SchemaRegistry-Props {
...
};
KSQL <-> Schema Registry: Basic HTTP Auth
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-secured-sr-long
SCHEMA_REGISTRY_OPTS=
-Djava.security.auth.login.config=
/path/to/jaas_config.file

75. 75
KSQL <-> Schema Registry: Basic HTTP Auth
authentication.method=BASIC
authentication.roles=user
authentication.realm=SR-Props
ksql.schema.registry.basic.auth
.credentials.source=USER_INFO
ksql.schema.registry.basic.auth
.user.info=ksqluser:password
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-secured-sr-long
authentication.method=BASIC
authentication.roles=user
authentication.realm=
SchemaRegistry-Props
SCHEMA_REGISTRY_OPTS=
-Djava.security.auth.login.config=
/path/to/jaas_config.file
SchemaRegistry-Props {
...
};

76. 76
Securing KSQL’s Connections
KSQL <-> Kafka KSQL <->
Schema Registry
Encryption TLS TLS
Authentication TLS
SASL
TLS
Basic HTTP Auth
Authorization ACLs
Quotas Network
CPU

77. 77
KSQL’s Connections
Schema
Registry
CLI
REST
UI
KSQL
Server
KSQL
Server
REST
REST

78. 78
KSQL Client <-> Server: TLS
listeners=
http://host.name:port
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-https

79. 79
KSQL Client <-> Server: TLS
listeners=
http://host.name:port
./bin/ksql http://hostname.port
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-https

80. 80
./bin/ksql
--config-file my-cli.properties
https://hostname.port
listeners=
https://host.name:port
ssl.keystore.location=
/path/to/keystore
ssl.keystore.password=xxxx
ssl.key.password=yyyy
KSQL Client <-> Server: TLS
ssl.truststore.location=
/path/to/truststore
ssl.truststore.password=xxx
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-https

81. 81
./bin/ksql
--config-file my-cli.properties
https://hostname.port
listeners=
https://host.name:port
ssl.keystore.location=
/path/to/keystore
ssl.keystore.password=xxxx
ssl.key.password=yyyy
KSQL Client <-> Server: TLS
ssl.truststore.location=
/path/to/truststore
ssl.truststore.password=xxx
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-https

82. 82
ssl.truststore.location=
/path/to/truststore
ssl.truststore.password=xxx
ssl.keystore.location=
/path/to/keystore
ssl.keystore.password=yyy
ssl.key.password=zzz
listeners=
https://host.name:port
ssl.keystore.location=
/path/to/keystore
ssl.keystore.password=xxxx
ssl.key.password=yyyy
ssl.client.auth=true
ssl.truststore.location=
/path/to/truststore
ssl.truststore.password=zzzz
KSQL Client <-> Server: TLS
./bin/ksql
--config-file my-cli.properties
https://hostname.port
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-https

83. 83
KSQL Client <-> Server: Basic HTTP Auth
authentication.method=BASIC
authentication.roles=user
authentication.realm=
KsqlServer-Props
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-basic-http-authentication

84. 84
KSQL Client <-> Server: Basic HTTP Auth
authentication.method=BASIC
authentication.roles=user
authentication.realm=
KsqlServer-Props
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-basic-http-authentication
KSQL_OPTS=
-Djava.security.auth.login.config=
/path/to/jaas_config.file
KsqlServer-Props {
...
};

85. 85
KSQL Client <-> Server: Basic HTTP Auth
authentication.method=BASIC
authentication.roles=user
authentication.realm=
KsqlServer-Props
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-basic-http-authentication
KSQL_OPTS=
-Djava.security.auth.login.config=
/path/to/jaas_config.file
KsqlServer-Props {
...
};

86. 86
./bin/ksql
--user username
--password mypassword
https://hostname.port
KSQL Client <-> Server: Basic HTTP Auth
authentication.method=BASIC
authentication.roles=user
authentication.realm=
KsqlServer-Props
Learn more:
https://docs.confluent.io/current/ksql/docs/installation/server-config/security.html#configuring-ksql-for-basic-http-authentication
KSQL_OPTS=
-Djava.security.auth.login.config=
/path/to/jaas_config.file
KsqlServer-Props {
...
};

87. 87
KSQL Client <-> Server: Custom Plugins
Learn more:
https://github.com/confluentinc/rest-utils/blob/b0418a69b8fd40a55446d31da98e4da3f25b6b93/core/src/main/java/io/confluent/rest/
RestConfig.java#L229
https://github.com/confluentinc/rest-utils/blob/b0418a69b8fd40a55446d31da98e4da3f25b6b93/core/src/main/java/io/confluent/rest/
Application.java#L454
rest.servlet.initializor.classes=my.java.namespace.MyCustomSecurityHandler

88. 88
KSQL Client <-> Server: Custom Plugins
public class MyCustomSecurityHandler implements Consumer<ServletContextHandler> {
@Override
public void accept(final ServletContextHandler context) {
final ConstraintSecurityHandler myHandler = new ConstraintSecurityHandler();
// ...
context.setSecurityHandler(myHandler);
}
}
Learn more:
https://github.com/confluentinc/rest-utils/blob/b0418a69b8fd40a55446d31da98e4da3f25b6b93/core/src/main/java/io/confluent/rest/
RestConfig.java#L229
https://github.com/confluentinc/rest-utils/blob/b0418a69b8fd40a55446d31da98e4da3f25b6b93/core/src/main/java/io/confluent/rest/
Application.java#L454
rest.servlet.initializor.classes=my.java.namespace.MyCustomSecurityHandler

89. 89
KSQL Client <-> Server: Custom Plugins
public class MyCustomSecurityHandler implements Consumer<ServletContextHandler> {
@Override
public void accept(final ServletContextHandler context) {
final ConstraintSecurityHandler myHandler = new ConstraintSecurityHandler();
// ...
context.setSecurityHandler(myHandler);
}
}
Learn more:
https://github.com/confluentinc/rest-utils/blob/b0418a69b8fd40a55446d31da98e4da3f25b6b93/core/src/main/java/io/confluent/rest/
RestConfig.java#L229
https://github.com/confluentinc/rest-utils/blob/b0418a69b8fd40a55446d31da98e4da3f25b6b93/core/src/main/java/io/confluent/rest/
Application.java#L454
rest.servlet.initializor.classes=my.java.namespace.MyCustomSecurityHandler
websocket.servlet.initializor.classes=my.java.namespace.MyCustomSecurityHandler

90. 90
Securing KSQL’s Connections
KSQL <-> Kafka KSQL <->
Schema Registry
KSQL Client <->
KSQL Server
Encryption TLS TLS TLS
Authentication TLS
SASL
TLS
Basic HTTP Auth
TLS
Basic HTTP Auth
Custom Plugins
Authorization ACLs Custom Plugins
Quotas Network
CPU

91. 91
Securing KSQL’s Connections
KSQL <-> Kafka KSQL <->
Schema Registry
KSQL Client <->
KSQL Server
Encryption TLS TLS TLS
Authentication TLS
SASL
TLS
Basic HTTP Auth
Custom Plugins
TLS
Basic HTTP Auth
Custom Plugins
Authorization ACLs Custom Plugins Custom Plugins
Quotas Network
CPU

92. 92
KSQL’s Connections
Schema
Registry
CLI
REST
UI
KSQL
Server
KSQL
Server
REST
REST

93. 93
User-Defined Functions (UDFs)
Learn more:
https://docs.confluent.io/current/ksql/docs/developer-guide/udf.html#ksql-custom-functions-and-security
@UdfDescription(
name = “myFunc”,
description = “my custom function”)
public class MyFunc {
// ...
}
SELECT MYFUNC(...)
FROM stream_foo;

94. 94
User-Defined Functions (UDFs)
● ksql.udfs.enabled
Learn more:
https://docs.confluent.io/current/ksql/docs/developer-guide/udf.html#ksql-custom-functions-and-security
@UdfDescription(
name = “myFunc”,
description = “my custom function”)
public class MyFunc {
// ...
}
SELECT MYFUNC(...)
FROM stream_foo;

95. 95
User-Defined Functions (UDFs)
● ksql.udfs.enabled
● ksql.udf.enable.security.manager
Learn more:
https://docs.confluent.io/current/ksql/docs/developer-guide/udf.html#ksql-custom-functions-and-security
@UdfDescription(
name = “myFunc”,
description = “my custom function”)
public class MyFunc {
// ...
}
SELECT MYFUNC(...)
FROM stream_foo;

96. 96
User-Defined Functions (UDFs)
● ksql.udfs.enabled
● ksql.udf.enable.security.manager
● <ksql.extension.dir>/resource-blacklist.txt
Learn more:
https://docs.confluent.io/current/ksql/docs/developer-guide/udf.html#ksql-custom-functions-and-security
@UdfDescription(
name = “myFunc”,
description = “my custom function”)
public class MyFunc {
// ...
}
SELECT MYFUNC(...)
FROM stream_foo;
# resource-blacklist.txt
java.lang.Compiler$
java.lang.Process

97. 97
Logging
● Log4j
Learn more:
https://docs.confluent.io/current/ksql/docs/developer-guide/processing-log.html

98. 98
Logging
● Log4j
● Record processing log
Learn more:
https://docs.confluent.io/current/ksql/docs/developer-guide/processing-log.html
{
“type”: 1,
…,
“deserializationError”:{
“errorMessage”: “org.apache.kafka.connect.errors.DataException: [...]”,
“recordB64”: “TWFuIGlzIGRpc3Rpbmd1aXNoZWQsIG5vdCBvbmx5”
}
}

99. 99
Logging
● Log4j
● Record processing log
○ ksql.logging.processing.topic.auto.create
Learn more:
https://docs.confluent.io/current/ksql/docs/developer-guide/processing-log.html
{
“type”: 1,
…,
“deserializationError”:{
“errorMessage”: “org.apache.kafka.connect.errors.DataException: [...]”,
“recordB64”: “TWFuIGlzIGRpc3Rpbmd1aXNoZWQsIG5vdCBvbmx5”
}
}

100. 100
Logging
● Log4j
● Record processing log
○ ksql.logging.processing.topic.auto.create
○ ksql.logging.processing.topic.name
Learn more:
https://docs.confluent.io/current/ksql/docs/developer-guide/processing-log.html
{
“type”: 1,
…,
“deserializationError”:{
“errorMessage”: “org.apache.kafka.connect.errors.DataException: [...]”,
“recordB64”: “TWFuIGlzIGRpc3Rpbmd1aXNoZWQsIG5vdCBvbmx5”
}
}

101. 101
Logging
● Log4j
● Record processing log
○ ksql.logging.processing.topic.auto.create
○ ksql.logging.processing.topic.name
○ ksql.logging.processing.rows.include
Learn more:
https://docs.confluent.io/current/ksql/docs/developer-guide/processing-log.html
{
“type”: 1,
…,
“deserializationError”:{
“errorMessage”: “org.apache.kafka.connect.errors.DataException: [...]”,
“recordB64”: “TWFuIGlzIGRpc3Rpbmd1aXNoZWQsIG5vdCBvbmx5”
}
}

102. 102
Limitations and Futures
● Impersonation
● Authorization and quotas
● End-to-end encryption
● Shared TLS configs
● UDF whitelisting
● Resolving external passwords: KIP-421
Learn more:
https://docs.confluent.io/current/ksql/docs/capacity-planning.html
https://github.com/confluentinc/ksql/blob/cf29742512378106ccbd50c47b8ebb2d2204afc6/ksql-common/src/main/java/io/confluent/
ksql/util/KsqlConfig.java#L121
https://github.com/confluentinc/ksql/issues/1821
https://cwiki.apache.org/confluence/display/KAFKA/KIP-421%3A+Support+resolving+externalized+secrets+in+AbstractConfig

103. 103
Takeaways
● Works in a secure Kafka environment
● Lock down KSQL by using headless mode
○ Or secure KSQL’s REST endpoint
● Deploy separate KSQL clusters for different use cases
● Consider: UDFs and record processing log

104. 104
Questions?