This document provides an overview of Cloudera's Navigator Key Trustee, which is a key management server that acts as a proxy between CDH components and an external key store. It discusses how Key Trustee uses encryption zone keys stored in an external hardware security module to encrypt data encryption keys, which are then used to encrypt data at rest in HDFS. The document also covers Key Trustee's architecture, deployment considerations, access control lists, and troubleshooting steps.

1. 1© Cloudera, Inc. All rights reserved.
Hadoop Distributed File System (HDFS)
Encryption with Navigator Key Trustee
Protecting Enterprise Data Hubs
Luke Hebert, Customer Operations Engineer, Security SME

2. 2© Cloudera, Inc. All rights reserved.©2014 Cloudera, Inc. All rights reserved.
Data Security Requirements
Protect data while preserving
application choice
Better alignment with key
management policies
Integrate with existing HSMs as
part of KMI (optional)
Data
Protecting data in the
cluster from
unauthorized visibility
InfoSec Concept:
Compliance
Key Trustee KMS &
Key Trustee

3. 3© Cloudera, Inc. All rights reserved.
“Virtual safe-deposit box” for managing encryption keys or other
Hadoop security artifact
Navigator Key Trustee
• Separates Keys from Encrypted Data
• Centralized Management
• Integration with HSMs from Thales,
and SafeNet
• Roadmap: Management of SSL
certificates, SSH keys, tokens,
passwords, Kerberos Keytab Files,
and more

4. © Cloudera, Inc. All rights reserved.
Key Trustee Key Management Server Proxy (KMS)
CDH Key Services

5. © Cloudera, Inc. All rights reserved.
• Acts as a broker between EDH and the backing Key Store.
• Is an extension used by the hadoop-kms component.
• Replaces the Java Key Store Key Provider with Key Trustee as the Key Store.
• Allows CDH components to retrieve Encryption Zone Keys as required.
• It has a single primary use case today.
• Data Encryption at Rest within HDFS
What is Key Trustee KMS ?

6. © Cloudera, Inc. All rights reserved.
• Implements a REST API which is utilized by components.
• Provides Key Caching.
• Provides a Key Pool to the NN.
• Modifies the behavior of several components.
• Handles retrieval of delegation tokens for jobs.
• Uses SPNEGO to facilitate authentication when Kerberos is
enabled.
• Implements ACLs which protect key accessibility.
• Allows for HA Communication with the Key Trustee Backing Key Store
and other KMS Proxies.
What does Key Trustee KMS Provide.

7. © Cloudera, Inc. All rights reserved.
Architecture
How does it work?

8. © Cloudera, Inc. All rights reserved.RESTRICTED -- DO NOT DISTRIBUTE © Cloudera, Inc. All rights reserved.
Key Trustee
Topology

9. © Cloudera, Inc. All rights reserved.
A Few Key Concepts.
• Encryption Zone Key (EZKEY)
• This key much like a mount key is associated
with an encryption zone in HDFS.
• Encrypted Data Encryption Key (EDEK)
• This is an encrypted copy of a Data
Encryption Key.
• Data Encryption Key (DEK)
• This is the real data encryption key used to
encrypt data stored within a file, zone, or
block device. This particular key concept is
used in both Navigator Encrypt and HDFS
Transparent Data Encryption (TDE).

10. © Cloudera, Inc. All rights reserved.
A Few Key Concepts.

11. © Cloudera, Inc. All rights reserved.
KMS Proxy Deployment considerations.
• KMS Proxy Servers
• Deployed as Service Role Instances within a Managed CDH cluster.
• Should be on isolated and protected Hardware.
• Should be installed on a clean Operating System.
• Same requirements as CDH Components for Install.
• Isolate from other services and avoid co-location. (Hardens Security)
• Requires the KEYTRUSTEE parcel be installed.
(As opposed to the KEYTRUSTEE_SERVER Parcel)
• Multiple KMS Proxies supported without LB.
• CDH Components internally enable the KMS client when configured.

12. © Cloudera, Inc. All rights reserved.
KMS Proxy Deployment considerations.

13. © Cloudera, Inc. All rights reserved.
KMS Proxy: High Level Overview
● Encryption occurs on the requesting client.
○ Data is encrypted before it lands on disk.
○ The KMS encrypts and decrypts specific key components.
○ The KMS does not encrypt content.
○ The KMS does not store keys.

14. © Cloudera, Inc. All rights reserved.
KMS Key Operation (Write)
● The EZ Key encrypts the data encryption keys (DEKs) that are used in turn to encrypt each file.
● DEKs are encrypted with the EZ key to form an encrypted data encryption key. (EDEK)
● The EDEK is stored on the NameNode via an extended attribute on the file.
● The EZ Key is stored on the backing Key Store (Key Trustee Server)

15. © Cloudera, Inc. All rights reserved. ‹#›© Cloudera, Inc. All rights reserved.
ACLs
Controlling Access to Keys

16. © Cloudera, Inc. All rights reserved.
• Hadoop has no concept of a Key Admin.
• Cloudera is creating a framework for Key Management based on roles.
• Creating this role allows for better compliance.
• Separating Key Management operations will ensure a separation of duties.
• In order to build this framework an administrator must lay down the correct ACLs.
• There are multiple classes of ACLs connected to the KMS.
• The ACLs are implemented in the upstream Hadoop Core KMS.
ACLs

17. © Cloudera, Inc. All rights reserved.
• There are 5 distinct ACL Classes available for use in the KMS.
• hadoop.kms.acl.<op>
• Controls permission to perform KMS level operations or access features.
• hadoop.kms.blacklist.<op>
• Controls permission to perform KMS level operations or access features.
• key.acl.<key-name>.<op>
• Controls permission to perform operations for a specific key.
• default.key.acl.<op>
• Controls permission to perform operations for keys that are not otherwise
specified by key.acl.<key-name>.<op>
• whitelist.key.acl.<op>
• Controls permission to perform key operations across all keys.
ACL Classes

18. © Cloudera, Inc. All rights reserved.
KMS ACL Flow
© Cloudera, Inc. All rights reserved.

19. © Cloudera, Inc. All rights reserved.
• Key Access
• In order to perform an operation, <OP>, on a key <KEY> a user
• Must be allowed by <hadoop.kms.acl.OP>
• Not disallowed by <hadoop.kms.blacklist.OP>
• and allowed by any of the 3 conditions below.
• <key.acl.KEY.OP>
• <whitelist.key.acl.OP>
• <default.key.acl.OP> if there is no <key.acl.KEY.OP> entry
Allowing user access

20. © Cloudera, Inc. All rights reserved.
Troubleshooting
How to get the information you need.

21. © Cloudera, Inc. All rights reserved.
• The KMS client cannot communicate with the server using the defined ports.
• Deposits and retrievals fail.
• The KMS or Key Trustee server is down or unable to handle incoming request.
• Deposits and retrievals fail.
• The HSM backing Key Trustee is unreachable or misconfigured.
• Deposits and retrievals fail.
• The server SSL certificates are invalid or expired.
• Communication Between KMS and Key Trustee Server will timeout.
• Low Entropy
• Key operations will be slow or hang indefinitely.
• Client registration will be slow or hang indefinitely.
• /var/lib/kms-keytrustee is out of sync when using multiple KMS Proxies.
• CDH component request for keys will result in random access to a subset of keys.
• Transparent Encryption may randomly fail for different components.
Common Issues

22. © Cloudera, Inc. All rights reserved.
Logs and places to look for errors.
• Attempt to replicate the operation and capture stdout/stderr
• Inspect messages.
• Ensure the right auth mechanism is set for all components. (Kerberos/Simple)
• Make sure zookeeper is working if you are in HA mode.
• Look for low level hardware problems.
• Logs on the kms client
• /var/log/kms-keytrustee
• /var/run/cloudera-scm-agent/process/<id>-keytrustee-KMS_KEYTRUSTEE
• Logs on Key Trustee
• /var/lib/keytrustee/logs/
• /var/run/cloudera-scm-agent/process/<id>-keytrustee_server-KEYTRUSTEE_ACTIVE_SERVER
(Managed)
• /var/run/cloudera-scm-agent/process/<id>-keytrustee_server-
KEYTRUSTEE_PASSIVE_SERVER (Managed)

23. © Cloudera, Inc. All rights reserved.
● The value returned.
○ An estimate of entropy available in the entropy pool.
● Low entropy.
○ Slow Key Operations
○ Key Generation Failures
○ Client Registration Failures
● Values below 500.
○ Considered a low entropy condition.
○ Requires injection of entropy from a source such as a DRNG, rngd, or haveged.
Checking Entropy Available
[root@server-1 ~]# cat /proc/sys/kernel/random/entropy_avail
3711

24. © Cloudera, Inc. All rights reserved.
Verifying server availability
[root@kms-01 ~]# curl -kv https://keytrustee-1.vpc.cloudera.dev:11371/?a=fingerprint
* About to connect() to keytrustee.cloudera.dev port 11371 (#0)
…
> GET /?a=fingerprint HTTP/1.1
…
* Closing connection #0
4096R/A71981C5F9E3F70C6484C5244BBC98C031F593DA
● Basic test of service availability from the client to the server.
○ A fingerprint return should indicate that the Key Database and Server are online.
● If the certificates are self-signed
○ You may need to use the -k flag in order to disable certificate validation.
● Operations are performed over HTTP you can increase the verbosity of curl.
○ When using -v you can inspect the server responses and headers.

25. © Cloudera, Inc. All rights reserved.
Verify KMS Fingerprint (gpg)
[root@kms-01 ~]# gpg --homedir /var/lib/kms-keytrustee/keytrustee/.keytrustee --fingerprint
gpg: WARNING: unsafe ownership on homedir `/var/lib/kms-keytrustee/keytrustee/.keytrustee'
/var/lib/kms-keytrustee/keytrustee/.keytrustee/pubring.gpg
----------------------------------------------------------
pub 4096R/31F593DA 2015-08-25
Key fingerprint = A719 81C5 F9E3 F70C 6484 C524 4BBC 98C0 31F5 93DA
uid keytrustee (keytrustee Server Key) <keytrustee@keytrustee-1.vpc.cloudera.com>
sub 4096R/D6017A05 2015-08-25
pub 4096R/E3D4EDD2 2015-08-25
Key fingerprint = 359B BCFF 965C FC18 2F5A A107 F15C 6514 E3D4 EDD2
uid keytrustee (client) <kms@kms-1.vpc.cloudera.com>
sub 4096R/193290BB 2015-08-25
[root@kms-01 ~]#
Note: GPG Keyring used for Message Authentication, Privacy, Message Encryption and Identity.

26. © Cloudera, Inc. All rights reserved.
● hadoop key list
○ Is the KMS Online.
○ Can hadoop access key material which is cached or otherwise.
○ Do you get a consistent list of keys returned from multiple attempts.
○ If you stop and start the KMS role can you still obtain key information.
Basic Key Ops
[root@server-1 ~]# hadoop key list
Listing keys for KeyProvider:
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@5e026e6d
mykey
mykey2
[root@server-1 ~]#
RESTRICTED -- DO NOT DISTRIBUTE

27. © Cloudera, Inc. All rights reserved.
● hadoop key create mykey3
○ Is the KMS Online.
○ Can hadoop create key material.
○ Is the HSM responding to Key Deposit request.
○ Is Key Trustee online.
Basic Key Ops
[root@server-1 ~]# hadoop key create mykey3
mykey3 has been successfully created with options Options{cipher='AES/CTR/NoPadding',
bitLength=128, description='null', attributes=null}.
org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider@5457487e has been updated.
RESTRICTED -- DO NOT DISTRIBUTE

28. © Cloudera, Inc. All rights reserved.
Thank you
Questions?
RESTRICTED -- DO NOT DISTRIBUTE