Kerberos is the ubiquitous authentication mechanism when it comes to secure any Hadoop Services. With recent updates in Hadoop core and various Apache Hadoop components, inherent Kerberos support has matured and has come a long way. Understanding & configuring Kerberos is still a challenge but even more painful & frustrating is troubleshooting a Kerberos issue. There are lot of things (small & big) that can go wrong (and will go wrong!). This talk covers the Kerberos debugging part in detail and discusses the tools & tricks that can be used to narrow down any Kerberos issue. Rather than discussing the issues and their resolution, we will focus on how to approach a Kerberos problem and do's / dont's in Kerberos scene. This talk will provide a step by step guide that will equip the audience for troubleshooting future Kerberos problems. Agenda is to discuss: - Systematic approach to Kerberos troubleshooting - Kerberos Tools available in Hadoop arsenal - Tips & Tricks to narrow down Kerberos issues quickly - Some nasty Kerberos issues from Support trenches Some prior knowledge on Kerberos basics will be appreciated but is not a prerequisite. Speaker: Vipin Rathor, Sr. Product Specialist (HDP Security), Hortonworks

1. 1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Troubleshooting Kerberos
In Hadoop :
Taming the Beast
DataWorks Summit
Sept 2017

2. 2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Author Profile
Vipin Rathor
Sr. Product Specialist (HDP Security)
Contributed to Kerberos, Apache Zeppelin, Apache Atlas
vrathor@hortonworks.com / @VipinRathor46

3. 3 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Agenda
• Why Kerberos?
• Where is Kerberos used across the Hadoop Stack?
• What is Kerberos & how does it work
• Realms, Principals and Keytabs
• Systematic Approach to Kerberos Nirvana
• Tools available in Hadoop
• Native Kerberos Tools / Debug Options
• Kerberos Checklist
• Most Common Kerberos Error Messages (& their meanings)

4. 4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Why Kerberos?
• Universal Authentication mechanism for Hadoop stack
• Integrates with enterprise user management (e.g. Active Directory)
Solves:
• How can parts of a cluster trust each other
(NameNodes, DataNodes, YARN, HBase, ZooKeeper...)
• How can users trust the system?
• How can the system trust users?
• Foundation for: how can users delegate rights to applications?
• Without Kerberos: your cluster has NO security
Hadoop clusters are some of the largest Kerberos systems ever!!

5. 5 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Where is Kerberos used across the Hadoop Stack?
• Ubiquitous End-User / Hadoop Service Authentication mechanism
• Hadoop DelegationToken (Delegated authentication to NameNode)
• != Kerberos Tickets
• Bootstrapped with Kerberos authentication token
• HTTP Authentication
• Using SPNEGO (RFC 4559)
• Via Browsers / cURL (curl --negotiate)
• RPC Authentication
• Using Simple Authentication & Security Layer aka SASL (!= SSL)
• Java API Based Kerberos login
• Using JGSS / JAAS
• GSS-API (RFC 2743)

6. 6 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
What is Kerberos
• Open source, Developed by MIT
• Password is NEVER transmitted over wire
• Central trusted authority – Key Distribution Center (KDC)
• Symmetric key (common shared key)
• Flavors:
• MIT Kerberos
• Active Directory
• Heimdal Kerberos (OS X)

7. 7 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
How does Kerberos work
End User
- Does kinit (1 & 2)
- Runs HDFS
command (3 - 6)
Hadoop NameNode
- Starts up with nn.service.keytab
- Verifies user and gives access to
HDFS
KDC
- Provisions user keys and
service keytabs (e.g.
nn.service.keytab)
- Provides TGT and TGS

8. 8 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Realms, Principals and Keytabs
• Realm
• User Principal
• E.g. user1@HWX.COM
• ken/admin@HWX.COM
• ken/sandbox.hortonworks.com@HWX.COM
• Service Principal
• E.g. HTTP/sandbox.hortonworks.com@HWX.COM
• nn/node1.hortonworks.com@HWX.COM
• dn/node2.hortonworks.com@HWX.COM
• dn/_HOST@HWX.COM
• Keytabs
• Service keytabs (for service)
• Headless keytabs (for user)

9. 9 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Systematic Approach to Kerberos Nirvana
• Identify the involved parties (user, service, keytabs, nodes)
• Identify the stage where Kerberos is failing
• Based on stage & error message, narrow down between client-side or service-
side issue
• Check & verify configurations for correctness using the appropriate tools
• Repeat as necessary

10. 10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Kerberos Tools Available in Hadoop
• Kdiag
• Runs a series of diagnostic checks & gives suggestions
• hadoop org.apache.hadoop.security.KDiag

11. 11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Kerberos Tools Available in Hadoop (cntd..)
• HadoopKerberosName
• Checks Auth_to_local rules (Kerberos Principal to Unix user name conversion)
• hadoop org.apache.hadoop.security.HadoopKerberosName
nn/bali1.openstacklocal@LAB.HORTONWORKS.NET

12. 12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Native Kerberos Tools / Debug Options
• via command line
• kinit
• klist -eaf / klist –kte
• kvno
• kdestroy
• export KRB5_TRACE=/tmp/krb5-curl.out
curl -ivL --negotiate -u: "http://namenode-host:50070/webhdfs/v1/?op=LISTSTATUS"
• via debug messages
• export HADOOP_JAAS_DEBUG=true
• export HADOOP_ROOT_LOGGER=DEBUG,console
• via Java library
• -Dsun.security.krb5.debug=true
• -Dsun.security.spnego.debug=true
• export OPTS=“$OPTS -Dsun.security.krb5.debug=true”

13. 13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Kerberos Checklist
• FQDN
• Name Resolution
• If DNS is configured, then check reverse lookup
• Date/Time sync (< 5 minutes)
• Configuration file - /etc/krb5.conf
• Principal Names
• Stale Keytabs (via kvno)
• Credential Cache location (JAAS config)
• Which Java suite, JCE policy
• KDC log file - /var/log/kerberos/krb5kdc.log

14. 14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Most Common Kerberos Error Messages (& their meaning)
• <unknown client> for <unknown service> 
• Decrypt Integrity Check Failed
• AES256 EncType not supported
• Clock skew too great
• Kerberos service principal not found in the database
• Client not found in the database
• No valid initial credential found

15. 15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
References
• http://web.mit.edu/kerberos/
• http://www.kerberos.org/software/tutorial.html
• https://github.com/steveloughran/kerberos_and_hadoop

16. 16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Thank you !