Securing Hadoop's REST APIs with Apache Knox Gateway Hadoop Summit June 6th, 2014

K
Kevin MinderSenior Member of Technical Staff à Hortonworks
© Hortonworks Inc. 2014
Securing Hadoop’s REST APIs
Apache Knox Gateway
Hadoop Summit 2014
Kevin Minder
Larry McCayhttp://knox.apache.org/
user (at) knox.apache.org
dev (at) knox.apache.org
© Hortonworks Inc. 2014
What is Apache Knox?
• The Apache Knox Gateway is…
• an extensible reverse proxy framework
• for securely exposing REST APIs and HTTP based services at a
perimeter
• out of the box it provides:
• support for several of the most common Hadoop services
• integration with enterprise authentication systems
• several other useful features
© Hortonworks Inc. 2014
What the Apache Knox Gateway isn’t
• Not an alternative to Kerberos for strong Hadoop core authentication
• Not a channel for high volume data ingest or export
© Hortonworks Inc. 2014
History and Status of the Apache Knox Gateway?
• 2013-02: Accepted into Apache Incubator
• 2013-04: Released 0.2.0
• 2013-10: Released 0.3.0
• 2014-02: Graduated to Apache TLP
• 2014-04: Released 0.4.0, Included in HDP 2.1
© Hortonworks Inc. 2014
Why Knox?
Simplified Access
• Kerberos encapsulation
• Extends API reach
• Single access point
• Multi-cluster support
• Single SSL certificate
Centralized Control
• Central REST API auditing
• Service-level authorization
• Alternative to SSH “edge node”
Enterprise Integration
• LDAP integration
• Active Directory integration
• SSO integration
• Apache Shiro extensibility
• Custom extensibility
Enhanced Security
• Protect network details
• Partial SSL for non-SSL services
• WebApp vulnerability filter
© Hortonworks Inc. 2014
Layers Of Hadoop Security
Perimeter Level Security
• Network Security (i.e. Firewalls)
• Apache Knox (i.e. Gateways)
Authentication
• Kerberos
• Delegation Tokens
OS Security
• File Permissions
• Process Isolation
Authorization
• MR ACLs
• HDFS Permissions
• HDFS ACLs
• HiveATZ-NG
• HBase ACLs
• Accumulo Label Security
• XA Security Policies
Data Protection
• Transport
• Storage
© Hortonworks Inc. 2014
REST API
Hadoop
Services
What does Perimeter Security really mean?
Gateway
REST API
Firewall
User
Firewall
required at
perimeter
(today)
Knox Gateway
controls all
Hadoop REST
API access
through firewall
Hadoop
cluster
mostly
unaffected
Firewall only
allows
connections
through specific
ports from Knox
host
© Hortonworks Inc. 2014
What REST APIs does Hadoop support?
Service URL Example
WebHDFS http://localhost:50070/webhdfs
WebHCat (aka Templeton) http://localhost:50111/templeton
Oozie http://localhost:11000/oozie
HBase (via Stargate) http://localhost:60080
Hive (HiveServer2) http://localhost:10001/cliservice
jdbc:hive2://localhost:10001/?hive.server2.transport.mode=http;hive.server2.thrif
t.http.path=cliservice
© Hortonworks Inc. 2014
Basic Knox Operation & Extensibility
© Hortonworks Inc. 2014
Authentication and Identity Propagation
1. REST API Request
2. HTTP Basic Auth Challenge
kminder:secret
3. Authenticate kminder:secret
knox
keytab
4. Authenticates as
knox via SPNego
(i.e. Kerberos)
5. REST API Request
doAs kminder
0. Configure
knox user to be
known as
trusted proxy
LDAP
© Hortonworks Inc. 2014
Scalability and Fault Tolerance
Hadoop
Apache HTTPD+mod_proxy_balancer
f5 BIG-IP
HAProxy
Knox Cluster
(no shared state)
Really any
traditional
web tier
load balancer
© Hortonworks Inc. 2014
Extensibility: Providers and Services
• Both are dynamically discovered on the class path via Java’s ServiceLoader
• Providers
• Add new features to the gateway that can be used by Services
• Typically result in one or more filters being added to one or more chains
• Services
• Add new endpoints to the gateway to expose a specific service
• Assemble filter chains to enable specific features via providers
• Includes providing configuration to providers
• For example URL rewrite rules
• Associates endpoints with filter chains
© Hortonworks Inc. 2014
Topology Files
• Describe the services that should be exposed for a specific cluster
• Found in <GATEWAY_HOME>/conf/topologies
• Name of topology file dictates URL component
• sandbox.xml -> http://localhost:8443/gateway/sandbox/webhdfs/…
<topology>
<gateway>
<provider>
<role>authentication</role>
<name>custom</name>
</provider>
</gateway>
<service>
<role>WEBHDFS</role>
<url>http://localhost:50070</url>
</service>
</topology>
Location of
WebHDFS in
target cluster
Selects an
authentication
provider
implementation
© Hortonworks Inc. 2014
Enhanced Security
© Hortonworks Inc. 2014
Topology Leakage: WebHDFS Example
• WebHDFS direct
curl -i -X PUT 'http://localhost:50070/webhdfs/v1/user/guest/file1?op=CREATE&user.name=guest’
HTTP/1.1 307 TEMPORARY_REDIRECT
Location:
http://sandbox.hortonworks.com:50075/webhdfs/v1/user/guest/file1?op=CREATE&user.name=guest&namenoderp
caddress=sandbox.hortonworks.com:8020&overwrite=false
• WebHDFS via Knox
curl -u guest:guest-password -i -k -X PUT 'https://localhost:8443/webhdfs/v1/user/guest/file2?op=CREATE’
HTTP/1.1 307 Temporary Redirect
Location:
https://localhost:8443/gateway/sandbox/webhdfs/data/v1/webhdfs/v1/user/guest/file2?_=AAAACAAAABAAAACAg
UDT7-QQZlpkcm09lxrxI0Bgo9d-
Egghp_qxmd4pQsmm3zvYc3M_LrDBQpMBNA48DnMS9QOhyzywCMl1WAShyX4RUETPjEcZa6x9Jwz7TMANj
SRKMR6F3rKf93ME-VsI2Phe8CX72L6oiI778--8F9DQCO8LHFHzLL70iB13Hm2BLyj-x9p3tn7FOHxkbPl5d-
eHxVop7Dk
RPC and
HTTP address
of DataNode is
leaked
unnecessarily
to REST client
Encrypted query param contains
dispatch information used by gateway
when redirect followed
© Hortonworks Inc. 2014
Topology Leakage: Oozie Example
• Oozie direct
<configuration>
<property>
<name>oozie.wf.application.path</name>
<value>hdfs://foo:9000/user/bansalm/myapp/</value>
</property>
...
</configuration>
• Oozie via Knox
<configuration>
<property>
<name>oozie.wf.application.path</name>
<value>/user/bansalm/myapp/</value>
</property>
...
</configuration>
• Example of submitting an Oozie job from Apache docs
• https://oozie.apache.org/docs/4.0.1/WebServicesAPI.html
• HTTP POST XML below to /oozie/v1/jobs
REST client
must know
RPC address
of NameNode
© Hortonworks Inc. 2014
Partial SSL for non-SSL enabled services
REST API REST API
WebHCat
DMZ
Desktop
Gateway
HTTPS HTTP
First “hop”
through
public/corp
networks
protected with
SSL
Last “hop”
within
secure
network
non-SSL
© Hortonworks Inc. 2014
WebApp Vulnerability Filter
• The Knox WebAppSec provider allows for the plugin of vulnerability prevention filters
• Cross Site Request Forgery CSRF is currently provided
• Uses common required header technique
• Later releases will include more filters based on standard techniques
<provider
<role>webappsec</role>
<name>WebAppSec</name>
<enabled>true</enabled>
<param><name>csrf.enabled</name><value>true</value></param>
<param><name>csrf.customHeader</name><value>X-XSRF-Header</value></param>
<param><name>csrf.methodsToIgnore</name><value>GET,OPTIONS,HEAD</value></param>
</provider>
© Hortonworks Inc. 2014
Simplified Access
© Hortonworks Inc. 2014
Knox Service URLs vs. direct URLs
Service Direct URL Knox URL
WebHDFS http://namenode-host:50070/webhdfs https://knox-host:8443/webhdfs
WebHCat http://webhcat-host:50111/templeton https://knox-host:8443/templeton
Oozie http://ooziehost:11000/oozie https://knox-host:8443/oozie
HBase http://hbasehost:60080 https://knox-host:8443/hbase
Hive http://hivehost:10001/cliservice https://knox-host:8443/hive
Masters could
be on many
different hosts
One hosts,
one port
Consistent
paths
© Hortonworks Inc. 2014
Hadoop CLIs require full server configs
/etc/hive/conf/hive-site.xml
<property>
<name>hive.server2.thrift.http.port</name>
<value>10001</value>
</property>
<property>
<name>hive.server2.thrift.http.path</name>
<value>cliservice</value>
</property>
/etc/hadoop/conf/core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://sandbox.hortonworks.com:8020</value>
</property>
/etc/hadoop/conf/hdfs-site.xml
<property>
<name>dfs.namenode.http-address</name>
<value>sandbox.hortonworks.com:50070</value>
</property>
/etc/hadoop/conf/yarn-site.xml
<property>
<name>yarn.resourcemanager.address</name>
<value>sandbox.hortonworks.com:8050</value>
</property>
/etc/hive-webhcat/conf/webhcat-site.xml
<property>
<name>templeton.port</name>
<value>50111</value>
</property>
/etc/oozie/conf/oozie-site.xml
<property>
<name>oozie.base.url</name>
<value>http://sandbox.hortonworks.com:11000/oozie</value>
</property>
HBase – Command line
These files
may all be
on different
nodes on
the cluster
too!
© Hortonworks Inc. 2014
Kerberos Encapsulation
1. REST API Request
2. HTTP Basic Auth Challenge
kminder:secret
3. Authenticate kminder:secret
knox
keytab
4. Authenticates as
knox via SPNego
(i.e. Kerberos)
5. REST API Request
doAs kminder
0. Configure
knox as trusted
proxy
The client isn’t
even aware the
cluster is secured
with Kerberos
© Hortonworks Inc. 2014
REST API REST API
Hadoop
REST API Reach: Intranet Access Model
DMZ
Desktop
Gateway
Users will
discover novel
ways to use easily
accessible REST
APIs
© Hortonworks Inc. 2014
HTML/JS REST
Hadoop
REST API Reach: Middleware Access Model
Web Tier / DMZ
Browser
“Give the APIs to the Apps”
GatewayApp
Server
REST
Most enterprises
cannot deal with
Kerberos in the
web tier and don’t
have CLI access
© Hortonworks Inc. 2014
REST API REST API
Hadoop
REST API Reach: Internet Access Model
DMZ
“Give the APIs to the Everyone”
Gateway
Internet
HaaS vendors
are exposing
Hadoop REST
APIs to the
internet. What
does the API tell
these clients to
know about your
cluster?
© Hortonworks Inc. 2014
Multi-Cluster Support
Gateway
http://knox:8443/gateway/green/webhdfs/v1 http://knox:8443/gateway/blue/webhdfs/v1
green
Production
Cluster
blue
Research
Cluster
One hosts,
one port for
many
clusters
© Hortonworks Inc. 2014
Simplified Client Certificate Management
hdfs
cert
hive
cert
hbase
cert
knox
cert
knox
pubkey
hive
pubkey
hbase
pubkey
hdfs
pubkey
• User only needs to trust Knox’s cert
• Admin only needs to manage multiple keys on Knox hosts
© Hortonworks Inc. 2014
Centralized Control
© Hortonworks Inc. 2014
SCP/SSHLogin Hadoop CLIs
Hadoop
Client Edge Node CLI Access Model
DMZ
Edge Node
Desktop
“Take the Users to the CLI”Limited
auditing on
edge node
CLI too hard
to install on
desktops
© Hortonworks Inc. 2014
REST APILogin REST API
Hadoop
Improved auditing and access control
DMZ
Desktop
Gateway
All activity
audited
consistently
Additional
authorization
control
available
© Hortonworks Inc. 2014
Service Level Authorization
• Control access to services by user, group or IP address
<provider>
<role>authorization</role>
<name>AclsAuthz</name>
<enabled>true</enabled>
<param>
<name>WEBHDFS.acl</name>
<value>*;admin;127.0.0.1</value>
</param>
</provider>
© Hortonworks Inc. 2014
XA Secure Integration
1. REST API Request
0. Distribute
policy
3. REST API Request
Policy Server
Agent
2. Service level
authorization decision
Agent
integrated as
authorization
provider
Policies
authored in
the portal and
distributed by
the policy
server
© Hortonworks Inc. 2014
KNOX-250: SSH Bastion Auditing Functionality
• Community is developing an extension
• Based on Apache MINA SSHD
• Provides administrative SSH access via Knox
• Further centralizes auditing of cluster administration
• https://issues.apache.org/jira/browse/KNOX-250
© Hortonworks Inc. 2014
KNOX-250: SSH Bastion Auditing Functionality
SSHLogin Hadoop CLI
Hadoop
DMZ
Desktop
Gateway
All activity
audited
consistently
© Hortonworks Inc. 2014
Enterprise Integration
© Hortonworks Inc. 2014
Apache Shiro Authentication Provider
• Apache Shiro is the primary authentication provider for Knox
• Used for both LDAP and Active Directory
• Apache Shiro is a popular JEE and JSE security framework
• Very modular and flexible architecture
• Many community extensions
• Integrated into Knox as a servlet filter
© Hortonworks Inc. 2014
Apache Shiro Authentication Provider
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<param>
<name>main.ldapRealm</name>
<value>org.apache.shiro.realm.ldap.JndiLdapRealm</value>
</param>
<param>
<name>main.ldapRealm.userDnTemplate</name>
<value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.url</name>
<value>ldap://localhost:33389</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
<value>simple</value>
</param>
<param>
<name>urls./**</name>
<value>authcBasic</value>
</param>
</provider>
© Hortonworks Inc. 2014
SSO Integration
• Similar in concept Hadoop’s trusted proxy model
• Preconfigured for SiteMinder use case
• HTTP Headers used to propagate pre-authenticated user and group info
• Only acceptable for use in a tightly controlled network environment
<provider>
<role>federation</role>
<name>HeaderPreAuth</name>
<enabled>true</enabled>
<param>
<name>preauth.validation.method</name>
<value>preauth.ip.validation</value>
</param>
<param>
<name>preauth.ip.addresses</name>
<value>127.0.*</value>
</param>
</provider>
© Hortonworks Inc. 2014
OAuth 2
• OAuth is becoming the defacto standard for communicating a user’s
identity to REST APIs
• It allows for explicit authorization by the user for the application to
access resources
• It has a number of ways to represent the user and authentication
information to go over the wire
• JSON Web Token (JWT) is an emerging standard for representing the
various claims, attributes and scopes of an identity
• Can be used as a bearer token, URL parameter or Header
• OAuth is also gaining popularity as a federation token for SSO
integrations
© Hortonworks Inc. 2014
KNOX-393: OAuth Resource Provider
• Community investigating OAuth Federation Provider extension
• Considering Apache Oltu
• Warning: Diagram dramatically oversimplified
• There are a number of other potential flows
2. REST API Request
Authorization: Bearer <token>
3. validateAccessToken(<token>)
4. Authenticates as
knox via SPNego
(i.e. Kerberos)
5. REST API Request
doAs kminder
0. Configure
knox user to be
known as
trusted proxy
1. requestAccessToken(JWT)
return Bearer token
kminder
© Hortonworks Inc. 2014
What is next for Knox?
Jira Assignee Description
KNOX-393: OAuth Resource Provider for
Middleware and Application Integration
COMMUNITY OAuth 2 federation provider potentially based on Apache
Oltu for external application SSO to Knox and Hadoop
KNOX-355: Support Knox Authentication
Provider based on Hadoop Auth Module
(SPNEGO)
KNOX Team SPNEGO authentication support for Knox clients
KNOX-250: SSH Bastion Auditing Functionality COMMUNITY SSH tunneling and auditing functionality in addition to
REST gateway services.
KNOX-353: Support Hadoop Java Client URLs KNOX Team In order to be used Hadoop CLIs that can use REST, we
need to support the expected URLs. This is in addition to
the extended URLs for multiple Hadoop cluster support
by Knox.
KNOX-242: LDAP Authentication
Enhancements
KNOX Team Search attribute based authentication rather than simple
LDAP bind.
KNOX-74: Support YARN REST API KNOX Team Add support for the YARN REST API
KNOX-66: Support Ambari REST API access
via the Gateway
KNOX Team Add support for the Ambari REST API
TBD TBD What is important to you?
© Hortonworks Inc. 2014
Interested?
• We’re hiring!
• http://hortonworks.com/careers/open-positions/
• Especially hands on platform level development experience with
• Kerberos
• LDAP
• OAuth
• SAML
• JAAS/GSS-API
• Crypto
1 sur 42

Recommandé

Hadoop REST API Security with Apache Knox Gateway par
Hadoop REST API Security with Apache Knox GatewayHadoop REST API Security with Apache Knox Gateway
Hadoop REST API Security with Apache Knox GatewayDataWorks Summit
12.9K vues45 diapositives
Hdp security overview par
Hdp security overview Hdp security overview
Hdp security overview Hortonworks
11.6K vues36 diapositives
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 edition par
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 editionHadoop and Kerberos: the Madness Beyond the Gate: January 2016 edition
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 editionSteve Loughran
1.8K vues38 diapositives
Apache Ranger par
Apache RangerApache Ranger
Apache RangerRommel Garcia
5.1K vues17 diapositives
Hadoop security par
Hadoop securityHadoop security
Hadoop securityShivaji Dutta
5.3K vues43 diapositives
Improvements in Hadoop Security par
Improvements in Hadoop SecurityImprovements in Hadoop Security
Improvements in Hadoop SecurityDataWorks Summit
4.2K vues39 diapositives

Contenu connexe

Tendances

Apache Knox setup and hive and hdfs Access using KNOX par
Apache Knox setup and hive and hdfs Access using KNOXApache Knox setup and hive and hdfs Access using KNOX
Apache Knox setup and hive and hdfs Access using KNOXAbhishek Mallick
2.7K vues13 diapositives
An Approach for Multi-Tenancy Through Apache Knox par
An Approach for Multi-Tenancy Through Apache KnoxAn Approach for Multi-Tenancy Through Apache Knox
An Approach for Multi-Tenancy Through Apache KnoxDataWorks Summit/Hadoop Summit
1.8K vues19 diapositives
Hadoop Security Today and Tomorrow par
Hadoop Security Today and TomorrowHadoop Security Today and Tomorrow
Hadoop Security Today and TomorrowDataWorks Summit
5.5K vues26 diapositives
Nl HUG 2016 Feb Hadoop security from the trenches par
Nl HUG 2016 Feb Hadoop security from the trenchesNl HUG 2016 Feb Hadoop security from the trenches
Nl HUG 2016 Feb Hadoop security from the trenchesBolke de Bruin
3.2K vues24 diapositives
2014 sept 4_hadoop_security par
2014 sept 4_hadoop_security2014 sept 4_hadoop_security
2014 sept 4_hadoop_securityAdam Muise
3.4K vues49 diapositives
Curb your insecurity with HDP - Tips for a Secure Cluster par
Curb your insecurity with HDP - Tips for a Secure ClusterCurb your insecurity with HDP - Tips for a Secure Cluster
Curb your insecurity with HDP - Tips for a Secure Clusterahortonworks
416 vues29 diapositives

Tendances(20)

Apache Knox setup and hive and hdfs Access using KNOX par Abhishek Mallick
Apache Knox setup and hive and hdfs Access using KNOXApache Knox setup and hive and hdfs Access using KNOX
Apache Knox setup and hive and hdfs Access using KNOX
Abhishek Mallick2.7K vues
Nl HUG 2016 Feb Hadoop security from the trenches par Bolke de Bruin
Nl HUG 2016 Feb Hadoop security from the trenchesNl HUG 2016 Feb Hadoop security from the trenches
Nl HUG 2016 Feb Hadoop security from the trenches
Bolke de Bruin3.2K vues
2014 sept 4_hadoop_security par Adam Muise
2014 sept 4_hadoop_security2014 sept 4_hadoop_security
2014 sept 4_hadoop_security
Adam Muise3.4K vues
Curb your insecurity with HDP - Tips for a Secure Cluster par ahortonworks
Curb your insecurity with HDP - Tips for a Secure ClusterCurb your insecurity with HDP - Tips for a Secure Cluster
Curb your insecurity with HDP - Tips for a Secure Cluster
ahortonworks416 vues
Hadoop Security Architecture par Owen O'Malley
Hadoop Security ArchitectureHadoop Security Architecture
Hadoop Security Architecture
Owen O'Malley30.2K vues
Hadoop & Security - Past, Present, Future par Uwe Printz
Hadoop & Security - Past, Present, FutureHadoop & Security - Past, Present, Future
Hadoop & Security - Past, Present, Future
Uwe Printz2.5K vues
Distilling Hadoop Patterns of Use and How You Can Use Them for Your Big Data ... par Hortonworks
Distilling Hadoop Patterns of Use and How You Can Use Them for Your Big Data ...Distilling Hadoop Patterns of Use and How You Can Use Them for Your Big Data ...
Distilling Hadoop Patterns of Use and How You Can Use Them for Your Big Data ...
Hortonworks4.6K vues
Open Source Security Tools for Big Data par Rommel Garcia
Open Source Security Tools for Big DataOpen Source Security Tools for Big Data
Open Source Security Tools for Big Data
Rommel Garcia3.3K vues
Hadoop and Kerberos: the Madness Beyond the Gate par Steve Loughran
Hadoop and Kerberos: the Madness Beyond the GateHadoop and Kerberos: the Madness Beyond the Gate
Hadoop and Kerberos: the Madness Beyond the Gate
Steve Loughran2.3K vues
Extending Apache Ranger Authorization Beyond Hadoop: Review of Apache Ranger ... par DataWorks Summit
Extending Apache Ranger Authorization Beyond Hadoop: Review of Apache Ranger ...Extending Apache Ranger Authorization Beyond Hadoop: Review of Apache Ranger ...
Extending Apache Ranger Authorization Beyond Hadoop: Review of Apache Ranger ...
DataWorks Summit1.4K vues
Apache Knox - Hadoop Security Swiss Army Knife par DataWorks Summit
Apache Knox - Hadoop Security Swiss Army KnifeApache Knox - Hadoop Security Swiss Army Knife
Apache Knox - Hadoop Security Swiss Army Knife
DataWorks Summit1.2K vues
Hortonworks Technical Workshop: Interactive Query with Apache Hive par Hortonworks
Hortonworks Technical Workshop: Interactive Query with Apache Hive Hortonworks Technical Workshop: Interactive Query with Apache Hive
Hortonworks Technical Workshop: Interactive Query with Apache Hive
Hortonworks6.4K vues
Hadoop Operations: How to Secure and Control Cluster Access par Cloudera, Inc.
Hadoop Operations: How to Secure and Control Cluster AccessHadoop Operations: How to Secure and Control Cluster Access
Hadoop Operations: How to Secure and Control Cluster Access
Cloudera, Inc.4.1K vues
Hadoop security overview_hit2012_1117rev par Jason Shih
Hadoop security overview_hit2012_1117revHadoop security overview_hit2012_1117rev
Hadoop security overview_hit2012_1117rev
Jason Shih2.8K vues
Apache ranger meetup par nvvrajesh
Apache ranger meetupApache ranger meetup
Apache ranger meetup
nvvrajesh1.2K vues

En vedette

Treat your enterprise data lake indigestion: Enterprise ready security and go... par
Treat your enterprise data lake indigestion: Enterprise ready security and go...Treat your enterprise data lake indigestion: Enterprise ready security and go...
Treat your enterprise data lake indigestion: Enterprise ready security and go...DataWorks Summit
1.6K vues33 diapositives
Information security in big data -privacy and data mining par
Information security in big data -privacy and data miningInformation security in big data -privacy and data mining
Information security in big data -privacy and data miningharithavijay94
6.4K vues30 diapositives
Troubleshooting Kerberos in Hadoop: Taming the Beast par
Troubleshooting Kerberos in Hadoop: Taming the BeastTroubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the BeastDataWorks Summit
2.4K vues16 diapositives
Big Data and Security - Where are we now? (2015) par
Big Data and Security - Where are we now? (2015)Big Data and Security - Where are we now? (2015)
Big Data and Security - Where are we now? (2015)Peter Wood
3.3K vues35 diapositives
Apache Knox Gateway "Single Sign On" expands the reach of the Enterprise Users par
Apache Knox Gateway "Single Sign On" expands the reach of the Enterprise UsersApache Knox Gateway "Single Sign On" expands the reach of the Enterprise Users
Apache Knox Gateway "Single Sign On" expands the reach of the Enterprise UsersDataWorks Summit
3.8K vues22 diapositives

En vedette(18)

Treat your enterprise data lake indigestion: Enterprise ready security and go... par DataWorks Summit
Treat your enterprise data lake indigestion: Enterprise ready security and go...Treat your enterprise data lake indigestion: Enterprise ready security and go...
Treat your enterprise data lake indigestion: Enterprise ready security and go...
DataWorks Summit1.6K vues
Information security in big data -privacy and data mining par harithavijay94
Information security in big data -privacy and data miningInformation security in big data -privacy and data mining
Information security in big data -privacy and data mining
harithavijay946.4K vues
Troubleshooting Kerberos in Hadoop: Taming the Beast par DataWorks Summit
Troubleshooting Kerberos in Hadoop: Taming the BeastTroubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the Beast
DataWorks Summit2.4K vues
Big Data and Security - Where are we now? (2015) par Peter Wood
Big Data and Security - Where are we now? (2015)Big Data and Security - Where are we now? (2015)
Big Data and Security - Where are we now? (2015)
Peter Wood3.3K vues
Apache Knox Gateway "Single Sign On" expands the reach of the Enterprise Users par DataWorks Summit
Apache Knox Gateway "Single Sign On" expands the reach of the Enterprise UsersApache Knox Gateway "Single Sign On" expands the reach of the Enterprise Users
Apache Knox Gateway "Single Sign On" expands the reach of the Enterprise Users
DataWorks Summit3.8K vues
Hadoop and Data Access Security par Cloudera, Inc.
Hadoop and Data Access SecurityHadoop and Data Access Security
Hadoop and Data Access Security
Cloudera, Inc.10.1K vues
OAuth - Open API Authentication par leahculver
OAuth - Open API AuthenticationOAuth - Open API Authentication
OAuth - Open API Authentication
leahculver22.3K vues
Hadoop Internals (2.3.0 or later) par Emilio Coppa
Hadoop Internals (2.3.0 or later)Hadoop Internals (2.3.0 or later)
Hadoop Internals (2.3.0 or later)
Emilio Coppa39.6K vues
HADOOP TECHNOLOGY ppt par sravya raju
HADOOP  TECHNOLOGY pptHADOOP  TECHNOLOGY ppt
HADOOP TECHNOLOGY ppt
sravya raju38.6K vues
Cours Big Data Chap1 par Amal Abid
Cours Big Data Chap1Cours Big Data Chap1
Cours Big Data Chap1
Amal Abid15.6K vues
Hadoop Overview & Architecture par EMC
Hadoop Overview & Architecture  Hadoop Overview & Architecture
Hadoop Overview & Architecture
EMC64.6K vues
Hadoop et son écosystème par Khanh Maudoux
Hadoop et son écosystèmeHadoop et son écosystème
Hadoop et son écosystème
Khanh Maudoux9.3K vues
Seminar Presentation Hadoop par Varun Narang
Seminar Presentation HadoopSeminar Presentation Hadoop
Seminar Presentation Hadoop
Varun Narang81.5K vues
What is Big Data? par Bernard Marr
What is Big Data?What is Big Data?
What is Big Data?
Bernard Marr585.3K vues

Similaire à Securing Hadoop's REST APIs with Apache Knox Gateway Hadoop Summit June 6th, 2014

Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox par
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache KnoxFortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache KnoxDataWorks Summit
498 vues27 diapositives
Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho... par
Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...
Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...huguk
1.7K vues29 diapositives
Apache Kafka Security par
Apache Kafka Security Apache Kafka Security
Apache Kafka Security DataWorks Summit/Hadoop Summit
6.9K vues39 diapositives
An Apache Hive Based Data Warehouse par
An Apache Hive Based Data WarehouseAn Apache Hive Based Data Warehouse
An Apache Hive Based Data WarehouseDataWorks Summit
8.1K vues32 diapositives
HA Deployment Architecture with HAProxy and Keepalived par
HA Deployment Architecture with HAProxy and KeepalivedHA Deployment Architecture with HAProxy and Keepalived
HA Deployment Architecture with HAProxy and KeepalivedGanapathi Kandaswamy
1.8K vues41 diapositives
Kafka Security par
Kafka SecurityKafka Security
Kafka SecurityDataWorks Summit/Hadoop Summit
2.6K vues43 diapositives

Similaire à Securing Hadoop's REST APIs with Apache Knox Gateway Hadoop Summit June 6th, 2014(20)

Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox par DataWorks Summit
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache KnoxFortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox
Fortifying Multi-Cluster Hybrid Cloud Data Lakes using Apache Knox
DataWorks Summit498 vues
Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho... par huguk
Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...
Apache Argus - How do I secure my entire Hadoop cluster? Olivier Renault @ Ho...
huguk1.7K vues
HA Deployment Architecture with HAProxy and Keepalived par Ganapathi Kandaswamy
HA Deployment Architecture with HAProxy and KeepalivedHA Deployment Architecture with HAProxy and Keepalived
HA Deployment Architecture with HAProxy and Keepalived
Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS par Hortonworks
Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFSDiscover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS
Discover HDP 2.1: Apache Hadoop 2.4.0, YARN & HDFS
Hortonworks4.8K vues
Discover Enterprise Security Features in Hortonworks Data Platform 2.1: Apach... par Hortonworks
Discover Enterprise Security Features in Hortonworks Data Platform 2.1: Apach...Discover Enterprise Security Features in Hortonworks Data Platform 2.1: Apach...
Discover Enterprise Security Features in Hortonworks Data Platform 2.1: Apach...
Hortonworks4.7K vues
Discover HDP 2.1: Interactive SQL Query in Hadoop with Apache Hive par Hortonworks
Discover HDP 2.1: Interactive SQL Query in Hadoop with Apache HiveDiscover HDP 2.1: Interactive SQL Query in Hadoop with Apache Hive
Discover HDP 2.1: Interactive SQL Query in Hadoop with Apache Hive
Hortonworks4.2K vues
How to create a multi tenancy for an interactive data analysis par Tiago Simões
How to create a multi tenancy for an interactive data analysisHow to create a multi tenancy for an interactive data analysis
How to create a multi tenancy for an interactive data analysis
Tiago Simões573 vues
One Click Hadoop Clusters - Anywhere (Using Docker) par DataWorks Summit
One Click Hadoop Clusters - Anywhere (Using Docker)One Click Hadoop Clusters - Anywhere (Using Docker)
One Click Hadoop Clusters - Anywhere (Using Docker)
DataWorks Summit1.8K vues
OpenStack APIs: Present and Future (Beta Talk) par Wade Minter
OpenStack APIs: Present and Future (Beta Talk)OpenStack APIs: Present and Future (Beta Talk)
OpenStack APIs: Present and Future (Beta Talk)
Wade Minter1.9K vues

Dernier

Keep par
KeepKeep
KeepGeniusee
75 vues10 diapositives
Navigating container technology for enhanced security by Niklas Saari par
Navigating container technology for enhanced security by Niklas SaariNavigating container technology for enhanced security by Niklas Saari
Navigating container technology for enhanced security by Niklas SaariMetosin Oy
12 vues34 diapositives
SAP FOR CONTRACT MANUFACTURING.pdf par
SAP FOR CONTRACT MANUFACTURING.pdfSAP FOR CONTRACT MANUFACTURING.pdf
SAP FOR CONTRACT MANUFACTURING.pdfVirendra Rai, PMP
11 vues2 diapositives
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge... par
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...Deltares
17 vues12 diapositives
A first look at MariaDB 11.x features and ideas on how to use them par
A first look at MariaDB 11.x features and ideas on how to use themA first look at MariaDB 11.x features and ideas on how to use them
A first look at MariaDB 11.x features and ideas on how to use themFederico Razzoli
45 vues36 diapositives
Software evolution understanding: Automatic extraction of software identifier... par
Software evolution understanding: Automatic extraction of software identifier...Software evolution understanding: Automatic extraction of software identifier...
Software evolution understanding: Automatic extraction of software identifier...Ra'Fat Al-Msie'deen
7 vues33 diapositives

Dernier(20)

Navigating container technology for enhanced security by Niklas Saari par Metosin Oy
Navigating container technology for enhanced security by Niklas SaariNavigating container technology for enhanced security by Niklas Saari
Navigating container technology for enhanced security by Niklas Saari
Metosin Oy12 vues
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge... par Deltares
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...
DSD-INT 2023 Delft3D FM Suite 2024.01 2D3D - New features + Improvements - Ge...
Deltares17 vues
A first look at MariaDB 11.x features and ideas on how to use them par Federico Razzoli
A first look at MariaDB 11.x features and ideas on how to use themA first look at MariaDB 11.x features and ideas on how to use them
A first look at MariaDB 11.x features and ideas on how to use them
Software evolution understanding: Automatic extraction of software identifier... par Ra'Fat Al-Msie'deen
Software evolution understanding: Automatic extraction of software identifier...Software evolution understanding: Automatic extraction of software identifier...
Software evolution understanding: Automatic extraction of software identifier...
Tridens DevOps par Tridens
Tridens DevOpsTridens DevOps
Tridens DevOps
Tridens9 vues
Headless JS UG Presentation.pptx par Jack Spektor
Headless JS UG Presentation.pptxHeadless JS UG Presentation.pptx
Headless JS UG Presentation.pptx
Jack Spektor7 vues
Elevate your SAP landscape's efficiency and performance with HCL Workload Aut... par HCLSoftware
Elevate your SAP landscape's efficiency and performance with HCL Workload Aut...Elevate your SAP landscape's efficiency and performance with HCL Workload Aut...
Elevate your SAP landscape's efficiency and performance with HCL Workload Aut...
HCLSoftware6 vues
DSD-INT 2023 - Delft3D User Days - Welcome - Day 3 - Afternoon par Deltares
DSD-INT 2023 - Delft3D User Days - Welcome - Day 3 - AfternoonDSD-INT 2023 - Delft3D User Days - Welcome - Day 3 - Afternoon
DSD-INT 2023 - Delft3D User Days - Welcome - Day 3 - Afternoon
Deltares15 vues
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t... par Deltares
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...
DSD-INT 2023 Thermobaricity in 3D DCSM-FM - taking pressure into account in t...
Deltares9 vues
What Can Employee Monitoring Software Do?​ par wAnywhere
What Can Employee Monitoring Software Do?​What Can Employee Monitoring Software Do?​
What Can Employee Monitoring Software Do?​
wAnywhere21 vues
DSD-INT 2023 The Danube Hazardous Substances Model - Kovacs par Deltares
DSD-INT 2023 The Danube Hazardous Substances Model - KovacsDSD-INT 2023 The Danube Hazardous Substances Model - Kovacs
DSD-INT 2023 The Danube Hazardous Substances Model - Kovacs
Deltares8 vues
El Arte de lo Possible par Neo4j
El Arte de lo PossibleEl Arte de lo Possible
El Arte de lo Possible
Neo4j39 vues
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx par animuscrm
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
2023-November-Schneider Electric-Meetup-BCN Admin Group.pptx
animuscrm13 vues
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ... par Deltares
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
DSD-INT 2023 Wave-Current Interaction at Montrose Tidal Inlet System and Its ...
Deltares10 vues
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra... par Marc Müller
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra....NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra...
.NET Developer Conference 2023 - .NET Microservices mit Dapr – zu viel Abstra...
Marc Müller38 vues
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ... par Donato Onofri
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...
Unmasking the Dark Art of Vectored Exception Handling: Bypassing XDR and EDR ...
Donato Onofri773 vues

Securing Hadoop's REST APIs with Apache Knox Gateway Hadoop Summit June 6th, 2014

  • 1. © Hortonworks Inc. 2014 Securing Hadoop’s REST APIs Apache Knox Gateway Hadoop Summit 2014 Kevin Minder Larry McCayhttp://knox.apache.org/ user (at) knox.apache.org dev (at) knox.apache.org
  • 2. © Hortonworks Inc. 2014 What is Apache Knox? • The Apache Knox Gateway is… • an extensible reverse proxy framework • for securely exposing REST APIs and HTTP based services at a perimeter • out of the box it provides: • support for several of the most common Hadoop services • integration with enterprise authentication systems • several other useful features
  • 3. © Hortonworks Inc. 2014 What the Apache Knox Gateway isn’t • Not an alternative to Kerberos for strong Hadoop core authentication • Not a channel for high volume data ingest or export
  • 4. © Hortonworks Inc. 2014 History and Status of the Apache Knox Gateway? • 2013-02: Accepted into Apache Incubator • 2013-04: Released 0.2.0 • 2013-10: Released 0.3.0 • 2014-02: Graduated to Apache TLP • 2014-04: Released 0.4.0, Included in HDP 2.1
  • 5. © Hortonworks Inc. 2014 Why Knox? Simplified Access • Kerberos encapsulation • Extends API reach • Single access point • Multi-cluster support • Single SSL certificate Centralized Control • Central REST API auditing • Service-level authorization • Alternative to SSH “edge node” Enterprise Integration • LDAP integration • Active Directory integration • SSO integration • Apache Shiro extensibility • Custom extensibility Enhanced Security • Protect network details • Partial SSL for non-SSL services • WebApp vulnerability filter
  • 6. © Hortonworks Inc. 2014 Layers Of Hadoop Security Perimeter Level Security • Network Security (i.e. Firewalls) • Apache Knox (i.e. Gateways) Authentication • Kerberos • Delegation Tokens OS Security • File Permissions • Process Isolation Authorization • MR ACLs • HDFS Permissions • HDFS ACLs • HiveATZ-NG • HBase ACLs • Accumulo Label Security • XA Security Policies Data Protection • Transport • Storage
  • 7. © Hortonworks Inc. 2014 REST API Hadoop Services What does Perimeter Security really mean? Gateway REST API Firewall User Firewall required at perimeter (today) Knox Gateway controls all Hadoop REST API access through firewall Hadoop cluster mostly unaffected Firewall only allows connections through specific ports from Knox host
  • 8. © Hortonworks Inc. 2014 What REST APIs does Hadoop support? Service URL Example WebHDFS http://localhost:50070/webhdfs WebHCat (aka Templeton) http://localhost:50111/templeton Oozie http://localhost:11000/oozie HBase (via Stargate) http://localhost:60080 Hive (HiveServer2) http://localhost:10001/cliservice jdbc:hive2://localhost:10001/?hive.server2.transport.mode=http;hive.server2.thrif t.http.path=cliservice
  • 9. © Hortonworks Inc. 2014 Basic Knox Operation & Extensibility
  • 10. © Hortonworks Inc. 2014 Authentication and Identity Propagation 1. REST API Request 2. HTTP Basic Auth Challenge kminder:secret 3. Authenticate kminder:secret knox keytab 4. Authenticates as knox via SPNego (i.e. Kerberos) 5. REST API Request doAs kminder 0. Configure knox user to be known as trusted proxy LDAP
  • 11. © Hortonworks Inc. 2014 Scalability and Fault Tolerance Hadoop Apache HTTPD+mod_proxy_balancer f5 BIG-IP HAProxy Knox Cluster (no shared state) Really any traditional web tier load balancer
  • 12. © Hortonworks Inc. 2014 Extensibility: Providers and Services • Both are dynamically discovered on the class path via Java’s ServiceLoader • Providers • Add new features to the gateway that can be used by Services • Typically result in one or more filters being added to one or more chains • Services • Add new endpoints to the gateway to expose a specific service • Assemble filter chains to enable specific features via providers • Includes providing configuration to providers • For example URL rewrite rules • Associates endpoints with filter chains
  • 13. © Hortonworks Inc. 2014 Topology Files • Describe the services that should be exposed for a specific cluster • Found in <GATEWAY_HOME>/conf/topologies • Name of topology file dictates URL component • sandbox.xml -> http://localhost:8443/gateway/sandbox/webhdfs/… <topology> <gateway> <provider> <role>authentication</role> <name>custom</name> </provider> </gateway> <service> <role>WEBHDFS</role> <url>http://localhost:50070</url> </service> </topology> Location of WebHDFS in target cluster Selects an authentication provider implementation
  • 14. © Hortonworks Inc. 2014 Enhanced Security
  • 15. © Hortonworks Inc. 2014 Topology Leakage: WebHDFS Example • WebHDFS direct curl -i -X PUT 'http://localhost:50070/webhdfs/v1/user/guest/file1?op=CREATE&user.name=guest’ HTTP/1.1 307 TEMPORARY_REDIRECT Location: http://sandbox.hortonworks.com:50075/webhdfs/v1/user/guest/file1?op=CREATE&user.name=guest&namenoderp caddress=sandbox.hortonworks.com:8020&overwrite=false • WebHDFS via Knox curl -u guest:guest-password -i -k -X PUT 'https://localhost:8443/webhdfs/v1/user/guest/file2?op=CREATE’ HTTP/1.1 307 Temporary Redirect Location: https://localhost:8443/gateway/sandbox/webhdfs/data/v1/webhdfs/v1/user/guest/file2?_=AAAACAAAABAAAACAg UDT7-QQZlpkcm09lxrxI0Bgo9d- Egghp_qxmd4pQsmm3zvYc3M_LrDBQpMBNA48DnMS9QOhyzywCMl1WAShyX4RUETPjEcZa6x9Jwz7TMANj SRKMR6F3rKf93ME-VsI2Phe8CX72L6oiI778--8F9DQCO8LHFHzLL70iB13Hm2BLyj-x9p3tn7FOHxkbPl5d- eHxVop7Dk RPC and HTTP address of DataNode is leaked unnecessarily to REST client Encrypted query param contains dispatch information used by gateway when redirect followed
  • 16. © Hortonworks Inc. 2014 Topology Leakage: Oozie Example • Oozie direct <configuration> <property> <name>oozie.wf.application.path</name> <value>hdfs://foo:9000/user/bansalm/myapp/</value> </property> ... </configuration> • Oozie via Knox <configuration> <property> <name>oozie.wf.application.path</name> <value>/user/bansalm/myapp/</value> </property> ... </configuration> • Example of submitting an Oozie job from Apache docs • https://oozie.apache.org/docs/4.0.1/WebServicesAPI.html • HTTP POST XML below to /oozie/v1/jobs REST client must know RPC address of NameNode
  • 17. © Hortonworks Inc. 2014 Partial SSL for non-SSL enabled services REST API REST API WebHCat DMZ Desktop Gateway HTTPS HTTP First “hop” through public/corp networks protected with SSL Last “hop” within secure network non-SSL
  • 18. © Hortonworks Inc. 2014 WebApp Vulnerability Filter • The Knox WebAppSec provider allows for the plugin of vulnerability prevention filters • Cross Site Request Forgery CSRF is currently provided • Uses common required header technique • Later releases will include more filters based on standard techniques <provider <role>webappsec</role> <name>WebAppSec</name> <enabled>true</enabled> <param><name>csrf.enabled</name><value>true</value></param> <param><name>csrf.customHeader</name><value>X-XSRF-Header</value></param> <param><name>csrf.methodsToIgnore</name><value>GET,OPTIONS,HEAD</value></param> </provider>
  • 19. © Hortonworks Inc. 2014 Simplified Access
  • 20. © Hortonworks Inc. 2014 Knox Service URLs vs. direct URLs Service Direct URL Knox URL WebHDFS http://namenode-host:50070/webhdfs https://knox-host:8443/webhdfs WebHCat http://webhcat-host:50111/templeton https://knox-host:8443/templeton Oozie http://ooziehost:11000/oozie https://knox-host:8443/oozie HBase http://hbasehost:60080 https://knox-host:8443/hbase Hive http://hivehost:10001/cliservice https://knox-host:8443/hive Masters could be on many different hosts One hosts, one port Consistent paths
  • 21. © Hortonworks Inc. 2014 Hadoop CLIs require full server configs /etc/hive/conf/hive-site.xml <property> <name>hive.server2.thrift.http.port</name> <value>10001</value> </property> <property> <name>hive.server2.thrift.http.path</name> <value>cliservice</value> </property> /etc/hadoop/conf/core-site.xml <property> <name>fs.defaultFS</name> <value>hdfs://sandbox.hortonworks.com:8020</value> </property> /etc/hadoop/conf/hdfs-site.xml <property> <name>dfs.namenode.http-address</name> <value>sandbox.hortonworks.com:50070</value> </property> /etc/hadoop/conf/yarn-site.xml <property> <name>yarn.resourcemanager.address</name> <value>sandbox.hortonworks.com:8050</value> </property> /etc/hive-webhcat/conf/webhcat-site.xml <property> <name>templeton.port</name> <value>50111</value> </property> /etc/oozie/conf/oozie-site.xml <property> <name>oozie.base.url</name> <value>http://sandbox.hortonworks.com:11000/oozie</value> </property> HBase – Command line These files may all be on different nodes on the cluster too!
  • 22. © Hortonworks Inc. 2014 Kerberos Encapsulation 1. REST API Request 2. HTTP Basic Auth Challenge kminder:secret 3. Authenticate kminder:secret knox keytab 4. Authenticates as knox via SPNego (i.e. Kerberos) 5. REST API Request doAs kminder 0. Configure knox as trusted proxy The client isn’t even aware the cluster is secured with Kerberos
  • 23. © Hortonworks Inc. 2014 REST API REST API Hadoop REST API Reach: Intranet Access Model DMZ Desktop Gateway Users will discover novel ways to use easily accessible REST APIs
  • 24. © Hortonworks Inc. 2014 HTML/JS REST Hadoop REST API Reach: Middleware Access Model Web Tier / DMZ Browser “Give the APIs to the Apps” GatewayApp Server REST Most enterprises cannot deal with Kerberos in the web tier and don’t have CLI access
  • 25. © Hortonworks Inc. 2014 REST API REST API Hadoop REST API Reach: Internet Access Model DMZ “Give the APIs to the Everyone” Gateway Internet HaaS vendors are exposing Hadoop REST APIs to the internet. What does the API tell these clients to know about your cluster?
  • 26. © Hortonworks Inc. 2014 Multi-Cluster Support Gateway http://knox:8443/gateway/green/webhdfs/v1 http://knox:8443/gateway/blue/webhdfs/v1 green Production Cluster blue Research Cluster One hosts, one port for many clusters
  • 27. © Hortonworks Inc. 2014 Simplified Client Certificate Management hdfs cert hive cert hbase cert knox cert knox pubkey hive pubkey hbase pubkey hdfs pubkey • User only needs to trust Knox’s cert • Admin only needs to manage multiple keys on Knox hosts
  • 28. © Hortonworks Inc. 2014 Centralized Control
  • 29. © Hortonworks Inc. 2014 SCP/SSHLogin Hadoop CLIs Hadoop Client Edge Node CLI Access Model DMZ Edge Node Desktop “Take the Users to the CLI”Limited auditing on edge node CLI too hard to install on desktops
  • 30. © Hortonworks Inc. 2014 REST APILogin REST API Hadoop Improved auditing and access control DMZ Desktop Gateway All activity audited consistently Additional authorization control available
  • 31. © Hortonworks Inc. 2014 Service Level Authorization • Control access to services by user, group or IP address <provider> <role>authorization</role> <name>AclsAuthz</name> <enabled>true</enabled> <param> <name>WEBHDFS.acl</name> <value>*;admin;127.0.0.1</value> </param> </provider>
  • 32. © Hortonworks Inc. 2014 XA Secure Integration 1. REST API Request 0. Distribute policy 3. REST API Request Policy Server Agent 2. Service level authorization decision Agent integrated as authorization provider Policies authored in the portal and distributed by the policy server
  • 33. © Hortonworks Inc. 2014 KNOX-250: SSH Bastion Auditing Functionality • Community is developing an extension • Based on Apache MINA SSHD • Provides administrative SSH access via Knox • Further centralizes auditing of cluster administration • https://issues.apache.org/jira/browse/KNOX-250
  • 34. © Hortonworks Inc. 2014 KNOX-250: SSH Bastion Auditing Functionality SSHLogin Hadoop CLI Hadoop DMZ Desktop Gateway All activity audited consistently
  • 35. © Hortonworks Inc. 2014 Enterprise Integration
  • 36. © Hortonworks Inc. 2014 Apache Shiro Authentication Provider • Apache Shiro is the primary authentication provider for Knox • Used for both LDAP and Active Directory • Apache Shiro is a popular JEE and JSE security framework • Very modular and flexible architecture • Many community extensions • Integrated into Knox as a servlet filter
  • 37. © Hortonworks Inc. 2014 Apache Shiro Authentication Provider <provider> <role>authentication</role> <name>ShiroProvider</name> <enabled>true</enabled> <param> <name>main.ldapRealm</name> <value>org.apache.shiro.realm.ldap.JndiLdapRealm</value> </param> <param> <name>main.ldapRealm.userDnTemplate</name> <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value> </param> <param> <name>main.ldapRealm.contextFactory.url</name> <value>ldap://localhost:33389</value> </param> <param> <name>main.ldapRealm.contextFactory.authenticationMechanism</name> <value>simple</value> </param> <param> <name>urls./**</name> <value>authcBasic</value> </param> </provider>
  • 38. © Hortonworks Inc. 2014 SSO Integration • Similar in concept Hadoop’s trusted proxy model • Preconfigured for SiteMinder use case • HTTP Headers used to propagate pre-authenticated user and group info • Only acceptable for use in a tightly controlled network environment <provider> <role>federation</role> <name>HeaderPreAuth</name> <enabled>true</enabled> <param> <name>preauth.validation.method</name> <value>preauth.ip.validation</value> </param> <param> <name>preauth.ip.addresses</name> <value>127.0.*</value> </param> </provider>
  • 39. © Hortonworks Inc. 2014 OAuth 2 • OAuth is becoming the defacto standard for communicating a user’s identity to REST APIs • It allows for explicit authorization by the user for the application to access resources • It has a number of ways to represent the user and authentication information to go over the wire • JSON Web Token (JWT) is an emerging standard for representing the various claims, attributes and scopes of an identity • Can be used as a bearer token, URL parameter or Header • OAuth is also gaining popularity as a federation token for SSO integrations
  • 40. © Hortonworks Inc. 2014 KNOX-393: OAuth Resource Provider • Community investigating OAuth Federation Provider extension • Considering Apache Oltu • Warning: Diagram dramatically oversimplified • There are a number of other potential flows 2. REST API Request Authorization: Bearer <token> 3. validateAccessToken(<token>) 4. Authenticates as knox via SPNego (i.e. Kerberos) 5. REST API Request doAs kminder 0. Configure knox user to be known as trusted proxy 1. requestAccessToken(JWT) return Bearer token kminder
  • 41. © Hortonworks Inc. 2014 What is next for Knox? Jira Assignee Description KNOX-393: OAuth Resource Provider for Middleware and Application Integration COMMUNITY OAuth 2 federation provider potentially based on Apache Oltu for external application SSO to Knox and Hadoop KNOX-355: Support Knox Authentication Provider based on Hadoop Auth Module (SPNEGO) KNOX Team SPNEGO authentication support for Knox clients KNOX-250: SSH Bastion Auditing Functionality COMMUNITY SSH tunneling and auditing functionality in addition to REST gateway services. KNOX-353: Support Hadoop Java Client URLs KNOX Team In order to be used Hadoop CLIs that can use REST, we need to support the expected URLs. This is in addition to the extended URLs for multiple Hadoop cluster support by Knox. KNOX-242: LDAP Authentication Enhancements KNOX Team Search attribute based authentication rather than simple LDAP bind. KNOX-74: Support YARN REST API KNOX Team Add support for the YARN REST API KNOX-66: Support Ambari REST API access via the Gateway KNOX Team Add support for the Ambari REST API TBD TBD What is important to you?
  • 42. © Hortonworks Inc. 2014 Interested? • We’re hiring! • http://hortonworks.com/careers/open-positions/ • Especially hands on platform level development experience with • Kerberos • LDAP • OAuth • SAML • JAAS/GSS-API • Crypto