This talk covers why Apache Zookeeper is a good fit for coordinating processes in a distributed environment, prior Python attempts at a client and the current state of the art Python client library, how unifying development efforts to merge several Python client libraries has paid off, features available to Python processes, and how to gracefully handle failures in a set of distributed processes.

1. DISTRIBUTED COORDINATION
WITH PYTHON
BenBangert
mozilla

2. Toolsof theTrade

3. DISTRIBUTED COORDINATION IS NOT...
• Distributed Databases (Cassandra, Riak)
• Distributed Computing (Hadoop, etc.)
• Distributed Event Analysis (Storm)

4. TheCommonElement

5. Apache Zookeeper

6. ZooKeeperisacentralizedservicefor
maintainingconfigurationinformation,
naming,providingdistributedsynchronization,
andprovidinggroupservices.

7. ZOOKEEPER

8. WHY NOT USE...
• Memcached?
• MongoDB?
• Postgres/MySQL?

11. Hierarchical data structure in znodes

13. • Session Based
• Znode watches
• Ephemeral and Sequential Znodes

14. • Last for duration of client session
• Session dies when connection is closed or expires
• Can’t have children znodes
EPHEMERAL ZNODES

15. SEQUENTIAL ZNODES
• Supply a node name (or not), get node name back with a trailing sequence
number (0001, 0002, 0003, etc.)
• Can be combined with ephemeral flag

16. BASIC COMMANDS
• create(PATH, DATA...)
• get(PATH...)
• get_children(PATH...)
• set(PATH, DATA...)
• delete(PATH...)

18. PYTHON CLIENTS
• txzookeeper
• kazoo
• unified client that works with gevent
• implements wire protocol in pure Python

19. USE KAZOO

20. EASY TO USE
from kazoo.client import KazooClient
client = KazooClient()
client.start()

21. USE CASES

22. CONFIGURATION
• Store settings in node data
• Organize node structure
• Set watches on nodes of interest

24. PARTY MEMBERSHIP
• Join a party, find out who else is around
• Elect a leader if desired
• Recipe in Kazoo

25. LOCKS
• Lock a resource for a single client
• Lock a resource for multiple clients (Semaphore)
• Hard to write properly
• Recipe in Kazoo

26. BUILDING HIGHER LEVEL
ABSTRACTIONS
ON
ZOOKEEPER

27. CAVEAT

28. DO NOT IMPLEMENT YOURSELF
USE THE RECIPE

30. BASIC STEPS
• Create lock parent node if needed
• Create ephemeral+sequence node under parent, store node name
returned
• Get children of lock node
• Sort children list by sequence number
• First child in the list has the lock!

31. THINGS TO WATCH OUT FOR
• Avoid the thundering herd, use watches only when needed
• When our node isn’t the lowest, watch the one in front of us
• Only one client wanting a lock is ‘woken’ when the lock is released by a
different client

32. HANDLING FAILURE

33. ROBUST CODE TAKES EFFORT
• What happens when a server fails?
• What happens when the client fails?
• What happens when we don’t know if the server has failed?

34. STOPPING WHEN UNCERTAIN

35. A BIT BETTER VERSION...

36. EVEN BETTER

37. FAILURE WILL HAPPEN
• Fail fast, fail completely.
• Session expiration is a good time to sys.exit
• Always include jitter (kazoo includes jitter on its connection and command
retry operations)
• Consider what exceptions can occur in any code relying on a distributed
system

38. • Distributed systems are hard
• Use existing battle-proven tools (Zookeeper, Kazoo)
• Always consider everything that can fail, and how
• Be wary of tools that don’t tell you how they fail
• Read Kyle Kingsbury’s Jepsen posts to see examples of
systems failing: http://aphyr.com/tags/jepsen

39. FIN

40. QUESTIONS?