Xiang Li, Head of Distributed Systems at CoreOS SF CoreOS Meetup May 25th, 2016

1. Xiang Li
xiang.li@coreos.com
etcd - mission-critical key-value store
CoreOS Meetup - May 25th

2. Uncoordinated Upgrades

3. ... ... ...
... ... ...
Unavailable
Uncoordinated Upgrades

4. Motivation
CoreOS cluster reboot lock
- Decrement a semaphore key atomically
- Reboot and wait...
- After reboot increment the semaphore key

5. 3
CoreOS updates coordination

6. CoreOS updates coordination
3

7. ...
CoreOS updates coordination
2

8. ... ... ...
CoreOS updates coordination
0

9. ... ... ...
CoreOS updates coordination
0

10. ... ...
CoreOS updates coordination
0

11. ... ...
CoreOS updates coordination
0

12. ... ...
CoreOS updates coordination
1

13. ... ...
...
CoreOS updates coordination
0

14. CoreOS updates coordination

15. Store Application Configuration
config

16. config
Start / RestartStart / Restart
Store Application Configuration

17. config
Update
Store Application Configuration

18. config
Unavailable
Store Application Configuration

19. Requirements
Strong Consistency
- mutual exclusive at any time for locking purpose
Highly Available
- resilient to single points of failure & network partitions
Watchable
- push configuration updates to application

20. Requirements
CAP
- We want CP
- We want something like Paxos

21. Common problem
GFS
Paxos
Big Table
Spanner
CFS
Chubby
Google - “All” infrastructure relies on Paxos

22. Common problem
Amazon - Replicated log powers ec2
Microsoft - Boxwood powers storage
infrastructure
Hadoop - ZooKeeper is the heart of the ecosystem

23. ETCD KEY VALUE STORE
Fully Replicated, Highly Available,
Consistent

24. PUT(foo, bar), GET(foo), DELETE(foo)
Watch(foo)
CAS(foo, bar, bar1)
Key-value Operations

25. DEMO
play.etcd.io

26. Runtime Reconfiguration
Point-in-time Backup
Extensive Metrics
etcd Operationality

27. ETCD v3
Successor of etcd v2

28. ETCD v3
Better Performance

29. ETCD v3
More Efficient APIs

30. Multi-Version
Put(foo, bar)
Put(foo, bar1)
Put(foo, bar2)
Get(foo) -> bar2

31. Multi-Version
Put(foo, bar)
Put(foo, bar1)
Put(foo, bar2)
Get(foo, 1) -> bar

32. Tx.If(
Compare(Value("foo"), ">", "bar"),
Compare(Version("foo"), "=", 2),
...
).Then(
Put("ok","true")...
).Else(
Put("ok","false")...
).Commit()
Mini-Transactions

33. l = CreateLease(15 * second)
Put(foo, bar, l)
l.KeepAlive()
l.Revoke()
Leases

34. w = Watch(foo)
for {
r = w.Recv()
print (r.Event) // PUT
print (r.KV) // foo,bar
}
Streaming Watch

35. ETCD v2
machine coordination -> Thousands

36. ETCD v3
app/container coordination -> Millions

37. Performance 1K keys

38. Performance
Snapshot caused
performance degradation
etcd2 - 600K keys

39. Performance etcd2 - 600K keys
Snapshot triggered
elections

40. ZooKeeper Performance
Non-blocking full snapshot
Efficient memory management

41. Performance ZooKeeper default

42. Performance
Snapshot triggered
election
ZooKeeper default

43. Performance
Snapshot
ZooKeeper default

44. Performance
GC
ZooKeeper snapshot disabled

45. Reliable Performance
- Similar to ZooKeeper with snapshot disabled
- Incremental snapshot
- No Garbage Collection Pauses
- Off-heap storage

46. Performance etcd3 /ZooKeeper snapshot disabled

47. Performance etcd3 /ZooKeeper snapshot disabled

48. Memory
10GB
2.4GB
0.8GB
512MB data - 2M 256B keys

49. Reliability
99% at small scale is easy
- Failure is infrequent and human manageable
99% at large scale is not enough
- Not manageable by humans
99.99% at large scale
- Reliable systems at bottom layer

50. HOW DO WE ACHIEVE RELIABILITY
WAL, Snapshots, Testing

51. Write Ahead Log
Append only
- Simple is good
Rolling CRC protected
- Storage & OSes can be unreliable

52. Snapshots
Torturing DBs for Fun and Profit (OSDI2014)
- The simpler database is safer
- LMDB was the winner
Boltdb an append only B+Tree
- A simpler LMDB written in Go

53. Testing Clusters Failure
Inject failures into running clusters
White box runtime checking
- Hash state of the system
- Progress of the system

54. Testing Cluster Health with Failures
Issue lock operations across cluster
Ensure the correctness of client library

55. TESTING CLUSTER
dash.etcd.io

56. etcd/raft Reliability
Designed for testability and flexibility
Used by large scale db systems and others
- Cockroachdb, TiKV, Dgraph

57. etcd Reliability
Do it Really Well

58. ETCD v3.0 BETA
Efficient and Scalable

59. BETA AVAILABLE TODAY
github.com/coreos/etcd

60. FUTURE WORK
Proxy, Caching, Watch Coalescing,
Secondary Index

61. Xiang Li
xiang.li@coreos.com
etcd - mission-critical key-value store
Thank you!