This document summarizes different approaches to achieving high availability in HBase. It discusses HBase region replicas, asynchronous WAL replication, and timeline consistency introduced in HBase 1.1 to provide increased read availability. It also describes WanDisco's non-stop HBase implementation using Paxos consensus and Facebook's HydraBase which uses RAFT consensus, with HydraBase designating region replicas as active or witness. The document compares these approaches on attributes like consensus algorithm, read/write availability, strong consistency, and support for multi-datacenter deployments.

1. HBASEHIGHAVAILABILITY
HBASE HIGH
AVAILABILITYPAST, PRESENT AND FUTURE
JUN // 11 // 2015
SUDARSHAN KADAMBI, BLOOMBERG
DEVARAJ DAS, HORTONWORKS

2. HBASEHIGHAVAILABILITY
HBASE BASICS – QUICK RECAP
2
Region
Memstore
RegionServer
HDFS
Client
Region
MemstoreRegion
Memstore
Get/Scan/Put
Memstore

3. HBASEHIGHAVAILABILITY
3
HBASE MTTR – AN OVERVIEW

4. HBASEHIGHAVAILABILITY
BEYOND MTTR ….
30 seconds downtime (aggressive MTTR time) is still too
much
CERTAIN CLASSES OF APPS ARE WILLING TO TOLERATE
DECREASED CONSISTENCY GUARANTEES IN FAVOR OF
AVAILABILITY
» Especially for READs
INTRODUCED REGION REPLICA & TIMELINE CONSISTENCY
IN HBASE
 Community effort in the Apache jira HBASE-10070
4

5. HBASEHIGHAVAILABILITY
HBASE-10070: “REGION REPLICA”
5
FOR EVERY REGION OF THE TABLE, THERE CAN BE MORE THAN ONE REPLICA
» Each region replica is hosted by a different region server
ONE REPLICA PER REGION IS THE “PRIMARY”
» Only this can accepts WRITEs
» All reads from this region replica return the most recent data
OTHER REPLICAS, ALSO CALLED “SECONDARIES” FOLLOW THE PRIMARY
» Via Asynchronous WAL replication
» May not have received the recent data
» Serve data from the same data files (hfiles) as primary

6. HBASEHIGHAVAILABILITY
HBASE-10070: “ASYNC WAL REPL”
Client1
X=1WAL
MEM
Replica_id=0 (primary)
Replica_id=r1
Replica_id=r2
replication
replication
X=3
WAL
MEM
WAL
MEM
X=1X=1Write

7. HBASEHIGHAVAILABILITY
HBASE-10070: “ASYNC WAL REPL”
Client1
X=1
Client2
WAL
MEM
Replica_id=0 (primary)
Replica_id=r1
Replica_id=r2
replication
replication
X=3
WAL
MEM
WAL
MEM
X=1
X=1
X=1
X=1
X=1
X=1Read
X=1Read
X=1Read

8. HBASEHIGHAVAILABILITY
HBASE-10070: “ASYNC WAL REPL”
Client1
X=1WAL
MEM
Replica_id=0 (primary)
Replica_id=r1
Replica_id=r2
replication
replication
X=3
WAL
MEM
WAL
MEM
X=1
X=1
X=1
X=2Write X=2
X=2

9. HBASEHIGHAVAILABILITY
HBASE-10070: “ASYNC WAL REPL”
X=1
Client2
WAL
MEM
Replica_id=0 (primary)
Replica_id=r1
Replica_id=r2
replication
replication
X=3
WAL
MEM
WAL
MEM
X=1
X=1
X=1
Read
Read
X=1Read
X=2
X=2
X=2
X=2
X=2
X=2
Reads and Scans can be performed,
returning possibly stale data

10. HBASEHIGHAVAILABILITY
HBASE-10070: “TIMELINE CONSISTENCY”
10
PUBLIC ENUM CONSISTENCY {
STRONG,
TIMELINE
}
GET GET = NEW GET(ROW);
GET.SETCONSISTENCY(CONSISTENCY.TIMELINE);
...
RESULT RESULT = TABLE.GET(GET);
…
IF (RESULT.ISSTALE()) {
...
}

11. HBASEHIGHAVAILABILITY
THE CURRENT STATE OF HBASE-10070
• Available in HBase 1.1.0
• Provides HA reads but not writes
• Simplest to implement
• Consensus-based HA more work than
having No consensus
11

12. HBASEHIGHAVAILABILITY
EXTENSIONS TO HBASE-10070
• Standby-aware Assignment Manager
• Standby-aware DataNode placement
• Pluggable retry policies
• Pluggable consistency models
12

13. HBASEHIGHAVAILABILITY
HIGH AVAILABILITY – SOLUTION
PARAMETERS
• Where does the WAL go?
• Are Region Servers Active-Active or Active-
Standby?
• Do you use quorum?
• What consensus algorithm to achieve
quorum?
13

14. >>>>>>>>>>>>>>
WANDISCO’S NON
STOP HBASE

15. HBASEHIGHAVAILABILITY
WAN DISCO: ARCHITECTURE
• WAL on Local file system
• Multiple Region Servers host a region
• Reads and writes to a key served by any
replica
• Paxos-based Consensus
15

16. HBASEHIGHAVAILABILITY
WAN DISCO: WRITE PATH
• All replicas equal for reads and writes
• Client sends the Put to any one replica
• Chosen replica co-ordinates with peers
• Memstore and WAL updated after
consensus is achieved
16

17. HBASEHIGHAVAILABILITY
WAN DISCO: WRITE PATH
Client
WAL
MEM
Replica_id=R1
Replica_id=R2
Replica_id=R3
Networkpartition
WAL
MEM
WAL
MEM
X=2
X=2
X=2
WRITE X=2
X=2
X=2

18. HBASEHIGHAVAILABILITY
WAN DISCO: READ PATH
• Reads go to any one replica in the set
• On timeout, client does co-ordinated read
to other replicas
• Could potentially return stale data
18

19. HBASEHIGHAVAILABILITY
WAN DISCO: READ PATH
Client
WAL
MEM
Replica_id=R1
Replica_id=R2
Replica_id=R3
WAL
MEM
WAL
MEM
X=2
X=2
X=2
READ X
X=2

20. HBASEHIGHAVAILABILITY
WAN DISCO: READ PATH
Client
WAL
MEM
Replica_id=R1
Replica_id=R2
Replica_id=R3
WAL
MEM
WAL
MEM
X=2
X=2
X=2
READ X
X=2

21. HBASEHIGHAVAILABILITY
WAN DISCO: REGION SERVER FAILURE
• No downtime on reads or writes upon RS
failure
• Failed RS will not persist writes
• Failed RS might be on the read path, client
re-issues read on timeout
21

22. >>>>>>>>>>>>>>
FACEBOOK’S
HYDRABASE

23. HBASEHIGHAVAILABILITY
HYDRABASE: ARCHITECTURE
• WAL on Local file system
• Multiple Region Servers host a given region
• Reads and writes to a key served by active
leader only
• RAFT-based Consensus
23

24. HBASEHIGHAVAILABILITY
HYDRABASE: ARCHITECTURE
• Replicas are in either active or witness
mode
• Active replicas write HFiles
• Witness replicas only do WAL replication
24

25. HBASEHIGHAVAILABILITY
HYDRABASE: ARCHITECTURE
25
cc, Image Courtesy https://code.facebook.com/posts/321111638043166/hydrabase-the-evolution-of-hbase-facebook/

26. HBASEHIGHAVAILABILITY
HYDRABASE: FAILOVER
26
cc, Image Courtesy https://code.facebook.com/posts/321111638043166/hydrabase-the-evolution-of-hbase-facebook/

27. HBASEHIGHAVAILABILITY
HYDRABASE: FAILOVER
27
cc, Image Courtesy https://code.facebook.com/posts/321111638043166/hydrabase-the-evolution-of-hbase-facebook/

28. HBASEHIGHAVAILABILITY
COMPARISON MATRIX
28
INTER-
CLUSTER
REPLICATION
HBASE-10070 WAN
DISCO
HYDRABASE
CONSENSUS NO
CONSENSUS
NO
CONSENSUS
PAXOS RAFT
READ
AVAILABILITY
YES YES YES YES
WRITE
AVAILABILITY
YES NO* YES YES
STRONGLY
CONSISTENT
HA READS
NO NO NO YES

29. HBASEHIGHAVAILABILITY
COMPARISON MATRIX
29
INTER-
CLUSTER
REPLICATION
HBASE-10070 WAN DISCO HYDRABASE
STRONGLY
CONSISTENT
HA WRITES
NO NO YES YES
MULTI-
DATACENTER
YES NO* YES YES
PLUGGABLE
CONSISTENCY
MODEL
NO NO NO NO
LICENSE OPEN
SOURCE
OPEN
SOURCE, 1.1
COMMERCIAL PROPRIETARY
*

30. HBASEHIGHAVAILABILITY
ACKNOWLEDGEMENTS
• Ryan Rawson & Jagane Sundar, WanDisco
• HBase team at Facebook
30

31. HBASEHIGHAVAILABILITY
QUESTIONS?