27. NonStop HBase – Making HBase
Continuously Available for Enterprise
Deployment
Dr. Konstantin Boudnik
WANdisco
28. Non-Stop HBase
Making HBase Continuously Available for
Enterprise Deployment
Konstantin Boudnik – Director, Advanced Technologies, WANdisco
Brett Rudenstein – Senior Product Manager, WANdisco
29. WANdisco: continuous availability company
WANdisco := Wide Area Network Distributed Computing
We solve availability problems for enterprises.. If you can’t afford 99.999% - we’ll help
Publicly trading at London Stock Exchange since mid-2012 (LSE:WAND)
Apache Software Foundation sponsor; actively contributing to Hadoop, SVN, and others
US patented active-active replication technology
Located on three continents
Enterprise ready, high availability software solutions that enable globally distributed
organizations to meet today’s data challenges of secure storage, scalability and
availability
Subversion, Git, Hadoop HDFS, HBase at 200+ customer sites
32. HA is (mostly) a glorified backup
Redundancy of critical elements
- Standby servers
- Backup network links
- Off-site copies of critical data
- RAID mirroring
Baseline:
- Create and synchronize replicas
- Clients switching in case of failure
- Extra hardware allaying idly spinning “just in case”
35. WANdisco Active-Active Architecture
/ page 35
100% Uptime with WANdisco’s patented replication technology
- Zero downtime / zero data loss
- Enables maintenance without downtime
Automatic recovery of failed servers; Automatic rebalancing as workload increases
HDFS Data
36. Multi-threaded Server Software:
Multiple threads processing client requests in a loop
Server
Process
make change to state (db)
get client request e.g.
hbase put
send return value to
client
OP OP OP OP
OP
OP
OP OPOP OP
OP
OP
thread
1
thread
3
thread
2
thread
1
thread
2
thread
3
acquire
lock
release
lock
38. Using a TCP Connection to send data to three
replicated servers (Load Balancer)
serve
r3
Server
Process
OP OP
serve
r2
Server
Process
OP OP OP OP
serve
r1
Server
Process
OP OP OP OP
Client
OP OP OP OP
Load
Balancer
Load
Balancer
39. HBase WAL replication
State Machine (HRegion contents, HMaster metadata, etc.) is modified first
Modification Log (HBase WAL) is sent to a Highly Available shared storage
Standby Server(s) read edits log and serve as warm standby servers, ready to take
over should the active server fail
41. HBase WAL tailing, WAL Snapshots etc.
Only one active region server is possible
Failover takes time
Failover is error prone
RegionServer failover isn’t seamless for clients
43. Three replicated servers
serve
r3
Server
Process
OP OP OP OP
Distributed
Coordination
Engine
serve
r2
Server
Process
Distributed
Coordination
Engine
OP OP OP OP
serve
r1
Server
Process
OP OP OP OP
Distributed
Coordination
Engine Paxos
DConE
Clie
nt
Clie
nt
Clie
nt
Clie
nt
Clie
nt
Paxos
DConE
OP
OPOP
OP
45. HBase Single Points of Failure
Single HBase Master
- Service interruption after Master failure
Hbase client
- Client session doesn’t failover after a RegionServer failure
HBase Region Server: downtime
- 30 secs ≥ MMTR ≤ 200 secs
Region major compaction (not a failure, but…)
- (un)-scheduled downtime of a region for compaction
52. HBase RegionServer replication using
WANdisco DConE
Shared nothing architecture
HFiles, WALs etc. are not shared
Replica count is tuned
Snapshots of HFiles do not need to be created
Messy details of WAL tailing are not necessary:
- WAL might not be needed at all (!)
Not an eventual consistency model
Does not serve up stale data