At StampedeCon 2012 in St. Louis, Pritam Damania presents: Reliable backup and recovery is one of the main requirements for any enterprise grade application. HBase has been very well embraced by enterprises needing random, real-time read/write access with huge volumes of data and ease of scalability. As such, they are looking for backup solutions that are reliable, easy to use, and can co-exist with existing infrastructure. HBase comes with several backup options but there is a clear need to improve the native export mechanisms. This talk will cover various options that are available out of the box, their drawbacks and what various companies are doing to make backup and recovery efficient. In particular it will cover what Facebook has done to improve performance of backup and recovery process with minimal impact to production cluster.
5. What is HDFS ?
▪ Distributed FileSystem
▪ Runs on top of commodity hardware
▪ Scale to Petabytes of data
▪ Tolerates machine failures
6. HDFS Data Model
▪ Data is logically organized into files and directories
▪ Files are divided into uniform-sized blocks
▪ Blocks
are distributed across the nodes of the cluster and are replicated
to handle hardware failure
▪ HDFS keeps checksums of data for corruption detection and recovery
▪ HDFS exposes block placement so that computation can be migrated to
data
10. HBase in a nutshell
§ distributed, large-scale data store
§ can host very large tables, billions of rows x millions of columns
§ efficient at random reads/writes
§ open source project modeled after Google’s BigTable
11. HBase Data Model
• An HBase table is:
• a sparse , three-dimensional array of cells, indexed by:
RowKey, ColumnKey, Timestamp/Version
• sharded into regions along an ordered RowKey space
• Within each region:
• Data is grouped into column families
▪ Sort order within each column family:
• Row Key (asc), Column Key (asc), Timestamp (desc)
12. HBase System Overview
Database Layer
HBASE
Master Backup Master
Region Region Region ...
Server Server Server
Storage Layer Coordination Service
HDFS Zookeeper Quorum
Namenode Secondary Namenode ZK ZK ...
Peer Peer
Datanode Datanode Datanode ...
13. HBase Overview
HBASE Region Server
....
Region #2
Region #1
....
ColumnFamily #2
ColumnFamily #1 Memstore
(in memory data structure)
HFiles (in HDFS) flush
Write Ahead Log ( in HDFS)
16. Hbase Backups – The Problem
▪ Need a consistent, point in time backup
▪ Issues :
▪ Live cluster, with traffic
▪ Data in MemStore
▪ Flushes and Compations in the background
▪ Regionserver death
▪ Regions moving
17. CURRENT OPTIONS – Export Table
▪ Pros :
▪ Can export part or full table
▪ Map-Reduce job downloads data to output path provided
▪ Supports start time, end time and versions so could provide a
consistent backup
▪ Can specify which Column Families to export
▪ Cons :
▪ Only one table at a time
▪ Full scans and random reads
18. CURRENT OPTIONS - Copy Table
▪ Tool to copy existing table to a intra/inter cluster
▪ Pros :
▪ Another parallel replicated setup to switch
▪ Supports start time, end time, and versions
▪ Cluster being copied to could be in different setup
▪ Can specify which Column Families to export
▪ Cons :
▪ Keep another HBASE cluster up and ready
▪ Full scans and random reads
20. Backups V1
Log(Put
A)
Application Backup
Cluster
Log(Put
A)
Put A Dedup
HBase Verify
21. Backups V1 – Pros and Cons
▪ Pros :
▪ Simple solution
▪ Consistency in backup
▪ Point in time restore
▪ Verification of backups
▪ Cons :
▪ Requires replay of large amount of transactions
▪ Requires double writes and deduplication
22. Backups V2
Flush Region
RegionServe Get File List Mapper
r
Flush Copy
Files
HDFS
.regioninfo
23. Backups V2 – Tuning
▪ Locality based mappers
▪ Use in rack replication
▪ Increase .Trash retention for HDFS
▪ Fault tolerant
▪ Use Backups V1 for point in time
24. Backups V2 – Restore
▪ Rewrite backed up .regioninfo
▪ Move backup copy in place
▪ Add regions to .META using .regioninfo
25. Backups V2 – Pros and Cons
▪ Pros :
▪ Faster restore
▪ Backup entire data in hours
▪ Consistency in backup
▪ Point in time restore
▪ Resilient to RS death, region moves
▪ Cons :
▪ Affects production cluster
▪ Not scalable with data growth
26. Backups V2 – HDFS Improvements
▪ Overhead of copying large files
▪ Use locality of data
▪ HDFS HFiles are immutable
▪ HDFS blocks are immutable
▪ Hardlinks at block level!
27. Fast Copy workflow
Source Destination
B1 B2 ……………….. B1’ B2’ ……………………
….
FastCopy Client Add Block
Create Destination
Get Source NameNode
Copy Block
B1 B1’ B1 B1’ B1 B1’
B2 B2’ B2 B2’ B2 B2’
Date Node1 Date Node2 Date Node3
28. FastCopy – Pros and Cons
▪ Pros :
▪ Extremely fast
▪ Lots of space saving
▪ Minimal impact to production cluster
▪ Cons :
▪ NameNode not aware
▪ Hardlinks lost on datanode death
▪ Balancer not aware.
29. Operations
▪ Messages Use Case :
▪ 3 stage (same cluster, off cluster, off data center)
▪ Stage 1 : once/ day
▪ Stage 2 : once / 10 day
▪ Stage 3 : once / 10 day
▪ Retention based on capacity
35. Further Work
▪ Backup HLogs
▪ Point in time backups
▪ Namenode level Hard links
▪ Code and JIRAs :
▪ HBASE 4618
▪ HDFS code in github (https://github.com/facebook/hadoop-20)