HBase Snapshots
•Télécharger en tant que PPTX, PDF•
7 j'aime•2,381 vues
Overview of the upcoming snapshot feature in HBase.
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à HBase Snapshots
Similaire à HBase Snapshots (20)
Dernier
Dernier (20)
HBase Snapshots
- 1. HBase Snapshots HBase User Group Meetup 10/29/12 Jesse Yates
- 2. So you wanna…. • Prevent data loss • Recover to a point in time • Backup your data • Sandbox copy of data
- 3. Problem!
- 4. a BIG Problem… • Petabytes of data • 100’s of servers • At a single point in time • Millions of writes per-second
- 5. Solution!
- 6. Solutions!
- 8. Built-in • Export – MapReduce job against HBase API – Output to single seqeunce file • Copy Table – MapReduce job against HBase API – Output to another table Yay • Simple • Heavily tested • Can do point-in-time Boo • Slow • High impact for running cluster
- 10. Replication • Export all changes by tailing WAL YAY • Simple • Gets all edits • Minimal impact on running cluster Boo • Turn on from beginning • Can’t turn it off and catch up • No built-in point-in-time • Still need ETL process to get multiple copies
- 11. (Facebook) Solution!1 Mozilla did something similar2 1. issues.apache.org/jira/browse/HBASE-5509 2. github.com/mozilla-metrics/akela/blob/master/src/main/java/com/mozilla/hadoop/Backup.java
- 12. Facebook Backup • Copy existing hfiles, hlogs Yay • Through HDFS – Doesn’t impact running cluster • Fast – distcp is 100% faster than M/R through HBase Boo • Not widely used • Requires Hardlinks • Recovery requires WAL replay • Point-in-time needs filter
- 13. Backup through the ages Export Copy Table Replication HBase HBASE-50 HDFS Facebook
- 14. Maybethis is harder than we thought…
- 16. We did some work…
- 17. Hardlink workarounds • HBASE-5547 – Move deleted hfiles to .archive directory • HBASE-6610 – FileLink: equivalent to Windows link files Enough to get started….
- 18. Difficulties • Coordinating many servers • Minimizing unavailability • Minimize time to restore • Gotta’ be Fast
- 20. Snapshots • Fast - zero-copy of files • Point-in-time semantics – Part of how its built • Built-in recovery – Make a table from a snapshot • SLA enforcement – Guaranteed max unavailability
- 21. Snapshots?
- 22. We’ve got a couple of those…
- 23. Snapshot Types • Offline – Table is already disabled • Globally consistent – Consistent across all servers • Timestamp consistent – Point-in-time according to each server
- 24. Offline Snapshots • Table is already disabled • Requires minimal log replay – Especially if table is cleanly disabled • State of the table when disabled • Don’t need to worry about changing state YAY • Fast! • Simple!
- 25. But I can’t take my table offline!
- 26. Globally Consistent Snapshots • All regions block writes until everyone agrees to snapshot – Two-phase commit-ish • Time-bound to prevent infinite blocking – Unavailability SLA maintained per region • No Flushing – its fast!
- 27. What could possibly go wrong?
- 28. Cross-Server Consistency Problems • General distributed coordination problems – Block writes while waiting for all regions – Limited by slowest region – servers = P(failure) • Stronger guarantees than currently in HBase • Requires WAL replay to restore table
- 29. I don’t need all that, what else do you have?
- 30. Timestamp Consistent Snapshots • All writes up to a TS are in the snapshot • Leverages existing flush functionality • Doesn’t block writes • No WAL replay on recovery
- 32. Put/Get/Delete/Mutate/etc. MemStore Timestamp in snapshot? Yes No Snapshot Store Future Store
- 33. I’ve got a snapshot, now what?
- 34. Recovery • Export snapshot – Send snapshot to another cluster • Clone snapshot – Create new table from snapshot • Restore table – Rollback table to specific state
- 35. Export Snapshot • Copy a full snapshot to another cluster – All required HFiles/Hlogs – Lots of options • Fancy dist-cp – Fast! – Minimal impact on running cluster
- 36. Clone Table • New table from snapshot • Create multiple tables from same snapshot • Exact replica at the point-in-time • Full Read/Write on new table
- 37. Restore • Replace existing table with snapshot • Snapshots current table, just in case • Minimal overhead – Handles creating/deleting regions – Fixes META for you
- 38. Whew, that’s a lot!
- 40. Goodies • Full support in shell • Distributed Coordination Framework • ‘Ragged Backup’ added along the way • Coming in next CDH • Backport to 0.94?
- 41. Special thanks! • MatteoBertozzi – All the recovery code – Shell support • Jon Hsieh – Distributed Two-Phase Commit refactor • All our reviewers… – Stack, Ted Yu, Jon Hsieh, Matteo
- 42. Thanks! Questions? Jesse Yates @jesse_yates jesse.k.yates@gmail.com
Notes de l'éditeur
- Data flying around,HBase is just chugging along. Your adding servers weekly – daily? – to handle the excess capacity; life is good. But wait, one of your DBAs fat fingers a command a deletes a table, a column family, the database. Or maybe your dev’s want to test out some new features – not on my production server!Or a customer makes a mistake and wants to get back to last Tuesday at 6PM.
- HBase has been around for a few years and well, these aren’t exactly new problems.
- OK, if you’ve thought about this problem for at least 5 minutes, you’ve probably seen these before. You probably are even running them them already
- Ok, we can do better…
- Just get a list of all the hfiles/hlogs and copy them over. Use hardlinks to ensure that we have the same state for the tableThis is getting better – we aren’t directly impacting the cluster (except for bandwidth).
- General trend down the stack – more knowledge of individual files, layout in HDFS, low-level functionality. Also trending towards a minimal impact on the running cluster – only take the hit on the wire, not through the HBase layer.HBASE-50:Internalhardlinks using reference counting in META, massive patch including restore, offline and online snapshots. WAY too much to review
- And for a few years people we really sad and made do with existing tooling. We are starting to run HBase in some large companies though and have stringent data requirements
- Story-ize the problem
- Focus on TADA of the snapshots
- Imagine you have 1000 servers, each with in memory state. How would you save it? How would you save it fast? Any problems?
- Example for stronger guaranettes than hbase - Currently, we only support transactions on a single row on a single server. This gives you a semi-omniscent view over all servers hosting a table – full cross server consensus over multiple rows. WAY more than HBase gives you now.
- Guarantee that all writes are filtered on a timestamp, flushing on the regionserver so all the information in the snapshot is present entirely in HFiles – NO WAL REPLAY!
- http://www.flickr.com/photos/69382656@N04/6744068967/in/photostream