Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

図でわかるHDFS Erasure Coding

4 316 vues

Publié le

Illustration of HDFS Erasure Coding

Publié dans : Données & analyses
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

図でわかるHDFS Erasure Coding

  1. 1. 図でわかる
 HDFS Erasure Coding Kai Sasaki Treasure Data Inc.
  2. 2. Who am I 佐々木 海(Kai Sasaki) Software Engineer at Treasure Data Inc.
 http://www.treasuredata.com Hadoop, Spark, DL4J
  3. 3. Agenda • Erasure Coding • Under the Namespace • Writing Side • Reading Side
  4. 4. Erasure Coding
  5. 5. Replication Block HDFS
  6. 6. Replication Block Block Block Block HDFS
  7. 7. Replication Block Block Block Block HDFS
  8. 8. Replication Block Block Block Block HDFS Capacity Overhead 
 x3
  9. 9. Replication Block Block Block Block HDFS Redundancy 2 Capacity Overhead 
 x3
  10. 10. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block RS-6-3
  11. 11. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9
  12. 12. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9
  13. 13. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9
  14. 14. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block6 out of 9
  15. 15. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block BlockRedundancy 3
  16. 16. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block Capacity Overhead 
 x1.5 Redundancy 3
  17. 17. Erasure Coding Block Block HDFS Block Block Block Block Block Block Block Block BlockGroup
  18. 18. Under the Namespace
  19. 19. INode and BlockInfo BlockInfo INode
  20. 20. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo…
  21. 21. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo… … Block Block Block
  22. 22. INode and BlockInfo BlockInfo INode BlockInfo BlockInfo… … Block Block Block BlockGroup
  23. 23. BlockInfo … Block Block Block BlockGroup
  24. 24. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup
  25. 25. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup index GroupId 4bit 60bit
  26. 26. long BlockId 0 64 BlockInfo … Block Block Block BlockGroup index GroupId 4bit 60bit index 0 index 2 index 1 Saving memory
  27. 27. Writing Side
  28. 28. Data 0
  29. 29. Data64KB 0
  30. 30. Data BlockGroup 64KB 0
  31. 31. Data BlockGroup 0
  32. 32. Data BlockGroup 0
  33. 33. Data BlockGroup 0
  34. 34. Data BlockGroup 0
  35. 35. Data BlockGroup 0
  36. 36. Data BlockGroup 0
  37. 37. Data BlockGroup 0
  38. 38. Data BlockGroup 0 Data Block
  39. 39. Data BlockGroup 0 Data Block Parity Block
  40. 40. Data BlockGroup 0 Data Block Parity Block Stripe
  41. 41. Data BlockGroup 0
  42. 42. Data BlockGroup 0
  43. 43. Data BlockGroup 0
  44. 44. Data BlockGroup 0
  45. 45. Data BlockGroup 0
  46. 46. Data BlockGroup 0
  47. 47. Data BlockGroup 0 0
  48. 48. Data BlockGroup 0 0
  49. 49. Data BlockGroup 0 0 Saving diskspace usage
  50. 50. Reading Side
  51. 51. BlockGroup 0 0
  52. 52. BlockGroup 0 0 200kb 500kb
  53. 53. BlockGroup 0 0 200kb 500kb
  54. 54. BlockGroup 0 0 200kb 500kb
  55. 55. BlockGroup 0 0 200kb 500kb
  56. 56. BlockGroup 0 0 200kb 500kb
  57. 57. BlockGroup 0 0 200kb 500kb
  58. 58. BlockGroup 0 0 200kb 500kb
  59. 59. BlockGroup 0 0 200kb 500kb Saving reading time
  60. 60. まとめ • Namespace -> Saving memory
 BlockInfoStriped, BlockIdManager • Writing Side -> Saving diskspace usage
 INodeFile • Reading Side -> Saving reading time
 DFSStripedInputStream
  61. 61. ありがとうございました

×