Hw09 Hadoop Development At Facebook Hive And Hdfs
•Télécharger en tant que PPT, PDF•
10 j'aime•6,291 vues
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Hw09 Hadoop Development At Facebook Hive And Hdfs
Similaire à Hw09 Hadoop Development At Facebook Hive And Hdfs (20)
Plus de Cloudera, Inc.
Plus de Cloudera, Inc. (20)
Dernier
Dernier (20)
Hw09 Hadoop Development At Facebook Hive And Hdfs
- 1. Hadoop and Hive Development at Facebook Dhruba Borthakur Zheng Shao {dhruba, zshao}@facebook.com Presented at Hadoop World, New York October 2, 2009
- 6. Data Flow into Hadoop Cloud Web Servers Scribe MidTier Network Storage and Servers Hadoop Hive Warehouse Oracle RAC MySQL
- 7. Hadoop Scribe: Avoid Costly Filers Web Servers Scribe Writers Realtime Hadoop Cluster Hadoop Hive Warehouse Oracle RAC MySQL Scribe MidTier http://hadoopblog.blogspot.com/2009/06/hdfs-scribe-integration.html
- 9. Archival: Move old data to cheap storage Cheap NAS Hadoop Archival Cluster Hadoop Archive Node NFS Hive Query Hadoop Warehouse http://issues.apache.org/jira/browse/HDFS-220
- 13. Hive Architecture HDFS Map Reduce Web UI + Hive CLI + JDBC/ODBC Browse, Query, DDL Hive QL Parser Planner Optimizer Execution SerDe CSV Thrift Regex UDF/UDAF substr sum average FileFormats TextFile SequenceFile RCFile User-defined Map-reduce Scripts
Notes de l'éditeur
- 1GB connectivity within a rack, 100MB across racks? Are all disks 7200 SATA?
- This is the architecture of our backend data warehouing system. This system provides important information on the usage of our website, including but not limited to the number page views of each page, the number of active users in each country, etc. We generate 4TB of compressed log data every day. All these data are stored and processed by the hadoop cluster which consists of over 600 machines. The summary of the log data is then copied to Oracle and MySQL databases, to make sure it is easy for people to access.
- The Network attached storage is replaced by a hadoop cluster. This hadoop cluster has real-time characteristics and is configured as failfast. ThebBenefits are no storage fragmentation and easier to manage. Overlapped clusters for HA.
- The Network attached storage is replaced by a hadoop cluster. This hadoop cluster has real-time characteristics and is configured as failfast. ThebBenefits are no storage fragmentation and easier to manage.