4. 中国移动大云技术架构
CMCC IT Supporting Systems Internet App IDC 。。。
Application
• IT Supporting System of CMCC
• IDC and Internet Applications
Cloud Storage Data Mining Search Engine
System Management :CloudMaster
System Management :CloudMaster
BC-NAS BC-PDM BC-SE Enabler
• BC-PDM: Cloud base Data Mining
• BC-NAS: File and Object Storage with
Sturcture Data Storage
web Interface and REST API
HugeTable
• BC-SE: Search Engine
CloudSecurity
CloudSecurity
Hadoop MapReduce
with CMRI Extension Platform
• MapReduce & HDFS: based on Hadoop
Object Storage Distributed Filesystem and with some extensions by CMRI
based on
oNest
Hadoop HDFS :
• HugeTable:Structure Storage with
SQL interface
Elastic Computing: BC-EC :
•oNest:Object Storage for Web Apps.
Linux, Xen/KVM
• CloudMaster: System Management
Resource
• PC Server and SATA Disk based
• BC-EC: IaaS based on OpenNebula
• Based on FOSS: Linux, KVM, Xen
5. 大云与Hadoop
CMCC IT Supporting Systems Internet App IDC 。。。
Development based on Hadoop
Parallel ETL and Data Mining
Cloud Storage Data Mining Search Engine based on MapReduce
System Management :
System Management :CloudMaster
BC-NAS BC-PDM BC-SE Search Engine based on
MapReduce
HugeTable (Structure data
Sturcture Data Storage
HugeTable
storage for data warehouse)
based on Hive, HBase & MR
CloudSecurity
CloudSecurity
Hadoop MapReduce
with CMRI Extension Development extending Hadoop
Object Storage Distributed Filesystem Volume Management of
based on DataNode in HDFS
oNest
Hadoop HDFS
NameNode Cluster for HDFS
Elastic Computing: BC-EC Multi-queue scheduler with
Linux, Xen/KVM queue priority enhancement
External facilities for Hadoop
Test tools for Hadoop HDFS
Inside job performance
evaluation tool
MapReduce Job Submission
Web Interface
6. Development on Hadoop in CMRI
Contributing to Mainline
Online Volume Management of DataNode (by Wang Xu etc., HDFS-
1362)
Off-Tree and Opened
NameNode Cluster for HA (by Wang Xu, hosted in GitHub)
Off-Tree and not Maintained
hdfs-fuse (by Zhao Peng, hosted in Google Code)
Multi-queue scheduler with queue priority enhancement (by Guo Leitao)
External Facilities
hadoop-test (by Wang Xu, hosted in Google Code)
MapReduce Job Submission Web Interface and Inside job performance
evaluation tool (by Guo Leitao, etc.)
Bug Fixes
7. DataNode Online Volume Management
http://github.com/gnawux/hadoop-cmri
https://issues.apache.org/jira/browse/HDFS-1362
Current State:
Disk failure Node
Decommission
Online Volume
Management:
Online removal of
failed disk
Migrate Data in faild
volume if still
readable
Change Disk online