Hw09 Hadoop Based Data Mining Platform For The Telecom Industry

Parallel Data Mining Platform in Telecom Industry -- Big Cloud based Parallel Data Mining Platform Friday, Oct 2, 2009 NYC Research Institute of China Mobile Communication Corporation Feng Cao

Outline ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Large scale data in China Mobile Communication Corporation (CMCC) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Large Scale Data Applications and current solution ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],The Requirements Current solution Clemetine Enterprise Miner Intelligent Miner

What’s BASS ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Challenges and limitations of BASS ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

What is the BC-PDM ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

BC-PDM Architecture ,[object Object],[object Object],[object Object],[object Object],DE DT ,[object Object],[object Object],[object Object],Data mining App

Features of BC-PDM (I) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Features of BC-PDM(II) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Case I – Mapreduce based ETL ,[object Object],[object Object],Input Data Set the targe fields to Key, other fields to Value Reduce the same key, read from the value list and write once Output Data Define the target fields (one or all) Set the targe fields to Key, other fields to Value Set the targe fields to Key, other fields to Value MapTasker 1 MapTasker 2 MapTasker n ReduceTasker 1 Reduce the same key, read from the value list and write once ReduceTasker m

关键技术方案 - 并行 ETL- 冗余删除功能冗余删除操作实现了针对所有数据样本中完全相同的两条或多条记录进行删除，只保留相同记录中的一条记录。指标 1 ）实现数据表冗余删除的并行化 2 ）正确性与串行结果完全一致 3 ）加速比接近线性， TB 级处理时间千秒级参考方案数据库中的串行冗余删除我们的方案 1 ）通过 map 对待处理数据进行分块处理，每个数据块对应一个处理节点； map 中输入的 key 为默认值——每行数据的偏移量， value 为该行数据的文本形式，以此方式实现在每块中依次读入每行数据； map 任务输出中间 <key,value> 对，其中， key 从整行数据文本， value 为空文本； 2 ）对具有相同 key 值的数据由 reduce 输出： key 为整行数据， value 值为空，即可实现同样的数据记录仅保留一条数据记录；将 reduce 输出结果存储到分布式文件系统。

Case II – Mapreduce based DM Algorithm ,[object Object],[object Object],Input Data Set the frequent k-1 length item sets to Key, appear times to Value Reduce the same key, read from the value list and sum Output Data Set the frequent k-1 length item sets to Key, appear times to Value Set the frequent k-1 length item sets to Key, appear times to Value MapTasker 1 MapTasker 2 MapTasker n ReduceTasker 1 Reduce the same key, read from the value list and sum ReduceTasker m Output rules satisfy both minimum support value and minimum confidence value

关键技术方案 - 并行关联规则算法 -PApriori 功能 Apriori 是基于统计频繁项集的策略发现属性间的关联关系指标 1 ）实现查找频繁 k 项集的并行化 2 ）正确性与串行结果完全一致 3 ）扩展性优良， TB 级处理时间千秒级参考方案串行 Apriori 算法我们的方案 1 ）采用 Map/Reduce 机制逐层迭代方法来发现频繁项集，在查找每个频繁 k 项集时进行并行化； 2 ）将数据转换为中间 Key/Value 对输出： key 为候选 k 项集， value 为项集计数；将各处理节点输出的数据进行合并处理，满足最小支持度阈值的作为频繁 k 项集； 3 ）由频集产生强关联规则，输出满足最小可信度阈值的关联规则。

Experiment Environment Software Hardware ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Evaluation of BC-PDM(Phase I) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Conclusions ,[object Object],[object Object],[object Object],[object Object]

Future works ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

People(cloud computing team from CMRI) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Collaborations are welcome! Thanks and Questions? fengcao@chinamobile.comluozhiguo@chinamobile.com [email_address] Cloud Computing E-Channel (in Chinese) http://labs.chinamobile.com/cloud

Hw09 Hadoop Based Data Mining Platform For The Telecom Industry

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (8)

Similaire à Hw09 Hadoop Based Data Mining Platform For The Telecom Industry

Similaire à Hw09 Hadoop Based Data Mining Platform For The Telecom Industry (20)

Plus de Cloudera, Inc.

Plus de Cloudera, Inc. (20)

Dernier

Dernier (20)

Hw09 Hadoop Based Data Mining Platform For The Telecom Industry

Notes de l'éditeur