Pattern Mining To Unknown Word Extraction (10

Pattern Mining to Chinese Unknown word Extraction 資工碩三 955202037 楊傑程 2008/10/14

Outline ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Introduction ,[object Object],[object Object],[object Object]

Introduction ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Introduction- types of unknown words ,[object Object],Types of Chinese unknown words Organization names Ex: 華碩電腦 Ex: 總經理、電腦化 Abbreviation Proper Names Ex: 中油、中大 Personal names Ex: 王小明 Derived Words Compounds Ex: 電腦桌、搜尋法 Numeric type compounds Ex: 1986 年、 19 巷

Introduction- unknown word identification ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Introduction- unknown word identification ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Introduction- applied techniques ,[object Object],[object Object],[object Object],[object Object],[object Object]

Related Works- particular methods ,[object Object],[object Object],[object Object],[object Object],[object Object]

Related Works- general methods (Rule-based) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Related Works- general methods (Machine Learning-based) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Related Works – Imbalanced Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Unknown Word Detection & Extraction ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Unknown Word Detection & Extraction Unknown Word Detection (Detection Rule Mining) Judge Judge Unknown Word Extraction (Machine Learning- Classification) 8/10 corpus + detection tags (Initial Segmentation) 8/10 corpus 1/10 corpus (Validation) 1/10 corpus (Initial Segmentation) Classification Decision 1/10 corpus + detection tags training testing Phase 1 Phase 2 Rules 1/10 corpus (Validation) Mining tool (Prowl) Model POS tagging POS tagging

Unknown Word Detection ,[object Object],[object Object],[object Object],[object Object]

Unknown word detection- Pattern Mining ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Unknown word detection- Continuity Pattern Mining ,[object Object],[object Object],[object Object],[object Object],[object Object]

Encoding ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Create Detection Rules ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],( 葡 (Na), 萄 (Na), ) ： 2 ( 葡 (Na) Y, 萄 (Na), ) ： 1 ( 葡 (Na) N, 萄 (Na) N, ) ： 1 ( 葡 (Na) Y, 萄 (Na) Y, ) ： 1 ( 葡 , 萄 , ) ： 2 ( 葡 (Na), 萄 , ) ： 2 ( 葡 , 萄 (Na), ) ： 2

Unknown Word Extraction ,[object Object],[object Object],[object Object]

Unknown Word Extraction- feature ( Pos) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Unknown Word Extraction- feature ( term_attribute) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],運動會 () 　‧ () 　四年 () 　甲班 () 　王 (?) 　姿 (?) 　分 (?) 　‧ () 　本校 () 　為 () 　響 () 　應 () ps() dot() ds() ds() ms(?) ms(?) ms(?) dot() ds() ms() ms() ms()

Data Processing- Sliding Window ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],t3 t2 t1 prefix t0 3-gram suffix t4

EX: 3-gram Model discard negative negative negative positive 運動會 () 　‧ () 　四年 () 　甲班 () 　王 (?) 　姿 (?) 　分 (?) 　‧ () 　本校 () 　為 () 　響 () 　應 () 運動會 ‧ 四年甲班王 (?) ‧ 四年甲班王 (?) 姿 (?) 四年甲班王 (?) 姿 (?) 分 (?) 甲班王 (?) 姿 (?) 分 (?) ‧ 王 (?) 姿 (?) 分 (?) ‧ 本校

Unknown Word Extraction- feature (Statistical Information) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],t3 t2 t1 prefix t0 3-gram suffix t4

Data presentation ,[object Object],[object Object],term_attribute (6) pos (55) t2 term_attribute (6) pos (55) term_attribute (6) pos (55) prefix t1 …… … statistics (7) term_attribute (6) pos (55) …… suffix

Experiments ,[object Object],[object Object]

Unknown Word Detection ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],Threshold (Accuracy) Precision Recall F-measure (our system) F-measure (AS system) 0.7 0.9324 0.4305 0.589035 0.71250 0.8 0.9008 0.5289 0.66648 0.752447 0.9 0.8343 0.7148 0.769941 0.76955 0.95 0.764 0.8288 0.795082 0.76553 0.98 0.686 0.8786 0.770446 0.744036 0.76158 0.9092 0.6552 29 0.77033 0.780085 0.787466 0.795082 F-measure 0.8995 0.8932 0.8819 0.8288 Recall 0.6736 0.6924 0.7113 0.764 Precision 19 11 7 3 Fre>=

Unknown Word Extraction ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Unknown Word Extraction ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Extraction result ,[object Object],[object Object],[object Object],[object Object],[object Object],0.627 68.2% 58.1% Total 0.614 67.1% 56.7% 2-gram 0.707 80% 63.3% 3-gram 0.426 70.3% 30.6% 4-gram F1-score Recall Precision n-gram

Ensemble Method Improvement 0.426 0.703 0.306 0.707 0.8 0.633 0.614 0.671 0.567 Censemble 0.336 0.59 0.238 0.66 0.765 0.583 0.594 0.653 0.544 Caverage 0.412 0.662 0.299 0.669 0.776 0.587 0.587 0.645 0.538 C12 0.335 0.554 0.24 0.667 0.74 0.607 0.593 0.668 0.533 C11 0.344 0.662 0.232 0.655 0.723 0.599 0.596 0.661 0.543 C10 0.321 0.635 0.215 0.645 0.715 0.587 0.598 0.657 0.548 C9 0.309 0.486 0.226 0.676 0.813 0.579 0.6 0.673 0.541 C8 0.325 0.703 0.211 0.648 0.691 0.611 0.604 0.66 0.557 C7 0.333 0.608 0.23 0.641 0.735 0.568 0.582 0.636 0.536 C6 0.299 0.554 0.205 0.644 0.779 0.549 0.603 0.66 0.555 C5 0.42 0.676 0.305 0.667 0.796 0.574 0.598 0.645 0.557 C4 0.28 0.378 0.222 0.664 0.81 0.563 0.58 0.633 0.535 C3 0.338 0.743 0.219 0.7 0.791 0.627 0.61 0.657 0.569 C2 0.315 0.419 0.252 0.649 0.808 0.542 0.572 0.64 0.518 C1 F1-Score Recall Precision F1-Score Recall Precision F1-Score Recall Precision 4-gram 3-gram 2-gram 分類模型

Experiment- One phase ,[object Object],[object Object],0.627 68.2% 58.1% Two Phases 0.52 71.4% 40.8% One Phase F-score Recall Precision Classification Performance

Conclusions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Pattern Mining To Unknown Word Extraction (10

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (10)

Similaire à Pattern Mining To Unknown Word Extraction (10

Similaire à Pattern Mining To Unknown Word Extraction (10 (20)

Dernier

Dernier (20)

Pattern Mining To Unknown Word Extraction (10