SlideShare une entreprise Scribd logo
1  sur  85
Télécharger pour lire hors ligne
Introduction to Semi-
                        Supervised Learning



Thursday, January 21, 2010
Outline
                    •
                    •
                             •   Self Traininga
                             •   Multiview Algorithms
                             •   Generative Models
                             •   S3VMs(TSVMs)
                             •   Graph-Based Algorithms
                    •
                    •
Thursday, January 21, 2010
?


Thursday, January 21, 2010
•
                             •
                    •                    costly...
                             •
                                 •
                                 •   :

Thursday, January 21, 2010
•

                    •        :


                                 ? => SSL



Thursday, January 21, 2010
?


Thursday, January 21, 2010
•
                    •




                             http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, January 21, 2010
SSL

                    • Self Training
                    • Multiview Algorithms
                    • Generative Models
                    • S3VMs(TSVMs)
                    • Graph-Based Algorithms

Thursday, January 21, 2010
Self Training



Thursday, January 21, 2010
Self Training

                    •                    :


                    •                :             p(y=1|x)   0.99
                                               1
                                 !

                             •

Thursday, January 21, 2010
Self Training

                    •
                             f

                    •                f


                    •(           )


                    •
Thursday, January 21, 2010
Self Training
                    •
                             •         !

                             •
                    •
                             •
                             •
                             •
Thursday, January 21, 2010
Multiview Algorithms



Thursday, January 21, 2010
Co-training

                    •             Self Training

                             =>
                    •                              ?
                    •


Thursday, January 21, 2010
: Web




                                     http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, January 21, 2010
•
                             •      )
                                                  (BOF


                    •
                    •        Co-training   idea

                             •
                             •

Thursday, January 21, 2010
Co-training

                    •
                    •

                    •


Thursday, January 21, 2010
Co-training




                               http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, January 21, 2010
Co-training
                    •
                             •
                             •   Self Training
                    •
                             •

                             •
Thursday, January 21, 2010
Co-training
                    • Co-EM
                     •                     EM

                    • Fake feature split
                     •
                    • Multiview
                     •
                     •          =>
Thursday, January 21, 2010
Generative Models



Thursday, January 21, 2010
Generative Models

                    •


                    •


                                      http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, January 21, 2010
Generative Models
                    •


                    •

                    •

                                      http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, January 21, 2010
Generative Models


                    •




                                      http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, January 21, 2010
Generative Models

                    •

                    •



                                      http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, January 21, 2010
Generative Models


                    •            :


                    •                (   )


                             •



Thursday, January 21, 2010
Generative Models
                    •
                             •
                             •   EM

                    •
                             •
                             •   EM

                    •
                             •
                             •   Baum-Welch algorithm
Thursday, January 21, 2010
•
                    •
                    •        : PRML
                                      => EM   MLE




Thursday, January 21, 2010
•                           (
                                   )

                    •                  y            2


                    •                      EM
                             MLE




Thursday, January 21, 2010
E-step and M-step




                                      http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, January 21, 2010
Generative Models
                    •
                             •
                             •
                    •
                             •
                             •   EM   (   )

                             •
                                              =>   page



Thursday, January 21, 2010
Generative Models




                                      http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf



Thursday, January 21, 2010
•

                    •
                             λ(λ < 1)




Thursday, January 21, 2010
S3VMs(TSVMs)



Thursday, January 21, 2010
TSVMs
                    •
                             (                    )




                                 http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf

Thursday, January 21, 2010
(RKHS)

                    • SVM
                    •
                                 =>
                             (RKHS:reproducing kernel Hilbert space)
                    •

Thursday, January 21, 2010
k



                                     k


                             f




Thursday, January 21, 2010
•
                             •
                    •                (
                                 )
                    •

Thursday, January 21, 2010
•
                             (F        )

                    •                          x
                                           F       (F
                                  !)



Thursday, January 21, 2010
•f         F



                    •
                    •              f               f(   x
                                       )   => f
                             => F                 =>
                                                            !!


Thursday, January 21, 2010
Thursday, January 21, 2010
SVM


                                   RKHS   !




             hw       RKHS




Thursday, January 21, 2010
hinge loss




                                               hinge loss                          !!




                                          http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf

Thursday, January 21, 2010
•
                 •                         ?

                       •     f(x)>1 or f(x)<-1


                       •

                                                 !
                     =>
                                                     http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf

Thursday, January 21, 2010
• Joachims(1999)
                    •                                branch-
                             and-bound search(   )



                             • 10000

Thursday, January 21, 2010
S3VM

                    •

                             •
                    •
                                   http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf




Thursday, January 21, 2010
CCCP
                        (Concave-Convex procedure)
                    •        Yuille (2003)

                    •
                    •                        )
                                                                (


                             •   Difference of Convex (DC   )


                    •        Update



Thursday, January 21, 2010
CCCP
                        (Concave-Convex procedure)
                    • update
                     •
                             •   (update   )



                    •                            !!

Thursday, January 21, 2010
TSVMs + CCCP
                    • CCCP           TSVMs

                    •
                    • Iteration      2




                                    http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
                                    http://www.stat.umn.edu/~xshen/paper/tsvm.pdf

Thursday, January 21, 2010
L. Wang (2007)
                    •                                           SVM
                                                         TSMVs CCCP
                             TSMVs




             http://www.stat.umn.edu/~xshen/paper/tsvm.pdf
Thursday, January 21, 2010
Graph-Based
                              Algorithms


Thursday, January 21, 2010
:
                    •
                    •




                                 http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, January 21, 2010
•                     !!
                             •
                             •   etc




                                       http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, January 21, 2010
•        sparseness




                             http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf
Thursday, January 21, 2010
http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf

                             https://www.aaai.org/Papers/ICML/2003/ICML03-118.pdf
Thursday, January 21, 2010
Graph-based SSL
                    •                :                   edge           node


                    •                                           :

                    •                (instance                      )

                             •   kNN :                   k              1
                                 0 =>              sparse


                             •                       :

                                 •               dense



Thursday, January 21, 2010
Graph-based SSL



                    • mincut
                    • harmonic
                    • manifold regularization


Thursday, January 21, 2010
•
                    •        0   1
                                     fix



                    •

Thursday, January 21, 2010
•                                           =>
                             •   sink
                                            source


                             •          :




                                                     http://john.blitzer.com/tutorial/ssl_tutorial.pdf
Thursday, January 21, 2010
source                          sink




              : source       ∞   : sink      -∞



Thursday, January 21, 2010
harmonic => Laplacian
                    •                      0-1           y

                    • Zhu        (2003a)

                                       Goldberg (2006)


                             •

Thursday, January 21, 2010
harmonic => Laplacian
                                         (SVM   )




                 (
                             L
            a U              b       f    ...
                             )   →                  !!


Thursday, January 21, 2010
http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf
Thursday, January 21, 2010
y_i
         hat{y}_i




                             http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf
Thursday, January 21, 2010
Laplacian


                    •f
                                    0
                             ! =>




Thursday, January 21, 2010
Goldberg (2006)

                    •        SVMR Metric labeling


                    •                         SSL


                    •


                                            http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf


Thursday, January 21, 2010
Manifold regularization
                    • Harmonic
                     •                     ...
                    •
                             • RKHS
                             •        :)


Thursday, January 21, 2010
Manifold regularization
                    •
                             •
                    • SVM

                    •               !


Thursday, January 21, 2010
:
                    •


                             •       2
                    •            (       )RKHS

                    •
Thursday, January 21, 2010
•
                             f



                    •            f




Thursday, January 21, 2010
•


                    •

                    •


Thursday, January 21, 2010
•

                             • Iterated Laplacians
                             • Heat semigroup
                              •
                             • Squared norm of the Hessian
                    •                 Belkin     (2004)
                                           http://www.geocities.co.jp/Technopolis/5893/publication/kernel.pdf


Thursday, January 21, 2010
Laplacian Regularized
                       Least Squres(LapRLS)
                    •
                             +
                                                0
                                          Regularized Least Squres


                    •            closed



Thursday, January 21, 2010
Laplacian Support
                             Vector Machines
                    • SVM

                    •

                    0        SVM



Thursday, January 21, 2010
?
                    •            (Harmonic
                             )
                    •
                    •


Thursday, January 21, 2010
Thursday, January 21, 2010
mixture model, EM

                                  TSMVs

                                Co-training
                                                 i.i.d
                                                         (
                               Graph-based
                                                 )
Thursday, January 21, 2010
• no pain, no gain
                    • no model assumption, no gain
                    • wrong model assumption, no gain, a lot of
                             pain




Thursday, January 21, 2010
(      | )
                    •
                             •   Co-boosting

                             •   bootstrap

                             •   Directed graphs

                             •   Information Regularization

                             •   Structural Learning

                    •        Large Data

                    •
                             •   PAC Statistical Learning Theory
Thursday, January 21, 2010
Thursday, January 21, 2010
(Tutorial etc)
                    •
                             •   NAACL 2006 Tutorial: Inductive Semi-supervised Learning
                                 with Applicability to NLP, A. Sarkar and G. Haffari.
                             •   ICML 2007 Tutorial: Semi-supervised Learning, Xiaojin Zhu.
                             •   Blitzer, J. and Zhu, J. (2008). ACL 2008 tutorial on Semi-
                                 Supervised learning. http://ssl-acl08.wikidot.com/.
                    •
                             •   X. Zhu. Semi-supervised learning literature survey. Technical
                                 report, Computer Sciences, University of Wisconsin-
                                 Madison, 2007.
                             •   Zhu, X. (2005). Semi-supervised learning with graphs.
                                 Doctoral dissertation, Carnegie Mellon University. CMU-
                                 LTI-05-192.
Thursday, January 21, 2010
(Generative model)
                    •        Nigam, K., McCallum, A. K., Thrun, S., & Mitchell, T. (2000). Text
                             classification from labeled and unlabeled documents using EM.
                             Machine Learning, 39, 103–134.

                    •        Liu, B., Lee, W. S.,Yu, P. S., & Li, X. (2002). Partially supervised
                             classification of text documents. Proceedings of the Nineteenth
                             International Conference on Machine Learning (ICML).
                    •        Lee, W. S., & Liu, B. (2003). Learning with positive and unlabeled
                             examples using weighted logistic regression. Proceedings of the
                             Twentieth International Conference on Machine Learning (ICML).

                    •        Denis, F., Gilleron, R., & Tommasi, M. (2002). Text classification from
                             positive and unlabeled examples. The 9th International Conference
                             on Information Processing and Management of Uncertainty in
                             Knowledge-Based Systems(IPMU).



Thursday, January 21, 2010
(TSVMs)
                    •        Joachims, T. (1999). Transductive inference for text classification
                             using support vector machines. Proc. 16th International Conf.
                             on Machine Learning (pp. 200– 209). Morgan Kaufmann, San
                             Francisco, CA.

                    •        Yuille, A.L., Rangara jan, A. The concave-convex procedure.
                             Neural Computation 15(4) (2003) 915–936.

                    •        L. Wang, X. Shen, and W. Pan. On transductive support vector
                             machines. In J. Verducci, X. Shen, and J. Lafferty, editors,
                             Prediction and Discovery. American Mathematical Society, 2007.

                    •        R. Collobert, et al. (2006). Large Scale Transductive SVMs.
                             Journal of Machine Learning Research 7:1687-1712.



Thursday, January 21, 2010
(Graph-based)
                    •        Blum, A., & Chawla, S. (2001). Learning from labeled and unlabeled data using graph
                             mincuts. Proc. 18th International Conf. on Machine Learning.

                    •        Zhu, X., Ghahramani, Z., & Lafferty, J. (2003a). Semi-supervised lear ning using
                             Gaussian fields and harmonic functions. The 20th International Conference on
                             Machine Learning (ICML).

                    •        Shi, J., & Malik, J. (2000). Normalized cuts and image segmentation. IEEE Transactions
                             on Pattern Analysis and Machine Intelligence, 22, 888–905.

                    •        Pang, B., & Lee, L. (2004). A sentimental education: Sentiment analysis using
                             subjectivity summarization based on minimum cuts. Proceedings of the Association
                             for Computational Linguistics (pp. 271–278).

                    •        Goldberg, A., & Zhu, X. (2006). Seeing stars when there aren’t many s tars: Graph-
                             based semi-supervised learning for sentiment categorization. HLT-NAACL 2006
                             Workshop on Textgraphs: Graph-based Algorithms for Natural Language
                             Processing. New York, NY.

                    •        Belkin, M., Niyogi, P., & Sindhwani, V. (2004b). Manifold regularization: A geometric
                             framework for learning from examples (Technical Report TR-2004-06). University
                             of Chicago.
                    •        M. Belkin & P. Niyogi (2002). `Using Manifold Structure for Partially Labelled
                             Classification'. In NIPS, pp. 929+.
Thursday, January 21, 2010
(                    )
                    •        Seeger, M. (2001). Learning with labeled and unlabeled
                             data (Technical Report). University of Edinburgh.

                    •        François Denis, Bat M, Universit'e De Lille I. PAC
                             Learning from Positive Statistical Queries. Proc. 9th
                             International Conference on Algorithmic Learning
                             Theory - ALT '98

                    •
                    •

Thursday, January 21, 2010

Contenu connexe

Tendances

画像認識の初歩、SIFT,SURF特徴量
画像認識の初歩、SIFT,SURF特徴量画像認識の初歩、SIFT,SURF特徴量
画像認識の初歩、SIFT,SURF特徴量
takaya imai
 
グラフィカルモデル入門
グラフィカルモデル入門グラフィカルモデル入門
グラフィカルモデル入門
Kawamoto_Kazuhiko
 

Tendances (20)

合成変量とアンサンブル:回帰森と加法モデルの要点
合成変量とアンサンブル:回帰森と加法モデルの要点合成変量とアンサンブル:回帰森と加法モデルの要点
合成変量とアンサンブル:回帰森と加法モデルの要点
 
機械学習 入門
機械学習 入門機械学習 入門
機械学習 入門
 
CV分野におけるサーベイ方法
CV分野におけるサーベイ方法CV分野におけるサーベイ方法
CV分野におけるサーベイ方法
 
Rでのtry関数によるエラー処理
Rでのtry関数によるエラー処理Rでのtry関数によるエラー処理
Rでのtry関数によるエラー処理
 
機械学習と主成分分析
機械学習と主成分分析機械学習と主成分分析
機械学習と主成分分析
 
Skip Connection まとめ(Neural Network)
Skip Connection まとめ(Neural Network)Skip Connection まとめ(Neural Network)
Skip Connection まとめ(Neural Network)
 
研究効率化Tips Ver.2
研究効率化Tips Ver.2研究効率化Tips Ver.2
研究効率化Tips Ver.2
 
大規模な組合せ最適化問題に対する発見的解法
大規模な組合せ最適化問題に対する発見的解法大規模な組合せ最適化問題に対する発見的解法
大規模な組合せ最適化問題に対する発見的解法
 
機械学習による統計的実験計画(ベイズ最適化を中心に)
機械学習による統計的実験計画(ベイズ最適化を中心に)機械学習による統計的実験計画(ベイズ最適化を中心に)
機械学習による統計的実験計画(ベイズ最適化を中心に)
 
ELBO型VAEのダメなところ
ELBO型VAEのダメなところELBO型VAEのダメなところ
ELBO型VAEのダメなところ
 
画像認識の初歩、SIFT,SURF特徴量
画像認識の初歩、SIFT,SURF特徴量画像認識の初歩、SIFT,SURF特徴量
画像認識の初歩、SIFT,SURF特徴量
 
One Class SVMを用いた異常値検知
One Class SVMを用いた異常値検知One Class SVMを用いた異常値検知
One Class SVMを用いた異常値検知
 
GiNZAで始める日本語依存構造解析 〜CaboCha, UDPipe, Stanford NLPとの比較〜
GiNZAで始める日本語依存構造解析 〜CaboCha, UDPipe, Stanford NLPとの比較〜GiNZAで始める日本語依存構造解析 〜CaboCha, UDPipe, Stanford NLPとの比較〜
GiNZAで始める日本語依存構造解析 〜CaboCha, UDPipe, Stanford NLPとの比較〜
 
[DL輪読会]Reward Augmented Maximum Likelihood for Neural Structured Prediction
[DL輪読会]Reward Augmented Maximum Likelihood for Neural Structured Prediction[DL輪読会]Reward Augmented Maximum Likelihood for Neural Structured Prediction
[DL輪読会]Reward Augmented Maximum Likelihood for Neural Structured Prediction
 
顕著性マップの推定手法
顕著性マップの推定手法顕著性マップの推定手法
顕著性マップの推定手法
 
【論文紹介】How Powerful are Graph Neural Networks?
【論文紹介】How Powerful are Graph Neural Networks?【論文紹介】How Powerful are Graph Neural Networks?
【論文紹介】How Powerful are Graph Neural Networks?
 
深層学習の非常に簡単な説明
深層学習の非常に簡単な説明深層学習の非常に簡単な説明
深層学習の非常に簡単な説明
 
非ガウス性を利用した 因果構造探索
非ガウス性を利用した因果構造探索非ガウス性を利用した因果構造探索
非ガウス性を利用した 因果構造探索
 
Semi supervised, weakly-supervised, unsupervised, and active learning
Semi supervised, weakly-supervised, unsupervised, and active learningSemi supervised, weakly-supervised, unsupervised, and active learning
Semi supervised, weakly-supervised, unsupervised, and active learning
 
グラフィカルモデル入門
グラフィカルモデル入門グラフィカルモデル入門
グラフィカルモデル入門
 

En vedette

NIPS読み会2013: One-shot learning by inverting a compositional causal process
NIPS読み会2013: One-shot learning by inverting  a compositional causal processNIPS読み会2013: One-shot learning by inverting  a compositional causal process
NIPS読み会2013: One-shot learning by inverting a compositional causal process
nozyh
 
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...
Databricks
 
Active Learning 入門
Active Learning 入門Active Learning 入門
Active Learning 入門
Shuyo Nakatani
 
はじめてでもわかるベイズ分類器 -基礎からMahout実装まで-
はじめてでもわかるベイズ分類器 -基礎からMahout実装まで-はじめてでもわかるベイズ分類器 -基礎からMahout実装まで-
はじめてでもわかるベイズ分類器 -基礎からMahout実装まで-
Naoki Yanai
 
パターン認識 第10章 決定木
パターン認識 第10章 決定木 パターン認識 第10章 決定木
パターン認識 第10章 決定木
Miyoshi Yuya
 
Simple perceptron by TJO
Simple perceptron by TJOSimple perceptron by TJO
Simple perceptron by TJO
Takashi J OZAKI
 
機会学習ハッカソン:ランダムフォレスト
機会学習ハッカソン:ランダムフォレスト機会学習ハッカソン:ランダムフォレスト
機会学習ハッカソン:ランダムフォレスト
Teppei Baba
 
FIT2012招待講演「異常検知技術のビジネス応用最前線」
FIT2012招待講演「異常検知技術のビジネス応用最前線」FIT2012招待講演「異常検知技術のビジネス応用最前線」
FIT2012招待講演「異常検知技術のビジネス応用最前線」
Shohei Hido
 

En vedette (20)

NIPS読み会2013: One-shot learning by inverting a compositional causal process
NIPS読み会2013: One-shot learning by inverting  a compositional causal processNIPS読み会2013: One-shot learning by inverting  a compositional causal process
NIPS読み会2013: One-shot learning by inverting a compositional causal process
 
論文紹介 Semi-supervised Learning with Deep Generative Models
論文紹介 Semi-supervised Learning with Deep Generative Models論文紹介 Semi-supervised Learning with Deep Generative Models
論文紹介 Semi-supervised Learning with Deep Generative Models
 
NIPS2015読み会: Ladder Networks
NIPS2015読み会: Ladder NetworksNIPS2015読み会: Ladder Networks
NIPS2015読み会: Ladder Networks
 
IIBMP2016 深層生成モデルによる表現学習
IIBMP2016 深層生成モデルによる表現学習IIBMP2016 深層生成モデルによる表現学習
IIBMP2016 深層生成モデルによる表現学習
 
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...
 
機械学習とコンピュータビジョン入門
機械学習とコンピュータビジョン入門機械学習とコンピュータビジョン入門
機械学習とコンピュータビジョン入門
 
Deep Learning on Apache Spark: TensorFrames & Deep Learning Pipelines
Deep Learning on Apache Spark: TensorFrames & Deep Learning Pipelines Deep Learning on Apache Spark: TensorFrames & Deep Learning Pipelines
Deep Learning on Apache Spark: TensorFrames & Deep Learning Pipelines
 
Semi-Supervised SVM
Semi-Supervised SVMSemi-Supervised SVM
Semi-Supervised SVM
 
多数のグラフからの統計的機械学習 (2014.7.24 人工知能学会 第94回人工知能基本問題研究会 招待講演)
多数のグラフからの統計的機械学習 (2014.7.24 人工知能学会 第94回人工知能基本問題研究会 招待講演)多数のグラフからの統計的機械学習 (2014.7.24 人工知能学会 第94回人工知能基本問題研究会 招待講演)
多数のグラフからの統計的機械学習 (2014.7.24 人工知能学会 第94回人工知能基本問題研究会 招待講演)
 
Active Learning 入門
Active Learning 入門Active Learning 入門
Active Learning 入門
 
トピックモデルを用いた 潜在ファッション嗜好の推定
トピックモデルを用いた 潜在ファッション嗜好の推定トピックモデルを用いた 潜在ファッション嗜好の推定
トピックモデルを用いた 潜在ファッション嗜好の推定
 
はじめてでもわかるベイズ分類器 -基礎からMahout実装まで-
はじめてでもわかるベイズ分類器 -基礎からMahout実装まで-はじめてでもわかるベイズ分類器 -基礎からMahout実装まで-
はじめてでもわかるベイズ分類器 -基礎からMahout実装まで-
 
パターン認識 第10章 決定木
パターン認識 第10章 決定木 パターン認識 第10章 決定木
パターン認識 第10章 決定木
 
今日から使える! みんなのクラスタリング超入門
今日から使える! みんなのクラスタリング超入門今日から使える! みんなのクラスタリング超入門
今日から使える! みんなのクラスタリング超入門
 
Simple perceptron by TJO
Simple perceptron by TJOSimple perceptron by TJO
Simple perceptron by TJO
 
バンディットアルゴリズム入門と実践
バンディットアルゴリズム入門と実践バンディットアルゴリズム入門と実践
バンディットアルゴリズム入門と実践
 
決定木学習
決定木学習決定木学習
決定木学習
 
SVMについて
SVMについてSVMについて
SVMについて
 
機会学習ハッカソン:ランダムフォレスト
機会学習ハッカソン:ランダムフォレスト機会学習ハッカソン:ランダムフォレスト
機会学習ハッカソン:ランダムフォレスト
 
FIT2012招待講演「異常検知技術のビジネス応用最前線」
FIT2012招待講演「異常検知技術のビジネス応用最前線」FIT2012招待講演「異常検知技術のビジネス応用最前線」
FIT2012招待講演「異常検知技術のビジネス応用最前線」
 

Plus de syou6162

Kernel20110619
Kernel20110619Kernel20110619
Kernel20110619
syou6162
 
Optimization In R
Optimization In ROptimization In R
Optimization In R
syou6162
 
Syou6162 Dbcls
Syou6162 DbclsSyou6162 Dbcls
Syou6162 Dbcls
syou6162
 

Plus de syou6162 (20)

自分の目的に合った統計量と そのバラ付きを計算しよう ~NPSを例に~(統計学勉強会)
自分の目的に合った統計量と そのバラ付きを計算しよう ~NPSを例に~(統計学勉強会)自分の目的に合った統計量と そのバラ付きを計算しよう ~NPSを例に~(統計学勉強会)
自分の目的に合った統計量と そのバラ付きを計算しよう ~NPSを例に~(統計学勉強会)
 
オープンセミナー岡山 これから始めるデータ活用
オープンセミナー岡山 これから始めるデータ活用オープンセミナー岡山 これから始めるデータ活用
オープンセミナー岡山 これから始めるデータ活用
 
正確な意思決定を阻む 問題・障害との向き合い方
正確な意思決定を阻む 問題・障害との向き合い方正確な意思決定を阻む 問題・障害との向き合い方
正確な意思決定を阻む 問題・障害との向き合い方
 
カスタマーサクセスのためのデータ整備人の活動記録
カスタマーサクセスのためのデータ整備人の活動記録カスタマーサクセスのためのデータ整備人の活動記録
カスタマーサクセスのためのデータ整備人の活動記録
 
Mackerel Drink Up #9 ロール内異常検知の正式化
Mackerel Drink Up #9 ロール内異常検知の正式化Mackerel Drink Up #9 ロール内異常検知の正式化
Mackerel Drink Up #9 ロール内異常検知の正式化
 
Mackerelのロール内異常検知の設計と運用
Mackerelのロール内異常検知の設計と運用Mackerelのロール内異常検知の設計と運用
Mackerelのロール内異常検知の設計と運用
 
教師なし学習によるMackerelの異常検知機能について 〜設計/運用/評価の観点から〜
教師なし学習によるMackerelの異常検知機能について〜設計/運用/評価の観点から〜教師なし学習によるMackerelの異常検知機能について〜設計/運用/評価の観点から〜
教師なし学習によるMackerelの異常検知機能について 〜設計/運用/評価の観点から〜
 
機械学習を活用したサービスにおける工夫紹介
機械学習を活用したサービスにおける工夫紹介機械学習を活用したサービスにおける工夫紹介
機械学習を活用したサービスにおける工夫紹介
 
機械学習を使った趣味サービスにおける工夫紹介
機械学習を使った趣味サービスにおける工夫紹介機械学習を使った趣味サービスにおける工夫紹介
機械学習を使った趣味サービスにおける工夫紹介
 
今日から始める機械学習〜はてなの事例〜
今日から始める機械学習〜はてなの事例〜今日から始める機械学習〜はてなの事例〜
今日から始める機械学習〜はてなの事例〜
 
機械学習を用いたMackerelの異常検知機能について
機械学習を用いたMackerelの異常検知機能について機械学習を用いたMackerelの異常検知機能について
機械学習を用いたMackerelの異常検知機能について
 
Mackerel Anomaly Detection at PyCon mini Osaka
Mackerel Anomaly Detection at PyCon mini OsakaMackerel Anomaly Detection at PyCon mini Osaka
Mackerel Anomaly Detection at PyCon mini Osaka
 
はてなにおける機械学習の取り組み
はてなにおける機械学習の取り組みはてなにおける機械学習の取り組み
はてなにおける機械学習の取り組み
 
はてなにおける機械学習の取り組み
はてなにおける機械学習の取り組みはてなにおける機械学習の取り組み
はてなにおける機械学習の取り組み
 
異常検知ナイト LT登壇資料 はてな id:syou6162
異常検知ナイト LT登壇資料 はてな id:syou6162異常検知ナイト LT登壇資料 はてな id:syou6162
異常検知ナイト LT登壇資料 はてな id:syou6162
 
Duolingo.pptx
Duolingo.pptxDuolingo.pptx
Duolingo.pptx
 
今日からできる構造学習(主に構造化パーセプトロンについて)
今日からできる構造学習(主に構造化パーセプトロンについて)今日からできる構造学習(主に構造化パーセプトロンについて)
今日からできる構造学習(主に構造化パーセプトロンについて)
 
Kernel20110619
Kernel20110619Kernel20110619
Kernel20110619
 
Optimization In R
Optimization In ROptimization In R
Optimization In R
 
Syou6162 Dbcls
Syou6162 DbclsSyou6162 Dbcls
Syou6162 Dbcls
 

半教師あり学習

  • 1. Introduction to Semi- Supervised Learning Thursday, January 21, 2010
  • 2. Outline • • • Self Traininga • Multiview Algorithms • Generative Models • S3VMs(TSVMs) • Graph-Based Algorithms • • Thursday, January 21, 2010
  • 4. • • costly... • • • : Thursday, January 21, 2010
  • 5. • : ? => SSL Thursday, January 21, 2010
  • 7. • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 8. SSL • Self Training • Multiview Algorithms • Generative Models • S3VMs(TSVMs) • Graph-Based Algorithms Thursday, January 21, 2010
  • 10. Self Training • : • : p(y=1|x) 0.99 1 ! • Thursday, January 21, 2010
  • 11. Self Training • f • f •( ) • Thursday, January 21, 2010
  • 12. Self Training • • ! • • • • • Thursday, January 21, 2010
  • 14. Co-training • Self Training => • ? • Thursday, January 21, 2010
  • 15. : Web http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 16. • ) (BOF • • Co-training idea • • Thursday, January 21, 2010
  • 17. Co-training • • • Thursday, January 21, 2010
  • 18. Co-training http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 19. Co-training • • • Self Training • • • Thursday, January 21, 2010
  • 20. Co-training • Co-EM • EM • Fake feature split • • Multiview • • => Thursday, January 21, 2010
  • 22. Generative Models • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 23. Generative Models • • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 24. Generative Models • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 25. Generative Models • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 26. Generative Models • : • ( ) • Thursday, January 21, 2010
  • 27. Generative Models • • • EM • • • EM • • • Baum-Welch algorithm Thursday, January 21, 2010
  • 28. • • : PRML => EM MLE Thursday, January 21, 2010
  • 29. ( ) • y 2 • EM MLE Thursday, January 21, 2010
  • 30. E-step and M-step http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 31. Generative Models • • • • • • EM ( ) • => page Thursday, January 21, 2010
  • 32. Generative Models http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 33. • λ(λ < 1) Thursday, January 21, 2010
  • 35. TSVMs • ( ) http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 36. (RKHS) • SVM • => (RKHS:reproducing kernel Hilbert space) • Thursday, January 21, 2010
  • 37. k k f Thursday, January 21, 2010
  • 38. • • ( ) • Thursday, January 21, 2010
  • 39. (F ) • x F (F !) Thursday, January 21, 2010
  • 40. •f F • • f f( x ) => f => F => !! Thursday, January 21, 2010
  • 42. SVM RKHS ! hw RKHS Thursday, January 21, 2010
  • 43. hinge loss hinge loss !! http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 44. • ? • f(x)>1 or f(x)<-1 • ! => http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 45. • Joachims(1999) • branch- and-bound search( ) • 10000 Thursday, January 21, 2010
  • 46. S3VM • • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 47. CCCP (Concave-Convex procedure) • Yuille (2003) • • ) ( • Difference of Convex (DC ) • Update Thursday, January 21, 2010
  • 48. CCCP (Concave-Convex procedure) • update • • (update ) • !! Thursday, January 21, 2010
  • 49. TSVMs + CCCP • CCCP TSVMs • • Iteration 2 http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf http://www.stat.umn.edu/~xshen/paper/tsvm.pdf Thursday, January 21, 2010
  • 50. L. Wang (2007) • SVM TSMVs CCCP TSMVs http://www.stat.umn.edu/~xshen/paper/tsvm.pdf Thursday, January 21, 2010
  • 51. Graph-Based Algorithms Thursday, January 21, 2010
  • 52. : • • http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 53. !! • • etc http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 54. sparseness http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf Thursday, January 21, 2010
  • 55. http://pages.cs.wisc.edu/~jerryzhu/pub/sslicml07.pdf https://www.aaai.org/Papers/ICML/2003/ICML03-118.pdf Thursday, January 21, 2010
  • 56. Graph-based SSL • : edge node • : • (instance ) • kNN : k 1 0 => sparse • : • dense Thursday, January 21, 2010
  • 57. Graph-based SSL • mincut • harmonic • manifold regularization Thursday, January 21, 2010
  • 58. • 0 1 fix • Thursday, January 21, 2010
  • 59. => • sink source • : http://john.blitzer.com/tutorial/ssl_tutorial.pdf Thursday, January 21, 2010
  • 60. source sink : source ∞ : sink -∞ Thursday, January 21, 2010
  • 61. harmonic => Laplacian • 0-1 y • Zhu (2003a) Goldberg (2006) • Thursday, January 21, 2010
  • 62. harmonic => Laplacian (SVM ) ( L a U b f ... ) → !! Thursday, January 21, 2010
  • 64. y_i hat{y}_i http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf Thursday, January 21, 2010
  • 65. Laplacian •f 0 ! => Thursday, January 21, 2010
  • 66. Goldberg (2006) • SVMR Metric labeling • SSL • http://pages.cs.wisc.edu/~goldberg/publications/goldbergTextgraphs.pdf Thursday, January 21, 2010
  • 67. Manifold regularization • Harmonic • ... • • RKHS • :) Thursday, January 21, 2010
  • 68. Manifold regularization • • • SVM • ! Thursday, January 21, 2010
  • 69. : • • 2 • ( )RKHS • Thursday, January 21, 2010
  • 70. f • f Thursday, January 21, 2010
  • 71. • • Thursday, January 21, 2010
  • 72. • Iterated Laplacians • Heat semigroup • • Squared norm of the Hessian • Belkin (2004) http://www.geocities.co.jp/Technopolis/5893/publication/kernel.pdf Thursday, January 21, 2010
  • 73. Laplacian Regularized Least Squres(LapRLS) • + 0 Regularized Least Squres • closed Thursday, January 21, 2010
  • 74. Laplacian Support Vector Machines • SVM • 0 SVM Thursday, January 21, 2010
  • 75. ? • (Harmonic ) • • Thursday, January 21, 2010
  • 77. mixture model, EM TSMVs Co-training i.i.d ( Graph-based ) Thursday, January 21, 2010
  • 78. • no pain, no gain • no model assumption, no gain • wrong model assumption, no gain, a lot of pain Thursday, January 21, 2010
  • 79. ( | ) • • Co-boosting • bootstrap • Directed graphs • Information Regularization • Structural Learning • Large Data • • PAC Statistical Learning Theory Thursday, January 21, 2010
  • 81. (Tutorial etc) • • NAACL 2006 Tutorial: Inductive Semi-supervised Learning with Applicability to NLP, A. Sarkar and G. Haffari. • ICML 2007 Tutorial: Semi-supervised Learning, Xiaojin Zhu. • Blitzer, J. and Zhu, J. (2008). ACL 2008 tutorial on Semi- Supervised learning. http://ssl-acl08.wikidot.com/. • • X. Zhu. Semi-supervised learning literature survey. Technical report, Computer Sciences, University of Wisconsin- Madison, 2007. • Zhu, X. (2005). Semi-supervised learning with graphs. Doctoral dissertation, Carnegie Mellon University. CMU- LTI-05-192. Thursday, January 21, 2010
  • 82. (Generative model) • Nigam, K., McCallum, A. K., Thrun, S., & Mitchell, T. (2000). Text classification from labeled and unlabeled documents using EM. Machine Learning, 39, 103–134. • Liu, B., Lee, W. S.,Yu, P. S., & Li, X. (2002). Partially supervised classification of text documents. Proceedings of the Nineteenth International Conference on Machine Learning (ICML). • Lee, W. S., & Liu, B. (2003). Learning with positive and unlabeled examples using weighted logistic regression. Proceedings of the Twentieth International Conference on Machine Learning (ICML). • Denis, F., Gilleron, R., & Tommasi, M. (2002). Text classification from positive and unlabeled examples. The 9th International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems(IPMU). Thursday, January 21, 2010
  • 83. (TSVMs) • Joachims, T. (1999). Transductive inference for text classification using support vector machines. Proc. 16th International Conf. on Machine Learning (pp. 200– 209). Morgan Kaufmann, San Francisco, CA. • Yuille, A.L., Rangara jan, A. The concave-convex procedure. Neural Computation 15(4) (2003) 915–936. • L. Wang, X. Shen, and W. Pan. On transductive support vector machines. In J. Verducci, X. Shen, and J. Lafferty, editors, Prediction and Discovery. American Mathematical Society, 2007. • R. Collobert, et al. (2006). Large Scale Transductive SVMs. Journal of Machine Learning Research 7:1687-1712. Thursday, January 21, 2010
  • 84. (Graph-based) • Blum, A., & Chawla, S. (2001). Learning from labeled and unlabeled data using graph mincuts. Proc. 18th International Conf. on Machine Learning. • Zhu, X., Ghahramani, Z., & Lafferty, J. (2003a). Semi-supervised lear ning using Gaussian fields and harmonic functions. The 20th International Conference on Machine Learning (ICML). • Shi, J., & Malik, J. (2000). Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22, 888–905. • Pang, B., & Lee, L. (2004). A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. Proceedings of the Association for Computational Linguistics (pp. 271–278). • Goldberg, A., & Zhu, X. (2006). Seeing stars when there aren’t many s tars: Graph- based semi-supervised learning for sentiment categorization. HLT-NAACL 2006 Workshop on Textgraphs: Graph-based Algorithms for Natural Language Processing. New York, NY. • Belkin, M., Niyogi, P., & Sindhwani, V. (2004b). Manifold regularization: A geometric framework for learning from examples (Technical Report TR-2004-06). University of Chicago. • M. Belkin & P. Niyogi (2002). `Using Manifold Structure for Partially Labelled Classification'. In NIPS, pp. 929+. Thursday, January 21, 2010
  • 85. ( ) • Seeger, M. (2001). Learning with labeled and unlabeled data (Technical Report). University of Edinburgh. • François Denis, Bat M, Universit'e De Lille I. PAC Learning from Positive Statistical Queries. Proc. 9th International Conference on Algorithmic Learning Theory - ALT '98 • • Thursday, January 21, 2010