Eacl 2006 Pedersen

Language Independent Methods of Clustering Similar Contexts (with applications) Ted Pedersen University of Minnesota, Duluth http://www.d.umn.edu/~tpederse [email_address]

Language Independent Methods ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Clustering Similar Contexts ,[object Object],[object Object],[object Object],[object Object],[object Object]

Applications ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Tutorial Outline ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

General Info ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

SenseClusters ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Many thanks… ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Headed and Headless Contexts ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Headed Contexts (input) ,[object Object],[object Object],[object Object],[object Object],[object Object]

Headed Contexts (output) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Headless Contexts (input) ,[object Object],[object Object],[object Object],[object Object],[object Object]

Headless Contexts (output) ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Web Search as Application ,[object Object],[object Object],[object Object],[object Object]

Email Foldering as Application ,[object Object],[object Object],[object Object],[object Object],[object Object]

Clustering News as Application ,[object Object],[object Object],[object Object],[object Object]

What is it to be “similar”? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

General Methodology ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Identifying Lexical Features Measures of Association and Tests of Significance

What are features? ,[object Object],[object Object],[object Object]

Where do features come from? ,[object Object],[object Object],[object Object],[object Object]

Feature Selection ,[object Object],[object Object],[object Object],[object Object],[object Object]

Lexical Features ,[object Object],[object Object],[object Object],[object Object],[object Object]

Bigrams ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Co-occurrences ,[object Object],[object Object],[object Object],[object Object],[object Object]

Bigrams and Co-occurrences ,[object Object],[object Object],[object Object],[object Object],[object Object]

“ occur together more often than expected by chance…” ,[object Object],[object Object],[object Object],[object Object],[object Object]

2x2 Contingency Table 100,000 300 !Artificial 400 100 Artificial !Intelligence Intelligence

2x2 Contingency Table 100,000 99,700 300 99,600 99,400 200 !Artificial 400 300 100 Artificial !Intelligence Intelligence

2x2 Contingency Table 100,000 99,700 300 99,600 99,400.0 99,301.2 200.0 298.8 !Artificial 400 300.0 398.8 100.0 000.12 Artificial !Intelligence Intelligence

Interpreting the Scores… ,[object Object],[object Object]

Interpreting the Scores… ,[object Object],[object Object],[object Object]

Measures of Association ,[object Object],[object Object],[object Object]

Measures Supported in NSP ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

NSP ,[object Object],[object Object],[object Object],[object Object],[object Object]

Summary ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Related Work ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Context Representations First and Second Order Methods

Once features selected… ,[object Object],[object Object],[object Object]

First Order Representation ,[object Object],[object Object],[object Object]

Contexts ,[object Object],[object Object],[object Object],[object Object]

Unigram Feature Set ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

First Order Vectors of Unigrams 1 0 1 0 1 Cxt4 0 0 0 0 0 Cxt3 1 1 0 1 0 Cxt2 1 1 1 1 1 Cxt1 child magic curse black island

Bigram Feature Set ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

First Order Vectors of Bigrams 1 0 1 1 0 Cxt4 0 1 1 0 0 Cxt3 1 0 0 0 1 Cxt2 1 0 0 1 1 Cxt1 voodoo child serious error military might island curse black magic

First Order Vectors ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Second Order Features ,[object Object],[object Object],[object Object],[object Object],[object Object]

Second Order Representation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Word by Word Matrix 120.0 0 69.4 0 0 voodoo 0 89.2 0 21.2 0 serious 0 54.9 100.3 0 0 military 73.2 0 0 189.2 0 island 43.2 0 0 0 123.5 black child error might curse magic

Word by Word Matrix ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

There was an island curse of black magic cast by that voodoo child. 120.0 0 69.4 0 0 voodoo 73.2 0 0 189.2 0 island 43.2 0 0 0 123.5 black child error might curse magic

Second Order Co-Occurrences ,[object Object],[object Object]

Second Order Representation ,[object Object],[object Object]

There was an island curse of black magic cast by that voodoo child. 78.8 0 24.4 63.1 41.2 Cxt1 child error might curse magic

Summary ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Related Work ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Dimensionality Reduction Singular Value Decomposition

Motivation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Many Methods ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Effect of SVD ,[object Object],[object Object]

Effect of SVD ,[object Object],[object Object],[object Object]

How can SVD be used? ,[object Object],[object Object],[object Object],[object Object]

Word by Word Matrix 4 2 0 0 0 3 0 1 box 0 1 2 2 1 2 0 0 memory 0 0 0 1 0 0 2 0 organ 0 2 0 3 2 0 0 0 debt 0 1 0 3 1 0 0 2 linux 0 1 0 3 2 0 0 0 sales 3 0 2 2 0 3 0 0 lab 1 0 2 0 0 1 2 0 petri 0 1 0 0 2 0 0 1 disk 1 0 2 0 0 0 3 0 body 0 0 0 3 1 0 0 2 pc plasma graphics tissue data ibm cells blood apple

Singular Value Decomposition A=UDV’

U -.52 .39 -.48 .02 .09 .41 -.09 .40 -.30 .08 .31 .43 -.26 -.39 -.6 .20 .00 -.00 -.00 -.02 -.01 .00 -.02 -.00 -.07 -.3 .14 -.49 -.07 .30 .25 .56 -.01 .08 .05 -.01 .24 -.08 .11 .46 .08 .03 -.04 .72 .09 -.31 -.01 .37 -.07 .01 -.21 -.31 -.34 -.45 -.68 .29 .00 .05 .83 .17 -.02 .25 -.45 .08 .03 .20 -.22 .31 -.60 .39 .13 .35 -.01 -.04 -.44 .08 .44 .59 -.49 .05 -.02 .63 .02 -.09 .52 -.2 .09 .35

D 0.00 0.00 0.00 0.66 1.26 2.30 2.52 3.25 3.99 6.36 9.19

V -.20 .22 -.07 -.10 -.87 -.07 -.06 .17 .19 -.26 .04 .03 .17 -.32 .02 .13 -.26 -.17 .06 -.04 .86 .50 -.58 .12 .09 -.18 -.27 -.18 -.12 -.47 .11 -.03 .12 .31 -.32 -.04 .64 -.45 -.14 -.23 .28 .07 -.23 -.62 -.59 .05 .02 -.12 .15 .11 .25 -.71 -.31 -.04 .08 .29 -.05 .05 .20 -.51 .09 -.03 .12 .31 -.01 .02 -.45 -.32 .50 .27 .49 -.02 .08 .21 -.06 .08 -.09 .52 -.45 -.01 .63 .03 -.12 -.31 .71 -.13 .39 -.12 .12 .15 .37 .07 .58 -.41 .15 .17 -.30 -.32 -.27 -.39 .11 .44 .25 .03 -.02 .26 .23 .39 .57 -.37 .04 .03 -.12 -.31 -.05 -.05 .04 .28 -.04 .08 .21

Word by Word Matrix After SVD 1.1 1.0 .98 1.7 .86 .72 .85 .77 memory .00 .00 .17 1.2 .77 .00 .84 .00 organ .00 1.5 .00 3.2 2.1 .00 .00 1.2 debt .13 1.1 .03 2.7 1.7 .16 .00 .96 linux .41 .85 .35 2.2 1.3 .39 .15 .73 sales 2.3 .18 2.5 1.7 .35 2.0 1.7 .21 lab 1.4 .00 1.5 .49 .00 1.2 1.1 .00 germ .00 .91 .00 2.1 1.3 .01 .00 .76 disk 1.5 .00 1.6 .33 .00 1.3 1.2 .00 body .09 .86 .01 2.0 1.3 .11 .00 .73 pc plasma graphics tissue data ibm cells blood apple

Second Order Representation ,[object Object],[object Object],[object Object],[object Object],1.0 .72 memory .00 .00 organ .13 1.1 .03 2.7 1.7 .16 .00 .96 linux .00 .91 .00 2.1 1.3 .01 .00 .76 disk Plasma graphics tissue data ibm cells blood apple

Relationship to LSA ,[object Object],[object Object],[object Object],[object Object],[object Object]

Feature by Context Representation 0 1 0 0 serious error 1 0 1 1 voodoo child 0 1 0 0 military might 1 0 0 1 island curse 1 0 1 1 black magic Cxt4 Cxt3 Cxt2 Cxt1

References ,[object Object],[object Object],[object Object],[object Object]

Clustering Partitional Methods Cluster Stopping Cluster Labeling

Many many methods… ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

General Methodology ,[object Object],[object Object],[object Object],[object Object],[object Object]

Agglomerative Clustering ,[object Object],[object Object],[object Object]

Measuring Similarity ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Agglomerative Clustering ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Average Link Clustering 1 2 4 S3 1 2 4 S3 0 2 S4 0 3 S2 2 3 S1 S4 S2 S1 0 S4 0 S2 S1S3 S4 S2 S1S3 S4 S1S3S2 S4 S1S3S2

Partitional Methods ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Partitional Methods ,[object Object],[object Object],[object Object],[object Object],[object Object]

Random Initial Centroids (k=2)

Partitional Criterion Functions ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Intra Cluster Similarity ,[object Object],[object Object],[object Object],[object Object]

Ball of String (I1 Internal Criterion Function)

Flower (I2 Internal Criterion Function)

Inter Cluster Similarity ,[object Object],[object Object],[object Object]

The Fan (E1 External Criterion Function)

Hybrid Criterion Functions ,[object Object],[object Object],[object Object],[object Object],[object Object]

Cluster Stopping ,[object Object],[object Object]

Criterion Functions Can Help ,[object Object],[object Object],[object Object],[object Object]

SenseCluster’s Approach to Cluster Stopping ,[object Object],[object Object],[object Object]

H2 versus k T. Blair – V. Putin – S. Hussein

PK2 ,[object Object],[object Object],[object Object]

PK2 predicts 3 senses T. Blair – V. Putin – S. Hussein

PK3 ,[object Object],[object Object],[object Object],[object Object]

PK3 predicts 3 senses T. Blair – V. Putin – S. Hussein

References ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Cluster Labeling ,[object Object],[object Object]

Results of Clustering ,[object Object],[object Object],[object Object],[object Object]

Label Types ,[object Object],[object Object]

Evaluation Techniques Comparison to gold standard data

Evaluation ,[object Object],[object Object],[object Object],[object Object]

Evaluation ,[object Object],[object Object],[object Object]

Baseline Algorithm ,[object Object],[object Object]

Baseline Performance ,[object Object],170 55 35 80 Totals 170 55 35 80 C3 0 0 0 0 C2 0 0 0 0 C1 Totals S3 S2 S1 170 80 35 55 Totals 170 80 35 55 C3 0 0 0 0 C2 0 0 0 0 C1 Totals S1 S2 S3

Evaluation ,[object Object],[object Object],[object Object],[object Object],[object Object],170 55 35 80 Totals 65 10 5 50 C3 60 40 0 20 C2 45 5 30 10 C1 Totals S3 S2 S1

Evaluation ,[object Object],[object Object],[object Object],170 80 55 35 Totals 65 50 10 5 C3 60 20 40 0 C2 45 10 5 30 C1 Totals S1 S3 S2

Analysis ,[object Object],[object Object],[object Object],[object Object]

Practical Session Experiments with SenseClusters

Things to Try ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Experimental Data ,[object Object],[object Object],[object Object],[object Object]

Creating Experimental Data ,[object Object],[object Object],[object Object],[object Object],[object Object]

Headed Clustering ,[object Object],[object Object],[object Object]

Headless Contexts ,[object Object],[object Object]

Thank you! ,[object Object],[object Object],[object Object],[object Object],[object Object]

Eacl 2006 Pedersen

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (12)

En vedette

En vedette (20)

Similaire à Eacl 2006 Pedersen

Similaire à Eacl 2006 Pedersen (20)

Plus de University of Minnesota, Duluth

Plus de University of Minnesota, Duluth (18)

Dernier

Dernier (20)

Eacl 2006 Pedersen