This PowerPoint helps students to consider the concept of infinity.
Smai Project: Topic Modelling
1. Topic Modelling
Assigning topic to any text
Team SMM
Mohit Sharma 201505508
Hari Naga Raghavendra Manohar 201505551
K S Chandra Reddy 201505544
2. Aim
To identify small number of topics/categories that best characterize a given document.
The Categories we considered are:
wiki, art, reference, people, culture, books, design, politics, technology,
psychology, interesting, wikipedia, research, religion, music, math, development,
theory, philosophy, article, language, science, programming, history and software.
6. We can use LDA to classify documents in different tags. We know that LDA divides the given corpus in fixed no. of
topics and can also provide which topics are contained in a document and with what probability. For the
experiments performed using LDA, we don’t need to worry about internal implementation of LDA. We used
gensim’s implementation of LDA. To use the library, we just need to know few points about input and output
format. Read the documentation on following link.
https://radimrehurek.com/gensim/wiki.html
During Learning phase
INPUT:
We provide all the wiki documents in single XML file zipped in bz2 format.
LEARNT MODEL:
Word distribution for each topic eg: “topic #0: 0.009*river + 0.008*lake + 0.006*island + 0.005*mountain +
0.004*area + 0.004*park + 0.004*antarctic + 0.004*south + 0.004*mountains + 0.004*dam”
7. During Testing phase
INPUT:
We provide the document to be classified in bag of words form to the learnt model
OUTPUT:
Topic distribution for a the text eg: “[(34, 0.023705742561150572), (60, 0.017830310671555303), (62,
0.023999239610385081), (83,0.029439444128473557), (87, 0.028172479800878891), (90, 0.1207424163376625),
(116,0.022904510579689157)]” represents the probabilities of the doc to fall under topics like 34,60,62….
Major challenge in classification:
It seems to be fairly simple to classify a document in different topics as we can see in output of testing phase. But
our aim is to classify the document under different tags like “politics, science” etc. and not under topic numbers.
8. Possible Solutions
Clearly we need some way to map all the topics learnt by LDA to the most suitable tags. If we are able to do this
then we simply test the unknown text against the model learnt by LDA and then report the tag corresponding to
the topic given by LDA in output. We tried two different solutions to map topics to the tags:
1. As each topic of LDA is represented by distribution of words. We can create a query by combining those
words and find best matched document on tf-idf basis for that query. That particular document must be the
best match for that topic. So we can map the topic to tag of best matched document.
2. We can find probability distribution of topics for all the documents. Represent each document as a topic
vector. Now find the closest document or the most similar document for each topic. Map the topic to the tag
of that particular document.
9. Approach 1
We can specify the major steps to implement this approach as follows:
1. Divide the documents in training and test data with 4000 docs in test data.
2. On training data run gensim's LDA and save the learnt model. Set the number of topics as 300.
3. Save all the topics in a file and convert them to queries.
Example topic:
2016-04-06 00:05:52,466 : INFO : topic #299 (0.003): 0.014*insurance + 0.009*scott + 0.007*samurai +
0.007*hipster + 0.006*forecasting + 0.006*fbi + 0.006*imf +
0.005*skeptical + 0.005*bass + 0.005*hidden
Query corresponding to above topic#299:
299:insurance scott samurai hipster forecasting fbi imf skeptical bass hidden
4. For each query, retrieve the most relevant document in training set on tf-idf basis and create topic to doc
Id mapping.
Example:
299:cae3757420fbc4008bbfe492ab0d4cb5
10. 5. Create a topic to tag mapping using the docId to tag mapping (already available in tagData.xml) and doc ID
to topic mapping created in above step.
Example docId to tag from tagData.xml:
cae3757420fbc4008bbfe492ab0d4cb5 : ['wiki', 'en', 'wikipedia,', 'activism', '-‘, 'political', 'poetry', 'free',
'person', 'music', 'encyclopedia', 'the', 'biography', 'history']
Example topic to docId:
299:cae3757420fbc4008bbfe492ab0d4cb5
Example topic to tag:
299:['wiki', 'en', 'wikipedia,', 'activism', '-', 'political', 'poetry', 'free', 'person', 'music‘, 'encyclopedia', 'the',
'biography', 'history']
Now each topic is mapped to multiple tags.
6. For each of the test documents (from 4000 docs in test data), find out the relevant topics using learnt LDA
model. Combine the tags corresponding to them and match them against already available target tags
(from tagData.xml) for that particular document.
If even one tag is matched, we say that document is correctly classified.
12. Target tags for this particular doc from tagData.xml:
['reference', 'economics', 'wikipedia', 'politics', 'reading', 'resources']
Accuracy from this approach: 97%
Problem with this approach:
1. If there is any match between our found tags and true tags, then we call it as correctly classified. Probability
of such scenario is very high as we have multiple found tags and multiple true tags. So even if we are doing
something wrong, chances of getting good accuracy is very high.
2. As we are doing tf-idf based matching then there is high chance that the document we get on top is not best
match for that particular topic. It can also happen because we are not considering all the representative
words of a particular topic to frame the query, we just considered top 10.
13. Approach 2
After analyzing the data we found that only 25 of the tags represent around 19K documents out of 20K. Which
simply means that we can eliminate the less frequent tags and docs corresponding to them. Which means we
have to divide the corpus among 25 topics at most. Which makes it easier to implement approach 2, as each
document can be easily represented in 25 dimensional topic space. We can specify the major steps to implement
this approach as follows:
1. Eliminate the less frequent tags and documents related to them. keep only top 25. Docs left will be around
19K.
2. On complete data run gensim's LDA and saved the learnt model. Set number of topics set as 25.
3. Save all the topics in a file and convert them to queries as in previous approach.
4. Test each of 19K documents against the learnt model and find the topic distribution eg:
“42d1d305d10b4b025e01e8237c44c87e:0 0 0 0 0.0242823647949 0 0.037682372871 0 0 0 0.0988683434224
0.0113662521741 0.0157100377468 0 0 0.182273317591 0.205447648234 0 0.0524222798936 0.167240557357
0 0.178899361052 0 0 0” represents the probabilities of the doc with given id in 25 different topics.
5. Using above distribution find out the most relevant document for a particular topic and map it to the tag of
that document. It gives the similar topic to tag mapping as in previous approach.
6. Now many topics must have matched to more than one tag. Manually check which tag is best suited for that
particular topic depending on words contained in the topic. As a result we have each topic mapped to at
most one tag.
7. Now perform the testing as done in step 6 of previous approach but on all 19K docs.
14. Accuracy from this approach: 88%
Problem with this approach:
1. Mapping topics to tags manually is an issue. We can’t always find out the best suited tag just by seeing the
topic words. Sometimes tags don’t reflect anything eg: ‘wikipedia’, ‘wiki’, ‘reference’ create problem.
Modification:
Performed the above experiment again but just with meaningful tags i.e. no tag like ‘wikipedia’, ‘wiki’,
‘reference’ etc. After eliminating these documents left were 17K. But the approach posed another issue:
1. There are similar tags which can represent a topic at the same time eg: [research, science], [web, internet],
[programming, math], [literature, language].
If we keep all such similar tags then accuracy is : 80% but if we strictly keep just one tag then accuracy drops
to 65%.
Reason for the drop is possibly manual work. We can’t surely say which tag should be kept when both tags are
same.
Conclusion: 2ND
approach is better as there is very less chance of false good accuracy and accuracy is also not
bad considering just ~19K documents for learning. Time reuired for leaning is aound 10 minutes and
assigning topic to any new text is intantaneous.
