Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

IRE- Algorithm Name Detection in Research Papers

213 vues

Publié le

Algorithm Name Detection in Research Papers in Computer Science.

Publié dans : Formation
  • Identifiez-vous pour voir les commentaires

  • Soyez le premier à aimer ceci

IRE- Algorithm Name Detection in Research Papers

  1. 1. Algorithm Name Detection in Computer Science Research Papers Information Retrieval & Extraction Course IIIT HYDERABAD Submission By: Team 41 Allaparthi Sriteja [201302139] Deeksha Singh Thakur [201505627] Sneh gupta [201302201]
  2. 2. Aim of project ● Processing the contents of the research document ● List out the name of algorithms being discussed in the paper ● Assist the users to find research papers specific to a domain without actually opening and reading each of them. Extraction of Algorithm Name from Research Paper
  3. 3. Converting pdf to text Input : A research paper in the pdf format. Output : Need to convert that pdf to text format. Processing : Using PDFMiner pdf2txt.py -O myoutput -o myoutput/myfile.text -t text myfile.pdf Usage: pdf2txt.py [options] filename.pdf Options: -o output file name -t output format (text/html/xml/tag[for Tagged PDFs]) -O dirname (triggers extraction of images from PDF into directory)
  4. 4. Named Entity Recognition Input : Research paper in the text format. Output : Noun phrases (NNPS and NNs) Processing : ● Sentence tokenization ● Merging the divided words at the end of the line [ex: div - n ision] ● Removing the part before the Abstract and after the Reference. ● Find the citation sentences and extract them ● Do pos_tagging for those sentences. ● Now extract the NNPS and NN. combine the NNPS occurring adjacent to each other in a sentence.
  5. 5. Filtration of the Named Entities Input : Named Entities with author names, University names, places. Output : stemmed desired named entities using porter stemmer. Processing: ● Designed the list of authors and universities and places. ● And compare the named entities with these lists and filter them. ● Search for the word algorithm or technique to give more weightage to that particular word as the probability of getting the algorithm name will be high in such sentences. ● Stem these remaining named entities using Porter Stemmer
  6. 6. Phase II
  7. 7. Input : Named Entities from Research Papers - From each research paper in the corpus, we obtain a set of Named Entities Eg. - These NE’s are filtered for author name geographical locations organization names dataset names BUT THE DATA STILL CONTAINS NOISE!!! neighborhood sparselinearmethod movi slim tabl matrixfactor hoslim ratingpredict
  8. 8. TASK : Separate noisy data from names of actual algorithms Using WORD2VEC From Gensim library Gensim is a FREE Python library that allows - Making and Importing word2vec models - Determine similarity between words in the model - Determine topN most similar words to a given word
  9. 9. WORD2VEC MODEL : The word2vec model under consideration contains - word2vec word vectors trained on ~4.3lac computer science papers, 3.7B tokens A 300 dimensional vector representation of all 1 word algorithm names Used as model[‘word’] = {[300 dimension vector], dtype: float}
  10. 10. Classifying the tokens : Form a list,(manually by going through some papers) - true positives[containing name of actual computer science algorithms] false positives [most common noise components in each paper]. Compare each named entity extracted from paper with these lists of TPs and FPs and find the similarity between them. If the similarity between a word and another word in TP is greater than a threshold value (0.4 considered in our case), classify it as the TP, otherwise FP.
  11. 11. TOKEN TRUE POSITIVES 'Svm' 'Knn' 'Neuralnetwork' 'Decisiontree' 'Lda' 'Backprop' 'Spade' 'search’ 'plsa' 'machinelearn' 'cluster' 'randomforest' 'Network' 'markov' 'reinforcementlearn' 'Cart' 'regressiontre' FALSE POSITIVES ‘Concept' 'dataset' 'database' 'approach' 'method' 'success' 'Algorithm' 'analysi' 'model' model.similarity(token,true_positives)<model.similarity(false_positives)