1. Query Expansion
Cluster based using N Grams
UMA K L (201305514)
SPANDAN VEGGALAM (201307674)
MAHAVER CHOPRA (201101011)
AKSHAT KANDELWAL (201001095)
INTERNATIONAL INSTITUTE OF INFORMATION TECHNOLOGY -HYDERABAD
2. Query Expansion
• Key feature of Search Engine
• In many cases it is difficult to find the search intent of user
• Users do not always formulate query in the best way
• Query recommendation is to help users in formulating queries to certain extent
• Improves the search retrieval performance, user selects the alternate query input from
suggestions which is relevant to his intent.
• Increases recall of Information Retrieval System
3. Expanding Queries
Following techniques are used for expanding queries
1. Spell Corrections
2. Finding and searching with “Synonyms” of input query terms
3. Augmenting query with terms
In our approach we focus on Augmenting queries and Searching with synonyms
4. Our Approach
Two Phases
1. Offline Phase
1. Add Synonyms to documents
2. Cluster the documents, in order to group similar documents into a cluster
3. Index and Label the clusters
Only nouns are indexed.
2. Online Phase
1. Search for clusters as Phrase query
2. Predict words for query augmentation
3. Re-weight the query and suggest top queries as query recommendations
Only Nouns are considered as augmented words
5. Our Approach – Offline Phase
Why Clustering?
1. Clustering improves the scope of suggesting queries for different contexts
2. Documents are clustered together, and indexed
3. Search is performed on cluster index.
4. Relevant clusters are considered to find augmented terms
5. Top N query suggestions from each cluster are considered
Clustering Parameters Used
Algorithm: K-Means
Number of Clusters: 150
6. Our Approach – Offline Phase
Adding Synonyms
1. Allows user to search with synonyms as well
2. Ideally system should accept synonyms and is expected to retrieve same relevant documents
3. Top 5% of words from each document are considered, and synonyms are added to these words
Labeling Clusters
1. Clusters are tagged with most relevant terms
2. Label contain set of terms which can distinguish it from other clusters
7. Our Approach – Online Phase
1. Retrieve relevant Clusters for given input query.
2. Select top ‘N’ Clusters
3. If given query can be represented in N Grams
1. As the words are sequential and from same document, intent of user is clear. Next word in
document can be suggested as augment word
2. Retrieve next sequential word from the cluster, which is set of documents
3. Augment the query with these predicted words, retrieve top queries are present the user as query
recommendations
8. Our Approach – Online Phase
4. Else if the query terms are separated with some distance
1. Predict next word for each input term, add terms to a list.
2. Identify the tags for clusters, and them to list. Tags are words that gives information about the input
words together
3. Here user intent is not clear so, tags which gives category/topic/context of the document are also
considered for augmentation
4. Augment the query with these predicted words, retrieve top queries are present the user as query
recommendations
5. If the given words are far from each other, it is very difficult to co-relate each word
1. Sequential words cannot be used as augment words, the input words may be from different
contexts and is hard to retrieve relevant documents thereby reduces the recall
2. Cluster tags are words which gives some information about the input words together, and are
considered as augmented terms
11. Tools & Data Set Used
Tools
◦ Word Net: Used to identify synonyms
◦ Cluto: Clusters the documents
◦ Doc2Mat: Represents documents in matrix format
◦ Apache Lucene: Used for indexing and querying.
◦ Stanford NLP POS tagger: Identifying Part of speech of word
Data Set
◦ Data set consists of Telegraph Calcutta news paper stories
◦ Stories are categorized into
◦ “FrontPage”, “Nation, “Calcutta”, “Bengal”, “Foreign”, “Business”, “Sports”, “Opinion”, “Metro”
◦ Format of each story in data set is
◦ <DocNo>*</DocNo><Text>*</Text>
12. Evaluation
We have run the augmented queries over the data set to retrieve relevant documents and found
considerable increase in recall and precision values.
Following bar diagram gives change in precision and recall values for random augmented queries
formulated over 30 input queries.
14. Future Work
1. Approach can be extended to implement query logs
◦ Query logs can be used as knowledge base for suggesting queries, and also helps in asynchronous way of
suggesting queries
2. User preferences can be used to filter the documents based on relevancy
3. Different Mixed versions of Markov models can be used to achieve the best balance among accuracy
and coverage both in terms of data (objective) and user (subjective) centric evaluation metrics
4. Different N gram variations can be used to make it ideally suitable for real time Search engine
15. Conclusion
1. We have explained our approach for query expansions, which uses Clustering method to
extend suggestions from various contexts, N gram and Markov model to determine augment
terms
2. We have applied sequential probabilistic model as it is suitable for the task of online query
recommendation
3. Achieved accuracy and coverage in terms of data.
4. Time and memory complexities of our application is measured and found it is suitable for
real time search engine
