WCRE08.ppt

221 vues

Publié le

Publié dans : Technologie
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

WCRE08.ppt

  1. 1. Prereqir: Recovering Pre-Requirements via Cluster Analysis Jane Huffman Hayes, Giuliano (Giulio) Antoniol and Yann-Gaël GuéhéneucWCRE 2008, Antwerp 1/21
  2. 2. Content Problem Statement PREREQUIR Idea PREREQUIR Process Technologies WEB Browser Requirements Case Study Results ConclusionsWCRE 2008, Antwerp 2/21
  3. 3. The Challenge A few years after deployment, the RS may no longer exist. If it exists, it will be almost surely outdated. My customers may desire new functionalities or technologies that my system may or may not implement. I poll my stakeholders: programmers, managers, testing team members, marketing personnel, and end users;WCRE 2008, Antwerp find out what they believe the system should do. 3/21
  4. 4. PREREQIR in Essence We need a pre-requirement document: what the competitor systems do; what our customer base needs. Obtain and vet a list of requirements from diverse stakeholders. Structure requirements by mapping them into representation suitable for grouping via pattern- recognition and similarity-based clustering. Analyze clustered requirements to divide them into set of essential and set of optionalWCRE 2008, Antwerp requirements. 4/21
  5. 5. The PREREQUIR Process Textual documents Vector space/clustering 001010 browser support zoom unzoom page detail rp Requirements rp Recovered PRI ri and oj Browser support print 101110 Split, Stop-word Removal, Clustering Labelling Stemming TF-IDF PAM/AGNES Clusters Tokenization Requirements ri ri 1 2 3 4WCRE 2008, Antwerp 5/21
  6. 6. PREREQUIR Technology Standard information retrieval vector space model. Indexing process: Stopper; Stemmer; Thesaurus (not vital but helps); TF-IDF indexing. Clustering PAM and AGNES.WCRE 2008, Labeling: still an open question. Antwerp 6/21
  7. 7. Step 1 – Collect Stakeholders RS By means of questionnaires, collect stakeholders requirements. We favor a non-intrusive lightweight approach such as a WEB based questionnaire. Minimize the risk of influencing stakeholder. There is risk that: he/she did not really understand the task; the granularity and level is very different between respondents; the respondent population is not heterogeneous enough; the sample size is small.WCRE 2008, Antwerp 7/21
  8. 8. Step 2 – Vector Space Mapping The goal is to group single requirements by different users into clusters representing the same functionality/concept. By means of standard IR tools, map the collected requirements into a vector space. Stopper, stemmer, and TDF/IDF plus thesaurus expansions: certain stakeholders may use cryptic terms such as RFC or test/benchmark acronyms.WCRE 2008, Antwerp 8/21
  9. 9. Step 3 – Clustering Transform similarity into a distance. Apply robust partition around medoids. Estimate the number of clusters (different requirements) silhouette: . >0.70 very strong structure 0.50 … 0.70 reasonable structure b(i ) − a(i ) 0.25 … 0.50 weak structure s (i ) = max{a(i ), b(i )} < 0.25 no structure. a(i) average distance to the other PRI in the cluster; b(i) is the average distance to PRI in the nearest cluster.WCRE 2008, Take the flex close to max value of the average Antwerp silhouette. 9/21
  10. 10. Step 3 Bis – Tree Structure If there is a weak structure, check for a requirement tree organization. Re-cluster with AGNES. Compute the Agglomerative Coefficient (AC). AC measures the strength of the hierarchical structure discovered. AC > 0.9 a very strong hierarchical structure.WCRE 2008, Impose a threshold on the average similarity to Antwerp avoid grouping “too different” things. 10/21
  11. 11. Step 4 – Label Clusters Process each PRI of a cluster: stopping, stemming; build cluster-specific dictionary; weight each word by its frequency in the cluster: If a word is in all the PRI in a cluster, its weight is 1.00. If a word appears in half of the PRI, its weight is 0.50. For a given stemmed PRI, calculate a score: sum up the weights of the stems present in the cluster dictionary to obtain a positive weight; count the number of words in the cluster-specific dictionary that are absent in the current PRI: obtain a negative weight. Assign a score to the PRI computed as: the ratio positive weight / negative weights. Label the cluster:WCRE 2008, Antwerp take the PRI with the highest score. 11/21
  12. 12. Case Study Mimic the recovery process for a Web browser. Pool via e-mail to a set of users (about 200). 25 answers out of which we kept 22, overall 433 user needs: mostly male (20), age varies, average 36, standard deviation 9.5; respondents: 10 researchers, five lecturers/professors, four students, one programmer, and two project managers.WCRE 2008, Antwerp 12/21
  13. 13. PAM - AGNES We did not find a strong or evident cluster structure: silhouette about 0.26; region between 167 – 170 cluster: say 170 clusters or less. AGNES reports a strong structure: AC above 0.9. Grouping via AGNES grows a tree starting from leavesWCRE 2008, Antwerp 13/21
  14. 14. Outliers Setting a cluster internal similarity threshold decides top level clusters singleton clusters - outliers inner nodes The “non kept” are also important: single user needs; more expert users may use acronyms must comply with ACID2 “too generic: sentences: it should be fast.WCRE 2008, Antwerp 14/21
  15. 15. 15/21 Antwerp WCRE 2008, 1000 100 200 300 400 500 600 700 800 900 0 Threshold 0.035 0.075 0.115 0.155 0.195 0.235 0.275 0.315 0.355 0.395 0.435 0.475 0.515 0.555 0.595 0.635 0.675 0.715 0.755 0.795 0.835 0.875 0.915 Tops Overall Leaves Outliers Intermediate AGNES Clusters
  16. 16. Manual Verification Two people reviewed cluster and cluster labeling. IR measures precision and recall. Precision measures the quality of the clusters. A conservative approach: “Yes” was assigned if both authors said “Yes”; “No” was assigned if one of the authors said “No”; “Maybe” was assigned in the other cases.WCRE 2008, Antwerp 16/21
  17. 17. Precision Recall – 0.36 128 Common User Needs, 181 Outliers 1 0,8 Precision 0,6 Recall Percentage of Outliers 0,4 0,2 0 Threshold 0,645 0,525 0,585 0,885 0,705 0,765 0,825 0,285 0,225 0,345 0,405 0,465WCRE 2008, Antwerp 17/21
  18. 18. Traceability Task PRI for a Web browser provided: Web site: www.learnthenet.com. There are 20 LtN PRI: textual PRI ranging from 5 to 73 words, having on average 23.5 words. LtN10: “The toolbar should include a Reload or Refresh button to load the web page again.” Trace via vector space retrieval with tf-idf.WCRE 2008, Antwerp Similarity threshold of 0.20. 18/21
  19. 19. Manual Evaluation by Two Authors 14 of the 20 LtN PRI are traced: the 14 PRI were all marked as “Yes” by both authors. If we also include the two marked as “Maybe” there are 16 LtN PRI out of 20 traced. Overall, between 70% (“Yes” only) and 80% (“Yes” and “Maybe”) of the LtN PRI are also found in the PRI obtained from the respondents.WCRE 2008, Antwerp 19/21
  20. 20. Threats to Validity External validity: only one system and 22 answers out of 200, impact of vocabulary is not known. Construct validity: computation performed using widely adopted toolsets, other tool can produce different results. Reliability validity: material will be made available.WCRE 2008, Internal validity: subjectivity introduced by Antwerp experts, “Yes” if and only if both agrees. 20/21
  21. 21. Conclusion AGNES clusters PRI with an accuracy of 70%. A similarity threshold of about 0.36, about 55% of the PRI were common to two or more stakeholders and 42% were outliers: 128 – 181. We automatically label the common and outlier PRI with 82% of the labels being correct. The method achieves roughly 70% recall andWCRE 2008, 70% precision when compared to a ground truth. Antwerp 21/21

×