An Assessment of Test-Driven ReusePromises and PitfallsMehrdad NurolahzadeRobert J. WalkerFrank Maurer{mnurolah, walker, m...
Test-Driven Development (TDD)An Assessment of Test-Driven Reuse 1Feature EFeature BFeature AFeature FFeature DFeature CFea...
Test-Driven Development Reuse (TDR)Developing Locating and reusing sourcecode through the provision of test casesdescribin...
An Assessment of Test-Driven Reuse 3Source: http://stackoverflow.com/questions/663374/java-ordered-map
TDR ProcessAn Assessment of Test-Driven Reuse 4Test CaseDeveloperFeature✔✖✖✖????Test CaseSource CodeRepository➊➋➌➍
Related WorkBehavior Sampling (Podgurski and Pierce)Test-Driven Reuse– Code Conjurer (Hummel et al.)– CodeGenie(Lemos et a...
Test-Driven ReuseToolTDR Process: Current ToolsAn Assessment of Test-Driven Reuse 6PotentialCandidatesTest CaseInterfaceTr...
• Are evaluation functions realistic?• Why arbitrary functions cannot be found?• What attributes of evaluation functions m...
Experiment Overview• 10 realistic TDR tasks– Developer forums– Programming tutorials– Example source code catalogs• Soluti...
Measures• Relevance:the extent of the features satisfied• Effort:the amount of work anticipated to adapt• Why not quantita...
ResultsTask Code Conjurer CodeGenie S61 ~ ✓ ✕2 ✕ ~ ✓3 ~ ✕ ✕4 ✕ ✕ ✕5 ✕ ✕ ✕6 ✕ ✕ ✕7 ✕ ✕ ✕8 ✕ ✕ ✕An Assessment of Test-Driven...
External Validation• Have the subjective scores been assigned fairly?• A random subset (12 out of 109 recommendations)• 5 ...
An Assessment of Test-Driven Reuse 12Source: http://stackoverflow.com/questions/663374/java-ordered-map
Why TDR Tools Do NOT Work• A developer may not always provide preciseinput.• TDR filters candidates based on lexical andsy...
An Assessment of Test-Driven Reuse 14
Why TDR Tools Do NOT Work• TDR evaluation tasks seek common variations– Easy to retrieve• Variants expose the same interfa...
Why TDR Tools Do NOT Work• Compiling and Running Source Code– Not working as intended• Difficult to automate– Resolving de...
An Assessment of Test-Driven Reuse 17
Implications for Future Research• Leveraging contextual facts– Helper types– Constraints: pre and post conditions– Data fl...
Implications for Future Research• Pipeline similarity matching• Multi-indexingAn Assessment of Test-Driven Reuse 19f1 f2 f...
Conclusion• Test cases can express a variety of structureand semantics that TDR can utilize.• Current TDR prototypes put t...
Prochain SlideShare
Chargement dans…5
×

An Assessment of Test-Driven Reuse: Promisses and Pitfalls

452 vues

Publié le

ICSR 2013 research paper presentation

Publié dans : Technologie, Business
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

An Assessment of Test-Driven Reuse: Promisses and Pitfalls

  1. 1. An Assessment of Test-Driven ReusePromises and PitfallsMehrdad NurolahzadeRobert J. WalkerFrank Maurer{mnurolah, walker, maurer}@ucalgary.caUniversity of Calgary13th International Conference of Software ReusePisa, Italy20 June 2013
  2. 2. Test-Driven Development (TDD)An Assessment of Test-Driven Reuse 1Feature EFeature BFeature AFeature FFeature DFeature CFeature GNew FeatureNewFeature’sTest
  3. 3. Test-Driven Development Reuse (TDR)Developing Locating and reusing sourcecode through the provision of test casesdescribing the behavior of the feature ofinterest.An Assessment of Test-Driven Reuse 2
  4. 4. An Assessment of Test-Driven Reuse 3Source: http://stackoverflow.com/questions/663374/java-ordered-map
  5. 5. TDR ProcessAn Assessment of Test-Driven Reuse 4Test CaseDeveloperFeature✔✖✖✖????Test CaseSource CodeRepository➊➋➌➍
  6. 6. Related WorkBehavior Sampling (Podgurski and Pierce)Test-Driven Reuse– Code Conjurer (Hummel et al.)– CodeGenie(Lemos et al.)– S6 (Reiss)An Assessment of Test-Driven Reuse 5A few hundredexecutablesMillions of sourcefiles
  7. 7. Test-Driven ReuseToolTDR Process: Current ToolsAn Assessment of Test-Driven Reuse 6PotentialCandidatesTest CaseInterfaceTransformedCandidatesCompiledCandidatesResultsExtractSearchCompileTestTransformDisplayWrite
  8. 8. • Are evaluation functions realistic?• Why arbitrary functions cannot be found?• What attributes of evaluation functions makesthem retrievable by TDR tools?An Assessment of Test-Driven Reuse 7Study Motivation
  9. 9. Experiment Overview• 10 realistic TDR tasks– Developer forums– Programming tutorials– Example source code catalogs• Solutions were confirmed to be in the toolrepositories.• Top 10 recommendations were qualitativelyanalyzed.An Assessment of Test-Driven Reuse 8
  10. 10. Measures• Relevance:the extent of the features satisfied• Effort:the amount of work anticipated to adapt• Why not quantitative measures?– Precision vs. accuracy– How poor vs. why are they poorAn Assessment of Test-Driven Reuse 9
  11. 11. ResultsTask Code Conjurer CodeGenie S61 ~ ✓ ✕2 ✕ ~ ✓3 ~ ✕ ✕4 ✕ ✕ ✕5 ✕ ✕ ✕6 ✕ ✕ ✕7 ✕ ✕ ✕8 ✕ ✕ ✕An Assessment of Test-Driven Reuse 10✓ Good ~OK ✕ Bad
  12. 12. External Validation• Have the subjective scores been assigned fairly?• A random subset (12 out of 109 recommendations)• 5 participants• Example and guidelines• Inter-rater reliability (Spearman’s ρ)An Assessment of Test-Driven Reuse 11ParticipantP1 P2 P3 P4 P5Relevance 0.86 0.89 0.93 0.84 0.81Effort 0.72 0.75 0.82 0.74 0.72
  13. 13. An Assessment of Test-Driven Reuse 12Source: http://stackoverflow.com/questions/663374/java-ordered-map
  14. 14. Why TDR Tools Do NOT Work• A developer may not always provide preciseinput.• TDR filters candidates based on lexical andsyntactic similarity.• TDR expects the developer to provide preciseinput.An Assessment of Test-Driven Reuse 13
  15. 15. An Assessment of Test-Driven Reuse 14
  16. 16. Why TDR Tools Do NOT Work• TDR evaluation tasks seek common variations– Easy to retrieve• Variants expose the same interface– But behave differently• TDR fails at retrieving variants of commonfunctions.An Assessment of Test-Driven Reuse 15
  17. 17. Why TDR Tools Do NOT Work• Compiling and Running Source Code– Not working as intended• Difficult to automate– Resolving dependencies– Provision of runtime environment or resources– AdaptationAn Assessment of Test-Driven Reuse 16
  18. 18. An Assessment of Test-Driven Reuse 17
  19. 19. Implications for Future Research• Leveraging contextual facts– Helper types– Constraints: pre and post conditions– Data flow– Control flowAn Assessment of Test-Driven Reuse 18
  20. 20. Implications for Future Research• Pipeline similarity matching• Multi-indexingAn Assessment of Test-Driven Reuse 19f1 f2 f3f1f2f3BigIndexIndex1 Index2 Index3
  21. 21. Conclusion• Test cases can express a variety of structureand semantics that TDR can utilize.• Current TDR prototypes put too muchemphasis on lexical and syntactic similarity.• Future TDR research should leverage otheraspects of test cases.An Assessment of Test-Driven Reuse 20

×