Adaptive Bug Prediction By Analyzing Software History

1. Adaptive Bug Prediction By Analyzing Project History Dissertation Defense, Aug 21, 2006 Sunghun Kim <hunkim@cs.ucsc.edu> University of California, Santa Cruz

17. Commits, Transactions & Configurations transactions configurations CVS file commits Added feature X Fixed null ptr bug Modified button text Added feature Y log message

18. Kenyon Processing SCM Repository Filesystem Extract Automated configuration extraction Save Persist gathered metrics & facts Kenyon Repository (RDBMS/ Hibernate) Analyze Query DB, add new facts Analysis Software (e.g., IVA) Compute Fact extraction (metrics, static analysis) Kenyon

20. Creating Corpus – Identifying bug-introducing changes Development history of foo.java SCM log message: “Bug #567 fixed” “ bug-fix ” Bug #567 entered into issue tracking system (bug finally observed and recorded)

21. Creating Corpus – Identifying bug-introducing changes Bug #567 entered into issue tracking system (bug finally observed and recorded) Software change that introduces the bug “ buggy ” … . … . … . … . … . … . Bug-fix change

22. Creating Corpus – Identifying bug-introducing changes buggy Foo rev 1 Foo rev 2 Foo rev 3 Foo rev 4 Foo rev 5 Foo rev 6

23. Creating Corpus – Identifying bug-introducing changes buggy Foo rev 1 Foo rev 2 Foo rev 3 Foo rev 4 Foo rev 5 Foo rev 6 clean

24. Creating Corpus – Analyzed projects SVN SCM C Subversion (SVN) SVN Bug tracker Java Scarab (SCA) CVS DBMS C/C++ PostgreSQL (POS) SVN Content management Python Plone (PLO) CVS Java Development/IDE Java Eclipse JDT (ECL) CVS Web browser C/C++/JS/XML Mozilla (MOZ) CVS Editor Java Jedit (JED) CVS Collaborative development PHP GForge (GFO) CVS Instance messenger C/C++ Gaim (GAI) CVS Mail client Java Columba (COL) CVS Bug tracker Perl Bugzilla (BUG) SVN HTTP server C Apache HTTP 1.3 (A1) SCM Software type Language Project

25. Bug Cache

33. (1) check/miss (2) fetch Cache Operation/Evaluation 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 Bug cache (1) check/miss (2) fetch 1 (1) check/miss (2) fetch 9 Bug cache 1 9 (1) check/ hit! (1) check/miss (2) fetch 2 (1) check/ hit! 1 2 9 (1) check/ hit! (3) replace 3 1 2 3 Bug cache: Bug prone entity list Bug-introducing change

34. Results - File Level Predictive accuracy at file level, 10% cache size

35. Results - Function/Method Level Predictive accuracy at function/method level, 10% cache size

36. 10 % cached entity LOC/total LOC About 16~33% of LOC

37. Various Cache Size and Hit Rates (no pre-fetch)

38. Cache Replacement File level: LRU or BUG Function/method level: mostly BUG

40. Future Work – Example IDE integration

41. Change Classification

42. When we introduce bugs? Foo rev 1 Foo rev 2 Foo rev 3 Foo rev 4 ……

43. When we introduce bugs? Foo rev 1 Foo rev 2 Foo rev 3 Foo rev 4 ……

44. Basic Idea Foo rev 1 Foo rev 2 Foo rev 3 Foo rev 4 Foo rev n Foo rev n-1 …… ?

49. Male/Female Example M/F height weight male 6.5 160 female 5.6 120 female 5.8 130 male 6.0 170 label feature

50. Male/Female Example M/F height weight male 6.5 160 female 5.6 120 female 5.8 130 male 6.0 170 label feature 6.1 150 ?

51. Male/Female Example M/F height weight male 6.5 160 female 5.6 120 female 5.8 130 male 6.0 170 label feature 6.1 150 ? height female male weight

52. Male/Female Example M/F height weight male 6.5 160 female 5.6 120 female 5.8 130 male 6.0 170 label feature 6.1 150 ? height female male weight

53. Male/Female Example M/F height weight male 6.5 160 female 5.6 161 female 5.8 130 male 6.0 170 label feature 6.1 165 ? height female male weight ?

54. Extracting Features Author : hunkim Check-in time : March 23, 2006 11:30 AM Log message : Added foo function … . … . … . … . … . … . Rev 10 Rev 11 /src/jdt/core/ast/ASTParser.java

56. Projects Analyzed 150,940 33.2* 3,518 8,272 N/A N/A Total 14,856 13.0 288 1,925 01/2002-03/2002 500-1000 SVN 5,710 50.5 366 358 06/2001-08/2001 500-1000 SCA 23,247 24.2 273 853 11/1996-02/1997 500-1000 POS 6,127 19.6 112 457 07/2002-02/2003 500-1000 PLO 16,192 10.1 67 592 10/2001-11/2001 500-750 ECL 13,648 29.9 169 395 08/2003-08/2004 500-1000 MOZ 13,879 37.5 377 626 08/2002-03/2003 500-750 JED 8,996 49.6 334 339 01/2003-03/2004 500-1000 GFO 9,281 37.8 451 742 08/2000-03/2001 500-1000 GAI 17,411 29.4 530 1,270 05/2003-09/2003 500-1000 COL 10,148 73.7 417 149 03/2000-08/2001 500-1000 BUG 11,445 23.6 134 566 10/1996-01/1997 500-1000 A1 # of features % of buggy changes # of buggy changes # of clean changes Period Revisions. Project

57. Extracting Features

59. 10-fold-cross validation 1 2 3 4 5 6 7 8 9 10 Corpus

60. 10-fold-cross validation 1 2 3 4 5 6 7 8 9 10 T est set Training set

61. 10-fold-cross validation 1 2 3 4 5 6 7 8 9 10 T est set Training set

62. 10-fold-cross validation 1 2 3 4 5 6 7 8 9 10 T est set Training set

63. 10-fold-cross validation 1 2 3 4 5 6 7 8 9 10 T est set Training set

66. Eclipse/Mozilla Accuracy by Feature Combination A - Added delta D - Deleted delta F - Directory/file name L - Change log N - New source code M - Metadata C - Complexity metrics ~X - all but X

67. Average Accuracy by Feature Combination A - Added delta D - Deleted delta F - Directory/file name L - Change log N - New source code M - Metadata C - Complexity metrics ~X - all but X

68. Best Accuracy Feature Combination A - Added delta D - Deleted delta F - Directory/file name L - Change log N - New source code M - Metadata C - Complexity metrics ~X - all but X

69. Important Features for Some Projects

70. Important Features for Some Projects

71. Why ‘author’ is not a significant feature? One revision (author) can have buggy and clean changes at the same time

87. Adaptive Bug Prediction By Analyzing Project History Dissertation Defense, Aug 21, 2006 Sunghun Kim <hunkim@cs.ucsc.edu> University of California, Santa Cruz