1. Are Automated Debugging Techniques Actually Helping Programmers? Chris Parnin Georgia Tech @chrisparnin (twitter) Alessandro (Alex) Orso Georgia Tech @alexorso(twitter)

2. Finding bugs can be hard…

3. Automated debugging to the rescue! I’ll help you find location of bug!

4. How it works (Ranking-Based) I have calculated most likely location of bug! Give me a failing program. Calculating… Here is your ranked list of statements.

5. How it works (Ranking-Based) I have calculated most likely location of bug! Give me input. But how does a programmer usea ranked list of statements? Calculating… Here is your rankedlist of statements.

6. Conceptual Model 1) 2) 3) 4) … Here is a list of places to check out Ok, I will check out your suggestions one by one.

7. Conceptual Model 1) 2) 3) 4) … Found the bug!

8. A Skeptic Does the conceptual model make sense? Have we evaluated it?

9. Let’s see… Over 50 years of researchonautomated debugging. 2001. Statistical Debugging 1999. Delta Debugging 1981. Weiser. Program Slicing 1962. Symbolic Debugging (UNIVAC FLIT)

10. Did you see anything?

13. Assumption #1: Perfect bug understanding must also exist when using automated tool. Do you see a bug?

14. Assumption #2 Programmer inspects statements linearly and exhaustively until finding bug. Is this realistic?

15. How can we evaluate the conceptual model? How could we measure the benefit of an automated tool?

16. Conceptual model: What if gave a developer a list of statements to inspect? How would they use the list? Would they be able to see the bug after visiting it? Is ranking important?

17. Benefit: What if we evaluate programmers with and without automated debuggers? > ? We also could observe what works and what doesn’t.

18. Study Setup 34 Developers 2 Debugging Tasks Automated debugging tool

19. Study Setup Participants: 34 developers MS/Phd Students Different levels of expertise (low,medium,high)

20. Study Setup Software subjects: Tetris (2.5 kloc) NanoXML (4.5 kloc)

21. Study Setup 21 Tools: Traditional debugger Eclipse ranking plugin (logged activity)

22. Study Setup Tasks: Debugging fault 30 minutes per task Questionnaire at end

23. Bugs Bug #1: Pressing rotate key causes square figure to move up!

24. Bugs When running the NanoXML program (main is in class Parser1_vw_v1), the following exception is thrown: Exception in thread "main" net.n3.nanoxml.XMLParseException: XML Not Well-Formed at Line 19: Closing tag does not match opening tag: `ns:Bar' != `:Bar' at net.n3.nanoxml.XMLUtil.errorWrongClosingTag(XMLUtil.java:497) at net.n3.nanoxml.StdXMLParser.processElement(StdXMLParser.java:438) at net.n3.nanoxml.StdXMLParser.scanSomeTag(StdXMLParser.java:202) at net.n3.nanoxml.StdXMLParser.processElement(StdXMLParser.java:453) at net.n3.nanoxml.StdXMLParser.scanSomeTag(StdXMLParser.java:202) at net.n3.nanoxml.StdXMLParser.scanData(StdXMLParser.java:159) at net.n3.nanoxml.StdXMLParser.parse(StdXMLParser.java:133) at net.n3.nanoxml.Parser1_vw_v1.main(Parser1_vw_v1.java:50) The input, testvm_22.xml, contains the following input xml document: <Foo a=”test”> <ns:Bar> <Blah x=”1” ns:x=”2”/> </ns:Bar> </Foo> Bug #2: Exception on input xml document.

25. Study Setup: Groups

26. Study Setup: Groups 26 A B

27. Study Setup: Groups C D Rank Rank

28. Results

29. How do developers use a ranked list? Lowperformers did follow list. Survey says searched through statements. 37% of visits jumped avg. 10. Navigation pattern zig-zagged (avg. 10 zigzags)

30. Is perfect bug understanding realistic? Only 1 out of 10 programmers who clicked on bug stopped investigation. The others spent on average ten minutes continuing investigation.

31. Are automated toolsspeeding up debugging? = Traditional Automated group No ✘

32. Are automated toolsspeeding up debugging? No ✘ = Traditional Automated group Rank No = ✘ Traditional Automated group

33. Are automated toolsspeeding up debugging? = Traditional Automated group No ✘

34. But… Stratifying Participants High Performers Medium Performers Low Performers ✔ ✔ ✘ ✔ ✘ ✘

35. Significant difference for “experts” High Performers On average, 5 minutes faster ✔ ✔

36. Are automated toolsspeeding up debugging? No ✘ = Traditional Automated group Yes! ✔ Experts Experts > Traditional Automated group

37. Observations Developers searched through statements. Developers without tool fixed symptoms (not problem). Developers wanted explanations rather than recommendations.

38. Future directions

39. Moving beyond fault space reduction 39 We can keep building better tools. But we can’t keep abstracting away the human.

40. Performing further studies Does different granularity work better for inspection? Documents? Methods? How does different interfaces or visualizations impact technique? Do other automated debugging techniques fare any better? 40

41. How do developers use a ranked list? Is perfect bug understanding realistic? Are Automated Debugging Tools Helpful? Human studies, human studies, human studies!

42. 64,000,000 miles 800,000 miles 35 years of Scientific Progress 1969 2004 42

43. 352 LOC (median 8 programs) 63.5 LOC (median 4 programs) 30 years 30 years of Scientific Progress 1981 2011 43