Manual debugging is driven by experiments—test runs that narrow down failure causes by systematically confirming or excluding individual factors. The BUGEX approach leverages test case generation to systematically isolate such causes from a single failing test run—causes such as properties of execution states or branches taken that correlate with the failure. Identifying these causes allows for deriving conclusions as: “The failure occurs whenever the daylight savings time starts at midnight local time.” In our evaluation, a prototype of BUGEX precisely pinpointed important failure explaining facts for six out of seven real-life bugs.

1. Isolating Failure Causes
through
Test Case Generation
Jeremias Rößler • Gordon Fraser • Andreas Zeller • Alessandro Orso
Saarland University / Georgia Institute of Technology

2. Joda Time Brazilian Date
public void testBrazil() {
LocalDate date = new LocalDate(2009, 10, 18);
DateTimeZone dtz =
DateTimeZone.forID("America/Sao_Paulo");
Interval interval = date.toInterval(dtz);
}

3. Joda Time Brazilian Date
public void testBrazil() {
LocalDate date = new LocalDate(2009, 10, 18);
DateTimeZone dtz =
DateTimeZone.forID("America/Sao_Paulo");
Interval interval = date.toInterval(dtz);
}

4. Statistical Debugging

5. Statistical Debugging

6. Statistical Debugging

7. Statistical Debugging
Statement ~

8. Statistical Debugging
Statement ~

9. Statement ~ Failure

10. Statement ~ Failure
5.5% – org.joda.time.chrono.BasicGJChronology#<clinit>:58
5.5% – org.joda.time.chrono.ISOChronology#assemble:169
5.5% – org.joda.time.field.UnsupportedDurationField#getInstance:48
5.5% – org.joda.time.DateTimeZone#<clinit>:123
5.5% – org.joda.time.DateTimeZone#<clinit>:130
5.5% – org.joda.time.field.ZeroIsMaxDateTimeField#<init>:46
5.5% – org.joda.time.tz.CachedDateTimeZone#<clinit>:45
5.5% – org.joda.time.chrono.BasicGJChronology#getMonthOfYear:94
5.5% – org.joda.time.chrono.GregorianChronology#getInstance:117
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:425
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:437
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:446
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:509
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:521
1.4% – org.joda.time.DateTimeZone#setNameProvider0:491
1.4% – org.joda.time.DateTimeZone#setProvider0:392

11. Statement ~ Failure
5.5% – org.joda.time.chrono.BasicGJChronology#<clinit>:58
5.5% – org.joda.time.chrono.ISOChronology#assemble:169
5.5% – org.joda.time.field.UnsupportedDurationField#getInstance:48
5.5% – org.joda.time.DateTimeZone#<clinit>:123
5.5% – org.joda.time.DateTimeZone#<clinit>:130
5.5% – org.joda.time.field.ZeroIsMaxDateTimeField#<init>:46
5.5% – org.joda.time.tz.CachedDateTimeZone#<clinit>:45
5.5% – org.joda.time.chrono.BasicGJChronology#getMonthOfYear:94
5.5% – org.joda.time.chrono.GregorianChronology#getInstance:117
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:425
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:437
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:446
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:509
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:521
1.4% – org.joda.time.DateTimeZone#setNameProvider0:491
1.4% – org.joda.time.DateTimeZone#setProvider0:392

12. Statement ~ Failure
5.5% – org.joda.time.chrono.BasicGJChronology#<clinit>:58
5.5% – org.joda.time.chrono.ISOChronology#assemble:169
5.5% – org.joda.time.field.UnsupportedDurationField#getInstance:48
static– {org.joda.time.DateTimeZone#<clinit>:123
5.5%
...
5.5% – org.joda.time.DateTimeZone#<clinit>:130
5.5% – minSum = 0;
long org.joda.time.field.ZeroIsMaxDateTimeField#<init>:46
5.5% – org.joda.time.tz.CachedDateTimeZone#<clinit>:45
for (int i = 0; i < 11; i++) {
5.5% – org.joda.time.chrono.BasicGJChronology#getMonthOfYear:94
long millis =
5.5% – org.joda.time.chrono.GregorianChronology#getInstance:117
1.4% –MIN_DAYS_PER_MONTH_ARRAY[i] *
org.joda.time.DateTimeZone#getDefaultProvider:425
1.4% –MILLIS_PER_DAY;
org.joda.time.DateTimeZone#getDefaultProvider:437
minSum += millis;
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:446
...
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:509
}
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:521
} 1.4% – org.joda.time.DateTimeZone#setNameProvider0:491
1.4% – org.joda.time.DateTimeZone#setProvider0:392

13. Statement ~ Failure
5.5% – org.joda.time.chrono.BasicGJChronology#<clinit>:58
5.5% – org.joda.time.chrono.ISOChronology#assemble:169
5.5% – org.joda.time.chrono.ISOChronology#assemble:169
5.5% – org.joda.time.field.UnsupportedDurationField#getInstance:48
5.5% – org.joda.time.DateTimeZone#<clinit>:123
protected void assemble(Fields fields) {
5.5% – org.joda.time.DateTimeZone#<clinit>:130
if (getBase().getZone() == DateTimeZone.UTC) {
5.5% – org.joda.time.field.ZeroIsMaxDateTimeField#<init>:46
5.5% Use zero based century and year of century.
// – org.joda.time.tz.CachedDateTimeZone#<clinit>:45
fields.centuryOfEra =
5.5% – org.joda.time.chrono.BasicGJChronology#getMonthOfYear:94
5.5% –new DividedDateTimeField(
org.joda.time.chrono.GregorianChronology#getInstance:117
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:425
ISOYearOfEraDateTimeField.INSTANCE,
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:437
DateTimeFieldType.centuryOfEra(), 100);
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:446
...
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:509
}
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:521
} 1.4% – org.joda.time.DateTimeZone#setNameProvider0:491
1.4% – org.joda.time.DateTimeZone#setProvider0:392

14. Statement ~ Failure
5.5% – org.joda.time.chrono.BasicGJChronology#<clinit>:58
5.5% – org.joda.time.chrono.ISOChronology#assemble:169
5.5% – org.joda.time.field.UnsupportedDurationField#getInstance:48
5.5% – org.joda.time.DateTimeZone#<clinit>:123
public org.joda.time.DateTimeZone#<clinit>:130
5.5% – static
UnsupportedDurationField
5.5% – org.joda.time.field.ZeroIsMaxDateTimeField#<init>:46
5.5% – org.joda.time.tz.CachedDateTimeZone#<clinit>:45
getInstance(DurationFieldType type) {
5.5% – org.joda.time.chrono.BasicGJChronology#getMonthOfYear:94
UnsupportedDurationField field;
5.5% – org.joda.time.chrono.GregorianChronology#getInstance:117
if (cCache == null) {
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:425
cCache = new HashMap(7);
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:437
field = null;
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:446
} else {
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:509
field = (UnsupportedDurationField) cCache.get(type);
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:521
}
1.4% – org.joda.time.DateTimeZone#setNameProvider0:491
1.4% – org.joda.time.DateTimeZone#setProvider0:392
...

15. Statement ~ Failure
5.5% – org.joda.time.chrono.BasicGJChronology#<clinit>:58
5.5% – org.joda.time.chrono.ISOChronology#assemble:169
5.5% – org.joda.time.field.UnsupportedDurationField#getInstance:48
5.5% – org.joda.time.DateTimeZone#<clinit>:123
5.5% – org.joda.time.DateTimeZone#<clinit>:130
5.5% – org.joda.time.field.ZeroIsMaxDateTimeField#<init>:46
5.5% – org.joda.time.tz.CachedDateTimeZone#<clinit>:45
5.5% – org.joda.time.chrono.BasicGJChronology#getMonthOfYear:94
5.5% – org.joda.time.chrono.GregorianChronology#getInstance:117
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:425
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:437
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:446
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:509
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:521
1.4% – org.joda.time.DateTimeZone#setNameProvider0:491
1.4% – org.joda.time.DateTimeZone#setProvider0:392

16. Statement ~ Failure
5.5% – org.joda.time.chrono.BasicGJChronology#<clinit>:58
5.5% – org.joda.time.chrono.ISOChronology#assemble:169
5.5% – org.joda.time.field.UnsupportedDurationField#getInstance:48
5.5% – org.joda.time.DateTimeZone#<clinit>:123
5.5% – org.joda.time.DateTimeZone#<clinit>:130
5.5% – org.joda.time.field.ZeroIsMaxDateTimeField#<init>:46
5.5% – org.joda.time.tz.CachedDateTimeZone#<clinit>:45
5.5% – org.joda.time.chrono.BasicGJChronology#getMonthOfYear:94
5.5% – org.joda.time.chrono.GregorianChronology#getInstance:117
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:425
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:437
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:446
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:509
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:521
1.4% – org.joda.time.DateTimeZone#setNameProvider0:491
1.4% – org.joda.time.DateTimeZone#setProvider0:392

17. Statement ~ Failure
5.5% – org.joda.time.chrono.BasicGJChronology#<clinit>:58
5.5% – org.joda.time.chrono.ISOChronology#assemble:169
5.5% – org.joda.time.field.UnsupportedDurationField#getInstance:48
5.5% – org.joda.time.DateTimeZone#<clinit>:123
5.5% – org.joda.time.DateTimeZone#<clinit>:130
5.5% – org.joda.time.field.ZeroIsMaxDateTimeField#<init>:46
5.5% – org.joda.time.tz.CachedDateTimeZone#<clinit>:45
5.5% – org.joda.time.chrono.BasicGJChronology#getMonthOfYear:94
5.5% – org.joda.time.chrono.GregorianChronology#getInstance:117
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:425
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:437
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:446
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:509
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:521
1.4% – org.joda.time.DateTimeZone#setNameProvider0:491
1.4% – org.joda.time.DateTimeZone#setProvider0:392

18. Statement ~ Failure
5.5% – org.joda.time.chrono.BasicGJChronology#<clinit>:58
5.5% – org.joda.time.chrono.ISOChronology#assemble:169
5.5% – org.joda.time.field.UnsupportedDurationField#getInstance:48
5.5% – org.joda.time.DateTimeZone#<clinit>:123
5.5% – org.joda.time.DateTimeZone#<clinit>:130
5.5% – org.joda.time.field.ZeroIsMaxDateTimeField#<init>:46
5.5% – org.joda.time.tz.CachedDateTimeZone#<clinit>:45
5.5% – org.joda.time.chrono.BasicGJChronology#getMonthOfYear:94
5.5% – org.joda.time.chrono.GregorianChronology#getInstance:117
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:425
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:437
1.4% – org.joda.time.DateTimeZone#getDefaultProvider:446
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:509
1.4% – org.joda.time.DateTimeZone#getDefaultNameProvider:521
1.4% – org.joda.time.DateTimeZone#setNameProvider0:491
1.4% – org.joda.time.DateTimeZone#setProvider0:392

19. Statement ~ Failure
0 %0 %0 %
0 %0 %0 %
0 %0 %0 %
0 %0 %0 % 6 % BasicGJChronology#<clinit>:58
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 6 % ISOChronology#assemble:169
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 6 % UnsupportedDurationField#getInstance:48
0 %
0 %
0 %
0 %
0 %
1 %
1 %
1 %
1 % 6 % DateTimeZone#<clinit>:123
1 %
1 %
1 %
1 % 6 % DateTimeZone#<clinit>:130
1 %
1 %
1 %
1 % 6 % ZeroIsMaxDateTimeField#<init>:46
1 %
1 %
1 %
1 % 6 % CachedDateTimeZone#<clinit>:45
1 %
1 %
1 %1 %1 % 6 % 6 % BasicGJChronology#getMonthOfYear:94
... GregorianChronology#getInstance:117

20. Statistical Debugging

21. Two Issues

22. Two Issues
1. Many weak correlations

23. Two Issues
1. Many weak correlations
2. Only location

24. Two Issues
and two solutions
1. Many weak correlations
2. Only location

25. Two Issues
and two solutions
1. Many weak correlations
– strengthen correlations
2. Only location

26. Two Issues
and two solutions
1. Many weak correlations
– strengthen correlations
2. Only location – provide explanations

27. Two Issues
and two solutions
1. Many weak correlations
– strengthen correlations
2. Only location – provide explanations
BugEx

28. Two Issues
and two solutions
1. Many weak correlations
– strengthen correlations
2. Only location – provide explanations
BugEx

29. Test Case Generation
Statement ~

30. Test Case Generation
Statement ~

31. Test Case Generation
stronger!
Statement ~

32. Test Case Generation
stronger!
Statement ~

33. Test Case Generation
directed
stronger!
Statement ~

34. Joda Time Brazilian Date
public void testBrazil() {
LocalDate date = new LocalDate(2009, 10, 18);
DateTimeZone dtz =
DateTimeZone.forID("America/Sao_Paulo");
Interval interval = date.toInterval(dtz);
}

35. Statistical Debugging
0 % 0 %0 %
0 % 0 %0 % 6 %
0 % 0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 6 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 6 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
1 %
1 %
1 %
1 % 6 %
1 %
1 %
1 %
1 % 6 %
1 %
1 %
1 %
1 % 6 %
1 %
1 %
1 %
1 % 6 %
1 %
1 %1 %1 %1 % 6 % 6 %

36. Statistical Debugging
0 % 0 %0 %
0 % 0 %0 % 6 %
0 %
0 %
0 %
0 %
0 %
0 % 0 %
0 % 0 %
…
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 6 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 6 %
0 %
T F 0 %
0 %
0 %
0 %
1 %
1 %
1 %
1 % 6 %
1 %
1 %
1 %
1 % 6 %
T F T F 1 %
1 %
1 %
1 % 6 %
1 %
1 %
…
…
…
…
1 %
1 % 6 %
1 %
1 %1 %1 %1 % 6 % 6 %

37. Statistical Debugging
0 % 0 %0 %
0 % 0 %0 % 6 %
0 %
0 %
0 %
0 %
0 %
0 % 0 %
0 % 0 %
…
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 6 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 6 %
0 %
T F 0 %
0 %
0 %
0 %
1 %
1 %
1 %
1 % 6 %
1 %
1 %
1 %
1 % 6 %
T F T F 1 %
1 %
1 %
1 % 6 %
1 %
1 %
…
…
…
…
1 %
1 % 6 %
1 %
1 %1 %1 %1 % 6 % 6 %

38. Directed Generation
0 % 0 %0 %
0 % 0 %0 % 6 %
0 %
0 %
0 %
0 %
0 %
0 % 0 %
0 % 0 %
…
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 6 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 6 %
0 %
T F 0 %
0 %
0 %
0 %
1 %
1 %
1 %
1 % 6 %
1 %
1 %
1 %
1 % 6 %
T F T F 1 %
1 %
1 %
1 % 6 %
1 %
1 %
…
…
…
…
1 %
1 % 6 %
1 %
1 %1 %1 %1 % 6 % 6 %

39. Directed Generation
0 % 0 %0 %
0 % 0 %0 % 6 %
0 %
0 %
0 %
0 %
0 %
0 % 0 %
0 % 0 %
…
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 6 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 6 %
0 %
T F 0 %
0 %
0 %
0 %
1 %
1 %
1 %
1 % 6 %
1 %
1 %
1 %
1 % 6 %
T F T F 1 %
1 %
1 %
1 % 6 %
1 %
1 %
…
…
…
…
1 %
1 % 6 %
1 %
1 %1 %1 %1 % 6 % 6 %

40. Directed Generation
0 %0 %0 %
0 %0 %0 %
0 %
0 %0 %0 %
0 %
0 %
0 %
0 %
0 %0 %
0 %0 %
…
0 %
0 %
0 %
0 % 0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 % 11 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 6 %
0 %
0 %
0 %
0 %
T F 0 %
0 %
1 %
1 %
1 %
1 % 6 %
1 %
1 %
1 %
1 % 6 %
1 %
T F T F 1 %
1 %
1 % 6 %
1 %
1 %
1 %
…
…
…
…
1 % 6 %
1 %
1 %
1 %
1 %
1 % 6 %
6 % 6 %

41. Directed Generation
0 %0 %0 %
0 %0 %0 %
0 %
0 %0 %0 %
0 %
0 %
0 %
0 %
0 %0 %
0 %0 %
…
0 %
0 %
0 %
0 % 0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 % 11 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 6 %
0 %
0 %
0 %
0 %
T F 0 %
0 %
1 %
1 %
1 %
1 % 6 %
1 %
1 %
1 %
1 % 6 %
1 %
T F T F 1 %
1 %
1 % 6 %
1 %
1 %
1 %
…
…
…
…
1 % 6 %
1 %
1 %
1 %
1 %
1 % 6 %
6 % 6 %

42. Directed Generation
0 %0 %0 %
0 %0 %0 %
0 %0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %0 %
…
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 16 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
T F 1 %
1 %
1 % 5 %
1 %
1 %
1 %
1 %
1 % 5 %
1 %
1 %
T F T F 1 %
1 %
1 % 5 %
1 %
1 %
1 % 5 %
1 %
…
…
…
…
1 %
1 %
1 %
1 % 5 %
5 % 5 %
5 %

43. Directed Generation
0 %0 %0 %
0 %0 %0 %
0 %0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %0 %
…
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 16 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
T F 1 %
1 %
1 % 5 %
1 %
1 %
1 %
1 %
1 % 5 %
1 %
1 %
T F T F 1 %
1 %
1 % 5 %
1 %
1 %
1 % 5 %
1 %
…
…
…
…
1 %
1 %
1 %
1 % 5 %
5 % 5 %
5 %

44. Directed Generation
0 %0 %0 %
0 %0 %0 %
0 %0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %0 %
…
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 16 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
T F 1 %
1 %
1 % 5 %
1 %
1 %
1 %
1 %
1 % 5 %
1 %
1 %
T F T F 1 %
1 %
1 % 5 %
1 %
1 %
1 % 5 %
1 %
…
…
…
…
1 %
1 %
1 %
1 % 5 %
5 % 5 %
5 %

45. Directed Generation
0 %0 %0 %
0 %0 %0 %
0 %0 %0 %
0 %
0 %
0 %0 %
0 %
0 %
0 %
0 %
…
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 17 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
T F 0 %
1 %
1 %
1 % 3 %
1 %
1 %
1 % 6 %
1 %
1 %
1 %
1 %
T F T F 1 % 6 %
1 %
1 %
1 % 6 %
1 %
1 %
…
…
…
…
1 %
1 % 6 %
1 %
1 %
1 %
6 % 6 %
6 %

46. Directed Generation
0 %0 %0 %
0 %0 %0 %
0 %0 %0 %
0 %
0 %
0 %0 %
0 %
0 %
0 %
0 %
…
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 17 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
T F 0 %
1 %
1 %
1 % 3 %
1 %
1 %
1 % 6 %
1 %
1 %
1 %
1 %
T F T F 1 % 6 %
1 %
1 %
1 % 6 %
1 %
1 %
…
…
…
…
1 %
1 % 6 %
1 %
1 %
1 %
6 % 6 %
6 %

47. Directed Generation
0 %0 %0 %
0 %0 %0 %
0 %
0 %0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %0 %
…
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 % 19 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
T F 1 %
1 %
1 %
1 % 1 %
1 %
1 %
1 % 5 %
1 %
1 %
1 %
T F T F 1 %
1 %
5 %
1 %
1 %
1 % 5 %
1 %
…
…
…
…
1 %
1 %
1 % 5 %
1 %
1 %
5 % 5 %
5 %

48. Directed Generation
0 %0 %0 %
0 %0 %0 %
0 %
0 %0 %0 %
0 %
0 %
0 %
0 %
0 %0 %
…
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
T F 0 %
0 %
0 %
0 %
0 %
0 % 37 %
1 %
1 %
1 %
1 %
1 %
1 %
1 %
T F T F 1 %
1 %
1 %
1 %
1 %
1 %
…
…
…
…
1 %
1 % 2 %
1 % 2 %
1 %1 % 2 %
1 %1 % 2 % 2 % 2 %
1 % 2 % 2 %

49. Directed Generation
…
T F
T F T F
…
…
…
…
99.9 %
DateTimeZone#getOffsetFromLocal:864

50. Statement ~ Failure
0.01% org.joda.time.chrono.BasicGJChronology#<clinit>:58
99.9% – org.joda.time.DateTimeZone#getOffsetFromLocal:864
0.01% – org.joda.time.chrono.ISOChronology#assemble:169
0.01% – org.joda.time.field.UnsupportedDurationField#getInstance:48
0.01% – org.joda.time.DateTimeZone#<clinit>:123
0.01% – org.joda.time.DateTimeZone#<clinit>:130
0.01% – org.joda.time.field.ZeroIsMaxDateTimeField#<init>:46
// if the offsets differ, we must be
0.01% – org.joda.time.tz.CachedDateTimeZone#<clinit>:45
0.01% – org.joda.time.chrono.BasicGJChronology#getMonthOfYear:94
// near a DST boundary
5.5% – org.joda.time.chrono.GregorianChronology#getInstance:117
if (offsetLocal != offsetAdjusted) {
0.001% – org.joda.time.DateTimeZone#getDefaultProvider:425
…
0.001% – org.joda.time.DateTimeZone#getDefaultProvider:437
0.001% – org.joda.time.DateTimeZone#getDefaultProvider:446
}
0.001% – org.joda.time.DateTimeZone#getDefaultNameProvider:509
0.001% – org.joda.time.DateTimeZone#getDefaultNameProvider:521
0.001% – org.joda.time.DateTimeZone#setNameProvider0:491
0.001% – org.joda.time.DateTimeZone#setProvider0:392

51. Statement ~ Failure
0.01% org.joda.time.chrono.BasicGJChronology#<clinit>:58
99.9% – org.joda.time.DateTimeZone#getOffsetFromLocal:864
0.01% – org.joda.time.chrono.ISOChronology#assemble:169
0.01% – org.joda.time.field.UnsupportedDurationField#getInstance:48
0.01% – org.joda.time.DateTimeZone#<clinit>:123
0.01% – org.joda.time.DateTimeZone#<clinit>:130
0.01% – org.joda.time.field.ZeroIsMaxDateTimeField#<init>:46
// if the offsets differ, we must be
0.01% – org.joda.time.tz.CachedDateTimeZone#<clinit>:45
0.01% – org.joda.time.chrono.BasicGJChronology#getMonthOfYear:94
// near a DST boundary – failure condition!
5.5% – org.joda.time.chrono.GregorianChronology#getInstance:117
if (offsetLocal != offsetAdjusted) {
0.001% – org.joda.time.DateTimeZone#getDefaultProvider:425
…
0.001% – org.joda.time.DateTimeZone#getDefaultProvider:437
0.001% – org.joda.time.DateTimeZone#getDefaultProvider:446
}
0.001% – org.joda.time.DateTimeZone#getDefaultNameProvider:509
0.001% – org.joda.time.DateTimeZone#getDefaultNameProvider:521
0.001% – org.joda.time.DateTimeZone#setNameProvider0:491
0.001% – org.joda.time.DateTimeZone#setProvider0:392

52. Two Issues
and two solutions
1. Many weak correlations
– strengthen correlations
2. Only location – provide explanations
BugEx

53. Two Issues
and two solutions
1. Many weak correlations
– strengthen correlations
2. Only location – provide explanations
BugEx

54. Directed Generation
0 %0 %0 %
0 %0 %0 %
0 %
0 %0 %0 %
0 %
0 %
0 %
0 %
0 %0 %
…
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
T F 0 %
0 %
0 %
0 %
0 %
0 % 37 %
1 %
1 %
1 %
1 %
1 %
1 %
1 %
T F T F 1 %
1 %
1 %
1 %
1 %
1 %
…
…
…
…
1 %
1 % 2 %
1 % 2 %
1 %1 % 2 %
1 %1 % 2 % 2 % 2 %
1 % 2 % 2 %

55. Directed Generation
0 %0 %0 %
0 %0 %0 %
0 %
0 %0 %0 %
0 %
0 %
0 %
0 %
0 %0 %
…
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
0 %
T F 0 %
0 %
0 %
0 %
0 %
0 % 37 %
1 %
1 %
1 %
Arbitrary runtime features 1 %
1 %
1 %
1 %
T F T F 1 %
1 %
1 %
1 %
1 %
1 %
…
…
…
…
1 %
1 % 2 %
1 % 2 %
1 %1 % 2 %
1 %1 % 2 % 2 % 2 %
1 % 2 % 2 %

56. Directed Generation
…
T F
• Branches
Arbitrary runtime features
T F T F
…
…
…
…

57. Directed Generation
…
T F
• Branches
Arbitrary runtime features • State predicates
T F T F
…
…
…
…

58. Directed Generation
…
T F
• Branches
Arbitrary runtime features • State predicates
T F T F
• Thread schedules
…
…
…
…

59. Directed Generation
…
T F
• Branches
Arbitrary runtime features • State predicates
T F T F
• Thread schedules
• Def-use pairs
…
…
…
…

60. Directed Generation
…
T F
• Branches
Arbitrary runtime features • State predicates
T F T F
• Thread schedules
• Def-use pairs
…
…
…
…
• and more…

61. Directed Generation
…
T F
• Branches
Arbitrary runtime features • State predicates
T F T F
• Thread schedules
• Def-use pairs
…
…
…
…
• and more…

62. State Predicates
• Encode features of objects:
attribute | inspector
<|>|≤|≥|=|≠
attribute | inspector | constant

63. State Predicates
• Encode features of objects:
attribute | inspector
<|>|≤|≥|=|≠
attribute | inspector | constant
• e.g. “shape.area() ≥ 0” or
“square.width = square.height”

64. Base64 Lookup
public void test() {
byte[] byteArray = { -125, -10, 64 };
Base64.isArrayByteBase64(byteArray);
}

65. Base64 Lookup
public void test() {
byte[] byteArray = { -125, -10, 64 };
Base64.isArrayByteBase64(byteArray);
}

66. Base64 Lookup
public void test() {
byte[] byteArray = { -125, -10, 64 };
Base64.isArrayByteBase64(byteArray);
}

67. Base64 Lookup
public void test() {
byte[] byteArray = { -125, -10, 64 };
Base64.isArrayByteBase64(byteArray);
}
arrayOctect.length() == 3

68. Base64 Lookup
public void test() {
byte[] byteArray = { -125, -10, 64 };
Base64.isArrayByteBase64(byteArray);
}
arrayOctect.length() == 3
octect <= 0

69. Base64 Lookup
public void test() {
byte[] byteArray = { -125, -10, 64 };
Base64.isArrayByteBase64(byteArray);
}
octect <= 0

70. Base64 Lookup
public void test() {
byte[] byteArray = { -125, -10, 64 };
Base64.isArrayByteBase64(byteArray);
}
octect <= 0
base64Alphabet[octect]

71. Quantitative Results
Branches Predicates

72. Quantitative Results
Branches Predicates

73. Quantitative Results
Branches Predicates
Brazilian Date bug 2,380 s 1 13,55 s 25
Base64 Lookup bug 38 s 1 737 s 2
Sparse Iterator bug 216 s 8 10,267 s 9
Base64 Decoder bug 214 s 1 1,339 s 23
Western Hemisphere bug 8,422 s 7 30,937 s 9
Vending Machine bug 19 s 1 56 s 1
Parse French Date bug 1,577 s 15 n/a n/a

74. Quantitative Results
Branches Predicates
Brazilian Date bug 2,380 s 1 13,55 s 25
Base64 Lookup bug 38 s 1 737 s 2
Sparse Iterator bug 216 s 8 10,267 s 9
Base64 Decoder bug 214 s 1 1,339 s 23
Western Hemisphere bug 8,422 s 7 30,937 s 9
Vending Machine bug 19 s 1 56 s 1
Parse French Date bug 1,577 s 15 n/a n/a

75. Quantitative Results
Branches Predicates
Brazilian Date bug 2,380 s 1 13,55 s 25
Base64 Lookup bug 38 s 1 737 s 2
Sparse Iterator bug 216 s 8 10,267 s 9
Base64 Decoder bug 214 s 1 1,339 s 23
Western Hemisphere bug 8,422 s 7 30,937 s 9
Vending Machine bug 19 s 1 56 s 1
Parse French Date bug 1,577 s 15 n/a n/a

76. Quantitative Results
Branches Predicates
Brazilian Date bug 2,380 s 1 13,55 s 25
Base64 Lookup bug 38 s 1 737 s 2
Sparse Iterator bug 216 s 8 10,267 s 9
Base64 Decoder bug 214 s 1 1,339 s 23
Western Hemisphere bug 8,422 s 7 30,937 s 9
Vending Machine bug 19 s 1 56 s 1
Parse French Date bug 1,577 s 15 n/a n/a

77. Quantitative Results
Branches Predicates
Brazilian Date bug 2,380 s 1 13,55 s 25
Base64 Lookup bug 38 s 1 737 s 2
Sparse Iterator bug 216 s 8 10,267 s 9
Base64 Decoder bug 214 s 1 1,339 s 23
Western Hemisphere bug 8,422 s 7 30,937 s 9
Vending Machine bug 19 s 1 56 s 1
Parse French Date bug 1,577 s 15 n/a n/a

78. Quantitative Results
Branches Predicates
Brazilian Date bug 2,380 s 1 13,55 s 25
Base64 Lookup bug 38 s 1 737 s 2
Sparse Iterator bug 216 s 8 10,267 s 9
Base64 Decoder bug 214 s 1 1,339 s 23
Western Hemisphere bug 8,422 s 7 30,937 s 9
Vending Machine bug 19 s 1 56 s 1
Parse French Date bug 1,577 s 15 n/a n/a

79. Quantitative Results
Branches Predicates
Brazilian Date bug 2,380 s 1 13,55 s 25
Base64 Lookup bug 38 s 1 737 s 2
Sparse Iterator bug 216 s 8 10,267 s 9
Base64 Decoder bug 214 s 1 1,339 s 23
Western Hemisphere bug 8,422 s 7 30,937 s 9
Vending Machine bug 19 s 1 56 s 1
Parse French Date bug 1,577 s 15 n/a n/a

80. Qualitative Results
Branches Predicates
• Correctbug
Brazilian Date results after2,380seconds 13,55 s 25
20 s 1
in six out of seven cases
Base64 Lookup bug 38 s 1 737 s 2
Sparse Iterator bug 216 s 8 10,267 s 9
Base64 Decoder bug 214 s 1 1,339 s 23
Western Hemisphere bug 8,422 s 7 30,937 s 9
Vending Machine bug 19 s 1 56 s 1
Parse French Date bug 1,577 s 15 n/a n/a

81. Qualitative Results
Branches Predicates
• Correctbug
Brazilian Date results after2,380seconds 13,55 s 25
20 s 1
in six out of seven cases
Base64 Lookup bug 38 s 1 737 s 2
•
Sparse Iterator bug effort 216 s
No additional 8 10,267 s 9
Base64 Decoder bug 214 s 1 1,339 s 23
Western Hemisphere bug 8,422 s 7 30,937 s 9
Vending Machine bug 19 s 1 56 s 1
Parse French Date bug 1,577 s 15 n/a n/a

82. Qualitative Results
Branches Predicates
• Correctbug
Brazilian Date results after2,380seconds 13,55 s 25
20 s 1
in six out of seven cases
Base64 Lookup bug 38 s 1 737 s 2
•
Sparse Iterator bug effort 216 s
No additional 8 10,267 s 9
•
Base64 Decoder bug of branchess reported
Small number 214 1 1,339 s 23
Western Hemisphere bug 8,422 s 7 30,937 s 9
Vending Machine bug 19 s 1 56 s 1
Parse French Date bug 1,577 s 15 n/a n/a

83. Qualitative Results
Branches Predicates
• Correctbug
Brazilian Date results after2,380seconds 13,55 s 25
20 s 1
in six out of seven cases
Base64 Lookup bug 38 s 1 737 s 2
•
Sparse Iterator bug effort 216 s
No additional 8 10,267 s 9
•
Base64 Decoder bug of branchess reported
Small number 214 1 1,339 s 23
• Directly leads to failure19 s 1
Western Hemisphere bug 8,422 s 7
Vending Machine bug
cause
30,937 s
56 s
9
1
in six out of seven cases
Parse French Date bug 1,577 s 15 n/a n/a

84. Future Work
• More runtime features

85. Future Work
• More runtime features
• Improve test case generation

86. Future Work
• More runtime features
• Improve test case generation
• Large-scale quantitative evaluation

87. Future Work
• More runtime features
• Improve test case generation
• Large-scale quantitative evaluation
• User study (like seen yesterday)

93. http://www.st.cs.uni-saarland.de/bugex/

95. Comparing Branches
Statistical
BugEx Debugging

96. Comparing Branches
Statistical
BugEx Debugging

97. Comparing Branches
Statistical
BugEx Debugging
Brazilian Date bug 2,380 s 1 25,223 s 24
Base64 Lookup bug 38 s 1 512 s 17
Sparse Iterator bug 216 s 8 1901 s 14
Base64 Decoder bug 214 s 1 496 s 51
Western Hemisphere bug 8,422 s 7 25,341 s 28
Vending Machine bug 19 s 1 n/a n/a
Parse French Date bug 1,577 s 15 25,542 s 26

98. Comparing Branches
Statistical
BugEx Debugging
Brazilian Date bug 2,380 s 1 25,223 s 24
Base64 Lookup bug 38 s 1 512 s 17
Sparse Iterator bug 216 s 8 1901 s 14
Base64 Decoder bug 214 s 1 496 s 51
Western Hemisphere bug 8,422 s 7 25,341 s 28
Vending Machine bug 19 s 1 n/a n/a
Parse French Date bug 1,577 s 15 25,542 s 26

99. Comparing Branches
Statistical
BugEx Debugging
Brazilian Date bug 2,380 s 1 25,223 s 24
Base64 Lookup bug 38 s 1 512 s 17
Sparse Iterator bug 216 s 8 1901 s 14
Base64 Decoder bug 214 s 1 496 s 51
Western Hemisphere bug 8,422 s 7 25,341 s 28
Vending Machine bug 19 s 1 n/a n/a
Parse French Date bug 1,577 s 15 25,542 s 26

100. Comparing Branches
Statistical
BugEx Debugging
Brazilian Date bug 2,380 s 1 25,223 s 24
Base64 Lookup bug 38 s 1 512 s 17
Sparse Iterator bug 216 s 8 1901 s 14
Base64 Decoder bug 214 s 1 496 s 51
Western Hemisphere bug 8,422 s 7 25,341 s 28
Vending Machine bug 19 s 1 n/a n/a
Parse French Date bug 1,577 s 15 25,542 s 26

101. Comparing Branches
Statistical
BugEx Debugging
Brazilian Date bug 2,380 s 1 25,223 s 24
Base64 Lookup bug 38 s 1 512 s 17
Sparse Iterator bug 216 s 8 1901 s 14
Base64 Decoder bug 214 s 1 496 s 51
Western Hemisphere bug 8,422 s 7 25,341 s 28
Vending Machine bug 19 s 1 n/a n/a
Parse French Date bug 1,577 s 15 25,542 s 26

102. Comparing Branches
Statistical
BugEx Debugging
Brazilian Date bug 2,380 s 1 25,223 s 24
Base64 Lookup bug 38 s 1 512 s 17
Sparse Iterator bug 216 s 8 1901 s 14
Base64 Decoder bug 214 s 1 496 s 51
Western Hemisphere bug 8,422 s 7 25,341 s 28
Vending Machine bug 19 s 1 n/a n/a
Parse French Date bug 1,577 s 15 25,542 s 26

103. Comparing Branches
Statistical
BugEx Debugging
Brazilian Date bug 2,380 s 1 25,223 s 24
Base64 Lookup bug 38 s 1 512 s 17
Sparse Iterator bug 216 s 8 1901 s 14
Base64 Decoder bug 214 s 1 496 s 51
Western Hemisphere bug 8,422 s 7 25,341 s 28
Vending Machine bug 19 s 1 n/a n/a
Parse French Date bug 1,577 s 15 25,542 s 26

104. Performance
Branches
Minutes

105. Joda Time
5.2%
Brazilian Date Test
94.8%
23.7%
3,497 Tests
76.3%

106. Commons Codec
1.7%
Base64 Lookup Test
98.3%
14.2%
185 Tests
85.8%