1. Automated Unit Test Generation
Vassil Popovski, VMware

4. The Cost of Software Bugs?
$59.9 B per year in USA (NIST, 2002)*
$312 B per year worldwide (Cambridge University, 2012)**
(*) http://www.nist.gov/director/planning/upload/report02-3.pdf
(**) http://www.roguewave.com/DesktopModules/Bring2mind/DMX/Download.aspx?
entryid=1606&command=core_download&PortalId=0&TabId=607

5. The Cost of Software Bugs?
$312,000,000,000

6. The Cost of Software Bugs?

7. Total market capitalization for all 3
companies(*):
!
$177.36 B
(*) Google finance - as of Nov 2nd, 2013

8. The Cost of World Hunger?
$40B per year(*)
(*) According to United Nations

10. The Cost of Fixing Bugs?
http://www.amazon.com/Assessments-Benchmarks-Addison-Wesley-Information-Technology/dp/
0201485427?ie=UTF8&s=books&qid=1209056706&sr=1-1

11. Software Testing Pyramid
E2E/GUI testing:
1-5%
Integration testing: 4-15%
Unit Testing:
http://blogs.agilefaqs.com/2011/02/01/inverting-the-testing-pyramid/
80-90%

12. Techniques/Tools
The goal: High (close to 100% code coverage*)
1. Random data generation
2. Search based
3. Symbolic/Concolic based

13. Anatomy of a Unit Test
Arrange
Act
Assert

14. Anatomy of a Unit Test
Arrange Data
Arrange Mocks/Stubs
Act
Assert

15. Anatomy of a Unit Test

16. Evaluation of Tools
Arrange/
Generate Data
Arrange/
Generate
Mocks/Stubs
Act/Invoke
methods
Assert/Verify
the results

17. Techniques/Tools
x != 100
x>=z
x == 100
x<z

18. Techniques/Tools
Random data generation:
!
x != 100
x == 100
Feed x,y with random data
x>=z
x<z

19. Demo

20. Evaluation of Tools
junit-quickcheck
feed4JUnit
Arrange/
Generate Data
Arrange/
Generate
Mocks/Stubs
Act/Invoke
methods
Assert/Verify
the results
yes - some
no
no
no

21. Techniques/Tools
1. Random Data Generation:
• junit-quickcheck (Java)
https://github.com/pholser/junit-quickcheck/
• Feed4JUnit & Feed4TestNG + Benerator (java)
http://databene.org/feed4-tools.html
http://databene.org/databene-benerator.html
• Randoop (Java + .NET)
https://code.google.com/p/randoop/
• T2 (Java)
https://code.google.com/p/t2framework/

22. Techniques/Tools
Search based approach:
!
x != 100
x == 100
1. Start with random/initial
data (x=1, y=1)
x>=z
x<z

23. Techniques/Tools
x=1, y=1
Search based approach:
!
x != 100
x == 100
1. Start with random/initial
data (x=1, y=1)
x>=z
x<z

24. Techniques/Tools
x=1, y=1
Search based approach:
!
x != 100
x == 100
1. Start with random/initial
data (x=1, y=1)
x>=z
x<z

25. Techniques/Tools
x=100, y=1
Search based approach:
!
1. Start with random/initial
data (x=1, y=1)
2. Based on coverage
information - select new
value(s) (x==100)
x != 100
x>=z
x == 100
x<z

26. Techniques/Tools
x=100, y=1
Search based approach:
x != 100
!
1. Start with random/initial
data (x=1, y=1)
2. Based on coverage
information - select new
value(s) (x==100)
!
x>=z
x == 100
x<z

27. Demo

30. Evaluation of Tools
junit-quickcheck
feed4JUnit
Arrange/
Generate Data
Arrange/
Generate
Mocks/Stubs
Act/Invoke
methods
Assert/Verify
the results
yes - some
EvoSuite
yes
no
no
no
yes
no
yes

31. Techniques/Tools
2. Search Based:
• EvoSuite (Java)
http://www.evosuite.org/
• Testful (Java)
https://code.google.com/p/testful/

32. Techniques/Tools
x=X, y=Y, z=2*Y
Symbolic based approach:
!
x != 100
x == 100
1. Evaluate path conditions (PC)
PC: X!=100, Y
PC: X == 100, Y
on every branch
2. For each leaf - calculate X & Y
x>=z
x<z
using a constraints solver:
1. X!=100, Y
=> (x=1, y=1)
2. X=100, X>=2*Y =>(x=100, y=1)
3. X=100, X<2*Y =>(x=100, y=51)
PC: X == 100, PC: X == 100,
X>=2*Y
X<2*Y

33. Techniques/Tools
Concolic based
approach:
x != 100
x == 100
!
Optimization of symbolic
execution - but use real
values for some of the
branches
x>=z
x<z

34. Demo

35. Evaluation of Tools
junit-quickcheck
feed4JUnit
Arrange/
Generate Data
Arrange/
Generate
Mocks/Stubs
Act/Invoke
methods
Assert/Verify
the results
yes - some
EvoSuite
PEX (+ Moles)
yes
yes
no
no
no
yes
yes
no
yes
yes
yes - some

36. Techniques/Tools
3. Symbolic/Concolic based:
• Microsoft Pex (.NET)
http://research.microsoft.com/en-us/projects/pex/
• jCUTE (Java)
http://osl.cs.uiuc.edu/software/jcute/
• jalangi (Java Script)
https://github.com/SRA-SiliconValley/jalangi
• AgitarOne (Java, commercial)
http://www.agitar.com/solutions/products/
agitarone.html
• Jtest (Java, commercial)
http://www.parasoft.com/jsp/products/jtest.jsp

37. Best Usage of Auto Generated Unit
Tests
1.Regression testing of legacy code
!
2.To support your unit testing / TDD

38. Q&A

39. Thank you
!
Vassil Popovski
!
Sr. Director, R&D, VMware
vpopovski@vmware.com