Software testers and quality assurance engineers are often pressured to cut testing time to ensure on-time product releases. Usually this means running fewer test cycles with the risk of worse software quality. As companies embrace a continuous integration (CI) that require frequent build and test cycles, the pressure to speed up automated testing is intense. Tanay Nagjee shows how you can cut the time to run an automated test suite by 80%—for example, from two hours to under 25 minutes. Find out how Tayay’s team broke down their test suites into bite-sized test that could be executed in parallel. Leveraging a cluster of computing horsepower (either on on-premise physical machines or in the cloud), you can refactor large test suites to execute in a fraction of the time it takes now. With real example data and a live demonstration, Tanay outlines a three-step approach to achieve these results within different test frameworks.

1. !
!
T2#
Session!
6/25/2015! !
10:15!AM!
!
!
!
!
“Reduce#Test#Automation#Execution#Time#by#
80%”##
Presented#by:#
Tanay#Nagjee#
Electric#Cloud#
#
#
#
#
#
Brought#to#you#by:#
#
#
#
#
#
#
340!Corporate!Way,!Suite!300,!Orange!Park,!FL!32073!
888D268D8770!E!904D278D0524!E!sqeinfo@sqe.com!E!www.sqe.com!
!

2. !
!
!
!
!!!!! !
!!!!
A software engineer for more than five years, Tanay Nagjee has developed several core
features of a highly scalable orchestration/automation platform. As a solutions architect, Tanay
tackles complex software delivery problems every day. He works with software companies of
all sizes, each with a unique delivery lifecycle and a myriad of tools. Needs common to these
companies are automation, orchestration, and acceleration. Tanay shares his experiences with
the product team to incorporate customer feedback into the product, which contributes to the
architecture of a product suite that is both flexible and intuitive.
Tanay Nagjee
Electric Cloud

3. © Electric Cloud | electric-cloud.com
Stop Sacrificing
Comprehensive
Testing to
Save Time
Tanay Nagjee
Solutions Architect
Electric Cloud

4. © Electric Cloud | electric-cloud.com

5. © Electric Cloud | electric-cloud.com
Healthcare.gov
problems in 2014
“Poorly tested”
But, it’s not a laughing matter…
Software errors cost US
economy $59.5B annually
– US National Institute of Standards and
Technology, 2002
Mass death
in WoW
Knight Capital, 2012

6. © Electric Cloud | electric-cloud.com
Software tester’s toolbox
Unit test
tools
Static
Analysis
tools
Code
Coverage
Tools

7. © Electric Cloud | electric-cloud.com
Business Pressures that Impact Testing
Get-it-out-the-door mentality
Will test it more later
New feature commitments
More frequent builds (CI)
(continuous integration)

8. © Electric Cloud | electric-cloud.com
As development cycles get shorter and more frequent…
We start limiting how
frequently we run:
• Unit tests
• Static analysis
• Code coverage
Which will let bugs go undetected

9. © Electric Cloud | electric-cloud.com
Typical fixes
• Manually parallelize test jobs to do
more in the same amount of time
• Buy faster hardware
• Accept long CI cycles
• Cut down on tests and analyses that
are run during CI cycle

10. © Electric Cloud | electric-cloud.com

11. © Electric Cloud | electric-cloud.com
How do I reach testing nirvana?
Shorter CI cycles without sacrificing tests/coverage/analysis
Comprehensive testing
Scale no matter how large the code base grows or how frequent
the CI cycle is
Minimize new hardware investment
Continue to use existing tools
Learn what tests take the longest and their dependencies

12. © Electric Cloud | electric-cloud.com
How do I start?

13. © Electric Cloud | electric-cloud.com
What’s next?
fine grain controlled
Parallelism
to do more stuff at once

14. © Electric Cloud | electric-cloud.com
Automation + acceleration
Build
Compile,
Package
Test
Functional,
non-functional
Deploy
Provision,
Configure, Deploy
weeks
days
+ automation
minutes+ acceleration
How long does it take to get a software update through a delivery pipeline?

15. © Electric Cloud | electric-cloud.com
Test acceleration with ElectricAccelerator
ElectricAccelerator dramatically accelerates software builds
and tests by safely parallelizing jobs across shared clusters of
physical or cloud CPU’s.

16. © Electric Cloud | electric-cloud.com
ElectricAccelerator
Massive
parallelization and
distribution for fast
builds and tests
Actionable
visualization and
reports for optimal
parallelization
Automatic
dependency
detection for faster,
more accurate builds
Efficient resource
utilization. Create
build/test clouds to
lower costs

17. © Electric Cloud | electric-cloud.com
Use case: static test analysis acceleration
• Large aerospace customer
• Static analysis tool: Parasoft
• Time to run static analysis:
! Before: 5m25s
! After: 1m25s
4x

18. © Electric Cloud | electric-cloud.com
Try to speed up the commit build. Continuous Integration on a
build of a few hours is better than nothing, but getting down to
that magic ten minute number is much better. This usually
requires some pretty serious surgery on your code base to do as
you break dependencies on slow parts of the system.
- Martin Fowler

19. © Electric Cloud | electric-cloud.com
Use case: unit test & code coverage acceleration
• A customer evaluated ElectricAccelerator (EA) to
reduce their cycle time to 10 minutes to enable gated
check-ins and continuous delivery.
• They are sticklers for test-driven development, and
need to keep the cycle time low as their test and
coverage suites grow over time.
• Tools: Boost, MS Code Coverage, Bullseye
• Time to run tests & coverage:
! Before: 7m30s
! After: 3m11s on 2 machines & 39s best case
2.4x2 Machines
11.5xBest Case

20. © Electric Cloud | electric-cloud.com
Demo

21. © Electric Cloud | electric-cloud.com
“If it hurts, do it more
frequently, and bring
the pain forward.”
Martin Fowler
Thank You!