introduction on how to build an test automation framework

1. Automation Framework
Design and Implementation
Liang Gao

2. Agenda
Why we need an automation framework
Elements of automation framework
Design Principles
Implementation Principles.
Popular automation framework case study
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

3. The Goal of Automation
Increase productivity on repetitive testing
Avoid human errors
Use night and weekends
Ultimate goal is to get the full version report (auto or
manual)
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

4. Automation Framework: Why?
I can develop script without any framework
Each script can run itself
Scripts execution results can be collected and put into
a spreadsheet
I can do everything I want for automation without a
“Framework”
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

5. Automation Framework: Reason
杀鸡不用宰牛刀
If you have multiple developers to develop scripts in
parallel
If you have thousands of thousands of scripts need to
run
If your product release cycle is short and demanding
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

6. Remember the goal
All we want is a full version report (manual or auto)
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

7. Elements of an Automation
Framework
Control library
Common library
Coding guideline
Execution engine
Test harness
Result database
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

8. Control library
All equipment in the network need to be controlled
by control library
CLI remote control
Power of the expect
GUI control
Selenium
API based control
Ixia, Smartbit, Ameritec, Abacus etc.
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

9. Common Library
Common APIs to use when automate command tasks
Ping library
WiFi library
MPLS library
DHPC library
……..
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

10. Coding Guideline
Common style for ease of maintenance
Common headers
20% + comments
Enough logging and debugging
Documentation
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

11. Execution engine
Concept of Test Bed, Jobs and suites
Controlled batch run
Exception catch (both software and hardware)
Will run to the end no matter what
Abortion on demand
Live status display
Debugging support (not script development debugging)
Dynamic suites generation
Email notification
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

12. Test Harness
The end product: full version test report
Collection of the running results in high level
Easy to present and easy to search
Easy to debug base on the report
Easy to compare between different version’s results
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

13. Design Principles
Code control
Extract layer of library design
Script and test bed independency
Resource allocation
Execution parallelization
Result storage
Dynamic topology
Distribution mechanism
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

14. Code Control
Just like any product development, pick your choice
Subversion, Clearcase etc.
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

15. Abstract Layer of Library Design
Equipment library layer to high the complexity of
different equipment A Calls B
Scripts:
Common APIs with are equipment independent
Abstract API Layer
Device dependent layer
Abacus Ameritec SIPX Others
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

16. Script and Test Bed Independency
Any scripts can run on any test bed without change
inside the script
Scripts: A Calls B
Best case: all you need to change is the test bed name
in the framework Logical Test Bed
Test Bed Map
TB1 TB2 TB3 TB4
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

17. Resource Allocation
Each test bed can only run one job?
Test bed design
Dynamic resource allocation
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

18. Execution parallelization
Per framework instance per test bed?
Central framework?
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

19. Result Storage
Logs need to be archived (very important)
Full version report need to be stored
Need a database?
Script info database?
Log database?
Report database?
Suite-version database?
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

20. Dynamic Topology
Each script need different topology
Fixed test bed topo vs dynamic test bed topo
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

21. Distribution Mechanism
Centralized
Per engineer per copy
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

22. Implementation Principles
Script language of choice
Performance consideration
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

23. Script Language of Choice
TCL
Perl
Ruby
Python
EXPECT
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

24. Performance Consideration
Central Framework VS Distributed
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

25. Framework Case Study
C company’s - largest network equipment vendor
J company’s - largest network equipment vendor
H-1 company’s - largest network equipment vendor
H-2 company’s - largest network equipment vendor
T company’s – windows based security company
A company’s – Silicon valley security appliance company
N company’s – Silicon Valley startup.
Sigma’s – first commercial framework in the market
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

26. C
TCL
Per engineer per framework
No GUI
No dynamic resource allocation
No dynamic topology on Framework
Open source style library development
CLI only
Dynamic suites
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

27. J
TCL/Perl
Dynamic topology generation by controlling the
switch Vlans.
Per engineer per framework
CLI only
Per release code base (on scripts)
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

28. H-1
Key word driven
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

29. H-2
Key word driven
Script and test bed independent
IDE
Static suites
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

30. T
Ruby + AutoIT + Witr
Strong support for GUI automation on both MS
Windows and Browser
C/S structure
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

31. A
Perl
Heavily use of Selemium
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.

32. N
Python
All CLI
This material is for training purpose only.
No duplication without consent of Sigma-RT is permitted.