1. The Art Of
System and Solution Testing

2. Example
• A software developer/tester convention was being held. On the train to the convention, there were a
bunch of developer majors and a bunch of tester majors. Each of the developer majors had his/her
train ticket. The group of testers had only ONE ticket for all of them. The developer majors started
laughing and snickering. Then, one of the software testers said, "here comes the conductor" and
then all of the testers went into the bathroom. The developer majors were puzzled. The conductor
came aboard and said "tickets please" and got tickets from all the developer majors. He then went to
the bathroom and knocked on the door and said "ticket please" and the testers stuck the ticket under
the door. The conductor took it and then the testers came out of the bathroom a few minutes later.
The developer majors felt really stupid. So, on the way back from the convention, the group of
developer majors had one ticket for the group. They started snickering at the testers, for the whole
group had no tickets amongst them. Then, the tester lookout said "Conductor coming!" All the
testers went to one bathroom. All the developer majors went to another bathroom. Then, before the
conductor came on board, one of the testers left the bathroom, knocked on the other bathroom, and
said "ticket please."
Lesson learned: Any test that passed in unit testing can fail in system testing.

3. Goal
• Find bugs that can not be fund by feature
level testing but will most likely be found on
customer’s Usage.
• If no time, can even only do system level
and solution testing.
• Development is a science, testing is an art.

4. What Is a System Level Testing
• Company N has a testbed called “Mother of
All Testbed (MOAT). It is the superset of
almost all Company N’s product features.
• To fully execute a cycle, it takes 2 weeks.
• It has just one test case. Compare to feature
level testing where you have thousands of
test cases

5. What Is a Solution Level Testing
• Company C has a dedicated solution testing
team
• Each solution is an end-to-end application
and has a code name
– VoIP end-to-end solution
– Security end-to-end solution
– MPLS solution
– Cable/DSL solution.

6. Difference?
• Solution is customer oriented. Not just one
company’s product.
• System testing is product oriented.
• Feature testing is code oriented. (need a
functionality Specification as input)
• System/Solution: 30% strategy, 70%
experience
• Feature:60% strategy, 40% experience

7. What makes a System/Solution tester
• System/Solution testing is to test the
“Customer”.Input
Tester
Output
User Manual
Customer Secenrios.
Experience
Test Plan/Test Cases
System Level
Defects

8. System Tester’s 360 Degree Advantage
• They interface with Customer (understand
customer)
• They interface with SE, TME, Product
manager (understand marketing)
• They interface with developer (understand
development)
• They interface with feature tester.
(understand testing)

9. Trademark of System and Solution Testing
• Race condition
• Memory corruption
• Algorithm performance (routing protocol
convergence time for both uni-cast and
multicast)
• How system perform in low memory and
high CPU (timers will be stretched out)

10. Typical Bugs Found on System Testing
• Crash: Infrastructure design issue
• Crash: Race condition:
Malloc memory twice, crash only happens
when 1st one success and 2nd one fail.
• Solution : Interoperability with 3rd party
products

11. System Testing is a Time Capsule
• 99.9999% up time = 10 seconds down time
in 3 years
• What your customers will experience in 3
years?
• Emulate 3 years in 2 weeks
• That is why lots of stress testing

12. System Testing is a Time Capsule
正常 异常 1 异常 2 异常 3 异常 4 异常 5 异常 6
1-2 年
2 周

13. Ideas on Building Your System
Cases
• Defined the baseline
– Background traffic.
– # of clients
– Complexity of the configuration
• Identify scalability (performance) index.
• Get the typical customer scenarios and
topologies.
• Design test cases.

14. Build Your Background Traffic
• Background traffic need to drive CPU and
memory to 30%-40%.
• Need to be diverse among typical
applications
• Need to be measurable on
– Latency
– Transit loss.
• IMIX traffic may be considered

15. No Performance Index, No Testing
• Performance index means scalability
• Size of the routing tables
• Number of VPN tunnels
• Voice call per second
• Routing protocol convergence time
• Traffic throughput.
• Peer count (P2P)
• Connection Rate (P2P)
• Concurrent connections

16. QC Meets Customer
• The product is a solution. If the problem
isn't solved, the productdoesn't work.
• The best test case can ever be designed on
your product is in the customer’s real
network.
• The complexity of your test cases should be
equal to your most valuable customers.
• Customers like QC visit from vendor
company very much. You will be surprised

17. Define Your Scenarios
• This is the way customer use it.
• Different location, different area (signal
coverage)
• Barrtery
• Different combination way of using our
product
– GPRS browsing with MP3 in the background
while have an unidentified call
• Different speed for mobile devices

18. 20/80 Rule in the consumer
market
• Satisfied customers seldom speak up
• Unhappy customers will complain
• 1% chance of reproducible rate is ok in the
network equipment market, but not ok in
the consumer market
• Higher testing standard in the consumer
market

19. Simulation Is Your Friends
• Customer level’s scalability can be achieved
in our lab by simulator
• Traffic simulator (commercial testing
equipment are expensive).
• Client simulator
• Protocol Simulator.
• Device Simulator.

20. Simulation Is Your Friends
• A 32 timer reverse Bug, happen once in
every 2 years, how you test that?

21. Steps to Design a System Level
Testing
• One liner for your testing goals.
• List the features you want to cover.
• Define your features performance index
(metrics)
• Design your topology
– Customer feedback
– Performance index requirement
• Pick your test simulator (budget conscious)

22. Give a Detail Example on DHCP server testing
• Definition: Carrier grade high availability DHCP server System level testing.
• Goals: testing system instability especially during the failover.
• Performance index:
– IP address release per second.
– Size of the database.
– Size of the log.
– Command and Control traffic (admin traffic)
– DNS traffic.
• Customer’s topology (simple)
• Simulator: DHCP client simulator

23. Give a Detail Example on DHCP server testing
• Scenarios:
1 Primary down, then come back after secondary switch over
2 Primary down, then come back right away before the switch over.
3 Primary down, then come back after switch over, the down again.
4 Primary keep flapping – up, down, up, down for a while (both before
and after the switch over).
5 Secondary down, then come back.
6 Secondary keep flapping – up, down, up, down for a while (both before
and after the switch over).
7 Both primary and secondary keep flapping for a while.
8 Both shut down then come back the same time.
9 Both shut down, the secondary come back first.
10 Both shut down, the primary come back first.

24. Get the Pass/Fail Criteria.
• DHCP servers can response DHCP request.
• No server performance down grade, the
DHCP can still response to a high rate of
DHCP request during the failover and
recovery
• The failover relationship between primary
and secondary, and the failover state is
working as expected.
• Database sync correctly. And the data
integrity is fine.

25. Overlap your System Test Cases
• Unlike feature testing, the impact among
features are welcomed
• Overlap your test cases one on top of
another increase the feature interaction.
• Carefully record your execution steps.
(reduce non-reproducible bugs)

26. Example: Firewall Instability
Testing• Definition: Testing a firewall system’s instability
• Goals: Discover critical issues when firewall is under severe network conditions
• Performance index:
– # of current sessions
– # of VPN tunnels
– # of content filtering rules.
– # of policies
– # of the NAT
• Customer’s topology (simple)
• Simulator: DHCP client simulator

27. Example: Firewall Instability
Testing
• Customer’s topology

28. Example: Firewall Instability
Testing
• Baseline traffic (pre-conditions)
– Valid traffic
• L3 traffic
• L7 traffic (HTTP/FTP, voice, rich application)
– Invalid traffic (noise)
• To the box
• Through the box
• Trust and un-trust zone
• Malicious (hacker traffic) and just invalid

29. Example: Firewall Instability
Testing
• Pass and fail criteria
– No crash
– No severe performance downgrade on
• Packet loss
• Long latency
– Major functionality working on each of the
performance index.

30. Example: Firewall Instability
Testing
• Customer Secenrios
– HA
– VPN
– Routing
– QoS
– DPI

31. 14 Steps Firewall TestingPhase1: start valid traffics
Phase2: layer3 and layer4 attacks
Phase3: Inject firewall attack traffic
Phase4: Inject invalid frame
Phase5: Drop In
Phase6: management attacks
Phase7: standard security testing
Phase8: Internet applications
Phase9: IP fragmentation, ICMP
Phase10: HA function
Phase11: VPN function
Phase12: dynamic routing function
Phase13: QoS
Phase14: contents filter

32. System and Solution Automation
• System/Solution automation is to make
your life easy
• End-to-end full automation is hard, try to do
semi-automation first.