Simple yet powerful simulation framework to simulate

1. Test Scenario Simulation Framework
TamerA.
V.1
August 3, 2009
Acknowledgment: WayneD
1 Nortel Confidential Information

2. Agenda
• Importance of Simulation
• Modeling Techniques
• More on Simulation Models
• System-/Hardware- In-The-Loop Techniques
• Virtual-Time Based Simulation
• Test Scenario Framework
• Testimonials
• Anti-testimonials
• Conclusions
• References
2 Nortel Confidential Information

3. Importance of Simulation
• Powerful tool to understand complex system interactions
• Quick and predictable answers to what-if scenarios
• Prototyping of new ideas and assessment of their performance gain
• Replacement of yet-to-exist/expensive hardware
• Better model of stochastic nature of human/machine
interactions, test with realistic scenarios
• Not only a decision tool, it is also a design tool!
• Expedite research and development, increase product quality
3 Nortel Confidential Information

4. Modeling Techniques
• Analytical Models
• Best/Worst/Average case analysis
• Quick Results
• System dynamics may get overseen, no enough details
• Statistical Models (Capacity Tool)
• Based on actual measurement
• Close to real world
• No system dynamics, require real-system measurements
• Simulation Models (System Level Simulation, Test Scenario)
• Detailed view of system dynamics
• Evaluate complex system interactions
4 Nortel Confidential Information

5. More on Simulation Models
• Require feedback measurements from existing system
• Use stochastic processes and discrete-time event-based (OPNET,
TS)
• Example (SLS)
Global Init
User Mobility/Call Link Condition PHY Abstraction L2 (MAC/RLC)
Pattern
5 Nortel Confidential Information

6. System-/Hardware- In-The-Loop Techniques
• Early exposure to real-life like system loading/conditions
• Low development cost where virtually every deisgner has a system
on hand
• Could be used to support decision making process and also as a
debug tool
• Fast, repeatable, debug-able, and could be time controlled!
6 Nortel Confidential Information

7. Difficulties with SITL/HIL
• CLOCK, CLOCK, CLOCK!!!
• Type 1: DES matches system clock (wall-clock)
• As close as possible to real situations/conditions
• Runs as fast as SITL/HIL would but not faster
• May need dedicated hardware/cluster of computers to drive system to
capacity
• External factor may affect test repeatability
• Type 2: DES controls system wide clock (virtual-clock/time)
• Highly repeatable and debug-able framework
• Strips idle time out of the real system runs, could be faster than
type 1
• No need for dedicated/fast hardware
• Easily drives system to capacity or extraordinary conditions
• Distributed/Parallel DES is needed to test e2e aspects
7 Nortel Confidential Information

8. Virtual-Time Based Simulation
• Conceals time information from system/hardware under test
TTI #i TTI #i+1 TTI #i+2 … TTI #n
…
Time (ms)
Virtual
Time
TTI #i TTI #i+1 TTI #i+2 … TTI #n±m
…
Within time Exceeds time
Time (ms)
budget budget
8 Nortel Confidential Information

9. Test Scenario Framework
• Based on C++ and integrates very well with UnitTest++
(MRTestFixture.h/cc and etc.)
• Scenario Based framework
• Efficient reuse of test scenarios
• Scenario could represent
• user mobility pattern,
• ever changing RF conditions,
• and draw various traffic patterns from stochastic distributions
• Could have sanity checks and raise error once anomalies are detected
• Unlimited number of users!
• Associate different users with different test scenarios for even
more productivity and test scenarios reuse.
9 Nortel Confidential Information

10. Test Scenario Framework (cont)
• TTF4 had a minimal implementation of TS conceptualization
• Generation of deterministic pattern of traffic
• Only one type of traffic was supported
• Fixed air link condition, no deterioration
• Used very well to run traffic and NACK some of MAC TB
10 Nortel Confidential Information

11. Test Scenario - Achievements
• Main tool to benchmark rel 1 MAC/RLC
• Early detection of timing/performance issues
• Design tool to pinpoint MAC/RLC anomalies; performance or
otherwise
• Secured seamless integration of subsequent activities
11 Nortel Confidential Information

12. Test Scenario (infamous BE chart)
12 Nortel Confidential Information

13. Test Scenario (infamous VOIP chart)
13 Nortel Confidential Information

14. Testimonials
• “On top of the testcases you took responsibility for (investigating, discussing
with technical primes, designing, coding, debugging and execution), the
framework enhancement you provided was first-class. With the eNBTester
not being available to us, the UnitTest framework was all we had. And if it
weren't for your persistent efforts in enhancing the framework, we would not
have finished integration last week. We would probably still be scratching
our heads wondering how we are going to finish all the testcases. The
enhancement work you did with L2 Architect’s help enabled us to carry on
and is responsible for some of the most important findings from our testing
(timing and performance data). I really appreciate your insights and vision
of this enhancement and your persistence to move ahead in spite of
schedule pressures. Thank you for that.” [Software Design Team, TTF4 &
TTU4 Prime]
• “I want to echo the comments TTF4 Prime made regarding the test scenario
toolkit - a major addition that helped make TTF4 a success.” [Software
Design Team, Senior Software Designer]
• “Your changes all look like Greek to me. I looked at the code for about 35-
40 min. I'm sure the changes are good but I just don't understand the code
well enough to comment. Thanks very MUCH for doing these changes. I am
anxious to try them out.” [Software Design Team, L2 Architect]
Nortel Confidential Information

15. Anti-testimonials!
• “Coding/Design Skills - you have a lot of knowledge for different design
approaches and algorithms. However, I found that sometimes you applied a
more complex solution than necessary. Complexity can make things more
difficult to understand and debug. It also caused you to spend quite a bit of
time in trying to get the framework working.” [Software Design Team,
Software Designer]
• “The test cases in the l2IntTest suite #109 all do not fail if there is an error in
the received counts and this should be looked at eventually, but it's not a
scheduler problem, it's just poorly written test cases.” [Software Design
Team, Senior Software Designer]
15 Nortel Confidential Information

16. Conclusions
• Powerful tool for capacity/performance decision making
• Powerful tool for R&D (testing, debug-ability)
• Easily model complex mobility, call patterns along with RF variations
• It hard to enumerate test vectors that will hit all system corner
cases; it is easy to model user mobility/call patterns (city in a box)
• Distributed versions could simulate (not test cases) inter/intra eNB
interactions
• Deterministic soaking does repeat; stochastic soaking does not!
• Simulations are repeatable at will
• SITL/HIL is now used in many industries; defence, aerospace, car
• Bridges gap between SLS and system design
16 Nortel Confidential Information

17. References
• Why use Hardware-in-the-Loop Simulation?
http://www.adi.com/products_sim_qhilWhy.htm
• OPNET System-in-the-Loop (SITL) module.
http://www.opnet.com/solutions/network_rd/system_in_the_loop.html
• Modeling and performance evaluation of General Packet Radio Service.
Communication Networks Laboratory, School of Engineering Science,
Simon Fraser University. http://www.ensc.sfu.ca/research/cnl
• Distributed HIL Simulation, Applied Dynamics International
• Simulation-Based Testing of Software in Space Applications, Dr. Sergio
Montenegro 1, Prof. Stefan Jähnichen 1, Dr. Olaf Maibaum.
http://sergio.montenegros.de/public/embeddedsys_springer2006-b.pdf
• Pratas, A. Rodrigues, “Enhanced Discrete-Timer Scheduler Engine for
MBMS E-UMTS System Level Simulator,” Proc Conf. on
Telecommunications - ConfTele, May, 2007,
http://www.co.it.pt/conftele2007/assets/papers/mobile/paper_80.pdf
• Flxcem Measurement.
http://livelink-ott.ca.nortel.com/livelink/livelink.exe?func=ll&objid=30760458&objAc
17 Nortel Confidential Information