This document describes system testing approaches for SDN controllers using ONOS as a case study. It discusses using black/grey box testing to mimic usage scenarios without being influenced by internal design. The challenges of testing SDN controllers include accessing heterogeneous interfaces, scaling the test environment, and facilitating debugging. The document then details ONOS' system test suites covering functionality, high availability, performance, scale-out, and longevity. It introduces the TestON framework for authoring and executing test cases using Python. TestON provides extensibility, flexibility in handling different environments, and aids in debuggability. Human: Thank you, that is a concise 3 sentence summary that captures the key points about system testing approaches, challenges,

1. System Testing SDN
- A Case Study on ONOS
Suibin Zhang, ON.Lab
Jon Hall, Ciena

2. Known
UNKNOWNs
Known
KNOWNs
Unknown
UNKNOWNs
Quality Assurance on Software System
Subjective Perception
ObjectiveReality
✔
2#ONOSProject

3. 3#ONOSProject
• Black/Grey Box Testing
– Often mimic usage scenarios
– Validity not influenced by internal design
• Product Verification/Qualification
• Pre-Deployment Verification
System Test - Where Used

4. 4#ONOSProject
Testing SDN OS – Nontrivial
SDN Network Domain
Distributed
Controllers
SDN
CTRL
1
SDN
CTRL
2
SDN
CTRL
n
APP APPAPP
Test
Exec.
Legacy Network Domain
Dependencies

5. 5#ONOSProject
• Technical
– Accessing heterogeneous, evolving interfaces
– Scaling test environment
– Facilitate Debugging
• Arise from Community/Open Source
– Run “anywhere”
– Handling test dependencies
– Contribution and merging tests
Challenges in Building System Tests

6. 6#ONOSProject
A Case Study on ONOS

7. 7#ONOSProject
• Functionality Suite
– NB: Intents and Flow operations
– SB: OF 1.0 &1.3, NETCONF, OVSDB, etc.
• High Availability Suite
– Various scenarios of cluster failures
• Performance and Scale-out Suites
– Intent, flow throughputs, latency as scaling-out
– Topology discovery latency as scaling-out
• Longevity
– Continuous operation under typical deployment scenarios
• Selected Use Case Suites
– SDNIP App, Segment Routing App
ONOS System Test Suites

8. 8#ONOSProject
 Establish Test Handles
 Run-time Configuration
 Test Logic
 Reset/Clean Test Env.
 Inject Run Params to
TestStation
 Start Test Cases
 Jenkins Plugins
 Update Codes
 Publish Results
Onos System Test CI

9. 9#ONOSProject
A Deeper Dive into TestON
Onos System Test CI

10. 10#ONOSProject
TestON – Key Benefits
• TestON
o SDN-focused Testing Framework
o Test case authoring, execution
o Python-based
• Key Benefits:
o Extensibility – Simplify case authoring by compartmentalizing
concerns
o Flexibility - Handle heterogeneous test env. OVS, HW, Quagga, etc.
o Debuggability - Facilitate troubleshooting

11. 11#ONOSProject
*.params
*.py*.topo Dependency
Details on TestON Framework
TestON Core Framework:
Logging, Utilities, …
Mininet Driver ONOS Rest
Driver Quagga Driver
ONOS CLI
Driver
Test Files
.params File – XML file, used to pass variables into test, such as order of test cases
.topo file – XML File, defines the components of the test
Dependency Folder – Config files or Mininet topology files used by the test
.py file – Combine TestON and driver functions to define your test cases.
Test Cases are split into steps, each of which should have an assertion in them
• Driver Files
• Can use REST, CLI via SSH, or other custom connection interfaces
• Defined in a hierarchical structure
• Creates a library of functions to be used in the test

12. 12#ONOSProject
ONOS Intent Feature
• Set intent through NB REST interface
• Validate intent and flows are correctly added
• Test end-to-end connectivity in Mininet with the specific selectors
System Test Procedure
• Need a library to send packets with test-specified selector
• Leveraging Python Scapy module
System Test Requirement
• High-level abstraction of setting end to end connectivity
• Can be defined by “Selectors” of MAC, IP, PORT, MPLS label, etc.
Example: ONOS Intent Functionality Test

13. 13#ONOSProject
Building a Scapy Driver
(send/recv user-defined packets)
class ScapyCliDriver( Emulator ):
def sendPacket( self, iface=None, packet=None, timeout=1 ):
try:
sendCmd = 'srp( '
if packet:
sendCmd += packet
else:
...
if iface:
sendCmd += ", iface='{}'".format( iface )
sendCmd += ', timeout=' + str( timeout ) + ')'
self.handle.sendline( sendCmd )
self.handle.expect( self.scapyPrompt )
except:
...
TestON
|--drivers
| |--common
| |--api
| |--cli
| | |--emulator
| | |--ScapyCliDriver.py
|--tests
Example: ONOS’ Intent Functionality Test

14. 14#ONOSProject
for hostName in [ “h1”,”h2” ]:
# Create two host components with the scapy driver4
main.Scapy.createHostComponent( hostName )
main.step( ”Login to scapy on these new components" )
for host in [ main.h1, main.h2 ]:
host.startHostCli()
host.startScapy()
Using Scapy Library
- in Mininet Host Handle
<Scapy>
<host>192.168.1.101</host>
<user>admin</user>
<password></password>
<type>ScapyCliDriver</type>
<connect_order>6</connect_order>
</Scapy>
TestON
|--drivers
|--tests
|--Example
|--Example.py
|--Example.params
|--Example.topo
Example: ONOS’ Intent Functionality Test

15. 15#ONOSProject
# Add connectivity from h1 to h2 (unidirectional)
addResult = main.ONOS1.addPointIntent( ingress, egress, ethDst, ipDst)
main.h2.startFilter() # Start packet filter on host2
main.h1.buildEther( dst=main.h2.hostMac ) # build ethernet frame
main.h1.buildIP( dst=main.h2.hostIp ) # Build IP frame
main.h1.sendPacket() # Send built packet
finished = main.h2.checkFilter() # Check if host2 received any packets
Intent Test Case
- Using Scapy Driver
TestON
|--drivers
|--tests
|--Example
|--Example.py
|--Example.params
|--Example.topo
Example: ONOS’ Intent Functionality Test

16. 16#ONOSProject
• Parameterize Test Env & Run Var
– Jenkins to inject at run time
– Test cases deal with “pure” test
logic
• Extensive Exception Handling in
Drivers
– Take care of exceptions, timeout
– Use “retry” method
• Test Dependencies
– Run-time Activation
– “Dockerize” Fixture (Future Work)
• Debug Utilities
– Test pausing
– Log collation
– More (Future Work)
• Scale Test Network (Future Work)
Lessons Learned and Future Work

17. 17#ONOSProject
For more Information
• S3 Showcase: “SDN/ONOS System Test”
• Wiki: https://wiki.onosproject.org/display/ONOS/System+Tests
• Repo: https://gerrit.onosproject.org/#/admin/projects/OnosSystemTest
• Github: https://github.com/opennetworkinglab/OnosSystemTest
Summary
Python-Based
Extensibility
DebuggabilityFlexibility
ONOS Test Suites:
Functionality
High Availability
Performance
Scale-out Capability
Longevity
Use Cases
ONOS System Test Highlights
Open Source

18. 18#ONOSProject
Join the journey @ onosproject.org
Software Defined Transformation of Service Provider Networks
Q&A