Functional testing is key to validate the installation and the correct behavior of the OPNFV platform. This session will cover how to prepare and execute the Functest framework over a fresh installed Arno and possibly some spoiler of the second release. It will also walk though the different tools used in Arno: Rally, Tempest, vPing and Robot.

1. FUNCTEST IN DEPTH
Jose Lausuch (Ericsson)
OPNFV Summit 2015
jose.lausuch@ericsson.com

2. Agenda
09/11/2015 Functest in Depth, OPNFV Summit 2015 2
What is Functest?
Test cases
Repository structure
CI & Automation
Brahmaputra Release

3. OPNFV Projects ecosystem
09/11/2015 Functest in Depth, OPNFV Summit 2015 3
TEST Projects
Yardstick
Qtip
VPerf
…
FEATURE Projects
VPN
SFC
SDN Contr.
Resource mgmt.
…
PHAROS
Testbed
OCTOPUS (CI)
RELENG
INSTALLER Projects
Fuel
Foreman
JOID
Compass
…
Functest
TestbedTestbed TestbedTestbed

4. What is Functest?
09/11/2015 Functest in Depth, OPNFV Summit 2015 4

5. What we do…
Focus on verifying the OpenStack deployment and
the SDN Controllers
Have a full integration and automation mechanism
Provide comprehensive testing methodology
09/11/2015 Functest in Depth, OPNFV Summit 2015 5
Base System Functionality Testing

6. 09/11/2015 Functest in Depth, OPNFV Summit 2015 6
ETSI NFV Reference
WE DON’T:
Performance tests
HW tests
Frameworks

7. The gang…
09/11/2015 Functest in Depth, OPNFV Summit 2015 7

8. The project…
09/11/2015 Functest in Depth, OPNFV Summit 2015 8
source: Bitergia (http://projects.bitergia.com/opnfv)
date: November 2015

9. Functest Testcases
Arno
09/11/2015 Functest in Depth, OPNFV Summit 2015 9

10. Functest Testcases
vPing test case
Create machines and verify connectivity
ODL test case
Robot framework, ODL functional testing
Rally bench tests
Benchmark the OpenStack deployment
Tempest test
OpenStack native tests (100+ smoke-tests)
09/11/2015 Functest in Depth, OPNFV Summit 2015 10

11. Functest Testcases
vPing test case
Create machines and verify connectivity
09/11/2015 Functest in Depth, OPNFV Summit 2015 11

12. VM2
vPing test case
09/11/2015 Functest in Depth, OPNFV Summit 2015 12
VM1
provisional Neutron network

13. Functest Testcases
ODL test case
Robot framework, ODL functional testing
09/11/2015 Functest in Depth, OPNFV Summit 2015 13

14. ODL test case
09/11/2015 Functest in Depth, OPNFV Summit 2015
14
Dashboard/API
(Horizon)
Compute Storage
Networking
(Neutron)
…..
ML2 Plug-in
Northbound API
ODL – Neutron REST API
OVSDB
OVS Nodes

15. ODL test cases
09/11/2015 Functest in Depth, OPNFV Summit 2015 15
Borrowed from ODL repository
4 basic test cases:
Check ODL Restconf
Create/Delete/Check Network
Create/Delete/Check Subnet
Create/Delete/Check Port

16. Functest Testcases
Rally bench tests
Benchmark the OpenStack deployment
09/11/2015 Functest in Depth, OPNFV Summit 2015 16

17. Rally
09/11/2015 Functest in Depth, OPNFV Summit 2015 17
Rally is a
benchmarking tool that
answers the question:
“How does OpenStack
work at scale?”.

18. Rally - scenario example
09/11/2015 Functest in Depth, OPNFV Summit 2015 18
Runners:
• Serial: launch scenario N times in one thread
• Constant: launch scenario N times in M parallel threads
• RPS: generates N threads per second M times
Success criteria (SLA)
• You can define a maximum time per iteration
• It will skip the test if 1 iteration fails
Context:
• Contexts needed for this scenario (tenants/users)

19. Rally - output
09/11/2015 Functest in Depth, OPNFV Summit 2015 19

20. Functest Testcases
Tempest test
OpenStack native tests (100+ smoke-tests)
09/11/2015 Functest in Depth, OPNFV Summit 2015 20

21. Tempest test
09/11/2015 Functest in Depth, OPNFV Summit 2015 21
Tempest is a set of integration tests to be run against
a live OpenStack cluster
Rally performs the tempest installation and generation
of tempest.conf

22. Functest Structure
today
09/11/2015 Functest in Depth, OPNFV Summit 2015 22

23. Directory structure
09/11/2015 Functest in Depth, OPNFV Summit 2015 23
Main configuration script. Installs needed tools (Rally, Robot, …)
Configuration parameters
SDN Controllers test suites
Openstack related tests (Rally, Tempest)
vPing test case
Dockerfile and automation scripts
Format results to dashboard ready and other utils
vIMS test case

24. Configuration file
09/11/2015 Functest in Depth, OPNFV Summit 2015 24
Configuration yaml file
Predefined tested values
Can be given as an input to Functest
Possible to specify commit IDs of the
used repositories
Versioning control

25. CI & Aut
09/11/2015 Functest in Depth, OPNFV Summit 2015 25
tion

26. CI & Automation
09/11/2015 Functest in Depth, OPNFV Summit 2015 26
Functest Jenkins job
Triggered automatically
after a successful
deployment
One macro per test case
You can execute a single
test From Jenkins

27. CI & Automation
09/11/2015 Functest in Depth, OPNFV Summit 2015 27
Daily verification of OPNFV platform
New
Features
JENKINS
BUILD
DEPLOYMENT
FUNCTEST
YARDSTICK
….

28. Execution of tests
09/11/2015 Functest in Depth, OPNFV Summit 2015 28
Target POD
Jumphost
Don’t worry sir,
I’ll do it for you
jjob

29. Brahmaputra
release
09/11/2015 Functest in Depth, OPNFV Summit 2015

30. Functest in Brahmaputra release
Dockerization
Test Result collection
Dashboard
Virtual IMS test case
+ SDN Controllers tests
Feature projects
09/11/2015 Functest in Depth, OPNFV Summit 2015 30

31. Docker -ization
09/11/2015 Functest in Depth, OPNFV Summit 2015 31

32. Functest Docker
09/11/2015 Functest in Depth, OPNFV Summit 2015 32
$docker run -t opnfv/functest $dir/run_test.sh

33. Functest Docker
09/11/2015 Functest in Depth, OPNFV Summit 2015 33
Without Docker With Docker
• Have to install needed libraries in
your jumphost
• Maintain in the documentation
the needed libraries and
dependencies
• Possible OS dependencies
• Not portable
• Needed libraries preinstalled in
the docker image
• Update image when there is a
new software dependency and
push to the OPNFV Docker Hub
• No OS dependencies
• Portable

34. Execution of tests
09/11/2015 Functest in Depth, OPNFV Summit 2015 34
Target POD
Jumphost
Docker Hub
Don’t worry sir,
I’ll do it for you
jjob

35. MRF PCRF
CSCF
HSS
vIMS test case
09/11/2015 Functest in Depth, OPNFV Summit 2015 35

36. vIMS test case
Full automation of a core Open Source
vIMS solution on OPNFV from Clearwater
Deployment of an orchestrator (Cloudify)
to manage the VNF
Run suite of signaling tests on vIMS
Successfully tested from Jenkins on
LinuxFoundation POD2
09/11/2015 Functest in Depth, OPNFV Summit 2015 36
SIP

37. vIMS test case
09/11/2015 Functest in Depth, OPNFV Summit 2015 37
1) Deploy the orchestrator
2) Create the tenant
3) Create the VNF (10 VMs)
4) Launch test suite (more
than 100 signaling tests)
5) Collect results
6) Clean-up

38. SDN Controllers
09/11/2015 Functest in Depth, OPNFV Summit 2015 38

39. SDN Controllers
Extension of OpenDaylight test cases
Integration of ONOS test framework
Open Contrail test
09/11/2015 Functest in Depth, OPNFV Summit 2015 39

40. Test results collection
09/11/2015 Functest in Depth, OPNFV Summit 2015 40

41. Test results collection
09/11/2015 Functest in Depth, OPNFV Summit 2015 41
Test projects
Mongo
DB
REST API
VPerfYardstick QTIP BottlenecksFunctest …

42. Test results collection - API
Filter per test case
– http://<DB_Public_IP>/results?case=vPing
Filter per pod
– http://<DB_Public_IP>/results?pod=opnfv-jump-2
Filter per project
– http://<DB_Public_IP>/results?project=yardstick
Combination of filters
– http://<DB_Public_IP>/results?project=functest&case=Tempest&pod=opnfv-jump-2
09/11/2015 Functest in Depth, OPNFV Summit 2015 42

43. Dashboard
09/11/2015 Functest in Depth, OPNFV Summit 2015 43

44. Dashboard
Show all the results using simplistic graphs
Possible to filter per project/test case/…
Comparison of previous results with recent ones
…
09/11/2015 Functest in Depth, OPNFV Summit 2015 44

45. Dashboard Draft
09/11/2015 Functest in Depth, OPNFV Summit 2015 45

46. Beyond Brahmaputra
09/11/2015 Functest in Depth, OPNFV Summit 2015 46
New functional test (vCPE, vHGW, vCDN, …)
Results analytics
Better Rally usage
More SDN controllers test integration
Broaden test coverage (security, O&M, …)

47. Lessons learned in Arno
09/11/2015 Functest in Depth, OPNFV Summit 2015 47
Need for stable labs some weeks before the release
Need to run the tests on different PODs from CI
Complex troubleshooting
Functest cannot solve all problems

48. Challenges
09/11/2015 Functest in Depth, OPNFV Summit 2015 48
Different OpenStack installers
Different SDN controllers
Many features
Lots of combinations to be tested !!
…
…

49. Summary
09/11/2015 Functest in Depth, OPNFV Summit 2015 49
First test project in OPNFV
Validate a fresh OPNFV deployment
Umbrella for testing projects
Umbrella for feature projects
We don’t bite, come to us with questions 

50. 09/11/2015 Functest in Depth, OPNFV Summit 2015 50

51. Thank you!
09/11/2015 Functest in Depth, OPNFV Summit 2015 51