NFV – Present & Future Risks and Challenges in NFV Testing Building SDN / NFV Assurance Process Creating an NFV test strategy More information can be found here: http://www.qualitestgroup.com/solutions-overview/by-industry/telecom-testing/network-functions-virtualization/

1. Testing -The Gatekeeper for
Quality in NFV

2. 3
Bryon Lowen
Vice-President Telco at
QualiTest
Benny Sand
NFV Strategy Lead at
QualiTest
Webinar Presenters

3. Agenda
NFV – Present & Future
Risks and Challenges in
NFV Testing
Summary & Questions
Creating an NFV test
strategy
Building SDN / NFV
Assurance Process
Introduction
4

4. Agenda
NFV – Present & Future
Risks and Challenges in
NFV Testing
Summary & Questions
Creating an NFV test
strategy
Building SDN / NFV
Assurance Process
Introduction
5

5. About
QualiTest
Group
| World’s 2nd Largest IndependentTesting Company
| QualiTest was recognized as the only visionary pure
play QA and testing Service Company in the “
Magic Quadrant”
6

6. Partial list of
Telecom
Customers
7

7. Agenda
NFV – Present & Future
Risks and Challenges in
NFV Testing
Summary & Questions
Creating an NFV test
strategy
Building SDN / NFV
Assurance Process
Introduction
8

8. OPEX
impact
| Shared computing resources
| Reduced power, space, cooling requirements
| Executing trials of new services without incurring much risk
| Dynamic recovery from failures
| Proof of Concepts (POCs) and trials can be run faster
9

9. CAPEX
impact
| The use of commodity servers reduces hardware costs
| No dependence on specialized hardware
| A single, common server architecture can be used within the data
center environment
| Migration to a distributed cloud architecture allows operators to
deploy backup in a cost-effective way
| Cycling the hardware more frequently to improve the overall
performance of the network
10

10. Predications
| NFV planning phase is shifting gears
| Operators are moving along the NFV curve
| Finical forecasts indicate a significant growth in NFV investments
| NFV will start to influence budgets and purchasing decisions
| NFVAdaption pace is progressing faster then expected
11

11. Agenda
NFV – Present & Future
Risks and Challenges in
NFV Testing
Summary & Questions
Creating an NFV test
strategy
Building SDN / NFV
Assurance Process
Introduction
12

12. Strategic
Challenges
| NFV enables accelerated time-to-market
| NFV enables organizations to implement services anywhere in
the world at any time
| NFV involves very complex integration of various software
elements
| Everything known becomes unknown
| NFV is more than a mere shift of technologies; it’s a cultural
transformation inTelecom organizations
13

13. Testing
Challenges
| Network testing expertise versusVirtualization and Software
testing
| Agile orientation will be introduced in Network testing
| Variety of NFV infrastructures requires more testing from the
carriers
| Frequent updates in the Network services will require more
management and orchestration testing
| More test automation is required to support agility, impact
testing and regression testing
| NFV test strategies are needed to accelerate and ensure the
delivery of NFV solutions with guaranteed quality
14

14. Agenda
NFV – Present & Future
Risks and Challenges in
NFV Testing
Summary & Questions
Creating an NFV test
strategy
Building SDN / NFV
Assurance Process
Introduction
15

15. Building SDN
/ NFV
assurance
process
16
Build strong assurance process to capitalize SDN / NFV
potential
| Renegotiate contracts
| Right Price
| Comprehensive Service Level
Agreements (SLA)
| Increasingly Complex
| Not affect Quality of Service
(QOS)

16. QualiTest NFV
validation
process
QualiTest aims to create a unified end-to-end approach for
service assurance and network performance
17
NFV
Management
Orchestration Infrastructure Migration

17. QualiTest NFV validation process
Assure full
compliance to ETSI
ISG Standards.
•Define NFV Testing
process
•Test Requirements
•NFV Test
Environment
•Risk Assessment
•Roles and
Responsibilities
•Team Skills and
Training
•Test Inputs / Basis
Assure full
compliance to NFV
requirements for
any network
equipment provider
and streamline the
ability of vendors
and users to be able
to implement
compatible NFV
solutions.
Test Environment:
simulation tools,
network analytics,
QoS and QoE
probes.
Automated
Certification Process
for each VNF before
it is introduced to
the network
Test management
and Orchestration
prior to VNF
migration, including
each component
(e.g. Orchestrator,
Vnfm, VIM).
Test the VNF
architecture and
implementation to
assure smooth
transition.
NFV
Test Policy
ETSI NFV ISG
Compliance
(NFV-Co)
Selecting
Mano Vendors
(NFV-Ven)
NFV Staging
Environment
(NFV-SE)
MANO Testing
(MANO-T)
VNF Migration
(VNFM-T)
18

18. QualiTest NFV validation process
Assure full
compliance to
ESTI ISG
Standards
ETSI ISG Standards Test procedure
Assuring compliance aims to find the
deviations from standards
• MANO
• VNF/NE
• Analyse ISG current standards
• Prepare compliance requirements
• Analyse each NFV component / int
• Define gap points and analyse
• Repeat steps 1-6 for each VNF and NE
NFV
Test Policy
ESTI NFV ISG
Compliance
(NFV-Co)
Selecting
Mano Vendors
(NFV-Ven)
NFV Test
Environment
(NFV-TE)
MANO Testing
(MANO-T)
VNF Migration
(VNFM-T)
19

19. QualiTest NFV validation process
Assure full
compliance to NFV
requirements for
any network
equipment provider
and streamline the
ability of vendors
and users to be able
to implement
compatible NFV
solutions.
NFV
Test Policy
ESTI NFV ISG
Compliance
(NFV-Co)
Selecting
Mano Vendors
(NFV-Ven)
NFV Test
Environment
(NFV-TE)
MANO Testing
(MANO-T)
VNF Migration
(VNFM-T)
ETSI specifications on Network Functions Virtualization
• NFV Infrastructure; Hypervisor Domain
• NFV Resiliency Requirements
• NFV Architectural Framework
• NFV Terminology for Main Concepts in NFV
• NFV Infrastructure; Compute Domain
• NFV Infrastructure; Network Domain
• NFV Service Quality Metrics
• NFV Management and Orchestration
• NFV Virtual Network Functions Architecture
• NFV Security; Security and Trust Guidance
• NFV Performance & Portability Best
Practices
• NFV Proofs of Concept; Framework
• NFV Infrastructure; Methodology to
describe Interfaces and Abstractions
• NFV Security; Problem Statement
• NFV Use Cases
• NFV Virtualization Requirements
20

20. QualiTest NFV validation process
Build a
comprehensive Test
Environment,
including Control
plane and Data
plane workload
simulation tools,
network analytics,
QoS and QoE
monitoring probes.
NFV
Test Policy
ESTI NFV ISG
Compliance
(NFV-Co)
Selecting
Mano Vendors
(NFV-Ven)
NFV Test
Environment
(NFV-TE)
MANO Testing
(MANO-T)
VNF Migration
(VNFM-T)
NFV provides great
flexibility:
• Build parallel test
environments
• Scale up and down
compute resources
• Copy any existing
environment
Environment build
guidelines:
• Management and
Orchestration should be
similar to the
production
• NS and VNF
Instantiation should be
similar to production
environment
• Same level of resources
should be allocated.
Test Appliances:
• Simulate workload
traffic on the client /
server
• Simulate both data
plane & control plane
traffic
• Measure key metrics
data & control plane
Virtual Appliances
vs. Physical Test
Appliances:
• Virtual test appliances
are more cost effective
• Physical test appliances
are recommended for
highest levels of data-
plane performance or
microsecond-level
timing accuracy
21

21. QualiTest NFV validation process
Test
management and
Orchestration
prior to VNF
migration,
including each
component (e.g.
Orchestrator,
VNFM, VIM).
NFV
Test Policy
ESTI NFV ISG
Compliance
(NFV-Co)
Selecting
Mano Vendors
(NFV-Ven)
NFV Test
Environment
(NFV-TE)
MANO Testing
(MANO-T)
VNF Migration
(VNFM-T)
The first step is to test NFV Management
and Orchestration Architectural Framework
NFV Orchestrator
VNF Manager
(VIM) and NFV
Infrastructure (NFVI)
VIM– VNF Manager
(Vi-Vnfm)
Orchestrator – VIM
(Or-Vi)
The VNF is deployed as one
or more Virtual Machines;
src ETSI GS NFV 002
v1.2.1
22

22. QualiTest NFV validation process
Test the VNF
architecture and
implementation
to assure smooth
transition.
NFV
Test Policy
ESTI NFV ISG
Compliance
(NFV-Co)
Selecting
Mano Vendors
(NFV-Ven)
NFV Test
Environment
(NFV-TE)
MANO Testing
(MANO-T)
VNF Migration
(VNFM-T)
Static testing VNF-Mano Integration
Conformance testing VNF Internal functionality
Scalability/Performance End-to-end/system test
23

23. Agenda
Risks in testing
NFV & SDN
Challenges with NFV –
what, when and why to
verify with SDN/NFV?
Summary & Questions
Creating an NFV test
strategy
Building SDN / NFV
assurance process
Introduction
24

24. NFV
framework –
Test areas
Overall, the NFV framework consists of three main
components.
1. Virtualized Network Functions (VNF)
2.NFV Infrastructure (NFVI)
3. Network FunctionsVirtualization Management and
Orchestration Architectural Framework (NFV-MANO
Architectural Framework)
25

25. Example:
MMSVNFTest
Strategy
Multimedia Messaging Service (MMS) is similar to the Short
Message Service (SMS) in that a message is passed between an
Originating Party (OP) and a Destination Party or Parties (DP).
The core Network Element is the Multimedia Messaging Service
Centre (MMSC) which acts as a central store and forward server
between the OP and DP, as shown below.
26

26. MMSVNFTest
strategy -
Approach
| Static testing (drive defect curve)
| Scope (requirements/functionality ofVNF)
| Shape test (risk based / ISO9126)
| Test environment (NFV-TE):
| Simulation tools
| Network analytics
| QoS / QoE probes
| Onboarding Automated Certification Process.
27

27. Test Phases –
FAT and NFVI int
Factory AcceptanceTesting
Compliance testing for eachVNF introduced to the network:
• Step 1: Analyze ISG current standards
• Step 2: Prepare compliance requirements
• Step 3: Analyze each NFV component and interfaces
• Step 4: Define gap points and analyze
• Step 5: Prepare correction plan
• Step 6: Monitor correction plan execution
• Step 7: Repeat steps 1-6 for eachVNF and NE
Requirements coverage
NFVI –test facility integration
Ensure full validation against NFVI in labs
28

28. VNF
compliance and
End2end
testing
VNF compliance testing:
VNF Compliance Criteria – functional criteria:
• VNF Design Patterns
• VNF Update and Upgrade
• VNF's Properties
• VNFTopological Characteristics
• VNF States andTransitions
• VNF Fault Management.
End to end testing:
• Test service end to end to ensure full integration
• TestingVNF performance together with otherVNFs & nonVNF
• Validate that introducing the newVNF doesn’t affect otherVNF’s functionality and
performance.

29. MMSVNFTest strategy – example Metrics
30
1 NFV Service Metrics
VM Provisioning Latency, instantiation latency. Time between VM instantiation and first available packet
VM Stall (event duration and frequency)
VM Scheduling Latency
2 QOS and Data-plane metrics:
Latency on each of tens of thousands of data streams
Throughput and forwarding rate
Frame loss rate
Packet-delay variation and short-term average latency
Dropped frames and error frames
Service Disruption Time for Fail-over Convergence
3 QoE and Control-plane metrics:
HTTP: page load time, load time variance
Video: MOS-AV score, range = 2-5 with 5 being the best
HTML5 video - AS score, 100 % score as the maximum
Direct metrics:
Peak Signal to Noise Ratio (PSNR)
Structural Similarity (SSIM) - compare the original image with the received image
Video Quality Metric (VQM)
Mean Opinion Score (MOS) - This metric combines delays, perceived jitter at application layer, codec used for communication and packet loss at application layer
Indirect metrics:
Startup time: Time difference between sending the request for content and the time when the user actually received the content
Delivery synchronization - In a multicast many-to-many scenario it is important that the content is received by all participants at the same time.
Freshness: The time difference between the time when the content is actually generated and the time when the users receives it.
Blocking: When the buffers on the receiver are empty and the user has to wait for content.
Connections establishment rate, and transactions per second
Total number of connections, round trip time and goodput

30. Non Functional
-Load and
Migration
Load / Performance
• Ensure auto-scale works properly
• Ensure resources consumption is efficient
• Measure performance meets SLA in high performance workloads
Failover Convergence testing
• Convergence measurement involves the measurement of processing time of the
trigger event in the control plane and the traffic switchover time.
• It is important in a multipleVNF deployment scenario that the convergence time of
anyVNF is not impacted by the otherVNFs on the same physical server.
Level % of BHTR Used for
Low 5% Functional Testing
Average 50% Scalability, on-going, fail-
over
Busy 100% Performance
Stress 150% Stress Testing, auto-scaling
31

31. Test strategy
goals
| Build an NFV ecosystem
| Drive quality and governance
| Continuous integration model
| Reduce cost of integration
32

32. Agenda
NFV – Present & Future
Risks and Challenges in
NFV Testing
Summary & Questions
Creating an NFV test
strategy
Building SDN / NFV
Assurance Process
Introduction
33

33. Summary
| NFV allows agile real-time shifting of assets to where they are
needed
| Testing, in the new world of NFV, changes dramatically and
becomes a core discipline that is central to the overall goal of
pleasing customers in a cost-effective way
| NFV –testing is the change agent from the physical networks to
the virtualized ones
| NFV –Testing challenges deep interdependencies and lightning
fast technology introductions; hence it becomes the quality
gatekeeper for NFV
| If an operator is to find benefits in the new technology, they
primarily want reliable services, at the right price, supported by
strong SLAs.
| The move to a software-centric architecture means becoming
more software-focused (skills pivot)
| Move towards network DevOps
34

34. Q&A
35

35. www.QualiTestGroup.com