1. Here Now - an Open Source Project Near
You
The Linaro LNG Open Data Plane Initiative
Mike Christofferson, Enea
In conjunction with Ola Liljedahl, Arm

2. FOUNDED
1968
TEN OFFICES
IN NORTH
AMERICA,
EUROPE AND
ASIA
REVENUE
~70 M
USD
NO. OF
EMPLOYEES
426
 Increasing data traffic in communication devices
require new and innovative software solutions to
handle bandwidth, performance, and power
requirements, as well as scalable systems
management and availability solutions
 A robust product portfolio
 Enea operating systems software is heavily
used in wireless Infrastructure (Macro, small
cell), gateway, etc. Enea Solutions run in
more than 50% of the world’s 8.2M radio
base stations.
 Enea provides a commercial Linux
distribution, built by Yocto, with focus on real-
time
 Proven, mature middleware solutions for over
10 years – High Availability, Systems
Management, and real-time database
 Global presence, global development, and
headquartered in Stockholm, Sweden
Enea - Powering Distributed, Connected Systems

3. 0
20
40
60
80
100
120
140
160
1995 2000 2005 2010 2015 2020 2025 2030
• Super-linear growth in:
– Number of users
– Number of connected devices
– Number of over the top (OTT)
applications and protocols
– Number of (standard)
protocols (RFC’s)
– Bandwidth usage
– Power consumption of
network infrastructure
What Is Happening
Logos and trademarks are used for illustration only and remain the property of their respective owners.

4. • Increasing and varying QoS requirements
– Realtime (e.g. VoIP, video conferencing, gaming)
– Streaming (e.g. music, video)
– Messaging (e.g. IM, Ajax, M2M/IoT)
– Bulk (web data, file sharing, OTA updates)
• More functionality and services implemented in the
network
– Web caching
– Content delivery (CDN)
– Intrusion detection and prevention (IDS/IPS)
– User-specific service level agreements (SLA)
Increasing Diversity and Functionality

5. Consequences
• Need flexibility of software and programmable hardware
– Trend towards software-realised networking - function defined by
software
• Need familiar programming environments with robust tools
– For TimeToMarket-driven development of new protocols and
services
• Need portability
– Move functionality and applications between hardware platforms
optimised for different power/performance/cost points
• Need high abstraction
– To enable innovation in efficient implementations
– E.g. OpenGL/OpenMAX
• Need efficient support for virtualization
– Decouple functionality (SW) from capacity (HW)
– Dynamic partitioning of common HW for different functions
– Simple, robust and incremental deployment of new services

6. Solution – HW/OS
Develop and deploy networking applications on general
purpose processors/architectures
• Increasingly ARM and x86
• Users and partners are drawn to the big ecosystems around these architectures
Networking applications running in Linux user space
Develop, debug and deploy using standard Linux tools
• Robust user space access to networking HW resources
Linux enhanced to provide bare metal-like environment
Bare Metal Linux
Avoid TLB misses, interrupts, context switches, system calls, thread migration
Direct HW access from user space
Applications run isolated in user space on dedicated cores, unaffected by the Linux
kernel and other applications
Optional real-time support
As needed by some wireless subsystems (<10μs interrupt response time)

7. What Is Open Data Plane?
• ODP is an open source cross-platform framework
for data plane applications
• Common API for application portability
• Multiple implementations tuned to different
platforms for performance
• Result: Easy app portability and performance
Application Environment
• Applications run in Linux user space with
essentially zero system overhead
Open Data Plane Overview

8. Open Data Plane: The Time has Come
• Networking silicon vendors have evolved data plane SDKs for years
– No cross-industry group has sanctioned any common interface on diverse
silicon
• The Linaro Networking Working Group - a consortium of 12 networking
stakeholders surveyed the open source landscape
– Consensus: No ideal “one-size fits all”, implementation for diverse
hardware/software approaches
• A truly open source & open contribution & cross-platform data plane interface,
driven by a cross section of stakeholders, is needed
• Based on the OpenGL model: A software API at a higher level of abstraction,
that could offer flexibility of implementations underneath that suit diverse
needs.
– The Linaro non-profit open source software engineering organization is launching
just such a collaboration…
• So Linaro created…OpenDataPlane(ODP).org with charter contributors…

9. Open Data Plane API
• Standardized data plane API to enable Linux-based
networking applications across any architecture
– Open support for ARM, Intel, MIPS & PowerPC !
• Structured to enable future innovation
– Lightweight abstraction preserves performance without
prescribing lower –level processing structure
– Access and management of HW accelerators
– Supports optional schedulers to provision easy
management and traffic load balancing
• Proprietary SDKs sit underneath for OEM/operator
software platform simplification (e.g. Supports
DPDK on x86, USDPAA on QorIQ, etc)
9
Enabling an efficient, truly cross-platform standardized
data plane processing model
Application and
services
portability
across a choice
of hardware
platforms

10. ODP Foundational Principles
Event Machine
– Work-driven many core data plane processing
SoC Abstraction
– Portable API’s for access to HW/SoC resources
Bare Metal Linux (a.k.a. Bear Metal Linux)
– Minimal overhead and deterministic execution in Linux user
space
EM
SoCA
BML
Application

11. ODP Foundational Principles (2)
• A data plane/networking API and runtime
– Loosely based on the NSN Event Machine
– Event/work-driven and polled programming models
– Portable API’s for accelerators and offloads
– Runs in user space under Bare Metal Linux for best performance and
determinism
• Common API, optimised implementations
– Separately owned and maintained API (e.g. OpenGL)
– Generic portable reference implementation from Linaro
– HW-optimised (possibly proprietary) implementations from
networking SoC vendors
– Linaro maintains ODP for x86/DPDK

12. API and Concepts
• ODP loosely based on Event Machine, originally developed by
NSN
– Generic framework for scalable multi-core programming, not limited
to packet processing
• Event based abstraction and programming model for handling
IO
– Supports packet flows, physical and virtual network interfaces,
accelerators, SW endpoints, etc.
– Events represent different types of data: packets, timers, baseband
data, HW notifications, SW messages...
• Supports scheduler based programming model (both HW &
SW)
– Scheduling of IO events using different algorithms and knowledge of
work in progress
– Implicit synchronisation and mutual exclusion between threads
• Supports different IO load balancing approaches
– Chose best configuration for traffic profile, latency/throughput
requirements, and HW characteristics without changing the
application
• Proven, already half a dozen HW-specific EM implementations

13. EM Basic Concepts
Queue groups
Queues with
events
Scheduler
Cores/threads associated
with queue groups
Idle
core
Queues can be
dynamically created
and added to and
removed from queue
groups
Cores can be
dynamically added to
and removed from
queue groups
Event handlers associated
with queues
IP-fwd
NAT
GTP-U
DPI
RoHC

14. Work Scheduling
• Actual scheduling algorithms implementation dependent
• Scheduler can enforce ordering/mutual exclusion
– Parallel, parallel/ordered and atomic queues
– Application doesn’t need software mutex for protecting per-flow state
• Logical flows/queues mapped to hardware queues (if available)
‘Pull’ work, on
demand scheduling
Clusters/Cores/ThreadsLogical flows/queues
thousands to millions
Flows/
QoS classes WRR
SP
Scheduling algorithms
WRR - Weighted Round Robin
SP - Strict Priority
Work
Scheduler
Processing packet
from flow A
Processing packet
from flow B
Idle (power-gated)
because of low load

15. Dynamic vs. Static Load Balancing
• Networking SoC’s have hardware
suitable for dynamic load balancing
– Queues associated with producer
– Queue and buffer mgmt in hardware
• Server NICs designed for termination
– Static load sharing (based on hashing)
– Queues associated with consumer
– Queue and buffer management in
software/shared memory
• Static increases average and worst case
latency and buffer space
– OK for ~8 cores… but not many-core ready
(some networking SoC’s already have 30+
cores/HW-threads)
• Static makes core elasticity very
difficult (per-core state with application
level seamless/lossless handovers)
– Limits opportunity for power scaling
0
20
40
60
80
100
120
140
1 2 4 8 16 32 64 128
Averagelatency/uS
Number of cores
Static load-
sharing
Dynamic load-
balancing
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
0% 20% 40% 60% 80% 100%
Power
% Load
Static Load Sharing
with typical DVFS
Dynamic Load
Balancing Power
gating + DVFS

16. Elasticity and Multi-core Load Balancing
Issues
– Traffic load and pattern varies over time
– Industry trend is to use more cores and more power-efficient cores
– To hit the sweet spot PPA (Power/Performance/Area)
– Enabled by inherent parallelism in networking
– Ideally use as many (or few) cores as traffic load and SLA’s require and use them
efficiently
ODP
 Supports hardware scheduler and dynamic load balancing
 Cores can be added and removed
 No fixed allocation of cores for specific application
 Enables power or clock gating of idle cores
 Cores can share load dynamically
 Increased throughput,
 Decreased packet latencies,
 Increased core utilization

17. Scalable and Elastic Timer Support
Issues
• Many protocols need timers, often several timers per
flow/connection
– Millions of flows in core network means millions of timers
• Timers, like packets, are associated with flows/connections
– Need mutual exclusion of flow context when processing a timer event
ODP
 ODP schedules timers together with packets
 Timers and packets can be synchronized and load balanced together

18. Power and Performance Management
Issues
• Traffic load varies
– Daily variation and intermittent bursts
• Use as many or as few cores needed to meet bandwidth and QoS
requirements
– Add and remove worker threads/cores
– Adjust clock frequency of active cores
– Power or clock gate inactive cores
ODP
 Supports power/performance management
 Provides API for observing queue lengths
 Idle worker threads may yield to OS for background tasks or power down core
 Application can monitor traffic load and quickly react to increasing load

19. vSwitch Integration
Issues
• Efficient and robust integration with software or hardware-accelerated
vSwitch
• No loss of performance for virtualised networking applications using
the dataplane API
ODP
 ODP’s queue-based I/O hides actual device implementation
 A queue may represent an actual network interface, a vSwitch port, a
pipeline of further processing stages (e.g. for encryption or
encapsulation) etc.
 Allows for HW to copy packets between application and vSwitch
 No shared memory between application and vSwitch

20. Openness and Cross-platform
ODP provides:
 Support for multiple architectures and platforms (e.g., ARM,
x86, and MIPS)
 Open source and an open collaboration
 Not controlled by any single company
 Anyone may join in
 Reference implementations are open source
 Based on the Event Machine which currently is implemented
on a number of different HW targets (using
ARM/MIPS/PPC/x86 processors)
 Proven cross-platform support

21. Status Core API Definitions
API Component Description Status
BUFFER Shared memory, buffer pools, buffer types and access
functions
Preliminary done, but still
work in progress
CLASSIFICATION Ingress packet classification Preliminary work underway,
CRYPTO Algorithmic and protocol offload for crypto, hashing, RNG Proposal being implemented
IPC Inter-process communication control plane/data plane TBD
PACKET I/O Network interface abstraction Done
QUEUE Buffer queue management Done
TIMER Protocol timers, periodic ticks Done
SCHEDULER Ingress scheduling and distribution to threads/cores Done
Version 0.2. of the API spec available now
Version 1.0 by year end 2014

22. Status Implementations
Platform Description Status
linux-generic Generic, portable reference implementation, uses Linux
facilities (e.g. NetMap, crypto)
Implements
BUFFER/CRYPTO/PACKET-
IO/QUEUE/TIMER/SCHEDULE
R
linux-dpdk Implementation for x86 using DPDK as the acceleration
layer.
Just started
linux-keystone2 HW-accelerated implementation for TI Keystone2 Tracking linux-generic
linux-qoriq HW-accelerated implementation for FSL DPAA In progress
Other implementations outside LNG also in progress...

23. • Cisco will demonstrate real app running on multiple HW-
implementations of ODP
– Usage of API’s
– Usage of HW acceleration through ODP API’s (e.g. ordered and atomic
scheduling, crypto)
– Portability
• NSN has had early influence on general architecture and APIs
• Huawei is promoting ODP in public presentations and
expressing their support in meetings
• Ericsson’s new influence in this project
Demos at Linaro Connect USA in Sep’ 14
Contributions and Interest from Major TEMs
The ODP API Specification can be influenced by anyone in the open
community

24. What’s next?
• Adding more members to the ODP team
– Several companies in discussions, e.g. Aricent, Juniper & others
downloading and commenting on ODP
• Developing NFV PoCs with ARM ecosystem partners, building
on ODP
– Hardening and optimizing the performance of ODP implementations
• Developing liaisons with OpenDayLight, Open Networking
Foundation, NFV Working Group
– Additionally, a new open source initiative to integrate open NFV
building blocks with ODP
• Evangelizing ODP in the broad community

25. Thanks for Attending
For more….
Visit us in booth #201
Go to
opendataplane.org
linaro.org/projects/networking