ONIE defines an open source “install environment” that runs on this management subsystem utilizing facilities in a Linux/BusyBox environment. This environment allows end-users and channel partners to install the target network OS as part of data center provisioning, in the fashion that servers are provisioned.
ONIE enables switch hardware suppliers, distributors and resellers to manage their operations based on a small number of hardware SKUs. This in turn creates economies of scale in manufacturing, distribution, stocking, and RMA enabling a thriving ecosystem of both network hardware and operating system alternatives.
5. Understanding Characteristics of a Leaf Switch
5cumulusnetworks.com
10/40 Gigabit
spine uplink ports
Serial
console port
Ethernet Out-of-Band
Management Port
1/10 Gigabit Ethernetnetwork access ports
* SFP+ ports can be grouped together into a single QSFP 40G port via reverse connecting breakout cable options
* QSFP ports can be broken out into four SFP+ ports via copper or optical transceiver options
6. Understanding Characteristics of a Spine Switch
6cumulusnetworks.com
Serial
console port
Ethernet Out-of-Band
Management Port
* QSFP ports can be broken out into four SFP+ ports via copper or optical breakout cable options
40G / 10G Fabric ports
7. Bare Metal Switch Provisioning
§ Similar approach to installing OS on server
§ BIOS + PXE = U-Boot + ONIE (Open Network Install Environment)
§ Supported hardware (HCL) preloaded with ONIE
§ ONIE available on GitHub
• http://onie.github.io/onie/
cumulusnetworks.com 7
bare metal server
operating system
app app app
BIOS and PXE
bare metal switch
operating system
app app app
U-Boot and ONIE
10. What Is ONIE?
§ Network OS Install Environment
§ Provides an environment for network OS installer discovery
and execution
§ Like a pre-installed kickstarter
§ Defined by its behaviors
§ Implemented using a modern Linux kernel and BusyBox
§ An Open Source Project within OCP
§ http://www.onie.org/
§ Evolving …
cumulusnetworks.com 10
13. ONIE: Anatomy of a Network Switch
cumulusnetworks.com 13
(
Management
Interfaces
)
(
Data
Plane
)
CPU
SoC
DRAM
Boot
Flash
Mass
Storage
Switching
ASIC
Serial
Console
Ethernet
Mgmt
Port
10Gb
Port
40Gb
Port
…
10Gb
Port
40Gb
Port
…
PCIe
14. ONIE: Uses Management Interfaces
cumulusnetworks.com 14
(
Management
Interfaces
)
(
Data
Plane
)
CPU
SoC
DRAM
Boot
Flash
Mass
Storage
Switching
ASIC
Serial
Console
Ethernet
Mgmt
Port
10Gb
Port
40Gb
Port
…
10Gb
Port
40Gb
Port
…
PCIe
15. ONIE: Bare Metal Install – First Time Boot Up
cumulusnetworks.com 15
Boot Loader
(HW Vendor Supplied)
ONIE
(HW Vendor Supplied)
Installer
(OS Vendor)
Boot
Loader
• Low
Level
boot
loader,
configures
CPU
complex
• Loads
and
boots
ONIE
ONIE
• Linux
Kernel
with
Busybox
• Configures
management
Ethernet
interface
• Locates
and
executes
an
OS
installer
• Provides
tools
and
environment
for
installer
OS
Installer
• Available
from
network
or
USB
• Linux
executable
• Installs
vendor
OS
into
mass
storage
Network OS
(OS Vendor Supplied)
Fetches
Installs
16. ONIE: Subsequent Reboots – Vendor’s OS is Already Installed
cumulusnetworks.com 16
Boot Loader
(HW Vendor Supplied)
ONIE
(HW Vendor Supplied)
Boot
Loader
• Low
Level
boot
loader,
configures
CPU
complex
• Loads
and
boots
OS
vendor’s
installed
OS
Network
OS
• Configures
Switching
ASIC
• Runs
Network
Protocols
• Provides
CLI
Network OS
(OS Vendor Supplied)
ONIE
• SSll
exists,
but
is
not
used
• Available
for
uninstall
/
re-‐install
operaSons
17. ONIE: Network OS Installer Discovery and Install Behavior
cumulusnetworks.com 17
Configure Network
Interface
Locate Installer
Run Installer
• Uses
DHCPv4,
DHCPv6
• Configures
Ethernet
interface
for
IPv4
/
IPv6
• Configures
DNS
and
hostname
• Determines
the
locaSon
of
an
installer
executable
• Examines
local
file
systems,
e.g.
USB
flash
drives
• Uses
DHCP
opSons,
DNS
Service
Discovery,
MulScast
DNS
and
IPv6
Neighbors
• Downloads
installer
via
URL
• Passes
various
environment
variables
to
installer
• Launches
installer
18. Why
Not
Use
PXE?
cumulusnetworks.com 18
• IPv6
and
HTTP
out
of
the
box
• Use
exisSng
Linux
device
drivers.
No
need
to
write
new
ones
• Integrated
automaSon
19. ONIE: Other Behaviors
§ Provides a mechanism for a network OS to invoke
the above behaviors.
cumulusnetworks.com 19
Reinstall Remove currently installed OS and return to
the “out of box” provisioning state
Uninstall Completely wipe out everything, except ONIE
Rescue Reboot box into ONIE for repair, debug and
forensics
Update Install a new ONIE version
Diag Run HW Vendor’s diag (optional)
20. Lessons Learned
§ Hardware Platforms
§ Diverse mix of CPU complex designs, even within a
single CPU family.
§ HW Vendors need freedom to customize ONIE
§ Need mechanism to run HW vendor diag
§ Recurring Themes
§ TLV based EEPROM format widely adopted
§ Common HW designs reduce development time
cumulusnetworks.com 20
21. ONIE Ongoing Development
§ Support Multiple CPU Architectures
§ PowerPC – Today
§ x86 – VM Prototype available today
§ Thinking about ARM, MIPS
§ Maintain ONIE behaviors across architectures
§ Testing and Compliance
§ Enhance and extend regression test suites
§ Develop compliance test suites
cumulusnetworks.com 21
22. ONIE Ongoing Development
§ New Features
§ DHCPv6
§ DNS Service Discovery / Multicast DNS discovery
§ HW Vendor diagnostic
§ Releases
§ Quarterly release cadence
§ Ongoing maintenance
§ Enhancements
cumulusnetworks.com 22
23. ONIE x86 Strategy
§ Use existing BIOS from hardware vendors
§ During manufacturing install GRUB2 and ONIE-x86 on
the mass storage block device
§ NOS Installer adds partitions, installs software and
updates GRUB2 configuration.
cumulusnetworks.com 23
24. Block Device Partitioning
§ Partition using GUID Partition Table (GPT) format
§ GPT supported by Linux, GRUB, UEFI
§ Supports dual-booting multiple operating systems
cumulusnetworks.com 24
Partition # Name R/W Notes
1 GRUB Boot r/o Used by GRUB and
GPT
2 ONIE BOOT r/o ONIE Kernel
3 ONIE CONFIG r/w Configuration
4+ For NOS use
29. Back Up Slides – Cumulus Background
cumulusnetworks.com 29
30. Pushing Changes Down
cumulusnetworks.com 30
CPU, RAM, Flash, etc. Switch Silicon
Front Panel Ports
lldpd
Routing Tables ARP Table
Devices
Bridge FDB Filter Tables
Bonds VLANs
LinuxKernel
Virtual Kernel Ports
Bridging
mstpd
ACLRouting Suite
Quagga
snmpd
vconfig
iptable
ebtable
ip6tableiproute2
VXLAN Bridges
Switch HAL
brctl
Switch
Driver
UserSpace
Quagga daemon,
Quagga.conf, and vtysh
CLI and /etc/network/
interfaces
switchd
31. Cumulus Linux Hardware Compatibility List
Cumulus Networks and Partner Confidential
40G Portfolio Part number Description Merchant Silicon Cumulus Linux Release
Dell S6000 32 x 40G-QSFP+ Trident II 2.1 or later
Edge-Core AS6700-32X 32 x 40G-QSFP+ Trident II 2.0 or later
Penguin Computing Arctica 3200XL 32 x 40G-QSFP+ Trident II 2.0 or later
Quanta QCT T5032-LY6* 32 x 40G-QSFP+ Trident II 2.0 or later
10G Portfolio Part number Description Merchant Silicon Cumulus Linux Release
Agema AG-7448CU 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident 1.5.0 or later
Dell S4810 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident 2.0.x or later**
Edge-Core AS5600-52X 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.0 or later
Edge-Core AS5710-54X 48 x 10G-SFP+ and 6 x 40G-QSFP+ Trident II 2.1 or later**
Edge-Core AS5710-96X 96 x 10G-SFP+ and 8 x 40G-QSFP+ Trident II 2.1 or later**
Penguin Computing Arctica 4804X 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.1 or later
Quanta QCT T-3048-LY2* 48 x 10G-SFP+ and 4 x 40G-QSFP+ Trident+ 1.5.0 or later
Quanta QCT T5048-LY8* 48 x 10G-SFP+ and 6 x 40G-QSFP+ Trident II 2.1 or later**
1G Portfolio Part number Description Merchant Silicon Cumulus Linux Release
Edge-Core AS4600-54T 48 x 1G-T and 4 x 10G-SFP+ Apollo2 2.0 or later
Penguin Computing Artica 4804i 48 x 1G-T and 4 x 10G-SFP+ Triumph2 1.5.1 or later
Quanta QCT T1048-LB9* 48 x 1G-T and 4 x 10G-SFP+ FireBolt3 1.5.0 or later
31