Internet Research Lab at NTU, Taiwan.
SIGCOMM HotSDN 2012 is the first conference workshop focused on SDN. This presentation provides a survey of selected papers in HotSDN'12 and revisits concepts and frameworks of SDN. Example applications are also presented.
Boost PC performance: How more available memory can improve productivity
Software-Defined Networking , Survey of HotSDN 2012
1. SDN Concepts and Applications
Survey of SIGCOMM HotSDN 2012
Presenter: Jason, Tsung-Cheng, HOU
Advisor: Wanjiun Liao
Sep. 20th, 2012
1
2. Agenda
• Background
• Review of Onix, Net OS / Controller
• Hierarchical Policies for SDN
• State Distribution Tradeoffs
• Offloading Control Applications
• Verifying Network-wide Invariants
• Debugger and Network Backtrace
• Programming Network for Online Applications
• Programmable Enterprise WLANs
• Research Directions 2
3. Background
• SDN: Software-Defined Networking
– Refactoring networks by offering control platforms
– A unified system-layered abstractions
– Programmability, flexibility, and extensibility
• Network OS / Controller:
– Handling important but tedious functions
Configuring net devices, channeling states
– Presenting net abstractions, states and graphs
– Inter-network-applications control, providing API
c.f. inter-process control
3
5. SDN Abstraction / Application
5
http://onrc.stanford.edu/research_unified_control_architecture.html
6. SDN Research?
• How do we scope research in SDN?
• SIGCOMM HotSDN 2012
– First Workshop in SDN
– Acceptance: 22 of 71 submissions, 31%
– 5 Stanford/Berkeley, 2 UIUC, 1 Princeton/Cornell
– 1 Nicira, 2 IBM, 2 HP, 1 Microsoft
• Mostly, implementations of newly proposed
systems, frameworks, or applications
• Very few modeling or theoretical work
• Better understanding of SDN
6
7. Agenda
• Background
• Review of Onix, Net OS / Controller
• Hierarchical Policies for SDN
• State Distribution Tradeoffs
• Offloading Control Applications
• Verifying Network-wide Invariants
• Debugger and Network Backtrace
• Programming Network for Online Applications
• Programmable Enterprise WLANs
• Research Directions 7
8. Concept of Virtualization
• Decoupling HW/SW by abstraction & layering
• Using, demanding,
but not owning or configuring
• Resource pool: flexible to slice, resize,
combine, and distribute
• A degree of automation by software
HOST 1 HOST 2 HOST 3 HOST 4,
VMs
Hypervisor:
Turns 1 server into many “virtual machines” (instances or VMs)
(VMWare ESX, Citrix XEN Server, KVM, Etc.) 8
10. Network Controller / OS
• DCN relies on controller for many functions:
– Address mapping / mgmt / registration / reuse
– Traffic load scheduling / balancing
– Route computation, switch entries configuration
– Logical network view ↔ physical construction
• An example: Onix
– Distributed system
– Maintain, exchange &
distribute net states
• Hard static: SQL DB
• Soft dynamic: DHT
– Asynchronous but
eventually consistent
10
11. Onix Functions
Control Plane / Applications
API
Provides
Abstraction
Logical Forwarding Plane / Network Graph
Control Logical States
Provides Commands Abstractions
Distributed Mapping Network Info Base
System Network Hypervisor Network OS
Distributes, Configures Real States
OpenFlow
11
12. Abstraction Layer
Net Control MapReduce / GFS Cluster / Tenant
Applications Master Master
Network Programming API
Language-Policy Interpreter
Network Graph
Policy-Rule Compiler Network Info Base
Configuration Manager Object DB
Verification-Debugger
Backtrace Collector
12
13. Agenda
• Background
• Review of Onix, Net OS / Controller
• Hierarchical Policies for SDN
• State Distribution Tradeoffs
• Offloading Control Applications
• Verifying Network-wide Invariants
• Debugger and Network Backtrace
• Programming Network for Online Applications
• Programmable Enterprise WLANs
• Research Directions 13
14. Hierarchical Policies
• Useful in many contexts in which resources
are shared among multiple entities.
• Express delegation of authority and the
resolution of conflicts
• Hierarchical Flow Tables (HFT): a framework
for specifying hierarchical policies in SDN
• Also proposed a compiler and runtime system
to realize policy trees
AD Ferguson et. al., “Hierarchical policies for software defined networks”, HotSDN 2012
14
15. Hierarchical Policies
A Policy Tree
Types of Hierarchy:
• Policy Priority, ex: Access Control > Routing
• IP Prefix, Longest Matching, Subnets
• Flow / Traffic Type, ex: Control PKT > Data 15
16. Hierarchical Flow Table
A rigorous definition attempting
to formalize network policies.
←Observe the symbols used by
authors here !
Matches a pkt with a policy, node internal
conflict resolution
Matches a pkt with a policy tree, using cmb recursively to
children, parent-child or inter-sibling conflict resolution
Declare policies
↓
Construct HFT
↓
Linearize & Compile HFT
Turn into flow entries on devices 16
17. Runtime HFT
Turn policy trees into
priority-indexed linear
and high-level flow
tables ↓
Turn high-level flow
tables into device
entries →
17
18. Abstraction Layer
Net Control MapReduce / GFS Cluster / Tenant
Applications Master Master
Network Programming API
Language-Policy Interpreter
Network Graph
Policy-Rule Compiler Network Info Base
Configuration Manager Object DB
Verification-Debugger
Backtrace Collector
18
19. Agenda
• Background
• Review of Onix, Net OS / Controller
• Hierarchical Policies for SDN
• State Distribution Tradeoffs
• Offloading Control Applications
• Verifying Network-wide Invariants
• Debugger and Network Backtrace
• Programming Network for Online Applications
• Programmable Enterprise WLANs
• Research Directions 19
20. Logically Centralized
• Logically centralized: multiple physical
machines hosting distributed controllers
• Net states are distributed among controllers
– What are the impact of inconsistent network view?
– How does distributed SDN states impact
performance of a logically centralized control
application?
• Depict state exchange points and trade-offs
• Provide a customized flow level simulator
D Levin et. al., “Logically centralized?: state distribution trade-offs in software
defined networks”, HotSDN 2012 20
21. State Exchange Points
State exchange points:
• App. state mgmt layer (Net. OS/Hypervisor)
• State mgmt layer network devices 21
22. Trade-Offs
• Strongly consistent
– Imposes overhead and delay, and limits
responsiveness, can lead to suboptimal
• Eventually consistent:
– Presents a temporarily inconsistent view, may
cause incorrect behavior
• Trade-off:
– performance (optimality), vs state distribution
overhead (staleness)
– Application complexity, vs robustness to
inconsistency in states
22
23. Results
• Simulation on very simple topo and workload
• Only discuss impacts on load-balancing app
• View staleness significantly impacts optimality
• Application robustness to inconsistency
increases when the application logic is aware
of distribution
• Not a very complete discussion, scenarios
are relatively simple
23
24. Abstraction Layer
Net Control MapReduce / GFS Cluster / Tenant
Applications Master Master
Network Programming API
Language-Policy Interpreter
Network Graph
Policy-Rule Compiler Network Info Base
Configuration Manager Object DB
Verification-Debugger
Backtrace Collector
24
25. Agenda
• Background
• Review of Onix, Net OS / Controller
• Hierarchical Policies for SDN
• State Distribution Tradeoffs
• Offloading Control Applications
• Verifying Network-wide Invariants
• Debugger and Network Backtrace
• Programming Network for Online Applications
• Programmable Enterprise WLANs
• Research Directions 25
26. Offloading Control ?
• Frequent and resource-exhaustive events:
– Such as flow arrivals and network state collection
– Stress the control plane and limit scalability
– As discussed in DevoFlow
• Current solutions: View as intrinsic, or try to
address by modifying switches
• How can we move control functionalities
toward datapaths, without introducing new
datapath mechanisms to switches?
S Hassas Yeganeh et. al., “Kandoo: a framework for efficient and scalable
offloading of control applications”, HotSDN 2012 26
27. Local Controllers
DevoFlow: Devolve tasks to switches
Here, perhaps switches are equipped with local controllers
Or, more local, light controllers installed near datapaths
What’s the difference?
Programmability and delay 27
28. Examples
• Does not need global network states • Shield from root controller
• Proxies of root controller • Local controllers do not propagate an
• Scale with network size event, unless root controller subscribes
• Hierarchy of controllers • Consume less bandwidth
28
29. Agenda
• Background
• Review of Onix, Net OS / Controller
• Hierarchical Policies for SDN
• State Distribution Tradeoffs
• Offloading Control Applications
• Verifying Network-wide Invariants
• Debugger and Network Backtrace
• Programming Network for Online Applications
• Programmable Enterprise WLANs
• Research Directions 29
30. Why Online Verifying?
• SDN eases development of network
applications, but bugs are still problematic
– Complexity of software will increase.
– SDN allows multiple applications or even multiple
users to program the same physical network,
potential conflicting rules
• Rule verification latency in seconds is not
enough to ensure real-time response
– Require network-wide states
– Processing churn introduce scaling challenges.
A Khurshid et. al., “VeriFlow: verifying network-wide invariants in real time”,
HotSDN 2012 30
31. VeriFlow Framework
• Prototype: Only (forwarding rule,
reachability) verification
• Equivalent classes: rules or policies
affect same type of packets or
flows, verify per class not per rule
• Rules in equivalent class are stored
as trie or prefix tree
• Generate forwarding graphs for
each class
• Processing reachability verification
by traversing forwarding graph
• Incremental, only check latest
changes, fast enough
31
32. Agenda
• Background
• Review of Onix, Net OS / Controller
• Hierarchical Policies for SDN
• State Distribution Tradeoffs
• Offloading Control Applications
• Verifying Network-wide Invariants
• Debugger and Network Backtrace
• Programming Network for Online Applications
• Programmable Enterprise WLANs
• Research Directions 32
33. Why Debugging ?
• When debugging a program:
– Pause execution at a breakpoint
– Shows the history of function calls leading to
breakpoint, a backtrace
• What bugs in SDN?
– Race condition: policy on-the-way, behind flows
– Controller logic error
– Switch implementation or policy insertion
N Handigol et. al., “Where is the debugger for my Software-Defined Network?”,
HotSDN 2012 33
36. Mechanisms
• Proxy: modify flow entry modification msg to
cmd switches producing postcards
• Flow Table State Recorder (FTSR): stores flow
table versions w.r.t. flow entry modifications
• Breackpoint: user-specified filtering rules
• Postcard: a truncated copy of the packet’s
header, augmented with the matching flow entry,
switch, and output port (links back to FTSR)
• Keep sending postcards to controller, all along
the way a packet traverses
• Reconstruct backtrace when a packet or a
postcard matches a filter breakpoint
36
40. Abstraction Layer
Net Control MapReduce / GFS Cluster / Tenant
Applications Master Master
Network Programming API
Language-Policy Interpreter
Network Graph
Policy-Rule Compiler Network Info Base
Configuration Manager Object DB
Verification-Debugger
Backtrace Collector
VeriFlow vs Debugger: Policy verification vs errant event tracing
40
41. Agenda
• Background
• Review of Onix, Net OS / Controller
• Hierarchical Policies for SDN
• State Distribution Tradeoffs
• Offloading Control Applications
• Verifying Network-wide Invariants
• Debugger and Network Backtrace
• Programming Network for Online Applications
• Programmable Enterprise WLANs
• Research Directions 41
42. Network Sensitive Applications
• Tightly integrate applications with the network
to improve performance and utilization
• To support cross-layer network control, the
SDN controller is interfaced to the master
node for each individual application, such as
the Hadoop scheduler or HBase master
• For big data applications, the SDN controller
provides an interface that accepts traffic
demand matrices from application controllers
G Wang et. al., “Programming your network at run-time for big data applications”,
HotSDN 2012 42
43. MapReduce Distributed Execution
Master also deals with:
• Worker status updates
User
• Fault-tolerance
Program
• I/O Scheduling
fork fork • Automatic distribution
fork
• Automatic parallelization
Master
assign assign
map reduce
Input Data Worker
write Output
local Worker File 0
Split 0
read write
Split 1 Worker
Split 2 Output
Worker File 1
Worker remote
read,sort
↑↑↑↑↑
Shuffle & Sort
46. Masters Work Together
How to coordinate configuration requests
among different kinds of applications?
46
47. Agenda
• Background
• Review of Onix, Net OS / Controller
• Hierarchical Policies for SDN
• State Distribution Tradeoffs
• Offloading Control Applications
• Verifying Network-wide Invariants
• Debugger and Network Backtrace
• Programming Network for Online Applications
• Programmable Enterprise WLANs
• Research Directions 47
48. Programmable WLAN
• In WiFi settings, clients choose what APs to
associate, broadcast, hard to track devices
• Interactions among APs and clients are
decentralized, hard to add functionalities
• Build a virtual AP abstraction to simplify client
management, without any client modificatoins
• Virtual AP: enables seamless mobility and
load balancing
L Suresh et. al., “Towards programmable enterprise WLANS with Odin”, HotSDN
2012 48
51. Reference
• Teemu Koponen et al., “Onix: A distributed control platform for large-scale production
networks”, OSDI, Oct, 2010
• AD Ferguson et. al., “Hierarchical policies for software defined networks”, HotSDN 2012
• D Levin et. al., “Logically centralized?: state distribution trade-offs in software defined
networks”, HotSDN 2012
• S Hassas Yeganeh et. al., “Kandoo: a framework for efficient and scalable offloading of
control applications”, HotSDN 2012
• A Khurshid et. al., “VeriFlow: verifying network-wide invariants in real time”, HotSDN 2012
• N Handigol et. al., “Where is the debugger for my Software-Defined Network?”, HotSDN
2012
• G Wang et. al., “Programming your network at run-time for big data applications”, HotSDN
2012
• L Suresh et. al., “Towards programmable enterprise WLANS with Odin”, HotSDN 2012
51
Editor's Notes
This is the distributed overview1.User run it’s program, the input data will be split into many pieces, each pieces is 64MB2. The program will copy to many machine, one of those is master, and the master will assign some worker be mapper, some workers be reducer.3. Mapper will red the content of the corresponding input split. It pass each key-value pair to the map function, and the intermediate generate by mapper will be stored in memory4. The mapper will write intermediate data to local disk periodically. 5. After all mapper finished. The reducer will read the corresponding intermediate data and sort each key-value pair by key. This make sure that the data with same key will be group together.6. The reducer run reduce function and output the result.7. When all map tasks and reduce task finish, the mapreduce job is finished.