Open-Extension Container (OXC) Framework in Linux

1. Design and Development of Real-Time
Multi-Processor Bandwidth Control Mechanisms in
General Purpose Operating Systems
Youcheng Sun
Graduate Program in Computer Science and Engineering
September 14, 2012
Y. Sun (GPCSE) Master thesis September 14, 2012 1 / 26

2. Outline
1 What is an ideal CPU bandwidth control mechanism for Linux?
2 A brief background
3 The OXC framework
4 Game time!
5 Conclusion
Y. Sun (GPCSE) Master thesis September 14, 2012 2 / 26

3. Outline
1 What is an ideal CPU bandwidth control mechanism for Linux?
2 A brief background
3 The OXC framework
4 Game time!
5 Conclusion
Y. Sun (GPCSE) Master thesis September 14, 2012 3 / 26

4. Four golden rules
Hierarchical CPU bandwidth distribution Tasks in a Linux system
can be organized in a flat or hierarchy. In modern systems, the
hierarchyical way is preferred. A good mechanism should be able
to distribute the CPU bandwidths hierarchically.
Serve all kinds of tasks In Linux, there are different types of tasks
that scheduled by different schedulers. A good mechanism should
be able to serve all kinds of tasks.
Support multi-processor platforms
Provide real-time guaruntee Linux is used to serve various
scenarios, among which some have temporal requirements. A
good mechanism should be able to provide real-time guarantee.
Y. Sun (GPCSE) Master thesis September 14, 2012 4 / 26

5. Before our work, no existing mechanisms can satisfy
all four golden rules
Previous work includes RT throttling, CFS bandwdith control,
AQuoSA, IRMOS real-time framework, etc.
One common failure is rule 2.
Y. Sun (GPCSE) Master thesis September 14, 2012 5 / 26

6. Before our work, no existing mechanisms can satisfy
all four golden rules
Previous work includes RT throttling, CFS bandwdith control,
AQuoSA, IRMOS real-time framework, etc.
One common failure is rule 2. Each previous mechanism can
only work for a specific type of tasks in Linux.
Y. Sun (GPCSE) Master thesis September 14, 2012 5 / 26

7. Before our work, no existing mechanisms can satisfy
all four golden rules
Previous work includes RT throttling, CFS bandwdith control,
AQuoSA, IRMOS real-time framework, etc.
One common failure is rule 2. Each previous mechanism can
only work for a specific type of tasks in Linux.
In our work, a framework that satisfies all golden rules is
proposed.
Y. Sun (GPCSE) Master thesis September 14, 2012 5 / 26

8. Outline
1 What is an ideal CPU bandwidth control mechanism for Linux?
2 A brief background
3 The OXC framework
4 Game time!
5 Conclusion
Y. Sun (GPCSE) Master thesis September 14, 2012 6 / 26

9. Resource reservation
The Constant Bandwidth Server (CBS) algorithm is used to
reserve CPU bandwidth in the work.
A CBS is characterized by an ordered pair (Q, P), where Q is
called maximum budget and P is period.
A CBS can reserve Q every P time units from a CPU.
Y. Sun (GPCSE) Master thesis September 14, 2012 7 / 26

10. Linux scheduling
Linux adapts modular scheduling design. Each scheduler is an
instance of the struct sched_class.
s t r u c t sched_class {
const s t r u c t sched_class ∗n e x t ;
void (∗ enqueue_task ) ( s t r u c t r q ∗rq , s t r u c t t a s k _ s t r u c t ∗p , i n t f l a g s ) ;
void (∗ dequeue_task ) ( s t r u c t r q ∗rq , s t r u c t t a s k _ s t r u c t ∗p , i n t f l a g s ) ;
void (∗ y i e l d _ t a s k ) ( s t r u c t r q ∗r q ) ;
bool (∗ y i e l d _ t o _ t a s k ) ( s t r u c t r q ∗rq , s t r u c t t a s k _ s t r u c t ∗p , b o o l preempt ) ;
void (∗ check_preempt_curr ) ( s t r u c t r q ∗rq , s t r u c t t a s k _ s t r u c t ∗p , i n t f l a g s ) ;
s t r u c t t a s k _ s t r u c t ∗ (∗ p i c k _ n e x t _ t a s k ) ( s t r u c t r q ∗r q ) ;
void (∗ p u t _ p r e v _ t a s k ) ( s t r u c t r q ∗rq , s t r u c t t a s k _ s t r u c t ∗p ) ;
...
};
These hooks are all scheduling operations for a scheduler.
Y. Sun (GPCSE) Master thesis September 14, 2012 8 / 26

11. Linux scheduling
Linux adapts modular scheduling design. Each scheduler is an
instance of the struct sched_class.
s t r u c t sched_class {
const s t r u c t sched_class ∗n e x t ;
void (∗ enqueue_task ) ( struct rq *rq , s t r u c t t a s k _ s t r u c t ∗p , i n t f l a g s ) ;
void (∗ dequeue_task ) ( struct rq *rq , s t r u c t t a s k _ s t r u c t ∗p , i n t f l a g s ) ;
void (∗ y i e l d _ t a s k ) ( struct rq *rq ) ;
b o o l (∗ y i e l d _ t o _ t a s k ) ( struct rq *rq , s t r u c t t a s k _ s t r u c t ∗p , b o o l preempt ) ;
void (∗ check_preempt_curr ) ( struct rq *rq , s t r u c t t a s k _ s t r u c t ∗p , i n t f l a g s ) ;
s t r u c t t a s k _ s t r u c t ∗ (∗ p i c k _ n e x t _ t a s k ) ( struct rq *rq ) ;
void (∗ p u t _ p r e v _ t a s k ) ( struct rq *rq , s t r u c t t a s k _ s t r u c t ∗p ) ;
...
};
These hooks are all scheduling operations for a scheduler.
Y. Sun (GPCSE) Master thesis September 14, 2012 9 / 26

12. The Linux scheduling is runqueue centered
Every hook in a scheduling class deals with the struct rq. A
struct rq instance is called runqueue in Linux.
The struct rq is a per CPU structure. That is, there is a
runqueue per CPU in Linux.
A scheduling system is built around the runqueue in each CPU.
Different per CPU scheduling systems cooperate with each other
through task migration strategy defined by schedulers.
Y. Sun (GPCSE) Master thesis September 14, 2012 10 / 26

13. The scheme of per CPU scheduling in Linux
Y. Sun (GPCSE) Master thesis September 14, 2012 11 / 26

14. Outline
1 What is an ideal CPU bandwidth control mechanism for Linux?
2 A brief background
3 The OXC framework
4 Game time!
5 Conclusion
Y. Sun (GPCSE) Master thesis September 14, 2012 12 / 26

15. The Open Extension Container (OXC) structure
The idea of ox container is
simple: any data
structure that contains a
runqueue inside is
regarded as the ox
container
Y. Sun (GPCSE) Master thesis September 14, 2012 13 / 26

16. Pass an ox container’s local runqueue to a scheduling
operation
For a scheduler, it needs a runqueue so as to deal with its tasks.
This runqueue can be the per CPU runqueue or an ox container’s
local runqueue.
When an ox container’s local runqueue is used by a scheduler,
tasks will run in this local runqueue.
Now, besides per CPU scheduling system, there is the per ox
container scheduling system in Linux.
Y. Sun (GPCSE) Master thesis September 14, 2012 14 / 26

17. Per oxc scheduling + per CPU scheduling
coexist in Linux
Y. Sun (GPCSE) Master thesis September 14, 2012 15 / 26

18. The oxc scheduling can be structured in a hierarchy
Y. Sun (GPCSE) Master thesis September 14, 2012 16 / 26

19. To allocate CPU bandwidth for an ox container
Different schedulers can use an ox container to enqueue, operate
and dequeue their tasks.
Suppose a fraction of CPU bandiwdth is allocated to an ox
container, all kinds of tasks inside the container will use it as if
running on a less powerful CPU.
Y. Sun (GPCSE) Master thesis September 14, 2012 17 / 26

20. To allocate CPU bandwidth for an ox container
Different schedulers can use an ox container to enqueue, operate
and dequeue their tasks.
Suppose a fraction of CPU bandiwdth is allocated to an ox
container, all kinds of tasks inside the container will use it as if
running on a less powerful CPU.
Based on this idea, the oxc (scheduling) framework is developed to
control CPU bandwidth allocated to tasks, which can be any type, in
a real-time way.
Y. Sun (GPCSE) Master thesis September 14, 2012 17 / 26

21. A concrete ox container that can reserve CPU
bandwidth
This an ox container that can
reserve bandwidth from a CPU
according to CBS rules. oxc_runtime and oxc_period
are the maximum budget and pe-
s t r u c t oxc_rq { riod parameters in CBS.
unsigned long o x c _ n r _ r u n n i n g ;
int oxc_throttled ;
u64 oxc_runtime ;
ktime_t oxc_period ;
u64 o x c _ d e a d l i n e ;
u64 oxc_time ;
u64 o x c _ s t a r t _ t i m e ;
struct hrtimer oxc_period_timer ;
raw_spinlock_t oxc_runtime_lock ;
b o o l oxc_needs_resync ;
s t r u c t r q rq_ ;
s t r u c t r q ∗r q ;
s t r u c t rb_node rb_node ;
};
Y. Sun (GPCSE) Master thesis September 14, 2012 18 / 26

22. A concrete ox container that can reserve CPU
bandwidth
This an ox container that can
reserve bandwidth from a CPU oxc_runtime and oxc_period
are the maximum budget and pe-
according to CBS rules. riod parameters in CBS.
s t r u c t oxc_rq {
unsigned long o x c _ n r _ r u n n i n g ;
int oxc_throttled ;
u64 oxc_runtime ;
ktime_t oxc_period ;
u64 o x c _ d e a d l i n e ;
u64 oxc_time ;
u64 o x c _ s t a r t _ t i m e ;
struct hrtimer oxc_period_timer ;
raw_spinlock_t oxc_runtime_lock ;
b o o l oxc_needs_resync ;
s t r u c t r q rq_ ; the local runqueue of an ox con-
s t r u c t r q ∗r q ;
s t r u c t rb_node rb_node ;
tainer
};
Y. Sun (GPCSE) Master thesis September 14, 2012 18 / 26

23. SMP support
An ox container can only
reserve CPU bandwidth from a
CPU. cpus_allowed specifies the CPU
that bandwidths are reserved from.
The hyper ox container
structure is defined for
reserving bandwidths from
multiple processors.
s t r u c t hyper_oxc_rq {
cpumask_var_t cpus_allowed ;
s t r u c t oxc_rq ∗∗ oxc_rq ;
};
Y. Sun (GPCSE) Master thesis September 14, 2012 19 / 26

24. SMP support
An ox container can only
reserve CPU bandwidth from a cpus_allowed specifies the CPU
that bandwidths are reserved from.
CPU.
The hyper ox container
structure is defined for
reserving bandwidths from
multiple processors.
s t r u c t hyper_oxc_rq { The ox containers to reserve band-
cpumask_var_t cpus_allowed ; widths from corresponding CPUs
s t r u c t oxc_rq ∗∗ oxc_rq ;
};
Y. Sun (GPCSE) Master thesis September 14, 2012 19 / 26

25. Outline
1 What is an ideal CPU bandwidth control mechanism for Linux?
2 A brief background
3 The OXC framework
4 Game time!
5 Conclusion
Y. Sun (GPCSE) Master thesis September 14, 2012 20 / 26

26. Experiment set-ups
Candidates :
RT throttling : CPU bandwidth control work, without real-time
guarantee, for RT tasks.
CFS bandwidth control : CPU bandwidth control work, without
real-time guarantee, for CFS tasks.
oxc control : Our CPU bandwidth control work, with real-time
guarantee, for any type of tasks.
Experiment rules:
Two tbench connections will be set up in the system; each
connection is dedicated to one CPU in a dual processor platform.
The CPU bandwidth allocated to tbench traffic is restricted using
the above work individually.
Y. Sun (GPCSE) Master thesis September 14, 2012 21 / 26

27. Results: oxc control vs. RT throttling
Y. Sun (GPCSE) Master thesis September 14, 2012 22 / 26

28. Results: oxc control vs. CFS bandwidth control
Y. Sun (GPCSE) Master thesis September 14, 2012 23 / 26

29. Results
Given that our work is still a prototype, such results are very
encouraging.
Y. Sun (GPCSE) Master thesis September 14, 2012 24 / 26

30. Results
Given that our work is still a prototype, such results are very
encouraging.
In addition: we can provide real-time gurantee ;-)
Y. Sun (GPCSE) Master thesis September 14, 2012 24 / 26

31. Results
Given that our work is still a prototype, such results are very
encouraging.
In addition: we can provide real-time gurantee ;-)
What I am really trying to sell :
Our one-for-all solution for CPU bandwidth control in Linux
is feasible!!!
Y. Sun (GPCSE) Master thesis September 14, 2012 24 / 26

32. Outline
1 What is an ideal CPU bandwidth control mechanism for Linux?
2 A brief background
3 The OXC framework
4 Game time!
5 Conclusion
Y. Sun (GPCSE) Master thesis September 14, 2012 25 / 26

33. Conclusion
In mainline Linux there are two schedulers.
Numerous schedulers have been or will be implemented based on
Linux for various reasons.
There exists CPU bandwidth control work dealing with some of
these schedulers. But each such mechanism can only work with
for one scheduler (one specific type of tasks).
Y. Sun (GPCSE) Master thesis September 14, 2012 26 / 26

34. Conclusion
In mainline Linux there are two schedulers.
Numerous schedulers have been or will be implemented based on
Linux for various reasons.
There exists CPU bandwidth control work dealing with some of
these schedulers. But each such mechanism can only work with
for one scheduler (one specific type of tasks).
The oxc framework can provide real-time CPU bandwidth
control for tasks of all these schedulers!
The project is just started; more work is coming.
Y. Sun (GPCSE) Master thesis September 14, 2012 26 / 26