Just my master thesis presentation. It is about the implementation of the BandWidth Inheritance real-time synchronization protocol within a QoS framework for Linux called AQuoSA (http://www.aquosa.sf.net). Nothing particularly refined (just master thesis quality :-P), but it was fun to do it!
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
1. Dario Faggioli, October 2 2007
Università di Pisa Facoltà di Ingegneria
Corso di Laurea Specialistica in
Ingegneria Informatica
Tesi di Laurea Specialistica
Implementation and study
of the BandWidth Inheritance protocol
in the Linux kernel
Candidato: Relatori:
Dario Faggioli Prof. Paolo Ancilotti
Prof. Giuseppe Lipari
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
2. Implementation and study of the
BandWidth Inheritance protocol in the Linux kernel
Real-time systems
− Definitions
− Real-time scheduling and resource sharing
− Server based scheduling and resource sharing
− Soft real-time systems, QoS and Resource Reservation
The AquoSA Framework
− Framework objectives and design goals
− AquoSA Framework architecture and usage
BWI implementation inside AquoSA
− Design and goals
− Specific contribution
− Implementation characteristics
BWI implementation verification
BWI experimental simulations results
BWI implementation overhead evaluation
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
3. Implementation and study of the
BandWidth Inheritance protocol in the Linux kernel
Real-time systems
− Definitions
− Real-time scheduling and resource sharing
− Server based scheduling and resource sharing
− Soft real-time systems, QoS and Resource Reservation
The AquoSA Framework
− Framework objectives and design goals
− AquoSA Framework architecture and usage
BWI implementation inside AquoSA
− Design and goals
− Specific contribution
− Implementation characteristics
BWI implementation verification
BWI experimental simulations results
BWI implementation overhead evaluation
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
4. Real-time systems
Definitions
Some definitions of a real-time system
− A set of activities with timing requirements
− A system interacting with the external environment which directly imposes its rigid
requirements on timing behaviour, usually as constraints on response and/or worst
case computation time
− A system the correctness of which depends both on the accuracy of the
computation results as well as on the time at which those results are produced and
presented as output
Differences with “traditional” systems
− “traditional” systems aims at achieving maximum possible throughput, fairness and
complete exploitation of available resources
− real-time systems aims at achieving predictability and determinism
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
5. Real-time systems
Definitions
Hard real-time systems
An hard real-time system completely fail if its timing constraints are not
honored, even only one single time
An hard real-time system typically handles critical activities
− flight control systems
− defense systems
− nuclear power plants control systems
− ...
An hard real-time system causes, if it fails, catastrophic consequences
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
6. Real-time systems
Definitions
Soft real-time systems
A soft real-time system can tolerate some miss to happen considering such an
event only a temporary failure
A soft real-time systems typically are are non critical systems
− audio/video players
− multimedia applications
− (multimedia) streaming applications
− VR or video games and entertainment systems
− ...
A soft real-time system produces, if occasional constraints violations occur, no
cataclysmic effects
A soft real-time system produces, if occasional constraints violations occur, a
degradation of the Quality of Service perceived by the user
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
7. Real-time systems
Definitions
Real-time theory
the scientific discipline that provides solutions to the problem of being able to design
systems with full predictable timing behavior, also showing (mathematically)
evidences that they meets all their timing requirements
Real-time tasks
An “activity” to be performed inside a real-time system is usually called a task and is
characterized by some parameters
periodic task ti, job j and j+1 non periodic task ti, job j and j+1
ai,j ci,j fi, j di,j ai,j ci,j fi, j di,j ai,j+1 c fi, j+1
i,j+1
ai,j+1 ci,j+1 fi, j+1 di,j+1 Ti,j,j+1
Di,j=Ti Di di,j+1
ci max j {c i, j }
Utilisation factor (for each task) U i= =
T i min j { T i, j, j+1 }
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
8. Real-time systems
Real-time scheduling and resource sharing
Real-time scheduling
Trying to find an order of execution of all the jobs of all the tasks in the system so that
any of them meets their deadlines
Real-time scheduling algorithms
− Rate Monotonic (RM)
tasks are executed in priority order, with the highest priority task being the one with
the shorter period
1
c
U = U i = < n2 n −1
i
T i
− Earliest Deadline First (EDF)
tasks are executed in priority order, with the highest priority task being the one with
the earliest (absolute) deadline
c
U = U i = <1
i
T i
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
9. Real-time systems
Real-time scheduling and resource sharing
Sharing resources and synchronization between tasks
In complex systems tasks may need to compete or cooperate by means of shared
resources collocated in shared memory areas and protected by mutex semaphores
providing mutual exclusive access
Priority inversion
The use of classical semaphores is prone to the well known priority inversion problem
an high priority task is blocked by a low priority one for an unbounded amount of time due to
the interference of some other task, with medium priority between the two
Caused problem also during Mars Pathfinder mission !!
Priority Inversion avoidance:
− Non Preemptive Protocol (NPP) T1 WR SR
R
− Highest Lock Priority Protocol
(HLPP) T2
− Priority Inheritance Protocol (PIP)
− Priority Ceiling Protocol (PCP) T3 WR SR
R R R
− Stack Resource Policy (SRP)
Priority inversion !
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
10. Real-time systems
Real-time scheduling and resource sharing
PIP rules
The PIP achieve its objectives through two simple rules
− when a task TH is blocked accessing a critical section whose mutex is already
locked by TL (and if TH has higher priority than TL) then TL inherits the priority of TH
− when TL releases the mutex it is returned the priority it had before acquiring it
PIP Example
T1 WR SR
R
T2 T1 WR SR
R
T3 WR SR
T2
R R R
Priority inversion ! T3 WR SR
R R
Priority inheritance
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
11. Real-time systems
Server based scheduling and resource sharing
Server based scheduling
A server Si is an entity characterized three parameters
− Qi the budget or capacity
− Pi the period
Qi
Ui=
− the server bandwidth
Pi
Servers are the actual scheduling entities
A server serves one or more tasks
T1
− hard or soft Hard T2
tasks EDF
− periodic, sporadic or aperiodic T3
ready queue
CPU
− typically T4 S1
Soft
T5
sporadic soft real-time tasks tasks S2
T6 S3
aperiodic non real-time tasks
When a server is scheduled its tasks are given a chance to run
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
12. Real-time systems
Server based scheduling and resource sharing
Server based scheduling
While a server executes its actual budget is decreased
When a server's budget reaches zero it is periodically recharged
It is usually possible to ensure a task it will run at least for Qi units of time
during each Pi period (depending on the server algorithm)
Some server algorithms
− Polling Server (PS),
− Deferrable Server (DS)
− Sporadic Server (SS)
− Total Bandwidth Server (TBS)
− Constant Bandwidth Server (CBS)
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
13. Real-time systems
Server based scheduling and resource sharing
Constant Bandwidth Server (CBS) based scheduling
Each CBS server is also characterized by
− qi (qi(t)) the current server budget at time instant t
− di (di(t)) the current server deadline at time instant t
The CBS algorithm works as follows:
− CBS initialization: for each server Si, qi=Qi , di=0
− CBS rule A: when a new job of Ti, within Si, arrives the server
checks qi ( di−ai )U i
if true nothing has to be done
if false qi=Qi, di=di+Pi
− CBS rule B: when a Si executes for Dt time units qi=qi-Dt
− CBS rule C: when qi=0 qi=Qi , di=di+Pi
CBS fundamental property
The contribution of the tasks served by Si to U is guaranteed to be not greater than Ui
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
14. Real-time systems
Server based scheduling and resource sharing
Some CBS properties
The basic ideas behind the algorithm are
− when a request for a task execution arrives it is assigned a deadline calculated
taking into account its server's bandwidth
− when a task tries to execute more than what it is expected its deadline is postponed,
so that it is slowed down
The algorithm works
− without requiring the WCET to be a-priori known
− without requiring the period or MIT to be a-priori known
The algorithm provides the Bandwidth Isolation Property (BIP)
− the fraction of the CPU bandwidth of a CBS server is U=Q/P
independently from the computation time of the served tasks
independently from the arrival time pattern of the served tasks
− the fraction of the CPU bandwidth the server can exploit is U=Q/P
independently from the computation time of other tasks in the system
independently from the arrival time pattern of other tasks in the system
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
15. Real-time systems
Server based scheduling and resource sharing
Sharing Resources with CBS
Within CBS algorithm no resource sharing is allowed
if a task blocks itself we are no longer sure the executing server
is the one with the earliest deadline
PIP can not be “simply” applied here
− what happens if SL exhausts its budget
after it inherited SH's priority (deadline)?
S1=(4, 16) WR SR
− which deadline has to be postponed R
and whose budget recharged ?
This leads to S2=(4, 24)
− priority inversions ! WR SR
S3=(9, 36)
R R R
− BIP breakout !
deadline miss !
S4=(1, 4)
0 2 4 6 8 10 12 14 16 18
Priority
inversion !
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
16. Real-time systems
Server based scheduling and resource sharing
BandWidth Inheritance protocol
Only adds two rules to the CBS algorithm
− CBS rule D: when Ti blocks while accessing R owned by Tj, then Tj is added to the
list of tasks served by the server Si.
If even Tj is blocked the chain of blocked tasks is followed and all the encountered
tasks are added until one that is not blocked is found
− CBS rule E: when Tj releases R and Ti is resumed the server Si discards Tj from its
list and all the servers that has previously added Tj replace it with Ti
deadline postponements
S1=(2, 6) WR SR SR
R R R
S2=(2, 6)
S3=(6, 18) WR
R
0 2 4 6 8 10 12 14 16 18 20
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
17. Real-time systems
Server based scheduling and resource sharing
BandWidth Inheritance protocol
Natural extension of Priority Inheritance Protocol toward the CBS
− bandwidth isolation property
− bounded blocking delay and no priority inversions
Real inheritance protocol
when a task blocks himself on a mutex the lock-owner inherits its whole server
Addresses the open issues of PIP and CBS
− what happens if SL exhausts its budget deadline postponements
after it inherited SH's priority (deadline)?
A task is able of being scheduled with S1=(2, 6) WR SR SR
R R R
the earliest deadline between all the
eligible servers it has been added to
S2=(2, 6)
− which deadline has to be postponed
and whose budget recharged ?
They are the ones of the server S3=(6, 18) WR
in which the task is executing R
0 2 4 6 8 10 12 14 16 18 20
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
18. Real-time systems
Server based scheduling and resource sharing
BandWidth Inheritance protocol and isolation
Interacting tasks
− Ti and Tj have to be considered interacting tasks if
Ti and Tj share a resource R
a chain of tasks T1, T2, T3, ... Tn exists such that Ti=T1 and Tj=Tn and,
for each k=1, 2, 3, ..., n, Tk and Tk-1 share a resource Rk
BWI is able to provide isolation among non-interacting task
It is impossible to provide isolation among interacting tasks !!
S1=(4, 16) WR SR S1=(4, 16) WR SR
R R R
S2=(4, 24) S2=(4, 24)
WR SR W SR
S3=(9, 36) S3=(9, 36) RR
R R R R
deadline miss !
S4=(1, 4) S4=(1, 4)
0 2 4 6 8 10 12 14 16 18 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
19. Real-time systems
Soft real-time systems, QoS and Resource Reservation
Building a soft real-time system
Soft real-time applications are characterized by more relaxed constraints
− some deadline misses can be tolerated up to a certain degree
− control on the system performance and resource usage is still required
The main goal dealing with soft real-time applications is not to prevent all the
deadline to be missed, beside to keep this phenomenon under control
What it is desirable in a soft real-time system is
− maximize the utilization of the system resources (not too pessimistic assumptions)
− remove any need of precise a-priori knowledge about the applications
− achieve graceful degradation of the performances
− provide isolation to each application, so that an overrun in any of them do not
influence the normal execution of the others
An hard real-time approach would mean an unacceptable waste of resource
and it is definitely not appropriate
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
20. Real-time systems
Soft real-time systems, QoS and Resource Reservation
Soft real-time systems and Quality of Service (QoS)
The user's satisfaction is a decreasing function of
− the number of the timing constraints violations
− the severity (e.g. mean tardiness) of the constraints violations
An optimal measure of the provided QoS could right be the
ratio of deadline miss over a given time interval
the lower the deadline miss ratio, the better the QoS
The main challenges with soft real-time systems are
− increasing efficiency in resource usage
− keep the overall cost of the system low
− removing any assumptions on the exact knowledge of execution times
− ensure some guarantees on the performances of the system
not an absolute “yes-or-no” as by the notion of QoS
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
21. Real-time systems
Soft real-time systems, QoS and Resource Reservation
Resource Reservation
A very effective approach to soft real-time systems
− temporal protected (isolated) access to shared system's resources (e.g. CPU)
each application is reserved a fraction of a system's resource utilisation time
its ability to meet the timing constraints is not influenced by any other application
Each soft real-time task is assigned a fraction of the CPU time (bandwidth)
and is scheduled in such a way it will never demand more that its bandwidth
If the task requires to execute more than its bandwidth it is slowed down
− overruns occurring on Ti will only affect and delay the execution of Ti itself
− no one is able to steal any crumb of bandwidth reserved to the other tasks
No reciprocal tasks interference allowed
− a task executes on a dedicated virtual processor running at a fraction of the speed
of the real one equal to its reserved bandwidth
− the problem of schedulability analysis reduces to the estimation of the computation
time of a task, without considering the rest of the system, in order to assign it the
proper bandwidth
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
22. Real-time systems
Soft real-time systems, QoS and Resource Reservation
Resource Reservation and CBS
We can effectively realize the RR utilising the CBS algorithm
The BIP provided by the CBS is the perfect vehicle for providing temporal isolation
Resource Reservation and BWI
We can also utilise the BWI-modified CBS algorithm
− the arrival times of the tasks need no to be known
− the execution times of the tasks need not to be know
− the mutual exclusive resources a task will access is not to be known
− the duration of each critical section is not to be known
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
23. Implementation and study of the
BandWidth Inheritance protocol in the Linux kernel
Real-time systems
− Definition
− Real-time scheduling and resource sharing
− Server based scheduling and resource sharing
− Soft real-time systems, QoS and Resource Reservation
The AquoSA Framework
− Objectives and design goals
− Architecture and usage
BWI implementation inside AquoSA
− Design and goals
− Specific contribution
− Implementation characteristics
BWI implementation verification
BWI experimental simulations results
BWI implementation overhead evaluation
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
24. The AQuoSA Framework
Objectives and design goals
AquoSA Framework
Adaptive Quality of Service Architecture (AquoSA)
− Free Software project developed at the Real-Time Systems Laboratory
(RETIS Lab, Scuola Superiore Sant'Anna, Pisa)
− aims at providing Quality of Service management on a GNU/Linux system
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
25. The AQuoSA Framework
Objectives and design goals
AquoSA Framework
What GNU/Linux offers to soft real-time applications
− a low-latency and fully preemptable kernel
− high-resolution timing support
− (almost) all the real-time capabilities defined in the POSIX real-time extensions
What GNU/Linux lacks
− a predictable scheduler with timing and/or QoS guarantees
− a temporal protection and isolation mechanism so that
an application can access the soft real-time scheduler
but not exploit it to starve all the other applications in the system
an application can have the guarantees it has been provided respected
in all system's load conditions and independently on misbehaviour of any other one
AQuoSA wants to fill this gap
provides a standard GNU/Linux system with all it's needed to offer time-sensitive
applications a guaranteed behaviour and the temporal protection property
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
26. The AQuoSA Framework
Objectives and design goals
Some AQuoSA design goals
AQuoSA is developed trying to achieve the following key goals
− Portability:
layered structure where kernel dependent code is confined in the lowermost level
− Backward compatibility:
no change to the API and ABI of a standard Linux kernel
− Flexibility:
easy to introduce new scheduling algorithms and policies
− Efficiency:
overhead introduced reduced to minimal measure
− Security:
control on the assigned bandwidth by the system administrator in order to prevent
erroneous or voluntary denial of services to happen
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
27. The AQuoSA Framework
Architecture and usage
AquoSA architecture
Resource Reservation (RR) layer
− provides the system with
the EDF scheduler
the RR mechanism (IRIS)
the reservation supervisor QoS Managed QoS Reserved GNU/Linux
Application Application Application
− is responsible for all that regards
the reservations creation
the reservations deletion QoS Manager
layer
the enforcement of the timing behaviour QoS Resource Reservation
of the reserved tasks layer
QoS Manager layer GNU/Linux system
a set of predictor and controllers for
feedback based scheduling
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
28. The AQuoSA Framework
Architecture and usage
AQuoSA architecture and components:
The architecture is realized by the following main components:
− Generic Scheduler Patch (GSP):
a small patch to the Linux kernel extending the Linux scheduler functionalities
− Kernel Abstraction Layer (KAL):
a set of functions and macros abstracting all we need from the kernel
− QoS Reservation component:
resource reservation kernel module (rresmod) which implements EDF and the RR
scheduling algorithms
resource reservation supervisor kernel module (qresmod) which grants no system
overload to occur
resource reservation and supervisor libraries (qreslib and qsuplib) which provides the
application level APIs
− QoS Manager component:
QoS manager loadable kernel module (qmgrmod)
QoS manager library (qmgrlib)
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
29. The AQuoSA Framework
Architecture and usage
AQuoSA architecture and components:
● The architecture is realized by the following main components:
● Generic Scheduler Patch (GSP):
a small patch to the Linux kernel extending the Linux scheduler functionalities
● Kernel Abstraction Layer (KAL):
a set of functions and macros abstractingApplication need from the kernel
QoS Managed
Application
all we
QoS Reserved GNU/Linux
Application
● QoS Reservation component:
user-space
● resource reservation kernel module (rresmod) which implements EDF and the RR scheduling
algorithms qmgrlib qsuplib qreslib glibc
● resource reservation supervisor kernel module (qresmod) which grants no system overload to
occur QoS
Manager qmgrmod
component
qresmod
● resource reservation and supervisor libraries (qreslib and qsuplib) which provides the
application level APIs EDF
Scheduler
rresmod
● QoS Manager component: kernel-space
Kernel Abstraction Layer
● QoS manager loadable kernel module (qmgrmod)
QoS
Reservation
component
● QoS manager library (qmgrlib) GSP
Linux kernel
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
30. The AQuoSA Framework
Architecture and usage
AquoSA usage
QoS resource reservation library
− from within application code
From the Command Line Interface
qres and qres-wrap utilities
with all the same
API functionalities
From the Graphical User Interface
AQuoSA Monitor much more
user friendly interface
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
31. Implementation and study of the
BandWidth Inheritance protocol in the Linux kernel
Real-time systems
− Definition
− Real-time scheduling and resource sharing
− Server based scheduling and resource sharing
− Soft real-time systems, QoS and Resource Reservation
The AquoSA Framework
− Framework objectives and design goals
− AquoSA Framework architecture and usage
BWI implementation inside AquoSA
− Design and goals
− Specific contributions
− Implementation characteristics
BWI implementation verification
BWI experimental simulations results
BWI implementation overhead evaluation
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
32. BWI implementation inside AquoSA
Design and goals
BWI implementation
Before this work AQuoSA offered no support for critical sections access
when a task blocks it is assumed an instance is finished !!
We started this work with the following specifications
− to realize a full and functional implementation of the BWI protocol inside AQuoSA
− to make it possible to enable or disable the protocol at compile time
− to make it be possible to use or not use the protocol at run time
(for each specific mutex and critical section)
− to alter as little as possible impact as possible the AQuoSA code and architecture
− to add as little as possible overhead both in space (memory occupation)
and in execution time
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
33. BWI implementation inside AquoSA
Specific contributions
Some contributions
Lightweight deadlock check
Since each time a task Ti blocks we walk through the whole chain of blocked tasks
starting from it, it is sufficient we to encounter Ti again to identify a deadlock
situation with zero overhead
Adding only one task
− As a performance optimization each time Ti blocks and we follow the chain of
blocked tasks till a running one, we can add only such an able to run task
− This means, since each task Ti can, at each time instant t, be blocked at most by
only one other task Tj, the following property always holds:
BWI one blocked task property:
at each time instant t each task Ti is being blocked by at most one task Tj,
and so only Tj is executing inside its server Si due to Ti itself
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
34. BWI implementation inside AquoSA
Specific contributions
Some contributions
Nested critical section handling enhancement
A slight but effective modification of the algorithm
− the CBS rule D does not account for the case a task blocks when it already has
been added to some servers
(i.e. while holding at least a mutex with at least one another blocked task)
− we have modified it into the CBS rule D' (and implemented the latter)
CBS rule D': when Ti blocks while accessing R owned by Tj, then Tj is added to all the
list of tasks of all the servers which also serves Ti.
If even Tj is blocked the chain of blocked task is followed and all the encountered
tasks are added to all these servers' list, until one that is not blocked is found
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
35. BWI implementation inside AquoSA
Specific contributions
Some contributions
Nested critical section handling enhancement
A slight but effective modification of the algorithm
− the CBS rule D does not account for the case a task blocks when it already has
been added to some servers
(i.e. while holding at least a mutex with at least one another blocked task)
− we have modified it into the CBS rule D' (and implemented the latter)
CBS rule D': when Ti blocks while accessing R owned by Tj, then Tj is added to all the
list of tasks of all the servers which also serves Ti.
If even Tj is blocked the chain of blocked task is followed and all the encountered
tasks are added to all these servers' list, until one that is not blocked is found
CBS rule D: when Ti blocks while accessing R owned by Tj,
then Tj is added to the list of tasks served by the server Si.
If even Tj is blocked the chain of blocked tasks is followed
and all the encountered tasks are added until one
that is not blocked is found
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
36. BWI implementation inside AquoSA
Specific contributions
Some contributions
Nested critical section handling enhancement
A slight but effective modification of the algorithm
− the CBS rule D does not account for the case a task blocks when it already has
been added to some servers
(i.e. while holding at least a mutex with at least one another blocked task)
− we have modified it into the CBS rule D' (and implemented the latter)
CBS rule D': when Ti blocks while accessing R owned by Tj, then Tj is added to all the
list of tasks of all the servers which also serves Ti.
If even Tj is blocked the chain of blocked task is followed and all the encountered
tasks are added to all these servers' list, until one that is not blocked is found
CBS rule D: when Ti blocks while accessing R owned by Tj,
then Tj is added to the list of tasks served by the server Si.
If even Tj is blocked the chain of blocked tasks is followed
and all the encountered tasks are added until one
that is not blocked is found
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
37. BWI implementations inside AquoSA
Implementation characteristics
Description of the implementation
Non intrusive approach
Do not modify the core features of AquoSA, that is
the scheduler and scheduler related functions
the RR enforcing mechanisms
the servers handling (list of tasks, dispatching, etc.) mechanisms
BWI added functions only called in very few situations
− when a task is attached or detached to/from a server
− when a task blocks or unblocks on/from a mutex
Quantitative data
− only 4 core functions, 3 utility ones and some KAL macro added
− only 2 files have been newly created
− only 6 files lightly modified modified
− totally 260 lines of code added
− totally 6 lines of code modified
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
38. BWI implementations inside AquoSA
Implementation characteristics
Description of the implementation
QoS Managed QoS Reserved GNU/Linux
Application Application Application
Non intrusive approach
user-space
− Do not modify the core features of AquoSA, that is
qmgrlib qsuplib qreslib glibc
the scheduler and scheduler related functions
QoS
the RR enforcing mechanisms
Manager qmgrmod
component the servers handling (list of tasks, dispatching, etc.) mechanisms
qresmod
EDF
BWI added functionsrresmod called in very few situations
only
Scheduler rres_bwi
− when a task is attached or detached to/from a server
Kernel Abstraction Layer kal_bwi
kernel-space
QoS
Reservation
−
component when a task blocks or unblocks on/from a mutex
GSP
Quantitative data Linux kernel
− only 4 core functions, 3 utility ones and some KAL macro added
− only 2 files have been newly created
− only 6 files lightly modified modified
− totally 260 lines of code added
− totally 6 lines of code modified
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
39. BWI implementations inside AquoSA
Implementation characteristics
Description of the implementation
Non intrusive approach
− Do not modify the core features of AquoSA, that is
the scheduler and scheduler related functions
the RR enforcing mechanisms
the servers handling (list of tasks, dispatching, etc.) mechanisms
task_struct task_list server_t
BWI added functions only called in very few situations
T1 ...
private_data
...
task
others
...
tasks
srv ...
bwi_blocked_on
− when a task is attached or detached to/from a server bwi_wait_task
bwi_orig_srv
blocked on bwi_entries S1
− when a task blocks or unblocks on/from a mutex
T2
Quantitative data task_struct
...
task_list
task
server_t
...
private_data others tasks
... srv ...
− only 4 core functions, 3 utility ones and some KAL macro added bwi_blocked_on
bwi_wait_task
bwi_orig_srv S1
− only 2 files have been newly created bwi_entries
task_list
− only 6 files lightly modified modified task
others
srv
− totally 260 lines of code added bwi_blocked_on
bwi_wait_task
bwi_orig_srv
− totally 6 lines of code modified bwi_entries
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
40. Implementation and study of the
BandWidth Inheritance protocol in the Linux kernel
Real-time systems
− Definition
− Real-time scheduling and resource sharing
− Server based scheduling and resource sharing
− Soft real-time systems, QoS and Resource Reservation
The AquoSA Framework
− Framework objectives and design goals
− AquoSA Framework architecture and usage
BWI implementation inside AquoSA
− Design and goals
− Specific contribution
− Implementation characteristics
BWI implementation verification
BWI experimental simulations results
BWI implementation overhead evaluation
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
41. BWI implementation verification
Verification
Static analysis
− during the whole process of code development
− assertions placed throughout the code to ensure consistency in some critical points
Dynamic analysis (testing)
− a lot of test programs have been run checking
the output they produces is exactly what we could have expected
none of the assertions added during static analysis is violated
Stress tests
− a lot of test programs have been run caring of
all the code paths are covered
all the identified abnormal situations are correctly handled
− all the test programs have been run in different system load conditions
(Inverse) integration tests
− all test programs have been run whit BWI not being used by them
− all test programs have been run with our code compiled out
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
42. BWI implementation verification
Verification
Static analysis
− during the whole process of code development
− assertions placed throughout the code to ensure consistency in some critical points
Dynamic analysis (testing)
− a lot of test programs have been run checking
the output they produces is exactly what we could have expected
E. W. Dijkstra:assertions added during static analysis is violated
none of the
Stress tests can only reveal the existence of programming errors,
<<testing
− a not prove their absence>> caring of
lot of test programs have been run
all the code paths are covered
all the identified abnormal situations are correctly handled
− all the test programs have been run in different system load conditions
(Inverse) integration tests
− all test programs have been run whit BWI not being used by them
− all test programs have been run with our code compiled out
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
43. Implementation and study of the
BandWidth Inheritance protocol in the Linux kernel
Real-time systems
− Definition
− Real-time scheduling and resource sharing
− Server based scheduling and resource sharing
− Soft real-time systems, QoS and Resource Reservation
The AquoSA Framework
− Framework objectives and design goals
− AquoSA Framework architecture and usage
BWI implementation inside AquoSA
− Design and goals
− Specific contribution
− Implementation characteristics
BWI implementation verification
BWI experimental simulations results
BWI implementation overhead evaluation
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
44. BWI experimental results
Experimental simulations
We build and ran some simulations in order to
− prove BWI is an effective solution for handling critical sections
and resource sharing in a real and running RR based soft real-time system
− point out some if its fundamental properties and benefits it introduces
List of the simulations we ran
− BWI enforcement
bwi_test_bwi_1
bwi_test_bwi_2
− Priority inversion control
bwi_test_inversion
− Compliance with the BIP
bwi_test_isolation
− Full bandwidth exploitation
bwi_test_bandwidth_1
bwi_test_bandwidth_2
bwi_test_bandwidth_3
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
45. BWI experimental results
Experimental results
bwi_test_bwi_1
deadline misses !
− BWI not used
S0 S0
Th. A=(10, 10, 100) W0 W0
0 0
W0 S0 W0 S0
Th. B=(30, 10, 100)
0 0 0 deadline hit 0 0 0
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 320 340 360 380 400 420 440 460 480 500 520
deadline misses !
− BWI used
W0 S0 W0 S0 W0 S0 W0 S0
Th. A=(10, 10, 100)
0 0 0 0 0 0
W0 S0 W0 S0
Th. B=(10, 10, 100) 0 0 0 0
deadline hit
Thread A's tardiness is significantly decreased
407.80196 msec instead of 1905.8146 msec after 10 runs
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
49. BWI experimental results
Experimental results
bwi_test_bandwidth_1
(2 thread, 1 mutex)
Full Bandwidth exploitation
Boosted finishing time
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
50. BWI experimental results
Experimental results
bwi_test_bandwidth_1
(3 thread, 2 mutex)
Full Bandwidth exploitation
Boosted finishing time
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
51. BWI experimental results
Experimental results
bwi_test_bandwidth_1
(8 thread, 5 mutex)
Full Bandwidth exploitation
Boosted finishing time
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
52. Implementation and study of the
BandWidth Inheritance protocol in the Linux kernel
Real-time systems
− Definition
− Real-time scheduling and resource sharing
− Server based scheduling and resource sharing
− Soft real-time systems, QoS and Resource Reservation
The AquoSA Framework
− Framework objectives and design goals
− AquoSA Framework architecture and usage
BWI implementation inside AquoSA
− Design and goals
− Specific contribution
− Implementation characteristics
BWI implementation verification
BWI experimental simulations results
BWI implementation overhead evaluation
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
53. BWI implementation overhead evaluation
Overhead evaluation
Space/Memory
look at the size of difference between the two cases BWI is compiled or not of
the rresmod.ko file, i.e. the loadable kernel module
its .text ELF section, i.e. the code actually loaded and executed
BWI not enabled BWI enabled Difference Difference %
module size (byte) 563055 609796 46741 8.3
.text length (byte) 31852 42196 10344 32.48
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
54. BWI implementation overhead evaluation
Overhead evaluation
Time/Execution
− profile the functions in which the call to our 4 core routines have been inserted
when BWI is compiled but not used
when BWI is compiled and used
− look at the differences between mean and maximum execution time obtained values
and the case BWI is not compiled in the framework
Max. Execution Time (usec)
BWI compiled out BWI not used Difference BWI used Differences
rres_attach_task_nosched 7 8 1 7 0
rres_block_hook 6 6 0 7 1
rres_unblock_hook 9 9 0 12 3
rres_detach_task_nosched 6 6 0 6 0
Mean Execution Time (usec)
BWI compiled out BWI not used Difference BWI used Differences
rres_attach_task_nosched 4.840 5.080 0.240 5.000 0.160
rres_block_hook 3.467 3.477 0.010 3.636 0.169
rres_unblock_hook 5.721 5.773 0.052 5.837 0.116
rres_detach_task_nosched 4.720 4.920 0.200 4.960 0.240
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel
55. BWI implementation overhead evaluation
Overhead evaluation
Context Switches
− investigate how BWI behaves with respect to context switches
does it introduce any additional context switch ?
does it reduce the number of context switch ?
− count the context switches and evaluate the differences between the cases
BWI is not used
BWI is used
bwi_test_bwi_1 (periodic tasks)
BWI not used BWI used Difference
Thread A 26 24 -2
Thread B 34 34 0
bwi_test_bandwidth_2 (greedy tasks)
BWI not used BWI used Difference
Thread A 607 343 -264
Thread B 414 316 -98
Thread C 405 405 0
02/10/2007, Dario Faggioli Implementation and study of the BandWidth Inheritance protocol in the Linux kernel