(Embedded Linux Conference Europe 2014)
Linux uses many kind of embedded products. The products include not only consumer electronics but also control systems such as programmable logic controllers. There are many type of infrastructure systems and each system has different technical requirements. The requirements include not only real-time performance but also reliability-related functions. The infrastructure systems have to meet all the requirements. This presentation gives a summary of our study and development to adapt the Linux to infrastructure systems. Then we discuss the direction of future development. Please note, this presentation doesn't focus on a specific product.
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Using Embedded Linux for Infrastructure Systems
1. Using Embedded Linux for
Infrastructure Systems
Embedded Linux Conference Europe 2014
14 Oct 2014
Yoshitake Kobayashi
2. 2Embedded Linux Conference Europe 2014
Scope of this presentation
Make a Place for collaboration
NOTE
This project currently belongs to CE workgroup, Linux
Foundation
Anyone interested in this activities, please contact us
3. 3Embedded Linux Conference Europe 2014
Outline
Definition of Social Infrastructure Systems
Problem statement
Expected goal
Project scope
Activities
Current status
4. 4Embedded Linux Conference Europe 2014
Definition of Social Infrastructure Systems (SI systems)
Embedded systems which uses for the following
infrastructure systems
Transport infrastructure
Energy infrastructure
Water management infrastructure
Communications infrastructure
Earth monitoring and measurement systems
The system includes
Automation systems
Controllers
PLC
Microcontrollers
Signal controller
Multi-purpose controller
Mini-server based controller (includes PC based controller)
Sensor network systems
7. 7Embedded Linux Conference Europe 2014
Infrastructure System becomes a part of IoT
sensor
THE INTERNET
OF THINGS
THE INTERNET
OF THINGS
Moving Forward
Controller
(e.g. PLC)
8. 8Embedded Linux Conference Europe 2014
Problem statement
The SI systems will be work with other systems
such as:
Cloud computing
IoT / IoE related systems or devices
9. 9Embedded Linux Conference Europe 2014
Problem statement
To connect with other systems, SI systems
Need to have same quality with new requirement
Need to have new technology to have new functionality
New requirements
Ensure a complicated performance requirement
Nondeterministic and deterministic programs runs on a
same board
Collaborate with cloud platform
New functionality
Use fast non-volatile memory (e.g. MRAM, FeRAM)
10. 10Embedded Linux Conference Europe 2014
Goal
Main goal
Develop reference implementations for SI system
Reference implementation may includes:
Linux kenel and filesystem for some target board
Target board: TBD
Open spec hardware is better for the implementation
Linux kernel related implementations should post to LKML
for mainlining
Test-cases
11. 11Embedded Linux Conference Europe 2014
Project scope
Security
Protect from irregular activities (e.g. viruses)
Integrity check
Dependability
Robustness
Run-Time Maintenance / Update
e.g. Kpatch / kGraft
Real-time
Hard real-time
Multi-purpose real-time system
Tests
Reliability test
Long Term Support
VLTS = Very Long Term Support
Legal/Compliance Issue
(Possibly) Functional Safety
The other scopes to be discussed
12. 12Embedded Linux Conference Europe 2014
Activities
This project will be separated into several
different sub-project by its key technology
Initial activity
Survey to determine current common key issues for SI
systems
Create a mail list
F2F meeting at Linux Foudation’s conference
Embedded Linux Conference / LinuxCon
Workgroup meeting will be arranged
Reviewed by members of the CE Workgroup Architecture
Group
13. 13Embedded Linux Conference Europe 2014
Estimated output
Publicly available pages on the elinux wiki which
includes the follows:
A list of issues
Specification for reference implementations
A list of link to reference implementations
All kernel related implementation should post to LKML
Reference implementations for key issues will be
available on GitHub or alternative service
14. 14Embedded Linux Conference Europe 2014
Current status
Under survey to determine actual issues by
discussion
Start to call for participation
15. 15Embedded Linux Conference Europe 2014
Project scope
Security
Protect from irregular activities (e.g. viruses)
Integrity check
Dependability
Robustness
Run-Time Maintenance / Update
e.g. kpatch / kGraft
Real-time
Hard real-time
Multi-purpose real-time system
Tests
Reliability test
Long Term Support
VLTS = Very Long Term Support
Legal/Compliance Issue
(Possibly) Functional Safety
The other scopes to be discussed
16. 16Embedded Linux Conference Europe 2014
Project scope
Security
Protect from irregular activities (e.g. viruses)
Integrity check
Dependability
Robustness
Run-Time Maintenance / Update
e.g. kpatch / kGraft
Real-time
Hard real-time
Multi-purpose real-time system
Tests
Reliability test
Long Term Support
VLTS = Very Long Term Support
Legal/Compliance Issue
(Possibly) Functional Safety
The other scopes to be discussed
17. 17Embedded Linux Conference Europe 2014
Real-time
Improve scheduring granularity for SCHED_DEADLINE
Hardware resource isolation to run RT and GP process
Test
Deta reliability evaluation for Linux filesystems
Other
Long-term testing by accelerated kernel
18. 18Embedded Linux Conference Europe 2014
Issue on SCHED_DEADLINE
Wakeup
Period
Task execution
Time
dl_timer
1.5ms
task_tick_dl
1ms
Wakeup
dl_timer
1.5ms
1ms
Step1: Refill budget
Step2: Use budget
Period
2ms
Difficult to keep task’s budget if the budget has
micro seconds granularity
Period: 1ms, Budget: 0.5ms
19. 19Embedded Linux Conference Europe 2014
Support for micro seconds granularity
Period: 1ms, Budget: 0.5ms
Wakeup
Period
Task execution
Set HRTICK here
Step1: Refill budget
Step2: Use budget
Time
3.2ms
1ms 2ms 3ms
Wakeup
When a task’s budget is less than 1ms, set HRTICK for the
rest of budget
Source code is available on GitHub ( https://github.com/ystk/sched-deadline )
20. 20Embedded Linux Conference Europe 2014
Let’s play a movie
ModifiedOriginal
Play the movie about 3 times faster than normal speed
Requirement for a frame decoding
Period: 10ms, budget: 3.6ms
When a frame decode is not finished in a period, the player shows “DEADLINE MISS” on screen and add 3ms
penalty
“Big Buck Bunny” http://www.bigbuckbunny.org/
(c) copyright 2008, Blender Foundation
License: Creative Commons Attribution 3.0
21. 21Embedded Linux Conference Europe 2014
Realtime
Improve scheduring granularity for SCHED_DEADLINE
Hardware resource isolation to run RT and GP process
Reliability
Deta reliability evaluation for Linux filesystems
Functionality
Long-term testing by accelerated kernel
22. 22Embedded Linux Conference Europe 2014
RT process and GP process runs on a board
CPU
Device 1 Device 2
HardwareApplicationsOS
Linux
RT App
Core 1 Core 2
GP App
Goal
Run RT processes and GP processes on a hardware platform
CPU
Device 1 Device 2
RT App
RT core GP core
GP App
Linux
Isolate from GP application’s
behavior
23. 23Embedded Linux Conference Europe 2014
An example for isolation demo
Eth0 SD UART
Core1Core0
Web
server
RT
control
Pandaboard
Workload
(Web clients)
Puppy
Motor
Sensors
(Gyroscope,
etc.)
Motor torque
Sensor values
CAN
RS232C
Serial-CAN
converter
Point
RT Control and UART
interrupts are isolated
from the Web server
and other interrupts.
IRQIRQ
25. 25Embedded Linux Conference Europe 2014
An issue while control the robot
Latency [μs]
Latency for RT app while running with GP appliation
Evaluation
Environment
Hardware
・ Pandaboard
- ARM Dual core 1.2GHz
- Memory 1GB
- SDcard
OS
・ Linux 3.6.11.6-RT38
Application
・ RT App
- Periodic task
Period = 300 micro sec
・ GP App
- Random I/O on SDcard
0
10
20
30
40
50
60
70
80
90
100
0 100 200 300 400 500 600 700 800 900
Numberofsamples
Before
(CPUset)
This latency cause a
failure
26. 26Embedded Linux Conference Europe 2014
Realtime performance improvement with Timer-Shielding
Timers Timers
Timer
management
Hardware
Applications
Linux
Kernel
RT core GP core
RT App GP App
Timers Timers
Timer management
Hardware
Applications
Linux
Kernel
RT core GP core
RT App GP App
Migrator
Timer-Shield ONTimer-Shield OFF
Hardware Partition Management
(Create / Modify / Delete)
Migrator
Prohibit to resister
non RT related
timers
Migrate
Migrate non RT relate
timers
API to manage
hardware partitions
Register
Issue
Some kernel task
does not able to
allocate to target core
27. 27Embedded Linux Conference Europe 2014
An issue while control the robot
0
10
20
30
40
50
60
70
80
90
100
0 100 200 300 400
Numberofsamples
Latency [μs]
After
Latency is less
than 60us
Latency [μs]
Latency for RT app while running with GP appliation
Evaluation
Environment
Hardware
・ Pandaboard
- ARM Dual core 1.2GHz
- Memory 1GB
- SDcard
OS
・ Linux 3.6.11.6-RT38
Application
・ RT App
- Periodic task
Period = 300 micro sec
・ GP App
- Random I/O on SDcard
0
10
20
30
40
50
60
70
80
90
100
0 100 200 300 400 500 600 700 800 900
Numberofsamples
Before
(CPUset)
28. 28Embedded Linux Conference Europe 2014
Realtime
Improve scheduring granularity for SCHED_DEADLINE
Hardware resource isolation to run RT and GP process
Test
Deta reliability evaluation for Linux filesystems
Other
Long-term testing by accelerated kernel
29. 29Embedded Linux Conference Europe 2014
Data reliability test
Checker
Target
file
LOG
file
AAAAA
BBBBB
CCCCC
DDDDD
EEEEE
OK
FFFFF
AAAAA
BBBBB
CCCCC
DDDDD
EEEEE
OK
AAAAA
BBBBB
CCCCC
DDDDD
AAAAA
NG
AAAAA
BBBBB
CCCCC
DDDDD
NG
?
size mismatchdata mismatch
Estimated file contents
Estimated file size
Power off while running writer processes
Verify all data
File size
Contents
Test case is available on GitHub ( https://github.com/ystk/fs-test )
30. 30
Results of data reliability tests on Linux filesystems
0.00
0.50
1.00
1.50
2.00
EXT3-
ORDERED
EXT3-
JOURNAL
* * * JFS XFS
Errorrate[%]
45.9%
Errorrate[%]
0.00
0.50
1.00
1.50
2.00
EXT3-
ORDERED
EXT3-
JOURNAL
EXT4-
JOURNAL
EXT4-
ORDERED
EXT4-
WRITEBACK
* XFS BTRFS
82.4%84.7%
43.4% 41.4%
43.2%
Errorrate[%]
0.00
0.50
1.00
1.50
2.00
EXT3-
ORDERED
EXT3-
JOURNAL
EXT4-
JOURNAL
EXT4-
ORDERED
* JFS XFS
13.3%
kernel
2.6.18
kernel
2.6.31
kernel
2.6.33
File size mismatch rate
Data mismatch rate
Point 1:
An filesystem has
different characteristics
on different kernel
Point 2:
2.6.33 has high error
rate on ordered and
writeback mode
Point 3:
Ext4-journal and Btrfs
has good resluts
File system types
kernelversion
31. 31Embedded Linux Conference Europe 2014
Realtime
Improve scheduring granularity for SCHED_DEADLINE
Hardware resource isolation to run RT and GP process
Test
Deta reliability evaluation for Linux filesystems
Other
Long-term testing by accelerated kernel
32. 32Embedded Linux Conference Europe 2014
Long-term running test by accelerated kernel
START GOAL
Accelerate
Detect clock counter
related errors (e.g. time_t)
Accelerate clock counter
jiffies = jiffies + (1 * ratio)
Issue
• Long-term testing takes really long time
→ We want results as fast as possible
Things that NOT able to be
accelerated.
• CPU frequency
• HDD or SSD access speed
• Network link speed
• …
33. 33Embedded Linux Conference Europe 2014
Evaluation with accelerated kernel
Xdaliclock works as a stopwatch
About 40 times faster
to get 100% CPU usage
returned incorrect value
after about 450 days
34. 34Embedded Linux Conference Europe 2014
Enough?
Real-time latency is short enough
Data is safe enough
System works long enough
35. 35Embedded Linux Conference Europe 2014
Summary
There are many issues need to be solved for SI system
CE workgroup is considering to have sub project for it
Current status
Under survey for actual issues
Start call for participation
Initial activity will be the follows:
Clarifying issues to be solved
Create a mail list
Have F2F meetings at Linux Foundaton’s conference
Estimated output
Publicly available pages on the elinux wiki or Linux Foundation’s web site
which includes the follows:
A list of issues
Specification for reference implementations
A list of link to reference implementations
All kernel related implementation should post to LKML
All source code for key issues will be opened
36. 36Embedded Linux Conference Europe 2014
Discussion
Any comments and suggestions are welcome
Contact information
To get the latest information, please send an email to the
following adderss:
Yoshitake Kobayashi yoshitake.kobayashi@toshiba.co.jp
Noriaki Fukuyasu fukuyasu@linuxfoundation.org