introduction to embedded systems

Fall 2013SILICON VALLEY UNIVERSITY

CONFIDENTIAL 1

Introduction to Embedded Systems

Dr. Jerry Shiao, Silicon Valley University

SILICON VALLEY UNIVERSITY CONFIDENTIAL 2Spring 2014

Section 10 Real-Time Linux

Real-Time Linux What is Real-Time Processing? Real-Time does not mean raw OS speed, but means

an OS that controls processing so that critical operations get done by guaranteed times.

The physical world imposes its schedule on Real-Time application.

Physical world schedule sends events through device interrupts.

Require predictability from the system when event occurs.

Could sacrifice throughput for guarantee timely behavior.


Section 10 Real-Time Linux Real-Time Linux

Hard Real-Time has absolute predictability. Event absolutely handled in 3ms.

Soft Real-time has more relaxed requirement. Event 80% probability within 3ms, 99.9% within 5ms.

Real-Time Processing Required? Lateness safety issue or unacceptable quality of service (deadline

required)? External device generating event (interrupt) require event to be handled

within fixed time? External device interacting with system or application process has time-

bound requirements. Does the Real-Time application require higher priority than Linux

networking or file system services (background daemons or threads) ? Real-Time clock coarse grain (100+ MSec) or require fine-grain

granularity only?



Real-Time Latency Time elapsed between receive an interrupt and

when application processes it.

Task or

Device

ISR Task n

Interrupt Latency: The amount of time elapsed between an interrupt and execution of the Interrupt Service Routine (ISR). Interrupt handling delayed: system bus contentions, DMA operations, cache misses, interrupt masking.

Scheduler Latency: The amount of time for the scheduler to complete its current duties before yielding the CPU to the task (i.e. workqueue bottom half interrupt handler ).

Time

Time to dispatch a pending interrupt to the ISR can be long because external interrupts can be masked when critical sections are being executed.

Event


Section 10 Real-Time Linux Real-Time Kernel Requirements

Preemption Granularity Kernel has to be able to switch as soon as possible from

low-priority context to a high-priority task. Linux 2.4: As long as Kernel was handling activities required

by a system call or another user space request, Kernel would not stop in order to allow a new task (higher priority) to run.

Linux 2.6: Kernel was preemptive, except in critical functions when code has disabled local interrupts.

CONFIG_PREEMPT=y Priority Inheritence

Prevents low-priority task from delaying high-priority task from executing.

Critical Sections Interrupt Latency: Longest time needed for Kernel to dispatch

pending interrupt to appropriate software handler. Time lapse before handling external interrupt unacceptable in

Real-Time Kernel.



Critical Sections Long critical sections in Virtual Memory Management, Virtual

Filesystem, and networking stack. Mask External Interrupts:

Prevents concurrent tasks on the same processor from wrongly accessing nonshareable resource.

Delayed Timer Interrupts Accurate delivery of timed events is a fundamental resource

upon which Real-Time Systems. Real-Time Application acquiring data at a fixed frequency from

dedicated hardware. Timing Services (periodic scheduling) depends on the

accurate delivery of interrupts sent by timer hardware. Must be kept within acceptable bounds.



Users of Real-Time Computing Technology: Measurement and Control Industry

Timings in the conveyer belts in industry Plant. Incoming samples processed in the right pace to keep the

input and output processes perfectly synchronized. Sensor detection requires the application to get the alarm

notification and react within appropriate time frame to shut down malfunctioning device.

Aerospace Industry Distributed Real-Time computer network aboard an aircraft

supporting numerous applications involving safety criticality levels.

Simulation systems has to interact with real hardware equipment. Testing new devices in virtual environment allows engineers to predict whether overall airborne system specification are met.



Users of Real-Time Computing Technology: The Financial Services Industry

Distributed applications crucial to the business may have requirements for reliable and timely exchange of messages.

In Stock Exchange, too much delay in processing trade opportunities could mean significant losses.

The Multimedia Business Multimedia systems have to process visual and auditive

information in a time frame short enough for the audience to not perceive visible or audible artifacts.


Section 10 Real-Time Linux Linux Paths to Real-Time

Real-Time Operating System designed from beginning to provide perdictable response time.

Linux is NOT designed as Real-Time Operating System. Linux is a General Purpose Operating System that was initially

designed for: Fairness (sharing resources among users). Good Average Performance under sustained load conditions.

Real-Time Applications becoming more complex, need for a more sophisticated Operating System, not chosen just for perdictable response time.

Real-Time Application has interface to non-Real-Time components.

Need Networking capabilities. Device Drivers for off-the-shelf hardware. Rich graphic support. Following standards means interoperability and larger pool of

engineers (cost-effective computing platforms ).


Section 10 Real-Time Linux Linux Paths to Real-Time

Linux as Real-Time Operating System Benefit from begin General Purpose Operating System. Graphical environments ( Xwindows for KDE and Gnome ). Large number of device drivers supporting off-the-shelf

hardware. Reliable networking stack. Integrate “real-time” enhancements.

“Other” Real-time Operating Systems The rigid and down-scaled Real-Time Operating System

more difficult to enhance. Integrate ad hoc and proprietary solutions to meet General

Purpose requirements.


Section 10 Real-Time Linux Linux Co-Kernel Approach Do not trust standard Linux Kernel to honor Real-Time tasks,

because GPOS Kernel is inherently complex. Real-Time processes can be handled by small Co-Kernel.

Co-Kernel Real-Time Kernel (micro kernel) running with core Linux Kernel. Real-Time tasks are Linux loadable modules (only supported

devices) in micro kernel. Loaded or unloaded as needed. Unmodified Linux OS is a process, scheduled with other Real-Time

processes by the micro kernel. Device interrupts go through the micro kernel first.

Linux cannot block other interrupts with execution in critical section. Real-Time processes has predictable response time.

Virtual Programmable Interrupt Controller (PIC) Micro kernel intercepts hardware interrupt. Software-based interrupt mask between Linux Kernel and hardware

PIC. Defers Linux interrupt handler until Linux takes control of processor.


Section 10 Real-Time Linux Linux Co-Kernel Approach

Co-Kernel Real-Time processes at micro-kernel have low interrupt and

dispatch latencies. Linux activity does not affect Real-Time processes. RTOS maintenance simplified.

Co-Kernel Disadvantage Linux drivers and Libraries on standard Kernel does not benefit

Real-Time Operating System. Must port any Linux Device drivers to RTOS. Standard C Library causes unexpected latency when used by RTOS

processes.


Section 10 Real-Time Linux Fully Preemptible Kernel Approach Convert Linux into full RTOS.

Real-Time Processes run without experiencing interference from unpredictable or unbounded activities by non Real-Time processes.

Problem: Linux millions of lines of code. Very difficult to be 100 percent sure GPOS code will not affect RTOS. Code in Linux Core: Scheduler, Interrupt Handling, Internal locking, Creation

and Destruction of new threads, Memory Management, Filesystems, and Timers.

Search for all areas that might disable preemption or interrupts for long periods of time.

Analyze all Device Drivers. Code in device drivers and Kernel modules: Linux supports many different

devices and architectures (PowerPC, MIPS, ARM, X86). Analyze Linux Core.

Linux 2.6 contain large areas where preemption is disabled. Mutexes that can cause priority inversion. Interrupt Service Routines preempt any process (Real-Time included).


Section 10 Real-Time Linux Xenomai Real-Time System Dual Kernel Approach: Small Co-Kernel running with

Linux on same Hardware. Real-Time Task Time-Critical operations controlled by

Co-Kernel: Very low latency. Allows porting other Real-Time environment into Linux

by providing Real-Time API emulators that mimic the system calls used by RTOS vendors.

Xenomai Core is GPL. User Space Library released under LGPL (Library/Lesser GPL).

Supporting different architectures: PowerPC, ARM, X86. Uses Adeos (Adaptive Domain Environment for

Operating Syststems) Hardware Abstraction Layer (HAL) between hardware and Operating System. Interrupt Pipeline for priority interrupt dispatching.

Website: http://www.xenomai.org


Section 10 Real-Time Linux Xenomai Real-Time System

Common Operating System functionalities of different RTOS, abstracted into RTOS core.

Real-Time API emulators that mimic the system calls as described in the RTOS vendor specifications.

Virtual Programmable Interrupt Controller. Allows Xenomai and Linux maintain separate masks for interrupts and policies for handling interrupts.Xenomai is the highest priority domain in the I-Pipe (Interrupt Pipeline) and is the first to receive interrupts, system calls, processor faults, and other exceptions. I-Pipe dispatches to domains according to domain’s static priority.

Hardware and System Abstraction Layer: All CPU and platform-dependent code.


Section 10 Real-Time Linux Xenomai Real-Time System Xenomai Core and Nucleus

Generic building blocks (skins) for migrating Real-Time Applications from different RTOS by creating a framework of Real-Time APIs provided by the different RTOS.

Supply all the Operating System resources the skins may require to properly mimic the RTOS APIs.

Application can use any building block (skin) to port application from an RTOS.

Nucleus: Kernel Loadable Module containing the building blocks. Building Blocks (skins):

POSIX: Replacement for glibc services reimplemented over Xenomai Co-Kernel.

VxWorks Emulator of WindRiver Real-Time Operating System. pSOS Emulator of Real-Time Operating System. VRTX Emulator of Mentor Graphics Real-Time Operating System. uITRON (micro ITRON) Emulator for Japanese open standard for Real-Time

Operating System. RTAI Emulator for legacy applications over Xenomai. RTDM (Real-Time Driver Model) interface for dual kernel-based systems.


Section 10 Real-Time Linux Xenomai Real-Time System System Calls

Module for a skin exports set of services to the Xenomai Nucleus. System calls intercepted by I-Pipe, and dispatched to RTOS skin.

Disp

Dispatches the system calls to the skin module.


Section 10 Real-Time Linux Xenomai Real-Time System Interrupt Pipeline ( I-Pipe ) software layer used by each RTOS

Domain to maintain separate masks for interrupts and policies for handling interrupts.

Xenomai is the highest priority domain, ahead of the Linux Kernel.

I-Pipe dispatches events such as interrupts, system calls, processor faults, and other exception

Adeos implements a queue of signals. Operating Systems (OS Domain) are passed the signal and must accept, ignore, discard, or terminate the signal. Signal not handled or discarded by OS Domain are passed to the next OS Domain in the chain.


Section 10 Real-Time Linux Xenomai Real-Time System Domain Migration

Co-Kernel Environment, two kernels run concurrently without synchronizing their activities.

Xenomai Thread on Co-Kernel uses Xenomai’s deterministic system calls. Linux Thread uses Linux system calls with no Real-Time guarantees. Real-Time Thread migrates from Xenomai Scheduler to Linux Scheduler and

vice versa when system call is issued.

I-Pipe used by the system calls, moves the Real-Time application between the Xenomai Scheduler and the Linux Scheduler as various interrupts, system calls, and other events require their services.


Section 10 Real-Time Linux Xenomai Real-Time System Real-Time Driver Model

Common Framework to develop drivers and tools for data acquisition. RTDM is a mediation layer connecting Real-Time Application process to services

exported by a device driver.

Two classes of Device Drivers:Protocol Drivers expose socket interface. Handles standard socket system calls ( socket(), send()/sendto(), recv()/recvfrom(), etc).

Named Device are RTDM layer device and similar to Linux Device Drivers, but identified by device name (text label). Handles I/O system calls ( open(), read()/write(), ioctl(), etc ) into Xenomai system calls.



Real Time Application Interface (RTAI)Deterministic Real-Time processing.University of Milan, Italy.Nonprofit organization to support Real-Time

Linux community.GPL license.New APIs to deal specifically with the Real-

Time environment and task management.“Interrupt Abstraction” approach.

Entire Linux Kernel preemptible by having separate hardware handling software to manage actual hardware interrupts on the system.



Real Time Application Interface (RTAI) Co-Kernel

Real-Time Kernel (micro kernel) running with core Linux Kernel.

Real-Time tasks are Linux loadable modules (only supported devices) in micro kernel. Loaded or unloaded as needed.

Unmodified Linux OS is a process, scheduled with other Real-Time processes by the micro kernel.

Device interrupts go through the micro kernel first. Linux cannot block other interrupts with execution in critical

section. Real-Time processes has predictable response time.

Virtual Programmable Interrupt Controller (PIC) Micro kernel intercepts hardware interrupt. Software-based interrupt mask between Linux Kernel and

hardware PIC. Share interrupts between two kernels.



Real Time Application Interface (RTAI)Limitations

Real-Time tasks are Linux kernel loadable modules.

Kernel loadable module utilize APIs that are not POSIX APIs.

GPL requires user Real-Time tasks to be released to general public.

Work underway to use the Real-Time scheduler use Real-Time processes in user-space (must avoid APIs and system calls that can cause blocking, i.e. networking, disk I/O, memory allocations).



Real Time Application Interface (RTAI)

Linux Process 1

Linux Process 2

Linux Kernel RTAI Process RTAI Process

RTAI Micro kernel

Virtual PIC

Linux cannot enable and disable hardware interrupts directly.

Hardware PIC

Core port of Linux for microkernel:

1)Scheduling of processes.

2)Interrupt handling.

3)Internal locking.

4)Creation and destruction of new threads.

5)Memory management.

6)Filesystems.

7)Timers.


Section 10 Real-Time Linux Linux 2.6 Real-Time Modifications

Linux originally designed around fairness, sharing resources among its uses and to perform under sustained load conditions.

General purpose complexity too unpredictable for real-time. Preemption

Linux 2.6 system call was preemptable. A process forced to release CPU.

Real Time Scheduler Linux 2.6 scheduler determined time slice for each process

and which process to run based on runqueue per priority level. Active and Expired priority arrays.

High Resolution POSIX Timers. Time hardware with microsecond resolution.

Minimize execution of interrupt critical sections (interrupt latency).

SILICON VALLEY UNIVERSITY CONFIDENTIALSpring 2014


Real-Time Patch To Native Linux KernelLinux Kernel in Real-Time Environment:

Preemption Improvements. Reduce time in non-preemptible code (interrupt context) to

minimize interrupt latency and scheduling latency. Low latency patches in Kernel, add points where Kernel thread

relinquishes non-preemption.

Unbounded Priority Inversion avoided.

1 3

2 6

5

4

Normal Execution

Critical Section Execution

Unbounded Priority Inversion

Task H

Task M

Task L

Priority



Real-Time Patch To Native Linux Kernel Priority Ceiling:

Shared resource has predefined priority ceiling. Task acquires shared resource, task priority temporarily

raised to priority ceiling. Prevents deadlocks by stopping nested locks. Static analysis to determine priority ceiling of each

shared resource. Every task must be known.

Task H

Task M

Task L

Normal Execution


- Task L priority raised to priority ceiling.- Executes critical section.

- Task H request resource. Priority raised to priority ceiling.

- Executes critical section.



Real-Time Patch To Native Linux Kernel Priority Inheritance:

Low-priority task has priority raised until completes critical section (releases shared resource).

Nested resource locks can lead to deadlock. Avoid deadlock by allowing each task to own one shared

resource. Do not allow nested locks.

Overhead raising priority and then lowering priority.

Task M

Task L

Normal Execution


- Task M request resource.- Task L priority raised.

Executes critical section.

- Task L Completes critical section. Priority lowered.

- Task M receives resource. Executes critical section.




Handle IRQs as Threads. Lowers latency caused by softirqs and tasklets.

Softirqs preempts all threads, including kernel. Default priority IRQ threads (PID = IRQ_n, n is interrupt

vector) is 50. chrt utility used to modify the priority of the ISR thread.

Interrupts can be prioritized, even though hardware does not support prioritized interrupts.

User process can be prioritized higher than IRQ thread.

CPU affinity of IRQ thread based on CPU affinity of the interrupt. Prevents expensive cache flushes.




Real-Time Patch has infrastructure for most hard real-time.

Size of the code is too large and general purpose design makes it difficult to meet all RT Patch needs.

Only necessary drivers are loaded (bulk of Linux are drivers).

Drivers must be audited to not disable interrupts for long periods.

Kernel configuration options must be optimized. Only enable options that are needed.

http://people.redhat.com/mingo/realtime_preempt

introduction to embedded systems

Documents