CSCI1600: Embedded and Real Time SoftwareLecture 27: Real Time Linux

Steven Reiss, Fall 2015

Conventional Operating Systems Memory model

Independent address spaces

Allocation and deallocation

Stack management

Device support Abstraction based on descriptors

Clock and other basic devices built in

High-level I/O support (e.g. networking)

Thread / process model (with scheduling)

IPC support (pipes, sockets, shared memory) Synchronization primitives

Conventional OS Problems Size (for embedded)

Micro-kernel based architectures help here

Unpredictable (and poor) latency

Slow interrupt handlers

Unpreemptible kernel code

Latency might be built into hardware

System management interrupts on x86

DMA bus mastering; USB handling; VGA console; Page faults

Unpredictable Scheduling

Desktop/server heuristics

Scheduling overhead grows with workload

Poor support for precise timing

Timing accuracy related to tick overhead

Are the Problems Fixable?

Want to retain the bulk of prior work Hundreds of device drivers

Many abstractions make programming easier

TCP/IP, File systems, IPC, Memory

How can we do this?

Two approaches to this are used Fixing the problems

Sidestepping the problems

RT-Linux

Assume most of the program is non-real time There is a small hard (soft) real-time component

Attempt to side step the problem Run the OS as a real time task

Allow other RT tasks (outside the OS)

Provide the benefits of an OS and still have a real time environment Except that Linux tasks are now sporadic

Linux on a RTOS We’ve seen this before

Scheduling sporadic tasks

Bandwidth servers (Total and Constant)

Group all the sporadic tasks together

Provide a real-time task (with fixed bandwidth)

This executes the various sporadic tasks

Using their priorities

This is basically RT-Linux Linux is “ported” to use RTOS primitives instead of hardware

Like running on a virtual machine

RT Linux

RT-Linux Details RTLinux core

Non-preemptible

Hard real time scheduler

Basic OS functionality

Deterministic interrupt-latency ISRs run in core

Others are transferred to Linux via a FIFO

Primitive tasks

Only statically allocated memory, no virtual memory, fixed priority

Run as real time threads

RT-Linux Details

Real time tasks No address space protection

Run with disabled interrupts

FIFO (message queue) Connects real time tasks with LINUX

Shared memory synchronization

Non-real time tasks Run as Linux processes

RT-Linux Pros and Cons Pros

Hard real-time guarantees (low tens of microseconds)

OS code is virtually unchanged

Applies equally well to other OS’s

Supports BSD as well as Linux

Cons

Real time task lose the convenience of a mature well-understood API

Communication with Linux processes is ad-hoc

Message queues – but user needs to define the messages

Fixing the Problems

Problems to tackle Scheduling, latency, timing

Techniques Priority scheduling and priority inheritance

Limit un-preemptible code

Offer high-resolution timers

Scheduling Real time schedulers are actually simpler

Support for fixed (absolute) priority

Ability to bind tasks to specific processors (cores)

Create new classes of processes (RR, FIFO) Round robin runs for a fixed time slice

FIFO is never guaranteed

Enhance mutexes to support priority (PIP,PCP)

Problem: scheduling latency Support fixed priorities

But scheduling decisions might be deferred

Interrupt handlers cannot be preempted

Spin-locked code can not be preempted

Minimizing Latency Simple Solution:

Threaded interrupt handling

Move interrupt handlers into kernel threads; true ISR is now very short

Handling can be preempted

Preemptible spin locks

Turn spin locks into mutexes

This almost works Timer interrupts need to be handled directly

Individual handlers are still atomic

Spinlocks not always equivalent to mutexes

Programming devices, modifying page tables, in the scheduler

Read-copy-update synchronization in Linux make non-preemption assumptions

New type: raw_spinlock_t

Other Improvements Dynamic tick

Ask for timer IRQ as needed, not periodically

High resolution timers

O(1) scheduler

Robust mutexes

O(1) kernel memory allocation

Ability to lock pages into memory (avoid page faults)

Most problems can be minimized But retrofitting them to an OS make hard real time unlikely

Solaris Attempt to fix the problems

Starting with the initial OS design

Not retrofitted

Features of Solaris for real time Interrupts are handled by threads

Kernel is fully preemptible and multithreaded

Real time scheduling is supported

Deterministic with real time priorities

Fixed priority scheduling supported

Solaris Real Time Threads Separate classes of threads

Fixed-priority (non-varying)

But lower priority than Operating System

Real-Time (fixed priority with fixed quantum)

Higher priority than operating system

Real time threadss Lock their pages in memory

Can call OS (but then run at lower priority)

Solaris Features Priority inheriting synchronization primitives

Processor sets The ability to isolate one or more CPUs for scheduling

Ability to lock tasks to a particular core

Full support for the Posix real time standard Real time threads

High-precision timers and clocks

Fast TCP/IP (zero copy)

Memory locking

Early binding of shared libraries

Solaris

Has been used as a hard real time environment

But not as an embedded environment Too large

Embedded Linux

Linux is a common tool for embedded systems Linux itself is relatively small

Linux is fairly modular

It can run in 1M (or less) of memory

Requirements on RT/Embedded

Safety Requirements The car won’t crash

Trains won’t crash

Both traffic lights won’t be green at the same time

You won’t blow a fuse

The system won’t deadlock

The heat and air conditioning won’t be on at the same time

More Requirements: Timing Forms

X will occur within k ms of event Y

X will occur every 10 +/- 1 ms

Examples The solenoid will trigger within 10 ms of the bumper being

hit

The switches will be sensed every 20 ms

The heat will come on within 5 minutes of detecting the room is too cool

More Requirements: Fairness Forms

Eventually X will occur

If X occurs, then eventually Y will occur

X will occur infinitely often

As long as Y is true, X will occur infinitely often

Examples Eventually you will add K to the score when you hit Y

As long as the heating system is on, the room will eventually be comfortable

Requirements

Are often mandatory (hard)

Failure is not a viable option Car or plan crashes

Broken devices

Heart stops pumping

Water is supplied unfiltered

The nuclear plant does not shut down

This is what hard real time means

What Are Your Requirements?

Problems

How do you show the system is safe

How do you get confidence in the system’s safety

When can you reasonably deploy the system

Can the system be used in a medical device

Can peoples lives depend on the system

Exercise

Suppose you wanted to sell your tic-tac-toe box What would you want as requirements

How would you check these?

Take the ankle-mounted obstacle finder example What would be the requirements

How would you check these?

Helpful Techniques: SE Good requirements analysis

Understanding all the possible problems and solutions

Good specifications Accurate modeling

Showing the models are correct

Petri net and FSA validation

Design for security Defensive coding

Building monitors as part of the code

Enter a safe state if the monitor detects anything wrong

Enter a safe state if the monitor detects anything unusual

Checking tools: lint, compiler, modeling analysis

Homework

Read chapter 13