Priority Inversion

BAE5030

Advanced Embedded Systems

9/13/04

Priority Inversion

• It’s not a good thing

• It can have disastrous results– Mars Pathfinder– Therac-25

• It can (and does) happen to the very best of the best experts!

Priority Inversion• General Description

– A low priority thread blocks a ready and willing high-priority thread indefinitely (Samek, p. 231)

– The highest priority task is prevented from running and completing a task on time

• The Dining Philosophers (Dijkstra)– 5 philosophers are eating around a table– Spaghetti in the middle– Takes 2 forks to eat slippery spaghetti– Fork between each philosopher

Dining Philosophers

• Philosophers alternate periods of eating and thinking

• When a philosopher wants to eat, he tries to acquire 2 forks

• If he is successful, he gets to eat for awhile, puts down the forks and continues to think

• How do we make this happen continuously without getting stuck? (resource allocation)

• Can’t all eat at once or think at once

• Phil. C is the most important philosopher but “very thin”

• Phil. B is a middle level philosopher• Phil. A is an amateur, low-level philosopher, very “portly”Scenario:• Phil C picks up a fork … Phil B grabs the other fork before

Phil C can get to it.• In the meantime, Phil A picks up two forks and eats while C

and B are gridlocked• The group of philosophers are never blessed with the

important philosophy of Phil. C (because he starves to death) and have to settle for the inane and inaccurate philosophy of portly Phil. A!

• That’s priority inversion!!!

Priority Inversion

Illustration (memory allocation)

Solutions/preventions:

• Semaphore: a protected variable and is a classic method for restricting access to shared resources

• Mutex: mutually exclusive semaphore - allows multiple threads to synchronize access to a shared resource

• Priority inheritance mutex: – A low priority task inherits the priority of any higher priority

task pending on a resource they share.– Priority changes as soon as the high-priority task begins to

pend and ends when the resource is released.– Requires help from the operating system

Priority Inheritance

• Priority ceiling mutex: – Associates a priority with each resource– Scheduler transfers that priority to any task

that accesses the resource– Priority assigned to the resource is the priority

of the highest-priority user, plus one.– When a task is finished with the resource,

priority returns to normal.– Semaphores are not needed, tasks can share

resources simply by changing priorities

Solutions/preventions:

Priority Ceiling

• If ceilings are chosen properly (not too high or too low)…priority ceiling mutex is:– faster – causes fewer context switches– much easier for static timing analysis… than priority inheritance mutex

• Bulletproof: priority ceiling protocol (Sha, et al.)…Combination of ceiling and inheritance

Solutions/preventions:

Mars Pathfinder

• Bus manager tasks communicated through a pipe along with a low-priority meteorological science task.

• Some medium priority tasks preempted the low-priority science task and kept the high-priority distribution manager waiting too long.

• Another bus scheduler became active and checked on the high-priority distribution manager, noticed its task wasn’t complete on time and caused a system reset.

• The fix: Windriver (software people) had left an inactive workaround in place. JPL had to enable it remotely. (Barr, Embedded Systems Programming)

Mars Pathfinder

Therac-25

• Computer-controlled radiation therapy machine

• Severely overdosed 6 people causing 2 painful deaths

• Intended dose: 100-200 rads

• Delivered dose: 13,000-25,000 rads

• The problem: – If treatment data was submitted through the

console within 8 seconds, the unit could end up partially set for xray treatment and partially set for electron treatment

– Impossible to determine mode of operation at any given time

-Samek: bottom-up design process and the problem still exists today only with some concurrency patches included.

Therac-25

Summary: Priority Inversion

• Important tasks miss deadlines because less important tasks are allowed to run instead

• The fix: – Priority inheritance– Priority ceiling– Priority ceiling protocol

• IT COULD HAPPEN TO YOU (US)! BE PREPARED.

Questions?