Using Hard Disks in Real-Time Systems

Mark Stanovich

Context Real-time systems Raw disk I/O Hard disks with built-in scheduler/queue Mixed Workload

Disk requests with deadline/response time requirements

Background/best effort requests Want to guarantee real-time deadlines

Difficulties Real-time schedulability analysis generally

relies on knowing worst-case execution times (WCET)

Non-preemption makes guarantees even more difficult

Variability of disk service times is extreme (tens of milliseconds to several seconds)

Result is that hard disks are not prevelant in the critical path of a real-time system

Meeting guarantees vs optimizing disk utilization

Value of Research

Provides more information to provide schedulability analysis

Large capacity of hard disks can be utilized in the critical path of a real-time system

Less burden on other resources Lower latencies

Applications

Multimedia Data Logging Webservers Data Analysis

Disk Scheduling Allow internal scheduler to schedule requests

less burden on the CPU device driver can be at a lower priority fine-grained internal state does not have to be

maintained disk specific characteristics can be utilized

All scheduling performed inside the OS more control over request service order scheduling policy can be changed

Disk Scheduling Rotational-Position-Aware Real-Time Disk

Scheduling Using a Dynamic Active Subset (DAS)

A Real-Time Disk Scheduler for Multimedia Integrated Server Considering the Disk Internal Scheduler

Dynamic Active Subset

upon each scheduling decision, the calculation of a subset of the outstanding disk requests such that all service guarantees can be enforced under worst-case assumptions

schedule the subset based on the rotational position of requests in order to improve scheduling decision

Response Times

Response Times

Worst-Case Execution Time

seek time rotional delay * number of rotations to settle access time for some number of sectors

time per sector varies depending on the zone overhead time

disk controller processing data transfer between disk and host system

skew time * v time to switch next cylinder and next disk head v depends on maximum request size and minimum

size of a single track

Number of Rotations to Settle

Rare cases the disk head needs some additional rotations to settle on the destination track

Provoke worst-case by alternately issuing requests to the innermost and outermost region of the disk

Worst-Case Execution Times

Hiding Overhead Times

substantial amount of time communicating with the disk without media access

use TCQ to minimize these times send 2 requests so that as one request is

transferred to or from the disk the other will be executing

Real-Time Disk Scheduling

Execute all real-time requests at the beginning of each period limits the scope of scheduling optimizations to

request classes DAS

construct a subset of the outstanding requests such that no service guarantee will be violated regardless of which request is executed

all scheduling algorithms can be used while ensuring deadlines

dynamic nature of DAS does not allow scheduling inside the disk controller's hardware

DAS

Performance

Autonomous

345 468 589 642

disk head location

= data locations to retrieve

AutonomousWorst-case Rotational Latencies

5400 RPM = 11.1 msec7200 RPM = 8.3 msec10K RPM = 6 msec

disk head location

Autonomous cons

without detailed knowledge of hard disk internals, service times are difficult to predict for real-time requests

Lack of Preemption Capablility

Real-time requests must wait for current request to finish if current request takes too long, even if we start

the real-time request immediately, it may fail to meet its deadline

NCQ does not allow requests to be pushed to the head of the queue now we may have to wait for all requests on the

disk to be processed first

Response Time

Simplistic Bound rotation latency + full stroke seek time Example: Maxtor 73G 10K RPM drive

worst case latency: 6 msec + 11 msec = 17 msec

Response Time

backgroundtask real-time task

Response Time

RT I/O Scheduler

simple no merging no sorting

accomodates I/O priorities

VFS

block I/O

device driver

elevator

application

user space

kernel space

Response Times

Response Times

Problems disk unaware of request priorities starvation of requests

new background requests sent to the disk are serviced before older requests

better performance to keep disk head in a certain region, less disk head movement

Response Times

Response Times

Solutions use round based scheduling with ordered tag to

prevent background requests from being serviced before real-time requests [Kim 2003]

rely on disk starvation prevention algorithm draining of disk queue limiting on disk queue depth

Draining

Draining

allow disk to service request already on the disk without sending any new requests

drain_time(n) maximum time to service n disk requests with no

subsequent requests being sent to the disk condition to send new request:

if (current time + drain_time(x)) <= earliest deadline, where x is the size of the on-disk queue depth

Draining

determing drain_time(n) on-disk scheduling logic unknown, therefore makes

analytical analysis difficult can empirically determine

send n number of requests to the disk and measure the time to completion

how to know when the worst case response time has been reached

Draining

Limiting On-Disk Queue Depth

max_depth maximum number of outstanding requests

permitted to be sent to the disk condition to send new request:

if number of outstanding requests < max_depth

Implementation

VFS

block I/O

device driver

elevator

application

user space

kernel space

Experimental Verification

periodic real-time task requests data from disk period = deadline = 250 msec 256KB request size

450 constant background asynchronous requests sent to same disk

Experimental Verification

Experimental Verification

Conclusion

more intelligent, autonomous hard drives increase the complexity of scheduling requests

command queuing provides some assistance, but does not address all real-time disk I/O issues

draining and limiting the on-disk queue can be used to maintain deadline constraints

several aspects of disk behavior is still unexplained and until these are resolved, no absolute guarantees can be made

Previous Work

Pro's takes advantage of drives internal mechanisms can guarantee most requests in a round uses Linux, a commodity OS uses an admission controller for the real-time

requests Con's

constrained to the time interval of a round all requests of one round are treated as equal

priority does not mention about priority of disk device

driver

constrain the internal features of a disk in order to provide some idea of reserved bandwidth

start with periodic requests (benefit from the knowledge of upcoming requests)

as time gets close to the real-time request reduce admission of best-effort requests to the disk (gradually lower on-disk requests)

may also want to constrain the region in which the disk can do work if the region of the real-time request is known

reserve bandwidth for some time interval

Work in progress

Work in progress

disk drive has a worst-case bandwidth admission control can allocate up to this parameter after that cannot guarantee anything else for hard

real-time constraints best-effort requests can fit before real-time

requests as long as the requests do not jeopardize the upcoming real-time request (worst-case service time)

real-time requests usually will not fill the entire time allocated for the worst case bandwidth therefore time will be available for best-effort requests

use ordered tag to send best-effort requests after ALL known real-time requests are issued to the disk

Work in progress

Differences between read/writes writes normally require a longer settle time

time for head position to stabilize on the selected track

Work in progress

Metrics extent to which real-time request exceeded

deadline (for hard real-time this should be 0) average response time of real-time request in

comparison with calculated worst-case time bandwidth of best-effort requests average/min/max of on-disk queue depth used

NCQ allows for only 32 on-disk requests are more really needed such as the SAS drives with

256 requests (SCSI TCQ has a maximum queue length of 2^64)

stress an actual implementation displaying video with best-effort applcations in the

background (compile kernel, copy large files, etc.)

Constraining the Disk Reduces Efficiency

Not necessary to send ordered tag right away sending the ordered tag may put unnecessary

constraints on the internal scheduler may be better to stop future requests until the real-

time request is completed or to some minimum internal disk queue length