Measuring I/O Channel Performance on Windows and UNIX Systems

Dr. Bill PierceTeraCloud Corp.

7-8-04

Business Goals Of I/O Performance Measurement

• Meet human expectations for application responsiveness

• Achieve the highest possible level of service at the lowest possible cost

• If you don’t measure, you can’t evaluate and compare options

Why measure I/O Channel Performance?

• I/O speed has increased much more slowly than processor

• I/O stacks have become more complex, hard to predict

• So, measure, verify and monitor expected levels of service

• Tune a stack for maximum speed

Application

C library

System Call

Filesystem Driver

OS and Buffer Cache

Device Driver

Hardware/Firmware of Adapter

Network

Switches and Routers

Network

Channel Adapter

Director

Layers of Cache

Internal SBOD

Disk Cache

Arm and Rotating Platter

Host Layer

NetworkLayer

StorageDevice Layer

Characterization of I/O Workload

• Characteristic sizes [bytes]• I/O rate at each characteristic size [1/s]• % read vs. write at each size [%]• Probability of cache hit [%]• Burstiness [distribution of request deltas]

Channel Characterization

• Throughput (better for large sequential access) [bytes/s]

• IOPS (better for small random access) [1/s]• Latency (Service Time) [s]• Queue Length [1]

Some Relationships

• Utilization Law:– (Utilization) = (Throughput) × (Service Time)

• Little’s law:– For utilization < 1– (Queue Length) = (Arrival Rate) × (Service Time)

How to Measure

• Active vs. Passive measurement tools

• Measure the whole I/O Stack• OS vs. application measures

– When 1 byte/s != 1 byte/s– Transfer overhead– Which do you care about more?

Active Passive

Windows IOmeter, IOzone io_profile

Sysmon, Task Manager

UNIX Bonnie/Bonnie++, IOmeter, tiobench, IOzone, io_profile

iostat, sar

I/O Spectral Profiling

• More is learned by the shape of the profile than a point measurement

• Use a measure that is self-normalizing, such as IOPS (as a function of size)

• Tune the channel for the characteristic I/O sizes. Watch the effect of tuning across sizes.

DAS, SAN-RAID5 Comparison, No Cache

0

500

1000

1500

2000

2500

100 1000 10000 100000IO size (bytes)

IOs/

s

writes/s DAS no cache

writes/s SAN no cache

reads/s DAS no cache

reads/s SAN no cache

The io_profile Spectral Profiling Utility

create_test_filesystem();

do_io(write); do_io(read);

end;

function do_io(mode) {

for io_size {

start_timer();

for repetitions {

file = choose_random_file()

open(file);

seek_to_random_location(file);

read_or_write_(mode, io_size);

close(file);

}

time = stop_timer();

}

The io_profile Spectral Measurement Tool

bash-2.04$ ./io_profile -h

Usage: io_profile [-svfh] [-m min_io] [-M max_io] [-r reps] [-l file_len]

Options:

-s single file mode (default is filesystem)

-b use low-level I/O to bypass the cache

-v print version and exit

-h print this help

-m min_io minimum I/O in bytes

-M max_io maximuim I/O in bytes

-r reps repetitions at each io size

-l file_len length of each test file in bytes

The Importance of the Buffer Cache

• Reuses recently accessed data without making round-trips to storage

• Fast memory acccess compared to network transit and mechanical storage mechanisms

• Writes can be made asynchronous.

• Reads can come from cache hits.

Application

BufferCache

IO Channel

Memory

The Importance of the Buffer Cache

• Which do you wish to measure:

• With caching or without?Influence of Buffer Cache

0

500

1000

1500

2000

2500

100 1000 10000 100000I/O size (bytes)

IOs/

s

writes/s IDE DAS

writes/s DAS no cache

reads/s IDE DAS

reads/s DAS no cache

Comparisons of Some Standard Channels (preliminary)

DAS, SAN-RAID5, NAS Comparison, with Cache

0

500

1000

1500

2000

2500

1 10 100 1000 10000 100000IO size (bytes)

IOs/

s

writes/s IDE DAS

writes/s SAN

writes/s NAS

reads/s IDE DAS

reads/s SAN

reads/s NAS

Conclusions of the Measurements(preliminary)

• Reads are generally faster than writes, with read speed falling off at large I/O sizes (strongly when cache is used).

• With cache, DAS and SAN-RAID5 and SAN-JBOD (not shown) were about the same and much faster than NAS over CIFS.

• Without cache (not shown), SAN-RAID5 (particularly reads) appear to be accelerated over DAS and SAN-JBOD drives.

• But don’t take my word for it, measure your systems for yourself…

Tool Sources

tool source

Sysmon Ships with Windows NT product line

IOmeter www.iometer.org

IOzone www.iozone.org

io_profile www.teracloud.com/utilities.html

Bonnie http://www.textuality.com/bonnie/

Bonnie++ www.coker.com.au/bonnie++/

tiobench http://sourceforge.net/projects/tiobench/

iostat ships UNIX OS

sar ships with UNIX OS

See www.acnc.com/benchmarks.html for more.

Questions?