benchmarkingwlans

8/3/2019 BenchmarkingWLANS

1/8

A Farpoint Group Technical Note

Document FPG 2006-314.1October 2006

Benchmarking Wireless LANs: Recommended Practice


2/8

Benchmarking Wireless LANs - Recommended Practice 1

Farpoint Group Technology Note October 2006

B enchmarking has been used throughout the history of computers and networks toprovide a rational, quantifiable basis for the comparison of two or more similar prod-ucts or services. One might wish, for example, to compare the processor perform-ance of two PCs, or the net throughput of a given network configuration based on two differ-ent Ethernet switches. The key to successful benchmarking is holding the values of as many

variables as possible constant in order to isolate the contribution(s) of the specific elementsbeing compared. For example, when comparing the performance of two computers, oneshould endeavor to assure that the configurations of the two are as similar as possible, run-ning the same OS (at the same patch or revision level), having identical main memory, etc.,unless, of course, one wishes to evaluate the performance of different OSes or revisionsthereof or the effect that varying the amount of memory might have on the results.

And deciding what results to obtain is often a further challenge. Typically, benchmarkingany network gear is about throughput in terms of megabits per second (Mbps) for a givensynthetic workload, the benchmark application. Note that megabits of throughput in thebenchmarking environment may or may not correlate to actual throughput in the real world,

although a good chance of correlation can be achieved if the benchmark application is care-fully designed to simulate traffic typical of a given real-world application. More commonly,however, it is simply important to apply the same synthetic workload to the two or moreconfigurations being compared, again holding other variables constant, and comparing theresults obtained. While benchmarking is not an exact science, careful attention to what is be-ing measured, and eliminating the possibility that other sources of variability or error mightaffect the outcome, will result in information useful in quantifying the differences betweenthe products under test.

Wireless, however, introduces a potentially broad range of variables that are not factors inother network benchmarks, most notably artifacts in the radio environment relating to radio

propagation and interference. As we noted above, care must be taken to hold these variablesas constant as possible as well if valuable results are to be obtained. For this reason, wirelessbenchmarks are often run in Faraday cages [ http://en.wikipedia.org/wiki/Faraday_cage ]or other environments where external RF energy cannot affect the test results. Such is usu-ally impractical in end-user-oriented tests, but, as we will discuss below, the effects of exter-nalities can be mitigated to a very great degree with a simple set of precautions and proce-dures that introduce only minimal additional cost to the exercise. Indeed, what we proposeherein as a Recommend Practice is designed to be simple, broadly applicable, and a sourceof results valuable to those comparing multiple wireless LAN products and systems.

What to MeasureWhile most networking benchmarks are concerned only with net (of protocol and other over-head) throughput resulting from a given synthetic workload, wireless benchmarks must takeinto consideration the variability in throughput that results from differences or changes in thedistance between endpoints. There is, in general, an inverse relationship between distanceand throughput in terrestrial wireless systems the greater the range, the lower the effectivethroughput. Moreover, note that Wi-Fi-compliant wireless LANs have the ability to


3/8



downshift and upshift the signaling rate in effect at any given moment in time. But whilethe IEEE 802.11 standard specifies this ability, it does not say under what conditions suchchanges are to be applied. As a consequence, this behavior in specific implementations isamong those that are ultimately measured via a benchmarking exercise.

We generally recommend using a relatively short distance (a few meters) between the endpointsto establish a baseline for the performance of any given connection. We then recommend usinglonger distances, up to a range dependent upon the parameters of a given test, to evaluate rate-vs.-range performance , important in many deployments. The most test points, the greater theconfidence one can have in the results. We suggest that distances of up to 30 meters, includingsome number (even up to five or six) of intervening walls and floors, can be very useful instress-testing WLANs, on the assumption that good long-range performance implies bettershort-range performance as well. Again, as long as all products are tested under the same condi-tions, the specific test geometry and building construction are of less importance; it is regardlessimpossible to predict the exact path radios waves will take in any case. Testing the same equip-ment in multiple venues, however, can further improve confidence in the results obtained.

As long as the same benchmark, with identical parameters, is used consistently within a givenexercise, the specific benchmark chosen can be any that simulate a real-world workload. Theseapplications include popular (and free) Iperf benchmark [ http://dast.nlanr.net/Projects/Iperf/ #related ] and Ixias IxChariot [ http://www.ixiacom.com/products/ performance_applications/pa_display.php?skey=pa_ixchariot ], among others, as well asproduction applications. We recommend multiple runs of both upstream and downstream(separately, not simultaneously) traffic so as to fully evaluate the performance of both ends of the connection. While we usually evaluate only one client on one AP at a time, enterprise-classbenchmarking can be performed with multiple APs and clients if desired. Again, as long as pa-rameters are held constant across runs, almost any configuration can be tested and evaluated.

An example benchmark configuration is shown in Figure 1. This is the one we used in theMIMO router testspublished in Net-work World in Au-gust of 2006 [ http:// www. network-world.com/ reviews/2006/ 080706-mimo-router-test.html ]. In

this case, we usedtwo copies of Iperf server run on thenotebook computerwith the PC Card.Iperf client was runon two PCshardwired to the

Figure 1 - A sample WLAN benchmark configuration. Source: Farpoint Group


4/8



wireless router (used only as a switch and AP) under test. Note that all IP addresses as static;this makes it much easier to debug a configuration should connectivity problems occur.

The statistical evaluation of benchmark results is largely up to the user. We usually make dowith simple averages, but more complex techniques, such as cumulative distribution functions

(CDFs) [ http://en.wikipedia.org/wiki/Cumulative_distribution_function ], may prove usefulin some circumstances.

Dealing with the Radio Environment

As was noted above, the single largest variable in any benchmarking exercise is likely to be theradio environment itself. Unlike the nice, neat little electromagnetic world that exists on wire,cable, or fiber, the freespace behavior of RF energy contains numerous opportunities for statisti-cally-variable and even entirely anomalous (and thus unreliable) results unless the impact of the

RF environment is carefully managed from the start.We recommend, therefore, the use of a spectrum analyzer or related functionality to measure,monitor, and continuously evaluate the RF environment. Commercial spectrum analyzers, avail-able from Agilent [ http://www.home.agilent.com/USeng/nav/-536902453.0/pc.html ], An-ritsu [ http://www.us.anritsu.com/products/ARO/North/Eng/Spectrum-Analyzers-(Wireless).aspx?lc=Eng&cc=US&rc=ARO&cat=1&cat2=3&cat3=46&cat4=0 ], and others,are electronic test equipment and thus can be quite complex to operate and usually require anengineer for useful results. We have lately been using Cognios Spectrum Expert [http:// www.cognio.com/solutions_mobile.html ], a PC card/software product also available embed-ded in WLAN tools such as AirMagnet Spectrum Analyzer [ http://www.airmagnet.com/

products/spectrum.htm ], and recommend this product highly. Were even seeing very inex-pensive ($100) spectrum analyzers are starting to appear such as Wi-Spy [ http:// www.metageek.net/ ], although these are hardly suitable for professional applications. WithWLANs becoming more than common in office environments everywhere, finding a clear (or atleast usable) channel can be a challenge. We therefore often recommend that benchmarks berun overnight, or even longer, in order to average out the effect of any destructive interference.Regardless, spectrum-analysis tools can be quite useful in picking a (mostly!) clear channel foruse in benchmarking, as well as documenting the environment for later analysis.

A few additional RF-environment-related factors must also be considered, as follows:

Antenna orientation Radio propagation is highly dependent upon the physicalenvironment, the geometric relationship of the transmitter and receiver, and the type andorientation of the antenna(s) used. Farpoint Group recommends the use of turntables(see Figure 2) on client PCs in order to eliminate the possibility that a given client is in adeep fade relative to the access point. Deep fades (also called nulls ) exist where a com-bination of specific frequency, transmit power, and antenna type and orientation com-bine to result in a significant loss of receive signal strength, resulting in a very weak sig-nal at the receiver. While this can (and does) occur in production environments, a user


5/8



will often note the problem via the tool-tray glyph or other mechanism and manu-ally correct for it. A slowly-revolvingturntable (1 RPM is recommend) is suffi-cient to eliminate this problem during a

benchmark run, and also compensates forthe inability to place a computer preciselyin the same spot after swapping PC Cards.These turntables are widely available onthe Web and are typically used to powerrotating display cases in retail stores.

Maintain identical locations - It is almostimpossible, given variances in antenna im-plementations and packaging, to maintainexact locations for infrastructure equipment as APs are swapped. One should try to get

as close as possible, but some difference is unavoidable and will likely have little effecton the results. Note that the number of APs may be varied when benchmarking specificconfigurations of enterprise-class products. The turntable on the other end at least par-tially compensates for any variability here.

Repeatability Note that even if we hold all variables constant, the vagaries of radiopropagation may yield differing results for two otherwise identical benchmark runs. Forthis reason, we recommend performing multiple runs and averaging the results. As analternative, as was noted above, a given benchmark can be run for a long period of time we have in the past run benchmarks for a full day (24 hours). Usually, however,performing three consecutive runs and averaging the results is sufficient.

Additional Core Recommendations

As we noted above, it is critical to do an apples to apples comparison in any benchmarkingexercise, and failure to consider all variables can produce results that are misleading or even

just plain erroneous. In addition to procedurally factoring out the effect of the radio environ-ment and using a common workload, the following must be considered when benchmarkingwireless LANs:

Router and driver settings The decision on whether to use other than the default set-tings must be carefully considered. In general, both WLAN chipset and finished-goodsvendors have told us that they ship their products with optimal driver settings in place,and modifications should not be necessary beyond changing the SSID and adjusting se-curity settings. Given the huge number of possibilities and the large amount of time thatwould be involved in experimentation with alternatives, we almost always stick with thedefaults, and carefully document when we do not. Security settings are an exception,and, as we have always suggested WPA as a minimal over-the-air security strategy,

Figure 2 - A turntable, in our lab. Source: Farpoint Group


6/8



thats what we have traditionally used in benchmarks. As WPA2 is now a requirementfor Wi-Fi certification, we prefer to use WPA2 in all tests. We suggest against using anyupper-layer security additions (such as a VPN or 802.1X), as well as avoiding any otherupper-layer software other than the benchmark application itself, so as to minimize theimpact of additional variables, but such can be utilized if desired.

Understand the entire value chain While we focus on the airlink, other interfaces canhave a dramatic effect on the outcome of a given benchmark. For example, as Figure 1shows, we use two computers to implement two streams because the wireless link in aMIMO-based WLAN could be much faster than a single 100 Mbps Ethernet port. Card-bus controllers can also be bottlenecks; we expect that the advent of ExpressCard[http://www.expresscard.org/web/site/ ] will largely address this issue. Regardless,some analysis of every interface is warranted. Its also worth noting here that some note-book computers just work better than others, and a number should be tried.

Use the latest software revisions Similarly, it is good practice to check that one is us-

ing the latest firmware and driver revisions available for all wireless products. We sug-gest that all products being at the latest revision levels provides the best basis for a com-parison. Note that both routers/APs and client devices need to be so configured, alongwith any intervening switches if these are used.

Use production hardware Its important to verify that all hardware be in productionand not based on any special tweaking (these are sometimes referred to as juicedunits). This can normally be assured by buying products through retail channels, but se-rial numbers should be listed in any reports especially for equipment received directlyfrom a vendor.

Understand Mbps Most vendors make specific claims as to the throughput of theirproducts. With contemporary MIMO-based wireless LANs, it is not at all unusual to seenumbers such as 240 Mbps and even 300 Mbps in marketing materials and in theconnection speeds reported by vendor utilities (packaged with driver software). Thesenumbers are physical-layer (PHY) connection speeds and have little to do with realiz-able throughput. The impact of protocol (IP and related) overhead and constantly-varying radio conditions must also be accounted for, and, perhaps not surprisingly, wehave found products advertising lower PHY speeds significantly outperform those ad-vertising higher rates. And, again, the Mbps numbers obtained from a benchmark runmay or may not correlate to those realized in production environments and applications,but will regardless be a much better guide to throughput than manufacturer claims.

Future Opportunities and Directions

In this document, we have defined objectives and procedures for basic rate-vs.-range perform-ance testing of wireless LANs. In the future, we intend to extend this Recommend Practice tocover two other elements, as follows:


7/8



Roaming Given that roaming in 802.11-based WLANs is somewhat non-deterministic(to put it kindly), testing the performance of a WLAN system while roaming can bemost challenging indeed. We are now experimenting with robotic techniques using inex-pensive robots from Evolution Robotics [ http://www.evolution.com/er1/ ]. These ro-bots can be programmed to move in a repeatable fashion, and such approaches mayprove valuable in quantifying future roaming benchmarks and other tests.

Voice, Video and Multimedia Given that these forms of communication are designedto deliver streaming information to a human and not a computer, the quality of the infor-mation delivered over time must be analyzed. Such evaluation can range from the sub-

jective (for example, a Mean Opinion Score , or MOS ) to the analytical ( R-values ) andback to the purely subjective evaluations required with respect to video quality. We willcover this issue and present a Recommended Practice for benchmarking such applica-tions in an upcoming Tech Note.

In addition, we are interested in benchmarkingtechniques other than those using purelyfreespace testing, via the use of analytical andsynthetic methods. These involve utilizingsimulations which are either purely numericalor the use of specialized test equipment thatworks in a closed (RF-controlled) environ-ment. With respect to the former, it shouldcome as no surprise to technical professionalsthat we believe much future benchmarking willbe done purely numerically via computational

simulations. After all, modern jet aircraft andother mission-critical, high-value items are de-signed and tested this way, and we expect tohave enough practice and experience over thenext few years to construct reliable mathemati-cal models, especially as implementations of 802.11n are in place and suitable for modelingand verification against real-world products. Inthe meantime, we expect that specializedWLAN test equipment, such as that availabletoday from such vendors as Azimuth Systems [ http://www.azimuthsystems.com/ ] (seeFigure 3) and Veriwave [ http://www.veriwave.com ] will be quite useful in performing closed-environment testing, where the variability of the radio environment can be both simulated andcontrolled, and tests for security, roaming, and other elements conducted in a highly-repeatableenvironment. The key in this case is in implementing accurate real-time channel models , andsome progress towards this goal has already been made by such firms as Azimuth and Elektro-bit Ltd. [ http://www.propsim.com ], among others. We expect, regardless, that free-spacetesting will remain very common for at least the next decade, and that the techniques outlined inthis Tech Note will remain very valuable well into the future.

Figure 3 - A closed-environment test system.Source: Azimuth Systems


8/8

Ashland MA 01721508-881-6467

[email protected]

Copyright 2006 All rights reserved

Permission to reproduce and distribute this document is granted provided this copyright notice is includedand no modifications are made to the original.

The information and analysis contained in this document are based upon publicly-available information sourcesand are believed to be correct as of the date of publication. Farpoint Group assumes no liability for any inaccura-cies which may be present herein. Revisions to this document may be issued, without notice, from time to time.

benchmarkingwlans

Documents