DOCUMENT RESUME

ED 327 546 TM 015 032

AUTHOR Sykes, Robert C.TITLE Stability of IRT b-Values over Time and Positlon.PUB DATE 89

NOTE 30p.

PUB TYPF Reports - Research/Technical (143)

EDRS PRICE MF01/PCO2 Plus Postage.DESCRIPTORS Analysis of Covariance; Certification; Comparative

Testing; *Item Response Theory; *LicensingExaminations (Professions); Predictive Measurement;*Research Methodology; *Test Items

IDENTIFIERS *8 Values; *Invariance Principle; Item Position(Tests); Rasch Model

ABSTRACTAn analysis-of-covariance methodology was used to

investa-ate whether there were population differences between tryoutand operational Rasch item b-values relative to differences betweenpairs of item response theory (IRT) b-values from consecutiveoperational item administrations. This methodology allowed theevaluation of whether any such differences could be attributed todifferences in item position after controlling for the backgroandeffect of scale drift on b-values. Usable real or scored items in alarge lIcensure examination were selected for study. The final sampleconsisted of 406 administration difference pairs. For each item, arecord was created containing: (1) administration dates; (2) itempositions for several administrations ending with July 1987; (3)classification code; (4) type of administration (real or tryout); and(5) b-value and standard error. The block-wise placement of tryoutitems had no apparent effect on their b-values. Neither differencesin booklet nor test item locations were significantly associated withdifferences in b-values across consecutive pairs of administrations.A small but significant increase in b-value differences in pairshaving earlier administration (between February 1981 and July 1986)suggested that scale drift increased over this period. Three datatables and eight figures are included. (Author/SLD)

* Reproductions supplied by EDRS are the best that can be made *

* from the original document. *

r- i

U.S. oenutTmerr or aoucoatotsOface of Educational Ressordi and Improvement

EDUCATIONAL RESOURCES INFORMATIONCENTER fERICI

lohe document has been reproduced asrecerved from the person or onmnitahonmoneampt

0 Minor changes have n made to improvereproduchon qualify

Punts of wow or opintons stated +rt this docu-ment do not necessarily represent officialOEM poulton or poky

"PERMISSION TO REPRODUCE THISMATER:AL HAS BEEN GRANTED BY

Roecti- C. sixes

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC) "

Stability of IRT b-values over

Time and Position

Robert C. Sykes

CTB/MacMillan/McGraw-Hill

2

Introduction

The use of item preequating to calibrate tryout items onto an IRTscale provides a means to build large pools of calibrated items. ,

The grouping of these tryout items as blocks of consecutive items,affords cost efficiencies in the printing and scoring of test .

booklets, but raises questions concerning whether the restrictedplacement of tryout items relative to operational items inducescontext or order effects. An Analysis of Covariance methodologywas employed in this study to investigate whether there werepopulation differences between tryout and operational Rasch itemb-values relative to differences between pairs of b-values fromconsecutive operational item administrations. The Analysis ofCovariance methodology allowed an evaluation of whether any suchdifferences could be attributed to differences in item positionafter controlling for the background effect of scale drift on b-

values.

As noted by Wainer and Kiely (1987) in their discussion of thepotential impact of context effects on CAT (Computerized Adaptive-Testing) scores or item parameters, the influence that itemlocation within a test may have on item parameter estimates is

one type of undesirable item context effect. Several studieshave documented differences in item difficulty parameterestimates as a function of their location Whitely and Dawis(1976) reported that for 9 out of 15 verbal analogy itemsembedded in seven different unspeeded tests parameter estimatesdiffered when parameters were obtained for each test. Yen (1980) .

found that for reading comprehension items administered in anonspeeded test, items became more difficult the later in the

test they appeared. A preequating study by Eigner and Cook(1983) replicated Yen's findings. Wise et al (1989) also founditem position effects for Word Knowledge and Arithmetic Reasoning

tests.

If the effects of differences in item location are assessed onparameter estimates obtained from item administrations spanning alength of time, estimates of such differences may be biased bythe presence of background scale drift. Bock, Muraki, andPfeiffenberger (1988) document the presence of differentiallinear drift of location parameters of College Board PhysicsAchievement items over a 10-year period. Additionally, theypresent statistical procedures which may be used for detectingand estimating item parameter drift in item pools.

The potential presence of scale drift in long-term testingprograms would thus necessitate that any evaluation of change initem parameter estimates incorporate procedures for controlling,and perhaps estimating the magnitude of the drift if parameterestimates are not obtained from concurrent item administrationa.Incorporation, in a design methodology, of statistical proceduresfor controlling the effect of drift allows independent estimatesof the mean difference in item parameters across type ofadministration (tryout vs operational) as well as an independent

3

evaluation of the extent to which such changes may be attributedto differences in item location across test administrations.

Method

All usable real or scored items in a large licensure examinationwere selected for study. A predominant number of the test itemsfor the examination are typically associated with passages. For

each of the real or scored items in the four booklet test, itemstatistics for a 7/87 administration as well as all previousadministrations were obtained. For each administration of eachitem, a record was created containing:

1) Administration date2) Classification code3) Position in the test booklet and complete exam indices4) Type of administration, real or tryuut, and5) b-value and standard error.

The selected 7/87 form had been equated to previous forms bysetting the mean 7/87 b-value, obtained from all scored items,

equal to the mean of the b-values obtained from the lastadministration of the scored items.

The presence of item statistics or parameters from multipleadministrations of each item implied that any assessed itemstatistic would not be independent across the observationalunits, that is, the test administrations. In order to obtain anindependent unit of analysis, differences in b-values for allpairs of consecutive administrations (later administration minusearlier administration) were computed. Thus, if an item wasadministered on three other occasions besides 7/87, sAy 2/86,7/84, and 7/82, three b-value differences would be generated:the b-value in 7/87 minus the b-value obtained in 2/86, the 2/86b-value minus the 7/84 b-value, and the 7/84 b-value minus the

7/82 b-value.

Independent differences in b-values across pairs of consecutiveadministrations permitted an evaluation of the stability of b-

values using en Analysis of Covariance methodology. Thismethodology provided a means to:

1) Determine the presence of significant linearrelationships between difference3 in item position in

the test booklet or complete exam and changes in b-valuesacross administrations and

2) Assess possible "background" scale drift confoundingdifferences in b-values.

2

4

Furthermore, the Analysis of Covariance promised more powerfulsignificance tests of differences between tryout and real itemstatistics by regressing error on any position or time indexhaving a significant relationship with b-value differences.

The sample of paired, consecutive administration b-valuedifferences was used for a set of comparative analyses. In thefirst analysis, differences in administration b-values (noted intables and figures as BVDif) were assessed in a crossed,unbalanced design consisting of three classification variables:

1) Type of administration pair (TypePr): both realadministrations vs. a later real, earlier tryoutadministration.

2) Domain 1 (NP) classification of the item (5 classes) andall its consecutive administration pairs, and

3) Domain 2 (CN) classification of the item (4 classes) andall its consecutive administration pairs.

Paired administration b-value differences were compared acrosslevels of the classification variables after fitting the mostparsimonious model for the linear regression of the errors on the

covariables.

The initial regression model consisted of four covariables.Difference in book position (BP1-BP2) and difference in positionin the complete exam (TP1-TP2) were entered last and assessed for

sig,:ificance. Given the length of the exam, four test bookletsof 93 items given over the span of two days, it was conceivablethat fatigue or motivational effects may be more substantial nearthe end of the test than near the end of the first booklets. Themagnitude of associations between the two position indices and b-value differences might then be cgpected to reflect thesedifferences in effects. For both indices position in the second,earlier administration was subtracted from the position in thefirst, later administration.

The two other covariables served as controls for potential scaledrift and consequently were entered before the difference-in-position indices. The first of these control covariables was anindex of time elapsed, in months, from 2/81 (TimFr281) to theearliest administration in each consecutive administration pair.This covariable would demonstrate a significant positivecorrelation with the b-value differences if scale drift hadoccurred over the span of six and a half years, 2/81 through

3

5

7/87, and if the drift consisted of an ircrease or decrease inmean item difficulty or candidate capability at several occasionsover this period. Negative equating constants for everyadministration back to at least 2/85 suggested a shift in themean candidate capability during this period.

The second of the control covariables was time elapsed (inmonths) between the earlier and the later administrations in eachadministration pair (TimBtwAd). The purpose of this index was toensure that any difference between real-real administration b-value differences and real-tryout administration b-valuedifferences could not be attributed to differences in the elapsedtime between the two types of administration pairs. The shorter,on average, time elapsing bef-'..e a tryout can be administered asa real compared to the Gverage time a real item is retired beforereadministration might be expected to produce smaller meandiffexences for real-tryout administration pairs if significantscale drift occurs across the period spanned.

Analyses and Results

Analysis of b-value differences

Initial plots of b-value differences by the four initialcovariables were scanned for outliers. Six consecutiveadministration pairs were deleted because of extreme b-valuedifferences. A total of 406 administration difference pairsremained in the sample.

Means and standard deviations of the dependent variable and fourprospective covariates are presented in Table 1. In addition tothe BVDif index of change in b-values, the difference inconsecutive administration b-values relative to the standarderror of the later administration (BVDif/s.e.1) is provided,hereafter referred to as BVDifSt.

The average b-value difference of .01 does not differsignificantly from 0 when assessed against its standard error(.01). B-value differences are, on average, .29 of a real item

standard error. This is slightly more than 1.6 times thestandard error of the BVDifSt mean (.18) as opposed to the onestandard error of the BVDif mean represented by the averageBVDif.

The average time elapsed between the later (chronologically) realadministration and the earlier real or tryout administration inthe paired consecutive administrations was 22.99 months. Theaverage difference in test booklet position and test positionbetween the later and earlier administrations was negative forboth position indices, -23.78 and -58.37, respectively. Bothmean BP1-BP2 and mean TP1-TP2 are substantially less than 0

4

6

because of the substantial number of tryout administrations.Their higher (i.e., later on average) booklet placement andhigher complete test position are subtracted from lower, onaverage, book and test positions for the next realadministration.

TUrning to the correlations and their significance levels (sign.)also presented in Table 1, a small but significant positivecorrelation of .13 is found between the b-value differences.andelapsed time from 2/81. B-values of the later administrationtend to increase relative to the earlier administration as timefrom 2/81 increases. The positive association between the b-value differences and elapsed time from 2/81 is slightly morepronounced when the differences are representad relative to astandard error, viz, a significant .16 correlation betweenBVDifSt and TimFr281. A significant positive association betweenthese two indices might be expected in the presence of a scaledrift that increased between 2/81 and 7/87.

The significant associations demonstrated between TimFr281 andboth TimBtwAd and TP1-TP2, as well as the significant, smaller itabsolute value, negative association between TP1-TP2 and TimBtwAd

are artifacts. For example, a substantial negative correlatiorof -.51 was observed between TimFr281 and TimBtwAd because thelonger the time elapsed from 2/81, the shorter the maximumpossible time between administrations, given the upper bound of

the 7/87 administration dat-3.

The differences in consecutive administration b-values, less thesix paired observations deleted as outliers, were plotted against

the four prospecti covariates for signs of nonlinearity betweenthe dependent variable and the covariates. Thew plots may befound in Figures 1-6b. Figures 1 and 2 provide no indicationthat b-value differences across groups or classification cellsare nonlinearly related to differences in test position (TP1-TP2)or differences in book position (BP1-BP2). respectively. Anexamination of plots of b-value differences and differences inbook position within type of administration pair in Figures 3aand 3b also revealed no sign of nonlinearity and similar meandifferences of approximately 0.

The plots of b-value differences against the elapsed timeindices, TimBtwAd and TimFr28i, in Figures 4 througn 6b alsoindicate no sign of nonlinearity. The marginal distributions ofb-value differences within type of administration pair in Figures

6a and 6b appear homogenous.

podel-fittinq

The significance of the linear associations between the fourprospective covariates--TimFr281, TimBtAd, BP1-BP2, and TP1-TP2--and b-value differences were evaluated in a step-wise Aralysis of

5

7

RegreJsion. In order to perform a statistical test ofparallelism of regression slopes, it was necessary to initiallysimplify the model. A multiple regression was performedcontrolling for any group effects. These results are presentedin Table 2.

Two covariates, TimBtAd and TP1-TP2, were eliminated because ofnonsignificant linear relationships with b-value differences.When entered last in the regression, Fs' of .12 and .00 wereobtained, with respective p values of .73 and .96. Afterelimination from the model, the slopes of the b-value differencesregressed on the two remaining covariates did not differsignificantly across groups.

Of the two remaining covariates, Timrr281 and BP1-BP2, neitherexplained significant variation in b-value differences whenentered last though TimFr281 was marginally insignificant (p =.06 and p = .96, respectively, in Table 2). TimFr281 wassignificant when entered first (F = 5.94, p = .02).

Given the significance of the single predictor, TimFr281, theclassification effects were next assessed in the Analysis ofCovariance summarized in Table 3. As in the Analysis ofRegression, terms were evaluated in a step-wise fashion but, forthe covariance analysis, the classification variables weraassessed after controlling for TimFr281.

Neither the three-way nor any of the three two-way interactionswere significant after controlling for lower-order interactionsor main effects (no p < .21). The effect of type ofadministration pair, tested only after any systematic differencesin b-values due to content classification had been accounted for,

was insignificant at p = .22. Differences in the mean b-valuedifferences across Domain 2 (CN) levels not attributable to theDomain 1 (NP) classification were significant at p = .04.Because NP was the first classification effect entered into themodel, the insignificant F (.49) for the Type I sum of squaresindicated no significant difference in b-value differences aftercontrolling for TimFR281 but ignoring the other classificationeffects.

A final model incorporating a sirgle covariate, TimFr281, and theCN classification was fitted. A test of the parallelism of theregression of b-value differences on TimFr281 within each CNclassification was insignificant at P = .48, indicating thesufficiency of a common regression slope fitted acre s the fourCN levels. The single-factor model with one covariate explainedsignificant variation in BVDif (p < .005).

6

Only one of three estimable CN level effects was significant at P

< .05: Physiological Integrity. The final fitted model for b-value differences across consecutive administration pairs glPhysiological Integrity items was consequently:

Est.9-value difference = -.131 + .082 + .002 (TimPr281 - 46.12) + l

For the other three CN levels, 0 replaces .082.

Discussion

The assessment of change in item b-values across consecutivepairs of item administrations indicated no signs that theplacement of tryout items had a significant effect on the tryout

b-values. The absence of linear relationships between the b-

value differences and differences in book position or completetest position, coupled with the fact that the test booklets (andcomplet exam) are unspeeded, suggests that general motivationalor fatigue effects, which might influence performance on lengthyexams, are not conspicuously present for tryout or real items.There appeared to be no low-order effect of item position on thepredominant-4 passage-related items. Furthermore, plots of b-value differences provided no indication that there werenonlinear or more localized effects of item placement on either

real or tryout b-values.

The positive relationship between time elapsed from 2/81 and b-value differences (i.e., the significant .002 regressioncoefficient) implies the presence of an Increasing drift in thescale in the direction of increasing item difficulty between 2/81and approximately 7/86, the latest (chronologically) "early"member of a consecutive administration pair. This finding isconsistent with the equnting constants for RN 2/85 through 7/86being increasingly negative (-.032. -.G40. -.048, and -.060,

respectively). A prediction of whether the drift has continuedto increase after 7/86 would require nxtrapolation from themodel which is hazardous. Equating constants for the exams afterthe 7/86 and excluding the 2/88 (-.079, -.46, -.141, and -.069for 7/86 through 2/89) do not support a trend of invariably morenegative equating constants though, even excluding 7/88, the

average post 7/86 equating constant (-.065) is less than theaverage of the four previous exam equating constants (-.045).

Finally, the fitted model for b-value differences imputes adifferential scale drift over Domain 2 categories. The averageb-value of one category of CN items increased faster than theaverage item b-value across all content classifications.

7

9

Conclusions

1) The blockwise placement of tryout items had no apparent effecton the b-values of the tryout items.

2) Neither differences in booklet or complete exam item locationswere significantly associated with differences in b-valuesacross consecutive pairs of administrations. This result issomewhat surprising given the predominance of test items whichwere passage related.

3) A small but significant increase in b-value differencesin pairs of consecutive administrations having earlieradministrations between 2/81 and 7/86 suggests that scaledrift had increased over this period.

4) B-values increased to a significantly greater extent forone category of Domain 2 (CN) items than they did, on average,across all content categories.

8

A:foresees

Bock, R.D. (1975). Multivariate statistical metbods inbehavioral research. New York, NY: McGraw-Hill, 1975.

Hock, R.D, Muraki, E. & Pfeiffenberger, W. (1988). Itmm poolmaintenance in the presence of ites parameter 2rift. JournalZducatianassursmsnt, 25, 275-285.

Eignor, D.R., & Cook, L.L. (1983). An investigation of thefeasibility of using item response theory in the preeguating of

aptitude tests. Paper presented at the annual meeting of theAmerican Educational Research Association, Montreal.

Wainer, H., & Kiely, G.L. (1987). Item clusters and 'computerizedadaptive testing: A case of testlets. Journal of EducationalMeasurement, 21, 185-201.

Whitely, S.E., & Dawis, R.V. (1976). The influence of testcontext on item difficulty. Educctional and PsychologicalNeasurement, 329-337.

Wise, L.L., Chia, W.J. & Park, R.Y. (1989). Effects of itemsposition on IRT parameter estisates and item statistics. Paperpresented at the annual meeting of the American EducationalResearch Association.

Yen, W.M. (1980). The extent, causes, and importance of contexteffects on item parameters for two latent trait models.journal of Education Measurement, 12, 297-311.

Yen, W.H. (1981). Using simulation renults to choose a latenttrait model, ApsliesLaygbologigaLkitaLaursamt, 2, 245-262.

Table 1

Means, Standard Devistions, and Correlations of

the Consecutive Administrstion ItemStatistic Differences

(II 406)

Mil EWA TierZel Iimlimad ELM ItIlti

Seen .01 .29 46.12 22.99 -23.72 -51.37

ims .26 3.64 15.11 9.48 32.55 174.93

Mitsign.

1.00

OVDifSt .98 1.00

sign. .00

T1err281 .13 .16 1.00

sign. .01 .00

TimiltwAd -.07 -.09 -.51 1.00

sisn. .15 .07 .00

8.1-11P2 -.02 -.03 .02 -.01 1.00

sign. .65 .57 .74 77

T101-TIP2 .07 .07 .36 -.19 .03 1.00

sign. .18 .15 .00 .00 .60

Table 2

Analysis of Regression

Dependent Variable: b Value Difference

UMW df Iza..Lid Lidos Zr.2.1 InsilLid Labs '421

IlegressiQn

efects

4 .434 1.67 .10

aloinetine design

offsets

Tier281 1 .385 5.94 .02 .225 3.48 .06

Tisiltwad 1 .006 0.09 .76 .008 0.12 .73

OPIDP2 1 .043 0.66 .42 .043 0.66 .42

TP14102 1 .000 0.00 .96 .000 0.00 .96

Reduced residual 3oi 2:1.474

Error 366 23.908

13

Tata 3

Analysis ef Covariance

Dependent Variable: b Velum Difference

himDepression

it Intim Luba tr.2.1

Tinfr281 1 .444 6.89 .01

4 .126 .49 .74

CV 3 .337 2.08 .04

TypePr I .099 1.53 .22

NP * CM 12 .781 1.01 .44

TypePr * MP 4 .277 1.07 .37

TypePr * CM 3 .066 .34 .79

TypePr MP CM 12 1.013 1.31 .21

Reduced Residual 365 23.523

Total (Corrected) 405 26.887

FIGURE 1

PLOT OF B VALUE DIFFERENCE BYDIFFERENCE IN TEST POSITION

(TP1TP2)

PLOT or 11601F4DTP1_1112 SYMBOL IS VALUE OF TYPEPII

0.6 4I

2

2

I

0.6 I

I 2

2

2I I

2 2

2

I e 2 2

I I 2 I 2 I I I

0.4 I 22 2 2 1 2 2

I 2 1 III 2

2 2 2 I I 1 2

I 22 2 I 12 12 2 1 12 I 2

I I 2 1 II 12 221

1

0.2 I 2 2 I I II 1211I 1 I I 12 12

I I 22

2 2 2 I 1 2 2 2 I 1 2 2 I 2

I I 2 I I I 22 I I 1 2 2 I

I 2 12 21 112211 2 2221 II 1

2 2 2 12 I I I I I 1 21 122 1111 22

0.0 14 2 2 I 2 I 22 II II II I I I I 2 I

2 1 212 12121 2 I 111 I I II 11 2

OVOIF I I I 2 2 I 2211 2 2 I II I 2 22 21 222

I 212 111 121 1 I 2

2 2 2 2 I 2 2 2 I 2 I I 2 12 2 2

-0.2 2 2 22 2 2 2 21111 2 I I

2 1 12 II 11 11 2

2 I 2 I 2 I I I I 2 2

2 I 1 2 I 2 1222

-0.4 12 21 21 2

2 12 21I 2

I

2 I 2

2 2 I

21

2

1 1

I 2

1

2

2

-0.6 4

I

62 2

-n.e a

-1.0 I..4 4 4 -4 4 4 4 4 4 4 4 4 4 4 4 4

-600 -340 -480 -420 -360 -300 -240 -180 -120 -60 0 60 120 160 240 300

TPI_TP2

Non!15

69 06S MIDOEN

16

Mr OF OVOIPI_OP:

0.0

0.11I

I0.44 2

I

1

0.2 2

I

2

2210.0 6

I

2

6VD1F

-0.2

I

2

;47.4 I

I-0.6

I-0.6

I-1.0

FIGURE 2

PLOT OF B-VALUE DIFFERENCE BY

DIFFERENCE IN TEST BOOKLET POSITION

SYMBOL IS VALUE OF TYPEPR(BPI -BP2).

t

2

2 2

2

22

2

2

2

22

2

2

21

2

21

!

1

1

1

1

1

1

2

2222

12

221 22

212 21

121 2

2

12

2

1

I 2

2

1 1

2 1

1 22111 211III 2

1 2 1

2 1

11 1

222 12

.

112 21 1 2

1 222 2

1

2 22

1 2

I

1

2

2

1 222 1

2

22II I

2 2 I

1 212 1

21 22 1

22 12 I

1 1

2 1 212 112

22211 211 I22 1 1 1

21 222 1

III 1 II

212 2221 2

22 222 1

2 2 2212 1 2

II

222 1

21 1 2

21

2

2

1 1

1 1

1

2111 111

1

1

2111 12111 1

21

1 1

1

1

1 "0

1

2

2

l 21i

12 2

ill

I 1

2 211

II II

II 1

1

2

112

12 1

11,

2122

22

2

I

2

1

2

1

2

1

2

2

2

1

1

1

1

1

1

1

I

2

2

2

21

1

2

2

1

2

1

1

1

I

4 2 2

..4 4 4 4 4 4 4 4 4 4 4 4

-100 -60 -60 -40 -20 0 20 40 60 eo 100 120 140

BPI_BP2

NO1E: 73 ons utootm 1 8

FIGURE 3a

PLOT OF B -VALUE DIFFERENCE BY

DIFFERENCE IN TEST BOOKLET POSITION

REAL-REAL ADMINISTRAT:JNS (Typeirs1)

111111110

111.01 OF OVOIMPI_8112 LEOENO: A 1 085. 8 2 OM ElC.OVOIF

0.9

0.8

0./

0.6

1

I A

A A

A

A

0. 5A A

A AA A A

0.4 A

A A A A 8

A A A A

O. 3 A

A A A A is A 8 A A

0.2AAAAAA

A A 8 AA A A

A AA

A A A A A A A AA

A O AA s

0. 1 A AA A. A CAA A A

A A A AA A A 8 AAA AA

A A A A A A A A A

0.0 A A C 0 8 AA A A

A 8 AC AAO A 88 A A

A A OA A AA A

-0. 1 A A A A A

A 8 *A A A A A A A A

A A A

-0. 2 A A A A A A

eA A A AA

A AA A

-0.3 A A A A

AA A

A A1

1 A A

-0.4 AA A

-0.5 AA

A1

A

A-0.8

--A

-100 -SO -60 -40 -20

4 4

0 20

OPI_OP2

4

AO

4 4 4 4

60 80 100 120 140

20

FIGURE 3h

PLOT OF BVALUE DIFFERENCE BY

DIFFERENCE IN TEST BOOLIT POSITION

REALTRYOUT ADMINISTRATIONS (TypePro42)

111.6141.2

a

PLOF OF OVO1F41/11.02 LEOENOt A 1 005. 0 2 On. ETC.

0.11 4

AA

8

AA0.6 A

A A

AA

A A A0.4 1 A A A A A A

A AA A AA

A AA A A AA AA AA OA A A 11

0.2 4 A A A AA A A A A

AA A

8 A A cA A A A AA A AA A A A

A A A 0 8 8 A AA A A0.0 A A A A A AA

A 6A OA A A A AOVU1F I A A A AAA

aA A AAAA

A A A AO AA A

AA A A L A AA A A A A A

0.2 1 A A A 0 A BAAA 0 8A 0

A AAA AA A A A A A

A A A A MA-0.4 A A A

A AA A A A

A A A-0.6 4

A A

-0.6 1

-1. 0-4 4 4 4 4 4 4 4 I 4 4 4 I 4 -

-100 -90 -90 -70 -60 -50 -40 -30 -20 -10 0 10 20 30 40 50

0111_0P2

21 22

PLOT OF OVOIFTIMOTMAD

OVOIF0.75 *

0.50 *

0.25

0.00

-0.28

-0.50

-0.75

1-1.00

-1.25

-1.50

FIGURE 4

PLOT OF B -VALUE DIFFERENCE BY

TIME BETWEEN ADMINISTRATIONS (Timbtwad)

SYMBOL IS VALUE OF TYPEPR

2

2

2 1

1 2 2 1

2 1

2 1

1 2 2

2 2 2

2 1 1 2

2 2 1 1 1 1 2

1 1 2

1 1 1 1 ,' 2

1 2 2 1

1 1 1 1 1 2

2 2 1 2

2 1 2

1 1 2 2 2

2 2 2 1 1 2

1 2 2 1 2

2 2 1 2

1

2 2 21

2 2

1

2

1

1

i* + 4 * + * *

5 ND 15 20 25 30 35 40 45 50 55 60

71MBIWA0

NOTE: 269 085 HIDDEN

2324

1

65 70 75

PLOT Of SValfTIMF11261

0.6 *

I0.11

0.4

0.2

I0.0

OVOIf

-0.11

-0.6

- 1.0

I

o

s

1

II

I

I

I

I

4

NOTE: 236 06S 11100EN

FIGURE 5

PLOT OF B -VALUE DIFFERENCE BY

TIME FROM 2/81 TO EARLIEST ADMINISTRATION (Tier281)

SYMBOL IS VALUE Of TTPEPR

222

I2222

22 2

2

222222

2

22

22

2

2

2

2

1

2

r2

1

1

1

1

1

1

1

1

21

1

1

1

1

I1

1

2

221

21

1

21

1

1

1

22221

21

21

22

1

1

21

21

1

21

1

21

1

1

2

1

1

2

1

2

2

1

1

1

21

1

1

1

1

1

1

1

1

1

1

1

1

1

1

1

21

1

21

2

2

1

2221

2

1

2

1

22

2

1

22

2

2

2

2

15 20 25 30 35 40

TIMFR26I

45 50 55 60 65 70 75

26

FIGURE 6.

PLOT OF B-VALUE DIFFERENCE BY

12PEPR1

KW OF 91/0171114711291IVON

0.9 4

0.11

0.7

TIME FROM 2/a1 TO EARLIEST ADMIPTSTRATION (TlaFr281)

REAL-REAL ADMINISTRATIONS (TypePrI)

UMW A 1 OBS. 9 2 093. UC.

A

AA

0.11 4

A

A

0.9

la

A A

8 A A

0.4A

IA AA

A A

0A

0.2A

AAA 8

A E .

E

Ce

0.2 C

AA

aa c

A

A 0 0

0.1 A a a a e

AC A 11 5

AC

8 C

0.0o 0 o 08 A F 1

0 8 8. A

-0.

IA

A cA

c A

AC

aA

A A

-0.2 AA C A

Is

8 8 A

C e

-0.3A is A

AA A

48 A A

-0.4 4A

AA

-0.9A

AA

-0.6 I I

06 4 4 4 4 4 4 4 4 4 4

6 10 15 20 25 30 35 40 AS 50 55 60 65 70 75

7111FR291

',MRS 6bPLOT OPAI-VALUE-DIFPBRENCS-BY

TIME PROM 2/81 TO EARLIEST ADMINISTRATION (T1m1r281)

REAL-TRYOUT ADMINISTRATIONS (rypePt...2)

TYPEPP2

PLO, Of 8VOIP0TINT1128i LEGEND: A 1 On. 8 2 OM ETC.

0.9

1

0.4 4

I

0.4 T

1 IS

0.2 A

I

88

A

0.0 1 A

OVOIF

0A

A

C

-0.21 C

I

e

0A

-0.4 4 A

I

A

A

AA

-OA4

A

I.

A AA A

A AA

A8 A

A A88 A

AA 8

A

A AA

A A

-0.9

-1.0

2 9

AA

8A

A AA

A A

A A

A6 A

6 A

6 A

A C A

AA A 9

O S A

A F 0 A

A 6A C A

A. 0A

8A A

A

A A

A A

A A

1A A

II

16 18 20 21 24 26 28 30 32 34 36 38 40 42 44 46 48 SO 52 54 56 58 60 62 64

TIMFR28I

30