the relationship of test performance to isdp rating in organic chemistry texts

The Relationship of Test Performance to ISDP Rating in Organic Chemistry Texts

Soo-Young Choi Instructional Science Department Brigham Young University Provo, UT84602

M. David Merrill University of Southern California Los Angeles, CA 90007

Ed Callahan Courseware, Inc. 10075 Carroll Canyon Road San Diego, CA 93131

Richard T. Hawkins Chemistry Department Brigham Young University Provo, UT84602

Robert F. Norton Center of Evaluation and Management Kuwait University Box 5969 Kuwait

Organic Chemistry Subcommittee. The test has 70 multiple-choice items.

Prior to this study the Chemistry De- partment at BYU had selected a textbook based upon student performance (ACS organic chemistry test) and intui- tion; however, no empirical study such as the one described in this article had been conducted to assist with the selection process. Therefore, this study was designed t o find out which textbook is "better" from an instructional science perspective and to provide data bearing on the Chemistry Department's textbook selection decision.

Merrill, Olsen, and Coldeway (1976) have developed an analytic tool to evaluate instructional materials. This tool is called the Instructional Strategy Diag- nostic Profile (ISDP) (Note 1). The ISDP is an instrument to judge the consistency and adequacy of a content/task objective, presentation, and evaluation.

quacy. Consistency must be determined before adequacy can be assessed.

The consistency criterion is met if the instructional objectives, test items, and the instructional presentation are consistent. Determining consistency is ac- complished in two steps. First, the instructional objectives and test items are classified on two dimensions: (a) the performance, or task level, required of the student and (b) the type of instructional content. These two dimensions then are combined to form a task/content classification matrix, which is used to classify objectives, test items, and instructional presentation. This matrix is illustrated in Figure 1. If an objective and its corresponding test item can be classified in the same cell of the matrix, they are considered to be consistent. The second step involved rating the consistency between instruction and objective/test items. The ISDP requires that

Introduction

The Chemistry Department at Brigham Young University (BYU) has used two different textbooks for the beginning organic chemistry course. The two textbooks are Introduction to Organic Chemistry by Streitwieser and Heath- cock (1976) and Organic Chemistry by Morrison and Boyd (1973). The two different textbooks were used to determine the most appropriate textbook for the organic chemistry course.

For the evaluation of students' performance, a national standardized organic chemistry test was used. Stu- dents' scores on the test represented student competency in this area. Therefore, this study used Form 1974 American Chemical Security (ACS) Cooperative Exam in organic chemistry, prepared by the Examination Committee of the Divi- sion of Chemical Education of the American Chemical Security through its

" this s tudy w a s des igned to f ind out w h i c h t ex tbook is 'better' f rom an instruct ional sc ience perspect ive . . . . "

O v e r v i e w of the I n s t r u c t i o n a l Strategy Diagnost ic Profile 1

The ISDP is an analytic tool, in the form of a set of prescriptions, that facili- tates the evaluation and revision of existing instruction and the design of new instruction. The ISDP is designed to evaluate instructional materials on two main criteria: consistency and ade-

1This information is taken verbatim from a report by M. D. Merrill, R. E. Richards, R. V. Schmidt, and N. D. Wood entitled Empiri- cal Validation of Selected Instructional Strategy Diagnostic Profile Prescriptions (NPRDC Tech. Rep. 77-43, September 1977) pages I through 5.

different components of instructional presentation, called primary presentation forms, be present for different com- binations of task level and content type. If the combination of primary presentation forms required for the task level and content type of each objective/ test item is present, then the instruction is consistent with the objective/test items.

Once it has been determined that instructional materials are consistent, the adequacy criterion is assessed. This is done by determining whether or not the instructional presentation adequately communicates the "to-be-learned" information. Variables that are hypothesized as affecting instructional ade-

16 JOURNAL OF INSTRUCTIONAL DEVELOPMENT

Use

Remember Paraphrase Generalities

Remember Paraphrase Examples

Remember Verbatim Generalities

Remember Verbatim Examples

i iii iiil ii! iiij ii ii !iii! i !ii!ii!iiiiiiiiiiiiiiiiii!l I [!i!!!i!iiiiiiiiliiii iiiiiiiiiiiii i i ! i!iiiili

[.::::;: :-: : : ::: :-:.:.:-:-: : : :,:.:.:.:.:.:.1 ]

�9 i:i:i:~:~:i:~:]:]:]:i:i:~:~:~:~:!:!i!~i~]i~i- i:]:i:-~:~:i:i:i:!:i?~:]:]:::::i:~':~:]?~:!

�9 : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

iiiii.ii.iiiiiiiiii.iiiiiiiill I iiiiii!i!iii!ii!ii!iiiiiiiiiiiiiiiiiiiiiiiii!iii!, ,

Iiiiiiiiiiiiiiii Facts Concepts Procedures

Content Type

Note: Facts can be tested only at the Remember Verbatim level.

Principles

FIGURE 1. ISDP task/content classification matrix.

quacy during ISDP development include the following: (a) isolation (i.e., is the relevant information separated and clearly identified?), (b) help (i.e., is ex- planatory or nmemonic information provided?), and (c) matching (i.e., are the examples and practice items matched?). Thus, instruction is based on these variables to obtain an adequacy index. Each primary presentation form within the instruction may be rated as more or less adequate.

In the following sections, the task/ content matrix will be described in greater detail, the primary presentation forms will be discussed, and validation experiments testing the consistency and adequacy assumptions will be presented�9

Task/Content Classification Matrix As shown in Figure 1, the task dimen-

sion of the task/content classification matrix is comprised of several levels, the broadest of which consists of the strate-

gies Use and Remember. Use is defined as the act of applying a general relationship to a specific situation where it has not been previously applied, and remember, as the act of bringing to mind something that has been previously encountered. Thus, a use item (or objective) would require the student to respond by applying a generality to a newly encountered example; that is, one that has not been previously dis- played to the student as part of the instructional presentation. A remember test item (or objective) would require the student to respond by recognizing or recalling a generality or example that has been previously encountered. Gen- erality i s defined as a statement of definition or relationship that can be applied to more than one specific object or event; and example, as a specific object or event or its representation that does or could exist in the real world.

The use level cannot be divided into sublevels--it always requires newly en-

countered examples. The reason for this is obvious--if an example had been previously encountered, the test item or objective would be classified at the remember level. The remember level, however, can be divided into sublevels, either paraphrase or verbatim. Paraphrase means equivalent in meaning but ex- pressed in other words; verbatim, word-for-word or exactly the same. Thus, a paraphrase generality means that synonyms have been substituted for the substantive words (nouns, verbs, and modifiers) of the original state- ments; and paraphrase example means that the same object or event is presented but that the form or representation used to exhibit this object or event has been modified. A verbatim generality~example requires the student to recognize or restate the same words that were used previously to present the generality/example. All paraphrase and verbatim generalities and examples have been previously encountered by the student.

As shown in Figure 1, the content dimension of the matrix involves four mutually exclusive content categories: facts, concepts, procedures, and principles. Except for facts, for which there can be no generalities, all types can be tested at any of the task levels. These categories are defined as follows:

A fact is a one-to-one association of a symbol and a specific object or event.

A concept is a class of objects, events, or symbols that'(a) share critical at- tributes, (b) can be referenced by a n~me or symbol, and (c) have discriminately different individual members.

A procedure is a series of steps required to produce an example of an out- come class. Each step may involve the temporal or spatial ordering of specific objects, events, or symbols or a branch- ing decision, based either on a fact or the classification of an example of a concept. A procedure is often characterized as "how to do something."

A principle is a predictive relationship between specific examples of a concept, or among a set of related concepts, which explains why an example of a particular class is produced as aresult of a particular manipulation.

Further information concerning these content categories is provided in Merrill (Note 1), which was designed to teach users how to classify test items and objectives according to the ISDP task/ content classification matrix. (It should be noted that some of the variables

FALL 1979, VOL. 3, NO. I 17

identified as affecting instructional adequacy are not applicable to all content types and task levels.)

Primary Presentation Forms The ISDP defines the instructional

presentation form or display as the fundamental unit of instructional strategy. As indicated previously, the instructional presentation forms must meet consistency and adequacy require- ments.

Four primary presentation forms or displays, which represent the various

Introduction to Organic Chemistry (by Streitwieser and Heathcock) and two of the classes used the Organic Chemistry text (by Morrison and Boyd).

Materials The materials of this study consist of

two organic chemistry texts used at BYU: Introduction to Organic Chem- istry and Organic Chemistry mentioned in the preceding paragraph.

Design Student posttest scores were obtained

" . . . the text with the higher ISDP rat ings . . , also produced higher student performance."

ways that information can be presented, have been defined (Merrill, Note 1) :

1. Tell via generality (TG)--A display that presents a definition of a concept, an algorithm that describes a procedure, or a proposition that expresses a principle.

2, Tel via example (Teg)--A display that illustrates how a generality applies to a specific example.

3. Question via generality (QG)-- Not used in this research study.

4. Question via example (Qeg)-- A display that presents an example and requires the student to respond to the example or it presents a name or generality and requires the student to respond by providing an example.

on the National Standard Organic Chemistry test, the 1974 ACS Coopera- tive Exam in organic chemistry. A discrimination analysis was conducted on each item of the test comparing test item scores for those students who used the Introduction to Organic Chemistry text with test item scores for those students who used Organic Chemistry. 2 A discrimination index was arrived at using the following formula : D = (number of incorrect responses for each textbook)/ (total number of respondents). Specific test items were selected for analysis if they had a percentage difference greater than 11 percent between the discrimination indices. A list of 18 items of the

step-by-step procedure used:

Step 1. Task level of item was determined.

Step 2. Content type of item was determined.

Step 3. The correct ISDP form was selected for the item dependent upon the task/content of the item. (See Figures 2 to 5.)

Step 4. Both texts were surveyed for lo- cation of presentation of information relevant to test item.

Step 5. Presentation-test consistency was determined (i.e., presence or ab- sence and number of prescribed components for each test item).

Step 6. Presentation adequacy was determined (i.e., degree to which presentation components meet ISDP criteria).

When all of the selection posttest items were profiled according to the ISDP, a cumulative rating of overall text consistency and overall text adequacy was computed for each item.

Statistical analyses were conducted on the data to address the stated hypotheses of the study. The following are the specific analyses conducted:

1. A t test to compare ISDP consistency and adequacy cumulative ratings of the two texts.

2. A t test to compare posttest scores of the students using the two texts.

3. A linear regression of ISDP cumulative ratings on posttest scores.

4. Another comparison was made be-

H y p o t h e s e s

The hypotheses of this study are: 1. There is a significant difference be-

tween ISDP ratings for the two texts. 2. There is a significant difference on

test performance between students using the two different texts.

3. There is a significant linear relationship between ISDP ratings and test performance of the two texts.

4. There is a significant difference between the text selected by the Chemistry Department and the text with the lower ISDP rating and lower performance.

M e t h o d s

Subjects Four undergraduate organic chem-

istry classes at BYU were selected for this study. Two of the classes used

" . . . we suggest that the present materials be reassessed using the complexity of performance task

t l measure . . . .

total possible 70 items of the posttest were selected for analysis. (The test items were numbers 2, 13, 17, 18, 27, 29, 33, 40, 43, 48, 54, 56, 57, 62, 66, 68, and 69 of the Form 1974 ACS Coop- erative Exam in organic chemistry.)

An ISDP was determined for each posttest item. A systematic procedure was followed for the determination of each item's profile. The following is the

2This discrimination analysis was orig- inally conducted using data from all available sources using these texts.

tween the text selection of the Chemistry Department with the findings from the ISDP.

Results

Hypothesis I stated: "There is a significant difference between ISDP ratings for the two texts." The results of this study support the above hypothesis. The Morrison and Boyd text received higher ISDP ratings than the Streit- wieser and Heathcock text (t = 3.104,

1 8 JOURNAL OF INSTRUCTIONAL DEVELOPMENT

I i

Os~ECr~vES" iii i

OBJECTIVE N(3. _ TIC Level of ~biectlve?

F C Pro Prt H n ,~

ti li . . - iiiiii!i ;i!}',ii!i}iiii! ',ii~i:.ii',~',::',::i:,;

,~ . iiiiiii:i!ili!it;;i?!i:ii!ii::: iii;i',i::i;:i;ii;i

Talk Lmm~ of Course?

I Object ive

n q

OBJECTIVE N ~ T/C Levet ot Objective?

F C Pro Prt i =

~176 " ii~ !i: ;!i ! li',;i~ii',; iiiiiii' apG i]::i]::]]]]i]],]

, . [ ] [ ] ' " "] '" " ] " "f!" " l ' , " ' ! ' ; ' j iiii ,'i'iiill li!ili;',iiiiiili ii Taadr i ot Course?

I

UGeg Component Consistency PRESEI~TATION ...... EVALUATION

i , i i n i

LOC. OF PRES.

TG T,,MI O~,q odi

oo~1 1 I ~ a "rQ ~'qq a~g a ~

I 1 I , , l l pI1GIILEMIi

~ O Cons is ten~ J~

~3 = L O C OF P R E S .

TQ T~ ~ QG

o"-,I I f ~ t*of-l:.-i=l

PIIQILJEMS

TEST ITEM NO. F C

i R ~

T-O Consistency ~_~

TEST ITEM NO_

a ~

Rpe~l

T-O Consistency ~-~

TE~-i" ITEM NO.

F C

Rp~

R ~

TEST ITEM NO.

F C

RpG ' t R~Q

Rr Rv~

T-Q Con~stency C

TEST ITEM NO.

F C ~ ~t UG=~

RvG

Rp~

R,I,~I

ii i

T-Q CQnzistoncy

TEST ITEM N O . .

uG.j

Rc~

RvG

Rr

R~qi

i m i i i i

T-O Consistency ~ T-Q C~m|lstency

TEST ITEM NO....._

F

a ~

Rv~l

TEST ITEM NO.

Iii R~aNI , ~

obj.c,,v. J..,,,,-~l I i~o co..,.,..,3 = I ~Qp f "~m Se -a Com 'z t~a re me. ~

FIGURE 2. Sample ISDP form: UGeg component consistency.

FALL 1979, VOL. 3, NO. 1 19

, m , , , , , m i r a , I

Presentat ion

UGeg Adequacy Questions

:I.TG (Tell v ia G e n e r a l i t y )

A d e q u a c y for Use

u.sns!scenc" ~

~Is there at lease one TG dispJay' I i f :here are ~o TG'~, no zo

tile T=Q ~=czion.'I Isolar

Is each TG separate?

is each TG i den t i f i ed?

Hel'o

Is one of the TG'~ an

!s ~ mnemonic included?

O3J. NO. T~ST ITEM NOS.

IZ] F ] I l l

UGeg [.

P R O F I L E

TG Tog QecJ QG

I N D E X [

P R O B L E M S

.~z~..,~,,~ F G I N D E X ~'.~.~;.. :.~Y~ �9 ,,. ,~..4L4~.'r :~ I

i lI.Teg(Tell v IB E x a m p l e ) P R O F I L E I N D E X i

Cons ~ ~t.enc "_-.,i Is there at leas: ~r.e Te~ display? L_I ( I f n o t , go to gag sect lon. )

[sc.lazton ~_ ~---j~] !

Are "-he Tec.'s s~o.?r.a.'..?~ . . . . . . . . . . . . . .

Are the Teg's identified?

He~o @'7 Are *..he Tag's acco:oanied by help? . . . . . . . . . . F - - - l Does at leas': o.~ Te~ use a I I si.~pl, f led :'::. 'esenta Zion?

,~atch~ no

Are ~o~t T.~g's notched?

i Do Tag's represent a range o i l ~ " I . . . . . . . . . . . .~ 1 ~ ' . , . t r a .~ . . . . . . . . . . . . . . : . . . . . . . . . , ~,re TeD's ~i ' ;ercent ~)n I " var iable a t t r ibu tes?

[ T ~ ~No~x ! ~ , ......... ~,.~I,:~ . J ~ ' E =

!11. Oeg(o~,t~o.~i~ s=~=p~ PROFILE INDEX

cohsisz:.c~ ' ~ l l IS there at leas: one 0ea display?

[so. i a z l o ~

~.re ~.lle fee.,~.b(:c~ { l S ; ) / a y $ II �9 : c, er:~:1~ ? |

LOC. OF P~ES.

CONTENT

I

I i

P R O B L E M S

i i

PROBLEMS

Are ~,e qee's v o i c Of he!p? r-----l'""" . . . . . . . . . . . . . . . . ~... . . . . . . . . . . . . . . . . . . . i i :~o the fee~oack Jisplays

inc)ude help? ~.a tchin&

Are the Qeg's randomll sequenced on ;~atchivq, d~ffJc,:~t?, etc.? - - _

Sa.~o I i n: ~-h.,r .--~,- DO :he QegLs represent" a range Le.:_,uU~ . . . . . . . . . 9f. di . ; f !cu1:v? ~ - - ~

Are t~e ~eg~s "~l{~'e'~-'~ 66 " variat, i e a t t r ibu{es?

O e g I N D E X - ~ ~ ~ ~ : ~ , ~ . . , OVERALL INDEX ,:.;.::~ - ' ~ t "

. . . . . . . . !,.',;.;~-',

G e n e r a l i t y E - : a m p l e

O~J. NO. TE-$I- HE:A Nr'jS. LOC. OF P~EL;.

F] I I I ]

UGeg

TG Tag Oeg QG CONTENT

m l . l

i PROF1LE I N D E X

D PROBLEMS

P R O F I L E I I N D E X

@.~

......... F

-- L

........... ~~-~

PRO.=J LE .~,t S

1

.......... i

.......... i

~ : . - ' : ; t ~ - ~ 2 ~ ! = L~,..'_-z. . ..-3 - 2 ~ 1 i v !

PROBI.EMS

FIGURE 3. S a m p l e ISDP f o r m : P r e s e n t a t i o n a d e q u a c y for use generality example .

2 0 JOURNAL OF INSTRUCTIONAL DEVELOPMENT

I ii

O B J E C T I V E S

OBJECTIVE N O . ~ T/C Leve l o f O b j e c t i v e ?

F C Pm P~t

: : : ' : : 1 : :2 : : : : 2 : - : . : - : . 2 "2 " : - : - i

::::::::::::::::::::::::::::::::::::::::::::::: ~ G

Rpeg

Task Levet o f C o u r s e ?

O ble c t l ve J u s t i f i e d ? ~ , ii i

OBJECTIVE N C ) . ~ T/C Lmml o f O b j e c t i v e ?

F C Pro Prt

U G e g " " " : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : :

"'"""" :': "'""""' :': ~:::i:i:" "~" " �9 . u . . . . . . . . . . / r . ' . . . ' / . - . . . . . 1 . : . ' . ' . . . . . :

RpG " " ' " ' " " "'"''"'"". ' " " '"'""" " '.-. .- ... ' " " " "" : : ~:::':::: ~:'~'::::""' I

R v G ' " ' " ' ' " * " "'" '" '""" ' j "" "'"" ::::::::: ::::: |::::::::::::::: :::i:::::::i:i:!:l

ii!iiii!!!!!i!ii: ii!:! :i'i!:!" :!:!:;:i::::'!::

Task Leve l o f C o u r s e ?

ii

o, , . r Justi,,ed, i ~

i i i i

E V A L U A T I O N , i i i =

TEST ITEM N O . ~ TEST ITEM N O . ~ F C I=m ~

ROG ~ G

i

i ,-O~=,,sten~ ! i i I

TEST ITEM N O . ~ TEST ITEM N O . ~

UG~i UGec]

RvG RvG

R~IIj Rpeg

i

T-O C o n s i s t e n c y , ~

T E S T I T E M N O . ~

UG4g

Rr 1

RV~l

T-O C o n s i s t e n c y i__.

TEST ITEM N O . ~

u ~ u=.q I I i J

R,a ~a I I i 1

! i I

I " ~ 1 7 6 1 7 6 1 7 6 1 7 6 7 .1 ~-~ c ~ , , , , , n ~ , 1 .... i. ~= ]0 tD r -~m ~ l l - I C o u r l m :~ . ~ 1T~

ii i i i i i i i i

Rpecj C o m p o n e n t C o n s i s t e n c y iii i i i i i

P R E S E N T A T I O N

LOC. OF PRES.

TG l"eq Qeq QG

[ ~ I I I R I C T 1"~i rQq Q ~ QG

I J ( I t PROBLE~

O C o n s i s t e n c y T-O C o n s i s t e n c y ],) -

i 1 ~ i I

LOC. OF PRES. TEST ITEM NC}.__

RvG

I l v ~ [--"

TG T4q Cleq QG

, 0 - l i I ~=! ,-ocon=-t-~ r-] m r.= a~ as TEST ITEM NO.__

[ I i I I , ~ a L E M S

P'O Conslstsncy/7 = J N ~

FIGURE 4. Sample ISDP form: Rpeg component consistency.

FALL 1979, VOL 3, NO. I 2 1

P r e s e n t a t i o n

R p e g

A d e q u a c y

Q u e s t i o n s

Adequacy OdJ. rao. TEST !I'EY~ NOS.

[i"ii" I l J

I N D E X

~ ~,,~ r ,~ ~ : ~ . )

. ' ~ T h r~,, , ~ , ~ : : : - - . ~ ~'.&-~t~e-J,z

~-_ip. ,:,re "[~ r ~ = r , - ~ , . , , ~ . = . ~ : . ~ i ~ ,

. . . . . . . . . . . . . . ~.~:.~e: . . . . . . . . . . . . . . . .

$r t fle-=/

~ 1 I~['

Are .--..'=. res~tal:1~a= a t f,,~ Ta~I r ,~,r3~ t ;

Teg INDEX

I I . Q e g <c~.,~:;o,, , , ~ ~ . . , , . o, .~

�9 .~.~...~ ~.::-~:.'.~:. :'~'.....'7.~L ,..~ .................................. ..~.~. ~:. ~':..~....':~.~L

I I I I I

IZ]

�9 i j I

,!

I t I

PcK3F;LE I N D E X /

, ] , , - ~ ,

! i

~lTi,i

T G ,"=~1 O ' * ~ O G

R o , ~ q ~ i i l " -

i

for Remember Parsphrase E x a m p l e "LGC, OF ~qE3 . G;JL %O. TE~T :T~,.~ "~C,S. LC.,C- G~: ~NES. !

K] F ~ .-Vi CONTENT TG reo ~ e q 0_.,~ C~NTENT

R p �9 ~ ":':'"">

. . . . . i _ :-:'::': . . . . - - 1

J .... 1

I

�9162

l i i

j:::::::::il]

. . . . . . . . . . . l i Are ~ re=,~,~w.,~ ~..-.~,--~.*

�9 ,, ,

................................ ! 1

!1 i I i

. . . . . . . . . . .

I N O E X ~;'; ..s :t - 1 - ~ , - _[ _ i

O V E R A L L I N D E X

P~OOLE~S

r

' . ' U

' ~ ' ' ~ 1 7 6 1 7 6 1 7 6 1 7 6 1 7 6

P R O B L E M S

> .,.',.'-~ ~.~4 f"~ ~ -- i |

FIGURE 5. Sample ISDP form: Presentat ion adequacy for remember paraphrase example.


TABLE 1. ISDP ratings on the 18 selected test items

Morr i son & Boyd text Streitwieser & Heathcock Text ACS ISDP index ISDP Index test item TG Teg Qeg OI TG Teg Qeg OI

2 1.00 .40 .00 .38 0 .20 0 .08 13 .67 .40 .00 .31 .33 0 0 .08 17 .33 .80 .80 .69 .33 0 0 .08 18 a 27 .67 .80 0 .46 .33 .40 .20 .31 29 .67 1.00 0 .54 .33 0 0 .08 33 .33 .80 0 .38 .67 .60 .40 .54 40 .67 0 .40 .31 0 .60 0 .23 43 1.00 .60 0 .46 .67 .60 .49 .54 48 .33 0 0 .08 .33 0 0 .08 54 .33 .60 0 .31 .67 .20 0 .23 56 .67 .60 .40 .54 .67 .20 0 .15 57 .33 .60 0 .31 .67 .40 0 .31 59 1.00 .60 0 .46 0 .40 0 .15 62 .67 0 0 .15 .33 .60 0 .31 66 .67 0 0 .15 0 0 0 0 68 . 6 7 .80 .60 .69 .33 .40 0 .23 69 .33 .60 .20 .38 .33 .40 0 .23

Note. For the Morrison & Boyd text, ~ = .3882 and SD = .0432. For the Streitwieser & Heath- cock text, ~ ~ .2135 and SD = .0388. TG = tell via generality. Teg = tell via example. Qeg = question via example. OI = overall index.

aItem number 18 has no ISDP ratings reported because no presentation could be found in either text on the content of the item.

TABLE 2. ACS organic chemistry test performance

Teacher I

Teacher 2

Morr i son & Boyd Streitwieser & Heathcock Text Text

Winter 1978 Winter 1977

= 46.0 ~ = 46.42 SD = 1.04 SD = 1.46

n = 77 n = 52

Winte r 1978 Winter 1977

= 47.5 ~ = 43.4 SD = 1.12 SD = 1.36

n = 74 n = 49

Note. t : 1.374, p < .10. For the Morrison & Boyd text, mean : 46.66 and SD : .760. For the Streitwieser & Heathcock text, mean = 44.96 and SD = 1.000.

p ~ .005); the respective means are .3882 and .2135. 3 (See Table 1.)

Hypothesis 2 s tated: "There is a significant difference on test performance between students using the two different texts." The results of this s tudy show partial suppor t for the above hypothesis. Students using the Morr i son and Boyd text (1973) had higher test per-

3The ISDP ratings are based upon 0.00 to 1.00 scale.

formance than the students using the Streitwieser and Heathcock text (1976) on the 70-item posttest (t = 1.3745, p <~ .10); the means respectively are 46.66 and 44.96. (See Table 2.) Also noted were marginal differences between the two groups on the 18 selected items (t = 1.52, p ~ .10); the means respectively are 61.25 and 55.08. (See Table 3.)

Hypothesis 3 s tated: "There is a significant linear relat ionship between

ISDP ratings and test performance of the two texts." The above hypothesis was not supported by the results of the study (t = -0.057, p ~ .10) . The regression equat ion was y = 59.99 - 6.852x. See Figure 6 for a plot of the data.

Hypothesis 4 stated: "There is a significant difference between the text selected by the Chemistry Depar tment and the text wi th the lower ISDP rating and lower performance." The Chemis- try Depar tment selected the Morr i son and Boyd text. The results of this study tend to suppor t the above hypothesis. Students given the Morr i son and Boyd text outperformed overall the students given the Streitwieser and Heathcock text and the Morr ison and Boyd text had higher ISDP ratings than the Streit- wieser and Hea thcock tex t.

D i s c u s s i o n

The Morr i son and Boyd text (1973) received higher ISDP ratings than the Streitwieser and Heathcock text (1976), and a l though student performance was only marginal ly significant, it was in the expected direction. Tha t is, the text with the higher ISDP ratings, the Morr i son and Boyd text, also produced higher s tudent performance. O the r available data s imilar to the data of this s tudy were analyzed. A separate analysis was conducted on this data because the authors felt that the subject populat ions were characteristically different. The results of this analysis fully suppor t the second hypothesis (t -- 2.137, p~ .025) , wi th the results in the expected direction. The Morr i son and Boyd text group outperformed the Streitwieser and Heathcock group. The respective means were 48.26 and 43.44.

Al though a linear relat ionship between posttest performance and ISDP rat ing was hypothesized, this par t icular research s tudy found no such relationship. Merrill , in previous research (Note 2), has found that wi th similar content a l inear relat ionship may not exist. In par- t i tular , the subject mat ter Merril l in- vest igated contained test items of con- siderable complexity that the ISDP rating form possibly could not handle. The ISDP forms are designed to index one- to-one correspondence; however, cer- tain subject areas do not fol low a one- to-one correspondence between presenta t ion and test items. Merril l has found a course in physics to be in this category (Note 3). Several of the items in the Organic Chemistry Exam were of

FALL 1979, VOL. 3, NO. 1 23

TABLE 3. ISDP ratings and test performance in organic chemistry course on the selected 18 test items

ACS test

i tem

Morr i son & Boyd text ISDP index a

Streitwieser & Heathcock text ISDP index a

TG Teg Qeg OI TG Teg Qeg OI

Number of Correct Responses & Percentb

Teacher I Teacher 2

Winter Winter Winter Winter 1978 1977 1978 1977

#c % # a % #~ % #f %

2 1.00 .40 .00" .38 0 .20 0 .08 13 .67 .40 .00 .31 .33 0 0 .08 17 .33 .80 .80 .69 .33 0 0 .08 18 27 .67 .80 0 .46 .33 .40 .20 .31 29 .67 1.00 0 .54 .33 0 0 .08 33 .33 .80 0 .38 .67 .60 .40 .54 40 .67 0 .40 .31 0 .60 0 .23 43 1.00 .60 0 .46 .67 .60 .49 .54 48 .33 0 0 .08 .33 0 0 .08 54 .33 .60 0 .31 .67 .20 0 .23 56 .67 .60 .40 .54 .67 .20 0 .15 57 .33 .60 0 .31 .67 .40 0 .31 59 1.00 .60 0 .46 0 .40 0 .15 62 .67 0 0 .15 .33 .60 0 .31 66 .67 0 0 .15 0 0 0 0 68 .67 .80 .60 .69 .33 .40 0 .23 69 .33 .60 .20 .38 .33 .40 0 .23

68 88 47 90 69 93 43 88 43 56 38 73 53 72 29 59 50 65 25 48 55 74 23 47 60 78 30 58 43 58 27 55 30 39 17 33 24 32 15 32 40 52 17 33 36 49 21 43 39 51 15 29 28 38 16 33 59 77 38 73 65 88 40 82 46 60 31 60 59 80 27 55 54 70 28 54 48 65 27 55 37 48 28 54 48 65 10 20 22 29 28 54 35 47 25 51 56 73 29 56 59 80 25 51 63 8 2 36 69 52 70 36 73 50 65 36 69 36 49 30 61 24 31 27 52 27 36 20 41 33 43 33 63 30 41 27- 55 64 83 35 67 68 78 23 47

= .3882 ~ = .2135 SD --- .0432 SD = .0388

= 6 0 ~ = 57 ,~ = 6 2 ~ = 53 S D = 4 . 3 9 S D = 3 . 7 7 S D = 4 . 5 7 S D = 4 . 1 1

Note. For the Morrison & Boyd text, ~ = 61.25. For the Streitwieser & Heathcock text, ~ = 55.08. TG Qeg = question via example. OI = overall index.

at = 3.104, p <~ .005. dpossible number of correct responses = 52.

bt = 1.52, p < .10. epossible number of correct responses = 74.

Cpossible number of correct responses = 77. fPossible number of correct responses = 49.

= tell via generality. Teg = tell via example.

fhe same type as described by Merrill . Apparent ly the approach taken in this

s tudy can be helpful in textbook selection. The text that had higher ISDP rating and higher posttest s tudent performance was also the text selected by the Chemist ry Depar tment .

R e c o m m e n d a t i o n s

Because,a l inear relat ionship was not found wi th this data and because the authors feel that the ISDP form m ay be limited to one-to-one correspondence type materials, it is recommended that fur ther analysis be conducted on those test items from this s tudy which are one- to-one correspondence types. The expectat ion is that the analysis will find a significant linear relat ionship for those items which have a one-to-one correspondence. Such research would be helpful in predicting student perform-

ance based on one-to-one correspondence type materials ISDP ratings.

O n the other hand, Wood, Richards, and Merril l (1976) have conducted research investigating complex objectives or test items similar to those found in this s tudy using ISDP rat ing forms. Wood used a complexity of performance task measure defined as the number and type of t a sk /con ten t relations inherent in individual test items. (His findings indicate that a high correlat ion exists between student performance ratings and ISDP ratings of the presenta- t ion materials.) Therefore, we suggest that the present materials be reassessed using the complexity of performance task measure employed by Wood in his study. The expectat ion is that the more complex the t a sk /con ten t relations are the higher the probabi l i ty of incorrect manipula t ion of one or more task/ content. Such research would enable developers to isolate on those test items

where problems arise to do the following: (a) Identify i tems that should be simplified or b roken down to smaller components of evaluat ion, if possible, (b) conduct " rear-end" analysis for diagnostic purposes, and (c) provide in- format ion as to wha t needs to be taught and included in the instruct ional materials.

The authors r ecommend other instructional mater ia ls be likewise stud- ied, especially when material selection decisions are to be made. Also, research on teacher effects (i.e., supplemental materials, specific emphasis , l abora to ry experiences, etc.), wh ich were not con- trolled in this present s tudy need to be addressed.

The authors suggest that all instructional materials be rated according to the ISDP and that areas of low rat ings receive deve lopment th rough a joint contr ibut ion f rom conten t experts and ISDP experts.


Test Performance

100.

80.-

60. *

40. *

20."

2 .... * *

* : e

2-

2 3 2 *

2 - - * * 2

* 2 *

I I I I 1 0.00 0.15 0.30 0.45 0.60 0.75

ISDP rating

FIGURE 6. Plot of the c o r r e l a t i o n o f the ISDP ratings and test performance.

Reference Notes

1. Merrill, M. D., Richards, R. E., Schmidt, R. V., & Wood, N. D. The instructional strategy diagnostic pro-

file training manual (NPRDC Spec. Rep. 77-14). San Diego: Navy Per- sonnel Research and Development Center, September 1977.

2. Merrill, M. D. Personal communica-

tion, April 1979.

3. Merrill, M. D., Schmidt, R. V., & Norton, R. F. Validation of the instructional strategy diagnostic profile in physics 100 (Instructional Rep. No. 64). Provo, Utah: Courseware, Inc., 1977.

References

Merrill, M. D., Olsen, J. B., & Colde- way, N. A. Research support for the instructional strategy diagnostic profile. Technical Report No. 3. Provo, Utah: Courseware, Inc., 1976.

Morrison, R. T., & Boyd, R. N. Organic Chemistry. Boston: Allyn and Bacon, Inc., 1973.

Streitwieser, A., Jr., & Heathcock, C. H. Introduction to Organic Chemis- try. Macmillan Publishing Co., Inc.,

1976. Wood, N. D., Richards, R. E., & Mer-

rill, M.D. Prediction of Student Performance on Rule Using Tasks from the Diagnosis of Instructional Strategies. Provo, Utah: Brigham Young University Research Paper, 1976.

In Search of f Graduate Degree Programs

In Instructional Technology 1978-79 I n t e r e s t e d in w h a t i n s t i t u t i o n s offer g r a d u a t e degree p r o g r a m s in i n s t r u c t i o n a l t e c h n o l o g y . . . c o u r s e r e q u i r e m e n t s . . , scholar - sh ips a n d fe l lowships?

N o w you can f ind all t h e answers in t he new, up - to -da t e pub l ica - t ion G r a d u a t e D e g r e e P r o - g r a m s In I n s t r u c t i o n a l T e c h - no logy . C o m p i l e d by J o h n W. El l ison a n d Cla ra DeFel ice of t he

S t a t e Un ive r s i ty of N e w York a t Buffalo, the book prov ides valu- able i n f o r m a t i o n for t h e prospec- tive s t u d e n t , t he counse lor , t h e AV p r o g r a m director , or a n y o n e in t e re s t ed in the s t a t u s of I T in- s t ruc t i on at t h e g r a d u a t e level.

T h e book provides desc r ip t i ons of g r adua t e p r o g r a m s a t 61 inst i - tu t ions , wi th de t a i l ed i n fo rma- t ion.

G r a d u a t e D e g r e e P r o g r a m s In I n s t r u c t i o n a l T e c h n o l o g y 1978-79 is ava i lab le from A E C T P u b l i c a t i o n Sales, Dept 8A, 1126 16th St . , N.W., W a s h i n g t o n , D.C. 4 ~ 20036. A E C T Member Price: $3.50; N o n m e m b e r Price: $4.50. Orders u n d e r $15 mus t be p repa id . A $1 charge plus pos tage will be added to bil led orders.

FALL 1979, VOL. 3, NO, 1 25

the relationship of test performance to isdp rating in organic chemistry texts

Documents