ed 367 684 tm 021 134 author rothman, robert title 21p. · pdf filescale programs;...

ED 367 684

AUTHORTITLE

INSTITUTION

SPONS AGENCY

PUB DATECONTRACTNOTEPUB TYPE

EDRS PRICEDESCRIPTORS

IDENTIFIERS

ABSTRACT

DOCUMENT RESUME

TM 021 134

Rothman, RobertAssessment Questions: Equity Answers. Proceedings ofthe 1993 CRESST Conference (Los Angeles, California,September 12-14, 1993). Evaluation Comment.California Univ., Los Angeles. Center for the Studyof Evaluation.; Center for Research on Evaluation,Standards, and Student Testing, Los Angeles, CA.Office of Educational Research and Improvement (ED),Washington, DC.94R117G1002721p.

Collected Works - Conference Proceedings (021)

MF01/PC01 Plus Postage.Access to Education; *Conferences; CostEffectiveness; Cultural Awareness; CulturalDifferences; Definitions; *Educational Assessment;Educational Change; Educational Objectives;Elementary Secondary Education; *Equal Education;Outcomes of Education; Portfolios (BackgroundMaterials); Racial Differences; *Standards; *TestBias; Test Construction; Testing; Testing Programs;Test InterpretationAlternative Assessment; *Authentic Assessment; LargeScale Programs; *Performance Based Evaluation; ReformEfforts

Focusing on one of the critical questions in theshift to new forms of assessment, researchers, policymakers, andteachers met at the 1993 Cent'er for Research on Evaluation,Standards, and Student Testing (CRESST) to consider equity inassessment. The opening remarks of CRESST co-director Robert Linnstressed that equity is at the center of debates over assessment andnew standards. Synopses of the remarks of a number of speakers aregrouped under the following headings; (1) the definition of equity;(2) the evaluation of the fairness of assessments; (3) data fromlarge-scale assessment programs; (4) costs of performance assessment;(5) equity and assessment design; (6) portfolios; (7) groupassessment; (8) equity and the interpretation of assessment results;(9) equity and the research agenda; and (10) reports from workinggroups on the design of equity-sensitive performance assessmenta. Theconcluding remarks of Adam Urbanski of the Rochester (New York)Teachers Association pointed out that reforming schools is a long anddifficult process that must involve the entire community. A list ofCRESST and Center for the Study of Evaluation reports available isincluded. (SLD)

***********************************************************************

Reproductions supplied by EDRS are the best that can be madefrom the original document.

*****w*****************************************************************

. titer tor thc 'Ninth ot liltiitoii kt<entcr tor (uNC.Irill (111 1. .1111.111(M, lIdt.:11I I L'SCI1114 1.

EVALUATION COMMENT=NM

U.& DEPARTMENT OF EDUCATIONOnce of Educat+onar Researcer, and Improvement

ED 1C ATIONAL RESOURCES INFORMATIONCENTER tERICI

Ttus dOcument has bean reproduced asrecopied from the Parson or orgaruzatronongrnseng a

C MIor changes nave been made to 'reprovereproductron gummy

Peonts of we or oo"Ont, stated . fhsdOCument do not necessanty represent officrarC)ERI posoron or poCy

tt

Assessment Questions: Equity AnswersProceedings of the 1993 CRESST Conference

Robert Rothman, CRESST./UCLA

Zeroing in on one ofthe mostcritical issues in the shift to

new forms of assessment, morethan 200 researchers, policy niak-ers, and teachers gathered at theUCLA campus September 12 -14,1993 to discuss questions sur-rounding equity.

Meeting at the 1993 CRESSTconference, entitled "AssessmentQuestions: Equity Answers," par-ticipants outlined many of theconcerns associated with the topicand offered _some possible solu-tions.

CRESST Criteria of FairnessCRESST Co-director Robert

Linn said in opening remarks thatequity is at the center of debatesover standards and assessments.Fairness is one of the most critical

of the criteria developed byCRESST to evaluate new assess-ments, he noted. At the same

time, he said, the report of theNational Council on EducationStandards and Testing and theClinton Administration's Goals2000 legislation have put equityat the top of the agenda in thefederal and state governments.

"States and the national effortare focusing heavily on trying toestablish ambitious content stan-dards," Linn said. "And assess-ment has been central to all thework put forward in curriculumreform. This has led to demandsfor high standards of student per-formance assessed by new assess-

ments congruent with the con-tent standards."

"If you put thc two together,"Linn said, "that leads immedi-ately to concerns about equity:what it means to give students afair shot [at meeting the stan-dards], especially if there are highstakes attached."

BEST COY AVAILABLE 2

Winter 1994"PERMISSION TO REPRODUCE THISMATERIAL HAS BEEN GRANTED BYj: C. th-L-4

TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)."

Dean Theodore R. Mitchell ofthe UCLA Graduate School ofEducation said ths. current de-bate represents a "unique histori-cal moment." "For the first time,"he said, "Americans are discuss-ing both equiry and excellence,taking into account both inputsand outcomes. We are at thethreshold of a breakthrough,"Dean Mitchell said.

But CRESST Associate Direc-tor Joan Herman cautioned thatresearchers do not yet have all the

See pages 13-16 for de-scriptions of eight newCRESST/CSE technicalreports.


answers to enable us to walkthrough that door. "Most don'tagree on what the central ques-tions are," she said.

Defining EquityWhile focusing their attention

on the role of assessment, manyagreed that the issue of equityinvolves the education system asa whole. "If we can adequatelyteach all youngsters, we don'tneed to be as concerned withequity," said Edmund W. Gor-don, CRESST/City Universityof New York and Yale University.

"However," Gordon added,"that fact does not let those in-volved with assessments of: thehook. Assessments themselves can

he made more equitable," he said,

"and assessments can help makeinputs more equitable."

To attain equity...opportunities for dem-onstrating compe-tency must be diverse.

But while agreeing on equity asa goal, researchers offered differ-ing perspectives on how to definethc concept and how it might bcachieved. To Gordon, equity isnot equivalent to equality, butrather, sufficiency. To attain eq-

uity, resources must be distrib-uted sufficiently and opportuni-ties for demonstrating compe-tency must be diverse. "You mayexpose all persons to the samestandard," Gordon said, "but ifthe manner in which the standardis presented isn't one that matchesthe characteristics of each per-son, one cannot assert that it hasbeen presented equitably."

Gordon also laid down a chal-lenge to the assessment commu-nity and outlined three ways tomake assessment more equitable.

First, make better use of theinformation provided from assess-ment to allocate resources equita-

bly. Second, develop new instru-ments and procedures to tap stu-dents' affective traits, not just their

cognitive skills. And third, con-duct research and developmentwork to build on what is knownabout pluralism. Although port-folios appear promising as a wayof assessing students' 'abilitiesthrough diverse ways, Gordonwarned against latching onto port-folios as the solution to everyproblem.

Gordon alSo cautioned that theproblem of inequity is a problemof the larger society outsideschool. As long as society contin-ues to reward wir -iers and toscreen out people "not like us,"

3

inequities will remain regardlessof what happens to assessments.

Lauren B. Resnick, CRESST/LRDC, University of Pittsburgh,argued that assessment can play amajor role in alleviating inequi-ties. Like Gordon, Resnick saidthat the real issue is learning, andshe said that changing assessmentis vital to creating opportunity tolearn for all Americans.

"._equity...is theright to achieve at lev-els sufficient to par-ticipate productively,and in a rewardingway, economicallyand civically."

Resnick argued that we areheading down a revolutionarypath, at the end of which all stu-dents will have a right to achieve."What we mean by equity," shesaid, "is the right to achieve atlevels sufficient to participate pro-

ductively, and in a rewarding way,

economically r.nd civically."

Reaching such equity, she said,will require defining the level ofachievement all children must at-tain, a process that is now underway through the development ofnational standards. In addition, itwill require holding ourselves re-sponsible for providing the op-


portunity for all students toachieve to the desired level. Al-though the right to achieve couldeventually become a legal stan-dard that would enable youngpeople to demand the opportuni-ties to achieve, in the meantimethis right could serve as a moralobligation for society.

Lorraine McDonnell, CRESST/

University of California, SantaBarbara, argued that attainingequity is a political, not a moral,challenge. She noted that Ameri-cans have long supported whatshe called procedural equity, or aprocess that ensures that every-one has access to valued goods.But substantive equity, or equalresults, has never enjoyed publicsupport, in part because it de-mands redistribution ofresources.Achieving that type of equity,McDonnell argued, demands ap-pealing to voters' self-interest fora better society in which youngpeople are better educated. "Onehopes for altruism, but it's hard tobuild a political majority that way,"

she said.

Similarly, Jeannie Oakes of theUCLA Graduate School of Edu-cation also doubted that moralsuasion would be sufficient toensure equity. Many schools, shesaid, lack the human and material

resources needed to create theconditions to provide high levels

of instruction for all students,particularly low-income, minor-ity students. What is needed arcopportunity-to-learn standardsthat would define a fair share ofresources for schools. But that initself may not be enough, Oakesadded, because it is "astonishinghow quickly good data disappear."

Instead, she said, schools shouldmake the data-collection process a

part of the improvement process."My best guess at this point,"

Oakes said, "is that whatever pro-cess we engage in for the regularbusiness of collecting informa-tion and feeding it back into oursystems probably better lookpretty much like the complex kind

of teaching and learning we'rehoping for in schools. Ideally, theprocess of assessing the quality of

students' opportunities will be-come indistinguishable from thevery effort to create and improvethose opportunities."

Evaluating the Fairness of As-wssments

Looking specifically at the issue

of fairness and assessment,CRESST Co-director RobertLinn noted that the traditionalmethods of evaluating whethertests are fairimpact analyses,which measure differences ingroup performances; sensitivity re-

4

views, which examine test con-tent with an eve toward eliminat-ing offensive or stereotyping ma-terial; and statistical analysesare inadequate for use with per-formance assessments.

Math assessments thatdemand substantiallinguistic ability max,be unfair to those Lick-ing in that skill.

Linn proposed additional factorsthat must be considered to deter-mine if new assessments are fair.These new factors include:

The intent of the measure,or the extent to which as-sessments measure ancillary

skills that might provide anadvantage to a particulargroup. As an example, Linn

noted that math assess-ments that demand a sub-stantial amount of linguis-tic abiEtv could prove a dis-

advantage to those withmath skills who lack read-ing and writing ability.

Comparability, or the ex-tent to which the assess-ments allow variability informat or scoring.

4


Choice of task. By allowing

students to choose theirtasks, such as reading abook, we may be providingan advantage to those whoare already familiar with the

books they choose.

Delivery standards. To as-sure equity, one must looknot only at assessments, but

the provision ofthe instruc-tional experiences to stu-dents.

Leigh Burstein, CRESST/UCLA, also said that examiningstudents' opportunity to learn isessential to evaluate the fairness

f new assessments. "You can'tmeasure achievement withoutknowing the instructional condi-tions under which achievementoccurs," he said. But he cautioned

that studies of opportunities tolearn have thus far only takenplace in low-stakes environments,

not in situations where schoolswere held accountable for provid-

ing such opportunities.Burstein pointed out two ways

in which students' opportunitiescould have an effect on their per-formance on alternative tbrms ofassessments. First is the students'

own experiences. Students accus-torned to multiple-choice tests

may balk at assessments that de-mand that they write responses,particularly in subjects other thanEnglish. He pointed out that inthe 1990 National Assessment ofEducational Progress, many stu-dents simply skipped over open-ended items, and there were eth-nic group differences in the omitrates.

In addition to the variations instudent experiences, differencesin teachers' experiences can alsoaffect students' performance,Burstein said. In reform environ-ments, urban teachers tend to have

fewer chances to participate indeveloping and scoring new as-sessments than do teachers fromsuburban and rural areas, since itcosts districts to send teachers tosuch sessions.

Burstein described two studiesdesigned to examine students' cur-

ricular experiences. The first looks

at classroom "artifacts"text-books, logs of daily activities,homework assignments, in-classquizzes, and major assignmentsand compares the findings withsurveys of teachers that attemptto get at students' learning op-portunities. The second, con-ducted as part of the CaliforniaLearning Assessment System, asks

studcnts and teachers if they havedone problems in their mathemat-

ics classes like the open-endedones on the assessment.

Performance-basedassessments use a smallnumber of tasks thatmay favor one groupover another.

H.D. Hoover of the Universityof Iowa, however, said it is un-clear whether differences in groupperformance on performance as-sessments reflect differences incurricular experiences or test bias.H-z said that, unlike multiplechoice tests, which can include awide range of questions, perfor-mance-based assessments use asmall number of tasks that mayfavor one group over another. Acertain reading passage and itscorresponding questions may ap-peal more to a student who has aninherent interest in the subjectmatter of that passage.

"For fairness, you need a diver-sity ofcontent and contexts," saidHoover. "Those of us who buildstandardized teststhat's what wedo. We ask lots of questions, andbalance questions."

Data From Large-Scale Assess-ment Programs

Whether because of differencesin opportunity to learn or because


of the assessments themselves,the gaps between advantaged and

disadvantaged students arc notclosing as schools shift to newforms of assessment. In fact, re-searchers who have studied newlarge-scale assessment programshave found that the gaps may bewidening.

Daniel Koretz, CRESST/RAND, said that some teachersin Vermont agree that that state'spioneering portfolio assessmentprogram improved education fortraditionally low-performing stu-dents. However, he noted, suchimprovements were erratic.

The use of portfolioassessments requiresteachers to adapt toeverchanging curricu-lar and instructionaldemands.

"We see sign after sign aftersign that teachers vary enor-mously in response to perfor-mance assessment," Koretz said."They vary in how quickly theyadapt to demands, and to what isexpected of them."

"Moreover," Koretz said, "theprogram itself of course has done

nothing to alleviate the condi-tions that have plagued low-per-forming students, such as pov-

erm" He concluded that improv-ing the level of student perfor-mance through the use of portfo-lios is a "hard, arduous task" thatwill require money and an infra-structure to enable teachers toadapt to changing curricular andinstructional demands.

Likewise, CRESST partnerMary Lee Smith from ArizonaState University found that thefirst year of implementation ofArizona's performance-based as-sessment program failed to ad-dress the disparities between low-performing and high-performingschools. Although teachers hadbelieved that the program wouldbe a low-stakes exercise that would

enable them to improve instruc-tion, in part through developingand scoring the assessment, inpractice the Arizona Student As-sessment Program (ASAP),proved quite different. The scor-ing was done by a commercialpublisher, not teachers, and thestakes went up when newspapersranked school districts accordingto test scores. The state, more-over, provided little professionaldevelopment to boost the capaci-ties of schools.

Whcther the ASAP program will

eventually narrow the gaps inschool performance is difficult topredict.

5

We have seen teach-ers break an integratedunit into bits, andteach toward masteryof the bits.

"It is not yet clear," said Smith,"how ASAP will affect instruc-tion, because or in spite of itshigh-stakes function, althoughthere are already foreshadowings.It is predictable that districts withadequate resources will do what isnecessary to raise low scores.Whether they will take the highroadundertaking the time-con-suming and expensive professional

and curriculum developmentwork necessary to teach towardambitious standards and a think-ing curriculumor the lowroadfinding the tricks to inflatescoresremains to be seen. Atthis point, we have already docu-mented such activities as teachersfocusing pupils' attention on thoseparts of the assessment that willbe scored. We have also seeninstances of what we call dis-inte-

grating, in which teachers wholack a thorough understanding ofconstructivist teaching take whatwas designed to be an integratedunit, break it into bits, and teachtoward mastery of the bits."

As with the large-scale pro-

6


grams, Lorrie A. Shepard,CRESST/Universitv olColoradoat Boulder, said that in classroomassessments, gains in students'ability to perform well on perfor-mance tasks come slowly and un-evenly. Drawing from herCRESST research involving thirdgraders in three Colorado schools,

Shepard cited a problem that asked

the students to complete a tableto determine how many pitchers(of 4 cups each) it would take tohave enough cups for all the stu-dents in the class. There were nodifferences in performance be-tween Anglo and Latino students,but initially very few students ineither group could even attemptthe problem. In the second yearof the program, the majority ofstudents could complete the tableand wrote more to explain theiranswers than they did the firstyear, Shepard explained. Butmany still had a long way to go."Anyone who thinks this can beput in place in a year or two isprobably crazy," she said.

But Jennifer Hanrev, a teacherat Cherry Drive ElementarySchool in Thornton, Colorado,who is part of the CRESST/Uni-versity of Colorado at Boulderstudy, said even small gains arevaluable. By using a variety ofalternative assessment methods,

Harvey now knows her studentsbetter than she ever did and shehas evidence oftheir progress thatshe can show to parents. She alsocited the case of one of her stu-dents, Jeff.

"I had given him a runningrecord in October," said Harvey;"he read about a page and a halfin 10 minutes. At the end of thattime, I couldn't bear to watch hisface anymore, it was so painful.So we stopped.... [He read] theexact same page in January. Andhe blew right through it. He didvery well. Again, he was thirdgrade, it was a preprimer, butthat was progress. And I showedit to him and said, 'Look whatyou've done.' And he beamedand said, 'I'm getting better,aren't I?' That's where it's at, forme."

Ideally, policy makersshould weigh the costsand benefits of a pro-posed reform beforeimplementation.

Costs of Performance Assess-ment

The benefits of performanceassessment should also have a bear-

ing on policy makers' decisionsabout whether to implement

them, since policy makers shouldideally weigh the costs and ben-efits of a proposed reform. Butquantifying the potential benefitshas proved elusive, noted DavidMonk of Cornell University, andas a result, most of the discussionof costs has focused solely on ex-penditures.

"You can't talk about cost with-out dealing with the benefit sideof the equation," Monk said."That's a problem with perfor-mance assessment. The simple factis, we don't know very much about

what performance assessment pro-

duces, or what kind of levels ofresources are required for this totake place. In the absence of thatkind of knowledge, you're in a bitof a dilemma trying to carry out acost analysis that's more than anexpendituic analysis."

Monk said that an analysis heconducted for the New StandardsProject produced a range of costestimates for performance assess-mcnts, depending on the extentto which the assessment is addedon to existing programs and theextent to which every student istested. In the worst case scenario,in which the assessment was anaddition to existing programs andevery student in three grades wastested, the assessment cost $97.4million oi $29 per pupil for alarge state (Texas), $27.9 million


or $28 per pupil for a medium-sized state (Virginia), and $3.5million or S37 per pupil for asmall state (Vermont ). In all cases,

a little less than a fourth of theexpenditures went toward statfdevelopment, and the total ex-penditures amounted to between0.6 percent and 0.7 percent ofeach state's education budget.

Fritz Mulhauser of the U.S.General Accounting Office(GAO), which conducted ananalysis of the cost of testing tbrCongress, said that the GAO study

came up with an estimate for aproposed national test that wasfar lower than previous esti-matesfrom $42 million for amultiple-choice test and littleadded time to $209 million for atest including short performance-based questions and 30 minutesof added testing time. ButMulhauser noted that the truecost depends on the purpose forsuch a test, and he urged Con-gress to be clear about the pur-pose before determining the pro-posed cost.

Similarly, Lawrence Picus,CRESST/University ofSouthernCalifornia, also raised a numberof questions that need to be an-swered before determiningwhether the benefits of perfor-mance assessment exceed thecosts. Among these questions are:

whether we want to compare in-dividual students or school dis-tricts, how many tasks are neededto provide a reliable estimate ofstudent abilities, how much train-ing is needed for teachers, andwhat are the competing claims forresources.

As in other issues of publicpolicy, Picus said, decidingwhether to invest in performanceassessment involves tradeoffs:"How much of this do you wantto do versus how much can youafford to do in terms of time andresources?"

Equity and Assessment Design

In looking ahead to new pro-grams, researchers also discussedpossible ways to ensure equity inassessment.

Equity is one of thekey criteria by whicheducators can judgetests.

One step toward that end isbeing taken by the National Coun-

cil of Teachers of Mathematics(NCTM). In that group's Assess-ment Standards for School Math-ematics, equity is one of the keycriteria by which educators canjudge tests, according to Thomas

7

A. Romberg of the University ofWisconsin, Madison. Rombergsaid that the NCTM defines eq-uity as providing all students theopportunity to demonstrate theirmathematical power, and he notedthat current tests do not matchthe council's goals.

cc ...[Natiye Ameri-cans] also value pa-tience, whereas testsdemand rapid re-sponses and irnmedi-ate decisions."

Other researchers suggestedthat new assessments must takeinto account the needs of diversestudents in order to be equitable.Michael Pavel of the UCLAGraduate School ofEducation saidthat Native American students areill-served by traditional tests. Na-tive Americans value placidity, a

characteristic that may result intheir being viewed as slow or back-

ward, he said. Similarly, patienceis valued, whereas tests demandrapid responses and immediatedecisions.

He added that assessmentshould be better adapted to Na-tive American people who needto improve on their academic per-

formance and make teachers more


aware of how they can addressthese academic needs. Therefore,to assist Native Americans, assess-

ment results should be used toguide student learning, refine cur-riculum, and improve instruction.

Charlene Rivera of GeorgeWashington University said as-sessment reform must also con-sider students for whom Englishis not the native language. In ad-dition to learning content knowl-edge and skills, she pointed out,"English language learners"(ELLS) are also learning a secondlanguage, something native En-glish speakers do not have to do.

"To datc, reform efforts havenot considered needs of ELLSstudents," Rivera said. "The guid-ing assumptions have been: Whatwill work for monolingual stu-dents will also work for ELLSstudents. Once ELLS studentslearn a little English, new im-proved assessment systcms will fit

them too. However, experiencedoesn't support this assumption.While ELIS students can and dolearn to high standards, assessingtheir achievements in the sameway as their monolingual peerswill greatly underestimate theiraccomplishments and potential."

Rivera added that schools cur-rently include English languagelearners in inappropriate testing

programs or Ise exempt themfrom tests altogether. "I am onthe side of trying to develop amiddle ground where there is ac-countability for student learning,"

she said. "If you exclude themcompletely, they are not consid-ered in the policy debate. Buthow to assess them? There is nodefinitive answer. The best prac-tices will result from experimen-tation."

Experiments alreadyunderway suggest pos-sible solutions for tra-ditionally underservedgroups.

Some experiments already un-der way suggest possible solu-tions for English language learn-ers and other traditionallyunderserved groups. One suchexperiment is a primary-gradesassessment currently being devel-oped by a consortium of six states

under the auspices of the Council

of Chief State School Officers.Jackie Cheong of the Universityof California. Davis, said one ofthe guiding principles of that ef-fort is to provide multiple assess-

ment strategies to tap a range ofwhat students from diverse back-grounds know and are able to do.

9

Establishing the link betweenclassroom assessments, such asportfolios, and accountability isone way to further the use ofmultiple assessments.

PortfoliosLikewise, portfolios can also

provide diverse opportunities forstudents to demonstrat their skillsand knowledge. A school projectin Pasadena, California, that inte-grates language arts and visualand performing arts instructionby asking students to create worksof art and to reflect on their ownand other works has shown dra-matic results among a group offormerly low-achieving children,according to Pam Aschbacher,CRESST/UCLA.

But Aschbacher also pointedout that implementing portfoliosalone may not improve educa-tional opportunities for all youthsand the portfolios themselves can

demonstrate this. In a separateproject, Aschbacher examinedportfolios to open a window onclassroom instruction and foundthat teachers were not alwaysmeeting reform objectives. Un-der the program, known asHumanitas, students were ex-pected to be personally investedin the work that they're doing inclass, to make interdisciplinaryconnections, to engage in corn-

Assessment Questions: Equity AnswersProoz.edings of the 1993 CRESST Conference

plex reasoning, and to evaluatethemselves and show some growth

over time, among other goals.The portfolios showed a direct

relation between the kind of as-signments teachers gave their stu-dents and what students learned.For example, she found that stu-dents' work showed more higherorder thinking skills and interdis-

ciplinary connections when theirclass assignments required themto make those connections and touse complex thinking. "That's astrong message," said Aschbacher,"to these teachers that says: 'Look,when you use assignments thatdo not explicitly ask kids to do the

kind ofthinking this program callsfor, they don't do it. You getwhat you ask for."

Similarly, Catherine Smith ofthe

Michigan Department of Educa-tion said that a program in thatstate to use portfolios to gaugestudents' workforce readiness alsoraised equity concerns. "Studentsin inner-ciry and rural areas wereless likely than those in suburbs to

be aware that they could use evi-dence from part-time jobs and other

a fterschool activities to demonstrate

readiness skills," she said.

Maryl Gearhart, CRESST/UCLA, also reported that imple-menting portfolio assessmentalone may not have much impacton classroom practice. Deserib-

ing two R&D projects on class-room assessment, one focused onwriting and the other on math-ematics, Gearhart said, "I havefound that elementary teachersdo not typically develop in theirunderstandings of student workwithout a substantive focus onsubject matter. New assessmentsrequire teachers to make informed

judgments," she said, "but teach-ers cannot judge work that theydo not understand. In ourprojects, the goal is to create pro-totypes of assessment practicesinttsrated with reform curricula.We can't simply exhort teachersto collect student work and assessit," said Gearhart. "We need togive them specific models builtupon specific curriculum."

Group AssessmentSimilar caution flags went up

over another potential avenue forensuring equity: the use of groupwork. Wayne Neuberger of theOregon Department of Educa-tion suggested that allowing stu-dents to work in groups in ad-vance of an assessment could level

the play* field by enabling stu-dents to compare notes and en-sure that they all had the samebackground to prepare for theassessment, The New StandardsProject, of which Oregon is a

1 0

9

member, provides an opportu-nity to do just that.

But Noreen Webb, CRESST/UCLA, noted that group inter-actions are complex and that sim-ply asking students to cooperatemay not result in everyone's ac-quiring shared knowledge andunderstandings. Factors such asthe composition of the groupand the incentives they have forworking together can influencewhether group interactions arefunctional or dysfunctional,Webb said.

Moreover, she added, relyingon group assessments as a mea-sure of student abilities may ad-versely affect equity, Dependingon the way groups work together,she said, group performances may

not be a valid measure of theperformance of individual mem-bers, and, as a result, group per-formances may mask individualdeficiencies. If that is the case,Webb warned, low performersmay miss out on needed instruc-tional help.

Equity and Interpreting As-sessment Results

In addition to the design ofnew assessments, the way resultsare interpreted also has a bearingon questions of equity. DavidBayless of Bayless and Associates

I t)


said that the traditional way ofanalyzing assessment results is tolook at subgroups, such as gen-der, socioeconomic status, andethnicity. But those analyses canprovide little guidance for im-provement, because they cannotbe changed. "It's difficult tochange (a student's] gender," hesaid. Rather than select demo-graphic subgroups that are easy tomeasure, Bayless argued, analysts

should examine assessment resultsaccording to factors that can beinfluenced by intervention, suchas opportunity to learn.

Examining single testscores may mask im-portant differences insubgroups' perfor-mance.

Bengt Muthen, CRESST/UCLA. said that examining single

test scores may mask importantdifferences in subgroups' pertbr-mance. In one study, Muthen said

he is using multivariate analysis to

gain a broader picture of studentperformance on the National As-sessment of Educational Progress.

Preliminary findings suggest thatwhile female and male studentsperform about equally well over-all, male students outperform fe-

males on certain subscales, suchas measurement. Muthen is alsoanalyzing the results of thc Lon-gitudinal Study of AmericanYouth to detect the effects oftrack-

ing and coursetaking on studentperformance over time.

Looking at a separate way ofexamining possible sources ofbias,

Jamal Abedi of CRESST/UCLAsaid that different methods ofanalyzing interrater reliability may

yield different estimates. As a re-sult, he suggested using differentapplicable approaches. "I cannotname a single best approach toestablishing interrater reliability,"

Abedi said. "Depending on thetbrm of data, compute as manyapplicable approaches as you can,and then draw your conclusionsbased on those outcomes."

Michael T. Nettles of the Uni-versity of Michigan proposed anadditional step. After citing eth-nic group differences in perfor-mance on various assessments, hesaid that we should look at assess-

ments that show good results vith

traditionally low-performing stu-dents.

Under the auspices of the FordFoundation, Nettles said thatduring the next year he expects todevelop a second symposium onequity and educational testing and

assessment and develop a pro-

gram to identitY exemplary as-sessment programs. One of thecriteria for being considered ex-emplary should be that the assess-ments have demonstrated mak-ing a contribution to improvingthe success of poor and minoritystudent populations.

Equity and the Research AgendaAlthough some answers to the

equity questions surrounding newforms of assessment are begin-ning to emerge, the researchagenda remains long. PaulineBrooks, CRESST/UCLA, out-lined a host of questions in lightof the unequal representation ofgender, socioeconomic status, and

diverse cultures throughout thehistory of testing and assessment.Included on her list were:

1 1

the effects of the teacher'scultural expectations onhis/her judgments of cul-turally diverse student per-formances;

the extent to which the new

assessments provide vary-ing opportunities for stu-dents of different cultural/SES/language backgroundsto demonstrate theirknowledge;

the correlation between

1 I


performance on standard-

ized achievement tests and

current performance assess-

mentsdo the racial/cul-tural and SES gaps narrow,

remain about the same, or

widen vith the use of per-formance assessments?

the characteristics of teach-

ers' interactions with dif-ferent cultural groups ofstudents, in both instruc-tional and assessment set-

tings; and

relative levels ofsupport for

the new assessments among

communities that vary eco-

nomically and culturally.

Working Group ResultsIn addition, working groups

representing various key constitu-

encies outlined action plans tbrdesigning equity-sensitive perfor-

mance assessments. The groups

recommended the following:

Policy Makers:Clearly articulate the pur-poses of new assessments,

so that the public under-stands that assessments are

aimed at meaningful andeffective accountability, not

an attempt to evade account-

ability.

Make sure assessment for

"classroom utility" and as-

sessment for public ac-countability are in sync, so

that teachers have an incen-

tive to focus on improvedinstruction.

Keep expectations high for

all students to eliminate any

incentive to relegate low-performing students to asecond-class education.

Practitioners:Make the purposes andlearning outcomes of new

assessments clear and make

sure that large-scale andclassroom-level assessments

are integrated.

Encourage flexibility so that

all students have the timeand opportunities to suc-ceed on new assessments.

Provide choices in assess-

ment alternatives to chil-dren that are appropriate to

their ethnicity, their gen-der, the possible existence

of handicap, the languagethat they use, etc. Incorpo-

rate diverse groups during

the design and piloting pro-

cess so that we engage all

12

communities andconstitu-encies.

Develop university-school

partnerships to share knowl-

edge about children's learn

ing and provide ongoingstatfdeve l op ment for teach -

erS.

Conduct longitudinal andcomparative studies to ex-

amine the impact of newassessment strategies onteachers and students.

Foundations:Make sure new programs

are practical and feasibleso that they can be imple-

mented in schools andyield useful findings.

Make sure programs arecredible to parents, teach-

ers, and funding agencies

and that the programs are

actually implemented.

Try to focus on the "hard-

to-crack" cases.

Business Community:Educators, businesses, and

the community at largeneed to find a better wayto collaborate to produce

I 2


diagnostic assessments thatenable our students to ad-vance their own goals in the

classroom and in the work-place.

Assessments must also bepredictive and show stu-dents' readiness for theworkplace as well as theirability to transfer skills from

school to work.

Consider whether all assess-

ments must be administered

in school or by teachers orwhether other approachesare possible, depending onthe purpose.

Media:Highlight the standards andcontent of new assessmentsas a way of showing what all

students arc expccted tolearn.

Report opportunity to learndata and classroom envi-ronments to provide a bet-tcr understanding of eq-uity.

Develop links between re-searchers and the media toprovide context for test-score data.

_

Parents:Involve parents in all as-pects of the developmentot' new assessments via ad-vocates or site-based man-agement, helping to ensurethat they are fair and betterfor all children.

Collect data on instruction,testing participation, growth

in student performance, andteacher grading, to accom-pany test-score data.

Conduct research on par-ent involvement, toxicschools, the role ofchurches,

and communication of test-ing information.

In ConclusionIn concluding remarks, Adam

Urbanski, the president of theRochester (NY) Teachers Asso-ciation, pointed out that reform-ing schools so that all studentslearn at high levels is a long anddifficult process that must involve

the entire community. But he said

that setting standards and devel-oping new assessments is the es-sential starting point of such ef-forts in order to ensure bcth ex-cellence and equity.

"I respectfully suggest that allstudents are learning already," said

Urbanski. "They're just not learn-ing the same things. Some stu-dents are learning math and En-glish and foreign languages andthe arts and physics. And some are

learning a lot about exclusion andfailure and discrimination and lack

ofopPortunities and low expecta-tions of them. But you can't stopstudents from learning. You canonly assist in channeling them tocertain 'kinds' of learnings. Thatis why the whole issue of stan-dards and assessment is so ger-mane. They are absolutely essen-tial and not only essential, butindeed the necessary starting pointfor all other reforms. Schoolchange need not be a choice be-tween making things better ormaking things fair. We're capableof doing both. We must makethings better and be fair in doingit."

Robert Rothman is a visitingresearcher at CRESST I UCLA.

13

1

CRESST/CSE TECHNICAL REPORTS

The following assessment reports are now available by calling (310) 206-1532.Or you may fill in the order form on page 20 and mail to CSE/CRESST Annex,

Graduate School of Education, 405 Hilgard Ave., Los Angeles, CA 90024-4108.

VERMONT UPDATECan Portfolios Assess StudentPerformance and Influence In-struction? The 1991-92 Ver-mont ExperienceDaniel Koretz, Brian Sather,Stephen Klein, Daniel McCaffrey,

and Edward Deibert, RANDCSE Technical Report 371, 1993($9.00)

Vermont's statewide assessmentinitiative program has garneredwidespread attention nationwidebecause of its reliance on portfo-lios of student work. This 145-page CRESST/RAND report de-scribes results of a multifacetedevaluation of thc program andprovides information about imple-

mentation of the Vermont assess-ment, program effects on educa-tional practice, reliability and va-lidity of portfolio scores, and ten-

sions that exist between assess-ment and instructional reform.

"Findings from the evaluation,"

said the research team, "suggestthat the assessment program re-sulted in changes in curriculumcontent and instructional style."Additionally, the researchersnoted that the amount of class-room time devoted to problem

NEW REPORT

solving increased, as did theamount of time students workedin small groups. Finally, portfo-lios seem to increase teachers'enthusiasm for their subjects andfor teaching.

While there was widespreadsupport for the reform at theschool level throughout thestatenearly one-half of theschools were voluntarily expand-ing the use of portfolios to othergrade levelssubstantial prob-lems remain. The mathematicsportfolio assessment created newburdens for principals, teachersand students, including demandson teachers' time and school re-sources. Over 80% of fourthgrade teachers and over 60% ofeighth-grade teachers reportcdthat they often had difficulty cov-

ering the required curriculum.Researchers anticipate that in time

some of these demands are likelyto decline, although others repre-sent continuing burdens.

"The Vermont experience hasimportant implications for reforms

that are undenvay or under con-sideration in other jurisdictions,"said the researchers, "but onlytime and careful scrutiny will show

1 4

1 3

how filly the goals of the pro-gramand of similar reform pro-grams centered on performanceassessmentcan be met."

MORE ABOUT VERMONTInterim Report: The Reliabil-ity of Vermont Portfolio Scoresin the 1992-93 School YearDaniel Koretz, Stephen Klein,Daniel McCaffrey, and BrianStecher, RANDCSE Technical Report 370,1993($3.00)

This interim report provides thefirst results of the second year ofthe Vermont assessment program,focusing on program implemen-tation, effects on education, andthe quality of performance data.

"The program was altered inmany ways in 1992-93," said theCRESST/RAND research team,"which resulted in a clear increase

in the reliability with which math-

ematics portfolios were scored."However, the researchers addedthat while this progress is encour-aging, scoring reliability in math-ematics needs to be increased fur-ther if the program goals are to beachieved. Refining or simplifyingscoring rubrics and placing fur-ther restrictions on types of tasks

1 4


considered acceptable for inclu-sion in mathematics portfolios areamong the types of clarificationsthat may result in increased reli-ability.

"In contrast," said the research-ers, "the reliability ofwriting port-folio scores did not improve sub-stantially and was considerablylower than in mathematics." Theresearchers believe that it is unre-alistic to expect a substantial rateof improvement in the reliabilityof the writing portfolio scoresunless the program itself is sub-stantially revised.

..`c Comparability Across Assess-

ments: Lessons From the Useof Moderation Procedures inEnglandElizabeth Burton and Robert L.LinnCSE Technical Report 369, 1994

($4.00)Although there is considerable

interest in developing a system ofperformance-based examinationsin the United States, there is ageneral lack of agreement on howto compare the results ofdifferentperformance assessments to setsof common national standards.This paper addresses the "com-parison" problem, drawing on two

major approaches used in En-gland, moderation by inspection

and ste:tistical ;noderation, to link

performance assessments to setsof common standards.

Although the United States willprobably not develop a programof assessments exactly like thoseused in England, it is likely thatthc procedures used to comparethe assessments will be similar.

"Currently English secondaryschool exams in various subjectsare developed and administeredby nine examination boards," saidBurton and Linn. "Individualschools are free to choose theexamination board that best titstheir standards. While local con-trol and high quality of assess-ments are maintained, the com-parison ofscores across the boards

is problematic," added the re-searchers.

In this report, Burton and Linndiscuss the advantages and prob-lems of moderation by inspectionand statistical moderation, to-gether with an explanation ofwhyneither approach is satisfactory by

itself. The authors concluded thatsome combination of the twoapproaches may be necessary."Neither a pure moderation byinspection nor a strict statisticalmoderation system is likely to meet

this [link between assessments and

standards] need," said Burton and

Linn. "It seems more likely that

some sort of hybrid system will be

required..."

-..`c Results From the New Stan-dards Project Big Sky ScoringConferenceLauren Resnick, Daniel Resnick,and Lizanne DeStefanoCSE Technical Report 368,1993($3.50)

Partially funded by CRESST,the New Standards Project is aneffOrt to create a state- and dis-trict-based assessment and pro-fessional development system thatwill serve as a catalyst for majoreducational reform. In 1992, aspart of a professional develop-ment strategy tied to assessment,114 teachers, curriculum supervisors, and assessment directors metto score student responses from afield test of mathematics and En-glish language arts assessment.The results of that meeting, theBig Sky Scoring Conference, were

used to analyze for comparabilityacross holistic and anaholistic scor-

ing methods."Interscorer reliability esti-

mates," said the researchers, "forreading and writing were in themoderate Fange, below levelsachieved with the use of large-scale writing assessment or stan-dardized tasks. Low reliability lim-

its the usc of [the] 1992 readingand writing scores for making

15


judgments about studenc perfor-mance or educational programs,"concluded the research team.

However, interscorer reliabilityestimates for math tasks weresomewhat higher than for literacy.

For six out of seven math tasks,reliability coefficients approached

or exceeded acceptable levels.The findings suggest that the

large number and varied nature ofparticipants may have jeopardizedthe production of valid and reli-able data. "Scorers reported feel-ing overwhelmed and overworked

after four days of training andscoring," said the researchers.

Despite these difficulties, evi-dence was provided that reliablescoring oflarge -scale performance

assessments can be achieved when

ample time is provided for train-ing, evaluation, feedback, and dis-cussion; clear definitions are given

of performance levels and the dis-tinctions between them; and well-chosen exemplars are used.

Parent Opinions About Stan-dardized Tests, Teacher's In-formation and PerformanceAssessmentsLorrie A. Shepard and CarribethL. Bliem

CSE Technical Report 367,1993($4.00)

Using parents of third-gradestudents in a working-class and

lower-middle-class school district,researchers .his study set forthto ascertain parents' opinionsabout assessment, including theiropinions about standardized testsversus performance assessments.The researchers sought answersto several questions including"How do parents in the samplerespond to Gallup Poll questionsabout the desirability ofstandard-ized national tests and the poten-tial uses for standardized test re-sults?" As part of the study, par-ents were given an opportunity toreview pertOrmance assessmenttasks and decide what type ofassessment, standardized or per-formance, was most suitable forclassroom use.

The results indicated that whenallowed to look closely at perfor-mance assessment problems, mostparents endorsed performanceassessments for district purposesand especially preferred their usein classroom contexts.

:( Teachers' Ideas and Prac-tices About Assessment and In-structionHilda Borko, Manrcne Flory, andKate CumboCSE Technical Report 366, 1993($4.00)

Participants involved in thisstudy were part of a year-long

16

15

intervention designed to helpteachers develop performance as-sessments in reading and math-ematics. Seeking to evaluate teach-

ers' knowledge, beliefs, and prac-tices about assessment and in-struction, the researchers alsostudied the changes that occurredto teachers during the first semes-ter of the intervention program.

Findings ftom the study indi-cated that the performance as-sessment development and imple-mentation process resulted inteachers having better under-standings and new insights intostudents' thinking and learningthan when teachers relied exclu-sively on more traditional formsof assessment. However, it wasnot clear to what extent teacherschanged their instructional pro-grams to take advantage of theirnewly gained insights. Based ontheir observations so far, research-

ers feel confident that as the pro-gram continues, more extensivechanges will occur.

=c Dilemmas and Issues inImplementing Classroom-Based Assessments for LiteracyElfrieda H. Hiebert and KathrynDavinroyCSE Technical Report 365, 1993($3.50)

Researchers in this study in-

1 0


vited third-grade teachers froman urban school district to col-laborate in a classroom-based lit-eracy assessment project. Thestudy focused on a series of lit-eracy workshops designed toimplement a long-standing per-spective on curriculum, instruc-tion and assessment adapted toclassroom-based assessment.

Some of the early outcomesfrom observations and transcrip-tions of the workshops indicatedthat teachers struggled with a va-riety ofissues including the task ofembedding assessments like run-ning records and written summa-ries into their instructional pro-grams. Despite many challenges,at least one of the schools movedquickly to implement the assess-ments and use the informationthe assessments provided.

Dilemmas and Issues forTeachers Developing Perfor-mance Assessments in Math-ematicsRoberta J. Flexer .7nd Eileen A.GerstnerCSE Technical Report 364,1993($4.00)

This paper examines some ofthe dilemmas and issues that arose

during the first two terms ofworkwith teachers participating in thedevelopment of assessments in

mathematics, and reports onchanges in their instruction andassessment as a result of theproject.

During the study, many dilem-mas and issues arose that wereunique to each ofthe three schools

studied, but the most challengingproblem was teachers' focus onwhat was important to teach (andtherefore assess), and how chil-d re n cou?d learn what wastaughtall within the constraintsof limited teacher time. As ex-pected, preliminary results of theproject were mixed, but hopeful.Researchers believe that futuredevelopment and implementationof performance assessments inthese classrooms hinge on teach-ers' beliefs in these assessments asuseful and practical tools.

Whose Work Is It? A '4uestionfor the Validity of Large-ScalePortfolio AssessmentMaryl Gearhart, Joan L. Herman,Eva L. Baker, and Andrea K.WhittaktrCSE Technical Report 363, 1993($3.00)

Portfolio assessment represents

a growing commitment to bridgethe worlds of public accountabil-ity and private classroom, andpolicy maker and child. Thus,within the move toward further

authenticity, portfolios supportperformance-based assessmentsthat may incorporate shared read-ings of common backgroundtexts, collaborative planning, andoppormnities fo- students to re-vise their work.

Based on an in-depth analysisof nine elementary school teach-ers actively using writing portfo-lios in their classrooms, the re-searchers of this study focused ona technical issue not yet directlyinvestigated in R&D studies ofportfolio assessment: "Whose work

is it?"If students collaborate withpeers or receive assistance fromparents or teachers, the author-ship of classroom work of thestudent is in question. Focusinghere just on the teachers' contri-butions to student work, the au-thors documented patterns of in-structional support across writingassignments and students. Thework raises technical issues con-cerning the meaningfulness of'student' scores derived from as-sessment of student portfolios.

r.

More CRESST/CSE TECHNICAL REPORTS

Performance-Based Assessmentand What Teachers Need

HiguchiCSE Technical Report 362, 1993(S4.00)

Sampling Variability of Perfor-mance Assessments

Shavelson, Gao, cis- BaxterCSE Technical Report 361, 1993

S-L.O.0;

Raising the Stakes of Test Admin-istration: The Impact on StudcntPerformance on NAEP

Kiplinger & LinnCSE Technical Report 360, 1993(S4.00)

Issues in Innovative Assessmentfor Classroom Practice: Barriersand Facilitators

AsehbacherCSE Technical Report 359, 1993(S4.50)

Writing What You Read: Assess-ment as a Learning Event

Wolf c.,'` GearhartCSE Technical Report 358, 1993($4.00)

Omitted and Not-Reached Itemsin Mathematics in the 1990 Na-tional Assessment of EducationalProgress

Korctz, LewisSkewes-Cox, &BursteinCSE Technical Rcport 357,1992,($4.00)

Latent Variable Modeling ofGrowth with Missing Data &Multilevel Data

MuthinCSE Technical Report 356, 1992(S2.50)

The Reliability of Scores From the1992 Vermont Portfolio Assess-ment Program

Koretz, Steelier, DelbertCSE Technical Report 355, 1993($3.00)

Assessment of Conative Constructsfor Educational Research andEvaluation: A Catalogue

Snow C- JacksonCSE Technical Report 354, 1992(S8.00)

The Apple Classrooms ofTomotrovirm: The UCLA Evalua-tion Studies

Baker, Gearhart, & HermanCSE Technical Report 353, 1993,(S3.50)

Collaborative Group Versus Indi-vidual Assessment in Mathemat-ics: Group Processes and Outcomes

Webb

CSE Technical Report 352, 1993,S4.00)

Educational Assessment: ExpandedExpectations and Challenges (1992Thorndike Award Address)

LinnCSE Technical Report 351, 1992,($3.50)

The Vermont Portfolio AssessmentProgram: Interim Report on Imple-mentation and Impact, 1991-1992School Year

KorerzCSE Technical Report 350, 1992($6.00)

18

17

Design Characteristics of SciencePerformance Assessments

Glaser, Raghavan, & BaxterCSE Technical Report 349, 1992($3.00)

Accountability and Alternative As-sessment

HermanCSE Technical Report 348, 1992($4.00)

Benchmarking Text Understand-ing Systems to Human Perfor-mance: An Exploration

Butler, Baker, Falk, Herl, Jang, ciMittel;CSE Technical Report 347, 1991($5.00)

The Influence of Problem Contexton Mathematics Performance

Webb & YasuiCSE Technical Report 346, 1992(S4.00)

Report on Multilevel and Longitu-dinal Psychometric Models: LatentVariable Models for Analysis ofGrowth

Muthin & NelsonCSE Technical Report 345, 1992($2.50)

Measurement of Workforce Readi-ness Competencies: Design of Pro-totype Measures

O'Neil, Jr., Allred, & BakerCSE Technical Report 344, 1992($4.00)

Measurement of Workforcc Readi-ness: Review of Theoretical Frame-works

O'Neil, Jr., Allred, & BakerCSE Technical Report 343, 1992($4.00 )

1 8


Will National Tests Improve Stu-dent Learning?

ShepardCSE Technical Report 342, 1991($3.00)

Implications for Diversity in Hu-man Characteristics for AuthenticAssessment

GordonCSE Technical Report 341, 1991(S2.00)

The Natural Language SourcebookRead, Dyer, Baker, Mutch, Butler,Quilici, & Reeves

CSE Technical Report 340, 1991($15.00)

Language Assessment Instruments:LAUSD Language DevelopmentProgram for African American Stu-dents

Butler, Herman, & TanurguchiCSE Technical Report 339, 1991(S4.00)

Discovering What Schools ReallyTeach: Designing Improved Indi-cators

McDonnell, Burstein, Ormseth,Carrera, & MoodyCSE Technical Report 338, 1990($5.00)

Writing Portfolios at the Elemen-tary Level: A Study of Methods forWriting Assessment

Gearhart, Herman, Baker, &WhittakerCSE Technical Report 337, 1992(S4.00)

A New Mirror for the Classroom:A Technology-Based Tool forDocumenting the Impact of Tech-nology on Instruction

Gearhart, Herman, Baker, Novak,& WhittakerCSE Technical Report 336, 1992($5.00)

Cross-State Comparability ofJudgements of Student Writing:Results From the New StandardsProject

Linn, Kiplinger, Chapman, &LeMahieuCSE Technical Report 335, 1992($5.50)

Effects of Standardized Testing onTeachers and LearningAnotherLook

Hertnan & GolanCSE Technical Report 334, 1991(S5.50)

Conceptual Considerations inInstructionally Sensitive Assess-ment

BursteinCSE Technical Report 333, 1990($2.00)

Multilevel Factor Analysis of Classand Student Achievement Compo-nents

MuthinCSE Technical Rcport 332, 1990($3.00)

Complex, Performance-Based As-sessment: Expectations and Vali-dation Criteria

Linn, Baker, & DunbarCSE Technical Report 331, 1991($3.00)

19

The Validity and Credibility of theAchievement Levels for the 1990NAEP in Mathematics

Linn, Koretz, Baker, & BursteinCSE Technical Report 330, 1991($6.00)

For a complete list of more than 140technical reports, monographs, andresource papers, p!case call Kim Hurstat (310) 206-1532.

nowUCLA's Center for the Study

of EvaluationThe National Center forResearch on Evaluation,

Standards, and Student TestingEva L. Baker, Co-director

Robert L. Linn, Co-dircctorJoan L. Herman, Associate Director

Ronald Dietel, EditorKatharine Fry, Editorial Assistant

Brenda R. Thomas, Layout

The work reported in this publica-tion was supported under the Educa-tional Research and DeveiopmentCenter Program cooperative agree-ment number R117G10027 andCFDA catalog number 84.117G asadministered by the Office of Educa-tional Research and Improvement,U.S. Department of Education. Thefindings and opinions expressed inthis publication do not reflect theposition or policies of the Office ofEducational Research and Improve-ment or the U.S. Department ofEducation.

MONOGRAPHS AND RESOURCE PAPERS

MONOGRAPHS

Assessing Student Achievement:A Profile of Classroom Practices

Dorr-Bremme & HermanCSE Monograph 11, 1986(S11.00)

Evaluation in School Districts:Organizational Perspectives

Bank & Williams Editors)CSE Monograph 10, 1981 (S7.50)

Values, Inquiry and EducationGideonse, Koff, & Schwab (Editors)CSE Monograph 9, 1980 (S11.00)

Toward a Methodology of Natural-istic Inquiry in EducationalEvaluation

GubaCSE Monograph 8, 1978 (54.50)

Thc Logic of Evaluative ArgumentHouseCSE Monograph 7, 1977 (54.50)

Achievement Test ItemsMethodsof Study

Harris, Pearlman, & WilcoxCSE Monograph 6, 1977 (S4.50)

RESOURCE PAPERS

Writing What You Read: AGuidebook for the Assessment ofChildren's Narratives

Wolf & GearhartCSE Resource Paper 10 ($4.00)

Improving Large-Scale AssessmentAschbacher, Baker, & HermanCSE Resource Paper 9 (S10.00)

Improving Opportunities forUnderachieving Minority Stu-dents: A Planning Guide forCommunity Action

Bain & HermanCSE Resource Paper 8 ($11.00)

Designing and Evaluating Lan-guage Programs for African-American Dialect Speakers: SomeGuidelines for Educators

BrooksCSE Resource Paper 7 (52.00)

A Practical Approach to LocalTest Development

Burry, Herman, & BakerCSE Resource Paper 6 (S3.50)

FOLD AND SECURE

19

Analytic Scales for AssessingStudents' Expository and Narra-tive Writing Skills

Quellmalz & Burr,CSE Resource Paper 5 ($3.00)

Criteria for Reviewing DistrictCompetency Tests

HermanCSE Resource Paper 4 ($2.00)

Issues in Achievement TestingBakerCSE Resource Paper 3 ($2.50)

Evaluation and Documentation:Making Thcm Work Together

BurryCSE Resource Paper 2 ($2.50)

An Introduction to Assessmentand Design in Bilingual Educa-tion

BurrvCSE Resource Paper 1 ($3.00)

UCLACSE/CRESST AnnexGraduate School of Education405 Hilgard AvenueLos Angeles, California 90024-4108Mail Code: 139448

PlacePostage

Here

20

Order FormAttach additional sheet if more room is needed. Form I:. pre-addressed on reverse.

CSE Reports/Monographs/Resource Papers/Videotapes

Report Number Title Number of copies Price per ,:opy Total Price

POSTAGE & HANDLING(Special 4th Class Book RAW

Subtotal of SO to $10S10 to $20S20 to S50

over $50

add S1.50add S2.50add S3.50add 10% of Subtotal

ORDER SUBTOTAL

POSTAGE & HANDLING (scale at left)

California residents add 8.25%

Your name & mailing addressplease print or type:

TOTAL

Orders of less than $10.00 must be prepaid

Payment enclosed Please bill me

1:1 I would like to receive free copies of theCRESST Line and Evaluation Commentpublications.

UCLACSE/CRESST AnnexGraduate School of Education405 Hilgard AvenueLos Angeles, California 90024-4108Mail Code: 139448

ADDRESS CORRECTION REQUESTED (EF 13)Absy

18114%.

Librarian

ERIC

UCLA

415 Hilgard, 96 Rowell Library1Los Angeles CA 90124 j,

NONPROFIT ORG.

U.S. i OSTAGE

PAID

U.C.L.A.

ed 367 684 tm 021 134 author rothman, robert title 21p. · pdf filescale programs;...

Documents