Arts Assessment - Past as Preview of the

Future?

Edward RoeberMarcia McCaffrey, NH Dept. of Ed

2

About the Presenter

• Work experience:• NAEP/ECS (1969-72) – Worked on NAEP

assessments in visual arts, music and other areas• MDE (1972-91) – Directed the state assessment

program and conducted state sample assessments in the music and visual arts

• CCSSO (1991-98) – Developed the SCASS program, which included the arts

3

About the Presenter

• Work experience• Measured Progress (1998-2003) – Worked on NY

State Education Department assessments of the arts (dance, music, theatre and visual arts)

• MDE (2003-07) – Directed the state assessment and accountability programs

• MSU (2007-Present) – Teach practitioners on assessment, particularly formative classroom assessment

4

Overview of this Presentation

• Past Assessments of the Arts• National Assessment of Educational Progress• States• Others

• Current Assessment Trends• Some Ideas and Possibilities for the Future

5

Goals of the Presentation

• To review some of the past efforts that might affect future action

• To show how some of the past activity might help direct future activity

• To help states and others create assessments that lead to constructive action by users of the assessment information

6

NAEP• NAEP has a long history of assessment in the

arts• Early Assessments (1970’s)• Recent Assessments (1997)• Current Assessments (2008)

• The best of NAEP was in the distant past• Each successive assessment has been less

innovative• Is there a future with NAEP?

7

Early NAEP Arts Assessments• NAEP included music and visual arts in its original

design in the 1960’s of Ralph Tyler• NAEP’s original design was much different (and

broader) than today:• Ages 9, 13, 17 (in- and out-of-school) and young adult (26-35)• Same items often used across 2, 3, or more age groups - especially

17’s and young adults• Considerable use of written response items• Also, considerable use of individually-administered items• Group-administered items used paced tapes to read questions to

students• Annual assessments of 2-3 content areas (10 total)• Entire NAEP budget: $6-7 million per year total

8

Early NAEP Arts Assessments

• Music assessments was conducted in 1971-72 and again in 1978-79

• Visual arts assessment was conducted in 1974-75 and 1978-79

• Items were reported individually and the reader decided whether the results were “good” or “not good”

9

1971-72 NAEP Music Framework• Perform a piece of music

• Sing (technical proficiency not required)• Play or sing (technical proficiency not required)• Invent and improvise (technical proficiency not required)

• Read standard musical notation• Identify elements of notation• Identify the correct notation for familiar pieces• Follow notation while listening to music• Sight-sing

• Listen to music with understanding• Perceive different elements of music, such as timbre, rhythm, melody• Perceive structure in music• Distinguish some differing types and functions of music• Be aware of (recognize) some features of historical styles of music

10

1971-72 NAEP Music Framework• Be knowledgeable about some musical instruments, terminology of

music, methods of performance, some of the standard literature of music and some aspects of music history• Know the meaning of common musical terms• Know standard pieces of music by title or composer• Know prominent composers and performers by name• Know something of the history of music

• Know about musical resources of the community and seek musical experiences by performing music• Know whether or not music libraries and stores are in the community and

where concerts are given• Seek to perform music by playing, singing, taking lessons, etc.

• Make judgments about music and value the personal worth of music• Distinguish parodies from their models• Be able to describe a personal “music” experience

11

1971-72 Example Exercises• Music

• Multiple-choice questions• Recognize instruments and instrument families• Recognize musical genre• Know music history questions

• Performance items• Sing in unison • Sing in harmony• Sing an accompaniment• Play a musical accompaniment• Play a prepared piece and sight read a piece of music

on the instrument you play

12

Changes for 1978• The 1978 assessment was similar to that of

1971-72, but tended to emphasize the affective domain more

• The objectives applied to all students, not just those involved in music

• No out-of-school 17’s or young adults participated

• Some items were repeated from the 1972 assessment

13

1978-79 NAEP Music Framework• Value music as an important realm of human experience

• Be affectively responsive to music• Be acquainted with music from different nations, cultures, periods,

genres and ethnic groups• Value music in the life of the individual• Make and support aesthetic judgments about music

• Perform music• Sing (without a score)• Play (without a score)• Sing or play a written score• Play or sing a previously-prepared piece

• Create music• Improvise• Represent music symbolically

14

1978-79 NAEP Music Framework• Identify the elements and expressive controls of music

• Identify the elements of music• Identify the relationships of elements in a given composition• Demonstrate an understanding of a variety of musical terms,

expression markings, and conducting gestures in a musical context• Identify and classify music historically and culturally

• Identify and describe the features that characterize a variety of folk, ethnic, popular, and art music

• Identify and describe the music and musical style of the various stylistic periods in Western civilization (e.g., Medieval, Renaissance, Baroque, Classical, Romantic)

• Identify composers of each period• Cite examples of ways in which man utilizes music in his social and

cultural life

15

1978-79 Example Exercises

• All of the exercises were group-administered multiple-choice exercises (no individual performance exercises)

• Stimuli used for exercises ranged from aural only to visual, or both

• Tapes were used again in group administered assessments

16

1973-74 and 1978-79 Visual Arts Assessments

• Much less information is available on these assessments - their frameworks and the assessment results

• I was not with NAEP when the 1973-74 VA assessment was administered and reported; hence have no materials on the results of this assessment in my files

17

Visual Arts Example Exercises – 1973-74 and 1978-79

• Recognize famous works of art• Contrast two works of art• Comment on the use of lines, shading, and

other visual arts aspects in works of art• Sketch a figure presented to them (Johnny

West doll)• Draw a model shown to them (a model car)• Sculpt a figure in clay (1978 only)

18

Early NAEP Exercises• Clearly, you can see that early NAEP tried to assess

as authentically as possible• Any production/performance items were

professionally scored at the contractor’s site by arts-educators

• These assessments were not without mechanical and other issues - the quality of the tapes or quality of reproductions, for example

• And, the assessments drew fire from those who believed that assessment destroys creativity

• However, these assessments were a tremendous advance for the time - unmatched today

19

1997 NAEP Arts Assessment• During the early 1990’s, interest in national

goals and improved assessment was rampant (in all content areas)

• Every content area was in a rush to develop national standards in their content area, based on the success of the NCTM standards in mathematics

• The arts was almost left out, but the advocates insisted that national standards also be created for the arts

20

1997 NAEP Arts Assessment• The result was that national standards were

created in the arts - in dance, music, theatre and visual arts

• Interest in assessment of the arts followed on the development of the national standards, as did interest in “authentic” assessment occur in many content areas

• ‘Authentic’ assessment meant looking for ways to actually use performance assessments in large-scale programs

21

1997 NAEP Arts Assessment• Interest in a NAEP arts assessment was

rekindled and pressure was placed on NAGB• The 1997 NAEP assessment of the arts

began with such promise:• Dance, music, theatre, and visual arts were all to

be assessed• A NAEP framework would be created in each area• Assessment would occur at all three grades (4, 8,

and 12)• Performance assessment would once again be

used in the NAEP assessments

22

1997 NAEP Music Framework• Creating • Performing• Responding• Content knowledge

• Context• Aesthetics• Form and structure• Processes

• Skills• Perceptual• Intellectual/reflective• Expressive• Technical

23

Arts Assessment Development• ETS subcontracted to CCSSO to develop a

portion of the assessment pool in 1992• States invited to participate; 10 were wanted• Over 20 states responded and 15 were selected• Each state needed to supply two 4-person writing

teams (minimum) for two areas of the arts• Writing occurred from May 1 to end of June• Each team met twice, taught/led by CCSSO• Each team wrote performance assessments• In independent reviews, the CCSSO items were

hands-down the best

24

1997 NAEP Arts Assessment• The reality of the 1997 NAEP assessment of

the arts was so different:• Dance was not assessed; theatre assessed poorly• Assessment occurred three grade levels (4, 8, 12)• While performance assessment was used in the

NAEP assessments, its use was limited by budget issues

• Only 2,275 students participated in 1997 (about 80,000 students participated in 1971)

• Cost of arts assessment was $11 million

25

1997 Example Exercises• Three types of items were used in 1997

• Multiple-choice items• Short- and extended constructed response• Performance items were used to assess performing

and creating - some involved one or the other, and some involved both

• Analyze features of music• Play rhythmic accompaniment• Play rhythmic embellishment to a melody on MIDI keyboard• Perform a solo of their choice

26

2008 NAEP Assessment• Used the 1997 framework• Only music and visual arts were assessed,

and only at grade 8• Only paper-and-pencil and written

performance assessments were used• Unfortunately, because of changes in scoring

procedures, results of the performance tasks used in 1997 and 2008 can not be compared

• 7,900 students participated, about half in each area

27

Sample NAEP Music Results

• 89 scale point range (on a 300 point scale) from highest to lowest performing students

• Racial-ethnic, gender, and poverty status differences

• 57% attended school where music is taught• About a quarter of the students participated in

music activities in school; about half listened to music at least once per month

28

Sample NAEP Visual Arts Results

• 89 scale point range from highest to lowest performing students

• Racial-ethnic, gender, and poverty status differences

• 47% attended school where visual arts is taught at least 3-4 times/week

• About 40% of the students participated in visual arts activities in school at least once a month

29

State Assessments of the Arts• A number of states have given arts

assessments at one time or another• Some “piggy-backed” on NAEP assessments• Some states require state assessment (which

might or might not be funded)• Other states require assessment by local

districts• Some states have created resources for local

arts education assessments

30

State Assessments of the Arts• There are a couple of examples that might

serve as models for assessment activity in the future• Michigan• New York

• This is not in any way meant to slight the good efforts that have occurred in other states – CO, FL, MD, MN, SC, TN, and others

• States that assess the arts rarely have sufficient resources to do so….

31

Michigan• In 1972, Michigan included a small set of

multiple-choice music items in its statewide assessment program at grades 4 and 7

• On two other occasions (1977 and 1983), Michigan conducted music performance assessments using statewide samples• 1977 very much modeled after the 1971 NAEP

Music assessment• Visual arts was never assessed although

such assessments were proposed on several occasions

32

Michigan• Both assessments included paper-and-pencil

exercises, as well as individually-administered performance items

• Items were written by interested college and local district music educators under direction of MDE

• Paper-and-pencil assessments used stimulus tapes, while the performance assessments used stimulus and response tapes

• About 5,000 students took part in each assessment, or about 600 students per item per grade

33

Michigan• One hallmark of both Michigan assessments is that

the exercise development, field administration, and scoring was done by volunteers

• The assessment administrators for the individually-administered exercises were faculty and graduate students of the universities in Michigan

• MDE drew a sample of schools, and each 4th and 7th grade student participated in the group-administered assessments, and a sample of these students were also given the individually-administered ones

34

Michigan• The result was a statewide picture of music

achievement, plus assessments that could be used locally

• It also resulted in teams of trained assessment administrators at universities who were • available to local districts to train them in the

assessment, • conduct the assessment for them • Interpret the assessment information

• Assessment cost less than $10,000 total

35

New York• Has a one-credit high school graduation

requirement in the arts, and students can elect dance, music, theatre or visual arts

• NY has a requirement that students’ proficiency must be assessed

• State developed a voluntary assessment that districts could use in 1999-2001

• Project was part of a federally-funded grant • Created a full-range of assessments in dance,

music, theatre and visual arts

36

New York• The project was run by Ulster BOCES and

resulted in a balanced approach to assessment• Summative assessments

• Paper-and-pencil items• Performance/production items

• Instructionally-embedded assessments done by students throughout the school year

• Portfolio to collect all of the assessment data and for students to document and reflect on their growth in their studies

37

New York

• The assessments developed included on-demand written assessments:• Multiple-choice items• Short constructed-response• Extended constructed-response

• And performance assessments• On-demand performance events• Performance tasks (products) • Showcase portfolio with reflection

38

New York• What was created was a balanced assessment of the

arts, with • Interim benchmark tasks for students to work on during

instruction (which could serve as end-of-instructional unit exams)• Summative assessments (the on-demand performance events

plus multiple-choice and constructed-response items)• Only thing missing was attending to the skills of teachers to

formatively assess students to impact instruction

• Like Michigan, the resource level was quite low and the value of the assessments quite high

• State has chosen not to make more than a “sampler” available to districts

39

CCSSO• SCASS program began in 1991 and the arts was

one of the 6 original groups that were founded - still functioning

• The group has met over the years and has accomplished much work including• Shared information and enthusiasm about assessing

the arts• Developed an electronic template for item

development• Created banks of assessment tasks for state or local

use

40

Thinking About the Future

• There are several lessons that can be drawn from these examples• National assessment is expensive and the

resources for a good assessment may not be available

• Good models have been (and are being) created at the state and local levels

• Such efforts take time, energy and a willingness to work a lot for little or no reimbursement (given unfortunate resources allocated to it)

41

The Future….• These are interesting times

• We’re seeing even more attention to test-based accountability for schools - what will the “new NCLB” require?

• Common core standards in mathematics and ELA have been developed

• States are planning for the development of common assessments with Federal support

• Some content areas want in, others want out• What about the arts? Should common standards

and assessments be pursued?

42

The Future….• IF the arts want to create common content

standards, how and with what will they do so?• Create their own common standards in the arts

by experts in each field?• Compile state standards• Use recent NAEP frameworks• Use the national standards from the 1990’s?• Other resources?

43

The Future….

• IF arts educators want to have assessments in the arts, what is the best way to do so?• Wait for NAEP?• CCSSO SCASS?• SEADAE?• Interested states should band together to create

the desired assessments and PD to use them?• Other ideas?

44

Assessment Purposes• There are several purposes for assessment that

should be considered• Individual student-level purposes

• Identify students with talent• Determine areas where students need help• Grade students• Reward high performing students• Other

• Program-level purposes• Accountability - is the program succeeding?• Educator effectiveness – how effective are teachers?• Improvement - where are weak areas and how can they be

improved?• Other?

45

Assessment Participation

• Who should be assessed?• All students (or a sample of students), regardless of

whether they have taken part in arts programs• Only students with past participation in the arts• Only students who are currently enrolled in an arts

course or program• Only students who have determined to major in the

arts after high school

46

Assessment Types• What is needed is a fully balanced

assessment system• Summative assessments - both paper-and-pencil

and performance • Benchmark assessments, with an emphasis on

performing/creating• Educators skilled in formative classroom

assessment – to help them become more agile with assessment that occurs as instruction is taking place and is used to guide instruction as it is occurring

47

The Future….• IF arts educators want to create

assessments in the arts, how would they go about it• Seek foundation support• Raise state funds• Work within existing groups• Create new groups or structures• Just do it

48

Assessment Administration• How could an assessment be administered?

• Provide the resources and the information (and training to use them) to local educators

• Conduct a statewide sample assessment of the group-administered exercises only

• Conduct a statewide sample administration of the individually-administered exercises, using within-state resources such as university faculty and graduate students, for district comparative use

• Blended system combining two or more of these ideas

49

Assessment Resources• There are substantial existing resources that

can be accessed in the arts - assessment development training, item banks, PD, etc

• It certainly makes sense to use these wherever possible

• The resources available will likely not increase, nor will public support for using them in the arts likely increase much

• Arts educators will need to make some important decisions in the coming months

50

Assessment Resources• Personally,I believe in the power of assessment

to affect instruction positively• Further, I believe that it is possible to develop

and use really good assessment strategies - balanced assessment ones - in the arts

• Not everyone agrees with these statements….• I would love to see a group or groups working

on common assessments in the arts -- if there are such groups, I would love to help them if desired

51

Questions/Comments/More Information

• Do you have any questions or comments on anything that I have covered?

• Edward Roeber

(517) 614-4877