What Really Goes on at an What Really Goes on at an AP Statistics Reading?AP Statistics Reading?

USCOTSUSCOTSMay 18, 2007May 18, 2007

Brad HartlaubBrad HartlaubKenyon CollegeKenyon College

Daren StarnesDaren StarnesFountain Valley School of ColoradoFountain Valley School of Colorado

The Growth Continues!The Growth Continues!

YEAR EXAMS % PASSINGYEAR EXAMS % PASSING

19971997 7,667 7,667 62.2 62.2

19981998 15,48815,488 59.7 59.7

19991999 25,24025,240 57.1 57.1

20002000 34,11834,118 53.7 53.7

20012001 41,60941,609 59.7 59.7

20022002 49,82449,824 56.8 56.8

20032003 58,23058,230 61.9 61.9

20042004 65,87865,878 59.9 59.9

20052005 76,78676,786 60.7 60.7

20062006 88,23788,237 60.2 60.2

20072007 96,13996,139 ? ?

Year

Num

ber

of O

pera

tional Exam

s

20062004200220001998

100000

80000

60000

40000

20000

0

Grading the AP ExamGrading the AP Exam

Rubrics (scoring standards) are drafted by the Rubrics (scoring standards) are drafted by the Chief ReaderChief Reader

The Test Development Committee reviews the The Test Development Committee reviews the rubrics and refines the free response questionsrubrics and refines the free response questions

Rubrics are revised and the Test Development Rubrics are revised and the Test Development Committee reviews the rubrics for a second time Committee reviews the rubrics for a second time (approximately 6 months after the first review)(approximately 6 months after the first review)

The Chief Reader makes final revisions to the The Chief Reader makes final revisions to the draft rubrics for the pre-reading meetingsdraft rubrics for the pre-reading meetings

Pre-Reading MeetingsPre-Reading Meetings

Four days before the reading begins question Four days before the reading begins question teams composed of high school teachers and teams composed of high school teachers and college and university professors arrive at the college and university professors arrive at the reading site (Louisville, Kentucky) to begin sample reading site (Louisville, Kentucky) to begin sample selectionselection

Student samples are selected from hundreds of Student samples are selected from hundreds of exam booklets to form training packets and exam booklets to form training packets and revisions are made to the draft rubricsrevisions are made to the draft rubrics

The leadership team, which will consist of 75 The leadership team, which will consist of 75 leaders in 2007, conducts practice briefings leaders in 2007, conducts practice briefings (training sessions) and final decisions are made on (training sessions) and final decisions are made on scoringscoring

A Typical Reading DayA Typical Reading Day

A briefing is held to train the readers on how to A briefing is held to train the readers on how to score a particular questionscore a particular question 445 readers will score approximately 100,000 exams in 445 readers will score approximately 100,000 exams in

20072007 Each reader will score at least two different questions Each reader will score at least two different questions

during the seven day readingduring the seven day reading

Readers go to reading rooms and work with a Readers go to reading rooms and work with a partner on one folder of exams (25 exams in a partner on one folder of exams (25 exams in a folder) and then begin scoring other foldersfolder) and then begin scoring other folders

Two leaders are in each reading room to answer Two leaders are in each reading room to answer scoring questions and backread scored exams scoring questions and backread scored exams

AP Statistics Exam: AP Statistics Exam: Structure and ScoringStructure and Scoring

The AP Statistics Exam consists of two sections:The AP Statistics Exam consists of two sections:40 Multiple Choice Questions40 Multiple Choice Questions 90 minutes90 minutes

counts 50% of exam scorecounts 50% of exam score6 Free Response Questions6 Free Response Questions 90 minutes90 minutes

5 open-ended items @ 12 minutes5 open-ended items @ 12 minutes1 Investigative Task @ 30 minutes1 Investigative Task @ 30 minutescounts 50% of exam scorecounts 50% of exam score

Purpose of the investigative task: to evaluate students’ understanding in several content areas and to assess their ability to integrate statistical ideas and apply them in a new context or in a nonroutine way.

Holistic GradingHolistic Grading

We use a holistic scoring system that is probably We use a holistic scoring system that is probably very different from any system you have ever very different from any system you have ever used to grade papers, projects, or exams:used to grade papers, projects, or exams:

• 4 Complete Response4 Complete Response• 3 Substantial Response3 Substantial Response• 2 Developing Response2 Developing Response• 1 Minimal Response1 Minimal Response• 0 0

Mock BriefingMock Briefing

Investigative Task from 2006Investigative Task from 2006 HandoutsHandouts

A Guide to Scoring Free-Response Statistics A Guide to Scoring Free-Response Statistics QuestionsQuestions

Question and Rubric (Scoring Standards)Question and Rubric (Scoring Standards) Student SamplesStudent Samples Scoring CommentaryScoring Commentary

Grade SettingGrade Setting

Each student receives a composite score (100 Each student receives a composite score (100 possible points – 50 for multiple choice section possible points – 50 for multiple choice section and 50 for the free response section)and 50 for the free response section)

Grades are set based on the distribution of the Grades are set based on the distribution of the composite scores.composite scores.

• 5 – Extremely well qualified5 – Extremely well qualified• 4 – Well qualified4 – Well qualified• 3 – Qualified3 – Qualified• 2 – Possibly Qualified2 – Possibly Qualified• 1 – No recommendation1 – No recommendation

Score Distributions for 2005 and 2006Score Distributions for 2005 and 2006

Score Score 2005 Percent2005 Percent 2006 Percent2006 Percent

55 12.512.5 12.512.5

44 22.822.8 22.322.3

33 25.225.2 25.425.4

22 19.319.3 18.318.3

11 20.220.2 21.521.5

Student Performance Student Performance (over 5 years)(over 5 years)

Year

Perc

ent

At

Sco

re

20062005200420032002

30

25

20

15

10

5

0

Score

5

34

Profiles of Each Passing Score

Scoring StatisticsScoring Statistics

QuestionQuestion 20052005 20062006

11 1.411.41 2.112.11

22 1.251.25 0.460.46

33 1.461.46 0.640.64

44 1.891.89 1.041.04

55 1.111.11 1.001.00

66 1.621.62 0.830.83

Professional and Social EventsProfessional and Social Events

Professional NightProfessional Night Best Practices NightBest Practices Night Statistical Papers NightStatistical Papers Night Social Lounges and ActivitiesSocial Lounges and Activities Question DebriefingsQuestion Debriefings Worst Practices NightWorst Practices Night

Professional Development OpportunityProfessional Development Opportunity

Please consider getting more involved in Please consider getting more involved in the AP Statistics Program by becoming a the AP Statistics Program by becoming a reader. reader.

We have been unable to hit our target of We have been unable to hit our target of 60% college and 40% school readers. 60% college and 40% school readers.

To complete the online application, just To complete the online application, just point your web browser to AP Central point your web browser to AP Central (apcentral.collegeboard.com)(apcentral.collegeboard.com)

Question and Answer SessionQuestion and Answer Session

Curricular IssuesCurricular Issues Course ContentCourse Content Statistical SoftwareStatistical Software Course ProjectsCourse Projects Post-Exam ActivitiesPost-Exam Activities

Test Development Test Development The ExamThe Exam RubricsRubrics The Reading The Reading Grade SettingGrade Setting