Arizona State Board of Education NOTICE OF PUBLIC MEETING Pursuant to Arizona Revised Statutes (A.R.S.) 38-431.02, notice is hereby given to the members of the Arizona State Board of Education and to the general public that the Boards will hold a special meeting, open to the public, on Friday, August 14, 2015, at 9:00 AM at the Arizona Department of Education, Room 122, 1535 W. Jefferson, Phoenix, AZ 85007.A copy of the agenda for the meeting is attached.The Board reserves the right to change the order of items on the agenda, with the exception of public hearings.One or more members of the Board may participate telephonically.Agenda materials can be reviewed online at http://azsbe.az.gov. Pursuant to A.R.S. 38-431.02 (H), the Board may discuss and take action concerning any matter listed on the agenda. Pursuant to A.R.S. 38-431.03(A)(3) and (4), the Board may vote to convene in executive session for discussion or consultation for legal advice from the Boards attorneys concerning any items on this agenda and/or for discussion or consultation with the Boards attorneys in order to consider its position and instruct its attorneys in pending or contemplated litigation or in settlement discussions conducted in order to avoid or resolve litigation. Persons with a disability may request a reasonable accommodation such as a sign language interpreter, by contacting the State Board Office at (602) 542-5057.Requests should be made as early as possible to allow time to arrange the accommodation. DATED AND POSTED this 31st day of July, 2015. Arizona State Board of Education By: _______________________________________________________ Christine Thompson Executive Director (602) 542-5057 Friday, August 14, 2015 9:00 AM Arizona Department of Education, Room 122 1535 W. Jefferson, Phoenix, AZ 85007 SPECIAL MEETING AGENDA ARIZONA STATE BOARD OF EDUCATION August 14, 2015 Page 2 9:00 a.m.CALL TO ORDER, PLEDGE OF ALLEGIANCE, MOMENT OF SILENCE, AND ROLL CALL 1.Presentation, discussion and possible action to adopt proposed AzMERIT performance levels (cut scores). 2.Presentation, discussion and possible action to adopt a Move on When Reading cut score for AzMERIT Grade 3, English Language Arts (ELA) 3.Presentation and discussion regarding proposed performance levels (cut scores) for the National Center and State Collaborative Alternate Assessment (NCSC). 4.Presentation and discussion regarding the administration of the Move On When Reading Program 5.Presentation, discussion and possible action regarding filling Board staff vacancies in the positions of Deputy Director and Administrative Assistant for the Investigative Unit, including consideration of the Superintendents recommendation and those of other Board members.Pursuant to A.R.S. 38-431.03(A)(3), the Board may vote to convene in executive session, which will not be open to the public, for discussion or consultation for legal advice with the Boards attorneys. 6.Presentation, discussion and possible action regarding the execution of the May 18, 2015 Board policy requiring the Superintendent to grant the employees of the State Board Investigation Unit access to necessary documents, records and electronic information. Pursuant to A.R.S. 38-431.03(A)(3) and (4), the Board may vote to convene in executive session for discussion or consultation for legal advice with the Board's attorneys. 7.Presentation, discussion and possible consideration regarding Douglas v. State Board of Education (CV2015-006171). Pursuant to A.R.S. 38-431.03(A)(3) and (4), the Board may vote to convene in executive session, which will not be open to the public, for discussion or consultation for legal advice with the Boards attorneys and/or for discussion or consultation with the Boards attorneys in order to consider its position and instruct its attorneys in pending or contemplated litigation or in settlement discussions conducted in order to avoid or resolve litigation. SPECIAL MEETING AGENDA ARIZONA STATE BOARD OF EDUCATION August 14, 2015 Page 3 8.CALL TO THE PUBLIC.This is the time for the public to comment.Members of the Board may not discuss items that are not specifically identified on the agenda.Therefore, pursuant to A.R.S. 38-431.01(H), action taken as a result of public comment will be limited to directing staff to study the matter, responding to any criticism or scheduling the matter for further consideration and decision at a later date ADJOURN Arizona State Board of Education Special Meeting August 14, 2015 Item 1EXECUTIVE SUMMARYPage 1 of 3 Contact Information:Irene Hunting, Deputy Associate SuperintendentLeila Williams, Associate Superintendent Issue:Presentation, discussion and possible action to adopt AzMERIT performance standards (cut scores) Action/Discussion Item Information Item Background and Discussion On November 3, 2014, the Board adopted AzMERIT as the statewide assessment to measure the Arizona English Language Arts and Mathematics standards. The March 2014 Board adopted values for the states new assessment guided the AzMERIT Standard Setting Workshop held July 13-17, 2015 in Phoenix, Arizona. The adopted values included expectations related to the transparency, validity, and inclusion of Arizona stakeholders in the processes associated with AzMERIT. Specifically, the values included test results thatmeasure a students mastery of the Arizona standards and progress towards college and career readiness,provide valid, reliable and timely data to educators and policy makers to advance the academic success of Arizona students and inform the States accountability measures, communicate results to students, parents, and educators, in a clear and timely manner to guide instruction,provide an accurate perspective of the quality of learning occurring within classrooms and schools, and allow meaningful national or multistate comparisons of school and student achievement. In February 2015, Arizonas Technical Advisory Council (TAC), comprised of nationally recognized assessment experts, reviewed all of the planned standard setting processes and planned studies necessary for the establishment of the AzMERIT vertical scale and for the determination of mode (paper-based, computer-based) comparability.All recommendations from the TAC were incorporated in the final standard setting procedures and plans for supporting studies. On July 22, 2015, the TAC met again to review the results of the completed standard setting process and related studies.The TAC endorsed the standard setting process and the findings and conclusions of the studies. Eighty-one Arizona educators participated in the AzMERIT Standard Setting Workshop.These educators were divided into eight panels representing four grade bands (3-4, 5-6, 7-8, and 9-11) for each subject (ELA and math). Using the Bookmark method, these panelists recommended performance standards (cut scores) for AzMERIT that measure student progress toward college and career readiness and allow for meaningful national Arizona State Board of Education Special Meeting August 14, 2015 Item 1 EXECUTIVE SUMMARYPage 2 of 3 and multistate comparisons of school and student achievement. The recommended AzMERIT performance standards are generally comparable to performance standards for NAEP and Smarter Balanced. For AzMERIT ELA 11 and AzMERIT Algebra II, the recommended performance standards indicate a college readiness at least as rigorous as ACTs college readiness. A complete description of the standard setting process is included in the attached report, Recommending AzMERIT Performance Standards ELA Grades 3-11, Math Grades 3-8, Algebra I, Geometry, and Algebra II. ADErecommendsthattheBoardadoptthesescalescorerangeswhichreflectthe performance standards recommended by the AzMERIT Standard Setting panelists. AzMERIT ELA Scale Score RangesMinimally Proficient Partially ProficientProficient Highly Proficient Grade 32395-24962497-25082509-25402541-2605 Grade 42400-25092510-25222523-25582559-2610 Grade 52419-25192520-25422543-25772578-2629 Grade 62431-25312532-25522553-25962597-2641 Grade 72438-25422543-25602561-25992600-2648 Grade 82448-25502551-25712572-26032604-2658 Grade 92454-25542555-25762577-26052606-2664 Grade 102458-25662567-25802581-26052606-2668 Grade 112465-25682569-25842585-26072608-2675 AzMERIT Math Scale Score RangesMinimally Proficient Partially ProficientProficient Highly Proficient Grade 33395-34943495-35303531-35723573-3605 Grade 43435-35293530-35613562-36053606-3645 Grade 53478-35623563-35943595-36343635-3688 Grade 63512-36013602-36283629-36623663-3722 Grade 73529-36283629-36513652-36793680-3739 Grade 83566-36493650-36723673-37043705-3776 Algebra I3577-36603661-36803681-37193720-3787 Geometry3609-36723673-36963697-37423743-3819 Algebra II3629-36893690-37103711-37503751-3839 Theadoptionofthesescalescorerangeswillresultinthefollowingestimated performance on the Spring 2015 AzMERIT assessments. Arizona State Board of Education Special Meeting August 14, 2015 Item 1 EXECUTIVE SUMMARYPage 3 of 3

AzMERIT ELAest. % of students for Spring 2015 Minimally Proficient Partially ProficientProficient Highly Proficient Grade 344%15%31%10% Grade 443%19%33%5% Grade 537%33%27%3% Grade 639%27%30%4% Grade 741%26%29%4% Grade 840%27%26%6% Grade 947%26%21%6% Grade 1049%21%22%8% Grade 1154%20%17%8% AzMERIT Math est. % of students for Spring 2015 Minimally Proficient Partially ProficientProficient Highly Proficient Grade 327%31%27%15% Grade 429%29%32%10% Grade 529%31%27%13% Grade 638%30%21%11% Grade 748%22%18%13% Grade 843%24%20%13% Algebra I45%23%23%9% Geometry47%24%24%6% Algebra II47%24%23%6% Recommendation to the Board It is recommended that the Board adopt the performance standardsfor AzMERIT as proposed by the ADE in these materials. Recommending AzMERIT Performance Standards English Language Arts Grades 3-11 Math Grades 3-8, Algebra I, Geometry, and Algebra II Technical Report June 27, 2015 Arizona Assessments Summer 2015 Standard Setting Recommending Performance Standards for Arizonas Measurement of Educational Readiness to Inform Teaching (AzMERIT) ELA Grades 3-11 Math Grades 3-8, Algebra I, Geometry, and Algebra II Technical Report J une 27, 2015 Prepared by American Institutes for Research (AIR) in collaboration with the Arizona Department of Education AzMERIT StandardSettingTechnicalReportiiTABLE OF CONTENTS Introduction....................................................................................................................................1PerformanceStandardsandValidityofTestScoreInterpretations...........................................1OverviewofStandardSettingApproach........................................................................................5WorkshopDesign........................................................................................................................5WorkshopLocation.....................................................................................................................7WorkshopStaffing......................................................................................................................7WorkshopPanelists....................................................................................................................7HigherEducationPanel...............................................................................................................8WorkshopTraining....................................................................................................................10StandardSettingMaterialsandProcedures.................................................................................12PerformanceLevelDescriptors.................................................................................................12OrderedItemBooklet...............................................................................................................13AzMERITBookmarkPlacement.................................................................................................15BenchmarkInformation............................................................................................................17PanelistFeedbackandImpactData..........................................................................................18VerticalArticulation..................................................................................................................19WorkshopEvaluation................................................................................................................20RecommendedPerformanceStandardsandImpactData...........................................................21EvaluationoftheStandardSettingWorkshop.............................................................................28PanelistEvaluationofStandardSettingWorkshop..................................................................28IndependentObserverReviewofStandardSettingWorkshop...............................................29References....................................................................................................................................30AppendixAWorkshopAgendas................................................................................................A1AppendixBCompositionofPanels...........................................................................................B1AppendixCTrainingPresentations...........................................................................................C2AppendixDPerformanceLevelDescriptors.............................................................................D1AppendixETestBlueprints.......................................................................................................E1AzMERIT StandardSettingTechnicalReportiiiAppendixFSummaryofOrderedItemBooklets......................................................................F1AppendixGOIBItemDataPlots...............................................................................................G1AppendixHBookmarkPlacementReadinessForms.................................................................H1AppendixIInvestigationofEquatingStudentScoresAcrossAzMERITTestAdministrationModes..................................................................................................................I1AppendixJAzMERITVerticalLinkingStudy...............................................................................J1AppendixKPanelistEvaluationForms......................................................................................K1AppendixLRecommendPerformanceStandardsbyRound.....................................................L1AppendixMConvergenceofBookmarksacrossRounds........................................................M1AppendixNEstimatedPercentageofStudentsatEachPerformanceLevelforPanelistRecommendedPerformanceStandards,OverallandbyGenderandEthnicity.........................N1AppendixOSummaryofPanelistEvaluations..........................................................................O1AppendixPIndependentObserverReporttoStateBoardofEducation.................................P1AzMERIT StandardSettingTechnicalReport 1American Institutes for Research Introduction In2010,ArizonaadoptednewacademiccontentstandardsinEnglishlanguagearts(ELA)andmath.TheArizonaCollegeandCareerReadyStandardsaredesignedtoensurethatstudentsacrossgradesarereceivingtheinstructiontheyneedtobeontrackforcollegeandcareerbythetimetheygraduate.Inspring2015,theArizonaDepartmentofEducation(ADE)administeredforthefirsttimeArizonasMeasurementofEducationalReadinesstoInformTeaching(AzMERIT)toassessproficiencyonthenewArizonaCollegeandCareerReadyStandards.TheAzMERITmeasuresEnglishlanguageartsingrades311,andmathingrades38andfollowingcompletionofhighschoolcourseworkinAlgebraI,Geometry,andAlgebraII.TheAzMERITisaseriesoffixedformassessmentsthatareintendedtobeadministeredonline,althoughtheassessmentisofferedasadualmode,onlineandpaper,assessmenttoaccommodateschoolsthatarenotreadytotransitiontotheonlinetestingenvironment.Acommonoperationalbaseformwasadministeredtoallstudentswithinagiventestgradeandsubject.Eachassessmentiscomprisedoftwotothreediscretetestsessions.ThefirstoperationaladministrationoftheAzMERITassessmenttookplaceinspring2015.OnlineadministrationoftheAzMERIToccurredfromMarch30throughMay8,2015.ThepaperversionoftheAzMERITwasadministeredbetweenApril13andApril24,2015.Followingthecloseofthetestadministrationwindows,theAmericanInstitutesforResearch(AIR),undercontracttoADE,convenedeightpanelsofArizonaeducatorstorecommendperformancestandardsontheassessments.Thisdocumentdescribestheproceduresusedtoconductthestandardsettingworkshopsaswellastherecommendedperformancestandardsandresultingimpacts.Performance Standards and Validity of Test Score Interpretations Validityreferstothedegreetowhichtestscoreinterpretationsaresupportedbyevidence,andspeaksdirectlytothelegitimateusesoftestscores.Establishingthevalidityoftestscoreinterpretationsisthusthemostfundamentalcomponentoftestdesignandevaluation.TheStandardsforEducationalandPsychologicalTesting(AmericanEducationalResearchAssociation,AmericanPsychologicalAssociation,andNationalCouncilonMeasurementinEducation,2014)provideaframeworkforevaluatingwhetherclaimsbasedontestscoreinterpretationsaresupportedbyevidence.Withinthisframework,theStandardsdescribetherangeofevidencethatmaybebroughttobeartosupportthevalidityoftestscoreinterpretations.1Thekindsofevidencerequiredtosupportthevalidityoftestscoreinterpretationsdependcentrallyontheclaimsmadeforhowtestscoresmaybeinterpreted.Moreover,thestandardsmakeexplicitthatvalidityisnotanattributeoftests,butrathertestscoreinterpretations.Sometestscoreinterpretationsmaybesupportedbyvalidityevidence,whileothersarenot.1ResponsivetoStandardsforEducationandPsychologicalTesting:Standard9.13AzMERIT StandardSettingTechnicalReport 2American Institutes for Research Thus,thetestitselfisnotconsideredvalid,butratherthevalidityoftheintendedinterpretationanduseoftestscoresisevaluated.Centraltoevaluatingthevalidityoftestscoreinterpretationsisdeterminingwhetherthetestmeasurestheintendedconstruct.Suchanevaluationinturnrequiresacleardefinitionofthemeasurementconstruct.ForArizonasnewAzMERITassessments,thedefinitionofthemeasurementconstructisprovidedbytheArizonaCollegeandCareerReadyStandards.TheArizonaCollegeandCareerReadyStandards(ACCRS)specifywhatstudentsshouldknowandbeabletodobytheendofeachgradelevelinorderforstudentstograduatereadyforpostsecondaryeducationorentryintotheworkforce.BecausedirectlymeasuringstudentachievementagainstofeachbenchmarkintheACCRSwouldresultinanimpracticallylongtest,eachtestadministrationisdesignedtomeasurearepresentativesampleofthecontentdomaindefinedbytheStandards.ToensurethateachstudentisassessedontheintendedbreadthanddepthoftheStandards,testformconstructionisguidedbyasetoftestspecifications,orblueprints,whichindicatethenumberofitemsthatshouldbesampledfromeachcontentstrand,standard,andbenchmark.Thus,thetestblueprintsrepresentapolicystatementabouttherelativeimportanceofcontentstrandsandstandardsinadditiontomeetingimportantmeasurementgoals(e.g.,sufficientitemstoreportstrandperformancelevelsreliably).BecausethetestblueprintdetermineshowstudentachievementoftheArizonaCollegeandCareerReadyStandardsisevaluated,alignmentoftestblueprintswiththecontentstandardsiscritical.ADEhaspublishedtheAzMERITtestblueprintsthatspecifythedistributionofitemsacrossreportingstrandsanddepthofknowledgelevels.AlignmentoftestcontenttotheArizonaCollegeandCareerReadyStandards(ACCRS)2ensuresthattestscorescanserveasvalidindicatorsofthedegreetowhichstudentshaveachievedthelearningexpectationsdetailedintheACCRS.However,theinterpretationoftheAzMERITtestscoresrestsfundamentallyonhowtestscoresrelatetoperformancestandardswhichdefinetheextenttowhichstudentshaveachievedtheexpectationsdefinedintheACCRS.AzMERITtestscoresarereportedwithrespecttofourproficiencylevels,demarcatingthedegreetowhichArizonastudentshaveachievedthelearningexpectationsdefinedbytheArizonaCollegeandCareerReadyStandards.ThecutscoreestablishingtheProficientlevelofperformanceisthemostcritical,sinceitindicatesthatstudentsaremeetinggradelevelexpectationsforachievementoftheArizonaCollegeandCareerReadyStandards,thattheyarepreparedtobenefitfrominstructionatthenextgradelevel,andthattheyareontracktopursuepostsecondaryeducationorentertheworkforce.ProceduresusedtoadoptperformancestandardfortheAzMERITassessmentsarethereforecentraltothevalidityoftestscoreinterpretations.FollowingthefirstoperationaladministrationoftheAzMERITassessmentsinspring2015,astandardsettingworkshopwasconductedtorecommendtotheArizonaStateBoardofEducationasetofperformancestandardsforreportingstudentachievementoftheArizonaCollegeandCareerReadyStandards.ThisdocumentdescribesthestandardizedandrigorousproceduresthatArizonaeducators,servingasstandardsettingpanelists,followedto2ResponsivetoStandardsforEducationandPsychologicalTesting:Standard12.8and12.10AzMERIT StandardSettingTechnicalReport 3American Institutes for Research recommendperformancestandards.TheworkshopsemployedtheBookmarkprocedure,awidelyusedmethodinwhichstandardsettingpanelistsusetheirexpertknowledgeoftheArizonaCollegeandCareerReadyStandardsandstudentachievementtomaptheperformanceleveldescriptorsadoptedbytheArizonaStateBoardofEducationontoanordereditembookbasedonthefirstoperationaltestformadministeredtostudentsinspring2015.Panelistswerealsoprovidedwithcontextualinformationtohelpinformtheirprimarilycontentdrivencutscorerecommendations.PanelistsrecommendingperformancestandardsforthehighschoolassessmentswereprovidedwithinformationabouttheapproximatelocationoftherelevantACTcollegereadyperformancestandardforthegrade11ELAandAlgebraIIassessments,andProgrammeforInternationalStudentAssessment(PISA)performancestandardsforthegrade10ELAandGeometryassessments.Panelistsrecommendingperformancestandardforthegrade38summativeassessmentswereprovidedwiththeapproximatelocationofrelevantNAEPperformancestandardsatgrades4and8,aswellasinterpolatedvaluesforgrade6.PanelistswereprovidedwiththeapproximatelocationsoftheSmarterBalancedperformancestandardsforthegrade38and11assessmentsinELAandmathtoprovideadditionalcontextaboutthelocationofperformancestandardsforstatewideassessments.Additionally,panelistswereprovidedthecorrespondinglocationsforthepreviousAIMSperformancestandards.Panelistswereaskedtoconsiderthelocationofthesebenchmarklocationswhenmakingtheircontentbasedcutscorerecommendations.Whenpanelistsareabletousebenchmarkinformationtolocateperformancestandardsthatconvergeacrossassessmentsystems,validityoftestscoreinterpretationsisbolstered.Inaddition,panelistswereprovidedwithfeedbackabouttheverticalarticulationoftheirrecommendedperformancestandardssothattheycouldviewhowthelocationsoftheirrecommendedcutscoresforeachgradelevelassessmentsatinrelationtothecutscorerecommendationsattheothergradelevels.Thisapproachallowedpaneliststoviewtheircutscorerecommendationsasacoherentsystemofperformancestandards,andfurtherreinforcestheinterpretationoftestscoresasindicatingnotonlyachievementofcurrentgradelevelstandards,butalsopreparednesstobenefitfrominstructioninthesubsequentgradelevel.Basedontherecommendedcutscores,Table1showstheestimatedpercentageofstudentsmeetingtheAzMERITproficientstandardforeachassessmentinspring2015.Table1alsoshowstheapproximatepercentageofArizonastudentsthatwouldbeexpectedtomeettheACTcollegereadystandard,andthepercentageofArizonastudentsmeetingtheNAEPproficientstandardsatgrades4and8.Table1alsopresentstheexpectedproficientratefortheSmarterBalancedAssessments,systemwide,basedonthespring2014fieldtestadministration.AsTable1indicates,theperformancestandardsrecommendedAzMERITassessmentsarequiteconsistentwithrelevantACTcollegeready,andtheNAEPandSmarterBalancedproficient,benchmarks.Moreover,becausetheperformancestandardswereverticallyarticulated,theproficiencyratesacrossgradelevelsaregenerallyconsistent.AzMERIT StandardSettingTechnicalReport 4American Institutes for Research Table1.EstimatedPercentageofStudentsMeetingAzMERITandBenchmarkProficientStandards.PercentofStudentsMeetingStandardAssessmentAzMERITProficientArizonaACTCollegeReadyArizonaNAEPProficientProjectedSBACELAGrade3 41% 38%Grade4 38% 28% 41%Grade5 30% 44%Grade6 34% 41%Grade7 33% 38%Grade8 32% 28% 41%Grade9 27% Grade10 30% Grade11 25% 34% 41%MathematicsGrade3 42% 39%Grade4 42% 42% 38%Grade5 40% 33%Grade6 32% 33%Grade7 31% 33%Grade8 33% 32% 32%AlgebraI 32% Geometry 30% AlgebraII 29% 36% 33%AzMERIT StandardSettingTechnicalReport 5American Institutes for Research Overview of Standard Setting Approach TheBookmarkmethod(Mitzel,Lewis,Patz,&Green,2001)wasusedtorecommendperformancestandardsfortheAzMERIT.ADEpreviouslyusedtheBookmarkmethodtorecommendperformancestandardsfortheAIMSassessment.TheBookmarkmethodwasimplementedintworounds,providingpanelistswithbenchmarkinformationpriortoRound1andpanelistfeedbackandimpactdatapriortoRound2.Tofacilitateverticalarticulationofperformancestandardsacrossgrades,workshoppanelistsbeganbyrecommendingperformancestandardsforgrades4,6,8,10,and11(GeometryandAlgebraIIformath),followingstandardBookmarkprocedures.Fortheremainingintermediategrades,followingaverticalmoderationsessiontoarticulateperformancestandardsacrossgrades,panelistswereprovidedwithinterpolatedperformancestandardsbasedontherecommendedstandardsfromtheanchorgrades.Fortheintermediategrades,thejudgmenttaskusedbypanelistswasmodifiedsomewhat.Foreachperformancestandard,panelistswereaskedtoexaminetheitemontheinterpolatedpageandjudgewhetherstudentswhojustbarelyaredescribedbytheperformanceleveldescriptorcouldrespondsuccessfullytotheitem,andifso,toendorsetheinterpolatedOIBpageastheperformancestandard.IftheycouldnotendorsetheinterpolatedOIBpageastheperformancestandard,panelistswereaskediftheycouldlocateanitemverynearthelocationoftheinterpolatedOIBpagethatstudentsjustbarelymeetingthestandardcouldrespondtosuccessfully.Panelistsweretaskedwithrecommendingthreeperformancestandards(PartiallyProficient,Proficient,andHighlyProficient)thatresultedinfourperformancelevels(MinimallyProficient,PartiallyProficient,Proficient,andHighlyProficient).Workshop Design TorecommendperformancestandardsforeachoftheAzMERITassessments,ADEconvenedeightpanelsrepresentingfourgradebands(34,56,78,and911)foreachsubject.Thepanelsconsistedofeducatorsfromtherespectivegradebandsandcontentareas.Thepanelistsrecommendedperformancestandardsbasedprimarilyoncontentconsiderationswithadditionalcontextprovidedbyrelevantbenchmarkinformationfromstatewide(SBAC),national(NAEP),international(PISA),andcollegeentrance(ACT)exams,aswellasestimatedstudentperformanceontherecommendedstandardspriortoRound2.PanelistsusedOrderedItemBooklets(OIBs)andPerformanceLevelDescriptors(PLDs)toplaceperformancestandardsforallthreeperformancelevels,PartiallyProficient,Proficient,andHighlyProficient,intworounds.Firstpanelistsrecommendedperformancestandardsfortheanchorgrades,4,6,8,11/AlgebraII(grade10/Geometrywerealsoconsideredanchorgrades).Afterrecommendingperformancestandardsfortheanchorgrades,amoderationsessionwasconductedwiththetableleadersfromeachofthepanelstoreviewtheverticalarticulationoftheperformancestandards,andtoimplementanyadjustmentstotheanchorgraderecommendationstofacilitateverticalarticulation.Followingtheverticalarticulationsession,panelistscontinuedontorecommendperformancestandardsfortheremaininggradelevelassessments,usingtheAzMERIT StandardSettingTechnicalReport 6American Institutes for Research interpolatedstandardstoprovidefurthercontextualinformationaboutthelikelylocationofperformancestandards.TheAzMERITStandardSettingworkshopswereconductedoverfourdays,withthehighschoolpanels,whichhadtorecommendperformancestandardsforthreeassessments,beginningonMonday,andtheremaininggradelevelpanelsconveningonTuesday.AbroadoverviewoftheworkshopcalendarispresentedinTable2.DetailedagendasforthestandardsettingworkshopsareincludedasAppendixA.Table2.CalendarDatesfor2015GradeLevelandHighSchoolELAandMathStandardsettingWorkshopsWorkshop Monday,July13 Tuesday,July14 Wednesday,July15 Thursday,July16GradeLevel N/AStandardSettingDay1StandardSettingDay2StandardSettingDay3HighSchoolStandardSettingDay1StandardSettingDay2StandardSettingDay3StandardSettingDay4Theworkshopsbeganwithabrieftableleaderorientationtoreviewwithtableleaderstheirroleandresponsibilities.Theworkshopproperbeganwithalargegrouptrainingtoprovidepanelistswithanoverviewoftheworkshopactivitiesandinitialtraininginthebookmarkingprocedures.Followingthelargegroupsession,theworkshoppanelsconvenedintheirmeetingrooms,andbegantheirworkbyparticipatinginthesameAzMERITonlineassessmentthatwasadministeredtotheirstudentsinthespring.Paneliststhenspentseveralhoursworkingthroughtheperformanceleveldescriptors(PLDs)developedbyADE,anddevelopingmodifieddescriptorstocharacterizethespecialsubsetofstudentswhojustbarelyqualifyforentryintoeachoftheperformancelevels.Afterdevelopingdescriptorsforthejustbarelystudents,panelistsspenttheremainderofdayonereviewingtheirordereditembooks(OIBs).Panelistsdidnotbeginrecommendingperformancestandardsuntildaytwo,whichbeganwithtrainingonthebookmarkplacementtask.PaneliststhenworkedthroughtheirOIBsandplacedtheirbookmarksforRound1.AfterRound1,panelistswereprovidedfeedbackaboutthebookmarkplacementsoftheotherpanelistsanddiscussedthosebookmarkplacementsattheirtablesandacrosstheroommoregenerally.Panelistswerethenalsoprovidedwithimpactdatashowingtheestimatedpercentageofstudentswhowouldmeeteachoftheperformancestandardsandengagedinpaneldiscussionsaboutanyimplicationsofthoseproficiencyrates.Uponcompletionofpaneldiscussions,panelistsmadeasecondroundofbookmarkplacements,andthenbegantheprocessoveragainforthesubsequentassessment.AzMERIT StandardSettingTechnicalReport 7American Institutes for Research Workshop Location TheworkshopswereheldattheHyattRegency,locatedat122North2ndStreetinPhoenix,Arizona.ThelocationprovidedmeetingspacestoholdtheAzMERITworkshoppanels,aswellasapsychometricworkroomforcompletionofanalysisactivitiesandstoragespaceforsecurematerialsthroughouttheworkshop.Workshop Staffing Aseniorworkshopcoordinatorwastaskedwithleadingthecrossworkshopintroductorytrainingandverticalmoderationmeetings,andwasresponsibleforworkingwitheachfacilitatorandmonitoringtheflowofactivitiesacrossworkshops.AIRtestdevelopmentstaffservedasworkshopfacilitators,leadingeachpanelthroughtrainingactivitiesandexecutionofthestandardsettingprocess.Additionally,anAIRresearchassistantwasassignedtoeachpaneltosupporttheworkshopfacilitator.Becausetestdevelopmentstaffservedasworkshopfacilitators,theywerehighlyqualifiedtofacilitatethedevelopmentofjustbarelyperformanceleveldescriptors,andtoserveasasubjectmatterresourceforpanelistsastheynavigatedtheOIB.AteamofthreeAIRpsychometriciansmanagedpsychometricactivitiesinsupportoftheworkshop,includingensuringaccuratedatacaptureofbookmarkplacements,presentationofverticalarticulationresultsformoderationmeetings,andproductionoffinalresultsforthestandardsettingtechnicalreport.Inaddition,AIRprojectstafffacilitatedorganizationofmeetingspaceandmealsandprovidedsupporttopanelistsasnecessary.ADEstaffmonitoredallstandardsettingactivities,andalsoaddressedanypolicyortestdevelopmentquestionsforpanelists.WhileADEstaffansweredspecific,directquestions,theywerenotactivelyinvolvedinthefacilitationofthemeeting.Workshop Panelists ADEworkedtoobtainbroadlyrepresentativepanelsforthestandardsettingworkshopsthatreflectedtheteacherpopulationinthestateofArizonaintermsofgender,race,ethnicity,andgeographicalrepresentation.Diversegroupsofpanelistsbringawiderangeofperspectivesandexperiencetothestandardsettingeffort,ensuringthattherecommendationsthatareforwardedtotheStateBoardofEducationarethoughtfulandrepresentativeofbroadeducationalconstituencies,andrepresenttherangeofexpertiseandexperiencesfoundintheeducatorpopulationacrossthestate.WithineachoftheELAandmathpanels,atotalof12panelistspergradebandsubpanelwererecruitedtorecommendstandards.ADEtargetedthenumberofmaleandfemalepaneliststomirrorthepopulationofeducators.Inthesameway,ADEworkedtoincludeproportionalrepresentationofAmericanIndian/NativeAmerican,Asian/PacificIslanders,Black(NonHispanic),HispanicandWhite(NonHispanic)panelists,andaproportionalnumberofpanelistsfromrural,urban,andsuburbandistricts.Forcoursebasedassessmentsinmaththatrequirespecificcontentexpertise,ADEsoughttoincludeteacherswhohaveexpertiseinthecontentstandardsandcourseworkforallthreeareastheyrecommendedperformancestandardsfor:AzMERIT StandardSettingTechnicalReport 8American Institutes for Research AlgebraI,Geometry,andAlgebraII.Inaddition,ADEworkedtoincludespecialeducationandEnglishLanguageLearners(ELL)teachers.Withineachsubpanel,tableswerebalancedtoincludepanelistswithvaryingcontentexpertiseanddemographicrepresentationineachgroup.ADEdesignatedthreetableleadersforeachpanel.Tableleadersattendedanadditionalorientationmeetingandweretaskedwithassistingstandardsettingstaffbyfacilitatingdiscussionswithintheirtable;distributingandcollectingreadinessandrecordingsheetsandsecurematerials;alertingworkshopstaffofconfusionorconcernswithintheirtables;andrepresentingtheirtableandpanelduringverticalarticulationmeetings.Letterscontaininglogisticalinformationandremindersaboutthepurpose3oftheworkshopwereemailedtoconfirmedpaneliststwoweekspriortothestandardsettingworkshop.Intheweekprior,testingcontractorstaffcontactedallpanelistsviaphonetoconfirmreceiptofinformation.Throughouttheprocess,ADEcontinuedtorecruitreplacementsforpanelistswhowithdrewtheirparticipation.AppendixB4presentsthecompositionofthestandardsettingpanels.Foreachpanel,thetableincludesarecordforeachpanelistandindicatesthegeographicregionheorsherepresentsandhisorhergender,ethnicity,andmainexpertise.Whileitiscriticallyimportanttoincludearangeofstakeholdersinthestandardsettingprocess,experiencehasshownthatitisessentialforpaneliststohavedirectknowledgeofacademicstandardsandstudentgradelevelperformancetoparticipatemeaningfullyintheBookmarkingprocedure.Forthisreason,panelparticipationwasrestrictedtoclassroomteachersandcurriculumspecialistswithexpertiseinELAandmathcurriculumandinstruction.Higher Education Panel Priortothestandardsettingworkshops,ADEengagedahighereducationpanelintwoactivitiesintendedtosupporttheassertionthatstudentswhoachievetheProficientlevelonAzMERITinELA11andAlgebraIIareontracktobecollegereadyupongraduationfromhighschool.Thishighereducationpanelincluded10participantsrepresentingallthreeofArizonaspublicuniversitiesandthreeofthestatescommunitycollegesystems.EachwasfamiliarwiththerequirementsforstudentstobesuccessfulineithercreditbearingentrylevelcollegemathematicscoursesorcreditbearingentrylevelcollegeEnglishcourses.3ResponsivetoStandardsforEducationandPsychologicalTesting:Standard5.0,5.21,5.22,and7.04ResponsivetoStandardsforEducationandPsychologicalTesting:Standard7.5AzMERIT StandardSettingTechnicalReport 9American Institutes for Research ThefirstactivityforthispanelwasareviewofthedetailedPLDsforELA11andAlgebraIIheldatADEsofficesonMay13,2015.Tosetthestageforthisactivity,anoverviewofAzMERIT,thedetailedPLDs,andhowthedetailedPLDswouldbeusedintheupcomingAzMERITStandardSettingwasprovidedtoall10participants.ThepanelthenbrokeoutintoseparateELAandmathgroupstofirstdeterminethecollegecoursethatbestfitsthedescriptorcreditbearingentrylevelcollegecoursefortheircontentarea.ForELA,thatentrylevelcoursewasdeterminedtobeFreshmanComposition,whiletheentrylevelcourseformathematicswasdeterminedtobeCollegeMath.Whilestillintheirsubjectareagroups,thepaneliststhenreviewedanddiscussedtheskillsandabilitiesdescribedintheELA11orAlgebraIIdetailedPLDsforstudentsintheProficientlevelandwhetherthatlevelofskillorabilitywassufficienttobepreparedforentrylevelcoursework.TheconsensusdecisionofboththeELAgroupandthemathgroupwasthatstudentswhohadtheskillsandabilitiesdescribedintheProficientlevelwouldbeadequatelypreparedforthetargetentrylevelcourseupongraduation.Additionally,boththeELAgroupandthemathgroupfeltitwasimportanttoindicatethattheirendorsementofcollegereadinessincludedtheexpectationthatstudentswouldtakeonemoreyearofhighschoolEnglishaftertheELA11testandonemoreyearofhighschoolmathaftertheAlgebraIItest.ThisisnotanunreasonableexpectationsincemoststudentswouldbetakingtheELA11testattheendoftheirthirdoffourrequiredhighschoolEnglishcoursesandwouldbetakingtheAlgebraIItestattheendoftheirthirdoffourrequiredhighschoolmathematicscourses.ThesecondactivityforthispanelwasareviewoftheitemsincludedintheELA11andAlgebraIItesttodeterminewhichitemsdemonstratedtheskillsandabilitiesneededforstudentstobeadequatelypreparedforentrylevelcoursework.Toaccommodatevacationschedules,panelistsparticipatedinthisonlineactivityindividuallyatthetimeandlocationoftheirchoosinginearlyJuly.ThisonlineactivityincludedatrainingmodulefollowedbyanitemreviewbasedonavariationontheItemDescriptorMatchingprocedure(Ferrara,Perie,&Johnson,2008).LikethebookmarkingprocedureusedtorecommendperformancestandardsforAzMERIT,theIDMatchingprocedurereliesonanordereditembook(OIB).Thisbookcontainstestitemsthatappearinorderfromeasiesttomostdifficult,basedonstudentperformanceinthespring2015testadministration.ThevariationoftheIDMatchingprocedureusedforthisactivityaskedthepaneliststodeterminewhethertheknowledgeandskillsnecessarytoanswereachitemcorrectlywereprerequisiteskillsforsuccessinentrylevelcoursework,thatis,CollegeMathorFreshmanComposition.HighereducationpanelistsbeganbyreviewingtheOIBfollowingthesameproceduresusedbythestandardsettingworkshoppanelists.BeginningwiththefirstpageintheOIB,participantsansweredtwoquestionsastheyreviewedeachitem:Whatdoesastudentneedtoknowandbeabletodotosuccessfullyrespondtothisitem?Whyisthisitemmoredifficultthantheprecedingitems?AzMERIT StandardSettingTechnicalReport 10American Institutes for Research Thisactivitywasdesignedtofocusparticipantsontheknowledgeandskillsmeasuredbyeachitem,aswellastocommunicatetoparticipantsthefullrangeofknowledgeandskillsmeasuredintheassessments.UponcompletionoftheOIBreview,participantswerepreparedtoperformtheIDMatchingtask.ToperformtheIDMatchingtask,participantswereaskedtoconsiderwhatknowledgeandskillsareprerequisiteforsuccessinentrylevelcourseworkattheirrespectiveinstitutions.ParticipantsrepresentingCollegeMathperformedtheIDMatchingtaskfortheAlgebraIIOIB,whileparticipantsrepresentingFreshmanCompositionperformedtheIDMatchingtaskusingtheGrade11ELAOIB.ToperformtheIDMatchingtask,participantsjudgedwhethertheknowledgeandskillsnecessarytoanswertheitemsuccessfullywereprerequisitetosuccessintherelevantentrylevelcourse.ForeachitemintheOIB,participantsansweredyesornothattocorrectlyanswertheitemrequiredknowledgeandskillsthatareprerequisiteforsuccessintheentrylevelcourse.Becauseitemswereorderedbydifficulty,theexpectationwasthatparticipantswouldgenerallyidentifytwodistinctregionsofachievement,aloweronewhereitemsclearlywereprerequisiteforsuccessinentrylevelcollegecoursework,andahigheronethatreflectedachievementbeyondwhatwouldbeconsideredprerequisiteknowledgeforsuccessincollege.Itwasalsoexpectedthattherewouldbearegionofuncertaintybetweenthetwo,withthenotionthatalikelycollegereadyperformancestandardwouldliewithintheregionofuncertainty.Whentheresponsesoftheparticipantsweretabulatedtogetherhowever,therewasnodetectableregionintheOIBwheretheknowledgeandskillsassessedbytheitemswerereliablynotprerequisiteforsuccessinentrylevelcoursework.Inotherwords,itemsconsideredprerequisiteforcollegesuccesswerereliablyidentifiedacrosstheentirerangeoftheOIB.Therefore,itwasnotpossibletoprovidethestandardsettingworkshoppanelistswithaconstrainedregioninwhichacollegereadystandardmightbeidentifiedbaseduponthereviewbythispanelofArizonahighereducationrepresentatives.Workshop Training Thoroughtrainingisanessentialelementofastandardsettingworkshop.Trainingatthemeetingshelpedpanelistsbecomefamiliarwiththeassessmentsystemandthestandardsettingprocess.Italsoinvolvedareviewanddiscussionoftheassessments,thestudentpopulationsthatparticipatedineach,andtheperformanceleveldescriptors(PLDs).Inaddition,trainingincludedindepthdiscussionofconceptskeytobookmarkplacement,suchasthenotionofwhatwouldconstituteastudentjustbarelyinaperformancelevel.Allpanelistswereadministeredanoperationaltestinordertounderstandthetestcontent,thetestinginterface,andvariousitemtypesthroughwhichstudentknowledgeandskillswereassessed.AAzMERIT StandardSettingTechnicalReport 11American Institutes for Research sampleofthepresentationslidesusedtoconducttheintroductorytraining,andthoseusedtofacilitateeachworkshopareprovidedinAppendixC5.Tobegintheworkshop,thepanelistswereconvenedforabriefintroductorytrainingthatfocusedonthepurposeofthestandardsettingworkshopandareviewofthemainworkshopactivities.Followingthislargegroupintroduction,panelistsjoinedtheirassignedworkshoppanelswheretheworkshopleaderforeachassessmentguidedpaneliststhroughthestandardsettingactivitiesandprovidedindepthtrainingthroughoutthecourseoftheworkshop.Tableleadershadtheadditionalresponsibilitiesofensuringthattableactivitiesremainedfocusedonthetaskathand,helpingtoverifythatpanelistsunderstoodtheirtasks,andalertingworkshopleaderstoanyissuesencounteredbypanelistsastheyengagedintheirworkshoptasks.Tableleaderswerenotexpectedtoprovidetrainingtopanelistsbutratherserveasliaisonsbetweenthepanelistsandworkshopleaderstoensurethatworkshopactivitieswereimplementedcorrectly,alertingworkshopleaderstoanyissuesthataroseduringthecourseofconductingworkshopactivities,andrepresentingtheirtablesinthecrosspanelmoderationdeliberations.Atableleaderorientationmeetingwasconvenedpriortothestandardsettingworkshoptofamiliarizetableleaderswiththeirrolesandresponsibilities,includingsuggestionsonhowtoprovideleadershipatthetablesduringthestandardsettingprocessandhowtomanagethesecurematerials. 5ResponsivetoStandardsforEducationandPsychologicalTesting:Standard7.5AzMERIT StandardSettingTechnicalReport 12American Institutes for Research Standard Setting Materials and Procedures Performance Level Descriptors Performanceleveldescriptors(PLDs)definethecontentareaknowledgeandskillsthatstudentsateachperformancelevelareexpectedtodemonstrate.ThestandardsettingpanelistsbasedtheirjudgmentsaboutthelocationoftheperformancestandardsonthePLDsaswellastheArizonaCollegeandCareerReadinessStandards.Priortoconveningthestandardsettingworkshops,AIR,inconsultationwithADE,draftedPLDsforeachtestthatdescribedtherangeofachievementencompassedbyeachperformancelevelonthetest.ThePLDsweredesignedtobeclear,concrete,andreflectArizonasexpectationsforproficiencybasedontheArizonaCollegeandCareerReadyStandards.FollowingacycleofrevisionstothedraftPLDs,ADEinvitedArizonaeducatorstoreviewPLDsforeachoftheassessments.Basedonfeedbackfrom166educators,PLDswerefurtherrevised,andtheresultingdraftswereusedbystandardsettingpanelists.ADEconsideredanyneedforclarificationorrevisionthatarosethroughoutthestandardsettingprocesspriortopublishingthefinalversionsofthePLDsfollowingthestandardsettingworkshop.PerformanceleveldescriptorsthatwereusedbypanelistsinthestandardsettingworkshoparepresentedinAppendixD.Centraltotheirtraininginthebookmarkmethod,panelistsusedthePLDstodeveloparepresentationofstudentswhoarejustbarelydescribedbyeachoftheperformanceleveldescriptors.Duringthistrainingtask,panelistslearnedthatwhilePLDsarewrittentocharacterizetypicalmembersofeachperformancelevel,theirbookmarkplacementswouldbedirectedtowardcharacterizingandidentifyingthemostminimallyqualifiedmembersofeachperformancelevel.Characterizingjustbarelymeetsstudentsisnotanintuitivejudgmentandpanelistsworkedtoidentifytheminimumcharacteristicsofstudentachievementforentryintoeachperformancelevel.EachpanelproducedajustbarelyPLDtohelpguidetheirdiscussionsandbookmarkplacements.Todevelopacommonunderstandingamongpanelists,eachpanelwasaskedto1.reviewandparseperformanceleveldescriptors;2.discusscharacteristicsofstudentsclassifiednearthresholdsofperformancestandards;3.identifythecharacteristicsthatdistinguishstudentsjustabovetheperformancestandardfromthosejustbelow;4.determinewhatevidencewasnecessarytoconcludethatastudentpossessedtheminimumknowledgeandskillsneededtomeettheperformancestandard;and5.summarizeknowledgeandskillsofstudentswhojustbarelymeeteachperformancestandard,orarejustbarelydescribedbyeachperformanceleveldescriptorAzMERIT StandardSettingTechnicalReport 13American Institutes for Research Thesediscussionsyieldedcommondescriptionsofstudentsjustbarelycharacterizedbyeachperformanceleveldescriptorwithineachroom.Ordered Item Booklet Followingreviewofperformanceleveldescriptorsanddevelopmentofjustbarelyperformanceleveldescriptors,panelistsreviewedordereditembooklets(OIBs).AnOIBisacollectionoftestitemsorderedfromeasiesttomostdifficult.EachpageintheOIBcorrespondstoalevelofachievementontheAzMERIT,andpanelistsusetheOIBtorecommendtheminimumlevelofachievementrequiredtoenterintoeachperformancelevel.Composition of OIB WithineachELAandmathtest,allonlinetesttakerswereadministeredatestformwithacommonsetofitemsusedforoperationalscoring,aswellasasetofembeddeditemsusedforlinkingorfieldtesting.Theoperationaltestformwasalsoadministeredonpaperwithitemsubstitutionsforafewtechnologyenhanceditemsthatcouldnotberepresentedonpaper.Theoperationalitemsadministeredonlineservedasthebasisfortheordereditembook.Tominimizegapsintheordereditembooklets,theOIBswereaugmentedbyadditionalfieldtestitemstomorefullyrepresenttherangeofacademicachievementencompassedwithinthoseitembanks.EachmathOIBwasaugmentedwith1021fieldtestitems,andeachELAOIBwasaugmentedwith712fieldtestitems.AllfieldtestitemsselectedforinclusionintheOIBwerereviewedforstatisticalintegrity;itemsflaggedforfurtherreviewduetolowdiscriminationwereexcludedfromtheOIB.ItisimportanttonotethateachOIBwasaugmentedwithrespecttotheassessmentblueprint,whichspecifiesthecompositionofeachtestwithrespecttotherangeofcontentassessedbyeachoperationalform.TheaugmentedELAandmathOIBswereproportionaltotheoperationaltestblueprints;theblueprintsarepresentedinAppendixE6.Increasingthenumberofitemsacrosstherangeofitemdifficultiesprovidespanelistswithgreatercontexttoidentifyimportantshiftsintheknowledgeandskillrequirementsoftestitems.Oftenpanelistsbecomefocusedonthecognitivedemandsofasingleitemwhendeliberatingonthelocationofaperformancestandard.Thispropensityisexacerbatedwhentherearerelativelyfewitemsinagivenlocation,whichcancausejudgmentaboutoneitemtotakeontoomuchimportance.Evenwhentherearesufficientitemstoestablishreliableperformancestandardsforacentralproficientperformancestandard,therearetypicallyfeweritemsavailableinlocationsassociatedwithperformancestandardscategorizingachievementbelowandaboveproficient;thus,movementofthebookmarkbyevenapageortwomayresultinverylargeincreasesordecreasesinthepercentageofstudentsmeetingthestandard.AugmentingtheOIBmoderatestheimpactassociatedwitheachOIBpage,especiallyforperformancestandardsinthetailsoftheabilitydistribution.6ResponsivetoStandardsforEducationandPsychologicalTesting:Standard7.1and12.4AzMERIT StandardSettingTechnicalReport 14American Institutes for Research Itemswereorderedaccordingtotheirresponseprobability(RP)levelbasedontheirItemResponseTheory(IRT)parameters.InIRT,theitemcharacteristiccurveforeachitemindicatesthelikelihoodofrespondingcorrectlyforeachpointalongthestudentachievementdimension.Theresponseprobabilitycriterionreferstothelocationontheachievementscalethatcorrespondstoagivenprobabilityofsuccess.Incontextofthestandardsettingworkshop,thiscriterionisusedtodevelopacommonunderstandingofwhatconstitutesmasterywhenevaluatingwhetherastudentcanrespondsuccessfullytoanitem.AnRPvalueof0.67wasusedasthemasterycriterionforallofstandardsettingworkshopsexceptthehighschoolendofcourseassessmentsinmath.Panelistswereaskedtoconsiderwhether,forexample,ajustbarelyproficientstudenthada0.67likelihoodofansweringtheitemcorrectly.Theywerealsoencouragedtoaskthisquestioninotherrelatedways,includingwhetherofjustbarelyproficientstudentswouldanswertheitemcorrectly,orwhetherajustbarelyproficientstudentwouldrespondcorrectlytoitemtwoofthreetimes.Theendofcoursemathtestswereverydifficultandthenumberofitemsonwhichstudentscoulddemonstratethatlevelofmasterywasquitelow,resultinginaveryshortfunctionalOIB.Thus,anRPvalueof0.50wasadoptedfortheEOCmathtests,meaningthatajustbarelyproficientstudent,forexample,hada0.50likelihoodofrespondingcorrectly,orthatofjustbarelystudentscouldrespondsuccessfullytoanitem,orthatajustbarelystudentcouldrespondsuccessfullytotheitematleastoneoftwotimes.Dichotomouslyscored(e.g.,incorrectvs.correct)AzMERITitemswerecalibratedusingtheRaschmodel.Multipoint,partialcredititemswerecalibratedusingMasterspartialcreditmodelwithorderingofscorepointpagesintheOIBbasedonstepleveldifficulties.Theordereditembookletswerepresentedonline,allowingpaneliststoviewitemsinthesamecontextasstudenttesttakers.ThecompositionoftheELAandmathordereditembookletsbyassessmentandgradearesummarizedinTable3below.AtechnicalsummaryoftheOIBsarepresentedinAppendixF,includingforeachpageintheOIB,theitemscorepointassociatedwiththepresenteditem,thedifficultyrepresentedbythepage,andthestandarderrorofthedifficulty.Inaddition,theappendixindicatestheoverallpercentofstudentswhowouldscoreatorabovethestandardassociatedwitheachOIBpage,andthelocationofexternalbenchmarkswithinthebooklet.Table3.CompositionofOrderedItemBooklets NumberofItemsInOIB PagesinOIB(TotalPoints) Test Operational FieldTest TotalELA3 42 9 51 67ELA4 42 8 50 67ELA5 42 10 52 68ELA6 42 7 49 68ELA7 42 10 52 68ELA8 42 12 54 71ELA9 44 11 55 69AzMERIT StandardSettingTechnicalReport 15American Institutes for Research NumberofItemsInOIB PagesinOIB(TotalPoints) Test Operational FieldTest TotalELA10 44 9 53 68ELA11 44 9 53 67Math3 45 13 58 58Math4 45 19 65 66Math5 45 20 65 66Math6 47 10 57 57Math7 47 14 61 61Math8 47 18 65 70AlgebraI 47 21 68 70Geometry 47 15 62 66AlgebraII 47 15 62 66Review of Ordered Item Booklets ForeachitemintheOIB,panelistswereinstructedtoaskwhatastudentmustknowandbeabletodotoanswereachquestionandwhatmakeseachitemintheOIBmoredifficultthantheprecedingitem.ThisreviewoftheOIBallowedpaneliststogainnewperspectivesontheknowledgeandskillrequirementsofitemsandtoshareinformationregardingtheirthoughtsonthelocationofthethresholdregion.Duringthisdiscussion,theworkshopleadercirculatedthroughtheroomtomonitorprogress,toassistpanelistswhomighthavehadtroublewiththetask,andtoansweranyquestions.OneachpageintheOIB,panelistsviewedthecontentoftheitem,theassociatedpassage,contentalignment,andthescoringkeyorrubric.Inaddition,foreachpagethatpresentedawritingitem,ELApanelistswereprovidedasamplestudentessayresponsethatscoredattheparticularscorepoint.PanelistswereinitiallyprovidedanitemmaptousewhilenavigatingtheOIB,whichincludedpassageandcontentalignmentinformationforeachpageintheOIB.Inaddition,panelistswerepresentedwithanitemplotthatdisplayedagraphicalrepresentationofthedifficultyofeachpageintheOIB;thistoolshowedwherepageitemdifficultieswereclusteredtogetherversusspreadout.OIBitemplotsarepresentedinAppendixG.AzMERIT Bookmark Placement PriortomakingtheirRound1bookmarkplacements,panelistswereprovidedtrainingintheidentificationofperformancestandardsintheordereditembooklets.Aspartofthistraining,panelistslearnedtoidentifyalocationintheOIBthatbestdelineatestwoperformancelevels(e.g.,betweenpagesonwhichstudentsmustdemonstratemasterytomeettheminimumrequirementsformembershipinthePartiallyProficientlevelfromthoseitemsonwhichdemonstrationofmasteryisnotnecessary).AzMERIT StandardSettingTechnicalReport 16American Institutes for Research UsingtheirjustbarelyPLDsasaguide,thepanelistsweretheninstructedtosetabookmarkontheitemthatbestdelineatedeachoftheperformancelevels.Panelistswereremindedhowtosetbookmarks,andpriortomakinginitialplacements,facilitatorsledagroupactivitythatreviewedthekeyconceptsofthebookmarkprocedure,allowingfacilitatorstoprovideadditionaltrainingifnecessary.Priortoplacingrecommendedperformancestandardsineachround,panelistswereaskedtocompleteareadinessformtoindicatetheirpreparednesstorecommendperformancestandards.Thisformaskedpaneliststoasserttheirunderstandingofthetoolsusedtorecommendperformancestandardsineachround.Ifapanelistindicatesthattheydonotfeelpreparedtorecommendperformancestandards,theworkshopleaderprovidesadditionaltrainingandopportunitiesfordiscussion.Allpanelistshadtoindicatethattheyfeltpreparedtomoveforwardbeforetheyrecommendedacut.AllAzMERITstandardsettingpanelistsindicatedtheyunderstoodthetaskathandandfeltreadytorecommendperformancestandards.SamplesofreadinessformsusedforcompletingthebookmarktaskarepresentedinAppendixH.Bookmarkplacementwasconductedintworounds,allowingpaneliststomakeindependentjudgmentswhilestillbenefitingfromdiscussionwiththeirfellowpanelists.PanelistswereinstructedtoidentifytheirrecommendedcutsforProficient,PartiallyProficient,andHighlyProficientineachround.TheplacementofthebookmarkisillustratedinFigure1.EachpanelistusedtheirjustbarelyPLDstoidentifywhichitemrepresentedthelowerboundofeachperformancelevel.Intheexample,apanelistconcludedthatstudentswhowerejustbarelyattheProficientlevelwoulddemonstratemasteryontheitemonthepageindicatedbythearrow,whilestudentsbelowtheProficientlevelwouldnot.Therefore,thepanelistdecidedthattheProficientperformancelevelwouldbeginonthepageindicatedbyanarrow.ThepanelistbelievedthatstudentsbelowtheProficientperformancelevelwouldnotbeabletodemonstratemasteryofitemsbeyondtheindicatedpageintheordereditembooklet.AzMERIT StandardSettingTechnicalReport 17American Institutes for Research Figure 1. Example of Bookmark Placement Benchmark Information Panelistswerechargedtorecommendperformancestandardscomparabletootherimportantassessmentsystems,includingnationalandinternationalbenchmarkssuchasNAEP,otherstatewideassessments,andcollegeentryexams.TofacilitatecomparisonsofArizonaperformancestandardswithothernationalandinternationalbenchmarks,panelistswereprovidedwiththelocationsofperformancestandardsfromtheseotherassessmentssystemsintheirOIBs.Inparticular,performancestandardlocationsforthefollowingassessmentswereprovidedaspartofpanelistsOIBreview:SmarterBalancedELAandmathperformancestandardsingrades38and11/AlgebraII,PISAperformancestandardsingrade10ELAandGeometry,NAEPperformancestandardsinreadingandmathingrades4and8(andinterpolatedforgrade6),ACTcollegereadyperformancestandardingrade11ELAandAlgebraII,andArizonaspreviousAIMSassessment.AzMERIT StandardSettingTechnicalReport 18American Institutes for Research Panelist Feedback and Impact Data PriortoRound2,panelistswereprovidedfeedbackaboutthebookmarkplacementsmadebyfellowpanelists.AftermakingtheirRound1bookmarkplacements,panelistsreconvenedandbeganwithadiscussionofpanelistfeedbackaboutthebookmarklocationsrecommendedbyeachpanelist,beginningwithtablelevelfeedbackanddiscussion,andprogressingtoroomleveldiscussion.Eachtablespenttimereviewinganddiscussingcutscoreplacements,focusingonthelowestandhighestrecommendedperformancestandardsbothatthetableandacrossthepanel.Panelistswereaskedtoreviewtheitemsbetweenthelowestandhighestperformancestandardsattheirtable,discussingthestandardsandthejustbarelyPLDs.Discussionwasthenexpandedtotheroomlevel,witheachtablereviewingthebasisfortheirownrecommendationsforthegroupatlarge.Followingdiscussionofpanelistfeedback,panelistswerepresentedwithimpactdata,thepercentageofstudentsexpectedtoscoreatorabovetherecommendedRound1performancestandards.Panelistsdiscussedanyimplicationsoftheimpactdata,bothattheirtablesandacrossthepanelmoregenerally,focusingonwhethertheimpactwasinlinewiththeirexpectations.Followingpresentationofimpactdata,panelistswereprovided,foreachitemintheOIB,thepercentageofstudentsexpectedtoachievetheabilitylevelindexedbythatpage.Aftercompletingtheirdiscussions,panelistsagainworkedthroughtheOIB,placingtheirRound2bookmarksforallthreeperformancelevels,beginningwithProficientandfollowedbyPartiallyProficientandHighlyProficient.Estimating Student Performance Data WhiletheAzMERITOIBswereconstructedbasedoncalibrationoftheonlinetestingpopulation,thepercentageofstudentswithinthestatewhomeetorexceedeachpotentialperformancestandard(i.e.,eachpageintheordereditembooklet)wasestimatedbasedonallstudentsparticipatinginthefirstoperationaladministrationoftheassessment,includingstudentswhotestedonlineandstudentswhotestedonpaper.Amatchedsamplesapproachwasusedtoestimatetheeffectsofmodeonstudentperformance.Previousyearstudentachievementresults,aswellasdemographicinformation,includinggender,ethnicity,incomelevelstatus,Englishlanguagelearner(ELL)status,IndividualizedEducationProgram(IEP),wereusedtoidentifymatchedsamplesforthemodecomparabilityanalyses.Withmatchedsamplesinhand,itemparameterswerecalibratedseparatelyforthematchedsamplesofpaperandonlinetestadministrations,andthelinkingconstantsnecessarytobringthepaperitemparametersontotheonlinereferencescalewereidentified.Themodelinkingconstantswereuniformlyquitesmall,indicatingvirtuallynoeffectoftestadministrationmodeonstudentperformance.Nevertheless,forthepurposeofestimatingstudentimpactforthestandardsettingworkshop,themodelinkingconstantswereappliedtothepaperitemparameterstoestimatestudentabilityforpapertestadministrations.Thus,thepercentageofAzMERIT StandardSettingTechnicalReport 19American Institutes for Research studentsestimatedtomeetorexceedeachpotentialperformancestandardontheAzMERITwasbasedonallstudentswhoparticipatedintheoperationalassessment.AsummaryofthemodecomparabilitystudyispresentedinAppendixI7.PriortoRound2oftheBookmarkprocedure,thepercentageofstudentsmeetingthestandards,basedontheRound1mediancutscore,waspresentedtopanelists.Vertical Articulation Performancestandardsshouldideallybewellarticulatedacrossgrades.Unlesstherearesystemicdifferencesinthequalityofinstructionacrossgrades,theexpectationisthatstudentswhomeetthestandardsandarepreparedforinstructioninthesubsequentgradewilllikelycontinuetomeetstandardsastheyprogressthroughtheirschoolyears,andthatthereforewewouldnotexpecttoseelargechangesintheproficiencyratesfromgradetograde.WhilethisverticalarticulationisincorporatedintothedevelopmentoftheArizonaCollegeandCareerReadyStandardsaswellasthetestspecificationsforeachoftheAzMERITassessments,maintainingandreinforcingthecrossgradearticulationinthesettingofmeaningfulperformancestandardsisimportant,especiallyforELAandmath,wherestudentsareassessedannually.Lackofarticulationinthesesubjectscanresultinconfusion,especiallywhenthereareunreasonablylargeshiftsinstudentperformancelevelclassificationsfromgradetograde.Articulationwasconsideredfromtwoperspectives:thepercentofstudentsmeetingstandardsacrossgradesandcourses,andthelocationoftheperformancestandardsontheverticallylinkedAzMERITscale,whichallowedpaneliststoevaluatetheirrecommendedperformancestandardswithrespecttoexpectedstudentgrowthfromgradetograde.AdescriptionoftheproceduresusedtoyieldtheAzMERITverticalscaleispresentedinAppendixJ.Tohelpfosterconsistencyintheidentificationofperformancestandardsacrossgrades,afterperformancestandardswererecommendedfortheinitialgradelevelineachgradeband,tableleaderswereconvenedtoparticipateinaverticalmoderationsession.Tableleaderswereshownthepercentageofstudentsscoringatoraboveeachoftheperformancestandards,andthepercentofstudentsclassifiedateachperformancelevelacrosstests.Wherethepercentageofstudentsexpectedtomeetstandardsvariedgreatlybetweengradeorcoursebasedassessments,tableleaderswereaskedtoconsidermodificationstotherecommendedstandardsthatwouldachieveamorearticulatedsystemofstandards.Intheseinstances,tableleadersreviewedtheordereditembookletsandconsideredwhetherthecontentoftheOIBsupportedtheadjustment.Thus,whiletableleadersworkedtoarticulatestandardsacrossgrades,theyalsoensuredthatanychangesresultingfromthemoderationmeetingbeconsistentwiththeknowledgeandskillsdescribedinthePLDs.Withanchorgradeperformancestandardsinhand,AIRevaluatedbothimpactdatafromeachgradelevelassessment,aswellasstudentabilityestimatesfromtheverticallylinkedAzMERIT7ResponsivetoStandardsforEducationandPsychologicalTesting:Standard3.0,3.5,3.6,3.8,3.15,5.7,5.12,5.13,5.14,5.15,12.3,12.17,and13.6AzMERIT StandardSettingTechnicalReport 20American Institutes for Research scale,tointerpolatethelikelylocationofeachperformancestandardforeachoftheremaininggradelevelandEOCassessments.Torecommendperformancestandardsinthesenonanchorgradeassessments,thestandardbookmarkproceduresweremodifiedsothatpanelistswereinstructedtodeterminewhetherthejustbarelyPLDssupportedtheplacementofaspecificbookmarkontheinterpolatedpage.IfthePLDsdidnotsupporttheplacementofthebookmarkontheinterpolatedpage,thenpanelistswereaskedwhethertheycouldidentifyabookmarkplacementneartheinterpolatedpagethatwouldbesupportedbythePLDs.Panelistswereinstructedthattheirbookmarkplacementsmustbeguidedbycontentconsiderations,whethertheyrecommendedtheinterpolatedpageintheOIBoradifferentbookmarkplacement.Otherwise,bookmarkplacementsproceededaswiththeanchorgraderounds.FollowingRound1bookmarkplacements,panelistsreceivedfeedbackaboutthebookmarkplacementsofpanelistsattheirtable,andfortheroomasawholeandimpactdata.Afinalmoderationsessionwasconductedfollowingthecompletionofworkshopactivitiesfortheinterpolatedgrades.Thisfinalmoderationactivityensuredthattableleadershadanopportunitytoreviewtheentiresystemofrecommendedstandardsandtomakeanydesiredadjustmentspriortocompletionoftheworkshop.Aswiththeinitialmoderationsession,inthoseinstanceswheretableleaderschosetoadjustaperformancestandardduringthefinalmoderationsession,theyreviewedtheirordereditembookletstoensurethattheadjustmentshadabasisintestcontent.Theadvantageofthisapproachisthatitresultsinasystemofperformancestandardsthataremoreconsistentacrossgradelevels.Atthemostbasiclevel,itensuresthattherearenotwidefluctuationsintheproportionofstudentsmeetingeachperformancestandardacrossgrades.Crossgradearticulationinformedbytheverticalscalealsoensuresthattherearenoreversalsinrecommendedperformancestandardsacrossgrades.Workshop Evaluation Panelistswereencouragedtoprovidefeedbackconcerningtheproceduresandoutcomesofthestandardsettingworkshopthroughouttheprocess,viagroupdiscussions,practiceactivities,andcompletionofreadinessformspriortoplacingtheirbookmarks.Atthecompletionoftheworkshop,panelistswereaskedtocompleteaworkshopevaluationformdesignedtoelicitfeedbackonallaspectsoftheworkshop,includingclarityoftrainingandtasks,appropriatenessofthetimespentonactivities,andsatisfactionwiththeoutcomeoftheworkshop.SamplesoftheevaluationformsarepresentedinAppendixK.AzMERIT StandardSettingTechnicalReport 21American Institutes for Research Recommended Performance Standards and Impact Data FortheAzMERITinELAandmath,AppendixLpresentstheminimum,maximum,andmedianbookmarkplacementforeachroundofbookmarkplacements,aswellasanybookmarksplacedduringModerationsessions,andresultingfinalrecommendationsfollowingthestandardsettingworkshops.Aspanelistsdiscussedthereasonsfortheirbookmarkplacementsinthecontextoffeedbackfromotherpanelistsandimpactdata,variabilityacrosstablesoftendecreasedacrossrounds.ThefiguresinAppendixM,ConvergenceofBookmarksacrossRounds,illustratevariabilityinmediantablebookmarkplacementsforthethreeperformancestandardsoverthetworounds.Thesefiguresillustratehowvariabilityinbookmarkdecisionschangedfromthefirsttothesecondround.Ingeneral,therewasconsiderableconsistencyintheplacementofperformancestandardsacrossrounds.Foreachtest,finalrecommendedperformancestandardistheoutcomefromthefinalmoderation,orintheabsenceofmoderation,themedianbookmarkpagefollowingRound2.Thefinalrecommendedperformancestandardsforeachassessment,grade,andperformancestandardarepresentedinTable4,alongwiththeprojectedimpacteachperformancestandardwouldhaveonArizonapublicschoolstudentstestedin2015.ThefinalrecommendedOIBpagenumbersarethemedianbookmarksofeachpanelfollowingRound2bookmarkplacement,andsubsequentmoderation. AzMERIT StandardSettingTechnicalReport 22American Institutes for Research Table4.FinalRecommendedPerformanceStandardsforAzMERITTest PerformanceLevelOrderedItemBookletPage ThetaEstimatedPercentageofStudentsAtorAbovePerformanceStandardGrade3ELAPartiallyProficient 18 0.09 56Proficient 25 0.29 41HighlyProficient 49 1.36 10Grade4ELAPartiallyProficient 19 0.14 57Proficient 32 0.60 39HighlyProficient 57 1.80 5Grade5ELAPartiallyProficient 15 0.13 63Proficient 32 0.63 30HighlyProficient 53 1.80 3Grade6ELAPartiallyProficient 16 0.12 61Proficient 30 0.58 34HighlyProficient 58 2.03 4Grade7ELAPartiallyProficient 18 0.02 59Proficient 36 0.61 33HighlyProficient 61 1.90 4Grade8ELAPartiallyProficient 19 0.06 60Proficient 38 0.64 33HighlyProficient 62 1.72 6Grade9ELAPartiallyProficient 17 0.12 53Proficient 32 0.59 27HighlyProficient 56 1.57 6Grade10ELAPartiallyProficient 13 0.11 51Proficient 32 0.58 30HighlyProficient 59 1.42 8Grade11ELAPartiallyProficient 13 0.02 46Proficient 29 0.52 26HighlyProficient 52 1.27 8Grade3MathPartiallyProficient 10 0.16 73Proficient 33 1.04 42HighlyProficient 52 2.43 15Grade4MathPartiallyProficient 10 0.31 71Proficient 35 0.76 42HighlyProficient 58 2.20 10Grade5MathPartiallyProficient 4 0.65 71Proficient 27 0.41 40HighlyProficient 52 1.74 13AzMERIT StandardSettingTechnicalReport 23American Institutes for Research Test PerformanceLevelOrderedItemBookletPage ThetaEstimatedPercentageofStudentsAtorAbovePerformanceStandardGrade6MathPartiallyProficient 9 0.48 62Proficient 26 0.41 32HighlyProficient 46 1.55 11Grade7MathPartiallyProficient 11 0.19 52Proficient 30 0.59 30HighlyProficient 46 1.51 13Grade8MathPartiallyProficient 15 0.69 57Proficient 29 0.09 32HighlyProficient 47 1.15 13AlgebraIPartiallyProficient 17 0.69 55Proficient 33 0.03 32HighlyProficient 56 1.27 9GeometryPartiallyProficient 16 1.37 53Proficient 30 0.58 30HighlyProficient 52 0.96 6AlgebraIIPartiallyProficient 15 1.49 53Proficient 29 0.78 29HighlyProficient 49 0.57 6Note:Followingthestandardsettingworkshop,recommendationsaresubmittedtoArizonasStateBoardofEducation.PerformancestandardsarenotfinalpriortoapprovalandadoptionbytheBoard. AzMERIT StandardSettingTechnicalReport 24American Institutes for Research Table5showstheestimatedpercentageofstudentclassifiedateachperformancelevelbasedonfinalpanelistrecommendedstandardsforthestudentpopulationoverallacrossgradelevelsandcoursesfortheELAandmathassessments.TheresultsofTable5arerepresentedgraphicallyinFigure2,forELA,andFigure3formath.AppendixNpresentstheestimatedpercentageofstudentsclassifiedateachperformanceleveldisaggregatedbygenderandethnicity.Table5.PercentageofStudentsatEachPerformanceLevelbasedonFinalRecommendedPerformanceStandardsTest MinimallyProficient PartiallyProficient Proficient HighlyProficientELAGrade3ELA 44% 15% 31% 10%Grade4ELA 43% 19% 33% 5%Grade5ELA 37% 33% 27% 3%Grade6ELA 39% 27% 30% 4%Grade7ELA 41% 26% 29% 4%Grade8ELA 40% 27% 26% 6%Grade9ELA 47% 26% 21% 6%Grade10ELA 49% 21% 22% 8%Grade11ELA 54% 20% 17% 8%MathGrade3Math 27% 31% 27% 15%Grade4Math 29% 29% 32% 10%Grade5Math 29% 31% 27% 13%Grade6Math 38% 30% 21% 11%Grade7Math 48% 22% 18% 13%Grade8Math 43% 24% 20% 13%AlgebraI 45% 23% 23% 9%Geometry 47% 24% 24% 6%AlgebraII 47% 24% 23% 6%AzMERIT StandardSettingTechnicalReport 25American Institutes for Research Figure2.PercentageofStudentsatEachPerformanceLevelbasedonFinalRecommendedPerformanceStandardsAzMERITELA 44433739414047495413193327262726212033332730292621221710534 46 68 8G3ELA G4ELA G5ELA G6ELA G7ELA G8ELA G9ELA G10ELA G11ELAMinimallyProficient PartiallyProficient Proficient HighlyProficientAzMERIT StandardSettingTechnicalReport 26American Institutes for Research Figure3.PercentageofStudentsatEachPerformanceLevelbasedonFinalRecommendedPerformanceStandardsAzMERITMath 2729 293848434547 473129313022242324 2427322721 18202324 231510131113 1396 6G3Math G4Math G5Math G6Math G7Math G8Math AlgebraI Geometry AlgebraIIMinimallyProficient PartiallyProficient Proficient HighlyProficientAzMERIT StandardSettingTechnicalReport 27American Institutes for Research ADEintendstoreportstudentperformanceontheontheverticallylinkedAzMERITscale.Becauseabilityestimatesofextremelylowandhighscoringstudentsarelessprecise,testscoresforverylowandhighperformingstudentswillbemorepronetofluctuateovertime.Tominimizescalescoreinstabilityforverylowandhighscoringstudents,abilityestimateswillbetruncatedat+3.5onthewithingradescalebeforebeingtransformedtotheverticallylinkedscale.StudentabilityestimateswillthenbetransformedfromtheverticallylinkedRaschthetascaletothesubjectspecificAzMERITreportingscale:EIA Scolc Scorc = 2Suu + (Su 0)Hotb Scolc Scorc = SSuu + (Su 0)ApplyingtheAzMERITscalescoretransformationstotheperformancestandardsrecommendedbytheworkshoppanelsresultsinthesystemofscalescorerangesforeachoftheAzMERITperformancelevelclassificationsidentifiedinTable6.Table6.AzMERITScaleScoreRangesBasedonFinalRecommendedPerformanceStandardsTest MinimallyProficient PartiallyProficient Proficient HighlyProficientELAGrade3ELA 23952496 24972508 25092540 25412605Grade4ELA 24002509 25102522 25232558 25592610Grade5ELA 24192519 25202542 25432577 25782629Grade6ELA 24312531 25322552 25532596 25972641Grade7ELA 24382542 25432560 25612599 26002648Grade8ELA 24482550 25512571 25722603 26042658Grade9ELA 24542554 25552576 25772605 26062664Grade10ELA 24582566 25672580 25812605 26062668Grade11ELA 24652568 25692584 25852607 26082675MathGrade3Math 33953494 34953530 35313572 35733605Grade4Math 34353529 35303561 35623605 36063645Grade5Math 34783562 35633594 35953634 36353688Grade6Math 35123601 36023628 36293662 36633722Grade7Math 35293628 36293651 36523679 36803739Grade8Math 35663649 36503672 36733704 37053776AlgebraI 35773660 36613680 36813719 37203787Geometry 36093672 36733696 36973742 37433819AlgebraII 36293689 36903710 37113750 37513839 AzMERIT StandardSettingTechnicalReport 28American Institutes for Research Evaluation of the Standard Setting Workshop Panelist Evaluation of Standard Setting Workshop Followingthecompletionofstandardsettingtasks,panelistswereaskedtoevaluatedifferentaspectsoftheworkshop,andtheresultingrecommendations.Attheendoftheworkshop,allbutonepanelistindicatedthattrainingonthemaincomponentsandtoolsofthebookmarkprocedurewasadequate,andthattheyunderstoodhowtouseeachcomponent.Generally,panelistsindicatedthattheamountoftimeallottedfordifferentactivitieswithinthestandardsettingworkshopwasaboutright.Overall,panelistsexpressedgeneralsatisfactionwiththeworkshopandofferedsuggestionsforimprovingtheexperienceinfuturemeetings.Acrossallpanels,allbutoneparticipantindicatedtheyagreedthatstudentsclassifiedateachperformancelevelarefairlyclassifiedintoeachoftheperformancelevelclassificationsbasedontheknowledgeandskillsdescribedintheArizonaCollegeandCareerReadyStandards,assummarizedinTable7.AppendixOshowspanelistsresponsestotheevaluationforms.Table7.SummaryofPanelistEvaluationofRecommendedPerformanceStandardsWorkshopEvaluationQuestionStronglyDisagreeDisagree AgreeStronglyAgreeIamconfidentthatstudentsclassifiedasProficientdemonstrateafundamentalunderstandingofandabilitytoapplythecontentknowledgeandskillsneededtobeontracktowardsArizonasCollegeandCareerReadinessStandards(ACCRS)inMathematicsandEnglishLanguageArtsStandards.(Level3)1 0 35 44IamconfidentthatstudentsclassifiedasPartiallyProficientdemonstrateapartialunderstandingofandabilitytoapplythecontentknowledgeandskillsneededtobeontracktowardsArizonasCollegeandCareerReadinessStandards(ACCRS)inMathematicsandEnglishLanguageArtsStandards.(Level2)1 0 34 45IamconfidentthatstudentsclassifiedasHighlyProficientdemonstrateanadvancedunderstandingofandabilitytoapplythecontentknowledgeandskillsneededtobeontracktowardsArizonasCollegeandCareerReadinessStandards(ACCRS)inMathematicsandEnglishLanguageArtsStandards.(Level4)1 0 30 49AzMERIT StandardSettingTechnicalReport 29American Institutes for Research Independent Observer Review of Standard Setting Workshop ADEinvitedmembersoftheStateBoardofEducationtoattendandobservethestandardsettingworkshop.ThreedistrictobserversattendedandsubmittedareporttotheStateBoardofEducationdescribingtheirexperienceattheworkshop;thereportwasproducedindependentlywithoutinputorreviewfromADE.ThereportispresentedinAppendixP.AzMERIT StandardSettingTechnicalReport 30American Institutes for Research References Ferrara,S.,Perie,M.,&Johnson,E.(2008).Matchingthejudgmentaltaskwithstandardsettingpanelistexpertise:TheItemDescriptor(ID)Matchingprocedure.JournalofAppliedTestingTechnology,9(1).McLaughlin,D.,Scarloss,B.A.,Stancavage,F.B.,&Blankenship,C.D.(2005).UsingStateAssessmentstoImputeAchievementofStudentsAbsentfromNAEP:AnEmpiricalStudyinFourStates.Washington,DC:AmericanInstitutesforResearch.Retrievedfromwww.air.org/files/McLaughlin_AbsentStudents.pdfMitzel,H.C.,Lewis,D.M.,Patz,R.J.,&Green,D.R.(2001).TheBookmarkprocedure:Psychologicalperspectives.InG.Cizek(Ed.),Settingperformancestandards:Concepts,methods,andperspectives.Mahwah,NJ:ErlbaAzMERIT StandardSettingTechnicalReportAppendixAppendix A Workshop Agendas AzMERITStandard Setting Technical Report A-1American Institutes for Research Document A1. AzMERIT Mathematics & ELA Grades 3-8 Standard Setting Agenda July 14 - 16, 2015 (Grade 3-8 Panels) Tuesday, July 14, 2015 7:30 8:00Orientation for Table Leaders 7:30 8:00Registration and morning refreshments Panelists receive folders, sign security affidavit 8:00 8:15Welcome and introductions from Arizona Department of Education 8:15 9:30Large group introductory training Welcome and introductions Purpose of standard setting workshop Description of the AzMERIT test design General overview of standard setting procedures and key concepts oProficiency Level Descriptors oJust Barely oOrdered Item Book oMastery oBookmark task oBenchmark Information oPanelist feedback and impact data 9:30 9:45Break, and separate into small group rooms 9:45 10:00Introductions within panel 10:00 11:00Participate in AzMERIT assessment 11:00 12:00Review Performance Level Descriptors and develop J ust Barely PLDs Grades 4, 6, and 8 12:00 1:00Lunch 1:00 2:30Review PLDs and develop J ust Barely PLDs Grades 4, 6, and 8 (continued) 2:30 2:45Break 2:45 4:45Review of Ordered Item Book Grades 4, 6, and 8 Training on review of the OIB oWhat do students need to know and be able to do to respond correctly to each question? oWhy is each item more difficult than the preceding item? Individual review of the OIB Discuss areas of transition and skills with tables 4:45Adjourn Wednesday, July 15, 2015 7:30 8:00Registration and morning refreshments 8:00 9:00Training on Bookmark Placement task AzMERITStandard Setting Technical Report A-2American Institutes for Research (Grade 3-8 Panels) Review of Bookmark Placement key concepts oProficiency Level Descriptors oOrdered Item Book Training on mastery and 2/3 likelihood Training on bookmark placement judgment task, and procedure for recording bookmarks 9:00 10:15Round 1 bookmark placement for Proficient, Partially Proficient, and Highly Proficient Grades 4, 6, and 8 Review of bookmark procedures and key concepts Completion of Bookmark Placement Readiness Form Review OIB and place each bookmark oProficient oPartially Proficient oHighly Proficient 10:15 10:30Panelist Break, and concurrent production of feedback data10:30 11:30Review results of Round 1 Grades 4, 6, and 8 Presentation and discussion of Round 1 panelist agreement feedback data Review agreement feedback data Discussion of percent of students achieving the Round 1 recommended standards 11:30 12:30Lunch 12:30 1:30Round 2 bookmark placement for Proficient, Partially Proficient, and Highly Proficient Grades 4, 6, and 8 Completion of Bookmark Placement Readiness Form Review OIB and place each bookmark oProficient oPartially Proficient oHighly Proficient 1:30 3:30Review Performance Level Descriptors and develop J ust Barely PLDs Grades 3, 5, and 7 3:30 3:45Break 3:45 4:45Anchor Grade Moderation *Table leaders required to participate, all panelists invited to attend 4:45Adjourn Thursday, July 16, 2015 7:30 8:00Registration and morning refreshments 8:00 10:00 Review of Ordered Item Booklet Grades 3, 5, and 7 10:00 10:15 Panelist Break 10:15 10:45 Review Results of anchor grade vertical moderationAzMERITStandard Setting Technical Report A-3American Institutes for Research (Grade 3-8 Panels) 10:45 12:00 Round 1 bookmark placement for Proficient, Partially Proficient, and Highly Proficient Grades 3, 5, and 7 Completion of Bookmark Placement Readiness Form Review OIB and place each bookmark oProficient oPartially Proficient oHighly Proficient 12:00 1:00Lunch 1:00 2:00Review results of Round 1 Grades 3, 5, and 7 2:00 3:00Round 2 bookmark placement for Proficient, Partially Proficient, and Highly Proficient Grades 3, 5, and 7 Completion of Bookmark Placement Readiness Form Review OIB and place each bookmark oProficient oPartially Proficient oHighly Proficient 3:00 3:30Complete workshop evaluation forms 3:30 4:30Final vertical moderation (if needed) *Table leaders required to participate, all panelists invited to attend AzMERITStandard Setting Technical Report A-4American Institutes for Research Document A2. AzMERIT Mathematics & ELA EOC Standard Setting Agenda July 13 - 16, 2015 High School Panels Monday, July 13, 2015 7:30 8:00Orientation for Table Leaders 7:30 8:00Registration and morning refreshments Panelists receive folders, sign security affidavit 8:00 8:15Welcome and introductions from Arizona Department of Education 8:15 9:30Large group introductory training Welcome and introductions Purpose of standard setting workshop Description of the AzMERIT test design General overview of standard setting procedures and key concepts oProficiency Level Descriptors oJust Barely oOrdered Item Book oMastery oBookmark task oBenchmark Information oPanelist feedback and impact data 9:30 9:45Break, and separate into small group rooms 9:45 10:00Introductions within panel 10:00 11:00Participate in AzMERIT assessment ELA 11/ Algebra II 11:00 12:00Review Performance Level Descriptors and develop J ust Barely PLDs ELA 11/ Algebra II 12:00 1:00Lunch 1:00 2:30Review PLDs and develop J ust Barely PLDs Algebra ELA/11 II (continued) 2:30 2:45Break 2:45 4:45Review of Ordered Item Book ELA 11/ Algebra II Training on review of the OIB oWhat do students need to know and be able to do to respond correctly to each question? oWhy is each item more difficult than the preceding item? Individual review of the OIB Discuss areas of transition and skills with tables 4:45Adjourn Tuesday, July 14, 2015 7:30 8:00Registration and morning refreshments AzMERITStandard Setting Technical Report A-5American Institutes for Research High School Panels 8:00 9:00Training on Bookmark Placement task Review of Bookmark Placement key concepts oProficiency Level Descriptors oOrdered Item Book Training on mastery and 2/3 likelihood Training on bookmark placement judgment task, and procedure for recording bookmarks 9:00 10:15Round 1 bookmark placement for Proficient, Partially Proficient, and Highly Proficient ELA 11/ Algebra II Review of bookmark procedures and key concepts Completion of Bookmark Placement Readiness Form Review OIB and place each bookmark oProficient oPartially Proficient oHighly Proficient 10:15 10:30Panelist Break, and concurrent production of feedback data10:30 11:30Review results of Round 1 ELA 11/ Algebra II Presentation and discussion of Round 1 panelist agreement feedback data Review agreement feedback data Discussion of percent of students achieving the Round 1 recommended standards 11:30 12:30Lunch 12:30 1:30Round 2 bookmark placement for Proficient, Partially Proficient, and Highly Proficient ELA 11/ Algebra II Completion of Bookmark Placement Readiness Form Review OIB and place each bookmark oProficient oPartially Proficient oHighly Proficient 1:30 3:30Review Performance Level Descriptors and develop J ust Barely PLDs ELA 10/ Geometry 3:30 3:45Break 3:45 4:45Begin Review of Ordered Item Booklet ELA 10/ Geometry 4:45AdjournWednesday, July 15, 2015 7:30 8:00Registration and morning refreshments 8:00 9:00Complete OIB Review ELA 10/ Geometry 9:00 10:15 Round 1 bookmark placement for Proficient, Partially Proficient, and Highly Proficient ELA 10/ Geometry AzMERITStandard Setting Technical Report A-6American Institutes for Research High School Panels Completion of Bookmark Placement Readiness Form Review OIB and place each bookmark oProficient oPartially Proficient oHighly Proficient 10:15 10:30Panelist Break 10:30 11:30Review results of Round 1 ELA 10/ Geometry Presentation and discussion of Round 1 panelist feedback data Review agreement feedback data Discussion of percent of students achieving the Round 1 recommended standards 11:30 12:30Lunch 12:30 1:30Round 2 bookmark placement for Proficient, Partially Proficient, and Highly Proficient ELA 10/ Geometry Completion of Bookmark Placement Readiness Form Review OIB and place each bookmark oProficient oPartially Proficient oHighly Proficient 1:30 3:30Review Performance Level Descriptors and develop J ust Barely PLDs ELA 9/ Algebra I 3:30 3:45Break 3:45 4:45Anchor grade vertical moderation *Table leaders required to participate, all panelists invited to attend 4:45Adjourn Thursday, July 16, 2015 7:30 8:00Registration and morning refreshments 8:00 10:00 Review of Ordered Item Booklet ELA 9/ Algebra I 10:00 10:15 Panelist Break 10:15 10:45 Review results of anchor grade vertical moderation10:45 12:00 Round 1 bookmark placement for Proficient, Partially Proficient, and Highly Proficient ELA 9/ Algebra I Completion of Bookmark Placement Readiness Form Review OIB and place each bookmark oProficient oPartially Proficient oHighly Proficient 12:00 1:00Lunch AzMERITStandard Setting Technical Report A-7American Institutes for Research High School Panels 1:00 2:00Review results of Round 1 ELA 9/ Algebra I 2:00 3:00Round 2 bookmark placement for Proficient, Partially Proficient, and Highly Proficient ELA 9/ Algebra I Completion of Bookmark Placement Readiness Form Review OIB and place each bookmark oProficient oPartially Proficient oHighly Proficient 3:00 3:30Complete workshop evaluation forms 3:30 4:30Final vertical moderation (if needed) *Table leaders required to participate, all panelists invited to attend AzMERIT StandardSettingTechnicalReportAppendixAppendix B Composition of Panels AzMERIT StandardSettingTechnicalReport B-1American Institutes for Research TableB1.CompositionofELAGrades34PanelsTableTableLeaderUrban/Rural District Gender Ethnicity CurrentPositionGradeMostFrequentlyTaught1 Yes Rural ArizonaNavajoCentral Male NativeAmerican ElementaryEducation 31 n/a Urbanized PeoriaUnifiedSchoolDistrict Female Asian ElementaryEducation 31 n/a UrbanizedCAFAINC.LearningFoundationandPerformingArtsFemale White,nonHispanic SecondaryEducation 32 Yes RuralLegacyTraditionalSchoolsandAthlosTraditionalAcademyFemale HispanicorLatinoElementary,AdministrativePrincipal,Superintendent42 n/aUrbanizedMaricopaCountyEducationServiceAgencyFemale HispanicorLatino Elementary,Secondary 42 n/a UrbanClustersJ.O.CombsUnifiedSchoolDistrictFemale White,nonHispanic ElementaryEducation 62 n/a Urbanized MadisonElementaryDistrict Female White,nonHispanic ElementaryEducation 43 Yes UrbanizedWashingtonElementarySchoolDistrictFemale White,nonHispanicElementary,AdministrativePrincipal,Superintendent33 n/a UrbanizedCAFA,Inc.dbaLearningFoundationPerformingArtsSchoolFemale HispanicorLatino ElementaryEducation 43 n/a Suburban DeerValleyUnifiedDistrict Female White,nonHispanic Elementary,Administrative 3 AzMERIT StandardSettingTechnicalReport B-2American Institutes for Research TableB2.CompositionofELAGrades56PanelsTableTableLeaderUrban/Rural District Gender Ethnicity CurrentPositionGradeMostFrequentlyTaught1 Yes UrbanClusters MesaUnifiedDistrict Female Black ElementaryEducation 51 n/a Rural SuperiorUnifiedSchoolDistrict Female White,nonHispanic Elementary,Secondary 51 n/a Suburban MesaUnifiedDistrict Female White,nonHispanic ElementaryEducation 62 Yes UrbanClusters MayerUnifiedSchoolDistrict Female White,nonHispanic ElementaryEducation 52 n/a Urbanized LibertyElementaryDistrict Female White,nonHispanic ElementaryEducation 52 n/a Urbanized ChandlerUnifiedDistrict Female White,nonHispanic ElementaryandSpecialEducation 63 Yes Urbanized ParadiseValleyUnifiedDistrict Female White,nonHispanic ElementaryandSpecialEducation 63 n/a Urbanized AvondaleElementaryDistrict Female White,nonHispanic ElementaryEducation 53 n/a Urbanized PeoriaUnifiedSchoolDistrict Female White,nonHispanicElementary,AdministrativePrincipal,Superintendent6 AzMERIT StandardSettingTechnicalReport B-3American Institutes for Research TableB3.CompositionofELAGrades78PanelsTableTableLeaderUrban/Rural District Gender Ethnicity CurrentPositionGradeMostFrequentlyTaught1 Yes Suburban PeoriaUnifiedSchoolDistrict Female White,nonHispanic ElementaryEducation 71 n/a Urbanized AlhambraElementaryDistrict Female White,nonHispanicAdministrativePrincipal,Superintendent81 n/a UrbanClustersJ.O.CombsUnifiedSchoolDistrictMale White,nonHispanicSecondary,AdministrativePrincipal,Superintendent82 Yes Urban GeorgeGervinPrepAcademy Female HispanicorLatinoElementaryK8,SpecialEdK8,NBCTEnglishLanguageArts72 n/a UrbanClusters DysartUnifiedDistrict Female White,nonHispanic SecondaryEducation 82 n/a Urbanized MesaUnifiedDistrict Female White,nonHispanic ElementaryandSpecialEducation 73 Yes Suburban PeoriaUnifiedSchoolDistrict Female White,nonHispanic Elementary,JuniorHigh 83 n/a UrbanClusters KyreneElementaryDistrict Female White,nonHispanic SpecialEducation 73 n/a Urbanized LaveenElementaryDistrict Female White,nonHispanic SecondaryEducation 7 AzMERIT StandardSettingTechnicalReport B-4American Institutes for Research TableB4.CompositionofELAGrades911PanelsTableTableLeaderUrban/Rural District Gender Ethnicity CurrentPositionGradeMostFrequentlyTaught1 Yes UrbanizedAmericanCharterSchoolsFoundationd.b.a.WestPhoenixHighSchoolMale White,nonHispanic Secondary,AdministrativeDuties 91 n/a UrbanizedAmericanCharterSchoolsFoundationd.b.a.SunValleyHighSchoolFemale White,nonHispanicSecondary,AdministrativePrincipal,Superintendent101 n/a Rural RedMesaUnifiedSchoolDistrict Female White,nonHispanic SecondaryEnglish&Art 111 n/a UrbanClusters PrescottUnifiedDistrict Female MultiRacial SecondaryEducation 92 Yes Rural VailUnifiedDistrict Male White,nonHispanic SecondaryEducation 102 n/a UrbanClusters TucsonUnifiedDistrict Female White,nonHispanic SecondaryEducation 112 n/a UrbanizedGlendaleUnionHighSchoolDistrictFemale White,nonHispanic SecondaryEducation 103 Yes Rural J.O.CombsUnifiedSchoolDistrict Female White,nonHispanicSecondaryEducation,AdultEducation,APLanguage&Composition113 n/a UrbanizedTollesonUnionHighSchoolDistrictFemale White,nonHispanic 93 n/a Urbanized PimaPreventionPartnership Female Asian SecondaryEducation 11 AzMERIT StandardSettingTechnicalReport B-5American Institutes for Research TableB5.CompositionofMathematicsGrades34PanelsTableTableLeaderUrban/Rural District Gender Ethnicity CurrentPositionGradeMostFrequentlyTaught1 Yes Urbanized RodelFoundationofArizona Female White,nonHispanicElementary,AdministrativePrincipal,Superintendent41 n/a Urbanized ScottsdaleUnifiedDistrict Female White,nonHispanic ElementaryEducation 31 n/a Urbanized RodelFoundationofArizona Female White,nonHispanicElementary,AdministrativePrincipal,Superintendent32 Yes Rural PimaUnifiedDistrict Female White,nonHispanic ElementaryEducation 32 n/a Urbanized GilbertUnifiedDistrict Female White,nonHispanic ElementaryEducation 42 n/a Urbanized DeerValleyUnifiedDistrict Female White,nonHispanic ElementaryEducation 43 Yes Urbanized MesaUnifiedDistrict Female White,nonHispanic ElementaryEducation 33 n/a Urbanized CartwrightElementaryDistrict Female White,nonHispanic (notprovided) 43 n/a Urbanized MadisonElementaryDistrict Female White,nonHispanic ElementaryEducation 33 n/a Rural LakeHavasuUnifiedDistrict Female White,nonHispanic ElementaryEducation 4 AzMERIT StandardSettingTechnicalReport B-6American Institutes for Research TableB6.CompositionofMathematicsGrades56PanelsTableTableLeaderUrban/Rural District Gender Ethnicity CurrentPositionGradeMostFrequentlyTaught1 Yes Urbanized MesaUnifiedDistrict Female HispanicorLatinoElementary,AdministrativePrincipal,Superintendent51 n/a UrbanizedAvondaleElementarySchoolDistrictFemale White,nonHispanicElementary,AdministrativePrincipal,Superintendent61 n/a Urbanized ScottsdaleUnifiedDistrict Female White,nonHispanic ElementaryEducation 51 n/a UrbanClusters J.O.CombsUnifiedSchoolDistrict Female White,nonHispanic ElementaryEducation 62 Yes RuralWashingtonElementarySchoolDistrictMale White,nonHispanic ElementaryEducation 62 n/a Urbanized DeerValleyUnifiedDistrict Female White,nonHispanic ElementaryEducation 52 n/a Urbanized FowlerElementaryDistrict Female White,nonHispanic ElementaryEducation 62 n/a Urbanized KyreneElementaryDistrict Female Black ElementaryEducation 53 Yes Urbanized TanqueVerdeUnifiedDistrict Female White,nonHispanicElementary,AdministrativePrincipal,Superintendent63 n/a UrbanClusters GlendaleElementaryDistrict Female White,nonHispanic ElementaryEducation 53 n/a Suburban ChandlerUnifiedDistrict Female White,nonHispanic ElementaryEducation 6 AzMERIT StandardSettingTechnicalReport B-7American Institutes for Research TableB7.CompositionofMathematicsGrades78PanelsTableTableLeaderUrban/Rural District Gender Ethnicity CurrentPositionGradeMostFrequentlyTaught1 Yes Urban DeerValleyUnifiedDistrict Male White,nonHispanic SecondaryEducation 71 n/a Urbanized OsbornElementaryDistrict Female White,nonHispanic Elementary,Secondary 81 n/a Urbanized DeerValleyUnifiedDistrict Female White,nonHispanic SecondaryEducation 71 n/a UrbanClustersLitchfieldElementarySchoolDistrictFemale White,nonHispanicSecondary,Biology,MiddleGradesMathematics82 Yes Suburban MesaUnifiedDistrict Female HispanicorLatino Elementary,SecondaryMathematics 82 n/a Urbanized ArizonaSchoolfortheArts Female White,nonHispanic ElementaryEducation 72 n/a Suburban ChandlerUnifiedDistrict Female White,nonHispanic SecondaryMathematics 82 n/a Urbanized CartwrightElementaryDistrict Male HispanicorLatino ElementaryEducation 73 Yes Urbanized TucsonUnifiedDistrict Female White,nonHispanic ElementaryEducation 83 n/a UrbanClusters BuckeyeElementaryDistrict Male White,nonHispanic SecondaryEducation 73 n/a Urbanized MesaUnifiedDistrict Female White,nonHispanic Specia