essa innovative assessment designs: examples from science

ESSAInnovativeAssessmentDesigns:ExamplesfromScience

BrianGong,CenterforAssessmentAprilMcCrae,DelawareDepartmentofEduca;on

MichelleCenter,CaliforniaDepartmentofEduca;onKarenKidwell,KentuckyDepartmentofEduca;on

CCSSONa;onalConferenceonStudentAssessmentJune28,2017Aus;n,TX

AssessmentChallengesandSolutionApproachesforNextGenerationScienceStandards:

Domainde<inition,claims,evidencespeci<ications

BrianGongCenterforAssessment

Presenta;oninthesessionon“ESSAInnova;veAssessmentDesigns:ExamplesfromScience”

CCSSONa;onalConferenceonStudentAssessmentJune28,2017Aus;n,TX

Overview

•  Descrip;onofNGSSandassessmentcontext

•  Threeissues,somepossiblesolu;onapproaches

•  Summary

NGSSassessmentissues&approaches-6/28/17 3

NextGenerationScienceStandards

•  Designedtoreplacepreviousna;onal/widelyadoptedsciencestandards(Na;onalScienceEduca;onStandards,1996;ScienceforAllAmericans,1989)

•  FrameworkforK-12ScienceEduca:on(NRC,2012)

•  NGSSStandards,includingPerformanceExpecta;ons(NGSS,2013[Achieveand26states])

•  Othermaterials,e.g.,DevelopingAssessmentsfortheNextGenera:onScienceStandards(NRC,2014);AssessmentFramework/alignedtestitemcluster(WestEdforCCSSOScienceItemAssessmentCollabora;ve,2015)


SummaryofNGSSstructureandcontent

•  GradesK-highschool•  Threedimensions

–  ScienceandEngineeringPrac;ces(SEP)•  1)Askingques;ons(forscience)anddefiningproblems(forengineering);2)Developing

andusingmodels;3)Planningandcarryingoutinves;ga;ons;4)Analyzingandinterpre;ngdata;5)Usingmathema;csandcomputa;onalthinking;6)Construc;ngexplana;ons(forscience)anddesigningsolu;ons(forengineering);7)Engaginginargumentfromevidence;8)Obtaining,evalua;ng,andcommunica;nginforma;on

–  DisciplinaryContentIdeas(DCI)•  11Coreideas(4inlifesciences;4inphysicalsciences;3inearthandspacesciences)

•  39sub-ideas

•  Eachsub-ideaiselaboratedinalistofwhatstudentsshouldunderstandaboutthatsub-ideaattheendofgrades2,5,8,and12.

–  Cross-CugngConcepts(CCC)•  1)Paherns;2)Causeandeffect:Mechanismandexplana;on;3)Scale,propor;on,and

quan;ty;4)Systemsandsystemmodels;5)Energyandmaher:Flows,cycles,andconserva;on;6)Structureandfunc;on;7)Stabilityandchange


NGSSstructureandcontent-PEs

•  PerformanceExpecta;onsdevelopedaspartofNGSStodefineassessmenttargetsintermsofwhatstudentsshouldknowandbeabletodo

•  EachPEcombinesselectedDCI,SEP,andCCC

•  PEdevelopedwithinlifescience,physicalscience,andearth&spacesciencecontentareas;okenmul;plePEforeachdisciplineareabygrade


SamplePerformanceExpectation

NGSSassessmentissues&approaches-6/28/17 7Source:hhps://www.nextgenscience.org/resources/how-read-next-genera;on-science-standards

AssessmentcontextofNGSS

•  Adoptedby17statesandDCthusfar(2017).Severalotherstatesusingrelatedcontentframeworks.

•  Fourstateshavelaunchedopera;onalNGSS-alignedtests(IL,KA,NV,DC).AfewNGSSstatesarefield-tes;ngnewassessmentsin2016-17and/or2017-18;moststatesareinassessmentdesignstage.

•  FederalEveryStudentSucceedsAct(ESSA,2015)requiresstatestoassessscienceatleastonceingradespans3-6,6-9,and10-12.Assessmentmustbealignedtostate’scontentstandards—nofederalmen;onofNGSS.


ESSA<lexibilitysupportspossibleNGSSassessmentdesigns•  ESSAgivesstatesauthorityforseveralflexibili;esthatarerelevanttopossibleNGSS

assessmentdesigns

–  ESSAallowsstatestochoosetheircontentstandards;USEDmayallowstatestoassessasubsetofthe“instruc;onallearningcontentstandards”asPEareasubsetofthepossiblecombina;onsoftheNGSSSEP/DCI/CCC

–  ESSAexplicitlystatesthat“porqolios,performanceevents…”maybeusedinstateassessments;whilenotprohibitedbefore,thismaybereadassomeencouragement

–  ESSArequiresstatestohaveavalida;onargument;notclearhowrecep;veUSED/PeerReviewmightbetovalida;onargumentsaroundmorelimited(lessgeneralizable)claims.ThesemaybenecessarytoaddressthesparsesamplingintheNGSSPE

–  ESSAallows“banking”ofmathscoresofgrade8studentswhotakeastate’shighschoolmathtest

–  ESSAallowsstatestouseinterimassessmentstoproduceasumma;vescore,andtousea“locallyselected,na;onallyrecognizedhighschoolassessment”inlieuofthestateassessment.Thesemayprovidemodelsforhowtohandleinnova;veassessmentsthathavesimilarinnova;veaspects(e.g.,through-course;mul;ple,non-strictlycomparableassessments)

–  ESSAallowsaDemonstra;onAuthorityaroundcompetency-basedassessments,whichmayprovideguidanceforhowtohandlesimilarinnova;veNGSSassessments,suchasonesbasedonclaimsofcriterionmasteryandindividualizedadministra;oncondi;ons(e.g.,;ming:“moveonwhenready”)


Restofpresentation

•  Presentanevidence-centereddesignapproachtotestdevelopmenttodefinetermsandshowrela;onshipsofpartstoeachotherandtovalida;onargument

•  Discussthreedomaindefini;onissuesposedbytheNGSS,andsomepossiblesolu;onapproachesintermsofdomaindefini;ons,claims,andevidence

•  Forselectedpossiblesolu;onapproaches,discussafewmeasurementchallengesandpossiblesolu;ons


Evidence-CenteredDesignsummary

•  Assessmentisintended,bydesign,togatherevidencetosupportmakingaclaimaboutstudentperformanceinrela;ontosomething–  Definethe“something”—thedomain/”construct”

–  Definetheclaim(s)

–  Definehowtheclaimswillbereported

–  Definewhatevidenceisnecessarytosupporttheclaims

–  Definewhatinforma;onwillbegatheredandhowitwillbeprocessedtobecomeevidence

–  Designhowtheinforma;on-gatheringiscomprehensive,prac;cal,fair,etc.

•  Inprac;ce,usuallyitera;ve


Issue1:Verylarge,complexdomain

•  NGSSdefines–  8ScienceandEngineeringPrac;ces,39+DisciplinaryCoreIdeasub-parts,7Cross-CugngConcepts=2,184possiblecombina;ons[616ifrestrictedto11DCI]

•  Challenge:ThepossibledomaindefinedbythecompletecrossingsofSEPxDCIxCCCistoolargetoassessconven;onally(andprobablytoolargetolearnincurrentcondi;ons)

•  Solu;onapproach1:Reducedomainforlearning–  1A:Eliminateelementsorcombina;ons(manystates:usePEs)

–  1B:Combineelements(e.g.,“Inquiry”fromseveralSEP)


Large,complexdomain–2

•  Solu;onapproach2:Reducedomainforassessment

–  2A:Eliminateelementsorcombina;ons(NGSS=PEs)

–  2B:Combineelements,e.g.,combine8SEPinto2categoriesforrepor;ngandtestdesign

•  Solu;onapproach3:Reduceassessment

–  3A:Assessalimitedpor;onofthedomain(e.g.,assessDCI,notSEP)

•  Solu;onapproach4:Spreadoutassessment

–  4A:Rotatecoverageoveryears/forms

–  4B:Distributecoverageoverforms,matrixsampleoverstudents

–  4C:Assessovermul;pleyears


Issue2:NGSSPEsweakfordomainde<inition

•  PerformanceExpecta;onsdevelopedaspartofNGSStodefinewhatshouldbeassessedandspecifywhatevidence(i.e.,whatstudentsshouldbeabletodo)

•  Challenge1:ThesetofNGSSPEsaresosparselysampleditisdifficulttohaveenoughevidencetogeneralizeaboutindividualPEsortheircomponen;alDCI/SEP/CCCwithinoracrossgrades

•  Challenge2:ThesetofNGSSPEsdonothaveanobviousordocumentedstructureofhowthecombina;onofthemrelatetoaclaimabout“sciencelearning”inasinglegradeoracrossgrades


PEcoverageofSEPxDCIcontentarea,grades3-5


Grade AQDP DUM PCOI AID UMCT CEDS EAE OECIGrade 3

PS X XLS X X X XESS X X X

Grade 4PS X X X XLS X XESS X X X X

Grade 5PS X X X XLS X XESS X X X X

3-5 ETS X X X

NGSS Scientific &Engineering Practices (SEP) Addressed in the Performance Expectations

Scien;ficandEngineeringPrac;ces1.  Askingques;ons(forscience)anddefiningproblems(forengineering);2.  Developingandusingmodels;3.  Planningandcarryingoutinves;ga;ons;4.  Analyzingandinterpre;ngdata;5.  Usingmathema;csandcomputa;onalthinking;6.  Construc;ngexplana;ons(forscience)anddesigningsolu;ons(forengineering);7.  Engaginginargumentfromevidence;8.  Obtaining,evalua;ng,andcommunica;nginforma;on

DisciplinaryCoreIdeaAreas1.  PS=PhysicalSciences2.  LS=LifeSciences3.  ESS=EarthandSpaceSciences4.  ETS=Engineering,Technology,and

Society

PEcoverageofSEPxDCI,grade5

•  WhenthefullDCIsubareasandSEParelisted,thesamplingbyPEisextremelysparseandara;onaleforthesamplingisnotevident(tome)


AQDP DUM PCOI AID UMCT CEDS EAE OECI

PS1.A, Structure and Properties of Matter 5-PS1-1 5-PS1-3 5-PS1-2PS1.B, Chemical Reactions 5-PS1-4 5-PS1-2PS1.C Nuclear Processes PS2.A, Forces and Motion PS2.B, Types of Interactions 5-PS2-1PS2.C, Stability and Instability in Physical Systems PS3.A, Definitions of Energy PS3.B, Conservation of Energy and Energy Transfer PS3..C, Energy and Chemical Processes in Everyday Life PS3.D Waves and Their Applications in Technologies for Information Transfer

5-PS3-1

PS4.A, Wave Properties PS4.B, Electromagnetic PS4.C, Information Technologies and Instrumentation LS1.A, Structure and Function LS1.B, Growth and Development of Organisms LS1.C, Organization for Matter and Energy Flow in Organisms

5-LS1-1

LS1.D, Social Interactions and Group Behavior

LS2.A, Interdependent Relationships in Ecosystems

5-LS2-1

LS2.B, Cycles of Matter and Energy Transfer in Ecosystems

5-LS2-1

LS2.C, Ecosystem Dynamics, Functioning, and Resilience

LS2.D, Social Interactions and Group Behavior

LS3.A, Inheritance of Traits

LS3.B, Variation of TraitsLS4.A, Evidence of Common Ancestry LS4.B, Natural Selection LS4.C, Adaptation LS4.D, Biodiversity and Humans

ESS1.A, The Universe and Its Stars 5-ESS1-1

ESS1.B, Earth and the Solar System 5-ESS1-2ESS1.C, The History of Planet Earth ESS2.A, Earth Materials and Systems 5-ESS2-1ESS2.B, Plate Tectonics and Large-Scale System InteractionsESS2.C, The Roles of Water in Earth's Surface Processes

5-ESS2-2

ESS2.D, Weather and Climate ESS2.E, Biogeology

ESS3.A, Natural Resources

ESS3.B, Natural Hazards ESS3.C, Human Impacts on Earth Systems

5-ESS3-1

ESS3.D, Global Climate Change ETS1-A, Defining and Delimiting Engineering Problems

3-5-ETS1-1

ETS1-B, Developing Possible Solutions

3-5-ETS1-3

3-5-ETS1-2

ETS-C, Optimizing the Design Solution

3-5-ETS1-3

3-5

Engi

neer

ing

Des

ign

ETS1, Engineering Design

PS4, Wave Properties

LS1, From Molecules to Organisms:

Structures and Processes

LS2, Ecosystems: Interactions, Energy, and Dynamics

LS3, Heredity: Inheritance and

Variation of Traits

LS4, Biological Evolution: Unity

and Diversity

ESS1, Earth's Place in the

Universe

ESS2, Earth's Systems

ESS3, Earth and Human Activities

Phys

ical

Sci

ence

s Li

fe S

cien

ces

Earth

and

Spa

ce S

cien

ces

PS1, Matter and Its Interactions

PS2, Motion and Stability: Forces

and Motion

PS 3, Energy

Performance Expectations by DCI by SEP

Grade 5

EarthandSpaceSciencesEarth’sPlaceintheUniverse•  TheUniverseand

ItsStars(EAE)•  Earthandthe

SolarSystem(AID)

Domainandclaims

•  ItisnotclearhowthePEswereintendedtosupportaclaimaboutadomainatthegradelevelorgradespan.

•  Statesthatwanttomakeaclaimwillneedtocomeupwithoneontheirown;wecouldnotfindanyclaimsthatrequirethecombina;onofPEsasevidenceintheNGSSdocuments.•  Someillustra;vepossibleclaims:

•  “Thegrade3studentcandoW(SEP)withXDCI;thegrade4studentcandoY(SEP)withZDCI.”

•  “Thegrade3studentcandoW(SEP)withXDCI;thegrade4studentcandoWandX(SEP)withYandZDCI,assumingnoforgegng.”

•  “Ifthegrade3studentcandoW(SEP)withXDCI,wewillassumethestudentcandoall8SEPwithall15DCIbecauseallSEPxDCIweretaughtequallywell.”

•  “Ifthegrade4studentcandoX(SEP)withZDCI,wewillassumestudentcanalsodoW(SEP)withYDCIbecausethesePEareinadevelopmentallearningprogressionthatrequiresWxYinordertodoXxZ.”


Domainandclaims-2

•  ThePEsdonotappearselectedtosupportreliableclaimsaboutthewholedomain(nomaherhowreliablyanyonePEisassessed).•  IftheDCIatthelevelofLS/ES/PSweresubscores,the

opportuni;esforevidencewouldrangefrom2-4(outof8possible)foreachrowinaysinglegrade

•  IftheSEPweresubscores,nomorethanhalfwouldhavemorethanoneopportunityforevidence(PE)inanysinglegrade


NumberofSEP(outof8)withnumberof:mesevidenceiscalledforinNGSSdomains(outof3)

0 1 2 3

Grade3 1 5 2 0

Grade4 1 4 3 0

Grade5 1 3 2 2

PEsanddomainde<inition–3•  Solu;onapproach1:LimitclaimstoassessedPEs;donot

generalizebeyond–  1A:Treatmorelikediscretecriterion-referencedassessmentsthan

broadconstructassessments–  1B:MakeALDsandothersuppor;ngdocumentsappropriatelyspecific

•  Solu;onapproach2:SayPEsprovideevidenceforclaimsbroaderthanspecificallyassessedcontent/skills–  2A:TreatPEsasculmina;ngassessmentsoflearningprogressions,i.e.,

provideevidenceofmasteryofprecedingcontent/skills–  2B:SayPEsassessnarrowly,butwhatwaslearnedwaslikelybroader

becauseinstruc;onwasbroader

•  Solu;onapproach3:ModifythePEs(assessmenttargets)tosupportmoregeneralizedclaims


Reports

•  StateswillneedtodecidewhattoreportabouttheNGSSasthetotaland/or“subscores”

–  Eachofthreedimensionsseparately

–  Priori;zetwodimensions

–  Priori;zeonedimension

–  Reportallthreedimensionsasawhole

•  Stateswillneedtodecidehowto“scale”–  Uni-dimensional

–  Mul;-dimensional

–  Mul;pleseparatedimensionsandscales

–  Ver;cally;grade-spans;contentclusters;grade-level


Issue3:Evidencecollection

•  NGSScallforperformanceassessments;mostscienceeducatorshighlysuppor;veofperformanceassessments

•  Challenge1:Difficulttogatherenoughevidenceinrichinstruc;onalcontext

–  Solu;onapproach1:Provide(proxyfor)richinstruc;onalcontexts

•  1A:Simula;onsandothertechnology-enhancedassessmenttools(e.g.,NAEP)

•  1B:Usecurriculum/classroom-embeddedassessments


Evidencecollection-2

•  Challenge2:Evidencemaybegatheredunderlessstandardizedcondi;ons–  Solu;onapproach1:Worktostandardizecondi;ons

•  1A:Providegenerallyapplicablerubrics•  1B:Intensivescorertraining,monitoring,modera;on

–  Solu;onapproach2:Provideassurancesofcomparabilitythroughexternalevidence•  2A:Usecomparabilityoncommontaskstosupportclaimofcomparabilityonnon-commontasks

–  Solu;onapproach3:Makelessstandardizedcondi;onspartoftheconstructand/orclaim•  3A:Aspectsofstandardizedtestadministra;ondonotmaher,e.g.,Competency-basedandcombiningmul;pleinterimassessmentsinvolvenon-standardiza;onof;mingofgatheringevidence—criterion-referencedmasteryconstructsays;mingofperformancedoesnotmaher(e.g.,ESSAgrade8advancedmathscorebanking)


Evidencecollection-3

•  Challenge3:Performanceassessmentsmaynotgeneralizeacrosscontent,skill,context(e.g.,Shavelsonetal.,1993,19941)–  Solu;onapproach1:Increaseinstruc;onandprac;cetosupport

moregeneralizedknowledgeandskills

–  Solu;onapproach2:Incorporatemoreinforma;ontoprovidemorereliablescore

•  2A:Assessmoresamplesacrossrangeofreplica;oncondi;ons

•  2B:Incorporateinforma;onfromotherperformances(e.g.,Quals+Disserta;on)

–  Solu;onapproach3:Changeclaimstoreducegeneralizability•  3A:“Student’sbestwork”(likeKYWri;ngporqolios)

•  3B:“Individualizedareaofspecializa;on”(likeappren;ceshiporcapstoneproject,likeRI’spreviousPBGR)


1-hhps://www.researchgate.net/profile/Richard_Shavelson/publica;on/233284543_Generalizability_of_Large-Scale_Performance_Assessments_in_Science_Promises_and_Problems/links/53d12ff80cf220632f391ecf.pdf

Summary•  NGSSpresentconsiderablechallengesregardingdomaindefini;onanddesignofevidenceto

supportclaimsinalarge,complexdomaindefinedbySEP/DCI/CCCandsparselysampledPEscombinedwithacallforextensiveperformanceassessments

•  Therearepossiblesolu;onsforeachoftheNGSSchallengesdiscussed.Thereisnoone“rightanswer”tohowtoassesstheNGSS.Gegngastronganswerwillinvolvethoughqul,itera;vework.

•  ESSAprovidesstatesflexibilityinseveralwaysthatsupportthesetypesofpossiblesolu;ons,includingspecificapproachesalreadybeingexploredbystates

–  ESSAallowsstatestoadopt/specifycontentstandardsandconstructvalida;onargumentsregardingassessingclaimsbasedonthedefineddomain.StatesmayinterprettheNGSStocreatestate-specificdomaindefini;ons,claims(e.g.,regardinggeneraliza;on),andevidencedesigns,suchasALDsandrepor;ngcategories

–  ESSAexplicitlystatesthat“porqolios,performanceevents…”maybeusedinstateassessments(Severalstatespursuingperformanceassessmentinvariousforms)

–  ESSAallows“banking”ofmathscoresofgrade8studentswhotakeastate’shighschoolmathtest.(ThisapproachmaybeusedinNGSSdesign)

–  ESSAallowsstatestouseinterimassessmentstoproduceasumma;vescore,andtousea“locallyselected,na;onallyrecognizedhighschoolassessment”inlieuofthestateassessment.Thesemayprovidemodelsforhowtohandleinnova;veNGSSassessmentsthataresimilarinsomeaspects(e.g.,through-course;mul;ple,non-strictlycomparableassessments).

–  ESSAallowsaDemonstra;onAuthorityaroundcompetency-basedassessments,whichmayprovideguidanceforhowtohandlesimilarinnova;veNGSSassessments,suchasonesbasedonclaimsofcriterionmasteryandindividualizedadministra;oncondi;ons(e.g.,;ming:“moveonwhenready”).

•  Itwillbeexci;ngtoseeandworkwithstatesandUSEDtohelpbringhighqualityNGSSassessmentsintoopera;onalexistenceoverthenextfewyears!


Questions?Comments?

Thankyou!


Formoreinformation:

CenterforAssessmentwww.nciea.org

[email protected]

26NGSSassessmentissues&approaches-6/28/17

essa innovative assessment designs: examples from science

Documents