Emily JolleyLet’s Measure Together Project –

Constructed Response

Mentor: Dr. Tammy Howard, DPI August 6, 2013

kenanfellows.org dpi.state.nc.us/accountability

Kenan Fellows Externship

• 5 week externship this summer at the Testing and Accountability office of DPI

• Mentor is Dr. Tammy Howard – director of the testing and accountability division for DPI

• Attended department meetings, met with test development staff, item writers, content specialists, psychometricians, and just about everyone else involved in testing and accountability.

NC DPI Accountability Services Division

• The mission of the Accountability Services Division is to promote the academic achievement of all North Carolina public school students and to assist stakeholders in understanding and gauging this achievement against state and national standards.

NC DPI Accountability Services Division

The major thrust of this mission is three-fold: 1. the design and development of reliable

and valid assessment instruments2. the uniform implementation of and access

to suitable assessment instruments for all students

3. the provision of accurate and statistically appropriate reports.

EOCs and EOGs

• Fulfill requirements of Federal NCLB Act of 2001.• Developed by NCSU Technical Outreach for Public

Schools (TOPS) Program as directed by DPI• Questions are written by North Carolina teachers

and then edited by TOPS and DPI.• Standardized administration and grading statewide.• Rigorous development process that takes years to

complete.• Used to assess student learning and growth.

Measures of Student Learning/Common Exams

• Required by Race to the Top Grant.• Developed by NCSU Technical Outreach for Public

Schools (TOPS) Program as directed by NC D.P.I.• Questions are written by North Carolina teachers.• Not standardized, considered a local assessment

administered and graded by individual Local Education Authorities (LEAs)• Used to assess teachers for Standard 6

Item Writing Trainings

• Attended testing item writing training for teachers by TOPS.• Focused on breaking down the standards into

Revised Bloom’s Taxonomy verbs, standard objects, and supporting items.• After learning how to break down the

standards, basic instructions were given about writing items.

Item Writing Trainings

Revised Bloom’s Knowledge Dimensions:•Factual – basic elements•Conceptual – interrelationships•Procedural – how to do something•Metacognition – awareness of what you know

Revised Bloom’s Cognitive Processes:•Remember, Understand, Apply•Analyze, Evaluate, Create

Item Writing TrainingsFor example – breaking down a Civics & Economics standard: CE.C&G.2.7: Analyze contemporary issues and governmental responses at the local, state, and national levels in terms of how they promote the public interest and/or general welfare (eg: taxes, immigration, naturalization, civil rights, economic development, annexation, redistricting, zoning, national security, health care, etc)

Unifying Concept (found in unpacking doc)

ES/CO RBT Verb Object In Terms Of: Examples:

Government Systems, Structure, Function

CE.C&G.2.7

Analyze contemporary issues and governmental responses at the local, state, and national levels

in terms of how they promote the public interest and/or general welfare

taxes, immigration, naturalization, civil rights, economic development, annexation, redistricting, zoning, national security, health care, etc

Most essential standards for high school social studies fall under the understand or analyze range of conceptual knowledge. A few fall under the understand or evaluate range, and a couple standards are procedural knowledge (especially in World History & American History I & II).

Why Constructed Response?

• Constructed Response questions were included in MSLs for a variety of reasons:•Almost none of the new standards are based on

factual knowledge only. Because the new essential standards are more conceptual, having only multiple choice questions is not the best way to assess these standards•Many teachers have been asking for assessments

that more closely reflect students’ learning, and constructed response questions help facilitate that request

Why Constructed Response?

• As teachers, we need to ensure that the written, taught, and tested curriculum are aligned. •We can of course go beyond what the

standards require, but we need to make sure we’re teaching and assessing the students at least at the level the standards are written.

What about the Common Core State Standards?

• We also have literacy standards for history/social studies. These are not technically tested by MSLs, but we should still be teaching them with fidelity to help our students become better readers and writers. A sample reading standard:• CCSS.ELA-Literacy.RH.9-10.8: Assess the extent to which the

reasoning and evidence in a text support the author’s claims.

Unifying ConceptES/CO RBT Verb Object In Terms Of:

Reading in History/Social Studies

RH.9-10.8 Assess the extent to which reasoning and evidence in a text supports the author’s claims

n/a

Almost all of the Common Core State Standards for literacy in high school social studies fall under the apply, analyze, or evaluate ranges of procedural knowledge, with a few falling under the create range of procedural knowledge (especially the writing strand).

Essential Standards Breakdown

• Now, let’s practice breaking down our own standards into this basic format. Use the template from my site to start breaking down your standards:

• The unpacking documents can also be found on my site. It may be more useful (along with the examples in the standards) when figuring out what content to teach.

Unifying Concept (found in unpacking doc)

ES/CO RBT Verb Object In Terms Of: Examples:

Quick Assessment

•When you’ve broken down some of your standards, take the quick survey on my site to assess your comfort with this process.• The survey has some standards from each course. It

also has a few of the released constructed response questions from last fall. You should categorize each by RBT cognitive process and knowledge dimension.• I highly recommend breaking down all the standards

from your course this way (and looking through the unpacking document!)

Constructed Response Criteria

• Now we’re ready to write!• You need a few things to begin writing a

constructed response question. Basic format:• Stimulus or Context Statement (optional): chart,

map, picture, graph, text, quote, intro, or scenario•Prompt or Task Statement: directive to students•Requirements Statement: response specifics• Scoring Criteria: rubric, range 0-4 points

C&E Fall 2012 CR Released Example:

• Stimulus/Context Statement: Throughout its history, American democracy has upheld and strengthened citizens’ individual rights. • Prompt or Task Statement: Take a position that argues for or

against the above statement. • Requirements Statement: Provide at least two specific historic

examples (e.g., legislation, Supreme Court cases, executive orders, or other actions by the federal government) that support your position.

• Scoring Criteria: • Score 0 No response or the response does not address the prompt. • Score 1 Fulfills only 1 of 3 requirements of a level 3 performance • Score 2 Fulfills 2 of 3 requirements of a level 3 performance • Score 3 Takes a position that argues for or against the statement; provides one

historic example that supports the position; provides a second historic example that supports the position

C&E Fall 2012 CR Released Example:

• Throughout its history, American democracy has upheld and strengthened citizens’ individual rights. • Take a position that argues for or against the above

statement. Provide at least two specific historic examples (e.g., legislation, Supreme Court cases, executive orders, or other actions by the federal government) that support your position. • Standard Alignment: • CE.C&G.2.6 Evaluate the authority federal, state, and local

governments have over individuals’ rights and privileges (e.g., Bill of Rights, Delegated Powers, Reserved Powers, Concurrent Powers, Pardons, Writ of habeas corpus, Judicial Process, states’ rights, Patriot Act, etc.).• CE.C&G.3.8 Evaluate the rights of individuals in terms of how well

those rights have been upheld by democratic government in the United States.

Constructed Response Tips

• Start with the objective(s) you want to measure. Most constructed response questions will satisfy more than one objective. • You must use the same RBT level that the standard requires.

(You also must use the same content… obviously!)• You should adhere to the same basic scoring criteria

whether you are creating a 2, 3, or 4 point question. Remember, students don’t see this criteria on the MSL, so you should teach them how to pick out points (there’s a PowerPoint about this on my site). • Review the released questions and use the RBT question

stems if you need help getting started!

Constructed Response

• Choose at least 2 objectives from your course and write your own constructed response using the template provided.

• After writing your constructed response, please trade it with a partner. You should break down each other’s CR and determine if it fits the criteria and matches the content and RBT levels of the standard.

Public Google Spreadsheet

• Please load the constructed response question you created today into the Google spreadsheet found on my site: https://docs.google.com/spreadsheet/ccc?key=0AtoXZ1jIf_QadDJ5N3RDTC0zZlhsSnVpOHpJSEE4Wmc&usp=sharing • This is a public spreadsheet that anyone can

access. I will update it throughout the year with constructed response questions I write (mostly for C&E), and I hope you will do the same.• Please share with everyone!

Other Resources

• TOPS suggested these resources for writing constructed response questions and understanding concept-based instruction:• Lynn Erickson - Concept-Based Curriculum and

Instruction • Lorin Anderson & David Krathwohl - A Taxonomy

of Learning, Teaching, and Assessing•Robert Mager - Measuring Instructional Results •RBT Resources (Section 5):

http://tpri.wikispaces.com/space/content

Contact Info

• Please contact me if you have any questions:

Emily JolleyMillbrook High School

ejolley@wcpss.net

Preparing Students for the Real World

Takes Teachers Who Learn in the Real World

BECOME A KENAN FELLOW

•See first-hand how researchers and industry experts use STEM skills in real-world settings

•Apply that knowledge to better prepare students for the jobs of tomorrow

•Enrich your career through professional development opportunities

•Serve as an ambassador for educational excellence in your school, community and across the state

Find out more: kenanfellows.org

Kenan Fellows Program Timeline

Kenan Fellows Program Stay Connected

facebook.com/KenanFellows

@KenanFellows

Kenan Fellows Program Group

youtube.com/user/KenanFellows

pinterest.com/kenanfellows