division of performance accountability department of assessment division of performance...

Division of Performance Accountability

Department of Assessment

Division of Performance Accountability

Department of Assessment

DEVELOPING ASSESSMENT ITEMS

USING THE FCAT

SPECIFICATIONS (SPECS)


USING THE FCAT

SPECIFICATIONS (SPECS)Nancy E. Brito, NBCT, Instructional Specialist

Senella Johnson, Test Development Specialist

Dr. Carla Kendall, Manager, Online Assessment

561-434-8850, PX48850

Science: 2002

Mathematics: 2005

The FCAT Specifications referenced in this Power Point are:

The Spring 2010 FCAT items are based on the benchmarks in these FCAT Specifications.

Reading:

2001

What you should know about the Spring 2010 FCAT

Reading and Mathematics will contain Performance Tasks: Short Response (SR) and Extended Response (ER)

Science will NOT contain Performance Tasks: SR and ER

Begin by taking a Released FCAT or

Sample Test in your content area and grade

level/cluster.

Experience what your students will

experience!

TARGET:

To utilize the FCAT Test Items and Performance Task Specifications as a guide to identify/develop/modify grade level appropriate questions that are aligned to the Sunshine State Standards benchmarks and are of the same cognitive rigor as the Florida Comprehensive Assessment Test (FCAT).

PRINCIPLES FOR QUALITYCLASSROOM ASSESSMENT*

Principle 1. Quality assessments arise from and accurately reflect clearly specified, appropriate,and essential learning targets.

Principle 2. Quality assessments are specificallydesigned and focused to serve instructionalpurposes.Principle 3. Quality assessments accurately reflectthe intended target and serve the intendedpurpose.

* From Article: Assessment Literacy for Educators: Linking Instruction and Assessment by Carolyn T. Cobb, Jeane M. Joyner, and Janet L. Williamson

continued

Principle 4. Quality assessments promote equityby providing ample opportunities for students todemonstrate their learning.

Principle 5. Quality assessments provide sufficientevidence of learning to permit confident conclusions (inferences) about student achievement.Principle 6. Quality assessments are designed,developed, and used in ways that eliminate bias ordistortion that may interfere with the accuracy ofresults.

PRINCIPLES FOR QUALITYCLASSROOM ASSESSMENT*

A well designed classroom assessment provides information on the strengths and challenges of students.

What do my students know? What are my students able to do?

What do my students know? What are my students able to do?

KINDS OF ASSESSMENTS…

JOURNALS PORTFOLIOS PROJECTS TESTS

Written Oral Skill

ULTIMATELY we want to…

determine if students mastered a TAUGHT concept

check the comprehension of both the knowledge and skill TAUGHT

In order to do this we must set STANDARDS…

STANDARDS provide expectations for student

achievement

provide direction for student learning (content & process)

provide an instructional design that caters and focuses the curriculum

STANDARDS

Curriculum

Grades Nine Weeks ExamSubject Specifics

High-Stakes Tests

ACTFCATSAT

What is the Driving Force?

There must be Alignment.There must be Alignment.

Curriculum Instruction

Assessment

SSS

What is the STANDARD for Florida?

Sunshine State Standards

Academic Standards

What do you know about academic standards?

How do you use the Sunshine State Standards in your

classroom?

Academic StandardsAcademic Standards

• Are the WHAT of education while curriculum and instruction are the HOW

• Apply to ALL students

• Are based on the belief that, given the right opportunities, EVERY CHILD CAN LEARN

Continued…

Why academic standards?

• Show teachers how well their students are doing by looking at their progress towards mastery of standards

• Show stakeholders what schools are aiming to teach and what students must learn

• Serve as an important tool for equity

• Provide yardsticks for evaluating all aspects of schooling, that is, textbooks, staff development workshops, schedules, resources, personnel assignments, etc.

HOW WELL DO YOU KNOW THE

STANDARDS?

What are the Language Arts SSS strands?

A. ReadingB. WritingC. Listening, Viewing, and SpeakingD. LanguageE. Literature

Which are assessed by FCAT?

A. ReadingB. WritingE. Literature

For purposes of reporting FCAT scores reading and literature benchmarks are organized in clusters:

1. Words and Phrases in Context

2. Main Idea, Plot, and Author’s Purpose

3. Comparisons and Cause/Effect

4. Reference and Research

Strands

Standards

Benchmarks

GLEs

5 11 36 (Grade 2) 66 (Grade 2)

51 (Grades 3-5)

92 (Grade 3)

94 (Grade 4)

101 (Grade 5)

52 (Grades 6-10)

114 (Grade 6)

115 (Grade 7)

110 (Grade 8)

What is the scope of the Language Arts SSS?

A. Number Sense, Concepts & Operations

B. Measurement

C. Geometry & Spatial Sense

D. Algebraic Thinking

E. Data Analysis & Probability

What are the Mathematics SSS strands?


ALL Strands

What is the scope of the Mathematics SSS?

Strands

Standards

Benchmarks

GLEs

5 17 34 (Grades 2-5)

110 (Grade 2)

36 (Grades 6-10)

111 (Grades 3-5)

130 (Grades 6-8)

A. The Nature of Matter

B. Energy

C. Force & Motion

D. Processes That Shape the Earth

E. Earth & Space

F. Processes of Life

G. How Living Things Interact with Their Environment

H. The Nature of Science

What are the Science SSS strands?

1. Physical and Chemical Sciences

2. Earth and Space Sciences

3. Life and Environmental Sciences

4. Scientific Thinking Sciences

For purposes of reporting FCAT scores benchmarks are organized in clusters:


ALL Strands

What is the scope of the Science SSS?

StrandsStandar

dsBenchmar

ksGLEs

8 17 41 (Grade 2)54 (Grade

2)

58 (Grades 3-5)

55 (Grade 3)

60 (Grade 4)

59 (Grade 5)

73 (Grades 6-8)

97 (Grade 6)

98 (Grade 7)

100 (Grade 8)

77 (Grades 9-12)

NA

How should we use the Sunshine State Standards?

To

PLAN and GUIDE

classroom instruction

How should we use the Sunshine State Standards?

To DEVELOP

classroom

assessments

StandardsStandards

Your turn!

Reading, Mathematics, & Science

Activity time!

Pick a benchmark and identify the mayor

differences across the grades.

The scope of the tested benchmarks are presented in Appendix B of the FCAT Specifications.



FCAT Test Item Specifications

What are the 7characteristics of a

“good” item writer?

1. Knows the CONTENT being tested

2. Is aware of the OBJECTIVES being tested

3. Knows/understands the STUDENTS being tested

4. Has skills in written COMMUNICATION

Continued…

What are the 7characteristics of a

“good” item writer?

5. Has skills in TECHNIQUES of item writing

6. Knows the FORMAT of high stakes test items

7. Knows how to ALIGN the assessment with the FCAT

Is there a simple way for every teacher to help students be prepared for FCAT?

Yes! Incorporate FCAT-like questions into regular lessons, activities, and assessments.

GETTING INTO THE FCAT

SPECIFICATIONS (SPECS)

What do the tags mean? The tags below refer to the subject

specific FCAT Specification page number for the appropriate subject.

Reading Mathematics

Science

Page # Page # Page #

LET’S EXPLORE AND EXAMINE!

• Explore the Reading, Mathematics, and Science FCAT Item Specifications (SPECS)

• Examine the guidelines that apply to developing FCAT test items

WHAT’S THE TARGET?

By the end of this presentation you will be able to:

• Explain the purpose of the FCAT SPECS

• Use the FCAT SPECS as a resource to create classroom assessments that are aligned to Sunshine State Standards and follow FCAT item creation specifications

WHAT DO YOU NEED?

The FCAT Specification

• Reading,

• Mathematics, or

• Science

Posted at the FDOE Web Site below:

http://fcat.fldoe.org/fcatis01.asp#specs

http://fcat.fldoe.org/fcatis01.asp#specs

WHAT DO YOU NEED?

Released or Sample FCAT test

• Reading,

• Mathematics, or

• Science

Released tests at:

http://fcat.fldoe.org/fcatrelease.asp

Sample tests at:

http://fcat.fldoe.org/fcatitem.asp

http://fcat.fldoe.org/fcatrelease.asp

http://fcat.fldoe.org/fcatitem.asp

WHAT DO YOU NEED?

Cognitive Complexity Classification Level

• Reading,

• Mathematics, or

• Science

Posted at:

http://fcat.fldoe.org/pdf/cog_complexity-fv31.pdf

http://fcat.fldoe.org/pdf/cog_complexity-fv31.pdf

WHAT DO YOU NEED?

FCAT Test Design Summary

• Reading

• Mathematics

• Science

Posted at:

http://fcat.fldoe.org/pdf/fc05designsummary.pdf

http://fcat.fldoe.org/pdf/fc05designsummary.pdf

WHAT IS THE PURPOSE OF THE FCAT SPECS?

• Describe the test items and performance tasks found in the FCAT assessments

• Provide general and specific guidelines for development of all test items and performance tasks at the grade levels assessed by the FCAT

Reading Mathematics

Science

Page 1 Page 1 Page 1

Reading

• Address the criteria that apply to the selection of Reading passages for FCAT

PART I: GENERAL SPECIFICATIONS PART I: GENERAL SPECIFICATIONS

Mathematics & Science

• Address the topics and development considerations that apply to all Mathematics and Science test items

Reading Mathematics

Science

3-13 2 6

BIAS, SENSITIVITY, AND BARRIERS

Potential Bias and Distortion to Accurate Assessments

BIAS

Types of bias: Gender Race/ethnicity Geographic Religious Socioeconomic

Family

environment Physical

Visually impaired Linguistic

Word usage Regional words Spelling

Reading Mathematics

Science

6, 8 33 7

SENSITIVITY

Issues of potential concern to members of community Inappropriate materials Religious Cultural Ethnic Fears Phobias

Your turn!


Take a look!Become familiar with the General Specifications.

GENERAL SPECIFICATIONS!GENERAL SPECIFICATIONS!

Reading Mathematics

Science

3-13 2 6

• Types of Passages

• Sources of Passages

• Quality of Passages

• Potential Modifications to Passages

WHAT ARE THE CRITERIA FOR PASSAGE SELECTION?

• Diversity

• Graphics

• Reading Level

• Passage Length

Reading Mathematics

Science

3-13 NA NA

TYPES OF PASSAGESTYPES OF PASSAGES

Literary Text or Informational Text

Examples of Informational and Literary Text

Reading Mathematics

Science

3, 4 NA NA

SOURCES OF PASSAGESSOURCES OF PASSAGES

USE

• Magazines

• Books

• Newspapers

• Children’s & Young People’s Journals

• Advertisements

• Brochures

AVOID

• Basal Text

• State-adopted textbooks

• Trade books and magazines familiar to students

Reading Mathematics

Science

5 NA NA

QUALITY OF PASSAGESQUALITY OF PASSAGES

• Are excerpts or complete section

• Contain recognizable key concepts

• Are well written and authentic

• Are interesting and appealing to students at the grade level

• Are grade level appropriate

Passages…

Reading Mathematics

Science

5 NA NA

POTENTIAL MODIFICATIONS TO PASSAGES POTENTIAL MODIFICATIONS TO PASSAGES

• Remove, replace or footnote a word or phrase above grade level

• Use footnote – for grade 4 and above

• Use parenthesis to explain a word – for grade 3

Note: Passages should NOT be edited simply to satisfy length requirements.

Make sure meaning is NOT altered

Reading Mathematics

Science

5 NA NA

CONTENT OF PASSAGESCONTENT OF PASSAGES

• Are relevant to the student

• Are free from bias or stereotyping

Passages…

Appendix A

• Are based on topics presented in the Sunshine State Standards

• Contain content for at least 8 to 12 items

• Are engaging and age-appropriate

Reading Mathematics

Science

6 NA NA

Passages reflect…

• A variety of interests and backgrounds of Florida student population (characters, setting & situations)

DIVERSITYDIVERSITY

• A range of cultural diversity (written by and about different cultures and races)

Reading Mathematics

Science

6 NA NA

READING LEVELSREADING LEVELSPassages…

• Are at the grade level of the students tested (Lexile)

• Represent a range (appropriate for both the beginning and the end of the grade)

PASSAGE LENGTHPASSAGE LENGTHPassages…

• vary within grade levels

Reading Mathematics

Science

7 NA NA

SUITABILITY FOR ITEMSSUITABILITY FOR ITEMS

Passages…

• Are substantial enough to support test items that cover a range of benchmarks

• Contain appropriate text for assessing certain standards, i.e. synonyms, antonyms, multiple meaning words

Reading Mathematics

Science

7 NA NA

GRAPHICS - READING PASSAGESGRAPHICS - READING PASSAGES

• May include photographs, illustrations, maps, charts, graphs, advertisements, and schedules

• Help students understand the text

• Provide supplemental information for answering the question

• Illustrate or support the context of the question

• Have appropriate labels, legend, keys, captions

Reading Mathematics

Science

6 NA NA

OVERALL CONSIDERATIONSOVERALL CONSIDERATIONS

Mathematics & ScienceItems…

• Are written to measure primarily one benchmark

• Are appropriate for students in terms of grade-level difficulty, life experiences, & reading level

• Range in difficulty from easy to challenging

• Are approximately one (Reading) grade level below the grade level of the test

• Do NOT disadvantage or disrespect any segment of the population

Reading Mathematics

Science

NA 2 6

ITEM CONTEXTITEM CONTEXTRefers to the situation in which a question is presented

• Designed to interest students at the tested level

• Presented in real-world context or related to real-world situations

• Presented through written text and/or visuals


Reading Mathematics

Science

NA 36 7

What are the criteria for FCAT Reading, Mathematics, & Science test items?

• Cognitive Levels – Updated July 2006 - see Cognitive Complexity Classification document

• Item Style Format – Updated September 2005 – see Test Summary and Design document

• Scope of Items

Reading Mathematics

Science

9 3 7-16

Application

Knowledge

Comprehension

Analysis

Synthesis

Evaluation

Level I

Level II

Old Classification

Bloom’s Taxonomy

LOW

MODERATE

HIGH

New Classification

Web’s Depth of Knowledge

& NAEP

VS

Summary of Tests and Designs Document

COGNITIVE COMPLEXITY CLASSIFICATION


Reading Mathematics Science

LowRead the line.

Require to recall, recognize or represent

basic facts; basic understanding of text.

Requires recalling and recognizing of concepts

and principles

Requires recalling and recognizing concepts and

principles.

ModerateRead between

the line.

Requires more than simple recall; apply skill; two-step process; simple

inference

Requires a response beyond the habitual; more than one step;

problem solving strategies

Requires a response that goes beyond the

habitual, is not specified, and ordinarily has more

than a single step or thought process.

HighRead beyond

the line.

Requires analysis and synthesis of information;

heavy demand on student thinking, abstract thinking and reasoning.

Requires abstract reasoning, planning,

analysis, judgment, and creative thought.

Requires abstract reasoning, planning,

analysis, judgment, and creative thought.

Cognitive Complexity Classification of FCAT SSS Test Items



COGNITIVE COMPLEXITY LEVELS!COGNITIVE COMPLEXITY LEVELS!

Reading

READING:EXAMPLES OF COMPLEXITY

Maria’s dog Spot almost knocked her down as she entered the door. Maria quickly noticed that all bowls were empty and that she’d forgotten to leave food out. Spot could hardly wait for Maria to fill the bowls as he hurriedly ate. Later when Maria returned to the kitchen, she couldn’t help but notice

the smile on Spot’s face as he slept.

Low- What was the name of Maria’s dog? (Read the Lines)

Moderate- What was Spot’s problem? (Read Between the Lines)

High- Why was Spot MOST LIKELY smiling in his sleep? (Read beyond the Lines)

L M H


Mathematics

MATHEMATICSEXAMPLES OF COMPLEXITY

LM H


Science

SCIENCE EXAMPLES OF COMPLEXITY

Mr. Molina’s class studied the mass and volume of three liquids.

What is the difference in density between the least and the greatest density of the liquids listed in the table?A. 0.09 g/cm3

B. 0.10 g/cm3

C. 0.16 g/cm3

D. 1.08 g/cm3

Liquid Mass (grams) Volume (cm3)

Corn syrup 10.8 10.0

Salad oil 23.0 25.0

Vinegar 30.3 30.0

LM H

Your turn!


Determine the cognitive complexity level of the items on the FCAT Released or Sample Test you experienced at the beginning of this presentation.

Check your answers against the Test Book with answers found in the FDOE web site.

COGNITIVE COMPLEXITY CLASSIFICATION LEVEL

COGNITIVE COMPLEXITY CLASSIFICATION LEVEL

ITEM STYLE AND FORMATITEM STYLE AND FORMAT

• Multiple Choice

• Short Response

• Extended Response

• Gridded Response



• Express as a question when possible

• Capitalize or use bold to emphasize a word (grade appropriate)

• Avoid gender-specific pronouns – use plural

• Use appropriate balance of male/female names



Reading Mathematics

Science

10-13 3-7 9-14

Your turn!


Look carefully at this section in the FCAT SPECS.

Important information is included that helps you prepare your students for the FCAT


Reading Mathematics

Science

10-13 3-7 9-14

GRAPHICS – MATHEMATICS AND SCIENCEGRAPHICS – MATHEMATICS AND SCIENCE

• Provide necessary information for answering the question

• Illustrate or support the context of the question

• Depend on the benchmarks

Used with 30% of the Mathematics items

Used with 40% of the Science items

Reading Mathematics

Science

NA 3 9

SCOPE OF ITEMSSCOPE OF ITEMS

Refers to the benchmarks (Appendix)


REFERENCE SHEETSREFERENCE SHEETS

• Contain appropriate formulas and conversions

• Are provided to students in grades 6 - 10 for use during test

• Are NOT provided for grades 3 – 5 (provided with each item)


Reading Mathematics

Science

NA 29-32 17-19

Reading Science

Mathematics

PART II: STRUCTURE OF THE SPECIFICATIONS

PART II: STRUCTURE OF THE SPECIFICATIONS

Reading Mathematics

Science

23 34-35 20

• Strand

• Standard

• Benchmark

• Benchmark Clarification

• Item Types

• Response Attributes

• Sample Items

• Content Limits

• Passage Attributes

• Definitions

• Item Context

SPECIFICATION SECTIONSSPECIFICATION SECTIONS

Dark Blue

Reading Mathematics

Science

14 34-35 22-23

STRAND: CATEGORY OF KNOWLEDGE STRAND: CATEGORY OF KNOWLEDGE

Strands assessed in Reading:

• Reading

• Literature


1. Words and Phrases in Context

2. Main Idea, Plot, and Author’s Purpose

3. Comparisons and Cause/Effect

4. Reference and Research

22

Reading Mathematics

Science

14 NA NA

STRAND: BROAD CONTENT AREA


Strands assessed in Mathematics:

• Number Sense, Concepts & Operations

• Measurement

• Geometry & Spatial Sense

• Algebraic Thinking

• Data Analysis & Probability

55

Reading Mathematics

Science

NA 34 NA



• The Nature of Matter

• Energy

• Force & Motion

• Processes That Shape the Earth

88• Earth & Space

• Processes of Life

• How Living Things Interact with Their Environment

• The Nature of Science

Strands assessed in Science:

1. Physical and Chemical Sciences

2. Earth and Space Sciences

3. Life and Environmental Sciences

4. Scientific Thinking Sciences


Reading Mathematics

Science

NA NA 22

STANDARDSTANDARD

• General statements of expected student achievement within each strand

• The same for all grade levels

Example:

The student selects or uses appropriate units and instruments for measurement to achieve the degree of precision and accuracy required in real-world situations.

Reading Mathematics

Science

14 34 22

BENCHMARK BENCHMARK

• Specific statements of expected student achievement

• Different for the different grade levels

Example:

MA.B.4.2.2. The student selects and uses appropriate instruments and technology, including scales, rulers, thermometers, measuring cups, protractors, and gauges, to measure in real-world situations.

Reading Mathematics

Science

14 34 22

ITEM TYPES ITEM TYPES Address the types of items used to assess each benchmark at each grade level

Reading Mathematics Science

MC MC MC

ER GR GR

SR SR SR

ER ER

MC = Multiple Choice GR = Gridded Response, SR = Short Response ER = Extended Response

Reading Mathematics

Science

14 34 22

Not applicable Spring 2010 FCAT Science

BENCHMARK CLARIFICATION BENCHMARK CLARIFICATION

• Contains the task(s) a student will perform when responding to questions

Example:

Students will identify appropriate tools or read measurements from tools.

Reading Mathematics

Science

15 34 22

DEFINITIONS DEFINITIONS

Describe terms used in the benchmarks that require further explanation

Example:

Nonstandard units are objects, such as pencils, crayons, or paper clips, used to obtain a measure.

CONTENT LIMITS CONTENT LIMITS

Contain the range of content knowledge and degree of difficulty that should and should not be assessed in the questions for the benchmark

Example:

Items will involve determining appropriate measure of length, weight, or capacity for common objects.

Customary and metric units may be used, but not in the same item.

Reading Mathematics

Science

16 16-28 14 (22)

STIMULUS ATTRIBUTES ND PASSAGE ATTRIBUTES

STIMULUS ATTRIBUTES ND PASSAGE ATTRIBUTES

Defines types of stimulus materials:

e.g.,

• appropriate use of articles/passages,

• graphic materials, and

• item context or content

Example:

Graphics should be used in at least 50% of these items.

Reading Mathematics

Science

16 35 22

RESPONSE ATTRIBUTES

GUIDELINES TO WRITERS

RESPONSE ATTRIBUTES

GUIDELINES TO WRITERS

• Give specific description of the distractors for Multiple-Choice items

• Contain specific directions for the types of responses allowable

Example:

When responses are measuring devices, graphics of the devices should be included as well as the device name.

Reading Mathematics

Science

Benchmark Specific 14

Benchmark Specific

Reading and Science

Mathematics

SAMPLE ITEMS SAMPLE ITEMS

• Provided for each type of question assessed

• Presented in a format like that used in the FCAT test

Reading Mathematics

Science

15 35 23

ITEM CONTEXT ITEM CONTEXT

• Identifies the broad subject areas represented by the content of the item

• Pertains only to the Mathematics FCAT test

Example:

The Arts, Science, Social Studies

Reading Mathematics

Science

NA 36 NA

APPENDIXES APPENDIXES Contain…

• FCAT topics

• Content assessed by the FCAT and item formats by benchmark

• FCAT Mathematics and Science Glossary

• FCAT Mathematics and Science Scoring Rubrics

By Grade Level

PUTTING IT ALL TOGETHER!PUTTING IT ALL TOGETHER!

Your turn!


Activity time!

Choose a benchmark and create an original item using the FCAT SPECS!

ORIGINAL ITEM!ORIGINAL ITEM!

Steps to developing an assessment item

Choose the benchmark you want to assess Choose a passage (Reading only) Go over the FCAT Specifications for that

benchmark Determine the Cognitive Complexity Level Determine item type Follow design and format Make the distractors diagnostic in nature

Statements about your assessment item

Measures the Benchmark Adheres to the

benchmark Content Limits

Is Grade Appropriate Has the Appropriate

Cognitive Complexity Has an Appropriate FCAT

Topic Has the Appropriate

Content Focus

Is clearly worded Has Only One Correct

Answer Has Appropriate Multiple

Choice Options Has Appropriate Use of

Graphics Includes an Answer Key Is in the Appropriate

Format

My item…

Congratulations!You are on your way to

developing quality assessment items that are

aligned to the SSS benchmarks and adhere to

FCAT Specifications.

Nancy E. Brito, NBCT, Instructional Specialist, 561-357-7521

Senella Johnson, Reading Test Development Specialist, 561-969-5842

Dr. Carla Kendall, Manager, Online Assessment & Mathematics Test Development Specialist, 561-969-5822

561-434-8850, PX48850

HAVE QUESTIONS?

CONTACT:

division of performance accountability department of assessment division of performance...

Documents