TILSA Alignment Tool Dissemination Workshop

July 25 and 26, 2005WYNDHAM Hotel

Boston, Massachusetts

Funded by the U.S. Department of Education through a contract to the state of Oklahoma and subcontracts to

CCSSO, WCER, HumRRO, and Tindal.

Alignment

Powerful Tool for Focusing

Instruction, Curricula, and

Assessment

AgendaJuly 25, 2005

8:30 to NoonSpeakers:

Norman L. WebbLauress L. WiseGerald Tindal

Noon Lunch

1:00 to 5:00 PMConcurrent sessions

Session A: Using the WATSession B: Interpreting reports and coordinating an alignment study

AgendaJuly 26, 2005

8:30 to 12:30 PMConcurrent sessions

Session A: Using the WATSession B: Interpreting reports and coordinating

an alignment study12:30 Lunch1:30 to 3:00

PlenaryTechnical issues with the CD alignment

systemGeneral questions and closing

Alignment Issues

Vertical Alignment Grade to grade content linkagesLauress Wise

Alternate Assessment AlignmentOperationalize the processGerald Tindal

Webb Alignment ProcessNorman Webb, Rob Ely, Meredith Alt, &

Brian Vesperman

Workshop Expectations

Set up of an alignment study

Responsibilities of a group leader

Responsibilities of reviewers

Coding procedures

Special features

How to conduct an alignment analysis of standards and assessments with the WAT

AlignmentThe degree to which expectationsand assessments are in agreementand serve in conjunction with one another to guide the system toward students learning what is expected.

 Standards  

 Curriculum Assessment

Degree of Alignment

Standards

Assess-ment

Assessment

Standards

Standards

StandardsAssessment

Assessment Items

Alignment Process

• Identify Standards and Assessments

• Select 6-8 Reviewers (Content Experts)

• Train Reviewers on DOK Levels

• Part I: Code DOK Levels of the Standards/Objectives

• Part II: Code DOK Levels and Corresponding Objectives of Assessment Items

Specific CriteriaContent Focus

A. Categorical Concurrence

B. Depth-of-Knowledge Consistency

C. Range-of-Knowledge Correspondence

D. Balance of Representation and

Source of Challenge

Depth of Knowledge

Level 1 Recall Recall of a fact, information, or procedure.

Level 2 Skill/Concept

Use information or conceptual knowledge, two or more steps, etc.

Level 3 Strategic Thinking

Requires reasoning, developing plan or a sequence of steps, some complexity, more than one possible answer.

Level 4 Extended Thinking

Requires an investigation, time to think and process multiple conditions of the problem.

Items by Objectives ReportLow Medium High

0 5.20202 48

root 

I  

1 7 8 19

38

45

 

1.1. 2 2 2 2 7 7 7 7 7 7 7 8 8 8 10

10

10

19

19

19

19

19

38

38

38

38

38

38

38

45

45  

1.2. 19

19

19

29

32

32

32

32

32

40

43

43

45

45

45

45

 

1.3. 29

36

38

45

 

1.4. 2 5 35

35

35

35

41

 

1.5. 2 2 2 2 7 8 8 10

10

25

25

25

25

29

36

41

45

 

What alignment is good enough?

Alignment Levels Using the Four Criteria

Alignment Level

Categorical Concurrence

Depth of Knowledge

Range of Knowledge

Balance of Representation

Acceptable 6 items per standard

50% 50% 70%

Weak --- 40%-49% 40%-49% 60%-69%

Unacceptable Less than 6 items per standard

Less than 40%

Less than 40%

Less than

60%

Categorical Concurrence State B Grade 8 Mathematics

0

2

4

6

8

10

12

14

16

18

20

Mean Hits

N G PR M PS/R PS COMP

Standards

Depth-of-Knowledge Consistency State B Grade 8 Mathematics

0

10

20

30

40

50

60

70

Mean Percent of Hits

N G PR M PS/R PS COMP

Standards

% Under

% At

% Above

0102030405060708090

100

Mean % of Objectives Hit

N G PR M PS/R GPS COMP

Standards

Range-of-Knowledge CorrespondenceState B Grade 8 Mathematics

Balance IndexState B Grade 8 Mathematics

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Balance Index

N G PR M PS/R P COMP

Standards

Coding Process Tips

• One Primary Objective and up to Two Secondary Objectives (if necessary)

• Source of Challenge (a correct/incorrect response for the wrong reason)

• Notes (any insights to share)

• Consider Full Range of Standards

• Use generic objectives sparingly

Structure of the Automated Alignment ProcessRegistration

Group LeaderReviewers

Standards/Goals/Objectives Entry Process

Training on Depth-of-Knowledge Levels

Phase I Consensus Process on Assigning DOK Levels to Objectives

Phase II Coding of Assessment Tasks

Phase III Analysis of Coding

Phase IV Reporting

Web Sites

http://facstaff.wcer.wisc.edu/normw/   

Alignment Toolhttp: //www.wcer.wisc.edu/WAT/index.aspx

 Survey of the Enacted Curriculum

http://www.SECsurvey.org 

  

EXAMPLE OF STANDARDS AND DEPTH-OF-KNOWLEDGE LEVELS

CONTENT AREA: GEOMETRY

Mathematics Standard Depth-of- Knowledge Level

State D Grade 8

VI. Geometric and Spatial Sense

VI.2 Explore transformations of geometric figures. 3 State B Grade 8

II. Geometry

II.4 Graph on a coordinate plane similar figures, reflections, and translations.

2

State A Grade 6

IV. Geometry and Spatial Sense

IV.D. Investigate and predict the results of transformations of shapes, figures, and models including slides, flips, and turns.

IV.D.1 Identify and describe the results of translations (slides), reflections (flips), rotations (turns), or glide reflections.

2

EXAMPLE OF STANDARDS AND DEPTH-OF-KNOWLEDGE LEVELS

CONTENT AREA: PROBABILITY AND STATISTICS

Mathematics Standard Depth-of-Knowledge

Level State D Grade 8

VII. Data Analysis, Probability & Statistics

VII.3 Formulate, predict, and defend positions taken that are based on data collected.

3

State B Grade 8

VI. Probability and Statistics

VI.1 Collect data involving 2 variables and display on a scatter plot; interpret results.

3

Which of the following numbers, whenrounded to the nearest thousand,becomes 27,000?

(a) 26,099

(b) 26,490

(c) 27,381

(d) 27,550

(e) 27,640

Which of these means about the same as the word gauge?

a. balance b. measure

c. select

d. warn

121 1) 190 13 2) 200 32 3) 290 + 34 4) N

A car odometer registered 41,256.9 miles when ahighway sign warned of a detour 1,200 feet ahead.What will the odometer read when the car reaches thedetour? (5,280 feet = 1 mile)

(a) 42,456.9

(b) 41,279.9

(c) 41,261.3

(d) 41,259.2

(e) 41,257.1

Did you use the calculator on this question?

Yes No

This question refers to pieces N, P, and Q.

In Mr. Bell’s classes, the students voted for their favorite shape for asymbol. Here are the results.

Class 1 Class 2 Class 3

Shape N 9 14 11

Shape P 1 9 17

Shape Q 22 7 2

Using the information in the chart, Mr. Bell must select one of theshapes to be the symbol. Which one should he select and why?

The shape Mr. Bell should select: __________________Explain:

Which of these conclusions is best supported by information from the passage? a. If a candidate meets the personal and educational

qualifications and is in fair physical shape, his or her chances of becoming an agent are very good.

b. Compared with other law enforcement agencies in the country, the F.B.I. has a low success rate for tracking down and apprehending suspected offenders.

c. The job of an agent is not for everyone; it takes someone with special training who is not afraid of danger and doesn’t mind being socially isolated at times.

d. The life of a federal investigator is not as interesting as most people think; agents spend most of their time working at desks.

New Cubes

Your school is planning a casino night to raise funds to construct a wall aquarium in your school. As a mathematics student, you are given the job of developing a dice game for this event.

A regular pair of “number dice” consists of two cubes, each with its faces numbered 1 through 6. Often, dice games are played by rolling the two dice and then finding the sum of the two numbers turned upward.

1. Show that, with a regular pair of number dice, the probability of rolling a sum of 7 is greater than the probability of rolling any other sum.

You decide to call your casino game “New Cubes.” To make it interesting, you decide to constructnew dice that have different numbers on their faces than regular dice. Here is how you will constructthem:

Only the single digits 0 through 9 can be used. Any digit can be used more than once. When the dice are rolled every sum from 4 to 14 must be possible and

no other sums can occur. The two dice do not have to be identical.

2. What numbers would you put on the 6 faces of each of the two dice so that the above conditions are met?

Die 1: _____ _____ _____ _____ _____ _____

Die 2: _____ _____ _____ _____ _____ _____

3. Which sum(s) do you think would turn up most frequently if your New Cubes were rolled 1000 times? Explain why.

Coordination of an Alignment Institute

Identify

• Content areas

• Grade levels

• Number of test forms

• Number of reviewers

• Computer facilities

• Standards and their structure

Coordination of an Alignment Institute

Ask if

• Tests include field test items

• Items have different point values

• Alternate assessments will be included

• English Language Learners will be included

WAT Adoption to State Needs

• Assessment development (front end alignment)

• District and local assessments

• Test to test comparison analysis

• Curriculum to standard analysis

Grades 9–12 Science Objectives and Depth-of-Knowledge (DOK) Levels for Michigan

Alignment Analysis

Standard

Number of Objs. Under

Standard

DOK Levels

of Objs.

# of Objs by

DOK Levels

% of Objs by

DOK Levels

Total 60

1234

104361

177210 1

Comparison of Six Science Assessments on Categorical Concurrence

Standard Categorical Concurrence

  

MEAP ACT PLAN SAT Biology

SAT Chemistry

SAT Physics

I 

Y Y Y Y N N

II 

N N N N N N

III Y N N Y N N

IV Y 

N N N Y Y

V Y N N N N N

Comparison of Six Science Assessments on Balance of Representation

Standard Balance of Representation

  

MEAP ACT PLAN SAT Biology

SAT Chemistry

SAT Physics

I 

Y Y W Y n/a n/a

II 

n/a n/a n/a n/a n/a n/a

III Y n/a n/a W n/a n/a

IV Y 

n/a n/a n/a W N

V W n/a n/a n/a n/a n/a

Cross-Assessment Summary

Three Analytic Methods

• Common Framework

• Expert Consensus

• Common Criteria

Survey of the Enacted Curriculum

 

Time on Topic

Number sense/Properties/Relationships

 Memorize Facts/etc.

 Perform

Procedures

 Demonstrate

Understanding of Mathematical

Ideas

0 1 2 3 Place value 0 1 2 3 0 1 2 3 0 1 2 3

0 1 2 3 Whole numbers 0 1 2 3 0 1 2 3 0 1 2 3

0 1 2 3 Operations 0 1 2 3 0 1 2 3 0 1 2 3

0 1 2 3 Fractions 0 1 2 3 0 1 2 3 0 1 2 3

0 1 2 3 Decimals 0 1 2 3 0 1 2 3 0 1 2 3

0 1 2 3 Percents 0 1 2 3 0 1 2 3 0 1 2 3

Enacted, Intended, and AssessedCurriculum

Enacted—What teachers teach

Intended—What standards require

Assessed—What state tests

Achieve MatrixGrade 3 Mathematics

Data Analysis and ProbabilityObj. # Text of Objective A B Content

CentralityType of Performance Centrality

Source of Challenge

Organize, describe and make predictions from existing data

10.A.1a Organize and display data using pictures, tallies, tables, charts, or bar graphs

81        

10.A.1b Answer questions and make predictions based on given data.

5        

  20        

  41        

  53        

  66        

  69        

Achieve Alignment Criteria

Item-Standard Match •Content Centrality •Performance Centrality •Source of Challenge

Instrument-Standard Match •Level of Challenge •Balance •Range

Alignment Process

• Identify Standards and Assessments

• Select 6-8 Reviewers (Content Experts)

• Train Reviewers on DOK Levels

• Part I: Code DOK Levels of the Standards/Objectives

• Part II: Code DOK Levels and Corresponding Objectives of Assessment Items