Using the Revised Bloom's Taxonomy (RBT) to

Redesign CTE Courses: Home Improvement or

Extreme Makeover?

““A Winter’s Day …”A Winter’s Day …” I met with Mary Jo Nason and others I met with Mary Jo Nason and others

at Pinehurst in December, 2004.at Pinehurst in December, 2004. Question: Will you come and meet Question: Will you come and meet

with the CTE consultants and discuss with the CTE consultants and discuss how the Revised Bloom’s Taxonomy how the Revised Bloom’s Taxonomy (RBT) can replace the original (RBT) can replace the original Bloom’s Taxonomy in our course Bloom’s Taxonomy in our course blueprints?blueprints?

““in a Deep and Dark in a Deep and Dark December”December”

When I accepted the invitation, I was When I accepted the invitation, I was expecting to help with some “home expecting to help with some “home improvement.”improvement.”

““Getting to Know You”Getting to Know You” In May and August, 2005, I met with the In May and August, 2005, I met with the

CTE Curriculum Management team for a CTE Curriculum Management team for a total of three days. total of three days.

Although our discussions focused on how Although our discussions focused on how to proceed in making the transition from to proceed in making the transition from the “old” Bloom’s Taxonomy to the the “old” Bloom’s Taxonomy to the RBT, people made me aware of RBT, people made me aware of problems with published course problems with published course materials. materials.

““Problems, Problems, Problems, Problems, Problems Problems

All Day Long ”All Day Long ” The course blueprints were too long, The course blueprints were too long,

containing too many objectives to containing too many objectives to teach in too short a time.teach in too short a time.

The curriculum guides were not used The curriculum guides were not used by relatively large numbers of by relatively large numbers of teachers. These teachers worked teachers. These teachers worked directly from the course blueprints, directly from the course blueprints, teaching the way they wanted to teach.teaching the way they wanted to teach.

“ “Trouble (Ya Got Trouble)”Trouble (Ya Got Trouble)” Concerns were raised about many Concerns were raised about many

students who did well on the “released” students who did well on the “released” test items but did not do as well on the test items but did not do as well on the “secure” test items. In addition, there “secure” test items. In addition, there were questions of the importance of some were questions of the importance of some of the items on an end-of-course test.of the items on an end-of-course test.

There was some concern about the quality There was some concern about the quality and usefulness of the published materials and usefulness of the published materials for a few courses. One set of materials for a few courses. One set of materials had to be pulled back after being had to be pulled back after being published. published.

““I Go to Extremes”I Go to Extremes” We brought in the course design We brought in the course design

teams for the first time in late October teams for the first time in late October and November, 2005.and November, 2005.

After working with the teams for After working with the teams for several months, I came to the several months, I came to the realization that this was not a Home realization that this was not a Home Improvement effort. We were Improvement effort. We were engaged in an Extreme Makeover.engaged in an Extreme Makeover.

And … extreme makeovers take time.And … extreme makeovers take time.

Solving the Course Blueprint Problem

Purpose of the Course Purpose of the Course BlueprintBlueprint

The purpose of the course blueprint is The purpose of the course blueprint is to determine, of all the things students to determine, of all the things students COULD learn in a course, what they COULD learn in a course, what they SHOULD learn. In the blueprint, what SHOULD learn. In the blueprint, what students SHOULD learn is specified as a students SHOULD learn is specified as a set of objectives. set of objectives.

Objectives help us differentiate what Objectives help us differentiate what students NEED to know and do from students NEED to know and do from what is NICE to know and do.what is NICE to know and do.

Purpose (continued)Purpose (continued)

As defined by the authors of the As defined by the authors of the original Bloom’s taxonomy, original Bloom’s taxonomy, objectives are “explicit formulations objectives are “explicit formulations of the ways in which students are of the ways in which students are expected to be changed by the expected to be changed by the educative process” (p. 26).educative process” (p. 26).

The Structure of CoursesThe Structure of Courses Objectives are the primary building Objectives are the primary building

blocks of a course. Rather than blocks of a course. Rather than including a “grocery list” of objectives, including a “grocery list” of objectives, however, competencies were used to however, competencies were used to organize objectives; units, in turn, were organize objectives; units, in turn, were used to organize the competencies. used to organize the competencies.

Each course must have AT LEAST two Each course must have AT LEAST two units, each unit must contain AT LEAST units, each unit must contain AT LEAST two competencies, and each two competencies, and each competency must contain AT LEAST two competency must contain AT LEAST two objectives.objectives.

How Many Competencies and How Many Competencies and Objectives?Objectives?

For each course, the objectives, competencies, For each course, the objectives, competencies, and units are determined consensually by a and units are determined consensually by a Course Design Team. Course Design Team.

Each objective included in a course blueprint Each objective included in a course blueprint has an associated “course weight.” As has an associated “course weight.” As described in the blueprint, the weight “shows described in the blueprint, the weight “shows the relative importance” of the objective and is the relative importance” of the objective and is used “to help determine the percentage of total used “to help determine the percentage of total class time that is spent on each objective.”class time that is spent on each objective.”

Rethinking the Blueprint Rethinking the Blueprint ProblemProblem

Because the smallest possible course weight Because the smallest possible course weight is 1%, a course cannot contain more than is 1%, a course cannot contain more than 100 objectives. 100 objectives.

In general, each blueprint page contains In general, each blueprint page contains from 10 to 15 objectives. You can estimate from 10 to 15 objectives. You can estimate the number of course objectives from the the number of course objectives from the number of pages included in the blueprint number of pages included in the blueprint (excluding the cover and general information (excluding the cover and general information pages).pages).

Beginning to Grasp the Beginning to Grasp the Blueprint ProblemBlueprint Problem

Assuming 10 to 15 objectives per page, Assuming 10 to 15 objectives per page, a 5-page blueprint would contain a 5-page blueprint would contain somewhere between 50 and 75 somewhere between 50 and 75 objectives. objectives.

If there are 50 objectives, the average If there are 50 objectives, the average course weight per objective is 2 percent. course weight per objective is 2 percent. If there are 75 objectives, the average If there are 75 objectives, the average course weight is 1.5 percent.course weight is 1.5 percent.

Understanding the Blueprint Understanding the Blueprint ProblemProblem

In the past, then, course blueprints In the past, then, course blueprints tended to contain relatively large tended to contain relatively large numbers of relatively “small” objectives. numbers of relatively “small” objectives. Using the terminology of the RBT, many Using the terminology of the RBT, many of these objectives were A1 objectives, of these objectives were A1 objectives, that is, objectives that required students that is, objectives that required students to Remember Factual Knowledge.to Remember Factual Knowledge.

Solving the Blueprint Solving the Blueprint ProblemProblem

To shorten the blueprints, then, we needed To shorten the blueprints, then, we needed to have fewer “larger” objectives. to have fewer “larger” objectives.

Because of the imposed constraints on the Because of the imposed constraints on the numbers of units, competencies, and numbers of units, competencies, and objectives mentioned earlier, the fewest objectives mentioned earlier, the fewest objectives that could be included in a objectives that could be included in a course would be 8 (2 units x 2 course would be 8 (2 units x 2 competencies per unit x 2 objectives per competencies per unit x 2 objectives per competency).competency).

Specificity & Generality of Specificity & Generality of ObjectivesObjectives

The authors of the RBT identified 3 levels The authors of the RBT identified 3 levels of objectives, ranging from general to of objectives, ranging from general to specific: global, educational, & specific: global, educational, & instructional.instructional.

They went on to say that educational They went on to say that educational objectives are “moderate in scope and objectives are “moderate in scope and provide the basis for planning units provide the basis for planning units containing objectives that require weeks containing objectives that require weeks or months to learn” (p. 17).or months to learn” (p. 17).

The Scope of the RBTThe Scope of the RBT

““Our framework is designed to Our framework is designed to facilitate working with EDUCATIONAL facilitate working with EDUCATIONAL objectives.”objectives.”

The RBT and Course The RBT and Course BlueprintsBlueprints

Because educational objectives require Because educational objectives require “weeks or months to learn,” we stipulated “weeks or months to learn,” we stipulated that, with few exceptions, the minimum that, with few exceptions, the minimum course weight for any objective would be 3%. course weight for any objective would be 3%.

In order to move to “3% or more” objectives, In order to move to “3% or more” objectives, Course Design Teams found themselves Course Design Teams found themselves moving away from the A1 cell of the RBT: to moving away from the A1 cell of the RBT: to the right on the Cognitive Process Dimension the right on the Cognitive Process Dimension and down the Knowledge Dimension.and down the Knowledge Dimension.

So What’s Different?So What’s Different?

The RBT course blueprints are shorter The RBT course blueprints are shorter than 3 pages, with many being 2 pages.than 3 pages, with many being 2 pages.

The structure of every course (units, The structure of every course (units, competencies, and objectives) is the competencies, and objectives) is the same for every course.same for every course.

So What’s Different?So What’s Different?

All verbs included in competencies & All verbs included in competencies & objectives are taken from the 6 cognitive objectives are taken from the 6 cognitive process categories and 19 specific process categories and 19 specific processes included in the RBT (see inside processes included in the RBT (see inside back cover of the text).back cover of the text).

The number of objectives that emphasize The number of objectives that emphasize memorization is reduced in every course.memorization is reduced in every course.

Exhibit AExhibit A

Portions of Course BlueprintsPortions of Course Blueprints 6208 Exploring Business 6208 Exploring Business TechnologiesTechnologies 7111 Early Childhood Education I7111 Early Childhood Education I

Solving the Curriculum Guide Problem

Purpose of the Curriculum Purpose of the Curriculum GuideGuide

The purpose of a curriculum guide is The purpose of a curriculum guide is to help teachers connect the to help teachers connect the objectives included in the course objectives included in the course blueprints with the students enrolled blueprints with the students enrolled in their courses.in their courses.

Unpacking the PurposeUnpacking the Purpose

Each curriculum guide is written BY Each curriculum guide is written BY teachers (along with, in some cases, teachers (along with, in some cases, teacher educators) FOR teachers.teacher educators) FOR teachers.

It is a GUIDE to help teachers, not a It is a GUIDE to help teachers, not a PRESCRIPTION to be followed blindly.PRESCRIPTION to be followed blindly.

Each curriculum guide MUST help Each curriculum guide MUST help teachers connect students likely to enroll teachers connect students likely to enroll in the course with the course objectives.in the course with the course objectives.

Connecting Students to Connecting Students to ObjectivesObjectives

Connecting students to course Connecting students to course objectives requires that objectives requires that course content be “unpacked” so course content be “unpacked” so

that it makes sense to students, and that it makes sense to students, and teachers must teach in ways that teachers must teach in ways that

actively engage students in the actively engage students in the process of learning.process of learning.

Content Outline Content Outline v.v. Unpacked Unpacked Content Content

Content OutlineContent Outline

Words OnlyWords Only Organized Organized

AlphabeticallyAlphabetically

Unpacked ContentUnpacked Content

Words and MeaningsWords and Meanings Organized Organized

StructurallyStructurally Written in “Language Written in “Language

of Students”of Students”

The “Language of Students”The “Language of Students”

Language of ExpertsLanguage of Experts Demand is the want Demand is the want

or desire to possess or desire to possess a good or service a good or service with the necessary with the necessary goods, services, or goods, services, or financial instruments financial instruments necessary to make a necessary to make a legal transaction for legal transaction for those goods or those goods or services. services.

Language of Language of StudentsStudents

Demand is the Demand is the desire to own desire to own something and the something and the ability to pay for it.ability to pay for it.

Power Point PresentationsPower Point Presentations

Most courses include Power Point Most courses include Power Point Presentations to provide an overview Presentations to provide an overview of the content contained in the of the content contained in the course.course.

These Power Point Presentations are These Power Point Presentations are intended for both teachers and intended for both teachers and students, particularly those teachers students, particularly those teachers who are lateral entry teachers and who are lateral entry teachers and those who may be teaching a those who may be teaching a particular course for the first time.particular course for the first time.

The Engaging Nature of The Engaging Nature of QuestionsQuestions

Once the content has been unpacked, Once the content has been unpacked, we turn our attention to the need to we turn our attention to the need to engage students in the process of engage students in the process of learning.learning.

From years of research we have learned From years of research we have learned that asking is more engaging than that asking is more engaging than telling. Consequently, “essential telling. Consequently, “essential questions” or “learning questions” are questions” or “learning questions” are included in course guides. included in course guides.

Exhibit BExhibit B

Examples of Unpacked Content and Examples of Unpacked Content and Essential/Learning QuestionsEssential/Learning Questions

6411 Computer Applications I6411 Computer Applications I 7075 Foods II – Food Technology7075 Foods II – Food Technology

Activities and Student Activities and Student EngagementEngagement

Stated simply, active students are more Stated simply, active students are more engaged in learning than passive engaged in learning than passive students. students.

For each objective, therefore, each For each objective, therefore, each curriculum guide includes a sequence curriculum guide includes a sequence of activities for teachers to follow.of activities for teachers to follow.

In addition, the relevance of each In addition, the relevance of each activity to the objective is specified, as activity to the objective is specified, as are the resources needed to support are the resources needed to support the activity.the activity.

Exhibit CExhibit C

Examples of Portions of Activity Examples of Portions of Activity SequencesSequences

6208 Exploring Business 6208 Exploring Business TechnologiesTechnologies 6626 Strategic Marketing6626 Strategic Marketing 7111 Early Childhood Education I7111 Early Childhood Education I

Graphic Organizers, Unpacked Graphic Organizers, Unpacked Content, and Student Content, and Student

EngagementEngagement Graphic organizers can be used not Graphic organizers can be used not

only to engage students in learning only to engage students in learning as they read material or listen to a as they read material or listen to a lecture, but also to help students lecture, but also to help students learn to unpack content on their own.learn to unpack content on their own.

Consequently, many courses include Consequently, many courses include graphic organizers.graphic organizers.

Exhibit DExhibit D

Examples of Graphic OrganizersExamples of Graphic Organizers

6158 Exploring Career Decisions6158 Exploring Career Decisions 6626 Strategic Marketing6626 Strategic Marketing 7075 Foods II – Food Technology7075 Foods II – Food Technology 7111 Early Childhood Education I7111 Early Childhood Education I

Making, Doing, & Student Making, Doing, & Student EngagementEngagement

Activities that require students to Activities that require students to make or do something are engaging make or do something are engaging for students.for students.

Consequently, both large- and small-Consequently, both large- and small-scale projects are included in scale projects are included in curriculum guides when objectives curriculum guides when objectives focus on Apply and Create.focus on Apply and Create.

Exhibit EExhibit E

Examples of ProjectsExamples of Projects

6158 Exploring Career Decisions6158 Exploring Career Decisions 6411 Computer Applications I6411 Computer Applications I 7075 Foods II – Food Technology7075 Foods II – Food Technology

Scoring Students’ ProjectsScoring Students’ Projects

Whenever a short- or long-term Whenever a short- or long-term project is assigned to students, the project is assigned to students, the curriculum guide includes a rubric curriculum guide includes a rubric that is to be used to score student that is to be used to score student performance on the project.performance on the project.

Exhibit FExhibit F

Examples of RubricsExamples of Rubrics

6208 Exploring Business 6208 Exploring Business TechnologiesTechnologies 7075 Foods II – Food Technology7075 Foods II – Food Technology

Unit OverviewUnit Overview

As mentioned earlier, courses are As mentioned earlier, courses are organized around units of instruction. In organized around units of instruction. In many ways, units are the “glue” that many ways, units are the “glue” that holds the competencies and objectives holds the competencies and objectives together.together.

Each course contains a Unit Overview Each course contains a Unit Overview that provides teachers with useful that provides teachers with useful information about the unit BEFORE they information about the unit BEFORE they begin teaching it. begin teaching it.

Exhibit GExhibit G

Examples of Unit OverviewsExamples of Unit Overviews

6411 Computer Applications I6411 Computer Applications I 7075 Foods II – Food Technology7075 Foods II – Food Technology

Solving the Curriculum Guide Solving the Curriculum Guide ProblemProblem

By providing a unit overview, unpacking By providing a unit overview, unpacking the content for each objective, using the content for each objective, using Power Point Presentations to provide Power Point Presentations to provide overviews of the content, and including overviews of the content, and including questions, activities, graphic organizers, questions, activities, graphic organizers, and projects to engage students, I and projects to engage students, I believe that we have made the believe that we have made the curriculum guides more “user friendly.” curriculum guides more “user friendly.”

Who Benefits from Curriculum Who Benefits from Curriculum Guides?Guides?

The curriculum guides should be The curriculum guides should be particularly helpful to lateral entry particularly helpful to lateral entry teachers and teachers teaching a teachers and teachers teaching a particular course the first time. particular course the first time.

For more experienced teachers, the For more experienced teachers, the curriculum guides should offer a set of curriculum guides should offer a set of ideas and strategies that represent the ideas and strategies that represent the best thinking of other experienced best thinking of other experienced teachers, ideas that in their busy worlds teachers, ideas that in their busy worlds they may not have had time to think of on they may not have had time to think of on their own.their own.

Solving the Test Item Problem

Importance and AlignmentImportance and Alignment

The test item problem can be The test item problem can be rephrased as two questions.rephrased as two questions. Do the items test learning that is truly Do the items test learning that is truly

important? important? Why do some students do well on the Why do some students do well on the

items in the classroom bank, but not on items in the classroom bank, but not on the items in the secure (VoCATS) bank?the items in the secure (VoCATS) bank?

When Does a Fact Become When Does a Fact Become Trivia?Trivia?

Not all facts are created equal. Facts Not all facts are created equal. Facts pertaining to safety, for example, are pertaining to safety, for example, are critical. So are facts that you use over critical. So are facts that you use over and over again (e.g., multiplication and over again (e.g., multiplication facts, names of frequently used tools). facts, names of frequently used tools). Facts pertaining to names of people, Facts pertaining to names of people, dates, figures, and locations, on the dates, figures, and locations, on the other hand, border on trivia.other hand, border on trivia.

Some Trivia ContendersSome Trivia Contenders

Who was the father of the Who was the father of the Kindergarten movement?Kindergarten movement?

Where is the backspace key located Where is the backspace key located on a standard computer keyboard?on a standard computer keyboard?

How many big screen TVs were How many big screen TVs were owned by Americans in 2000?owned by Americans in 2000?

What do the initials HACCP stand for? What do the initials HACCP stand for?

Objectives and Test ItemsObjectives and Test Items Because items are derived from Because items are derived from

objectives, the inclusion of large objectives, the inclusion of large numbers of items of questionable numbers of items of questionable importance derives from the inclusion of importance derives from the inclusion of large numbers of “low level” objectives.large numbers of “low level” objectives.

The best way to solve this aspect of the The best way to solve this aspect of the test item problem is to increase the test item problem is to increase the cognitive level of the course objectives. cognitive level of the course objectives. In comparing the RBT courses with prior In comparing the RBT courses with prior courses, you should see a shift from courses, you should see a shift from memorization (A1) to understanding memorization (A1) to understanding (B2).(B2).

Released and Secure ItemsReleased and Secure Items

Just as not all facts are created equal, Just as not all facts are created equal, so too, not all test items are created so too, not all test items are created equal. Some items are more difficult; equal. Some items are more difficult; others are easier. If released items others are easier. If released items are easier than secure items, on are easier than secure items, on average, then student performance average, then student performance on the released items will quite likely on the released items will quite likely be higher than on the secure items.be higher than on the secure items.

The Use of Item FormsThe Use of Item Forms

Item forms were used to prepare the Item forms were used to prepare the test items for all courses. The use of test items for all courses. The use of item forms makes it more likely that item forms makes it more likely that the items will be valid (that is, truly the items will be valid (that is, truly assess students’ learning of the assess students’ learning of the objective as written) and more objective as written) and more similar in their difficulty.similar in their difficulty.

Two Items of the Same Two Items of the Same FormForm

Lowes is targeting consumers living in the Lowes is targeting consumers living in the Southwest for a new product offering. It is Southwest for a new product offering. It is segmenting the consumer market based segmenting the consumer market based on: a) benefits, b) geographics, c) psycho-on: a) benefits, b) geographics, c) psycho-graphics, d) usage rate.graphics, d) usage rate.

Western Wake Hospital is targeting a 29-Western Wake Hospital is targeting a 29-year old, female consumer that earns year old, female consumer that earns $60,000 per year. It is segmenting the $60,000 per year. It is segmenting the consumer market based on: a) consumer market based on: a) demographics, b) geographics, c) demographics, b) geographics, c) psychographics, d) usage rate. psychographics, d) usage rate.

The RBT and Item FormsThe RBT and Item Forms Different item forms are needed to Different item forms are needed to

properly assess objectives falling into properly assess objectives falling into different cells of the RBT.different cells of the RBT.

For example, one very good way of For example, one very good way of assessing Understand Conceptual assessing Understand Conceptual Knowledge is to give students one or Knowledge is to give students one or more examples and have them more examples and have them determine in which category or determine in which category or classification the example (or classification the example (or examples) belong (see previous slide). examples) belong (see previous slide).

Assessing Conceptual Assessing Conceptual UnderstandingUnderstanding

Another good way to assess Another good way to assess Understand Conceptual Knowledge is Understand Conceptual Knowledge is to give students information to give students information presented in one form and ask them presented in one form and ask them to change it to another form. In the to change it to another form. In the language of the RBT, we are asking language of the RBT, we are asking students to interpret the information.students to interpret the information.

““Interpret” requires students to Interpret” requires students to change from one from one form (a change from one from one form (a graph)graph)

To Another Form (the Meaning To Another Form (the Meaning of a Single Point on the Graph)of a Single Point on the Graph)

Which point in the graph in the Which point in the graph in the figure above would illustrate the figure above would illustrate the equilibrium price for Demand Curve equilibrium price for Demand Curve 1 (D1)?1 (D1)?

A.A. PP11

B.B. PP22

C.C. QQ

D.D. SS

Assessing “Apply” Objectives Assessing “Apply” Objectives Using Multiple-Choice ItemsUsing Multiple-Choice Items

Since apply objectives involve “doing Since apply objectives involve “doing something” and typically involves something” and typically involves procedural knowledge, the emphasis procedural knowledge, the emphasis in assessing apply procedural in assessing apply procedural knowledge (C3) objectives is on knowledge (C3) objectives is on doing something or knowing what to doing something or knowing what to do or how to do it. do or how to do it.

Examples of C3 ItemsExamples of C3 Items You are developing a new food science You are developing a new food science

experiment. What should your first step be? experiment. What should your first step be? Collect the dataCollect the data Identify the problemIdentify the problem Identify variablesIdentify variables Write the procedureWrite the procedure

After putting together a product development After putting together a product development team, what is the appropriate first action to team, what is the appropriate first action to take? take? Do a cost breakdown of the productDo a cost breakdown of the product Hold a brainstorming meeting to develop ideasHold a brainstorming meeting to develop ideas Start doing the research for the productStart doing the research for the product Start trial formulations of the new productStart trial formulations of the new product

Assessing “Evaluate” Assessing “Evaluate” ObjectivesObjectives

In the context of the RBT, evaluate In the context of the RBT, evaluate means to “make judgments based on means to “make judgments based on criteria and standards.”criteria and standards.”

One of the major judgments that One of the major judgments that early childhood educators need to early childhood educators need to make is whether programs and make is whether programs and practices are “developmentally practices are “developmentally appropriate.”appropriate.”

Evaluating ECE Programs & Evaluating ECE Programs & PracticesPractices

An after-school program has centers with lots An after-school program has centers with lots of interesting activities all arranged on tables of interesting activities all arranged on tables around the room. Each child is required to around the room. Each child is required to spend a certain amount of time in each center spend a certain amount of time in each center each week. Which part of this program is NOT each week. Which part of this program is NOT developmental appropriate?developmental appropriate? Centers are not needed in after-school programs.Centers are not needed in after-school programs. Children would benefit from having more options.Children would benefit from having more options. Discipline would be better if children were seated at separate Discipline would be better if children were seated at separate

desks.desks. With so many activities, children may lose focus and cause With so many activities, children may lose focus and cause

behavior problems.behavior problems.

What Differences Will You See What Differences Will You See in the New Test Items?in the New Test Items?

A reduction in the number of memory or A1 items.A reduction in the number of memory or A1 items. Information at the beginning of items that provide Information at the beginning of items that provide

a context for the question that follows. The a context for the question that follows. The information may be a graph or a short vignette information may be a graph or a short vignette (generally from one to three sentences). (generally from one to three sentences).

The context makes it LESS likely that students will The context makes it LESS likely that students will rely on memory for the answer and more likely rely on memory for the answer and more likely they will have to demonstrate understanding, they will have to demonstrate understanding, application, analysis, or evaluation in order to application, analysis, or evaluation in order to respond correctly to the items. respond correctly to the items.

Field TestingField Testing

Although the use of item forms helps to Although the use of item forms helps to reduce the differences in item difficulty, reduce the differences in item difficulty, there is no guarantee that this will there is no guarantee that this will happen.happen.

The only way to make sure the released The only way to make sure the released and secure items are of equal difficulty, and secure items are of equal difficulty, on average, is to field test ALL items and on average, is to field test ALL items and then divide the items into “secure” and then divide the items into “secure” and “released” based on the results. “released” based on the results.

Field Test Trade-OffsField Test Trade-Offs

Field testing of all items involves a Field testing of all items involves a trade-off. The first year a course is trade-off. The first year a course is offered – the “field test year” – there offered – the “field test year” – there will be no released items. After the will be no released items. After the first year, however, you will have first year, however, you will have increased confidence in the increased confidence in the comparability of the released and comparability of the released and secure items in terms of their validity secure items in terms of their validity and difficulty. and difficulty.

Solving the Quality Problem

The Introduction of Peer The Introduction of Peer ReviewReview

We all make mistakes. We all get too We all make mistakes. We all get too “close” to our work. We all see things “close” to our work. We all see things that aren’t there and fail to see things that aren’t there and fail to see things that are. To truly improve the quality that are. To truly improve the quality of any product, we need of any product, we need At least one other pair of eyes (usually At least one other pair of eyes (usually

several), several), An openness to constructive criticism, andAn openness to constructive criticism, and A willingness to make the changes needed A willingness to make the changes needed

to make a good product even better. to make a good product even better.

To this end …To this end …

CTE within DPI has established a peer CTE within DPI has established a peer review process to ensure the products review process to ensure the products they publish are of the highest quality.they publish are of the highest quality.

Although this process is time-consuming Although this process is time-consuming and may initially cause defensiveness and may initially cause defensiveness and “hurt feelings,” it is critical for and “hurt feelings,” it is critical for curriculum building success.curriculum building success.

I applaud the CTE consultants and I applaud the CTE consultants and administration for taking this bold step. administration for taking this bold step.

““I Can See Clearly Now”I Can See Clearly Now”

My hope is that:My hope is that: You now understand why the revision You now understand why the revision

process has taken so long to complete.process has taken so long to complete.

““I Can See Clearly Now”I Can See Clearly Now”

My beliefs are that:My beliefs are that: The revised courses are of higher quality The revised courses are of higher quality

and are more teacher-friendly than the and are more teacher-friendly than the courses they are replacing.courses they are replacing.

As a consequence, the revised courses As a consequence, the revised courses will result in higher levels of teaching will result in higher levels of teaching effectiveness and, ultimately, higher effectiveness and, ultimately, higher levels of learning for larger numbers of levels of learning for larger numbers of students.students.

SongbookSongbook ““I am a Rock,” Simon and GarfunkelI am a Rock,” Simon and Garfunkel ““Getting to Know You,” Gertrude Getting to Know You,” Gertrude

Lawrence (or Marni Nixon)Lawrence (or Marni Nixon) ““Problems,” The Everly BrothersProblems,” The Everly Brothers ““Trouble (Ya Got Trouble),” Robert Trouble (Ya Got Trouble),” Robert

PrestonPreston ““I Go to Extremes,” Billy JoelI Go to Extremes,” Billy Joel ““I Can See Clearly Now,” Johnny I Can See Clearly Now,” Johnny

Nash (or Bob Marley)Nash (or Bob Marley)