1

You’re the Judge Now…Getting the Scores Right

Calibration Training, continued

Determining an Indicator’s Rating

Each Indicator often has several definitional narratives for each level of proficiency. However, Principals must enter only one (1) score as to each Indicator (1, 2, 3, 4, 5, NO, NA).

2

3

Determining an Indicator’s Rating

“Best fit” approach

3

The Goal of Rater Accuracy

To move from subjectivity to higher levels

of objectivity and insight.

4

How Do You Become More Accurate?

Practice, discipline, and

intentional study of the rubric.

5

Let’s Review!!

• Indicator 1• Indicator 6• Indicator 7

• Indicator 11• Indicator 12

• “I think that’s a Revulsion!”• “I like Chuck E. Cheese too!”• “If you can express your answer in

the numeric way, you can also express it in the geometric way and in words.”

• “I luv puzzles!!”• “In science, you have to be

flexible. Everyone be flexible”6

Calibration Activity One

7

Your First Job…

• Using your rubric, working in PAIRS, sort observation evidence into their appropriate indicator category or categories:– Indicator 1– Indicator 6– Indicator 7– Indicator 11– Indicator 12

8

Answer Key to Sorting the Evidence, Calibration Activity #1

9

EvidenceIndicator 1 B, O, R, TIndicator 6 C, F, H, L, Q (also I)Indicator 7 D, E, I, M, S (also H)Indicator 11 N, P (also A, G, J and S)Indicator 12 A, G, J, K

Your Second Job…

• Using your rubric, working in PAIRS, determine the “best fit score” for each piece of evidence. – Note that the evidence

typically relates to just one component of the indicator. Right now, you should ignore the other components of the indicator/rubric that are not addressed on the evidence sheet.

10

Suggested* Scores

11

Evidence Indicator ScoreA 11, 12 3B 1 1-2C 6 1D 7 2E 7 5F 6 2G 11, 12 4H 6, 7 5 (Ind. 6), 3-4 (Ind. 7)I 6, 7 1-2J 11, 12 1 (Ind. 12); 2 (Ind. 11)

Suggested* Scores

12

Evidence Indicator ScoreK 12 5L 6 4M 7 4N 11 2O 1 1-2P 11 2-3Q 6 3-4R 1 3-4S 7, 11 5T 1 3

Wait…there’s something we haven’t talked about!

13

Make Your Own “Professional Preference” Lists

• Pet Peeves

and

• Preferences

14

Common Pet Peeves/Preferences

Pet Peeves/Negative Bias--messy rooms--sloppy attire--food/drink in the classroom--accents, loud speech--typohs --particular strategies

Preferences/Positive Bias--neat room--attractive appearance--humor--casual demeanor--students in uniforms--particular strategies

15

Give me a 5 if you’re ready for a break!

16

We’re Back…ready or not!!!

17

Three Additional Indicators for Calibration

• Indicator 2: Clearly defined behavior

• Indicator 9: Active learning, questioning, incorporation of interests, and involving all students

• Indicator 13: Monitoring for progress

18

19

Indicator 2

Indicator 2—Teacher clearly defines expected behavior.

• Ensures that students are engaged and clear as to the expectations of the classroom with few reminders given.

• Monitors the behavior of students during whole-

class, small group and seat work activities and during transitions between instructional activities.

• Stops inappropriate behavior promptly and consistently with an appropriate voice level / word choice.

20

Clearly Defining Expected Behavior: What Do you Expect…

• teachers to be doing/saying if they are performing at an “effective” level?

• students to be doing/saying if teachers are performing at an “effective” level?

21

22

Indicator 9

Indicator 9—Engagement of learners: active learning/ involvement/ questioning/ incorporation of interests.

• Engages most students in active learning experiences 80 percent of the class time.

• Uses questioning techniques throughout the lesson, scaffolding to at least the mid-level of Bloom's taxonomy. Provides wait time for some student response and does random checking to ensure the involvement of all learners.

• Engages students by incorporating their general skills and interests into the lesson.

23

Engagement of LearnersWhat Do you Expect…

• teachers to be doing/saying if they are performing at an “effective” level?

• students to be doing/saying if teachers are performing at an “effective” level?

24

25

Indicator 13

Indicator 13—Teacher checks to determine if students are progressing toward stated objectives.

• When appropriate, moves to all areas of the room while students are working on guided practice to promote and reinforce positive student behaviors.

• Uses different types of student response techniques, both individual / group.

• Uses student response techniques to increase active engagement.

• Student’s understanding is evaluated by feedback. • Uses wait time of 3-5 seconds (more for more complex

questions) after voicing the question. Provides opportunity for students to formulate more thoughtful responses and allows time for the student to consider supporting evidence.

26

Checks/Monitors for ProgressWhat Do you Expect…

• teachers to be doing/saying if they are performing at an “effective” level…

• students to be doing/saying if teachers are performing at an “effective” level…

27

The Three-Step Process of Scoring

• Step One: Gather evidence

• Step Two: Sort evidence into indicators, as appropriate

• Step Three: Determine rating

28

29

Calibration Activity Two

30

• During the video– Step One, Part A: Take

notes, documenting the evidence you observed relevant to indicators 2, 9 & 13.

– We will pause briefly at the 5 min. marker to give you add’l time to write your evidence down on notepaper

• After the video– Step One, Part B:

Complete your notes and review them for the most significant evidence relevant to indicators 2, 9 & 13.

– Transfer the key evidence onto sticky notes—one per piece of evidence.

31

Practice Video 1…just indicators 2, 9 & 13Step One

Independent Work

Additional Notes About the Video

• Your “universe of evidence” is the video alone.– “consistently implements” = consistently within

the video– Disregard rubric requirements that require

knowledge outside of video (stricken in the rubric)

• Video format is similar to what will be used on the calibration test.– One screen on the board; one panoramic screen

on the class.

32

33

Everyone Ready?

Calibration Activity #2…Brief Practice Video …just 2, 9 & 13

CCOSA--Tulsa Model Video for Calibration Activity No. 2 (7th grade ELA) -- view first 12 minutes only

34

• During the video– Step One, Part A: Take

notes, documenting the evidence you observed relevant to indicators 2, 9 & 13.

– We will pause briefly at the 5 min. marker to give you add’l time to write your evidence down on notepaper

• After the video– Step One, Part B:

Complete your notes and review them for the most significant evidence relevant to indicators 2, 9 & 13.

– Transfer the key evidence onto sticky notes—one per piece of evidence.

35

Practice Video 1…just indicators 2, 9 & 13Step One

Independent Work

Step Two, Part A:

Using the worksheets as placemats, sort the evidence (on sticky notes) into the appropriate indicator category•If evidence applies to more than one indicator, you may copy the evidence onto multiple sticky notes

Step Two, Part B:

Share worksheets with Table Team

Consolidate the table’s findings evidence onto one placemat per indicator.

36

Practice Video 1…just indicators 2, 9 & 13Step Two

Independent Work Table Work

Pause and Share Evidence with the Room

• What evidence did you have for Indicator 2?

• What evidence did you have for Indicator 9?

37

Pause and Share Evidence with the Room

• What evidence did you have for Indicator 13?

38

Practice Video 1…just indicators 2, 9 & 13Step Three

Step Three

Using the placemats and the rubric, determine the appropriate rating for each indicator and write the “best fit” rating on the placemat in the designated spot.

39

Table Work

Pause and Share Ratings with the Room

• What rating did you have for Indicator 2?

Target: 2

• What rating did you have for Indicator 13?

Target: 2

• What rating did you have for Indicator 9?

Target: 2

40

41

Calibration Activity #3

Practice Test video!

42

Practice Video…eight indicators

• Rating only Indicators 1, 2, 6, 7, 9, 11, 12 & 13• Viewing just the first 20 minutes of an

Observation Engine video (secondary Math).– We will NOT stop at the video while we view it.

• We will:– Independently categorize evidence, and – Independently rate the indicators; then– Share our findings with table teams and report.

43

44

Everyone Ready to Cue Music?

Link to Practice Test Video (Secondary Math)

CCOSA--Tulsa Model Video Practice Test (Secondary Math) -- view first 20 minutes only

45

Practice Test -- Secondary

46

• Ind. 1: 3• Ind. 2: 3• Ind. 6: 3• Ind. 7: 2

• Ind. 9: 2• Ind. 11: 3• Ind. 12: 4• Ind. 13: 2

Target Scores…

47

Are We Ready?

48

Calibration Test Basics

• Required by the state, but it’s TPS-developed• Online platform called Observation Engine.• The testing time will last 90 minutes in all.• Scoring only the 20 minutes of 3 videos– Indicators 1, 2, 6, 7, 9, 11, 12, & 13– Customized by elementary/secondary (your choice)• Either test provides K-12 calibration certification

– After each video, you will have 10 minutes to score before we proceed with the next video.

49

Calibration Test Basics, continued

• Target scores determined by expert panel• Your “grade” is determined by comparing your

scores against the target scores.• Goal is 80% on target• The tests will be graded electronically using

the scantrons on which you enter your scores.• Pass/Fail data will be available in about 2

weeks, if not earlier.

50

Context of Calibration Test

• Your “universe of evidence” is the video alone.–“consistently implements” = consistently within the video–Disregard rubric requirements that require knowledge

outside of video (stricken in the rubric)

• Tips–Use the truncated rubric form for each video, focusing on

each required indicator separately. –Don’t stress! You will do fine!

– Re-takes are available (with new videos)

–Don’t let your professional preferences bias you.

Are we Ready? …. Questions?

52

For Tulsa Model or Calibration Questions…

Jana Burk – burkja@tulsaschools.orgGene S. Kleindienst - kleinge@tulsaschools.org

Katy Ackley – ackleka@tulsaschools.orgJennifer Gripado – gripaje@tulsaschools.org