c r e s s t / u c l a qualitative indicators: classroom observations and classroom practice ann m....

C R E S S T / U C L A

Qualitative Indicators: Classroom Observations and

Classroom Practice

Ann M. Mastergeorge

UCLA Graduate School of Education & Information StudiesCenter for the Study of Evaluation

National Center for Research on Evaluation, Standards, and Student Testing

Education Writers Association Conference “How To Tell If Schools Are Really Improving”

December 8, 2000


Classroom Observation “ What Things to Look For?”

An OverviewAn OverviewA Researcher Perspective: Classroom

Culture, Classroom Practice, Classroom Domains

Shaping Classroom Observations: Field notes and Protocols

Meaning in the Classroom: What do the words and ratings mean?


Characteristics of Qualitative Observations

Natural Setting As Direct Source of Data

Researcher = Key Instrument

Descriptive

Concerned with Process

Analyze Inductively

Meaning in Participant Perspectives


Classroom Observations: To See or Not to See

What is important to see in and out of the classroom?

How do I structure my fieldnotes and interviews?

Where is the story and how do I let it unfold?


Classroom Observations

Creative Tensions: Issues and Paradoxes

Understanding the classroom as a culture versus the classroom as a classroom

Entering the world as a ”novice" versus the world as an “expert”

Knowing the story versus finding the story and letting the story unfold


Classroom Observations (con’t.)

Notes versus fieldnotes

The whole picture versus a piece of the picture

The research community versus the school community

Quality of observations versus quantity of observations


Classroom Observations (con’t)

Creative Chaos: The Collision of Observers and Participants

The World As We See It: How to Share a Vision

Creating Understandings of the Observations: Perspectives, Insights, Barriers, Limitations

The Right Answer: Where is it?


A Day in the Life of a School Issues to Consider

The (sub)cultures

Perspectives

Chaos

Negotiation and Choices

Making the Familiar Strange

Making the Strange Familiar

Finding and Unfolding the story


Issues and Dilemmas in Fieldwork

At what level do I describe reality (e.g. how specific, how general should I be)?

How inferential should my field observations be?

How do I differentiate between what I see and hear and what I infer from my observations?


Issues and Dilemmas in Fieldwork (con’t)

Should I count "things" that I see?

When have I observed enough?

Should I take my participants points of view or my own when I do fieldwork?

What's important about what I am observing?

Have I seen everything, or is there more to see?



Should I take notes while I am observing? If so, should they be narratives, shorthand accounts, outlines, or....?

What should I watch--people, interactions, settings, all of the above?

Should I concentrate on watching a few things, or should I sample from many? If the latter, how do I sample?



Should I quote verbatim what people say or just the gist of their words?

What should be my style of observation/recording? Should I be a holistic ethnographer, a radical empiricist, both, neither?


Characteristics of Field Notes

Rough (jotted, mental notes) versus Finished Notes

Use of Mnemonics

Noting what is missed

Level of Detail


Characteristics of Field Notes (con’t.)

Observer Comments

Summarizing

Posing New Questions

Timely Write-up


Field Note Focus

Activities Individuals One Individual’s Life Roles

• Dyads• Times of Day• Periods in Routine• Types of Acts• “Windows” on Behavior

Observation Domains: DescriptiveDescriptive Focused Focused SelectiveSelective

Observation Types:


EXAMPLE 1:The students are observing Pillbugs first and then will look at Bess Beetles, first

in the cup, then dumping it into their hands to look at it. Each student has a piece ofpaper with space for a picture and thick lines on it, and two sentence starters separatedinto two columns: "A Pillbug is…" and "A Bess Beetle is..."

After about 10 minutes of working with this activity, the teacher claps for theirattention and tells them that they should be at the point where they are drawing anddescribing their bugs. One student starts writing immediately, and the teacher callsattention to her, commending her out loud on using the adjective "small" to describe herbug. The little girl smiles as she is complimented.

Field Note Example 1Field Note Example 1


After about 15 minutes of the kids working with Pillbugs, she hands each group a cupwith a Beetle in it. The kids are going nuts. They are so excited to touch these beetles andare using magnifying glasses to see them close-up.

Meanwhile the teacher continues to rove around. She claps for their attention andtells the kids, "I want to see all beetles back in their cups and I want you to start drawingand writing." The class gets quieter as they put the beetles away and start to write anddraw. Again, she roves around to check out spelling and printing, paying as muchattention to these issues as to content issues…

The kids are so engaged in this activity. They are squealing with excitement anddisgust as they hold the bugs. All of the students are actively involved in this activity.With the beetle, the kids are so excited, yelling and dancing around with excitementabout touching them and feeling them. They maintain excitement and engagement theentire time. The kids are excited and discuss with each other characteristics of the bugsas they write.

Field Note Example 1 Field Note Example 1 (con’t.)(con’t.)


EXAMPLE 2:She tells them she has a book called "Postcards from Pluto" that will help give

them some ideas about what to write on their postcards. She begins the read aloud and itis about a tour of the solar system. In the story, the students first fly by the sun, and thereis a postcard in the story from a student writing home to his parents about the sun, tellingthem it's a star, what it's made of, and what sun spots are. One boy yells out, "Man, they'dbe dead if they were that close." The teacher ignores these comments and a few others.

The next postcard is from a student visiting Mercury. The students make morecomments. Rather than discussing the comments or creating an instructional conversationthe teacher tells them to hold their questions until they are all done. She continues readingto them and tells them to be quiet so they can finish the story so they have enough time towrite their postcards.

Field Note Example 2Field Note Example 2


On the page from Earth, a student asks what the robot is saying, the teacher hasn'tread it yet. She tells him the robot is saying, "I wish those people on Earth would takebetter care of me." The boy says, "yeah because everyone pollutes." The teachercontinues reading. This is another missed opportunity to have a short educationalconversation.

At Mars a girl raises her hand and the teacher tells her to hold her question untilshe's finished reading. Between each planet there is a little discussion among the students,for instance about the moons in orbit around a certain planet or whichever a studentthinks is his favorite planet. The teacher ignores all of this and continues on with thereading. This goes on until she has read all the postcards from each of the nine planets.

The teacher cold have done a better job including the students in the introductorylesson, she had several opportunities to establish an instructional dialogue… She wasmore concerned with having enough time than maximizing student learning.

Field Note Example 2 Field Note Example 2 (con’t.)(con’t.)


Opportunities for engagement in discussions

Monitoring students’ work

Critical thinking skills

Quality feedback

Dialogue between students and teachers

Domains of Engagement


Domains of Engagement (con’t.)

Excitement

Use of prompts

Integrated approach

Engaging implementation of lesson/activity

Creative process and thinking


CLASSROOM OBSERVATIONSCClassroom PPractice OObservation PProtocol

Domains of Classroom Practice:Domains of Classroom Practice: Challenge of the Lesson Activities Implementation of Lesson Activity Proportion of Students On-Task Opportunity to Participate in Conversation Student Participation in Conversation Quality of Instructional Feedback


R a t i n g

4 . 03 . 02 . 01 . 0

C h a l l e n g e o f L e s s o n A c t i v i t y

Frequency

6 0

5 0

4 0

3 0

2 0

1 0

0

S t d . D e v = . 7 6

M e a n = 2 . 3

N = 9 7 . 0 0

7

2 8

5 2

1 0


R a t i n g

4 . 03 . 02 . 01 . 0

I m p l e m e n t a t i o n o f L e a r n i n g A c t i v i t y

Frequency

4 0

3 0

2 0

1 0

0

S t d . D e v = 1 . 0 4

M e a n = 2 . 7

N = 9 7 . 0 0

2 6

3 5

2 0

1 6


R a t i n g

4 . 03 . 02 . 01 . 0

P r o p o r t i o n o f S t u d e n t s O n - T a s k

Frequency

5 0

4 0

3 0

2 0

1 0

0

S t d . D e v = . 9 1

M e a n = 3 . 2

N = 9 7 . 0 0

4 3

3 6

1 1

7


R a t i n g

4 . 03 . 02 . 01 . 0

O p p o r t u n i t y t o P a r t i c i p a t e i n C o n v e r s a t i o n

Frequency

5 0

4 0

3 0

2 0

1 0

0

S t d . D e v = . 9 7

M e a n = 2 . 0

N = 9 7 . 0 0

7

2 3

2 7

4 0


R a t i n g

4 . 03 . 02 . 01 . 0

S t u d e n t P a r t i c i p a t i o n i n C o n v e r s a t i o n

Frequency

4 0

3 0

2 0

1 0

0

S t d . D e v = 1 . 1 4

M e a n = 2 . 2

N = 9 7 . 0 0

1 7

2 2

2 0

3 8


R a t i n g

4 . 03 . 02 . 01 . 0

Q u a l i t y o f I n s t r u c t i o n a l F e e d b a c k

Frequency

4 0

3 0

2 0

1 0

0

S t d . D e v = 1 . 0 9

M e a n = 2 . 3

N = 9 7 . 0 0

1 8

2 1

2 92 9


Site 1-12 Domain Averages

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

Challenge ofLesson Activity

Implementation ofLearning Activity

Proportion ofStudents On-Task

Opportunity toParticipate inConversation

StudentParticipation in

Discussion

Quality ofInstructional

FeedbackDomain

Rating

Site 1Site 2Site 3Site 4Site 5Site 6Site 7Site 8Site 9Site 10Site 11Site 12


TEACHER INTERVIEWSEmerging Themes

Learning that promotes student inquiry and provides students with meaningful experiences

Learners that are active, independent, and empowered

An expectation that test scores will improve and a concern that test scores have to improve


Student Technology AssignmentsTeacher Ratings of Technology Integration Assignments

Teacher Assignments and Student WorkTeacher Assigned Levels of Technology

Integration

Assignment Low Medium High

"PowerPoint" 2 0 0

"History of L.A." 0 2 2

"Projectile Lab" 0 2 0

"Drum Machine" 0 0 2

"Similar Triangle" 5 3 0

"Oceanography" 1 0 0

"A Fiesta" 5 0 0

"NASA" 0 0 3

"Texture Design" 0 5 0

"Self Portrait 0 0 3

Total 13 12 8


Classroom Observations Methodological Framework

Documentation

InterviewsO

bser

vatio

ns

c r e s s t / u c l a qualitative indicators: classroom observations and classroom practice ann m....

Documents