Sacramento Area Collaborative for Teacher Preparation

Using Qualitative Evaluation to Identify Common Elements of Successful Reformed

Courses

What is SACTE?

• Recruitment of secondary science teachers

• Revision of math and science course for preservice elementary teachers

• Faculty professional development

CSUS Context

• Teacher credential is 5th year program• Content preparation in Liberal Studies

or Child Development B.A. program• New “blended program” combines B.A.

and credential• 59% transfer students,

average age 23

Project Process

• Standards setting, RFP issued• Revision teams required to include

discipline faculty from both CSUS and community colleges, education faculty

• Team goals individualized to needs of program and discipline

Collaboratively-set standards• Incorporate information-gathering skills• Support wide range of learning styles and

cultural backgrounds• Connect content with K-12• Incorporate appropriate technology• Articulate with National and State

standards• Integrate foundational skills in math,

science and literacy• Student-centered learning

Required Products

• Syllabus

• Student manual

• Instructor’s manual

• Assessments

• Overall project manual for induction of new faculty

Evaluation Plan

• External reviewer monitored progress on monthly basis, trained faculty

• Surveys, student and faculty

• Interviews with faculty

• Observation of classes– CORE Evaluation– Qualitative observation

Observer’s role• Local evaluator experienced in educational evaluation• Observed all reformed courses at least once• Used revised CORE protocol• Listened to student groups as well as instructor• Recorded (in notes and on tape)

everything that happened in course, noting times of transitions betweeninstructional activities

• Analyzed data for common patterns, unusual events

Unexpected Happy Results

• Instructors designed courses to meet a common set of standards for quality instruction, but analysis shows the instructors converged on a set of common instructional structures and strategies not explicitlyspecified in the design standards for the courses

Cyclical Lab Structures

• Lab assignments were broken down and done in segments throughout the lab period.

• Each lab work segment “cycle” was done in groups, typically with continual instructor input, as well as feedback (either from instructor or students or both), before student groups moved to the next segment of the lab or day’s work.

Cycle Segments• Opening - info,

brainstorm, preassessment

• Student work in small groups

• Sharing ideas - students and/or instructor

• Instructor feedback• Class discussion

• More information presented

• Summarizing

Example: Physics• 37 minute sequence• Opening brainstorm: words connected to

electricity• Small group work: classify words into 2 lists -

electrical charges in motion and charges at rest

• Sharing ideas: whiteboardgallery walk, oral presentation

Example: Physics• Instructor feedback: Adds a new category -

historical context• Discussion: Are there words that were hard to

classify?• More info: helping students

review their sort, adding some new ideas

• Summarizing: interactive review of 3 examples of static electricity

• Next cycle starts

Common Instructional Activities

• Brainstorming• Classifying• Small-group discussion based in hands-on

activities

Find all the ways the

minerals in the box are

different from each other

Common Instructional Activities

• Whiteboarding– Group construction of ideas and explanations– Making thinking public– Presenting ideas to the class

Show in words and pictures what

happened to the salt at the atomic

level when you put the teabag in the

water

Common Instructional Strategies

• Labs and lessons broken into chunks and based around student progress

• Bookending lessons: intros that frame the concepts and wrap ups that validate the knowledge gained

• Lots of signposts in lesson to orientstudents to where they are in the process

Common Instructor Behaviors• Instructor poses lots of questions for students to

address• Frequent feedback from instructor, near constant

interaction with students• Explicitly making connection to K-6 classroom

Kinesthethic modeling of

salt dissolving in water

Common Instructor Behaviors

• Drawing connections between concepts, revisiting earlier concepts

• Frequent checks for comprehension• Just-in-time teaching - explanations

provided at the moment when needed, not before

• Establishing sense of camraderie with students - colleagues in learning

Common Student Behaviors

• Engagement increased with hands-on activities, decreased with lecture

• Enthusiastic involvement in class discussions

• “Rowdy” group with fluctuating engagement

• Even “rowdy” groups completed work with apparent comprehension

Common Challenges• Bringing students back from excursions into

everyday examples - e.g. bad hair days and static electricity

• Differentiating instruction• Dealing with mixed age groups

in small groups - mature self-starters vs. uninitiated freshmen

• Affective aspects of group work in diverse populations

Applying CORE Evaluation

• Instructional strategies– Many different kinds of activities happening at

once or in a short period of time• Hands-on activities involved small

group discussions, teacher interaction, and more.

– Categories describe behaviors, but not quality or context

• Student/teacher interaction could be procedural, modeling problem solving, enforcing class norms, eliciting student explanations, etc.

Applying CORE Evaluation

• Student engagement– Students tended to dip in and out of “on-task”

talk - hard to monitor in multiple groups– Student “off-task” talk helps cement social

relationships that allow risk taking• “rowdy”, apparently off-task group

created and sustained a dynamic whole-class discussion

Applying CORE Evaluation

• Cognitive level– Observer found it hard to distinguish

categories– Categorization notes what instructor is

doing, NOT what students are doing. Student cognitive level is best determined by analysis of student work.

Take-Home Lesson

• If we had just used the CORE evaluation, we would have missed lots of interesting things happening in our classrooms

• Qualitative research expands the range of things you can look for in quantitative research