(1 ut is cl (1) 0 n W. ft er 1, et Ili 0 0 i`i. 0 () Al 11 Po 0 ttl Ito 1-1 LI tt 1 %di od A 110 ti4, id m mi 0 ty 0 is 1-+ (A 13 (1) a a a ti tea )4 M t4t m as el 141 el 0 4.1 IA 0 1.1 a OA ft f) 1/1 Ai tri ei ill e`i i+ 0 n 6.1, ft tt 0 111 0 0 en 11.1 t11 111 14 tit /4 4tis. m 0 M ors 0 (AN ea 0 1 is IA M I- il) (1 tii IA Pi et Al A IN nth to to tu meN. a 11 HI m el m a tit 0 Is' 0 H. el a n 0 a a LI, lb Ilt toi iii rti 11 1 +I0 t..1HI M It Ps to et 1 H 1.1. at Al 4)w (31 (I. N et 1 1 0 N 1 01 1,1 1C1 1w btl 0/1-1. 1.11 1,11 11 til at 111 %CI 0 0 H. el It 0 H. 1.1 It tb H 1-1 n PI 1,1 /13 1 )0 Cr a CV Pt Oa d 0 0 11 al et 11 0 0 et II lit m 1,1 H 1,1 HI 411 amk

al 1..1 I4 F.A. CI n la. I I I t 1 VA 0 tr « tit va m 0 al tei 0 cu t4,1 0t1 4/1et tb et 0 I-o ft hat a tit lb a* In 0 n ri 0 tt 1 1t1 19 NJW' I a) ka i.. I 0 1 '4:4 W DI ... 11 ft Pt 111 a It & IV W LI IA0 re gx. til 64 H At 13 ro et CI r t3 RI 1-1 11. Att (It PO m CI a' no- 1-1, M n 14 11 ni 1d, 0 re H

IN 1,-.. N H. M i'a s'a Oo 11. H.1 r, n er n mo.i. m,r, n 0 ta n et w go 4 m 01 1.1. m 0 N H M et tr ft) ni to. ai (D et to H. re, VI ft 1-o t4 t-t. V P9', e1 t1 At0 M IA fo :.1 11t t1 0 C re Iv 0 in 0 fl, 0 tv t+ 11 et At P1 "1.11 * ie. * N4 141 ft * I* M hoi No tom% r) HI 1.4n 11 0 n ni n CL m 1,I CL W Oki 11 ti kJ fb tr DI tr 0 if tr n r) a Pt 11 al re lik, Pin M 11 iii -4) ?i A 13° 0tO C 0 0+ 1,1. 0 4 re 14. li 0 a la. 11 It) M n M M 1.1. M P) 0 111 It $11 a 14 n I .,i Ai itt oti Pt) hi 111 PIIn n CI li 14. tb IA 0 Al is I-a 0 m n W MI et, ot IA re 11 Hi H IP 1.Q a 1.1 n M tr hit PI 11 el 11 111 tO0 0 R. et. et, Ill 0 NI 15 IF 1-1. tl U 14 Hs et tt At At r1 14. 11 11 PV, w. n .: a ri It... e.., re r) IA In t) le, mI IN M Pi P ' 0 0 t2o 0 a tf IA Mira 11 (11 Ii. el ile o m 1-1, lir tow & M . o i,. n I g f m ri,

ots rt M 0 DI lj t) 13' 111 It 10 141 11 et h IN t J. 1...1 et lo, to P. H. to at 01 14 to hi 0. At 0 N... At V to tai (1% th a le PI !,..1 CI ri M n et IA t) 10 Ili H. 0 . I-o IV n Pi ill o I+ m ko.t n !I N ...d 13 1,41

10 t i' to 0 Co 0 ... et of to tr 0 et 0 0 a a i-+ n 0. ti tl ili 4 Od1 04 0 111 el tOin 11) M (1, 1.4 04 V 0 LI ID fh 1...+ e... n re re 0 a 0 0 n 0 rt. n t n il) tt 0 is fr) Pi t1 C.)- t3 0 11, IA - co 0 et, et- 1..o 0 M lai et trc3 Po el u 0 0 .ot 14, al IA M 0 el n al ri m to lA.1000 el N 01N 0 0+ I 1 ) %LI 0 0 N N II IA V' et 0 H et 41 M' I 0 0 0 0 0 tit

la et, 0 1-o 0 t9 11 0 (I) 0 11:1 n t4 13 tl C 11 H. (11 0 .. La. 0 til II et, C At )4 1d P P$ n Pa / ft Ms ?icit, er At isi a all ef GI eti 01 M et H N' A) ti ti f) lb 0 a M rig Or 11 Al CI 11/41 0 is ry pi 01 ri 010 ef 0 tD 113. 11 0 /.. V 1:3 H. tr ra I+ to nwth m....g114,1 Al sm. e) 0 11 ft to n 0 11 ti. tl %I " 11:t : g% ir' °I re t; t',.. `o'' " I') 4 tit

ata. 401 el re iv 0 el ..... 1-, 0 Is e er re '64 o iii, rl 1,11 0+ *nonat:tch cow :,

elin

a* I T Itt 0 rt 0 0. n 1- ti cu n n a 111 '''' 14 Di III Ws 1/1 11 4.1 0 111 el too 1.1IN H. 0 IC 14* C '" 0 0 0 Ho 0 et 0 fi* M 14 tit et Ilii 0 IC " l3 0 13 PO et tl 0 1et tA 0 II H 11:o 0 el' II NJ ft HI a) a 0 rr 1...11 0 it) IV

0 0 0 M4 0 fr 14 ott I-o ki 0 M 0 At IN 11 0it IV H. 0 11 6 1.4 11/ 141 1.I gi 1t4 Lit Ili111 0 0 tra n to is a to IV Iii tkt ifl

1 1 I-J, M I C ) ) ) (0 M 111 a M m th 64 el a . n o HI n or II to M Ot n t4 to 1/1OWWf+1110 1.41e1.istneretMiJg 000 WU. W Peet HI 14 tr A it) AI 0 fil 00 P. 1' /-1, HI A / f t lb a a 0 t j p i a M ID ti, 41 a a n H* qo Ira a In ti, Po o laF a rt. n t - 1 r , a 0 t 1.1. a a i l a 0 tt .... t o ti n tr m wa 4 ma ti co to o om id ('1iai 01 so ..) Pi 0 tA ti, n 0 M %.4 a 0 Fa n a 1.4 M ein a m *1 Pt ). fl4 M 0 II 0 et 14:1 ollH r 111 is rt 0 1J104 is M I-I A) 0 a rl IA lo et. AI M el, 0 la, H tr 0 Nt z t ' 0 o V i l 0 1 3 M . , 1 3 t i t ti s o 0 0 0 , 0 1 3 0 , 0 0 a t t i e l J i r ) to re n * tO II IN Im 0 WI n a on Pim a PI PO CD a ri PI 1-1 IA )41 1, It M ou 0 la, t4 a 11 1.d Da si r1 m M g.) U' 1.1. pi

IR c+ 0 1-4 11 ri IA to. la, 0 0 0 rt) es, 11 f+ li et I-o V AI to, 13 11 M L4.1 ..4 too Ho Ilt0 0 At 14. (0 al otrim(anto 1+ et 0' tr et H Fos HA 11 0 IN I+ et 0 NO as ti to0 C 0 Hi 41 H,I 0 1,1 el, 0 et tt. H 0 0 ill di it tr A MM el s AI if I I/1 11 Ci If4 ) 1- n i z a al a es.... 1411 n 11 al a iir n m i- 11.11 l'd Pr P1 .-1 Kt 0 0 11 lbfal Oa H. rtt et et p. n a 0 0 0 0 et. 0 0 n o to 1-1 t-41 0

M IA iwt Iti 61 0 0 0M 1.1. Ho 0 loo 00 IN M H II 0 et M n vat IA a Cu a * to 11 et 0 n 1di Ora

r o Pm th 0 0 )3 Pt3 CU fli 11,1 0. ti lir it to Pi (A 111 fl el' 01 Ho 0 IA M. PI re mt4 II CI t1 al 11 (11 1-14 1.1 0 Di 0 tA M CU 0 41 (I Id, Co a bs f r m IA I.+ 0 La 131 111Olk thaelD0P1 Mrlaii 1:1 1-401/1 It 11 ft a 0 14.i Ili ft 14. n I r et a 01ti ft) AI 0 IS' P1 t+ PC ()) Al fai 111 )1 tA P' IA U 0 hi 0 Is 141 ft WI tit 0 64 P. ItO Prni P. M l" MI P. (4. 01 li I.-1 I-o to, I+ ft 14 0 III r+0 0 et " 06 Po)) rt, u el et, 0 IP MM ai fl/ la. V a P. a a t...e tr e+ 41. 111 0 Id* 113 he to, 0 al H. Co t-1 *no 0 th Het (1) Al et 0 ft Po e+ tii W. AI 410011014% IN CI 0 I:1 P. n a a . pi

II a kr e+ 0 I+ di r+ at to, et. A IN II 1+ t.o. 11) ft) ei A) "4 is) tfi 0 MIA " 40 As m et it) er 0 1.-1 it 0 di id 0 . e# ft 1- n n N a a 0 pi ..1..1 M et%

Hi N H* in ft) Ho 12 01 0 li tn IA et 0 IN or1 et 111 el *" el 016.1 00 0 0 0 et (0 et to I-I 0 0 to P. et. CD lot, NA 14 Pi o rhre co to ts le 0 mrs, et. 11 u) m o a ro is. a a ep r+ 0 11' P11 et

tA IA N, 0) tti la. ti 0 M es MIv ill 1 121a ts,

(t) Al 0 o m 1... 0 cu P. m cu a or 0 m n U'0 ts tt id 0 0 ir Pm Cm ise id ri a 0 ir to il)M 1-i Pe 0 it In M a 0 n M

M in Pi a a a i tl rirt. lal

vS sEvaAirorteifer.ANIFICIEs.EINKAINIMEllsbElIFgagss isesum. earifs Tuff Of

...eworAllivest" t.C.f..,;;Vfaul ..uaS 110Ev steipso.:Loom Esa.r ss to",f wIED taffeA

igs curzaorez.vlovioalrow.I.-11:x 1.-:CialCd1001104

yrs/cz Arc/ wrEtiCSSaanmst.u/ "t2 4sSTAIVICOCttb 'Z' riv

C1--

THE DESIG% OF A sm.(

TO VALIDATE TEACHER COMPETEXCIES

IX TEM OF PUPIL CROWN

Jeffrey L. Lorentz

West Georgia College

A Paper presented at the 1976 Annual Meeting, American

Educational Research Association, April 19-23, 1976, San Francisco,

California.

INTRODIXTION

Encouraged by the federal support for the "Title" programs which

proliferated during the aid- sixties, researchers have had the resources

to Investigate a broad spectrum of educational programs. This federal

largess, however, was not without its drawbacks. Program guidelines,

funding cycles and other considerations often proved so restrictive or

unreasonable that the conclusions from such programs were insignificant,

not rcplicative, or even worthless. Nor were the initiators of many of

the studies, however, without fault. Bloom (1966) in his AERA presi-

dential address contended that, while a 2,000 precent increase in federal

funding for educational It & D provided a strong motive for studies of

education, only one in 1,000 studies reported --3 per year--were crucial

or significant. Consequently, while the era has been an exciting and

productive one for research, it has often been frustrating as well.

Program evaluators and statistical and design consultants working

with funded programs are often called upon aid -way into the studies.

They are handicapped by insufficient prior planning, inadequate liter-

ature review, Improper instrument selection, inadequate comparison

groups, unreasonably small samples, and unrealistically short periods in

which to test for treatment effect. Finally, the need to interrupt the

study at crucial times to produce Interim and final reports is

especially trying--since the reports are often due before the final and

all important data can be collected and analyzed.

1.)

2

A7

The above is presented in contrast ttikfh-environment in which

the present study has been conducted.

The Carroll County Competency eased Teacher Certification (CRTC)

Project, conducted by the School of Education, West Georgia College,

and Carroll County, Georgia, teachers, under, an ESEA Title III Grant

fr^n the Division of Program and Staff Development, Georgia State

Department of Education, is now in its third year of operation. The

rajor objective of the project is the development of a system for the

re-certification of experienced classroom teachers which is a potential

alternative to the program--approved approach currently in use in

Georgia.

The CFTC project has been characterized by a healthy freedom from

many of the problems alluded to above. From the outset it has been

proposed as a long-term study which would be likely to produce little

immediate results. The project design has been flexible and subject to

immediate modification as needed. The Georgia Department of Education,

as the grant agency has provided minimal guidelines apart from a broad

mandate to develop a competency based certification system, and has

requested few formal reports.

This absence of unnecessary guidelines and constant interference by

the grant agency along with a flexible design should not be viewed as a

weakness. On the contrary, the project is established on a sound

theoreticzl base (CBTC Librarian's Report, 1974). More important, it also

has the advantage of regular, continuous contact with a team of con-

sultants, each of whom is an outstanding authority on teacher effectiveness,

educational research or statistical analysis. Finally, this freedom and

3

flexibility has been deliberately cultivated due to the prototypical

nature of the project: one hoped to be worthy of replication.

The risks of such freedom are obvious and the investigators are

apprised of these risks as voila as the responsibility. The benefits

to be gained, however, are telt to far outweigh the potential risks.

The Project Desirn

A major goal of the project is the development of a model for the

identification and measurement of teacher competency areas; and the

assessment of the extent to which these teacher behaviors (i.e. "com-

petencies") affect student outcomes, and the extent to which they relate

to school goals and objectives. Although the CBTC project is limited

to Carroll County, Georgia, it is anticipated that the results of the

study will be applicable in other school districts both within and

outside the State.

The project design is based on the following theoretical framework:

The certification process will involve several stages including a

screening procedure, classroom observations by trained observers in

natural settings, and a study of certain relationships outside the

classroom. The collection of date relating to student outcomes will be

used during the developmental stages, but not-as a basis for individual

teacher certification.

The overall plan which was established and followed during the past

three years involved several steps including a) the establishment of a

competency list and related teacher behaviors and student outcomes;

b) the selection of appropriate measures of classroom behavior and

student growth; c) the collection of data in on-going classrooms;

d) the reduction of observation data to competency-related scores;

and e) examining anticipated relationships between observed competencies

and student growth.

Subsequent to the successful co lotion of these preliminary steps,

replication in additional classrooms would provide 2 validation of the

findings and confirmation that some competencies were indeed measurable

and their presence resulted in student growth. This in turn would pro-

vide the Department of Education with a preliminary set of measurable

teacher behaviors which had been shown to result in student growth and

which could then be used as part of the certification process.

METHOD

Introduction

The first year of the project (1973-74) was devoted to a review of

the literature, site visits by project staff to similar projects and

other developmental activities. Local teachers and administrators,

assisted by project staff and consultants developed a set of generic

teacher competency areas which were based in part on existing lists,

but considered essential to all local teachers. As each competency was

adopted as part of the definitive set, performance criteria were also

listed. This added specificity to the definition of the competency and

also provided guidelines for the selection of the measuring instruments

which will be the heart of the competency-based certification system.

Eleven generic competency arras and some 40 sub-competencies comprised

the resulting list.

5

During the first year also, potential instruments were identified,

additional staff were selected, and teachers were recruited to sarve as

subjects of observation. The second year (1974-75) was devoted to data,

collection and the present (1975-76) year given to additional data

collection along with the analysis of the initial observation data. The

remainder of the present paper is devoted to a discussion of these latter

activities.

Subjects

The subjects of the study are the 60 teachers who participated in

the CFTC Project during the 1974-75 school year and were selected as

follows: All teachers in Carroll County were contacted in the Spring of

1974 and shown a slide/tape about the CBTC project and asked to volunteer

as observers or observees. Three observers were selected and sixty

teachers agreed to serve as subjects. Letters of agreement were signed

outlining responsibilities of both parties. Summer workshops were con-

ducted to familiarize participating teachers with all phases of study

including competencies, observation instruments and student outcome

measures.

These teachers, being volunteers, represented grades 1 - 12, a

wide range of subject areas and varying levels of experience. Both

male and female and black and white teachers are represented in the

sample.

Student Achievement Measures

The Scott Foresman Initial Survey Test, Form A (IST) (Monroe, Manning,

Wepman and Gibb, 1972) was used as the pretest (fall) measure in the first

grade. Subtests include Reading and Math. Since the Scott Foresman

6

Reading program is used in Carroll County primary grades, Scott Foresnan's

1ST is used as a placement test in all first grade classes. Its validity

and reliability make the 1ST an appropriate instrurent for use in the

present study.

The selection of an end-of-year first grade measure uas node after

several different tests were tried with a small sample of students not

in the project. The Comprehensive Test of Basic Skills, Level 3, Form S

(CTBS) (CTB/McGraw-Hill, 1974) was found to be the most appropriate in

terms of difficulty. Students were able to respond to some of the items,

but did not reach the ceiling. (Both floor and ceiling effects are

problems with some tests in the lower elementary grades).

The CTBS includes a total Reading and total Math subtest. The

reported validity and reliability of the test are satisfactory.

The Iowa Test of Basic Skills, Primary Battery, Level 7, Form 6

(ITBS-P) (Hieronymus, Lindquist and others, 1972), a widely used and

well-known achievement test was used in the second grade.

The Iowa Tests of Basic Skills, Form 5 and 6, Levels Edition (ITBS)

(A. N. Hieronymus and E. F. Lindquist, 1971) was used with the remainder

of the classes in grades 3 through 8. The test includes Vocabulary,

Language, Work Study Skills, Reading, and Math Subtests. As with the

other measures, the test's reliability and validity are adequate. In

addition, the State of Georgia makes use of the ITBS Battery in the

4th and 8th grades in the state-wide testing program.

The Tests of Academic Progress, Form S (TAP) (Houghton Mifflin,

1972) are designed to measure the extent to which the objentives of a

basic area of high school instruction have been achieved. The six

7

sub:,..-t are: Social Studies, Composition, Science, Reading, Mathematics

and Literature. The appropriate subject area test along with the Reading

Test Was used in the various high school classes. The TAP is used in

the eleventh grade as part of the Georgia state-wide testing program.

Validity and reliability of the instrument are satisfactory.

Student Self-Concept Measures

I Feel - Me Feel (IFMF) (Yeasts and Bentley, 1970) was selected for

use with students in grades 1 - 3. This 40-item Likert-type scale

(using five faces which range from happy to sad rather than numbers) is

a self-concept measure appropriate for use with children at this level.

Scoring was based on locally-developed factor keys which were

entitled: Academic, Self, Frustration, Femininity, Fun and Independence.

Coefficient Alpha reliabilities for these scales ranged from .54 to .84

for the CBTC sample.

How I See Myself (HISN) (Gordon, 1968) was used as a measure of

self-concept in grades four through eight (Elementary Form) and grades

nine through twelve (Secondary Form). Both forms of the instrument were

scored using four of Gordon's Keys entitled Teacher-School, Physical

Appearance, Interpersonal Adequacy and Autonomy. Reliabilities for

these scales, using CBTC data ranged from .64 to .83.

The Junior Index of Motivation (JIM) (Frymier, 1970) was also used

with students in grades 9 - 12. This instrument is used to assess

students' motivation toward school. Fifty of the 80 items are scored to

produce a single index of motivation. Frymier (pp. 60-85) reports a

number of studies in which the validity and reliability of the instrument

were assessed.

111

8

Classroom Observation Instruments

The Coping Analysis Schedule for Educational Settings (CASES),

(Spaulding, 1970) is a category system designed to measure pupil

socialization and consists of 19 categories of student "coping" behaviors

which are identified by descriptive statements such as "Aggressive

Behavior," "Self-Directed Activity," "Observing Passively," etc. The

CASES categories are arranged with the more active coping categories

grouped at one end and the more passive categories at the other, but the

numbers do not represent a scale. CASES has been developed over a ten

year period as a result of more than 1,000 case studies in on-going

public school classrooms and o-her educational settings. Relationships

between CASES and student achievement have been established by McKinney

and others (1975).

The Florida Classroom Climate and Control System (FLACCS) (Soar,

Soar and Ragosta, 1971) examines the control tactics of teachers as well

as their affective behavior. It includes items relating to the nature of

classroom structure, teacher and student control strategies, and teacher

and student affective behaviors, both positive and negative. In Follow

Through Studies (Soar, 1973), data indicated that FLACCS discriminated

significantly between programs and related to pupil growth.

The Observation Schedule and Record, Form 5, Verbal (OSCAR 5V)

(Medley, 1955) looks at the verbal behavior of teachers as perceived by

students. It is based on 14 categories for teacher questions and state-

ments and four categories for pupil--initiated utterances; by examining

sequence it is possible to recognize some 600 transitions from one event

to another. OSCAR has been widely used for a number of years and has been

the basis for several other instruments.

i 0

9

The Teacher Practices Observation Record (TPOR) (Brown, 1972)

measures the congruency of observed teacher behavior with Dewey's

philosophy of Experimentalism. Half of the 62 items describe behavior

which reflects agreement with Experimentalism, while the other half

reflect disagreement. All of the items reflect behavior which is found

in public school settings.

Other Measures

Socioeconomic Status. An estimate of the socioeconomic status (SES)

was obtained from each student based upon the occupation of the father

(or the mother if she were the family's sole support). This occupati'nal

information was transformed to a 1-8 scale (1=high status, 8=low status)

using a modified version of Warner, Meeker and Eells (1960) classification

of occupations and le'vels. As an estimate of the reliability of the

assignment of SES, two raters independently assigned SES scores to a

small (N=77) sample of students using the procedure described above.

The resulting intra-class R was .94. Students with missing SES were

assigned the mean for their classroom, rounded to the nearest integer.

Interview. A fifty-five question interview, developed with the

assistance of Selection Research, Incorporated, of Lincoln, Nebraska,,

was designed to be used as a potentially useful screening instrument.

The questions are scored dichotomously by comparing each response with a

criterion. Responses which paraphrase the criterion are scored as

correct and those which do not reflect the criterion are not counted.

Items are grouped into eleven sets of five corresponding to the eleven

CBTC competency areas.

10

Data Collection

Student Outcomes. Achievement and self-concept measures were

administered early in the Fall of 1974 in the classrooms of 60 Carroll

County teachers, and posttests administered again in the classrooms of

the same 60 teachers--all of whom were still in the project in May, 1975.

A group of specially--trained testers administered the tests in

each classroom. Where necessary (i.e. in primary grades) the tests were

administered to small groups of students and care was taken to insure

that students understood the testing procedure, instructions, etc.

After the pretests had been administered, each of the approximately

1800 students in the 60 classes was observed using the CASES instrument.

This task was completed in three weeks. Each child's record was scored

to identify his coping style. At the end of the year, two observations

were again recorded on every student in the 60 classrooms.

Process data. After the pretests had been administered and the

CASES scored, six children with different predominant coping styles

were identified in each classroom. These six students were used in

subsequent observations using STARS. Six such observations were made

in each classroom.

During the year, each teacher was visited three times, and three

five-minute samples of behavior were recorded on each occasion on the

TPOR. Each teacher was also visited on six other occasions on each of

which the first five minutes of behavior were recorded on FLACCS, fhe

next ten minutes on OSCAR and the last five minutes on FLACCS.

Visits were scheduled in advance, but no attempt was made to pre-

select the activities to be observed, since the intent ras to record a

representative sample of the behavior present in each classroom.

ll

During the Fall, 1974 every teacher was interviewed with the CBTC

interldew schedule and has been interviewed again during the Fall, 1975.

Data Reduction

Initial Processing. Tests were hand or machine scored, keypunched

and verified. Observation data were keypunched from the data collection

forms and verified. Achievement tests were scored according to the keys

provided by the various pniblishers. The pelf concept measures were

reduced to subtest scores by summing various combin. -as of items_

CASES frequencies for each student were scor.1 using Spaulding's procedure

(personal communication, 1974) to identify predominant coping styles for

each student from Style A (aggressive, manipulative) to Style H (other-

directed, task-oriented), as well as a Composite Score (styles A -H).

Uben CASES observations were made, observers distinguished two types

of classroom settings: Teacher Directed (TD) and Program Directed (PD).

There i3 evidence from Spaulding's studies and from preliminary CBTC

Project analyses that changes in the classroo.d setting resulted in changes

in the student's CASES scores. Scores were therefore obtained for

a

students for the appropriate setting by combining data from both initial

visits only when collected in the same setting. The same procedure was

followed for the CASES data collected in final (post) visits. Thus

for one student, one or two sets of pre-CASES scores could be obtained

(e.g.: both observations occurred in TD Setting; both occurred in PD

Setting; one occurred in PD Setting and one occurred in TD Setting), and

one or two sets of post-CASES scores obtained.

In general, the following steps were taken to reduce the observational

data on teachers.

1e

12

STARS. The 475 cells in the student/teacher interaction matrix

were weighted and averaged to obtain an overall score and portions of

the matrix are combined to produce scores reflecting various teacher

types. These 12 scores are listed in Table 1. One set of scores was

obtained for each student observed during each observation period and

for the overall class if the teacher interacted with students -t

specifically observed. These individual scores were averaged for all

observations to provide class means to be used in subsequent analyses.

The 475 cells were also ret:ined as total fretNencies across all obser-

vations for each teacher.

OSCAR. The record of sequential teacher-student behaviors was

scored to produce a 600 cell matrix of occurrences of interest (out of

9999 possible). These were weighted and reduced to percents to produce

22 keys, twelve of which relate to the Carroll County Competency list.

These are also shown in Table 1. One set of 22 scores was obtained for

each classroom observation period, and average scores for the 6 obser-

vation periods were then produced. The 600 cell matrix of occurrences

for all observations was also retained for each teacher.

FLACCS and TPOR. These data were reduced from a series of items

(FLACCS=186, TPOR=62) for six series of observations to competency scores

for each teacher by Robert and Ruth Soar at the University of Florida.

These competency scores are based on a priori competencies identified

in the Carroll County CBTC Project. They were normalized for the total

project group (N = 60) by converting items to T scores. By combining

sets of these items, FLACCS yielded 28 scores and TPOR resulted in 21

scores. These are shown in Table 1. The 186 FLACCS item T scores and

62 TPOR T scores were also retained for each teachet

13

TABLE 1

SVBSCOMS: Fill STARS, OScAR,

FLACCS, TPOR AND EZiraIVIEll

VARIABLENUXBER KEY DESCRI7ION

1 S1 Overall composite2 S2 The Story Teller

3 S3 The Boring Lecturer

4 S4 The Examiner

5 S5 The Entertainer6 S6 The Controller7 S7 The Counselor8 S8 The Psuedo-Peer

9 S9 The Discovery Teacher10 S10 The Socratic Teacher11 Sll The Effective Manager

12 S12 The Effective Expository Teacher

0ScAR:

13 01 Deviant Behavior

14 02 Listening15 03 Pupil Involvement in Subject Matter

16 04 Pupil Involvement in Planning

17 05 Clear Instructions

18 06 Pause for Student to Answer Questions after Instruction

19 07 Clear Explanations

20 08 Pause for Qty_itions afte: Explanations

21 09 Pupil Speaks Freely

22 010 Self-control

23 011 Pupil Initiations

24 012 Question Difficulty

25 013 Teacher Statements

26 014 Confidence27 015 Feedback from Teacher to Students

28 016 Rebuking Behavior

29 017 Managing Behavior

30 018 Lecturing

31 019 Questions Source Modified

32 020 Question Difficulty Modified

33 021 Question Quality

34 022 Permissiveness of Teachers

It

14

7ABLE 1 (continued)

VARIABLENIMER KEY DES

FLACCS:

35 Fl Student reduced deviant behavior

36 F2 Student better physical, mental health

37 F3 Student enjoys class, happy smiles

38 F4 Student on task, actively involved

39 F5 Student absence of withdrawn behavior

40 F6 Teacher pauses, elicits and responds to studentquestions before proceeding

41 F7 Sage as F6 above but a sunnative measure

42 F8 Teacher uses a variety of methods, verbal, todeliver instructions

43 F9 Teacher uses a variety of methods, verbal, plusselected non-verbal, to deliver instructions

44 FIO Teacher uses a variety of methods--facial--gestural

45 Fil Teacher uses a variety of methods, verbal and non-

verbal to deliver instructions46 F12 Student self-directed to move toward task

47 FI3 Teacher demonstrates proper listening skills

48 FI4 Student able to speak freely

49 F15 Teacher utilizes non-verbal communication skills

50 F16 Student able to follow directions, on task

51 FI7 Students feel free to interrupt presentations

52 718 Student movement toward tasks

53 ''19 Teacher provides feedback to pupil on his behavior

54 F20 Teacher maintains self-control in various classroomsituations and interactions with student

55 F21 Teacher recognizes and treats individual student

behavior56 F22 Teacher uses more than one instructional activity

simultaneously

57 F23 Teacher evidence of a personal one-to-one relation-ship with each student

58 F24 Student evidence of importance as class member- -group

involvement

59 F25 Teacher supportive classroom management

60 F26 Student evidence of enthusiasm

61 F27 Teacher listens to students and provides feedback

62 F28 Student high interest

35

TABLE I (continued)

VARIABLE171T2LR REY DESCRIPTION

TPOR:

63 T1 leacher selects goals and objectives appropriate to

pupil need

64 12 Teacher matches student with appropriate materials

65 13 Student on task actively involved

66 14 Teacher gathers multi -level materials

67 T5 Teacher gives clear, explicit directions which are

understood by students

68 16 General task

69 17 Teacher respects individual's right to speak

70 18 Teacher helps pupil correct cognitive misperception

71 19 leacher accepts necessity of dealing with individual

students on individual basis

72 110 Student actively involved

73 111 Teacher- - evidence of praise and/or rewards in

operation

74 112 Teacher accepts and incorporates student ideas

75 113 Student expresses ideas and opinions different to

those of teacher and peers

76 114 leacher evidence of one-to-one counseling and

absence of ...rejection (brush-off)

77 115 Sum of 11, 12 and T3

78 116 Sum of 112 for teacher and T13 for S

79 117 Teacher control--general

80 118 Pupil choice of subject matter

81 119 Teacher choice of subject matter

82 120 Expansive treatment of subject matter

83 123. Restrictive treatment of subject natter

INTERVIEW:

84 11 Diagnosis

85 12 Organization

86 13 Teacher communication

87 14 Coping

88 15 Insightfulness

89 16 Student--teacher communication

93 17 Methodology

91 18 Self-concept

92 19 Emotional awareness

93 I10 Teacher growth

94 Ill Public awareness

1 r*

16

INTERVIEU. Each interview was tope recorded and the tapes were

transcribed. For each of the 55 questions, a correct response hzs been

established (which may be paraphrased or stated in the subject's own

words). Fire items on each of the eleven CRTC competency areas are

summed to produce a subscale =ore. These eleven scores, related to

the eleven Carroll County competencies, are shown in 'Table 1.

The item-level data for STARS (475 interaction cells), OSeAR

(600 occurrences), FLACCS (186 item I scores) and 7POR (62 item I scores)

were combined into a vector of 1322 variables for each of the 60 teachers_

Reduction of student data. The voluntary participation of teachers

in the CRTC Project, lack of equivalent tests across grade levels,

to_udent attrition, and related problems, resulted in some significant

complications in the analysis of the student data. In general, the

followinz steps were taken, subsequent to scoring described above for

achievement, self-concept and CASES.

On an instrument, by instrument basis (and within achievement tests,

by subtest), in order to maintain as large a number of subjects as

possible, student pretests and posttests were matched. Students with

missing pretest or posttest were dropped for that portion of the

analysis. In some instances, primary teachers taught one group of

students Reading and exchanged them with another project teacher for

Math. These students were assigned to the "correct" teacher for each

subtest.

JIBS sub tests for grades 3 - 8 were reduced to a common index by

converting raw scores to grade equivalent scores using norms tables, and

then subtracting students actual grade from his grade equivalent scores.

17

Vsigsg LY.D041' Amalysis of Covariance program (Dixon, 1974)

adjusted posttest means (i.e., regressed gains) were computed for each

variable. 1r every case, pretest and SES were used as covariates to

adjust the posttest scores. Resulting means were transcribed and key-

punched. The number of students and mean pretest and mean SES were

also recorded.

CASES gains were computed separately for PD and for ID settings.

Analysis Procedures

During the 1st year of observation and data collection, some 50,000

punched cards of raw data were stored on disk files and a large number

of variables were reduced to a snaller set of indices for each of the

60 classrooms. A major task during that year Was the developnent of

scoring routines and other analytical processes. Resultant statistical

analyses were generally descriptive of the sanple.

Due to the nature of the data collected (and supported by the

advice of project consultants) the overall design for data analysis

during the present year has been divided into three major phases. The

first phase, the initial reduction of observation data to a set of

" competency scoaes" and the computation of student gains, has been

described above. The reduction of the observation data to competency

measures related to the CFTC competency list was undertaken in response

to a major project objective. The second phase is the examination of

the relationships between the competency scores and student gains and

the third is an examination of the observational data for patterns and

an attempt to sort teachers into groups. The second and third phases

have been undertaken more or less concurrently.

is

Competency scores and student gains have been related in a series of

multiple regression analyses. Some of these are reported elsewhere at

this AERA meeting in papers by Soar and Spaulding. Groups of teachers

with sindlar tompetency scores (identified in the Phase 3 analyses) have

been contrasted, using student Baines as dependent variables, in a series

of discriminant analyses. The resulrs of these analyses are discussed

below.

Teacher grout's. One major phase in the analysis of observational

data has been an attempt to sort the teachers into groups with fairly

homozeneous teaching styles. Teachers with similar profiles on the

observational measures form similar clusters, or groups. To determine

which teachers belong together, it is necessary to examine the profiles

of all teachers and to find those with similar profiles. The number of

groups can range from one (i.e., all teachers exhibit similar scores on

all measures) to N, where N=the number of teachers (i.e., each teacher

is considerably different from every other).

Several alternative statistical methodologies addressing this pro-

blem of grouping are available. Transpose Factor Analysis, or Q- Analysis,

is one of these methods which was selected for the initial grouping.

The 19 x 25 (475 cell) matrix of STARS frequencies was summed

across observations within each classroom, resulting in a vector of 475

"scores" for each of the 60 teachers. This total 60 x 475 matrix was

transposed and factor analyzed using Guertin and Bailey's (1970) program

£0501. Four factors were selected for rotation. These four factors

represent an initial clustering of teachers according to STARS patterns.

19

The names of the teachers who had the highest loadings on each

factor were examined by project staff to determine if similarities

existed. It was apparent that neither grade level nor vubject natter

were held in com=on by the members of any group. In fact, the

differences were more apparent than the similarities.

As a next step, the four groups were contrasted on the 12 a priori

STARS keys through a discriminant analysis using Veldman's (1967)

program DISCRIM. Results indicated that the groups were significantly

different overall (Jilk's Lambda = 0.079; F = 5.05, p<.0001) and

significantly different on six of the twelve scales. At this stage,

the investigators felt that two conclusions were supported by the data.

First, it Was apparent that at least some of the a priori STARS keys

adequately described teacher characteristics and, second, and more

importantly, that the presence or absence of teacher characteristics,

or styles of teaching, could be measured by direct observation in live

settings.

The same four groupings (from the STARS 0- analysis) were sub-

sequently contrasted on the 22 OScAR scores, the 28 FLACCS scores, the

21 TPOR scores, and the 11 Interview scales through additional dis-

criminant analyses. Each of these confirmed that the groups differed

significantly overall on each of these measures (except the interview),

and that the groups were significantly different on several of the sub-

scales on each instrument including 8 OSCAR keys, 9 FLACCS keys, 5 TPOR

keys and 1 Interview key.

These results confirmed that the groups exhibited stable patterns

which could be observed directly through the use of different observation

20

schedules, by different observers and at different tines. It also

cQpfirmed that some of the behavioral characteristics of teachers, or

"competency scores" did exist in varying levels within the sample group.

It was of rarticular significance to the investigators that these

characteristics were exhibited by teachers working in natural settings

and were not artificially induced by the creation of a "test" environment.

In a subsequent analysis, the transposed (475 x 60) STARS matrix

Was normalized (i.e., converted to T-scores) and then factor analyzed.

Four factors were selected for rotation. These four rotated factors

comprised different teachers. These groups also differed significantly

overall and on 11 of the 12 STARS keys, 17 of 28 FLACCS keys, and 5 of

22 OScAR keys_ This suggests that multiple analyses are necessary to

adequately identify all the characteristics of the teacher groups.

Additional analyses have been undertaken. Two of these include

the transposed factor analysis of the 1322 x 60 matrix (columns 1-475 =

STARS, 476-1074 = OSCAR, 1075-1136 = TPOR and 1137-1132 = FLACCS) using

raw frequencies and frequencies normalized across all subjects.

Since CI-analysis, however, considers only the criterion of relative

parallelness of teacher profiles in grouping and ignores overall levels

of these profiles and relative scatter within profiles, two other

methodologies are being employed to establish groupings based on STARS,

OSCAR, FLACCS and TPOR. The relative discrimination afforded by these

groupings are being examined by means of Bottenburg and Ward's

hierarchical analysis, (Veldman, 1966) and Guertin's distance analysis.

(Guertin and Bailey, 1970)

Finally, in order to examine items in each instrument which dis-

criminate significantly, ea:h of the groups identified in the various

21

Q-analyses are contrasted on each of the 1322 items using a 1-way ANOVA.

The F-ratio is not used as an inferential statistic (since many would

be expected to be significant b' chance) but rather as an index of the

relative contribution of each variable to the overall discrimination

among the groups.

RESULTS

As discussed above, transposed factor analysis, or Q-analysis, of

the 60 x 475 cell STARS raw frequencies and the examination of the

resulting four groups ;),y discriminant analyses produced profiles which

were interpretable and meaningful to the investigators. This led to

further use of Q-analysis to identify teacher groups. A normalized

60 x 475 STARS matrix, a raw-frequency 60 x 1322 cell matrix, a

normalized 60 x 1322 cell matrix, and a standard score (z-score)

matrix were each subjected to Q-analysis and Varimax rotation of factors.

Following Q-analyses, the profiles of the teachers loading highest

on the factors rotated (from two through nine, depending on analysis)

were contrasted by discriminant analyses. STARS, FLACCS, OSCAR and TPOR

keys which discriminated significantly among the groups helped the

investigators to label these groups.

The group profiles tend to be stable for the groups across instru-

ments, but differences appear when the teachers are 7e-grouped on the

basis of successive analysis. This suggests both the need for multiple

instruments in assessing teacher behavior and the need to examine

multiple dimensions of behavior.

Factor analysis of the transposed raw-frequency 60 x 1322 cell

matrix and Varimax rotation of factors produced both two and three

2 r.

7'7

interpretable groups of teachers. The results of discriminant analyses

of the 3-group solution are discussei as an illustration of the findings.

Table 2 presents the Wilk's Lambda and F-ratios testing the

significance of overall group differentiation en profiles from four

instruments. These indicate that the groups are significantly different

on these dimensions. Group centroids are plotted on the two discriminant

axes in Figure 1 to illustrate the separation of the groups, and to

illustrate that the several instruments reveal different patterns of

discrimination.

Univariate F-tests revealed that the three groups differed

significantly on 6 STARS keys, two FLACCS and one 0ScARkey, and nine

TPOR keys. These are shown, along with group means in Table 3. Group

I teachers are characterized in part by relatively high scores on the

"Counselor" (S7), "Discovery" (S9), arid "Student expresses different

ideas" (T13) keys. The Group 2 teachers are partly characterized by

higher scores on "Boring Lecturer" (S3) and "Teacher respects student's

right to speak" (T7), and lower scores on "Controller" (S6), "Listening"

(02), "Student actively involved" (T10). Finally, some of the

significant characteristics of the 3rd group are higher scores on

"Examiner" (S4), "Student movement toward task" (F18), and "Teacher

helps pupil correct cognitive misperception" (T8). Other differences

can also be observed in Table 3.

Finally, to'determine if the different groups produce significant

differences in student gain, gain measures were contrasted in dis-

criminant analyses. The student gain on ITBS subtests did not differ

significantly. Neither did self concept (HISM) nor CASES setting 1 gains.

24

23

TABLE 2

RESULTS OF DISCRIMINANT ANALYSES

CONTRASTING 3 GROUPS ON STARS, FLACCS. OScAR & TPOR

INSTRUMENT WILKSLAMBDA

df

STARS 0.213 24, 92 4.464 0.0000

FLACCS 0.000 56, 63 1587.041 0.0000

OScAR 0.001 44, 72 41.479 0.0000

TPOR 0.044 42, 74 6.598 0.0000

2i

0

0A

S.0 4.7

55

5.:2

3.5

f. 5

to

2-0 4-1

2.5 4-.0

itf

si:5

24

Keyd =-- Group 10=-' Group Z.0= Group 3

0

0

-1.0 -3.5 -.3.0 -2-5 .01 .02. .03 .011 .05 .04 .07

STARS OSCAR 5V

.40 ,43 .46 4f ,11. .15

FLACCS

Fig.]. Group

.50

./5

.10

-.35

.30

.25

.20

s

A

0

0

.1.0 ..15 .50 .55 .40 ,I5 .10

TPOR

CenFroid5 from if- Discr;inani Anolyses,r 1

2"0

25

TABLE 3

SICNIVICANT (p < -05) C7A3LAC3FISTICS

OF THREE GROUPS

FROM DISCRIMINANT ANALYSES

E'LY*

1

CROUP MEANS

2 3

F(di = 2,57)

p

53 16.8 37.7 18-4 28-32 .0000

S4 17-3 3.5-2 25.9 7.92 .0313

55 10.5 14.9 7.2 6-34 .0036

S6 13-4 6.6 13.7 13.71 .0001

S7 3.3 1-7 1.2 3-68 .0305

S9 10.3 6.4 7-5 4.49 .0152

F18 255.5 230.9 264-6 5.06 .0096

P22 50.9 45.2 55.0 3.57 -0335

02 4.8 2.9 6.4 4.04 .0272

11 0.6 0.5 0.5 33.62 .0000

72 0.5 0.7 0.5 76.61 .0000

13 0.5 0.4 0.7 29-07 .3000

17 16.8 32.2 18-4 28.32 .0000

18 17-3. 15-2 25.9 7.92 0013

T9 10.5 14.9 7.2 6.34 .0036

110 13.4 6.6 13.7 13.71 .0001

111 3.3 1.2 1.2 3.68 .0305

713 10.3 4.4 7.5 4.49 .0152

a

*Refer to Table 1 for description of keys.

27 .

26

Eowever gains in CASES Setting 3 shvwed significant differences for

Style E (in favor of CroLp 2) and fcr Style B (in favor of Croup 1).

A difficulty in the separation of groups and subsequent examination

of differences in student outcomes may be illustrated by the factor

lnadings on seven rotated factors from the analysis of the standardized

matrix_ Each of the seven factors had an approzi=ately equal number of

:Positive and negative high loadings. An identification of the names

of the teachers with such loadings revelled groups who appeared (to

CAC eu,ervers) LO lie ar cppesite cads of a continuum of behavior.

Thus, each factor, or group, is in reality bi-polar and within group

differences =ay counteract differences between groups.

SUMMARY AND CONCLUSIONS

The results of preliminary analyses suggest that systematic

classroom observation using low-inference measures produces inter-

pretable profiles of teacher behavior. Clear-cut relationships

between teacher competencies and student outcomes have not yet been

established; however, the data suggest that finer distinctions must be

made among the groups before the competency-outcome relations may be

uncc4ered.

The CBTC project has developed a prototype for the measurement of

competencies as veil as for the reduction of data related to teacher

types. The observation instruments used in the study are worthy of

retention. Each appears to identify unique characteristics of teachers,

while at the same time complementing and supporting the others.

Further investigation of the sane kind is needed using larger

teacher groups at fewer grade levels. The reduction of N to 1 or 2

'1

27

groups for sore studert measures =a& meznil-,gful and valid analyses of

s0=e of tbe student data

Finally, it is apparent from the prelinieary analyses that the

cost inportant relationships are probably non-linear. The task of

discovering which of the nany possible interactions anong the teacher

variables contribute to differences in student growth will be cost

difficult.

It is clear that the road to an enpiricallyfounded competency

based certification system will be a long one, filled with oJstacles and

However, the goal is one which can be reached and the CBTC

project has taken the first steps on that road.

25

LLZBERENCES

Moc.m, B. S. "Twonty-five Tear of Educational Research," AmericanEducational research J*,urnal 1966, 3, 211 -218.

:-Brown, B. B. "Experimentalism in Teachinz Practice," Journal of

Fe:;earch and Ite-,7eIcement in Education, 1970, 4, 14-22.

Carroll County Cumpetency Based Teacher Certification Project.Librarian's Derort. Mimeographed, 1974.

Cx-zirelensive Tests of Basic Skills. Monterey, California:McGraw-Hill, 1974.

Dizz,n, W. J. (Ed.) Biomedical Computer Programs (3rd Ed.). Berkeley:

University of Califo-nia Press, 1974.

Frynier, J. R. "Developmvnt and Validation of a Motivation Index,"

Theory into Practice, 1970, 9 (1), 56-88-

Cordon, I. J. A Test Manual for the How I See .Myself Scale.

Gainesville: Florida Educational Research and Developnent

Council, 1968.

Guertin, V. H. 4 Bailey, J. P., Jr. Introduction to Modern Factor

Analysis. Ann Arbor: Edwards Brothers, Inc_, 1970.

Hierany=s, A. N., 6 Lindquist, E. F. Iowa Tests of Basic Skills.

Boston: Houghton Mifflin Company, 1971.

Hieronymus, A. N., Lindquist, E. F., S Others. Iowa Tests of Basic

Skills. Boston: Houghton Mifflin Company, 1972.

McKinney, J. D., Mason, J., Perkerson, K., 6 Clifford M."Relationship Between Classroom Behavior and Academic Achieve-ment." Journal of Educational Psychology, 1975, 67 (2),

198-203.

Medley, D. M. observation Schedule and Record, Form 5, Verbal.Mimeographed, 1973.

Monroe, M., Manning, J. C., Wepnan, J. M., & Gibb, G. E. The Scott,

Foresran Initial Sunlej Test. Glenview, Illinois: Scott,

Foresman and Company, 1972.

Soar, R. S. Final Report -- Follow Through Classroom Process easure-ment and Pupil Growth, (1970-71). Gainesville: Institute For

Development of Human Resources, University of Florida, 1973.

g

-0,61 'sxcallny -1-Es4-6:1diFs;

I5To aW=.4or. i.mzua2 g .-a -J

-6961 '2a4scif tvIn

";i:Siaamy diiivg3 !epos -E 'sfloE 'za%4A-,4 'TS g20=CM

-ME 4-3111 4CO3S2tn Par '310H =Val M4: -1:aiiraps-iczosaccE4E arif.xag Ur.4amstaarlz csaui 2ess wr -a 'ae=2,1o.A

-ufir grmwsw co2q2coli =own p1.6i,,r.w Jo

zoctirly :450Z" ues (2damle4ACpac stsxietrq aatAmAi m...gtxxasrria -E daL;pincds

-Tz6i 'eplaou ;o Izysxoafaa 'saxxrci,E cr=q1. ;o smacuE4.---.7z

zo; ounrmuil :oEflas:au;c2 _1 c4;4 s.za;-xao-5::tj

Yo2ImA3 purr au-Igo cppd4A 'elsa/tE 9 -H zems -E 'lens

6Z