1

The Value of NCATE

A Study of Research Institutions and NCATE AccreditationAACTE, Friday, February 8, 2008

Emerson J. ElliottNCATE, February 2008

2

Initial report from a study

Research institutions that are accredited by NCATE

Interviews of deans, NCATE coordinators and faculty

Perceptions of NCATE accreditation

3

Topics in this sessionA. Setting for the study

B. Who participated

C. Interview questions

D. Response themes

E. Summing up

4

A. Setting for the study AACTE Fall 2004 study of deans CADREI fall 2006, and AACTE committee

Streamline Reduce burden and cost Address time commitment required of BOE and UAB

members Dovetail program review and unit review Make more collaborative, less punitive—more like some

other specialized accreditors

Brought to the NCATE Executive Board

A. Continued, Setting Executive Board also set goals for NCATE’s

management plan (Fall 2006-Spring 2007) Reduce the burden of accreditation Improve service to institutions Increase the value and perceived value of the

accreditation process to institutions

And NCATE was concerned that: Research universities are the source of new knowledge in

educator preparation. and for NCATE unit and specialized professional standards. . .

5

6

A. Continued, Setting But faculty from research universities are

infrequently involved with NCATE Research university faculty are often not among the

participants in standards writing And—prior to a change in 2007—research university

faculty had participated rarely as examiners or on NCATE’s policy boards

[3:15 session at AACTE today] Arranged meeting with research university deans,

January 2007

7

A. Continued, Setting: NCATE meeting with research universities Outcomes

Symbiotic relationship between research on teaching and learning produced by research universities and use of that research to improve educator preparation

Continue discussions with CADREI and AACTE Urge more BOE and board candidates from research institutions Streamlining to ensure an efficient and effective system

As one way to support these outcomes, and to ground the “value” goal in the management plan, NCATE decided to ask the research institutions themselves what they think about NCATE accreditation.

8

B. Who participated?

Five research institutions accredited by NCATE Five interviews in each institution

Dean Coordinator Three faculty designated by the dean

4 associate deans 4 department chairs 6 program directors and/or SPA coordinators 1 assessment coordinator and doctoral student

9

B. continued, Who participated: Interviews

All conducted by telephone All interviewees gave explicit permission to

record the interview All interviews followed the same interview

questions All interviewees were promised anonymity

All transcribed

10

B. Continued, Who participated: Selected institutions All in the East of the Mississippi Three are private not-for-profit, two are public Three participated in the NCATE program review process,

two did not All five are doctorate granting, four are Carnegie “high” R &

D ($10 to $150 million in 2005), one is Carnegie “very high” (over $250 million in 2005)

Two have around $20 million in social sciences R & D in 2005

Enrollments for 2004 range from around 12,000 to more than 42,000

B. Continued, Who participated: What is not reflected in the participating institutions?

Institutions that NCATE does not accredit Institutions in the West “All” NCATE accredited research institutions—

selected from two recent cohorts of UAB action Sample is small But:

Still found a range of differences across institutions and individuals

11

12

B. Continued,Who participated: Accreditation experience All had accreditation visits in 2004-2006

One was having an initial visit Four were continuing visits

Two had all standards met Two required focused visits related to

standards 1 and 2 and are now fully accredited

One has a coming focused visit related to standards 1 and 2

B. Continued, Who participated: Time warp

YEAR2001 2002 2003 2004 2005 2006 2007

Semester S F S F S F S F S F S F S F

UNIT New Stds1st phase,Transition

2nd phase 3rd phase Full Full Full Full

PROGRAM

Assessment, syllabi, and program evidence New 1st

cohort

2nd

cohort3rd

cohort4th

cohort5th

cohort6th

cohort7th

cohort

Prog Rpts submitted

PR submit

PR submit

PR submit

PR submit

VISIT V V V V

UAB action

UAB UAB UAB UAB

Interviews

I

13

14

C. Interview Questions Overall topics were:

Value that respondents associate with NCATE accreditation

How NCATE does its job Advice about how accreditation could be improved

Findings from 3 questions and then 6 themes—topics that recurred in the interview responses

15

C. Continued, questions: (1) Overall reactions 72% (18 respondents) “positive”

Positive, very positive, useful learning experience, important to do

Includes 3 deans

28% (7 respondents) “negative” Technical and prescriptive requirements Time and resource intensive Too much assessment, compliance, SPA instructions

changed Includes 2 deans—different experiences

16

C. Continued, questions: (1) Overall reactions Additional observations

Many at our institution think we don’t need national accreditation—we have really high standards

The process allowed for considerable flexibility in applying standards

The team were true professionals The self-study aspect is useful

17

C. Continued, questions: (2) Most helpful

MOST helpful Deans—reflection, self study, faculty

collaboration Coordinators—assessment Faculty—fitting assessments to standards,

collegial activities among the faculty

18

C. Continued, questions: (3) Least helpful

LEAST helpful Deans—time, cost, burden, team visit Coordinators—mixed: Praxis, changing rules,

duplicate NCATE and state requirements, lack of research institution team members

Faculty Assessments—confusion, rubrics for standards, changing

requirements BOE team—composition, logistics, use of web

information

D. Response “themes” Frequently recurring topics that were not

explicit interview questions Research institution self descriptions BOE teams Program review and SPAs Defining evidence for NCATE TEAC references States—contrasting perspectives

19

D. Continued, themes: (1) Research institutions 19 comments, 12 respondents, all 5 institutions; 24 points

Character of the institutions Accreditation is not the way they think Candidates arrive at the graduate level and have completed subject content

courses Specialization Emphasis on scholarship and research Always under study

Professional life of the faculty Own projects and funding NCATE takes time from grant writing Faculty change courses all the time, and their assessments Little motivation to volunteer for NCATE work

BOE teams often don’t understand their qualities20

D. Continued, themes: (2) BOE teams 48 references (largest number), 18 respondents in 5

institutions. Distinction between one institution that characterized its visit

as “disastrous” and all other

21

Difficult visit All other institutions

Positive comments 0% 43%

Negative comments 80% 21%

Other comments 0 11%

Recommendations 20% 25%

Total 100% 100%

D. Continued, themes: (2) BOE teams Positive statements about BOE teams--12

The team—true professionals, collegial, business-like, outstanding team, chair from a large research university, respected what we do, team understood who we are, all in this together, good team, worked hard

22

D. Continued, themes: (2) BOE teams

Negative statements about BOE teams The team in “all other” institutions—6

lack of preparation, did not use web-based exhibits, lack of research university peers, overemphasis on logistics

The team in the “difficult visit” institution—16 failed to ask for information, did not listen, did not use provided

documentation, tense exit interview, members not prepared, did not make good use of the Sunday evening poster session

23

24

D. Continued, themes: (2) BOE teams

Recommendations—11 of the 48 comments Chair/members from research institutions (8) Train members about research institutions Assure that teams arrive better prepared

To use electronic data For more collegial interaction To use time not just to “find data, but to clarify and

elaborate”

D. Continued, themes: (3) Program Review and SPAs 22 references from 10 respondents in all 5

institutions; total of 51 points made Inconsistencies and changes—20

Across SPAs and between SPAs and NCATE—some would accept GPA or Praxis data, others would not

Changes during the accreditation process Certain SPAs are difficult to work with Lack of fit between the standards and graduate level

initial preparation

25

D. Continued, themes: (3) Program review and SPAs Concurrences with the program review process—14

More consistent use of assessments Agreement on just 6 to 8 assessments Writing for national recognition helped to redirect

programs Faculty worked hard “because there’s this element of

pride” Our program’s gotten better Comments by reviewers were fair, right on the money

26

D. Continued, themes: (3) Program review and SPAs Complaints—9

Praxis data not aligned; can’t get sub scores Limited feedback Took too long Just wanted numbers

Others—2 Respondent’s own institution made changes Faculty would rather fight with their professional

organization than adopt standards

27

D. Continued, themes: (3) Program review and SPAs; Recommendations—6 More consistency

Equivalent demands for all SPAs More coordination across SPAs and between SPAs and NCATE Limit the number of assessments

Better fit with graduate level preparation Write standards for initial preparation that recognize prior candidate

preparation in subject content Provide more examples of evidence for “other” professional

preparation

Reconsider way decisions are made about SPA standards Need more “we’re in this together,” less territoriality, more at-large

partners28

D. Continued, themes: (4) Evidence 24 references by 10 respondents from 4 institutions 18 comments on evidence generally

NCATE is too prescriptive too much reliance on “numbers and tables” Not a research base to back up some data requirements Institutions use grades as evidence of accomplishment Assessment data cannot be aggregated meaningfully

across different levels of programs 6 comments on evidence for diversity

Numbers fail to inform the goal of “cultural competence”29

D. Continued, themes: (4) Evidence; Recommendations Use a broader definition, not just “numbers on tables” Let institutions decide

More anthropological—look at what we’re doing and how faculty and our publics interpret that

Let institutions self-define their mission, and ask for reporting of evidence within broad parameters—limiting the number of assessments

Address particular issues Take on measuring of student learning based on what good teaching

and learning are Focus on ethnic diversity and “cultural competence”

30

D. Continued, themes: (5) TEAC 10 references made by 6 respondents from 4 institutions; total

of 13 points 7 points were set in a context of perceived problems with

NCATE NCATE is overly prescriptive Some required pieces of data are not supported by research Differences between SPAs and NCATE frustrate faculty NCATE is “bean counting” NCATE is burdensome, tedious, time intensive, costly Important qualities of a program cannot be captured in numbers Standards don’t align with the way the state is going

31

D. Continued, themes: (5) TEAC Three respondents ventured impressions of TEAC or

recognized it as an alternative There were 3 recommendations

NCATE should look at what it does “from a business perspective”, but leave room for compromise

TEAC and NCATE representatives should “work out a whole new accreditation system. . . Good aspects of both, but neither is a perfect system”

BUT don’t go “so far in another direction that we don’t have a set of standards that people have to step up to meet”

32

D. Continued, themes: (6) States 13 references by 6 respondents from just 2 institutions Starkly different portraits of states, so state influence is a

strong factor in accreditation, at least for some institutions NCATE parallels the state, should keep coordinated, make state exam

pass rate count for NCATE, can defer to our state on state assessments, would not have completed NCATE accreditation if state had not insisted

Negativity about NCATE is really directed at the state; state report is different only in minor ways—a duplicate, unnecessary, requirement; disparity in the standards that “regular” and “alternative” programs are held to but state refuses to acknowledge

33

E. Summing up Respondents found much that was good, e.g.:

I like what I’m hearing about web-based submissions Electronic submission is very helpful to organize data,

other information, Positive that NCATE is interested in what approved

programs are saying and thinking Preparing for NCATE made the regional accreditation

easy

34

E. Summing up They made recommendations that NCATE needs to

sift and consider. Particularly, Evidence Teams and recruitment Program standards and standards decisions Regional accreditor experiences Continuing accreditation, candidate data only

35