the quality transformation faculty development assessment framework (215733086)

8/12/2019 The Quality Transformation Faculty Development Assessment Framework (215733086)

http://slidepdf.com/reader/full/the-quality-transformation-faculty-development-assessment-framework-215733086 1/16

The Quality Transformation

Faculty Development

Assessment Framework

Larry Ragan, Director--Center for Innovation andOnline Learning, [email protected]

Dedicated to

Carol McQuiggan, Manager, Faculty Center for Teaching & Instructional

Technology, [email protected]

mailto:[email protected]






Agenda

1. Background of faculty development initiatives

2. Review of Quality Transformation Faculty

Development Assessment Framework and

related metrics

3. Discussion of adjustments/improvements to

Framework



Start with WHY?

1. Relative “newness” of faculty

development/professional programs

2. Increased number of “faculty dev” programs

3. Historical lack of accountability regarding

evidence of impact

4. Increased demands on all aspects of online

learning regarding measurements of impact5. Difficulty in creating direct linkage of fac dev

services to student gains



Four integral sources

1. Kirkpatrick’s (1994) 4 steps of evaluation

2. Guskey’s (2002) 5 critical levels of information

3. Plank & Kalish’s (2010) guidance for working

through preassessment planning

4. Hines’s (2009) dimensions of quality program

assessment



Quality Transformation Faculty

Development Assessment Framework

1. Combined assessment techniques from the 4

sources

2. Need thorough understanding of expectationsfor/of faculty development initiative

3. Assessment strategies need to aligned with

institutional goals and values



Quality Transformation Faculty

Development Assessment Framework

4. Prioritize/employ those methods/approaches that

best match needs

5. Use of single relational database (single point ofdata collection)

6. You have to begin somewhere, even if it’s

imperfect!



WHATTECHNIQUES/STRATEGIES/M

ETHODS DO YOU USE TOASSESS FACULTY

DEVELOPMENT IMPACT?

Question:



Step 1: Participation Metrics

Sample outcomes Metrics

Faculty participation Number of program participants

Percentage of faculty participation Number of available faculty

participants vs. actual number ofparticipants

Reengagement Number of repeat attendees

Course completion Number/percentage of participants

who complete course

Participation by specific discipline,

school, department, or full-time or

part-time faculty

Number/percentage of participants

broken down by variables



Step 2: Faculty Input and Satisfaction

Metrics

Sample inquiries Metrics

Was the professional development program

offered at a convenient time?

Scaled response rate (very inconvenient to

very convenient).

Qualitative input: please indicate why this

time was inconvenient for you.

Was the duration (number of days orweeks) of the professional development

program reasonable for you?

Scaled response rate (very unreasonable tovery reasonable).

Qualitative input: please indicate why this

duration was unreasonable for you.

Which topics offered in the professional

development program were most beneficial

for you?

Provide checklist that enable faculty to

choose topics from program offerings.

What additional topics were not offered in

the professional development program that

you would have found beneficial?

Provide checklist of other topics (not

currently included) for faculty to choose

from.

What questions remain? Provide space for open-ended response.



Step 3: Attitude and Knowledge Metrics

Sample techniques Metrics

Knowledge quizzes and exams

embedded in faculty development

program

Standard grading scales and feedback

Postprogram survey instruments

(phone interviews, focus groups,

online surveys) examining attitudes

toward learning environment

Feedback regarding online instructor’s

preparedness (identify areas requiring

additional information or skill

development)

Collection of feedback from student

complaint and resolution services

Number of complaints; nature of

conflicts and resolutions

Midcourse and end-of-course student

survey on online instructor’s

preparedness and teaching practices

Student satisfaction and evaluation of

instructor preparedness for teaching

online



Step 4: Instructor Performance Metrics


Postprogram survey instruments

(phone interviews, focus groups,

online surveys) examining teaching

behaviors, conducted eitherimmediately following a program or

over a longer period

Description of teaching behaviors

and source of new skill (creating a

linkage to faculty preparation

program)

Midcourse and end-of-course student

survey on online instructor’s

preparedness and teaching practices

Student satisfaction and evaluation of

instructor preparedness for teaching

online

Collection of feedback on instructor

teaching behaviors

Evidence of the application of

specific instructional methods and

techniques by online instructor



Step 5: Student Performance


Examination of student retention

within program of study, student

progress toward degree, years to

program completion

Data gathered from student tracking

and record systems

Calculation of number of students

achieving passing grade

Program or institutionally defined

End-of-course (or maybe longer-

term) student survey regardingonline instructor’s preparedness

and teaching practices

Student report and evaluation of

instructor’s impact on coursesuccess



Final Thoughts

• Match desired outcomes of initiative tostrategies and metrics used to ascertain

effectiveness, efficacy, and impact

• Analyze multiple measures to increase

reliability

• Begin with clear understanding of program

goals and overlay selected techniques

• Use direct and indirect measures

• Operate within culture and context of institution

• Include program developers and faculty



Chapter/Book Information

• Chapter 7 in upcoming book published byStylus and edited by Kay Shattuck

Assuring Quality in Online Education:Practices and Processes at Teaching,

Resource, and Program Levels

• Publisher link http://bit.ly/AssuringQuality

http://bit.ly/AssuringQuality






Questions/Discussion

the quality transformation faculty development assessment framework (215733086)

Documents