2007/1/9 CSL Lab@ ICE.NTNU 1

Instructor: Ming-puu ChenPresenter : Ching-ting Lin

Considerations for Developing Considerations for Developing Evaluations of Online CoursesEvaluations of Online CoursesAchtemeier, S. D., Morris L. V., & Finnegan C. L. (2003). Considerations for developing evaluations of online courses. Journal of Asynchronous Learning Networks, 7(1), 1-13.

2007/1/9 CSL LabICE.NTNU 2

Contents• Introduction• Literature Review

– Seven Principle for Good Practice– Assessment Essentials– Guiding Questions for Planning Assessment– Applying the Seven Principles to the Evaluation

• Student-faculty contact• Cooperation among students• Active learning• Online Task

• Research Questions and Design• Conclusion

2007/1/9 CSL LabICE.NTNU 3

Introduction• These questions lead the researchers

– Investigate the definitions and principles of effective teaching and learning

– Perform a content analysis of instruments

– To develop considerations for design of evaluation instruments

2007/1/9 CSL LabICE.NTNU 4

Literature Review• Described “good” assessment• Summarized course evaluation

broadly• Identified principles specifically for

evaluation of online instruction

2007/1/9 CSL LabICE.NTNU 5

Seven Principles for Good Practice

• In undergraduate education• Supported by the American Association of Higher Education (AAHE)• Good practice in teaching and learning must do the following:

– Encourage student-faculty contact 師生互動– Encourage cooperation among students 合作– Encourage active learning 主動學習– Give prompt feedback 立即回饋– Emphasize time on task 完成任務時間– Communicate high expectations 高度期待– Respect diverse talents and ways of learning 因材施教

2007/1/9 CSL LabICE.NTNU 6

Assessment Essentials• Palomba and Banta address qualities that should lay the foundation for any assessment effort

– Assessment should be preceded by explicitly stated outcomes.– Assessment should distinguish between formative and summative uses.– Assessment should have strong faculty buy-in.– Multiple methods should be used.– Assessment results should be shared and used.– The assessment itself should be assessed.

2007/1/9 CSL LabICE.NTNU 7

Guiding Questions for Planning Assessment

• Joseph B. Cuseo in Assessment of the First-Year Experience– Why is the assessment undertaken?

• Is the focus student or faculty experience? Is it the design of the course? Is it the delivery?– What will change?– What outcomes are being assessed? At what level?– When should the assessment take place?– Where and how should the assessment take place?– Who should be involved?

2007/1/9 CSL LabICE.NTNU 8

Applying the Seven Principles to the Evaluation

• Applying the Seven Principles to the Evaluation of Web Based Distance Education discovered that key strengths in– encouraging active learning, student-

faculty contact, and diverse ways of learning

– But needed improvement in encouraging cooperation among students and giving prompt feedback

2007/1/9 CSL LabICE.NTNU 9

Student-faculty contact• Three web-specific requirements of

the faculty– Faculty need skill with asynchronous

conferencing tools– Faculty clearly and adequately

communicate their email response policy to students

– Burden on faculty to detect and contact students who are falling behind

2007/1/9 CSL LabICE.NTNU 10

Cooperation among students• Faculty should begin with

– structured activities that facilitate community among students

– Develop assignments that require meaningful peer interaction

2007/1/9 CSL LabICE.NTNU 11

Active learning• Faculty should ask students to present their work electronically• Prompt feedback

– Faculty should give immediate acknowledgement feedback 立即的確認– Prompt information feedback regarding the content of the student’s work

2007/1/9 CSL LabICE.NTNU 12

Online Task• To encourage students’ time on task

– Faculty should give structured assignment deadlines– Assignments should require resources that are easily accessible to the online students

• To communicate high expectation– Faculty provide examples to students of exemplary online performance 網路行為的模範

2007/1/9 CSL LabICE.NTNU 13

Research Questions and Design

• To examine the above recommendations for teaching and learning online, thirteen evaluation instruments were received

• Answers were sought to the following research questions:– To what degree do actual evaluation instruments try to

assess whether the Seven Principles for Good Practice in Undergraduate Education are taking place?

– To what degree are other principles of effective teaching identified in the literature being evaluated by these instruments?

– What other issues are being addressed regarding the evaluation of courses?

– What other issues are being addressed regarding the evaluation of faculty?

2007/1/9 CSL LabICE.NTNU 14

Research Results• Table 1. Numbers of institutions whose evaluation

instruments assess for the Seven Principles of Good Practice in Undergraduate Education.

• Table 2. Numbers of institutions whose evaluation instruments assess for the questions suggested in Principles of Effective Teaching in the Online Classroom.

• Table 3. Numbers of institutions whose evaluation instruments assess these additional questions regarding courses

• Table 4. Numbers of institutions whose evaluation instruments assess these additional questions regarding instructors

2007/1/9 CSL LabICE.NTNU 15

Table 1. Numbers of institutions whose evaluation instruments assess for the

Seven Principles of Good Practice in Undergraduate Education.

2007/1/9 CSL LabICE.NTNU 16

Table 2. Questions• Following a review of the Principles of Effective Teaching in the Online

Classroom1. Were the course goals, learning objectives and outcomes made

clear to you at the beginning of the course?2. Did you have the necessary technological equipment and skills

required for this course?3. Was there adequate technical support if you encountered difficulties?4. Was the format and page design of the online course easy to use?5. Were there sufficient instructions given for you to complete all

assignments?6. Did you feel hindered in your online course experience any way?

Please describe.7. Were standards for evaluation of assignments made clear?8. Did you receive prompt feedback on your completed assignments?9. Did you participate in online conversations with your instructor

during the course?10. Did you participate in online conversations with your classmates

during the course?11. What learning activities most influenced your learning in this course?

2007/1/9 CSL LabICE.NTNU 17

Table 2. Numbers of institutions whose evaluation instruments assess for the

questions suggested in Principles of Effective Teaching in the Online Classroom.

Important Questions to Ask

Online (4)

2-year (2)

4-year (3)

Master’s (2)

Doctoral (1)

Research (1)

Percent (of 13)

#1 Course goals clearly articulated

2 2 1 1 1 1 62%

#2 Student has skills & equipment necessary

2 1 23%

#4 Format & page design easy to use

3 1 1 38%

#5 Sufficient instructions given for all assignments

1 8%

#11 Satisfaction with learning activities

4 1 2 1 1 1 77%

2007/1/9 CSL LabICE.NTNU 18

Table 3. Numbers of institutions whose evaluation instruments assess these

additional questions regarding courses

2007/1/9 CSL LabICE.NTNU 19

Table 4. Numbers of institutions whose evaluation instruments assess these

additional questions regarding instructors

Other Questions Asked Regarding the Faculty

Online (4)

2-year (2)

4-year (3)

Master’s (2)

Doctoral (1)

Research (1)

Percent (of 13)

1. Knowledge of subject

3 2 3 2 77%

2. Availability for help

4 2 3 2 85%

3. Good job answering questions

2 2 1 2 1 62%

4. Enthusiasm 1 2 1 2 1 54%5. Overall rating of the faculty

2 2 1 1 46%

2007/1/9 CSL LabICE.NTNU 20

Conclusions• Seven principles + suggests eleven other

questions pertinent– Only 8 of the 18 principles were assessed by

those evaluation instruments– Notably missing were questions about

cooperation among students and active learning

– While one instrument asked about prompt feedback in general, none asked about prompt feedback on completed assignments

2007/1/9 CSL LabICE.NTNU 21

END Any advice is appreciated.