FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 1
Formative and Summative Evaluation Reports
Matt DeLaire, Ann-Marie Peirano, and Jennifer Watson
The University of Alabama
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 2
Formative Evaluation Report
Purpose
A previously conducted needs assessment report revealed that effective communication
in online undergraduate courses is a significant factor in the positive retention of online students.
Using this data, an instructional design plan was developed in order to provide guidance to
online instructors on how to successfully incorporate communication into the online courses they
teach. After the completion of the instructional design plan, this formative evaluation report was
undertaken in order to assess the effectiveness and quality of the instructional design plan. The
evaluation specifically focused upon the following issues: understandability of stated objectives,
appropriateness of the instruction for the target audience, alignment of performance objectives
with content activities, thoroughness of the instruction, adequate opportunities to practice usage
of the content, and adequate feedback on products (Rothwell & Kazanas, 1998). This
information will be used to make improvements and any necessary changes to the content of the
instructional design plan before the plan is finalized and potentially put into action (Morrison,
Ross, Kalman, & Kemp, 2011). The audience for this formative evaluation report is the
instructional design team who created the initial plan; they will undertake plan revisions to the
plan as required.
Methods
Data for the formative evaluation was gathered by administering a survey (see Appendix
A) to three members of the target population who could potentially receive the planned
instruction but who are also experienced online instructors. These three subject matter experts
(SMEs) each currently teach at least one online undergraduate course, and all three have been
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 3
teaching online courses for four academic years. After the completion of the initial instructional
design plan, the three survey participants were emailed the “Formative Evaluation Instrument”
(Appendix A) and one complete learning episode, termed module three (see Appendix B), from
the six episode instructional design plan. Module three was chosen for evaluation because it was
representative of the larger instructional design plan. Participants were asked to review the
learning episode and electronically complete and return the evaluation survey within three days,
and all three surveys were returned within this time frame. The data collected was analyzed
qualitatively with an explanation of the researchers’ interpretation of extant data.
Analysis
The formative evaluation instrument (Appendix A) consisted of 11 questions dealing
with the effectiveness and quality of the planned instruction. Each question allowed the SME to
answer “yes,” “no,” or “not applicable,” and for seven of the 11 questions, all of the SMEs gave
an answer of “yes” (Appendices C, D, and E). An answer of “yes” indicated that the content
pertaining to the question did not need to be revised in the opinion of the SME. The survey also
provided an area for comments for each question, and these comments provided the most
valuable feedback on the learning episode under review and the larger instructional design plan.
On the evaluation instrument, all of the SMEs reported that the performance objectives
were clearly stated and that the instructional materials support those objectives. The three SMEs
also answered “yes” when asked if the content of the instruction was complete and up to date,
and if the learners were given adequate feedback on real-world products created during the
training. In addition, all three survey respondents stated that the instructional methods were
appropriate for the learners and the instructional methods support the objectives. The four
remaining questions on the evaluation instrument received an answer of “no” or “not applicable”
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 4
from one or more of the SMEs.
All of the SMEs answered negatively or felt the question was not applicable when asked
about the alignment of instructional materials to learner characteristics in the learning episode. In
addition, two of the SMEs commented that they did not answer “yes” simply because they did
not know the specific learners who would be completing the instruction. The instructional design
team concluded that more information about the learners should have been provided to the SMEs
before administering the formative evaluation survey, but that the negative SME responses in
this area do not warrant adjusting the instructional deign plan. The other questions that received
negative responses dealt with giving learners adequate opportunities to receive information about
the content, and giving learners adequate opportunities to practice or apply what they have
learned. On the evaluation instrument, all three of the SMEs answered “no” to one or both of
these areas. Comments from the survey participants indicated that more time was needed than
was allotted in the learning episode to effectively present initial information. Likewise, SMEs
commented that more time was needed for participants to produce real world products, such as
revising a lesson plan, than was allotted in module three.
Conclusions
Due to the positive SME responses concerning the performance objectives and how the
instructional methods support those objectives, the instructional design team is confident that the
objectives themselves need no further revision. In addition, comments from the SMEs indicating
that the content of the instructional design plan was valid and that participants would receive
adequate feedback indicated to the instructional design team that the plan does not need to be
adjusted in these areas. However, SME concerns about the time allotment for the presentation of
instruction and the production of work products signals to the design team that modifications to
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 5
the plan must made in the area of timing. Therefore, in the instructional design plan, revisions
have been made to extend the time for the presentation of stimulus material and learner guidance
by 30 minutes. In addition, during instances in which learners are being asked to elicit a
performance, such as revising a course outline or lesson plan, the time given to complete these
tasks has been extended by 45 minutes.
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 6
Summative Evaluation Plan
Purpose/Goals
The summative assessment will utilize the Kirkpatrick four-level model to evaluate the
effectiveness of the coursework provided. This plan will outline the instruments that will be
used after the implementation of the training that is focused on improving communication in
online learning in order to increase retention. The course objectives that will be evaluated in the
Kirkpatrick model are as follows:
Instructors will understand and describe the basic principles of communication for online
coursework.
o Type of communication
o Importance of communication
o Synchronous verses asynchronous
Instructors will analyze and evaluate best practices in online communication and apply these to
real world products.
o Improved student satisfaction through communication
o Establishing reasons to revisit the course site
o Communication focused lesson plans
Instructors will prepare students in their courses for online learning.
o Accessibility
o Clear expectations and technical guidelines
The goal of the summative evaluation plan is to benchmark the following:
Satisfaction of the participants
Learning take-aways
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 7
Behavioral changes in course design
Positive effects on the retention rates of online students resulting from the training
Methods
Before beginning the online instruction modules, students will complete a pre-test to
determine their baseline level of knowledge related to the topic of communication in the online
medium (see Appendix G). At the conclusion of the entire online instruction modules, students
will once again take the described pre-test as a post-test to assess the impact of instruction as
indicated in level two of Kirkpatrick’s four levels of evaluationAlso, to address the level one
issue of student reactions, after completing the course, students will complete a course evaluation
(see Appendix F) that allows them the opportunity to rate their experience in the course and also
the effectiveness of the content and delivery medium.
The level three summative evaluation tool (see Appendix H) is designed to be given to senior
faculty in supervisorial roles. This evaluation utilizes a Likert scale and points will be assigned
to the scale and an overall rating for each instructor’s course will be evaluated. Scoring on the
Likert scale is as follows:
Strongly disagree = -2
Somewhat disagree = -1
Neutral = 0
Somewhat agree = 1
Strongly agree = 2
Scores greater than zero will be viewed as favorable and the possible correlation between
lower scores and retention rates will be investigated further. Additional communication
enhancing instruction will be recommended for lower scoring courses. The level four summative
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 8
evaluation tool (see Appendix I) included in our plan is a trend analysis of retention rates before
and after the introduction of our communication training. Data gathered over time will likely
draw a correlation between retention rates and our training course. This will provide
management with supportive data to implement future similar programs or expansions to the
existing program.
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 9
References
Morrsison, G. R., Ross, S.M., Kalman, H.K., & Kemp, J.E. (2011). Designing effective
instruction. New Jersey: John Wiley & Sons, Inc.
Rothwell, W.J., & Kazanas, H.C. (1998). Mastering the instructional design process. San
Francisco: Josey-Bass/Pfeiffer.
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 10
Appendix A
Formative Evaluation Instrument
Answer each question by placing an “X” next to yes, no, or not applicable. Please provide any additional comments that you feel will enhance the effectiveness of the instruction.
Question Yes No Not Applicable
Comments
1. Do instructional materials clearly state the performance objectives expected?
2. Do the instructional materials appear to match the learner characteristics?
3. Do the instructional materials support the performance objectives?
4. Is the content of the instruction complete?
5. Is the content of the instruction up to date?
6. Are learners given adequate opportunities to receive information about the content?
7. Are learners given adequate opportunities to practice or apply what they learn?
8. Are learners given adequate opportunities to receive feedback on how well they applied what they learned?
9. Are the instructional methods appropriate for the targeted learners?
10. Do the instructional methods support the performance objectives?
11. Are there other issues that you noticed that could improve the instruction? If yes please describe in the comments box.
Source: Rothwell, W.J., & Kazanas, H.C. (1998). Mastering the instructional design process. San Francisco: Josey-Bass/Pfeiffer.
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 11
Appendix B
The document below represents the instructional design planning sheet for learning episode three (module 3) from the instructional design plan. The performance objectives are included at the end of this document.
Learning Episode #3 (Module 3)
Terminal Objective 3 - Instructors will analyze and evaluate best practices in online communication and apply these practices to real world products.
Design Sequence Description Objective(s) Time Instructional
Strategy
Media
(Technology, Worksheets, etc.)
Gain Attention Pre-instructional strategy
3.1
3.2
3.3
3.4
3 minute Participants will view a Camtasia screencast that will display and orally present statistics and literature findings concerning the correlation between effective communication in online courses and the retention of online students. This information is designed to enlighten participants on the importance of involvement and communication in online courses. The following information will be displayed:
In a survey of 1,406 higher education students, Drouin (2008) discovered that students reported higher levels of satisfaction and sense of community membership when there was increased communication with the instructor and with other students enrolled in the course.
These findings were supported by Harrell (2008) who found that one of the key factors causing online graduate students to drop out was a sense of isolation.
Camtasia screencast
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 12
Inform Learners of Objectives
Pre-instructional strategy
3.1
3.2
3.3
3.4
5 minutes Using the Camtasia screencast and accompanying PDF handout participants will be provided with a written and oral summary of the importance of self-reflection upon the level of satisfaction that their own online students have experienced. Also, an overview of the process for evaluating existing lesson plans in order to incorporate opportunities for enhanced communication will be provided.
Camtasia screencast, PDF attachment/handout
Stimulate Recall of Prior Learning
Review Terminal objective 1
5 minutes The Camtasia screencast and accompanying PDF handout will continue and will briefly review the following concepts from previous modules: basics in communication for online coursework, advantages in communication for online courses and its importance, types of communication common in online courses, asynchronous communication, and synchronous communication.
Camtasia screencast, PDF attachment/handout
Present Stimulus Material
Recall 3.1 5 minutes The Camtasia screencast will continue and will visually and orally explain to participants Herbert’s (2006) survey concerning online student satisfaction and retention.
Camtasia screencast
Provide Learner Guidance
Recall 3.1 N/A The Camtasia screencast explaining Herbert’s (2006) survey will be accompanied by a PDF file that provides participants with a copy of the survey that can be downloaded or printed.
PDF attachment/handout
Elicit Performance
Elaboration 3.2 10 minutes Participants will take Herbert’s online student satisfaction survey from the point of view that they are a student in an online course that they currently teach. Participants will be asked to answer each question as honestly as possible. The survey will be provided in the module as a PDF file download.
PDF attachment/handout
Provide Feedback Integration 3.2 5 minutes Participants will post questions to a discussion board created for the online instructional module. The questions will be based upon the results of the satisfaction survey and the practical implications of the survey to the teaching of online courses. The instructional design team will address these questions.
Discussion Board
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 13
Present Stimulus Material
Recall 3.3 15 minutes The Camtasia screencast will continue with an explanation of the elements needed to create lesson plans with effective communication in online courses. Participants will be provided with a PDF document that outlines these elements and also gives an effective lesson plan template.
Camtasia screencast, PDF attachment/handout
Provide Learner Guidance
Recall 3.3 N/A Participants will be provided with a PDF document that gives several examples of lesson plans that utilize best practices in online communication.
PDF attachment/handout
Elicit Performance
Elaboration 3.4 45 minutes Participants will use the lesson template provided in the Camtasia screencast lecture to evaluate and revise a personally created existing lesson plan to incorporate opportunities for enhanced communication.
Camtasia screencast, PDF attachment/handout
Provide Feedback Recall 3.3 5 minutes Participants will post questions to a discussion board created for the online instructional module concerning the most effective methods for creating lesson plans that maximize communication. The instructional design team will address these questions.
Discussion Board
Assess Performance
Elaboration 3.1
3.2
3.3
3.4
10 minutes
Participants will have two weeks to offer feedback to other participants.
Participants will post responses to a discussion board created for the online instructional module concerning how they plan to improve communication in the areas in which they were weak on the survey. In addition, participants will post the revised lesson plan for feedback and suggestions from other participants and instructors.
Discussion Board
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 14
Enhance Retention and Transfer
Recall 3.1
3.2
3.3
3.4
N/A Participants will be provided with a quick reference PDF sheet that summarizes the assessment of online communication and the elements needed for an effective lesson plan.
PDF attachment/handout
Learning Objectives
Terminal Objective 3
3.0 Instructors will analyze and evaluate best practices in online communication and apply these practices to real world products.
3.1 Instructors will recall the importance of student involvement and student satisfaction regarding online course experiences.
3.2 Instructors will evaluate through self-reflection the level of involvement and satisfaction previous students have expressed regarding online course experiences.
3.3 Instructors will identify ways in which lesson plans can provide opportunities for enhanced communication in online courses.
3.4 Instructors will evaluate and revise existing lesson plans to incorporate opportunities for enhanced communication.
3.5 Instructors will identify meaningful opportunities for students to visit the online course’s site on a regular basis and the tools to provide these opportunities.
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 15
Appendix C
Subject Matter Expert Review #1 of Learning Episode
Formative Evaluation Instrument Answer each question by placing an “X” next to yes, no, or not applicable. Please provide any additional comments that you feel will enhance the effectiveness of the instruction.
Question Yes No Not Applicable
Comments
1. Do instructional materials clearly state the performance objectives expected?
x
2. Do the instructional materials appear to match the learner characteristics?
x I needed more info. about who will be completing the course to answer this question.
3. Do the instructional materials support the performance objectives?
x
4. Is the content of the instruction complete?
x
5. Is the content of the instruction up to date?
x
6. Are learners given adequate opportunities to receive information about the content?
x The time listed for “lecture type” material seems a little short to explain the information.
7. Are learners given adequate opportunities to practice or apply what they learn?
x
8. Are learners given adequate opportunities to receive feedback on how well they applied what they learned?
x
9. Are the instructional methods appropriate for the targeted learners?
x
10. Do the instructional methods support the performance objectives?
x
11. Are there other issues that you noticed that could improve the instruction? If yes please describe in the comments box.
X Forty-five minutes does not seem long enough for someone to completely revise a lesson plan.
Source: Rothwell, W.J., & Kazanas, H.C. (1998). Mastering the instructional design process. San Francisco: Josey-Bass/Pfeiffer.
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 16
Appendix D
Subject Matter Expert Review #2 of Learning Episode
Formative Evaluation Instrument Answer each question by placing an “X” next to yes, no, or not applicable. Please provide any additional comments that you feel will enhance the effectiveness of the instruction.
Question Yes No Not Applicable
Comments
1. Do instructional materials clearly state the performance objectives expected?
X
2. Do the instructional materials appear to match the learner characteristics?
X Don’t know the learner characteristics
3. Do the instructional materials support the performance objectives?
X
4. Is the content of the instruction complete?
X
5. Is the content of the instruction up to date?
X
6. Are learners given adequate opportunities to receive information about the content?
X
7. Are learners given adequate opportunities to practice or apply what they learn?
X I only put no because it will take most people longer than ten minutes to take the survey, and longer than 45 min. to revise a lesson plan.
8. Are learners given adequate opportunities to receive feedback on how well they applied what they learned?
X I like that both instructors and other participants will provide feedback.
9. Are the instructional methods appropriate for the targeted learners?
X
10. Do the instructional methods support the performance objectives?
X
11. Are there other issues that you noticed that could improve the instruction? If yes, please describe in the comments box.
X I really like the activities where participants get to interact with one another. It would be valuable to work more of this interaction into the larger plan.
Source: Rothwell, W.J., & Kazanas, H.C. (1998). Mastering the instructional design process. San Francisco: Josey-Bass/Pfeiffer.
FORMATIVE AND SUMMATIVE EVALUATION REPORTS: THE RETENTION CHALLENGE OF ONLINE STUDENTS 17
Appendix E
Subject Matter Expert Review #3 of Learning Episode
Formative Evaluation Instrument Answer each question by placing an “X” next to yes, no, or not applicable. Please provide any additional comments that you feel will enhance the effectiveness of the instruction.
Question Yes No Not Applicable
Comments
1. Do instructional materials clearly state the performance objectives expected?
x
2. Do the instructional materials appear to match the learner characteristics?
x
3. Do the instructional materials support the performance objectives?
x
4. Is the content of the instruction complete?
x It’s complete for the given module but I did not review the entire product.
5. Is the content of the instruction up to date?
x
6. Are learners given adequate opportunities to receive information about the content?
x
7. Are learners given adequate opportunities to practice or apply what they learn?
x The learners would benefit from revising more than one lesson plan; perhaps they could revise a whole unit or topic of a course.
8. Are learners given adequate opportunities to receive feedback on how well they applied what they learned?
x
9. Are the instructional methods appropriate for the targeted learners?
x
10. Do the instructional methods support the performance objectives?
x
11. Are there other issues that you noticed that could improve the instruction? If yes, please describe in the comments box.
Source: Rothwell, W.J., & Kazanas, H.C. (1998). Mastering the instructional design process. San Francisco: Josey-Bass/Pfeiffer.
FORMATIVE AND SUMMATIVE EVALUATION REPORT: THE RETENTION CHALLENGE OF ONLINE STUDENTS 18
Appendix F
Level I Summative Evaluation Tool
Incorporating Communication into Online Course Design - Course Evaluation
Directions: Using the Likert scale provided, please rate the design of the online course in
regards to communication. In the comment section please include any ideas that could help
improve clarity of expectations or communication aspects of the course.
Course Evaluation
Strongly Agree
Somewhat Agree
Neutral Somewhat Disagree
Strongly Disagree
1. The information presented was relevant to my job as an online instructor.
2. The course provided useful information on communication in online courses.
3. I feel more comfortable communicating with my online students.
4. The course objectives were clear.
5. The instructional materials supported my learning.
6. I would recommend this course to a friend.
7. The pace of the course lessons was appropriate.
8. I felt like I had the required background for the content covered in this course.
FORMATIVE AND SUMMATIVE EVALUATION REPORT: THE RETENTION CHALLENGE OF ONLINE STUDENTS 19
9. The course achieved its desired
objectives.
10. I feel confident I can incorporate the skills learned into my teaching practice.
What did you like best about the course?___________________________________________________-_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
What would you like to see changed about the course? _______________________________________-_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
Will you use any of the communication techniques in your courses? Which ones? __________________ _____________________________________________________________________________________-_____________________________________________________________________________________
_____________________________________________________________________________________
Please write any additional comments regarding the course. ____________________________________ _____________________________________________________________________________________-_____________________________________________________________________________________
_____________________________________________________________________________________
FORMATIVE AND SUMMATIVE EVALUATION REPORT: THE RETENTION CHALLENGE OF ONLINE STUDENTS 20
Appendix G
Level II Summative Evaluation Tool
Incorporating Communication into Online Course Design - Pre-Test / Post-Test
Corresponding Objective
Pre-Test / Post-Test Question
All Communication in online courses should be handled in exactly the same way as face to face courses.
True False
1.5 Provide two examples of asynchronous communication used in online courses:
_____________________________ _____________________________
1.5 Provide two examples of synchronous communication used in online courses:
_____________________________ _____________________________
3.2 Name three important factors in student retention in online courses
_____________________________ _____________________________ _____________________________
3.5 List three examples of meaningful reasons in which a student will visit an online course platform:
_____________________________ _____________________________ _____________________________
2.5 Email replies to students in online courses should be completed within this time frame:
12 Hours 24 Hours 3 Days 2 Weeks
FORMATIVE AND SUMMATIVE EVALUATION REPORT: THE RETENTION CHALLENGE OF ONLINE STUDENTS 21
2.4 Tone is always inferred uniformly in written communication, such as online course
content.
True False
FORMATIVE AND SUMMATIVE EVALUATION REPORT: THE RETENTION CHALLENGE OF ONLINE STUDENTS 22
Appendix H
Level III Summative Evaluation Tool
Incorporating Communication into Online Course Design Evaluation Form
Directions: Using the Likert scale provided, please rate the design of the online course in
regards to communication. In the comment section please include any ideas that could help
improve clarity of expectations or communication aspects of the course.
Objective 1: Instructors will understand the basics in communication for online coursework.
Strongly Agree
Somewhat Agree
Neutral Somewhat Disagree
Strongly Disagree
1. Communication features are readily accessible in the course.
2. Communication features of the course are simple and easy to use.
3. The course design clearly outlines and identifies the expectations on when to use the different types of communication mediums.
4. The course makes good usage of asynchronous communication.
5. The course makes good usage of synchronous communication.
6. Communication was a clear fundamental consideration in the design of this course.
Comments: __________________________________________________________________________
____________________________________________________________________________________
____________________________________________________________________________________
FORMATIVE AND SUMMATIVE EVALUATION REPORT: THE RETENTION CHALLENGE OF ONLINE STUDENTS 23
Objective 3: Instructors will implement best practices in online communication.
Strongly Agree
Somewhat Agree
Neutral Somewhat Disagree
Strongly Disagree
7. Substantial communication enhancements exist in this course design compared to previous versions of this course or most other online course experiences.
8. Communication between the instructor and student is the key focus of this course compared to previous versions of this course or most other online course experiences.
9. Increased student to student communication was a key focus compared to previous versions of this course or most other online course experiences.
10. Lesson plans have clear communication objectives built into them.
11. Reoccurring student visits to the course site are more likely compared to previous versions of this course or most other online course experiences.
12. An increase in student satisfaction was perceived due to a focus on communication.
Comments: __________________________________________________________________________
____________________________________________________________________________________
____________________________________________________________________________________
Objective 2: Instructors will prepare students in their courses for online learning, and make themselves available, communicate technical guidelines and clear expectations.
Strongly Agree
Somewhat Agree
Neutral Somewhat Disagree
Strongly Disagree
FORMATIVE AND SUMMATIVE EVALUATION REPORT: THE RETENTION CHALLENGE OF ONLINE STUDENTS 24
13. Course design promotes instructor
availability and willingness to help.
14. Virtual office hours or other arrangements for one on one communication are clearly outlined.
15. Reasonable response time for online correspondence is clearly indicated.
16. Technical assistance and/or proper links to contacts are available.
17. Technical guidelines for the course are accessible and clear.
18. The course has clearly stated centrally located expectations set for the students.
Comments: __________________________________________________________________________
____________________________________________________________________________________
____________________________________________________________________________________
FORMATIVE AND SUMMATIVE EVALUATION REPORT: THE RETENTION CHALLENGE OF ONLINE STUDENTS 25
Appendix I
Level IV Summative Evaluation Tool
Incorporating Communication into Online Course Design Evaluation Form
Directions:
Step 1: Gather Data
Data will need to be gathered from online course registration data and will need to be compiled
into a summary table that looks like the following:
Year/Semester Instructor Retention Rate Course completion
2011/Fall Dr. Bob 78% N
2011/Summer Dr. Bob 85% Y
2011/Spring Dr. Bob 82% N
2010/Fall Dr. Bob 74% N
2011/Fall Dr. Benson 100% Y
2011/Summer Dr. Benson 99% Y
The data will need to be filtered to include only those professors that not only attended the
course but that scored above a zero on the Level III evaluations showing that they were able to
successfully incorporate communication enhancements into their course design behaviors.
Step 2
FORMATIVE AND SUMMATIVE EVALUATION REPORT: THE RETENTION CHALLENGE OF ONLINE STUDENTS 26
Continued data collection over time for several semesters after the training course will allow a
proper assessment of the trending. A minimum of three data points, pre and post training, will be
required, but collecting more data over time is suggested.
Step 3
Compile a trend analysis to see if retention rates increase. This will be done for instructors who
have positively altered their behavior as it relates to communication in online courses. This will
be submitted to management as part of an impact report and will look like the following: