Weighing Costs versus Benefits: Evaluating the Impact

From Implementing a Course Management System

Richard E. WestUniversity of Georgia (rickwest@uga.edu)

Greg Waddoups, Meghan Kennedy, and Charles R. Graham

Brigham Young University

Technology improves everything … right?

Calvin and Hobbes by Bill Watterson

CMS

CMS

Research Questions

• How has the implementation of a CMS impacted teaching/learning at the university?

• What are the benefits/challenges from supporting this tool on an institution-wide basis?

Research Context

• Average university enrollment each semester is 30,000 students

• 8,000 academic courses for both fall and winter semesters in 2004

• There were slightly more than 4,000 Blackboard courses created during each semester

• Over 300,000 students and faculty are estimated to have used Blackboard at least once

Methods

• Open-ended/closed-ended survey questions (n=124 instructors; 163 students) administered to students and instructors over two semesters

• An analysis of the calls reported to the Blackboard support center on campus (n=1,341)

• Semi-structured interviews with instructors (n=53) and brief intercept interviews with students (n=17)

• Data was analyzed inductively, looking for patterns within major categories

Findings — Satisfaction

Table 1. (Satisfaction) — How satisfied students are with Blackboard and how comfortable they feel using the tool. Survey Question Population SD D N A SA Blackboard is easy to learn/use. Fall 04 Students 4.2% 6.3% 14.6% 68.8% 6.3% Winter 05 Students 6.2% 6.2% 14.2% 55.8% 17.7% I prefer that instructors use Blackboard in their courses. Fall 04 Students 10.2% 4.1% 18.4% 36.7% 30.6% Winter 05 Students 6.3% 7.1% 20.5% 42.9% 23.2% Note. SD=Strongly Disagree, D=Disagree, N=Neither, A=Agree, SA=Strongly Agree.

Findings — Satisfaction

Findings: Satisfaction

What Blackboard features do you like using in your class?

0.010.020.030.040.050.060.070.080.090.0

100.0

AnnouncementsCourse DocsDisc. BoardsAssessments

Email

Virt. ClassroomLghtwght ChatGradebook

Assign ManagerDig. Dropbox

Fall '04 Students Fall '04 Faculty Winter '05 Students

Conclusions: Satisfaction• When Blackboard works without technical

difficulties, most students and instructors at our university reported being satisfied with the tool because it is convenient for them to use, easy to learn, and helpful in their studies.

• Satisfaction and usage is high for only four features (announcements, email, posting of course documents, and gradebook). Satisfaction is low for synchronous chatting tools, discussion boards, assessments, and the digital dropbox

Findings — Stability

When Was The Last Time You Encountered a Technical Problem With Blackboard?

26.5

14.2

23.925.7

9.7

34.0

21.3 21.323.4

0.00.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

1-2 days ago Less than aweek ago

Less than amonth ago

More than amonth ago

Never

Students Faculty

Conclusions: Stability

• Blackboard has been unstable in some of its most critical areas

• Stability is more than a matter of convenience. It impacts instructor/student relationships and the instructional effectiveness of activities– Case Study A

Findings — Efficiency

On average, Blackboard saves me time in teaching/working on my classes.

0.0

10.0

20.0

30.0

40.0

50.0

60.0

StronglyDisagree

Disagree Neither Agree StronglyAgree

F '04 Students F '04 Faculty

W '05 Students W '05 Faculty

Conclusions: Efficiency

• Using a CMS seems to improve efficiency somewhat for instructors and students, especially in 4 ways:– distributing class materials– updating documents electronically– keeping the instructor and course

organized– providing quicker and more frequent

feedback to students

Conclusions: Efficiency

• “The potential for me is that I can change it once in one place and not have to pass out 30 paper.”

• “It allows (and makes) professors be more organized. For the class that doesn't use Blackboard, I am often confused at what material I should know, assignments, etc.”

• “It saves time at the end of the semester so you don't get a whole bunch of students saying, ‘C+, I thought I had a B.’”

However, some efficiency traps…• Homework creep• Unrealistic expectations• Instructor reliability• Encouragement to miss class• Using the entire CMS tool for one small

feature

Findings — LearningBlackboard improves student learning.

0.0

10.0

20.0

30.0

40.0

50.0

60.0

StronglyDisagree

Disagree Neither Agree StronglyAgree

F '04 Students F '04 Faculty

W '05 Students W '05 Faculty

Conclusions: Learning

• Many instructors feel the CMS either did nothing or improved their instruction

• Most instructors who felt it improved instruction were using it to support teacher-centered instruction– Access to materials– Improved notetaking during lectures– Just-in-time preparation for teaching– Saving in-class time for other activities

Conclusions: Teacher-centered learning• “I think one of the biggest things that helps the

learning thing is having my lectures online so they can pull them down and review or else they can get it if they miss the class, and that's huge.”

• “I think it's allowed a little more freedom in the classroom for students to ask questions and not feel they're going to miss something while they're writing it down.”

• “It's interesting because I can get through a lecture now 30% faster because I don't have to write it all out on the board. . . . I can get through the material so I can get to discussion and application, which is really nice.”

Findings: Student-centered learning• Many of the instructors who do not find

Blackboard to be a helpful tool are those instructors who prefer more hands-on, exploratory, or student-centered approaches to instruction. – Case Vignette 2

• These instructors often feel that Blackboard’s benefits do not outweigh the costs associated with using the tool

• These instructors are typically seeking out alternatives to Blackboard to fulfill their online instructional needs, such as blogs, wikis, personal websites, and instant messenging services

Why?• All feature controls are open only to

the teacher– Examples: Discussions, grading, posting

of documents, no outside guests, creating/adding modules

• Blackboard is not easily adaptable– “It is not flexible”

Overall Conclusions

• Do the benefits of a CMS outweigh its disadvantages?– It depends! What type of instruction do you

want to encourage? Are you satisfied with only four features, or will it only be worth the price tag if every feature is being used?

• Caution: are you getting “married” to the tool?– Even if they only use one or two features,

faculty can grow dependent on the tool, making a switch very difficult.

Future Questions• Can a CMS be built to support

constructivist, student-centered, problem-based, or other types of pedagogies?

• How can we help stakeholders properly evaluate a technology for its pedagogical strengths before purchasing it?

• If we have already purchased one that is not meeting our needs, how can we more easily maneuver towards other tools without isolating the instructors who now feel tied to the tool?

Case Studies

• “It looks like it works great, but there are glitches.”

• “It's really cumbersome.”