Computers & Education 58 (2012) 1094–1105

Contents lists available at SciVerse ScienceDirect

Computers & Education

journal homepage: www.elsevier .com/locate/compedu

Development and evaluation of a Web 2.0 annotation system as a learning tool inan e-learning environment

Yu-Chien Chen, Ren-Hung Hwang*, Cheng-Yu WangNational Chung-Cheng University, Taiwan

a r t i c l e i n f o

Article history:Received 14 June 2011Received in revised form21 November 2011Accepted 13 December 2011

Keywords:Annotation systemWeb 2.0LMSOnline learningUsability

* Corresponding author. Department of ComputerTaiwan.

E-mail address: rhhwang@cs.ccu.edu.tw (R.-H. Hw

0360-1315/$ – see front matter � 2011 Elsevier Ltd. Adoi:10.1016/j.compedu.2011.12.017

a b s t r a c t

The emergence of Web 2.0 technology provides more opportunities to foster online communication andsharing in an e-learning environment. The purpose of this study was to develop a Web 2.0 annotationsystem, MyNote, based on the Web 2.0 core concepts which emphasize ease of access and active sharingand then to gain an understanding about people’s perceptions of MyNote from a usability perspective.MyNote was employed on multimedia learning objects in a Learning Management System (LMS), and outof the LMS as well in this study. The evaluation results showed that, with factor analysis, interactivity,usefulness, helpfulness, and willingness for future use were categorized to represent the perceptions ofMyNote. It was also found that the factors of interactivity and helpfulness were statistically significant topredict the future use of MyNote. Lastly, the habit of taking notes also affected learners’ perceptions ofusing MyNote.

� 2011 Elsevier Ltd. All rights reserved.

1. Introduction

How to foster online interaction to help students learn has been an important issue in the development of online learning programs.Numerous studies have shown various instructional strategies, such as collaborative learning, or blended learning, from an instructionalperspective to engage students online while others are devoted to developing new tools and technology to afford rich learning environ-ments (Alexander, 2006; Avouris, Komis, Margaritis, & Fiotakis, 2004; Barak, Herscoviz, Kaberman, & Dori, 2009; LeeTiernan & Grudin,2001). The emergence of Web 2.0 technology provides an opportunity to develop online learning tools enabling students to not onlyparticipate in online activities more actively, but also to learn from their colleagues. The use of blogs and Wikis are examples of ways toutilize Web 2.0 technology in online learning environments. Many researchers have reported that Web 2.0 based learning tools or systemsprovide effective platforms for collaborative learning and knowledge sharing (Barak et al., 2009; Hwang, Wang, & Sharples, 2007; Jones,Blackey, Fitzgibbon, & Chew, 2010; Su, Yang, Hwang, & Zhang, 2010). However, in a Learning Management System (LMS), the use ofblogs orWikis mainly offers a supplemental place separated from themain learning content andmaterials for students to share or constructtheir knowledge. We believe that Web 2.0 technology should be able to provide a more intuitive way for students to learn and to interact atthe same time when they focus on learning multimedia materials.

Traditionally, in physical classrooms, students get used to making annotations and notes on paper-based textbooks or handouts. Takingnotes is a common learning behavior. Brown and Smiley (1978) indicated that students who take notes or make annotations would studybetter. Moreover, makingmeaningful annotations on readingmaterials benefits not only for annotators but also for future readers (Marshall,1997; Wolfe, 2002). The growth of online learning programs changes the way students gain knowledge (Bates, 2005; Laurillard, 2005;Prensky, 2001). Online multimedia learning objects provides an interactive way to learn but only on the interface of computer screens. Ifstudents still like to take notes when learning online, in what way can an online annotation system facilitate students to make annotations,especially with the application of Web 2.0 technology?

Furthermore, online learning is not only limited within the LMS; on the contrary, online learners always search supplemental infor-mation and resources out of the LMS. The purpose of this study is to develop a Web 2.0-based online annotation system, called MyNote,

Science & Information Engineering, National Chung Cheng University, 168 University Road, Min-Hsiung, Chia-Yi 621,

ang).

ll rights reserved.

Y.-C. Chen et al. / Computers & Education 58 (2012) 1094–1105 1095

which is able to make annotations on learning objects in the LMS as well as supplemental web documents out of the LMS. The researchquestions addressed in this study are:

(1) What characteristics does a Web 2.0 annotation system possess in order to foster online learning?(2) How do users perceive MyNote as an online learning tool?

2. Theoretical framework

How to facilitate online learning interaction has always been an issue in managing online learning environments. This study tries toexplore the effect of Web 2.0-based annotation technology to enhance online learning interaction. This section discusses the concepts ofonline learning communities and collaborative learning in order to find the way to foster online interaction. Moreover, Web 2.0-basedannotation systems are presented to demonstrate their benefits on enhancing online learning interaction.

2.1. Online learning communities

Online learning communities refer to virtual places which combine learning and community together (Downes, 1999). It is not only tobecome skilled at online courses but also to learn from the interaction with other participants. Clark (1998) proposes some principles toestablish learning communities. For example, a learning community is not built, but grown by itself. Clark indicates that a community will bestrong if it is grown by its members to create their own environment. Once they can realize that they are a part of this community, they cansupport each other, will this community be sustained. The other principle is that personal narrative is encouraged. Clark asserts thatpersonal narrative is “the sun that makes communities grow.” Exchanging experiences or opinions can make members feel closer to eachother and provide identity. In addition, Downes (1999) also points out that for learning communities, creating a sense of commitment isimportant. Once members are able to build their trust in this community, they will share more about learning and personal experiences.Downes suggests successful learning communities can be built through course management. He claims that content and communicationshould be integrated together, which implies that learning together cannot be isolated from the content.

2.2. Collaborative learning

Another important way to foster an online learning community is through collaborative learning. Collaborative learning involvesstudents to learn as a team and it contributes to cognitive learning (Klien, O’Neil, & Baker, 1998). Several studies have shown that collab-orative learning can enhance learning in the classroom (Liaw, Chen, & Huang, 2008; Wang, 2009). In a collaborative learning setting,students can verbalize what they think, help with each other, and complete a task together (Webb, Nemer, Chizhik, & Sugrue, 1995). In suchan environment, students are able to solve problems together through mutual interaction and discussion in order to achieve the learningobjectives. People who advocate Constructivism also claimed that knowledge is built through interaction with environment, includingmaterials and people (Jonassen, Peck, &Wilson,1999). An online collaborative learning environment emphasizes online communication andinteraction among peers in order to augment their experiences, thus enhancing their learning performance. From the perspective of socialconstructivism, students would learn more through the process of sharing experiences, discussion to build their knowledge (Vygotsky,1978). This process also enables the construction of online knowledge management. Some research has been devoted to developingonline tools to facilitate online communication and interaction to eliminate the “sociable spaces” (Karahalios & Donath, 2003).

2.3. Annotation system

In order to enhance peer interaction, many computer-mediated communication tools are involved in online learning environments, suchas discussions boards, learning journals, blogs, etc. However, the study of online annotation systems utilized in LMS has not beenemphasized until recently (Ahern, 2005; Annotea, 2001; Hwang & Wang, 2004; Hwang et al., 2007; Su et al., 2010). Since taking notes isa common behavioral occurrence in the classroom, there is necessity to provide annotation systems even in online learning environments aswell.

Several studies have shown the benefits of online annotation systems onwork efficiency and learning performance (Ahern, 2005; Cadiz,Gupta, & Grudin, 2000; Gupta, Condit, & Gupta, 2008; Hwang et al., 2007;Marshall & Brush, 2004; Nokelainen, Kurhila, Miettinen, Floreen, &Tirri, 2003; Quade,1996; Robert, 2009; Su et al., 2010). Quade (1996) indicated that students learnedmorewith computer-based annotation,rather than paper-based, in computer-based learning environments. Ahern (2005) found the use of online annotation software – Red Pencil,encouraged students to engage in learning activities. However, the interface design might interfere with students’ reading. Nokelainen et al.(2003) found that EDUCOSM, an annotation system fostered learner-centered collaborative learning and students who liked to use the toolelaborated more about their work. Another annotation system, VPen also showed its benefit by enhancing student’s learning performance(Hwang et al., 2007). In the workplace, web annotations successfully promote the work efficiency through context-based discussion withdocuments (Cadiz et al., 2000). Gupta et al. (2008) even created an annotation management system, Graphitti, to facilitate annotationamong heterogeneous objects in scientific data. It seems that online annotation systems can serve as a “mindtool” (Jonassen,1996) to inspirestudent thinking and may act as a catalyst to foster collaborative learning and peer interaction. In addition, how to share knowledge usingannotation was also discussed in several literatures (Robert, 2009). Recently, a Web 2.0 collaborative annotation tool, referred to aspersonalized annotationmanagement system (PAMS), was developed by Su et al. (2010). This system especially emphasized on collaborativeand knowledge sharingwhere users have full control of the groupmembers in order to manage the access control of annotation sharing. Theauthors found that learners are very interested in using this tool and feel the tool is useful for conducting learning tasks. They also showedthat use of PAMS can significantly increase learning achievements in collaborative learning environments.

Y.-C. Chen et al. / Computers & Education 58 (2012) 1094–11051096

However, most online learning annotation systems are mainly utilized on static web pages or documents while some are focused onphoto or video noting. Some even require user registration before making use of the annotation system. In LMS, learning objects arecommonly multimedia-based, which involve dynamic animations, video streaming, and slide presentations. Annotation on static pages ortext-based documents may not meet the need to directly make notes on multimedia materials. In addition, although in most annotationsystems, annotators can share their annotations with the public, not all of them can allow other users to put their comments back to theannotations. The interaction is passive, not mutual. However, in online learning environments, the conversations among annotations mightbe required and helpful for learners to gain a deeper understanding about the contents. In other words, in addition to sharing notes,interacting with notes is required for annotation system design in LMS, thus offering an opportunity to facilitate making personal anno-tations to public ones (Marshall & Brush, 2004). Another concern of annotation systems for online learning is its universal use acrossdifferent browsers and LMSs. Learning online is not only limited within a LMS. On the contrary, learners often search information orresources out of the LMS as supplemental materials. Therefore, annotation systems for online learning should be versatile to accommodatedifferent systems and interfaces.

Azouaou, Chen, and Desmoulins (2004) proposed three principles to develop annotation systems for e-learning, including usefulness,shareability, and usability. In online learning environments, annotation systems should take the teaching and learning contexts intoconsideration. For example, the annotation system can cooperate with the learning activities and correspond to learning objectives.Secondly, an e-learning annotation system can facilitate the communications of learners through annotation. Online learners can share andinteract with each other through the annotation system interface. Thirdly, the e-learning annotation system requires a user-friendlyinterface, which offers users an easy way to interact and does not interfere with learners’ interaction between learning objects andannotation (O’Hara & Wilensky, 1997).

2.4. Prototype of Web 2.0 annotation system

The concept of Web 2.0 brings out interactive social software (Alexander, 2006), which allows users to actively share opinions, engagein online activities, span all connected devices, not limited on the web but requiring lightweight programming models (O’Reilly, 2005).Jones et al. studied the student experience on current Web 2.0 social software and argued the need of design principles of learning withsocial software so that individual preferences can be considered. Important characteristics of Web 2.0 include rich user participation,dynamic content, metadata, and collective intelligence. For collaborative learning, we believe that the most important power of Web 2.0 isto harness collective intelligence. Therefore, the Web 2.0 technology offers a great niche to develop online learning annotation systems.Summarized from above, this study proposes a prototype of Web 2.0 annotation system. A Web 2.0 annotation system should provide aninterface with which learners can annotate within the context, especially on multimedia learning objects. They can annotate at any nearbyposition wherever they want to insert. In other words, embedded annotations within the context are allowed. Moreover, learners caninteract with peers on annotations and share them publicly as well based on their preferences. That is, it should serve as a platform toharness collective intelligence, just like blogs or Wikis with access control. The annotation system not only allows learners to makeannotations on multimedia learning objects, more importantly, it also allows them to read annotations made by others, reply to questionsposted by others, and search annotations via keywords or category, etc, on learning objects. Thirdly, Web 2.0 annotation systems enablelearners to collaborate with peers and cooperate on knowledge construction. Web 2.0 annotation systems can be utilized across variousbrowsers and LMS. The annotations can be filtered and searched easily. Lastly, the design of Web 2.0 annotation systems should be user-friendly. Based upon the concepts of Web 2.0 annotation systems, this study develops a Web 2.0-based annotation system, called MyNote,introduced as follows.

The contribution of MyNote is to exploit the Web 2.0 concept on collaborative learning through annotation. It allows learners to writenotes, share notes, and discuss notes with others; therefore, it fulfills theWeb 2.0 characteristics of user participation, dynamic content, andcollective intelligence. Technical novelty of MyNote includes using mashup tools to allow the annotation tool to be part of an LMS system. Itis a cutting-edge annotation tool on streaming media, including flash and videos. MyNote also allows learners to classify notes, in particular,as question or answer so that they could post question notes on the right place of the learning material and questions could be answered bytheir peers or instructors effectively. Lastly, it displays notes on streaming media based on the time stamp which expands notes to the timedomain, a new concept on annotating dynamic Web content.

3. MyNote annotation system

3.1. System architecture

The architecture of MyNote Annotation System (MyNote) is shown in Fig. 1. The MyNote Annotation system is embedded in a LearningManagement System (LMS). MyNote can be employed in two different scenarios: (1) make annotations of learning objects within the LMSand (2) make annotations of web documents out of the LMS. When MyNote is used on learning objects in the LMS, in order to avoidinterfering with users’ learning activities such as reading learning objects, MyNote passes annotations, URL, and user ID through Asyn-chronous JavaScript XML (AJAX) to save annotations. AJAX is a commonly used Web 2.0 technology used on the client side. It allows a webclient to send data to, and retrieve data from, a server in the backgroundwithout interfering with the display of current web page. Therefore,with AJAX, MyNote allows learners to continue their learning activities without notice of the operations done by MyNote. For example,when a learner is reading a learning object, making an annotationwill not cause a reload of current learning object or redirect to a new page.While it is used onweb documents out of the LMS, MyNote passes annotations and URL through HTTP POST request. The annotation systemwill make a copy of the original web pagewith annotations, and then return aweb page that shows an URL to point out where the copy pagewas just made. Learners can share this URL with others later on.

Fig. 1. The system architecture of MyNote.

Y.-C. Chen et al. / Computers & Education 58 (2012) 1094–1105 1097

3.2. User interface and functions

The setup of MyNote requires only one step to bookmark a hyperlink of MyNote. When the setup is completed, users can click thebookmark on any web page where users want to annotate. The MyNote toolbar appears on the web page after clicking the bookmark, asshown in Fig. 2.

The functions of MyNote Toolbar are shown as following:

1. Point: Allow the mouse back to its status as a point after users work on the functions of Write, Mark, Line, Circle, or Box.2. Write: Leave a sticky note on the web page. Users can write texts on it and classify the note into different categories. Default categories

include normal, question, answer, and discussion.3. Mark: Highlight selected texts.4. Line/Circle/Box: Drag mouse to draw a line/circle/box on the web page.5. Undo: remove last annotation activity (Write, Mark, Line, etc.).6. Hide_All/Show_All: Hide/Show all annotations, except Mark.7. Filter: Filter the annotations according to category. For example, users can hide or filter out all annotations and only display “question”

notes.8. and : The former is hiding MyNote toolbar, while the latter is closing MyNote toolbar.

3.3. MyNote in the LMS

When MyNote is used on multimedia learning objects in a LMS, the functions of SAVE, LOAD, Play, Manage, and Search may performdifferently from those out of the LMS. The following describes these functions in the LMS.

� SAVE: As shown in Fig. 3, learners can click on the “SAVE” button to save their annotations. When the “SAVE” function is invoked,MyNote will pass annotation and metadata through AJAX without redirecting learners to another web page. In this way, learners cankeep reading multimedia learning objects on the page.

� LOAD: Fig. 4 demonstrates that learners can load their own annotation. Or as in Fig. 5, they can also load several annotations from theirpeers who post their annotations on the same learning object. The load function is also implemented using AJAX.

� Play: A very unique feature of MyNote is that learners are able to make annotation on multimedia streaming content, such as flash andvideo. Fig. 5 shows annotation made on a video streaming learning object. Annotations made on flash objects are shown in Fig. 6. Whenlearners annotate on multimedia streaming learning objects in the LMS, MyNote places each annotation component a time stamp fromthe learning object. When learners click on the “Play” button, their annotations of the multimedia streaming content will appear on thestreaming media according to their associated time stamps, similar to the display of subtitles of a video.

Fig. 2. MyNote toolbar.

Fig. 3. SAVE in the LMS.

Y.-C. Chen et al. / Computers & Education 58 (2012) 1094–11051098

� Management: The MyNote manage function is a function of LMS, which is not shown on the MyNote toolbar. When learners of LMSinvoke the manage function, all annotations will be listed on a web page. Learners could filter notes by class (tag), share/hide notes, oradd/delete/modify class tag of a note. Note that classifying a note in MyNote is by adding a class tag to it.

� Search: The search function of MyNote is also built in the LMS. Learners can log on to the LMS to search their own annotations or publicannotations shared by their peers by keywords. Search scope could be unlimited or limited to a specific type of annotations. Four optionsof the search scope are implemented, including unlimited search, limited to question, answer or discussion only. A simplified version ofthe search function is also supported on MyNote toolbar, as shown in Fig. 2, which only supports search on the learning object beingexamined, such as a flash or video object.

Fig. 4. LOAD in the LMS.

Fig. 5. Load several annotations from peers in the LMS.

Y.-C. Chen et al. / Computers & Education 58 (2012) 1094–1105 1099

3.4. Learning objects out of the LMS

Fig. 7 demonstrates how MyNote is used on multimedia learning objects out of the LMS.When MyNote is used on learning objects out of the LMS, that is, on normal web pages, the functions are slightly different from those

within the LMS which are summarized as follows.

� LOAD and PLAY: These two functions cannot be used out of LMS.� SAVE: After learners click the SAVE function, MyNote creates a copy of the web page on which all annotations are included. Then thesystem replies an URL to inform learners where the saved annotation is, as shown in Fig. 7(c). Fig. 7(d) shows that if anyone browses theURL, they can use the MyNote toolbar on the web page directly without installing it.

Fig. 6. MyNote allows learners to make annotations on flash objects.

Fig. 7. (a). Starts to use MyNote out of LMS. (b). Using MyNote to annotate. (c). After save, MyNote system will reply with an URL to tell where the saved annotation is. (d). The copyof the original page, another user replies an answer on the web.

Y.-C. Chen et al. / Computers & Education 58 (2012) 1094–11051100

3.5. Potential benefits of MyNote to foster online learning

MyNote may have potential to be an online scaffolding tool. First, MyNote does not require a complicated installation process, whichmight impede online learners’motivation to use it. Second, MyNote offers several note categories for learners to classify their notes, such asa question or discussion. The classification of notes offers an organized way for knowledge management. In addition, sharing peers’annotations may stimulate further thinking when learners read learning objects. By that time, MyNote may become a scaffolding tool forcritical thinking. Furthermore, MyNote allows learners to read annotations and the learning objects at the same time; thus it may reduce theaction of going back and forth between annotations and learning objects if they are not located at the same space and time.

MyNote creates a newway of online interaction among learners, materials, and instructors. For example, if a learner has some questionsabout the learning objects, he does not need to write a lot of words to describe his question scenario, neither to post those words on a forumor discussion board. With this system, he only makes annotations on the learning object and set the note category to “Question” andmake itto “Public”. As a result, everyone can see the question on the learning object, and read the question. If someone answers the question andsets their annotation to public, too, then the learner who asked the question can find the answer by loading all public annotations on thelearning object with the note category of “Answer”. Similarly, the learner can useMyNote tomake annotations onweb documents out of theLMS and then share with others or put the annotations into their own knowledge database. These documents may be supplementalinformation for learning topics.

Y.-C. Chen et al. / Computers & Education 58 (2012) 1094–1105 1101

4. Evaluation of Web 2.0 annotation system

Since MyNote annotation system is just in its initial development stage as an educational online learning tool, this study has adopted theconcept of usability testing as its evaluation method to examine the effectiveness of MyNote annotation system and gained more under-standing to improve its interface and function as a more developed online learning tool. Usability refers to a degree of ease for people tolearn when they interact with an interface (Head, 1999). The purpose of usability testing is to gain users’ feedback in order to improve theinterface design and its effectiveness. It emphasizes empirical tests to gather data to evaluate the effectiveness of the interface (Torres, 2002;Treu, 1994). Several methods could be employed in the usability testing process (Nielsen, 1993) based upon the lifecycle stage of products.The questionnairemethodwas adopted in this study to gain users’ perceptions and experiences when they interact withMyNote annotationsystem.

4.1. Research methods

4.1.1. SettingAn online course “Educational Statistics” was chosen to examine the effectiveness of MyNote annotation system. The online course of

“Educational Statistics” consists of multimedia learning objects with streaming videos. The multimedia streaming videos involve Power-Point slides with texts, the instructor’s hand-writing, pictures, and sounds. Learners are able to control the learning speed on their ownwithforwarding, rewinding, or stopping the videos.

4.1.2. SubjectsFifty students from two classes of the course “Introduction of Educational Statistics” took part in this study. Twenty nine undergraduate

students were from a regular classroom setting who took a face-to-face classroom instruction (Group 1), while twenty one students werefrom an online masters programwho studied educational statistics online (Group2). They all majored in education and had experience theuse of LMS.

4.1.3. ProcedureThis study went through four steps. First, both groups of participants were asked to attend a computer lab. Before all the participants

started, they were informed about the purpose of the study (5 min). Then a demonstration of how to use MyNote was delivered, includingaccessing the online course, installing the system (i.e. creating the bookmark), and explaining all functions of the system, such as how to usethe toolbar, and how to respond to other people’s notes (15min). Next, the participants were given tasks towork on each function ofMyNoteafter the demonstration, including tasks in the LMS and out of the LMS (20 min). Following the tasks, the participants completed a ques-tionnaire regarding their experiences of using MyNote (10 min). The whole process took 50 min to complete.

4.1.4. InstrumentA paper-based questionnaire was developed in two parts to assess the subject’s experience interacting with MyNote and their personal

backgrounds. This study adapted the concepts of usability testing and developed seventeen questions regarding its function, interfacedesign, and users’ experience and perception with a 4-point Likert scale (1 ¼ completely disagree, 4 ¼ completely agree) in the first part(shown in Table 1). Before this questionnaire was implemented, it was reviewed by three experts to verify its validity. The overall scalegained reliability of Cronbach’s a ¼ 0.89. The second part of the questionnaire asked the subjects about their experiences of using varioustechnologies, and habit of taking notes. The habit of taking notes was categorized by frequency, including, usually, often, sometimes, andseldom.

Table 1Participants’ perceptions of using MyNote.

Experiences of using MyNote 4 3 2 1 Mean SD

1. I wanted to read the annotation that other people shared. 18 (36%) 24 (48%) 6 (12%) 2 (4%) 3.16 0.792. When I read people’s question annotation, I tried to answer it. 13 (26%) 26 (52%) 9 (18%) 2 (4%) 3.00 0.783. When I read people’s question annotation, I paid attention to the answers that

other people make.19 (38%) 25 (50%) 4 (8%) 2 (4%) 3.22 0.76

4. When I read people’s discussion annotation, I paid attention to the discussiontopic and content.

15 (30%) 31 (62%) 2 (4%) 2 (4%) 3.18 0.69

5. When I read people’s discussion annotation, I tried to join the discussion. 14 (28%) 24 (48%) 10 (20%) 4 (4%) 3.00 0.816. The integration of MyNote annotation and learning objects helped me understand

the content.8 (16%) 30 (60%) 11 (22%) 1 (2%) 2.90 0.68

7. The integration of MyNote annotation and learning objects helped me memorizethe content.

13 (26%) 31 (62%) 5 (10%) 1 (2%) 3.12 0.66

8. The Toolbar did not interfere my reading of multimedia learning objects. 10 (20%) 37 (74%) 2 (4%) 1 (2%) 3.12 0.569. The experience of using MyNote was interesting. 21 (42%) 27 (42%) 1 (2%) 1 (2%) 3.36 0.6310. The MyNote toolbar describes all the functions clearly. 21 (42%) 21 (42%) 8 (16%) 0 3.26 0.7211. Installing MyNote did not frustrate me. 21 (42%) 25 (50%) 4 (8%) 0 3.34 0.6312. After some practices, it was easy for me to work on every function. 18 (36%) 29 (58%) 3 (6%) 0 3.30 0.5813. MyNote is useful to annotate on multimedia learning objects. 17 (34%) 28 (56%) 5 (10%) 0 3.24 0.6214. The tools of circling, lining, and squaring are useful to annotate on multimedia

learning objects.10 (20%) 28 (56%) 11 (22%) 1 (2%) 2.94 0.71

15. I would like to keep using MyNote in the future. 13 (26%) 31 (62%) 4 (8%) 2 (4%) 3.10 0.7116. MyNote motivates me to annotate on multimedia learning objects. 12 (24%) 32 (64%) 6 (12%) 0 3.12 0.5917. I would like to use MyNote as my main web annotation tool. 11 (22%) 32 (64%) 6 (12%) 1 (2%) 3.06 0.65Overall 3.14 0.41

Y.-C. Chen et al. / Computers & Education 58 (2012) 1094–11051102

4.1.5. Data analysisSeveral statistical methods were employed for data analysis. First, descriptive statistics was used to present the participants’ responses

and perceptions about MyNote. Next, factor analysis was utilized to explore any systematic patterns in the questionnaire. Then the rela-tionships among the new variables generated from the results of factor analysis and background variables were analyzed with t-test,multiple regression and MANOVA statistical techniques. SPSS 15.0 for Windows was employed as the analytical tool in the paper.

4.2. Results

The results of personal backgrounds showed that when asked about the medium of taking notes or annotations, all the participantsanswered that they had the habit of taking notes with paper and pencil (pen). Only three participants (6%) from the online masters programused the electronic medium for taking notes. The participants who used paper as the major annotationmedium reported that theywere justused to using paper (70%), and it was convenient to use (40%). While for those who used the electronic medium to take notes, they indicatedthe advantages of the electronic medium are the easiest to organize and edit the note contents. Regarding the habit of taking notes, 58% ofthe participants always took notes, 30% took notes often, while 12% of the participants seldom did so.

Table 1 shows all participants’ perceptions after using MyNote. As we can see, most participants were satisfied with MyNote (Overallmean¼ 3.14, SD¼ 0.41) and they thought their experience of usingMyNotewas quite interesting (Mean¼ 3.36, SD¼ 0.63). When theywereasked their reaction to other people’s note, more than seventy percent of participants tried to pay attention to the peer’s response (item 1, 3,4, 5), and even tried to answer it (item 2). Over seventy percent of participants thought the integration of MyNote helped them learn online(item 6, 7). As for the interface design, most people felt it was not difficult to use MyNote as a learning tool (item 8, 10, 11, 12, 13, 14). Lastly,more than 80% of participants showed their satisfaction and willingness to use MyNote in the future (item 9, 15, 16, 17).

In order to explore more about users’ experiences and gain more systematic results from the questionnaire, a statistical technique ofexploratory factor analysis with varimax rotation was employed. Based upon the results of exploratory factor analysis, these seventeenquestions were categorized into four factors whose eigenvalues are greater than 1 (shown in Table 2): interactivity, helpfulness, usefulness,and willingness for future usage. The percentage of variance explained for each factor was: 37.83% for interactivity, 18.95% for helpfulness,7.54% for usefulness, and 6.14% for willingness for future usage (shown in Table 3). A total of 70.46% variancewas explained. In addition, eachfactor showed high internal consistency (Cronbach’s a¼ 0.91, 0.82, 0.76, and 0.73, respectively). Interactivity refers to learner involvement ininteracting within the system and sharing notes with peers (five questions). Helpfulness examines how the system helps learners learn (fourquestions). Usefulness relates to the degree of ease to use and how useful the system is (five questions). Willingness for future use relates tosubjects’will to use this system in the future (three questions). They thought MyNote was quite useful (average mean of usefulness ¼ 3.22,SD ¼ 0.47) and helpful (average mean of helpfulness ¼ 3.13, SD ¼ 0.51). These four factors were then applied for further analysis betweensubjects’ perceptions and personal backgrounds.

Although two different groups were involved in this study, the statistical results showed that these two groups did not violate theassumption of homogeneity of variance (p > 0.05). Furthermore, no statistical difference exists between these two groups for theirexperience responses of MyNote (shown in Table 4). This examination indicates that in spite of the learning background of these two groups,their perceptions of using MyNote may be equal. Therefore, the following analysis will no more treat all participants as from two groups.

Table 5 showed that genderwas a significant predictor on perceptions of usingMyNote. The result revealed thatmale participants tendedto gain higher appreciation on the factor of usefulness (t ¼ 3.40, p < 0.01), implying that male participants can adapt to working on thesystem more easily.

One important issue of developing MyNote is to assess users’ intention to employ this system in the future. Taking all the factors ofinteractivity, helpfulness, and usefulness into account, the result of multiple regression analysis showed that interactivity (b¼ 0.32, p< 0.01)and helpfulness (b ¼ 0.38, p < 0.01) are two indicators to predict users’ willingness to use MyNote in the future (see Table 6). This modeltotally explained 56% of the variance. The result implies that people who gained better experiences on interactive annotation activities andfound the helpfulness of the annotation system, may show more interest in using the system in the future.

Table 2Factor loading of experiences of using MyNote.

Experiences of using MyNote Component

1 2 3 4

1. I wanted to read the annotation that other people shared. 0.80 0.24 0.06 0.012. When I read people’s question annotation, I tried to answer it. 0.81 �0.07 0.01 0.373. When I read people’s question annotation, I paid attention to the answers that other people make. 0.85 0.12 0.01 0.204. When I read people’s discussion annotation, I paid attention to the discussion topic and content. 0.91 0.12 �0.07 0.015. When I read people’s discussion annotation, I tried to join the discussion. 0.76 0.04 �0.03 0.346. The integration of MyNote annotation and learning objects helped me understand the content. 0.21 0.74 0.01 0.377. The integration of MyNote annotation and learning objects helped me memorize the content. 0.24 0.64 0.17 0.338. The Toolbar did not interfere my reading of multimedia learning objects. 0.05 0.67 0.49 0.119. The experience of using MyNote was interesting. �0.01 0.84 0.13 0.0810. The MyNote toolbar describes all the functions clearly. �0.18 0.41 0.46 0.3711. Installing MyNote did not frustrate me. 0.11 0.08 0.84 0.2412. After some practices, it was easy for me to work on every function. �0.15 0.10 0.79 0.1413. The tools of circling, lining, and squaring are useful to annotate on multimedia learning objects. 0.08 0.50 0.67 �0.1614. MyNote is useful to annotate on multimedia learning objects. 0.23 0.41 0.13 0.7115. I would like to keep using MyNote in the future. 0.36 0.46 0.15 0.4316. MyNote motivates me to annotate on multimedia learning objects. 0.22 0.29 0.11 0.7417. I would like to use MyNote as my main web annotation tool. 0.37 0.05 0.40 0.61Eigenvalue 6.43 3.22 1.28 1.04

Table 3Experiences of using MyNote (n ¼ 50).

Factor Mean SD % variance explained Cronbach’s a

Interactivity (item 1, 2, 3, 4, 5) 3.11 0.66 37.83 0.91Helpfulness (item 6, 7, 8, 9) 3.13 0.51 18.95 0.82Usefulness (item 10, 11, 12, 13) 3.21 0.51 7.54 0.77Willingness for future use (item 14, 15, 16, 17) 3.13 0.52 6.14 0.82

Table 4Group differences on experiences of using MyNote.

Group 1 Group 2 t value

N ¼ 29 N ¼ 21

Mean SD Mean SD

Interactivity 3.02 0.70 3.24 0.59 �1.16Helpfulness 3.16 0.59 3.08 0.39 0.49Usefulness 3.16 0.54 3.29 0.47 �0.90Willingness for future use 3.09 0.57 3.18 0.44 �0.56

Y.-C. Chen et al. / Computers & Education 58 (2012) 1094–1105 1103

Taking notes as a learning habit might influence people’s behavior or perceptions when they make online annotations. Therefore, a one-way MANOVA was conducted with interactivity, helpfulness, usefulness, and willingness for future use as dependent variables while thehabit to take notes was treated as an independent variable. The effect of taking notes on subjects’ MyNote perceptions was significant(Wilk’sL¼ 0.60, F(8,88)¼ 3.20, p< 0.01). The followed univariate analysis showed that the factors of interactivity andwillingness for futureuse were statistically significant (F(2, 47) ¼ 6.64, p < 0.01, F(2, 47) ¼ 4.53, p < 0.05, respectively), indicating that people who are used totaking notes more would be more engaged in the annotation activities and tend to use online annotations system, such as MyNote, in thefuture.

5. Discussion and conclusion

This study has tried to build up a Web 2.0-based online annotation system and evaluated its effectiveness from a usability perspective.MyNote, an online annotation system, was created based upon the Web. 2.0 core concepts, with which, MyNote allows learners to annotateon multimedia learning objects and to share, search, and classify notes with their peers. The usability testing results showed that studentsperceived MyNote annotation system as a tool possessing the characteristics of interactivity, helpfulness, and usefulness, and theyconsidered to adapt it for future use. These characteristics echo the spirit of Web 2.0, the design core of MyNote. Moreover, male studentsfocused on the function of usefulness than female. The factors which would influence students’ willingness for future use of MyNote wereinteractivity and helpfulness. Furthermore, students’ habits of taking notes affected their perceptions about interactivity and willingness forfuture use of MyNote.

Among these factors, interactivity and usefulness are two major concerns for future use. This result corresponds to the requirements ofannotation system for online learning (Azouaou et al., 2004). Further, it implies that since students concern about communicating andsharing informationwith peers in online learning environments, that the annotation system should be useful and easy for students in orderto achieve their expectation. This study also found that the habit of taking notes was a significant factor in predicting online annotationbehaviors. Embedding an annotation system in LMS may help students learn and lead to more active interactions when they take onlinenotes more intuitively. Ahern (2005) indicates that the Red Pencil annotation systemwould get in the way of reading learning objects. Thisphenomenonwas not obvious in the study, showing that the design ofMyNote seemsmore user-oriented and friendly. Another advantage ofMyNote is that it enables students to annotate on the multimedia learning objects. Students do not have to go back and forth between theannotation system and the learning objects, whichmay reduce the cognitive load of finding sources (Mayer, 2001). In addition, putting notesjust on the learning objects may make learning process more coherent and more intuitive. As Downes (1999) suggests, the integration oflearning content and communication might foster learning. MyNote provides such an intuitive way to present annotations and allowsstudents to interact on it, demonstrating its great potential as an online learning tool. However, this study implies that gender may affectannotation behaviors. It seems easier for male students to adapt to working on the functions of MyNote. Colley and Comber (2003) alsofound that male adolescents were more confident and skillful on computers. As a result, male students may become familiar with computer

Table 5The effect of gender on experiences of using MyNote.

Male Female t value

N ¼ 18 N ¼ 32

Mean SD Mean SD

Interactivity 3.04 0.86 3.15 0.52 �0.54Helpfulness 3.29 0.39 3.03 0.55 1.77Usefulness 3.53 0.46 3.03 0.45 3.72a

Willingness for future use 3.22 0.56 3.08 0.49 0.94

a p < 0.01.

Table 6Predictors to affect willingness for future use.

b t value

Intercept 0.18 0.46Interactivity 0.32 3.95a

Helpfulness 0.38 2.97a

Usefulness 0.23 1.88

a p < 0.01, R2 ¼ 0.57.

Y.-C. Chen et al. / Computers & Education 58 (2012) 1094–11051104

tools more quickly and easily than female. Furthermore, because of their background and attitudes toward computers, male students maytend to focus on functionality of computer tools more than female (Bain & Rice, 2006), leading to their higher appreciation of the usefulnessof MyNote in this study. For future educational training, female students will need more time and practice to become familiarized with theannotation.

With more and more people involved in online learning programs as an approach to lifelong learning, taking notes online might be ananticipated online learning behavior. MyNote demonstrates an example ofWeb 2.0 annotation system and shows great potential to enhanceonline learning. Due to the limitation of the study, it is suggested that to discover annotation behaviors, the impact of MyNote on learningoutcomes, and student annotation interaction for future examination. In addition, the content analysis of annotations and conversationsamong annotations might be helpful for instructors to understand the student thinking process and improve learning object presentationsthrough the analysis. Also, since discussion forums are quite common in online learning environments, it is suggested to compare theeffectiveness between MyNote and discussion forums. Additionally, the interface design of MyNote should be continually improved andtested with different methods, such as think aloud protocol or interviews, in order to meet the users’ needs.

Acknowledgments

We would like to thank the peer reviewers for their valuable and constructive comments and suggestions. This work was supported inpart by National Science Council (NSC), Taiwan under Contract NSC-96-2520-S-194-002-MY3.

References

Ahern, T. C. (2005). Using online annotation software to provide timely feedback in an introductory programming course. Paper presented in 35th ASEE/IEEE Frontiers inEducation Conference, October 19–22, 2005, Indianapolis, IN.

Alexander, B. (2006). Web 2.0 – a new wave of innovation for teaching and learning? EDUCAUSE Review, 141(2), 32–44.Annotea. (2001). Annotea project. retrieved from. http://www.w3.org/2001/Annotea/.Avouris, N., Komis, V., Margaritis, M., & Fiotakis, G. (2004). An environment for studying collaborative learning activities. Educational Technology and Society, 7(2), 34–41.Azouaou, F., Chen, W., & Desmoulins, C. (2004). Semantic annotation tools for learning material. Paper presented in International Workshop on Applications of Semantic Web

Technologies for e-Learning, SW-EL04’.Bain, C. D., & Rice, M. L. (2006). The influence of gender on attitudes, perceptions, and uses of technology. Journal of Research on Technology in Education, 39(2), 119–132.Barak, M., Herscoviz, O., Kaberman, Z., & Dori, Y. J. (2009). MOSAICA: a Web-2.0 based system for the preservation and presentation of cultural heritage. Computers and

Education, 53(3), 841–852.Bates, A. W. (2005). Technology, e-learning and distance education (2nd ed.). New York: Routledge.Brown, A. L., & Smiley, S. S. (1978). The development of strategies for studying texts. Child Development, 49(4), 1076–1088.Cadiz, J. J., Gupta, A., & Grudin, J. (2000). Using web annotations for asynchronous collaboration around documents. In Proceeding of CSCW 2000. Retrieve Nov. 1 2011 from

http://research.microsoft.com/en-us/um/redmond/groups/coet/annotations/trs/00-44.pdf.Clark, C. J. (1998). Let your online learning community grow: 3 design principles for growing successful Email Listervs and online forums in educational settings. San Diego State

University. Retrieved Dec. 9, 2003 from. http://www.noendpress.com/caleb/ olc/3Principles_Online_Comm.pdf.Colley, A., & Comber, C. (2003). Age and gender differences in computer use and attitudes among secondary school students: what has changed? Educational Research, 45(2),

155–165.Downes, S. (1999). Creating an online learning community. Edmonton: Virtual School Symposium. Retrieved July 17, 2008 from. http://www.slideshare.net/Downes/creating-

an-online-learning-community.Gupta, S., Condit, C., & Gupta, A. (2008). Graphitti: An annotation management system for heterogeneous objects. Paper presented in International Conference on Data Engi-

neering (ICDE), 2008, April 7–12, Cancun, Mexico.Head, A. J. (1999). Design wise – A guide for evaluating the interface design of information resources. Medford, NJ: Cyberage Books.Hwang, W. Y., & Wang, C. Y. (2004). A study on application of annotation system in Web-based materials. In Proceedings of the 8th Global Chinese Conference on Computers in

Education (GCCCE 2004).Hwang, W. Y., Wang, C. Y., & Sharples, M. (2007). A study of multimedia annotation of web-based materials. Computers and Education, 48(4), 680–699.Jonassen, D. H. (1996). Computers in the classroom: Mindtools for critical thinking. Englewood Cliffs, N.J: Merrill.Jonassen, D. H., Peck, K. L., & Wilson, B. G. (1999). Learning with technology – A constructivist perspective. NJ: Prentice Hall, Inc.Jones, N., Blackey, H., Fitzgibbon, K., & Chew, E. (2010). Get out of MySpace! Computers and Education, 54(3), 776–782.Karahalios, K., & Donath, J. (2003). Scale, form and time: Creating connected sociable spaces. MIT Media Lab. Retrieved Dec. 1, 2003 from. http://persona.www.media.mit.edu/

SMG.Klien, D. C., O’Neil, H. F., & Baker, E. L. (1998). A cognitive demands analysis of innovative technologies. CSE Technical Report 454. National Center for Research on Evaluation

Standards, and Student Testing (CRESST).Laurillard, D. (2005). e-learning in higher education. In P. Ashwin (Ed.), Changing higher education: The development of teaching and learning. London: Routledge.LeeTiernan, S., & Grudin, J. (2001). Fostering engagement in asynchronous learning through collaborative multimedia annotation. Proceedings of Interact, 472–479.Liaw, S. S., Chen, G. D., & Huang, H. M. (2008). Users’ attitudes towardWeb-based collaborative learning systems for knowledge management. Computers and Education, 50(3),

950–961.Marshall, C. C. (1997). Annotation: from paper books to the digital library. In Proceedings of the Second ACM Conference on Digital Libraries (pp. 131–140).Marshall, C. C., & Brush, A. J. B. (2004). Exploring the relationship between personal and public annotations. In Proceedings of the 2004 ACE/IEEE Conference on Digital Libraries

(JCDL’04) (pp. 349–357).Mayer, R. E. (2001). Multimedia learning. New York: Cambridge University Press.Nielsen, J. (1993). Usability engineering. Academic Press Inc.Nokelainen, P., Kurhila, J., Miettinen, M., Floreen, P., & Tirri, H. (2003). Evaluating the role of a shared document-based annotation tool in learner-centered collaborative

learning. In Proceedings of the 3rd IEEE International Conference on Advanced Learning Technologies (ICALT’03) (pp. 200–203).O’Hara, K., & Wilensky, R. (1997). A comparison of reading paper and on-line document. In Proceedings of CHI 1997 (pp. 335–342). ACM Press.O’Reilly, T. (2005). What is web 2.0. Retrieved June 18, 2008 from. http://www.oreillynet.com/pub/a/oreilly/tim/news/2005/09/30/what-is-web-20.html.

Y.-C. Chen et al. / Computers & Education 58 (2012) 1094–1105 1105

Prensky, M. (2001). Digital natives, digital immigrants. Horizon, 9(5), 1–6, NCB University Press.Quade, A. M. (1996). An assessment of retention and depth of processing associated with notetaking using traditional pencil and paper and an on-line notepad during

computer-delivered instruction. In Proceedings of Selected Research and Development Presentations at the 1996 National Convention of the Association for EducationalCommunications and Technology (pp. 559–570).

Robert, C. A. (2009). Annotation for knowledge sharing in a collaborative environment. Journal of Knowledge Management, 13(1), 111–119.Su, A. Y. S., Yang, S. H., Hwang, W. Y., & Zhang, J. (2010). A Web 2.0-based collaborative annotation system for enhancing knowledge sharing in collaborative learning

environments. Computers and Education, 55(2), 752–766.Torres, R. J. (2002). Practitioner’s handbook for user interface design and development. NJ: Prentice Hall.Treu, S. (1994). User interface evaluation: A structured approach. NY: Plenum Press.Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Cambridge University Press.Wang, Q. (2009). Design and evaluation of a collaborative learning environment. Computers and Education, 53(4), 1138–1146.Webb, N. M., Nemer, K., Chizhik, A., & Sugrue, B. (1995). Using group collaboration as a window into students’ cognitive processes. CSE Technical Report 404. National Center for

Research on Evaluation Standards, and Student Testing (CRESST).Wolfe, J. (2002). Annotation technologies: a software and research review. Computers and Composition, 19, 471–497.