international conference of educational technology · improving the usability of e-learning user...
TRANSCRIPT
“Toward smart approaches to education: bridging learning theory, technology application, and teaching practice”
INTERNATIONAL CONFERENCE OF EDUCATIONAL TECHNOLOGY
ICET2013ICET2013INTERNATIONAL CONFERENCE OF
EDUCATIONAL TECHNOLOGY
November 23. 2013GwangGaeTo Building 8th / 15th Floor
SEJONG UNIVERSITY
�����A4.indd 1 2013-11-19 �� 5:48:55
1
- I -
Table of Content
Welcoming Address - Insook Lee (President of KSET, Sejong University) .......................................................3
Program Table............................................................................................................5
Plenary SessionHistory, Trends, and Issues in Educational Communications and Technology
- Marcus Childress (President of AECT, Emporia State University)..................................7
Keynote SpeechCognitive Load Theory and Educational Technology
- John Sweller (University of New South Wales and Honorary Professorial Fellow University of Wollongong).............................................................................................17
Invited PresentationAn Organization Model for Ubiquitous Learning Resource From Learning Object to Learning Cell
- Shengquan Yu (Beijing Normal University)...................................................................30
Designing Telepresence System for Distance Learning in Hyper-Aged Society
- Atsushi Hiyama (University of Tokyo)...........................................................................59
Teaching to Vitalize, Rather than Neglect, Students’ Motivation
- Johnmarshall Reeve (Korea University).........................................................................76
Concurrent Session I Virtual Worlds in Education: Changes Of Student’s Perspectives And Learning Outcome - Mihwa Kim (Seoul National University).........................................................................88
Using a Virtual Tutee System to Promote Academic Reading Engagement- SeungWon Park (Texas A&M University) / ChanMin Kim (University of Georgia Athens)......93
Factors influencing students’ adoption of MyGuru2 asynchronous online discussion in Malaysia- Mahizer Hamazah (Sultan Idris Education University)..................................................98
- II -
Relationships among Learners’ Efficacy Beliefs, Perceptions on Scaffolding, Learning Participation and Achievement in Team Project-Based Learning- Youngsoo Kim (Ewha Womans University) / Heeok Heo (Sunchon National University)
/ Youngsun Yang (Kwandong University)...................................................................104
CSCL Scripts for the Collective Working Memory Effect- Jihyun Si (Hanyang University)...................................................................................108
Fostering Students’ Team Shared Mental Model and Team Satisfaction using Debate in the College Classroom- Myongnam Jun (Daegu Haany University) / Heather L Allen (Daegu Hanny University)...112
Multimedia Learning for Speaking Fluency - Joohee Son (Columbia University)...............................................................................118
The Transfer of the Foreign Language Curricular Goals and the Implications for ELP Curricular Development- Luping Zhang (China University)................................................................................123
Concurrent Session IITechnology, Connectedness, and Learning in the Digital Age: A Conceptual Framework for Digital Learning Standards- HyeJeong Kim (Chung-Ang University) / Wonseok Suh (Chung-Ang University) /
Hanho Jeong (Chongshin University) / Youngju Lee (Korea National University of Education) / Hae-Deok Song (Chung-Ang University)................................................134
Transforming Teaching and Learning Innovation: The Center for Teaching and Learning (CTLE) Model- Thapanee Thammetar (Silpakorn University) / Chattiwat Wisa (Silpakorn University) /
Ruangrit Nammon (Silpakorn University) / Bangthamai Eknarin (Silpakorn University)....139
Relationship or Content First? - MiJar Lee (Gwangju National University of Education)................................................143
The Effects of Informal mentoring on Organizational Commitment and Organizational Citizenship Behavior- YoungRan Yoo (Ewha Womans University) / Myunghee Kang (Ewha Womans University)
/ JiHyun Kim (KRIVET) / Jiwon You (Ewha Womans University).................................149
Improving the Usability of E-Learning User Interfaces: Affordance-Based Design- Hae-deok Song (Chung-Ang University).....................................................................155
Developing Design Principles of Emotional Interface for Self-Regulated Learning in an E-Learning Environment- Cheolil Lim (Seoul National University) / Taejung Park (Seoul National University) /
Wonjoon Hong (Seoul National University) / Jungeun Park (Seoul National University)....161
- III -
Design of Curriculum Management System- Innwoo Park (Korea University) / Wonsuk Shin (Korea University) / SongYi Beak
(Korea University) / HyeYeong Kim (Korea University)...............................................169
Mobile Phone Use, & Lifelong Learning.- Ken Morrison (Hannam University).............................................................................173
Smart Use of LMS in Higher Education: Viewing Students’ Perceptions in a Framework of Activity Theory- Yeonjeong Park (Ewha Womans University) / IlHyun Jo (Ewha Womans University)...178
Design and Practice of Project-Based Collaborative Learning Between Korean and Japanese University Students- Shinichi Sato (Nihon Fukushi University) / Makoto Kageto (Nihon Fukushi University)
/ Jeeheon Ryu (Chonnam National University)............................................................185
Learner-Centered and Collaborative Learning through Designing Digital Textbook for Music Curriculum in South Korea- Dong Yub Lee (KICE) / Sahoon H. Kim (KICE) / Ji Hyun Park (KICE)........................190
Effects of the Types of Communication Media on Collaborative Problem Solving Tasks- Hyewon Kim (Dankook University) / Minjeong Kim (Dankook University) / MiYoung Lee
(Dankook University)...................................................................................................197
Student SessionThe Effects of Creativity and Flow on Learning through the STEAM Education on Elementary School Contexts- Boram Cho (Ewha Womans University) / Jeongmin Lee (Ewha Womans University)..206
The Effects of Desirable Difficulties on Collaboration Load and Learning Outcome in Collaborative Learning Environment- Yoonmee Kim (Hanyang Unversity) / Dongsik Kim (Hanyang University)...................211
The Effects of Academic Emotions on Motivation in e-Learning- Seungho Kim (Visang Education) / Insook Lee (Sejong University)............................215
How Do the Level of Complex Learning Task and the Part-task Sequencing Affect on Mental Model, Cognitive Load, and Learning Time?- Kyungjin Kim (Hanyang University) / Dongsik Kim (Hanyang University)...................220
Predictability of Presence on Learning Persistence and Learning Satisfaction in Facebook -Based Collaborative Learning Environment- Hyunmin Chung (Ewha Womans University) / Sungeun Oh (Ewha Womans University)
Jiyoon Moon (Ewha Womans University) / Jeongmin Lee (Ewha Womans University)......224
- IV -
Case study of the Need Assessment and Usability Issue in Designing and Developing e-Book- Hyungju Lee (Chonnam National University) / Sinok Kim (Chonnam National
University) Gwansun Hong (Chonnam National University) / Sanha Kang (Chonnam National University) Jeongah Woo (Chonnam National University) / Yoojin Hong (Chonnam National University)....................................................................................229
The Effects of Simulation Game-Based Learning on Academic Emotions and Achievement- Yunha Jung (Ewha Womans University) / KyuYon Lim (Ewha Womans University)...233
The Effects of Part-task Sequencing and the Level of Element Interactivity on Schema Automation and Cognitive Load- Hyejeong Lee (Hanyang University) / Dongsik Kim (Hanyang University)...................238
Effect of Conversational Gesture of Pedagogical Agent and Visual Cueing on Task Comprehension and Eye Fixation- Jewoong Moon (Chonnam National University) / Jeeheon Ryu (Chonnam National
University)...................................................................................................................242
Assessment of Virtual Patients on Realistic Performance- Sun Kim (Chonnam National University) / Jeeheon Ryu (Chonnam National University)....246
Usability Study of Visual Dashboard as Learning Analytics Interventions- Kunhee Ha (Ewha Womans University) / Sohye Lim (Ewha Womans University) /
Il-Hyun Jo (Ewha Womans University).........................................................................249
The Effects of Regulatory Learning Strategies on Collaboration Load and Collaboration Outcomes in Computer-Supported Collaborative Learning- Hyojin Lee (Hanyang University) / Dongsik Kim (Hanyang University).......................256
Improvement of Score Reading Skill By Music Composing Class with SMART Education- Hyerin Lee (Chunchon National University of Education)...........................................261
The Effect of Awareness Information on Affect-based Trust in Collaborative Problem-solving Learning: A Pilot Study- Jongsuk Song (Hanyang University) / Dongsik Kim (Hanyang University)..................266
Student’s Perception on Learning Analytics Dashboard (LAD) Presenting Online Activities in LMS- Stephanie Kang (Ewha Womans University) / Yeonjeong Park (Ewha Womans University)
/ Il-Hyun Jo (Ewha Womans University).....................................................................269
Student Session
205
The Effects of Creativity and Flow on Learning through the STEAM
Education on Elementary School Contexts
Boram Cho
Student
Ewha Womans University
Seoul, Korea
Jeongmin Lee
Professor
Ewha Womans University
Seoul, Korea
ABSTRACT
This study aims to examine the effects of STEAM education on elementary school student‟s
creativity (creative problem solving, creative personality) and flow on learning. STEAM education is
composed of 5 strands: Science, Technology, Engineering, Art, and Mathematics. The STEAM education
provides convergence education to explore diverse thinking and achieve future convergence human
resources. Previous studies on STEAM education have been done on the model development and concept
formulation. There was very little application research. In addition, test subjects were usually middle
school and high school students. Therefore, this study focused on the elementary school students to
investigate the effect of STEAM lessons. This study made STEAM lesson plans that strengthen the
linkages themes among the subjects. It helps students acquire the creative design and emotional experience.
Based on this purpose of study, there are two research hypotheses. First, STEAM education Improves
creativity (creative problem solving, creative personality) on elementary school students. Second, STEAM
education enhances the flow on learning on elementary school students. The subjects in this analysis were
6th graders, two classes from elementary schools. Each class was taught for 45minutes during 8weeks by
the same teacher and performed 3tests as time goes by. After the test, we examine the changes in student‟s
creativity and flow on learning. Creativity was measured by two aspects; cognitive aspect and emotive
personality. Applied statistical methods were two independent samples t-test. As the result, there were
significant differences in creativity (creative problem solving, creative personality) and flow on learning
through the STEAM education. The result indicates that STEAM education was helpful to improve
creativity (creative problem solving, creative personality) and flow on learning.
Keywords: STEAM education, Creativity (creative problem solving, creative personality), Flow on
Learning
INTRODUCTION
The wave of change occurs in fast pace in the information age. Even though the world united as one
with rapid technology development, new complex problems occur by cultural diversity, environmental
diversity, and multiple values. Based on these phase of times, Ministry of Education and Science
Technology(MEST) published Steam education to improve the elementary school student's creativity
improvement in December 17th, 2010. STEAM education stands for convergence education in area of
206
Science, Technology, Engineering, Art, and Mathematics, and it aims for developing talented person with
creativity and personality who can solve various problems in rapidly changing society (MEST, 2010).
STEAM education adopted in Korea to learn association of theoretical principles with reality by
linking science, mathematics, and technology which are difficult subjects for students to arts and
engineering. Students can practice of adopting in reality through engineering and technology, and they
can grow as talented creative person through sensibility of arts (Korean Educational Development
Institute, 2012).
In addition, new rising keyword in our society is creativity and communication (Noh & Ahn, 2012).
There was a limitation on developing talented person with personality of communication with STEM
education that developed in United States, so STEAM education that cultivates creativity and personality
linking imagination and sensibility with technological education was suggested.
Educational outcome of STEAM education can be divided as cognitive side and emotive side.
Improvement of problem solving, creativity, cooperative learning, concentration in subject, and critical
thinking was positively influenced by educational outcome of cognitive aspects (Kim et al., 2011;
Kim&Kim, 2012; Shin et al., 2013). As emotive side, there are positive influences on learner‟s interest,
motivation (Bae, 2011, Kim et al., 2011; Kim & Kim, 2012; Moon, 2009), and attitude (Bae, 2011, Moon,
2009).
Currently, basic research such as concept establishment and model development is in progress
from STEAM education, and it lacks research in program development and application. In addition, test
subjects were usually middle school and high school students. Program development and dissemination
of STEAM education is priority issue for invigorating STEAM education from elementary school
teacher's research (Kum & Bae, 2012). Also, the research from HeSook Han and HwaJung Lee (2012)
stated that STEAM program development and dissemination is the most needed issue for helping
teacher‟s comprehension of STEAM education. In addition, existing STEAM education programs were
lack of links between subjects (Kim, 2012). Looking at the existing problems of subject, technical
education lacks of scientific principle explanation and science education needed to strengthen the linkages
themes among the subjects. Math education focused problem solving oriented class (Baek, 2011). Thus,
according to the passage, this study is needed to make a STEAM lesson plan that based on real life
contents and strengthen the linkages themes among the subjects. After that, we examine the effects of
STEAM education about creativity and flow on learning. Creativity was measured by two aspects;
cognitive aspect and emotive personality. Cognitive aspect measured by creative problem solving and
emotive personality measured by creative personality. The purpose of this study is to analyze about
effects of STEAM education on elementary school contexts.
Research problems
Based on this purpose of study, there are three research questions as follows:
First, does STEAM education improve creative problem solving on elementary school students?
Second, does STEAM education improve creative personality on elementary school students?
Third, does STEAM education enhance the flow on learning on elementary school students?
METHOD
Sample and procedures The subjects in this analysis were 6th graders, two classes from elementary schools. Each STEAM
class was taught for 45minutes, once per week, during 8weeks by the same teacher. The developed
STEAM program was reviewed from 3 elementary specialists. Before and after the instruction, we
examine the changes in student‟s creative problem solving, creative personality, and flow on learning.
Paired t-tests were used for data analysis methods.
207
Measures Creative problem solving
Creative problem solving is developed by Jeong(2008). It is created by Korean Educational Development
Institution in 2001year, and then MI research team of Seoul National University developed „easy creative
problem solving development research(1) in 2004year. It consists of 20 questions. The question contents
are specific areas of knowledge, thinking skills, understanding and mastery of the technology, divergent
thinking, critical reasoning and motivation. The questions were rated on Likert response scale (1=“to a
very slight extent or not at all to 5=”to a very large extent”). A higher score indicates higher level of
creative problem solving. Cronbach α reliability for this scale was .90 in this study.
Creative personality
Creative personality is developed by Ha (2000). It was used to measure creative personality for
Elementary students. It consists of 22 questions. The contents are curiosity, self-confidence, imagination,
patience/obsession and humor. The questions were rated on Likert response scale. Cranach α reliability
for this scale was .88 in this study.
Flow on learning
The flow on learning developed by Seok & Kang (2007) was used to measure flow on learning in
elementary school students. It consists of 35 questions. The sub factors are combination of challenges and
skiils, clear goals, specific feedback, the integration of action and awareness, sense of control,
concentration challenge, loss of consciousness, distorted sense of time, experience and self-purposive.
The questions were rated on Likert response scale. Cronbach α reliability for this scale was .94 in this
study.
Results
<Table1> Change of creative problem solving
n M SD df t p
Pre test 45 67.04 11.91 44 -4.24 0.000
*
Post test 45 73.11 13.40
*P<.05
Based on Table 1, the pre-test average was 67.04 and standard deviation was 11.91. The post-test
average was 73.11 and standard deviation was 13.40. For difference between pre and post test results,
this study verified statistical significance, t statistic value was -4.24, significance probability was .000
(p<.05). The result indicated that STEAM education improve creative problem solving significantly.
<Table2> Change of creative personality
n M SD df t p
Pre test 45 70.90 11.07 44 -2.40 0.021
*
Post test 45 73.80 13.39
*P<.05
208
Based on Table 2, the pre-test average was 70.90 and standard deviation was 11.07. The post-test
average was 73.80 and standard deviation was 13.39. For difference between pre and post test results,
this study verified statistical significance, t statistic value was -2.40, significance probability was .021
(p<.05). The result indicated that STEAM education improve creative personality significantly.
<Table 3> Change of flow on learning
n M SD df t p
Pre test 45 69.09 12.92 44 -4.02 0.000
*
Post test 45 75.47 13.79
*P<.05
Based on Table 3, the pre-test average was 69.09 and standard deviation was 12.92. The post-test
average was 75.47 and standard deviation was 13.79. For difference between pre and post test results,
this study verified statistical significance, t statistic value was -4.02, significance probability was .000
(p<.05). The result indicated that STEAM education improve flow on learning significantly.
CONCLUSION
The purpose of this study is examining the effects of creativity and flow on learning through
STEAM education on elementary school context. There is a brief summary of results. This study
conducted making a STEAM education lesson plan and then taught STEAM education program 45min/
once a week for 8weeks. After the 8weeks, the results showed that there were statistically significant
(p<.05) differences in creativity (creative problem solving, creative personality) and flow on learning.
Therefore, the results have implications that the STEAM education was helpful to improve creativity and
flow on learning on elementary school students. The results of the study provide useful information for
future researches on STEAM education. The previous research also supports current study‟s results. The
following is suggestions of future research directions. First, future research should be developed
structured model and theoretical framework. Second, STEAM education should reflect teacher, student
and parents‟ needs of the program in the development stage. Third, The STEAM education should create
a long-term integrated curriculum as project 21 in US.
REFERENCES
Bae, S. A. (2011). The Development and Application of Activity-Centered STEM Education
Program of Electricity, Electronics Technology area in Middle School. The Journal of Korean Institute of
Industrial Education, 36(1), 1-22.
Baek, Y. S., Park, H. J., Kim, Y. M., Noh, S. K., Park, J. Y., Lee, J. Y., Jeong, J. S., Choi, Y. H.,
Han, H. S.(2011). STEAM Education in Korea. Journal of Learner-Centered Curriculum and Instruction,
11(4), 149-171.
Ha, J. H. (2000). A Development Creative Personality Scale. Korean Educational Research
Assorciation, 14(2), 187-210. Han, H. S., Lee, H. J. (2012). A Study on the Teachers‟ Perceptions and Needs of STEAM
Education. Journal of Learner-Centered Curriculum and Instruction , 12(3), 573-603.
209
Jeong, U. Y. (2008). (The) Effects of Squeak Etoys based informatics education on elementary
school student's creative problem solving ability. Unpublished master's thesis, Choongbuk: Korea
National University of Education.
Kim, J. A., Kim, B. S., Lee, J. H., Kim, J. H. (2011). A Study of Teaching-Learning Methods for the
IT-Based STEAM Education Model With Regards to Developing People of Interdisciplinary Abilities.
The Korea Society for Fisheries and Marine Sciences Education, 23(3), 445-460.
Kim, T. H., Kim, J. H. (2012). A Development of Android Application for Physics Learning Based
on STEAM . The Korea Society for Fisheries and Marine Sciences Education, 24(1), 25-33.
Korean Educational Development Institute (2012). Understanding of STEAM education through
actual application case. 2012(2). 1-53. from
http://edpolicy.kedi.re.kr/EpnicForum/Epnic/EpnicForum02Viw.php?PageNum=3&S_Key=&S_Menu=
Ac_Code=D0010201&Ac_Num0=13090
Kum, Y. C., Bae, S. A. (2012). The Recognition and Needs of Elementary School Teachers about
STEAM Education. Korean Institute of Industrial Educators, 37(2), 57-75.
Ministry of Education and Science Technology:MEST (2010). Creative talented person and
advanced science technology with future Korea. 2011year working report.
Moon, D. Y. (2009). A Case Study on Elementary Students' Attitudes toward Engineering and
Engineering Problem Solving: Through Applying the Education Program of STEM Integration Approach.
Korean Educational Research Association, 22(4), 51-66.
Noh, S. W., Ahn, D. S. (2012). A Study on Direction of Development in STEAM Education. The
Education Research, 10(3), 75-96.
Seok, I. B. (2007). Construction of Flow on Learning: Scale, Personality, Condition, Participaation.
Unpublished doctoral dissertation, Kyeongbuk: Kyeongbuk University.
Shin, J. H., Nam, K. J. D., Kim, Y., Park, S. S., Jo, J. B., Lee, Y. M., Han, J. Y. (2013).
Development and Adaptation of Art-Oriented STEAM Converged Education Program through Toy-
Drama. Journal of Learner-Centered Curriculum and Instruction, 13(1), 215-240.
Shin, Y. J., Han, S. K. (2011). A Study of the Elementary School Teachers' Perception in
STEAM(Science, Technology, Engineering, Arts, Mathematics) Education. The Elementary Science
Education, 30(4), 514-523.
210
The effects of desirable difficulties on collaboration load
and learning outcome in collaborative learning environment
Younmi Kim
Doctoral Student
Hanyang University
Seoul, Korea
Dongsik Kim
Professor
Hanyang University
Seoul, Korea
ABSTRACT The purpose of this study was to investigate how collaboration load and learning outcome were
affected by the experience of desirable difficulties and the way of designing the experience. We
suggested that problematizing approach might be advantageous for learners who were required to
conduct learning tasks in collaborative environment, although many studies indicated that
scaffolding strategies were useful. 135 students in three classes were participated in this study and
classes were assigned to each group with full script, simple script and delayed full script. The results
revealed that there were significant differences in task load, process satisfaction and flow of
collaboration load categories and no significant differences between individual learning outcomes.
Based on the results, this study discussed the effects of desirable difficulties and implications for
further study.
Keywords: desirable difficulty, collaboration load, collaborative learning environment
INTRODUCTION
Supporting and guiding learners in collaborative learning environment may not always guarantee
better learning outcomes. Some learners can benefit from various supports such as aids, tools, and scripts,
and others not.
There are two approaches for design of collaborative learning (Reiser, 2002): structuring approach
(such as guiding learning process, providing tool that restrict or lead a particular learning activities) and
problematizing approach (to make something in students’ work more problematic). Structuring
approaches provide supports for learners to avoid difficulties that can hinder their learning, whereas
problematizing approaches let them encounter some challenges that may be exposed in real world.
Majority of studies have focused on the structuring approach, but some studies revealed that too much
supports could have negative effects on learning outcomes.
Guiding learners can be advantageous to collaborative learning when unexperienced learners receive
some help from learning supports to perform a collaborative task. Although supporting learners in groups
with various scaffolding strategies and methods can be productive, some studies have shown that guiding
learners do not always lead to positive results of learning process.
Kapur(2008) has introduced the concept of ‘productive failure’ and his researches are based on
problematizing approach. Learners who were in lack of support in collaborative phases and experienced
productive failure showed higher scores with respect to individual achievement, even though they failed
in their collaborative task (Kapur, 2008; Kapur & Kinzer, 2009; Kapur & Bielaczyc, 2012). Also, studies
with script theory addressed that learners’ efforts to perform their tasks themselves without support could
211
be helpful in learning outcomes (Makitalo, Weinberger, Hakkinen, & Fischer, 2005; Fischer, Kollar,
Stegmann, & Wecker, 2013).
Based on problematizing approach, this study aim to provide that ‘desirable difficulties(this word
means insufficient and problematic situation)’ may have positive effects on learning and can be designed
with to improve collaborative learning.
Research questions
Research questions are as follows:
1. To what extent is the collaboration load affected by experience of desirable difficulties and the
way of designing desirable difficulties in collaborative learning environment?
2. To what extent are the individual and collective outcomes affected by experience of desirable
difficulties and the way of designing desirable difficulties in collaborative learning environment?
METHODS
Participants and Research design
Participants were N=135, freshmen from ‘Career Planning and Self-Development’ classes at
Dankook University in Korea. Three different classes were randomly assigned to group1, 2 and 3. Then,
students of each class were assigned to 3 or 4-person teams and required to solve a collaborative task.
Each group was in one of three conditions including different script types(full script, simple script, and
delayed full script) and required to submit the worksheet at the end of the class. Pretest and posttest were
asked to respond before and after class individually.
<Table 1>. The research design
Phase group1 group2 group3
1 pre-test (individually)
2 full script simple script Nothing full script
3 posttest(individually & collectively)
Instructional Conditions
All groups went through same procedures, during same time, by an instructor. Phrase2 were
conducted during normal classes. The students in a team were got together and presented with one of
three different types of script through smart phone. Then they were asked to complete the form of
worksheets on smart phone through discussion collaboratively. Students were permit to refer to teaching
materials and search on the internet by phone as needed.
Instruments and Data Analysis
The students were measured on collaboration load using a questionnaire with a five-point scale and
on individual and collective outcomes using scored worksheets. We adopted the questionnaire developed
by Jeong & Kim(2012) and graded the worksheets based on criterion developed by the instructor and
subject matter expert
One-way ANOVA tests were used to compare the collaboration load and individual and
collaborative learning outcomes. Then post-hoc Scheffe tests were also used to identify which two groups
had significant differences statistically.
212
RESULTS
Collaboration Load
One-way ANOVA showed that there were significant differences in task load, process satisfaction
and flow of collaboration load categories. And it was confirmed that there were significant difference in
task load between group1 and group3 by post-hoc Scheffe test. Group1 was significantly higher than
group2 and 3 in process satisfaction and flow.
<Table 2>. Collaboration load
Source SS df MS F p
Task load Between Groups 9.032 2 4.516 5.064 .008
Within Groups 117.710 132 .892 Total 126.743 134
Germane load Between Groups .934 2 .467 1.110 .333
Within Groups 55.509 132 .421 Total 56.443 134
Intrinsic load Between Groups 2.101 2 1.050 3.073 .050
Within Groups 45.128 132 .342 Total 47.229 134
Process satisfaction
Between Groups 5.377 2 2.689 5.600 .005 Within Groups 63.368 132 .480
Total 68.745 134
Performance satisfaction
Between Groups 1.358 2 .679 1.607 .204 Within Groups 55.801 132 .423
Total 57.159 134
Extraneous load
Between Groups 3.732 2 1.866 3.048 .051 Within Groups 80.802 132 .612
Total 84.534 134
flow Between Groups 9.787 2 4.893 8.695 .000
Within Groups 74.289 132 .563 Total 84.076 134
Individual and Collaborative Outcomes
The results showed no significant differences between mean scores of individual learning outcomes.
The individuals’ mean scores of group1were higher than others, but the difference was not significant
(p>.05). Otherwise, group 1 and 3 obtained much higher scores than group2 in collaborative outcomes
significantly.
DISCUSSION
In summary, we found that the experience of desirable difficulties affected collaboration load partially
and collaboration outcomes but individual outcomes. From the findings, it may not be sure that the
experience of desirable difficulties was meaningful in collaborative learning environment. Nevertheless,
we need to focus on the result that there was no significant difference between individual learning
outcomes although the experience of desirable difficulties seems to be negative for learning processes.
Despite learners who experienced desirable difficulties felt more pressure by task, were unsatisfied with
learning process, and less concentrated on learning, the individuals achieved similar result compared to
learners who have not experienced desirable difficulties. The findings may imply that desirable
difficulties have possibilities for facilitating learners in some way. Thus, it is necessary to investigate how
the learners get a benefit from desirable difficulties and how the experience of desirable difficulties can
be designed to improve learning in further study.
213
REFERENCES
Dillenbourg, P. (2002). Over-scripting CSCL: The risks of blending collaborative learning with
instructional design. In P. A. Kerschner(Eds.), Three worlds of CSCL: Can we support CSCL? (pp.
61-91). Heerlen: Open University of the Netherlands.
Fischer, F., Kollar. I., Stegmann, K., & Wecker, C.(2013). Toward a script theory of guidance in CSCL.
Educational psychologist, 48(1), 56-66.
Hyojeong, jeong, & Hyewon Kim(2012). An Exploratory Validation for the Constructs of Collaboration
Load. Journal of Educational Technology. 28(3), 619-640.
Kapur, M.(2008). Productive Failure. International Journal of Cognition and Instruction. 26:379-424.
Kapur, M., & Kinzer, C. K.(2009). Productive failure in CSCL groups. International Journal of
Computer-Supported Collaborative Learning, 4, 21-46.
Kapur, M., & Bielaczyc, K.(2012). Designing for productive failure. J. of the learning sciences. 21, 45-83.
Makitalo, K., Weinberger, A., Hakkinen, P., & Fischer, F.(2005). Online collaborative learning: Will
collaborationi scripts reduce uncertainty? Educational technology, 45(5), 25-29.
Reiser, B. J.(2002). Why scaffolding should sometimes make tasks more difficult for learners. In G.
Stahl(Ed.) Computer support for collaborative learning: foundations for a CSCL community,
proceedings of CSCL 2002(pp.255-264), Boulder, CL, Jan. 7-11, 2002.
Weinberger, A., Stegmann, K., Fisher, F., & Mandl, H.(2007). Scripting argumentative knowledge
construction in computer-supported learning environments. In F. Fisher, I. Kollar, & J. M.
Haake(eds.), Scripting computer-supported collaborative learning(192-211). NY: Springer.
Renkl, R. & Atkinson, R. K.(2010). Learning from Worked-Out Examples and Problem Solving. In J. L.
Plass, R. Moreno, & R. Brunken(eds.), COGNITIVE LOAD THEORY(91-108). NY: Cambridge
University Press.
214
The Effects of Academic Emotions on Motivation in e-Learning
Seungho Kim
Visang Education
Seoul, Republic of Korea
Insook Lee
Professor
Sejong University
Seoul, Republic of Korea
ABSTRACT
The purpose of this study is to examine the impact of students’ experienced academic emotions on
motivation in e-Learning. In terms of motivation, this study identifies meanings and importance of
students’ academic emotions in e-Learning and draws implications for instructional design. With middle
school students who enrolled an online mathematics course, the researcher measured academic emotions
that study participants experienced during the online lectures and then measured motivational factors after
online lecture by using self-reported instruments. The result indicates that correlations exist among several
academic emotions and motivational factors. Similar to advanced research, furthermore, frustration and
boredom negatively affected on student’s motivation and pride positively influenced on their motivation.
Keywords: Academic emotions, Emotions, Instructional design, Motivation, e-Learning
INTRODUCTION
Student’s academic emotions experienced in learning have not received sufficient attention of
instructional designers or educational researchers. But different types of emotion that student experience
in learning have impacts on concentration, memory, performance and learning process or achievement as
well(Cacioppo & Gardner, 1999; Lee, 2012). Especially, many motivation theories and related researches
include student’s emotional experience as major factor that influences to his/her motivation (Ainely,
2006; Linnenbrink, 2006; Meyer & Turner, 2006; Pekrun, 2006; Schutz et al., 2006). But most of the
advanced researches were conducted in face-to-face learning. e-Learning environment has different
factors that arouse student’s emotions from classroom (Hara & Kling, 2000; Ng, 2001; O’Regan, 2003;
Wegerif, 1998, etc.). Therefore we need to analyze impact of academic emotions that student experience
in e-learning process, and which elements of e-Learning environment have relevance to it.
THEORETICAL BACKGROUND
Emotion and Learning Emotion is individual and psychophysiological experience. It arises in interaction between human
being and surrounding environment (Myers, 2004). Also, human acts in a certain way under the influence
of their emotion (Linnenbrink & Pintrich, 2002). Learning situation is where learner can experience
diverse emotions. Considering the emotions’ role and function, emotions that learner experienced directly
and indirectly can effect to learning process (for example, learner’s concentration, self-regulation,
215
motivation, etc.) and academic achievement (Nummenmaa, 2007; Pekrun et al., 2002; Schutz & DeCuir,
2002).
Regarding cause of arising academic emotions, Elliot(1999), Frijda(1993), Lazarus(1991),
Pintrich(2000), Schutz & Davis(2000), Smith(1991) asserted that learner’s appraisal of learning
environment triggers his/her academic emotions and it related to their academic goal. Agreeing to this
assertion, Pekrun(2000, 2006) integrated various theories related to emotions and suggested Control-
Value Theory. Control-Value Theory assumes that learner conducts two types of cognitive appraisals
related to his/her emotional experience. One is that “can I control my academic activities or
achievement?” The other is that “This academic activities or achievement is valuable to me? (Positive or
Negative)” In short, Learner generally experiences emotion going through the process: 1) Event or
situation stimulates learner. 2) Learner conducts two types of cognitive appraisals. 3) Learner experiences
some kind of academic emotions.
e-Learning and Academic Emotions
e-Learning environment involves factors that cause learner’s emotions. Some of these factors are not
in face-to-face classroom. These relate to computer and web technology. e-Learner experiences various
emotions, because e-Learning environment has properties that arouse student’s academic emotions:
computer media, information network, web cyberspace. And emotions can be quantitatively as well as
qualitatively changed according to that how much e-Learning environment satisfies learner’s need or
expectation (Brave & Nass, 2002). O’Regan(2003) interviews 11 university students who participated in
online courses. He found that students experience frustration because internet connection and instability
of web site, complexity of web site’s structure, and so on. And students can experience anxiety, fear,
concern because time delay, submitting assignment to website, digital literacy, and so on. Also, they feel
excitement because convenience, accessibility to information for learning, connectivity to colleagues,
feedback of instructor. Pride arises during online learning due to that they make public their assignment to
website and feel tutors effectively manage their online course, get positive feedback from professor or
colleagues.
In research of emotional intelligence, there is evidence that appear importance of learner’s emotions.
Lee(2012) measured emotional intelligence of university students who participated in e-learning courses
and verified that their emotional intelligence have effect to academic achievement. In result, empathy of
emotional intelligence relevantly predicted to cognitive and attitude domain of academic achievement.
Also, comparing off-line and on-line learning, on-line cooperative learning, Kang & Goo(2007) found
that, in on-line learning, emotional intelligence significantly predicted learner’s academic achievement.
Academic Emotions and Motivation Weiner(1985) asserted attribution theory that human experiences emotion when they think about the
cause of consequence after behavior. And then, emotions influence their choice in next behavior.
Ford(1992) emphasized that learner’s emotions play a significant role in process of managing his/her
motivation. He found that academic emotions come from interaction between learners, learners and
teacher. Also, emotions play role as indicator that provides important information to motivation and
cognitive management. Pekrun have carried out many researches about emotions in learning. He assumed
that learner’s experienced academic emotions influence his/her motivation. Following the same purpose,
Pekrun et al.(2002) developed Achievement Emotions Questionnaire: AEQ) that measures learner’s
emotions(enjoyment, hope, pride, relief, anxiety, shame, anger, hopelessness, boredom) in classroom,
learning, test and examined middle school and university students’ emotion related to learning and
motivation, learning strategies, cognitive load, self-regulation, learning achievement. Analyzing the
results, they found that positive emotions and inner/external motivation have positive correlation. In
contrast, negative emotions and motivation have negative correlation. Since then, Pekrun(2006)
integrated empirical researches and theories, and suggested conceptual model that include cognitive
appraisal and learner’s emotions, academic achievement. This model appears that learner’s emotions
influence their learning strategies and cognitive resource, self-regulation, motivation (see pekrun, 2006).
These researches show that learner’s experienced emotions have relevance to their motivation, and
provide important information about their learning process.
216
RESEARCH METHODS
In that point of view, in order to investigate the impact of academic emotions (enjoyment, pride,
anxiety, frustration, boredom, and learning environment anxiety from Pekrun et al, 2002) on learning, the
current research analyzes relations between academic emotions and motivational factors
(intrinsic/extrinsic goal orientation, task value from Pintrich et al., 1991) that advanced research (Ainely,
2006; Linnenbrink, 2006; Meyer & Turner, 2006; Pekrun, 2006; Schutz et al., 2006) have suggested.
With middle school students who enrolled an online mathematics course, the researcher measured
academic emotions that study participants experienced during the online lectures and then measured
motivational factors after online lecture by using self-reported instruments. Correlation and regression
have been conducted for analysis of these data.
RESEARCH RESULTS
The result indicates that correlations exist among several academic emotions and motivational factors.
Furthermore, learning environment anxiety and frustration predict intrinsic goal orientation (R2 = 81%, p
< .01). Learning environment anxiety, frustration and pride predict extrinsic goal orientation (R2 = 72%, p
< .01) and learning environment anxiety and boredom predict task value (R2 = 52%, p < .01). Students
who participated in online mathematics course differently experienced both positive and negative
emotions to students who join in face-to-face classroom.
<Table 1>. Effect of Academic Emotions to Motivation
Motivation Academic Emotions R R2 t p
intrinsic goal orientation learning environment anxiety
0.9 0.81 18.276
* 0.00
frustration -8.321* 0.00
extrinsic goal orientation
learning environment anxiety
0.848 0.719
13.565* 0.00
frustration -9.004* 0.00
pride 3.267* 0.002
task value learning environment anxiety
0.721 0.52 9.258
* 0.00
boredom -8.459* 0.00
* p < .01
CONCLUSION
These results come from environmental properties and subjective characteristics that the online
course has. Similar to advanced research, furthermore, frustration and boredom negatively affected on
student’s motivation and pride positively influenced on their motivation. In contradistinction to advanced
research, but, learning environment anxiety negatively predicted to all factors of motivation. In our
research, learning environment anxiety related to guidance of e-learning system, stability of web site,
guidance of online course, etc. And a degree of learning environment anxiety was not serious for learning
(M = 3.91, SD = 0.96). These results present that learner who was motivated sensitively reacts to e-
Learning environment: guidance of e-learning system, stability of web site, guidance of online course, etc.
The present study results implicate that there is a need to considering learner’ emotional experience
when instructor designs e-Learning courses. Course designer checks e-Learning environment (especially,
related to technology) whether there are some problems that cause learner’s negative emotions (for
example, anxiety or frustration, anger, etc.) and resolve it. On the other hand, instructional strategies or
elements that boost learner’s positive emotions were included in e-Learning course or system (for
example, professor or colleagues’ positive feedback).
217
REFERENCES
Ainley, M. (2006). Connecting with Learning: Motivation, Affect and Cognition in Interest Processes.
Educational Psychology Review, 18(4), 391-405.
Brave, S., & Nass, C. (2002). Emotion in human-computer interaction. In J. Jacko, & A. Sears (Eds.),
Handbook of human-computer interaction (pp. 251-271). New York: Lawrence Erlbaum Associates.
Cacioppo, J. T., & Gardner, W. L. (1999). Emotions. Annual Review of Psychology, 50, 191-214.
Elliot, A. J. (1999). Approach and avoidance motivation and achievement goals. Educational Psychologist,
34, 149–169.
Ford, M. E. (1992). Motivating humans: Goals, emotions, and personal agency beliefs. Newbury Park,
California: Sage.
Frijda, N. H. (1993). The place of appraisal in emotion. Cognition and Emotion, 7, 357–387.
Hara, N. & Kling, R. (2000). Students’ distress with a Web-based distance education course: An
ethnographic study of participants’ experiences. Center for Social Informatics. Retrieved October
11, 2010, from http://www.slis.indiana.edu/CSI/wp00-01.html
Kang, Myunghee & Goo, Nahyun (2007). Predictive Validity of Emotional Intelligence on Various
Achievement in Blended Learning Environment. Research Institude of Curriculum Instruction,
11(1), 235-255.
Lazarus, R. S. (1991). Progress on a cognitive–motivational–relational theory of emotion. American
Psychologist, 46, 819–834.
Lee, In-sook. (2012). Research trends on emotional factors in the e-Learning context. International
Journal for Educational Media and Technology, 6(1), 14-22.
Linnenbrink, E. A. (2006). Emotion Research in Education: Theoretical and Methodological Perspectives
on the Integration of Affect, Motivation, and Cognition. Educational Psychology Review, 18(4),
307-314.
Linnenbrink, E. A., & Pintrich, P. R. (2002). Achievement goal theory and affect: An asymmetrical
bidirectional model. Educational Psychologist, 37, 69–78.
Myers, David G. (2004). Theories of Emotion. Psychology: Seventh Edition. New York: Worth
Publishers.
Meyer, D. K., & Turner, J. C. (2006). Re-conceptualizing Emotion and Motivation to Learn in Classroom
Contexts. Educational Psychology Review, 18(4), 377-390.
Ng, K-C. (2001). Using e-mail to foster collaboration in distance education. Open Learning, 16(2), 191–
200.
Nummenmaa, M. (2007). Emotions in a web-based learning environment. Turku: Turun yliopisto.
Doctoral thesis.
O'Regan, K. (2003). Emotion and e-learning. Journal of Asynchronous Learning Networks, 7(3), 78-92.
Pekrun, R. (2000). A social cognitive, control-value theory of achievement emotions. In J. Heckhausen
(Ed.), Motivational psychology of human development (pp. 143-163). Oxford, UK: Elsevier.
Pekrun, R. (2006). The Control-Value Theory of Achievement Emotions: Assumptions, Corollaries, and
Implications for Educational Research and Practice. Educational Psychology Review, 18(4), 315-
341.
Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students' self-regulated
learning and achievement: A program of qualitative and quantitative research. Educational
Psychologist, 37(2), 91-106.
Pintrich, P. R.(2000). Multiple goals, multiple pathways: The role of goal orientation in learning and
achievement. Journal of Educational Psychology, 92, 544-555.
Pintrich, P. R., Smith, D. A., Garcia, T., & McKeachie, W. J. (1991). A manual for the use of the
motivated strategies for learning questionnaire (MSLQ). Ann Arbor, Michigan: National Center for
Research to improve Post Secondary Teaching and Learning (NCRIPTAL). The University of
Michigan, Eric Document Reproduction Service, ED 338122.
Schutz, P. A., & Davis, H. A. (2000). Emotions and self-regulation during test taking. Educational
Psychologist, 35, 243–255.
218
Schutz, P. A., & DeCuir, J. T. (2002). Inquiry on emotions in education. Educational Psychologist, 37(2),
125-134.
Schutz, P. A., Hong, J. I., Cross, D. I., & Osbon, J. N. (2006). Reflection on Investigation Emotion in
Educational Activity Settings. Educational Psychology Review, 18(4), 343-360.
Smith, C. A. (1991). The self, appraisal and coping. In C. R. Snyder & D. R. Forsyth (Eds.), Handbook of
social and clinical psychology: The health perspective (pp. 116–137). Elmsford, NY: Pergamon.
Wegerif, R. (1998). The social dimension of asynchronous learning networks. Journal of Asynchronous
Learning Networks, 2(1), 34-49.
Weiner, B. (1985). An attributional theory of achievement motivation and emotion. Psychological
Review, 92, 548-573.
219
How Do the Level of Complex Learning Task and the Part-task
Sequencing Affect on Mental Model, Cognitive Load,
and Learning Time?
Kyungjin Kim
Doctoral course student
Hanyang University, Korea
Dongsik Kim
Professor at the department of Educational Technology
Hanyang University, Korea
ABSTRACT
This study was conducted to examine effects of the level of task complexity and the method of part-
task sequencing on a learner's mental model and cognitive load in the complex learning task. Task
complexity has been divided into three levels, high, medium, and low. The method of part-task sequencing
has been categorized into two, backward chaining with snowballing and simple backward chaining. In
order to examine the effects, the Computer-based Complex Task Performance Supporting (CTPS)
program was designed and developed. This study has suggested an alternative solution to realize a
complex learning task effectively by presenting instructional design principles with different part-task
sequencing methods depending on a task complexity level in implementing a complex task.
Keywords: Complex task, Part-task sequencing, Cognitive load
INTRODUCTION
A learning task taught in the school may be deemed well-structured in general, well-defined, and
comparatively easy to learn and teach, but is far from authentic problems in a real world. On the contrary,
numerous problems we are facing out of the school are not only not-well defined, but also ill-structured.
One significant goal of learning is to apply what is learned to solve a real problem in an ordinary
routine of life. Learning a simple task may help a learner obtain fact and information which he/she has not
been aware of, but it is not enough to solve a real task which requires complex elements such as
knowledge, skill, and attitude. Therefore, nowadays, researchers emphasize to provide learners in the
learning process with experiences on a complex task (Merrill, 2002; van Merriënboer, Kester, & Pass,
2006).
Recent studies have mainly dealt with instruction materials and instructional design that reduce a
learner's extraneous load in learning a complex task (Merrill, 2002; van Merriënboer, Kester, & Pass,
2006). It simply analyzed which variable influenced more on a cognitive load or not.
In this study, we aim to suggest which part-task sequencing method is most appropriate for effective
performance of a complex task, depending on the complexity level of complex task.
CTPS program development
220
A tool for this study, the CTPS Program (Complex Task Performance Supporting Program) has
been developed. There three main reasons: First, a complex task is composed of the same level of
multiple task types, and shall be designed by using a scaffolding method in which much more
instructional supports are provided in the beginning, but gradually reduced for performing learning in
each task type. Second, a complex task needs detailed learning guidance at an initial stage of learning,
and supports must be consistently provided available anytime during learning, as required. Third, most of
online lecture sites are composed only in order which is presented in the subject, and there are almost no
such sites that have part task sequencing. If there is no standardized program in sequencing part tasks for
a complex task, its result may be distorted due to inconsistency of treatment depending on researcher's
capability. The concrete and standardized program must be developed to ascertain the validity of study.
THEORETICAL BACKGROUND
Task Sequencing The task sequencing means that it determines a task class by categorizing learning tasks from easy
one to hard one, and sequences it. This theory has recently more developed and expanded to have 10 steps
(Van Merriënboer, & Kirschner, 2007). This study will focus on their opinions on task sequencing in
terms of these views.
Whole task sequencing
High
Complexity
Simple
Backward
GABCDEF-FABCDE-EABCD-DABC-CAB-BA-A
Medium
Complexity
Simple
Backward
EABCD-DABC-CAB-BA-A
Low
Complexity Simple
Backward
CAB-BA-A
High
Complexity
Snowballing
Backward
GABCDEF-FGABCDE-EFGABCD-DEFGABC-CDEFGAB-BCDEFGA-ABCDEFG
Medium
Complexity
Snowballing
Backward
EABCD-DEABC-CDEAB-BCDEA-ABCDE
221
Low
Complexity
Snowballing
Backward
CAB-BCA-ABC
Figure 1. Illustration of Sequencing
Figure 1 is part-task sequencing methods which a researcher illustrated according to a complexity
level.
METHODOLOGY
Groups were undergraduate students enrolled in liberal arts classes of community college located at
Seoul city. Each group underwent pre-tests to confirm group homogeneity; then assigned into high-level,
medium-level, and low-level groups in terms of task complexity; and again divided into two sets of
groups depending on which part-task sequencing method was utilized. They were informed with the
functions of main menus on the CTPS program. The CTPS program was given to the groups, they were
expected to experience different learning scenario by the natures of treatment groups. Finally post-tests
were administered to measure students’ mental model development, their cognitive loads, and their
learning time.
CONCLUSION
We found three main results. First, part-tasks sequencing with the backward chaining with
snowballing would best method instructional design in complex learning. Second, when complex learning
tasks are completed through the backward chaining with snowballing, mental model development may be
intensified but vastly more so would cognitive load, with possible results of the learner avoiding a similar
task in the future or even quitting the task during. Hence, instructional design should be formatted with
strong considerations on which part-task sequencing method is adequate for the particular learner. Third,
learners performing complex tasks should be given enough time, and even more so when the complexity
level is higher. Moreover, sequencing part-tasks in the backward chaining with snowballing that requires
extraneous cognitive load would require longer learning time. Instructional design should reflect these
results to offer enough learning time to the learner.
In summary, the underpinning purpose of this study is to overcome the strategic limitations of
previously suggested instructional designs addressing complex learning and to determine which approach
to part-task sequencing would be most effective when completing complex learning tasks with varying
complexity levels. Conclusions are drawn that for elevated performance in completing complex tasks,
instructional methods should be designed to accommodate the level of Complexity part-tasks. And for
effectual mental model development, detailed instructive-learning strategies that motivate learners to
immerse in task completion should be developed and applied in practice. This study reports on the design
principles and strategies of instruction in complex learning environments with regard to learners’ mental
model, cognitive load, and learning time, which provides guideline to building alternative solutions in
instruction.
REFERENCES
Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development,
50(3), 43-59.
222
van Merriënboer, J. J. G., Kester, L., & Paas, F. (2006). Teahing complex rather than simple tasks:
Balancing intrinsic and germane load to enhance transfer of learning. Applied Cognitive Psychology,
20, 343-352.
van Merriënboer, J. J. G., & Kirschner, P. A. (2007). Ten steps to complex learning: A systematic
approach to four-component instructional design. Mahwah, NJ: Lawrence Erlbaum Associcates.
223
Predictability of Presence on learning persistence and learning satisfaction
in Facebook-based Collaborative Learning Environment
Hyunmin Chung [email protected]
Graduate student
Ewha Womans University
Seoul, Republic of Korea
Sungeun Oh
Graduate student
Ewha Womans University
Seoul, Republic of Korea
Jiyoon Moon
Graduate student
Ewha Womans University
Seoul, Republic of Korea
Jeongmin Lee
Professor
Ewha Womans University
Seoul, Republic of Korea
ABSTRACT
The purpose of this study was to analyze the effect of presence (cognitive, social, and emotional) on the
persistence and satisfaction in SNS-based collaborative learning environment. 95 college students who took a course
“career education” class participated in this research. Data were analyzed by correlation analysis and multiple
regression analysis. The research findings are summarized as follows: first, emotional presence predicted significantly
learning persistence, while cognitive presence and social presence did not predict learning persistence in SNS-based
collaborative learning environment. Second, cognitive presence and emotional presence predicted significantly
learning satisfaction. On the other hand, social presence did not predict learning satisfaction in SNS-based
collaborative learning environment. The implication of this study and future research were discussed in the conference.
Keywords: Social Network Service (SNS), Social learning, Facebook, presence, persistence, satisfaction.
224
PURPOSE OF THE RESEARCH
Social Network Service, an online flatform that enables to generate and strengthen communication, information
share and social relationship between users, has been rapidly spread with a diffusion of smartphone based on its
property of universal accessibility (Pimmer, Linxen & Grohbiel, 2012). Social Network Service like Facebook or
Twitter showed a diverse use in the fields of marketing, media, politics and society (Kang, Hhan & Kim, 2012).
In a domain of educational practice and research, several positive result of study (Ajjan & Hartshorne, 2008;
Mzaman & Usluel, 2010) supported that learners could actively participate, interact, cooperate and share within the
learning environment where all characteristics of SNS, including openness, cooperation and information sharing,
function as a tool for educational purpose.
Among the available service, Facebook users were distinctive in that the majority of users are in the age of 18 to
24 who were in higher education, establishing a sense of familiarity and having an experience by using their online
social activities (Choe & Kwon, 2013). Therefore, Facebook users were expected to share their feelings under the
educational setting of SNS through daily language use. The situation allowed emotional interaction with the increase
of familiarity and friendship. Based on the previous study, the learning effect would be maximized in the process of
achieving shared learning goal in those setting, arousing positive interaction between learner and teacher as well as
learner themselves provided by Facebook interface (Lim, 2011).
The opposite study regarded Facebook as an inappropriate educational tool or defines a limit undermining its
learning effect. Selwyn (2009) revealed that University students tend to use Facebook as a space for casual talk or
social relationship management so that Facebook use was highly seen as „backstage‟ rather than as „front-stage‟.
Moreover, a number of learners preferred face to face setting to Facebook-based one (Baran, 2010) and only a few
shows their active use for learning. Those who did not want to join the learning group explained the reason for
security and privacy issues (Kop, Fournier, & Mak, 2011).
Previous studies have shown conflicting results and few investigate the effectiveness with an experimental
approach, therefore, this study empirically explored the persistence and satisfaction as learning outcome in Facebook-
based learning environment.
Moreover, undemanding condition to produce information freely and actively, to cooperate and to share for new
encourages a sense of presence (Witmer & Singer, 1998) which was defined as the subjective experience of being in a
certain place or an environment. According to Wang and Kang (2006), there were three intersecting domains of
presence in teaching-learning environment: cognitive, social, and emotional and each domain describes learning
content during learning, virtual presence of instructors or peers and self-awareness for emotion. Based on the existing
theory that high cognitive, emotional and social presences increase learning flow connected to successful learning,
this study suggested a model of presence-gaining strategies by analyzing underlying factors of learning presence in
Facebook-based collaborative learning environment.
First, do cognitive presence, social presence and emotional presence predict learning satisfaction in Facebook-
based collaborative learning environment?
Second, do cognitive presence, social presence and emotional presence predict learning persistence in Facebook-
based collaborative learning environment?
METHOD
Subjects & Procedure
225
Ninety-five students participated in this research who took career education course opened at a university in Seoul. The
course was required as academic liberal arts for the first and second year students and all participants had to use Facebook group
for collaborative group task consisted of class interaction such as materials, discussion, feedback and learning-log. Before
collaborative group task by using Facebook group, orientation session was provided.
Learning presence, satisfaction and learning persistence were measured and the results have been reliably analyzed, firstly,
each measuring instrument had item reliability established through Cronbach's α in order to verify internal consistency, secondly,
normality of the distribution was confirmed and then correlation analysis adopted to figure out the relationship between variables,
lastly, hypothetical research model was developed used for multiple regression analysis through SPSS.
Measurement
To measure learning presence, items developed by Wang and Kang (2006) were revised with the use of 5-point Likert scale.
The sense of cognitive presence consisted of 13 test items like „I am able to discuss what I learn from the course‟, 11 items for
social presence like „I feel like learn together with others‟ and emotional presence written as a question: „I easily express my
feeling on Facebook‟. Inter-item consistency had a Cronbach's α of .733 for cognitive presence, of .911 for social presence and
of .837 for emotional presence.
Also, learning satisfaction was measured with the revised version (five items, Cronbach's α = .875) of Shin & Chan (2004).
A sample of the questionnaire item as follows: “I felt a high sense of accomplishment through this course”. Lastly, learning
persistence was measured with the revised version (four items, Cronbach's α = .816) of Kim & Kang (2010) comprised of „I am
willing to take related course‟.
CONCLUSION
The purpose of this study was to analyze the effect of learning presence (cognitive, social, and emotional) whether they predict
learning outcome and investigate each variance‟s relative power of prediction in Facebook-based learning environment. To set
SNS-based collaborative learning environment, the research made use of Facebook group page and performed during 8 weeks.
The result of the study summarized as follows:
Among the sub-factors of learning presence-cognitive, social and emotional, values of two factors was meaningful;
cognitive (ß = .355, p <.05) and emotional (ß = .456, p <.05) aspects. The result showed that the higher competence for self-
cognition in learning, emotional state, the degree of expression and regulation hold, the more positive response for learning
experience followed. It was consistent with previous research that learners with cognitive presence or strategies for acquiring
cognitive presence felt more satisfaction toward their learning (Kang, Kim, Park, 2008), and the same result came from another
research about the effect of emotional presence toward satisfaction for learning (Kang, et, al, 2008).
Second, cognitive (ß = .217, p <.05) and emotional presence (ß = .152, p <.05) influenced on learning persistence. The
findings we obtained were consistent with the result that emotional presence as well as cognitive one meaningfully worked to
make learners persist on their learning (Joo, Kim, & Park, 2009), emotional presence (Kang & Kim, 2006).
Moreover, higher cognitive presence, control of emotional state and self-regulation competency were connected to learning
persistence; willingness to persist their own learning was increased. As a result, we confirmed learning persistence could be
changed depending on the cognitive and emotional presence. That is, the way to increase those factors should be regarded
importantly when designing Facebook based learning for learning persistence.
On the contrary to the fact that cognitive and emotional presence produce meaning value, social presence didn‟t obtain
meaningful result; satisfaction (ß = -.088, p >.05) and learning persistence (ß = .111, p >.05). There was a prior research
explaining that Facebook-based learning could strengthen social presence in that consolidate strong sense of solidarity and
intercommunication and connected to the outcome, comparing to the online or web-based learning environment. (Kang, Hhan, &
Kim, 2012; Omar, Embi, & Yunus, 2012) The result denied the expectancy towards the potential power of network learning.
The rationale behind the failure of prediction was that learners‟ resistance to use their profile information for teaching-
learning environment was unsuccessful to build social presence. In the light of this, Jones, Blackey, Fitzgibbon & Chew (2010)
referred to the fact that using their private account for social networking services with the same as for the learning environment
made leaners feel insecure; learners did not want to intersect the circles they involved. It suggested learners‟ scope of learning
environment needed to be separated from that of social life.
226
Moreover, a lack of understanding the concept of social presence occurred because of the number of the group member;
With 95 members on a group, the characteristic of group collaboration would be shown in a different way comparing to small
group based work. In fact, previous research about the effect of social presence to learning persistence or satisfaction confined
learning situation as small group or major class under 40. According to Jang & Chang (2013), small group discussion with 25
members took advantage of Facebook services in learning like instant responses, convenience, a sense of solidarity and
familiarity and thus, it created higher social presence, learning flow, satisfaction, learning achievement. Also, from the research
concerned with the effect of social presence. Cho, Han (2010) pointed out that social presence could successfully predict learning
achievement, participation and satisfaction. The experiment conducted on 200 learners attending a class opened at a cyber
university but divided into 10 groups to limit the number of participants in each group.
Through the previous and this research, considering social presence could be meaningful when designing SNS-based
teaching. Since a strategy-based approach enhancing social presence and its underlying factors to increase social presence-
coexistence, influences and cohesiveness effectively worked for small size group working collaboratively, it is meaningful to
apply the implication to SNS learning environment.
Finally, there are two limitations of this investigation that are important to be underscored: the data collected to measure
learning presence and missed or hidden factors related to the learner variables and learning environment. With regard to the first,
learners endorsed questionnaire items indicating their level of learning presence. If message analysis by quantitative method
implemented for the experimental design of this investigation, the result could be more reliable. Finally, as learning presence was
regarded as a complex structure, deep understanding of learning presence has to be preceded. While the research was not
designed to make this determination, it is nevertheless important to encourage further research to distinguish underlying factors to
increase learning presence and its interaction between the characteristic of instructors and leaners, the process of learning and
learning environment. Then, meaningful implication could be given for a practical and various course design.
REFERENCES
Ajjan, H., & Hartshorne, R. (2008). Investigating faculty decisions to adopt Web 2.0 technologies: Theory and empirical tests.
The Internet and Higher Education, 11(2), 71-80.
Cho, E., & Han, A. (2010). The Effect of Social Presence on Learning Flow and Learning Effects In Online Learning Community.
The Journal of Educational Information and Media, 16(1), 23-43.
Choe, J., & Kwon, S. (2013). Teaching Strategies in Learning with Facebook to Improve Learner‟s Interaction. Journal of
Lifelong Learning Society, 9(2), 155-180.
Jang, E., & Chang, H. (2013). The differences between Web-based debate and Social Network Service(SNS)-based debate on
social presence, learning flow, satisfaction and self-evaluation. Journal of Educational Technology 29(1), 1-25.
Jones, N., Blackey, H., Fitzgibbon, K., & Chew, E. (2010). Get out of My Space!. Computers & Education, 54(3), 776-782.
Joo, Y., Kim, E., & Park, S.,Y. (2009). The Structural Relationship among Cognitive Presence, Flow and Learning Outcome in
Corporate Cyber Education, The Journal of Educational Information and Media, 15(3) 21-38.
Kang, M., Hhan, J., & Kim, J. (2012). Exploring students‟ learning experiences of Facebook in Service-Learning. The Korean
Journal of Educational Methodology Studies. 24(4), 771-796.
Kang, M., Kim, J., & Park, M. (2008). Investigating Presence as a Predictor of Learning Outcomes in E-learning Environment.
In J. Luca & E. Weippl (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and
Telecommunications 2008 (pp. 4175-4180). Chesapeake, VA: AACE.
227
Kang, M., & Kim, M. (2006). Investigation the Relationship among Perceived Social Presence, Achievement, Satisfaction and
Learning Persistence in e-Learning Environment. Journal of Educational Technology, 22(4), 1-27.
Kang, M., Park, M., Jung, J., & Park, H. (2009). The Effect of Interaction and Learning Presence on Learning Outcome in Web-
Based Project Learning. The Journal of Educational Information and Media, 15(2), 67-85.
Kim, J., & Kang, M. (2010). Structural Relationship among Teaching Presence, Learning Presence, and Effectiveness of e-
Learning in the corporate setting. Asian Journal of Education. 11(2). 29-56.
Kop, R., Fournier, H., & Mak, J. S. F. (2011). A pedagogy of abundance or a pedagogy to support human beings? Participant
support on massive open online courses. International Review of Research in Open and Distance Learning, 12(7), 74-93.
Lim, K. (2010). A Case Study on a Learning with Social Network Services on Smartphones: Communication Contents and
Characteristics Analyses of the Applications. The Korean Journal of Educational Methodology Studies. 22(4), 91-114.
Omar, H., Embi, M. A., & Yunus, M. M. (2012). ESL Learners‟ Interaction in an Online Discussion via Facebook. Asian Social
Science, 8(11), 67.
Pimmer, C., Linxen, S., & Gröhbiel, U. (2012). Facebook as a learning tool? A case study on the appropriation of social network
sites from mobile phones in developing countries. British Journal of Educational Technology, 43(5), 726-738.
Selwyn, N. (2009). Faceworking: exploring students' education‐related use of Facebook Learning. Media and Technology, 34(2),
157-174.
Shin, N. M., & Chan, J. (2004). Direct and indirect effects of online learning on distance education. British Journal of
Educational Technology, 35(3), 275-288.
Wang, M. J., & Kang, M. (2006). Cybergogy for engaged learning: A framework for creating learner engagement through
information and communication technology. In M. S. Khine (Ed.), Engaged Learning with Emerging Technologies (pp.
225-253): New York: Springer Publishing.
Witmer, B. G., & Singer, M. J. (1998). Measuring presence in virtual environments: A presence questionnaire. Presence.
Teleoperators and virtual environments, 7(3), 225-240.
228
A Case Study of The Need Assessment and Usability Issue
in Designing and Developing e-book
Hyungju Lee
Undergraduate Student
Chonnam National University (Gwangju, Korea)
SinOk Kim
Undergraduate Student
Chonnam National University (Gwangju, Korea)
Gwansun Hong
Undergraduate Student
Chonnam National University (Gwangju, Korea)
SanHa Kang
Undergraduate Student
Chonnam National University (Gwangju, Korea)
JeongAh Woo
Undergraduate Student
Chonnam National University (Gwangju, Korea)
YooJin Hong
Undergraduate Student
Chonnam National University (Gwangju, Korea)
ABSTRACT
The goal of this study is to assess the needs of international students and to evaluate the usability in
designing and developing e-book. The e-book is designed for the first arrival students at Chonnam
National University (CNU). This study will apply survey, interview and pilot-test with small sample size
to meet the purpose of e-book design so the result of study may not be appropriate for generalization.
However, this study will provide context-driven experience of designing and developing e-book for
international students. The value of this study will focus on what needs and usability issues should be
considered for the multicultural readers. Questions were made based on experience with international
students who visited CNU as exchange students. The study is only aimed at international students who
study at CNU in Gwangju, Korea.
The e-book covers the mainly campus life, culture differences, and hot places for lunch and dinner
near the university. That will be used for international students studying at CNU, as it contains a number
of contents about campus life of CNU.
This study will proceed processes of needs assessment and usability evaluation. These are the main
components in the interface of this research. First, needs assessment will be conducted to identify
international students' needs, desired conditions or wants. The assessment is to measure the actual
students' needs specifically and to shed some light on the e-books' contents which are suitable. It can also
improve the quality of e-book by containing international students' needs. Second, usability evaluation is
important to increase the utility of e-book. The evaluation depends on familiarity which the students has
with it. The less familiar the e-book is, the less usable it is. By performing a usability evaluation with
familiarity, the e-book will be optimized for international students..
Keywords: need assessment, usability, e-book
229
INTRODUCTION
An e-book is “an electronic version of a printed book which can be read on a personal computer or
hand held device designed specifically for the purpose” (K.T. Anuradha, 2006). As smart technology
becomes essential in daily life, students are no longer constrained by time and location. They can access
and share information anytime and anywhere. The number of international students at CNU has steadily
increased in recent years (Yang, 2009). Many have to spend weeks on becoming familiar with this
unfamiliar environment. As strangers, a personal and portable location-based service can help them to be
accustomed to their new environment and lead them to interesting sites. Although CNU has provided a
campus guide in PDF format and included many things for the students to feel more comfortable on the
campus, they have difficulties in using the facilities at CNU, treating their peers in class and selecting
restaurants around CNU owing to the lack of information. This study is the try to solve those problems by
applying „contiguity principle‟ and using „e-book‟.
METHOD
The target of this survey was the international students at CNU. The questionnaire was intended to
elicit information for needs assessment. The survey for needs assessment organized two sections. In the
first section, there were multiple questions about CNU’s facilities and functions. It also contained the
question about culture in class such as which one was more difficult when they treat or converse with,
Korean professors or students. In the next section, the participants answered which factors were
preferable or uncomfortable when they chose their food or restaurants. Off-campus life team used „Likert
scale‟ to measure the preference. After needs assessment, each team analyzed their data and proceeded
four steps, brainstorming, storyboarding, paper modeling, and developing e-book. The e-book was
applied „contiguity principle‟ for design and development.
RESULT
Needs assessment In the first section, the participants taking part in the survey were thirty three. In the second
section, the participants partaking in the survey were thirty two. The participants joining in the survey
were almost equal. The needs assessment discovered by on-campus life team was 1) the most visited
areas on campus, 2) the difficulty in treating Korean friends. The participants answered that B, C and D
section were the most visited areas. Each section is an area of CNU. The team tried to assume the reason
why the respondents visited those areas. A section was excluded when on-campus life team analyzed the
data because there are not many available facilities. The respondents probably went to B section because
there was a big lecture hall taking place many lectures. In addition to, the participants frequently visited B
section because there was a student dormitory in B section. If the participants had a free time to do
something, they would often go to D section because there was a playground to exercise. Interestingly,
there was little gap between C section and D section in the responses. The gap of the responses is just 0.3
percentage. In C section, there was the student union which has multiple functions for international
students to use such as medical checkup, post office and restaurants. The second assessment discovered
by the team was the difficulty in classroom culture. The students responded that they faced the difficulty
in treating their Korean friends rather than Korean professors. Comparing that 7 participants usually or
always met difficulty in treating Korean professors, 12 participants usually or always felt difficulty in
treating Korean students.
The needs assessment found by off-campus life team was that 1) taste and service were
important factors when they chose food and restaurants and 2) restaurants didn‟t give enough information
230
such as food ingredients, menu and tip. The respondents regarded taste and service as the important
factors. According to the Table.1, the mean of taste was 4.72 and that of service was 4.59. Interestingly,
the mean of distance was only stayed at 3.59 on Table.1. The respondents didn‟t consider how far the
restaurants are. The second assessment was that the participants thought the restaurants didn‟t provide
enough information. When off-campus life team gave the question, „Do you think the information about
food or restaurants was offered sufficiently?‟, fifteen of participants answered that they thought the
restaurants didn‟t give much information.
Development The development process consisted of 4 steps. In the first step, brainstorming, each team
discussed and decided contents, cognitive load theories, expectation for needs assessment in designing
and developing the e-book. The second step was storyboarding. After each team made non-example and
example, every team compared example applying „contiguity principle‟ with non-example not applying
„contiguity principle‟. The third step was paper modeling. Based on the storyboarding, each team made
hand-made book specifically before developing e-book. The final step was to develop and design e-book.
Although each team separately conducted needs assessment and the development of the contents of e-
book, the contents were finally merged. The contents of the e-book consisted of two parts and types. The
first part was information about Restaurants around CNU. The second part was the structure of the
student union. The teams developed the e-book into two types, but they couldn‟t evaluate the usability of
non-example and example.
<Table 1>. Preference of food & restaurant (n=32)
Options Mean SD
Service 4.59 0.50
Taste 4.72 0.52
Quality 4.53 0.57
Quantity 3.81 0.78
Distance 3.59 0.87
Communication 3.63 0.87
Price 4.25 0.76
Ingredient 4.00 0.88
CONCLUSION
The conclusion through this study is 1) difference from expectation to needs assessment, 2) theory-
driven approach and 3) limitation: usability issues about device compatibility. Before conducting needs
assessment, on-campus life team expected that international students felt more difficulty in treating
Korean professors than Korean students. However, that expectation didn‟t match needs assessment.
Rather, the students thought that they had more difficulty in treating Korean friends. Additionally, off-
campus life team anticipated that distance was a quite important factor when they chose food and
restaurants. Likewise, on-campus life team did, the expectation was not agreed. On the other hand, the
taste and service were more significant than the distance. The teams had a question about which one is
better between „theory-driven one‟ and „the looking good other‟. Referring to „e-learning and the science
of instruction‟, the teams identified theory-driven one is better than the looking good other. As a
limitation, during the steps in developing and designing e-book, the teams faced a problem because the e-
book was not well compatible with some electronic devices and applications. Even though „Galaxy tab‟
didn‟t well operate this e-book, „iPad‟ run well the e-book. At the beginning of this study, the teams were
231
scheduled to evaluate the usability of e-book. Owing to the problem with compatibility, the teams
couldn‟t conduct the usability evaluation.
REFERENCES
Clark, R. C., & Mayer, R. E. (2008). E-learning and the science of instruction: Proven guidelines for consumers and designers of multimedia learning. Wiley
232
The Effects of Simulation Game-Based Learning
on Academic Emotions and Achievement
Yun Ha JUNG
Master’s Student
Ewha Womans University
Seoul, Korea
Kyu Yon LIM
Assistant Professor
Ewha Womans University
Seoul, Korea
ABSTRACT
The purpose of this study is to examine the effects of simulation game-based learning on academic
emotions (positive, negative) and achievement (factual, conceptual, procedural knowledge acquisition). Sixty-
three learners (experimental group: 32, comparison group: 31) from a high school located in Seoul, Korea were
chosen to conduct an experiment. The results of the study demonstrate that there was a significant difference
between the comparison group and the experimental group in both positive and negative academic emotion.
However, there was no significant difference between the comparison group and the experimental group in
factual, conceptual, and procedural knowledge acquisition. These findings show that simulation game-based
learning brings more positive emotion and less negative emotion during learning compared to the instructor-led
lectures, although there were no significant effects on achievement.
Keywords: Simulation game-based learning, Academic emotion, Achievement
INTRODUCTION
There have been many efforts to create more effective, efficient, and engaging learning environment by
adopting educational technology. However there is still a gap between the learning design and the students called
digital native who prefers self-directed, enjoyable, and socially-connected experience (KERIS, 2012; Prensky, 2001).
In order to reduce this gap, researchers paid attention to a simulation game-based learning as an alternative teaching
method. Many early researches have focused on a game-based learning because of its attractive and entertaining
attributes (Anolli, Mantovani, Confalonieri, Ascolese, & Peveri, 2010; Garris, Ahlers, & Driskell, 2002), and situated
learning theory has insisted that simulation can make learning more meaningful (Choi & Hannafin, 1995; Howland,
Jonassen, & Marra, 2012). Simulation game-based learning is defined as a kind of instructional form using the critical
elements of both simulation and game, such as goal-oriented activities and competitive experiences for teaching and
learning (Baek, 2006). Previous research on simulation game has reported that there was a significant effect on
learners’ achievement (Akinsola, 2007; Sowunmi & Aladejana, 2013) and attitude toward learning (Akinsola, 2007).
However, there are some exceptions according to the types of games and instructional context (Sitzmann, 2011),
which requires further investigation on simulation game.
In this particular study, academic emotion has been selected as a major variable or interest, since students in
Korea have been struggling with academic stress due to highly competitive environment. According to Kim (2009),
academic emotion is an emotional state caused by specific subject of stimulation composed of enjoyment, pride,
angry, anxiety, and boredom. Many researchers who studies emotions claimed that academic emotions have
important function to affect student's motivation and learning strategy use to construct knowledge (Do, 2008; Yang &
Kim, 2010). Especially in the domain of game-based learning, researchers theoretically insisted that simulation game
tends to increase positive emotions such as enjoyment and pride, while reducing negative emotions such as angry,
233
anxiety, and boredom (Anolli et al, 2010; Astleitner & Leutner, 2000; Novak & Johnson, 2012). Based on this
theoretical approach, it is necessary to perform empirical study on academic emotion and simulation game-based
learning.
Therefore, this study aimed to examine the effects of a simulation game-based learning on academic emotions
and achievements. Specifically, the use of simulation game-based learning was the independent variable
(experimental group: simulation game-based; comparison group: lecture-oriented), while academic emotion (positive,
negative) and achievement (factual, conceptual, procedural knowledge) were dependent variables. Eventually, the
study results will provide practical implications for designing simulation game-based learning as well as theoretical
justifications for game and academic emotions.
Research questions 1. Are there any differences in academic emotions (positive, negative) between simulation game-based learning
group and traditional lecture-oriented learning group?
2. Are there any differences in achievement (factual, conceptual, procedural knowledge) between simulation game-
based learning group and traditional lecture-oriented learning group?
METHODS
Participants, 12 fonts, Bold) Sixty-three students from a high school located in Seoul, Korea were chosen through convenient sampling.
Among these, 32 were assigned to the experimental group with simulation game-based learning treatment, and 31
were assigned to the comparison group with traditional lecture-oriented treatment. Prior to the intervention,
researchers confirmed the homogeneity of the two groups in terms of prior level of academic emotions as well as
prior knowledge, meaning that there were no significant differences in both dependent variables between
experimental and comparison groups (pre-positive academic emotion: t = .057, p = .955; pre-negative emotion: t = -
1.481, p = .144; prior knowledge: t = -1.735, p = .088).
Treatments Basically, the content and the instructor were identical for both groups. The only difference was the practice
session: Simulation game-based group used simulation game, while lecture-oriented group used paper-pencil-based
exercise. The treatment used for the experimental group was a simulation game called 'Mock stock investment
simulation game' designed by the Bank of Korea. This game allowed students to virtually buy and sell stocks for 40
companies. Every single stock price was reflected by real stock trading market, and the return on investment made by
game players was calculated and announced every midnight. The instructor was able to control how much money the
students can make investment during the game, and the research participants were given 5,000,000 won at the
beginning of the simulation game.
Measurement instruments This study used three measurement instruments. First, in order to measure academic emotions, Academic
Emotion Questionnaire developed by Pekrun et al. (2011) was adopted. Positive academic emotions consisted of
items relevant to enjoyment and pride, and negative academic emotions consisted of items relevant to anger, anxiety,
and boredom. The Cronbach’s alphas calculated using the study data were .95 for positive emotion, and .94 for
negative emotion. Second, prior knowledge test used for checking the homogeneity of the two groups was developed
by two subject matter experts. Third, measurement instruments for the achievement for factual, conceptual, and
procedural knowledge acquisition were also developed by two subject matter experts. Prior knowledge test and
achievement test items were validated by another subject matter expert. Examples of measurement instruments are
illustrated in <Table 1>.
<Table 1> Measurement instruments
Variables Examples Scale # of
items
Academic
emotions
Positive Enjoyment: I enjoy being in class.
Pride: I am proud of the contribution I have
made in class.
Likert
5 16
234
Negative Anger: I feel anger welling up in me.
Anxiety: I worry the others will understand
more than me.
Boredom: Because I get bored, my mind begins
to wander.
24
Achievement Factual
Knowledge What is the appropriate terminology for
‘securities’?
10
points 10
Conceptual
Knowledge Which is incorrect about economic variables
and stock market?
10
points 10
Procedural
Knowledge What is the correct procedure to buy and sell
stocks in the market?
5
points 5
Procedure (Times New Roman, 12 fonts, Bold) Firstly, researchers assigned participants into experimental and comparison groups. Then all the participants
were provided with a 30-min orientation session for the research, and took pre-academic emotion questionnaire and
prior knowledge test. Secondly, each group participated in the intervention for 9 days, respectively. Experimental
group played a mock stock investment simulation game under the guidance of the instructor during their two 80-min
face-to-face sessions, and also played the game as homework by themselves. On the other hand, comparison group
received face-to-face lectures given by the same instructor, same learning contents, and also did paper-pencil-based
homework by themselves. Lastly, after the treatments completed, participants responded to the academic emotion
questionnaire and achievement test. The data were analyzed using independent sample t-test and MANOVA.
RESULTS
Effects on academic emotions Bold MANOVA on academic emotions revealed that there is a significant difference between the experimental group
and the comparison group in both positive and negative academic emotion (Wilks' = .820, P = .003). <Table 2>
summarizes the descriptive statistics and MANOVA results for two experimental conditions.
<Table 2> Descriptive statistics and MANOVA results on academic emotions
(n = 63)
Variable Treatment M SD F P η2
Academic
emotion
Positive Simulation game 4.09 0.639
10.162 .002* .143 Lecture 3.48 0.862
Negative Simulation game 2.27 0.624
9.134 .004* .130 Lecture 2.75 0.648
* p < .05
Effects on an Achievement MANOVA on achievement revealed that there are no significant differences between the experimental group
and the comparison group in factual, conceptual, and procedural knowledge (Wilks' = .977, P = .709). Descriptive
statistics and MANOVA results for two experimental conditions are described in <Table 3>.
<Table 3> Descriptive statistics and MANOVA results on achievement
(n = 63)
Variable Treatment M SD F p η2
Achievement
Factual Simulation game 7.22 1.773
.172 .680 .003 Lecture 7.42 2.062
Conceptual Simulation game 5.28 2.453
.095 .759 .002 Lecture 5.10 2.286
235
Procedural Simulation game 1.84 1.019
.585 .447 .009 Lecture 2.06 1.263
* p < .05
CONCLUSION
This study examined the effects of a simulation game-based learning on academic emotions and achievements.
In conclusion, the findings show that simulation game-based learning brings more positive emotion and less negative
emotion during learning compared to the instructor-led lectures, although there were no significant effects on
achievement.
In terms of academic emotions, similar to the previous studies, the results indicated that the simulation game-
based learning tends to bring more task values and more sense of control for students, which are relevant to the
positive emotions (Pekrun, 2006). In other words, there is a possibility that simulation game increased learners’
perception on task values because of its reality, and also provided a sense of control because of its participant-
oriented procedures (e.g. entering orders of selling and buying stocks). The results implied that these attributes
enabled by the simulation game are likely to create positive emotional environment. In terms of achievement, there is
a possibility that students had a cognitive overload because of its large amount of information to invest stocks (Sun &
Choi, 2013). In order to generate meaningful learning with simulation game, it is required for learners more time for
constructing knowledge. Although there was no significant effect on achievement, it still means that game-based
learning is surely more effective in the emotional aspect.
Limitations and implications for future research are suggested as follows: First, participants for this study were
sixty-three high school students, in Seoul, Korea, which requires caution for generalizing the results. Second, follow-
up studies need to further explore specific emotions such as pleasure, pride, anxiety, and boredom as separated
dependent variables, instead of using positive and negative categories. Third, it is necessary to consider other key
independent variables such as context of instruction and collaboration during simulation game-based learning.
REFERENCES (Times New Roman, 12 font, Bold, All Capital)
Akinsola, M. K. (2007). The effect of simulation-games environment on students achievement in and attitudes to
mathematics in secondary schools. The Turkish Online Journal of Educational Technology, 6(3), 113-119.
Anolli, L., Mantovani, F., Confalonieri, L., Ascolese, A., & Peveri, L. (2010). Emotions in serious games: From
experience to assessment. International Journal of Emerging Technologies in Learning, 5(3), 7-16.
Astleitner, H., & Leutner, D. (2000). Designing Instructional technology from an emotional perspective. Journal of
Research on Computing in Education, 32, 497-510.
Back, Y. K.(2006). Understanding and application of Game based learning. Seoul: Educational science.
Choi, J. I., & Hannafin, M. (1995). Situated cognition and learning environment: Roles, structures, and implications
for design. Educational Technology Research & Development, 43(2), 53-69.
Do, S. L. (2008). Issues and prospects of research on affect in education. The Korean Journal of Educational
Psychology, 22(4), 919-937.
Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and practice model.
Simulation & Gaming, 33(4), 441-467.
Gredler, M. E. (2004). Games and simulations and their relationships to learning. In D. H. Jonassen (Ed), Handbook
of research on educational communications and technology (pp. 571-581). Mahwah, NJ: Lawrence Erlbaum
Associates.
Howland, J. L., Jonassen, D., & Marra, R. M. (2012). Meaningful learning with technology (4th Ed.). Boston, MA:
Pearson Education.
KERIS (2012). Adapting education to the information age. Seoul, Korea: KERIS.
Kim, M. S. (2009). Emotion in Learning Context: Its Origins and Functions. Asian Journal of Education, 10(1), 73-
98.
Lindh, J., Hrastinski, S., Bruhn, C., & Mozgira, L. (2008). Computer-based business simulation games as tools for
learning: A comparative study of student and teacher perceptions. In Proceedings of the 2nd European
conference on games-based learning (ECGBL), 16–17 October 2008, Barcelona, Spain.
Novak, E., Johnson, T. E. (2012). Assessment of student's emotions in game-based learning. In D. Ifenthaler, D.
Eseryel., & X. Ge (Eds.), Assessment in game based learning: Foundations, innovations, and perspectives (pp.
379-399). NY: Springer.
236
Pekrun, R. (2006). The control-value theory of achievement emotions: Assumptions, corollaries, and implications for
educational research and practice. Educational Psychology Review, 18, 315–341.
Pekrun, R., Goetz, T., Frenzel, A. C., Barchfeld, P., & Perry, R. P. (2011). Measuring emotions in students’ learning
and performance: The academic emotions questionnaire (AEQ). Contemporary Educational Psychology 36, 36-
48.
Pekrun, R., Goetz, T., Frenzel, A. C., & Perry, R. P. (2011). Academic Emotions Questionnaire (AEQ). User's
manual (2nd ed.). Munich, Germany: Department of Psychology, University of Munich.
Prensky, M. (2001). Digital Game-Based Learning. NY: McGraw-Hill.
Sitzmann, T. (2011). Meta analytic examination of the instructional effectiveness of computer based simulation
games. Personnel Psychology, 64, 489-528.
Sowunmi, O., & Aladejana, F. (2013). Effect of simulation games and computer assisted instruction on performance
in primary science. Proceedings of the 2013 WEI International Academic Conference, USA, 10-15.
Sun, J, S., & Choi, W. (2013). An inquiry for the utilization strategies of cognitive tools in web-based PBL(Ploblem-
Based Learning) and an analysis of the effects on cognitive load. Journal of Educational Technology, 29(2),
349-382.
Yang, M. H., & Kim, E. J. (2010). The influence of emotional regulation on learning strategy: Mediated by
emotionality. The Korean Journal of Educational Psychology, 24(2), 449-467.
237
The Effects of Part-task Sequencing and the Level of
Element Interactivity on Schema Automation and Cognitive Load
Hyejeong Lee
Graduate Student
Hanyang University
Seoul, Korea
Dongsik Kim
Professor
Hanyang University
Seoul, Korea
ABSTRACT
This study is to reveal the most effective part-task sequencing according to the level of element
interactivity, and to find the appropriate part-task sequencing best suit for human cognitive structure. A
two-way ANOVA with schema automation and cognitive load as dependent variables, and the kind of
part-task sequencing and the level of element interactivity as independent variables was conducted. The
study results show that progressive chaining and snowballing chaining outperformed forward chaining
with regard to schema automation. However, there was no significant difference between progressive
chaining and snowballing chaining. Regarding cognitive load, higher intrinsic cognitive load was occurred
when the element interactivity is high compared to low element interactivity, regardless of the kind of
part-task sequencing. The lowest level of extraneous cognitive load and the highest level of germane
cognitive load were imposed when using progressive chaining.
Keywords: Part-task sequencing, Element interactivity, Cognitive load
INTRODUCTION
Element interactivity Complex learning task is characterized by high element interactivity, which means that learner have
to deal with a large number of interacting elements simultaneously. Element interactivity is described as
one of the sources of intrinsic cognitive load which closely related to the nature of learning task itself. It
was supposed that intrinsic cognitive load is relatively difficult or even impossible to be altered in
comparison with extraneous cognitive load (Sweller & Chandler, 1994; Sweller, van Merrienboer, & Paas,
1998). Thus, earlier studies regarding cognitive load mainly focused on reducing extraneous cognitive
load for effective instructional design (van Merrienboer & Sweller, 2005).
Pollock, Chandler, and Sweller (2002) were known as the first to put an effort to reducing intrinsic
cognitive load by sequencing method. They presented complex task through two phases which are
isolated element phase and interacting element phase: firstly learners studied individual information of the
whole complex task; in the second phase, learners were presented all the information elements at once.
The result of this study indicates that isolated-interacting elements technique is a useful instructional
238
sequencing method, especially for novice learners who suffer from high cognitive load due to the
limitation of their working memory.
There are some other researches that have investigated the effect of sequencing strategy on
decreasing intrinsic cognitive load (Clarke, Ayres, & Sweller, 2005; van Merriënboer, Kirschner, &
Kester, 2003). In the line of these kinds of efforts, this study also examined part-task sequencing as an
effective instructional technique for high element interactivity.
Part-task sequencing Part-task sequencing approach was initially proposed because it is hard to start learning with whole
authentic tasks which represent high complexity (Salden, Paas, & van Merrienboer, 2006). Human
working memory has a limited capacity for cognitive processing. If highly complex task is given, the
learner might feel overloaded, which can have a negative impact on learning (Sweller et al., 1998). Part-
task sequencing can decrease working memory load by decomposing a large multicomponent skills into
several separate components and reducing the complexity of the task.
Part-task sequencing is an effective and efficient method when learning complex task(Wickens,
Hutchins, Carolan, & Cumming, 2013). By breaking the complex task into individual elements, learners
have only to learn certain amounts of elements that can be processed in working memory at once, thus
intrinsic cognitive load would be decreased accordingly. Although part-task sequencing can be productive
for reaching isolated, specific objects, it does not always increase transfer when learners deal with whole
integrated objectives (van Merriënboer & Kirschner, 2007).
To increase learner’s understating and transfer over the complex task, learners should practice
whole elements together to learn how all relevant elements interact (Pollock et al., 2002). Snowballing
technique can be complement to such part-task sequencing as snowballing gives learners the opportunity
to practice whole complex task (van Merriënboer & Kirschner, 2007).
Part-task sequencing with snowballing technique look somewhat similar to Pollock et al. (2002)’s
‘isolated-interacting effect’. Even though Pollock et al. (2002)’s study gives us clear suggestion that
sequencing from isolated elements to interacting elements is advantageous for complex learning task, it
does not establish the exact guidelines indicating which ways can be effective when there are more than
two parts of isolated elements.
In this study, two part-task sequencing strategies within isolated-followed-by-interacting-elements
approach were examined to find out how different sequencing methods affect schema automation and
cognitive load when task complexity is converted into three isolated elements. Also this study was
investigated whether the effect of part-task sequencing strategies can be changeable according to the level
of element interactivity.
RESEARCH QUESTIONS
RQ1. To what extent do different part-task sequencing methods (forward, snowballing and
progressive chaining) and element interactivity (high and low) have an effect on participants’ schema
automation?
RQ2. How do different part-task sequencing methods (forward, snowballing and progressive
chaining) and element interactivity (high and low) affect cognitive load?
METHODS
Participants and Research Design To answer these research questions, a 2x3 factorial test design with six experimental conditions was
used. Participants in this study are 111 freshman students taking the course of Practical Training Methods
at Hanyang Women’s University. They were randomly assigned to one of the six experimental conditions
(see Table 1).
239
<Table 1>. Design of the experimental study
Part-task sequencing
Forward Snowballing Progressive
Element
interactivity
Low Group 1 (n=19) Group 2 (n=18) Group 3 (n=18)
High Group 4 (n=19) Group 5 (n=17) Group 6 (n=20)
Task and Procedure The students’ task was to draw various objectives using presentation software. Low element
interactivity groups were asked to draw two-dimensional objectives and high element interactivity groups
were asked to draw three-dimensional objectives. All instructions were presented in the form of softcopy
to standardize the procedure.
The experiment lasted for total four hours. Except the last 30 minutes which is assigned to test
learner’s performance and cognitive load, three hours and a half were designed to give leaners enough
time for practice.
Dependent Variables Dependent variables were learner’s schema automation and cognitive load. Schema automation was
measured by performance accuracy and time taken to complete the task. Performance accuracy and time
are the main key to evaluate to what extent to learners reach to schema automation. To assess the
performance accuracy, checklist was used.
Cognitive load was measured by 5-point self-rating questionnaires. Few studies exist measured
cognitive load by splitting it into extraneous, intrinsic, and germane cognitive load. This study measured
each cognitive load separately.
RESULTS
Schema automation A two-way ANOVA with performance accuracy as a dependent variable, and the kind of part-task
sequencing and the level of element interactivity as independent variables revealed a significant large-size
effect of the kind of part-task sequencing on performance accuracy (F(2,105)=18.94, p=.00). A post hoc
Tukey test showed that the forward group and the snowballing group differed significantly at p=.00 and
the forward group and the progressive group also differed significantly at p=.00. However, there was no
statistically difference between low element interactivity and high element interactivity (p=.82).
A two-way ANOVA showed a significant effect of the kind of part-task sequencing on the time
taken to complete a task by learners (F(2,105)=5.22, p<.01). A post hoc Tukey test showed that the forward
group and progressive group differed significantly at p<.01). There was also significant part-task
sequencing x element interactivity interaction effects (F(2,105)=8.91, p=.00).
Cognitive Load A two-way ANOVA was conducted to explore the effect of the kind of part-task sequencing and the
level of element interactivity on cognitive load.
Firstly, this analysis revealed a significant effect of the level of element interactivity on intrinsic
cognitive load (F(1,105)=12.08, p<.01). Second, a significant effect of the kind of part-task sequencing on
extraneous cognitive load was found (F(2,105)=3.76, p<.05). To find out where the differences line, a
Tuke’s post hoc was run. The forward group and the progressive group differed significantly at p<.05.
Third, with regard to germane cognitive load, there was a significant medium-size effect of the kind of
part-task sequencing (F(2,105)=3.34, p<.05). A post hoc Tukey test showed that the forward group and
progressive group differed significantly at p<.05).
240
DISSCUSION AND CONCLUSION
This study investigated the effect of different part-task sequencing and the level of element
interactivity on schema automation and cognitive load. To summarize the results, progressive chaining
and snowballing chaining outperformed forward chaining with regard to schema automation. However,
there was no significant difference between progressive chaining and snowballing chaining. Regarding
cognitive load, higher intrinsic cognitive load was occurred when the element interactivity is high
compared to low element interactivity, regardless of the kind of part-task sequencing. The lowest level of
extraneous cognitive load and the highest level of germane cognitive load were imposed when using
progressive chaining.
The results from this study provide same evidence with Pollock et al. (2002)’s study that isolated-
followed-by-interacting approach is effective to learners’ schema automation. However, the results cannot
tell us which sequencing strategy is better than others when considering isolated elements are divided into
more than two parts. Snowballing and progressive chaining does not make any significant difference. It
might be because that these two methods are all powerful technique for learning high element
interactivity.
REFERENCES
Clarke, T., Ayres, P., & Sweller, J. (2005). The impact of sequencing and prior knowledge on learning
mathematics through spreadsheet applications. Educational Technology Research and
Development, 53(3), 15-24.
Pollock, E., Chandler, P., & Sweller, J. (2002). Assimilating complex information. Learning and
instruction, 12(1), 61-86.
Salden, R. J., Paas, F., & van Merrienboer, J. J. (2006). A comparison of approaches to learning task
selection in the training of complex cognitive skills. Computers in human behavior, 22(3), 321-
333.
Sweller, J., & Chandler, P. (1994). Why some material is difficult to learn. Cognition and instruction,
12(3), 185-233.
Sweller, J., van Merrienboer, J. J., & Paas, F. G. (1998). Cognitive architecture and instructional design.
Educational psychology review, 10(3), 251-296.
van Merriënboer, J. J., & Kirschner, P. A. (2007). Ten steps to complex learning: A systematic approach
to four-component instructional design: Mahwah, NJ: Erlbaum.
van Merriënboer, J. J., Kirschner, P. A., & Kester, L. (2003). Taking the load off a learner's mind:
Instructional design for complex learning. Educational psychologist, 38(1), 5-13.
van Merrienboer, J. J., & Sweller, J. (2005). Cognitive load theory and complex learning: Recent
developments and future directions. Educational psychology review, 17(2), 147-177.
Wickens, C. D., Hutchins, S., Carolan, T., & Cumming, J. (2013). Effectiveness of Part-Task Training
and Increasing-Difficulty Training Strategies A Meta-Analysis Approach. Human Factors: The
Journal of the Human Factors and Ergonomics Society, 55(2), 461-470.
241
Effect of Conversational Gesture of Pedagogical Agent and Visual
Cueing on Task Comprehension and Eye Fixation in Types of
Information Formats
Jewoong Moon
Master Student
Dept. of Education, College of Education
Gwangju, Republic of Korea
Jeeheon Ryu
Associate Professor
Dept. of Education, College of Education
Gwangju, Republic of Korea
ABSTRACT
This study is to investigate the interaction effect of conversational gesture of pedagogical agent and
visual cueing on learning comprehension and eye fixation time. Pedagogical agent is a virtual character to
help students in multimedia environment. It can motivate learners to engage social interaction. Especially,
conversational gesture can play a crucial role to facilitate the interaction between learners and contents.
Because it provide a realistic human-like movement, learners can get higher social expectancy. Visual
cueing provide deictic information to allocate visual attention for students. It is non-content stimulus how
right direction learners can follow sequential process in learning visually. The participants of study were
sixty-four college students (male=19, female=45). The independent variables were presence of
conversational gesture and visual cueing, the dependent variables were comprehension scores and eye
fixation time. The results revealed that there were main effect (F=7.44, p=.008) and interaction effect
(F=4.57, p=.037) on comprehension score in text formats. The conditions not including gesture
outperformed than those groups including gesture. Furthermore, the condition of gesture and visual cueing
had lowest learning outcome. Rather, the condition only applied visual cueing has highest scores on
comprehension. It mean that visual cueing had a role for attentional guidance for learning. Rather, the
condition including both of gesture and visual cueing stimulated redundancy effect on learning
comprehension. In case of graphic formats, no significant difference by independent variables. Moreover,
it also had significant main effects on eye-fixation by gesture in text format (F=9.95, p=.003) and graphic
format (F=6.27, p=.015). As a result, the conditions except gesture had longer fixation time in
information area. Through these results, gesture of pedagogical agent had negative effect on dependent
variables. Because it might foster split-attention effect on visual attention. It could hinder sequential
learning process. Moreover, there were different results by types of information format. It might be due to
different mental representation with each formats.
Keywords: Pedagogical agent, Gesture effect, Visual cueing, Types of information format.
INTRODCUTION
Pedagogical agent is a cartoon-like to give instructional guidance for learners. Conversational
gesture of agent can foster more social interaction with learners. Because it can show realistic movement
242
like real human. Learners can easily perceive more social presence to animated character. But it could
also trigger distractions since that objectives show fancy stuffs visually beside instructional content. For
reducing split-attention effect by distraction, visual cueing can be applied to agents for allocating right
cognitive process. Visual cueing can guide learners how they should see the content sequentially. In line
with this arguments, types of information formats might also have differential results on using
pedagogical agents. Because text and graphic formats have different cognitive process. For exploring
these sequences, eye-tracking experiment were needed. In eye-mind hypothesis, eye-tracking study can
show human cognitive process through visual movement. Moreover, it can also provide unobtrusive result
for study.
Pedagogical Agent and Conversational Gesture Pedagogical agent is virtual object to provide guidance for learners. For more social interaction
between learners and animated agents, gesture was one of the necessary factors in communication.
Specifically, conversational gesture could provide affective context of communication with learners.
(Frechette & Moreno, 2010). It caused learners to engage learning environment with agent. Gesture is
also used as memory storage facilitating cognitive process, it required less cognitive capacity because it
reduces load automatically (Goldin-Meadow, Nusbaum, Kelly, & Wagner, 2001).
Visual Cueing Visual cueing was the addition of non-content information that captures attention to those objects
that are important in an complex materials (de Koning, Tabbers, Rikers, & Paas, 2007). In cognitive load
theory, if instructional material which are composed of pictorial and text information were allocated
separately, cueing strategy could reduce extraneous load by allocating visual attention. By leveraging
cognitive capacity for germane load, more elaborated mental models would be constructed. For
measuring visual attention, eye-tracking study might be useful tools for unobtrusive result on cognitive
process (Jarodzka, Scheiter, Gerjets, & Van Gog, 2010).
Types of Information Formats In multimedia learning, text and graphic information were normally delivered from two sensory
modality. According to Schnotz and Bannert (2003)’s cognitive model, text and graphical information has
different cognitive process. While text information was extracted from sematic and propositional meaning,
graphic one focused on intuitive perception visually. They classified two types of formats which are
depictive versus descriptive characteristics (Hochpöchler et al., 2012). Text as descriptive features were
heavily related to elicit imagery activity, but graphic as depictive ones could facilitate inference-making
process.
METHOD
Participants and Experimental Design In this study, sixty-four university students were participated in experiment as paid. Independent
variables were presence of gesture of agent and visual cueing. Gesture were used as conversational
movement for social interaction. Visual cueing was applied to red-colored circles, rectangular and arrows.
All participants were randomly assigned in each four conditions (n=16) by independent variables. The
dependent variables were comprehension scores and eye fixation time.
Apparatus All experiment were conducted in eye-tracking study. The SMI iViewX eye-tracker was used for
measuring the eye fixation time in AOI from participants. In experiment room, learners should watch a
video in front of eye-tracking device. The eye-tracker recorded whole experiment sessions. For
instructional materials, the agent was made by software ‘iClone5’. Narration were used to text-to-speech
(TTS) machine voice. For instructional materials, the video clip was used. The instructional materials
243
were about ‘formation of clouds and air mass’ in science lecture. The content was divided in types of
information formats between text-based and graphic-based materials. While text-based information
focused on core concepts and related meaning on theme, graphic-based one had a tendency with spatial
information which were specific locations or procedures.
Data Analysis Before seven days conducting the experiment, pretest were conducted for measurement of
covariance. The comprehension scores were 23 items divided between measurements with types of
information format. There were conducted pencil-to-paper test. For analyzing the eye fixation time, area
of interest (AOI) were designated to content area. The eye fixation time means sums of durations from all
fixation and saccades in designated area of interest (AOI). For measuring dependent variables, two-way
ANCOVA were applied. Covariate was pre-test scores with each types of information formats.
RESULT
Comprehension score In comprehension score, there were two types of score between text and graphic. In condition of text
condition, like figure 1 below, it had a significant main effect on gesture of pedagogical agent
(F(1,63)=7.44, p=.008). No gesture condition were higher comprehension score in text condition. Like
figure below, There had also significant interaction effect between independent variables (F(1,63)=4.57,
p=.037). On the other hand, in graphic condition, no significant result on all conditions.
Figure 1. Graphs of difference between comprehension score in text format
Eye fixation time As a result of eye fixation time in this study, with text formats, there were significant main effect on
gesture in eye fixation time (F(1,63)=9.95, p=.003). The conditions excluding gesture had longer fixation
time rather than those groups including gesture. Furthermore, in graphic condition, there was a main
effect on gesture (F(1,63)=6.27, p=.015) on eye fixation time with graphic formats. In light of previous
result, no gesture condition has higher fixation time rather than other condition.
CONCLUSION
Split-attention Effect by Gesture
244
Like previous results, there were significant differences by gesture of agent on comprehension score.
Especially, no gesture condition were outperformed than those groups including gesture in text format. In
line with this result, there were also significant differences between AOI analyses. In case of no gesture
condition, the longer fixation time were emerged in AOI. It means that learners couldn’t concentrate on
contents and be interfered with learning process. Through these results, gesture of pedagogical agent
might foster split-attention effect by distraction (Craig, Gholson, & Driscoll, 2002).
Attentional or Redundant Visual Cueing Visual cueing is a guidance for cognitive process visually. However, in case of this study, visual cueing
has a contrary results from their role. It was either attentional guidance or redundant information. The
condition with both of gesture and visual cueing had lowest score on comprehension. However, the
condition applied only visual cueing had highest score on comprehension than other conditions. While
only visual cueing could direct intentional cognitive process, the case which is simultaneously shown
from both of visual cueing and conversational gesture of agent could trigger redundancy effect on
multimedia environment.
Cognitive Processing with Information Format While comprehension score has different result in text condition, no significant results were found
in graphic ones. Hochpöchler et al. (2012) revealed that differences between text and graphic formats
were caused by cognitive processing from perception. Textual format was considered for imagery
elaboration, on the other hand, graphical one focused on formation of realistic mental models visually.
Especially, in this study, fixation count in graphic formats had significant differences. It mean that there
were more referential cognitive process in graphical ones because learners tried to comprehend only
visual information including their procedural and hierarchical structures. In contrast, text-based formats
were easily obtained information through sematic representation.
REFERENCES
Craig, S. D., Gholson, B., & Driscoll, D. M. (2002). Animated pedagogical agents in multimedia
educational environments: Effects of agent properties, picture features and redundancy. Journal
of Educational Psychology, 94(2), 428.
de Koning, B. B., Tabbers, H. K., Rikers, R. M., & Paas, F. (2007). Attention cueing as a means to
enhance learning from an animation. Applied Cognitive Psychology, 21(6), 731-746.
Frechette, C., & Moreno, R. (2010). The roles of animated pedagogical agents’ presence and nonverbal
communication in multimedia learning environments. Journal of Media Psychology: Theories,
Methods, and Applications, 22(2), 61.
Goldin-Meadow, S., Nusbaum, H., Kelly, S. D., & Wagner, S. (2001). Explaining math: Gesturing
lightens the load. Psychological Science, 12(6), 516-522.
Hochpöchler, U., Schnotz, W., Rasch, T., Ullrich, M., Horz, H., McElvany, N., & Baumert, J. (2012).
Dynamics of mental model construction from text and graphics. European Journal of
Psychology of Education, 1-22. doi: 10.1007/s10212-012-0156-z
Jarodzka, H., Scheiter, K., Gerjets, P., & Van Gog, T. (2010). In the eyes of the beholder: How experts
and novices interpret dynamic stimuli. Learning and Instruction, 20(2), 146-154.
Schnotz, W., & Bannert, M. (2003). Construction and interference in learning from multiple
representation. Learning and Instruction, 13(2), 141-156.
245
Assessment of Virtual Patients on Realistic Performance
Sun Kim
Master Student
The CaLT Lab, Department of Educational Technology, Chonnam National University
Gwangju, Korea
Jeeheon Ryu
Associate professor
The CaLT Lab, Department of Educational Technology, Chonnam National University
Gwangju, Korea
ABSTRACT
The purpose of this study is to find out a number of necessary items for assessing the virtual patients
on realistic performance. A virtual patient is defined as a computer program that simulates real-life clinical
scenarios for the purpose of healthcare and medical training, education. A virtual patient is being utilized
as a novel way to augment traditional methods of teaching and assessing clinical interviewing skills,
bioethics, basic patient communication, history taking and clinical decision-making skills. Furthermore, a
virtual patient can be used to simulate medical students with various medical conditions associated with
patients. However, there are a few studies that have reported the instrument for assessing virtual patient.
For this reason, this study will investigate the factors assessing virtual patient in terms of realistic
performance.
Keywords: Virtual patient, Assessment, Realistic performance
INTRODUCTION
One of the challenges in medical education concerns the need to teach students how to apply their
knowledge when they are dealing with clinical problems. Research has shown that students develop
clinical reasoning skills by seeing many patients, actively engaging in problem solving and receiving
sufficient feedback (Huwendiek et al., 2009). Especially, virtual patients are developed for training,
assessment and education of medical students. However, there are no suitable frameworks at present for
assessing virtual patients. As a teaching tool or an assessment tool, it is important to find out the
instrument items. For this reason, it needs to develop factors affecting realistic performance of virtual
patients.
THE LITERATURE REVIEW
This study will suggest that the factors having effects on realistic performance assessment of virtual
patients from the literature review. Wind and colleagues (2004) developed a valid, reliable and feasible
instrument to evaluate the performance of Standardized patients (SPs), based on 21questions. Their focus
246
was to access two sub-scales such as the authenticity of role play and the quality of feedback. However,
they focus on performance of SPs not VPs. It might have different perceptions to the SPs compared to the
VPs.
Tait and colleagues (2008) developed and evaluated the critical care e-learning scenario for student
nurses by surveying their attitude to the scenario. The questionnaire consisted of one item in a five-sub
scale that measured participant perceptions of 1) ease of use of the scenario; 2) benefit of its interactive
content; 3) its realism; 4) their increase in confidence in dealing with similar situations in real life; and 5)
overall participants’ attitude for scenario. Scenario-based learning has been implemented in a number of
ways including the use of video clips to develop students’ skills at handling difficult situations, clinical
simulation laboratories and e-learning scenarios. It will be useful to increase student nurses’ knowledge of
critical care and decision-making skills. Zary and colleagues (2006) also evaluated if it was possible to
develop a web-based virtual patient case simulation environment. The questionnaire that presented a Six-
Point Likert-type rating scale focusing on ease of use, experiences learning outcome and realism.
However, they investigated students’ attitude of scenarios, not student’ perception of VPs.
Bearman and Cesnik (2001) compared student attitudes to different models of the same virtual
patients. They used 10questions to measure student attitudes to the simulation. They concluded that the
majority of students were positive, although not wildly enthusiastic, towards the simulations. In addition,
Forsberg and colleagues (2011) investigated students’ opinions about feasibility of using Virtual patients
(VPs) for assessing clinical reasoning in nursing education. The questionnaire was constructed 7
questions to measure two sub-scales: 1) the use of virtual cases for assessment; and 2) the use of web-SP
system. They investigated that overall the students had very positive opinions about the use of virtual
patients as an assessment method. However, such factors are not relevant for performance of VPs.
FACTORS FOR ASSESSING VIRTUAL PATIENT
There is a need for virtual patient realistic performance. Evaluation of VP performance is
important to ensure the educational quality when they use VPs. To find out factors for the instrument, we
collected the instruments used in other studies that investigated VPs. After collating all 76items from
these studies, we selected appropriate factors from initial items and revised them as follows Table 1.
<Table 1>. Factors affecting realistic performance of virtual patients
Study Numbers of
Questions Initial Factors Revised Factors
Wind and
colleagues
(2004)
21
1) Authenticity
2) Feedback
3) Satisfaction
▪ Ease of use
relating to the use of virtual patients
▪ Realism
relating to the level of realism such
as facial expression and voice of the
virtual patients compared to real
patients
▪ Usefulness
relating to the learner’s perceptions
Tait and
colleagues
(2008)
21
1) Ease of use
2) Interactive content
3) Realism of scenario
4) Confidence
5) Satisfaction
Zary and
colleagues
(2006)
17
1) Ease of use
2) Experienced
learning outcome
3) Realism of scenario
247
Bearman and
Cesnik (2001) 10 1) Simulation
of whether they felt more useful
about dealing with similar situations
in real life after using the virtual
patient
▪ Satisfaction
relating to the overall attitude of the
learners to virtual patients
Forsberg and
colleagues
(2011)
7 1) Use of virtual scenario
2) Satisfaction
As shown table 1, there have something in common with factors such as ease of use, realism,
usefulness and satisfaction. So this study selected the factors which are ease of use, realism, usefulness
and satisfaction, and are described as shown table 1.
IMPLICATIONS
There are two limitations which could impact the generalization. The first, only a few studies are
reviewed and the studies are restricted to virtual patients. If other researches such as agents are reviewed,
factors affecting virtual patients’ realistic performance may be different. The second, the results of this
study are identified through literature review, not an experimental research. To make a valid, reliable and
feasible instrument through results of this study, a further study must be investigated.
REFERENCES
Bearman, M., & Cesnik, B. (2001). Comparing student attitudes to different models of the same virtual
patient. Studies in health technology and informatics, (2), 1004-1008.
Dickerson, R., Johnsen, K., Raij, A., Lok, B., Stevens, A., Bernard, T., & Lind, D. S. (2005). Virtual
patients: assessment of synthesized versus recorded speech. Studies in Health Technology and
Informatics, 119, 114.
Forsberg, E., Georg, C., Ziegert, K., & Fors, U. (2011). Virtual patients for assessment of clinical
reasoning in nursing—A pilot study. Nurse Education Today, 31(8), 757-762.
Huwendiek, S., Reichert, F., Bosse, H. M., De Leng, B. A., Van Der Vleuten, C. P., Haag, M., ... &
Tönshoff, B. (2009). Design principles for virtual patients: a focus group study among
students. Medical education, 43(6), 580-588.
Tait, M., Tait, D., Thornton, F., & Edwards, M. (2008). Development and evaluation of a critical care e-
learning scenario. Nurse Education Today, 28(8), 970-980.
Wind, L. A., Van Dalen, J., Muijtjens, A. M., & Rethans, J. J. (2004). Assessing simulated patients in an
educational setting: the MaSP (Maastricht Assessment of Simulated Patients). Medical
Education, 38(1), 39-44.
Zary, N., Johnson, G., Boberg, J., & Fors, U. G. (2006). Development, implementation and pilot
evaluation of a Web-based Virtual Patient Case Simulation environment–Web-SP. BMC medical
education, 6(1), 10.
248
Usability Study of Visual Dashboard as Learning Analytics
Interventions
Kunhee, Ha
Graduate Student
Ewha Womans University
Seoul, Korea
Il-Hyun, Jo
Associate Professor
Ewha Womans University
Seoul, Korea
Sohye, Lim
Assistant Professor
Ewha Womans University
Seoul, Korea
ABSTRACT
Learning analytics has been emerging as a breakthrough research and practice domain for
educational technology (Campbell, DeBlois, & Oblinger, 2007; Elias, 2011). Learning Analytics
Dashboard (LAD), which is a visual presentation of data mining results regarding individual learners‟
online learning behaviors, has been suggested as an effective intervention strategy for their learning and
performance (Jo & Yu, 2013, ; Van Barneveld, Arnold, & Campbell, 2012). However, only a paucity of
empirical studies regarding the design strategy of and usability of online dashboard interventions has been
reported (Koulocheri & Xenos, 2013).
The purpose of this study was to investigate the user experience with visual dashboard designed and
developed based on the learning analytics perspective. Information provided in visual dashboard was
composed of graphs representing the results of statistical analysis including total log-in time, total log-in
frequency, log-in regularity, visits on board, time spent on board and visits on repository. However, it was
assumed that these statistical representations may not be easily understood to students. As a result, it is
possible not to find maximum effects intended by the dashboard designers.
Therefore, as an empirical study, we conducted a usability test of LAD and investigated students‟
real-time responses on LAD. Qualitative research method including stimulated recall protocol was
employed. Users‟ experience on LAD was recorded through the software „Morae‟ and interviews were
conducted with 6 college students. The results of the study provide useful insights on how to design and
develop an instructionally effective and psychologically intuitive dashboard as an intervention of learning
analytics.
Keywords: Learning Analytics, Visual Dashboard, Usability Test
249
INTRODUCTION
Learning analytics is an emerging research field for both practical implications and the theoretical
ground in educational technology (Campbell, DeBlois, and Oblinger, 2007; Elias, 2011). An approach of
learning analytics suggests actionable interventions for students by observing, understanding and
predicting learning behaviors (Brown 2011). Unlike the application of data mining to analyze customers‟
behavior pattern in cooperation setting, the area of learning analytics aims not only to analyze students‟
learning pattern but also to provide effective interventions for learners to develop learning and
performance. Above all, the characteristics of online learning environment where the distance between
instructor and learners is rather extensive (Moore, 1993), and thus the range of direct observation and
interventions are narrow and limited (Scheuer, and Zinn, 2007). Thus, solutions such as Learning
Analytics Dashboard (LAD) can be effective strategies for students to be aware of their learning
behaviors, to reflect themselves, to have new insights, and to change their learning patterns and methods
(Leony, Pardo, de la Fuente Valentın, Quinones, and Kloos; van Barneveld, Arnold, and Campbell, 2012).
Thus, several previous researches have attempted to test and analyze usefulness of dashboards. Katrien,
Erik, Joris, Sten and José (2013) compared twelve dashboards with regard to usefulness, effectiveness.
Govaerts, Verbert, Duval and Pardo (2012) also analyzed that the use and usefulness of different
visualizations and discussed how such dashboards work on learners. These studies suggest that the
usefulness of dashboard be analyzed before verifying whether dashboard can be effective as a learning
intervention. However, the general function of dashboard was limited to presenting accumulated data
without considering effective instructional design to stimulate learners‟ cognition. Therefore
instructionally advanced design techniques have been required in order for learners to improve their
learning performance (Segedy, Sulcer, and Biswas, 2010; Essa, and Ayad, 2012). As an example, Course
Signal (CS) program at Purdue University played a role of early warning that enables instructors to
predict those who are at the risk level for appropriate academic achievement. The high level of risk was
shown with a red signal while the low risk level was indicated in green sign (Arnold, and Pistilli, 2012).
Purdue University‟s CS program has been positively evaluated for it develops students‟ meta-cognitive
strategies as well. In that case, the usefulness of dashboard is very important issue to instructor, but before
examining usefulness of dashboard, its usability should first be investigated. Jo, HEO, Lim, CHOI, and
NOH (2013) suggest usability, the degree of users‟ felt easiness of artifacts, is the necessary condition to
be tested prior to usefulness of them. In line of Jo et al‟s thought, LAD intervention should be tested its
usability before investigating its ultimate goal, usefulness of learning and performance. However, no
study systematically tested LAD‟s usability. The question we address in this study is: Is the learning
analytics dashboard we developed usable to target students? To answer the question, the first step to be
taken is to design and to develop a LAD prototype and then to test its usability because a learning
supporting tool might be effective when it is perceived to be useful tool by its users.
In short, we attempted to focus on students‟ perception and comprehension of the information in LAD.
Usability is the issue of affectiveness, while Usefulness is the issue of effectiveness. It is also the reason
why this study used stimulated recall protocol method which can make subjects to recall past activity
more clearly. In cognitive information processing, people can remember when they are provided with
detailed situation or distinguishing symbols (Jo & Kim, 2006). The usability test requires users‟ precise
memory because it should be designed and modified based on the users‟ experiences and perception.
METHOD
In order to answer the research questions, we adopted a qualitative research method in order to
focus on the usability test to investigate a small number of students‟ reactions and their perception on
LAD.
Participants The participants of this study were 6 college students in E women‟s university in Seoul. They were
sampled out of from an undergraduate course “Knowledge management”. All students were Educational
Technology major.
250
Learning Analytics Dashboard Institute for Teaching & Learning (ITL) of E university collaborated for the design and
development of dashboard. Dashboard is constructed with the log data accumulated for 8 weeks after the
fall semester began. 7 graphs were presented including the first graph that summarized the correlation
between the variables: total log-in time, total log-in frequency, log-in regularity, visits on the board, time
spent on the board and visits on repository. In that plane board, all students in the class were shown and
„I‟ was located as a red dot. It also indicated the average value of the class so that „I‟ could choose the
variables as a X-axis and Y-axis to figure out where „I‟ was and how far „I‟ was apart from average line
of each variable. The other was a bar graph in which trend lines of „my‟ weekly activity was visualized.
Figure 1 is the prototype display that the subjects were exposed to in the usability test.
Figure 1. Learning Analytics Dashboard (LAD)
Setting for Usability Test
To conduct a usability test, 6 participants were placed in a secure classroom equipped with
computers. Since the recording software „Morae‟ was installed on the computers, students‟ mouse
movement, the displayed screen, students‟ face and their voices were automatically recorded as shown in
Figure 2. At the beginning of test, the researcher explained the participants the purpose and the procedure
of the study. {Since their testing duration is related to our study, they could continue testing until they
wanted to quit. As a result, the mean of testing duration was about 10 minutes. After the test, we collected
the data and replayed using the manager program using „Morae‟ for the analysis.
251
Figure 2. The recorded data using „Morae‟ in usability test
Research Procedures
The test consists of four steps. First, as participants used LAD, we recorded their reactions using
software „Morae‟. Second, one day later, we conducted individual interviews with 6 participants. Third,
we analyzed all the data including observation notes, recorded video clips, and the transcribed interview
results.
The interviews were conducted based on the recorded data. The manager program in „Morae‟
provides Mark system, so that we could replay the recorded data and mark on each timeline when the user
reacted to something. The data extracted through the replay process composed the first part of interview.
In addition we proceeded to the second part of interview by asking the degree of understanding on LAD,
perceived usefulness and opinions including suggestions to improve the quality of LAD. The second part
of interview was identical to all 6 students. If a student had already answered the questions of the second
part in the first interview watching recording data, repeated answers were skipped.
We decided that the participants should be free to speak of their opinions so that all the interviews
were asked in the format of open-ended questions. For example, during the first interview, the interviewer
and participant watched the video clip together and paused on the point where the participant exhibited a
unique reaction. By asking the reason for the reaction “why did you do so?” or “what did you think about
it at the moment?” the participant could easily recall their perceptions and answer more precisely based
on their memory. Testing the usability of dashboard including the interviews with six participants were
conducted within two days in order to prevent further memory loss.
RESUSLS
Degree of Understanding
Most participants reported difficulties understanding the first summary graph. We could observe
that they were trying to compare the summary graph with the other graphs. One student mentioned:
“The Summary graph was not comprehensible at first, so I looked down the others…the summary
graph presents too many information.”
In this regard, students pointed out that they needed a description on the meaning of graphs. Student
„E‟ indicated:
“Additional descriptions are necessary for understanding the graph properly. If someone doesn‟t
have sufficient background knowledge in the statistics, these graphs are too hard to understand.
Also, the correlation between the summary graph and the bar graphs should be explained
enough.”
LAD display
The recorded user face
(with web cam)
252
Student „A‟ and „D‟ also answered that they could not understand why and how to see the suggested
data when shown. On the other hand, another student „B‟ mentioned that she could understand LAD
almost perfectly. She also explained how she could analyze the LAD.
“I wondered why the log-in frequency does not match the total log-in time in seventh week, so I
inferred that the data is not enough. I think more data should be updated.”
Through participants‟ diverse reactions on LAD, we discovered that the level of students‟
understanding, especially the literacy of graphic representation, varied and this may impact their
perceived usefulness and the effectiveness of LAD. This underlines the importance of providing enough
instructions and guidelines to help them understand the meaning of graphs on their online learning
patterns.
Perceived Usefulness
Several students commonly commented that the graphs illustrating their relative position in
comparison with other students were useful. Moreover, they added that the function of graph comparing
with other students would boost their motivation. Students „F‟ said:
“I usually wanted to know how other students are studying in the class. For example, I clicked all
the board each time whenever I logged in the cyber campus. I really wondered how and what they
do in there. Especially, in the forum board, students were interacting for their team project, so I
sought for other teams‟ performance as well. If they interacted more frequently than our team, I
was motivated to do harder.”
Students were positive when the information was related with their location in the class and
performance. What students wanted to understand was especially the correlation between their online
activity and actual learning performance. Student „C‟ critiqued:
“Do online activities have a relationship with real learning activities? The log data which is
visualized on the dashboard just shows me that „you are doing what you see‟, but additional
implications are not involved in data, I think. I just understand what I am doing without why the
dashboard presented all this data.”
However, some students negatively responded in regard to the total log in time. Student „B‟
mentioned that like: “Sometimes I log in cyber campus and do other things. I do not study all the time I
log in.” In addition, student „F‟ answered similarly:
“Some classes could leave cyber campus empty. The professor in that class just uploaded a
syllabus or some notices and did nothing with cyber campus. In that case, students did not use
forum or free board at all.”
Opinion and Suggestion
A few students answered that the correlation and interaction between students in the class is needed.
When the class uses forum board, how students interact with other students can be an indicator for their
participation in the group learning. Student „C‟ said:
“Sometimes I would log in but don‟t do anything in particular, so I think the number of reply
could be more helpful information for us. That information was about practical learning.”
Student „D‟ also supported such need for interaction information by mentioning: “Especially online
debates or communication is important. I could compare myself with others.”
In summation, students perceived LAD in a two ways. One is a reminder that they can reflect their
previous behavioral patterns and the other is a comparison tool that they could learn their relative rank in
the class. Lastly, the participants suggested for the number of opinions for modifying LAD. These
include:
253
“I needed the unit of information. For instance, this bar graph indicates the log-in regularity, but I
could not distinguish whether the unit of these numbers was minute or not.” (Student C)
“Is this the number of frequency? Or just time?”(Student D)
CONCLUSION
Usability is a separated concept from usefulness. LAD can be useful but it does not assure that it is
also usable. Why we conducted qualitative research method in this study is to figure out the affectiveness
of students using LAD. Learning interventions with LAD is closely related with a usage pattern. In other
words, how users approach LAD is related with how much LAD is usable and it could be finally
determine the effectiveness of online learning interventions by LAD.
With that regard, several implications on LAD in this study were drawn. First, students‟ response
indicated incompatibility about the perception of instructor. We had concluded in the previous study that
regularity of learning interval is a powerful indicator for learning performance (Kang, Kim, and Park,
2008; Jo, Yoon, and Ha, 2013). However, students did not know that log-in regularity information is
related with learning performance or it could predict learning outcomes. Student „B‟ replied that she did
not know that even log-in regularity could be suggested as a data, but most of students looked like that
they could not find which information relates to the learning performance to a greater extent which means
that the usefulness of information could be varying across the learners and the instructors. Thus,
designing a dashboard should take the learners‟ perception of their learning into account and explain the
reason for the information presented in the dashboard. Thus, dashboard without some descriptions to the
extent the learners can notify why instructor choose that data cannot serve the role of learning
interventions enough. In that regard, designing dashboard should consider the learners‟ perception of their
learning and explain why dashboard presents these kinds of information.
Second, LAD designed in this study is a preliminary step before developing an effective learning
intervention with which learners can observe their learning patterns, predict learning outcomes and adjust
their behaviors accordingly. For that reason, this study results can be the reference for instructional design
to develop dashboard as a learning intervention. As referred to earlier, LAD should be considered in two
perspectives: usefulness and usability. What we intended when design dashboard is that we could observe
how well learners understand the basic visualization of data and how they percept LAD emotively for
their continuous usage. Thus, based on this usability test, instructors could discuss what kind of
instructional design factors should be provided to help learners appropriately.
This study is not without its own limitations. In particular, the dashboard was not subject-specific
and this might have lowered the students‟ motivation to process the information in LAD. Moreover, the
online activity is not the whole learning data about a student, because most of classes are operated in
offline. Thus, the log data are limited to the analysis of the learners themselves, so the instructor needs to
consider this when they use LAD as a learning intervention.
REFERENCES
Arnold, K. E., & Pistilli, M. D. (2012). Course Signals at Purdue: Using learning analytics to increase
student success. In S. Buckingham Shum, D. Gašević, & R. Ferguson (Eds.), Proceedings of the
2nd International Conference on Learning Analytics and Knowledge (LAK ′12) (pp. 267-270). New
York: ACM.
Brown, M. (2011). Learning Analytics: The Coming Third Wave. EDUCAUSE Learning Initiative Brief,
Retrieved from http://www.educause.edu/Resources/LearningAnalyticsTheComingThir/227287.
Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic analytics: A new tool for a new
era. Educause Review, 42(4), 40.
Elias, T. (2011). Learning analytics: Definitions, processes and potential.Retrieved February, 9, 2012.
Essa, A., & Ayad, H. (2012, April). Student success system: risk analytics and data visualization using
ensembles of predictive models. In Proceedings of the 2nd International Conference on Learning
Analytics and Knowledge (pp. 158-161). ACM.
Gass, S. M., & Mackey, A. (2000). Stimulated Recall Methodology in Second Language Research,
Lawrence Erlbaum Associates; Mahwah, NJ.
254
Govaerts, S., Verbert, K., Duval, E., & Pardo, A. (2012, May). The student activity meter for awareness
and self-reflection. In CHI'12 Extended Abstracts on Human Factors in Computing Systems (pp.
869-884). ACM.
Jo, I., Heeok, HEO., Kyu Yon, Lim., Jeong-Im CHOI., & Jeongmin, NOH. (2013). Usability Study of
Middle School English Digital Textbook: A Stimulated Recall Approach. Educational Technology
International. 14(1). pp. 109-136.
Jo, I. (2012). On the LAPA (Learning Analytics for Prediction & Action) Model suggested. Future
Research Seminar. Korea Society of Knowledge Management. Seoul.
Jo, I., Kang, Y., Yoon, M., & Kang, M. (2012, Fall). Development of cluster-specific learning prediction
models: A learning analytics approach, Paper presented at the HYCU International Conference,
Seoul, Korea.
Jo, I., & Kim, Y. (2006). An Exploratory Study on the Validity of Stimulated Recall Method as a
Usability Test Tool for e-Learning. The Journal of Educational Information and Med. 12(2). pp. 71-
95.
Jo, I. & Kim, Y. (2013). Impact of Learner‟s Time Management Strategies on Achievement in an e-
learning Environment: A Learning Analytics Approach. The Journal of Educational Information and
Media. 19(1). 83-107
Jo, I., Yoon, M., & Ha, K. (2013). Analysis of Relations Between Learner‟s Time Management Strategy,
Regularity of Learning, and Learning Performance: A Learning Analytics Approach. e-Learning
Korea 2013 Conference.
Kang, M., Kim, J., & Park, I. (2008). The Examination of the Variables related to the Students‟ e-
Learning Participartion that Have an Effect on Learning Achievement in e-learnign Environment of
Cyber University. Journal of Korean Internet Information. 10(5). 135-143.
Kim, S. (2003). A correlation between Learning Behavior and Achievement Level of Learners in e-
learning. Master Thesis. Korean University of Technology and Education, Chung Nam.
Kim, Y. (2010). Impact of Regulated of Learning Interval, Total Learning Hours and Number of Access
to e-Learning in a Corporate e-Learning Environment on Academic Achievement. Master thesis,
Seoul, Ewha Womans University.
Kim, Y. (2011). Learning time management variables impact on academic achievement in corporate e-
learning environment. Master Thesis. Ewha Womans University, Seoul.
Leony, D., Pardo, A., de la Fuente Valentın, L., Quinones, I., & Kloos, C. D. Learning analytics in the
LMS: Using browser extensions to embed visualizations into a Learning Management System.
Moore, M. G. (1993). 2 Theory of transactional distance. Theoretical principles of distance education, 22.
Santos, J. L., Verbert, K., Govaerts, S., & Duval, E. (2013, April). Addressing learner issues with
StepUp!: an Evaluation. In Proceedings of the Third International Conference on Learning
Analytics and Knowledge (pp. 14-22). ACM.
Scheuer, O., & Zinn, C. (2007). How did the e-learning session go? The Student Inspector. FRONTIERS
IN ARTIFICIAL INTELLIGENCE AND APPLICATIONS, 158, 487.
Segedy, J., Sulcer, B., & Biswas, G. (2010, January). Are ILEs ready for the classroom? bringing teachers
into the feedback loop. In Intelligent Tutoring Systems (pp. 405-407). Springer Berlin Heidelberg.
van Barneveld, A., Arnold, K. E., & Campbell, J. P. (2012). Analytics in higher education: Establishing a
common language. ELI paper, 1, 2012.
Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning Analytics Dashboard
Applications. American Behavioral Scientist.
255
The effects of regulatory learning strategies on collaboration load
and collaboration outcomes in computer-supported
collaborative learning
Hyojin Lee
Doctoral Student
Hanyang University
Seoul, Korea
Dongsik Kim
Professor
Hanyang University
Seoul, Korea
ABSTRACT
The purpose of this study was to examine the effects of regulatory learning strategies on
collaboration load and collaboration outcome in CSCL. We focuses on collaborative learning effectiveness
in acquiring self-regulation skills and collaborative skills when a regulatory strategies is presented. This
study was driven by a couple of questions. First, What effects do regulatory learning strategies have on
collaboration load in CSCL? Second, What effects do regulatory learning strategies have on collaboration
outcome in CSCL?
Keywords: self-regulation, regulated learning, collaboration load, CSCL
INTRODUCTION
Self-regulated learning has been of interested to many researchers in education, learning
sciences(Shunk & Zimmerman, 2008; Winne & Hadwin, 1998; Volet, Summers, & Thurman, 2009;
Zimmerman, 2008). Similarly, scholars recognize the importance of metacognitive processes in
collaborative learning such as planning, setting goals, monitoring(Gibson, 2001, Rummel & Spada, 2005).
According to Jarvela and Hadwin‟s study(2013, p.28), three dimensions characterize metacognitive
processes in Computer-Supported Collaborative Learning(CSCL). The first dimension is “self-
regulation”, which is guided by environmental conditions that promote individuals to adopt, develop, and
refine strategies; monitor, evaluate, and set goals. This dimension suggests that successful team work
requires each group member to regulate his or her own cognitive process. The second dimension of
regulatory learning that Jarvela and Hadwin(2013, p.28) describes is that “co-regulation” occurs when
individuals‟ regulatory activities are guided by and with others. Co-regulation requires team members to
be aware of one another‟s goals and progress and to consider those in relation to the shared task. The third
constitutive dimension of regulatory learning according to Jarvela and Hadwin(2013, p.28) is the “shared
regulation”. This dimension occurs when groups regulate as a collective such as when they construct
shared goals. In this case, goals and standards are co-constructed, and regulation is distributed and shared
with multiple ideas and being weighed and negotiated until consensus is met.
Many prevailing accounts of self-regulated learning in individual contexts revealed that experts
generally come up with self-regulation strategies than novice(Plass, Kalyuga, & Leutner, 2010). In other
words, the more learners have prior knowledge, the more they acquire self-regulatory skills effectively.
256
The self-regulatory process in collaborative learning is seen as akin to the process in individual learning.
In collaborative learning settings, it is also important to enable individuals to utilize their relevant
disciplinary knowledge, while at the same time making use of self-regulated learning strategies.
However, we intentionally distinguish between the individual and group(and peers) regulatory
process. Contrary to self-regulation skills, when experts in a particular domain did not have collaborative
skills, they were difficult to acquire co-regulation and shared regulation skills because coordination and
communication are interrupted by confusion of interaction. Domain-specific knowledge is one of the
crucial aspects of successful self-regulatory learning, whereas co-regulated learning and shared regulated
learning are strongly related to collaborative skills or previous experience. Therefore, regulatory learning
in CSCL contexts only succeed with respective support for individual and group dimension.
STRATEGIED FOR SUPPORTING REGULATION IN CSCL CONTEXTS
Our approach to supporting self-regulated learning in CSCL is grounded in Cognitive Load
Theory(CLT). As we have described above, the majority of research examining self-regulatory learning
have appeared that the level of learner expertise was a critical factor that influenced the application of
self-regulation strategies(Plass et al., 2010). If a learner has sufficient knowledge to understand the
problem formulation, he or she may successfully apply self-regulatory strategies by reducing intrinsic
load. In contrast, for these less experienced learners, demands of information processing may induce
cognitive overload and interfere with self-regulation learning instead of assisting it.
Self-regulated learning is related to extraneous load as well as intrinsic load. Demands of
monitoring, controlling, reflecting activity might cause cognitive overload and lead to failure of both the
problem solving and learning process. Without adequate support in collaborative learning settings,
learners often fail to complete their joint task or find that it requires too much time and effort. Therefore,
self-regulated learning in CSCL contexts need to be supported to ensure controlling intrinsic load and
extraneous load and promoting germane cognitive process(Plass et al., 2010).
CLT offers principles and methods to design effective and efficient instructional interventions that
support self-regulatory learning in CSCL contexts. Kester, Paas and van Merriënboer(2010) showed the
possibility and promising potential of feedback strategies for the field. The study revealed that germane-
inducing method(i.e., delayed feedback) was more effective in solving complex task than structured
method(i.e., immediate feedback) (Kester et al., 2010). Giving delayed feedback refers to comparing the
current state of group work to a model of desired work after a certain period of time and intervene when
discrepancies between these two states are discovered. On the other hand, immediate feedback occurs
simultaneously with problem solving. Butler and Winne(1995) highlight feedback is generally an inherent
catalyst for all self-regulated activities. While learners monitor their engagement with tasks, internal
feedback is generated by the monitoring process(Butler & Winne,1995). The feedback methods are not
designed for collaboration but may be considered strategies for supporting self-regulation in CSCL.
However, some scholars suggest that germane-inducing methods priority may be limited to novice(Ross
& Kilbane, 1997). For low prior knowledge learners, what they make an effort to evaluate their
engagement with tasks on their own becomes a redundant activity that contributes little or nothing to
further learning and problem solving and causes cognitive overload. Despite the limitation of information
processing, some researchers acknowledge the importance of problematizing task which may make
learners encounter important concepts or processes on their own. Problematizing task have been found to
successfully trigger changes in group dynamics by provoking debate, deliberations, and decisions(Reiser,
2004). Thus, this method may not only cause cognitive load that contributes to learning asennd problem
solving but also lead to enhanced transfer.
Collaboration scripts have been shown to be a promising approach to support co-regulation and
shared regulation in CSCL. A common use of collaboration script in CSCL environments is to decrease
the coordinative effort both on the teacher's and the learner's part. CSCL script enables group members to
engage in the appropriate interaction with peers and the improvement the joint problem-solving(Wecker
& Fischer, 2007). CSCL script can be advantageous to regulation. Awareness of one another's regulatory
process and co-construction a shared task perception are central processes in collaborative learning tasks.
To ensure efficient regulatory activity in the CSCL environment, it is crucial to specify, sequence, and
257
distribute learning activities by using script. Collaboration script not only aims to foster the acquisition of
domain-specific knowledge, but also collaborative skills.
Two different types of scripts may be used to support collaboration: Scripting as a supporting tool
and scripting as a teacher‟s instruction. These approaches differ not only in the timing of the intervention,
but also coercion. The coercion suggests that the degree of freedom participants have in following the
script. Scripting as a supporting tool aims to create favorable conditions for learning by designing and
scripting the situation before the interaction begins(Jermann et al., 2004; Kirschner, & Erkens, 2013). The
method constrains the number of options learners have, thus guiding them along the lines of the
processes(Beers et al., 2005). On the contrary, scripting as a teacher‟s instruction supports co-regulation
and shared regulation by taking actions after the interaction has begun(Jermann et al., 2004; Kirschner, &
Erkens, 2013). The method such as providing students with process worksheets guides processes by
providing just-in-time support for them when a specific problem arises.
Despite the advantages of guiding interaction and problem-solving processes, many prevailing
empirical studies revealed that the coercion of script made learner's motivation dwindling and induced the
short-term effects(Rummel, & Spada, 2005). Enforcement structured interaction might cause redundant
activity and lead to failure of reflection on the „whys‟ of the scripting.
This study aims to examine the effects of regulatory learning strategies on collaboration load and
collaboration outcome in CSCL by designing feedback and script.
1) What effects do regulatory learning strategies have on collaboration load in CSCL?
2) What effects do regulatory learning strategies have on collaboration outcome in CSCL?
RESEARCH DESIGN
The experimental paradigm set up comprised two phases: a self-regulation phase and a group-
regulation phase. The goal of the self-regulation phase was the acquisition of self-regulation skills and
domain-specific knowledge about elements relevant for a good and potentially successful collaboration.
Aspects of the joint work – the collaboration outcome - after the group-regulation phase as well as
collaboration load were investigated as dependent variables. Further, collaboration skills about elements
of a good and potentially successful collaboration was assessed in a posttest.
Independent variables
Feedback
We considered three aspects of feedback:
Immediate feedback by instructor Learners was given feedback from instructors within minutes
whenever there appeared to be „discrepancies between current understanding and performance
and a learning goal‟.
Delayed feedback by instructor Learners were presented with a set of rubric to self-check their
self-regulation strategies and then they worked on some problem collaboratively. A week later,
instructor provided feedback on whether last strategies were effective or not to attain the goal.
Script
We considered two aspects of scripts:
Scripting as a supporting tool: Groups in script condition with collaboration script received a
collection of interaction-related prompts to structure their collaborative problem solving. This
script was targeted at improving collaborative skills. During the group-regulation phase, dyads
in the scripted collaboration condition were provided with a detailed script prescribing specific
phases for their interaction. The collaboration script involved different activities and roles for
each student during the collaborative problem-solving: (a) define objectives of task, (b) scan
case description for potential problems with understanding and formulate questions to the
partner, (c) mutually answer questions and determine course of action(content, time, roles), (d)
individually search related information and come up with idea, (e) exchange information and
258
discuss individual ideas, (f) revise individual ideas and formulate final solution for the problem,
(g) copy individual parts of solution in shared editor and integrate final check of entire joint
solution(Rummel & Spada, 2005). The script was structurally equivalent to the instruction.
Scripting as a teacher’s instruction: After pre-test, students were guided to a 3 hours training
phase in the face to face, which helped them get a first experience on how to handle the learning
environment and how to collaborate. This information was equivalent to the information
presented in the prompts of the collaboration scripts. The students in the instruction condition
received no support beyond training by their teacher to the strategy of collaborative problem
solving.
Dependent variables
Collaboration Outcome
Collaboration outcomes were comprised of (a) joint solution, (b) self-regulation skills, and (c)
collaborative skills.
Joint Solution: To measure collaboration outcomes, each team‟s final report was assessed. To
analyze the quality of the joint solution, a system of quantitative criteria was developed by
experts in the area of A. To analyze the quality of the joint solution, a system of quantitative
criteria was developed by experts.
Self-regulation Skills: To assess self-regulation skills, we adopted microanalytic methodology by
developed Kitsantas and Zimmerman(2002). Specific questions was used to measure well-
established self-regulatory processes and motivational beliefs or feelings at key points before,
during, and after learning. Learners were asked open- or closed-ended questions that produced
both qualitative and quantitative data, respectively. The questions were brief and task specific in
order to minimize disruptions in learning.
Collaboration Skills: As discussed earlier, we assumed that instructional support measures
would improve people‟s collaborative skills. We assessed collaborative skills with individual
posttest. We employed the coding scheme developed by Meier, Spada and Rummel(2007). This
coding sheme distinguished five theoretically and empirically grounded collaborative process:
communication, joint information processing, coordination, interpersonal relationship,
motivation. Table 3 provided detailed dimensions of these aspects.
Collaboration Load
To capture the collaboration load, we adopted subjective rating scale technique developed by Jung
and Kim(2012) for two important reasons. First, subjective rating of mental effort is the best one rather
than objective measures such as time-on-task, eye-tracking analysis because learners can be asked to rate
their perceived cognitive load subjectively. Second, this instrument directly examines three types of
load(e.g. extraneous collaboration load, instrinsic collaboration load, germane collaboration load).
Participants answered 26 questions on a seven-point Likert scale ranging from 1 “strongly disagree” to 7
“strongly agree”. There were 4 questions for physical effort, 4 questions for mental effort, 4 questions for
task difficulty, 4 qeustions for process satisfaction, 4 questions for outcome satisfaction, 2 questions for
environment availability, 4 questions for immersion.
CONCLUSION
In this study, we suggest the strategies to promote self-regulatory learning and collaborative
learning in CSCL contexts. We explored the theoretical framework of regulation‟s types and explained
each strategies to support self-regulation and co-regulation. To sum up, it is expected that the feedback
strategies can promote successful self-regulation activities and effectively induce germane cognitive
process, thereby resulting in learning enhancement. In addition, different types of scripts may cause
dissimilar cognitive load and lead to learning process. We suggest futher studies with empirical findings
that will validate the strategies developed in this study.
259
REFERENCES
Beers, P. J., Kirschner, P. A., Boshuizen, H. P. A., & Gijselaers, W. R. (2005). Coercing knowledge
construction in collaborative learning environments. Proceedings of the international conference on
CSCL 2005 (pp. 8-17), May 30-June 4, 2005, Taipai.
Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis.
Review of Educational Research, 65(3), 245–274.
Gibson, C. B. (2001). From knowledge accumulation to accommodation: cycles of collective cognition in
work groups, Journal of Organizational Behavior, 22(2), 121-134.
Hattie, J. A. C., & Gan, M. (2011). Instruction based on feedback. In R. Mayer, & P. Alexander (Eds.),
Handbook of research on learning and instruction (pp. 249–271). New York: Routledge.
Järvelä, S. & Hadwin, A. F. (2013) New Frontiers: Regulating Learning in CSCL. Educational
Psychologist. 48(1). 25-39.
Jermann, P., Soller, A., & Lesgold, A. (2004). Computer software support for CSCL. in J. W. Strijbos, P.
A. Kirschner, & R. L. Martens(eds.), What we know about CSCL? (pp. 141-166), NY: Kluwer
Academic Publishers.
Jung. H. J. & Kim. H. Y. (2012). An Exploratory Validation for the Constructs of Collaboration Load.
Journal of Educational Technology, 28(3), 619-640.
Kester, L., Paas, F., & van Merriënboer, J. J. G. (2010). Instructional control of cognitive load in the
design of complex learning environments. In J. Plass, R. Moreno, & R. Brünken (Eds.), Cognitive
load theory (pp. 109-130), New York: Cambridge University Press.
Kirschner, P. A. & Erkens. G. (2013). Towards a framework for CSCL research. Educational
psychologyist, 48(1). 1-8.
Kitsantas, A., & Zimmerman, B. J. (2002). Comparing self-regulatory processes among novice, non-
expert, and expert volleyball players: A microanalytic study. Journal of Applied Sport Psychology,
14, 91–105.
Meier, A., Spada, H., & Rummel, N. (2007). A rating scheme for assessing the quality of computer
supported collaboration processes. International Journal of Computer-Supported Collaborative
Learning, 2, 63–86.
Plass, J. L., Kalyuga, S., & Leutner, D. (2010). Individual differences and cognitive load theory. In J. L.
Plass, R. Moreno, & R. Brunken (Eds.), Cognitive load theory (pp. 65–87). New York: Cambridge
University Press.
Reiser, B. J. (2004). Why scaffolding should sometimes make tasks more difficult for learners. In G.
Stahl(Ed.) Computer support for collaborative learning: foundations for a CSCL community,
proceedings of CSCL 2002 (pp. 255-264), Boulder, CL, Jan. 7-11, 2002.
Ross, B. H., & Kilbane, M. C. (1997). Effects of principle explanation and superficial similarity on
analogical mapping in problem solving. Journal of Experimental Psychology: Learning, Memory,
and Cognition, 23, 427–440.
Rummel, N., & Spada, H. (2005). Learning to collaborate: an instructional approach to promoting
collaborative problem solving in computer-mediated settings. Journal of learning sciences, 14(2),
201-241.
Schunk, D. H., & Zimmerman, B. J. (Eds.). (2008). Motivation and self-regulated learning: Theory,
research, and applications. Mahwah, NJ: Lawrence Erlbaum.
Volet, S. E., Summers, M., & Thurman, J. (2009). High-level co-regulation in collaborative learning: how
does it emerge and how is it sustained? Learning and Instruction, 19(2), 128–143.
Wecker, C., & Fischer, F. (2007). Fading scripts in CSCL: the role of distributed monitoring. Mice, Mins
and Society: CSCL conference 2007(vol. 8) (pp. 763-771), New Brunswick, NJ:SUNJ.
Winne, P., and Hadwin, A. (1998). Studying as self-regulated learning. In Hacker, D., Dunlosky, J., and
Graesser, A. (eds.), Metacognition in Educational Theory and Practice, Erlbaum, Hillsdale, NJ, pp.
279–306.
260
Improvement of Score Reading Skill
By Music Composing Class with SMART Education
Hyerin Lee
Master’s degree student
Chuncehon National University of Education
ABSTRACT
Score reading skill and music composing are related each other deeply. So, to improve score reading
skill, music composing class is effective way. But there are some barriers. Every student needs music
instruments individually. It’s not that easy to give them music instruments each, however, especially in a
big size of class. Therefore, music teacher often made them compose as a group or use simplified music
instruments like a recorder or a melodeon. If we call these ways as a traditional teaching style, we can
overcome some limitations of traditional music composing class by using SMART education. The purpose
of this study is improvement students’ score reading skill in music composing class through SMART
education. For achieving this purpose, 16 of Grade 3 students participated in this study containing a
mental weakness student. They composed their own music individually with a Samik Piano application
and iPad. The period of this study is a week, 6 times of music class.
Students recognized about this study and answered to a paper questionnaire about their interests,
experiences, expectations about music composing class in 1st class. In 2nd~ 3rd class, students composed
own melody through a Samik Piano application by touching every tones of a virtual piano keys and wrote
their melody in a simplified score called ‘grid score’. In 4th class, students composed proper rhythm
according to their melody and wrote the rhythm in the ‘grid score’ In 5th class, students moved their
melody and rhythm from ‘grid score’ to a staff notation. In final 6th class, students played own music in
front of classmates using a Samik Piano application. Students do peer evaluation through 2 standards, flow
of melody and variation of rhythm. Also, students answered to a paper questionnaire the same with
pretest’s one.
In the conclusion, students answered they felt improvements of score reading skill and interests
about music composing. Also, students satisfied about their products as they were able to follow their own
learning speed and had enough time to think and edit. The limitations of this study were wasting time to
instruct students about iPad and worry about novelty effect.
Keywords: Score reading skill, Music composing, SMART education, iPad, Piano application
Needs Analysis
Before starting this study, students answered to a questionnaire about composing class.
<Table 1> Pretest questionnaire result
This questionnaire is designed by 4 questions and students check this. 16 of Gr.3 students
participated in this questionnaire.
1. Have you ever composed music on your
own?
Yes( very much, a little bit) : 9
No( very much, a little bit) : 7
2. Do you like music? Yes (very much, a little bit) : 11
No( very much, a little bit) : 5
3. Do you have confidence in music class? Yes (very much, a little bit) : 14
No( very much, a little bit) : 2
261
4. Can you read a staff notation? Yes (very much, a little bit) : 5
No( very much, a little bit) : 11
5. Do you have expectation about music
composing?
Yes (very much, a little bit) : 5
No( very much, a little bit) : 11
Low interests about music composing More than half of students had a music composing experience on their own. They said they had
hummed some melody not in normal music class but in break time or a home. Most of students liked
music. However, 5 students who didn’t like music were all boys. 11 students couldn’t read a staff notation.
And Most of students didn’t have expectations or interests about music composing. It was kind of ironic
that they had interests or confidence about music but they didn’t have those things about music
composing. It’s because they couldn’t have many music composing experiences before. If they had
successful experiences, they could think music composing positively and it would help to boost interests
of music and score reading skill.
Limitations of traditional music composing class Through teachers’ experiences, there were two major difficulties in traditional music composing
class. First, students can’t get music instruments individually. They can use some instruments like a
recorder or a melodeon. But students need to spend more time to learn to play a recorder. And a melodeon
has a limitation to realize students’ inspiration because that is simplified shapes of a piano. A real piano is
most easy and effective tools to compose.
Second, composing is entirely personal working. It means, students need enough time and follow
their own speed. But students needed to finish on time in traditional class. It made products’ quality low
and students feel less satisfaction and motivation about composing.
Literature Review
SMART education SMART education can support smart devices boosts collaborated learning, improving recognition
of learners, self-directed learning. (Sunhee Bang, 2012)
SMART education contains 5 compositions which are ‘Self-directed learning, Motivated, Adaptive,
Resource free, Technology embedded’. ‘Self-directed learning’ means, students can get knowledge by
themselves through teacher’s guideline. Also students can control learning process on their own.
‘Motivated’ means learners’ can be motivated by SMART education. As SMART education can help to
provide and share learners’ product immediately. In addition, SMART education can easily figure out
how many students achieve there objectives. So teachers can adaptive worksheets or review contents to
proper students. That is because SMART education is ‘Adaptive’. SMART education can provide
multimedia contents and searching system. Also, a learning group can gather students’ ideas with
technology such as cloud system or collaborative devices. These mean SMART education is ‘Resource
free’ and ‘Technology embedded’.(Jinsook Kim, 2012)
To overcome traditional music composing class’s difficulties, we can use SMART education.
Difficulties of traditional music composing were lack of individual music instrument to compose and
giving chance to control own working speed through need analysis. Among 5 compositions of SMART
education, 4 compositions(S, M, R, T) can help this problems.
Methods
This study was constructed with 6 times of music class. The whole plan of this study is like the
following.
262
Digital device and application Students used iPad and Samik piano application. Samik piano application is free contents in App
store. This application contains 88 keys of piano perfectly. Also, students can choose variety sounds.
Besides, example melody contained this application can help students to practice to play the piano and
recognize syllable names. So, this application makes students feel friendly about a piano.
Introduction and pretest In 1
st class, students recognized plans of this study and participated in pretest questionnaire. This
study composed of 6 music class.
Composing melody In 2
nd~3
rd class, students touched every keys of piano application to ready to compose own melody.
After that, they composed melody with natural flow of their mind. And then, they wrote down the syllable
names of melody in a simplified score called ‘Grid score’. ‘Grid score’ was made of 50 cubes. When they
finished to write ‘grid score’, a teacher tought them locations of tones in a staff notation.
<Picture 1> Composing melody and writing syllable names in ‘Grid score’
Composing rhythm In 4
th class, students reviewed some types of rhythm when they studied in previous music class. And
a teacher tought them how to draw notes in ‘rhythm grid score’. ‘Rhythm grid score’ is a upgraded
version of ‘grid score’ add some lines to put in notes.
<Picture 2> Composing rhythm and
write in ‘Rhythm grid score’.
Writing a staff notation Students wrote their products in a staff notation using ‘Rhythm grid score’. Many students felt
difficulties to principle of a staff notation still, so a teacher needed to teach them locations of tones in a
staff notation first. Also, it was hard to divide a staff notation through the beat, dividing the score was
skipped in this study. After writing tones, they wrote down the title of their product at the top of the score.
Composing melody by touching a
piano application freely.
Writing down own melody in
‘Grid score’.
A mental weakness student was
also participated in this study.
263
<Picture 3> Writing a staff notation in ‘Rhythm grid score’.
Peer evaluation and posttest Students played own product with a piano application or a recorder. And others evaluated by giving
a student from 1 to 5 points with 2 standards. First was ‘Is flowing of melody natural?’ and ‘Did he(or
she) use various types of rhythm?’. After peer evaluation, students were participated in posttest
questionnaire same with pretest one.
<Picture 4> Playing own product and do peer evaluation.
Findings and Interpretations
<Table 2> Posttest questionnaire result
1. Have you ever composed music on your
own?
This question is omitted because they already have
done music composing through this class.
2. Do you like music? Yes (very much, a little bit) : 0
No( very much, a little bit) : 16
3. Do you have confidence about music? Yes (very much, a little bit) : 0
No( very much, a little bit) : 16
4. Can you read a staff notation? Yes (very much, a little bit) : 0
No( very much, a little bit) : 16
5. Do you have expectation about music
composing?
Yes (very much, a little bit) : 15
No( very much, a little bit) : 1
Improvement of score reading skill Through posttest, every student answered that they could read a staff notation. Also, they could do
better in singing, playing music instruments class.
264
Improvement 4 compositions of SMART education This study could improve 4 compositions of SMART education among ‘Self-directed learning,
Motivated, Adaptive, Resource free, Technology embedded’.
Self-directed learning Students could follow own learning speed using iPad. Music composing needs enough time to think
and touching every tones to find good melody following.
Motivated
Students had rare experiences using a digital device in a class. So, there was novelty effect which
made students motivated. Also, there were a successful music composing result through peer evaluation.
12 of students got more than 8 points in 10 points and 4 students got between 6~7 points. After they got
own points, they could feel confidence , interests and motivation about music composing.
Resource free
Samik piano application was free contents in App store.
Technology embedded
Students could get the application and iPad individually.
CONCLUSION
The purpose of this study was improving students’ score reading skill in music composing class
through SMART education. To achieve this purpose, this study was conducted by SMART education to
overcome traditional composing class’s major difficulties. First, giving iPads with a piano application to
every student could help student to ‘motivate’ and give ‘resource free’, ‘technology embedded’
environment. Second, students could do ‘self-directed learning’. So, they could feel motivation about
music composing and satisfaction about their products.
Through this study, students could feel confidence and interests about music composing. Also, they
could improve score reading skill. Because they felt like that with music composing, and a music teacher
could observe they could singing and playing music instruments with score better than before this study.
In consequence, this study could fulfill the purpose by overcome limitations of traditional composing
class with SMART education. This conclusion means SMART education can enlarge the boundary of
teaching and learning realization.
The limitations of this study are, first, students needed much time to get used to iPad. It was first
time to input iPad to Gr. 3 students, they felt embarrassed when the application’s gone and didn’t know
basic instruction. So, if I would do this type of teaching again, I should teach them how to use digital
devices and applications above all. Second difficulty was novelty effect. Although this study was focused
on improvement of score reading skill, if this study input to classes which already got used to digital
devices like iPad, they might not feel much motivation and not achieve the purpose with less motivation
in the end. So, we need to think more how to motivate students not leaning on novelty effect from digital
devices in SMART education.
REFERENCES
Takako Mato(2009), Intergrating technology in the music classroom.
Sunhee Bang(2012), A study on strategies of self-directed learning to promote smart learning, In journal
of lifelong learning society.
Takako Mato(2009), Integrating technology in the music classroom, St. Mary’s college of Maryland.
Jinsook Kim(2012),SMART education school guidebook, KERIS.
265
The Effect of Awareness Information on Affect-based Trust
in Collaborative Problem-solving Learning: A Pilot Study
Jongsuk Song
Doctoral student
Hanyang University
Seoul, Republic of Korea
Dongsik Kim
Professor
Hanyang University
Seoul, Republic of Korea
ABSTRACT
This study was conducted as a pilot purpose. The initial purpose of the study was to explore the
effect of awareness information on affect-based trust and the development of competence-based trust.
However, it is beyond the scope of this study to examine the relationship between awareness information
and the development of competence-based trust. Two groups of four participants were involved. They
were asked to find solutions concerning school violence and bullying problems. As a result of the pilot
study, five issues (time limit, facilities, technical problems, selection of participants, and measuring
instruments) were revealed and should be more examined to achieve the purpose of this study.
Keywords: Collaborative learning, awareness information, affect-based trust
INTRODUCTION
Trust is one of the influential factors to determine the success of collaborative learning (Mayer,
1995; Kanawattanachai, 2002; Ge & Hsieh, 2005; Fransen et al., 2013). Of various dimensions of trust,
emphasis is placed on affect-based trust grounded in emotional bonds and friendship in this study.
Though emotional relationship or friendship among collaborators is one of the pivotal factors in
collaborative learning (Hartup, 1992, Kratzer et al., 2005), most of the current studies have shown that
there are several disadvantages. First, it is likely lead to free-riding or dominating effects, that is, group
members' attitude toward collaboration. (Fransen et al., 2013). Molleman (2005), for instance, argued that
affect-based trust may help strengthen mutual trust in a group however, it tends to result in someone’s
dominating behaviors (Fransen et al., 2013). While free-riding tends to occur when learners put less effect
into group learning due to their excessive trust or expectations on others' ability, dominating effects refers
to when one or two members dominate the entire group processes or activities (Dillenbourg, 2005) since
they may be overconfident about their ability in comparison with the others'.
To reduce the disadvantages of the negative aspects related to affect-based trust, one of the
instructional methods, as argued by Aggarwal (2004), is group composition. For free-riding effect, in
addition, the selection of learning tasks can be another alternative as there is high relevance between free-
riding and motivation (Wsezey et al., 1994). Similarly, Weibel and Frederique (2013) argued that high
interest in learning task can result in reduced free-riding behaviors.
Unlike the emphasis on the attitudes of collaborators’ activities toward collaborative learning, this
study was intended to highlight another aspect, that is, their converging activities such as discussion,
negotiation, and final decision making in collaborative problem-solving. Affect-based trust is often
associated with blind or biased trust, which may hinder balanced and accurate judgment or evaluation
(Nooteboom, 2002). Nooteboom (2002) further argues that such trust is likely lead to delusion which
266
keeps them from discerning the possibility of fallacy. In collaborative problem-solving environment, for
example, some contributions can be overestimated despite of possible errors or mistakes since the
contributors are considered the higher achieving ones, and vice versa.
It seems that group composition or choice of task discussed earlier is a possible instructional
solution however, awareness information is suggested as another possible answer in relation to the impact
of affect-based trust on converging processes of collaborative learning. Awareness information is defined
as information regarding collaborators' skills, knowledge, social activities, etc. (Janssen & Bodemer,
2013). In other words, they get information on who contributes more or less, who has more information,
who knows better about the given task, etc. When considering that students are mostly engaged in
collaborative learning under regular courses with any forms of grades and credits, it can be assumed that
they are sensitive to group performance and its results. This study, therefore, expects they use awareness
information when engaging in converging activities and less dependence on their affect-based trust,
probably resulting in biased and incorrect judgment and evaluation during converging processes.
RQ: To what extent does the provision of awareness information affect the collaborating learners’
dependence on their affect-based trust during converging activities?
METHOD
Two groups of four participants were involved in the pilot study. Participants in a control group
were 4-year college (S Univ.) students majoring in Education. Participants in an experimental group were
H middle school students. Each experiment was conducted in different places and at different times (the
control group on S Univ. campus on Oct 1st and the experimental group in a study room in Hanam City on
Oct 12th)
.
Both groups were given the same problem-solving task, “Solutions to school violence and bullying”.
The original task was to seek solutions from five groups concerned in the problem, that is, peer students,
teachers and schools, parents, local community, and government. While the control group was given this
original task, only two parties were included in the learning task of the experimental group for reasons of
(1) the procedures of collaborative learning in control group were simpler than the ones of the
experimental group, and (2) the participants in the experimental group are middle school students, so it
was taken into account that there would be need for the ease of the burden by the learning task.
There were two phases of collaboration, sharing the task-related materials and finding solutions
through converging activities including discussion and negotiation. The control group was only engaged
in the second type of collaboration (individual work in the rest of the task), instead, the experimental
group performed both.
Two online mapping tools were employed, Minddomo (www.minddomo.com) and Groupzap
(www.groupzap.com). To measure participants’ affect-based trust, Affect-based trust (5 items) in
McAllister’s Interpersonal Trust Measure and McGill Friendship Questionnaire – Friendship Functions
(MFQ-FF) was also used. To collect awareness information-related data, this study focused on two parts,
the number of interaction occurred for each contribution and each individual’s score earned (students
evaluate each others’ solutions at the final phase of collaborative learning on a scale of 1 to 5).
LESSIONS LEARNED
To achieve the purpose of this study, five significant aspects should be taken into consideration:
time limit, facilities, technical problems, selection of participants, and measuring instruments. The
following problems, in particular, were identified in the pilot study. First, students wanted to make sure
that they invest limited amount of time (3hrs), as a result, did not have enough time to concentrate on the
task. Also to meet their request, the researcher had to conduct an orientation session and the main learning
session at once. Eventually, data collection was partly failed, particularly for control group (Time limit).
Second, it was hard to find an appropriate place to carry out the experiment since it was not allowed
to use any computer laboratory on the campus. Fortunately, college students in the control group brought
their own laptop, however middle school students did not have their own laptop, so the researcher had to
find any available laptops and provide them (Facilities).
267
Next, during the control group experiment, Minddomo interface suddenly did not provide “invite”
icon even though the researcher made sure that there was not any technical problem using Minddomo in
two time previous test. As a result, the control group failed to proceed with their collaborative learning,
particularly the only sharing phase. To cope with the occurrence of the same problem, the researcher
replaced it with Groupzap for the experimental group, which is much simpler and provides less panels
and icons (Technical problems).
Fourth, both groups were consisted of different age groups since it was also difficult to recruit
participants. Therefore, the study may be not ready enough to answer any questions in terms of the
selection of participants such as “Are there any significant issues to affect the outcomes with respect to
recruitment of participants?”, etc. (Selection of participants).
Finally, there were two main issues were revealed concerning measuring instruments: (1) McGill
questionnaire was used to support or provide any detailed information for McAllister’s affect-based trust
consisted of too small number of items. However, there were no meaningful differences between the
results from the two instruments. In addition, many items in the McGill questionnaire seemed too similar
to one another to participants though deep investigation may show slight difference between them. Even
the research explained the differences, they still looked confused. (2) Another issue is associated with the
validity of their ratings. The researcher provided a descriptive-type questionnaire as an extra data
collection to the experimental group. Of four questions, they were asked to choose one of the group
members in two questions. Three students answered just “All”, which possibly indicates rating inflation
in order to avoid awarding low scores on the others.
CONCLUSION
This pilot study suggests that: (1) participants should be provided with enough time and appropriate
learning environments in order to concentrate on the give task; (2) the researcher should prepare available
alternatives to cope with any possible technical problems concerning an online mapping tool; and (3) for
more valid and accurate data collection, measuring instruments need to be more developed, or additional
instruments can be included.
REFERENCES
Kratzer, J., Leenders, R.T.A.J. and Van Engelen, J.M.L. (2005), 'Informal contacts and performance in
innovation teams', International Journal of Manpower, 26(6), 513–28.
Aggarwal. (2004). Educational Technology. Sarup & Sons: New Delhi.
Antoinette Weibel & Frederique Six. (2013). Trust and Control: the Role of Intrinsic Motivation.
Reinhard Bachmann & Akbar Zaheer (eds.). Handbook of Advances in Trust Research, pp.57-81.
Edward Elgar Publishing, Inc: Massachusetts.
Hartup, W. (1992) “Having Friends, Making Friends, and Keeping Friends: Relationships as Educational
Contexts.” ERIC Digest, ED345854. Urbana, IL: ERIC Clearinghouse on Elementary and Early
Childhood Education. Available at http://bern.library.nenu.edu.cn/upload/soft/0-article/013/14021.doc,
accessed on Oct 22, 2013.
Janssen, J., & Bodemer, D. (2013). Coordinated computer-supported collaborative learning: awareness
and awareness tools. Educational Psychologist, 48(1), 40-55.
Jos Fransen, Armin Weinberger & Paul A. Kirschner. (2013). Team effectiveness and team development
in CSCL. Educational Psychologist, 48(1), 9-24.
McAllister, D. (1995). Affect- and cognition-based trust as foundations for interpersonal cooperation in
organizations. Academy of Management Journal, 38(1), 24-59.
Nooteboom, B. (2002), Trust: Forms, Foundations, Functions, Failures and Figures, Edward Elgar:
Cheltenham.
Pierre Dillenbourg. (2005). Designing Biases That Augment Socio-Cognitive Interactions. Computer-
Supported Collaborative Learning Series, 5, 243-264.
Robert W. Swezey, Andrew L. Meltzer, and Eduardo Salas. (1994). Some Issues Involved in Motivating
Teams. Harold F. O'Neil & Michael Drillings. (eds.). Motivation: Theory and Research. pp.141-170.
Routledge: New Jersey.
268
Student's Perception on Learning Analytics Dashboard (LAD)
Presenting Online Activities in LMS
Stephanie Kang [email protected]
Graduate Student
Ewha Womans University
Seoul, Korea
Yeonjeong Park
Research Professor
Ewha Womans University
Seoul, Korea
Il-Hyun Jo
Associate Professor
Ewha Womans University
Seoul, Korea
ABSTRACT
The purpose of this study is to investigate how students react and perceive a dashboard treatment
“Learning Analytics Dashboard (LAD)” designed and developed by the researchers. 37 students in a large
university were invited as subjects. LAD represents individual learners’ activities such as online activity
summary, total log-in time, total log-in frequency, log-in regularity, visits on board, time spent on board,
and visits on repository. These data were analyzed and visualized by statistical algorithms and provided as
graphs on LAD. The survey for students’ perception and understandng were conducted as data collection
method and the repeated measure ANOVA was conducted as data analysis method. We found that
students easily understood the graphs and information in LAD, and felt medium conformity between the
actual online activities shown on the graphs and the online activity they perceived themselves. Also,
participants’ perceived usefulness and degree of understanding on each item were significantly different,
while the difference among items in conformity was not significant. This study is meaningful in terms of
suggesting implication for the development of more refined and effective dashboard treatment, and
providing specific directions for future research.
Keywords: Dashboard, Data mining, LMS
INTRODUCTION
As e-Learning develops, the role of Learning Management Systems (LMS) has expanded from a
provider of contents to a system that helps us managing contents and learning process at the same time by
enabling learners to participate in various learning activities and to manage their own learning time and
place in cyber space (Britton & Tesser, 1991; Zaiane & Luo, 2001). Since an increasingly large number
of educational resources have moved to online, an enormous amount of student’s behavioral data is
accumulated as web-log files. Learning Analytics is an emerging field that extracts valuable information
and knowledge from web-log data to improve students’ learning performance.
In relation to learning performance, learning analytics has addressed a range of issues such as
mining processes, performance prediction, and outlier detection. Since its recent birth, learning analytics
269
research has been flourishing in the area of analysis of learner behaviors and prediction for the outcomes
(Baker & Yacef, 2009; Siemens & Long, 2011). However, the research has yet gone further to the
development of instructionally effective treatment based on the analysis and prediction efforts. Design
and development of effective interventions to control the outcome of the behavior based on the analysis
and prediction efforts should be educational technologists’ ultimate task.
In educational research field, various kinds of educational treatments including tasks, learning
environment, instructional methods have been suggested. Among them, a visualized dashboard utilizing
data mining is widely used because it helps improvement of self-knowledge by providing information
about learner’s own activity (Verbert, Duval, Klerkx, Govaerts, & Santos, 2013), and it promotes self-
evaluation and self-encouragement (Duval, 2013). For example, Essa & Ayad (2012) analyzed three
factors of learner’s success and classified at-risk student, then provided those information with graphs.
Leony and his colleagues (2012) also introduced GLASS, the open-source program which analyzes and
visualizes leaner’s web-log data. However, unfortunately, these statistical representations were not
intuitively understood (Koulocheri & Xenos, 2013). Previous studies suggest thorough investigation on
how the learners perceive and react to the dashboard treatment.
The present study is to investigate the student’s perception on a visual dashboard system designed
and developed by the researchers. 37 students from two different classes in a large university were invited
as subjects. Our online dashboard called “Learning Analytics Dashboard (LAD)” represents individual
learners’ activities such as online activity summary, total log-in time, total log-in frequency, log-in
regularity, visits on board, time spent on board, and visits on repository. These data were analyzed and
visualized by statistical algorithms. The results were provided as dashboard treatment and the survey for
student perception and understanding was conducted.
METHOD
Participants 37 college students in E women’s university in Seoul, Korea were provided LAD via LMS of the
university. Answers of 22 students who answered to the survey were analyzed. Due to the nature of
university, all participants were women, and their year in school were considerably evenly distributed.
Dashboard Development The researchers have collaborated with Institute for Teaching & Learning (ITL) of E university on
LAD development project. The dashboard consists of 7 graphs: each representing the online activity
summary, total log-in time, total log-in frequency, log-in regularity, visits on board, time spent on board,
and visits on repository. As shown in Figure 1, the graph of online activity summary is the scatterplot that
individual learners can choose X-axis and Y-axis to locate their position in class. Other 6 graphs are
provided with trend line of their activity of every week along with average activity information of other
learners in the class. All of graphs in LAD are planned to update every week until the end of semester.
See Figure 1 for the screenshot of LAD. Name of student is covered to protect personal information.
Measurement Survey questionnaire was developed by researchers to measure conformity, perceived usefulness,
and degree of understanding. It consists of 24 questions (21 Likert 5-scale questions and 3 open-ended
questions) to gather the participants’ opinions and suggestions on LAD. Table 1 shows the summary of
survey questionnaire.
Procedure After development of LAD, the researchers provided a brief instruction regarding the dashboard and
asked for the survey verbally and by e-mails. At this point, the participants could access LAD freely (and
repeatedly if they want) via LMS they have been used. They were asked to complete the survey in 5 days
after they were provided LAD and 22 out of 37 participants completed the survey. The researcher
analyzed and compared the means of each questions.
270
Figure 1. Screenshot of “E Learning Analytics Dashboard (LAD)”
<Table 1> Summary of Survey Questionnaire
Part Contents Example N of
Questions Measure
Conformity Degree of conformity
between learner’s
perceived online activity
and real data
How much the total log-in
time graph conforms to your
perceived total log-in time? 7 Likert 5-Scale
Perceived
Usefulness
Degree of learner’s
perceived usefulness of
the information in LAD
How much do you think the
total log-in information will
help your learning process?
7 Likert 5-Scale
Degree of
Understanding
Degree of learner’s
understanding of the
graphs in LAD
How difficult it is to fully
understand the total log-in
time graph?
7 Likert 5-Scale
Opinion and
Suggestion
Participant’s opinion and
suggestion on LAD
Please suggest any opinion
you have on this dashboard. 3 Open-ended
RESULTS
As shown in Figure 2, in comparing the conformity, perceived usefulness, and degree of
understanding for all graphs in LAD, the result indicates that higher degree of understanding(Total Mean
=4.10), relatively lower degree of the perceived usefulness(Total Mean=3.22), and medium level of
271
conformity (Total Mean=3.70). That is, students perceived the information in dashboard reflected their
online activities in LMS. While they could understand the meaning of graphs quite well, it was somehow
weak to perceive this information as useful tools for their learning and performance.
Figure 2. Overview for Comparison (n=22)
<Table 2> Descriptive statistics of conformity, perceived usefulness, and level of understanding (n=22)
Categories
Variables
Conformity Perceived Usefulness Degree of
Understanding
Mean SD Mean SD Mean SD
Online Activity Summary 3.36 .790 3.36 .790 3.73 1.077
Total Log-in Time 3.55 .963 2.68 .839 4.23 .869
Total Log-in Frequency 3.86 .774 3.09 .971 4.32 .839
Log-in Regularity 3.73 .827 2.59 1.008 3.86 1.125
Visits on Board 3.77 .922 3.68 .839 4.14 .990
Time Spent on Board 3.73 .935 3.27 1.032 4.14 .941
Visits on Repository 3.91 .684 3.86 .889 4.32 .839
Total Mean 3.70 .842 3.22 .910 4.10 .954
Conformity The answers of conformity questions were moderately high (Total Mean=3.70) in general. Among
them, the online activity summary (Mean=3.36) and the total log-in time (Mean=3.55) showed relatively
low level of conformity. There were relatively more students who answered the total log-in time and the
log-in frequency information in LAD were different from what they though. In regard to this result, we
suspect that students often stayed online without logging-out even after they achieved their goal of action.
However, this result needs to be carefully observed since the result of repeated measures ANOVA
indicated that no significant mean difference among the 7 variables was detected (F5 = 1.20, p > .05).
272
Perceived Usefulness Students generally have medium perception on the usefulness of LAD (Total Mean=3.22). It means
that they think and expect the information from LAD would help their learning or learning process
somehow but not very much. Follow-up repeated measures ANOVA for Perceived Usefulness detected
significant large mean difference among the 7 variables (F5 = 13.52, p < .001). Participants perceived that
most useful item was the visit on repository, followed by the visit on board, the online activity summary,
the time spent on board, the total log-in frequency, the total log-in time, and the log-in regularity,
respectively. The total log-in time (Mean=2.68) and the log-in regularity (Mean=2.59) showed lowest
level of perceived usefulness. Also, perceived usefulness of the log in regularity is the only question that
received extremely negative answer, “I do not think this information will help my learning at all”.
Degree of Understanding In terms of degree of understanding, students felt it is easy to understand most of the graphs in LAD
(Total Mean=4.10). Another repeated measures ANOVA for Degree of Understanding reports noticeable
significant mean difference among the 7 variables (F5 = 3.17, p < .01). Degree of understanding presented
highest on the total log-in frequency and the visit on board, followed by the total log-in time. The visit on
board and the time spent on board have same mean values, and the lowest degree of understand was
shown in the log-in regularity and the online activity summary. There were many negative answers on the
online activity summary (Mean=3.73) and the log-in regularity (Mean=3.86) suggesting that students
were experiencing difficulties to fully understand those two graphs. Since the online activity summary is
the scatterplot that users should choose X-axis and Y-axis by themselves, it can be assumed that it might
be difficult to use it and understand the information without detailed explanation or manual. In case of the
log-in regularity, the difficulty that students experienced was due to the misunderstanding of the concept
itself. To solve this problem and help students’ understanding, it is necessary to provide the manual of
LAD including the concept of each item and how it is presented as graph.
Discussion & Conclusion
In this study, we found that students easily understood the graphs and information they represent,
and felt medium conformity between the actual online activities shown on the graphs and the online
activity they perceived themselves. Regarding the result of the repeated measure ANOVA, participants
answered consistently on all items on conformity. However, the answers for the open-questions provide
us meaningful implications. In the answers of open-ended questions, students mentioned: “total log-in
time is not the same as my study time”. Such response confirms that even though the time spent on
learning is important for students’ learning achievement (Rau and Durand, 2000), the time they stay
logged-in on LMS does not necessarily mean the time they spent on learning (Cotton and Savard, 1981).
Furthermore, unlike our expectation that participants would perceive LAD useful and feel easy to
understand on all items, the results of the repeated measure ANOVA represent that participants’ different
perceptions on perceived usefulness and degree of understanding. Specifically, participants answered that
the log-in regularity would not help their learning. This result is contradictory with the result of preceding
research that the log-in regularity predicts higher learning achievement (Jo & Kim, 2013). Considering
the answers for open questions (i.e. “It is hard to understand log-in regularity”, “What is the unit or
scale?”, etc.) and low degree of understanding on the log-in regularity, it is possible that participants
experienced difficulty to understand intuitively, so that they did not apprehend the concept of the regular
learning which presented in LAD.
Regarding to the purpose of this study to investigate the students’ perception and understanding on
LAD, in spite of relatively lower levels of perceived usefulness shown in this study, it is remarkable that
the results of the survey showed high perceived usefulness and degree of understanding. Only one
extremely negative answer was show up throughout the survey, even the relatively lower category
showed moderate positiveness. It indicates that students reacted to LAD positively in general, so that we
expect the quality and potential effectiveness of LAD.
The implications of this study raise the needs of empirical research on actual effect of LAD. More
specifically, whether this graphs and information they represent would help students’ learning or not
273
should be clarified. With those future researches, we would be able to refine or change the items and
information we present on LAD to improve its usefulness. For this process, we need to consider that
participants’ perceived usefulness and degree of understanding on each item are significantly different,
while the difference among items in conformity was not significant. It is necessary to carefully examine
the reasons of these results by follow-up empirical researches. In last, another comment from open-ended
question such as “the usefulness is up to how each class utilizes LMS” implies the usefulness and effect of
LAD would be very different between 100% online class and blended learning class. Therefore, as a
further study, it is necessary to examine the effect of LAD empirically in different contexts to prove its
usefulness more precisely.
In conclusion, this study analyzed how end-users react and perceive developed dashboard treatment.
This study is meaningful in terms of suggesting implication for the development of more refined and
effective dashboard treatment, and providing specific directions for future research.
REFERENCES
Baker, R. S. J. D., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future
visions. Journal of Educational Data Mining, 1(1), 3-17.
Britton, B. K., & Tesser, A. (1991). Effects of time-management practices on college grades. Journal of
educational psychology, 83(3), 405.
Cotton, K., & Savard, G. W. (1981). Time factors in learning. research on school effectiveness project:
Topic summary report. Retrieved from
http://search.proquest.com.access.ewha.ac.kr/docview/63544228? accountid=10785
Essa, A., & Ayad, H. (2012). Student success system: risk analytics and data visualization using
ensembles of predictive models. In Proceedings of the 2nd International Conference on Learning
Analytics and Knowledge. 158-161.
Jo, I. H., & Kim, Y. (2013). Impact of Learner’s Time Management Strategies on Achievement in an e-
learning Environment: A Learning Analytics Approach. The Journal of Educational Information
and Media, 19(1), 83-107.
Koulocheri, E., & Xenos, M. (2013). Considering formal assessment in learning analytics within a PLE:
the HOU2LEARN case. In Proceedings of the Third International Conference on Learning
Analytics and Knowledge (pp. 28-32). ACM.
Leony, D., Pardo, A., de la Fuente Valentín, L., de Castro, D. S., & Kloos, C. D. (2012). GLASS: a
learning analytics visualization tool. In Proceedings of the 2nd International Conference on
Learning Analytics and Knowledge. 162-163.
Rau, W., & Durand, A. (2000). The academic ethic and college grades: Does hard work help students to"
make the grade"?. Sociology of Education, 19-38..
Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. Educause
Review, 46(5), 30-32
Verbert, K., Duval, E., Klerkx, J., Govaerts, S., & Santos, J. L. (2013). Learning Analytics Dashboard
Applications. American Behavioral Scientis, 57(10). 1500-1509.
Zaïane, O. R., & Luo, J. (2001). Towards evaluating learners' behaviour in a web-based distance learning
environment. In Advanced Learning Technologies, 2001. Proceedings. IEEE International
Conference on (pp. 357-360). IEEE.
274
275