introducing the indicators project

The Indicators ProjectIdentifying effective learning: adoption, activity, grades and external factors

Colin BeerDavid JonesKen Clarkhttp://tr.im/FvRQ

Overview

http://www.flickr.com/photos/atbaker/1577665185/

Context

Feature usage

Participation, grades

and other factors

Future work and Reflections

Overview

Context

Feature usage

and other factors

http://www.flickr.com/photos/dnorman/251646154/

The data that we, educators gatherand utilize is all but garbage.(Wiley,

http://www.flickr.com/photos/riverap1/3875718119/

almost every university is planning to make use of an LMS (Salmon,

They are becoming ubiquitous at

universities around the world(Coates et al, 2005)

http://www.flickr.com/photos/dsifry/2452527887/

Are sites ready for start of term?

Are students using the site?How?

With what impact?

What makes “good” LMS use?

Are thereproblems?

If I do “x” what happens?

Is it worth it?

http://www.flickr.com/photos/80417459@N00/2471314610/

What’s going on within your LMS/course site?a) Asking staff and/or

studentsb) Manual checking of LMSc) LMS reporting facilitiesd) Data mining + other analyticse) Opinion/self-reportingf) Otherg) None

Mostly ad hoc

Some surveys

Mostly LMS reporting

One manual check (Gonch,

Peer review?

Mostly ad hoc

Some surveys

No widespread view

Mostly ad hoc

Some surveys

No widespread view

Only what is in the LMS

Limited views

Mostly ad hoc

Some surveys

No widespread view

Human recollection

Limited views

Mostly ad hoc

Some surveys

No widespread view

Human recollection

Time-consuming & error prone

Limited views

Mostly ad hoc

Some surveys

No widespread view

Human recollection

No lognitudinal examination

Limited views

Mostly ad hoc

Some surveys

No widespread view

Human recollection

No lognitudinal examinationNo cross LMS comparison

Limited views

Mostly ad hoc

Some surveys

No widespread view

Human recollection

No lognitudinal examinationNo cross LMS comparison

No cross institutional comparison

Limited views

“LMS”

Webfuse – 1997-2009

Pre-2010

WebCT – 1999-2003/4Blackboard – 2004-2009

Moodle

http://www.flickr.com/photos/anders-vindegg/3408838186/

We have lots of data

Examining it might reveal interesting patterns

Useful for L&T practice, management and research

http://indicatorsproject.wordpress.com

Enabling comparisons of LMS usageacross institutions, platforms and

http://www.flickr.com/photos/david_jones/36379215/

Institution A

LMS 1 LMS 2

Institution B

Institution C

LMS Independent

Useful Data/Informat

T&L Practice Research

Institution A

LMS 1 LMS 2

Institution B

Institution C

LMS Independent

Enabling

Institution A

LMS 1 LMS 2

Institution B

Institution C

LMS Independent

Enabling

Institution A

LMS 1 LMS 2

Institution B

Institution C

LMS Independent

Enabling

Institution A

LMS 1 LMS 2

Institution B

Institution C

LMS Independent

Enabling

WhyHow

Overview

Context

Feature usage

and other factors

Future work

http://www.flickr.com/photos/chokingsun/3495110670/

There are more similarities than differences among LMS (Black et al,

TransmittingContent

CreatingClass

Interactions

ComputerBased

Instruction

Move to features in unexplored categories, until each category is considered for different learning needs

Level 1: Most used Level 2: Moderately used Level 3: Rarely used

• Content/files• Announcements• Gradebook

More than 50% of courses More than 20% of courses (much) less than 20% of courses

EvaluatingStudents• Quizzes• Drop box

Evaluatingcourse and instructors

• Surveys• Chat• Discussion forum• Email

• Some quizzes• Adaptive release

(Malikowski et al, 2007)

What we did

Wf BbLMS logs

LMS Independent

What we did

Wf BbLMS logs

LMS Independent

What we did

Feature usage

Malikowskitop & bottomrange

Blackboard

Webfuse

http://www.flickr.com/photos/wadem/2730257498/

The particular trajectory of emergence is not wholly

determined either by the intentions of the human actors or by the material properties of technology, but rather by the interplay of the two(Jones,

http://www.flickr.com/photos/nedrichards/3234490934/

Future Work

Will Moodle be different?

What about other institutions?

Are Malikowski ranges out of date?

When is a feature adopted?

…….

Overview

Context

Feature usage

and other factors

http://www.flickr.com/photos/wolfgangstaudt/2279651479/

The results indicated a significant difference between low and high performing students in terms of the quantity of online session times attended during the course (Dawson et al,

Discussion forum activity has been demonstrated to be a sound indicator of future student academic performance. (Dawson, 2009; Morris et al,

Wf BbLMS logs

LMS Independent

What we did

Feature usage

Wf BbLMS logs

LMS Independent

What we did

Feature usage

Student records

Wf BbLMS logs

LMS Independent

What we did

Feature usage

Participation, grades,external factors

Student records

Hits on course site

Hits on course forum

Distance education students

CQ on-campus students

AIC students

Distance education students

# of replies

# of posts

CQ on-campus students

AIC students

Mixed design ANOVA

All reported effects are significant

Significant interaction effectbetween grades and participation

AIC = emphasis on face-to-face

FLEX/DE = no face-to-face

CQ = Quite a bit emphasis face-to-face

Course site designedby CQ-based staff(usually)

Taxonomy of critical success factorsfor quality web-supported learning(Fresen,

Lecturer interaction/facilitationAcademic background of lecturer/instructional design

http://www.flickr.com/photos/orinrobertjohn/3321067724/

Lecturer interaction/facilitationAcademic background of lecturer/instructional design

Course groups based on staff hits

Group Courses Student-courses

Students/course

Definition

High 678(17%)

156,770 231 x>3000 hits

Medium 980(25%)

143,314 146 3000>x>1000

Low 1474(37%)

102,214 69 1000>x>100

Super low 849(21%)

68,354 80 100>x>0

High staff participation courses

Replies & posts

High staff participation courses

Super low staff participation courses

Replies & posts

Super low staff participation courses

Posts comparisons

Superlow

Future Work

- Course size

Examination of super-low

Networks of interaction (Bakharia et al, 2009; Dawson,

Forum content

Lecturer interaction/facilitationAcademic background of lecturer

and instructional design

Course offerings with

Staff with grad certs in L&TSignificant input from instructional designerStaff who have received teaching awards

Staff teaching education coursesNone of the above

n=3288

None of the above

Grad cert in L&T

Teaching awards

Education discipline

Instructional design

Overview

Context

Feature usage

and other factors

Future work & Reflections

http://www.flickr.com/photos/mikebaird/2985066755/

http://www.flickr.com/photos/david_jones/36379215/

http://www.flickr.com/photos/dmatos/540688952/

http://www.flickr.com/photos/mwichary/2188958154/

http://www.flickr.com/photos/sylvar/1138341328/

http://www.flickr.com/photos/fncll/145149313/

Institution A

LMS 1 LMS 2

Institution B

Institution C

LMS Independent

WhyHow

EnablingLittle

Little bit

Little bit Not

http://www.flickr.com/photos/fncll/145149313/

http://www.flickr.com/photos/jamuraa/813966437/

http://tr.im/FvRQ

http://indicatorsproject.wordpress.com

Black, E., D. Beck, et al. (2007). "The other side of the LMS: Considering implementation and use in the adoption of an LMS in online and blended learning environments." Tech Trends 51(2): 35-39.

Coates, H., R. James, et al. (2005). "A Critical Examination of the Effects of Learning Management Systems on University Teaching and Learning." Tertiary Education and Management 11(1): 19-36.

Dawson, S. (2009). "'Seeing' the learning community: An exploration of the development of a resource for monitoring online student networking." British Journal of Educational Technology: in press.

Dawson, S., E. McWilliam, et al. (2008). Teaching smarter: How mining ICT data can inform and improve learning and teaching practice. Hello! Where are you in the landscape of educational technology? Proceedings ascilite Melbourne 2008. Melbourne: 221-230

Fresen, J. (2007). "A taxonomy of factors to promote quality web-supported learning." International Journal on E-Learning 6(3): 351-362.

Gonch, B. (2007). An inventory of Term 2 2006 online courses at CQU. Rockhampton, Central Queensland University: 18.

Jones, M. (1999). Information systems and the double mangle: Steering a course between the scylla of embedded structure and the charybdis of strong symmetry. Information Systems: Current Issues and Future Chalenges. T. Larsen, L. Levine and J. DeGross. Laxenburg, Austria, IFIP: 287-302.

Malikowski, S., M. Thompson, et al. (2007). "A model for research into course management systems: bridging technology and learning theory." Journal of Educational Computing Research 36(2): 149-173.

Morris, L., C. Finnega, et al. (2005). "Tracking student behavior, persistence, and achievement in online courses." Internet and Higher Education 8(3): 221-231.

Salmon, G. (2005). "Flying not flapping: a strategic framework for e-learning and pedagogical innovation in higher education institutions." ALT-J, Research in Learning Technology 13(3): 201-218.

Wiley, D. (2009). "The LHC and education." Retrieved 29 November, 2009, from http://opencontent.org/blog/archives/1098.

introducing the indicators project

good lms use

factorsfuture workhttp

lmscourse site

lmsblack et

worldcoates et

moderately usedlevel

interesting patternsuseful

lt practice

Education

introducing e - project maths

community project indicators framework (cpif)

union community indicators introducing the inaugural report

vital tillamook indicators project

introducing the recode project

introducing energy indicators - sustainable development ·...

introducing the mpa guidebook’s governance indicators...

introducing… scaling-up project project critical critical

entrepreneurship indicators project entrepreneurship...

community indicators project 2012-2013

progress report: health care quality indicators project ·...

the trade indicators project (tip)

aidsfree project monitoring and evaluation indicators...

community indicators project 2009-2010

introducing the ceylon project

introducing project director ss

introducing project time management

introducing conceptdraw project · introducing conceptdraw...

equity indicators project update - dallas

introducing the ianex project