Cooperative Learning Object Exchange (CLOE)

A Learning Object Project Case Study

Peter Tittenberger, July 2009

Snapshot history

• 1997?: 3 Ontario institutions join MERLOT as an institutional consortium [Guelph, Waterloo, York]. Each pays 1/3 of the membership fee

• 1998: 8 Ontario institutions in MERLOT consortium, each contributing share of fee + faculty time to participate

• 1998-2000: grant from TeleLearning NCE to investigate further community and collaboration models

• 1999: grant from Industry Canada for students to work on "learnware" • 2001-2003: grant from HRDC Office of Learning Technologies to fund

learning object development with some investment in infrastructure for CLOE

• 2004-2005: grant from Inukshuk to collaboratively develop learning objects

• 2006-2007: UWaterloo continues to subsidize CLOE operation, no success in identifying sources for ongoing funding

• 2008: operation effectively ceases, although website available until domain registration expires.

Via Tom Carey email, July 2009

CLOE included:

1. Collaborative design and learning object development

2. Peer Review process3. Learning object repository4. Administrative infrastructure

List of CLOE institutions:

Brock UniversityCarleton UniversityConestoga CollegeDurham CollegeUniversity of GuelphLakehead UniversityLa Cité CollégialeLambton CollegeLaurentian UniversityMcMaster UniversityMemorial University of NewfoundlandMinistry of Science and Technology, ThailandNiagara CollegeNipissing University

Queen's UniversityRyerson UniversityWendy FreemanSeneca CollegeSheridan CollegeTrent UniversityUniversity of ManitobaUniversity of Ontario Institute of TechnologyUniversity of OttawaUniversity of TorontoUniversity of WaterlooUniversity of Western OntarioUniversity of WindsorWilfrid Laurier UniversityYork University

1. Collaborative design and learning object development

• Grant funded– 2004-2005 Inukshuk fund ($2 million)

• Camp Cloe– a one week Instructional Design workshop.

CLOE partners attend and work in teams on the collaborative design of LOs and then these LOs are developed over the year.

Camp Cloe

Instructional Design & Learning Technology Workshop   •Learning Object Design Process     •Mock Learning Objects     •Project Management     •Understanding Learners     •Prototyping & User Testing     •Electronic Prototyping

http://web.archive.org/web/20051029195100/tlc.uwaterloo.ca/projects/cloe/CaseStory/

CLOE design and development methodology created a rigorous set of standards for learning object creation.

http://www.brocku.ca/learningobjects/flash_content/index.html

http://www.brocku.ca/learningobjects/flash_content/LO/Stroop_Effect.html

Access Restrictions:

These Learning Objects are being developed by ITS (Information Technology Services) in conjunction with CTLET (Centre for Teaching, Learning and Educational Technologies). All objects represent works near completion. As they are currently under development, they are authorized for use by Brock University personnel only, with permission.

Completed learning objects are peer reviewed and can  be accessed through CLOE (Co-Operative Learning Object Exchange). Please contact CTLET if you would like to apply for full access to CLOE or if you would like more information on Brock University's learning object development.

http://www.brocku.ca/learningobjects/flash_content/index.htmlLast updated: Friday, 01-Oct-2004 16:16:30 EDT

Form to add object to CLOE

2. Peer Review

CLOE partners concluded that a hallmark of CLOE should be that all LOs in CLOE should be Peer Reviewed.

CLOE created its own Peer Review process modeled after the MERLOT Peer Review process (http://taste.merlot.org/catalog/peer_review/)

The CLOE Guidelines to Authors are used by both (a) creators of LOs to inform their design and development process and (b) CLOE Peer Reviewers when conducting a peer review.

The process that the CLOE Editor-in -Chief uses when a new LO is submitted to CLOE:

Guidelines for authors creating los and for peer reviewers evaluating los.

CLOE peer review process created a rigorous methodology and set of standards for learning object submission and review.

3. Learning object repository

• Stand alone web site.• Access restricted.• No federated search.• D2L arrangement.• Reuse procedure (form).• Licensing.

CLOE lo ‘borrowing’ form.

4. Administrative infrastructure

• UWaterloo provided director, admin support and technical components and support

• Organized and hosted Camp Cloe• Memoranda of Understanding (MOUs)

Memorandum of Understanding

• When partners join CLOE a high-level official at that institution signs the CLOE Memorandum of Understanding (MOU).

• The original MOU stated that each partner’s primary contribution was to submit LOs to CLOE and to reuse LOs from CLOE.

Memorandum of Understanding

• Potential new partners indicated that they wanted to be full and equal partners in CLOE and yet some of them indicated that based on the size and nature of their institution that they might not submit any LOs to CLOE. To address this issue, CLOE partners have created a new expanded MOU which allows partners to choose their commitment from a list of potential commitments. Each partner is not required to choose a specific number of items from the list but rather to choose the items that they can contribute.

Memorandum of Understanding

• Collaboration in the design of a LO• Collaboration in the development and subsequent use of a LO• Collaboration in the formative evaluation of the use of a LO• Peer review of LOs• Collaboration in the adaptation of a LO for use at the partner institution (normally this would not involve modifying the LO but rather documenting the use of the LO in a setting the is different than the setting envisioned by the designers and developers. For example, a LO created for 1st year Psychology being reused in a 2nd year statistics course)•Reusing a LO from CLOE and providing summative evaluation to CLOE and the creators regarding the reuse.

List of potential CLOE partner commitments.

• Empower project leadership to set and achieve goals– Select leaders and key staff with requisite experience; clearly communicate mission and goals

of the organisation; and create an atmosphere that encourages an entrepreneurial spirit, including a willingness to test new ideas.

• Craft a strong value proposition– Create a resource that offers unique value and continue to add value to the resource based on

an understanding of users’ needs.

• Find creative ways to lower the direct costs of running the project.

– Secure contributions from the host institution; outsource work through vendors and other external partnerships; work with volunteers.

• Cultivate sources of revenue to cover both direct costs and ongoing upgrades.

– Experiment with different revenue models to find the ones that are the best fit for the project; show willingness to try new models; cultivate the ability to identify and communicate the value of the resource to the target audience (of customers, authors, subscribers and so forth).

• Establish a system of accountability and measurement of the success of the resource and the revenue model.

– Establish goals and targets and determine the balance between financial and mission-related returns; assess progress towards mission-based and financial goals and targets.

Sustaining Digital resources: An On-the-Ground View of Projects Todayhttp://www.ithaka.org/ithaka-s-r/strategy/ithaka-case-studies-in-sustainability/report/SCA_Ithaka_SustainingDigitalResources_Report.pdf

Key factors for Sustainability

Why did CLOE fail?

• LO Design and development?• Peer review?• Repository/reuse?• Administrative support?

Why did it fail?

LO Design and development

• Once grant funding ended no financial incentive to collaboratively design and develop.

• Few universities and colleges had technical capacity built in (most CLOE LO development by students).

• No demand by faculty to create objects for courses using CLOE methodology.

Why did it fail?

Peer review

• No recognition of peer review in promotion, retention, and tenure.

• No interest by faculty in peer reviewing without recognition

• Process too elaborate?

Why did it fail?

Repository/reuse

• Cumbersome registration/paperwork to reuse lo.• Low demand for lo reuse• D2L repository never becomes operational• The context question

– not an open resource– remix/repurpose restricted through no access

to source code or license– no demand because …

Impact on teaching practices:

• “The foremost reason for not using digital resources was that they did not support faculty’s teaching approaches.”

• Time to identify and adapt resources: “Lack of time was a major constraint, regardless of institution.”

• Reusing resources in new contexts: “Faculty, including those active and enthusiastic in their use of digital resources, identified many other obstacles to using these resources for teaching, including how to . . . reuse them in new contexts”

Opening Up Education http://mitpress.mit.edu/books/chapters/0262033712chap12.pdf

Why did it fail?

Administrative support

• UWaterloo pulls out support after funding ends• Unsustainable by volunteers/in kind contributions

Key factors for Sustainability

1. Assume grant funding will not always be available2. Define sustainability – determine need to generate revenue

beyond operating costs3. Understand, monitor and measure the demand side of the

project4. Consider options for long term governance5. Develop marketing plans and seek out strategic partnerships6. Be prepared to change direction in a rapidly changing

environment7. Leaders need to be fully dedicated, fully invested and focused8. Support a culture of experimentation and innovation

Sustainability and Revenue Models for Online Academic Resourceshttp://sca.jiscinvolve.org/files/2008/06/sca_ithaka_sustainability_report-final.pdf

• Empower project leadership to set and achieve goals– Select leaders and key staff with requisite experience; clearly communicate mission and goals

of the organization; and create an atmosphere that encourages an entrepreneurial spirit, including a willingness to test new ideas.

• Craft a strong value proposition– Create a resource that offers unique value and continue to add value to the resource based on

an understanding of users’ needs.

• Find creative ways to lower the direct costs of running the project.

– Secure contributions from the host institution; outsource work through vendors and other external partnerships; work with volunteers.

• Cultivate sources of revenue to cover both direct costs and ongoing upgrades.

– Experiment with different revenue models to find the ones that are the best fit for the project; show willingness to try new models; cultivate the ability to identify and communicate the value of the resource to the target audience (of customers, authors, subscribers and so forth).

• Establish a system of accountability and measurement of the success of the resource and the revenue model.

– Establish goals and targets and determine the balance between financial and mission-related returns; assess progress towards mission-based and financial goals and targets.

Sustaining Digital resources: An On-the-Ground View of Projects Todayhttp://www.ithaka.org/ithaka-s-r/strategy/ithaka-case-studies-in-sustainability/report/SCA_Ithaka_SustainingDigitalResources_Report.pdf

Key factors for Sustainability

Thank you.