OER Research Hub Evidence Hub Development BriefDate: 28.08.2014

OER Research Hub

OER Impact Map Development BriefContents1.0 Document Control...............................................................................................................3

1.1 Version History..................................................................................................................3

1.2 Distribution........................................................................................................................3

2. Purpose of document.........................................................................................................3

3. Summary..............................................................................................................................4

4. About the OER Research Hub project..............................................................................4

4.1 Project objectives..........................................................................................................5

4.2 Project approach: collaboration and openness.........................................................5

4.3 Project stakeholders......................................................................................................6

4.4 Role of the specified work in the project.....................................................................8

5. Rationale............................................................................................................................10

6. Specification......................................................................................................................11

6.1 File Types.........................................................................................................................11

6.2 Information Architecture................................................................................................12

6.3 Data and Metadata..........................................................................................................13

6.4 Authentication.................................................................................................................14

6.5 User Interface..................................................................................................................14

6.6 Usability & Accessibility................................................................................................15

7. Delivery..............................................................................................................................16

7.1 Phase I – Public Prototyping........................................................................................16

7.2 Phase II – Iterative Development...................................................................................17

7.3 Phase III - Legacy............................................................................................................18

1

OER Research Hub Evidence Hub Development BriefDate: 28.08.2014

7.4 Resources........................................................................................................................18

7.5 Liaison and Reporting....................................................................................................18

7.5 Intellectual Property.......................................................................................................19

7.6 Legacy..............................................................................................................................19

Appendix A: Background Information on OLnet Evidence Hub......................................20

Appendix B: Summary of UNESCO OER Mapping Project...............................................24

Appendix C: Wireframe for Survey Data Explorer……………………………………………26

References..............................................................................................................................27

Author: Dr Robert Farrow, Research Associate

IET

rob.farrow@open.ac.uk

2

OER Research Hub Evidence Hub Development BriefDate: 28.08.2014

1.0 Document ControlDocument Identifier:

OER Research Hub Evaluation Framework

Date due:

Class Deliverable:

OER Evidence Hub WP4 Submission date:

Project start date:

1 September 2012 State: Draft

Project duration:

24 months

1.1 Version HistoryDate Version Stage Summary of changes28/08/13 v0.1 Initial draft Initial draft by Rob Farrow28/8/14 v.1.0 Final draft Final draft incorporates Phase 2 and Phase 3

plans

1.2 DistributionName Organisation/Role

Author(s):

Rob Farrow Research Associate

Reviewers:

Martin Weller Co-Investigator

Patrick McAndrew Principal Investigator

Approvers:

Patrick McAndrew Principal Investigator

2. Purpose of document

3

OER Research Hub Evidence Hub Development BriefDate: 28.08.2014

The object of this document is to describe the technical work that is to be carried out for the development of the Evidence Hub (EH) for the OER Research Hub project (oerresearchhub.org).

3. Summary

The EH will build on the legacy systems developed during the OLnet project to provide a platform for collaborative research, discussion, and dissemination of OER news and reports. The value of this approach for focusing research and attracting collective interest has already been demonstrated both within OLnet (olnet.org) and outside the OER field, but the system needs to be developed and improved in a number of ways, including doing more to facilitate engagement with the target audience(s); a greater range of outputs (including new visualizations) and making the most of automated functions. There will need to be an agreed workflow for researchers using the Evidence Hub as well as on-going revision of effective use of the Evidence Hub as a tool for both research and dissemination.

4. About the OER Research Hub project

The OER Research Hub (OERRH) is a Hewlett Foundation funded project being led by the Institute of Educational Technology at the Open University (UK) (http://www.open.ac.uk/iet/main/). The project provides a focus for research, designed to give answers to the overall question ‘What is the impact of Open Educational Resources (OER) on learning and teaching practices?’ and identify the particular influence of openness. To achieve these aims the OERRH team are working in collaboration with projects across four education sectors (K12, college, higher education and informal) extending a network of research with shared methods and shared results.

The project combines:

A targeted collaboration program with existing OER projects;

An international fellowship program;

Networking to make connections; and

4

OER Research Hub Evidence Hub Development BriefDate: 28.08.2014

A hub for research data and OER excellence in practice.

These activities are illustrated in Figure 1.

Figure 1: The activities of the OERRH project

4.1 Project objectives

It is aimed that by the end of the OERRH project’s research phase evidence will have been gathered showing what works and when in terms of the use of OER in diverse teaching and learning contexts, and that the research team will also have also established methods and instruments for broader engagement in researching the impact of openness on learning. The evidence generated via research collaborations will contribute to this overall aim, whilst providing a legacy for policy change and practice.

5

OER Research Hub Evidence Hub Development BriefDate: 28.08.2014

4.2 Project approach: collaboration and openness

The OERRH project operates on a collaborative rather than partnership model. It features a dual research approach, combining both remote monitoring of publicly available data and the development of research instruments which are applied through a combination of direct and facilitated research.

The core group of collaborating projects is as follows:

The Flipped Learning Network Gulf of Maine Research Institute The Open Course Library (OCL) in Washington State The Community College Consortium for OER (CCCOER) Connexions/OpenStax College TESS-India School of Open/P2PU Bridge to Success OpenLearn

The OERRH approach to collaboration combines research with practical assistance and it is hoped that this will bring benefits to all involved and will also provide an enhanced flow of research data. The OERRH collaborations cover different sectors and issues, these include: the opening up of classroom based teaching to open content; the large-scale decision points implied by open textbooks for community colleges; the extension of technology beyond textbook through eBook and simulation; the challenge of teacher training in India; and the ways that OER can support less formal approaches to learning. By basing good practice on practical experience and research it is intended that the project will help tackle practical problems whilst building the evidence bank needed by all.

Meeting the challenges of openness requires research strategies which take account of the different aspects of OER impact. In the OER world, the way forward is associated with greater transparency and sharing of educational materials and research. Similarly, a more useful evidence base should emerge through taking advantage of working openly and in collaboration as it allows for the pooling of limited resources and greater scale and efficiency through co-ordinated action.

6

OER Research Hub Evidence Hub Development BriefDate: 28.08.2014

4.3 Project stakeholders

The project stakeholders will include (but are not limited to) The Hewlett Foundation; The Open University; IET; collaborating projects; the OER Research Hub project team; the OERRH Fellows, OER users and potential users (institutions, educators and formal/informal learners) and the OER and open education movements. Table 1 provides a fuller analysis of the OERRH stakeholders.

Stakeholder Interest/Stake Importance

Project management team Ensuring the provision of the resources that will allow the project to meet its objectives.

High

Project researchers Professional reputation; ensuring that work is carried out to accepted methodological, epistemological and ethical standards. 

High

Project principal investigator & co-investigator

Professional reputation; ensuring that work is carried out to accepted methodological, epistemological and ethical standards. 

High

The Open University Opportunities to develop a better understanding of the impact of OER on their base of teachers and students; contribution to research profile aligns with strategic priorities.

Medium

Institute of Educational Technology

As part of the IET research portfolio, OERRH contributes to maintaining a reputation for leading research into openness in education and in educational technology more generally.

Medium

Collaborating projects A chance to outsource research needs and better contextualise projects’ own work.  Good practice can be identified and disseminated. The national/international profile of OER projects can be raised.

Varied according to institutional or operational objectives

7

OER Research Hub Evidence Hub Development BriefDate: 28.08.2014

The Hewlett Foundation Getting value for funding input. Ensuring that project stated objectives, outputs and outcomes have been achieved. Furthering knowledge of what works in terms of OER use and development.

High

The OERRH Fellows  Opportunities for professional development; networking & collaboration; novel research opportunities

Medium

OER users and potential users (institutions, educators and informal/formal learners)

Improved understanding of OER impact should facilitate the development of better quality OER and pedagogies which make use of them.

Varied

The OER and open education movements

Raising the profile of the open education movement and of OER generally. Aggregated research should be of benefit to the movement as a whole.

Medium

Table 1: OERRH project stakeholders

More information about the project, its aspirations and techniques can be found on the project blog, which is located at http://oerresearchhub.org/.

4.4 Role of the specified work in the project

The key outcomes of the OERRH project will be:

Research: establishing a greater understanding of the priority research questions on the impact of OER on teaching and learning;

Collaborations: through the research and collaboration program, developing a shared framework that helps both address these questions and enable their research, allowing collaborating projects to benefit from development and advice from the OERRH project team and linked-fellows;

Fellowships: through the fellowship program, enhancing collaboration and communication across OER projects and providing the space and time for understanding of OER-related research issues;

8

OER Research Hub Evidence Hub Development BriefDate: 28.08.2014

Establishing and developing an on-going, sustainable OER Hub at the OU for informing decision making in this area.

The OERRH project comprises six work packages:

WP1: Management WP2: Collaborative Research WP3: Collaborations and Fellowships WP4: OER Evidence Hub WP5: Evaluation WP6: Dissemination

The EH thus has a central place in the project as an online research database and communication platform. The EH will function as a research ‘base’ with practical outcomes that supports the OER and education communities in understanding the issues and challenges.A “How to conduct OER research” pack produced first for use in the collaborations and then as a public, openly licensed document and section of the website. The open-license research instruments featured in the pack will include schedules for semi-structured interviews, guidance notes for conducting focus groups (including hypothesis-related topic guides), a bank of survey questions and guidance for the design of surveys incorporating these questions, an analysis strategy for analysis and interpretation of the quantitative and qualitative data collected through the survey questions, focus groups and interviews, and identification of quantifiable factors either gathered as secondary data or direct analytics;

The project logic model is shown in Figure 2.

9

OER Research Hub Evidence Hub Development BriefDate: 28.08.2014

Figure 2: The OERRH logic model

10

5. Rationale

The OER Evidence Hub will build on the legacy systems developed during the OLnet project to provide a platform for collaborative research, discussion, and dissemination of OER news and reports. The concept of the Evidence Hub developed as a response to the need for technology platforms which could support collaborative knowledge building in an emerging discourse [http://evidence-hub.net/]. There is an identified need for a web service where information about OER impact can be retained and recalled but which also helps users to make sense of research data, offering comparison across cohorts and in relation to international context. OER research is often fragmentary, or doesn't allow easy generalisation across contexts OERRH is guided by a series of research hypotheses which are investigated through both primary and secondary research. Our understanding of these hypotheses and their validity is to be supported by evidence gathered at the right level of granularity from a plurality of sources.

The EH needs to be able to present information to different audiences, and to enable users to navigate information that is relevant to them. This means that it needs to be able to help people to make sense of complex data and show how it might be relevant to them. It is both a tool for the research team that provides insights into 'the bigger picture' and a way for those involved in the open education movement to better understand their own contexts. OER advocates remain in need of a tool which will enable them to present a convincing evidence case for policy change and changes in practice. OERRH will support this by structuring the evidence and making arguments explicit. The ‘researcher pack’ will make OERRH research instruments available openly so that others can contribute to the evidence base.

By mapping OER evidence and OER initiatives round the world, the EH will help potential networks of support and collaboration to become aware of of one another. Another important function for the EH is as a place for disseminating the work of the OER Research Hub project. This means it will need to be able to accommodate a range of different types of raw and redacted data as well as multimedia. It must also conform with the guidance set out in the project Ethics Manual (http://oerresearchhub.org/project-documents-archive/reports/oerrh-ethics-manual/ ).

11

Here is a summary of comments generated by the OERRH team about what the EH should strive to be:

Easily navigable content Easy answers for funders

Clear evidence relating to OER impact

A presence that enhances OU reputation

A bank of resources for others to us Evidence-informed claims

Guidance for other OER researchers A catalyst for further research

‘Our story’ A narrative about changes in OER/OEP

Customised interfaces Signposting missing evidence

Geographical dataset A better understanding of each hypothesis

Information which maps to hypotheses

Help & evidence on OER issues

Infographics for each collaboration / hypothesis

Table 2: EH Design Guidelines

6. Specification

The platform and/or technical architecture have yet to be decided upon, but a number of system requirements have been identified (see below).

6.1 File Types

The Evidence Hub should support both stored and embedded content of a range of types including databases, text, audio and video. HTML pages should be able to accommodate these. Embedding content should be encouraged where possible, but we will need storage space at least for any data we add (e.g. survey results) which will need to conform to data protection policies. We should enable users to sort content in intelligent ways which suit their strategic needs. We should also ensure that it is possible to filter evidence by sector (School, Further Education, Higher Education, Informal Learning) and potentially by stakeholder.

12

6.2 Information Architecture

In OER Research Hub, a series of research hypotheses provide the focal points and provide a way of establishing the relevance and value of a particular piece of evidence. They are provided here with a shorthand tag.

a)  Use of OER leads to improvement in student performance and satisfaction [impact]

b) The open aspect of OER creates different usage and adoption patterns than other online resources [patterns]

c) Open education models lead to more equitable access to education, serving a broader base of learners than traditional education [access]

d)  Use of OER is an effective method for improving retention for at-risk students [retention]

e)  Use of OER leads to critical reflection by educators, with evidence of improvement in their practice [reflection]

f)  OER adoption at an institutional level leads to financial benefits for students and/or institutions [finance]

g)  Informal learners use a variety of indicators when selecting OER [indictors]

h)  Informal learners adopt a variety of techniques to compensate for the lack of formal support, which can be supported in open courses [support]

i)  Open education acts as a bridge to formal education, and is complementary, not competitive, with it [transitions]

j)  Participation in OER pilots and programs leads to policy change at institutional level [policy]

k)  Informal means of assessment are motivators to learning with OER [assessment]

The short form of the tagging system binds relevant content together. The advantage of doing this in relation to a hypothesis is that it will lets us quickly draw upon a body of evidence that all relates to that hypothesis even if the content is diverse.

13

6.3 Data and Metadata

Content for evidence nodes at the prototyping stage is to be provided by OERRH researchers. There are existing data sources which will be used for seeding the EH, including data currently stored on SurveyMonkey and imports from the legacy EH from the OLnet project.

Evidence entries themselves are likely to be mostly text but should support embedding of video, audio and other types of content through HTML. For any given piece of evidence we expect there to be key metadata which provides the basis for navigating, synthesizing and presentation of information. For example, it should be mandatory to tag any given entry with a relevant hypothesis.

Here is the anticipated structure for an evidence record:

Aspect Type Format(s) Notesa Title Textb Copy Text/HTML Likely to be entered

into a simple HTML editor which supports embedding

c Relevant Hypotheses Radio Buttons Mandatoryd Polarity (positive/negative) Radio Buttons Mandatorye Location Geotagging/GPS Supports mappingf Sector

(School-K12/College/Higher Education/Informal)

Radio Buttons

g Link(s) URL Further information for this entry

h Citation Text Academic reference relevant to this entry (may or may not include URL)

Table 3: Structure of Evidence

It may be desirable that some sort of 'featured image' functionality also be included in the data description. The data model should be able to support entries that are only a few sentences as well as more detailed 'case study' style entries which might include multimedia objects or embedded content.

The inclusion of geographical references is important for the user interface that the EH will be modeled upon (see Appendix B). It should also be possible to add other kinds of

14

entries to the EH, such as OER projects around the world, news highlights, or events. These may be added at a later date so the system used needs to be open to data which is unanticipated at this point.

6.4 Authentication

The public prototype version of the EH need not have a fully working system of user accounts, though this would be desirable if time and budgets allow. Open standards for authentication which facilitates sign-in through social media accounts (Twitter, Facebook, Google, Yahoo, Wordpress, etc.) or OpenID are preferable.

We ultimately want signed-in users from outside the team to be able to add evidence o the database. Using IP addresses or on-device GPS should help us to locate their contributions on the map and this functionality should remain a possibility regardless of which platforms or programming languages are used.

Ultimately, administrative privilege should include the facility to edit/delete any aspect of any entry and moderate comments made, and to create/edit/delete static pages and other elements of the site.

6.5 User Interface

The site design should be clean, uncluttered, intuitive, and consistent with the project branding and with the guidelines provided by The Open University. It should use the existing OERRH logo and branding guidelines which will be supplied to the successful candidate.

The site will need a few static content pages with information about the project with contact information. The homepage for the EH should contain some general information about the project extracted from oerresearchhub.org and possibly displaying the RSS feed for this site and for our Twitter account (OER_Hub).

Users should be able to browse collected evidence and OER organizations through a world map similar to Google Maps and preferably which makes use of ‘open’ platforms such as the OGC (2013) standards. It should also be possible to review the evidence associated with a particular hypothesis, read case studies, browse by sector, navigate through tags and/or make use of search box. All views should be working fro the same data set.

Content will be created by members of the OERRH team which means that users should be able to create new evidence entries with the structural elements described in 6.3.

15

At some point in the future we may wish to introduce the ability to form interest groups from users, or to ‘favourite’ particular pieces of evidence. We will also introduce the functionality for sharing evidence nodes directly to social media.

6.6 Usability & Accessibility

The work should conform to Web Content Accessibility Guidelines 2.0 (W3C, WCAG20). The OUICE template (http://www.open.ac.uk/webstandards/v3.0/index.php) may provide some guidance in this respect although the EH design should be consistent with OERRH rather than Open University branding.

The end-user facing site developed for this project will be tested against the following platform/browser combinations:

Desktop:Mac (OS X) PC (Windows 7)

Chrome (latest release) Chrome (latest release)Mozilla Firefox (latest release) Mozilla Firefox (latest release)

Internet Explorer 9.0Internet Explorer 8.0

Safari (latest release)

Mobile:Android iOS

Chrome on Samsung Galaxy S2 Safari on iPhone 4Chrome on Nexus 7 Safari on iPad Mini

Chrome on Samsung Galaxy Tab 2 Safari on iPad 2Table 4: Target Platforms/Browsers

6.7 Analytics

The EH should include calls to a Google Analytics account (details TBC) with all creation, editing and deletion of records logged in the database. No facility for accessing this log need be included in this work. The log is intended for potential future expansion of functionality and for reporting and analysis. The following information should be captured:

Date and time of the operation The operation performed The unique reference for that entry Relevant user ID

16

7. Delivery

Work on the EH needs to begin immediately in anticipation of upcoming project dissemination activities which will publicly present research data from the project. It is expected that there will several phases of development activity with discrete elements. Phase I refers to the public prototyping of the EH and Phase II comprises distinct elements of functionality (several of which may be combined in future commissioning).

Development work will take place on local developer machines with regular updates to the prototype EH which will be hosted on IET servers.

7.1 Phase I – Public Prototyping

Thus far, a basic prototype which aggregates evidence gathered within the project has been written in JSON (Javascript Object Notation) which shows how our data might be structured in ways which would support aggregation and database queries. JSON is a text-based open standard which is designed for human-readable data interchange and is compatible with a range of programming languages and web browsers. Our experiences of working from an evidence dump to narratives about each hypothesis shows how the process might work. The prototype includes information about hypothesis; a summary of the +ve/-ve evidence count; a list of evidence entries and key ideas.

The first phase of consultancy will build on this in order to deliver the following (including the creation of any required APIs (Application Programmable Interfaces), GUI (Graphical User Interfaces) and web interfaces for these tools):

(1) A demonstrator evidence hub capable of supporting:

browsing and displaying information about the OERRH hypotheses on a map navigation, filtering and presentation of data by variables and hypotheses

including: a simple user interface to allow evidence to be entered into the system administrative functions a summary page for each ( or at least some) hypotheses which presents

aggregated evidence embedding of diverse multimedia content

(2) A system of structured files or database content enabling functionality which can be expanded upon in future

conversion of OERRH research data into seeded content incorporation of imported data about OER projects (e.g. extracted from the OLnet

Evidence Hub) maintenance of data sets

17

(3) Data visualisations indicative of project research data, including surveys still images suitable for presentation prototyping/descriptions of visualisation tools which could be added to the site

 (4) Suggest design directions and provide a debriefing meeting with OERRH in person or remotely to help determine the next stage in the development (5) Write at least 2 blog posts working with the OERRH workpackage leader to help brief on the hypothesis hub and provide support to presenting the OERRH hypothesis hub prototype at the Open Access Unconference (San Jose, CA) and OpenEd 2013 (Park City, UT) in October-November 2013.

Planned Start Date: 9 September 2013 Planned End Date: 9 November 2013

7.2 Phase II – Iterative Development

The focus for this phase of work is substantive development of site which will enable fuller dissemination of research data.  This has been broken into three distinct sections of work so that they can be commissioned independently or together.

Phase II.1 (Jan 2014)

(1) Building a system to display and explore survey data that has been generated by the project. This data is exported from Survey Monkey and can be prepared by the OERRH research team once a protocol is agreed.  Mash-ups of existing and public data should be supported in future.

(2) Tools for helping users to make sense of survey data.  These could include filters for isolating different variables within the data set and the ability to run simple queries that will interrogate the data.  Data Visualisations that will present survey data might include showing trends plotting data against a heat map; maps which show results by country or state; interactive pie/bar charts and pivot tables.

(3) A (front-end) user interface for entering data into the system.  A pop-out UI should facilitate data entry for the evidence, project, location and policy post types. This will initially be for the use of the OERRH team but will likely be rolled out to other users in the future.  It should include a feature which warns the user when they may be entering information already in the database by comparing the citation with older posts.

(4) Introduction of a new polarity status of 'neutral' to be used exclusively with the 'Uncategorised' evidence category.  This will then be used to curate evidence which is not presently connected with a hypothesis but which may be important for future hypotheses.

18

(5) Refinement of the citation field which will detect when a URL has been provided and automatically provide links for these while retaining the option to add a textual academic citation without automatically converting this to a URL.

Phase II.2 (Feb 2014)

(6) Installation of plugin(s) to support improved integration with social networks: sign-in, share buttons, community-building (e.g. OpenID, BuddyPress).

(7) Installation of a plugin to facilitate curation through RSS (e.g. FeedWordPress) that will facilitate queueing of evidence for entry from bookmarking/curation services.

(8) Redesigned pages for the exploration of hypotheses. These pages need to be structured and developed in such a way as to make it easier for the user to preview and explore hypotheses with less scrolling and less text immediately visible on screen.  A summary page for each hypothesis comprised of panels showing a mix of visual representations of data (e.g. hyperbolic tree; scales of balance) as well as textual summaries (e.g. hypothesis description, total of evidence gathered, featured evidence, most-shared evidence) could work well for this.  These summary pages )(or infographics) should be print-optimised so that users can generate a quick report on the latest data.

(9) Redesigned pages for the exploration of evidence which help the user to make sense of the evidence as it is presented according to polarity and/or sector rather than providing a sorted list of all evidence posts meeting the criteria selected.

(10) A bookmarklet that will facilitate data entry after being added to internet browsers, providing a way to directly import basic information about a website into the data entry interface (similar to Scoop.It).

Phase II.3 (Mar 2014)

(11) A method for publishing customised map 'stories' about research data (e.g. Mapbox) which can be used to present a case relating to a particular sector or hypothesis by taking the user on a path through a collection of evidence, projects, and/or policies and providing additional commentary through an additional panel (See https://www.mapbox.com/tutorial-sherlock/ for an example.)   It should also be feasible that this method be used in relation to scraping tweets and displaying them on a map.

(12) A redeveloped policy map which can show the range of (geographical) influence that the collected policies have in addition to (potentially multiple).  This could work like a heatmap, showing where there are areas with a lot of active policies.  Policies need to be able to apply to several (or all) sectors simultaneously.

(13) Creation of RSS feeds for evidence, project and policy post types.

(14) Re-specification of the CSS active on the site to meet project branding guidelines.

19

(15) Installation of a plugin for search engine optimization to improve the visibility of the site.  This should be configured so as to make the custom post types available for search.

In addition and alongside these packages, the developer(s) should write:

(16) Three blog posts (to be written in February, March and April) on the OERRH project website to document the work being done and the challenges/solutions encountered.

A audit of site usability and accessibility will also be carried out as part of Phase II – this will be conducted by specialists within IET.  This may identify some bugs or fixes which should be carried out in this phase of development.  

Please note that developmental work relating to the Hewlett OER Map project may be carried out alongside the work described here if the bid id successful.   This will be described in a separate document.

Planned Start Date: Jan 2014 Planned End Date: April 2014

7.3 Phase III – Legacy

This phase of development will address any outstanding issues with previous development as well as introduce features designed for improving the user experience as well as improving the long-term sustainability of the site and its services.

(1) Debugging/finalization of work specified in previous phases of development, namely:

Open ID sign-in and sharing from within records Specification of the CSS (especially BLOCKQUOTE) to meet project branding

guidelines RSS feeds for the different custom post types (evidence, policy, project) Buddypress social network plug-in Installation of an alternative search plugin Ensuring that map pointers don’t bunch so close together that they obscure one

another (e.g. Nairobi)

(2) Redevelopment of the interface for exploring OERRH survey data in accordance with the provided wireframe. The database needs to be replaced with an updated and cleaned version of the survey data. (See Appendix C.)

(3) Creation of a second ‘Tweetmap’ based on the #oerrhub and #oerrh hashtags and the @oer_hub handle – the relevant TAGS archives are available from

20

https://docs.google.com/spreadsheet/ccc?key=0Ak-iDQSojJ9adGFaQkNpMmVlT2RqbWFYd3pBQjhia1E&usp=drive_web and https://docs.google.com/spreadsheet/ccc?key=0Ak-iDQSojJ9adDdMdXJMT3ZvMlByZ0ViakFfOG9fc2c&usp=drive_web#gid=36. This map and the existing #oermap Tweetmap should be integrated into the native mapping system rather than hosted on Google maps/fusion tables. It I desirable that it is possible to manually add entries to this map (URL + custome post type so as to use it for the basis of a map that show the connections and influence of the OERRH project.

(4) Refinement of the UI to allow users to read full records in overlay style without having to leave the map view. (See http://www.mira.org.br/ for an example of how this might work.) Alternatively, the map ‘pop-out’ box can be redesigned to compress the metadata (Type/Hypothesis/Sector/Polarity/Locale/etc.) into a banner (which could also include quick links for sharing) or visual representation and expand the box to accommodate the full text record.

Other UI/display tweaks: Colouring the central circle of evidence nodes according to relevant sector Legends for all maps Installation of the following Wordpress theme(s):

o Forever (https://wordpress.org/themes/forever)

(5) An interface/shortcode for creating new custom post types (or another solution such as tagging) so that it will be possible for the project team to create new bespoke maps in the future as data becomes. Four new maps are anticipated as part of this phase of development:

(a) Building a new map view for hosting data from the TAACCCT grantees (http://www.doleta.gov/taaccct/pdf/TAACCCT_One_Pagers_All.pdf). This map will be USA only and the data will provided by one of the OERRH project fellows. This may involve the creation of a new custom post type but specific metadata requirements are TBC. There are 57 grantees spread throughout the USA.

(b) Building a new map view based around a series of case studies of notable OER initiatives (which could be linked to ‘projects’ in the ontology). This map will present detail on 8-10 initiatives through a structured narrative and may be structured as a simple blog post with geo locations. (Content provided by project team.)

(c) Building a new map for recording MERLOT impact studies; these will include a simple ontology no special metadata requirements (similar to the OER Policy Map). Data will be provided by a forthcoming call to the MERLOT user base. It is expected that there will be up to fifty of these to be displayed on either a global map or map of North America.

21

(d) Building a new map for displaying impact stories for the TESS-India project. This would also have a simple data structure, but one idea that has been discussed for the India map is the idea of parent/child relationships to show primary/secondary impacts that have resulted from interventions. It would be good if there was an intuitive way to explore this (e.g. click to reveal).

Suggested ontology for these maps: Name of entry Geographical location Impact statement Sector Additional fields as required by map type

There may be a couple of custom fields required for some of these. (E.g., the India map may include an ‘interaction type’ field.)

Some data collected for (a) – (d) might qualify as evidence in the main map – in which case it will be added to the OER Evidence Map separately by the project team.

(6) Full documentation of API end points and facilitation of import/export of site data though API. Users with appropriate permissions should be able to interface with the database in such a way as to facilitate the reuse of open data from both the maps and the survey data explorer. It should also be possible to interface with open data from the Hewlett OER Map prototypes (MIRA, ISKME, HBZ) and import, publish or cross-reference this with the OERRH maps.

(7) Introduction of a digital badging system (e.g. BadgeOS) which would allow awards to be made for:

evidence contribution voting on evidence entries completion of the OERRH School of Open course (begins Sept 2014)

(8) Suggesting, agreeing upon and implementing technology/ies for presenting narratives around the OERRH project and its impact (e.g.)

Aesop Story Engine http://wordpress.org/plugins/aesop-story-engine/ Scrollkit http://www.scrollkit.com/ (nb. Bought out by Wordpress) Killing Lincoln (NatGeo) http://killinglincoln.nationalgeographic.com/ Bear 71 http://bear71.nfb.ca/#/bear71 Docubase.mit

(9) Installation of Wordpress Ubuntu font and any other necessary plugins. Slider https://wordpress.org/plugins/slider-image/

22

(10) (a) Importing 416 projects (or ‘companies’) from the Little Bird data provided by Lumen Learning into OER Projects Map. (Relevant fields are ‘’Full Name’; ‘Description’; ‘Related Website’ and ‘Location’.)

(b) The larger data set also shows influence on Twitter and could potentially be used for data visualization(s). These should attribute the data to

Little Bird (http://getlittlebird.com/) and Lumen Learning (http://lumenlearning.com/).

(11) Adding ‘Sector’ to the metadata for projects

(11) Optimization and debugging of final code.

(12) In addition, the developer(s) should write one blog post each month to document the work being done and record the challenges encountered and solutions that were adopted.

Planned Start Date: Jul 2014 Planned End Date: Oct 2014

7.4 Resources

The OERRH Development Consultant is required to address the need for an online implementation giving access to findings linked to the hypotheses used in the OERRH. A series of consultancies are planned to achieve this in place of the Developer post indicated in the proposal. This reflects an unsuccessful round of recruitment for that post.

A prototype system is required for demonstration in November 2013. The system requirements are well specified and data is in place to enable the demonstrator. The initial consultancy will focus on this task. Further consultancies may follow in order to complete the development requirements set out in the proposal. The Hypothesis Hub work can also build on the phase 1 evaluation work that is reporting in September. In order for debriefing to be effective potential expenses are included to cover travel to/from MK campus on occasions.

Maximum Fee: £6000 Estimated Maximum Expenses: £1000

Resources made available for subsequent phases of development will be subject to budgetary review.

23

7.5 Liaison and Reporting

To ensure clarity of communication, OERRH will provide a single point of contact: Dr. Rob Farrow (rob.farrow@open.ac.uk) who is a Research Associate on the project and leading the relevant work package.

Note that at times it may be necessary to liaise and communicate effectively with the project academic and non-technical staff to manage technical and non-technical project objectives. The developer is expected to document the development of the software and ensure that the technical deliverables of projects have gone through appropriate quality control and quality assurance process.

7.5 Intellectual Property

All work created as part of any consultancy is to be owned by OER Research Hub, and any code is granted an indefinite non-exclusive license for use by The Open University.

OERRH is committed to a policy of open dissemination, as set out in the project Ethics Manual. Specifically, all project materials for which it is thought relevant are to be released openly on a CC-BY Creative Commons licence. This may ultimately include code and copy generated for the development of the EH.

7.6 Legacy

OER Research Hub is funded until September 2014 and website development is likely to continue as we work toward project completion. Once the project ends the site could be able to be managed with a minimum of input and should continue to present the evidence base constructed during the project while remaining open to new hypotheses and new data.

24

Appendix A: Background Information on OLnet Evidence Hub

The Open Learning Network (OLnet) was a project based at The Open University’s Institute of Educational Technology (IET) and Carnegie Mellon University. In addition to supporting and commissioning research into OER and their impact, OLnet comprised a fellowship scheme and had an explicit focus on developing software tools which would facilitate and support collective intelligence and identify potential solutions to the challenges facing the OER movement.

OLnet’s mission was to ‘to gather evidence and methods about how we can research and understand ways to learn in a more open world, particularly linked to OER, but also looking at other influences.’ The idea of gathering evidence for OER impact was thus at the core of the project and continues to inform work on OER Research Hub. OLnet included a number of academics involved in different aspects of OER research. The evidence hub was envisioned as a tool to disseminate their work while linking key aspects to wider discussions and events that were taking place.

The decision was taken to develop an evidence hub that could accommodate a wide range of possible forms of evidence but in a way that allows sense-making to take place. Why might we need to take such an approach to open education? One of the reasons that it might be appropriate to investigate open education through collective sense-making might be the fragmentary and partial nature of much of the evidence that is available. Where evidence does exist, it is often tightly focused on a particular initiative rather than on open education as a whole.

The designers of the OLnet Evidence Hub used the concept of ‘contested collective intelligence’ (De Liddo & Buckingham Shum 2010; De Liddo & al 2012). This approach recognises that most ‘knowledge’ is contested, and different backgrounds, cultural assumptions and intellectual traditions. It was used to develop a social-semantic web application for human annotation and knowledge mapping called Cohere (http://kmi.open.ac.uk/technologies/name/cohere).

Information is classified and assigned to the following nodes:

Issues

Potential Solutions

Research Claims

Evidence

Resources

Organisations/Projects

25

The benefit of breaking information down to this level of granularity is that it can promote nuanced consideration of the relationships between particular claims or pieces of evidence. Users can add nodes and make new connections as well as provide argument or counterargument and vote on the plausibility of claims or solutions. The OLnet Evidence Hub was thus designed to provide the following functions:

Explore and debate the key challenges for the Open Educational movement.

Allow the community to express the importance of each challenge (challenges can be promoted or demoted)

Add a description of an OER project, including geographical location

Post, explore and discuss OER related issues and questions

Propose solutions to tackle the major challenges facing Open Education

Share relevant evidence and Web resources for the OER community to contribute to the evidence base of OER impact on teaching and learning

To make and investigate new claims of OER effectiveness

Cohere allows for some very fine-grained distinctions in textual analysis. Researchers on OLnet used it to analyse 125 reports from previous Hewlett grantees working with OER. This thematic analysis was used to seed the Evidence Hub, to create the tagging system, and to identify a number of ‘issues’ and ‘solutions’ which were used. It also provided details about organizations which initiated the ‘mapping’ process. However, the full version of Cohere was not thought to be one which with users would engage, and so a simplified version was used. Envisaged as a ‘vertical’ version of a horizontal tool, this version trades nuance in expression for relative ease of use (Buckingham-Shum, 2012).

The knowledge-mapping functions of the OLnet Evidence Hub are largely provided by a tagging system which can indicate that discrete nodes of information have some common connection.

26

Fig. 1 – OLnet Evidence Hub Themes/Tags

During the implementation phase a new category of ‘Key Challenges’ was introduced in order to provide a clearer focus for the information that had been entered. The evidence model went through a number of revisions over the course of the project. Here is the final (simplified) data model for the OLnet Evidence Hub:

Fig. 2 – OLnet Evidence Hub Data Model

At the time of writing there are the approximately 1500 records in the OLnet Evidence Hub, most of which were entered by IET staff:

300+ OER projects and organisations 129 research claims 79 OER issues 323 evidence nodes 553 resource nodes

It’s worth noting that, with the exception of someone expressing a view in a comment, the information contained within the hub can only be metadata: no materials from the resources were stored within the hub and only items with a URL can be added as resources. This arguably lessened the value of the hub for users who want a ‘one stop shop’ for OER information and engagement.

27

Given the number of possible connections it is possible to make between nodes, the resulting database is very complex, which resulted in some slow loading times.

The work carried out using the OLnet Evidence Hub resulted in a number of research outputs. These included two conference presentations (De Liddo et al, 2012; McAndrew et al, 2012b) and a journal paper (McAndrew et al, 2012a). The research into knowledge mapping also provided outputs for KMi although these were not necessarily part of OLnet. (e.g. De Liddo et al, 2011). The key challenges themselves were well received among the OER community and may have acted as a focal point for discussion and planning.

The website was promoted at a number of high profile events, including Open Ed 2011, Cambridge 2012  (jointly organised by OCW Consortium and SCORE (Support Centre for Open Resources in Education), and a number of OU events and webinars (including the Learn About Fair and OpenOpen: Sharing Research and Practice). OLnet Fellows were also encouraged to interact with the service. While these promotional activities generated a respectable level of interest in the site, users never really engaged with the site in numbers which would authenticate claims to have drawn on collective intelligence, and so the Hub was primarily seeded with information brought in through email lists, RSS, social networks and curation services. Perhaps the most effective use of the tool was in analysing discrete documents, such as the JISC OER impact report or the UNESCO / Commonwealth of Learning 2010 policy announcement. Summarising lengthy documents like these is a service which is often valued by those with time pressures, and which may also have pedagogical benefits for researchers. However, this is a time-intensive task which requires skills in analysis and communication. Furthermore, the purpose of the hub is not primarily to provide textual analysis.

Arguably the OLnet Evidence Hub did not meet its aspirations as a dissemination tool. Very little of the research output of the project is recorded there, and little user activity which would indicate adoption by the OER community. However, recent site statistics are more encouraging. The site presently receives around 7-8 visitors a day, most of whom are new to the site and from either the UK or the USA.

The OLnet Evidence Hub was not subjected to a formal evaluation, though a considerable amount of user testing took place. Users included IET staff and OLnet fellows as well as developers from KMi. The Learning and Teaching Development Team also carried out an audit for accessibility and usability.

Most users found it difficult to orient themselves with the interface, and as these representative comments from the different evaluation activities show they typically did not feel inclined to continue using the site.

The process of data entry was onerous, with multiple forms to complete Some users felt unsure what to do when they had linked nodes together

28

Some felt that there should be additional ways of making sense of the data while others found the present ontology too complicated

Some felt that the purpose of the site was unclear: resource? tool? news site? Excessive terminology was identified as a barrier to use The design was often thought visually unappealing, with too many text lists and

windows No clear process for evaluating evidence or claims without large numbers of users Some errors in loading maps (possibly Java issues) Lack of a definition of ‘evidence’ and other key terms Hard to understand the thinking behind the website Some 404 errors and slowly loading pages Visual representations of arguments add little value Not always clear who is making a claim: the writer or the cited source Some issues surrounding ownership of materials on the site

The OLnet Evidence Hub was intended to provide a software structure which could support sense-making discourses around OER. It was supposed to augment everyday communication mechanisms by crystalizing existing discourse and arranging it in ways which allow collective intelligence to filter claims while preserving the contested nature of knowledge. The site has never had a sufficient number of users who acted in the ways envisioned by the designers, so the approach of the designers has not necessarily been invalidated.

However, the slow take-up of a tool for which there is a recognised need within the OER movement indicates that the approach has not been appropriate for the audience we need to reach. It particular, it seems evident that any user unfriendly interface will not encourage community engagement, and users need to feel that they are receiving value from their activity on the site. There also needs to be a clearer sense of what the objectives of such a service should be.

Appendix B: Summary of UNESCO OER Mapping Project

Undertaken at the same time as the main body of work on the OLnet Evidence Hub – and working towards a commonly identified need – the UNESCO OER Mapping Project attempted to record the landscape of OER use to support communication, connection and collaboration. In 2005 the International Institute for Educational Planning (IIEP) of UNESCO convened an international OER community with the objective of raising awareness of OER. The UNESCO/Commonwealth of Learning Chair in OER at Athabasca University was created with a focus on promoting OER at institutional, national and international levels.

29

There was a feeling that “[t]he growth of the OER movement over the past decade has meant that it has become more and more difficult to have an overview of OER initiatives globally, and even locally. A map of the “OER world” could illuminate its scope, making initiatives or projects more visible and enabling connection, collaboration and partnerships” (D’Antoni, 2012).

The process of reflecting on the topic began with a period of collaboration between the AU OER mapping project and the OLnet project at The Open University (OU), a partner institution of the UNESCO Chairs in OER. The interaction in the Athabasca University OER community of almost 900 members was organised over a three-week period with a suggested focus for each week:

1. What could an OER world map look like?2. Could a world map be built collaboratively?3. Reflection and next steps

There was some reflection on what was needed – whether it was a geographic map or a database of OER projects. In fact, a map is generated from a database, but these remarks point to the fact that the community may wish to have various representations and these may be textual or visual. For example, it would be useful to have a listing of initiatives by country, or by initiative language.

The discussion which took place around this time led to a number of recommendations about the design of a mapping system. It could be used to:

make visible the scale and geographic range of OER initiatives

broaden the OER scenario beyond the well-known projects and players

trace the life cycles of OER projects over time

identify a global roster of OER experts

enable collaboration and partnerships among OER people and projects

awareness raising to educate those outside the OER community

communication and advocacy with policy and decision makers

inclusive community-building based on a common goal

It was noted that the mapping process should be simple and sustainable with the capacity to draw on a range of sources. With this in mind, the following ‘essential’ data was requested for initiatives so that they could be added to the map.

1 OER initiative name

2 OER initiative web site

3 OER initiative working language(s)

30

4 Contact person name

5 Contact person email

6 Latitude and Longitude

More complex metadata requirements were considered, but ultimately rejected in the discussion as too onerous on the user. It was suggested that case studies be attached to entries where possible.

Although the UNESCO OER Mapping Project was never formally implemented, though a prototype version which mapped information from OCWC was used for demonstration purposes.1 However, the process of consultation led to a number of central recommendations (D’Antoni, 2012):

The first point of consensus was that Free and Open Source Software (FOSS) be used for the mapping exercise – for both the development of a database and for any visual representations of the data.

Information should be described by the appropriate metadata, and classified using open international standards.

It is important to have a centralised function for final quality control and loading of the data

Mapping should combine local data collection with centralised quality control

Tools should be flexible enough to be adapted by different communities for different purposes

Mapping services need to think about how visually impaired users might access information

Here is what the report concludes about information visualization and its value for an OER mapping system:

“A visual presentation of OER initiatives worldwide could communicate different stories or messages, and a well-drawn picture conveys its message largely without words. Maps can present different combinations of data to give different pictures or perspectives. Geographic representation of projects by country would present one picture, while representation by the working language of projects would present another. A timeline created from information on the launching of projects would indicate, for example, periods of intense development and highlight new initiatives… The different techniques used in mapping – such as colour coding or size of marker – all contribute to conveying a message, and there are many options for presenting data visually.” (D’Antoni, 2012)

1 http://oerworldmap.oerknowledgecloud.org/index.php?lat=20&long=0&zoom=2&filter=1

31

Appendix C: Wireframe for Survey Data Explorer

SECTOR HE College School/K12 Inf. Learners

ROLES Educators

Formal Students

Informal Learners

Librarians

COUNTRY (Optional) [Drop down list]

GENDER Male Female Other

AGE <15

15-18

19-24

25-34

35-44

45-54

55-64

65-74

75<

DISABILITY YES NO

EDUCATION No formal

Vocational

School College

Undergraduate

Postgrad

QUERY

Select one of the following options: (see below)*

TEACHING & LEARNING PROFILESOER BEHAVIOURSCHALLENGES & SOLUTIONSOER IMPACT

32

GO

[OUTPUT]

CHART TABLE

Or maybe this works better given the template…

Chart

Table

Notes on Data Explorer

In the database for the survey data explorer (ALL_Compiled_SurveyData_June14.csv) row 1 indicate filters (those variables relevant for the checkboxes).

All checkboxes are selected by default – but users can deselect variables to hone the sample.

Row 2 presents the key areas used for running the queries. Row 3 gives question wordings. The interface for selecting query areas could be a panel where you select one, or a

drop down box, or something more creative. Users will just deselect any variables they need to in order to refine the sample,

then select a query area and press ‘Go’. This will update the output chart(s) and table(s) as well as the marker map. The other elements are anticipated to be static.

33

Further queries are generated by changing the

*Query Areas

Here are the areas that comprise each query area. Each contains 2-5 relevant questions as indicated below. The output should include all of the data for that query area according to the filters that have been selected/deselected as well as the questions asked. (The code in parenthesis shows where to look up this data in the file ALL_Compiled_SurveyData_June14.csv.)

TEACHING AND LEARNING PROFILES Subject areas (AG2) Internet access (AY2) Reasons for using OER (IK2)

OER BEHAVIOURS OER use patterns (CK2) Types of OER used (CQ2) Purpose(s) of OER use (DF2) Repositories used (DW2) Factors affecting OER selection (GZ2)

CHALLENGES AND SOLUTIONS Challenges faced (EN2) Support techniques used (HQ2)

IMPACT Impact on teaching practice (FD2) Impact on learners (learner perspective) (FP2) Impact on learners (teacher perspective) (GC2) Prediction of future behaviour(s) (ID2) Impact on motivation (IV2)

Outputs should include a world map which updates according to the sample that has been selected as well as simple graphs and tables showing the data relevant to the query that has been entered.

References

Buckingham-Shum, S. (2012). Evidence Hubs: a new way to interact with idea networks. Available from http://technologies.kmi.open.ac.uk/cohere/2012/01/27/evidence-hubs-a-new-way-to-interact-with-idea-networks/.

34

D’Antoni, S. (2012). A world map of Open Educational Resources initiatives: Can the global OER community design and build it together? Summary report of an international conversation: 12 – 30 November 2012.

De Liddo, A., Buckingham-Shum, S., McAndrew, P., & Farrow, R. (2012) “The Open Education Evidence Hub: A Collective Intelligence Tool for Evidence Based Policy”. Proceedings of Cambridge 2012: Innovation and Impact – Openly Collaborating to Enhance Education, OCW Consortium and SCORE, Cambridge, UK, April 16–18 2012, Milton Keynes, The Open University, pp. 396-404, http://oro.open.ac.uk/33640/.

De Liddo, A., Sandor, A., & Buckingham-Shum, S. (2011). Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study, CSCW Journal (21(4-5), pp. 417–448

De Liddo, A; Sándor, Á; Buckingham Shum, S (2012) Workshop 'Collective Intelligence as Community Discourse and Action, CSCW conference in Seattle, February 11. 2012.  http://events.kmi.open.ac.uk/cscw-ci2012/  http://events.kmi.open.ac.uk/cscw-ci2012/wp-content/uploads/2012/02/CSCW2012-CI-Workshop-final.pdf

De Liddo, A. and Buckingham Shum, S. (2010). Cohere: A prototype for contested collective intelligence. In: ACM Computer Supported Cooperative Work (CSCW 2010) - Workshop: Collective Intelligence In Organizations - Toward a Research Agenda, February 6-10, 2010, Savannah, Georgia, USA. http://oro.open.ac.uk/19554.

McAndrew, P., Farrow, R., Law, P., & Elliot-Cirigottis, G. (2012a) “Learning the Lessons of Openness” in Comas-Quinn, A., Fitzgerald, A., & Fairweather, I. (eds.) Journal of Interactive Media in Education. Special Issue Cambridge OER 2012. Available fromhttp://jime.open.ac.uk/article/2012-10/html.

McAndrew, P., Farrow, R., Law, P., & Elliot-Cirigottis, G. (2012b) “Learning the Lessons of Openness”. Proceedings of Cambridge 2012: Innovation and Impact – Openly Collaborating to Enhance Education, OCW Consortium and SCORE, Cambridge, UK, April 16–18 2012, Milton Keynes, The Open University, pp. 193-202, http://oro.open.ac.uk/33640/.

OER Research Hub (2013) OERRH Ethics Manual. Available from http://oerresearchhub.org/project-documents-archive/reports/oerrh-ethics-manual/.

OER Research Hub (2013) Project Website at http://oerresearchhub.org/.

OGC (2013) Open Geospatial Consortium: KML. Available from http://www.opengeospatial.org/standards/kml/.

35

Open Learning Network (2013). Evidence Hub for Open Education. Available from http://ci.olnet.org/.

36