An assessment of the potential benefits of Blackboard Analytics for Learn: meeting the needs of administrators, instructors and researchers

Matthew Bernacki Assistant Professor, Educational Psychology

Michael Wilder

Instructional Design Coordinator, Office of Online Education

Overview

As the University becomes increasingly reliant upon WebCampus’s Blackboard Learn software (Blackboard) for face-to-face, hybrid- and online courses, instructors, instructional designers, and administrators face new challenges with respect to 1) monitoring student use of technology for learning and 2) instructors’ use of technology to teach. Blackboard Analytics for Learn may have the potential to meet the needs of these stakeholders, and may also facilitate research about the quality of learning and instruction in technology-enhanced courses. The results of these studies could then be leveraged to improve student-learning outcomes by adapting instructors’ practices. There may be more precise and economical solutions. This document examines the individual needs of University administrators, instructors, and researchers, identifies the functionality that Blackboard Analytics for Learn can and cannot provide to address these needs, and then renders a recommendation regarding the utility of the analytics package.

Statement of needs Student Accountability In hybrid and online courses, students are required to access course materials that are hosted on Blackboard. At the course level, Blackboard Learn itself provides some data for faculty. User activity by content area (folders, assessments, assignments, etc.), user activity in forums, course activity overview is available. At times, the detail of this information is insufficient. For example, an instructor may encounter a student who claims to have conducted an action on Blackboard (e.g. completed a test), but no record of the test can be found in the instructor’s course interface. Ideally, the instructor could request a report from Blackboard and examine a log of student use of Blackboard for the period in question. At present, the software provides very basic reports that show:

Report 1: whether a student logged in that day, but not where they went OR Report 2: How many students accessed the course content item in question, but not which students accessed it (or when).

Stakeholder #1 need: Instructors of hybrid and online courses need a greater degree of detail regarding student actions in the system in order to be able to confirm the

learning efforts students claim to have made and to be able to make informed decisions regarding the efficacy of instruction. Instructor Accountability In hybrid and fully online courses, instructional designers and instructors collaborate to produce and deliver content for students to use when learning in a course. In some instances, instructional designers build courses that are used by multiple instructors, and are tasked not only with designing the material, but also with monitoring the fidelity of instruction (such as course logins, time engaged in a course, etc.). The Blackboard software that is currently in operation at UNLV does not provide a complete record of instructor actions in a course, limiting the ability of instructional design staff and administrators in academic units to monitor instructors’ activity in courses that meet via Blackboard. Stakeholder #2 need: Instructional designers and Academic Unit administrators need additional functionality from Blackboard software to ensure instructors are providing a course experience of sufficient quality. Opportunities to Understand and Improve Learning In addition to being able to confirm students’ actions for the purposes of accountability, instructors have a desire to assess the utility of the instructional materials the use in the technology-enhanced courses. To evaluate whether materials are beneficial for learning, instructors want to examine how individual students use Blackboard-hosted content items, and which approaches result in superior learning outcomes. At present, instructors cannot look at this data in sequence, or at the student level. Stakeholder #3 need: A fine level of detail about student learning to improve educational experiences. Similar to the needs of faculty and instructional staff (Need #1), educational researchers are in need of a more refined data management system that captures student learning at a fine-grained level of detail. To better understand how students learn best when using technology, researchers need to be able to see the sequence and duration of learning events, and the relations between students' actions. As described above, the current report structure available in Blackboard Learn does not provide this level of detail.

Affordances of Blackboard Analytics for Learn to address University needs As stated in the first section, the needs of administrators and chairs of academic unit as well as faculty and researchers can be addressed with two specific types of functionality. In order to support all these stakeholders, we need a learning management system that can:

1) Track student use of Blackboard features in a fine-grained fashion. By obtaining a log of student actions, we can help faculty monitor students’ use of

learning materials for both student accountability and the evaluation of their own instructional approaches. We can also facilitate research on the benefits of learning activities in online and LMS-supported hybrid environments.

2) Track instructor use. A fine grained log of instructor use is necessary for the Office of Online Education to monitor PTIs instruction in courses, and can be useful for Department Chairs to evaluate faculty use of the LMS system, where warranted.

Following is an example of possible reports obtained from Blackboard Analytics for Learn for Learn:

• Student at-a-glance • Faculty at-a-glance • Most active instructors • Aggregate activity by organizational unit • Logins by day of week • Days since last login • Length of sessions • Course tool use • Average discussion posts per user • Courses with no content • Building block usage

Based on an evaluation of the literature we have received that describes the functionality of Blackboard Analytics, it is unclear that the Analytics package can provide the fine degree of detail that educational researchers may require. Most of the functionality comes in the form of new reports, which summarize data, rather than provide fine-grained details. Blackboard Analytics for Learn may provide useful information for instructional design and for University administration, however, at an aggregate level. This could be useful for administrators who wish to examine enrollment in courses (e.g. Registrar, Provost, etc.) or for Academic Unit heads to plan course offerings, but it does not offer much to individual faculty members. Faculty members may find some other tools useful, such as their tools that ply logistics regression models to predict student achievement based on completion of certain activities in Blackboard (e.g. completing the first paper on time predicts a B or better on final grade). However, many of these findings are likely going to be intuitive, and can be supported with some simple support in other ways (e.g. partnering with an Educational Psychology or assessment and evaluation faculty member) using either basic Blackboard functionality or an available Blackboard building block.

Potential alternative solutions Do-it-yourself data analysis Following the model established by Cal State Chico, a wealth of information can be obtained by querying the database directly. Given access to the data, a qualified database administrator should be able to generate basic reports (such as student/faculty logins, empty courses, basic tool usage, etc.) using standard data querying methods (SQL statements). Building block to log student behavior At an OIT-sponsored meeting with a Blackboard engineer on 11/14, we were informed of a building block designed for Financial Aid officers that tracks student behaviors online. The building block will be available in the coming months, and is reported to log all student actions for the purposes of verifying their involvement in courses in order to approve their financial aid. We have yet to verify the reports made available by this tool but are in touch with a Blackboard engineer (Liam Ferris) who has offered to help us explore its functionality. If the building block works as described, it should address faculty and researcher needs regarding student use of the LMS. Pass-through software to log student behavior Because UNLV-sponsored coursework on Blackboard is hosted via the University’s servers, we could also capture student use of the LMS via pass-through software. In this approach, software would be set up so that, when students access Blackboard, their requests are routed through software that logs the students’ Blackboard account information and their actions, without influencing the way they use the LMS interface. This transparent method would allow students to continue using the LMS as normal, but would also produce a locally housed log of student behaviors. This log can be sorted by course and provided to instructors as requested for the purposes of monitoring student behavior for accountability or course improvement. It can also be anonymized and processed for researcher use. Andy Stefik, a faculty member in Computer Science, has expressed a willingness to partner with the co-authors and OIT leadership to set up this system to facilitate a research proposal that is to be submitted to the National Science Foundation in January (Research on Education and Learning program; REAL).

Outstanding Need

While our alternative solutions could meet the needs of faculty and researchers, solutions to meet the needs of administrators who wish to examine LMS use by instructors and examine LMS use across courses have not yet been identified. It is not clear that the functionalities provided by the Analytics package will satisfy the needs of these stakeholders.

Additional effort should be undertaken by the Office of Online Education to identify potential data solutions for LMS user logging across courses, and some polling of Academic Unit leads is warranted to determine their needs re: evaluation of course offerings and instructor use.

Recommendation

Because the cost to acquire Blackboard Analytics for Learn is high, time to implement to long, and the likelihood that the package meets all of our needs is low, we currently recommend not moving forward with a purchase at present. Instead, it would be good for the Office of Instructional Technology, the Office of Online Education and a core of faculty and researchers (potentially from the CMSCC committee and others with expertise) to pilot DIY database analytics, building blocks, and pass-through software in a subset of courses or time periods to examine whether they offer satisfactory solutions. It is possible to attempt these approaches in the Spring, spend the Summer considering their viability and, if warranted, acquiring a third-party analytics package for Fall 2014. Note: As University officials consider how UNLV will approach learning technologies, we also provide a set of common questions governing the use of technologies at postsecondary institutions.

Questions to consider Following is a set of important questions that need to be considered prior to investment in any learning analytics solution:

• What are the specific goals of the data analytics? • Who will gather the data? • Who will act upon the results? • Who will maintain any processes/software used to consolidate the data? • Who will provide funding?

If proprietary software is used to consolidate the data:

• Will findings be treated as proprietary and will not be made available to other researchers?

• How are various factors weighted in an algorithm? • Are the concepts being analyzed the rights ones? • Can researchers adjust the algorithms of vendor tools to conduct experiments of

other factors that might impact learning? Questions concerning ethics and privacy

• Should students be told that their activity is being tracked? • How much information should be provided to students, faculty, parents, issuers of

scholarships and others? • How should faculty members react? • Do students have an obligation to seek assistance? • What data is appropriate to collect about student? What data is inappropriate? • Who should be able to access the data and view results? Which data should be

reported anonymously? • What is the impact of showing faculty modeling results? Do any of the data bias

faculty instruction and evaluation of students? Questions regarding the interpretation of data

• Who gets to interpret the data? • Who owns the data? • Who benefits from the data? The institution, the instructor, or the student? • Can data be use against instructors/departments/institutions to determine

employment, tenure or funding? • Is time in an LMS representative of engagement? Simply because educational

content pages are open in a browser, does it mean that a student is processing the information?