Failing forward in research around technology enhanced learning

Associate Professor Michael SankeyRMIT University

Rachel WhitsedCharles Sturt University

Introduction

Let’s start with a popular quote

“…we all favour evidence that supports our existing beliefs rather than alternative explanations. We also tend to downplay our responsibility and place undue blame on external or situational factors when we fail, only to do the reverse when assessing the failures of others – a psychological trap known as fundamental attribution error.” (Edmondson, 2011)

The two cases

• We’ll share two instances where our research did not go as expected, but some clear lessons were learnt

• The 1st from CSU, looking at the success and efficacy of an application called “SkillBox”

• The instrument itself appears to be successful, but the research around it did not produce the anticipated results  

• The 2nd from USQ - an experiment conducted with 10 group of students using 6 multimodal learning environments

• The results did not match the hypothesis due to poor methodological issues

Both institutions:

• Strong online & DE providers clearly focused on L&T

• Higher numbers of off-campus students among Aussie Uni’s

• CSU - 22,967 (60.8%), USQ - 17,284 (72%)

• A strong focus on L&T research, ensuring students are receiving an equivalent or, as some have found, better learning experience than other modes of delivery (Lundberg, Castillo, Dahmani, 2008; Ya Ni, 2013).

• We see significant quantities of SoTL being conducted

• But it’s not as highly regarded as discipline based research

• This increases the pressure to produce research outcomes without sufficient time in workloads to meet these demands.

But how important is SoTL

“Although the number of journal articles pertaining to SoTL is increasing and the concept is gaining momentum in higher education, both nationally and internationally, it may not be universally accepted or well understood and not valued equally with that of discipline-specific research. (Lanning et.al. 2014, p.1353)

• But for strong teaching institutions, research into L&T needs to be valued as highly as discipline based research.

• We need rigorous approaches to questioning our teaching and designing, as this provides the affordance necessary to elevate our good teaching to appropriate scholarship (Burcham & Shaw, 2010).

Researching SkillBox• In 2014 SkillBox was developed by 2 academics at CSU

• A set of curated online adaptively scaffolded resources to guide students through a single knowledge area, at their own pace and in their own time

• Research into SkillBox was encouraged by a CSU DE innovation grant designed to stimulate a variety of areas related to online learning

• SkillBox was & remains a useful contribution to innovative L&T practices, but the research component has not been as successful as it might have been

• Student pre and post surveys collected quant & qual data looking at changes in attitude, knowledge and confidence in the topic area and on use

• 2015 Phase 1, students used SkillBox V1 in 2 small (< 20 enrolments) online courses, with a strong 36% response rate

• 2016 Phase 2, 100’s of students used SkillBox V2 across 8 courses

• But very low response rates threatened to sabotage the successful outcome of the research project.

• The academics had no formal SoTL training. Though ethics approval was gained, so the research plan had been scrutinised, many good practices were unintentionally not followed

• E.G. the survey was not piloted, some questions that should have been asked, were not, questions were not phrased well

• So the survey had to be redesigned between research phases, causing inconsistent results/analysis

• Technology proved a barrier. Surveys had to be held separate to SkillBox, allowing students to opt out, but still access SkillBox

• Many issues resulted in a disproportionate number of clicks to access the surveys and then SkillBox. The LMS may also have hindered progress. All contributed to a very low response rate

• While academics should be encouraged to undertake SoTL, more support is needed in the form of advice and collaboration in the research design and implementation.

Research multimodal learning environments• At USQ multimodal learning is a big thing. So research into this is important 

• The pressure to perform research is high, but not so much SoTL, unfortunately some activities not so well thought through, plus poor workload allocation

• Looking at the impact of multiple representations of teaching content on learning outcomes across different learning styles

• A quasi-experimental design was used for groups of students using different configurations of materials and then measuring their learning performance

• 60 students, 10 x 6 experimental groups. A small incentive was offered. Each undertook a learning styles inventory. 2 participants from each of the 5 learning styles allocated to each of the 6 groups (not quite)

• Students attended the test venue - undertook a pre-test. They then did 2 of 6 study conditions containing different combinations of materials (Text and Study Guide through to Text, Study Guide, printed PPTs, PPTs + audio, and interactive diagrams with script and audio. Then a post-test and an online survey about their experience.

• The methodology complicated the analysis, due to the limited number of participants (60) and the limitations of the quasi-experimental model

• Was an improvement in the scores from pre- and post-test (to be expected) the quant data did not indicate they performed better because of representations

• Qual data: students perceive the learning resources with additional content helped them understand, retain content, were more interesting and enjoyable

• The small sample size had a predominance of: higher-achieving students; multimodal learners; and a lack of aural and visual learners

• Multimodal learning may be of more benefit to lower-achieving students, while higher achieving students perform well regardless of how it’s presented

• If repeated we need a much larger sample & a higher group of lower-achieving students, and better representation across the different learning styles

• The unnatural study conditions and difficulties in controlling for extraneous factors in the design should be addressed

• The difficulties with the quasi-experimental design may explain the dearth of empirical data on the impact of multimodal teaching content on learning styles.

Lessons learned• The pressure to be active in SoTL can lead to academics being underprepared

and under-supported

• Discipline academics can lack the theoretical grounding for particular type of social research, so underestimate the preparation and planning required

• Time, funding and support from SoTL specialists (mentors) are crucial for academics to develop successful SoTL research programs

• Clearly communicating expectations between researchers and participants is critical. When not clearly set out, we can miss collecting valuable data

• Cultural barriers can limit the quality of data collected

• Be honest about limitations in the research. When response rates are low, it can be tempting to make claims that can’t be substantiated

• There are critical questions that need addressing around professional development, support, awards and promotion based on innovations in SoTL

• It is possible that students are over-surveyed

Conclusion

• We are left with a lot of academics demonstrating an extraordinary amount of good-will because they want to make their teaching practice hit the mark

• This means some projects will not go as well as others, but that’s OK.  Importantly, we need to learn from each other, from the communities of practice that exist around us, not just sharing the wins, but also sharing some of the losses

• “Determining what went wrong in a situation has value. But taking that analysis another step and figuring out how to use it to your benefit is the real difference maker when it comes to failing forward. Don't let your learning lead to knowledge; let your learning lead to action.” (Maxwell, 2000)

• The action in this case is better outcomes for our students.