student digital experience tracker pilot results 2016

PowerPoint Presentation

Student digital experience tracker 2016Results from the pilot project - Tabetha Newman and Helen Beetham 13/06/2016

Executive summaryJisc developed a set of 11 questions, designed to provide a snapshot of students' digital experience while engaged in a course of studyIn February-April 2016 Jisc delivered two versions of the questions (HE, and FE and skills) as a pilot tracker service for institutionsThe questions were delivered in BOS and piloted across 24 institutions (12 HE and 12 FE and skills, the latter including two specialist colleges)Pilot sites were supported through the process with identical guidance to simulate a real-world service delivery setting and to ensure equivalence of dataA total of 10,753 students completed the tracker questionsWe have a reliable snapshot of the situation with respect to device ownership and use, access to digital services, typical digital activities on course, and how students rate different aspects of their digital experienceThe data reveal significant differences between students in HE and in FE and skills13/06/2016Student digital experience tracker pilot results 2016 3

13/06/20163Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

ContentsIntroductionDevices used to support students learningAccess to institutional digital resourcesDigital skills within the course experienceInstitutional digital guidance and supportStudent digital experienceDigital technologies in learning: what students want to start, stop and keep

13/06/2016Student digital experience tracker pilot results 2016 4

Introduction


Development of the trackerA series of consultations carried out by Jisc during 2014-2015 found a demand for a better understanding of students' digital experiences and expectationsThe tracker questions were based on research by the Jisc Digital student project including studies of students in HE, FE and skillsA live consultation workshop with staff and students refined the proposed questions from 36 to 12Cognitive interviewing with FE and HE students, and early beta testing further refined the questions for piloting13/06/2016Student digital experience tracker pilot results 2016 6

Implementation of the trackerThe survey was implemented in BOS onlinesurveys.ac.uk, a system developed for UK HE and FE and already in use at many of the pilot sites50+ education providers applied to take part in the pilot: 24 were chosen based on a spread of institutional sizes, types and locationsPilot sites were required to engage stakeholders and explore their reasons for taking part in the pilotPilot sites were responsible for customising the survey, releasing it to learners, engaging learners to complete it, and analysing their own data13/06/2016Student digital experience tracker pilot results 2016 7

Implementation of the trackerSix separate guides were produced to support the pilot process and to ensure parity of the data collectedSites had only the guides and collective email/blog postings for support, to reproduce service delivery conditionsThe project negotiated for benchmarking data to be made available to pilot sites within the BOS system13/06/2016Student digital experience tracker pilot results 2016 8

Evaluation of the trackerSynthesis data has undergone preliminary analysis (this report)Further analysis is planned by this and other teams within JiscEach pilot site is supported to analyse, understand and respond to its own data, with the involvement of key stakeholdersStandardised feedback about the pilot process has been collected from the lead contact at all 24 pilot sitesThe experience of six contrasting sites is being explored in more detail, including feedback from learners and other stakeholders13/06/2016Student digital experience tracker pilot results 2016 9

Sample sizes and response ratesA total of 10,753 students across the 24 pilot institutions answered at least one question in the Tracker: a median average of 245 responses per institution7,425 students from twelve HE and 3,328 students from twelve FE and skills institutions completed the tracker (an average of 285 responses per HE institution, and 219 per FE institution)The maximum number of responses from one institution was 3,592 (from a university with about 20,000 students); the minimum number of responses was 25 (from a specialist college with about 40 students)Questions were optional, hence the exact sample size differs for every questionA total of 11 of our 24 institutions (46%) collected over 300 responses, and 19 (79%) collected at least 100 responses from students


Analysis and statisticsData is presented graphically where possible, usually as percentages of students choosing each optionWhere a dont know option was available, this data is usually included unless: HE student data is compared to FE and skills student data where possibleComparisons between proportions (eg between FE and skills versus HE students) were carried out using a Z Test calculator set at a 99.9% confidence level (higher than the minimum level of 95% in order to reflect the number of times that comparisons were run with this data set)The magnify glass (see below right) is positioned at the bottom right of pages where a particularly interesting result has been found


Q1: What digital devices do you use to support your learning?13/06/2016Student digital experience tracker pilot results 2016 12

Headline Q1 findingsWith regards to use of digital devices to support learning:Of the devices listed, students are most likely to use institutional desktops and printers to support their learningWhen it comes to personal devices, students are more likely to use laptops, smartphones and tablets than desktops to support their learning90% of HE students use personal laptops, in comparison with only 66% of FE and skills students40% of FE and skills students use personal desktops, in comparison with only 30% of HE studentsFE and skills students are significantly more likely to use institutional desktops, laptops, tablets and smartphones to support their learning in comparison with HE students


Removed dont know for Q3 because this is not a relevant option to show an audience. Instead more important to see the difference in the number of people who chose YES and not NO13/06/201613Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Personal devices13/06/2016Student digital experience tracker pilot results 2016 14HE, and FE and skills students are more likely to use personal laptops, smartphones and tablets than desktops to support their learning

HEFE and skills

Personal: FE and S students use more desktops, HE use more laptopsNote: at present Z tests are all significant at 0.05 (because of huge sample size) so need to counter for repeat testing of data set so changed to 0.001 (99.9 confidence level) via online calc below and have used this. Need to double check in SPSShttps://www.mccallum-layton.co.uk/tools/statistic-calculators/z-test-for-proportions-independent-groups-calculator/#z-test-for-proportions-independent-groups-calculator 13/06/201614Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Personal devices HE v FE differencesThere were two significant differences between HE versus FE and skills students in terms of personal use of digital devices to support their learning:90% of HE students use personal laptops, in comparison with only 66% of FE and skills students40% of FE and skills students use personal desktops, in comparison with only 30% of HE students


Note: at present Z tests are all significant at 0.05 (because of huge sample size) but need to counter for repeat testing of data set so changed to 0.001 (99.9 confidence level) via online calc below and have used this. Need to double check in SPSShttps://www.mccallum-layton.co.uk/tools/statistic-calculators/z-test-for-proportions-independent-groups-calculator/#z-test-for-proportions-independent-groups-calculator 13/06/201615Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Institutional devicesOf all the devices listed, HE and FE and skills students are most likely to use institutional desktop computers and institutional printers to support their learningOther digital devices used personally or in institutions included Kindles, scanners, 3D printers, iPods, cameras and games consoles13/06/2016Student digital experience tracker pilot results 2016 16

HEFE and skills


Personal devices HE v FE differencesFE and skills students are significantly more likely to use institutional desktops, laptops, tablets and smartphones to support their learning in comparison with HE students:Desktops: 84% of FE and skills students in comparison with 74% of HE studentsLaptops: 36% of FE and skills students in comparison with only 21% of HE studentsTablets: 12% of FE and skills students in comparison with only 7% of HE studentsSmartphones: 12% of FE and skills students in comparison with only 5% of HE students


Note: at present Z tests are all significant at 0.05 (because of huge sample size) but need to counter for repeat testing of data set so changed to 0.001 (99.9 confidence level) via online calc13/06/201617Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Q2: How often can you access the following digital services at your institution?13/06/2016Student digital experience tracker pilot results 2016 18

Headline Q2 findingsWith regards to accessing institutional digital services:Significantly more FE and skills students could frequently access computers and printers in comparison with HE studentsSignificantly more HE students could frequently access Wi-Fi in comparison with FE and skills studentsSignificantly more HE students could frequently access personal information (eg. grades, module choices) online in comparison with FE and skills studentsA significantly higher % of FE and skills students said they could never access personal information online in comparison with HE students



Computers and printers64.1% of HE students could access university computers and printers most of the time73.2% of FE and skills students could access college/learning provider computers and printers most of the timeSignificantly more FE and skills students could frequently access computers and printers in comparison with HE students13/06/2016Student digital experience tracker pilot results 2016 20

HEFE and skills

Wi-Fi in buildings79.5% of HE students could access Wi-Fi in university buildings most of the time70.0% of FE and skills students could access Wi-Fi in college/learning provider buildings most of the timeSignificantly more HE students could frequently access Wi-Fi in comparison with FE and skills students13/06/2016Student digital experience tracker pilot results 2016 21

HEFE and skills

E-books and e-journals89.7% of HE students could access e-books and e-journals some or most of the time; 5% said they could never access them and 5% said they didnt know53.0% of HE students could access e-books and e-journals most of the timeA further 36.7% could access e-books and e-journals sometimesThis question was not asked to FE and skills students13/06/2016Student digital experience tracker pilot results 2016 22

HE

Online course materials71.2% of HE students could access online course materials most of the time60.4% of FE and skills students could access online course materials most of the timeSignificantly more HE students could frequently access online course materials in comparison with FE and skills students13/06/2016Student digital experience tracker pilot results 2016 23

HEFE and skills

Other learning resources87.6% of FE and skills students could access other learning resources materials most or some of the time51.6% of FE and skills students could access other learning resources most of the time4% said they could never access other learning resources, and 9% chose dont know (perhaps this option was too ambiguous?)This question was not asked to HE students13/06/2016Student digital experience tracker pilot results 2016 24

FE and skills

Suggest we consider removing this in the revised Tracker13/06/201624Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Personal information online70.9% of HE students could access personal information (eg grades, module choices) online most of the time54.9% of FE and skills students could access personal information (eg grades, module choices) online most of the timeSignificantly more HE students could frequently access personal information (eg grades, module choices) online in comparison with FE and skills studentsA significantly higher % of FE and skills students said they could never access personal information online in comparison with HE students


HEFE and skills

Large disparity between HE and FE&S in terms of % choosing dont know option. So checked by removing the dont know, and this result still stands13/06/201625Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Online file storage80.0% of HE students could access online file storage (eg shared or Google drives, or in The Cloud) most or some of the time82% of FE and skills students could access file storage most of the timeThere was no statistical difference between HE and FE and skills students in terms of their access to online file storage


HEFE and skills

Q3: As part of your course, in the last six weeks have you done the following activities?13/06/2016Student digital experience tracker pilot results 2016 27

Headline Q3 findingsHE students are significantly more likely than FE/skills students to have produced work in a digital format, and to have worked online with others

During the last six weeks of their course:Nine in every 10 students found information onlineSeven in 10 students produced work in a digital formatFive in every 10 students worked online with othersThree in every 10 students created a personal record of their learning



Found information onlineAs part of their course, 96.0% of HE and 94.1% of FE and skills students had been asked to find information online in the past six weeksAlthough the percentage difference here is very small, there is a significant difference, with HE students being asked to complete this action more than FE and skills students in the last six weeksThis result is possible because of the large volume of data collected, allowing for a robust comparison between the two groups of students13/06/2016Student digital experience tracker pilot results 2016 29HEFE and skills


Worked online with othersAs part of their course, 62.8% of HE and 46.6% of FE and skills students had been asked to work online with others in the last 6 weeks (eg using discussion boards, Skype, Facebook groups, Twitter, Google+) There is a significant difference, with HE students being asked to complete this action more than FE and skills students in the last six weeks13/06/2016Student digital experience tracker pilot results 2016 30HEFE and skills

Produced work in digital formatAs part of their course, 79.1% of HE and 71.7% of FE and skills students had been asked to produce work in a digital format in the past six weeks (eg using presentations, web pages, info-graphics, animations)There is a significant difference, with HE students being asked to complete this action more than FE and skills students in the last six weeks13/06/2016Student digital experience tracker pilot results 2016 31HEFE and skills

Created personal record of your learningAs part of their course, 34.9% of HE and 32.6% of FE and skills students had been asked to create a personal record of their learning in the last six weeks (eg using a blog or e-portfolio)There is no significant difference between the proportion of HE and FE and skills students who had been asked to create a personal learning record in the last six weeks13/06/2016Student digital experience tracker pilot results 2016 32HEFE and skills

Q4: Do you get enough guidance and support to help you with the following?13/06/2016Student digital experience tracker pilot results 2016 33

Headline Q4 findingsWith regards to guidance and support:Less than half of all students say they have received guidance on: modifying devices to suit their individual needs, and creating a positive online presenceLess than two thirds of all students say they have received guidance on: bringing your own device, and developing digital skills relevant to their courseSignificantly more FE and skills students than HE students had received guidance on: bringing your own device, developing digital skills relevant to their course, behaving safely and respectfully online, and modifying devices to suit their needs



Using own device (bring your own - BYO)58.5% of HE students and 66.1% of FE and skills students agreed that yes, they had received enough guidance and support to help them use their own device (BYO) for learning in the institution (eg using their laptop, tablet or phone)Significantly more FE and skills students received guidance to BYO in comparison with HE studentsAbout two in ten students had never tried to ask for BYO support13/06/2016Student digital experience tracker pilot results 2016 35

HEFE and skills

Develop digital skills relevant to course56.1% of HE students and 67.1% of FE and skills students agreed that yes, they had received enough guidance and support to help them develop the digital skills relevant to their courseSignificantly more FE and skills students received this guidance in comparison with HE students


HEFE and skills

Behaving safely/respectfully online63.7% of HE students and 81.9% of FE and skills students agreed that yes, they had received enough guidance and support to help them behave safely and respectfully onlineSignificantly more FE and skills students received this guidance in comparison with HE students


HEFE and skills

Modify devices to suit individual needs32.0% of HE students and 39.3% of FE and skills students agreed that yes, they had received enough guidance and support to help them modify digital devices to suit their learning needs (eg text-to-speech, assistive software)Significantly more FE and skills students received this guidance in comparison with HE students


HEFE and skills

Create positive online profile46.3% of HE students and 46.5% of FE and skills students agreed that yes, they had received enough guidance and support to help them create a positive online profile (eg LinkedIn, CV, e-portfolio)There was no significant difference between HE and FE and skills students


HEFE and skills

Q5: How much do you agree with the following statements about the student digital experience?13/06/2016Student digital experience tracker pilot results 2016 40

Headline Q5 findings13/06/2016Student digital experience tracker pilot results 2016 41With regards to the overall student digital experience:Approximately seven in ten of students believe that when technology is used by teaching staff it enhances their learning experience Around six in 10 students believe that digital assessments are delivered wellFE and skills students are significantly more likely than HE students to say that they know how their personal data is used by their provider, and that they feel involved in decisions about the digital environment


Technology use by teaching staff72.2% of HE students and 70.1% of FE and skills students agreed that when technology is used by teaching staff, it helped their learning experienceThere was no significant difference in the opinions of HE and FE and skills students


HEFE and skills

Removed dont know here, as the proportion of those who dont know isnt of interest. Its between 4% (FE students, teaching staff) and 19% (FE students, support staff question) of student answers per question.13/06/201642Title of presentation (Insert > Header & Footer > Notes and Handouts > Header > Apply to all)

Technology use by support staff65.8% of HE students and 59.7% of FE and skills students agreed that support staff (learning resources, admin. etc.) use digital technologies confidentlySignificantly more HE students agreed with this statement in comparison with FE and skills students, who were more likely to have a neutral opinion


HEFE and skills

Im told how personal data is handled31.5% of HE students and 44.5% of FE and skills students agreed that they are told how their personal data is stored and usedSignificantly more FE and skills students agreed with this statement in comparison with HE students, who were more likely to disagree


HEFE and skills


Online assessments are delivered well58.1% of HE students and 55.8% of FE and skills students agreed that online assessments are delivered and managed wellThere was no significant difference between HE and FE and skills students


HEFE and skills

Learners are involved in decision-making37.5% of HE students and 43.2% of FE and skills students agreed that learners are given the chance to be involved in decisions about digital servicesSignificantly more FE and skills students agreed with this statement in comparison with HE students, who were more likely to disagree


HEFE and skills

Q6, 7, 8: What should we start, stop and keep doing when it comes to digital technologies?13/06/2016Student digital experience tracker pilot results 2016 47

Headline Q6,7,8 findings13/06/2016Student digital experience tracker pilot results 2016 48Students were asked to describe that they would like their institution to start, stop and keep doing with regards to using digital technologies The main issues are summarised in the following slides


HE: What to start doingThe following themes were often mentioned as something that HE institutions should start doing:Recording lecturesBetter use of VLEs: standardise use by staff, add presentations, teach students how to use it effectively, and improve access (eg mobile access)Improve online services: more online resources/activities, assessment submissionsAccess to better/more computers13/06/2016Student digital experience tracker pilot results 2016 49

FE and skills: What to start doingThe following themes were often mentioned as something that FE and skills institutions should start doing:Access to better/more computers and laptopsBetter Wi-Fi accessTeach students and staff how to best use technology (hardware and software) effectively and efficientlyPut more resources online and make them easy to find


HE: What to stop doingThe following themes were often mentioned as something that HE institutions should stop doing:Stop sending cluttering emails! (LOTS of students said this is there a better way to match students with relevant emails?)Early morning lecturesDeath by PowerPoint or very long lectures


FE and skills: What to stop doingThe following themes were often mentioned as something that FE and skills institutions should stop doing:Blocking access to websitesGiving multiple assignments at the same timeStop using Chrome machines, and google docs


HE: What to keep doingThe following themes were often mentioned as something that HE institutions should keep doing:Online 24/7 access to as much content as possible24/7 library accessPutting as much a possible (lecture notes, slides etc) on Blackboard, and making it accessible any time any placeUsing technology, and embracing new technology services and resources


FE and skills: What to keep doingThe following themes were often mentioned as something that FE and skills institutions should keep doing:Providing support for learnersAllowing access to social media and the internetAllowing home access to course materialsGood Wi-Fi connectivityEmbedding technology in the teaching learning experienceUsing google classroom


For more information contactSarah KnightSenior co-design [email protected]/06/2016Student digital experience tracker pilot results 2016 55

jisc.ac.uk

student digital experience tracker pilot results 2016

Education