sustaining digital learning provision gem conf 2011

Sustaining digital learning provision

GEM conference 2011 Norwich8 Sep 2011

Martin Bazley

Martin BazleyPreviously• Teaching (7 yrs)• Science Museum, London,

Internet Projects (7yrs)• E-Learning Officer, MLA South East

(3yrs)

Martin Bazley• Current• Vice Chair of Digital Learning Network

DLNET• Developing online resources, websites, user

testing, evaluation, training, consultancy…Martin Bazley & Associateswww.martinbazley.com

Slides and notes available online afterwardswww.slideshare.net/martinbazley

http://www.martinbazley.com/

http://www.slideshare.net/martinbazley

www.martinbazley.com

Note to self: check stats tomorrow to see if anyone looked up the website

To sustain your digital learning offer:• (a) find out and

(b) stay in touch with what your users want – gather views and data (online audience research)

• Use iterative review as part of planning and developing your online offer

• Make use of free and low cost tools and services

Session elements:

planning, developing and evaluating online resources

intro to planning digital audience research

free / low cost digital tools and services / techniques for developing online resources

not only resources labelled as ‘learning’: teachers and others are using content in different ways – teachers mainly look for images, videos, etc

also learning experiences using digital technology – see New Media handbook

the key point is that review and evaluation are essential to sustaining digital learning offer

Various advice and guidance from:

onlineaudiences.wordpress.com (this link will be repeated later)

www.martinbazley.com (for contact details)

http://onlineaudiences.wordpress.com/


Website users• Who uses your website (or resource) ?

• Why would they want to use it?

• How would they find it?

• What do they get out of it?

• What do they dislike about it?

How do you get it right for everyone?

• Answer:• You can’t get it right for everyone.

• You have to make choices, and stick to them:

• Who is it for? • What..• How…

Who for…?Who for…?What for?What for?

How will they use it?How will they use it?

Who for what for ...• Who for? (audience)

Need to be clear from start• mum + 2 children looking for something to do

this weekend• teachers of yr5/6 in local area with whiteboards• men interested in gadgets

Who for what for ...• What ‘real-world’ outcomes?

What will they do as a result of using the site?

• make a donation• plan a visit to a museum• buy a train ticket• think differently about learning disability

Who for what for ...• How will they use it? (user experience)

What do they actually do on the site? • browse and read articles

• working alone or in pairs? (learning resources)• lean forward or sit back?• Browsing, following, searching…

• Also Where, When and Why?

Testing is an iterative process

Testing isn’t something you do once

Make somethingMake something=> test it => test it

=> refine it=> refine it=> test it again=> test it again

Developing a learning resource: iterative review

your content curriculum (find a match)

Ch

eck

Does it match your audience’s specific needs?

If so TEST - and then amend

Learning activities Learning outcomes (find a match)

When to evaluate or test and why

• Before funding approval – project planning

• Post-funding - project development

• Post-project – summative evaluation

Before funding – project planning• *Evaluation of other websites

– Who for? What for? How use it? etc– awareness raising: issues, opportunities– contributes to market research– possible elements, graphic feel etc

• *Concept testing – check idea makes sense with audience– reshape project based on user feedback

Focus group

Research

Post-funding - project development• *Concept testing

– refine project outcomes based on feedback from intended users

• Refine website structure– does it work for users?

• *Evaluate initial look and feel – graphics,navigation etc

Focus group

Focus group

One-to-one tasks

Post-funding - project development 2

• *Full evaluation of a draft working version – usability AND content: do activities work, how

engaging is it, what else could be offered, etc

Observation of actual use of website

by intended users,

using it for intended purpose,

in intended context – workplace, classroom, library, home, etc

Post-funding - project development 3

• Acceptance testing of ‘finished’ website– last minute check, minor corrections only– often offered by web developers

• Summative evaluation– report for funders, etc– learn lessons at project level for next time

Website evaluation and testingNeed to think ahead a bit:

– what are you trying to find out?

– how do you intend to test it?

– why? what will do you do as a result?

The Why?Why? should drive this process

Martin BazleyOnline experience consultant

Evaluating online learning resources in the classroom

Key point:

for a site designed for schools,

the most effective user testing

observations

will be made in a real classroom situation

National Archives Moving Here projectalso Victorians (with V&A) – about to be released

For teachers of 8 – 14 yr olds

History Geography and Citizenship

Features: Interactives, activity sheets, audio and video

clips

Moving Here Schools:For 8 – 14 yr olds studying:

History Geography and Citizenship

Features:

Interactives, activity sheets, audio

and video clips

1. preliminary testing sessions –

conventional user-testing with teachers (at TNA)

2. in-class testing – teachers used the Moving Here Schools site with pupils in their own classrooms This meant sitting at the back of the

classroom observing and taking notes…

Evaluation: 2-phase approach

Site ready in parts – but not too ready:

The environment had a significant impact on how the site was used.

The class dynamic within the different groups contributed to how much the students learned.

The environment and social dynamics

The environment had a significant impact on how the site was used.

The class dynamic within the different groups contributed to how much the students learned.

in-class testing picked up elements not there in conventional user testing.

teachers in preliminary user testing did not spot some problems until actually in the classroom. For example…

interactive activities:

looked big enough when viewed on a screen nearby…

… but text/images too small for some children to see from the back of the class…

…so interactives needed to be viewable full-screen

Only spotted during in-class testing:

…so interactives needed to be viewable full-screen

content: when students tried to read text out loud, teachers realised some text was too difficult or complex

activity sheets:

some sheets did not have spaces for students to put their names - caused confusion when printing 30 at same time…

Manchester Art Gallery art interactive

For teachers of 8 – 11 yr olds, and for pupils

History Art and Citizenship

Features: interactive with built in video, quiz, etc,

plus activity sheets and background info

Martin Bazley

www.ICT4Learning.com

'This classroom user testing is all very well, but...'

How can you see everything in a class of 30 children – don't you miss things?

You see things in a classroom that don't arise in one-to-one testing

They are the real issues


Doesn't using a specific class with particular needs skew the results?

» For example, low ability, poor English, equipment not working, behaviour issues, etc - are results as reliable as those in a 'neutral' environment?

» ‘neutral environment’ ? – no such thing - any test will be subjective, and in any case:

» Testing is to make website work well in classroom, - need to see effects of factors like those.



Can't my Web developer do the testing for us? » best not to use external developer to

do user testing - conflict of interest » also likely to focus more on the

technical aspects of the site than on effect on the teacher and pupils.

» observe classes yourself but use an independent evaluator for key decision points



Can't my Web developer do the testing for us? » best not to use external developer to

do user testing - conflict of interest » also likely to focus more on the

technical aspects of the site than on effect on the teacher and pupils.

» visit a classroom yourself but use an independent evaluator for key decision points



I don't have the time or budget to do this!

» need cost no more than conventional user testing. one person could attend a one-hour class session in a school, giving the teacher the same small token payment

» This programme had evaluation built into project: 6.7% of total Schools site budget.

» Allow 5 -10% of total project budget for user testing

=> videos


Video clips • Moving Here key ideas not lesson

plans etc http://www.vimeo.com/18888798

• http://www.vimeo.com/18892401 Lesson starter

• Time saver http://www.vimeo.com/18867252 S

http://www.vimeo.com/18888798



User test early

Testing one user early on in the project…

…is better than testing 50 near the end

Two usability testing techniques

“Get it” testing- do they understand the purpose, how it

works, etc

Key task testing- ask the user to do something, watch how

well they do

Ideally, do a bit of each, in that order

User testing – who should do it?• The worst person to conduct (or interpret)

user testing of your own site is…– you! you!

• The 2nd person is…– Your web developer! Your web developer!

• Beware of hearing what you want to hear…

• Useful to have an external viewpoint• First 5mins in a genuine setting tells you

80% of what’s wrong with the site

Plan, evaluate, do audience research

o How to plan and conduct research into online audiences:http://sca.jiscinvolve.org/audience-publications/

o Focus on why you are measuring what you plan to measure

o Web stats, online questionnaires, focus groups (concept testing), user testing – various ways of gathering data

o Build costs into the project for regular evaluation – avoid expensive remedial action later in project

http://sca.jiscinvolve.org/audience-publications/

How can we get a sense of who our online visitors are and what they do with our online content?

How do we gather data to help us improve what we do?

How do we measure success from the user's point of view, and against our own objectives and constraints?

For example, how justify investment (or lack of it) in digital, including social networks etc?

Tools available

• Qualitative – focus groups, “free text” questions in surveys, interviews

• Quantitative – web statistics, “multiple choice” questions in surveys, visitor tracking

• Observational – user testing, ethnographic

examples

• SurveyMonkey online survey

• Google Analytics website statistics

http://www.surveymonkey.com/

http://www.google.com/analytics

Define audience research goal

Analyse data

Collect data Use results to guide changes

Plan methodology

Martin Bazley0780 3580 737


More information / advice / ideas


Freebies and cheapies

o WordPress and other ‘free’ CMSs for creating websites, blogging, or prototyping

o Online survey tools, online event booking

o YouTube, Vimeo, podcasting

o Social media: Facebook, Twitter

o Google maps, Google site search, Google Analytics

o TripAdvisor, etc

Making websites - CMSs

http://wordpress.com - free website creation service (pay extra for features like own domain name etc). All hosting and upgrading etc is done for you


http://wordpress.org - free website creation service – like .com except you have to install it on your server and you are responsible for updating, hosting costs, etc


www.contentcurator.net - free open-source CMS specially developed for cultural and heritage sector. Powerful and easy to use e.g. in-place editing


www.cmsmadesimple.org - free website creation service – you install it on your server and you are responsible for updating, hosting costs, etc - very similar to wordpress.org

Prototyping with WordPress

http://ruskinelementsofdrawing.wordpress.com

http://educationonline.ashmolean.org/ruskin

Video – maximise impact

http://vimeo.com

http://ruskinelementsofdrawing.wordpress.com

Social media – maximise impact?

Has potential to increase engagement but be aware:– long term commitment essential so financial

implications in resourcing– Can also work against you. DLNet vs. Dulwich Picture Gallery blog

Online audiences - links

Advice and links on online audience data gathering, evaluation, New Media handbook etc

http://onlineaudiences.wordpress.com/

More info / advice / ideas

Martin Bazley

0780 3580 727


sustaining digital learning provision gem conf 2011

Education

online learning resources

online resources

digital learning offer

funding project planning

online offer

website evaluation

learning disability

project outcomes