Report for the Student Services Web Page Management Group

FROM: Web Page Usability Testing Team SUBJECT: Web Page Usability Test 1st December 2004DATE: 1st February 2005

BACKGROUNDThe members of the Web Page Usability Testing Team had intended, individually, for some time, to use students to test web page usability before the opportunity to develop this exercise arose. The catalyst for action was the decision to change the Things Not Going Right / Signposts Questionnaire page.

Web Usability Testing Team Members:

Gill Bartlett (Learner Support), who organises focus groups of new students to identify additional support requirements and has had a long term interest in testing the web pages.

Scott Castle (Team Leader of SSiD and Student Services Web Page Manager), who has had some previous experience in usability testing in the systems development environment.

Gill Hughes (Information Assistant in SSiD), who gained recent exposure to many different web pages when helping her son decide which universities to apply to and initiated the development of the Useful Contacts for Current Students Page (www.shef.ac.uk/ssid/contacts.html)

SSID Web PagesThere are some 800 pages in the SSiD suite of web pages. The ‘site’ receives over 200,000 hits (50,000 visits) from 25,000 unique sites each month. Some of the more popular pages are Your Record (7,000), Course Information (10,000) and Examination Timetable (up to 20,000).

Most of the changes undertaken in the last year have been to improve accessibility to these pages, devolve responsibility for updating content and a limited amount of time has been available to improve individual page design, navigation and usability. This exercise was seen as a good opportunity to increase the priority of time given to usability, navigation and individual page design issues.

1. THE AIMS OF THE EXERCISEA conscious effort was made to ensure that we were collecting data relating to the users actual behaviour rather than their just their opinions.

“To design an easy-to-use interface, pay attention to what users do, not what they say.” Nielsen (2001)

Furthermore we wanted the information collected to support changes rather than determine changes to the web pages.

- 1 -

“The purpose of testing is not to prove or disprove something. It’s to inform your judgement.” Krug (2000) p142

The students were not selected from specific groups but from those who had been used in a previous focus group and were known for their ability to express themselves.

Finally, the exercise was seen as a pilot test: an opportunity to test and refine the experimental procedure.

2. THE EXERCISE: PREPARATION, EXECUTION AND REVIEW2.1 Preparation

2.1.1 Format Following from our research into the subject and some discussion, it was agreed that:

a) the usability tests would be conducted over the course of one working day. There is published research to indicate that there is no need for big numbers of web testers. Even one will show you things that you can do to improve your site and 5 users will tend to uncover 85% of a site’s usability problems.

b) the tests would involve observing students navigating the current students (SSiD) web pages trying to achieve objectives which would start with easy tasks but become increasingly challenging.

c) the tests would be conducted with one student at a time and involve a facilitator (to guide and prompt the student) and an observer (to take notes).

d) only first year students would be recruited for the tests as they were more likely to be relying purely on the usability of the site to find information. Second or third year students were more likely to have memorised where information could be found or be already familiar with our terminology through past experience of using our web pages.

e) The members of the testing team would take it in turns to act as facilitator and observer in order to build up expertise in both roles.

f) that the test for each student should last for approx 50 minutes.

2.2.2 ResourcesApart from our staff time, the resources required to complete the exercise were negligible.

As students were being tested individually, a large room was not required. The Student Services Meeting Room was used because it had adequate space, communications and was convenient. The room was set up with computer, audio and video equipment (more details about the hardware later in the report). All the necessary equipment was provided and set up by Audio Visual Services free of charge.

Five undergraduate students were recruited (of which four turned up) and provided with a £10 book voucher as a token of our appreciation for their help. Apart from the book tokens there were no other direct costs involved in the exercise.

- 2 -

Of the 4 students who were involved in the usability testing:

2 were male and 2 were female 1 was a very mature student 2 were local or ‘commuter’ students they were from varied departments/disciplines (Archaeology, Law,

Mechanical Engineering and Psychology)

2.2.3 The ScriptThe team started with a script suggested by Krug (2000) (a version of which was downloaded from www.sensible.com/). However significant changes were soon made to include the specific requirements for the test (See Appendix 3 for the script that was eventually used).

The intention was that the facilitators would stick to the script, although a certain amount of improvisation was allowed. This obviously, did not include leading the student or giving clues as to what they should do.

The script started by:

informing student web testers why we were conducting the usability tests reassuring them that it was the web site being testing and not them –

therefore they couldn’t do anything wrong. asking them to be as honest and as critical as they liked without worrying

about hurting anyone’s feelings reminding them to do all their thinking out loud – i.e. to vocalise throughout

the exercise what was going through their minds.

The students were asked for their permission to record both their dialogue and their navigation around the computer screen. (None objected).

2.2 The Test Itself

2.2.1 The Tasks: a) Contextual Questions, to put the student at ease and gain an idea of their

web usage experience and any bias. b) Four ‘Scenarios’, which challenged students to find a particular piece of

information. Each scenario consisted of two or more objectives: the initial task and a related progression e.g. find exam timetable (task), find exam clash form (progression). (See Appendix 1)

c) Home page evaluation, which consisted of asking the student their impression the Current Students ‘home’ page (http://www.shef.ac.uk/ssid/)

d) Evaluation of another university’s Current Students ‘home’ page.

2.3 Evaluation of the Testa) Only minutes into the first test, it was obvious that some useful insights were

being gained into how these web pages were used. Afterwards, once the

- 3 -

recordings had studied it was realised that the exercise had indeed obtained a large wealth of information with minimal effort and cost.

b) The key task testing approach, using the four scenario exercises which involved the students finding information and performing key tasks, had proved to be particularly useful.

c) The least useful part of the test had been the evaluation and comparison of the ‘current students’ home pages for this and another institution. Although a few interesting comments had been made, students seemed to struggle to find things to say. This may have been due to several factors: high expectations by testers, reticence of the students, or the inappropriateness of the test for this task.

d) It was decided that the exercise would be repeated in February.

3. INFORMATION CAPTURE

3.1 Video and Audio recordingThe equipment set up (scan converter, video recorder, microphone, monitor) allowed camera video taping and capture of the click streams (video) and student comments (audio).

3.2 Observers Note Taking

a) Contemporaneous notes. Notes were made by the observer during the test. These notes were generally reflections of the observer rather than the comments of the student which were colleted more comprehensively later from the audio recording.

b) Audio recording. A sound file was created (Wave format) from the Video, that could be listened to at a later time via a PC so that more detailed notes, from the students comments, could be made.

c) The notes collected in a) and b) above were recorded on a Word document. (See Appendix 2: Usability Test 011204 (Notes).

3.3 Tracking User Click StreamsUsing the video, the user click streams were recorded onto Excel Spreadsheet (See Appendix 5: Usability Test 011204 (Clicks).

4. FINDINGS

4.1 Content VolumeThe students commented on several occasions that there was a lot of information to read and that if it was not a test environment they would cease searching for the

- 4 -

information. In Scenario 4 – Things Not Going Right, one student commented that, if you were distressed or anxious, you would not take the time to read four hundred words.

4.2 Accessing SSiD pagesThere were a variety of methods that students used to access the SSiD web pages. These included clicking on current students on the table of contents, typing SSID in the left hand search box and going to the ‘departments’ page. [10] [13]

4.3 Students not knowing what they did not know.When trying to find specific information the students experienced frustration by not knowing the difference between information that may meet their requirements and information that was totally irrelevant.

a) There was confusion between what would be the most appropriate document showing examination results: Confirming letter / Certified Copies of Degree Certificate / Transcript etc. [4] [14]

4.4 Finding Specific Information Sets There were several issues generated when the student tried to find a Transcript:

a) Most did not know that a Transcript was an ‘official’ summary of examination results. [4]

b) All expected the link to the appropriate information to be located on ‘Your Record’ page (http://www.shef.ac.uk/ssid/record).

4.5 Search Box Effectiveness.The left hand search box did not seem to produce useful results and generated several negative comments from the students. [11] [15]

4.6 Choosing Appropriate Information MediumIn respect of scenario four, all the students said they would approach their personal tutor rather than trying to find information on the web. [7] [14]

4.7 Link DescriptionsSome link descriptions were identified as either inappropriate or leading to pages that provided content that was not expected or wanted.

a) A student followed a link on the Student Welfare page (www.shef.ac.uk/ssid/welfare) titled ‘Student Support & Guidance’ anticipating that it would be a link to useful information relevant to the issue she was trying to resolve. The link went to administrative information about the Student Support and Guidance section in the Student Services Department. As the student put it: “When I clicked on Student Support and Guidance I thought I would get something about Student Support and Guidance” [8]

b) More than one student noticed that when they clicked on a link title ‘Things not Going Right’ they were presented with a page that was titled ‘Signposts Questionnaire’. [9]

4.7 Links to Further informationOne student came across a section on a page relating to Short Term Loans and assumed that this was the exhaustive content relating to the subject. No link existed to take the student to the definitive page. [16]

- 5 -

Notes referenced above can be found in Appendix 1: Web Page Usability Testing – Notes

- 6 -

5. CHANGES MADE TO WEB PAGES

5.1 Things Not Going Righthttp://www.shef.ac.uk/ssid/welfare/signposts/index.htmla) The introductory page was rewritten. The number of words on the page was

substantially reduced. b) Links were also added to the introductory page to categories that were separated

into subject areas on a series of (five) new pages.

5.2 Examples Pageshttp://www.shef.ac.uk/ssid/exampleCreated a series of (ten) new pages which display a picture of particular documents that can be obtained from SSiD.

5.3 Current Students (SSiD) home page changes. http://www.shef.ac.uk/ssid/index.htmla) A series of images (8) which show two pictures: a picture of a student using

the SSID facilities to find information for themselves and a picture of a student talking to Information Assistant at the SSiD front desk.

b) Revised the middle section of the page so that it looked more consistent with University home page style.

5.4 Summary of Results pagehttp://www.shef.ac.uk/ssid/record/results.htmlA new page was created to detail the purpose, components and appearance of Summary of Results report.

5.5 Your Record Linkshttp://www.shef.ac.uk/ssid/recordIntroduced links to the Transcript of Qualification and new Summary of Results pages. Also changed the order of the to make the ‘Changing your Address’ section visible when the page is first entered.

5.6 Certificate of Student Status pagehttp://www.shef.ac.uk/ssid/forms/certify.htmlChanges to this page to illustrate the variety of certifying letters there are available on request in SSiD. (taught, taught with fee, bank and term time or home address)

5.7 Expanded Frequently Asked Questions pagehttp://www.shef.ac.uk/ssid/faqArising out of the change to the ‘Things Not Going Right’ page, 84 new questions were added and the subject areas organised into nineteen categories.

5.8 Revised the A to Z index and Other Useful links pageshttp://www.shef.ac.uk/ssid/atoz.htmlhttp://www.shef.ac.uk/ssid/general.htmlSeparate links into those that are internal to the University and Union (A to Z index) and those that are external (Other Useful Links). Secondly made the list of links single column and added a navigation link to return the user to the top of the page.

- 7 -

5.9 Student Support and Guidance http://www.shef.ac.uk/ssd/ssgChanges were made to the ‘home’ page of the Student Support and Guidance section. This involved redesigning the page to make it read more like an informative page to students rather than a list of administrative aims and objectives.

5.10 New Frequently Asked Questions category: What is it?http://www.shef.ac.uk/ssid/faq/what.htmlA new page was created on the FAQ section. The questions try to explain some of the University terminology for the student and remove some confusion. Questions include: A Transcript of Qualifications – What is it? Access to Learning Fund – What is it?

5.11 Other Minor Changeshttp://www.shef.ac.uk/ssid/transcript/index.htmlMinor changes were made so that the page was more attractive and usable, a link to Summary of Results was introduced and an explanation provide, up front, about the likelihood of charges for the service.

Not all of the above changes were in direct response to student comments, suggestions or behaviour. Some changes were indirectly inspired by the exercise as a whole.

6. CHANGES MADE TO THE TEST MODEL6.1 The Current Students (SSiD) home page evaluation will be deferred to a later date and not included in the February test. Further investigation needs to be made into how to make the test more effective / students more forth coming with views.

6.2 One or two additional scenarios can be added to February test as a result of a).

6.3 Some script changes will be made. In particular, more guidance will be given to the facilitator, to ensure that each student starts the particular task at the same point.

6.4 Specific scenarios to be repeated in the February test. This will be done principally to test changes we have made as a result of the December 2004 test.

6.5 The facilitator will demonstrate ‘thinking aloud’ to the student before the first scenario begins. Suggested by Nielsen (93) p 197

6.7 Scenario 4 will be developed so that the student will be encouraged to refer to the web pages before talking to a tutor. This could be achieved by adding the caveat that it is late at night or it is a subject you do not want to talk to your personal tutor about.

- 8 -

Bibliography

Advanced Common Sense Home page at http://www.sensible.com/

Krug, Steve (2000) “Don’t Make Me Think! A Common Sense Approach to Web Usability” New Riders Publishing

Neilsen, Jacob & Tahir, Marie (2002) “Homepage Usability: 50 Sites Deconstructed” New Riders Publishing

Neilsen, Jacob (1998) “Cost of User Testing a Website” May 3, 1998 http://www.useit.com/alertbox/980503.html

Neilsen, Jacob (2000) “Designing Web Usability: The Practice of Simplicity” New Riders Publishing

Neilsen, Jacob (2000) “Why You Only Need to Test With 5 Users” March 19, 2000 http://www.useit.com/alertbox/20000319.html

Nielsen, Jacob (1993) “Usability Engineering” Morgan Kaufmann Publishers Inc.

Rosenfeld, Louis & Morville, Peter (2002) Information Architecture for the World Wide Web” ORielly & Associates

Rubin, Jefferey (1994) “Usability Testing: How to plan, design and conduct effective Tests” John Wiley & Sons

Nielsen, Jacob (2001) ‘First Rule of Usability? Don’t Listen to the Users’ August 5, 2001 http://www.useit.com/alertbox/20010805.html

- 9 -

Appendix 1: Scenarios for Web Page Usability Test 1st December 2004

SCENARIO 1: ExaminationsCan you imagine that you have exams coming up that involve using a calculator. Find the dates of your exams and information about using calculators in exams.

Progression: You have checked the dates and times and realise that you are timetabled to take two exams at the same time. A friend has told you that you can report a clash of exams online. Find the Examination clash form.

SCENARIO 2: Personal RecordsYou are just about to graduate, move out of your term time accommodation and go on holiday. You want to make sure that the tickets for your graduation ceremony arrive at your new address. Find a way to change your address online.

Now you have found this page explain the difference between a home and correspondence address.

Progression: Now try and find how you can get a document that details all your modules and exam results.

SCENARIO 3: Finance Imagine you are a final year student experiencing severe financial difficulties. A friend tells you that the University can provide additional Financial Support through something called “Access to Learning Funds”. Try and find out whether you would be eligible for this.

Progression: You complete the application form and are told that it takes 3 weeks to process. You also find out that your next student loan payment is going to be 3 weeks late because the computer system at your home LEA has crashed. Try to find information about an emergency loan that can help you until your funding comes through.

SCENARIO 4: Help with Problems Imagine that your course has not turned out to be what you expected. You are not enjoying it and have started to miss lectures and tutorials. You feel that you are performing poorly. You wish you had chosen another course and feel increasingly despondent. You suspect that you will fail your exams and are wondering whether you should withdraw from the course. What should you do? Can you try to find guidance on what you should do from these web pages.

- 10 -

Appendix 2: Web Page Usability Testing – Notes 1st December 2004

The Tasks in the ExerciseSee Appendix 1 for the tasks (including scenario descriptions) that were presented to the student as part of the Web Page Usability Testing exercise.

Tracking User Click StreamsSee Appendix 5 Usability Test 011204 (Clicks) for the user click streams of each student. Each note below is itemised on the spreadsheet at the click at which it was generated.

Student 1Scenario 1: Examination Timetable details1. Examination identified by Unit code. Suggested that a search facility (by unit code or title) may be useful or a drop down menu. 2. The link to the clash form should be at the top of the timetable page rather than on the previous page.

Scenario 2: Changing Address details3. Home addresses are usually parental addresses. With mature students this would be your own home. The student found that a lot of the terminology used is assuming that all students fall into the non-mature category.4. The student was unsure about what kind of document may summarise exam results. When asked later, she had not heard the term Transcript of Qualification “I expected it to be some part of Your Record”“I am wondering whether it is to do with certified copies of degree….letter of degree confirmation…I wouldn’t want that – it costs money”5. “Certified copies of degree certificate….expect it to be something to do with this …. But I have never seen one of these……” [What would help you decide] Perhaps a mock up of one of these.” [What would you do in real life] “I may give up and phone.”

Scenario 3: Application to Learning Fund6. This page confusing. Headings are not clear. Student thought she would have to work through each one in order. “Lot of stuff here – might even do a find on the page”

Scenario 4: Help with multiple problems – things not going right7. Generally, the student would not use the web to find advice regarding personal problems, she would talk to her personal tutor or other member of her academic department. 8. The student thought the term ‘student support and guidance’ might help her with this type of problem but was then disconcerted when faced with the aims and objectives of a University administrative section rather than some guidance. “When I clicked on Student Support and Guidance I thought I would get something about Student Support and Guidance”9. Did eventually notice the ‘Things Not going Right’ link but clicked through to the apparently unrelated page Signposts questionnaire. The student also balked at the volume of text that was presented on the page

- 11 -

Current Student (SSiD) Home Page evaluationhttp://www.shef.ac.uk/ssid/“It’s busy… there are links in the middle, on the left and on the right”“It says follow the links on the left. What about the ones on the right – aren’t they important”“I wouldn’t use FAQs unless I was desperate… I would probably use the contents list”“I like the pictures… they tell me I can go there”

Birmingham University Current Students page evaluationhttp://www.student.bham.ac.uk/“It looks cleaner”“Nice heading on the left…. What’s Happening”“It says welcome and you’ve got smiley people”“Oh it’s got an acronym directory. I come across acronyms all the time here”

Any other Comments“I don’t often go into MUSE. I don’t know what advantages I would get from using MUSE. It seems you can get access to stuff in MUSE and from an alternative. So why use MUSE?”“I tried for a long time to find out when libraries were open. I could find where they were but not when they were open in the same place. I think I phoned up in the end.”“Departmental learning resources and STAR are good.”“In STAR the keyword search is not very good.”“Webmail – I can’t add a name automatically so I just go use hotmail instead.”“I did use the website before I came to Sheffield and it was good to look at all the course information before I decided on coming here.”

Student 2Scenario 1: Examination Timetable details10. “I know this sort of thing is in SSiD. I can’t see an immediate link to SSiD from the Home Page, so I will stick it in the search engine.”

Scenario 2: Changing Address details11. The student knew it was a transcript but was unsure where he could access that information. “That would be a transcript. I’m not sure whether that would be in [Your] record. Perhaps Course information would be a good place to start. I’m not sure whether that would be through MUSE or not.”

Scenario 3: Application to Learning FundThe student progressed smoothly through this task.

Scenario 4: Help with multiple problems – things not going right12. “In that situation I would talk to my personal tutor first. I wouldn’t look on the web immediately. I may look on the web as a result of a discussion with my personal tutor.” “Things not going right…Student Support and Guidance…seems vague. It could encompass pretty much everything they do.” “I’m thinking this is just an introductory page that says we’ve got this questionnaire.”

- 12 -

Current Student (SSiD) Home Page evaluation“Seems to be set out in a pretty standard manner.”“Other pages you’ve been reading recently – that’s pretty useful.”“[Pictures suggest] there is an actual physical presence, not just an online facility.”“Perhaps you could have a box on the right to say what’s new.”

Southampton University Current Students page evaluationhttp://www.soton.ac.uk/NewandCurrentStudents/“With a web page, I’m not overly fussed about the design of a web page, so long as it doesn’t make the site unusable”

Student 3Scenario 1: Examination Timetable details13. Student found SSiD by going to departments.The student progressed smoothly through this task.

Scenario 2: Changing Address details14. Went through MUSE to change address details because he familiar with that route.Exam results in MUSE led to an out of date article: http://www.shef.ac.uk/ssid/news/spring_results.html

Scenario 3: Application to Learning Fund15. Student relied on the search facility which proved to be very ineffective in both sets of results returned.16. Short Term Loan is a paragraph of Financial Support page but does not have a link to the main Short Term Emergency Assistance page. The assumption by the student is that they have discovered all there is to read about Short Term Emergency Assistance.

Scenario 4: Help with multiple problems – things not going right17. “It’s quite good the University has anticipated that something should be put online. You might not want to talk about it face to face to start with. It can sum up your situation and perhaps point you in the right direction.”

Current Student (SSiD) Home Page evaluation“…obviously designed to provide a lot of information”“…relevant information in the centre…”“….might seem perplexing at first…”“….most important and recent things in the middle…”“[pictures] gives you an impression of what SSiD does; people helping at a counter…”

University of Central Lancashire Current Students page evaluationhttp://www.uclan.ac.uk/students/current.htm“….wondering what these girls were doing…” “Oh free laptop.” “Number of links on the left [ToC] seems to be right number” “No search facility on it” “Access to Learning Funds, the number of clicks is easier.”

- 13 -

Student 4Scenario 1: Examination Timetable details18. Although not use FAQ before in SSID, the student said she generally preferred to look at FAQs first. “…otherwise I would have gone to frequently asked questions. But it was in the purple box.” “I tend to look at the FAQ before the A to Z because it is categorised”19. Student confirmed what we know about how people look at web pages i.e. that she tended to look middle, then right then left hand side.

Scenario 2: Changing Address details20. The student perceived that summary of results could be obtained from Your University Record and became a bit disconcerted when she couldn’t find “I’m a bit lost, so I would type it in here [search]”21. After being prompted with the information that the document she was looking for was called a transcript of qualification the student used the search facility but only found a reference on a web page design for staff FAQs i.e. http://www.shef.ac.uk/ssd/faq/assessment.html22. After more attempts to find the page the student said that in real life “I would probably just ring up [SSiD]”

Scenario 3: Application to Learning FundThe student progressed smoothly through this task.

Scenario 4: Help with multiple problems – things not going right23. Student said generally she would not try and find this information on the web.“I would probably go to talk to one of my tutors.” “That’s strange. It’s not clear where the questionnaire is. I would probably find that a bit annoying. I just want a list of information of who I could go and talk to. It’s a bit confusing.”

Current Student (SSiD) Home Page evaluation“I would look at the A to Z before the search because the search doesn’t work particularly well. I tend to look at the FAQ before the A to Z because it is categorised”“The pictures are fine, showing staff / student contact.”“That’s most relevant to the times, at the moment it’s exams so … and that’s probably the thing you look at first…the purple box.” “I probably wouldn’t take much notice of the green box.”“I think that’s a good idea (pages you have been reading) as it’s the most popular pages, so it would be a good idea to look there first.”

Idaho University Current Students page evaluationhttp://www.isu.edu/current.shtml“Looks a little bit more modern – different fonts being used.”“All a bit confusing. Prefer it to be simple on the homepage.”

Any other Comments“I found the Sheffield home page a bit old fashioned when I applied compared to other universities.”“Search doesn’t work very well”

- 14 -

“Before I came I used the accommodation web page.”

- 15 -

Appendix 3: Script for Web Page Usability Test 1st December 2004

Time Facilitator Observer10.00 – 10.50 Scott Castle Gill Bartlett11.00 – 11.50 Gill Bartlett Gill Hughes13.00 – 13.50 Gill Hughes Scott Castle14.00 – 14.50 Scott Castle Gill Hughes15.00 – 15.50 Gill Bartlett Scott Castle

Timings are for guidance only

INTRODUCTION (Start no later than 5 past the hour)Hi, my name is ………………………., and I’m going to be talking you through this web testing session.

This is……… who will be helping me.

Let me explain why we’ve asked you to come here today. We’re testing parts of our web site so we can see what its like for people who actually use it.

It’s important for you to know that we’re testing the site, not you. That means you can’t do anything wrong.

We want to hear exactly what you think, so as we go along, please try to do all your thinking out loud. We want to know exactly what’s going through your mind so you need to tell us. Please don’t worry about hurting our feelings. We do want to improve the site so we need to know honestly what you think.

With your permission, we’re going to record what you have to say. We are not filming you, but we do have equipment in place which will allow us to record where you are clicking with your mouse, and this will be matched with your taped comments. The tape will only be heard by the people working on this project.

We estimate that the exercise will take between 30 and 40 minutes

The Exercise will consist of:

1. A series of general questions about your experience of the web.2. A series of scenarios where we will describe a situation and ask you to find some information on the web.3. An evaluation of 2 specific web pagesDo you have any questions before we begin?

CONTEXTUAL QUESTIONS? (Start no later than 10 past the hour)

1. In general, what do you use the web for?2. What do you use the University of Sheffield Web Site?3. Which pages of the University of Sheffield Web Site have you used before?

- 16 -

TASK 1 - SCENARIO 1: Examinations (Start no later than 15 past the hour)

Can you imagine that you have exams coming up that involve using a calculator. Find the dates of your exams and information about using calculators in exams.

You have checked the dates and times and realise that you are timetabled to take two exams at the same time. A friend has told you that you can report a clash of exams online. Find the Examination clash form.

Remember to tell us what you are thinking.

TASK 2 – SCENARIO 2: Personal Records (Start no later than 20 past the hour)

You are just about to graduate, move out of your term time accommodation and go on holiday. Your want to make sure that the tickets for your graduation ceremony arrive at your new address. Find a way to change your address online. Remember to vocalise all your thoughts.

Now you have found this page explain the difference between a home and correspondence address.

Now try and find how you can get a document that details all your modules and exam results.

TASK 3 - SCENARIO 3: Finance (Start no later than 25 past the hour)Imagine you are a final year student experiencing severe financial difficulties. A friend tells you that the University can provide additional Financial Support through something called “Access to Learning Funds”. Try and find out whether you would be eligible for this.

You complete the application form and are told that it takes 3 weeks to process. You also find out that your next student loan payment is going to be 3 weeks late because the computer system at your home LEA has crashed. Try to find information about an emergency loan that can help you until your funding comes through.

TASK 4 - SCENARIO 4: Help with Problems (Start no later than 25 to the hour)

Imagine that your course has not turned out to be what you expected. You are not enjoying it and have started to miss lectures and tutorials. You feel that you are performing poorly. You wish you had chosen another course and feel increasingly despondent. You suspect that you will fail your exams and are wondering whether you should withdraw from the course. What should you do? Can you try to find guidance on what you should do from this home page.

- 17 -

TASK 5 – CURRENT STUDENTS (SSID) HOME PAGE 1 (Start no later than 20 to the hour)

From www.shef.ac.uk/ssid

First, I’d like you to look at this page and tell me what you think it is, what strikes you about it and what you think you’d click on first.

For now don’t actually click on anything. Just tell me what you would click on.

And again, remember to think out loud as much as possible so we know what you’re thinking about.

TASK 6 - HOME PAGE 2 (Start no later than a quarter to the hour)

Select one page for each student from:http://www.student.bham.ac.uk/http://www.soton.ac.uk/NewandCurrentStudents/http://www.uclan.ac.uk/students/current.htmhttp://www.nottingham.ac.uk/current/http://www.isu.edu/current.shtml

First, I’d like you to look at this page and tell me what you think it is, what strikes you about it and what you think you’d click on first.

For now don’t actually click on anything. Just tell me what you would click on.

And again, remember to think out load as much as possible so we know what you’re thinking about.

- 18 -

Appendix 4: Video Tape Timings

Student 1Scenario 1 Start: 0:03:42Scenario 2 Start: 0:12:09Scenario 3 Start: 0:16:52Scenario 4 Start: 0:23:18Home page Evaluation Start: 0:31:48

Student 2Scenario 1 Start: 0:49:00Scenario 2 Start: 0:53:06Scenario 3 Start: 1:01:01Scenario 4 Start: 1:04:00Home page Evaluation Start: 1:07:39

Student 3Scenario 1 Start: 1:19:10Scenario 2 Start: 1:24:00Scenario 3 Start: 1:32:06Scenario 4 Start: 1:40:00Home page Evaluation Start: 1:45:00

Student 4Scenario 1 Start: 2:06:15Scenario 2 Start: 2:10:31Scenario 3 Start: 2:16:20Scenario 4 Start: 2:19:50Home page Evaluation Start: 2:22:26

- 19 -

Appendix 5: Usability Test 011204 (Clicks)Student 1Scenario 1: Examination Timetable Starting From: http://www.shef.ac.uk/Click Link Name Link Position To Notes

1 Current Students Table of Contents http://www.shef.ac.uk/ssid/2 Exam Timetable main body (featured) http://www.shef.ac.uk/ssid/admin/timetbl.html3 DRAFT timetable, main body (text) http://www.shef.ac.uk/ssid/draftexamtt/all_draft.html 1 & 2

Scenario Progression: Clash Forms Starting From: http://www.shef.ac.uk/ssid/draftexamtt/all_draft.html1 Back Menu Bar http://www.shef.ac.uk/ssid/draftexamtt/all_draft.html2 Examination Clash Form main body (text) http://www.sheffield.ac.uk/ssid/forms/exam.html

Scenario 2: Change of Address Starting From: http://www.shef.ac.uk/1 Current Students Table of Contents http://www.shef.ac.uk/ssid/2 Your University Record Table of Contents http://www.sheffield.ac.uk/ssid/record3 Your University Record main body (title) http://www.sheffield.ac.uk/ssid/record/pers_inf.html4 Changing your Address main body (text) http://www.shef.ac.uk/ssid/record/pin.html 3

Scenario Progression: Results of Exams Starting From: http://www.shef.ac.uk/ssid/record/pin.html1 Your University Record Breadcrumbs http://www.shef.ac.uk/ssid/record 42 Certified Copies of Degree Certificates main body (title) http://www.shef.ac.uk/ssid/record/degree_copies.html 5

Scenario 3: Access to Learning Fund Starting From: http://www.shef.ac.uk/1 Current Students Table of Contents http://www.shef.ac.uk/ssid/2 Financial Information Table of Contents http://www.shef.ac.uk/ssid/finance3 Undergraduate Table of Contents http://www.shef.ac.uk/ssid/finance/financeug.html 64 Financial Support administered by the University main body (title) http://www.shef.ac.uk/ssid/finance/finhlpug.html5 Financial Support for Full Time Courses main body (sub-title) http://www.shef.ac.uk/ssid/finance/ftguide.html

Scenario Progression: Short Term Loan Starting From: http://www.shef.ac.uk/ssid/finance/ftguide.htmlWould not try to find information on web. Would phone SSiD immediatelyScenario 4: Things not going right Starting From: http://www.shef.ac.uk/

1 Current Students Table of Contents http://www.shef.ac.uk/ssid/ 72 A to Z Contents List Table of Contents http://www.shef.ac.uk/ssid/atoz.html3 Student Support and Guidance main body (listed) http://www.shef.ac.uk/ssd/ssg 84 Student Services Information Desk Table of Contents http://www.shef.ac.uk/ssid/5 Student Welfare Table of Contents http://www.shef.ac.uk/ssid/welfare6 Things Not Going Right ? main body (title) http://www.shef.ac.uk/ssid/welfare/signposts/index.html 97 SSiD Home Page Table of Contents http://www.shef.ac.uk/ssid/8 Department Menu Bar http://www.shef.ac.uk/departments/9 Archeology main body (listed) http://www.shef.ac.uk/archaeology/10 Current Students Table of Contents http://www.shef.ac.uk/archaeology/current

Student would then talk to their department / personal tutor

- 20 -

Student 2: Scenario 1: Examination Timetable Starting From: http://www.shef.ac.uk/Click Link Name Link Position To Notes

1 Search for SSiD left search box http://www.shef.ac.uk/ssid/ 102 Timetable (DRAFT) right (information) http://www.shef.ac.uk/ssid/admin/timetbl.html3 DRAFT timetable, main body (text) http://www.shef.ac.uk/ssid/draftexamtt/all_draft.html

Scenario Progression: Calculator Starting From: http://www.shef.ac.uk/ssid/draftexamtt/all_draft.html1 SSiD Breadcrumbs http://www.shef.ac.uk/ssid/2 Examinations See Also box http://www.sheffield.ac.uk/ssid/admin/exam.html3 Calculator Approval main body (sub-title) http://www.sheffield.ac.uk/ssid/admin/calculator/index.html

Scenario Progression: Clash Form Starting From: http://www.sheffield.ac.uk/ssid/admin/calculator/index.html1 Back key stroke http://www.sheffield.ac.uk/ssid/admin/exam.html

2On-line Examination Clash Reporting Form main body (sub-title) http://www.shef.ac.uk/ssid/forms/exam.html

Scenario 2: Change of Address Starting From: http://www.shef.ac.uk/ssid/1 Your University Record Table of Contents http://www.sheffield.ac.uk/ssid/record2 Changing your Address main body (text) http://www.shef.ac.uk/ssid/record/pin.html

Scenario Progression: Results of Exams Starting From: http://www.shef.ac.uk/ssid/record/pin.html1 Course Information Table of Contents http://www.shef.ac.uk/ssid/admin/course.html2 Your University Record Table of Contents http://www.sheffield.ac.uk/ssid/record 113 Your University Record main body (title) http://www.sheffield.ac.uk/ssid/record/pers_inf.html

4Search for Exam Results in Your Record No results

5Search for Exam Results in whole web site No results

6 Back 2 times key stroke7 Check Your Record Table of Contents http://www.shef.ac.uk/ssid/record/pers_inf.html

Scenario 3: Access to Learning Fund Starting From: http://www.shef.ac.uk/ssid/record/pers_inf.html1 Student Welfare Table of Contents http://www.shef.ac.uk/ssid/welfare2 Financial Help Available main body (title) http://www.shef.ac.uk/ssid/finance/finhlp.html

3Financial Support for Full Time Courses main body (sub-title) http://www.shef.ac.uk/ssid/finance/ftguide.html

Scenario Progression: Short Term Loan Starting From: http://www.shef.ac.uk/ssid/finance/ftguide.html1 Back key stroke http://www.shef.ac.uk/ssid/finance/finhlp.html2 Short-term Emergency Assistance main body (title) http://www.shef.ac.uk/ssid/finance/shortterm.html

Scenario 4: Things not going right Starting From: http://www.shef.ac.uk/ssid/finance/shortterm.html1 Student Welfare Table of Contents http://www.shef.ac.uk/ssid/welfare 122 Things Not Going Right ? main body (title) http://www.shef.ac.uk/ssid/welfare/signposts/index.html3 Questionaire main body (link) http://www.shef.ac.uk/ssid/signposts.html

- 21 -

Student 3Scenario 1: Examination Timetable Starting From: http://www.shef.ac.uk/Click Link Name Link Position To Notes

1 Department Menu Bar http://www.shef.ac.uk/departments/ 132 Student Services Information Desk main body (listed) http://www.shef.ac.uk/ssid/3 Exam Timetable main body (featured) http://www.shef.ac.uk/ssid/admin/timetbl.html

DRAFT timetable, main body (text) http://www.shef.ac.uk/ssid/draftexamtt/all_draft.htmlScenario Progression: Calculator Starting From: http://www.shef.ac.uk/ssid/draftexamtt/all_draft.html

1 SSiD Table of Contents http://www.shef.ac.uk/ssid/2 Frequently Asked Questions main body (featured) http://www.shef.ac.uk/ssid/faq3 Examination main body (listed) http://www.shef.ac.uk/ssid/faq/examinations.html4 Calculator Approval main body (link) http://www.sheffield.ac.uk/ssid/admin/calculator/index.html

Scenario Progression: Clash Form Starting From: http://www.sheffield.ac.uk/ssid/admin/calculator/index.html1 Back 5 times Menu Bar http://www.sheffield.ac.uk/ssid/admin/exam.html2 On-line Examination Clash Reporting Form main body (sub-title) http://www.shef.ac.uk/ssid/forms/exam.html

Scenario 2: Change of Address Starting From: http://www.shef.ac.uk/ssid/forms/exam.html1 MUSE login 142 application Table of Contents3 Check / Amend Details main body (title)

Scenario Progression: Results of Exams Starting From: MUSE1 Back Menu Bar http://www.shef.ac.uk/2 Student Services right (Quick Links) http://www.shef.ac.uk/ssd/3 Registry Services Table of Contents http://www.shef.ac.uk/ssd/reg/4 Registration Office main body (title) http://www.shef.ac.uk/ssd/registration5 MUSE login6 Exam Results main body (title) http://www.shef.ac.uk/ssid/news/spring_results.html/

Scenario 3: Access to Learning Fund Starting From: http://www.shef.ac.uk/1 Search for access to learning left search box2 Search for access to learning fund left search box3 Department Menu Bar http://www.shef.ac.uk/departments/4 Student Services Information Desk main body (listed) http://www.shef.ac.uk/ssid/5 Financial Information Table of Contents http://www.shef.ac.uk/ssid/finance6 Funding for Undergraduate Studies main body (title) http://www.shef.ac.uk/ssid/finance/financeug.html7 Financial Support administered by the Uni main body (title) http://www.shef.ac.uk/ssid/finance/finhlpug.html8 Financial Support for Full Time Courses main body (sub-title) http://www.shef.ac.uk/ssid/finance/ftguide.html

Scenario Progression: Short Term Loan Starting From: http://www.shef.ac.uk/ssid/finance/ftguide.htmlScrolled down to relevant part of page. Short notes but no link 16Scenario 4: Things not going right Starting From: http://www.shef.ac.uk/

1 Current Students Table of Contents http://www.shef.ac.uk/ssid/

- 22 -

2 SSiD Services main body (featured) http://www.sheffield.ac.uk/ssid/services.html3 Student Welfare Information main body (listed) http://www.shef.ac.uk/ssid/welfare4 Things Not Going Right ? main body (title) http://www.shef.ac.uk/ssid/welfare/signposts/index.html 17

- 23 -

Student 4Scenario 1: Examination Timetable Starting From: http://www.shef.ac.uk/Click Link Name Link Position To Notes

1 Current Students Table of Contents http://www.shef.ac.uk/ssid/ 18 & 192 Calculator Approval right (information) http://www.sheffield.ac.uk/ssid/admin/calculator/index.html3 Approved models of calculators main body (link) http://www.sheffield.ac.uk/ssid/admin/calculator/approved.html

Scenario Progression: Examination Timetable Starting From: http://www.sheffield.ac.uk/ssid/admin/calculator/index.html1 Back 2 times http://www.shef.ac.uk/ssid/2 Exam Timetable main body (featured) http://www.shef.ac.uk/ssid/admin/timetbl.html3 DRAFT timetable, main body (text) http://www.shef.ac.uk/ssid/draftexamtt/all_draft.html

Scenario Progression: Clash Forms Starting From: http://www.shef.ac.uk/ssid/draftexamtt/all_draft.html1 SSiD Home Page Table of Contents http://www.shef.ac.uk/ssid/draftexamtt/all_draft.html2 Frequently Asked Questions main body (featured) http://www.sheffield.ac.uk/ssid/faq3 Examinations main body (listed) http://www.sheffield.ac.uk/ssid/faq/examinations.html2 Examination Clash Reporting Form main body (link) http://www.sheffield.ac.uk/ssid/forms/exam.html

Scenario 2: Change of Address Starting From: http://www.shef.ac.uk/ssid/1 Your University Record Table of Contents http://www.sheffield.ac.uk/ssid/record3 Changing your Address Table of Contents http://www.shef.ac.uk/ssid/record/pin.html

Scenario Progression: Results of Exams Starting From: http://www.shef.ac.uk/ssid/record/pin.html1 Your University Record Table of Contents http://www.shef.ac.uk/ssid/record 202 Check Your Record Table of Contents http://www.sheffield.ac.uk/ssid/record/pers_inf.html3 Course Information Table of Contents http://www.sheffield.ac.uk/ssid/admin/course.html

4Search for transcription of qualification (site) left search box http://www.shef.ac.uk/ssd/faq/assessment.html 21 & 22

Scenario 3: Access to Learning Fund Starting From: http://www.shef.ac.uk/ssid/1 Financial Information Table of Contents http://www.shef.ac.uk/ssid/finance2 Undergraduate main body (title) http://www.shef.ac.uk/ssid/finance/financeug.html

3Financial Support administered by the DfES main body (title)

4 Back5 Financial Support administered by the Uni main body (title) http://www.shef.ac.uk/ssid/finance/finhlpug.html4 Financial Support for Full Time Courses main body (sub-title) http://www.shef.ac.uk/ssid/finance/ftguide.html

Scenario Progression: Short Term Loan Starting From: http://www.shef.ac.uk/ssid/finance/ftguide.html1 Back Menu Bar http://www.shef.ac.uk/ssid/finance/finhlpug.html2 Short-term Emergency Assistance main body (title) http://www.shef.ac.uk/ssid/finance/shortterm.html

Scenario 4: Things not going right Starting From: http://www.shef.ac.uk/ssid/1 Student Welfare Table of Contents http://www.shef.ac.uk/ssid/welfare 232 Things Not Going Right ? main body (title) http://www.shef.ac.uk/ssid/welfare/signposts/index.html3 Signpost Questionaire main body (link) http://www.shef.ac.uk/ssid/signposts.html

- 24 -