database web application user test 2
TRANSCRIPT
USABILITY TESTING 2Database Web Application
Test Information
OVERVIEWTechSmith Morae was used on a laptop computer to conduct usability testing of the recently revised WVU Libraries Database
web application, using test questions from the first round of testing that were still relevant to the web application. This round of
usability testing was internal and focused on Undergraduate Student employees.
Test Information
TARGET AUDIENCE73%* of WVU Libraries users are undergraduate students who are 48% female or 52% male, are primarily Caucasian,
African American, Hispanic, or Asian in ethnicity, are United States citizens or are international students, are 21 years in their
average age, and are 49% from the state of West Virginia or 51% from other United States, territories, or countries.
Target audience numbers are populated from both survey polls and server statistics.*
Test Information
DATABASE USE96%* of the WVU Libraries Database web application usage is from non-Downtown Campus Library WVU employee
computer users ‒ 64% of which are returning users, and 61% of which are from ages 18 to 34 ‒ on and off Morgantown
campus, whereas 4% of Database web application usage is from WVU employee at the Downtown Campus Library. The
Geographic Breakdown of these 96% of users is as follows: 99% from the United States; 86% from the state of West
Virginia; 77% from Morgantown, West Virginia; 1% outside the United States.
Statistics are calculated by Google Analytics.*
Test Information
PARTICIPANTS
Ajay Hannah Kyle Robert Samantha Taija
https://testing.lib.wvu.edu/usability5/AJ.wmv
https://testing.lib.wvu.edu/usability5/Hannah.wmv
https://testing.lib.wvu.edu/usability5/Kyle.wmv
https://testing.lib.wvu.edu/usability5/Robert.wmv
https://testing.lib.wvu.edu/usability5/Samantha.wmv
https://testing.lib.wvu.edu/usability5/Taija.wmv
TASK ANALYSISTest 1 & Test 2
Test 1 & Test 2
TASKS1. Go to WVU Libraries’ Databases.
2. Find the database PubMed.
3. What is the earliest coverage in the database SportDiscus?
4. What is a database for the subject Marketing, that is also an
encyclopedia?
5. Your assignment is to listen to an aria from an opera. What
database will you use?
6. How many databases cover the subject of psychology?
7. Find a subject guide for engineering.
8. Go to the tutorial for the database Ad$pender.
9. What are the best databases for finding sources in biology?
10. What is the starting year of coverage for the database Web of
Science?
11. How can you get help with database selection or searching?
12. You need a brief video clip on chemistry lab safety. Where
would you find it?
13. Find 2 databases that contain statistics for market research.
14. What is a database for Theater that is contains Images?
15. How many primary sources can you find for West Virginia
databases?
■ - Omitted Tasks in Test 2
Test 1 Results:All Tasks
AVERAGE TIME* Average time includes delivery of tasks.
Test 2 Results:All Tasks
AVERAGE TIME* Average time includes delivery of tasks.
Test 1 & 2 Comparison:All Tasks
AVERAGE TIME* Average time includes delivery of tasks.
0.74
1.17
1.07
0.46
0.66
0.99
0.84
1.72
0.95
0.6
0.84
0.61
0.41
1.33
0.87
0.88
0.98
0.64
1.43
1.46
1.39
1.03
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8
Task 1
Task 2
Task 3
Task 4
Task 5
Task 6
Task 7
Task 8
Task 9
Task 10
Task 11
Average Time (Minutes)
Task
Test 2 Test 1
Test 1 Results:All Tasks
SUCCESSRATES* Percentages based on a 0 to 3 scoring system.
Test 2 Results:All Tasks
SUCCESSRATES* Percentages based on a 0 to 3 scoring system.
Test 1 & 2 Comparison:All Tasks
SUCCESSRATES* Percentages based on a 0 to 3 scoring system.
Test 1 Results:All Tasks
ERRORRATE* Error rate is a standard deviation based direct path to information as it is known to test administrator, and the rate of the user encountering a functionality or system error.
Test 2 Results:All Tasks
ERRORRATE* Error rate is a standard deviation based direct path to information as it is known to test administrator, and the rate of the user encountering a functionality or system error.
Test 1 & 2 Comparison:All Tasks
ERRORRATE* Error rate is a standard deviation based direct path to information as it is known to test administrator, and the rate of the user encountering a functionality or system error.
■ - Test 1 ■ - Test 2
SYSTEM USABILITY SCALETest 1 & Test 2
Test Information
*(SUS) QUESTIONS1. I think that I would like to use this system frequently
2. I found the system unnecessarily complex
3. I thought that the system was easy to use
4. I think that I would need the support of a technical person to
be able to use this system
5. I found the various functions in this system were well
integrated
6. I thought there was too much inconsistency in this system
7. I would imagine that most people would learn to use this
system very quickly
8. I found the system very cumbersome to use
9. I felt very confident using the system
10. I needed to learn a lot of things before I could get going with
this system
* System Usability Scale (SUS)
Test 1 Results:All Tasks
SUS SURVEY* Results are based on survey results taken after usability testing, and are concerned with the Database web application interface and functionality.
30%
70%
90%
50%
10%
20%
70%
30%
10%
50%
50%
40%
30%
20%
30%
30%
50%
70%
30%
60%
30%
50%
30%
40%
10%
I think that I would like to use this system frequently
I found the system unnecessarily complex
I thought that the system was easy to use
I think that I would need the support of a technical person to be able to use thissystem
I found the various functions in this system were well integrated
I thought there was too much inconsistency in this system
I would imagine that most people would learn to use this system very quickly
I found the system very cumbersome to use
I felt very confident using the system
I needed to learn a lot of things before I could get going with this system
Strongly Disagree Disagree N/A Agree Strongly Agree
Test 2 Results:All Tasks
SUS SURVEY* Results are based on survey results taken after usability testing, and are concerned with the Database web application interface and functionality.
49.98%
49.98%
49.98%
66.64%
33.32%
49.98%
49.98%
33.32%
33.32%
16.66%
33.32%
66.64%
33.32%
33.32%
16.66%
16.66%
16.66%
33.32%
66.64%
33.32%
66.64%
33.32%
16.66%
33.32%
33.32%
33.32%
I think that I would like to use this system frequently
I found the system unnecessarily complex
I thought that the system was easy to use
I think that I would need the support of a technical person to be able touse this system
I found the various functions in this system were well integrated
I thought there was too much inconsistency in this system
I would imagine that most people would learn to use this system veryquickly
I found the system very cumbersome to use
I felt very confident using the system
I needed to learn a lot of things before I could get going with thissystem
Strongly Disagree Disagree N/A Agree Strongly Agree
SUMMARYNotes & Observations
TASK 1WOuld type lib.wvu.eduWOuld click on Databse tabsWent to websiteWent to the library databases website, clicked on tab, then quick linkWent to google first to search for 'WVU Libraries' clicked on submeta info in googleDoesn't know how to get to databses.didnt click on databses quick lik - didnt scrolld own
TASK 2WOuld go by titlewent to databses by
TASK 3Would use Met Opera on Demand, Or Opera in VideoWould searc h in searchbox for opera, Met Opera on DemandWould say theatre...WOuld just use concert hallWould go to Subject/MusicWould click on viudeo and soundwould add title to search boxWent to Video and soundWent to video and soundWent to theatre in videoUser didn't know music wasn't a daabseTHought search was general search instead of just title
searchTHinks that Video & Sound is a databseStudent thought subjects was a datbaseOpera on demandMet Opera on demandDatabases from search results showed no sugegstions -errorChoose trail database because star
TASK 4Would Search - need to change to Databases by titleWould go click on title PWanting to search for subject in search box - Found 1Not seeing number for results - make largerMusic Dattbase didn't loadLeft justify results , and list at boottom?Isn't using subjects or scrolling dowqnIsn't seeign results immeadetlyIS using search box to search for subject - found 7Didn't notice databases by subject524ish
TASK 5Would click on EWas cobfused and searcheduser doesn't know what subject giuide is, went out to search onlieStudent sees two 'Engineering' databses, si confused
search box should possibly search annotations & titles & subjectsPossibly reword to include just 1 engineering subject, have subject guides as a navigation item, or show up search resultsis searching for subject guideDoesn't know what a subject guide isDeosnt't know what Subject Guide is./ Didnt see first time around
TASK 6We should state 'Database Title Search' that it only searches databse titlesUSer doesn;t know whicjh are best - agricola/academic search compelteStart HEre is sugegstedPossibly reword Start Here to signfy best databasesNot using yellow databases.mising yellow box as imporatnace or suggestionLisated at topI would just search for oneI really dont kno a good journal... normally journals would have imapct scoresDidn't associate yellow databses with best databases.
Live Notes
1/2
TASK 7Would click on Ask A librarianWould just clickon Ask A Librarianscrolle ddown to help with databses - would callI would call to talk to a librarian or talk to someone in personDidnt knwo there was text measssaging optionCOntacting someone at bottom or Ask A LibrarianCLick Aska librarianAsk A real person'Oh, I didn't know there was a text messaging.'
TASK 8Would use seach box as a keyword sechwould use films on demandWould search through individual Databases under chemistryWould go to YouTubeWOuld go to Chemistry/Would go to Subject GUide/Would go to chemistry or video/soundWould c lik on first one to searchWent to video and soundWent to subject guide but nothing about videos was there. Was this a good question?Went to Chemistry,m but is reading through for videosWent to chemistryUser would search in search boxSearch Box results dont suggest databasespossibly change video/sound from subject o mediaNo suggested databses for search results
Need to show video resources or medua resourccesDont see anythign that says videoscan't find chemistry video; searched in search boxCan get toi chemistry, but doesn't seem to know how to get specific media
TASK 9Would use ProQuest Statistical Insight and Richard K MilelrWould go to marketingWent to marketingUser is not using in-page searchingStatista & iLibrary StatisticsStatistaReading through finding research or statisticsRead to find statistics, OECD and StatistaQuestion for eResources-- should there be a Statistics subject area?Keeps trying to search in boxIS rreading through for media typesIs reading through Marketing Subject - Ibis WOrld, and PAssportIndividually reading through definitions of Marketing databasesChose EMIS MAket research and IBS World
TASK 10Would search and go to Theatre in Video
Would just use google for searchingWould just try to click one that says video to see whats in itWould choose encylcopedia of cltohing and fashonWent to threatre, searching for media typesSearching subject in search boxSearched in page of theatere for imagesScroleld to theatreJStor includes high-quqality mages (Student was readinthrough)Is there an advanced search here?IDKFOund some magazine ones... would choose thosecant find medium types
TASK 11Would think newspapers are primary sourcesWent to WV, found 2User is reading through for 'Primary SOurces'THought the titles under subject were the databse titlessearched for west virginia instead of using link; search box should search subjects as well as titles?not going to subjects/ WVI would search west virginaiFOund 2Didn't knwo what rpimary sources meant . said 11 resultsCLicked West VIrignia411 results
Live Notes
2/2
Average Time- 54% of task completion required more time in Test 2 when compared to
Test 1- 36% of task completion required less time in Test 2 when compared to
Test 1- 10% of task completion required the same amount of time in Test 2
when compared to Test 1
Success Rate- Success rates for both 'Completed with ease' and 'Completed' were
largely reduced in Test 2 when compared to Test 1- Success rates for both 'Completed with difficulty' and 'Failed to
complete' largely increased in Test 2 when compared to Test 1
Error Rate/Standard Deviation- The standard deviation of error rates have majorly increased in Test 2
when compared to Test 1
SUSThe following 70% of SUS responses decreased in favorable response:
- I needed to learn a lot of things before I could get going with this system
- I found the system very cumbersome to use- I thought there was too much inconsistency in this system- I found the various functions in this system were well integrated- I thought that the system was easy to use- I think that I would like to use this system frequently
Data Observation
TRENDS
ENHANCEMENTSSuggested
Data from the second round of usability testing indicates a False-Consensus Effect (from client to user) - which can be seen in the Average Time, Success Rate, and Error Rate metric comparison - and is illustrated on page 24 of this report.
The majority of enhancements to content and user interface are maintenance-specific only; however, potential major impact areas reside with the development of the search box, increasing its functionality, and through conducting further user testing.
Both search box development and customer experience, or CX-based user testing should provide insight/dramatically reduce the percentage of failure rates, and decrease the amount of time for task completion.
Content- Change ‘Database Search’ to ‘Database Title-Only Search’ or
‘Database Title Search’- Database descriptions rewritten/reformatted and simplified to be more
readable/scan-able as users scroll- Add ‘Image’ type to ‘Database Subjects’, or revise ‘Database Subjects’
to exclude media types (see *)- Change verbiage from popular databases from ‘Start Here’ to
something more assistive/top-picks orientated
Development- Database search box searches database description and subject- Fix error on search box ‘no results’ to return popular databases.- Integrate chat box into the desktop user interface- Integrate a subject guide list- Modal information for primary resources, subject guides, and types
(see *)
Usability Testing- Shift usability testing to be more formative and away from summative- Conduct future usability testing focusing only on target audience- Conduct future usability testing focusing on qualitative results and user
desirability
User Interface- Prioritize ‘Database by Subject’ section over ‘Trial Databases’ section- Make item or media types stand out in Database description, or add
media keywords/tags to the bottom of Database description *
Other- Determine both the stakeholder’s and the user’s goals/priorities for
project
ENHANCEMENTS