imprint pro usability assessment pratik jha, julie naga and dr. raja parasuraman george mason...
TRANSCRIPT
IMPRINT Pro Usability Assessment
Pratik Jha, Julie Naga and Dr. Raja Parasuraman
George Mason University
2
Introduction
A “professional” (Pro) version of IMPRINT is currently under development
+ More Modular
+ Added Functionality
+ Greater Flexibility
This has also resulted in series of User Interface changes
+ Multiple windows to simultaneously display information
+ Tree structure for navigation
+ Pallete to show common used functionality
3
Project Goals
To Uncover Usability Issues of IMPRINT Pro– Does the new interface allow users to conduct their analyses as
effectively as with older versions of IMPRINT?
– Is the added functionality and flexibility understandable and usable?
– Is the new system easy to learn for a user trained on earlier versions?
– Does the IMPRINT Pro interface allow users to navigate through the space of modeling options in an effective and timely manner?
4
Method
2 step approach– Heuristic evaluation (first cut)
– Usability walkthrough (confirmatory)
5
Heuristic Evaluation Methodology
Familiarization phase– Team read product manuals and other background material– Conducted high level task analysis – Gathered information on user characteristics– Attended IMPRINT workshop and tutorial
Review Phase– 3 reviewers conducted independent usability evaluation – 7 well established heuristics were used for evaluation– Several group meetings were held to discuss reviews and
consolidate evaluation
6
Heuristics
Ease of Navigation
Look and Feel
Providing functionality consistent with
actions, terms and situation
Maximize RecognitionEnabling User Options
Preventing User Error
Heuristics
User Feedback
7
IMPRINT Pro Areas Evaluated1. Overall look and
feel (windows, menu structure, legibility of text and icons etc.)
2. Analysis tree3. Navigational tab4. Properties window5. Adding a mission
from library6. Adding warfighter7. Adding mission8. Adding equipment
8
Tasks
1. Add new analysisa. Add name of analysisb. Add brief description and
purpose of the analysis
2. Add a warfightera. Add nameb. Select MOSc. Select workload strategy
3. Add a mission3. Add mission name and
description4. Add task network (functions
and task)5. Add task parameters or
change properties
4. Add a equipmenta. Add equipment name and
descriptionb. Add or change parameters
5. Save an analysis6. Add analysis from the
library
9
Results
37 usability issues were uncovered from heuristic analysis
Results suggest some major usability problems in IMPRINT PRO software that could lead to
– Poor user experience– Increased training time and poor learnability– Negative training transfer effect because the
software is inconsistent with the previous versions
Majority of the usability issues are severe or moderate in nature
10
Results – Usability Issues by Severity
Usability issue by severity
3
21
13
0
5
10
15
20
25
Low Moderate Severe
No
. o
f id
enti
fied
pro
ble
ms
Severity
11
Results – Usability Issues by Heuristic Criteria
Usability Issues by Heuristic Criteria
11
65
32
3
10
0
2
4
6
8
10
12
Nu
mb
er
of
Usab
ilty
Issu
es
Un
co
vere
d
12
Results Ease of Navigation
– Software does not lead the user to the next step
– User options are not visible– Number of user clicks are
not optimized– Analysis tree and
navigational bread crump gets cluttered fairly soon
Look and Feel– Display clutter due to
multiple windows– Poor contrast between
background and text
13
Results
Functionality– Pane seems to be of limited use and adds to display clutter– Windows functionality in the view menu is not enabled– Parameters cannot be changed in the properties window
Enabling user options– A new version is added instead of new analysis – Cant drag and drop missions from other analysis
14
Results
Maximize Recognition– Excessive use of
Acronyms– MOS categories and
workload strategies are not defined
Preventing User Error– Incorrect user input sort
(for example user is asked to input probability in percentage)
15
Results
Providing User Feedback– Navigational tab and analysis
tree are not synchronized
– No feedback provided to the user if the analysis is saved or not
– Property windows are not labeled
16
Usability Walkthrough
We conducted a usability walkthrough to validate results of heuristic analysis
Methodology– Tested 4 participants on Pro (2 experts and 2 novice)– Participants performed a series of task– Participants were asked to think aloud– Session was video taped for future analysis
17
Task
1. Add new analysis – Name it NCops– Add brief description about the
purpose of the analysis for reference
2. Add new Warfighter– Name the warfighter as soldier 1– Select appropriate MOS
category– Select appropriate workload
management strategy
3. Add new mission– Name the mission Network
Centric Warfighter– Add a brief description
4. Add task network– Add function and name it Gather
Information– Add a task to above function and
name it Situation Monitoring (task 1)
– Add another task to above function and name it Scan Head Mounted Display (task 2)
– Change the priority of task 1 to 1 (Priority can be changed from the Effects screen of task properties)
– Change the priority of task 2 to 2– Review the task properties in
snapshot view
5. Save your analysis and exit
18
Data CollectedBackground questionnaire
– IMPRINT usage– Experience– Others
Subjective ratings – Overall reaction to the application– Screen – Terminology and system
operations– Learning– System
Think aloud data
Comments
19
Results – Overall Reaction to Application
Overall Reaction
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
Terrible toWonderful
Difficult to Easy Frustrating toSatisfying
Inadequate toAdequate Power
Dull to Stimulating Rigid to Flexible
Rati
ng
s
20
Results
Terminology and System Information
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
Use of Termsthroughout
System
ApplicationTerminology is Related to theTask you are
doing
Position ofmessages on
screen
Messages onscreen which prompt you for
input
System keeps yoinfomred
about what it isdoing
Se
lf R
ate
d A
ve
rag
es
Screen
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
Characters on computerscreen
Organization of information Sequence of Screens
Self
Rate
d Av
erag
es
Learning Feedback Averages
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
Learning to operatethe system
Exploring new featuresby trial and error
Remembering namesand use of commands
Tasks can beperformed in a
straight-forwardmanner
Se
lf R
ate
d A
ve
rag
es
System Feedback
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
The layout andpresentation of Information was
clear
It was easy to movearound
different parts of thesystem
The system allowedsufficient flexibility to work in the way
you wanted
The system washelpful in coping
with any errors thatwere made
The system wasresponsive to your
inputs
Se
lf R
ate
d A
ve
rag
es
21
Results – User Comments
Feedback" It is very different than 7 and will require old IMPRINT
users to have new training because it is vastly different“
“ There is no feedback on saving an analysis."
22
Recommendations
1. A detailed task analysis should be conducted to improve the navigation and refine the functionality of IMPRINT Pro
2. An analysis should be performed to investigate if the radically different interface of IMPRINT Pro from the previous versions could cause a negative transfer training effect
3. Functionality aimed at minimizing training time or to provide users help in conducting the analysis should be explored. For example, the efficacy of “wizards” that help users in building an analysis should be evaluated.
23