user-centered research on the paying for college website and tools - edui 2014
DESCRIPTION
The Paying for College website is designed to help consumers make informed decisions about college finance. The Consumer Financial Protection Bureau (CFPB) began development with a user-centered design process for this tool-set, which is now in its 4th iteration. The college cost and financial aid comparison tool, a central feature of these resources, supports efforts by the Department of Education to standardize financial aid disclosures. During this session, we’ll cover the most recent rounds of usability testing conducted with multiple groups across the U.S. We’ll highlight difficulties when designing and testing for multiple audiences with different needs as well as testing and iterating with live and prototype versions of the site. Data will also be emphasized as we share collection methodologies (click paths, eye tracking, questionnaires, etc.) and the importance of each. We’ll also provide insights into planning, execution, and reporting and how these findings informed major changes on the website.TRANSCRIPT
Paying For Education: User-Centered Research on the Paying For College Website and Tools September 30, 2014 Jennifer Romano Bergstrom & Lamar Alayoubi @romanocog
Eye Tracking in UX – Oct 16
2
User Focus – Oct 17 eyetrackingux2014.com October 16, 2014 EDUI = 20% OFF
2014.userfocus.org October 17, 2014 EDUI = 20% OFF
Usability = “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.” ISO 9241-11
+ emotions and perceptions = UX
@romanocog #eduiconf
4 User Experience Design (P. Morville): http://semanticstudios.com/publications/semantics/000029.php
User Experience
@romanocog #eduiconf
5
When to test
@romanocog #eduiconf
6
What we design for…and reality
Krug, S. (2000) Don’t Make Me Think. Pearson Education *NEW 3rd Edition (2014)*
@romanocog #eduiconf @skrug
7
OBSERVATIONAL + Time on page/task
+ Reaction time + Selection/click behavior
+ Success/fail rate + Conversion rate
IMPLICIT + Eye tracking
+ Electrodermal activity (EDA) + Behavioral analysis
+ Verbalization analysis + Pupil dilation
+ Facial expression analysis
SUBJECTIVE + Satisfaction & difficulty ratings
+ Verbal responses + Moderator follow up
+ Real-time +/- dial
UX Data @romanocog #eduiconf
Traditional UX research is good at explaining what people say and do, not what they think and feel.
Why should we measure implicit?
@romanocog #eduiconf
9 User Experience Design (P. Morville): http://semanticstudios.com/publications/semantics/000029.php
@romanocog @richmondux
Implicit Data
@romanocog #eduiconf
10
Modern eye tracking @romanocog @dcaapor
Eye Tracking in the Past
11
Modern eye tracking
Modern Eye Tracking
UX Research on Paying For College
12
Tested the User Experience (UX) of: • Overall navigation • Comparing the cost of colleges side-by-side • Organization of information:
• Federal vs. private loan information • Federal loan interest rates • Financial aid disbursement
information • College bank accounts.
Task Creation
SATQ Items
Other materials
Debriefing
Reports
.
. .
.
. .
.
Iterative Process with Client
@romanocog #eduiconf
“A spectator sport” @skug
Testing Iterations & Rounds
13
High School Students
Influencers
Round 1: Austin, TX
Round 3: Arlington, VA
Round 2: Spokane, WA
Live site version 1 Live site version 1 Live site version 1 OR Prototype site version 2
Iteration 1:
February/ March 2014
High School Students
College Graduates
Round 1: Atlanta, GA
Round 3: Arlington, VA
Round 2: Milwaukee, WI
Live site version 2 Live site version 2 Live site version 2
Iteration 2:
April 2014
@romanocog #eduiconf
14
Methodology @romanocog #eduiconf
Testing Environment
Eye Tracker
Moderator & Participant
Methods: UX Metrics
15
OBSERVATIONAL + Time to complete tasks
+ Time to first click + First click accuracy
+ Task accuracy
IMPLICIT + Eye tracking
SUBJECTIVE + Difficulty and satisfaction ratings
+ Think aloud protocol + Debriefing interview
• Eye tracking [implicit] tells us what participants focus on. • Think aloud & debriefing [subjective] tell us why they focus there. • Observational measures tell us how they go about completing a task. • Combination of subjective, observational, & implicit metrics paints the whole picture.
Use eye tracking to: • Gain insights into attracting visual elements and the order that users observe • Identify the amount of text that respondents read (e.g., terms/instructions) • Classify how respondents complete tasks and the order in which they process elements
@romanocog #eduiconf
UX Metrics in Action: Live Website Testing
16
@romanocog #eduiconf
Navigation should be intuitive.
TASK: Let’s say that you have your list narrowed down to the two schools that you have financial aid offers from. You are interested in graduation rates of those two schools. Which university graduates a higher percentage of students who enroll?
17
First clicks: Compare financial aid offers (14),
Compare costs (3), search function (1)
17
1
Successful
Iteration 2 Measure: First-click accuracy
Unsuccessful
6 6 Successful
Unsuccessful
First clicks: Compare costs (4),
Research schools (3), Compare get started (2),
Search function (1), Home (1), Graduate (1)
Iteration 1 Measure: First-click accuracy
50% Accuracy:
94% Accuracy:
@romanocog #eduiconf
Navigation should be intuitive.
TASK: Let’s say that you have your list narrowed down to the two schools that you have financial aid offers from. You are interested in graduation rates of those two schools. Which university graduates a higher percentage of students who enroll?
18
First clicks: Compare financial aid offers (14),
Compare costs (3), search function (1)
17
1
Successful
Iteration 2 Measure: First-click accuracy
Unsuccessful
6 6 Successful
Unsuccessful
First clicks: Compare costs (4),
Research schools (3), Compare get started (2),
Search function (1), Home (1), Graduate (1)
Iteration 1 Measure: First-click accuracy
50% Accuracy:
94% Accuracy:
UX Best Practices & Recommendations:
• Ensure that the primary navigation and other navigation options are intuitive and match user expectations.
UX Metrics:
• First Click Accuracy: If people make the correct first click, the design is intuitive (for that particular task).
@romanocog #eduiconf
13
3
Successful
Unsuccessful
First clicks: Student loans (7),
More about student loans (5), Student financial guides (1),
Get Assistance – Info for students (1), Participate – Student loans (1), Compare financial aid offers (1)
11
1 Unsuccessful
Successful
First clicks: Choose a loan - timeline (8),
Choose a loan – Get started (3), Home (1)
Iteration 1 Measure: First-click accuracy. Task: “What are the types of loans available?”
Iteration 2 Measure: First-click accuracy. Task: “What are the types of loans available?”
94% Accuracy:
81% Accuracy:
Navigation should be intuitive and consistent. @romanocog #eduiconf
TASK: Let’s say you have decided on the university you want to attend and you know you’re going to need to take out a loan to pay for tuition. What are the different types of loans that are available to you?
13
3
Successful
Unsuccessful
First clicks: Student loans (7),
More about student loans (5), Student financial guides (1),
Get Assistance – Info for students (1), Participate – Student loans (1), Compare financial aid offers (1)
11
1 Unsuccessful
Successful
First clicks: Choose a loan - timeline (8),
Choose a loan – Get started (3), Home (1)
Iteration 1 Measure: First-click accuracy. Task: “What are the types of loans available?”
Iteration 2 Measure: First-click accuracy. Task: “What are the types of loans available?”
94% Accuracy:
81% Accuracy:
Navigation should be intuitive and consistent.
UX Best Practices & Recommendations:
• Ensure that navigation options are consistent; for a horizontal navigational layout, either all headings should have dropdown options, or none should.
• Use category headings that accurately convey what users will find on those pages.
@romanocog #eduiconf
TASK: Let’s say you have decided on the university you want to attend and you know you’re going to need to take out a loan to pay for tuition. What are the different types of loans that are available to you?
Getting started with a tool should be intuitive.
21
Iteration 1: Gaze plots for 4 participants on Compare Costs tool main page. Disorderly gaze patterns suggest that it isn’t readily apparent where to start on this page.
Iteration 2: Gaze plots for 4 participants on Compare Costs tool main page. More orderly gaze patterns clustered at the top of the page suggest that it is more apparent where to start on this page.
27 sec Time to start:
14 sec Time to start:
@romanocog #eduiconf
TASK: Let’s say that you have your list narrowed down to the two schools that you have financial aid offers from. You are interested in graduation rates of those two schools. Which university graduates a higher percentage of students who enroll?
Getting started with a tool should be intuitive.
22
Iteration 1: Gaze plots for 4 participants on Compare Costs tool main page. Disorderly gaze patterns suggest that it isn’t readily apparent where to start on this page.
Iteration 2: Gaze plots for 4 participants on Compare Costs tool main page. More orderly gaze patterns clustered at the top of the page suggest that it is more apparent where to start on this page.
27 sec Time to start:
14 sec Time to start:
UX Best Practices & Recommendations:
• It should be apparent to the user how to get started using a web tool.
• Use a prominent location and, if necessary, highlight the button that users need to get started.
UX Metrics:
• Time to First Click: If people make the correct first click quickly, the design is intuitive (for that particular task).
• Eye Tracking Gaze Plots: Gaze patterns inform us of the order in which a user processes
information. Disorderly gaze patters imply a poor design.
@romanocog #eduiconf
TASK: Let’s say that you have your list narrowed down to the two schools that you have financial aid offers from. You are interested in graduation rates of those two schools. Which university graduates a higher percentage of students who enroll?
23
Input methods should be apparent. • When attempting to put a number in a box that
requires a previous box to be completed, no indication is given that participants have to fill out the prior information first, resulting in participants becoming increasingly frustrated.
• Participants commented that there weren’t enough boxes to put in all of the information they had; specifically when entering scholarship information, there is only one blank for “other scholarships and grants,” prompting several participants to say that they wished there were additional blanks.
• The Cost of Attendance dropdown automatically opens, but the Money for School dropdown does not. As a result, some participants did not notice that they could enter additional information, resulting in confusion.
“It made me think that the site was incomplete or I missed something.” “I don’t understand. What’s the point if you can’t compare the costs. Maybe I did it wrong.”
Iteration 1: Compare tool input fields.
Iteration 2: Compare tool input fields.
@romanocog #eduiconf
TASK: Let’s say that you have your list narrowed down to the two schools that you have financial aid offers from. You are interested in graduation rates of those two schools. Which university graduates a higher percentage of students who enroll?
24
Input methods should be apparent. • When attempting to put a number in a box that
requires a previous box to be completed, no indication is given that participants have to fill out the prior information first, resulting in participants becoming increasingly frustrated.
• Participants commented that there weren’t enough boxes to put in all of the information they had; specifically when entering scholarship information, there is only one blank for “other scholarships and grants,” prompting several participants to say that they wished there were additional blanks.
• The Cost of Attendance dropdown automatically opens, but the Money for School dropdown does not. As a result, some participants did not notice that they could enter additional information, resulting in confusion.
“It made me think that the site was incomplete or I missed something.” “I don’t understand. What’s the point if you can’t compare the costs. Maybe I did it wrong.”
Iteration 1: Compare tool input fields.
Iteration 2: Compare tool input fields.
UX Best Practices & Recommendations:
• Provide relevant error messages when users enter information in the incorrect order (or skip a field) to promote recovery and minimize frustrations.
• Provide enough places to enter information – match user expectations. In this particular example, more options for entering scholarship amounts into the tool would have been clear and less cognitively taxing for the user.
UX Metrics:
• Think Aloud Protocol: Users verbalize their experience while completing tasks and tell us in-the-moment when they are frustrated. We quantify the percentage of users that verbalize a particular problem and gain insight into how severe the problem is.
@romanocog #eduiconf
TASK: Let’s say that you have your list narrowed down to the two schools that you have financial aid offers from. You are interested in graduation rates of those two schools. Which university graduates a higher percentage of students who enroll?
Infographics should be easily understandable.
25
• Because participants are confused about how to enter their information into the tool correctly, the output graphics that the tool generates are also confusing.
• The Left to pay number is not calculated into the debt burden, leading to several participants being confused as to why they’ll have so little debt at graduation.
• No explanation is provided for where salary is pulled from, leading several participants to question the accuracy of that number.
Iteration 2: Compare tool output graphics.
@romanocog #eduiconf
TASKS: After you graduate, what would be the debt burden for [THE SECOND SCHOOL THEY ENTER]? After you graduate, what is the monthly payment for the loans you would take out while attending [THE FIRST SCHOOL THEY ENTER]? How much debt burden would you have at [SECOND SCHOOL] vs. [FIRST SCHOOL]?
Infographics should be easily understandable.
26
• Because participants are confused about how to enter their information into the tool correctly, the output graphics that the tool generates are also confusing.
• The Left to pay number is not calculated into the debt burden, leading to several participants being confused as to why they’ll have so little debt at graduation.
• No explanation is provided for where salary is pulled from, leading several participants to question the accuracy of that number.
Iteration 2: Compare tool output graphics.
UX Best Practices & Recommendations:
• Thoroughly explain how to use a diagram; do not assume that the user will be able to figure out how to interpret an infographic.
• Use hover-over pop-up boxes to explain what each portion of the infographic actually represents.
• Identify sources of information whenever appropriate. For example, salary information and graduation rates should be explained and sourced in this infographic.
@romanocog #eduiconf
TASKS: After you graduate, what would be the debt burden for [THE SECOND SCHOOL THEY ENTER]? After you graduate, what is the monthly payment for the loans you would take out while attending [THE FIRST SCHOOL THEY ENTER]? How much debt burden would you have at [SECOND SCHOOL] vs. [FIRST SCHOOL]?
Users concentrate on the most important site features.
27
Participants in the Graduate group were asked what content on the site was relevant to them. They looked at the primary features of the site in order to locate relevant content.
PFC main homepage.
Fixation count heat map of 12 participants on the main page, fixating mostly on the PFC header and on the four main menu tabs, including the Repay Student Debt tab.
@romanocog #eduiconf
Users concentrate on the most important site features.
28
Participants in the Graduate group were asked what content on the site was relevant to them. They looked at the primary features of the site in order to locate relevant content.
PFC main homepage.
Fixation count heat map of 12 participants on the main page, fixating mostly on the PFC header and on the four main menu tabs, including the Repay Student Debt tab.
UX Metrics:
• Eye Tracking Heat Maps: Heat maps allow us to see concentrations of people’s fixations; in this instance, people fixate the introductory text and the 4 labels of the main navigation, which is exactly where we want them to look.
@romanocog #eduiconf
UX Metrics in Action: Prototype Testing
29
@romanocog #eduiconf
Repay Student Debt Prototype Methodology
30
*Prototypes presented were non-functional PDF prototypes of the starting page of the repayment tool.
CFPB proposed testing three non-functional prototypes of the main page of the Repay Student Debt tool.* The look and feel of those prototypes differed in each round of testing.
Borrower Type: Borrower type allowed participants to choose whether they were a typical borrower or had special circumstances. Version 1 of the Borrower prototype included images, while Version 2 did not. All participants saw one version of this prototype.
Loan Type: Loan type allowed participants to select what type(s) of loans they have. This was similar to how the live site worked. Version 1 of the Loan prototype included images, while Version 2 did not. All participants saw one version of this prototype.
Repayment Type: Repayment type allowed participants to choose what kind of repayment plan they were interested in. Only Round 2 participants saw this prototype.
Goal Type: Goal type allowed participants to choose whether they wanted to lower their monthly payment, pay less interest over time, or inquire about other programs. Only Round 3 participants saw this prototype. The main purpose of the prototype testing was to determine which starting point participants thought was the most logical. Four different starting points were tested:
@romanocog #eduiconf
Participants “clicked” the ‘Get Started’ content on each prototype first.
31
A majority of participants said that they would click on the Get Started content on every version of all prototypes except the repayment type prototype.
*Repayment Type prototype was only seen by Round 2 participants; the Goal Type prototype was only seen by Round 3 participants.
0
4
8
12
16
Borrower Type Loan Type Repayment Type*
Goal Type*
First “Clicks” on Any of the Get Started Buttons
Round 2 Round 3 Borrower Type Version 2.
@romanocog #eduiconf
Participants “clicked” the ‘Get Started’ content on each prototype first.
32
A majority of participants said that they would click on the Get Started content on every version of all prototypes except the repayment type prototype.
*Repayment Type prototype was only seen by Round 2 participants; the Goal Type prototype was only seen by Round 3 participants.
0
4
8
12
16
Borrower Type Loan Type Repayment Type*
Goal Type*
First “Clicks” on Any of the Get Started Buttons
Round 2 Round 3 Borrower Type Version 2.
UX Metrics:
• First Clicks: PDF prototypes were not clickable. We ask participants where they would click first and gain insight into how users would interact with these pages if they were live.
@romanocog #eduiconf
Prototype Summary & Recommendations
33
• There was no overwhelming favorite among participants regarding which prototype was their favorite. – 4 out of 18 (22%) participants preferred the Borrower Type prototype – 6 out of 18 (33%) participants preferred the Loan Type prototype – 4 out of 8 (50%) participants preferred the Repayment Type prototype – 4 out of 10 (40%) participants preferred the Goal Type prototype
Recommendations: Start with either the goal or the loan prototype. Although there was no real consensus around one prototype, almost all participants said something positive about the goal prototype. Having a goal-oriented way of getting into the tool matches how users think of the process. The loan prototype was basically a different-looking version of the live site, which participants found generally intuitive. Include a ‘Help Me Decide’ button. Help Me Decide was a popular first click button on the one prototype it was shown with. It would allow users who don’t necessarily have a goal a way to get in. Include graphics. Although graphics were removed from the second iteration of the prototypes for content-testing purposes, the feedback on them was generally positive in the first round of testing. Remove content below the fold. Most participants clicked on one of the Get Started options; however, some clicked on the other content below the fold. If the main purpose of this site is to get people to use the tool, then content below the fold could be removed to simplify the page.
@romanocog #eduiconf
Prototype Summary & Recommendations
34
• There was no overwhelming favorite among participants regarding which prototype was their favorite. – 4 out of 18 (22%) participants preferred the Borrower Type prototype – 6 out of 18 (33%) participants preferred the Loan Type prototype – 4 out of 8 (50%) participants preferred the Repayment Type prototype – 4 out of 10 (40%) participants preferred the Goal Type prototype
Recommendations: Start with either the goal or the loan prototype. Although there was no real consensus around one prototype, almost all participants said something positive about the goal prototype. Having a goal-oriented way of getting into the tool matches how users think of the process. The loan prototype was basically a different-looking version of the live site, which participants found generally intuitive. Include a ‘Help Me Decide’ button. Help Me Decide was a popular first click button on the one prototype it was shown with. It would allow users who don’t necessarily have a goal a way to get in. Include graphics. Although graphics were removed from the second iteration of the prototypes for content-testing purposes, the feedback on them was generally positive in the first round of testing. Remove content below the fold. Most participants clicked on one of the Get Started options; however, some clicked on the other content below the fold. If the main purpose of this site is to get people to use the tool, then content below the fold could be removed to simplify the page.
UX Metrics:
• Debriefing Interviews: After the tasks and testing is over, a debriefing interview allows us to ask users about their preferences. We can then quantify those preferences to gain insight into which prototypes users prefer.
@romanocog #eduiconf
• Combination of subjective, observational, and implicit data to evaluate the UX. • Many UX elements improved from Iteration 1 to Iteration 2; some need further exploration.
35
0%
0%
17%
56%
28%
0%
8%
8%
83%
0%
Extremely Difficult
Very Difficult
Moderately Difficult
Slightly Difficult
Not Difficult at All
0%
0%
6%
44%
50%
8%
0%
33%
58%
0%
Extremely Difficult
Very Difficult
Moderately Difficult
Slightly Difficult
Not Difficult at All
“Please rate your overall experience with the Paying for College website.”
“Please rate your difficulty with navigating on the Paying for College website.”
Iteration 1 Iteration 2
Study Impacts @romanocog #eduiconf
• Combination of subjective, observational, and implicit data to evaluate the UX. • Many UX elements improved from Iteration 1 to Iteration 2; some need further exploration.
36
0%
0%
17%
56%
28%
0%
8%
8%
83%
0%
Extremely Difficult
Very Difficult
Moderately Difficult
Slightly Difficult
Not Difficult at All
0%
0%
6%
44%
50%
8%
0%
33%
58%
0%
Extremely Difficult
Very Difficult
Moderately Difficult
Slightly Difficult
Not Difficult at All
“Please rate your overall experience with the Paying for College website.”
“Please rate your difficulty with navigating on the Paying for College website.”
Iteration 1 Iteration 2
Study Impacts
UX Metrics:
• Satisfaction Questionnaire: After the tasks and debriefing, asking users to rate their satisfaction allows us to make comparisons in iterative testing.
@romanocog #eduiconf
UX Research Success: User Types
37
High School Students
Influencers
Round 1: Austin, TX
Round 3: Arlington, VA
Round 2: Spokane, WA
Live site version 1 Live site version 1 Live site version 1 OR Prototype site version 2
Iteration 1:
February/ March 2014
High School Students
College Graduates
Round 1: Atlanta, GA
Round 3: Arlington, VA
Round 2: Milwaukee, WI
Live site version 2 Live site version 2 Live site version 2
Iteration 2:
April 2014
@romanocog #eduiconf
UX Research Success: Client Collaboration
38
High School Students
Influencers
Round 1: Austin, TX
Round 3: Arlington, VA
Round 2: Spokane, WA
Live site version 1 Live site version 1 Live site version 1 OR Prototype site version 2
Iteration 1:
February/ March 2014
High School Students
College Graduates
Round 1: Atlanta, GA
Round 3: Arlington, VA
Round 2: Milwaukee, WI
Live site version 2 Live site version 2 Live site version 2
Iteration 2:
April 2014
@romanocog #eduiconf
Task Creation
SATQ Items
Other materials
Debriefing
Reports
.
. .
.
. .
.
Iterative Process with Client
“A spectator sport” @skug
UX Research Success: Multiple Data
39
OBSERVATIONAL + Time to complete tasks
+ Time to first click + First click accuracy
+ Task accuracy
IMPLICIT + Eye tracking
SUBJECTIVE + Difficulty and satisfaction ratings
+ Think aloud protocol + Debriefing interview
• Eye tracking [implicit] tells us what participants focus on. • Think aloud & debriefing [subjective] tell us why they focus there. • Observational measures tell us how they go about completing a task. • Combination of subjective, observational, & implicit metrics paints the whole picture.
@romanocog #eduiconf
UX Research Success
40
• All relevant user types participated. • Students • Graduates • Educators • Parents
• All relevant parties were involved. • Designers • Developers • Key stakeholders
• Multiple data sources provided insight into the complete experience. • Iterative process: Test, improve, test again
• Ensure changes improve the UX. • Ensure something else did not break.
@romanocog #eduiconf
Thank you! • Twitter: @forsmarshgroup • LinkedIn: http://www.linkedin.com/company/fors-marsh-group • Blog: http://www.forsmarshgroup.com/categories/blog/
Jennifer Romano Bergstrom
@romanocog [email protected]
Lamar Alayoubi
Eye Tracking in UX – Oct 16
42
User Focus – Oct 17 eyetrackingux2014.com October 16, 2014 EDUI = 20% OFF
2014.userfocus.org October 17, 2014 EDUI = 20% OFF