introducing a new ux maturity metric: team engagement...
TRANSCRIPT
![Page 1: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/1.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Introducing a New UX Maturity Metric: Team Engagement Score
(TES) During Usability Testing
Bill Albert
Executive Director, Bentley University User Experience Center
Josh Rosenberg
User Experience Researcher, DonorsChoose.org
June 6, 2017
@BentleyUXC | @UXMetrics
![Page 2: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/2.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Two Short Stories
@BentleyUXC | @UXMetrics
![Page 3: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/3.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Why Team Engagement Matters
• Adoption of UX recommendations
• Efficiency in the design process and product development
• Culture of UX throughout the organization
• Sparking innovation
• Value to co-workers
@BentleyUXC | @UXMetrics
![Page 4: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/4.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Our Motivation
• Formalize our thinking through development of a team engagement metric
• Launch discussions internally – explore the value of a team engagement metric
• Share with others in the UX community
• Promote team engagement
![Page 5: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/5.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
TES: Team Engagement Score
TES = Observation + Collaboration + Behavior
• Observation = Observing usability sessions
• Collaboration = Being part of the team during testing
• Behavior = Being actively engaged when observing
@BentleyUXC | @UXMetrics
![Page 6: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/6.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Component 1: Observation
@BentleyUXC | @UXMetrics
# Actual Observed Sessions
# Total Observable Sessions
![Page 7: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/7.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
25 Actual Observed Sessions100 Total Observable Sessions
Component 1: Observation
@BentleyUXC | @UXMetrics
0.25=
![Page 8: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/8.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Component 2: Collaboration
# Feasible In-Person Observations
# In-Person Observations
![Page 9: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/9.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Component 2: Collaboration
2 In-Person Observers
5 Possible Observers
= 0.40
![Page 10: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/10.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Component 3: Behavior
@BentleyUXC | @UXMetrics
10
1-10 Rating
![Page 11: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/11.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Component 3: Behavior
@BentleyUXC | @UXMetrics
Took Notes
Follow-up Questions
Paid Attention
Brainstormed Solutions
Etc.
Rating: 8
✓
✓
✓
Rating: 8
10
= 0.80
![Page 12: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/12.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Component 3: Behavior
@BentleyUXC | @UXMetrics
Component 3: B
@BentleyUXC | @UXMetrics
• Attention (0 -2)• Low = meetings, multi-tasking; High = undivided attention
• Conversation (0-2)• Low = none; High = creative brainstorming based on insights
• Notes (0-2)• Low = no notes; High = took detailed notes
• Attitude (0-2)• Low = discredits participants; High = empathizes with participants
• Questions (0-2)• Low = no questions; High = a lot of questions
![Page 13: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/13.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Calculating TES
• TES = Observation + Collaboration + BehaviorTES = ((# actual observations/ # total possible observations) +
(# in-person observers / # feasible in-person) +
(engagement score/10)) / 3
• Average the three numbers• TES = (0.25 + 0.4 + 0.8) = 1.45, divided by 3 = 0.48
• TES runs from 0 – 1 (higher the better)
• All three dimensions weighted equally, but can apply different weights
• TES works for large or small teams
@BentleyUXC | @UXMetrics
![Page 14: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/14.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Scenario 1: High Engagement
@BentleyUXC | @UXMetrics
• Observation = 10 sessions (2 days), 8 attend all 10 sessions, 2 attend the first day only (5) = 90/100 = 0.90
• Collaboration = 5 people local, all attend in person = 5/5 = 1.0
• Behavior = highly engaged = 9/10 = 0.9
• TES = 0.9 + 1.0 + 0.9 = 2.8/3.0 = .93
![Page 15: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/15.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Scenario 2: Low Engagement
@BentleyUXC | @UXMetrics
• Observation = 10 sessions (2 days), 4 attend day 1 only, 6 don’t attend any sessions = 20/100 = 0.20
• Collaboration = 5 people local, 2 attend in person = 2/5 = 0.4
• Behavior = low engagement = 3/10 = 0.3
• TES = 0.2 + 0.4 + 0.3 = 0.9/3.0 = .30
![Page 16: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/16.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Scenario 3: Remote Only Studies
@BentleyUXC | @UXMetrics
• Observation = 10 sessions (2 days) with 10 invited - 3 attend both days, 5 attend day 1 only, 2 don’t attend any sessions = 45/100 = 0.45
• Collaboration = all remote, so this component is not considered
• Behavior = moderate engagement = 6/10 = 0.6
• TES = 0.45 + 0.6 = 1.05/2.0 = .525
![Page 17: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/17.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Changing Weights
@BentleyUXC | @UXMetrics
• Change weights if you value one component more than another
• Observation = 0.5 x 2 = 1.0 (weighted twice)
• Collaboration = 0.4 x 0.5 = 0.2 (weighted half)
• Behavior = moderate engagement = 0.6 (no change)
• TES = 1.0 + 0.2 + 0.6 = 1.8/3.0 = 0.6
![Page 18: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/18.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Interpreting TES
@BentleyUXC | @UXMetrics
0.1 0.2 0.3 0.4 0.60.5 0.7 0.90.8
Big Problem Fully EngagedOpportunity to ImproveWork to Do
Take it Seriously
![Page 19: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/19.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Improving TES
• Formalize the process
• Bake into design schedule
• Senior-level support
• Correlate metrics to individual/team performance reviews
• Free food is no longer viable!
@BentleyUXC | @UXMetrics
![Page 20: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/20.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
How Would You Measure Team Engagement?
@BentleyUXC | @UXMetrics
![Page 21: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/21.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Value of TES
• Easy to collect and measure
• Track team engagement over time
• Report with other UX metrics
• Roll into larger UX maturity score
• Is this something you would use?
@BentleyUXC | @UXMetrics
![Page 22: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/22.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Limitations of TES
• Not a validated metric, no normative data
• Engagement is more than participating in usability testing
• Not all observers are the same, not all sessions are the same
• Potential political ramifications
• Other dimensions of engagement are hard to measure
@BentleyUXC | @UXMetrics
![Page 23: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/23.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Delivering Value
@BentleyUXC | @UXMetrics
![Page 24: Introducing a New UX Maturity Metric: Team Engagement ...d2f5upgbvkx8pz.cloudfront.net/sites/default/files/inline-files/UXPA2017_Albert...Introducing a New UX Maturity Metric: Team](https://reader034.vdocument.in/reader034/viewer/2022050110/5f47ab7344fedb154d7c3ac1/html5/thumbnails/24.jpg)
#UXPA2017
www.uxpa2017.orgDownload our mobile app UXPA.org/2017app to fill out our conference survey!
Questions?
Bill Albert
Executive Director
Bentley University User Experience Center
www.bentley.edu/uxc
@BentleyUXC | @UXMetrics