setting the right goals and objectives for your beta test
TRANSCRIPT
1
Setting the Right Goals for Your Beta
2
Meet the Speaker
John S. Little
Director of Research @ Centercode
Agenda
1. What are goals?
2. How to set good goals for your beta test
3. How to maintain your goals throughout the beta phase
4. How to connect the insights from your test to your goals
5. Q&A
3
4
Who is Centercode?
Services Fully managed tests and programs
Software Complete beta program management platform
Testers Access to 150k+ deeply profiled global candidates
Content Free tools and best practices
Customer validation is our core competency
We've been collecting customer feedback for tech companies since 2001
GoalsIntroduction
5
Webinar Goals
Establish good goals for beta tests
Accomplish goals of the beta test
Leverage information to make a better product
6
7
Goals Refresher
Business Definition
An observable and measurable end result
having one or more objectives to be achieved
within a more or less fixed timeframe.
http://www.businessdictionary.com/definition/goal.html#ixzz4KBQiUDi6
8
Goals for betas
“Goal is something that you want to achieve”
How am I going to do it? - Method
Why am I going to do it? - Return
Useful tools● S.M.A.R.T.● Definition of Done● Commander’s intent
9
Quote
If one does not know to which port one is sailing, no wind is favorable. – Seneca
10
11
Good Goals
Beta Goal Types
1. Overall Test Goal
2. Feature Set Level(s)
3. Attributes
12
Overall Test Goals - Finding a Focus
What is the main goal of the beta test?
● Natural usage● Coverage of features● Opinion of features● Partial, focused coverage● Unique environments and use cases
13
Real vs. Complete
Natural usage Coverage of features
14
Feature Set Goals - Finding a Focus
Where do I find the areas of the product that need focus?
● Product/Market Requirements Document (PRD/MRD)● Marketing Messaging● Packaging Design● Key Stakeholder Requests
15
Priorities for Constraints
16
Constraints
● Time● Products● Cost● Tester Availability
Feature Sets
● Setup● Documentation● Website● App● Settings● Out of box● WiFi● Feature 1 - Feature n
VS
Attributes
17
Overall Satisfaction
Perceived Performance
Ease of Use
Usefulness
Preference
Comparative
Self Reported / Attitudinal
Communicate Goals
Document, distribute, and follow-up
● Key Stakeholders
● Immediate team members
● Extended team members (build awareness)
● Testers (if appropriate)
18
Goals for Feature Sets
19
Bad Good
Collect feedback on feature Measure user’s satisfaction with ease of use of feature
Measure usability of the product Evaluate customer acceptance of product
Measure the performance of the product Measure perceived performance while initiating application
How long did it take to setup the product? Evaluate satisfaction with time to set up
Will they pay $150 for it? Conjoint analysis
After Scenario Questionnaire (ASQ)
20
Example
Product Example
Internet of Things Light
● Mobile App● Preset Colors● Manual Color Selection● Timer ● Alarm● Auto-dimmer
21
Beta Goals
● Evaluate total customer experience● Assess MVP readiness● Roadmap validation● Competitive analysis● Identify bugs● Real-world impact of known issues● Regression testing● Analyze real-world performance
● Test compatibility● Study user experience● Assess new user experience● Test documentation● Test support processes● Collect mock reviews● Increase product confidence● Collect case studies
22
Objectives
Quality Customer Acceptance Natural Use
● Test compatibility● Assess MVP readiness● Competitive analysis● Identify bugs● Regression testing● Analyze real-world
performance
● Evaluate total customer experience
● Assess new user experience● Collect mock reviews● Increase confidence● Real-world impact of known
issues
● Study user experience● Test documentation● Test support processes● Collect case studies● Roadmap validation
23
SMART Beta
SMART Criteria
Specific Customer acceptance of product
Measurable Quantifiable by survey(s) & severity of issues
Achievable Product state / Tester availability / Tool
Relative Feedback will go into this version or next
Timebound 2 weeks of test -> 2 weeks regression -> GM
24
Definition of Done
❏ All new feature in the product tested by customers
❏ Measurable feedback on the customer acceptance of the product
❏ All discovered bugs prioritized and sent to QA
❏ Tester suggestions provided to product management
❏ Support alerted of specific pain points that are not being addressed
25
Commander’s Intent
Complete a beta study of the customer acceptance of the product
in order to
understand and make critical improvements prior to release or general improvements in the next revision
26
Feature Set vs. Constraints
Features● Packaging● Easy Setup● Preset Colors● Manual Color Selection● Timer and Alarm● Auto-dimmer
Research Attributes● Ease of Use● Stickiness● Usefulness● Feature Satisfaction
Constraints
Units = 32
Time = 2 Weeks
Tools = Centercode
Testers = Betabound
Test Team = ½ Time (me)
27
Feature Set vs. Constraints
Features● Packaging● Easy Setup● Preset Colors● Manual Color Selection● Timer and Alarm● Auto-dimmer
Research Attributes● Ease of Use● Stickiness● Usefulness● Feature Satisfaction
Constraints
Units = 32
Time = 2 Weeks
Tools = Centercode
Testers = Betabound
Test Team = ½ Time (me)
28
Establish Goals
Primary Test Goal = Customer Acceptance
Feature Sets = Easy Setup; Preset Colors; Manual Color Selection; Timer; Alarm
Attributes/Feedback
● Satisfaction, Ease of use, etc. (Surveys)● Bugs (Issue reports)
Week 1
● Setup; Manual Colors; Timer
Week 2
● Alarm; Preset Colors; Timer29
Centercode Test Plan
30
One more time
1. Identify test objectives
2. Group and provide reasonable options
3. Ensured that the goal is well crafted
4. Identified feature set goals & attributes
5. Prioritized based on constraints
6. Document and send out for review
Start
Finish
31
32
Maintaining Goals
Surveys
Goal: Measure Attributes
Pros
Pointer to areas that might have problems
Very simple to send to large numbers of people
Cons
Qualitative aspects difficult to analyze
Attitudinal satisfaction with attributes of items33
Bugs and Suggestions
Goals: Unique Environments and Use Cases
Pros
Facts & opinions from testers
Build both - let it sort for you
Cons
Difficult to quantify without tools
Can require back-and-forth for clarity 34
Journals
Goal: Natural Use Information
Pros
Diary type study
Requires limited guidance to testers
Cons
Requires more time for analysis
Could be limited35
36
Goal Dangers
Type Mismatch
Article - https://www.nngroup.com/articles/which-ux-research-methods/
Study Type Results
Attitudinal vs. Behavioral What they say vs. what they do
Qualitative vs. Quantitative Why/how to fix vs. how many / how much
37
Miscommunication
Goals not agreed upon
Feedback not solicited
New or unaware stakeholders
Misguided testers
Wrong methodology
38
¯\_(ツ)_/¯
39
Goals to Insight
Connect Feedback Types
Surveys and Bugs
Bugs and Suggestions
Surveys and Suggestions
Journals and …
Marketing, UX, Support, Sales, QA
40
Focus on Product Goals
Focus results on original goals - they were goals for a reason
Bring in updates to the information that helped you define the goals
Only highlight other areas if they are significant
41
Generate Top X List
Information overload happens to everyone
Prioritize for your own sake
Short list will get more attention than long list
Bring in the “good/promotes” from the test
Everyone likes to have a couple wins, and so does the customer
42
Problem Elevator Pitch
20 - 30 second rundown
Customer’s point of view
Focused on selling the item to the team
One for each item on “Top X” list
43
44
Key Takeaways
If you remember nothing else...
1. Clearly define your goals - always; and associate
the best method to achieve them before signing off
on the project.
2. Stay focused on goals, unless significant change
3. Beware of research myopia
4. If you could use help, contact us!
45
46
Q&A
John S. Little
Director of Research @ Centercode
Sign up for our newsletter and our upcoming webinars
For more resources, visit our library
47
Thank You!