Download - Running a Successful Beta Test Program
Running a SuccessfulBeta Test Program
Adam Long (CTO)Darren Gage (QA Mgr) Sam Soubra (Dev Mgr)
September 2010
Agenda
• About Us• Running a Beta Test Program• Runtime Intelligence Service– Overview– Code Demo
• Questions
About our Company
Established in 1995
Independent Software Vendor
based in Melbourne, Australia
90% of revenue from exports
> 400,000 customers in 150 countries
About our Software
• Qualitative Data Analysis– Unstructured Data
• Documents• Audio / Video• Images• Spreadsheets
• What MS Excel does for structured numerical data, we do for unstructured non-numerical data
About our Software
• NVivo 9– .NET 3.5
• Win Forms• WPF• WCF
– SQL Server 2008 R2 Express
• NVivo Server 9– Silverlight 3– SQL Server 2008 R2
Standard
Beta Test
What• Beta testing is conducted by
a sample of customers to identify flaws in feature complete software
• Generally occurs after alpha testing (unit & system)
Why• Validate objectives
– Market– Features– Quality
• Improve outcomes– Collaboration / Innovation
• Assist marketing– Quotes / Case Studies
• Plan future direction– Opportunities
7 Steps of a Beta Test Program
Plan
2 weeks
Promote
4 weeks
Select
2 weeks
Collect
4 weeks
Analyse
2 weeks
Prioritize
1 week
Recognize
1 week
1. Plan
• Scope– Goals & Success Criteria
• Approach– Open vs. Closed
• Responsibilities– Management, Communications, Defects, Suggestions
• Schedule– Balance between having enough features available to
test versus enough time to incorporate feedback
2. Promote
• Internal– Sales, Marketing, Training
• External– Existing Customers– Potential Customers
• Website & Social Media announcements
• Customer Newsletter• Email Select Customers
• Best Beta Testers– Interested– Independent– Vocal
• Avoid pressure to include– Sales Leads– Evaluators
3. Select
• Online Application– Outline Program Expectations– 20 Questions
• Technical Environment• Market Segment• Research Background• Competitor Software• Motivation
• Result– 523 Applicants– 78 Testers Selected
• Selection Process– Top Previous Beta Testers– Top Forum Posters– Subject Matter Experts– Random Applicants
• Simulation Program to Optimize Spread of Answers to Questions
• Obtain Signed Agreement
4. Collect
• Online Webcasts (Citrix GoToWebinar)– 5 recorded videos & 88 feedback forms
• Online Forums (IP.Board)– 412 suggestion posts & 335 defect posts
• Online Survey (Survey Monkey)– 58 survey responses
• Application Analytics (RIS)– 2,448 hours of usage statistics
5. Analyse
• Identify– Defects– Suggestions
• Understand– Most liked features– Most disliked features– Most important
improvements needed
• Dogfooding– Used NVivo Beta to
analyse feedback• Videos• Forums• Surveys
– In conjunction with• Application Analytics
6. Prioritize
• Prioritize Suggestions– 1 Must Address for RTM– 2 Preferably Address for RTM– 3 Consider for Service Pack– 4 Defer for Future Release– 5 Rejected
• Raise Defects in Bug Tracking
7. Recognize
• Active Participants (85%)– Activated the software & provided some feedback– Average of 37 hours testing over 4 weeks– Receive free copy of software
• Top 20 Participants (25%)– Also receive Amazon voucher
Beta Test Tips
• Allocate sufficient time & resources• Observe usage patterns using Application
Analytics and encourage testers to explore specific features
• Reprioritize effort based on final feedback• Hold weekly status meetings• Communicate outcomes to internal
stakeholders
Runtime Intelligence – What is it?
• The purpose of runtime intelligence as a component of SDLC management is to support better development, support and R&D investment decision making.
• Leverages technologies, managed services and practices for the collection, integration, analysis, and presentation of application usage levels, patterns and practices
Runtime Intelligence – Why use it?Gauge
Effectiveness of the beta
test
Collection Phase
Feature usage
Analysis Phase
Platform usage
Beta tester effectiveness
Runtime Intelligence – Why use it?
Improve Decision Making
Usage context while Analysing
feedback
Prioritization PhaseAnalysis
Phase
Usage context while Prioritising
feedback
Understand Quality
Runtime Intelligence – Tool Choice
• PreEmptive RIS– Features matched
those in competitor’s offerings
– We had used other PreEmptive tools (Dotfuscator)
– Integrated into Visual Studio 2010
• Other tools out there;– EQATEC Analytics– Gibraltar Software
Runtime Intelligence – Architecture
11
2
2
3
3
Instrumented App is run
Usage data sent to Data Repository
Dashboard and Reports
Runtime Intelligence – Implementation
• Credentials, Entry point, Endpoint & Security
• Features & Custom data
• System and Performance
• Opt-in policy
Configure analytics
• Inject Instrumentation
• Package and Deploy
Application distribution
• Beta Mgmt., UX program, Support
• Business Intelligence & ROI
• Operations Mgmt.
Runtime Intelligence Services
Runtime Intelligence – Services
Preemptive Cloud
Yours
Aggregate
Validate
ManagePublish
Runtime Intelligence endpointRuntime Intelligence repository
Aggregate, validate, manage, & publishManaged or on-premise
For internal dev. and multi-tenant constituencies
Runtime IntelligenceReports & Dashboards
Owner, application, feature, user, & customRESTful API and export
IDE, CRM, ERP, & IT Operations
Runtime Intelligence – Reports and Dashboard
• PreEmptives RIS Portal
Runtime Intelligence – Export
• Data is rich
• You will need to spend time to mine it!
Runtime Intelligence – Export
Circle Graph Dendrogram Horizontal Dendrogram Vertical Mds 2d Mds 3d
Total 36 65 52 82 92
5
15
25
35
45
55
65
75
85
95
Similarity Diagram Type
# us
es o
f fea
ture
s
Runtime Intelligence – Export
Windows 7 Enter-
prise
Windows 7 Home
Premium
Windows 7 Profes-
sional
Windows 7 Ultimate
Windows XP Home Edition
Windows XP Pro-fessional
Windows(R) XP Pro-fessional x64 Edi-
tion
Windows Vista
Business
Windows Vista Home
Premium
Windows Vista
Starter
Windows Vista Ul-timate
Not De-fined
Total 6 9 14 13 1 27 1 8 9 1 5 1
2.5
7.5
12.5
17.5
22.5
27.5
OS Edition
# w
orks
tatio
ns
Runtime Intelligence – Where to next?
• Move into Production
• Improve Development Decision Making for future releases
Development• P
erformance tuning
• Usability
QA• R
isk based testing
Documentation• M
aterial coverage
• Help usage
Application Usage
Code Demo
Dotfuscator by PreEmptive Solutions• Runtime Intelligence defines several message
types:– Application and Session Start – Application and Session Stop – Feature – Performance Probe – System Profile – Tamper Detected
Runtime Intelligence Portal’s dashboards
To have your application send these messages, you must:
• Be a Runtime Intelligence Service subscriber• Activate Runtime Intelligence from within
Dotfuscator• Attribute your application with Runtime Intelligence
attributes, including Setup and Teardown• Run your application through Dotfuscator with the
Send Analytics Messages option turned on
Business Attribute
Application Attribute
Setup Attribute
Teardown Attribute
Feature Attribute
Sample Application
Sample Code
ILDASM Before
ILDASM After
Analytical Data
Questions?