alpha and beta testing

8/15/2019 Alpha and Beta Testing

1/72

Alpha and Beta TestingByrne Reese

October 2, 2006


2/72

Product Manager, Six Apart

Founder, Test Run LLC

Open Source Developer

[email protected]

http://www.majordojo.com/

AIM: byrnereese

Byrne Reese


3/72

A test-centric development methodology.

Real world use cases.

What I am here to talk

about


4/72

“Since you know version 1 is

going to suck , hurry up and

ship it so you can start

working on version 2.”

Google Proverb


5/72

Alpha and Beta testing is notabout flushing out bugs…


6/72

It’s about enabling afeedback loop.


7/72

Stop thinking about

alpha and beta testingas phases in a

development cycle.


8/72

Alpha and Beta Testing is an integral part

of emerging development methodologies.

And for that, you have to look at the

whole process.


9/72

Phase OneRequirements


10/72

Wait, wait, I forgot…

Before you begin you need to:a) define your target audience.

b) start collecting sign-ups.


11/72

Phase OneRequirements


12/72

Test on “paper.”Test on “paper.”


13/72

Iterate on paper, whiteboards,

post it notes, index cards...Test team members’ ideas with

one another.


14/72

And remember:

It doesn’t haveto look perfect

(It’s a frick’n wireframe)


15/72

Then, as soon as you can…


16/72

STOPtesting on paper!


17/72

Don’t get trapped inwireframe-land.


18/72

Phase Two

Development


19/72

Create a tangible mock up.

Use what you create.

Watch others use what you create.

Learn from that experience.


20/72


21/72

Watching others…


22/72

Select your subjects carefully. Do they align with

your target market?

Get an impartial party trained to presentoptions and collect feedback from subjects.

Record the session.

Then have the whole team watch it.


23/72

Tools I Use:

Macromedia Breeze

Ethnio


24/72

Establish a

development

rhythm early.


25/72

Create an engineering pipeline.

Requirements

Release

Design

Requirements Design

Development Testing

Requirements

Development


26/72

Phase Three

User Testing


27/72


28/72

Create as many open

communicationchannels as you can.


29/72


30/72

Create a discussion forum.Answer questions.


31/72

Buy a help ticketing system.Build your knowledge base.


32/72


33/72

Talk to your customers.


34/72

Listen to them too.


35/72


36/72

even if it is easy to do.


37/72

But definitely do some of the

things they tell you.

Make your customers

feel heard.


38/72

Make it crazy-easy

for customersto report bugs.


39/72

Use automated error reports.


40/72


41/72

Log the error anyway.Send a daily report of all errors to the

entire team.

Meet daily to review open bugs and

to schedule their fix.

Schedule “bug-a-thons.”


42/72

Don’t just record errors.

Record everything.

Understand your users’ behavior.


43/72


44/72


45/72

Don’t forget: early beta testers are

perhaps your most important asset.

They will form the core

of your user base.


46/72

Reward your betatesters.


47/72

Praise them for helping.

Give them credit for their ideas.

Send them product discounts.

Buy them something off their wish list.

Hire some of them.

Release frequently.


48/72

Lessons from


49/72

Sandoz Laboratory produced a methodfor isolating the active ingredients of

medicinal plants in order to create

proper dosages for patients.

And now?


50/72


51/72


52/72

The Internet was created to build acommunications network impervious

to nuclear assault.

And now?

“Tubes”


53/72


54/72

Ludicorp saw what users werereally using and adapted.

And now?


55/72

Early on Friendster actively stoppedpeople from friending movies,

products and other “stuff.”

And now?


56/72

96,697 friends and counting…


57/72

Blogspot was originally a simplenews board in a larger project

management software suite.

And now?


58/72


59/72

After 6 months of beta testing, Swivelshut the product down and

went back to the drawing board.

And now?


60/72

Summary


61/72

The customer really is

always right.


62/72

The customer really is

always right.(most of the time)


63/72

Don’t be afraid tomake Hard Decisions.


64/72

Alpha and Beta Testing

is not just aboutflushing out bugs…


65/72

It’s abouttesting your ideas.


66/72

It’s abouttesting your process.


67/72


68/72

And it’s aboutflushing out bugs.


69/72

fin.


70/72

The Slide Graveyard:

where slides go to die.


71/72

Alpha and Beta testingare not the last phases of

development.

Requirements Development QARelease

alpha beta


72/72

They are the last phase of every

phase of development.

Requirements Development User TestingRelease

alpha betaalpha betaalpha beta

alpha and beta testing

Documents