michael bolton developsense - step-in forum bolton - a rapid...1 a rapid introduction to rapid...

96
1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January 2007

Upload: buithien

Post on 15-Mar-2018

223 views

Category:

Documents


4 download

TRANSCRIPT

Page 1: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

1

A Rapid Introduction to Rapid Software Testing

Michael BoltonDevelopSenseSTeP-IN Conference

Bengalooru, IndiaJanuary 2007

Page 2: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

2

Who I Am

Michael Bolton(not the singer, not the guy in Office Space)

DevelopSense, Toronto, Canada

[email protected]+1 (416) 992-8378

http://www.developsense.com

Page 3: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

3

Acknowledgements

James Bach, senior author of this course− http://www.satisfice.com− Co-author of Lessons Learned in Software Testing− James and I both teach this course, of which I am co-author

Much of this presentation is influenced by the work of our colleagues and mentors, including Jerry Weinberg, Elisabeth Hendrickson, Doug Hoffman, Payson Hall

Page 4: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

4

Acknowledgements

Cem Kaner, Ph.D., J.D.− http://www.kaner.com− http://www.testingeducation.org

− Senior author of Lessons Learned in Software Testing− Cem presents an online course on Black Box Software Testing,

written by himself and James Bach− http://www.testingeducation.org/BBST/index.html− university level− online videos, course notes, supplementary reading

−free

Page 5: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

5

Rapid Testing

Rapid testing is a mind-setand a skill-set of testing

focused on how to do testingmore quickly,

less expensively, with excellent results.

This is a general testingmethodology. It adapts to

any kind of project or product.

Page 6: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

6

KEY IDEA

The tester’s job is

to think critically about the software,

so that we can provide information,

so that management can make informed decisions

about the product and the project.

Page 7: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

7

Quality is value to some person.

A bug is anything that threatens the valueof the product.

These definitions are designed to be inclusive.Inclusive definitions minimize the chance that testers will inadvertently overlook an important problem.

Testers light the way.

Page 8: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

8

This is our role.We see things for what they are.

We make informed decisions about quality possible, because we think critically about software.

Testers light the way.

45

Page 9: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

9

The “questions” consist of various ways of configuring and operating the product. (Asking questions about the product without operating it is usually called review rather than testing)The product “answers” by exhibiting behavior, which the tester observes and evaluates.To evaluate a product is to infer from its observed behavior howit will behave in the field, and to identify important problems in the product.

Testing is questioning a product in order to evaluate it.

Testers light the way.

Page 10: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

10

Questions = tests; right answers = credible observationsTraditionally, testing has been more focused on getting the right answers, through carefully documented, controlled, and repeated tests (confirmation)BUT… there are so many questions worth asking (investigation), we have to consider the tradeoffs between ultimate care in getting the right answers and the danger of simply not doing enough testing.

Asking the right questionsvs.

Getting the right answers

Two Sides of Testing

Page 11: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

11

What is a Test?

The test is what youthink and do!

The test is not the test script.

Only a fraction of a skilled tester’s thinkingand behavior can be represented in a script

(about which more later).

Page 12: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

12

KEY IDEA

“Rapid Testing” means

the fastest,

least expensive testing

that fulfills the mission.

Page 13: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

13

The Triangle Program

This program takes three numbers as input.The numbers represent the dimensions of a triangle.When you click on the check button, the program tells you what kind of triangle the sides represent:

− scalene (no side equal to any other)− isosceles (two sides are equal)− equilateral (all sides are equal)

I want you to test this program.Any questions?

Page 14: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

14

Some Follow-Up Questions

What was your mission?What problems did you find?How did you know they were problems?How did you find them?What questions did you ask?Can you report credibly?Did you need more time?Is the Triangle program good?

Page 15: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

15

The Universal Test Procedure, v1.0

“Try it and see if it works.”

Get itSet it upRun itRun it again, maybe

Choose where to lookSee what’sthereSee what’s not there

Read the specSee if the product matches the specFind problems……especially the bad ones

Procedures OraclesCoverage

Page 16: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

16

Tests

ProjectEnvironment

ProductElements

QualityCriteria

PerceivedQuality

A Heuristic Test Strategy Model

Page 17: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

17

ProjectEnvironment

Tests

ProductElements

QualityCriteria

PerceivedQuality

A Heuristic Test Strategy Model

Page 18: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

18

The Universal Test Procedure, v2.0“Try it and see if it works.”

There are numerous waysof doing these

things. Patterns thatdescribe how to do

this are called“test techniques.”

1. Model the test space.2. Determine coverage.3. Determine oracles.4. Determine test procedures.5. Configure the test system.6. Operate the test system.7. Observe the test system.8. Evaluate the test results.9. Report test results.

Page 19: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

19

Assumptions of Rapid Testing

Testers have at least some control over the design of tests and some time to create new tests.One goal of testers is to find important bugs fast.Testing is done under conditions of uncertainty and severe time pressure.Testers have control over how they think and what they think about.Testers are, or become, very good at software testing.

Page 20: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

20

A Rapid Tester’s Mandate

To be able to test a productwhen it has to be tested right now,under conditions of uncertainty,

in a way that stands up to scrutiny.

Page 21: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

21

A Rapid Tester’s Mandate

Testers should always be thinkinglike expert testers.

Do that well, and they’ll beable to handle any testing situation.

Page 22: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

22

The Themes of Rapid Testing

Put the tester's mind at the center of testing.Learn to deal with complexity and ambiguity.Be a service to the project community, not an obstacle.Consider cost vs. value in all your testing activity.Develop your testing skills.Use heuristics to guide and structure your process.Diversify your testing and your test team.Your context should drive your choices, both of which evolve over time.

Page 23: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

23

-Idea-Idea

What Does Rapid Testing Look Like?Skill + Heuristics Makes Powerful Testers

1. Do this2. Then do this3. Then do this4. Then do this5. And then this…

This…

…not this.Hey! Testing is not

a clerical process.

A heuristic is a falliblemethod for solving

a problem.

Page 24: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

24

Rapid testing may not be exhaustive,but it is thorough enough and quick enough. It’s less work than ponderous testing. It might be less work than slapdash testing.

It fulfills the missionof testing.

How does Rapid Testing Compare with Other Kinds of Testing?

Management likes to talkabout exhaustive testing, butthey don’t want to fund it andthey don’t know how to do it.

You can always test quickly...

But it might be poortesting.

When testing is turned into an elaborate set of rote tasks,

it becomes ponderous without really being thorough.

Mor

e W

ork

&

Tim

e(C

ost)

Better Thinking & Better Testing(Value)

SlapdashMuch faster, cheaper,

and easier

PonderousSlow, expensive,

and easier

RapidFaster, less expensive,

still challenging

ExhaustiveSlow, very expensive,

and difficult

Page 25: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

25

Is Rapid Testing Agile?

Two cheers for AgilityWe love the idea that programmers write unit tests− that saves us from having to report silly bugs that could be, should

be caught early

We like the idea that programmers are even talking about testing

Testing is not merely rote confirmationTesting is also (and, I believe, more importantly) active investigation

Page 26: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

26

KEY IDEA

Page 27: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

27

Do You Test UnderTime Pressure?

You have less time than you’d like.Faster is better.Your plans are often interrupted.You are not fully in control of your time.

What is this like?

Page 28: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

28

Excellent rapid technical workbegins with you…

When the ball comes to you…Do you know you have the ball?

Can you receive the pass?

Do you know what yourrole and mission is?

Do you know whereyour teammates are?

Do you know your options?

Is your equipment ready?

Can you read the situation on the field?

Are you aware of the criticality of the situation?

Are you ready to act, right now?

Can you let your teammates help you?

Page 29: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

29

Rapid test teams are about diverse talents cooperating− We call this the elliptical team, as opposed to the well-rounded team.− Some important dimensions to vary:

− Technical skill− Domain expertise− Temperament (e.g. introvert vs. extrovert)− Testing experience− Project experience− Industry experience− Product knowledge− Educational background− Writing skill

− Diversity makes exploration far more powerful− Your team is more powerful because of your unique individual contribution

…but you don’t have to be great at everything.

Page 30: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

30

The Tester is at the Center

skill

procedureenvironment

supervisionThe greater the skill of

the tester, the lessprescribed procedure,

supervision, orfavorable environment

is required.

Page 31: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

31

Rapid Testing MindsetAssess Cost vs. Value

VALUE− Coverage. The value of the elements of the product that are exercised by the test.

− Power. The probability that if a problem exists in the area covered, the test will reveal it.

− Pop. The potential for it to reveal unanticipated problems as well as those you anticipate.

− Integrity. The degree to which the test can be trusted, because its results are valid and consistentacross multiple executions of that test.

− Technical Necessity. The degree to which the information we want to discover exists; and the degree to which that information matters.

− Business Necessity. The degree to which a test is expected to be performed by your clients.

COSTS (remember that time = money)− Opportunity Cost. Performing it may prevent you from doing other tests.

− Development. The cost of getting ready to perform the test.

− Execution. The cost of performing it as needed.

− Transfer. The cost of getting someone else ready to run that test.

− Maintenance. The cost of keeping it running.

− Accounting. The cost of explaining the test, justifying it, show that it was performed.

Page 32: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

32

Happy PathDocumentation TourSample Data TourVariability TourComplexity TourContinuous Use

A quick test is a cheap test that has some valuebut requires little preparation, knowledge,

or time to perform.

Cost vs. ValueTry quick tests as well as careful tests

InterruptionsUnderminingAdjustmentsDog PilingFeature InteractionsAsk for Help

Page 33: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

33

Input Constraint AttackClick FrenzyShoe TestBlink TestError Message HangoverKeyboard Tour

A quick test is a cheap test that has some valuebut requires little preparation, knowledge,

or time to perform.

Cost vs. ValueTry quick tests as well as careful tests

Folder TourResource StarvationMultiple InstancesCrazy ConfigsCheap ToolsBuild Your Own Tools

Page 34: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

34

Blink Tests

Informed by Malcolm Gladwell’s BlinkBlink tests are based on removing some information and using human pattern recognitionBlink test example: ProSum− thank you, Earl Everett!

Blink test example: Excel SheetBlink test example: another Excel Sheet

30

Page 35: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

35

Blink Precepts

Snap judgments are a central part of our decision-making processSnap judgments are vulnerable to corruption by forces we don’t recognize or understandSometimes we can improve the quality of our judgments by removing informationInstead of changing the decision-maker, change the context in which the decision is made

30

Page 36: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

36

Motivations for Quick Tests

Finding important bugs quicklyAchieving broader coverage quicklySmoke (or “build acceptance”) testingDeveloping, discovering, or refining a test planBuying time for more testingWhen investigation is more important than confirmation

Page 37: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

37

Thinking Scientifically

Ability to pose useful questions.Ability to observe what’s going on.Ability to describe what you perceive.Ability to think critically about what you know.Ability to recognize and manage bias.Ability to form and test conjectures.Ability to keep thinking despite already knowing.Ability to analyze someone else’s thinking.

40

Page 38: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

38

General Systems

systems are made up of parts comprising a whole, with dynamic connections and relationships between theminput, output, control, and feedback influence the behaviour of systemssystems can be decomposed into parts or subsystems, which are themselves systemssystems are components of larger systems

Page 39: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

39

General Systems Thinking

General systems thinking is a way of observing and determining the way things (tend to (tend to)) workA means of analyzing, mastering, and learning to live with complexityA means of simplifying problems in useful waysAny view of a system is necessarily a model− “All models are wrong; some are useful.”

− George Box− “The map is not the territory.”− “When the map and the territory disagree, believe the territory.”

− Jerry Weinberg, quoting the Swedish Army

Page 40: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

40

General Systems in Testing

The product is composed of functional componentsThe product runs on platforms that are also systemsThe product is part of a business processThe business process is part of the work done by an organizationThe organization is part of a communityThe community is part of a societyDo your tests reflect these ideas?− many testers stop at the first line!

Page 41: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

41

Heuristics:Generating Solutions Quickly

adjective:“serving to discover.”

noun:“a fallible method for finding a solution.”

“Heuristic reasoning is not regarded as final and strictbut as provisional and plausible only, whose purpose

is to discover the solution to the present problem.”- George Polya, How to Solve It

Page 42: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

42

Some Everyday Heuristics

It’s dangerous to drink and drive.A bird in hand is worth two in the bush.Nothing ventured, nothing gained.Sometimes people stash their passwords near their computers. Try looking there.Stores are open later during the Holidays.If your computer is behaving strangely, try rebooting. If it’s very strange, reinstall Windows.If it’s a genuinely important task, your boss will follow-up, otherwise, you can ignore it.

Page 43: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

43

How Heuristics Differ fromOther Procedures or Methods

A heuristic is not an edict. Heuristics work implicitly under the control of a competent practitioner.Heuristics are presumed fallible and context-dependent.Heuristics aid or focus an open-ended solution search.Heuristics can substitute for complete and rigorous analysis.

Page 44: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

44

KEY IDEA

Page 45: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

45

Oracles

An oracle is the principle or mechanismby which you recognize a problem.

“..it works”

“...it appeared at least once to meet some requirement to some degree.”

Page 46: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

46

All Oracles Are Heuristic

There is no single oracle that can tell us whether a program (or a feature) is working correctly at all times and in all circumstances.Oracles are fallible and context-dependent.Multiple oracles may increase our confidence, but even combinations of oracles are fallible.Oracles can be contradicted by other oracles.Recognizing a different problem usually requires a different oracle.A test designer doesn’t need to be aware of an oracle in advance of the observation, unless the test is designed to be run by rote.Any time you see a problem, you must be using an oracle…so what is it?

Page 47: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

47

General Oracles: Consistency

History: Present system is consistent with past versions of the same system.

Image: System is consistent with an image that the organization wants to project.

Comparable Products: System is consistent with comparable systems.

Claims: System is consistent with what people say it’s supposed to be.

User’s Expectations: System is consistent with what we think users want.

Product: Elements of the system are consistent with comparable elements within the same system.

Purpose: System is consistent with its purposes, both explicit and implicit.

Statutes: System elements are consistent with applicable laws.

Use consistency heuristics to support inferences about possible bugs.

Page 48: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

48

Was the file saved?

You’re testing Microsoft Word.You work with a file, save, and close it. You note that a file with the expected name appears in the expected folder.You reopen the file immediately, and the reopened document looks the same as it was just before you saved it.

Page 49: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

49

Does Word save files properly?MAYBE…

we’re opening an identical file from the wrong folder.the file wasn’t saved at all, and we’re opening an identical file from thisfolder.we weren’t warned, even though we were replacing an existing file.the file we opened was in the old Word file format, but we weren’t asked to save the file in the most recent Word file format.the file did get saved, but clobbered another file in the process.the new document did get saved, but the default filename that Word offered didn’t match the first few words of our file. the new document did get saved to the right filename, but left a copy of ~WRD####.tmp lying around on the disk.the file got saved, but it took an unacceptably long time to do it.the “protect document” features are reset, instead of preservedthe bit of the document that we looked at is okay, but the rest of it is messed up − we looked at the text, but we forgot to look at the graphics− some of the file contents got saved, but the macros didn’t.− some of the file contents got saved, but unauthorized new macros

got saved too.− some of the file got saved, but the change tracking information

didn’t get saved.− some of the file got saved, but the font data was saved incorrectly

somewhere in the file.− the font data got saved, but the colors didn’t.− all the font information got saved, but the styles didn’t.− the text got saved, but all the graphics were missing.− it looks identical to a casual glance, but on a pixel-by-pixel basis,

the graphics quality has degraded− the file didn’t save any auto-text entries.

the file got saved, but the metadata under the File / Propertiesmenu didn’t get savedthe file got saved, but the metadata in the file system is inconsistent with File / Propertiesthere was no room on the disk, so the save operation should havefailedif we saved it with a password, we didn’t get prompted for a password when we reopened the filethe file got saved as an .RTF file, but WordPad can’t read it; only this version of Word can read it.the file can be reopened, but not edited.the file got saved, but when we press Shift-F5, the cursor doesn’t return to the point where it was when the document was saved.you weren’t prompted to update a linked documentyou were not prompted to save the Normal template, even though the option to save the Normal template was checked.you can open the file from the document management system where you saved it, but you won’t be able to save subsequent changes.when you saved it, Word only let you save the file as a template.the saved file did not get listed in the Recent Documents folder.the saved file did not get added to the “most recently used” list on the File/Open menu.Word didn’t maintain case-sensitivity in saving the file name.the autorun macros contained in the document did not execute, even though you weren’t holding the Shift key.

Page 50: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

50

Simplicity vs. Complexity

Sometimes simple functions can conceal enormous complexity.Sometimes complex things can be simplified.

Let’s look at DiskMapperDiskMapper is a utility designed to display the relative amounts of space taken up by your files.A DiskMapper display can display hundreds of rectangles, each with a specific size and color.How can we look at a DiskMapper display and know that it is correct?

Page 51: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

51

DiskMapper

Page 52: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

52

Tests Are Oracles PLUS Coverage

“We haven't tried very hard to make it fail, and we haven't been running it very long or under very diverse conditions, but so far we haven't seen any failures, though we haven't

been looking too closely, either.”

According to Jerry Weinberg, “it works”

Page 53: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

53

KEY IDEA

Page 54: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

54

Coverage

There are as many kinds of coverageas there are ways to model the product.

StructureFunctionalDataPlatformOperationsTime

Product coverage is the proportion of theproduct that has been tested.

CapabilityReliabilityUsabilitySecurity

Scalability

PerformanceInstallability

CompatibilitySupportability

Testability

MaintainabilityPortability

Localizability

Page 55: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

Product Elements:Structural Coverage

platform

input outputTest what it’smade of.

Print testing example− Files associated with printing− Code modules that implement printing− Code statements inside the modules− Code branches inside the modules

Page 56: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

Product Elements:Functional Coverage

input output

platform

functionsTest what it does.

Print testing example− Print, page setup and print preview− Print range, print copies, zoom− Print all, current page, or specific range

functions

Page 57: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

Product Elements:Data Coverage

functions&

structureinput output

platform

Test whatit does it to.

Print testing example− Types of documents− Items in documents, size and structure of documents− Data about how to print (e.g. zoom factor, no. of copies)

Page 58: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

Product Elements:Platform Coverage

Test what itdepends upon.

Print testing example− Printers, spoolers, network behavior− Computers− Operating systems− Printer drivers

functions&

structureinput output

platform

Page 59: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

Product Elements:Operations Coverage

Test howit’s used.

Print testing example− Use defaults− Use realistic environments− Use realistic scenarios− Use complex flows

input output

platform

Page 60: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

Product Elements:Time Coverage

Test howit’s affected

by time.

Print testing example− Try different network or port speeds− Print one document right after another, or after long intervals− Try time-related constraints--spooling, buffering, or timeouts− Try printing hourly, daily, month-end, and year-end reports− Try printing from two workstations at the same time

input output

platform

Page 61: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

61

Sometimes your coverage is disputed…

“No user would do that.”

“No user I can think of, who I like,would do that on purpose.”

Who aren’t you thinking of?Who don’t you like who might really use this product?

What might good users do by accident?

Page 62: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

62

KEY IDEA

Page 63: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

63

Thirty-Six Testing Heuristics

CustomersInformationDeveloper relationsTeamEquipment & toolsScheduleTest ItemsDeliverables

StructuresFunctionsDataPlatformsOperationsTime

CapabilityReliabilityUsabilitySecurityScalabilityPerformanceInstallabilityCompatibilitySupportabilityTestabilityMaintainabilityPortabilityLocalizability

Function testingDomain testingStress testingFlow testingScenario testingClaims testingUser testingRisk testingAutomatic testing

15

Page 64: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

64

General Test Techniques

Function testing Domain testingStress testingFlow testingScenario testingUser testingRisk testingClaims testingAutomatic Testing

For any one of these techniques,there’s somebody, somewhere,

who believes that is theonly way to test.

Page 65: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

65

MISSION:The most important part

Find important problemsAssess qualityCertify to standardFulfill process mandatesSatisfy stakeholdersAssure accountability

Advise about QAAdvise about testingAdvise about qualityMaximize efficiencyMinimize timeMinimize cost

The quality of testing depends on which of thesepossible missions matter and how they relate.

Many debates about the goodness of testingare really debates over missions and givens.

Page 66: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

66

Good Enough Software...

…with what level of confidence?…to meet ethical obligations?…in what time frame?…compared to what?…for what purpose?…or else what?…for whom?

Perspectiveis

Everything

Page 67: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

67

Heuristic Model:The Four-Part Risk Story

Victim. Someone that experiences the impact of a problem. Ultimately no bug can be important unless it victimizes a human.

Problem: Something the product does that we wish it wouldn’t do.

Vulnerability: Something about the product that causes or allows it to exhibit a problem, under certain conditions.

Threat: Some condition or input external to the product that, were it to occur, would trigger a problem in a vulnerable product.

Someone may be hurt or annoyedbecause of something that might go wrong while operating the product,

due to some vulnerability in the productthat is exploited by some threat.

Page 68: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

68

KEY IDEA

Page 69: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

69

Test Design and Execution

How do you effectivelyguide a tester?

TestIdea

How do youarrive at excellent

test design?

Page 70: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

70

Exploratory Testing Defined

Exploratory testing is simultaneouslearning, test design, and test execution.

pure scripted freestyle exploratory

When I say “exploratory testing” and don’t qualify it, I mean anythingon the exploratory side of this continuum.

chartersvague scriptsfragmentarytest cases roles

Page 71: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

71

Exploratory Testing Is…

an approach to software testing…

that emphasizes the personal freedom and responsibility of each tester to continually optimize the value of his work…

by treating learning, test design and test execution as mutually supportive activities that run in parallel throughout the project.

(applicable to any test technique)

(optimize how?)

Page 72: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

72

Test DesignTesting to Learn vs. Testing to Search

Testing (primarily) to Learn− Forming a mental model of the product.− Learning what the product is supposed to do.− Inventorying the product elements you may want to test.− Looking at consistency relationships and trying various oracles.− Generating test data.− Considering testability and trying potentially useful tools.− Experimenting with different ways of testing it.− Reporting bugs that you find.

Testing (primarily) to Search− Using your detailed product knowledge, and any relevant tools, to

systematically exercise the product in every important way.− Using careful observation and good oracles to detect important bugs.− Modifying and diversifying your tests to make them more powerful.− Reporting bugs that you find.

Page 73: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

73

Test DesignTesting to Learn vs. Testing to Search

− Starting a new project.

− Seeing a new feature for the first time.

− Testing a product deeply to reveal important bugs.

− Investigating a particular bug.

− Re-testing a product after a change.

− Repeated execution of detailed procedural test scripts.

LL

LSSS

Compare these Situations:

L

L

S

S

S

Page 74: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

74

The Process of Test Design

Page 75: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

75

How to Start?Pick a Useful, Fun, or Easy Starting Point.

What do I need to produce for my client?What has already been done?

What kind of testingam I good at?

What obstacles threatenmy success?

What do I know?What do I need to know?

Do I have a product?What can I do with

the product?

Page 76: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

76

Contrasting Approaches

In scripted testing, tests are first designed and recorded. Then they may be executed at some later time or by a different tester.

In exploratory testing, tests are designed and executed at the same time, and they are not necessarily recorded, but may be.

Product

TestScripts

Test Ideas

Page 77: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

77

Contrasting Approaches

Scripted testing is likemaking a prepared speech.It is guided by pre-conceived ideas.

Exploratory testing is like having a conversation. It is self-guided.

Product

TestScripts

Test Ideas

Page 78: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

78

Contrasting Approaches

Scripted testing is aboutcontrolling test execution.

Exploratory testing is about improving test design.

Product

TestScripts

Test Ideas

Page 79: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

79

Contrasting Approaches

Scripted testing emphasizes predictability, accountability and decidability.

Exploratory testing emphasizes adaptability, credibility and learning. Product

TestScripts

Test Ideas

Page 80: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

80

The Cognitive Processof the Tester Matters!

The test is what youthink and do!

The test is not the test script.

Only a fraction of a skilled tester’s thinkingand behavior can be represented in a script.

Page 81: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

81

Why Script Anything?

IF any of the following are both true and important…− An agent needs to do something specific…− Those actions are known prior to the time of performing the action…− The agent would not otherwise know what to do unless told…− The agent might know what to do, but forget…− The script organizes information for more efficient, reliable access…− The script helps speed up execution…− The script helps coordinate the behavior of multiple agents…− The script helps multiple stakeholders come to a shared decision…− The scripts defines a pass/fail decision rule…− The script helps account for or assess what happens…− The script can be created at a reasonable cost…− The script is itself a product, not just a tool…

…THEN scripting an action MIGHT be a good idea.

Page 82: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

82

Why Work Without a Script?

An agent DOES NOT need to do something very specific…Specific actions are NOT known prior to the time of performing the action…The agent knows what to do WITHOUT being told…There’s LITTLE danger of forgetting what to do…When a script DOESN’T organize information more accessibly…When a script DOESN’T help speed up execution…When there are NO multiple agents, or they are ALREADY coordinated…When NO shared decision is necessary…When you DON’T need to decide pass/fail criteria in advance…When sufficient accounting and assessment ALREADY occurs…When a good script would be TOO EXPENSIVE or take TOO LONG…When the script is NOT otherwise a product, but just a tool.When unscripted testing would be far more productive.When a script is a hindrance to the future instead of a help.

Page 83: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

83

Blending Scripted & Exploratory

Generic scripts: specify general test procedures and apply them to different parts of a test coverage outline.Vague scripts: specify a test step-by-step, but leave out any detail that does not absolutely need to be pre-specified.Improvisation: have scripts, but encourage deviation from them, too.Fragmentary cases: specify tests as single sentences or phrases.Test Coverage Outline: use outline of product elements and have tester construct tests from it on the fly.Risk Catalog: specify types of problems to look for, then construct tests on the fly to find each one.Exploratory Charters: specify 90 minutes of testing in two sentences or less.Roles: Give each tester a standing role to test a certain part of the product. Leave the rest up to them.Heuristics: Train exploratory testers to use standardized test design heuristics.SBTM: Consider using Session-Based Test Management, a formalized method of exploratory test management. (http://www.satisfice.com/sbtm).

Page 84: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

84

Test Design and Execution

Guide testers with personal supervision and concise documentation of test ideas. Meanwhile, train them so that they can guide themselves and be accountable for increasingly challenging work.

TestIdeas

Achieve excellent test design by exploring different test designs

while actually testing Product

Productor spec

Page 85: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

85

KEY IDEA

Page 86: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

86

Common Claims AboutTest Documentation

It’s needed for new testers.It’s needed to deflect legal liability.It’s needed to share testing with other testers.It’s needed for “repeatability.”It’s needed for accountability.It forces you to think about test strategy.

Something is wrong with these claims. Can you see the problem?

Page 87: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

87

The First Law of Documentation

“That should be documented.”

“That should be documentedif and when and how it serves our purposes.”

Who will read it? Will they understand it?Is there a better way to communicate that information?

What does documentation cost you?

Page 88: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

88

Common Problems withTest Documentation

Distracts, and even prevents, testers from doing the best testing they know how to do.Authors don’t understand testing.Authors don’t own format.Templates obscure poor thinking. Full of fluff.Fluff discourages readers and increases cost of maintenance.No one knows documentation requirements.Too much formatting increases the cost of maintenance.Reference and tutorial information mixed together.Catering to rare audiences instead of everyday user.Disregard for the social life of information.Long term and short term goals conflict.

Page 89: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

89

Documentation: Product or Tool?

ProductTool

Audience:

Self Team Customers Regulators

Purpose:Recollection Organization Communication Demonstration

Paradigm:

5

Page 90: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

90

Rapid Testing Documentation

Determine where your documentation is on the continuum: product or tool?Keep your tools sharp and lightweightObtain consensus from others as to what’s necessary and what’s excess in productsEliminate unnecessary clerical work

Page 91: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

91

Accountability for Rapid Testing:Session-Based Test Management

Charter− A clear, concise mission for a session of testing

Time Box− 90-minute (+/- 30), long enough for setup and focused work; short

enough to make sure things don’t get off track

Reviewable Result− next slide!

Debriefing− conversation between tester and manager− problems, bugs and issues can be discussed− new risks can be identified− coaching and mentoring can happen

Page 92: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

92

Reviewable Results

Charter− #AREAS

Start TimeTester Name(s)Breakdown− DURATION− TEST DESIGN AND EXECUTION− BUG INVESTIGATION AND

REPORTING− SESSION SETUP− CHARTER/OPPORTUNITY

Data Files

Test NotesBugs− BUG

Issues− ISSUE

CHARTER-----------------------------------------------Analyze MapMaker’s View menu functionality and report on areas of potential risk.

#AREASOS | Windows 2000Menu | ViewStrategy | Function TestingStrategy | Functional Analysis

START-----------------------------------------------5/30/00 03:20 pm

TESTER-----------------------------------------------Jonathan Bach

TASK BREAKDOWN-----------------------------------------------

#DURATIONshort

#TEST DESIGN AND EXECUTION65

#BUG INVESTIGATION AND REPORTING25

#SESSION SETUP20

A scannable session sheet

Page 93: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

93

KEY IDEA

Page 94: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

94

How Do We Justify Rapid Testing?

Recognize that you’re using it already− but that you might not realize it…− …or if you realize it, you might not be admitting it…− …to important people…− …who deserve to know if you’re producing good

results by avoiding their mythodologies

Ask people what they value− more documentation, or more testing?− they might well value the documentation− if so, are they ways to make that more rapid?

Page 95: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

95

How Do We Justify Rapid Testing?

Do a bit of it using disposable timeTry a pilot project of a few hours a week− and then deploy some of those metrics, of which certain

managers seem to be so fond

Recognize that some of the best testing and development work is, at its heart, exploratory and investigative… and rapid

Page 96: Michael Bolton DevelopSense - STeP-IN Forum Bolton - A Rapid...1 A Rapid Introduction to Rapid Software Testing Michael Bolton DevelopSense STeP-IN Conference Bengalooru, India January

96

Additional References

Our Web sites and blogs− http://www.developsense.com− http://www.satisfice.com

My Test Connection column in Better Software MagazineLessons Learned in Software Testing− Cem Kaner, James Bach, Bret Pettichord

How To Break Software− James Whittaker

How To Break Software Security− James Whittaker

The Hacking Exposed Series− Various authors

Blink− Malcolm Gladwell

ANYTHING by Jerry Weinberg− Gerald M. Weinberg

Still more references available via email or my USB key