starcanada 2013 keynote: lightning strikes the keynotes
DESCRIPTION
Throughout the years, Lightning Talks have been a popular part of the STAR conferences. If you’re not familiar with the concept, Lightning Talks consists of a series of five-minute talks by different speakers within one presentation period. Lightning Talks are the opportunity for speakers to deliver their single biggest bang-for-the-buck idea in a rapid-fire presentation. And now, lightning has struck the STAR keynotes. Some of the best-known experts in testing—Jon Bach, Michael Bolton, Fiona Charles, Janet Gregory, Paul Holland, Griffin Jones, Keith Klain, Gerard Meszaros, and Nate Oster—will step up to the podium and give you their best shot of lightning. Get ten keynote presentations for the price of one—and have some fun at the same time.TRANSCRIPT
Featuring
Michael BoltonJon Bach Fiona Charles
Griffin Jones
Keith Klain Gerard Meszaros
Janet Gregory Paul Holland
Nate Oster
Michael Bolton
Attention,Deficit,Disorder.
Michael Bolton
http://www.developsense.com
aircanada.com
aircanada.com
Error MessageVia Rail, between Montreal and Toronto, 2007
But I can’t contact my… oh, never mind.
Error MessageVia Rail, between Montreal and Toronto, 2007
Hyatt Regency, 2008
If you can’t do math, it’s a nickel extra.
Hyatt Regency, 2008
Why you shouldn’t let an unsupervised algorithm choose your sponsored links (1).
Vimeo’s Web PageSpring 2010
Why you shouldn’t let an unsupervised algorithm choose your sponsored links (2).
Vimeo’s Web PageSpring 2010
Why you shouldn’t let an unsupervised algorithm choose your sponsored links (3).
Vimeo’s Web PageSpring 2010
Google Chrome
Google Calendar Update
OK, fine.
Don’t Know Who This Was
Adobe Acrobat
Adobe Acrobat
Microsoft Outlook
Intuit Quicken
Intuit Quicken
Intuit Quicken
Here’s another err.
Intuit Quicken
ibm.com
Windows
I think I’ll shoot my own trouble for now.
Windows
Windows
I don’t know!
Windows
iTunes
Testing is more than checking.
Welcome to the new world.
“The World of Heineken”Heineken Experience, Amsterdam, 2006
Paul Holland
Functional Specification Blinders - My Experiences with Creativity
Paul Holland Test Consultant and Teacher at Testing Thoughts
Test Idea Creation
• A “Typical” method of developing test cases or test ideas:
1. Start by reading functional specification2. Write down test ideas (or create TCs)3. Try to think of other ways to test product4. If you think of any, write those down too5. When product is available, execute test ideas6. While testing try to think of more test ideas
April, 2013 ©2013 Testing Thoughts 34
What I Do on my Teams
• Over the past five years as a test manager at Alcatel-Lucent, I would reverse two steps:
1. Think of all ways you can to test product2. Write down test ideas3. Only then you read the functional specification4. Write any new ideas down5. When product is available, execute test ideas6. While testing try to think of more test ideas
April, 2013 ©2013 Testing Thoughts 35
How To Model Without FS
• Oh no! I can’t read the functional specification. How can I possibly think of test ideas?
1. If the product is available, then play with it2. If not, then talk to designers, architects,
marketing, customers, project managers, etc.3. Find out why the new feature/product is being
developed and how it will be used4. Play with competitive/similar products/features5. Learn as much as you can at a high level
April, 2013 ©2013 Testing Thoughts 36
Justification
• For years I had asked my team to think of their own test ideas before reading the functional specification
• I now have (statistically invalid) proof that reading the specification first can stifle individual creativity and limit the testing ideas that are generated
April, 2013 ©2013 Testing Thoughts 37
Experience Report
• During a modeling exercise of the Rapid Software Testing course November, 2012:
– Students were asked to identify all the dimensions of an object that may be relevant to testing it
April, 2013 ©2013 Testing Thoughts 38
Experience Report
• Students typically start with intrinsic dimensions:– Height, width, material, volume, etc.
• They then progress to relative dimensions:– Durability, dishwasher safe, contents, place of
manufacture, where it is now, where it has been, etc.
April, 2013 ©2013 Testing Thoughts 39
Experience Report
• Some students mention “conformance with standards”
• There is an ISO specification for a specific version of this object– A few have found this ISO specification exists– One student of mine found the spec and read it
April, 2013 ©2013 Testing Thoughts 40
Experience Report
• He found the spec BEFORE he had tried to think of his own dimensions
• After he read the spec, he could not think of ANY other dimensions
• In other exercises, this student showed good levels of creativity
• In this exercise, his creativity had been completely stifled by putting on the “specification blinders”
April, 2013 ©2013 Testing Thoughts 41
Summary
• Try to think of your own test ideas BEFORE reading the functional specification, previously existing test cases, or any other rigid documentation that may compartmentalize your thoughts
• Do not stifle your creativity. The functional specification will be waiting for you when you have done some of your own thinking
April, 2013 ©2013 Testing Thoughts 42
April, 2013 ©2013 Testing Thoughts 43
Griffin Jones
WRESTWorkshop on Regulated Software Testing
Software subject to review by an internal or external regulatory body
Purpose and Format• Share and generate ideas and
techniques• Provide a forum for people
interested in the topic• Participation is free and open to all• LAWST-style rules of engagement
Workshop Structure• Facilitated • Series of experiential
presentations and group discussions
• Atmosphere is collaborative, supportive and constructive
• Focus on the practical and useful
More Information• Next WREST: May 3rd 2013
Hosted by SQE at STAREAST
• Contact:Karen N. Johnson, John McConda, Griffin Jones
• Website: wrestworkshop.com
Gerard Meszaros
Premise:
Automated Tests are a Self-Verifying Behavior Specification
Because:
They Tell us What the System Should Continue to Do
After any changes or bug fixes we apply
The Challenge:
How to Avoid Creating
Tests/Checks/Specs
that Need Frequent Maintenance ?
My Belief:
Most Automated Tests Have Too Much Detail
Which is What Makes Them Require Frequent Maintenance
Need to specify at the right detail and scope
Behavior Specification at Right Level
Broad Narrow
Detail
Hig
h
Lo
w
Scope
Specify broad scope at minimum detail Specify most detailed req’ts at narrowest scope
Behavior Specification at Right Level
Rules &Algorithms
Multi-Use CaseWorkflows
Transactions(Use Cases)
IncompleteSpec
Too much detailUnmaintainable
Broad Narrow
Detail
Hig
h
Lo
w
Scope
Configurable Bank TX NotificationsExample:
Configure Notification
Rules
Suspend Notification
Resume Notification
AccountHolder
Process Transaction
TransactionSettlement
Testing Notifications - 1
Given: User and Accounts
Example:
When: Notification
Rule Configured
Then:Notification Rule
Configured and Active
Then:Notification Rule Configured and
Active
Testing Notifications - 2
Then: ExpectedNotifications
Medium Detail; Large ScopeToo much detail given the scope
Example:
When: The Transactions to be processed
Use Case: Process
Transactions
Issues With This Test
• Difficult to understand which TX should notify – because cause (rules) and effect (notification) are far
apart
• Only verifies one simple combination of rules– We will require many more tests to test all the other
combinations– Lots of repetition of workflow & data across test cases
• Tests will take a long time to run– Due to need to configure first, then run transaction
processing
Changing Level of Abstraction/Detail• Too detailed for verifying workflow and
• Too broadly scoped for checking the rules
• Need to Reduce Detail or Reduce Scope
Rules &Algorithms
Multi-Use CaseWorkflows
Transactions(Use Cases)
Too much detail
Unmaintainable
IncompleteSpec
Broad Narrow
Detail
Hig
h
L
ow
Scope
Reduce
Scope a Lot!Reduce
Both a bitRed
uce
Det
ail a
Lot
Specifying Notification WorkflowExample:
Broad Scope; Minimum Detail; Irrelevant Details Omitted!
Given: User &
Thresholds
When: Transactions
Are Processed
Then: Expected
Notifications
The overall workflow should be defined at a very high level
Changing Level of Abstraction/Detail• Need the detail to specify notification rules• Therefore, need to reduce scope
Rules &Algorithms
Multi-Use CaseWorkflows
Transactions(Use Cases)
IncompleteSpec
Too much detail
UnmaintainableBroad Narrow
Detail
Hig
h
Lo
w
Scope
Reduce
Scope a Lot!
Red
uce
Det
ail a
Lot
Reduce
Both a bit
Business Rule Specs
Configuration
Process Transaction
Threshold per Charge Type
High Detail; Narrow ScopeEasy to Understand
Example:
Given these rules
When we ask NotificationRequired
? with this transaction:
Then: The answer should be
NotificationLog
• Encapsulation of Rules in “Rules Component”– which can Expose a Test API
• Accept Rules via “Data Injection”– To Simplify Test Automation
• Making Automation a Dev’t Responsibility– To incent design-for-testability
System Under Test
Business Rules Tests Encourage:
NotificationRequired?
Do
Notification.
ProcessTransaction
ConfigureNotification
ThresholdNotification
Rules
Transaction
Interface
ConfigurationUser
Interface
NotificationRule Test
NotificationRules Fake
Gerard [email protected]
http://www.xunitpatterns.comhttp://ScopeVsDetail.gerardm.com
In Summary: Manage Scope vs. Detail
– to Avoid Unmaintainable Tests/Specs Provide the Tests/Specs to Dev’t Before
System Built– to allow Dev’t to automate the test – to ensure Design-for-Testability
Thank You!Jolt Productivity Award
winner - Technical Books
http://testingguidance.codeple
x.com/
Our Thanks To
Michael BoltonJon Bach Fiona Charles
Griffin Jones
Keith Klain Gerard Meszaros
Janet Gregory Paul Holland
Nate Oster