maybe we don’t have to test it
DESCRIPTION
Testers have been taught they are responsible for all testing. Some even say “It’s not tested until I run the product myself.” Eric Jacobson thinks this old school way of thinking can hurt a tester’s reputation and—even worse—may threaten team success. Learning to recognize opportunities where you may NOT have to test can eliminate bottlenecks and make you everyone’s favorite tester. Eric shares eight patterns from his personal experiences where not testing was the best approach. Examples include patches for critical production problems that can’t get worse, features that are too technical for the tester, cosmetic bug fixes with substantial test setup, and more. Challenge your natural testing assumptions. Become more comfortable with approaches that don’t require testing. Eliminate waste in your testing process by asking, “Does this need to be tested? By me?” Take back ideas to manage not testing including using lightweight documentation for justification. Not testing may actually be a means to better testing.TRANSCRIPT
![Page 1: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/1.jpg)
(Maybe We Don’t Have to)
Test It
![Page 2: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/2.jpg)
Preparing For Houseguests(A Story)
![Page 3: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/3.jpg)
TheProblem
![Page 4: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/4.jpg)
![Page 5: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/5.jpg)
![Page 6: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/6.jpg)
![Page 7: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/7.jpg)
11
1 1 100
001 00
Tester
11 1
00
01000
10 0
1 0
1 1001 100 00 000 1 11
11 1
![Page 8: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/8.jpg)
Video of arrogant tester.
![Page 9: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/9.jpg)
Routines are safe.
![Page 10: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/10.jpg)
“If it exists, I want to test it”
James Bach
![Page 11: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/11.jpg)
“If it exists, I want to test it. (The only exception is if I have
something more important to do.)”James Bach
![Page 12: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/12.jpg)
![Page 13: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/13.jpg)
![Page 14: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/14.jpg)
11
1 1 100
001 001
1 10
00100
10 0
1 0
1 10
0
1 100 00 000 1 1
11
1
100
100
100
11
1 1 00
01 0010 0
1 101 1001
01
100
1
![Page 15: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/15.jpg)
Patterns
![Page 16: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/16.jpg)
1No repro steps.
![Page 17: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/17.jpg)
Bug106621:DimMedia returns error intermittently.
![Page 18: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/18.jpg)
How can I repro this?
Beats me.
![Page 19: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/19.jpg)
I’ll play around in QA.
Have fun.
![Page 20: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/20.jpg)
Um, no luck yet.
You’re wasting time.
![Page 21: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/21.jpg)
Can’t you debug it or something?
No.
![Page 22: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/22.jpg)
How will we know if it’s fixed before we
release to prod?
We won’t.
![Page 23: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/23.jpg)
![Page 24: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/24.jpg)
![Page 25: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/25.jpg)
![Page 26: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/26.jpg)
![Page 27: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/27.jpg)
2Things that don’t go to
production.
![Page 28: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/28.jpg)
![Page 29: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/29.jpg)
Feature 106218:As a developer, I would like to convert TypeMock unit tests to Microsoft Fakes…
![Page 30: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/30.jpg)
Feature 96120:
Create visualization of
service operation
dependencies from SM
data.
![Page 31: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/31.jpg)
Feature 96190:
Create service
latency reports to
more easily
troubleshoot
problems
![Page 32: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/32.jpg)
![Page 33: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/33.jpg)
3Patches for critical
production problemsthat can’t get worse.
![Page 34: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/34.jpg)
Bug 102927:
Users unable to rework today’s schedule with empty alternate.
![Page 35: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/35.jpg)
Bug 103789:CopyEngine.ApplyVirtualCopyBySchedule causing perpetual LogDataMart timeout.
![Page 36: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/36.jpg)
4Cosmetic changes with
timely test setup.
![Page 37: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/37.jpg)
Feature 103540:
The word “unknown” is
spelled incorrectly on
the alt time constraint
validation message.
![Page 38: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/38.jpg)
Would you bother to test this?
Yes No0
10
20
30
40
![Page 39: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/39.jpg)
5Straight forward
configuration changes.
![Page 40: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/40.jpg)
Feature 106863:
TVE DS: Disable SCTE
events for CNN & HLN
![Page 41: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/41.jpg)
![Page 42: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/42.jpg)
Bug 94337:SolverPlatform jobs throwing error while while trying to serialize parameter.
![Page 43: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/43.jpg)
Maximum number of items that can be serialized or deserialized in an object graph is '65536'
![Page 44: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/44.jpg)
Trust but
verify
![Page 45: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/45.jpg)
6Too technical for a non-
programmer to test.
![Page 46: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/46.jpg)
Feature 106221:
Janus TypeInitializer
should recover if Janus
Admin service is down
upon initial service call
after app pool reset.
![Page 47: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/47.jpg)
7Non-tester on loan.
![Page 48: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/48.jpg)
Feature 100251:Brand change notification cosmetic enhancements.
![Page 49: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/49.jpg)
Feature 95947:
As a module user I
would like the module
search to be resizable so
I can see longer criteria.
![Page 50: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/50.jpg)
Feature 94052:
As a member of the Proof
Group, I would like the
proof categories to be
more intuitive.
![Page 51: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/51.jpg)
8Inadequate test data
or hardware.
![Page 52: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/52.jpg)
Bug 105481:
CDC not properly re-
initializing after re-
initializing replication.
![Page 53: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/53.jpg)
Bug 106271:Database script - users unable to open schedule for TNT, 1/11/2013 due to orphaned comment.
![Page 54: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/54.jpg)
![Page 55: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/55.jpg)
9Bugs not logged by
you.
![Page 56: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/56.jpg)
Techniques
![Page 57: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/57.jpg)
I’m not testing this because...
![Page 58: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/58.jpg)
Are you crazy?
![Page 59: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/59.jpg)
Only a fool would bother
testing this.
![Page 60: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/60.jpg)
I’ll just mark it
“Tested” without doing
much.
![Page 61: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/61.jpg)
An endorsement without careful thought or personal investment in the outcome.
![Page 62: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/62.jpg)
1.Some degree of risk analysis took place.
2.The impact to production would be low.
![Page 63: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/63.jpg)
RR 7/13/12: There is no feasible test
that can be done for this prior to
deployment due to replication not
happening in downstream
environments. The worst case if this
does not work, is CDC will still remain
broken. There is no risk of data loss.
Passing this work item along so it can
be deployed to production.
![Page 64: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/64.jpg)
3 Questions to Start Asking:
1. Does this need to ever be tested?
2. Does it need to be tested by me?
3. Who cares if it doesn’t work?
![Page 65: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/65.jpg)
“NOT testing” may still leave room for testing by
a broader definition
![Page 66: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/66.jpg)
We may be…
• Regression testing/checking.
• Questioning our team.• Learning product
architecture.• Evaluating others’ tests.• Reviewing code.
![Page 67: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/67.jpg)
Instead of saying “NOT testing”,
try saying“testing something more important”.
![Page 68: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/68.jpg)
1. Spend more time on valuable testing.2. Look for “not-test” opportunities. 3. Use my 9 patterns and find your own.4. Introduce a code word.
![Page 69: Maybe We Don’t Have to Test It](https://reader036.vdocument.in/reader036/viewer/2022062418/55625fd3d8b42ae87d8b4a65/html5/thumbnails/69.jpg)
Video of revised arrogant tester.