analytics and witch doctoring: a cure for the black box mentality
DESCRIPTION
Analytics and Witch Doctoring: A Cure for the Black Box Mentality. February 1, 2011 O’Reilly Strata Conference J.C. Herz, Batchtags LLC [email protected]. Analytics: An Intervention. Original Sin. Enterprise Pathologies. Critical Questions. Analytics: Occult Phenomenon. Very powerful - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/1.jpg)
Analytics and Witch Doctoring: A Cure for the Black Box
Mentality
February 1, 2011
O’Reilly Strata Conference
J.C. Herz, Batchtags LLC
![Page 2: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/2.jpg)
Analytics: An Intervention
• Original Sin
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
Enterprise Pathologies Critical Questions
![Page 3: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/3.jpg)
Analytics: Occult Phenomenon
QuickTime™ and a decompressor
are needed to see this picture.
Very powerful
Don’t understand it
Practitioners possess arcane knowledge
![Page 4: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/4.jpg)
Alchemy
QuickTime™ and a decompressor
are needed to see this picture.
![Page 5: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/5.jpg)
SecretAlgorithms
QuickTime™ and a decompressor
are needed to see this picture.
![Page 6: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/6.jpg)
Greek Letters Are Your Kryptonite
QuickTime™ and a decompressor
are needed to see this picture. QuickTime™ and a decompressor
are needed to see this picture.
![Page 7: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/7.jpg)
High Status Helplessness
• If you understood the technology, you’d be one of those people whose job it is to make technology work.
• You know, underlings
![Page 8: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/8.jpg)
Executive ADD - Kaching!
• Re-starts are where consulting shops make their money
QuickTime™ and a decompressor
are needed to see this picture.
![Page 9: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/9.jpg)
Mid-Life Crisis
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
![Page 10: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/10.jpg)
Shiny Pebble Syndrome
• Infoviz Porn: Visualization with no use case– Ex: Social Network visualization. Why?– START with a use case and work
forward
• Demo Envy: just because it looks slick doesn’t mean it’s possible, or even advisable, to pipe your data into it.
QuickTime™ and a decompressor
are needed to see this picture.
![Page 11: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/11.jpg)
A Ballad of Spectacular Information Display
• Time Magazine 1976• Telex text routing:
information off the wire goes to terminals, properly foldered
• Z8 terminal display awes executives
• Pneumatic system not eliminated
QuickTime™ and a decompressor
are needed to see this picture.
![Page 12: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/12.jpg)
Technical Reality vs. Leadership Attention Span
QuickTime™ and a decompressor
are needed to see this picture.
![Page 13: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/13.jpg)
Half Ass Syndrome
• Halfway into the project, jump off into the next problem.
• Haven’t refined results or hypothesis
• Failure blamed on technology, but it’s really loss of interest and desire for instant gratification
![Page 14: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/14.jpg)
Shelfware Syndrome
• The guy who was driving the program left...
• Approach-Avoidance conflict --> pilot-itis
• A US agency has $30M of software that hasn’t been installed…some of it with maintenance contracts.
• Base Model vs. Fully Loaded– One enterprise bought $12M worth of
Autonomy before figuring out that the add-ons they needed would be another $22M.
![Page 15: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/15.jpg)
Customization Before Testing
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
![Page 16: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/16.jpg)
Critical Question: What is the Validation Test?
• Formulating the validation test keeps both the customer and the developer focused - and honest
• Suggest pay for performance, and see if the developer or vendor freaks out.
• Make sure validation is ongoing - in case the ground is shifting
![Page 17: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/17.jpg)
Data Due Diligence & Auditing
QuickTime™ and a decompressor
are needed to see this picture.
![Page 18: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/18.jpg)
Shame
QuickTime™ and a decompressor
are needed to see this picture.
Shame
![Page 19: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/19.jpg)
Ugly Babies
QuickTime™ and a decompressor
are needed to see this picture.
![Page 20: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/20.jpg)
Critical Question: Data Quality
• How complete is it?
– Ex: 600 custom fields, only two have more than 50% coverage
• How accurate is it? How do you know?
• How consistent is it?
– Good test: make three calls to different parts of the company, to get an answer to a factual question that doesn’t require calculation.
![Page 21: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/21.jpg)
Critical Question: Half-Life of Data
• How long is the data accurate?
• How long is the data useful?
QuickTime™ and a decompressor
are needed to see this picture.
![Page 22: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/22.jpg)
Critical Question: Real World Context
• Without real world data, “behavioral” metrics are misleading
• Where is the transactional data that validates insights from non-transactional data?
• How would you prove the magic analytics WRONG?
![Page 23: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/23.jpg)
• Are you prepared to spend painful amounts of money cleaning up your data?
• Crack heads if people don’t share data?
• Make business units accountable for their data?
• Play hardball to make sure data is not stored in single-application proprietary formats?
Data: Gut Check
![Page 24: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/24.jpg)
Critical Questions: Workflow
• What workflow changes will this proposed capability require?
• People hate changing their workflow, even if it’s an improvement
• Never attribute to stupidity what can be attributed to laziness
• What is your plan for changing workflow? How do you enforce it?
![Page 25: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/25.jpg)
The perfect application that no-one uses is still worthless
![Page 26: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/26.jpg)
Process
QuickTime™ and a decompressor
are needed to see this picture.
![Page 27: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/27.jpg)
Politics
QuickTime™ and a decompressor
are needed to see this picture.
![Page 28: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/28.jpg)
What are you prepared to do?
QuickTime™ and a decompressor
are needed to see this picture.
![Page 29: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/29.jpg)
Critical Question: Consequences
• What actions are you willing to take on the basis of validated analytic insight?– Change your product?– Change your marketing budget?– Change people’s job descriptions?– Re-allocate R&D budgets?
• What actions are you not willing to take?
![Page 30: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/30.jpg)
Stakes
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
![Page 31: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/31.jpg)
Critical Question: Tempo
• How fast will a decision be made on the basis of analytic insight?
• Quarterly?
• Daily?
• Within seconds?
• Milliseconds?
• Never?
• Realtime vs. Continuous vs. Batch
![Page 32: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/32.jpg)
Precision vs. Accuracy
When precision exceeds accuracy, you’re setting yourself up for analytic failure
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.
![Page 33: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/33.jpg)
OODA Loop
Which of these does an analytic tool/technology do?
QuickTime™ and a decompressor
are needed to see this picture.
![Page 34: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/34.jpg)
Before You Rip ‘n’ Replace:
What is the exit cost of this technology?
Does “turnkey” mean monoculture?
![Page 35: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/35.jpg)
Business Payoff vs. Intellectual Appeal
QuickTime™ and a decompressor
are needed to see this picture.
Social Network Analysis
Operations Research
Market Segmentation
Competitive Intelligence
Pilots to test new analyst tools with tiny amounts of generic data
360º Lead Scoring
Validate Marketing Effectiveness
![Page 36: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/36.jpg)
Questions?
• J.C. Herz [email protected] (202) 213-3151
QuickTime™ and a decompressor
are needed to see this picture.
![Page 37: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/37.jpg)
Base Model vs. Fully Loaded
![Page 38: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/38.jpg)
“Never attribute to stupidity what can be explained by
laziness.”
![Page 39: Analytics and Witch Doctoring: A Cure for the Black Box Mentality](https://reader036.vdocument.in/reader036/viewer/2022081519/56814400550346895db09286/html5/thumbnails/39.jpg)
Ugly Babies & Pretty Babies
QuickTime™ and a decompressor
are needed to see this picture.
QuickTime™ and a decompressor
are needed to see this picture.