stupid metrics tricks.ppt - rbcs,...

14
Stupid Metrics Tricks And How to Avoid Them

Upload: lamcong

Post on 27-Apr-2018

222 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: Stupid Metrics Tricks.ppt - RBCS, Incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfStupid Metrics Tricks ... Ever seen someone do something stupid with metrics? Sure

Stupid Metrics TricksAnd How to Avoid Them

Page 2: Stupid Metrics Tricks.ppt - RBCS, Incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfStupid Metrics Tricks ... Ever seen someone do something stupid with metrics? Sure

I t d tiIntroductionEver seen someone do something Ever seen someone do something stupid with metrics?Sure ya have!T d ’ l k t t id Today, we’re gonna look at stupid stuff people do with metrics

What are the most common metrics i t k ? mistakes?

Why are they mistakes? Why do people make these mistakes? A ki h i k ? Are you making these mistakes? Why use metrics at all, when there are so many mistakes?

And, of course, we’ll look at ways to implement

Stupid Metrics Tricks Page 2

metrics that aren’t stupidwww.rbcs-us.com

Copyright (c) RBCS 2016

Page 3: Stupid Metrics Tricks.ppt - RBCS, Incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfStupid Metrics Tricks ... Ever seen someone do something stupid with metrics? Sure

Y Li ht Up M LifDo you find this slide

You Light Up My LifeDo you find this slide easier to read when the font is lighter or darker?gOr do you find that the changing brightness of the font attracts your g yattention?You are a victim of the Hawthorne effectWhat gets measured gets doneg g

Stupid Metrics Tricks Page 3www.rbcs-us.com

Copyright (c) RBCS 2016

Page 4: Stupid Metrics Tricks.ppt - RBCS, Incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfStupid Metrics Tricks ... Ever seen someone do something stupid with metrics? Sure

A d Wh t D ’t G t M d D ’t!And What Doesn’t Get Measured, Doesn’t!As this figure shows, As this figure shows, people can get distractedfrom important factsSome people measure their worth in followers, likes, retweets, etc.likes, retweets, etc.How many count whether they cross the street without

i hi b ?getting hit by a car?What happens if you measure things that are less important than what you don’t measure?less important than what you don t measure?

Stupid Metrics Tricks Page 4www.rbcs-us.com

Copyright (c) RBCS 2016

Page 5: Stupid Metrics Tricks.ppt - RBCS, Incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfStupid Metrics Tricks ... Ever seen someone do something stupid with metrics? Sure

S ti Y ’ P hi St iSometimes You’re Pushing on a StringWhat if you attach yrewards or punishments to a metric?That’ll cause it to move in the right direction, yes?Well, it might just , g jfrustrate the hell out of people and cause no movement at allOr, you might mistake random variation for movement

Stupid Metrics Tricks Page 5www.rbcs-us.com

Copyright (c) RBCS 2016

Page 6: Stupid Metrics Tricks.ppt - RBCS, Incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfStupid Metrics Tricks ... Ever seen someone do something stupid with metrics? Sure

D i ’ R d B d E p i tDeming’s Red Bead ExperimentNeed six volunteer operators, three teams of one p ,bead selector and one bead counter eachOperators are issued a paddle with nine bead gaps and a bucket with mixed beadsand a bucket with mixed beadsOperators are to separate white beads from a mix of red and white beadsOperators are to submit one paddle every 15 Operators are to submit one paddle every 15 seconds, 10 paddles totalBead counters are to note any red beads or gaps in the paddle submitted by the selectorthe paddle submitted by the selectorScore per paddle is 1 point per white bead, 0 points per empty gaps, and -2 points per red beadT ith th t i t i iTeam with the most points wins a prize

Stupid Metrics Tricks Page 6www.rbcs-us.com

Copyright (c) RBCS 2016

Page 7: Stupid Metrics Tricks.ppt - RBCS, Incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfStupid Metrics Tricks ... Ever seen someone do something stupid with metrics? Sure

P t th Li i th C tPut the Lime in the CoconutQuick which shows a lower

3030

Quick, which shows a lower temperature?Are you sure?

10 10

2020

3030Are you sure?This illustrates a common mistake made by many

0 0

10 10y ytesters, test managers, and other managersOne example: taking the test pyramid literally rather than metaphorically

Stupid Metrics Tricks Page 7www.rbcs-us.com

Copyright (c) RBCS 2016

Blue Redthan metaphorically

Page 8: Stupid Metrics Tricks.ppt - RBCS, Incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfStupid Metrics Tricks ... Ever seen someone do something stupid with metrics? Sure

Ri f th M hiRise of the MachinesIt’s not unusual for people to have a goal of 100% test automationDoes this make sense?Does this make sense?Does this mean you don’t test areas that aren’t automatable?automatable?Does this mean you automate tests that run rarely?Is testing that consists entirely of Is testing that consists entirely of verification (with no validation) really complete?Who’s wagging whom?Who s wagging whom?

Stupid Metrics Tricks Page 8www.rbcs-us.com

Copyright (c) RBCS 2016

Page 9: Stupid Metrics Tricks.ppt - RBCS, Incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfStupid Metrics Tricks ... Ever seen someone do something stupid with metrics? Sure

I t bl ? I l t?Inscrutable? Irrelevant?Two questions from Two questions from assessments:

Do you understand the test results?test results?Do the test results help you do your job?

Common answers:No and noY b t nYes, but no

Always be able to answer the “so what” questionq

Stupid Metrics Tricks Page 9www.rbcs-us.com

Copyright (c) RBCS 2016

Page 10: Stupid Metrics Tricks.ppt - RBCS, Incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfStupid Metrics Tricks ... Ever seen someone do something stupid with metrics? Sure

Th W t C f R D tThe Water Cannon of Raw DataSome tools make it easy to deliver too much data yand low/no informationHow often have you seen 20 page test reports, slicing and dicing test case and bug metrics?slicing and dicing test case and bug metrics?Does that actually answer key questions like:

What’s been tested and what hasn’t?and what hasn t?What is still at risk?If we release now, what can we have confidence in and what can’t we?

These are the kinds of questions management q gactually cares about

Stupid Metrics Trickswww.rbcs-us.com

Copyright (c) RBCS 2016 Page 10

Page 11: Stupid Metrics Tricks.ppt - RBCS, Incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfStupid Metrics Tricks ... Ever seen someone do something stupid with metrics? Sure

S Wh B th ith M t i ?So Why Bother with Metrics?So, with all these potential mistakes, why bother?, p , yBecause it’s better to make decisions based on facts, not opinions, first impressions, opinions, first impressions, and common senseHuman cognition and humanpsychology do not lend psychology do not lend themselves to good choices without the aid of factsIgnorance and bad ideasIgnorance and bad ideasthrive in low-fact settings Pick one: proof by assertion or proof by evidence?or proof by evidence?

Stupid Metrics Tricks Page 11www.rbcs-us.com

Copyright (c) RBCS 2016

Page 12: Stupid Metrics Tricks.ppt - RBCS, Incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfStupid Metrics Tricks ... Ever seen someone do something stupid with metrics? Sure

Wh t’ N d d t G th G d M t iWhat’s Needed to Gather Good MetricsStart with the objectives, then derive metrics for Start with the objectives, then derive metrics for effectiveness, efficiency, and satisfactionKeep the metrics set concise, relevant, and actionableEnsure proper tool setup to gather the metricsP id t i i i th f th t lProvide proper training in the use of the toolsRemove any direct or indirect incentives or nudges to corrupt datanudges to corrupt dataTailor the metrics and their presentation to each audience

Stupid Metrics Tricks Page 12www.rbcs-us.com

Copyright (c) RBCS 2016

Page 13: Stupid Metrics Tricks.ppt - RBCS, Incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfStupid Metrics Tricks ... Ever seen someone do something stupid with metrics? Sure

C l iConclusionsThe very act of measurement can changeThe very act of measurement can changebehaviorMeasurement also re-directs attentionEven well-intentioned metrics can be misused, with dire resultsB f th i t k id Be aware of the mistakes so you can avoid themBut don’t reject metrics simply because it’s But don t reject metrics simply because it s hard to use them rightInstead, work diligently and methodically

d i i lto put good metrics in placeStupid Metrics Tricks Page 13

www.rbcs-us.com Copyright (c) RBCS 2016

Page 14: Stupid Metrics Tricks.ppt - RBCS, Incrbcs-us.com/site/assets/files/1391/stupid_metrics_tricks-1.pdfStupid Metrics Tricks ... Ever seen someone do something stupid with metrics? Sure

T C t t RBCSFor over 20 years, RBCS has delivered consulting, training, and expert services to clients, helping them with software and hardware testing Employing the industry’s most

To Contact RBCShelping them with software and hardware testing. Employing the industry s most experienced and recognized consultants, RBCS advises its clients, trains their employees, conducts product testing, builds and improves testing groups, and hires testing staff for hundreds of clients worldwide. Ranging from Fortune 20 companies to start-ups, RBCS clients save time and money through improved product development decreased tech support clients save time and money through improved product development, decreased tech support calls, improved corporate reputation and more. To learn more about RBCS, visit www.rbcs-us.com.Address: RBCS, Inc.

31520 Beck Road31520 Beck RoadBulverde, TX 78163-3911USA

Phone: +1 (830) 438-4830E il i f @ bE-mail: [email protected]: www.rbcs-us.comTwitter: @RBCS, @MisterSDET, @LaikaTestDog Facebook: @TestingImprovedbyRBCS

Stupid Metrics Tricks Page 14

LinkedIn: https://www.linkedin.com/in/rex-blackYouTube: https://www.youtube.com/user/RBCSINC

www.rbcs-us.com Copyright (c) RBCS 2016