blinding: examples and approaches · everyone knows french wines are the best in the world, at...

Blinding: examples and approaches

September 1, 2015

Outine of talk

� The role of blinding in experimental design

� Examples of its importance and utility

� Historical examples

� Examples in medicine

� Examples in physics

� The relevance to forensic science

� A case study: fingerprints

� Proficiency vs. verification

The Gold Standard

Properly designed experiments should

� use controls

� these should be randomly allocated

� assessments should be “double-blind”

Classic examples illustrating the importance of these:

� The 1954 Salk polio vaccine trial

� The portacaval shunt

The Gold Standard

� use controls

The Gold Standard

� use controls

The Gold Standard

� use controls

The Gold Standard

� use controls

The Gold Standard

� use controls

The Gold Standard

� use controls

The 1954 Salk polio vaccine trial

This was a very well designed study:

� Over a million (!) young children participated

� randomized placebo controls: some children inoculated with asalt solution instead of vaccine

� double-blind: children, parents, and evaluating physicians keptin the dark as to who received vaccine vs. placebo.

The result: definitive proof of the efficacy of the vaccine.

(Paul Meier, “The biggest public health experiment ever: the 1954field trial of the Salk poliomyelitis vaccine”.)

The portacaval shunt: a study of studies

Design of experiment Enthusiasm

Marked Moderate None

No controls 24 7 1

Nonrandomized controls 10 3 2

Randomized controls 0 1 (3) 3

What is going on here? It is a well-understood phenomenon that

the human mind often sees what it expects to see, hopes to see, or

wants to see.

The portacaval shunt: a study of studies

Design of experiment Enthusiasm

Marked Moderate None

No controls 24 7 1

Nonrandomized controls 10 3 2

Randomized controls 0 1 (3) 3

What is going on here? It is a well-understood phenomenon that

the human mind often sees what it expects to see, hopes to see, or

wants to see.

The Salk polio vaccine trial

The preceding study illustrates why randomized controls are impor-tant, but what about blindness?

For most of us, it is something of a shock to be told that

competent and dedicated physicians must be kept in ig-

norance lest their judgments be colored by knowledge of

treatment status. We should keep in mind that it is not

deliberate distortion of findings by the physician that con-

cern the medical experimenter. It is rather the extreme

difficulty in many cases of making an uncertain decision

that, experience has shown, leads the best of investigators

to be subtly influenced by information of this kind.

The Judgement of Paris

Everyone knows French wines are the best in the world, at least until

May 24, 1976: two blind tastings are conducted in Paris by StevenSpurrier, a British purveyor of French wines.

The lineup:

� white: Chardonnays

� red: French Bordeaux and California Cabernet Sauvignon

In both case a California wine won!

(One of the judges, Odette Kahn, then tried – unsuccessfully – towithdraw her ballot.)

The lineup:

Franklin and Mesmerism

Franz Mesmer (1734–1815), German physician

1774: Proposes a theory of “animal magnetism” (mesmerism)an invisible natural force having curative powers

1777: Mesmer leaves Vienna for Paris (under a cloud)

1784: Louis XVI appoints four member commission to determine ifthe phenomenon of animal magnetism exists

Members include Lavoisier and Benjamin Franklin

Experiments conducted March–June, 1784

I t E P O R T

Dr. BENJAMIN FRANKLIN,

;ANO .OTHER. -

C 0 M M I S S I O N E K. S,

C H A R G E D B Y T H E

K I N G O F F R A N C E ,

WITH THE EXAMINATION OF THE

A N I M A L M A G N E T I S M ,

AS NOW PRACTISED AT PARIS.

T R A N S L A T E D PROM THF.FRENCH.

WITH AM

HISTORICAL INTRODUCTION.

L O N D O N :

HUNTED FOR J . JOHNSON, ( NO. 72 ) ST. PATFLV CHURCH YARD. I785.

An early example of experimental blinding

The commission reported that

of eleven persons that underwent the experiment, one

only had been sensible to the magnetism.

Thusthe business was only to protect her from the illusions

of the imagination, or at least to leave her imagination

without any thing to direct its operations.

The subject was

accordingly blindfolded and magnetised; the phenomena

no longer answered to the places towards which the mag-

netism was directed.

The subject was

N-rays

Background: major discoveries in physics – Roentgen (x-rays), 1895;Becquerel (radioactivity), 1896; Thompson (electrons), 1897.

Prosper-Rene Blondlot: distinguished physicist at the University ofNancy; corresponding member of the Academie des Sciences

Timeline:

� 1903: Blondlot announces discovery of n-rays, a supposed newform of radiation

� Many scientists publish papers on n-rays, even claims of priority!

� But some physicists such as Lord Kelvin express skepticism

N-rays

Timeline:

N-rays

Timeline:

N-rays

Timeline:

N-rays

Timeline:

Wood and n-rays: a “negative control”

Robert W. Wood (1868–1955), a physicist at Johns Hopkins, visitedBlondlot’s laboratory in 1904.

In dark room, W. removes a prism but Blondlot still sees n-rays.

Blondlot’s suspicious assistant repeats the experiment:

As soon as the light was lowered, I moved over towards

the prism, with audible footsteps, but I did not touch

the prism. The assistant commenced to turn the wheel,

and suddenly said hurriedly to Blondlot in French, “I see

nothing; there is no spectrum. I think the American has

made some derangement”. Where upon he immediately

turned up the gas and went over and examined the prism

carefully.

Some other historical examples

1. Mendel and Fisher

� 1866 Mendel publishes his famous paper on the basis of heredity

� 1936 R. A. Fisher: Mendel’s results are too good to be true!

� The hypothesis of the obliging assistant (!)

2. Clever Hans: a horse who could add (by taping out the answerwith his hoof) . . . as long as the audience didn’t the answer.

3. Millikan drop experiment: successive measurements drifted fromMillikan’s incorrect result to the current correct one.

In order to avoid such debacles, experiments must be properlydesigned.

Feynman on Millikan

We have learned a lot from experience about how to handle

some of the ways we fool ourselves. One example: Millikan

measured the charge on an electron by an experiment with

falling oil drops, and got an answer which we now know

not to be quite right. It’s a little bit off because he had

the incorrect value for the viscosity of air. It’s interesting

to look at the history of measurements of the charge of an

electron, after Millikan. If you plot them as a function of

time, you find that one is a little bit bigger than Millikan’s,

and the next one’s a little bit bigger than that, and the

next one’s a little bit bigger than that, until finally they

settle down to a number which is higher.

Feynman’s postmortem

Why didn’t they discover the new number was higher right

away? It’s a thing that scientists are ashamed of–this

history–because it’s apparent that people did things like

this: When they got a number that was too high above

Millikan’s, they thought something must be wrong–and

they would look for and find a reason why something might

be wrong. When they got a number close to Millikan’s

value they didn’t look so hard. And so they eliminated the

numbers that were too far off, and did other things like

Do statins have side-effects?

Statins (such as atorvastatin, aka Lipitor) are widely used to lowercholesterol and decrease the likelihood of cardiovascular disease.

These drugs are effective and inexpensive, but . . .

Some patients complain of side-effects such as muscle pain and lossof memory.

As a result more expensive alternatives are coming onto the market.

Is this really a problem?

Gina Kolata, NYT August 29, 2015

Dr. Peter Libby said that in his role as a physician, “the

patient is always right” But, he added, “as a scientist,

I find randomized, large-scale, double-blind studies more

persuasive than anecdote.”

Muscle pain

The statin trials, which involved tens of thousands of peo-

ple, found no more muscle aches in patients who took

statins than in those who took placebos.

Memory loss:

Dr. Jane Armitage (Oxford), studied memory problems in

20,000 patients randomly assigned to take a statin or a

placebo. “There was absolutely no difference.”

Muscle pain

Memory loss:

Muscle pain

Memory loss:

Arthroscopic surgery

“Sham surgery (placebo surgery) is a faked surgical intervention thatomits the step thought to be therapeutically necessary” (Wikipedia).

Example:“A controlled trial of arthroscopic surgery for osteoarthritis of theknee” (New England Journal of Medicine, 347 (2002), pp. 81–88).

Conclusion:In this controlled trial involving patients with osteoarthri-

tis of the knee, the outcomes after arthroscopic lavage or

arthroscopic debridement were no better than those after

a placebo procedure.

Some examples in physics

During the past decade, blind analysis has become a widely

used tool in nuclear and particle physics measurements. A

blind analysis avoids the possibility of experimenters bias-

ing their result toward their own preconceptions by pre-

venting them from knowing the answer until the analysis

is complete. There is at least circumstantial evidence that

such a bias has affected past measurements, and as exper-

iments have become costlier and more difficult and hence

harder to reproduce, the possibility of bias has become a

more important issue than in the past.

Joshua R. Klein and Aaron Roodman, Blind analysis in nuclear and

particle physics, Annual Reviews in Nuclear and Particle Science2005. 55:14–63

Some examples

� speed of light measurements: early measurements agree moreclosely with each other than the current accepted value, andthere are several values during the 1930–1940 period that differby several standard deviations from the true value.

� Einstein-de Haas experiment: experimental estimates off by afactor of 2, agreeing with an incorrect theory.

� fractional charges ±e/3(1980s): Luis Alvarez proposed that arandom charge be added to the data before being analyzed; theeffect then disappeared.

Some examples

The elephant in the room: confirmation bias

The definition:

Confirmation bias refers to our tendency to search for and

interpret information that confirms our prior belief

One simple example: people who believe arthritis pain is influencedby the weather will notice their pain more during bad weather, butmay pay less attention when the weather is fine.

There is a famous example of confirmation bias in forensic science.

The definition:

The Mayfield affair

� March 11, 2004: terrorist bomb attack, Madrid train station,resulted in 191 deaths and some 2,000 people injured

The Spanish authorities found a bag of detonators near the siteof the explosion with a fingerprint on it not matching any intheir databankAfter searching its fingerprint database, FBI located a possiblematch in the prints of Mr. Brandon Mayfield, an attorney inPortland, OregonFBI examiners concluded print a “100% positive identification,”and informed Spanish authorities on April 2, 2004.

The Mayfield affair

� � � � March 11, 2004: terrorist bomb attack, Madrid train station,resulted in 191 deaths and some 2,000 people injuredThe Spanish authorities found a bag of detonators near the siteof the explosion with a fingerprint on it not matching any intheir databank

After searching its fingerprint database, FBI located a possiblematch in the prints of Mr. Brandon Mayfield, an attorney inPortland, OregonFBI examiners concluded print a “100% positive identification,”and informed Spanish authorities on April 2, 2004.

The Mayfield affair

� � � � March 11, 2004: terrorist bomb attack, Madrid train station,resulted in 191 deaths and some 2,000 people injuredThe Spanish authorities found a bag of detonators near the siteof the explosion with a fingerprint on it not matching any intheir databankAfter searching its fingerprint database, FBI located a possiblematch in the prints of Mr. Brandon Mayfield, an attorney inPortland, Oregon

FBI examiners concluded print a “100% positive identification,”and informed Spanish authorities on April 2, 2004.

The Mayfield affair

� � � � March 11, 2004: terrorist bomb attack, Madrid train station,resulted in 191 deaths and some 2,000 people injuredThe Spanish authorities found a bag of detonators near the siteof the explosion with a fingerprint on it not matching any intheir databankAfter searching its fingerprint database, FBI located a possiblematch in the prints of Mr. Brandon Mayfield, an attorney inPortland, OregonFBI examiners concluded print a “100% positive identification,”and informed Spanish authorities on April 2, 2004.

Not so fast . . .

� � � The Spanish disagreed.

� April 13, 2004: Spanish authorities tell the FBI the match is“conclusively negative”.

� The FBI found 15 points of agreement for the fingerprint, theSpanish only 7

� April 21, 2004: FBI meet with Spanish officials in Madrid.

� They agree to disagree.

� May 6, 2004: Mayfield arrested on a material witness warrant.Fortunately for Mayfield, the Spanish persist and shortly afterannounce they’ve matched the print to an Algerian, OuhnaneDaoud.

� Mayfield soon freed and later settles for $2 million from US.

Not so fast . . .

The Spanish disagreed.

Not so fast . . .

The Spanish postspective

Pedro Luis Melida Lledo, head, SNP fingerprint unit:

They had a justification for everything . . . but I just

couldn’t see it.

You’re trying to match a woman’s face to a picture . . . but

you see that woman has a mole, and the face in the picture

doesn’t. Well, maybe it’s covered up with make-up, you

say. O.K., but the woman has straight hair and it’s curly

in the picture. Maybe the woman in the picture had a

permanent?

Proficiency and verification

Proficiency tests: unless blind . . .

� test for basic competence,

� NOT performance under field conditions.

Verification (as in ACE-V):

� goes to reproducibility (“reliability”),

� NOT “validity” of the procedure

Blindness is also very important in verification

(otherwise such a “second opinion” isn’t independent of the first)

A modest proposal

One could follow the IRS lead and institute a random fraction ofblind verifications

Selecting a return for audit does not always suggest that an

error has been made. Returns are selected using a variety

of methods, including:

Random selection and computer screening – sometimes

returns are selected based solely on a statistical formula.

The IRS finds this useful because

� Discover new facts(you can’t learn about returns you don’t look at)

� Discourages non-compliance

A modest proposal

A concluding word from Richard Feynman

We’ve learned from experience that the truth will come

out. Other experimenters will repeat your experiment and

find out whether you were wrong or right. Nature’s phe-

nomena will agree or they’ll disagree with your theory.

And, although you may gain some temporary fame and

excitement, you will not gain a good reputation as a sci-

entist if you haven’t tried to be very careful in this kind

of work. And it’s this type of integrity, this kind of care

not to fool yourself that is missing to a large extent in

much of the research in cargo cult science.

(1974 CalTech Commencement Address, emphasis added)

blinding: examples and approaches · everyone knows french wines are the best in the world, at...

Documents