modern studies of assessing the competence and credibility of human sources of evidence dave schum...
Post on 17-Jan-2018
220 Views
Preview:
DESCRIPTION
TRANSCRIPT
MODERN STUDIES OF ASSESSING THE COMPETENCEAND CREDIBILITY OF HUMAN SOURCES OF EVIDENCE
DAVE SCHUM
SCHOOL OF INFORMATION TECHNOLOGY AND ENGINEERING
SCHOOL OF LAW
GEORGE MASON UNIVERSITY
UNIVERSITY COLLEGE LONDON
JANUARY 12, 2007
2
SOME POINTS I WILL ADDRESS IN MY TALK
• I HAVE ALWAYS BELIEVED THAT ONE OF THE MOST INTERESTING AND DIFFICULT ISSUES IN PROBABILISTIC REASONING CONCERNS THE CREDIBILTIY OF HUMAN SOURCES OF TESTIMONIAL EVIDENCE.
• EARLY PROBABIILISTIC STUDIES OF THE “CREDIBILITY-TESTIMONY PROBLEM” ARE VERY INTERESTING BUT NOT PARTICULARLY HELPFUL.
• MY CONCERNS ABOUT ATTRIBUTES OF THE COMPETENCE AND CREDIBILITY OF HUMAN SOURCES.
• SOME THOUGHTS FROM EPISTEMOLOGY, LAW, AND COMMON EXPERIENCE SUGGEST WHAT COMPETENCE AND CREDIBILITY ATTRIBUTES OUGHT TO BE CONSIDERED.
•A MAJOR CONTRIBUTION FROM LAW: A LEGACY OF IMPORTANT QUESTIONS TO ASK ABOUT CREDIBILITY ATTRIBUTES.
• A RETURN TO PROBABILITY AGAIN IN THE ANALYSIS OF CHAINS OF REASONING IN WHICH CREDIBILITY CONSIDERATIONS FORM THE FOUNDATION FOR THESE CHAINS.
•THESE VARIOUS THOUGHTS ARE COMBINED IN SYSTEM I HAVE BEEN WORKING ON FOR QUITE SOME TIME TO ASSIST OUR INTELLIGENCE ANALYSTS DO A MORE CAPABLE JOB OF ASSESSING SOURCES OF HUMINT [HUMAN INTELLIGENCE].
3
EXAMPLE: GEORGE HOOPER, [1640 - 1727]. BISHOP OF BATH AND WELLS
STRENGTH OF BELIEF IN AN EVENT AFTER N REPORTS REPORTS OF IT OVER TIME, FROM WITNESSES WHO EACH HAVE CREDIBILITY P = [P]N
STRENGTH OF BELIEF IN AN EVENT PROVIDED SIMULTANEOUSLY BY N WITNESSES WHO EACH HAVE CREDIBITY P =
1 - (1 - P)N.
4
{H1, H2}, Any two mutually exclusive hypotheses.
{E, EC}
How important is event E?
How credible is our source of evidence E* about event E?
E*
5
I SPENT MANY YEARS STUDYING HOW THE FORCE OF EVIDENCE IS RELATED TO THE CREDIBILITY OF ITS SOURCES. HERE IS A BAYESIAN ANALYSIS OF THE FORCE OF EVIDENCE IN A SIMPLE CHAIN OF REASONING.
THIS REASONING CHAIN BEGINS BY ASKING TWO OBVIOUS QUESTIONS [IN A MOMENT WE WILL SEE THAT THERE ARE OTHER QUESTIONS WE SHOULD BE ASKING]:
HOW MUCH WEIGHT DOES EVIDENCE E* HAVE ON {H1, H2}?
NOTE THE DISTINCTION BETWEEN EVIDENCE E* AND THE ACTUAL OCCURRENCE OF EVENT E.
6
LP E H P E EH P E E H P E E HP E H P E EH P E E H P E E HE
c c
c c*
( | )[ ( *| ) ( *| )] ( *| )( | )[ ( *| ) ( *| )] ( *| )
=− +− +
1 1 1 1
2 2 2 2
EVENT IMPORTANCE AND RARENESS.
NOTE : WE CAN HOLD THE RATIO P(E|H1)/P(E|H2) CONSTANT BUT VARY THEIR DIFFERENCES TO INDICATE RARENESS OF E; FOR EXAMPLE:
P(E|H1) = 0.90; P(E|H2) = 0.09
P(E|H1) = 0.09; P(E|H2) = 0.009
P(E|H1) = 0.009; P(E|H2) = 0.0009
CREDIBILITY-RELATED INGREDIENTS.
P(E*|E -) IS CALLED A “HIT” PROBABILITY,
P(E*|EC -) IS CALLED A “FALSE-POSITIVE”
PROBABILITY.
BUT
BAYES' RULE SAYS WE HAVE LEFT OUT A VERY IMPORTANT ADDITIONAL LINK.
A LIKELIHOOD RATIO EXPRESSION FOR THE FORCE OF EVIDENCE:
WILL SHOW HOW RARENESS AND CREDIBILITY INTERACT.
Answers to the questions posed by the early probabilists regarding rare events.
{H1, H2}, Any two mutually exclusive hypotheses.
{E, EC}
How important is event E?
How credible is our source of evidence E* about event E?
E*
THE MISSING LINK; SHOULD WE ADD IT?
THIS MISSING LINK INVOLVES THE CONCEPT OF CONDITIONAL DEPENDENCE, THE MAJOR VEHICLE IN BAYES' RULE FOR CAPTURING EVIDENTIAL SUBTLETIES.
WHAT THIS MISSING LINK WILL ASK US TO EXAMINE IS WHETHER THE SOURCE’S HIT AND FALSE-POSITIVE PROBABILITIES ALSO DEPEND ON H1 OR H2. IF THEY DO, THIS WILL TELL US SOMETHING VERY INTERESTING ABOUT THE EVIDENCE IN THIS CASE.
WHAT IT WILL TELL US IS THAT THE FORCE OF EVIDENCE E* FROM THIS SOURCE CAN BE GREATER THAN THE FORCE OF KNOWING FOR SURE THAT EVENT E OCCURRED.
7
LP E H h f fP E H h f fE*
( | )[ ]( | )[ ]
=− +− +
1 1 1 1
2 2 2 2
LP E H h f fP E H h f fE*
( | )[ ]( | )[ ]
=− +− +
1
2
(1)
(2)
When hit and false probabilities do depend upon H1 and H2, we have:
When hit and false probabilities do NOT depend upon H1 and H2, we can sever the additional link, in which case we have:
8
WHAT I NEEDED TO BE ABLE DO WAS TO DECOMPOSE THE LINK BETWEEN E* AND E FOR TESTIMONIAL EVIDENCE IN ORDER TO CAPTURE ATTRIBUTES OF THE CREDIBILITY OF HUMAN SOURCES. BUT WHAT ARE THESE ATTRIBUTES?
MY SEARCH FOR THE ATTRIBUTES OF THE CREDIBILITY OF HUMAN SOURCES OF EVIDENCE:
THE “STANDARD ANALYSIS”OF KNOWLEDGE AND THE CHAIN OF REASONING IT SUGGESTS [controversial, but a good heuristic]
9
A SOURCE “KNOWS” THAT EVENT E OCCURRED IF:
E DID OCCUR,
THE SOURCE GOT NON-DEFECTIVE EVIDENCE THAT E OCCURRED, AND
THE SOURCE BELIEVED THE EVIDENCE THAT E OCCURRED.
THIS SOURCE NOW TELLS US THAT E OCCURRED BASED ON AN OBSERVATION HE/SHE ALLEGEDLY MADE. WE HAVE UNCERTAINTY ABOUT THESE THREE EVENTS.
OUR INFERENCE ABOUT EVENT E, BASED ON THIS SOURCE’S TESTIMONY E*
HOW GOOD WAS THE EVIDENCE? DID EVENT E OCCUR?
DID THE SOURCE BASE THIS BELIEF ON SENSORY EVIDENCE?
DOES THIS SOURCE BELIEVE THAT E OCCURRED?
THE SOURCE’S REPORT E* THAT EVENT E OCCURRED.
OBSERVATIONAL SENSITIVITY
OBJECTIVITY
VERACITY
HOW GOOD WAS THE EVIDENCE? DID EVENT E OCCUR?
DID THE SOURCE BASE THIS BELIEF ON SENSORY EVIDENCE?
DOES THIS SOURCE BELIEVE THAT E OCCURRED?
THE SOURCE’S REPORT E* THAT EVENT E OCCURRED.
SOURCES OF SUPPORT FOR THIS CREDIBILITY-RELATED CHAIN OF REASONING:
VERACITY
OBJECTIVITY
OBSERVATIONALSENSITIVITY
WIGMORE
WAS THIS EVIDENCE ADEQUATE?
10
WITNESS TESTIMONY BASED ON PERSONAL OBSERVATION
WITNESS DO NOT ALWAYS TESTIFY WHAT THEY BELIEVE
DID THE WITNESS BASE A BELIEF ON SENSORY EVIDENCE/
COMMON EXPERIENCE
A PERSON TELLS US WHAT HE/SHE OBSERVED
PEOPLE DO NOT ALWAYS BELIEVE THE THINGS THEY TELL US
PEOPLE DO NOT ALWAYS BELIEVE WHAT THEIR SENSES RECORD, BUT BELIEVE WHAT THEY EXPECT OR WISH TO HAVE OCCURRED
HUMAN SENSE ARE FALLIBLE, ESPECIALLY UNDER CERTAIN CONDITIONS
MORE ON VERACITY
11
WHAT DOES IT MEAN TO SAY THAT A PERSON IS TELLING THE TRUTH?• IN EARLY AND IN SOME MODERN STUDIES OF THE “CREDIBILITY-TESTIMONY PROBLEM”, A PERSON WAS SAID TO BE TRUTHFUL IF THE EVENT HE/SHE REPORTED ACTUALLY OCCURRED.
• THE OED DEFINES “VERACITY” AS: “THE CHARACTER OF SPEAKING THE TRUTH. IT GOES ON TO SAY THAT VERACITY IS IN CORRESPONDENCE WITH TRUTH OR FACTS; CORRECTNESS, ACCURACY.
• SOME TROUBLES HERE. THESE ACOUNTS CONFOUND VERACITY AND OTHER ATTRIBUTES OF THE CREDIBILITY OF HUMAN SOURCES.
• EXAMPLE: PERSON X TELLS US THAT EVENT E OCCURRED, BUT WE LATER FIND OUT FOR SURE THAT IT DID NOT OCCUR. QUESTION: WAS X LYING TO US? ANSWER: NOT NECESSARILY. PERSON X MAY HAVE BEEN MISTAKEN OR UNOBJECTIVE AS AN OBSERVER.
EVEN MORE ON VERACITY
WE WOULD NOT SAY A PERSON IS BEING UNTRUTHFUL IF THIS PERSON TOLD US SOMETHING THAT HE/SHE ACTUALLY BELIEVED.
A SAMPLE OF SOURCES:
• Sisela Bok [Lying: Moral Choice in Public and Private Life, 1989]: “When we undertake to deceive others intentionally, we communicate messages meant to mislead them, and make them believe what we ourselves do not believe”.
• J. H. Wigmore [Science of Judicial Proof, 1937]: “ A lie is the intentional introduction into another’s mind of a belief that is not in harmony with what the actor himself supposes to be the truth”.
• Thomas Paine [The Age of Reason, 1794]: “Infidelity does not consist in believing or disbelieving; it consists in professing to believe what he does not believe”.
• St. Augustine [Enchiridion]: “Every liar says the opposite of what he thinks in his heart, with purpose to deceive.
SO: VERACITY CONCERNS WHETHER A HUMAN SOURCE IS REPORTING WHAT HE/SHE BELIEVES TO BE TRUE.
12
ON OBJECTIVITY:THE OVERLOOKED CREDIBILITY ATTRIBUTE
13
I HAVE FOUND ALMOST NO STUDIES OF HUMAN SOURCE CREDIBILITY THAT ACKNOWLEGE THE IMPORTANCE OF HOW AN OBSERVER FORMED HIS/HER BELIEF ABOUT THE EVENT BEING REPORTED.
• WAS IT BASED ON SENSORY EVIDENCE OR ON WHAT THE OBSERVER EXPECTED OR WISHED TO OCCUR OR NOT TO OCCUR.
• THE OED SAYS THAT AN OBJECTIVE OBSERVER IS ONE WHO IS ABLE TO PRESENT OR VIEW FACTS UNCOLOURED BY FEELINGS, OPINIONS, OR PERSONAL BIAS.
• MAJOR PROBLEM: OUR BELIEFS ARE SUPPLE OR ELASTIC; THEY MAY CHANGE OVER TIME. SO, WE HAVE TO ASK WHETHER THIS SOURCE’S BELIEFS ARE THE SAME NOW DURING THIS PERSON’S TESTIMONY AS THEY WERE AT THE TIME THIS PERSON MADE AN OBSERVATION. MEMORY BECOMES AN ISSUE HERE AS WELL AS OTHER EVENTS THAT COULD HAVE CHANGED THIS PERSON’S MIND ABOUT WHAT HE/SHE BELIEVED AT THE TIME OF THE OBSERVATION.
FIRST CONTRIBUTION FROM LAW:
DISTINGUISHING BETWEEN THE COMPETENCE AND THE CREDIBILITY OF HUMAN SOURCES
COMPETENCE:• Appropriate sources,
• In a position to observe,
• Understanding of what was observed,
• Ability to communicate. CREDIBILITY:• Veracity,
• Objectivity
• Observational Sensitivity
ORTHOGONAL
CHARACTERISTICS:
ONE DOES NOT ENTAIL THE OTHER.
14
THESE TWO CHARACTERISTIC ARE FREQUENTLY CONFUSED LEADING TO SERIOUS INFERENTIAL ERRORS.
EXAMPLE OF A FREQUENT MISTAKE:
“We can believe what X tells us because he had good access to his sources”
15
A MAJOR CONTRIBUTION FROM LAW: GROUNDS FOR IMPEACHING OR SUPPORTING WITNESS CREDIBILITY
VARIETY IS THE SPICE OF LIFE IN CREDIBILITY ASSESSMENT
RETURNING TO PROBABILITY: DECOMPOSING THE LINK BETWEEN EVIDENCE E* AND EVENT E
LP E H h f fP E H h f fE*
( | )[ ]( | )[ ]
=− +− +
1
2
For h = P(E*|E) =
P(ES|E)[P(EB|ES) - P(EB|ESC)][P(E*|EB) - P(E*|EB
C)] + P(EB|ESC)
[P(E*|EB) - P(E*|EBC)] + P(E*|EB
C).
For f = P(E*|EC) =
P(ES|EC)[P(EB|ES) - P(EB|ESC)][P(E*|EB) - P(E*|EB
C)] + P(EB|ES
C)[ P(E*|EB) - P(E*|EBC)] +P(E*|EB
C).
16SOURCE”S TESTIMONY E*
DOES THE SOURCE BELIEVE THAT E OCCURRED?
P(E*|EB) AND P(E*|EBC)
[VERACITY]
DID THE SOURCE BASE THIS BELIEF ON SENSORY EVIDENCE OF E?
P(EB|ES) AND P(EB|ESC)
[OBJECTIVITY]
DID EVENT E OCCUR? [HOW GOOD WAS THE SENSORY EVIDENCE?]
P(ES|E) AND P(EB|EC)
[SENSITIVITY]
THE MACE SYSTEM NOW DESCRIBED USES THESE ALGORITHMS FOR DETERMINING h AND f.
THE MACE SYSTEM:
MACE = METHOD FOR ASSESSING THE CREDIBILITY OF EVIDENCE
MACE INITIALLY DESIGNED FOR CIA IN 1990; BUT THEY WERE NOT READY FOR IT. SINCE THE EVENTS OF 9/11/01 AND SUBSEQUENT EVENTS, THEY ARE MORE THAN READY FOR IT NOW. MACE CONSISTS OF TWO PARTS LEADING TO TWO DIFFERENT HEDGES ON A CONCLUSION ABOUT WHETHER TO BELIEVE WHAT A HUMAN SOURCE TELLS US.
PART I.
HOW COMPLETE IS THE EVIDENCE WE HAVE ABOUT THIS SOURCE? [A BACONIAN QUESTION]. MACE ALLOWS THE USER TO:
• MARSHAL ANSWERS TO THE COMPETENCE AND CREDIBILITY QUESTIONS SUGGESTED BY 500 YEARS OF EXPERIENCE IN LAW, • JUDGE WHETHER THE ANSWERS FAVOR OR DISFAVOR THE SOURCE’S COMPETENCE AND ATTRIBUTES OF HIS/HER CREDIBILITY,
• DECIDE WHETHER THE EVIDENCE JUSTIFIES A BELIEF IN WHAT THE SOURCE HAS REPORTED [THIS PART MIGHT STAND ON ITS OWN].
PART II.
HOW STRONG IS THE EVIDENCE WE HAVE ABOUT THIS SOURCE? [A BAYESIAN QUESTION]. MACE ALLOWS THE USER TO:• ASSESS LIKELIHOODS ASSOCIATED WITH THE THREE CREDIBILITY ATTRIBUTES, BASED ON EVIDENCE MARSHALED IN PART I.
• COMBINES THESE LIKELIHOOD ASSESSMENTS USING THE ALGORITHM JUST GIVEN.
• CALCULATES POSTERIOR ODDS ON WHETHER WE SHOULD BELIEVE WHAT THIS SOURCE HAS TOLD US.
17
PART I IN MACE ALLOWS THE MARSHALING OF ANSWERS TO THESE COMPETENCE AND CREDIBILITY QUESTIONS COMING FROM LAW
COMPETENCE1. ACCESS TO INFORMATION
2. KNOWLEDGE AND
EXPERTISE.
3. OBSERVATIONAL
CAPABILITIES.
4. MEETING BEHAVIOR
5. MOTIVATIONAL
CONSISTENCY
6. RESPONSIVENESS.
VERACITY1 PRIOR INCONSISTENCIES
2. OUTSIDE INFLUENCES
3. EXPLOITATION POTENTIAL
4. CONTRADICTION AND
CONFLICT
5. CORROBORATION AND
CONFIRMING
6. CHARACTER
7. REPORTING RECORD
8. TAILORING INFORMATION
9. COLLATERAL DETAILS
10. INTERVIEW BEHAVIOR
OBJECTIVITY1.OBSERVATIONAL
EXPECTATIONS
2.OBSERVATIONAL DESIRES
3. BELIEF CONSEQUENCES
4. MEMORY EFFECTS
5. CONTRADICTION AND
CONFLICT
SENSITIVITY1. SENSORY CAPACITY
2. OBSERVATIONAL
CONTEXT
3. PAST ACCURACY
4. CONTRADICTION AND
CONFLICT
5. COLLATERAL DETAILS
USERS RESPOND TO ANSWERS TO THESE QUESTIONS
BY CHECKING ONE OF THE FOLLOWING BOXES
ANSWER IS FAVORABLE TO AN ATTRIBUTE:
ANSWER IS DISFAVORABLE TO AN ATTRIBUTE:
I CAN’T DECIDE:
NO ANSWERS TO THIS QUESTION:
OVERALL BACONIAN ASSESSMENT IN PART I OF MACE
YOUR OVERALL JUDGMENT ABOUT THIS REPORT: INTELLIGENCE SUPPORTS THAT THE EVENT REPORTED BY THIS SOURCE DID/WLL OCCUR:
INTELLIGENCE FAILS TO SUPPORT THAT THE EVENT REPORTED BY THIS SOURCE DID/WLL OCCUR:
CAN’T DECIDE:
JUDGEMENT BASED ON THIS SUMMARY:AN EXAMPLE
ANSWERED QUESTIONS UNANSWERED QUESTIONS
COMPETENCE [5 Qs]
VERACITY [10 Qs]:
OBJECTIVITY [5 Qs]:\
SENSITIVITY [5 Qs}:
2 1 2
52 3
CREDIBILITY SUMMARY:
1 1 3
2 2 1
2 3 8 7 19
MACE PART II. BAYESIAN ASSESSMENTS OF THE STRENGTH OF EVIDENCE REGARDING THE CREDIBILITY OF A HUMINT SOURCE.
ASSESSMENTS OF PAIRS OF LIKELIHOODS FOR EACH CREDIBILITY ATTRIBUTE:
1.0
1.00
20
1.0
1.00
FOR VERACITY FOR OBJECTIVITY
1.0
1.00
FOR SENSITIVITY OR ACCURACY
P(E*|EB)
P(E*|EBC)
P(EB|ES)
P(EB|ESC)
P(ES|E)
P(ES|EC)
• ASSESSMENTS BASED ON ALL EVIDENCE FOR EACH ATTRIBUTE
• ASSESSMENTS MADE IN GRAPHICAL FORM AS SHOWN NEXT.
AN EXAMPLE OF LIKELIHOOD ASSESSMENT: FOR VERACITY
ANALYSTS, AS WELL AS OTHERS, RESIST HAVING TO MAKE NUMERICAL ASSESSMENTS OF PROBABILITIES. SO, MACE ALLOWS THEM TO DRAW BOXES IN A TWO DIMENSIONAL PROBABLITY SPACE [ONE BOX PER ATTRIBUTE].
21
1.0
1.00
P(E*|EB)
P(E*|EBC)
LOCATION OF BOX TELLS HOW STRONGLY THE EVIDENCE FAVORS OR DISFAVORS VERACITY.
SIZE OF BOX SHOWS USER CONFIDENCE IN HIS/HER ASSESSMENT
AUTOMATIC CALCULATIONS MADE BY THE MACE SYSTEM
FIRST: MACE FINDS THE FOUR NUMERICAL CORNER POINTS OF EACH OF THE THREE BOXES DRAWN BY THE USER; ONE BOX PER ATTRIBUTE.
SECOND: MACE FORMS ALL 64 COMBINATIONS OF THESE CORNER POINTS.
THIRD, FOR EACH COMBINATION, MACE CALCULATES h = P(E*|E) AND f = P(E*|EC) USING THE ALGORITHM PREVIOUSLY GIVEN:
For h = P(E*|E) =
P(ES|E)[P(EB|ES) - P(EB|ESC)][P(E*|EB) - P(E*|EB
C)] + P(EB|ES
C)[P(E*|EB) - P(E*|EBC)] + P(E*|EB
C).
For f = P(E*|EC) =
P(ES|EC)[P(EB|ES) - P(EB|ESC)][P(E*|EB) - P(E*|EB
C)] + P(EB|ES
C)[ P(E*|EB) - P(E*|EBC)] +P(E*|EB
C).
FORTH, MACE FINDS ALL POSSIBLE VALUES OF THE 64 POSSIBLE h/f = P(E*|E}/P(E*|E) RATIOS AND FINDS AN INTERVAL CONTAINING ALL OF THEM. NOTE:THE RATIO h/f = P(E*|E}/P(E*|E) SHOWS THE BAYESIAN FORCE OF EVIDENCE IN AN INFERENCE ABOUT WHETHER E IS TRUE, BASED ON THE SOURCE’S REPORT E*. 22
FINALLY, MACE PROVIDES THE FOLLOWING GRAPHICAL DISPLAY: A SINGLE EXAMPLE
LOG ODDS, LOG LIKELIHOOD RATIO
0 __
FAVORS EC +FAVORS E
PRIOR ODDS INTERVAL. GIVEN INITIALLY BY USER
LIKELIHOOD RATIO INTERVAL CALCULATED BY MACE
POSTERIOR ODDS ON E , GIVEN THE SOURCE’S REPORT.
23
COMING WORK ON MACE
• EXTENDING MACE TO DEAL WITH ADDITIONAL CREDIBILITY MATTERS WHEN HUMAN SOURCES PROVIDE TANGIBLE EVIDENCE.
• EXTENDING MACE TO COPE WITH “WILDERNESS OF MIRRORS” PROBLEM, WHEN WE HAVE CHAINS OF SOURCES ALL COMMENTING ON EACH OTHER’S CREDIBILITY.
• EXTEND MACE TO ALLOW BETTER RECOGNITION OF “DOUBLES”.
END OF STORYMANY THANKS FOR ALLOWING ME TO TELL IT
24
top related