objective evaluation of subjective decisions
DESCRIPTION
Objective Evaluation of Subjective Decisions. Mel Siegel & Huadong Wu Robotics Institute – School of Computer Science Carnegie Mellon University - Pittsburgh PA 15232 USA. SCIMA-2003 Soft Computing Techniques in Instrumentation, Measurement and Related Applications - PowerPoint PPT PresentationTRANSCRIPT
Objective Evaluation of Subjective Decisions
Mel Siegel & Huadong WuRobotics Institute – School of Computer Science
Carnegie Mellon University - Pittsburgh PA 15232 USA
SCIMA-2003Soft Computing Techniques in Instrumentation,
Measurement and Related Applications
Brigham Young University – Provo UT USA2003 May 17
outline• background: problem of “sensor fusion for
context aware computing”• approach: development of an “adaptive
weighted Dempster-Shafer (D-S)” algorithm• issue (= the talk’s title): objective evaluation
of subjective decisions– meta-issue: is it really an issue?
• discussion: “receiver operating characteristic”• closing the loop: ROC D-S ?
background
• “context detection” for HCI– e.g., your cell phone could ring louder if it could
know it is in your briefcase
• context detection requires subjective evaluation of “ordinary” sensor signals
• sensor fusion required when we have multiple detectors, none of them very good
• sequence of algorithms culminates in an “adaptively weighted Dempster-Shafer” method
Focus-of-AttentionFocus-of-Attention decisiondecisionby fusion of video and audio databy fusion of video and audio data
Focus-of-AttentionFocus-of-Attention decisiondecisionby fusion of video and audio databy fusion of video and audio data
Camera View
sensor fusion alternativessensor fusion alternatives
#1. complementary
#2. competitive
#3. cooperativeParametric template,Figures of merit,Syntactic pattern recognition… …
Logical templateAI rule-based reasoning,Heuristic inferenceNeural network… …
Classic InferenceSensor i: Pi( x detected | x appeared )
Simple & effective for “x vs. ¬x” problems
Priori knowledge and pdf are required to combine multiple sensor outputs, priori
assessments are not used, do not have enough reasoning power
Voting FusionAssociate pdf with confidence estimation, and
provide a way to predict the result probabilities of their boolean combinations
Though big improvement over Classic Inference method, still not powerful enough to reason at
fine granularity
Bayesian NetworkLikelihood of a hypothesis is updated using a previous likelihood estimation and additional
evidence
“cannot distinguish between lack of belief and disbelief”, cannot address a problem like “its
likely either user A or user B”
Fuzzy Logic No pdf required, very cheap in computation
It doesn’t make sense that a person is assigned as “0.6 membership of user A”, “0.7
membership of user B”, and “0.9 membership of either user A or B”
Neural NetworkFlexible, powerful, no pdf needed, cheap
computational cost in classification process
Local minimal problem, results cannot be easily explained, not suitable for dynamic
configuration of sensors
our problem: Bayes can’t do itour problem: Bayes can’t do it
head pan
left
observed pan
sensor noise
straight
left straight right
right
rightstraight
right
approach:approach:the Dempster-Shafer methodthe Dempster-Shafer method
• “Frame of Discernment” Θ lists all possibilities: {A}={ {L}, {S}, {R}, {L | S}, {S | R}, {L | R}, {L | S | R} }
•
•
,)()(Belief
AE
kii
k
EmA
AE
kiii
k
EmAA )(1)(Belief1)(tyPlausibili
)(yprobabilit:)( kki AEm
a theory of evidencea theory of evidencea theory of evidencea theory of evidenceallows belief and plausibilityallows belief and plausibilityallows belief and plausibilityallows belief and plausibility
quantifies both knowledge and ignorancequantifies both knowledge and ignorancequantifies both knowledge and ignorancequantifies both knowledge and ignorance
a generalization/extension of Bayesian inference networka generalization/extension of Bayesian inference networka generalization/extension of Bayesian inference networka generalization/extension of Bayesian inference network
sensor fusion using “classical” sensor fusion using “classical” Dempster-Shafer Theory of Evidence Dempster-Shafer Theory of Evidence
sensor fusion using “classical” sensor fusion using “classical” Dempster-Shafer Theory of Evidence Dempster-Shafer Theory of Evidence
'
'
'
'
)()(1
)()(
)(kEkE
kjki
AkEkEkjki
EmEm
EmEm
ji Amm
{L}=0.3 {R}=0.6 {L|R}=0.1
{L}=0.4 {L}=0.4x0.3 {Φ}=0.4x0.6 {L}=0.4x0.1
{R}=0.5 {Φ}=0.5x0.3 {R}=0.5x0.6 {R}=0.5x0.6
{L|R}=0.1 {L}=0.1x0.3 {R}=0.1x0.6 {L|R}=0.1x0.1
'
'
'
'
)()(1
)()(
)(kEkE
kjki
AkEkEkjki
EmEm
EmEm
ji Amm
extension of Dempster-Shafer: extension of Dempster-Shafer: evidence weighted by sensors’ reliabilitiesevidence weighted by sensors’ reliabilities
kEjEkji
AkEjEkji
EmEm
EmEm
i
iiii
iii
Amm
mwwm
AAmwAm
)]()('[1
)]()('[
)(
)(1'
,)()('
further extension of Dempster-Shafer: further extension of Dempster-Shafer: weights change according to performance historyweights change according to performance history
overcomes sensor drift problem! overcomes sensor drift problem! overcomes sensor drift problem! overcomes sensor drift problem!
kEjEkji
AkEjEkji
EmEm
EmEm
i
iiii
iii
Amm
mtwtwm
AAmtwAm
)]()('[1
)]()('[
)(
)()()(1'
,)()()('
1.0) (0.0, factor remnance
predictionincorrect 1
predictioncorrect 0)(
)()1()(0
p
tc
ptntcptwn
ni
an arbitrary effectiveness measurean arbitrary effectiveness measure
-5%
0%
5%
10%
15%
20%
25%
30%
35%
40%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
focus-of-attention: meeting experiments - user subject
perc
en
tag
e r
eg
ain
ed
fro
m t
he v
isu
al sen
so
rs' lo
ss
linear probability combination standard Dempster-Shafer
Weighted Dempspter-Shafer Dynamically weighted Dempster-Shafer
generalizing via a simulation ...generalizing via a simulation ...
head pan
left
observed pan
sensor noise
straight
left straight right
right
rightstraight
right
... yields an intriguing result... yields an intriguing resultwhen sensor precisions are very differentwhen sensor precisions are very different
-10%
-5%
0%
5%
10%
15%
20%
25%
1 2 3 4 5 6
experiments
se
ns
or
fus
ion
's l
os
s/g
ain
fo
r S
1 m
iss
ed
da
ta
linear probability combination
standard Dempster-Shafer
weighted Dempster-Shafer
dynamically weighted Dempster-Shafer
the issue ...
• objective evaluation of subjective decisions– a meta-issue: is it really an issue?
“objective” vs. (?) “subjective”
• in medicine the distinction is sharp:– subjective: means what the patient tells the
physician about his/her complaint, what he/she thinks is the problem, etc
– objective: means what the physician observes (and his/her instruments report) about the condition of the patient
• statisticians talk about “rational gambling”• but in most contexts it feels fuzzier ...
• and even physicians make subjective decisions• whose quality we can evaluate objectively!:
patientreally has SARS
patient really doesn’t have SARS
physician says patient has
SARS
TRUEPOSITIVE
FALSEPOSITIVE
physician says patient doesn’t
have SARS
FALSENEGATIVE
TRUENEGATIVE
receiver operating characteristic• originally developed for target analysis• considers ratio of signal to signal-plus-
noise vs. the discriminator level set• adopted and extensively developed in the
medical diagnostic test community– { TP, TN } signal, { FP, FN } noise
– most physicians understand a test’s sensitivity == TP/(TP+FN) andspecificity == TN/(TN+FP)vs. the chosen “cut point” of the test
(dotted) ideal(dashed) useless(a) reliable(b) typical
ROC
-- increasing cut point increases TPs (good) and FNs (bad)-- decreasing cut point increases TNs (good) and FPs (bad)
closing the loop? ...
• ROC D-S ?
0 1
evidence that supports X-- fever-- white tongue-- headache
evidence that rules out X-- no virus detected-- had disease once before-- over age 55
“belief”
“plausibility”
Dempster-Shafer0 1
cut point
TP TN
FP FN
ROC
conclusions / questions• adaptive weighted D-S seems to contribute
an incremental but real improvement in appropriate sensor fusion applications
• “objective”/“subjective” distinction is fuzzy• maybe ROC and related “cut point analysis”
techniques can help us set neural net, fuzzy system, etc, parameters that are now set either arbitrarily or iteratively (hence slowly)
• is the apparent connection between D-S and ROC superficial, or real at some deep level?