components of judgmental skill thomas r. stewart, ph.d. center for policy research rockefeller...
TRANSCRIPT
![Page 1: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/1.jpg)
Components of judgmental skill
Thomas R. Stewart, Ph.D.Center for Policy Research
Rockefeller College of Public Affairs and PolicyUniversity at Albany
State University of New [email protected]
Public Administration and PolicyPAD634 Judgment and Decision Making Behavior
![Page 2: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/2.jpg)
components-of-skill.ppt 2
Fundamental tenet of correspondence research
"Human competence in making judgments and decisions under uncertainty is impressive. Sometimes performance is not. Why? Because sometimes task conditions degrade the accuracy of judgment."
Hammond, K. R. (1996). Human Judgment and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable Injustice. New York, Oxford University Press (p. 282).
![Page 3: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/3.jpg)
components-of-skill.ppt 3
Brunswik's lens model
Distal variable Judgment
X
Cues
Ye Ys
![Page 4: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/4.jpg)
components-of-skill.ppt 4
TrueDescriptors
Subjective
Judgment
CuesCues
Expanded lens model
Distal variable
![Page 5: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/5.jpg)
Environmental predictability
Fidelity of the information system
Match between environment and judge
Reliability of information acquisition
Reliability of information processing
TrueDescriptors
Subjective
Judgment
CuesCues
components-of-skill.ppt 5
Components of skill and the lens model
Distal variable
![Page 6: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/6.jpg)
components-of-skill.ppt 6
Area under ROC curve
Correlation (rYO)
O is the observation, or “gold standard”
Y is the judgment
Mean square error:
Skill score:
Alternative measures of performance(correspondence approach)
O - Yn1
= MSE 2iiY
B
Y
MSEMSE
- 1 = SS 2iB )O O(
n1
= MSE
![Page 7: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/7.jpg)
components-of-skill.ppt 7
Decomposing accuracy--the skill score
Murphy (1988):
Skill = Correlation - Conditional bias - Unconditional bias
score (regression bias) (base rate bias)
22
2
OO
Y
YOYO sO Y
- ss
- r - r = SS
![Page 8: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/8.jpg)
components-of-skill.ppt 8
Decomposing accuracy: The Lens Model Equation (Tucker 1964)
Judgment
Cues
"Truth"
O.Xn2O.X
E + X , ... ,X ,XM = O1
Y.Xn2Y.X
E + X , ... ,X ,XM = Y1
2
Y.X
2
O.XX.YO.XYOR - 1 R - 1C + R G R = r
![Page 9: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/9.jpg)
Decomposition of Skill Score
rYO
( )2
-
rYO
- ( / ) ][ - [ ( Y - O ) / ]sY
_
sO
2 2s
O
Squared correlation
Conditional Unconditional
bias bias
SS =
RO.X
G RY.X
GSS
rYO- - ( / ) ][ - [ ( Y - O ) / ]s
Ys
O
2 2s
O
( )2
RO.T
_
Murphy
Tucker
VT.X G R
Y.U
Expandedlens
model
Components of skill:
1. Environmental predictability
2. Fidelity of the information system
3. Match between environment and judge
4. Reliability of information acquisition
5. Reliability of information processing
6. Conditional/regression bias
7. Unconditional/base rate bias
Step 1:
Step 2:
Step 3:
(regression) (base rate)
rYO - ( / ) ][ - [ ( Y - O ) / ]s
Ys
O
2 2s
O
_-
(1988)
(1964)
1 2 3 4 765
_
_
_[ ]
2
SS = Skill Score = 1 - ( MSE Y MSE B/ )
~=
SS ~= VU.X
![Page 10: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/10.jpg)
components-of-skill.ppt 10
Decomposition of skill score
processing
ninformatio
ofy reliabilit
nacquisitio
ninformatio
ofy reliabilit
forecaster and
tenvironmen
between match
system
ninformatio
the offidelity
litypredictabi
talenvironmen
bias
nalunconditio
bias
lconditiona
Skill score =
![Page 11: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/11.jpg)
Components of skill addressed by selected methods for improving judgments
Component of Skill* Method for improving judgments 1 2 3 4 5 6 7 Identify new descriptors through research X Develop better measures of true descriptors X Develop clear definitions of cues X Training to improve cue judgments X Improve information displays X Bootstrapping--replace judge with model X Require justification of judgments X X Combine several judgments X Decompose judgment task X Mechanical combination of cues X Train judge about environmental system X Experience with problem X X XCognitive feedback X Train judge to ignore non-predictive cues X Statistical training X XFeedback about nature of biases in judgment X XSearch for discrepant information X Statistical correction for bias X X
*1. Environmental predictability 2. Fidelity of the information system 3. Reliability of information acquisition 4. Reliability of information processing 5. Match between environment and judge 6. Conditional/regression bias 7. Unconditional/base rate bias
components-of-skill.ppt 11
![Page 12: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/12.jpg)
components-of-skill.ppt 12
1. Environmental predictability
• Environmental predictability is conditional on current knowledge and information. It can be improved through research that results in improved information and improved understanding of environmental processes.
• Environmental predictability determines an upper bound on performance and therefore indicates how much improvement is possible through attention to other components.
Components of skill
![Page 13: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/13.jpg)
components-of-skill.ppt 1313
Environmental predictability limits accuracy of judgment
![Page 14: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/14.jpg)
components-of-skill.ppt 14
2. Fidelity of information system
• Judgmental skill may be degraded if the information system that brings data to the judge does not accurately represent actual conditions, i.e., if the cues do not accurately measure the true descriptors. Fidelity of the information system refers to the quality, not the quantity, of information about the cues that are currently being used.
• Fidelity is improved by developing better measures, e.g., though improved instrumentation or increased density in space or time.
Components of skill
![Page 15: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/15.jpg)
components-of-skill.ppt 15
3. Match between environment and judge
• The match between the model of the judge and the environmental model is an estimate of the potential skill that the judge's current strategy could achieve if the environment were perfectly predictable (given the cues) and the judgments were unbiased and perfectly reliable.
• This component might be called “knowledge.” It is addressed by training and experience. If the judge learns to rely on the most relevant information and ignore irrelevant information, this component will generally be good.
Components of skill
![Page 16: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/16.jpg)
components-of-skill.ppt 16
Reliability
• Reliability is high if identical conditions produce identical judgments.
• Humans are rarely perfectly reliable.
• There are two sources of unreliability:– Reliability of information acquisition
– Reliability of information processing
Components of skill
![Page 17: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/17.jpg)
components-of-skill.ppt 17
Reliability
• Reliability decreases as amount of information increases.
Components of skill
Theoretical relation between amount of information and
accuracy of judgment
![Page 18: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/18.jpg)
components-of-skill.ppt 18
Reliability decreases as environmental predictability decreases
Components of skill
18
![Page 19: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/19.jpg)
components-of-skill.ppt 19
4. Reliability of information acquisition
• Reliability of information acquisition is the extent to which the judge can reliably interpret the objective cues.
• It is improved by organizing and presenting information in a form that clearly emphasizes relevant information.
Components of skill
![Page 20: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/20.jpg)
components-of-skill.ppt 20
Visual display of information
Components of skill
Books by Edward R. Tufte
The Visual Display of Quantitative Information.
Envisioning Information
Visual Explanations
Beautiful Evidence
The Cognitive Style of Powerpoint: Pitching Out Corrupts Within
Cheshire, Connecticut: Graphics Press
![Page 21: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/21.jpg)
components-of-skill.ppt 21
Visual display of information
Components of skill
Image from the videotape “Study of a Numerically modeled severe storm,” National Center for Supercomputing applications, University of Illinois (Tufte, Visual Explanations, p. 20)
Tufte’s questions:How big is the cloud?What direction is it moving?What are the dimensions of the grid?
![Page 22: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/22.jpg)
components-of-skill.ppt 22
Visual display of information
Components of skill
Redesign by Tufte and Bushell (Tufte, Visual Explanations, p. 21)
Restored quantitative informationDirectional arrowsGrid size given (and reduce the dominance of the grid pattern)Time scale, with small clouds depicting storm history
![Page 23: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/23.jpg)
components-of-skill.ppt 23
Visual display of information
Components of skill
Chart prepared by Morton Thiokol for Challenger commission. Tufte, Visual Explanations, p. 47
Visual display problems
Disappearing legend.The O-ring damage legend was on another slide.
ChartjunkObscures cause and effect
Temperatures turned sidewaysO-ring anomalies depicted by scattered little marks
Wrong orderDate of launch rather than temperature
![Page 24: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/24.jpg)
components-of-skill.ppt 24
Visual display of information: Scatterplot of temperature vs. O-ring damage
Components of skill
Tufte, Visual Explanations, p. 45
![Page 25: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/25.jpg)
components-of-skill.ppt 25
Visual display of information influences mode of thought
Components of skill
Use of images tends to induce intuition
Use of numbers tends to induce analysis
![Page 26: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/26.jpg)
components-of-skill.ppt 26
5. Reliability of information processing
Decreases with increasing information and with increasing environmental uncertainty
Methods for improving reliability of information processing: Limit the amount of information used in making
judgments. Use a small number of very important cues. Use mechanical methods to process information. Combine several judgments. Require justification of judgments.
Components of skill
![Page 27: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/27.jpg)
Acc
urac
y
Amount of Information
Actual accuracy
Theoretical limit of accuracy
Perfectaccuracy
Noaccuracy
Effect of limited
information and
environmental uncertainty
Effect of
limitations in
information processing
components-of-skill.ppt 27
Theoretical relation between amount of information and accuracy
![Page 28: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/28.jpg)
components-of-skill.ppt 28
The relation between information and accuracy depends on environmental
uncertainty
Low predictability task
Amount of Information
Acc
ura
cy
Theoretical Limit of Accuracy
Actual Accuracy
Effect of limited information and environmental
uncertainty
Effect of limitations in information processing
No accuracy
Perfect accuracy
High predictability task
Amount of Information
Acc
ura
cy
Theoretical Limit of Accuracy
Actual Accuracy
Effect of limited information and environmental
uncertainty Effect of limitations in information processing
No accuracy
Perfect accuracy
- - - - - Theoretical limit of accuracy——— Actual accuracy
- - - - - Theoretical limit of accuracy——— Actual accuracy
![Page 29: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/29.jpg)
components-of-skill.ppt 29
6 and 7. Bias -- Conditional (regression bias) and unconditional (base rate bias)
Together, the two bias terms measure judgment "calibration.”
Reducing bias: Experience Statistical training Feedback about nature of biases in judgment Search for discrepant information Statistical correction for bias
Components of skill
![Page 30: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/30.jpg)
components-of-skill.ppt 30
Calibration of judgments depends on the task
Calibration data for precipitation forecasts (Murphy and Winkler, 1974) Heideman (1989)
![Page 31: Components of judgmental skill Thomas R. Stewart, Ph.D. Center for Policy Research Rockefeller College of Public Affairs and Policy University at Albany](https://reader035.vdocument.in/reader035/viewer/2022062314/56649ee05503460f94bf00d2/html5/thumbnails/31.jpg)
components-of-skill.ppt 31
Conclusion
• Problem: Improving judgmental accuracy– Understanding and improving judgment requires
understanding the task and the environment.– Decomposing skill can aid in identifying the factors
that limit judgmental accuracy.