human factors aspects of anomaly detection...
TRANSCRIPT
NRC COHSI 2/12/20091
Human Factors Aspects of Anomaly Detection Systems
Thomas Sanquist, Thomas Sheridan, John Lee, Nancy Cooke
Committee on Human-System IntegrationNational Research Council
February 12, 2009
NRC COHSI 2/12/20092
Background
• COHSI outreach to government agencies in September, 2008
• An underlying theme among a number of agencies seemed to be anomaly detection related to various applications:
– Safety
– Security
– Traffic management
– Diagnosis
• Consensus on committee to explore general area in greater depth at current meeting
• Many other applications beyond these areas: medicine, industrialprocess monitoring, aircraft predictive maintenance, energy system management, etc.
• Discussion format
NRC COHSI 2/12/20093
Session Overview
• Characterize general features of human-mediated anomaly detection systems
• Review issues and engineering approaches to anomaly detection
• Review selected aspects of application examples
– Radiation detection
– Landmine detection
– Visual scene surveillance
– Shipping traffic patterns/maritime domain awareness
– Operator Impairment Detection (Lee)
– Group Communication Anomalies (Cooke)
• Common human factors issues across types of systems
• Knowledge gaps – how to address them?
• Role of NRC workshops and consensus studies
NRC COHSI 2/12/20094
Anomaly Detection Defined
• Finding patterns in data that do not conform to expected behavior.– Anomalies, outliers, discordant observations, exceptions, aberrations, surprises,
peculiarities, contaminants.– Most of our training emphasized the use of averages, but there is also value in
studying extreme values in a variety of fields
• Applications: – Fraud detection– Insurance & health care– Intrusion detection– Fault detection in safety critical systems– Surveillance for military and security
• The term anomaly detection first came into the literature in the mid-1980s in the realm of computer network intrusion detection systems (IDS)
NRC COHSI 2/12/20095
The Principal Messages
• Very little HFE work in these types of systems (except air passenger screening & landmine detection)
• Basic research dominated by algorithm development and refinement
• Applied research involves demonstration projects
• Demonstrations do not entail systematic data collection from operators– This feedback loop can help to improve technical aspects of
systems
• Systems engineering risk:– Large-scale technical systems deployed which require constant
staffing to compensate for technical performance issues
NRC COHSI 2/12/20096
Selected Resources
• Chandola, Banerjee and Kumar (2009). Anomaly Detection: A Survey. ACM Computing Surveys, in press.
• Axelsson, S. (2000). The Base-Rate Fallacy and the Difficulty of Intrusion Detection. ACM Transactions on Information and System Security. 3(3). 186 – 205.
• Chen, H., et al. (2005) Imaging for Concealed Weapon Detection. IEEE Signal Processing Magazine. 52.
• Dee, H.M and Velastin, S.A. (2008). How close are we to solving the problem of automated visual surveillance? Machine Vision and Applications. 19: 329 – 343.
• Kristin M. Schweitzer; Andrew S. Bodenhamer (2007). Visual Detection of Land Mines. ARL-TR-4073.
• MacDonald, J., et al. (2003). Alternatives for Landmine Detection. RAND Monograph Report 1608.
• Parasuraman, Sheridan, Wickens (2000). A Model for Types and Levels of Human Interaction with Automation. IEEE Transactions on Systems, Man, and Cybernetics Part a: Systems and Humans, Vol. 30, No. 3.
• USCG Acquisition Directorate, NAIS Fact Sheet http://www.uscg.mil/ACQUISITION/programs/pdf/NAIS.pdf
• DHS National Plan to Achieve Maritime Domain Awareness. http://www.dhs.gov/xlibrary/assets/HSPD_MDAPlan.pdf
• Valera and Velastin (2005). Intelligent Distributed Surveillance Systems: A Review. IEE Proceedings on Visual Imagery and Signal Processing. 152(2). 192 – 204.
• very few human performance studies in this area
NRC COHSI 2/12/20097
A Personal Example
• Call from Columbus, OH credit union before 8AM PDT Monday morning (how do I know who they really are....?)
• Unusual transactions flagged by Bank of America neural net software• Have you traveled to Atlantic City recently?• Do you have your credit card? (no....uh-oh....)• Cash transfer transactions on card with $9K limit:
– Saturday evening: $12, Sands Swingers– Sunday: $539.99, Harrah’s– Sunday: $2627.99, Bally’s Park Place– Sunday: $2629.99, Sands Hotel & Casino– Sunday: $2629.99, Harrah’s
• Anomalies were detected, reported to credit union keeping normal business hours.
• System failed to alert in time to prevent loss • Numbers and timing suggest perpetrators knew how system
worked• My loss covered by credit union insurance
NRC COHSI 2/12/20098
Why are anomalies important?
• Associated with significant actionable information:
– Credit card fraud
– Excessive work hours, worker impairment
– Smuggling
– IED location
– Terrorist activity
– Medical problem
• From scientific standpoint, study of outliers enriches our understanding of human capabilities and limitations, such as:
– “short” and “long” sleepers
– “larks” and “night owls”
– Neuropsychological studies, e.g., HM,
AJ
NRC COHSI 2/12/20099
Three general types of human interaction
PhysicalSignal
Amplify
Condition PresentOr Absent?
PhysicalSignal
Amplify, Process, Alarm Criteria
Operator Evaluationand Resolution
PhysicalSignal
Condition PresentOr Absent?
?
NRC COHSI 2/12/200910
Human Interaction and Levels of AutomationLEVELS OF AUTOMATION IN SIGNAL DETECTION
COMPUTER HUMAN
1 NA Senses raw data
Decides S or N
2 Senses raw data Observes display
Displays it in human friendly format Decides S or N
3 Senses raw data Observes display
Performs initial filtering of noise Decides S or N
Displays result to human friendly format
4 Senses raw data Observed display
Decides S or N Considers computer decision
Also displays human friendly filtered data Makes final decision S or N
5 Senses raw data Observed display
Decides S or N Considers computer decision
Also displays human friendly filtered data Also considers confidence etc.
Also displays confidence or other parameters Makes final decision S or N
NRC COHSI 2/12/200911
Human Factors and Anomaly Detection
• Provide a means for human awareness of sensory or non-sensory phenomena, and/or alert human attendants to events of interest on the basis of automated decision criteria
• There are varying levels of human involvement in this process
• Human operators can provide: physical detection, decision makingverification, interpretation and classification, context.
• New jobs/tasks created to handle resolution of imperfect system output
• Influenced by many “traditional” human factors such as workload,signal-to-noise ratio, pace of data flow, length of watch.
• System-specific human factors such as low base rate of events, complexity of masking, transparency of algorithms, high false alarm rates.
NRC COHSI 2/12/200912
NRC COHSI 2/12/200912
The Trust Issue
Calib
rate
d Tru
st
Overtrust:Trust exceedssystem capabilities
Distrust:Trust falls short of system capabilities
Trust
Automation Capability(trustworthiness)
Types of information underlying trust
• Purpose: Intended application
• Process: Sensors and data processing algorithms
• Performance: Precision and consistency
NRC COHSI 2/12/200913
Challenges
• Difficult to define region encompassing all possible normal behaviors.– Boundary between normal and anomalous not precise (e.g., taking
pictures at Penn Station)
• Malicious adversaries adapt to make anomalous observations appear normal
• Across domains the definition of normal evolves
• Exact notion of anomaly varies across domains:– Small deviation in body temp might be anomalous
– Small deviation in stock market normal (not that we have seen any of these lately…..)
• Availability of labeled (ground truth) data for training of models use for anomaly detection a major issue
• Data often contain noise which looks similar to anomalies
NRC COHSI 2/12/200914
Types of anomalies
Collective anomaly – multiple data instances which by themselves are not anomalous, but contiguous occurrence makes them so
Point – individual data instance deviant with respect to the rest
Contextual – anomalous with respect to surrounding context
NRC COHSI 2/12/200915
Many Data Processing Techniques and Variants
• Statistical Profiling
• Neural Networks
– Multi-layered perceptrons, Neural trees, Adaptive Resonance Theory, Radial Basis Function, Hopfield Networks, Oscillatory Networks
• Support Vector Machines
• Rule-based systems
• Bayesian Networks
• Clustering Algorithms
• Spectral Analysis
• Nearest Neighbor Analysis
NRC COHSI 2/12/200916
A typical paragraph from an anomaly detection article….why we need the HF perspective
NRC COHSI 2/12/200917
Comparison of decision architectures
• Traditional 4-stage human performance model
• Typical Signal Processing Algorithm for Land Mine Detection
Sensory Processing
Perception/Working Memory
Decision MakingResponse
Selection & Action
Preprocessing Detection Discrimination Decision
Output
NRC COHSI 2/12/200918
Comparison of decision architectures, continued
• Visual surveillance systems
NRC COHSI 2/12/200919
Taxonomy for Anomaly Detection Systems –a means for comparing across domains
• Main Purpose
• Domain of Application
• Time Frame of Detection (real-time, post-hoc)
• Role of Human Operator (sensory, interpretation, adjudication....)
• Secondary applications
• Nature of Anomaly Data (point, collective, contextual...)
• Data Processing Approach
• Data-to-Construct linkage
• Base rate of occurrence
• False positive rates
• Positive Predictive Value of Alarm
• Ground truth data availability
NRC COHSI 2/12/200920
Some specific anomaly detection systems
NRC COHSI 2/12/200921
Radiation Detection
NRC COHSI 2/12/200922
Radiological Threat
– Radiation Portal Monitoring (RPM) program, Domestic Nuclear Detection Office & Customs and Border Protection
– Nuclear Weapons
• State Weapon
• Improvised Nuclear Weapon
– Nuclear Weapons Material (Special Nuclear Material - SNM)
– Radiation Dispersal Device (RDD)
– Radiological Material for use in Construction of a RDD
– Other Illegal or Illicit Radioactive Material (i.e. ContaminatedSteel, inappropriately manifested/marked material, other radiological contraband)
NRC COHSI 2/12/200923
Elements of RPM system
NRC COHSI 2/12/200924
Basis of human factors problem with radiation detection:poor threat classification
NRC COHSI 2/12/200925
Fundamental human factors issues inradiation portal screening
• Low probability – high consequence events
– Low base rate of illicit nuclear material movement (estimate = 4.5 smuggling events per 11,000,000 commercial truck transits)
– Comparatively higher rate of “nuisance alarms” (multiple times per day) – Naturally Occurring Radioactive Material: ceramics, fertilizer, etc. (1 –2 NORM alarms per 100 vehicles)
– Very low probability of true threat alarm (1 every 2- 4 years)
• Frequent attention to non-informative alarms leads to mistrust in automation, complacency, possibly ignoring/disabling alarms
• Dedicated manpower requirement to resolve nuisance alarms –estimated cost for one major port = $15M
• Potential solution: threat likelihood alarms
– Requires extending the simple point anomaly detection based on standard deviation of measure over background radiation
– Incorporate more sophisticated sensors (spectral), energy windowdiscrimination, and fusion of manifest and notice-of-arrival data.
NRC COHSI 2/12/200926
Radiation Detection Anomaly Detection Summary
• Main Purpose: Threat Detection and Interdiction
• Domain of Application: Radiation Detection at Border Crossings
• Time Frame of Detection: Real-time
• Role of Human Operator: Evaluation and adjudication of system output
• Secondary applications: Prevention of contaminated material entering country
• Nature of Anomaly Data: Point
• Data Processing Approach: Statistical, Spectral
• Data-to-Construct linkage: Moderate (range of threat types considerable)
• Base rate of occurrence: unknown but assumed to be extremely low(estimates from IAEA)
• False positive rates: Very high
• Positive Predictive Value of Alarm: Very low
• Ground truth data availability: yes, based on secondary exam and manifest declaration
NRC COHSI 2/12/200927
Land Mine Detection
NRC COHSI 2/12/200928
The Land Mine Problem
• 15 – 20,000 victims per year in 90 countries
• 40 – 50 million mines remain to be cleared
– 100K per year are cleared
– 1.9 million new mines laid every year
• Extensive contamination of agricultural land in Afghanistan reduces usage by up to 80%
• Devastating human and economic consequence
NRC COHSI 2/12/200929
Fundamental Human Factors Issues inLand Mine Detection
• High consequence event, high probability in certain areas• Demining tools similar to those employed in WWII using electromagnetic
induction (EMI).• Audible signal provided to operator• Limitation is inability to discriminate mines from non-mine metal clutter• False alarm rate = 99.7%, true positive % = 0.3% (500K/200M)• Sensitivity varies by detector, location and soil type• 1 de-miner killed for every 1000 – 2000 mines cleared• Excessive time spent investigating false alarms leads to fatigue and
carelessness. All buried items signaled by detector are investigated manually
• Visual cues are ignored with excessive focus on detector (Davison, ARL)
NRC COHSI 2/12/200930
Solution Approaches for Landmine Problem
• Other technologies being researched: ground penetrating radar, acoustic/seismic, vapor trace detection (very basic level), nuclear quadrupole resonance.
• Dual sensor program to combine EMI and GPR into Handheld Standoff Mine Detection System (HSTAMIDS)– Evolved to AN/PSS-14 (Army-Navy/Portable Special Search)
• Uses dual outputs to operator – this would be an area for further HFE development
• ARL studies show that 33% of simulated mines can be detected on basis of visual cues alone – recommend further training to reinforce this modality, and studies of unique auditory signatures of specific mines.
• Further research with fused multi-sensor data into single output for operator.
NRC COHSI 2/12/200931
Improving Detection of Land Mines Jim Staszewski
Carnegie-Mellon University
• Problem: Poor land mine detection
• Solution: identify expert and bootstrap expert’s detection strategy
• Field Implementation:
– New training adopted by Army
• Evidence of Success:
– Improved detection rates after training
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
0.90
1.00
PRE POST
P(D
)
CEBES
Training
Traditional
Training
NRC COHSI 2/12/200932
Landmine Detection System Summary
• Main Purpose: Landmine detection and clearing• Domain of Application: Previously mined areas• Time Frame of Detection: Real-time• Role of Human Operator: Evaluation and adjudication of system
output• Secondary applications: None• Nature of Anomaly Data: Point• Data Processing Approach: • Data-to-Construct linkage: Weak - Moderate (range of mines
considerable, much clutter in environment)• Base rate of occurrence: high in identified areas• False positive rates: Very high• Positive Predictive Value of Alarm: low• Ground truth data availability: Yes, but only following physical
investigation
NRC COHSI 2/12/200933
Visual Surveillance Systems
http://www.cernium.com/WMV/Belo_Cernium.asf
NRC COHSI 2/12/200934
Threat Basis
• Crowded public areas
• Unusual or suspicious behavior
• Unauthorized presence, crowd formation
• Left-behind packages/luggage
• Concealed explosives
NRC COHSI 2/12/200935
Watch the Texas Border
• http://www.blueservo.net
• This system links cameras along the Rio Grande to the public via streaming video
• 500 lbs of marijuana seized in December as a result of report from this system
NRC COHSI 2/12/200936
Fundamental Human Factors Issues inVisual Surveillance Systems
• Looking for precursors to low probability, high consequence events (e.g., leaving a bomb behind)
• Data overload: screen to camera ratio = 1:4 to 1:30; ratio of operators to screens up to 1:16
• MABA-MABA – humans are better at detecting unusual circumstances, machines are better at detecting small changes in static or clutter environments. Good for exclusion zone monitoring.
• Boredom – some officers play “hide and seek” with on-the-ground personnel, some train the cameras on their own vehicles, etc.
• There is much contextual knowledge that has yet to be codified as systems are in a relatively primitive state
– PNNL estimates that COTS is 60% accurate at best, with tendency to have higher miss rate than false alarm rate
NRC COHSI 2/12/200937
Visual Surveillance System Summary
• Main Purpose: Unusual event detection in peopled space
• Domain of Application: Public spaces, controlled spaces
• Time Frame of Detection: Real-time, post-hoc
• Role of Human Operator: Evaluation and adjudication of system output
• Secondary applications: None
• Nature of Anomaly Data: Point, contextual, collective
• Data Processing Approach: HMM, Bayesian, Numerical Clustering
• Data-to-Construct linkage: Weak (range of behaviors very high)
• Base rate of occurrence: low
• False positive rates: High (high miss rate too)
• Positive Predictive Value of Alarm: Very low
• Ground truth data availability: Limited
NRC COHSI 2/12/200938
Maritime Domain Awareness
NRC COHSI 2/12/200939
Maritime Domain Awareness (DHS national plan)
• The effective understanding of anything associated with
the global maritime domain that could impact the United States’ security, safety, economy, or environment.
• Achieved by improving our ability to collect, fuse, analyze, display, and disseminate actionable information and intelligence to operational commanders and decision makers.
• Integrate relevant Cold War Command, Control, Communications, Computers, Intelligence, Surveillance and Reconnaissance (C4ISR)legacy systems and operational concepts with current and emerging sensor capabilities and applicable procedures. These capabilities
must be fused in a common operating picture that is
available to maritime operational commanders
NRC COHSI 2/12/200940
Automatic Identification System: An MDA “feed” from a civilian-based system (IMO)
• AIS signal includes:
– Ship ID
– Course
– Ship dimensions
– Cargo
– Destination
– ETA
• VHF system using GPS transponder data and ship navigational instruments (gyrocompass, speed indicator, etc)
NRC COHSI 2/12/200941
Nationwide AIS (from NAIS fact sheet)
• AIS data collected by NAIS will be combined with other government intelligence and surveillance data to form a holistic, overarching view of maritime traffic within or near U.S. and territorial waters.
• How will the Automatic Identification System help to increase security? .... by increasing awareness of vessels in the maritime domain, especially vessels approaching U.S. ports. AIS corroborates and provides identification and position of vessels not always possible through voice radio communication or radar alone.
• Corroboration seems to be a key concept that would be served by anomaly detection.
• NAIS personnel also concerned about using anomaly detection to determine if vessels are “spoofing” the system.
• Traffic management applications of AIS are well-underway; security applications just starting.
• NAIS is a system-of-systems
NRC COHSI 2/12/200942
Nationwide Automated Identification System
NRC COHSI 2/12/200943
Maritime Scenarios
Anomaly detection in the maritime domain
Proc. SPIE, Vol. 6945, 2008; Jean Roy, Defense Canada.
NRC COHSI 2/12/200944
MDA/NAIS operational sequence (adapted from National Plan for Maritime Domain Awareness)
• Intel that IND being carried by cargo vessel
• Electronic Notice of Arrival filed by vessel
• ATS notes anomaly in cargo manifest
• AIS vessel track flagged
• COP shows available security assets which are deployed
• Vessel interdicted and cargo seized
NRC COHSI 2/12/200945
MDA system human factors
• Threat indicators need to be fused from diverse systems which are not yet integrated
• Transparency of this process to human system agents will be more complex than stand-alone systems
• Area charts (maps) used when operational tempo is urgent (wide area, persistent, natural interaction, multiple resource tracking)
• Range of threat scenarios extremely broad
• Lack of operational experience
NRC COHSI 2/12/200946
NAIS/MDAAnomaly Detection Summary
• Main Purpose: Threat Detection and Interdiction• Domain of Application: Maritime vessel, cargo, personnel• Time Frame of Detection: Predictive, Real-time• Role of Human Operator: Detection, evaluation and adjudication of
system output• Secondary applications: Traffic Management (?), • Nature of Anomaly Data: Point, Collective, Contextual• Data Processing Approach: Multiple• Data-to-Construct linkage: Weak (range of threat types
considerable)• Base rate of occurrence: unknown but assumed to be extremely low
despite recent spike in piracy• False positive rates: unknown• Positive Predictive Value of Alarm: unknown• Ground truth data availability: probably if vessel boarded
NRC COHSI 2/12/200947
NRC COHSI 2/12/200947
Operator Impairment Detection
NRC COHSI 2/12/200948
Real-time, behavior-based impairment detection complements alcohol interlocks
§ Sensor limits and low baserate
§ Pharmacodynamics—BAC levels can increase while driving
§ Drinking while driving
§ Impairment under .08 BAC
NRC COHSI 2/12/200949
Behavior-based sensors of alcohol impairmentBehavior-based sensors of alcohol impairment
Sensor technology largely available for production vehicles
NRC COHSI 2/12/200950
Decoupling of eyes and steering
Cross-correlogram—steering and eyes
40ml Vodka 100ml Vodka
Marple-Horvat etl al (2008)
Corr
ela
tion
NRC COHSI 2/12/200951
NRC COHSI 2/12/200951
Behavior-based Impairment Detection Summary
• Main Purpose: Real-time behavior-based impairment• Domain of Application: Driving and other safety-critical situations• Time Frame of Detection: Real-time, post-hoc• Role of Human Operator: Interpretation and behavior adjustment• Secondary applications: Fatigue, distraction, and prescription drug
impairment countermeasures• Nature of Anomaly Data: Point, contextual, collective• Data Processing Approach: SVM, Bayesian• Data-to-Construct linkage: Moderate (neurological basis of alcohol
impairment well understood)• Base rate of occurrence: low• False positive rates: High (potentially high miss rate too)• Positive Predictive Value of Alarm: Moderate• Ground truth data availability: Limited unless there is secondary
investigation.
NRC COHSI 2/12/200952
Detection of Teamwork Failures through Communications Monitoring
and Analysis
NRC COHSI 2/12/200953
Team Failures in Collaboration, Communications, Coordination, Command-and-Control
• USS Vincennes shoots down Iranian airbus (1988)
• Challenger/Columbia accidents tied to poor organizational decision making (1986/2003)
• Response to 9/11 reveals communication breakdowns (2001)
• Katrina response lacked coordination (2005)
• Sago Mine disaster report cites poor command-and-control (2006)
• VA Tech communications substandard (2007)
• Friendly fire incidents• Various health care mishaps attributed to
poor teamwork• Unmanned Aerial Systems
• Real-time detection of teamwork breakdowns needed for just-in-time intervention and prevention
• Team communications (voice, text chat, email) provide ongoing data stream for monitoring
• Identify anomalous patterns; detect change
NRC COHSI 2/12/200954
Detecting Meaningful Patterns in Communication Data
Selection of interesting data using cheapest fastest methods
(e.g., number of words, time speaking)
Analysis identifies data in need of further processing
(e.g., communication flow patterns)
Most expensive/detailed analysis on select data
(e.g., content-based analysis)
Communication
timing stability
ChainMaster,
Procedural
Networks
(PRONET),
transition analysis
Semantic
correlations,
Latent Semantic
Analysis Lag
Coherence
DYNAMIC
Avg. time of
following behavior
Following behavior
(Dominance)
Avg. # of words,
Latent Semantic
Analysis,
Communication
Density
STATIC
TIMINGFLOWCONTENT
Abeg
AendPend
Dbeg
Dend
Pbeg
Dbeg
Dend
Pbeg
Dbeg
Pend Aend
Abeg
P-D fight
PRONET: Communication Flow Analysis
Tie patterns to team performance
Tiered application of analytic methods
NRC COHSI 2/12/200955
ChainMaster Deviations from Expected Enron Email Patterns Map onto Organizational Change
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
0.75
1 2 3 4
Time Period
Co
ntr
ol-
Exp
eri
men
tal
Sim
ilari
ty
Enron Files for Bankruptcy
Skilling
Resigns
More
Change
Less
Change
Connect Detected Patterns to Team State
NRC COHSI 2/12/200956
1 sec.
(0,1,1) (0,1,
0)
(0,0,1)
(0,0,0)
(0,0,1)
(1,0,0)
Represent Data to Human
0 500 1000 1500 2000 2500 3000 3500 4000-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5x 10
6
Time (s ec)
Va
r(Y
)
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.50
0.5
1
1.5
2
2.5
3
3.5x 10
13 Spectral Slope = -1.0576
Frequency
Po
we
r
Sonification of change in flow patterns
NRC COHSI 2/12/200957
NRC COHSI 2/12/200957
Teamwork Failure Detection Summary
• Main Purpose: Real-time detection of teamwork failures• Domain of Application: Team communications• Time Frame of Detection: Real-time• Role of Human Operator: Initial analysis, monitoring, detection,
interpretation, and intervention • Secondary applications: Team training, threat assessment• Nature of Anomaly Data: Point, contextual, collective• Data Processing Approach: Sequential data analysis, dynamical systems
modeling, latent semantic analysis, spectral analysis• Data-to-Construct linkage: Moderate (better for structured tasks; can detect
change and anomalies; diagnosis more difficult)• Base rate of occurrence: low• False positive rates: ?• Positive Predictive Value of Alarm: Moderate• Ground truth data availability: Dependent on other existing measures of
team performance and outcome
NRC COHSI 2/12/200958
Summary
• Threat-oriented anomaly detection systems try to address areas where humans underperform machines– Physical occurrences undetectable by human senses
– Making many repetitive observations
– Continuous operations
• Nuisance alarms and low rates of occurrence for events of interest limit utility
• Across a range of phenomena, from basic atomic quanta to individual and collective behavior, the data-to-construct linkage can be strengthened
• Noise in the system, i.e., normal variation, requires human agents to provide context and interpretation
NRC COHSI 2/12/200959
Research Needs
• Catalogue the range of anomaly systems, beyond threat detection,to identify systems with greater and lesser utility
• Identify performance improvements that can be facilitated by human factors analysis
• Foster cross-domain information exchange
• Promote research approaches that incorporate systematic HF studies into technology demonstrations
• Evaluate cost-benefit implications of low utility system staffing versus developmental or no-action alternatives
• Develop systems on basis of positive predictive value of alarms provided
• Approach through workshops and/or consensus studies via NRC