institute for collaborative innovation (ici) 2006: rigor in...
TRANSCRIPT
Institute for Collaborative Innovation (ICI) 2006: Rigor in Information Analysis
Emily S. Patterson, PhD
Research Scientist Associate Director, CPoD
Ohio State University Proprietary – please contact [email protected] !
Overview of research/design on data overload
Rigor
Study
Strategies
Process Metric
Angola 2011 Scenario-based exploration of design ideas
Outline
Ohio State University (OSU) consortium:
Innovate solutions to data overload (beyond “tweaks”)
Advance methods for envisioning useful support for information
analysis and comprehension (IA&C)
Develop interdisciplinary talent (Cognitive Systems Engineering,
Design)
Innovate by pursuing leverage points:
Process vulnerability with high consequences for failure
Innovative technological or organizational capability
Grounded basis for predicting performance improvement
Converging Perspectives on Data (CPoD)
Cross-Cutting Research Themes
1. Integrated end-to-end workflow
2. Break hypothesis fixation using multiple perspectives
3. Find critical data via emergent collaboration
4. “Actionable” intelligence by judging analytic rigor
Down Collect Conflict and Corroboration
Hypothesis Exploration
CPoD Intelligence Analysis Studies
1: Interview 5 analysts on workflow differences
2: Perspectives case studies: Libyan shootdown, Zaire, Middle East
3: Observe 4 teams collaborating on WWII counter-insurgency
4: Interview 10 analysts on process rigor behind two products
Observe 10 experts on Ariane 501 rocket accident Interview 46 analysts, instructors, support personnel Observe team during Stability and Support Ops (SASO) Interview 6 experts who critique novice on Ariane 501
Foundational Research: Challenges in Intelligence Analysis
Targeted Inquiry on Research Themes
©2000 Christoffersen, Woods, Malin
Jumpstarting Innovation with Research
Design Seed
Modular design concept
Generalizes across software, architecture, scenario, domain
Instantiated in a case
Animated mockup (ani-mock)
Addresses process vulnerability
Leverages technological advance
Multiple levels for relying on machine processing
Elicits feedback on concept “usefulness” and state of technology
CPoD “Seed Book” for Intelligence Analysis
“High profit” attribute search Magnified phrases workspace Circular reporting alerts Automated “bookkeeping” functions Automated event detection Mixed-initiative update detection Visual narratives workspace
Theme 1: Integrated workflow
Query expansion based on analyst perspective (bias) Automated detection of similar queries & analyses
Theme 2: Multiple perspectives
“Open” spaces for cross-checking Meta-tag ontologies for “snippets”
Theme 3: Emergent collaboration
Institute for Collaborative Innovation (ICI) 2006
Theme 4: Improve and assess rigor
Definition: Rigor
for
of
Strategies to Increase Rigor (apprenticeship learning from gurus)
Peer review Specialist consults Devil’s advocates
Personal “vetted” files “Read-on” immersion
Read for and against Prediction diversity sampling Perspective (bias) analysis
Read old to new Timelines
“Build a house” around question 360 degree analysis
Corroborate data Separate facts and assumptions Reminders of uncertainty Track “to be included” info
Organize documents Notesheet organization
Referencing
NASA Columbia Accident Investigation
"Another lack of rigor cited by the panel is the widespread use of PowerPoint presentations in lieu of actual engineering data and analyses.”
Strategies to Infer Rigor
Relationship to others: - Consensus view - Extreme opinion
Reasoning: - Warrant for inferences - Believability - Certainty
Analyst competence: - Reputation (individual, organization) - Ability to answer questions - Credentials - Professionalism Presentation:
- Clarity - Polish
Down Collect Conflict and Corroboration
Hypothesis Exploration
Broadening checks (“up” arrows) increase rigor
Elm, W., Potter, S., Tittle, J., Woods, D.D., Grossman, J., Patterson, E.S. (2005). Finding decision support requirements for effective intelligence analysis tools. Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting.
Aspects of (Sufficient) Process Rigor
Analyst - Analyst
Analyst - Supervisor
Supervisor - Policymaker
Comparison of Two Processes
The End
©1999 Tinapple, Woods, and Patterson
Exploration of Concept Usefulness
Example: Al Qaeda Iraq Pre-War Connection
Principle Component 1
Principle Component 2
Little or no connection
Neutrals Strong connection
High profits
Compressed semantic space
“911 Report”
Mylroie testimony
Query
“The Real Connection”
Example: Search Results
Query: Al Qaeda Iraq Connection
*Rigor by sampling best of multiple perspectives
Nearest Neighbor (google ‘98)
Diversity (CMU)
Tuned Diversity* (Our vision)
1 Selling An Iraq Al Qaeda Connection
1 Selling An Iraq Al Qaeda Connection
911 Report
2 General…Connection …is Nonexistent
2 Bush Defends Assertions…
Mylroie Testimony
3 Ties…Footnote Fahrenheit 3 Poverty and Low Education…
The Real Connection
Example: Hypotheses and Sources
Connection Hypotheses Key Sources
Strong Cell leader worked for Saddam 911 Report
& Mylroie testimony
Medium Cell leader met with Saddam 911 Report
& Mylroie testimony
Saddam developed insurgent Al Qaeda contacts
Weak Saddam made a mural celebrating 911 The Real Connection
Masri and Zarqawi started a Bagdad cell in 2002
None There was no connection. The Real Connection
Confirmed Disconfirmed
Future Directions
In down-collect, analyst wants: Awareness of multiple perspectives → diversity goal High profit documents → tuning goal
Planned Contributions: “meta-methods” to tune index and search parameters
Insights (e.g., which settings foster speed and profit) NOT replace existing indexing methods and search engines
Run Analyst … Engine Parm. 4 # Errors Speed (ms)
1 Fred … 0.92 4 62
2 Sally … 0.20 0 33
40 Sally … 0.20 3 115
Example experimental plan
The End
Innovations for Information Analysis and Comprehension A Consortium at The Ohio State University
Distributed Tasking
Institute for Collaborative
Innovation
for Hypothesis Generation
Kidnapping is for ransom from oil
company
(electronics are expenditure)
Xt = Xt-1
espionage
on-line propaganda
insurgent-style video terrorism
cyber-attacks
selling stolen equipment for $
The End
Likelihood
Ris
k
H0
H1
H2
H3
Data 1
Data 27
Data 33
Data 45
Likelihood
Ris
k
H0
Data 1
Data 27
Data 33
Data 45
Likelihood
Ris
k
H0
H1
H0.1
H2
H3
Rebel/Congo joint attack planned
Likelihood
Ris
k
Rebels have gained outside support
Rebels planning attack
Chinese language intercepted on Govt. channels
Rebel attacks increase Rebels have gained support from Congo
Rebels have gained support from Mid East
Day 1 Day 2
Evidence of rebel planning activity
Evidence of rebel planning activity
Evidence of rebel planning activity
No French or Kituba language traffic
Rebels planning attack Rebel/Mid
east terrorist joint attack planned
Arabic language traffic on rebel channels
Day 3
Mid East business deal is benign
Chinese planning to invest in Angolan oil
Rebels/Mid East allied to destroy embassy/ derail Chinese oil investment.
Chinese embassy implicated as rebel target
Chinese message translated: “… oil facility security…”
Arabic signals are regarding a benign business deal
Mid east bomb-making materials being smuggled
Anti-rebel actions have had an effect
The End
BRIEFING INTERACTION
BRIEFING INTERACTION
MESSAGE BRIEFER
AUDIENCE LIMITED FEEDBACK
Briefing Interactions: Traditional Model
BRIEFING INTERACTION CYCLE
BRIEFING ���ANALYST AUDIENCE���
PARTICIPANTS
ANALYSIS BRIEFING
INFORMING
TASKING
Analyst���Supervisor
Decision Maker
Briefing Interactions: Participatory Exchange Model
BRIEFING INTERACTION CYCLE
BRIEFING ���ANALYST AUDIENCE���
PARTICIPANTS
ANALYSIS BRIEFING
INFORMING
TASKING
ANALYSIS REPLANNING
REPLANNING ANALYSIS
CONVERGING PERSPECTIVE
Analyst���Supervisor
Decision Maker
Briefing Interactions: Participatory Exchange Model
The Participatory Exchange
• Driven by Participatory Interaction
• Conversation with the Audience
• Supports Re-Tasking
• Approach is Supported by...
- Empirical Analyst Reports
- LNG Scenario Walkthrough Study
- Laws of Cognitive Work
The Participatory Exchange: Summary
Active Participant Familiarity with discussion topic Understanding of unique viewpoint
Guides through data space Integrates feedback Unifies idea themes Controls flow of the exchange
Presenter Audience
The End