amir shirkhodaienagi/muri/muri/pdf...network based hard/soft information fusion amir shirkhodaie...
TRANSCRIPT
1
Amir Shirkhodaie Center of Excellence for Battlefield Sensor Fusion
Tennessee State University
3rd MURI Annual Research Progress Review
Meeting
University of Buffalo Buffalo, NY
September 24, 2012
Network Based Hard/Soft Information Fusion Amir Shirkhodaie (Co-PI)
Objectives Develop robust processing and fusion techniques
for inference of group activities based on imaging and acoustic signatures.
Conduct human-in-the-loop experiments for testing and evaluation of new fusion techniques.
Support integration/transition of hard sensor fusion effort for MURI project.
DoD Benefits Automatic techniques for multi-modality hard
sensor data fusion and tagging.
Scientific/Technical Approach
Infer Suspicious group activities via object
involvement probabilities modeling. Bayesian framework for fusion of acoustic and
imagery sensor data. Apply a Kernel-based Frequency Similarity Matching technique to recognize urban environment sounds. Apply Speed-Up Robust Features (SURF) technique for
Salient Events Detection. Conduct human-in-the-loop experiments to generate
multi-modality sensor data for testing and evaluating fusion algorithms and techniques
Generate tagged annotation of hard sensor data.
Accomplishments Demonstrated a methodology for semantic
annotation of group activities based on robust imagery and acoustic sensor data processing.
Demonstrated effectiveness of acoustic and imagery data fusion techniques for robust characterization of complex group activities.
Challenges Application of decision rules for robust fusion of
acoustic and imagery data. Achievement of low false-alarm rate despite of noise
acoustic and imagery data.
3
Group Activity recognition and tagging based
on Imagery and Acoustic Signatures.
Translation of Fused Sensor Information into
TML Messages to Facilitate H/S Fusion.
Conduct of Human-in-the-loop Experiments to address “Real” challenges of this
Transformation process.
Main Scientific/Technical Accomplishments
Research Focus
4
Project Statistics and Summary
STUDENTS INVOLVEMENT Three Ph.D. Students:
Vinayak Elangovan (Imagery Data Fusion)
Amjad Alkalani (Acoustic Data Fusion)
Mohammad Habibi* (Network-based Visual Analytics)
One Graduate Student:
Jerry Sweafford (Sensors Deployment)
Three Undergraduate Students:
Omar Saleem (Simulation Modeling)
Courtney Schuetz (Algorithms Testing)
Shad Studd (Algorithms Testing)
PUBLICATIONS
2009-2010: 7 Papers, 2010-2011: 9 Papers, and 2011-2012: 9 Papers
TECHONOLOGY TRANSITION
Attended SPIE Defense and Security Conference, Orlando FL, April 2012.
Attended 3rd Annual Human and Light Vehicle Detection Workshop, University of Mississippi , May 2012.
* Not supported under MURI funding
5
Main Scientific/Technical Accomplishments
(Years 1 and 2)
Group dynamics are typically involved with:
Multiple entities
Some kind of transportation(s)
Exhibit certain physical behaviors
Unexpected arrival
Loading/Unloading of objects
Exchange of objects/vehicles
Negotiation/dealing
Interactions with environment
Devious departure
Research Accomplishments (Year -3) Understanding of Group Dynamics
Understanding of Group Dynamics is crucial for:
Battlefield Applications
Homeland Security Applications
Taxonomy of Group Dynamics
Three Taxonomy of Group Dynamics: 1. Human-Human Interaction (HHI) 2. Human-Object Interaction (HOI)
3. Human-Vehicle Interaction (HVI)
Three Taxonomy of Group Dynamics: 1. Human-Human Interaction (HHI) 2. Human-Object Interaction (HOI)
3. Human-Vehicle Interaction (HVI)
Three Taxonomy of Group Dynamics: 1. Human-Human Interaction (HHI) 2. Human-Object Interaction (HOI)
3. Human-Vehicle Interaction (HVI)
8
Group Activity Micro-Vignette Related to Sunni Message Sets
(Experiment 1)
9
Group Activity Micro-Vignette Related to Sunni Message Sets
(Experiment 2)
10
Group Activity Micro-Vignette Related to Sunni Message Sets
(Experiment 3)
11
Group Activity Micro-Vignette Related to Sunni Message Sets
(Experiment 4)
12
Group Activity Micro-Vignette Related to Sunni Message Sets
(Experiment 5)
14
Experiment: Effectiveness of Kinetic Camera in Dark Outdoor Environment
16
Group Activity Recognition And
Semantic Annotation
Taxonomy of Acoustic Sounds Identification and Classification
2nd-Year
Identification of Acoustic Sounds Related to SYNCOIN Operations
Acoustic Signature Analysis Based on STFT Spectrogram
23
Stages of Acoustic Signatures
Processing and Annotation
24
Local Keypoint descriptor
Image Processing Approach for
Target Detection and Recognition
26
Annotation of Hard Sensor Data
(Joint TSU-PSU Off-Campus Experiment)
30
Future Research Plan (2012-2014)
Research Goals:
Specific Research Objectives (2012-2013):
Extend the HVI knowledge representation scheme (viz. , the semantic and parametric representation of actions, entities and events) to more general activities, events and entities found in the SYNCOIN and related data collections.
Participate with PSU on additional data collections and analyses in the off-campus experiments.
Develop and approach for distributed group behavior through entity-role interactions.
Participate in distributed fusion collection/analysis experiments and implementation.
Extend the group activity knowledge representation scheme (viz. , the semantic and
parametric representation of actions, entities and events) to more complex SYNCOIN operations;
Develop a hybrid (virtual/physical) environment for testing simulated human-in-the-loop experiments at TSU
Develop a client-server capability for sensor data processing and integration with PSU/UB SOA data fusion architecture
31
Future Research Plan (2013-2014)
Specific Research Objectives (2013-2014):
Adapt Hard sensor data fusion algorithms to real-time SYNCOIN related activities;
Develop sensor data fusion techniques for accommodating multi-modality participatory sensor data;
Participate with PSU on additional data collections and analyses both off-campus and in virtual environments experiments;
Develop and approach for distributed group behavior through entity-role interactions; and
Participate in transition of technology to the Army.
32
Conclusion
Developed an Architectural Framework for Analysis of Group Activity Characterization, Tracking, and Classification.
Demonstrated techniques for classification of acoustic and imagery sensor data via different noise tolerant classifiers
Collaborated with PSU and UB in generating and analysis of SYNCOIN Data
Sets.