damage assessment from social media imagery data during disasters
TRANSCRIPT
Damage Assessment from Social Media Imagery Data During Disasters
Dat T. Nguyen, Ferda Ofli, Muhammad Imran, Prasenjit MitraQatar Computing Research Institute, Qatar
The Pennsylvania State University, University Park, PA, USA
Partners & Clients:
New York (Suffolk) Emergency Management Dept.
Types of Information on Twitter
- Twitter data from 13 recent crises
- Over 100,000 tweets
- Information types
- Types of sources
Source: Qatar Computing Research Institute - Published in World Humanitarian Data and Trends 2014 (UN OCHA)
The Value of Timely Information During Disasters
Based on FEMA large-scale survey among emergency management professionals across the US.
Info
rmat
ion
val
ue
When information is too late
The Value of Timely Information During Disasters
Based on FEMA large-scale survey among emergency management professionals across the US.
Info
rmat
ion
val
ue
When information is too late
2013 Pakistan EarthquakeSeptember 28 at 07:34 UTC
2010 Haiti EarthquakeJanuary 12 at 21:53 UTC
Social Media Data and Opportunities
Social MediaPlatforms
Availability of Immense Data:
Around 16 thousands tweetsper minute were posted duringthe hurricane Sandy in the US.
Opportunities:
- Early warning and event detection
- Situational awareness
- Actionable information
- Rapid crisis response
- Post-disaster analysis
Disease outbreaks
“A picture is worth a thousand words.”Images from 3 Different Disasters
Time-Critical Events and Information Gaps
Info. Info. Info.
Disaster event (earthquake, flood) Destruction, Damage
Information gathering
Humanitarian organizations and local administrationNeed information to help and launch response
Information gathering, especially in real-time, is the most challenging part
Relief operations & reconstruction
Disaster
Government orgs.
Tweet4Act: Automatic Image Processing Pipeline
Presented at ASONAM’17 as demo
Damage Severity Assessment from Images
Task: Our Task is to classify each incoming imageInto one of the three classes.
Challenges
• Task complexity: lack of labeled data, ill-defined objects
• Poor signal-to-noise ration: social media data is extremely noisy. E.g., duplicates, irrelevant
• Task subjectivity: confusion between damage severity classes “severe” and “mild”
• Cold-start issue: first few hours of a disaster are critical, learning ML classifiers needs labeled data
Images Datasets: Twitter + Google
Twitter messages collected using
- Damaged building- Damaged road- Damaged bridge
Queries we used:
Human Annotations
We used AIDR (volunteers) and Crowdflower (paid workers)
The purpose of this task is to assess the severity of damage shown in an image…
1. Severe damage
Substantial destruction, a non-livableOr non-useable building, a non-crossable Bridge, a non-drivable road
2. Mild damage
Damage generally exceeding minor (e.g., 50% of a building is damaged), partial loss of amenity/roof, part of bridge is unusable or needs repairs
3. Little-to-no damage
Images that show damage-free infrastructureOr small cracks, wear and tear due to age
Three classes:
Instructions:
Human Annotations
We used AIDR (volunteers) and Crowdflower (paid workers)
Crowdflower annotations
AIDR was used during the actual event.
Learning Schemes
1. Baseline (PHOW + SVM): Pyramid Histogram of Visual Words (PHOW) featureswith linear SVM
2. Pre-trained CNN as feature extractor: We used VGG-16 network trained on the ImageNet dataset1.2M images and 1000 classes. We used fc7 layer i.e., removed the last layerto get a 4097-dimensional vector for every image.
3. Fine-tuning a pre-trained CNN: Used existing weights of a pre-trained CNN as an initialization for our datasetWhere last layer representing our task (3 classes)
Learning Settings
1. Event-specific setting: Training, development, and test sets are form the same eventTrain: 60%, Dev = 20%, Test = 20%
2. Cross-event setting: Scenario: no labeled data for the target event. Labeled data from past events is abundant.
Cross-event: train on past events (source) and test on current event (target)
For example: Train: Nepal earthquake + Ecuador earthquakeTest: Typhoon Ruby
We use Google data assuming no past event data is available
Event-Specific Results
Cross-Event using Ecuador and Matthew as Test
Ecuador earthquake (20%) as fixed test set and all sources with 60%
Hurricane Matthew (20%) as fixed test set and all sources with 60%
Event-Specific Precision-Recall Curves and AUC
Cross-Event Precision-Recall Curves and AUC
Conclusions
• We presented results for the task of damage assessment from social media images
• We used real world datasets
• Compared non-deep learning, deep learning and transfer learning approaches
• In the event-specific case, transfer learning approach performs better
• In the cross-event case, we observed the more the data the better, same event data always helps
Thanks – Q & A@aidr_qcri