towards understanding crisis events on online social networks through pictures
TRANSCRIPT
Towards Understanding Crisis Events On Online Social Networks Through Pictures
IEEE/ACM Conference on Advances in Social Networks Analysis and Mining (ASONAM), 2017
Prateek Dewan, Anshuman Suri, Varun Bharadhwaj, Aditi Mithal, Ponnurangam Kumaraguru
Precog@IIITD
Indraprastha Institute of Information Technology – Delhi (IIITD)
http://precog.iiitd.edu.in
Who am I?
• PhD student at IIIT-Delhi, India• 2012 – present
• Masters (Information Security), IIIT-Delhi (2010 – 2012)
• Funded by the Government of India, IIIT-Delhi, IBM, National Internet eXchange of India (NIXI)…
• Part of Precog@IIITD• Privacy, eCrime, Online Social Networks, Data Science for Security and Privacy
• Research interests• Privacy and Security in Online Social Media, Web Security, Machine Learning
• Data Scientist at Apple
2
http://precog.iiitd.edu.in
The Human Brain: Images versus text
• Human brain processes images 60,000 times faster than text
4
http://precog.iiitd.edu.in
“A Picture Is Worth A Thousand Words”
• Images are the latest way of communicating on OSNs• 1.8 billion+ pictures shared on Online Social Networks every day
• Images attract much more attention and engagement as compared to text• Tweets with images get 18% more clicks, 150% more retweets
• 93% of most engaging content on Facebook has an image
5
http://precog.iiitd.edu.in
Are we doing enough to "understand" images?
• Most research to analyze social media content focuses on text• Topics are understood using topic modelling on text
• Sentiment is understood by subjecting textual content to linguistic techniques
• Is that enough? Does it capture everything?
• Studies related to images are limited to small scale• Few hundred images manually annotated and analyzed
• What can be done?• Automated techniques for image summarization using Deep Learning and
Convolutional Neural Networks (CNNs) to scale across large no. of images
• Domain transfer learning: Using existing knowledge in one domain to understand another domain
• Optical Character Recognition
6
http://precog.iiitd.edu.in
What do we study?
• Crisis event• Terrorist attacks in Paris, France in November 2015
• Images on Social Networks• Facebook
• Data collection – Facebook Graph API Search• #ParisAttacks
• #PrayForParis
7
Unique posts 131,548
Unique users 106,275
Posts with images 75,277
Total images extracted 57,748
Total unique images 15,123
http://precog.iiitd.edu.in
Methodology
• 3-tier pipeline for extracting high level image descriptors from images
8
Images
Themes(Inceptionv3)
ImageSentiment(DeCAF trainedon
SentiBank)
OpticalCharacterRecognition
Humanunderstandabledescriptors
TextSentiment(LIWC) +Topics(TF)
Manualcalibration
Tier1:VisualThemes
Tier2:ImageSentiment
Tier3:Textembeddedinimages
http://precog.iiitd.edu.in
Tier I: Visual Themes
• ImageNet Large Scale Visual Recognition Challenge (ILSVRC), 2012• 1.2 million images, 1,000 categories
• Winner: Google’s Inception-v3 (top-1 error: 17.2%)• 48-layer Deep Convolutional Neural Network
9
http://precog.iiitd.edu.in
Tier I: Visual Themes contd.
• All images labeled using Inception-v3
• Validation: • Random sample of 2,545 images annotated by 3 human annotators
• 38.87% accuracy (majority voting)
• Manual calibration• Renamed 7 out of the top 30 (most frequently occurring) labels
• New accuracy: 51.3%
• Why rename?
10
Bolo Tie(Inception-v3)
PeaceForParis(Our dataset)
http://precog.iiitd.edu.in
Tier II: Image Sentiment
• Domain Transfer Learning
• Inception-v3’s last layer retrained using SentiBank
• SentiBank• Images collected from Flickr using Adjective Noun Pairs (ANPs) as search
query
• ANPs: happy dog, adorable baby, abandoned house
• Weakly labeled dataset of images carrying emotion
• Final training set – 133,108 negative + 305,100 positive sentiment images
• 10-fold random subsampling
• 69.8% accuracy
11
http://precog.iiitd.edu.in
Tier III: Text embedded in images
• Optical Character Recognition (OCR)• Tesseract OCR (Python)
• 31,689 images had text
• Manually extracted text from a random sample of 1,000 images
• Compared with OCR output using string similarity metrics
• ~62% accuracy
12
Tesseract output:
No-one thinks that these people are representative of Christians. So why do so many think that these people are representative of Muslims?
http://precog.iiitd.edu.in
Findings: Top visual themes
14
Label Count Description
Website 12,416 Images of posts, tweets, banners, etc.
Book jacket * 5,383 Posters, banners, etc.
Comic book 3,803 Cartoons, animated posters and memes
Fountain 1,264 Fountain in front of the Louvre museum, other fountains
Envelope * 1,248 Posters, banners, etc.
Suit (clothing) 1,246 People wearing suits, at gatherings etc.
Stage 1,135 Stages during public speeches, mass gathering events, etc.
Candle waxlight 1,021 Lit candles and lamps offering support to victims
Malinois # 995 Police dog that died during the attacks
Scoreboard # 971 Images of sports stadium
http://precog.iiitd.edu.in
Image and post text had different topics
• Text embedded in images depicted more negative sentiment than user generated textual content
16
Text embedded in images User generated text
http://precog.iiitd.edu.in
Findings
• Image sentiment was more positive than text sentiment
17
0
0.1
0.2
0.3
0.4
0.5
0.6
8 24 40 56 72 88 104 120 136 152 168 184 200 216 232 248 264 280
SentimentValue/VolumeFraction
No.ofhoursaftertheattacks
PostText ImageText
Image VolumeFraction
http://precog.iiitd.edu.in
Contributions
• Insights into the visual side of content during crisis events on social networks
• Generalizable methodology / pipeline for analyzing large topical image datasets
19
http://precog.iiitd.edu.in
Limitations
• Object detection technique has limited accuracy• Retraining is costly; we prefer manual intervention over retraining
• Sentiment portrayed by an image can be subjective
• OCR does not always produce good results• Missing out on part of the content
20