1

Visual Processing for Social Media

Andrew C. GallagherTsuhan Chen

September 30, 2012

Cornell University

Outline

Social Media Overview Visual Processing Overview Social Media Insights Within the Image Social Media Insights From Sharing

2

3

Now, pictures of people Examples of how social data has helped

understand images of people Some things I’ve learned about people

from computer vision

4

Understanding images of people

5

What the computer sees

6

Faces in the lab

[Turk et al., Cog. Neuro. 1991]

[Belhaumer et al., PAMI 1997]

[Wiskott et al., PAMI, 1997] [Lucey et al., IJCV 2007]

[Blanz et al., PAMI 2003]

[Kanade, Kyoto U. 1973]

8

Is this a family?

9

The Loop

Images and Computer Vision

What we know about people

10

Understanding Images of People

Describe people: How tall? How old? Identify people: Who? Why are they together? Exploit the same context humans use!

11

Understanding Images of People

Capture Context Social Context

July 2, 20058:27 PMLat: 42.2902Long: 85.5361

June 25, 200510:50 AMLat: 42.3202Long: 85.1261

12

Understanding Images of People

Capture Context Social Context

Adult male height: 177 cmAdult female height: 163 cmMLE mother-child: 27 yearsMLE husband-wife: 2 yearsMLE |sibling-sibling|: 6 years

13

What is social context?Social Context: information about people and their society that is useful for understanding images. Distributions of ages and genders in social

groups Social relationships Face position in a group image First name popularity over time Anthropometric measurements

14

Group Images What we know and learn about people:

Group dynamics Computer vision task:

Measuring age, gender, of each person in a group

15

Images of Groups

Identify age and gender Recognize certain group events Consider context and appearance

[A. Gallagher, T. Chen, CVPR 2009]

16

Contextual Features

Absolute face position Size, position relative neighbor and group Minimal spanning tree degree

17

Evidence of Social Context

Relative positions of nearest neighbors depends on the social relationship

Mean distance is 306 mm

Neighbors Male to Female Other to Baby

19

Evidence of Social Context

Samples of faces based on image locationRandom samples

20

Use All Context

5080 images with 28,231 faces Classification improves with more

contextual features

21

Appearance Features

Project face into Fisher Space Nearest neighbor density estimation

Gender subspaceNearest neighbors

22

Gender Estimation

Context Appearance Combined

24

Context and Appearance

Context contributes more when appearance is weak.

Context Appearance Combined

All Faces Small Faces

25

Context for Scene Geometry

Find the face vanishing lineEstimated horizon from face positionsManually labeled horizon

26

Context for Dining Event

Group Structure = Activity

27

Row Segmentation

[A. Gallagher, T. Chen, ICME 2009]

29

Row Segmentation

30

Row Segmentation

31

Social Relationship Retrieval

Spouses

Mother-Child

[G. Wang, A. Gallagher, J. Luo, D. Forsyth, ECCV 2010]

32

Names as Context What we know and learn about people:

Government census data Computer vision task:

Matching names to faces. Guessing age and gender.

33

First names capture information about age and gender

First names are social context

Person A and Person B

First Names as Context

Mildred and Lisa

1900 1920 1940 1960 1980 20000

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Birth Year

Pro

babi

lity

Probability of Birth Year

LisaMildred

1900 1920 1940 1960 1980 20000

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Birth Year

Pro

babi

lity

Probability of Birth Year

MildredLisaNoraPeytonLinda

Source: U.S. Social Security Administration

[A. Gallagher, T. Chen, CVPR 2008]

34

First names and appearance? Tom_101 Ben_165 Caleb_337 Andrew_233 Brian_116 Zachary_431

1953 1956 2003 1984 1962 1996

Abigail_194 Heather_224 Alejandra_152 Juanita_192 Ethel_165 Gertrude_532002 1970 1977 1947 1926 1924

35

Gertrude_531924

Sort by Expected Age Tom_101

1953Ben_165

1956

Caleb_3372003

Andrew_2331984

Brian_1161962

Zachary_4311996

Abigail_1942002

Heather_2241970

Alejandra_1521977

Juanita_1921947

Ethel_1651926

36

First Names as Context

Mildred and Lisa

Name

Birth Year

AgeFeatures

Gender

GenderFeatures

Image-Based Features

37

More Context = Better Results

Appearance First Name Full Model

38

Recognition from First Name

The model improves name assignment, age estimation, and gender classification

39

Learning about people

39

Images and Computer Vision

What we know about people

40

Group Images How close do people stand in group

photos? Computer vision answer: 306 mm Sociology’s “Personal Space”: 457 mm

Do people suspend personal space needs during photograph?

41

Group Images: Gender Prior How do people end up in a group photo

anyhow?

42

Group Images: Gender Prior Bernoulli world?

Implicit prior, IID: Let’s look at the data!

43

“Group Shots”

Number of Females

Gender Distribution of 6 peopleBinomial Distribution

Number of Females 0 1 2 3 4 5 6

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 1 2 3 4 5 60

0.05

0.1

0.15

0.2

0.25

0.3

0.35

?

0 1 2 3 4 5 60

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Number of Females

?

“Family”

Actu

al D

istrib

ution

s

Genders of people in a image are not independent!

44

Group Shot Analysis Standing Order Frequency for 4 people (2

male, 2 female):0.13

0.11

0.19

0.13

0.30

0.15

But why?

45

Learning about people

4545

Images and Computer Vision

What we know about people

(what they do and think!)

46

Social Context Data Summary U.S. Social Security First Name Database

6693 first names, birth years, gender U.S. CDC National Center for Health

Statistics Physical growth tables Birth rates and other birth statistics Family structure statistics

Farkas, 1994 Facial anthropometric measurements

47

Conclusions Social context is useful for interpreting

single images or image collections Social context is learned from images or

other public sources Learning about people improves our

understanding of images of people

48

Related Publications J. Xin, A. Gallagher, L. Cao, J. Luo, J. Han

The Wisdom of Social Multimedia: Using Flickr For Prediction and Forecast

ACM MM 2009

G. Wang, A. Gallagher, J. Luo, D. Forsyth

Seeing People in Social Context: Recognizing People and Social Relationship

ECCV 2010

A. Gallagher, A. Blose, T. Chen

Jointly Estimating Demographics and Height with a Calibrated Camera

ICCV 2009

A. Gallagher, T. Chen Using Context to Recognize People in Consumer Images IPSJ Trans. on Comp. Vis. and Apps., 2009

A. Gallagher, T. Chen Understanding Images of Groups of People CVPR 2009

A. Gallagher, T. Chen Finding Rows of People in Group Images ICME 2009

A. Gallagher, C. Neustaedter, J. Luo, L. Cao, T. Chen

Image Annotation Using Personal Calendars as Context ACM MM 2008

A. Gallagher, T. Chen Estimating Age, Gender and Identity using First Name Priors

CVPR 2008

A. Gallagher, T. Chen Clothing Cosegmentation for Recognizing People CVPR 2008

P. Singla, H. Kautz, J. Luo, A. Gallagher

Discovery of Social Relationships in Consumer Photo Collections Using Markov Logic

CVPR SLAM 2008

A. Gallagher, T. Chen Using a Markov Network to Recognize People in Consumer Images

ICIP 2007

A. Gallagher, M. Das, A. Loui

User-Assisted People Search in Consumer Image Collections ICME 2007

A. Gallagher, T. Chen Using Group Prior to Identify People in Consumer Images CVPR SLAM 2007