working with big visual cultural data - 2015
DESCRIPTION
Analyzing Big Visual Data: theory, methods, examples Lev Manovich / [email protected] Professor of Computer Science, The Graduate Center, City University of New York (CUNY) Director, Software Studies Initiative www.softwarestudies.comTRANSCRIPT
softwarestudies.com
Analyzing Big Visual Data: theory, methods, examplesLev Manovich / [email protected] Professor of Computer Science, The Graduate Center, City University of New York (CUNY)Director, Software Studies Initiative www.softwarestudies.com
Summer 2015
1
softwarestudies.com 2
All projects included in these slides were created by members of Software Studies Initiative and our collaborators between 2008 and 2015.
Please refer to project descriptions on softwarestudies.com for credits and further details.
softwarestudies.com 3
Research at Software Studies Initiative
softwarestudies.com 4
BACKGROUND:MODERN ART, STATISTICS, SCIENCE,DATA SCIENCE,DATA VISUALIZATION
softwarestudies.com 5
1. TWO MODERN ABSTRACTIONS: STATISTISTICAL GRAPHS (1800-) ABSTRACT ART (1900-)
softwarestudies.com 6statistical graphs, early 1990s
softwarestudies.com 7
Piet Mondrian, 1909-1912
softwarestudies.com 8
2. MODERN ART VS. MODERN SCIENCE (1600-1960):ART: showing general types though the concrete (people, landscape, etc.) SCIENCE: modeling / explaining the regular; concerned with general laws (example: linear regression: y = xB + e)
softwarestudies.com 9
Example of how art represents general through the particular (Aleksander Deineka)
softwarestudies.com 10example of scientific method: modeling the general
softwarestudies.com 11
3. VISUALIZATION WITHOUT REDUCTION?how to combine concrete and abstract? can we create visualizations that show patterns but do not use aggregation and abstraction?
softwarestudies.com 12
One possible “visualization without reduction” method: in our lab we create visualizations show all images in a dataset without reducing them to points, bars, etc. By sorting the images in different ways we can see patterns.
softwarestudies.com 13
SOFTWARE STUDIES INITIATIVE GOALS AND METHODS, 2007-
1. LOOKING AT EVERYTHING AT ONCE 2. SEEING CONTINUOS CHANGES
3. THINKING WITHOUT CATEGORIES?
4. VISUALIZING THE SOCIAL
softwarestudies.com 14
1. LOOKING AT EVERYTHING AT ONCE
- using the complete data (or at least a larger sample that represents the phenomenon well)
- more inclusive cultural history - seeing what has been excluded - mapping contemporary cultural fields
softwarestudies.com 15
We were invited by MoMA to analyze their whole photo collection and contribute to the OBJECT : PHOTO exhibition book
softwarestudies.com 16
Seeing the museum collection: 20,000 photographs from MoMA,1844-1989.
Organized by year (top to bottom). Each bar shows photographs from a particular year.
softwarestudies.com 17
closeup, 1925-1929
softwarestudies.com 18
Visualization of 5000 paintings of French Impressionist artists
x and y - first two dimensions of PCA using 200 features
the familiar impressionist paintings (see closeup on next slide) turn to be only %10-20 of their whole creative output
softwarestudies.com 19closeup
softwarestudies.com 20
Visualizing time-based media and user experiences (films, animations, TV programs, playing a video game):
example: visualizations of films by Dziga Vertov - using one frame from every shot (collaboration with Austrian Film Museum)
softwarestudies.com 21“Kino Pravda” (1921)
softwarestudies.com 22“The Eleventh Year” (1928)
softwarestudies.com 23“A Man with a Movie Camera” (1929)
softwarestudies.com 24“Six Part of the World”
softwarestudies.com 25
Kingdom Hearts gameplay: 62.5 hours, 27 sessions over 20 days (left to right, top to bottom).
softwarestudies.com 26
2. SEEING CONTINUOS CHANGES- visualizing cultural and stylistic changes in time
- seeing continuos historical change (instead of discrete periods / stages)
softwarestudies.com 27
Mark Rothko, 393 paintings,1927-1970. X - year. Y - brightness mean.
softwarestudies.com 28
Animating artistic development: 128 paintings by Piet Mondrian. Animated PCAvisualization using60 features. Images that are visually similar in some ways appear closely together.
softwarestudies.com 29
4535 Time covers 1923-2009.
Organized by date, left to right, top to bottom.
Every pattern we observe is continuous, with changes taking places over years or decades.
softwarestudies.com 30
4535 Time covers 1923-2009.
closeup: 1920s
softwarestudies.com 31
4535 Time covers 1923-2009.
closeup: 1990s-2000s
softwarestudies.com 32
4535 Time covers 1923-2009 (left to right). Each cover is represented by a single vertical line.
softwarestudies.com
Image plots of 4535 Time covers, 1923-2009. X-axis = date; Y-axis = saturation mean.
33
softwarestudies.com 34closeup
softwarestudies.com 35covers that have highest saturation (1960s)
softwarestudies.com 36
Exhibition of our visualizations including Time covers, Graphic Design Museum Breda
softwarestudies.com 37
softwarestudies.com 38
3. THINKING WITHOUT CATEGORIES?
- from categories to continuos descriptions - computer describes properties of media using continuos variables (example: RGB color values) - instead of using a small number of categories, we extract hundreds or thousands of features from every object - - do features determine what we can see in the data?
softwarestudies.com
1 million manga pages x - standard deviation y - entropy
softwarestudies.com 40
Closeups of the bottom left corner and top corner (previous slide). Entropy feature sorts all pages according to low detail/no texture/flat - high detail/texture/3D dimension. Visualization reveals continuos variation on this dimension. This example suggests that our standard concept of "style" may not be appropriate when looking at particular characteristics of big cultural samples (because "style assumes presence of distinct characteristics, not continuos variation across a whole dimension).
softwarestudies.com 41
1 million manga pages plotted as points x - standard deviation y - entropy Some plot areas are densely filled in, while others are almost empty. Why manga visual language developed in this way? Visualization of a large number of samples allows us to map a cultural fields to see what is typical and what is rare, and what kind of clusters (if they exist) are present in this field.
softwarestudies.com 42
single short manga series (>1000 pages).
Does this manga series has a coherent style on the two analyzed dimensions?
softwarestudies.com 43
4. VISUALIZING THE SOCIAL (using visual social media)
- creating portraits of society though social media data
- using social media as lens into society - interactive interfaces for exploring
large visual social media - analysis of contemporary popular
digital photography
softwarestudies.com 44Phototrails project, 2013: analysis of 2.3 million Instagram photos collected in 13 global cities
softwarestudies.com 45
Example of data aggregation - reducing 2.3M photos to 13 data points (one point per city)
softwarestudies.com 46
Another plot of cities differences (using only color features)
softwarestudies.com 47
Comparing San Francisco and Tokyo using 50K image samples. Photos are organized by average brightness (distance to plot center) and average hue (angle).
softwarestudies.com 48Comparing NYC and Tokyo using 50K image samples shared over few days (organized by upload date/time.)
softwarestudies.com 49closeup of the visualization from previous slide
softwarestudies.com 50
Example of data aggregation: comparing Memorial and National Days in Israel (2012). Each plot shows locations of all Instagram images with geo locations shared on that day. Time of day is represented by colors (green-yellow-red). If a user shared photos within a small time interval, they are connected by lines.
softwarestudies.com 51
Instead of aggregating the data for all users, we plot locations of photos by each user separately.
Plots show locations of all Instagram photos by top 289 users in Tel Aviv for 3 months in 2012. Each user photos are in a separate plot.
softwarestudies.com 52
closeup of the visualization (previous slide)
softwarestudies.com 53
Selfiecity project, 2014: analysis of 3200 Instagram selfie shared in 5 global cities. http://selfiecity.net
softwarestudies.com 54One of the visualizations from Selfiecity
softwarestudies.com 55
Screenshot from interactive app selfiexploratory: http://selfiecity.net/selfiexploratory/
softwarestudies.com 56
softwarestudies.com 57SelfieSaoPaolo project, 2014. http://manovich.net/index.php/exhibitions/selfiesaopaulo
softwarestudies.com 58
SelfieSaoPaolo project, 2014. Different views of the animated projection.
softwarestudies.com 59On Broadway project, 2015. http://http://on-broadway.nyc/
softwarestudies.com 60On Broadway is an interactive installation shown at New York Public Library, 12/2004-1/2016
softwarestudies.com 61Artists team in front of On Broadway installation
softwarestudies.com 62Interface uses familiar multi-touch gestures to navigate Broadway street in Manhattan (21 km, 40M data points)
softwarestudies.com 63
Zoomed out view: all of Broadway is visible (21 km)
softwarestudies.com 64
Zoomed in view. User can move along Broadway in 30 meter intervals
softwarestudies.com 66
Video showing interaction with On Broadway interactive application on a 46-inch touch screen
softwarestudies.com 67
Video showing interaction with On Broadway interactive application on a 46-inch touch screen
The Exceptional & The Everyday project, 2014
softwarestudies.com 68
The Exceptional & The Everyday project, 2014 http://www.the-everyday.net/
The visualization shows 13,208 Instagram images shared by 6,165 people in the center of Kiev during 2014 Ukrainian revolution ( February 17 - February 22, 2014). The photos are organized chronologically (left to right, top to bottom). The right column shows summary of the events from Wikipedia page about the revolution.
A single condensed narrative history (Wikipedia text) vs. visual experiences of thousands of people (Instagram)? The second is potentially richer - but also more difficult to interpet.
Can we narrate history without aggregation and summarization? History as timelines of million of people?
softwarestudies.com 69
our projects, papers, free tools: www.softwarestudies.com contact: [email protected]