raskar emtech2010 mar final

Raskar, Camera Culture, MIT Media Lab

Camera Culture

Ramesh Raskar

Camera CultureMIT Media Lab http://raskar.info

http://cameraculture.info

Ramesh RaskarAssociate Professor

Future of Imaging

Film-like Digital Photography

Cameras Phones are EverywhereCameras Phones are Everywhere

Kush R. Varshney, “Jagannath Temple,” Puri, Orissa, India, Dec. 2007.

Wish List TodayWish List Today

• ConsumersConsumers– Super-human visionSuper-human vision– Microscope like resolutionMicroscope like resolution– High speed (burst, video, no blur)High speed (burst, video, no blur)– See inside the body (health)See inside the body (health)– Auto-triggerAuto-trigger– Battery lifeBattery life– Zero start or shutter delayZero start or shutter delay– Keep only ‘good’ picsKeep only ‘good’ pics– Find ‘relevant’ pics and better Find ‘relevant’ pics and better

archiving/accessarchiving/access

– Put photographer back in photo!Put photographer back in photo!

• CompaniesCompanies– CostCost– ResolutionResolution– Low-light sensitivity, HDRLow-light sensitivity, HDR– ((For cameraphoneFor cameraphone) Size and depth) Size and depth– Stereo and 3DStereo and 3D– Mecha-free zoom/focusMecha-free zoom/focus– Auto-tagging for sharingAuto-tagging for sharing– RecognitionRecognition

Can you look around a corner ?

Convert LCD into a big flat camera?

Where are the ‘camera’s?Where are the ‘camera’s?

We focus on creating tools to better capture and share visual information via a

new class of imaging platforms

Ramesh RaskarRamesh Raskar http://raskar.info

What is the Media Lab?

A Graduate Program in the Media Arts & SciencesHouses ~150 students and 30 PIs

A Research Lab that spans across disciplines and academic/ industrial lines65 sponsor companiesSponsors get free, non-exclusive licenses for ML IP

Founded in 1985 by Nicholas Negroponte and Jerome Wiesner

Close Ties With Industry

05/03/23 R E S E A R C H S T R A T E G Y 18

Our 65 corporate sponsors include some of the most creative companies in the world

Smart Cities

What if cars could stack like shopping carts in cities?Ryan Chin and Bill Mitchell

ADAPTABILTIY

Hyper -

Adaptability

Bio-Mechatronics

Music/Mind/Health

People -Sense

Sociable Robots

Neuro-MediaHUMAN

ADAPTABILITY

Bio-Mechatronics

Neuro-Media

Sociable Robots

People -Sense

Music/Mind/Health


Camera Culture

Ramesh Raskar

Ramesh RaskarAssociate Professor, MIT Media Lab http://raskar.info

New Emerging TechnologiesMedical Imaging

EntertainmentUser InterfacesIndustrial Vision

The impact of Next Billion CamerasMovie-making, news reporting

Social stability

Visual Social ComputingImage-based commerce

Future of Imaging

Ramesh Raskar, MIT Media Lab

Medical Imaging Today .. Medical Imaging Today ..

http://info.med.yale.edu/intmed/cardio/imaging/techniques/ct_imaging/http://info.med.yale.edu/intmed/cardio/imaging/techniques/ct_imaging/

Self Evaluation of Eye

What are the problems with ‘real’ photo in conveying information ?

Why do we hire artists to draw what can be photographed ?

Shadows

Clutter

Many Colors

Highlight Shape Edges

Mark moving parts

Basic colors

Cartoon CameraCartoon CameraUsing Depth EdgesUsing Depth Edges

Canny Intensity Edge Detection

Our Method

Photo Result

Convert LCD into a big flat camera?Beyond Multi-touch: 3D Gestures

Large Virtual Camera for 3D Interactive HCI and Video Conferencing

Matthew Hirsch, Henry HoltzmanDoug Lanman, Ramesh Raskar

Siggraph Asia 2009

BiDi Screen

Touch + Hover using Thin, Depth Sensing LCD Sensor

Sensing Depth from Array of Virtual Eyes in LCD


Camera Culture

Ramesh Raskar

Understand the World•Identify/recognize Objects

•3D Awareness•Interact with embedded

information

Can you ‘see’ around a corner ?

Femto-Photography: Higher Dimensional Capture

FemtoFlash

UltraFast Detector

Computational OpticsSerious Sync

Application: Rescue and Planning

Bokode

Defocus blur of Bokode

Coding in Angle

Mohan, Woo, Smithwick, Hiura, Raskar [Siggraph 2009]

camera

sensor

circle of confusion circle of information

Bokode(angle)

Quote suggested by Kurt Akeley

Encoding in Angle, not space, time or wavelength

cell-phone cameraclose to the Bokode

(10,000+ bytes of data)


Camera Culture

Ramesh Raskar

Understand the World•Identify/recognize Objects

•3D Awareness•Interact with embedded

information

Visual Computing

Forward (Synthesize)– Special Effects, Image Processing, Design

Inverse (Understand)– Human-like performance– Recognizing/finding objects, image search– Challenging !

Power ofthe

Processor

Power of

theNetwork

Powerof

People

1985 1995 2005 2015

Powershifts

CrowdsourcingCrowdsourcing

http://www.wired.com/wired/archive/14.06/crowds.html

Amazon Mechanical Turk: Steve Fossett search

ReCAPTCHA=OCR

Visual Social Computing

• Image-based commerceBy the people, for the people, of the people

• Next Trend in India ?Harnessing human power beyond language barriers

• Business Model ?Outsource → Image-based Crowdsource

Cameras in Developing CountriesCameras in Developing Countries

http://news.bbc.co.uk/2/hi/south_asia/7147796.stm

Community news program run by village women

Developing Countries: CAMForms• Paper forms with

barcodes• 83-bit 2D codes

(including seven bits of error correction)

Parikh (2005)

User Generated Content Visualization

• Google Map overlayed with geo-tagged photos

• Image-based Mashups

http://phototour.cs.washington.edu/

Photo Synth

Socio-Political Goals: B’TselemSocio-Political Goals: B’Tselem

From websiteof : Israeli Information Center for Human Rights in the Occupied Territories

Truth in ImagesTruth in Images

From Hany Farid

LA Times March’03


Camera Culture

Ramesh Raskar

How will the next billion cameras change the social culture ?

How can we augment the camera to support best ‘image search’ ?

How will camera improve trust and social stability ?

How will movie-making, news reporting change ?

Next model for image-based commerce ?

Visual Social Computing

MIT Media Lab Workshops

Session before lunch (12:15 to 1pm):How to Invent?Health, Entertainment and User Interfaces

Session after lunch: (by Invitation Only)How to innovateBrainstorming


Camera Culture

Ramesh Raskar

Ramesh RaskarAssociate Professor, MIT Media Lab http://raskar.info

New Emerging TechnologiesMedical Imaging

EntertainmentUser InterfacesIndustrial Vision

The impact of Next Billion CamerasMovie-making, news reporting

Social stability

Visual Social ComputingImage-based commerce

Future of Imaging

END Part 1

Start Part 2


Camera Culture

Ramesh Raskar



Ramesh RaskarAssociate Professor


After X, what is neXt

How to Invent?



Xd

X++

X X+Y

X

X

neXt



Simple Exercise .. Simple Exercise .. • Image CompressionImage Compression

– Save Bandwidth and storageSave Bandwidth and storage

What is neXt


Strategy #1: XStrategy #1: Xdd

• Extend it to next dimension (or some other) dimensionExtend it to next dimension (or some other) dimension

– Flickr to YoutubeFlickr to Youtube

– Wikipedia to .. ?Wikipedia to .. ?

• Text, Audio (Speech), Image, Video .. Whats next ?Text, Audio (Speech), Image, Video .. Whats next ?– CD ..CD ..

• Images to infrared, sound, ultrasoundImages to infrared, sound, ultrasound• Macro scale to microscaleMacro scale to microscale

• Airbag for car to airbag for .. ?Airbag for car to airbag for .. ?


Strategy #2: X+YStrategy #2: X+Y• Fusion of the dissimilarFusion of the dissimilar

– More dissimilar, more spectacular the outputMore dissimilar, more spectacular the output

• ExampleExample– Scientific imaging + PhotographyScientific imaging + Photography

• Coded apertureCoded aperture• TomographyTomography


Prototype cameraPrototype camera

4000 4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens× 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens

Contax medium format camera Kodak 16-megapixel sensor

Adaptive Optics microlens array 125μ square-sided microlenses


Example of digital refocusingExample of digital refocusing


Imaging in Sciences: Imaging in Sciences: Computer TomographyComputer Tomography

• http://info.med.yale.edu/intmed/cardio/imaging/techniques/http://info.med.yale.edu/intmed/cardio/imaging/techniques/ct_imaging/ct_imaging/

Self Evaluation of Eye


Strategy #3: X Strategy #3: X Do exactly the oppositeDo exactly the opposite

• Processing, Memory, BandwidthProcessing, Memory, Bandwidth– In Computing world, in any era, one of this is a bottleneckIn Computing world, in any era, one of this is a bottleneck– But overtime, they change. You can often take an older idea and do But overtime, they change. You can often take an older idea and do

exactly the opposite.exactly the opposite.– E.g. bandwidth is now considered virtually limitlessE.g. bandwidth is now considered virtually limitless

• Business Process Reengineering (BPR) Business Process Reengineering (BPR) – Michael Hammer, James Champy, 1990sMichael Hammer, James Champy, 1990s

• In imaging:In imaging:– SLR: Faster mirror flip or no mirror flipSLR: Faster mirror flip or no mirror flip

• Companies spent years improving mirror flip speedCompanies spent years improving mirror flip speed• Why not just remove it?Why not just remove it?

• More computationMore computation• Less lightLess light


• Toll Free callsToll Free calls• Reverse AuctionReverse Auction

Power ofthe

Processor

Power of

theNetwork

Powerof

People

1985 1995 2005 2015

Powershifts

Larval Trematode WormLarval Trematode Worm


Strategy #4: X Strategy #4: X • Given a Hammer ..Given a Hammer ..

– Find all the nailsFind all the nails– Sometimes even screws and boltsSometimes even screws and bolts

• Given a cool solution/technique/Opportunity Given a cool solution/technique/Opportunity – Find other problemsFind other problems– (Where to find them?)(Where to find them?)

• ExamplesExamples– Peltier effect: Peltier effect:

• Create a jacket that keeps you warm or coldCreate a jacket that keeps you warm or cold– Mobile phone opportunityMobile phone opportunity


Strategy #5: X Strategy #5: X • Given a nail, Given a nail,

– Find all hammersFind all hammers– Sometimes even screwdrivers and pliers may Sometimes even screwdrivers and pliers may

workwork• Given a problem, Given a problem,

– Find other solutionsFind other solutions– (Where to find them?)(Where to find them?)

• ExamplesExamples– App store (Apple) .. Open platform for all devicesApp store (Apple) .. Open platform for all devices– ....


Strategy #6: X++Strategy #6: X++• Pick your adjective ..Pick your adjective ..• Making it faster, better, cheaperMaking it faster, better, cheaper

neXt = adjective + XneXt = adjective + X


X++ : Add your favorite adjectiveX++ : Add your favorite adjective• Context aware, Context aware, • AdaptiveAdaptive• (temporally) Coherent, (temporally) Coherent, • Hierarchical, Hierarchical, • ProgressiveProgressive• EfficientEfficient• ParallelizedParallelized• DistributedDistributed

• Good example: Image or video compression schemesGood example: Image or video compression schemes


X++ : Add your favorite adjectiveX++ : Add your favorite adjective• Good example: Image or video compression schemesGood example: Image or video compression schemes• But X++ is a signBut X++ is a sign

– The field is maturing in terms of research but booming in business The field is maturing in terms of research but booming in business impactimpact

• KaizenKaizen– Small incremental changesSmall incremental changes– Japanese Management styles (6sigma, Kanban)Japanese Management styles (6sigma, Kanban)– Mainly to save money/time/resources. Not everyone can do it. GM, Mainly to save money/time/resources. Not everyone can do it. GM,

0.84 suggestions per employee vs Toyota 18. GM accepted 23%, 0.84 suggestions per employee vs Toyota 18. GM accepted 23%, Toyota 90%Toyota 90%


Hexagon Corners for Different StagesHexagon Corners for Different Stages

[Bruce Tuckman, 1965][Bruce Tuckman, 1965]

• FormingForming• StormingStorming• NormingNorming• PerformingPerforming• AdjourningAdjourning


PitfallsPitfalls• These six ways are only a start These six ways are only a start • They are a good mental exercise and will They are a good mental exercise and will

allow you to train as a researcherallow you to train as a researcher• Great for projectsGreat for projects• But But

– Maynot produce radically new ideasMaynot produce radically new ideas– Sometimes a danger of being labeled incrementalSometimes a danger of being labeled incremental– Could be into ‘public domain ideas’Could be into ‘public domain ideas’


What are Bad ideas to pursueWhat are Bad ideas to pursue• X then Y (then Z)X then Y (then Z)

– X+Y is great with true fusion, fusion of dissimilar is bestX+Y is great with true fusion, fusion of dissimilar is best– But avoid a ‘pipeline’ systems, where the output of one is But avoid a ‘pipeline’ systems, where the output of one is

THEN channeled into the input of the next stage, and non THEN channeled into the input of the next stage, and non of the components are novelof the components are novel

• Follow the hype (too much competition) Follow the hype (too much competition) • Do because it can be done Do because it can be done

– (Why do we climb a mountain? because it is there! )(Why do we climb a mountain? because it is there! )– But only the first one gets a credit. But only the first one gets a credit. – May make you strong, and give you a sense of May make you strong, and give you a sense of

achievement but not a research project.achievement but not a research project.


Xd

X++

X X+Y

X

X

neXt


End Part 2


Camera Culture

Ramesh Raskar




Extra Slides

raskar emtech2010 mar final

Technology