computer vision as universal controller for interactive media

8/8/2019 Computer Vision as universal controller for interactive media

1/50

Computer Vision as universal controller for interactive media

Denis Perevalov

[email protected]: www.youtube.com/perevalovds#g/u

lections: www.uralvision.blogspot.comInstitute of Mathematics and Mechanics, Ural Branch of Russian Academy of Sciences

http://www.instablogsimages.com/images/2008/01/04/aperture-interactive-display_48.jpg
http://www.uralvision.blogspot.com/http://www.youtube.com/perevalovds#g/umailto:[email protected]


2/50

Contents

1. What is controller2. Computer vision as universal controller3. Video cameras for computer vision4. Controllers based on computer vision I

5. Controllers based on computer vision II6. Programming technologies7. The prospects


3/50

1. What is controller


4/50

Definition of controller

Controller is any sensor, any source of data, which can be converted into digitalsignal.

Proximity sensor


5/50

Standard musical controllers

Midi padMidi keyboard

Midi track control

Standard instrumentswith MIDI outputBreath controller

Actually, these are musical instruments or tools for intruments, which output digidal dataabout the sound instead of sound.


6/50

Controllers of human motion

Biosensors

Motion Capture

http://neurocenter.unige.ch/groups/pun.php

Multitouch


7/50

Controllers of physical phenomenas

Waves on the waterAleatoric water musical instrumenthttp://www.youtube.com/watch?v=CZ_KijiwQHE


8/50

Stochastical controllers

Internet, data frommarket

Insects trajectories

"Debug - art by insects"http://vimeo.com/12645870


9/50

2. Computer vision asuniversal controller


10/50

What is computer vision

Computer vision is a branch of Computer Sciences, which studyautomatic image analysis, with images received from digital video andphoto cameras.


11/50

Examples of tasks, which can be solved using computer vision:

SegmentationAnalysis of motion(optical flow)



12/50

Examples of tasks, which can be solved using computer vision:

Object tracking andmeasurement

Motion Capture

http://armi.kaist.ac.kr/korean/UserFiles/File/MMPC.JPG



13/50

Computer vision as universal controller

Any parameters of the physical processes, which is realized in mechanicalmotion, shape or color or transparency change- can be digitized using computer vision.


14/50

Difference of computer vision from other sensor types:1) possibility to use huge amount of data (pixel array).

2) possibility to structurize this data, by extracting from the

image the needed parameters (for example, position and sizeof the objects).

Computer vision as universal controller


15/50

Limits of the controllers based oncomputer vision

1. Because of huge amount of data, such controllers normallywork with big latency.

2. Because of cameras see objects at a distance, a precisionof measuremens can be not enough.

3. The light is needed for cameras. (Visible or infrared, butneeded anyway).

(We'll see it in more details below)


16/50

3. Video cameras forcomputer vision


17/50

Camera's basic characteristics

For tasks of image analysis in realtimedifferent kinds of cameras can be needed.

The basic characteristics are:

1. Resolution2. Number of frames per second

3. Type of output data (visible, IR, 3d)


18/50

Resolution

This is dimensions of the camera image in pixels.

https://reader009.{domain}/reader009/html5/0

320 x 240precision for object size 1

meter:3.13

size of 30 frames:6.6 Mb

640 x 480

1.56

26.4

1280 x 1024

0.97

112.5


19/50

Number of frame per second

30 FPS

time between two frames33 msec

60 FPS

16 msec

150 FPS

6 msecCan be used for musical

instrument

http://www.youtube.com/watch?v=7iEvQIvbn8o


20/50

Type of output data

Color or graytone imageof visible light

Infrared image

Using invisible IR-light, suchcamera wil see during dark

performance conditions

Color image + depthinformation


21/50

Examples

Sony PS3 Eye

320 x 240 : 150 FPS640 x 480 : 60 FPS

Data type:visible light,IR (you need to remove IR-filter)

Price: 50$.

USB, CCD


22/50

Examples

Point Grey Flea3

648 x 488 : 120 FPS

Data type:- visible light,

- IR (?)

Price: 600$.

Model FL3-FW-03S1C-C

IEEE 1394b, CCD


23/50

Examples

Microsoft Kinect

640 x 480 : 30 FPS

Data type:color image + depth

Price: 150$.(depth is received using special IR pattern,so will not work in direct sunlight)USB, CMOS


24/50

Examples

Point Grey BumbleBee2

640 x 480 : 48 FPS

Data type:color image + depth

Price: 2000$.

(Depth is receiving using stereovision from two cameras)IEEE 1394b, CCD


25/50

4. Controllers based oncomputer vision I


26/50

Sliders and buttons - by camera

- Camera can read slider position without electronics.- Your finger can be slider or button, camera will read itsposition.


27/50

Additive synthesizer using optical measurement of

brightness


28/50

Using two color markers for creating virtual picture


29/50

Giroscope using camera: AR

Augmented Reality with Markers6-dimensions: X, Y, Z, 3 axes on rotation

http://www.edhv.nl/edhv/wp-content/uploads/2009/12/aug_Picture-10_no-border-450x337.jpg


30/50

Multitouch

FTIR multitouch

http://www.touchuserinterface.com/2010/02/lcd-multi-touch-using-inverted-ftir.htmlhttp://sites.google.com/site/ideolabsdocumentation/images/multitouchdiagram.png


31/50

Virtual reality gloves

Color gloves using one webcamera only, in MIT (prototype)

http://www.csail.mit.edu/videoarchive/research/gv/hand-tracking


32/50

Motion Capture

Microsoft Kinect for XBox(Motion capture is available only for XBox developers yet. But using OpenKinect library inMac/Windows you can get color and depth images - it is really good for many types of

projects. For official opinion of Microsoft on OpenKinect see http://www.thinq.co.uk/2010/11/22/microsoft-declares-openkinect-safe/)
http://www.thinq.co.uk/2010/11/22/microsoft-declares-openkinect-safe/http://www.thinq.co.uk/2010/11/22/microsoft-declares-openkinect-safe/


33/50

Conclusion

Not-camera sensors are more precise and rapid.

Computer vision merits:- fast implementstion for prototype

building- universal (one camera can be used for differents kinds ofsensors)


34/50

5. Controllers based oncoputer vision II


35/50

Motion areas detection

Result is coordinates of the areas with motion.


36/50

Optical flow calculation

Result is 2D-vector field of motion vectors.


37/50

Objects of interest detection

Result is coordinates and sizes of detected objects.


38/50

Non-visual structures on the image

- Sum of brightness of some special pixels on the image.

- Vector of special components of image Fast Fourier Transform.Such characteristics are "nonvisual", because have no direct relationwith areas on the image or some objects or its characteristics. Butthese are non-random and sometimes are useful.


39/50

Conclusion

Implementetion of such controllers using not-camera sensors isexpensive, and sometimes specific device constructing isneeded.


40/50

6. Programming technologies


41/50


42/50

Low-level libraries

"Open Graphics Library"

Open library for radip graphics, C/++.


43/50

Low-level libraries

"Open Computing Language"Open library for parallel computing, with GPU too, C/++.

Allows to speedup computings very much. OpenCV + OpenCL is preparinindevelopment stage now (by Intel).


44/50

Middle-level platforms

"Creative coding" platforms with large amount of functions forconvenient programming.

openFrameworksC/C++

ProcessingJavais too slow for computervision

CinderC/C++is young, but popularity isincreasing

f


45/50

High-level platforms

Visual programming platforms for musucians and designers.Almost no text programming needed. If needed, it can beextended by custom plugins, developed by programmers inlower-level platforms.

Max/MSP/JitterAudio-oriented

VVVVVideo effects -oriented

Quest3DHigh quality 3D -oriented


46/50

The prospects

T h l i l t


47/50

Technological prospects

- Manufacturing more models of cameras with FPS > 100 forusing it in live performance.

- Increasing resolution and processing speed for increasing

measurement precise in space and time scales.

A ith i l t


48/50

Agorithmical prospects

- Stable emotions and mimics recognition.

- 3D scenes with many objects recognition (using the cameraswith depth).

http://susiemander.files.wordpress.com/2010/10/facial-expression.jpg

Id t


49/50

Ideas prospects

1. Mutltitouch is not used as it should to be.2. How to use moton capture technology for effectiveperformance?

Id t


50/50

Ideas prospects

- Searching new and unusual processes, which can be visible

by camera.

- Searching new interesting structures in images and ordinalphenomenas.

computer vision as universal controller for interactive media

Documents