image, shape & multimedia resource discovery...image, shape & multimedia resource discovery...
TRANSCRIPT
![Page 1: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/1.jpg)
Image, Shape & MultimediaResource Discovery
Stefan RügerMultimedia & Information Systemsmmis.doc.ic.ac.uk
Frederic Fol LeymarieArts Computingdoc.gold.ac.uk/~ffl
![Page 2: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/2.jpg)
2
Outline
Part 1: New paradigms for media access
Part 2: Shape representation
![Page 3: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/3.jpg)
3
Background: Stefan Rüger• Trained as theoretical physicist• PhD in theory of neural networks• Imperial College London
– PostDoc in data mining and text mining (19971999)– EPSRC Advanced Research Fellow (19992004)– Research Lecturer (20002002), Lecturer (20022005),
Reader in Multimedia and Information Systems (20052006)• Professor of Knowledge Media, KMI, the Open University
Background: Frederic F. Leymarie• Trained as an engineer (electrical and physics)• PhD in 3D shape theory• Brown University (USA)
– Cofounder of the SHAPE Lab. (1999); NSF supported– SHAPE Lab manager (20022004)
• Professor of Arts Computing, Goldsmiths Digital Studios
![Page 4: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/4.jpg)
4
Outline
• New Paradigms in Media Access– Multimedia Management and Access– Semantic gap and Polysemy– Contentbased visual search– Lateral browsing– Commercial applications
• Shape representation– 2D– 2.5D, images, motion– 3D
![Page 5: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/5.jpg)
5
Research: Multimedia Management and Access
What have you got?
![Page 6: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/6.jpg)
6
Multimedia Management and Access
• Archive– text, video, images, speech, music, web content, …
• Query– text, stills, sketch, speech, humming, examples, …– contentbased
• Valueadded services– browsing, summaries, story boards– document clustering, cluster summaries
• Putting the user in the loop
![Page 7: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/7.jpg)
7
New types of Search Engines
text
videoimagesspeechmusicsketchesmultimedia
text
still
ssk
etch
spee
chso
und
hum
min
gm
otio
n
query doc
conventional text retrieval
hum a tune and get a music piece
you roar and get a wildlife documentarytype “floods” and get BBC radio news
Example
![Page 8: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/8.jpg)
8
Applications
• Medicine• Personal collections • Multimedia digital libraries• Media archives• Entertainment• Tourism• Elearning• Retail
![Page 9: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/9.jpg)
9
Outline
• New Paradigms in Media Access– Multimedia Management and Access– Semantic gap and Polysemy– Contentbased visual search– Lateral browsing– Commercial applications
• Shape representation– 2D– 2.5D, images, motion– 3D
![Page 10: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/10.jpg)
10
The semantic gap
1. 120,000 pixels witha particular spatial colour distribution
2. faces and a vaselike object
3. victory, triumph, ...
![Page 11: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/11.jpg)
11
Automated image annotation
water grass trees
the beautiful sun
le soleil beau
![Page 12: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/12.jpg)
12
• Training set, e.g., 32,000 images such as
animal, creature, face, grass, nature, outdoors, outside, portrait, teeth, tiger, vertical, water, wild, wildlife
Automated image annotationas machine learning problem
[with Magalhães, CIVR 2006][PhD work in progress: A Yavlinsky, Nov 2003, partly ORS
funded][PhD work in progress: J Magalhães, Oct 2004, funded by
FCT][with Magalhães, book chapter 2006][with Magalhães, SIGIR 2005][with Yavlinsky and Schofield, CIVR 2005][with Yavlinsky, Heesch and Pickering, ICASSP 2004]
![Page 13: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/13.jpg)
13
Automated image annotation: an example
Automated: water buildings city sunset aerial
![Page 14: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/14.jpg)
14
![Page 15: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/15.jpg)
15
Bridging the semantic gap
• Region classification (grass, water, sky, ...) + simple vocabulary
• Infer complex concepts from simple models (barbecue = grass+plates+people), e.g., using ontologies
[collaboration with AT&T Research, Cambridge 2001]
![Page 16: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/16.jpg)
16
Polysemy
![Page 17: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/17.jpg)
17
Relevance feedback
• endow system with plasticity (parameters)• system needs to learn from user
= change the parameters
[with Heesch and Yavlinsky, CIVR 2003][with Heesch, ECIR 2003]
![Page 18: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/18.jpg)
18
Search for images of blue doors by example image
![Page 19: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/19.jpg)
19Move relevant results closer to centre = more important
![Page 20: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/20.jpg)
20New query = query image + new position of initial results (user)
![Page 21: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/21.jpg)
21Next round: many more relevant results (blue doors)
![Page 22: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/22.jpg)
22
Outline
• New Paradigms in Media Access– Multimedia Management and Access– Semantic gap and Polysemy– Contentbased visual search– Lateral browsing– Commercial applications
• Shape representation– 2D– 2.5D, images, motion– 3D
![Page 23: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/23.jpg)
23
Contentbased Visual Search [book chapter: 2006 to appear][Choplin 1.3 opensource software, 3 Q 2006][iBase 1.2 opensource software, 3 Q 2006][keynote: EU infoday for a call on audiovisual search, 19 Dec 2005][invited talks: AMR 2005, National eScience workshop 2005][invited panels: EWIMT 2004, AMR 2005, EWIMT 2005][with Howarth, Information Retrieval, submitted 2005][with Howarth CIVR 2005][with Chawda, Craft, Cairns and Heesch, HCI 2005][with Howarth, ECIR 2005][with Heesch, Pickering, Howarth and Yavlinsky, JCDL 2004][with Yavlinsky, Heesch and Pickering, ICASSP 2004][with Howarth, CIVR 2004][with Pickering, Computer Vision and Image Understanding, 92(1), 2003][with Heesch, Pickering and Yavlinsky, TRECVID 2003][with Heesch and Yavlinsky, CIVR 2003][with Pickering, Heesch, O’Callaghan and Bull, TREC 2002][with Pickering and Sinclair, CIVR 2002][with Pickering, TREC 2001]
![Page 24: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/24.jpg)
24
Evaluation conference TRECVID
• TRECVID topic 102: Find shots from behind the pitcher in a baseball game as he throws a ball that the batter swings at
![Page 25: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/25.jpg)
25
Outline
• New Paradigms in Media Access– Multimedia Management and Access– Semantic gap and Polysemy– Contentbased visual search– Lateral browsing– Commercial applications
• Shape representation– 2D– 2.5D, images, motion– 3D
![Page 26: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/26.jpg)
26
Putting the user in the look
• Precompute browsable network within media connecting ``similar‘‘ stuff
[with Heesch, CIVR 2005][with Heesch, CIVR 2004][with Heesch, ECIR 2004]
![Page 27: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/27.jpg)
27
Nearest neighbours wrt features
Colour
Text
Structurequery
![Page 28: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/28.jpg)
28
Browsing example
• TRECVID topic 102: Find shots from behind the pitcher in a baseball game as he throws a ball that the batter swings at
In your mind
![Page 29: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/29.jpg)
29
[Alexander May: SET award 2004: best use of IT – national prize]
![Page 30: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/30.jpg)
30
![Page 31: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/31.jpg)
31
![Page 32: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/32.jpg)
32
![Page 33: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/33.jpg)
33
Properties
• Exposure of semantic richness• User decides which image meaning is correct• Network precomputed > fast and interactive• Supports search without query formulation• 3 degrees of separation (for 32,000 images)• scalefree, smallworld graph
![Page 34: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/34.jpg)
34
Outline
• New Paradigms in Media Access– Multimedia Management and Access– Semantic gap and Polysemy– Contentbased visual search– Lateral browsing– Commercial applications
• Shape representation– 2D– 2.5D– 3D
![Page 35: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/35.jpg)
35
Video summarystorylevel segmentation
keyframe summaryvideotext summarynamed entities
fulltext search
[technology being licensed by Imperial Innovations to industry][patent application 2004][finished PhD project: Pickering, now at HTB, London][with Wong and Pickering, CIVR 2003][with Lal, DUC 2002][Pickering: best UK CS student project 2000 – national prize]
![Page 36: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/36.jpg)
36
Multimedia Digital Libraries: Edutainment
![Page 37: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/37.jpg)
37
Digital Shoebox
• Personal photo collection
![Page 38: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/38.jpg)
38
Multimedia Digital Libraries
• Visual, geotemporal and text access to multimedia collections
• Novel visual browsing and search modes• Information visualisation• Collections from BBC, BL, V&A and 2 Unis
![Page 39: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/39.jpg)
39
Outline
• New Paradigms in Media Access– Multimedia Management and Access– Semantic gap and Polysemy– Contentbased visual search– Lateral browsing– Commercial applications
• Shape representation– 2D– 2.5D, images, motion– 3D
![Page 40: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/40.jpg)
From “Image search engines: an overview”, Gevers & Smeulders, 2003
ContentBased Image Retrieval
![Page 41: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/41.jpg)
Color
Shading
Shape
Texture
Motion
![Page 42: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/42.jpg)
From “On the Role of Medial Geometry in Human Vision”, Kimia, 2003.
Shape as a Representation / Unification
![Page 43: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/43.jpg)
Outline
• New Paradigms in Media Access– Multimedia Management and Access– Semantic gap and Polysemy– Contentbased visual search– Lateral browsing– Commercial applications
• Shape representation– 2D– 2.5D, images, motion– 3D
![Page 44: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/44.jpg)
Medial representations of shape
1 Consider a set of inflatable primitives centered midway between 2 or more outline segments and grown to osculate (touch) the object on these segments.
Special case: maximal contact spheres.
Medial loci: let the outline “propagate” (i.e., continuously deform) until it selfintersects.
Special case: Euclidean wave propagation (optics).
![Page 45: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/45.jpg)
Medial representations of shape
The Medial Axis Transform: 1st singular solutions of the Eikonal equation of geometric optics [Blum].
From Gert van Tonder's website:
3D cellular automata simulationby Leymarie and Kimia:
![Page 46: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/46.jpg)
Medial Axis representation of shape
Qualitative descriptions: e.g., branchingstructures of elongated objects.
Quantitative descriptions: encode the varying widthof objects via the radius function (of contact spheres).
Shape features: make explicit corners,bumps, surface ridges (sharp or smooth).
Builtin hierarchy of scales: mix of spatial and width properties.
Completeness: exact reconstruction possible.
Shape dynamics: modeling of deformations, kinematics (including animation, motion),genesis (growth).
![Page 47: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/47.jpg)
2D graph “regularization”• Transition removal, i.e., remove topological instability• 2D Boundary Smoothing2D Boundary Smoothing ordered by “scale”
Iterative removal of MA branches, ordered by boundary support (i.e., how much of the contour is represented), coupled with local boundary model adjustment, results in corner enhancement and small perturbations' smoothing.
(After Tek and Kimia, 2001)
![Page 48: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/48.jpg)
3D scaffold regularization
![Page 49: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/49.jpg)
Examples: parts, deformation
![Page 50: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/50.jpg)
Examples: parts, deformation
![Page 51: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/51.jpg)
Challenge: How to navigate the “shapespace”
![Page 52: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/52.jpg)
Challenge: How to navigate the “shapespace”
Build metrics on (graph) deformations
![Page 53: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/53.jpg)
Challenge: How to navigate the “shapespace”
Build metrics on (graph) deformations
![Page 54: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/54.jpg)
DB indexing by (2D) shape
After Sebastian, Kimia, et al., IEEEPAMI, 2004.
![Page 55: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/55.jpg)
Outline
• New Paradigms in Media Access– Multimedia Management and Access– Semantic gap and Polysemy– Contentbased visual search– Lateral browsing– Commercial applications
• Shape representation– 2D– 2.5D, images, motion– 3D
![Page 56: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/56.jpg)
DB indexing by (“2.5D”) shape
After Kimia, et al., 2000.
![Page 57: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/57.jpg)
Examples: multiple views, animation
![Page 58: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/58.jpg)
Image grouping / segmentation
After Kimia et al., 2004.
![Page 59: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/59.jpg)
Image grouping / segmentation
After Kimia et al., 2004.
![Page 60: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/60.jpg)
Shape as a Representation / Unification
From “On the Role of Medial Geometry in Human Vision”, Kimia, 2003
![Page 61: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/61.jpg)
How to represent complex moving forms
From “Capturing time”, Ilona Kovacks, 2006
Animation by Charles Lucassen, after Muybridge (1887)
![Page 62: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/62.jpg)
How to represent complex moving forms
From “Capturing time”, Ilona Kovacks, 2006
Chronophotographs, Marey, 1884, College de France
![Page 63: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/63.jpg)
How to represent complex moving forms
From “Capturing time”, Ilona Kovacks, 2006
Marey, 1883, College de France
![Page 64: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/64.jpg)
How to represent complex moving forms
From “Capturing time”, Ilona Kovacks, 2006
•human contrast sensitivity••neuronal responses in monkey occipital cortex
![Page 65: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/65.jpg)
Outline
• New Paradigms in Media Access– Multimedia Management and Access– Semantic gap and Polysemy– Contentbased visual search– Lateral browsing– Commercial applications
• Shape representation– 2D– 2.5D, images, motion– 3D
![Page 66: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/66.jpg)
3D
www.doc.gold.ac.uk/morpholingua/
LeymarieKimia:Keep only singularsingular
points of the flowpoints of the flow(radius) to build a
graph graph (2001).
![Page 67: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/67.jpg)
3D
Mangin et al., 2004; Cortical folds' recognition.
![Page 68: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/68.jpg)
3D : Towards Object Recognition
![Page 69: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/69.jpg)
3D : Towards Object Recognition
![Page 70: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/70.jpg)
Visual Search in large 3D DB
Challenge: robust automatic extraction of graphs,dealing with topological instabilities/events.
Graph matching via graduated assignementGraph matching via graduated assignement(presented at 3DPVT, Greece, Sept. 2004, Chang, Leymarie & Kimia)(presented at 3DPVT, Greece, Sept. 2004, Chang, Leymarie & Kimia)
a solution to thea solution to the Global Registration Global Registration problem.problem.
Digital MichelangeloStanford, Firenze, NRC
![Page 71: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/71.jpg)
Outline
• New Paradigms in Media Access– Multimedia Management and Access– Semantic gap and Polysemy– Contentbased visual search– Lateral browsing– Commercial applications
• Shape representation– 2D– 2.5D, images, motion– 3D
What next ...?
![Page 72: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/72.jpg)
![Page 73: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/73.jpg)
Visual Search in large 2D DBCan the shape representation lead to functional descriptors and help in automatically creating metadata?
Challenge: robust automatic extraction of graphs, dealing with topological instabilities/events.
![Page 74: Image, Shape & Multimedia Resource Discovery...Image, Shape & Multimedia Resource Discovery Stefan Rüger Multimedia & Information Systems mmis.doc.ic.ac.uk Frederic Fol Leymarie Arts](https://reader030.vdocument.in/reader030/viewer/2022040910/5e8326f759f9ee2322708241/html5/thumbnails/74.jpg)
Image, Shape & MultimediaResource Discovery
Stefan RügerMultimedia & Information Systemsmmis.doc.ic.ac.uk
Frederic Fol LeymarieArts Computingdoc.gold.ac.uk/~ffl