hand-eye coordination and vision-based interaction jag/courses/handeye / cmput610 martin jagersand
TRANSCRIPT
![Page 1: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/1.jpg)
Hand-Eye Coordination and Vision-based Interaction http://www.cs.ualberta.ca/~jag/courses/HandEye/
cmput610
Martin Jagersand
![Page 2: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/2.jpg)
Today:
1. Fun (hopefully) intro to some of the topics in the course. Give flavor of course.
2. Suggestions for some exciting projects you can choose. You are also welcome to propose your own.
3. (Boring) administrative details: Course schedule, room, participation and examination, class list etc…
![Page 3: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/3.jpg)
Main focus of course
Most other courses are about static vision. What is in the image?
This course: Vision for regestering, understanding and generating motion. How to act based on visual information.
Im Processing Symbol Interp
Images
Agent doing physical motion
Real world
![Page 4: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/4.jpg)
Where/What or Action/Perception division also in biological vision
![Page 5: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/5.jpg)
Main topics we will cover
1. Motion in the real world and its image projection
2. Vision and other sensory based control
3. Biological motor control.
4. User interfaces for computers, robots and other machines.
![Page 6: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/6.jpg)
Motion estimation:Image subtraction
50
Candidate areas formotion
Detecting motion:
![Page 7: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/7.jpg)
Motion estimation 2
A vector flow is a more useful representation
![Page 8: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/8.jpg)
Motion estimation 3
Non-rigid motions are also important
![Page 9: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/9.jpg)
Motion estimation 4
Generalize to many freedooms (DOFs)
![Page 10: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/10.jpg)
One application: Tracking
Goal: Stabilizing motion. Find move params
![Page 11: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/11.jpg)
Vision and Sensory based motion control
![Page 12: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/12.jpg)
Uncalibrated Visual Servoing
Let y = visual observation; x = motor control Linear system model:
y= Linear p-controller
Estimate the Visual-Motor Jacobian
Jêk+1 = Jêk+ É xTÉ x
(É ymeasuredà JêkÉ x)É xT
f (x) ù f (xk) + J (xk)(x à xk)
xç = K J +kyç
![Page 13: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/13.jpg)
Shortcut in Image based visual servoing
![Page 14: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/14.jpg)
Biological sensory-motor control
Sensory and motor areas in cortex
![Page 15: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/15.jpg)
Example: Eye movements
Relatively well studied area Yet complex nonlinear kinematics Interesting adaption behaviour
Paths: 1 flash Multple targets Remembered location
![Page 16: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/16.jpg)
User Interface applications:Gesture and motion tracking
![Page 17: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/17.jpg)
User Interfaces 2:Robot Control
Vision based “Tele Assistance”
Describe task and objects by gestures and pointing
Visual language maps to physical actions
![Page 18: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/18.jpg)
Composite Task: Solving a Puzzle
![Page 19: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/19.jpg)
Proposed projects 1
Neurosci Georgopoulos, Kalaska, Shieber...
PercAct Goodale, Lomis,
Characterizing motion: Activity recognition: Nelson,Alimonios Fleet, Jepson... Jagersand Matlab implementaton
Animating motion. Gleicher, Jagersand, Thalman
![Page 20: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/20.jpg)
Proposed projects 2
Tracking: Hager, Black, XVision: practical experiments with. Filtering, Bar-Shalom:
Robotics: Controller HW and SW for the small robot arm (or the PUMA's)
Real time systems Real time linux
![Page 21: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/21.jpg)
Participation and Examination
Course readings Class discussion, active class participation Present some papers of your choice Do your own reading or practical project and a
writeup
![Page 22: Hand-Eye Coordination and Vision-based Interaction jag/courses/HandEye / cmput610 Martin Jagersand](https://reader036.vdocument.in/reader036/viewer/2022070413/5697bf9a1a28abf838c92066/html5/thumbnails/22.jpg)
Administrative
Register for cmput610 Proposed schedule: Tue, Thu 9-10am Room TBA Course literature and readings are given on-line when
possible. Announcements via email. Make sure you are on the
email list. Course web page (important):
http://www.cs.ualberta.ca/~jag/courses/HandEye/