![Page 1: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/1.jpg)
Robot Vision SS 2005 Matthias Rüther 1
710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU
Matthias Rüther
Kawada Industries Inc. DLR
![Page 2: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/2.jpg)
Robot Vision SS 2005 Matthias Rüther 2
Organization
VO: Tuesday 14:15-15:45 Seminarraum ICG
Exam: Written Exam Oral Exam if Requested
KU:implementation of lecture topics in the real world (on the lab-robots)
Groups of three students Possible problems on the last slide Scheduling of topics: 8.3.2005 If you are interested: excursions to industrial vision
companies (Alicona Imaging, M&R)
![Page 3: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/3.jpg)
Robot Vision SS 2005 Matthias Rüther 3
Time Table
1.3. : Introduction and Overview
8.3. : Projective Geometry (1)15.3. : Projective Geometry (2)12.4. : Projective Geometry (3)19.4. : Projective Geometry (4)
26.4. : Camera Technologies
3.5. : Shape From X (1)10.5. : Shape From X (2)24.5. : Shape From X (3)
31.5. : Robot Kinematics (1)7.6. : Robot Kinematics (2)
14.6. : Tracking of Moving Objects
21.6. : Visual Servoing / Hand Eye Coordination
![Page 4: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/4.jpg)
Robot Vision SS 2005 Matthias Rüther 4
Literature
• Sciavicco, L., Siciliano, B., Modelling and Control of Robot Manipulators 2nd Ed., Springer, 2000
• Ballard D.H., Brown C.M., "Computer Vision", Prentice-Hall, 1982
• Sonka M., Hlavac V., Boyle Image Processing, Analysis and Machine Vision, Chapman Hall, 1998
• Nalva V.S., "A Guided Tour of Computer Vision", Addison-Wesley Publishing Company, 1993
• Horn B.K.P., "Robot Vision", MIT Press, Cambridge, 1986
• Shirai Y., "Three- Dimensional Computer Vision", Springer Verlag, 1987
• Faugeras O., Three-Dimensional Computer Vision A Geometric Viewpoint, MIT Press, 1993
• Hartley R., Zissermann A., Multiple View Geometry in Computer Vision, Cambridge, 2001.
![Page 5: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/5.jpg)
Robot Vision SS 2005 Matthias Rüther 5
Robotics
What is a robot?"A reprogrammable, multifunctional manipulator designed to move
material, parts, tools, or specialized devices through various programmed motions for the performance of a variety of tasks"
Robot Institute of America, 1979
… in a three-dimensional environment.
Industrial– Mostly automatic manipulation of rigid parts with well-known shape in a
specially prepared environment.
Medical– Mostly semi-automatic manipulation of deformable objects in a
naturally created, space limited environment.
Field Robotics– Autonomous control and navigation of a mobile vehicle in an arbitrary
environment.
![Page 6: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/6.jpg)
Robot Vision SS 2005 Matthias Rüther 6
Experimental/Industrial/Commercial Robots
![Page 7: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/7.jpg)
Robot Vision SS 2005 Matthias Rüther 7
Industrial Robots
![Page 8: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/8.jpg)
Robot Vision SS 2005 Matthias Rüther 8
Challenging Environments
![Page 9: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/9.jpg)
Robot Vision SS 2005 Matthias Rüther 9
Service and Assistance
![Page 10: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/10.jpg)
Robot Vision SS 2005 Matthias Rüther 10
FRIEND Project
![Page 11: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/11.jpg)
Robot Vision SS 2005 Matthias Rüther 11
Robot vs Human
Robot Advantages:
– Strength
– Accuracy
– Speed
– Does not tire
– Does repetitive tasks
– Can Measure
Human advantages:
– Intelligence
– Flexibility
– Adaptability
– Skill
– Can Learn
– Can Estimate
![Page 12: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/12.jpg)
Robot Vision SS 2005 Matthias Rüther 12
Robotics: Goals and Applications
Robotics does not intend to develop the artificial human![Whitney, D. E., Lozinski, C. A. and Rourke, J. M. (1986) Industrial robot forward calibration method and results.]
Goal: combine robot and human abilities.
Applications: – Automation (Production)
– Inspection (Quality control)
– Remote Sensing (Mapping)
– Man-Machine interaction („Cobot“)
– Robot Companion (Physically challenged people)
– See [Brady, M. et. al. (eds). „Robot Motion: Planning and Control“]
![Page 13: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/13.jpg)
Robot Vision SS 2005 Matthias Rüther 13
What can Computer Vision do for Robotics?
Accurate Robot-Object Positioning
Keeping Relative Position under Movement
Visualization / Teaching / Telerobotics
Performing measurements
Object Recognition (see LV „Bildverarbeitung u. Mustererkennung“, „Bildverstehen“, „AK Computer Vision“)
Registration
Visual Servoing
![Page 14: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/14.jpg)
Robot Vision SS 2005 Matthias Rüther 14
Combining Computer Vision and Robotics
Abstraction level
Motor Modeling: what voltage should I set now ?
Control (PID): what voltage should I set over time ?
Kinematics: if I move this motor somehow, what happens in other coordinate systems ?
Motion Planning: Given a known world and a cooperative mechanism, how do I get there from here ?
Bug Algorithms: Given an unknowable world but a known goal and local
sensing, how can I get there from here?
Mapping: Given sensors, how do I create a useful map?
Localization: Given sensors and a map, where am I ?
low
high
Vision: If my sensors are eyes, what do I do?
![Page 15: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/15.jpg)
Robot Vision SS 2005 Matthias Rüther 15
Computer Vision
What is Computer Vision?
"Computer Vision describes the automatic deduction of the structure and the properties of a (possible dynamic) three-dimensional world from either a single or multiple two-dimensional images of the world" [Nalva VS, "A Guided Tour of Computer Vision"]
Measurement– Measure shape and material properties in a 3D environment. Accuracy
is important.
Recognition– Cognitive systems interpret a 3D environment (object classification,
categorization). Systems are allowed to fail to a certain extent (similar to humans).
Navigation– Navigation Systems orient themselves in a 3D environment.
Robustness and time are important.
![Page 16: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/16.jpg)
Robot Vision SS 2005 Matthias Rüther 16
Measurement
„Shape from X“ techniques measure shape properties of objects from 2D digital images.
– Shape from Stereo: two cameras obeserve an object from different viewpoints (similar to human eye).
– Shape from focus: limited depth of focus allows to measure object-camera-distance.
– Shape from structured light: a light pattern is projected on the object, the pattern deformation gives shape information.
– Shape from Shading: an object is illuminated from a single direction. Light reflection depends on object shape and follows a reflectance function.
![Page 17: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/17.jpg)
Robot Vision SS 2005 Matthias Rüther 17
Shape from Stereo
![Page 18: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/18.jpg)
Robot Vision SS 2005 Matthias Rüther 18
Shape from Stereo
![Page 19: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/19.jpg)
Robot Vision SS 2005 Matthias Rüther 19
Shape from Focus
![Page 20: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/20.jpg)
Robot Vision SS 2005 Matthias Rüther 20
Shape from Structured Light
Structured Light Sensor
Figures from PRIP, TU Vienna
![Page 21: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/21.jpg)
Robot Vision SS 2005 Matthias Rüther 21
Shape from Shading
![Page 22: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/22.jpg)
Robot Vision SS 2005 Matthias Rüther 22
Navigation
SLAM: Simultaneous Localization and Mapping. – Where am I on my map?
– If the place is unknown, build a new map, try to merge it with the original map.
Visual Odometry: calculate the relative motion of the camera between two frames. Summing up the motion gives the camera path. Error propagation!
Visual Servoing: move to / maintain a relative position between robot end effector and an object.
Tracking: continuously measure the position of an object within the sensor coordinate frame.
![Page 23: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/23.jpg)
Robot Vision SS 2005 Matthias Rüther 23
SLAM
Mapping:
![Page 24: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/24.jpg)
Robot Vision SS 2005 Matthias Rüther 24
SLAM
The final map:
![Page 25: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/25.jpg)
Robot Vision SS 2005 Matthias Rüther 25
SLAM
Navigation:
![Page 26: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/26.jpg)
Robot Vision SS 2005 Matthias Rüther 26
Visual Odometry
![Page 27: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/27.jpg)
Robot Vision SS 2005 Matthias Rüther 27
Visual Servoing
![Page 28: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/28.jpg)
Robot Vision SS 2005 Matthias Rüther 28
Tracking
![Page 29: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/29.jpg)
Robot Vision SS 2005 Matthias Rüther 29
Tracking
![Page 30: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/30.jpg)
Robot Vision SS 2005 Matthias Rüther 30
Registration
Registration of CAD models to scene features:
Figures from P.Wunsch: Registration of CAD-Models to Images by Iterative Inverse Perspective Matching
![Page 31: Robot Vision SS 2005 Matthias Rüther 1 710.088 ROBOT VISION („Messen aus Bildern“) 2VO 1KU Matthias Rüther Kawada Industries Inc.DLR](https://reader030.vdocument.in/reader030/viewer/2022032517/56649cb95503460f9497f8b9/html5/thumbnails/31.jpg)
Robot Vision SS 2005 Matthias Rüther 31
KU: Student Problems
Shape from Stereo 3 students
Shape from Focus 3 students
Shape from Structured Light:Laser 3 students
Shape from Structured Light:Pattern 3 students
Shape from Shading 3 students
Robot Kinematics 3 students
2D Grip Planning 2..3 students
2D Visual Servoing 3 students
2D Tracking 3 students
Registration / Model Fitting 3 students
Visual Odometry + Randomized RANSAC 3 students