mindraces, first review meeting, lund, 11/01/2006 fovea-based robot control for anticipation studies...
TRANSCRIPT
![Page 1: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/1.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006
Fovea-Based Robot Control for Anticipation Studies in Various
Scenarios
Alexander Förster, Daan Wierstra, Jürgen Schmidhuber
IDSIA - Lugano - Switzerland
![Page 2: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/2.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 2
Overview
• Real world scenarios
• Fovea
• Software Framework
• Robertino Robot
• Simulated 3D scenarios
• Integration and Future Work
![Page 3: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/3.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 3
Instances of scenario 1: find an object
Still Image
Robot Lab
Office Room
Movie (LUCS)
2D E
nviro
nmen
ts3D
Environm
ents
![Page 4: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/4.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 4
Fovea Centralis Simulation
Three regions with increasing resolutions
![Page 5: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/5.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 5
Fovea Centralis Simulation
Three regions with increasing resolutions
Subsample each region to 13x11 pixels
![Page 6: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/6.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 6
Fovea Centralis Simulation
Three regions with increasing resolutions
Subsample each region to 13x11 pixels
Combine the subsampled image
![Page 7: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/7.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 7
?? ? ? ? ? ? ? ? ?? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley
Bins Objects
1-D world
Transformation
Fovea
Memory
State 0
![Page 8: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/8.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 8
?? ? ? ? ? ? ? ? ?? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 1
Bins Objects
1-D world
Transformation
Fovea
Memory
![Page 9: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/9.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 9
?? ? ? ? ? ? ? ? ?? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 2
Bins Objects
1-D world
Transformation
Fovea
Memory
![Page 10: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/10.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 10
?? ? ? ? ? ? ? ? ?? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 3
Bins Objects
1-D world
Transformation
Fovea
Memory
![Page 11: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/11.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 11
? ? ? ? ? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 4
Bins Objects
1-D world
Transformation
Fovea
Memory
![Page 12: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/12.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 12
? ? ? ? ? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 5
Bins Objects
1-D world
Transformation
Fovea
Memory
![Page 13: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/13.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 13
? ? ? ? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 6
Bins Objects
1-D world
Transformation
Fovea
Memory
![Page 14: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/14.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 14
? ? ? ? ?
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 7
Bins Objects
1-D world
Fovea
Memory
Transformation
![Page 15: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/15.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 15
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 8
Bins Objects
1-D world
Fovea
Memory
Transformation
![Page 16: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/16.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 16
Simulated Vision: 1-D Fovea Simulation
Task: Find the sad smiley State 9
Bins Objects
1-D world
Fovea
Memory
Transformation
![Page 17: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/17.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 17
Simulated Vision: abstract 2-D Simulation
• Extended 1-D Simulation• Only the centered sensor can differentiate between
objects
![Page 18: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/18.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 18
Simulated Vision: 2-D Simulation
Original image with 2 objects
![Page 19: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/19.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 19
Simulated Vision: 2-D Simulation
Original image with 2 objects
Simulated fovea images
Center of the fovea
Step 1
![Page 20: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/20.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 20
Simulated Vision: 2-D Simulation
Original image with 2 objects
Simulated fovea images
Center of the fovea
Detected!!!
Step 1 Step n
![Page 21: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/21.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 21
Software Framework
Server
Robertino
•Debian/GNU Linux operating system•CAN bus interface •FireWire (IEEE 1394) interface•Video4Linux library•Fovea simulation
Client
Robosim
•Linux or Windows•Ogre framework•Fovea simulation•Simple physical and collision detection system
•Same interface as the real robot
Robomon
•Linux or Windows•Experiment management•Interface for learning algorithms•Full remote control of the robot/simulation
TCP/IP
Previously recorded data
![Page 22: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/22.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 24
Robertino – Client Software
Manual control of the robot
Learning control interface
Fovea image
![Page 23: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/23.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 25
Robertino - Overview
• Diameter: 40 cm• Height: 43 cm• Weight: 6.5 kg. • Holonomic three wheeled drive• PC-103 (industry standard)
with a 500MHz Intel Mobile-Pentium II processor on-board
• WLAN (IEEE 802.11a)• 2 cameras, used to simulate
the fovea• Actuators: the three wheels
and the simulated foveawww.openrobertino.org
![Page 24: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/24.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 26
Omnidirectional Camera
Web cam
Omnidirectional mirror
Image of an office environment, as seen by the robot
![Page 25: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/25.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 27
Image Transformation
• Used only for human convenience for navigation• Simple transformation algorithm
![Page 26: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/26.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 28
High Quality Camera
Imaging Source DFK21 AF04Image of an office environment, as seen by the robot
![Page 27: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/27.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 29
Robot Lab Environment
• Robot lab 550 cm x 280 cm• Robot can navigate through the world and move the fovea• Robot can be observed from a top view camera and local camera
![Page 28: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/28.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 30
Robot Lab Environment
Camera image Fovea data Composition
No video for his presentation!
![Page 29: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/29.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 31
3D Simulation of the Robot
• Environment design in 3D Studio MAX or G-MAX. • Simulation in Ogre3D• Same objects as in the real world• Shadows (optional)
![Page 30: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/30.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 32
Simulated vs. Real Vision (Fovea)
Simulation Real Robot
• Fovea transformation brings both world more together• Artificial noise simulates camera noise (optional)
![Page 31: MindRACES, First Review Meeting, Lund, 11/01/2006 Fovea-Based Robot Control for Anticipation Studies in Various Scenarios Alexander Förster, Daan Wierstra,](https://reader036.vdocument.in/reader036/viewer/2022081519/56649e225503460f94b0f703/html5/thumbnails/31.jpg)
MindRACES, First Review Meeting, Lund, 11/01/2006 33
Integration and Future Work
• Develop and evaluate anticipatory learning algorithms in the simulated environments (UW, ÖFAI); compare them also to non-anticipatory ones
• Share scenarios for attentive vision (LUCS, NBU)• Real robot with simulation as anticipation for surprise studies (CNR) • Simulation-based learning of anticipation (UW)• Share fovea simulation (IST)• Evaluate transfer of learned behavior to the real robot• Systematically increase the complexity of simulated and real
environments
• Possibly use a movable camera and zoom lens mounted on the robot