human-robot outline collection - mcgill university
TRANSCRIPT
Human-Robot Collaboration for Data
Collectionin The Aqua Project
Gregory DudekDirector, School of Computer Science
James McGill ProfessorMember, Center for Intelligent Machines
McGill University
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Outline• Introduction: robotics & coral reef studies
• Hardware: our robots
• Recognition of places and surfaces
• Navigation and control
• Terrain learning, gait selection
• Human-Robot Interaction
• Conclusion
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
McGill, MontrealMcGillMontreal
Boston
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
McGillMontreal
BarbadosSan
Diego
McGillArcticstation
What is robotics?
TakingAction
Planning"Thinking"
In the Robot
OutsideWorld
Text
MeasuringFiguring outwhat we see
Spot the Robot
Microwave oven?
Measures something (how cooked)
Makes a decision (another 10 minutes).
Does something to the outside world
(cooks stuff).
About Robotics (in general).
• Robotics (broadly):
» The science and engineering of replicating the attributes of living beings, and humans in particular, in machines.
• Encompasses artificial intelligence, computational vision, machine learning, psychology, mechatronics and “traditional robotics”.
• Major resurgence of deep science, technological breakthroughs in the last 10 years.
• Really using robots depends on interacting with people.
• Significant economic impact already.
Applications Already Exist
Applications Already Exist
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
What are traditional robots best suited for?
• Environments that are... • dangerous, • inaccessible, • expensive to access, • tiring, • inhospitable.
• e.g. Exploration, radiation cleanup, military surveillance, hazardous waste assessment, factory production line, non-stop deliveries, ....
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
What are traditional robots best suited for?
• Environments that are... • dangerous, • inaccessible, • expensive to access, • tiring, • inhospitable.
• e.g. Exploration, radiation cleanup, military surveillance, hazardous waste assessment, factory production line, non-stop deliveries, ....
Undersea: inaccessible, dangerous, costly, demanding.
Most of the world is undersea, yet it’s the environment on earth we understand the least well!
Aqua Project: objectives
• Aqua is about developing a robot that can walk and swim, and which exhibits the ability to use vision to know where it is and what is near it.– Underlying science: model the world using artificial cognition.
– Current application is to assist a human biologist.
• Survey and monitor the conditions on a coral reef. – By being able to land on the bottom and move around, the robot
can make regular observations without disturbing the natural organisms.
• The ability to walk, swim and use vision underwater is unique to AQUA.
11
Aqua Project: objectives
• Aqua is about developing a robot that can walk and swim, and which exhibits the ability to use vision to know where it is and what is near it.– Underlying science: model the world using artificial cognition.
– Current application is to assist a human biologist.
• Survey and monitor the conditions on a coral reef. – By being able to land on the bottom and move around, the robot
can make regular observations without disturbing the natural organisms.
• The ability to walk, swim and use vision underwater is unique to AQUA.
11
G. Dudek, MRI presentation for CSA
Issues• How to control the robot.
– Just follow a diver, take instructions, work autonomously.
• How to model (describe) the bottom.
– By building 2-dimensional and 3-dimensional models.
• Remembering where we have been before.
– Using image-based localization methods.
– Remembering the appearance of the reef.
• Select “good” images.
• Image correction [e.g. with Luz-Abril Torres-Mendez].
Applications
! Marine life inspection.! Ship inspection, underwater cables/pipeline,
security.! Diver’s ”buddy:” Take instructions, follow diver,
help out.! Long-term surveillance, e.g. on coral reefs.
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Coral Reefs
• Oceans: 70% of earth’s surface.• Reefs: Greatest diversity / area of any marine
ecosystem• 4-5% of all species (91 000) found on coral reefs
• Significant to the health of the planet: • 1/2 of the calcium that enters the world’s
oceans /year is taken up and bound intoCoral Reefs as Calcium Bicarbonate.
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Coral Reefs• 20% of the world’s reefs have been destroyed.• 24% of reefs are under imminent threat of collapse
due to human pressure, 26% under longer term threat of collapse!• Dec. 2005 there was a terrible coral bleaching (and
destruction) in the Caribbean.• 95% of Jamaica’s reefs are dead or dying.
• We need to be able to measure the changes, both to understand and ameliorate.
• This is currently taxing, error-prone, tiring and dangerous.
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Reef Studies• Detailed mosaics of reef 2D and 3D structure.• Transects of reefs
• Effects of rugosity (texture), topology and structure of intra-species interaction and mobility.
With Katerine Turgeon and Don Kramer, Biology.
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Outline• Introduction: robotics & coral reef studies
• Hardware: our robots
• Recognition of places and surfaces
• Navigation and control
• Terrain learning, gait selection
• Human-Robot Interaction
• Conclusion
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Outline• Introduction: robotics & coral reef studies
• Hardware: our robots
• Recognition of places and surfaces
• Navigation and control
• Terrain learning, gait selection
• Human-Robot Interaction
• Conclusion
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Aqua Vehicle OverviewHigh-mobility amphibious capability
Walking Swimming
IEEE Computer Magazine, January 2007.
19
Aqua robots: Overview
! “Control Stack” uses standard QNX operating system
! The “Sensor Stack” runs a custom-built OS based on Linux
! Visual Sensing runs embedded, approximately 35 KLOC in C++ implementation
! 6-DOF flipper motion! Power-autonomous! 3 cameras! Inertial and depth
measurement
Aqua: Generations
Aqua Hardware
Three basic categories
! Locomotion: direct-drive legs! Computation: multiple computers! Sensing...
Hardware: Sensing
! 3 Firewire (IIDC1394) cameras! Inertial measurement unit! Internal sensors
(thermal, current, leg position via Hall Effect, etc.)! Depth sensing! Microphone
Underwater vehicles
UT-1 Ultra Trencher 7.8 x 7.8 x 5.6 meters
Argo
Autonomous Benthic Explorer (ABE)
1200 pounds and a little over 2 meters long.
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Ease of deployment
Robustness.
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Ease of deployment
Robustness.
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Outline• Introduction: robotics & coral reef studies
• Hardware: our robots
• Recognition of places and surfaces
• Navigation and control
• Terrain learning, gait selection
• Human-Robot Interaction
• Conclusion
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Outline• Introduction: robotics & coral reef studies
• Hardware: our robots
• Recognition of places and surfaces
• Navigation and control
• Terrain learning, gait selection
• Human-Robot Interaction
• Conclusion
Much of our work has been inspired by observing how people understand their world using vision.
Arbitrary Environments: what landmarks ?
G. Dudek
Key Principle: Feature Detection
! Detection of “unusual” points in the image (or the world itself).
! Features detected a saliency operator (SIFT, SURF, Harris edges). 29
Results: Place Recognition
Results: Place RecognitionBy remembering “interesting” points, we
recognize where we (probably) are.
Results: Place Recognition
Test Images
By remembering “interesting” points, we recognize where we (probably) are.
Results: Place Recognition
Test Images Training Images
Best match
By remembering “interesting” points, we recognize where we (probably) are.
Results: Place Recognition
Test Images Training Images
Best match
Best match
By remembering “interesting” points, we recognize where we (probably) are.
Performance Versus R
eco
gnitio
n R
ate
100%
Number of features
Note: 10 windows of size 15x15 meansusing only 0.7% of the total image
content.
Optional: skip?
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Outline• Introduction: robotics & coral reef studies
• Hardware: our robots
• Recognition of places and surfaces
• Navigation and control
• Terrain learning, gait selection
• Human-Robot Interaction
• Conclusion
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Outline• Introduction: robotics & coral reef studies
• Hardware: our robots
• Recognition of places and surfaces
• Navigation and control
• Terrain learning, gait selection
• Human-Robot Interaction
• Conclusion
•Thrust is generated by oscillating foils with amplitude Aflipper around angle !offset.
•Phase offset " between the flippers to reduce parasitic oscillations of robot.
Swimming Gait Description
Giguere & Dudek
Flexible Paddle Model•90% of the paddle is rigid and do not deflect•The paddle is separated into 100 elements of equal length•Assumption: paddle hinge is fixed relative to the surrounding fluid
Drag force:
Forces Acting on the Paddle3 forces are acting on the paddle:
1. Torque:
2. Hydrodynamic force: Hi
3. Resistive Force: F
The resistive force is given by:
Bending Moment1. Bending moment in continuous form:
2. Bending moment in Discrete form:
3. Elastic bending theory:
•Controlling moments are generated by modifying:
• !offset for pitch, roll
•Amplitude Aflipper for yaw
Swimming Gait (con’t)
Pitch Roll Yaw
Giguere & Dudek Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Outline• Introduction: robotics & coral reef studies
• Hardware: our robots
• Recognition of places and surfaces
• Navigation and control
• Terrain learning, gait selection
• Human-Robot Interaction
• Conclusion
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Outline• Introduction: robotics & coral reef studies
• Hardware: our robots
• Recognition of places and surfaces
• Navigation and control
• Terrain learning, gait selection
• Human-Robot Interaction
• Conclusion
Recall: it walks (hard real time)
Canadian Space Agency, simulated Mars terrain.
Gait and terrainHow we walk depends on what we are
walking on.
52
Various terrains, various gaits
G. Dudek, M. Jenkin, C. Prahacs, A. Hogue, J. Sattar, P. Giguere, A. German, H. Liu, S. Saunderson, A. Ripsman, S. Simhon, L. A. Torres-Mendez, E. Milios, P. Zhang, I. Rekleitis. A Visually Guided Swimming Robot, Proceedings of the 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1749-1754, 2005.
Various terrains, various gaits
G. Dudek, M. Jenkin, C. Prahacs, A. Hogue, J. Sattar, P. Giguere, A. German, H. Liu, S. Saunderson, A. Ripsman, S. Simhon, L. A. Torres-Mendez, E. Milios, P. Zhang, I. Rekleitis. A Visually Guided Swimming Robot, Proceedings of the 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1749-1754, 2005.
Unsupervised Terrain Identification
Giguere & Dudek, Robotics Science and Systems (RSS), 2008.Giguere & Dudek, Robotics, Autonomous Robots, 2009.
Unsupervised Terrain Identification
Giguere & Dudek, Robotics Science and Systems (RSS), 2008.Giguere & Dudek, Robotics, Autonomous Robots, 2009.
AQUA Contact Sensors
Unsupervised CaseLearning of Terrains and Places
Unsupervised CaseLearning of Terrains and Places
Learn the combination of features that distinguishes the classes.
Class 1
Class 2
Classifier Training
Class 1?
Classifier Training
Giguere & Dudek, 2008
cost function
• For unsupervised case, our cost function capture the notion of what a good classification would look like.
Features +Dimensionality Reduction
Features +Dimensionality Reduction
Features +Dimensionality Reduction
Giguere & Dudek, 2008
2 Terrains (Linear Separator)
Giguere & Dudek
More complex case
• Can we learn to distinguish many different terrain types?
More complex case
• Can we learn to distinguish many different terrain types?
5 Terrains Data
Giguere & Dudek
5 Terrains Data
Giguere & Dudek
5 Terrains (Gaussian Mix. Model)
Classification Rate: 91%
Giguere & Dudek
5 Terrains (Gaussian Mix. Model)
Giguere & DudekGregory Dudek, http://www.cim.mcgill.ca/~dudek
Outline• Introduction: robotics & coral reef studies
• Hardware: our robots
• Recognition of places and surfaces
• Navigation and control
• Terrain learning, gait selection
• Human-Robot Interaction
• Conclusion
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Outline• Introduction: robotics & coral reef studies
• Hardware: our robots
• Recognition of places and surfaces
• Navigation and control
• Terrain learning, gait selection
• Human-Robot Interaction
• Conclusion
Vision: soft real time
Vision: soft real time
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Guidance• Full sensor-based autonomy is just too difficult
– And the costs of failures is too high,• Select trajectories by (initially) following a diver.• Diver specifies specific actions as desired.• Diver specifies where and how data is collected.
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Traditional (former) Procedure• Operator on surface controls robot.• Submerged diver can communicate with operator, for
example with hand signals.• Tether user for communication• Deficits:
– Tether dynamics & bouyancy– Entanglement risk (danger to robot & diver)– Inattention by operator– Cognitive load on diver
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Diver-Operator Gesture Language
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Diverse Command Set
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Solution: servo-based control• Instruct the robot directly using robust (engineered)
markers.• Navigation by servo-control with respect to a diver.
• No “top-side” navigator required.– Multiple tracker subsystems
• color blobs, • probabilistic model of target motion (mean-shift tracker), • color combinations with adaptive learning,
• Spatio-chromatic filters
• human motion (Fourier tracker)
• Tracker provides feedback to gait selector and controller.
Xu, Sattar, Dudek, IEEE Int. Conf. Robotics and Automation (ICRA) 2008. Giguere, Sattar, Dudek, IJRR 2009.Skip tracker details
Tracking: Learning Spatio-chromaticity for tracking
! Filters are tuned to hue variations over space! Use Boosting to learn the color distribution of
target objects over space! Hence a Spatio-Chromatic tracker
Sattar & Dudek,IEEE Int. Conf. Robotics and Automation (ICRA) 2009
Spatio-Chromatic trackers! Uses color-based tracking
! Many possible simple color filters
! Distributed and oriented in space to capture color spread of the target object
! Four types based on distribution
Basic filter/tracker types
Much like the receptive fields of color cells in the primary human visual cortex.
Boosted Ensemble
Boosted Ensemble Boosted Ensemble
Weak Spatio-chromatic trackers
Boosted Ensemble
Weak Spatio-chromatic trackers
Trackers importance weights
Boosted Ensemble
Boosted Ensemble Tracker family size
! A very large number of weak trackers can be generated for training
! e.g. For type 4 trackers:
! The value of Colors lie in the real number space (for N-RGB)
Approach Example Training data
Tracker spread Some results
Servo-based Control Behavior Control• Since visual methods are used for servo-control, use them for navigation,
behavior & action control as well.• Want very robust methods and occasional landmarks.• Also make tether almost completely unnecessary.
• Achromatic visual symbolic markers • Redundant geometrically unique robust error-correcting encoding.• Variable-resolution content, partial information if viewing
conditions are bad.• Implemented using Fourier- Tags [Dudek, Sattar, et al.].
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Tag-based Control (v1)
• Visual tags provide language for symbolic content.
• Tags specify:• Atomic actions• Parameters• Complex behaviors.
• Diver carries an dictionaryof tags from which hecan select.
Fiducial-based Human Interface
Solutions:
Cube
Flipbook
Deltohedron
•! Problem: large expression vocabulary =a lot of markers needed for explicit encoding
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Operator initiated start & left turn
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Outline• Introduction: robotics & coral reef studies
• Hardware: our robots
• Recognition of places and surfaces
• Navigation and control
• Terrain learning, gait selection
• Human-Robot Interaction
• Conclusion
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Outline• Introduction: robotics & coral reef studies
• Hardware: our robots
• Recognition of places and surfaces
• Navigation and control
• Terrain learning, gait selection
• Human-Robot Interaction
• Conclusion
Additional contributors to this work include:Philippe Giguere, Junaed Sattar, Chris Prahacs,
Yogesh Girdhar, John-Paul Lobos, Mike Jenkin
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Conclusion• Navigation: internal model, vision based
sensing.
• Real-work behavior involves many levels of control and interaction.
• Need a person “in the loop” for many real tasks. Use both implicit interaction (follow the leader) and explicit cues (“do this”).
• Vision-based Human-Robot interaction is rich and effective.
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
Future Work• When should the robot ignore it’s
programming or otherwise act on it’s own?
• When should the robot ask for help?
• What constitutes a useful data item?
• Challenges of optimizing a language for human-robot interaction.
• Software systems that span multiple levels of control, but are easy to use. (You can ask about these...)
Gregory Dudek, http://www.cim.mcgill.ca/~dudekGregory Dudek, http://www.cim.mcgill.ca/~dudek
AQUA underwater robot
Gregory Dudek, http://www.cim.mcgill.ca/~dudek
AQUA underwater robot Sub-optimal legs (1)
• Good for walking, bearable but poor for swimming.
Sub-optimal legs (2)• Good for swimming, bearable but poor for walking.
Amphibious legs
• Fast
• Too fast
Amphibious legs
• Fast
• Too fast