roboticperceptionand.action.0project augmentedreality.for ... ·...
TRANSCRIPT
![Page 1: RoboticPerceptionand.Action.0Project AugmentedReality.for ... · M.#De#Cecco#)RoboticPerceptionandAction RoboticPerceptionand.Action.0Project AugmentedReality.for.the.“Man.in](https://reader033.vdocument.in/reader033/viewer/2022051601/5ac530f07f8b9a333d8dba38/html5/thumbnails/1.jpg)
M. De Cecco - Robotic Perception and Action
Robotic Perception and Action -‐ Project
Augmented Reality for the “Man in the Loop”
![Page 2: RoboticPerceptionand.Action.0Project AugmentedReality.for ... · M.#De#Cecco#)RoboticPerceptionandAction RoboticPerceptionand.Action.0Project AugmentedReality.for.the.“Man.in](https://reader033.vdocument.in/reader033/viewer/2022051601/5ac530f07f8b9a333d8dba38/html5/thumbnails/2.jpg)
M. De Cecco - Robotic Perception and Action
• Introduction to AR• AR interaction technologies• ToF cameras (HW)• UNITY basics• C#• State machine with UNITY• How to display an image on the projector with UNITY• A software based on a state machine structure able to interact with
the user with UNITY• Identification and localization module:
– Superquadrics– Optimization for fitting– RANSAC– Exercitation with matlab– PCL C++
• Calibration of Tof camera, traditional cameras and projectors on a plane
Project – prerequisites
![Page 3: RoboticPerceptionand.Action.0Project AugmentedReality.for ... · M.#De#Cecco#)RoboticPerceptionandAction RoboticPerceptionand.Action.0Project AugmentedReality.for.the.“Man.in](https://reader033.vdocument.in/reader033/viewer/2022051601/5ac530f07f8b9a333d8dba38/html5/thumbnails/3.jpg)
M. De Cecco - Robotic Perception and Action
• A projector to display on a table indications to the user (HW)
• A table with retro-‐reflective sheets? (HW)• A Kinect v2 ToF camera (HW)• A software based on a state machine structure able to interact with the user (sw)
• An identification and localization module (sw)• Calibration of Tof camera with the projector and with the table (table and projector could simply fit by moving properly the projector)
Project – components
![Page 4: RoboticPerceptionand.Action.0Project AugmentedReality.for ... · M.#De#Cecco#)RoboticPerceptionandAction RoboticPerceptionand.Action.0Project AugmentedReality.for.the.“Man.in](https://reader033.vdocument.in/reader033/viewer/2022051601/5ac530f07f8b9a333d8dba38/html5/thumbnails/4.jpg)
M. De Cecco - Robotic Perception and Action
An example of setup from literature
Jaakko Hyry, Max Krichenbauer, GoshiroYamamoto, TakafumiTaketomi, Christian Sandor, Hirokazu Kato, Petri Pulli, Design of Assistive TabletopProjector-‐Camera System for the Elderly with Cognitive and Motor Skill Impairments
![Page 5: RoboticPerceptionand.Action.0Project AugmentedReality.for ... · M.#De#Cecco#)RoboticPerceptionandAction RoboticPerceptionand.Action.0Project AugmentedReality.for.the.“Man.in](https://reader033.vdocument.in/reader033/viewer/2022051601/5ac530f07f8b9a333d8dba38/html5/thumbnails/5.jpg)
M. De Cecco - Robotic Perception and Action
• The user quest for help (voice, gesture or virtual button on the table)
• The system:1. Item := 1;2. Detect the item to manipulate3. Highlight it to the user4. Animate the action to be performed5. Verify that the item was moved6. Ask if the first step was performed correctly7. Item = item + 18. If not end-‐procedure, goto point 2
Project – interaction sequence
![Page 6: RoboticPerceptionand.Action.0Project AugmentedReality.for ... · M.#De#Cecco#)RoboticPerceptionandAction RoboticPerceptionand.Action.0Project AugmentedReality.for.the.“Man.in](https://reader033.vdocument.in/reader033/viewer/2022051601/5ac530f07f8b9a333d8dba38/html5/thumbnails/6.jpg)
M. De Cecco - Robotic Perception and Action
Project – modules to develop
Project Modules Description Software to use Groups
State Machine
> Main software that manages the HMI interaction.> Main projected interface.HW: virtual buttons with retroreflective material C# -‐ UNITY
Objects Localization with AR Toolkit
> Markers placed on the objects extraction (position and location on the projector space).> Calibration of camera/projector/furnituresHW: kitchen simulated furnitures C# -‐ UNITY
Skeleton acquisition> Skeleton acquisition with ToF (kinect)> Control of task execution UNITY
AnimationsDevelopment of proper anmations (for example with Blender) UNITY -‐ Blender
Objects Localization with ToFcamera
> Point cloud extraction and its use for objects recognition and localization C++ (PCL)
Sensor Fusion
> Sensor Fusion of Objects locations estimated with Ar Toolkit and the ToF camera UNITY
![Page 7: RoboticPerceptionand.Action.0Project AugmentedReality.for ... · M.#De#Cecco#)RoboticPerceptionandAction RoboticPerceptionand.Action.0Project AugmentedReality.for.the.“Man.in](https://reader033.vdocument.in/reader033/viewer/2022051601/5ac530f07f8b9a333d8dba38/html5/thumbnails/7.jpg)
M. De Cecco - Robotic Perception and Action
1. State Machine
![Page 8: RoboticPerceptionand.Action.0Project AugmentedReality.for ... · M.#De#Cecco#)RoboticPerceptionandAction RoboticPerceptionand.Action.0Project AugmentedReality.for.the.“Man.in](https://reader033.vdocument.in/reader033/viewer/2022051601/5ac530f07f8b9a333d8dba38/html5/thumbnails/8.jpg)
M. De Cecco - Robotic Perception and Action
Objects Localization with AR Toolkit
![Page 9: RoboticPerceptionand.Action.0Project AugmentedReality.for ... · M.#De#Cecco#)RoboticPerceptionandAction RoboticPerceptionand.Action.0Project AugmentedReality.for.the.“Man.in](https://reader033.vdocument.in/reader033/viewer/2022051601/5ac530f07f8b9a333d8dba38/html5/thumbnails/9.jpg)
M. De Cecco - Robotic Perception and Action
Skeleton acquisition
![Page 10: RoboticPerceptionand.Action.0Project AugmentedReality.for ... · M.#De#Cecco#)RoboticPerceptionandAction RoboticPerceptionand.Action.0Project AugmentedReality.for.the.“Man.in](https://reader033.vdocument.in/reader033/viewer/2022051601/5ac530f07f8b9a333d8dba38/html5/thumbnails/10.jpg)
M. De Cecco - Robotic Perception and Action
Animations
Link to some examples: https://www.youtube.com/watch?annotation_id=annotation_974074&feature=iv&src_vid=l-‐gMnB3XsDM&v=VVQ7UiX8HTk
2D: https://www.youtube.com/watch?v=RKEuxDXpcFA
![Page 11: RoboticPerceptionand.Action.0Project AugmentedReality.for ... · M.#De#Cecco#)RoboticPerceptionandAction RoboticPerceptionand.Action.0Project AugmentedReality.for.the.“Man.in](https://reader033.vdocument.in/reader033/viewer/2022051601/5ac530f07f8b9a333d8dba38/html5/thumbnails/11.jpg)
M. De Cecco - Robotic Perception and Action
Objects Localization with ToF camera
Link to our project: http://www.robosense.it/index.php/component/allvideoshare/video/tof-‐pallet-‐identification-‐mocup?Itemid=172
![Page 12: RoboticPerceptionand.Action.0Project AugmentedReality.for ... · M.#De#Cecco#)RoboticPerceptionandAction RoboticPerceptionand.Action.0Project AugmentedReality.for.the.“Man.in](https://reader033.vdocument.in/reader033/viewer/2022051601/5ac530f07f8b9a333d8dba38/html5/thumbnails/12.jpg)
M. De Cecco - Robotic Perception and Action
Sensor Fusion