![Page 1: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/1.jpg)
Haptic Sensing for Assistive Robots
Charlie Kemp, PhD
Associate Professor, Department of Biomedical Engineering
Adjunct, School of Interactive Computing
Adjunct, School of Electrical and Computer Engineering
Georgia Tech
![Page 2: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/2.jpg)
Haptic Sensing for Assistive Robots
• Quick overview of assistive robotics
• Data-driven models of forces
• Whole-arm tactile sensing
• Thermal tactile sensing
http://healthcare-robotics.com
Healthcare Robotics Lab
![Page 3: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/3.jpg)
Health-Related Physical Assistance
People with motor impairments, injuries and illnesses would often benefit from physical assistance.
Today in the US alone
>15,000 with ALS [1]
~290,000 with a spinal cord injury [2]
~14,000,000 aged 65 years and older have a severe disability [3]
[1] Paul Mehta, M. D. "Prevalence of Amyotrophic Lateral Sclerosis—United States, 2012–2013." MMWR. Surveillance Summaries 65 (2016).
[2] Singh, Anoushka, et al. "Global prevalence and incidence of traumatic spinal cord injury." Clin Epidemiol 6 (2014): 309-331.
[3] Brault, Matthew W. "Americans with disabilities: 2010." Current population reports 7 (2012): 0-131.
[image] http://www.nurseuncut.com.au/how-stressed-are-you/
![Page 4: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/4.jpg)
Types of Tasks
• Activities of Daily Living (ADLs)
• Feeding, toileting, transferring, dressing, and hygiene
• Predictive of ability to live independently
• Manipulation near the person’s body
• Instrumental Activities of Daily Living (IADLs)
• Housework, food preparation, taking medications, …
• Manipulation of objects in the environment
In 2010 in the US alone, ~12,000,000 “aged 6 years and older …
needed assistance with one or more ADLs or IADLs” [1]
[1] Brault, Matthew W. "Americans with disabilities: 2010." Current population reports 7 (2012): 0-131.
[images] found on the internet and used without permission
![Page 5: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/5.jpg)
Types of Tasks
• Activities of Daily Living (ADLs)
• Feeding, toileting, transferring, dressing, and hygiene
• Predictive of ability to live independently
• Manipulation near the person’s body
• Instrumental Activities of Daily Living (IADLs)
• Housework, food preparation, taking medications, …
• Manipulation of objects in the environment
In 2010 in the US alone, ~12,000,000 “aged 6 years and older …
needed assistance with one or more ADLs or IADLs” [1]
[1] Brault, Matthew W. "Americans with disabilities: 2010." Current population reports 7 (2012): 0-131.
[images] found on the internet and used without permission
![Page 6: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/6.jpg)
Provide independence
Robots preferred for some tasks [1]
24/7 personalized assistance
Robotic Opportunities
[1] Cory-Ann Smarr, Tracy L. Mitzner, Jenay M. Beer, Akanksha Prakash, Tiffany L. Chen, Charles C.
Kemp, and Wendy A. Rogers. Domestic robots for older adults: Attitudes, preferences, and potential.
International Journal of Social Robotics, 6(2):229–247, 2014.
[image] from Willow Garage
![Page 7: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/7.jpg)
Robotic Prostheses
Robotic Orthoses / Exoskeletons
Wheelchair Mounted Robot Arms
Desktop Robots
Mobile Manipulators
MySpoon by SECOM JACO by Kinova HAL by Cyberdine
DEKA Arm by DEKA
Assistive Robots
![Page 8: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/8.jpg)
Potential Benefits of Mobile Manipulators
• Operate independently from user
• No don/doff
• Assist diverse end users
• Mass market product
[image] from Willow Garage
![Page 9: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/9.jpg)
Our Data Suggest that Many People will be Open to Assistance from Mobile Manipulators
• Over 200 participants in studies about assistive mobile manipulation
• Most studies with representative end-users
• Older adults
• Nurses
• Able-bodied participants (mock generic patients)
• People with severe motor impairments
[Image] Chih-Hung King, Tiffany L. Chen, Zhengqin Fan, Jonathan D. Glass, and Charles C. Kemp,
Dusty: An Assistive Mobile Manipulator that Retrieves Dropped Objects for People with Motor
Impairments, Disability and Rehabilitation: Assistive Technology, 2011.
![Page 10: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/10.jpg)
Hai Nguyen, Matei Ciocarlie, Kaijen Hsiao, and Charles C. Kemp, ROS Commander (ROSCo):
Behavior Creation for Home Robots, IEEE International Conference on Robotics and Automation, 2013.
![Page 11: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/11.jpg)
Kelsey Hawkins, Phillip M. Grice, Tiffany L. Chen, Chih-Hung King, and Charles C. Kemp,
Assistive Mobile Manipulation for Self-Care Tasks Around the Head, 2014 IEEE Symposium
on Computational Intelligence in Robotic Rehabilitation and Assistive Technologies, 2014.
![Page 12: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/12.jpg)
How can haptic sensing help?
![Page 13: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/13.jpg)
Data-Driven Models of Forces for Robot-Assisted Tasks
![Page 14: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/14.jpg)
Statistics for the Target Force of
Shaving Strokes
Maximum 8.2 N
First Quartile 2.7 N
Median 1.9 N
Data-driven Models for Robot-Assisted Shaving
Kelsey Hawkins, Chih-Hung King, Tiffany L. Chen, and Charles C. Kemp, Informing Assistive Robots with
Models of Contact Forces from Able-Bodied Face Wiping and Shaving, IEEE International Symposium on
Robot and Human Interactive Communication (RO-MAN), 2012.
![Page 15: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/15.jpg)
Data-driven Models for Robot-Assisted Door Opening
door opening angle (deg)
ap
plied
forc
e (
N)
Advait Jain and Charles C. Kemp, Improving Robot Manipulation with Data-Driven Object-Centric
Models of Everyday Forces, Autonomous Robots, 2013.
![Page 16: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/16.jpg)
Advait Jain and Charles C. Kemp, Improving Robot Manipulation with Data-Driven Object-Centric
Models of Everyday Forces, Autonomous Robots, 2013.
Data-driven Models for Robot-Assisted Door Opening
![Page 17: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/17.jpg)
Recognition of Categories and Instances
Advait Jain and Charles C. Kemp, Improving Robot Manipulation with Data-Driven Object-Centric
Models of Everyday Forces, Autonomous Robots, 2013.
![Page 18: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/18.jpg)
Aggressive, Timid, or Smart?
Initial Force to Open
Advait Jain and Charles C. Kemp, Improving Robot Manipulation with Data-Driven Object-Centric
Models of Everyday Forces, Autonomous Robots, 2013.
![Page 19: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/19.jpg)
Advait Jain and Charles C. Kemp, Improving Robot Manipulation with Data-Driven Object-Centric
Models of Everyday Forces, Autonomous Robots, 2013.
Data-driven Models for Robot-Assisted Door Opening
![Page 20: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/20.jpg)
Data-Driven Models for Robot-Assisted Dressing
Ariel Kapusta, Wenhao Yu, Tapomayukh Bhattacharjee, C. Karen Liu, Greg Turk, and Charles C. Kemp, Data-Driven
Haptic Perception for Robot-Assisted Dressing, IEEE International Symposium on Robot and Human Interactive
Communication (RO-MAN), 2016.
![Page 21: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/21.jpg)
Daehyung Park, Zackory Erickson, Tapomayukh Bhattacharjee, and Charles C. Kemp,
Multimodal Execution Monitoring for Anomaly Detection During Robot Manipulation, IEEE
International Conference on Robotics and Automation (ICRA), 2016.
![Page 22: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/22.jpg)
Data-Driven Models of Forces for Robot-Assisted Tasks
• Common sense about forces during tasks
• Detect anomalies and unsafe situations
• Recognize object instances and classes
• Infer task-relevant state
• Forces are useful for sharing
• Data collection challenge
• Handheld device
• Human participants
![Page 23: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/23.jpg)
Whole-Arm Tactile Sensing
![Page 24: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/24.jpg)
Dominant Strategy for Robotic Manipulation has been to Avoid Contact
• Between the robot’s arm and the world
• Between the robot’s arm and other parts of its body
• Between the robot’s arm and people
![Page 25: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/25.jpg)
Reaching a high shelf
Tying a rope to a poleInstalling a car seat
PlumbingCleaning a car trunk Carrying boxes
Contact with the World is Common
[images] found on the internet and used without permission
![Page 26: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/26.jpg)
Self-contact is Common
[image] Results of Google search performed by Charles C. Kemp
![Page 27: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/27.jpg)
Contact with People is Common (e.g., when providing assistance)
[images] found on the internet and used without permission
![Page 28: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/28.jpg)
Geometric Simulation to Investigate Value of Contact
Phillip M. Grice, Marc D. Killpack, Advait Jain, Sarvagya Vaish, Jeffrey Hawke, and Charles C. Kemp, Whole-arm Tactile
Sensing for Beneficial and Acceptable Contact During Robotic Assistance, 13th International Conference on Rehabilitation
Robotics (ICORR), 2013.
![Page 29: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/29.jpg)
Unreachable End Effector Poses with Safety Margin of 4cm
Phillip M. Grice, Marc D. Killpack, Advait Jain, Sarvagya Vaish, Jeffrey Hawke, and Charles C. Kemp, Whole-arm Tactile
Sensing for Beneficial and Acceptable Contact During Robotic Assistance, 13th International Conference on Rehabilitation
Robotics (ICORR), 2013.
![Page 30: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/30.jpg)
Phillip M. Grice, Marc D. Killpack, Advait Jain, Sarvagya Vaish, Jeffrey Hawke, and Charles C. Kemp, Whole-arm Tactile
Sensing for Beneficial and Acceptable Contact During Robotic Assistance, 13th International Conference on Rehabilitation
Robotics (ICORR), 2013.
![Page 31: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/31.jpg)
Controllers that Allow Contact
• Assume
• Low contact forces have no associated penalty
• The robot has
• Low-stiffness compliant joints
• Whole-arm tactile sensing
Advait Jain, Marc D. Killpack, Aaron Edsinger, and Charles C. Kemp, Reaching in clutter with
whole-arm tactile sensing. The International Journal of Robotics Research, 32.4 (2013): 458-482.
![Page 32: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/32.jpg)
Whole-body tactile sensing is everywhere.
Nematode (~mm)
Human (~m)
Ant (~cm)
[image of nematode] Bianchi L, Mechanotransduction: Touch and feel at the molecular level as modeled in caenorhabditis elegans. Molecular Neurobiology
36(3): 254–271, 2007.
[image of ant] from the Dartmouth College Electron Microscope Facility, http://remf.dartmouth.edu/images/insectPart3SEM/source/31.html
[image of human] Lederman, Susan J., and Roberta L. Klatzky. Haptic perception: A tutorial. Attention, Perception, & Psychophysics 71.7, 1439-1459, 2009.
![Page 33: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/33.jpg)
Quasistatic Model Predictive Control
End Effector
Goal
Advait Jain, Marc D. Killpack, Aaron Edsinger, and Charles C. Kemp, Reaching in clutter with
whole-arm tactile sensing. The International Journal of Robotics Research, 32.4 (2013): 458-482.
At each time step• Generate quasi-static model
• Use quadratic programming
to find a change to the
equilibrium angles of the
joints that
• Minimizes the predicted
distance to the goal
• Subject to constraints on
predicted contact forces
![Page 34: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/34.jpg)
Advait Jain, Marc D. Killpack, Aaron Edsinger, and Charles C. Kemp, Reaching in clutter with
whole-arm tactile sensing. The International Journal of Robotics Research, 32.4 (2013): 458-482.
![Page 35: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/35.jpg)
Advait Jain, Marc D. Killpack, Aaron Edsinger, and Charles C. Kemp, Reaching in clutter with
whole-arm tactile sensing. The International Journal of Robotics Research, 32.4 (2013): 458-482.
![Page 36: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/36.jpg)
Marc D. Killpack, Ariel Kapusta, and Charles C. Kemp, Model predictive control for
fast reaching in clutter, Autonomous Robots, 2015.
![Page 37: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/37.jpg)
Tapomayukh Bhattacharjee, Advait Jain, Sarvagya Vaish, Marc D. Killpack, and Charles C.
Kemp, Tactile Sensing over Articulated Joints with Stretchable Sensors, IEEE World Haptics
Conference (WHC 2013), 2013.
Contact at the Joints
![Page 38: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/38.jpg)
Contact at the Joints
Tapomayukh Bhattacharjee, Advait Jain, Sarvagya Vaish, Marc D. Killpack, and Charles C.
Kemp, Tactile Sensing over Articulated Joints with Stretchable Sensors, IEEE World Haptics
Conference (WHC 2013), 2013.
Reaching Task
![Page 39: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/39.jpg)
Higher success rate with tactile sensing compared to per-link force-torque sensing
64,800 trials
in simulation
Advait Jain, Marc D. Killpack, Aaron Edsinger, and Charles C. Kemp, Reaching in Clutter with Whole-
Arm Tactile Sensing, The International Journal of Robotics Research (IJRR), 2013.
![Page 40: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/40.jpg)
attempt to keep force <= 5N
Advait Jain, Marc D. Killpack, Aaron Edsinger, and Charles C. Kemp, Reaching in Clutter with Whole-
Arm Tactile Sensing, The International Journal of Robotics Research (IJRR), 2013.
Higher contact forces with per-link force-torque sensing
![Page 41: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/41.jpg)
Real-time Haptic Mapping
Tapomayukh Bhattacharjee, Phillip M. Grice, Ariel Kapusta, Marc D. Killpack, Daehyung Park, and Charles C.
Kemp, A Robotic System for Reaching in Dense Clutter that Integrates Model Predictive Control, Learning,
Haptic Mapping, and Planning, IROS 2014 workshop: 3rd Workshop on Robots in Clutter: Perception and
Interaction in Clutter, 2014.
![Page 42: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/42.jpg)
Stretchable Fabric Tactile Sensors
• 41 discrete tactile sensor
elements (taxels)
• 3 on upper arm
• 22 on forearm
• 16 on the gripper
• Open hardware
Phillip M. Grice, Marc D. Killpack, Advait Jain, Sarvagya Vaish, Jeffrey Hawke, and Charles C. Kemp, Whole-arm Tactile
Sensing for Beneficial and Acceptable Contact During Robotic Assistance, 13th International Conference on Rehabilitation
Robotics (ICORR), 2013.
![Page 43: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/43.jpg)
Picking Up a Cloth and
Wiping Face in Bed
Phillip M. Grice, Marc D. Killpack, Advait Jain, Sarvagya Vaish, Jeffrey Hawke, and Charles C. Kemp, Whole-arm Tactile
Sensing for Beneficial and Acceptable Contact During Robotic Assistance, 13th International Conference on Rehabilitation
Robotics (ICORR), 2013.
![Page 44: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/44.jpg)
Grasping and Pulling up a Blanket in Bed
Phillip M. Grice, Marc D. Killpack, Advait Jain, Sarvagya Vaish, Jeffrey Hawke, and Charles C. Kemp, Whole-arm Tactile
Sensing for Beneficial and Acceptable Contact During Robotic Assistance, 13th International Conference on Rehabilitation
Robotics (ICORR), 2013.
![Page 45: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/45.jpg)
Henry Evans’s Original Comments
During the tests:
“It is very compliant”
“I like it.”
“I think it’s a good safety feature because it hardly presses against me even when I tell it to.”
“It really feels safe to be close to the robot.”
A week after the tests:
“Skin
Overall awesome
Feels VERY safe
Faster than motion planning
It just wriggles around obstacles”
“DEFINITELY keep developing this !”
Phillip M. Grice, Marc D. Killpack, Advait Jain, Sarvagya Vaish, Jeffrey Hawke, and Charles C. Kemp, Whole-arm Tactile
Sensing for Beneficial and Acceptable Contact During Robotic Assistance, 13th International Conference on Rehabilitation
Robotics (ICORR), 2013.
![Page 46: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/46.jpg)
Will contact be acceptable to others?
Phillip M. Grice, Marc D. Killpack, Advait Jain, Sarvagya Vaish, Jeffrey Hawke, and Charles C. Kemp, Whole-arm Tactile
Sensing for Beneficial and Acceptable Contact During Robotic Assistance, 13th International Conference on Rehabilitation
Robotics (ICORR), 2013.
![Page 47: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/47.jpg)
8 Able-bodied Participants
Near ConditionAway Condition
1: Strongly Disagree, 2: Disagree, 3: Neither Agree nor Disagree, 4: Agree, 5: Strongly Agree
Phillip M. Grice, Marc D. Killpack, Advait Jain, Sarvagya Vaish, Jeffrey Hawke, and Charles C. Kemp, Whole-arm Tactile
Sensing for Beneficial and Acceptable Contact During Robotic Assistance, 13th International Conference on Rehabilitation
Robotics (ICORR), 2013.
![Page 48: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/48.jpg)
Whole-Body Tactile Sensing for Unconventional Robots
![Page 49: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/49.jpg)
Perceptual Collaboration Between Robots
Ariel Kapusta, Yash Chitalia, Daehyung Park, and Charles C. Kemp, Collaboration Between a Robotic Bed
and a Mobile Manipulator May Improve Physical Assistance for People with Disabilities, RO-MAN 2016
Workshop on behavior adaptation, interaction and learning for assistive robots (BAILAR 2016), 2016.
![Page 50: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/50.jpg)
Whole-Arm Tactile Sensing
• Reach locations in clutter while keeping forces low
• Reaching around the human body
• Outperformed per-link force-torque sensing
• Challenge
• Immature technology for large area tactile sensing
• Permitting contact
• Makes more poses reachable
• Reduces line of sight sensing requirements
• Creates opportunities to sense through touch(e.g., incidental contact)
![Page 51: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/51.jpg)
Thermal Tactile Sensing
![Page 52: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/52.jpg)
Metal (room temperature)
Human Finger (above room temperature)
Heat Transfer
Active Thermal Sensing
Tapomayukh Bhattacharjee, Joshua Wade, and Charles C. Kemp, Material Recognition from Heat Transfer
given Varying Initial Conditions and Short-Duration Contact, Robotics Science and Systems (RSS), 2015.
![Page 53: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/53.jpg)
Tapomayukh Bhattacharjee, Joshua Wade, and Charles C. Kemp, Material Recognition from Heat Transfer
given Varying Initial Conditions and Short-Duration Contact, Robotics Science and Systems (RSS), 2015.
Active Thermal Sensing
![Page 54: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/54.jpg)
Materials
Thermal Effusivity
Tapomayukh Bhattacharjee, Joshua Wade, and Charles C. Kemp, Material Recognition from Heat Transfer
given Varying Initial Conditions and Short-Duration Contact, Robotics Science and Systems (RSS), 2015.
![Page 55: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/55.jpg)
Passive Thermal Sensing
[image] found on the internet and used without permission
Tapomayukh Bhattacharjee, Joshua Wade, Yash Chitalia, and Charles C. Kemp, Data-Driven Thermal
Recognition of Contact with People and Objects, IEEE Haptics Symposium, 2016.
![Page 56: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/56.jpg)
Data-Driven Thermal Recognition
Tapomayukh Bhattacharjee, Joshua Wade, Yash Chitalia, and Charles C. Kemp, Data-Driven Thermal
Recognition of Contact with People and Objects, IEEE Haptics Symposium, 2016.
![Page 57: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/57.jpg)
Tapomayukh Bhattacharjee, Joshua Wade, Yash Chitalia, and Charles C. Kemp, Data-Driven Thermal
Recognition of Contact with People and Objects, IEEE Haptics Symposium, 2016.
![Page 58: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/58.jpg)
Tapomayukh Bhattacharjee, Joshua Wade, Yash Chitalia, and Charles C. Kemp, Data-Driven Thermal
Recognition of Contact with People and Objects, IEEE Haptics Symposium, 2016.
![Page 59: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/59.jpg)
Tapomayukh Bhattacharjee, Joshua Wade, Yash Chitalia, and Charles C. Kemp, Data-Driven Thermal
Recognition of Contact with People and Objects, IEEE Haptics Symposium, 2016.
![Page 60: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/60.jpg)
Tapomayukh Bhattacharjee, Joshua Wade, Yash Chitalia, and Charles C. Kemp, Data-Driven Thermal
Recognition of Contact with People and Objects, IEEE Haptics Symposium, 2016.
![Page 61: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/61.jpg)
Passive thermal sensing performed best for humans vs. objects
98.75% with 3.65s of contact
Vs.
[images] found on the internet and used without permission
Tapomayukh Bhattacharjee, Joshua Wade, Yash Chitalia, and Charles C. Kemp, Data-Driven Thermal
Recognition of Contact with People and Objects, IEEE Haptics Symposium, 2016.
![Page 62: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/62.jpg)
Passive & active thermal sensing togetherperformed best for objects vs. objects
92.14% : generalizing to new locations in same environment84% : generalizing to different environments
with 3.65s of contact
[images] found on the internet and used without permission
Tapomayukh Bhattacharjee, Joshua Wade, Yash Chitalia, and Charles C. Kemp, Data-Driven Thermal
Recognition of Contact with People and Objects, IEEE Haptics Symposium, 2016.
![Page 63: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/63.jpg)
Force and Thermal Sensing with a
Fabric-Based Skin
A Prototype with 5 multimodal taxels : Force, Active Thermal, and Passive Thermal
Joshua Wade, Tapomayukh Bhattacharjee, and Charles C. Kemp, Force and Thermal Sensing with a
Fabric-based Skin, IROS Workshop on See, Touch, and Hear : 2nd Workshop on Multimodal Sensor-
based Robot Control for HRI and Soft Manipulation, 2016.
![Page 64: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/64.jpg)
5 Force Sensing Taxels
A Prototype with 5 Multimodal Taxels
Joshua Wade, Tapomayukh Bhattacharjee, and Charles C. Kemp, Force and Thermal Sensing with a
Fabric-based Skin, IROS Workshop on See, Touch, and Hear : 2nd Workshop on Multimodal Sensor-
based Robot Control for HRI and Soft Manipulation, 2016.
![Page 65: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/65.jpg)
A Prototype with 5 Multimodal Taxels
10 Actively Heated Thermistors10 Passive Thermistors
Joshua Wade, Tapomayukh Bhattacharjee, and Charles C. Kemp, Force and Thermal Sensing with a
Fabric-based Skin, IROS Workshop on See, Touch, and Hear : 2nd Workshop on Multimodal Sensor-
based Robot Control for HRI and Soft Manipulation, 2016.
![Page 66: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/66.jpg)
Experiments : Representative Manipulation Task 1
Distinguish between Aluminum and Pine
Joshua Wade, Tapomayukh Bhattacharjee, and Charles C. Kemp, Force and Thermal Sensing with a
Fabric-based Skin, IROS Workshop on See, Touch, and Hear : 2nd Workshop on Multimodal Sensor-
based Robot Control for HRI and Soft Manipulation, 2016.
![Page 67: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/67.jpg)
Experiments : Representative Manipulation Task 2
Distinguish between Aluminum and Pine
Joshua Wade, Tapomayukh Bhattacharjee, and Charles C. Kemp, Force and Thermal Sensing with a
Fabric-based Skin, IROS Workshop on See, Touch, and Hear : 2nd Workshop on Multimodal Sensor-
based Robot Control for HRI and Soft Manipulation, 2016.
![Page 68: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/68.jpg)
Results : Aluminum vs. Pine
Contact
Duration(s)
Recognition Accuracy
Pressing
Task
Sliding
Task
0.25 96% 73%
1.25 96% 88%
Joshua Wade, Tapomayukh Bhattacharjee, and Charles C. Kemp, Force and Thermal Sensing with a
Fabric-based Skin, IROS Workshop on See, Touch, and Hear : 2nd Workshop on Multimodal Sensor-
based Robot Control for HRI and Soft Manipulation, 2016.
![Page 69: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/69.jpg)
Thermal Tactile Sensing
• Less sensitive to contact mechanics than force sensing
• Recognize contact with
• Materials with distinct effusivities
• Human body
• Task-relevant objects
• Challenges
• Time to heat up
• Time for heat to transfer
![Page 70: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/70.jpg)
Henry Evans is severely impaired due to a brainstem stroke. He operates the
robot using a mouse pointer that he controls using motion of his head and his
fingers via an off-the-shelf head tracker and mouse buttons.
(Research was performed as part of the collaborative Robots for Humanity project.)
Henry pulls up a blanket and wipes his
face for himself while in bed at home using
a robot with intelligent tactile sensing.
Henry shaves himself at home using a
web-based application for shaving.
Henry operates devices in his house for himself
with autonomous robot actions.
General purpose robot from Willow
Garage used in this research.
Assistive mobile manipulation at home is feasible for people with severe motor impairments using conventional interfaces.
![Page 71: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/71.jpg)
Haptic Sensing for Assistive Robots
Haptic Sensing Capability Assistive Tasks
Data-driven
models of forces
Common sense
about forces
shaving, door opening,
dressing, feeding
Whole-arm
tactile sensing
Reach locations in
clutter
Reach locations around
the human body
Thermal tactile
sensing
Recognize contact
with task-relevant
categories
human vs. environment,
toothbrush vs. counter,
tactile foreground vs.
tactile background
![Page 72: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/72.jpg)
[images] found on the internet and used without permission
![Page 73: Haptic Sensing for Assistive Robots - Neuroinformatics Group · Haptic Sensing for Assistive Robots • Quick overview of assistive robotics • Data-driven models of forces • Whole-arm](https://reader030.vdocument.in/reader030/viewer/2022040311/5d53253188c993ff0e8be19f/html5/thumbnails/73.jpg)
http://healthcare-robotics.com
Credit
Phil GriceTapo Bhattacharjee
GRADUATING SOON!
Many thanks go to the students, postdocs,
collaborators, participants, and colleagues
who made this work possible.
To learn more, please visit: