faculty: manuela veloso, anthony stentz, alex rudnicky brett browning, m. bernardine dias students:...
TRANSCRIPT
![Page 1: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/1.jpg)
Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky
Brett Browning, M. Bernardine Dias
Students: Thomas Harris, Brenna Argall, Gil Jones
Satanjeev Banerjee
Dynamic Human-Robot Teams Engaged in Complex Adversarial Tasks Using Language-
Based Communication
Carnegie Mellon School of Computer ScienceBoeing Human-Robot Teams Project
![Page 2: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/2.jpg)
2Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Project Goals
Treasure hunt with two or more teams of humans and robots competing to locate target objects as they explore an unknown space
Research Goals/Questions: Specify team member’s roles and capabilities to
perform tasks Rapidly form ad-hoc heterogeneous teams of robots
and humans Robots and humans executing synchronized action
as a team, while communicating via speech Improve team performance with experience
![Page 3: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/3.jpg)
3Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Alignment with Boeing’s Objectives
Our project develops component technologies relevant to:
FCS – Force multiplication for human-robot teams NASA Code T – Emphasis on robots assisting and augmenting
humans in complex tasks Space activities (Other than Code T)
Automating launch sites Automating escape systems Automating maintenance inspections/repair Ice inspections
Aircraft maintenance Automating maintenance inspections/repair Ice inspections
Automated baggage handlers
In collaboration with Phillip Koons, Boeing
![Page 4: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/4.jpg)
4Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
STP: Overview
“Skills” for low-level control, “Tactics” for single robot, “Plays” for teamwork
APPLICABLE offenseDONE aborted !offense
ROLE 1 pass 3 mark best_opponentROLE 2 blockROLE 3 pos_for_pass R B 1000 0 receive_pass shoot AROLE 4 defend_lane
![Page 5: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/5.jpg)
5Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
STP: Implementation
Play selection from playbook Dynamic role assignment Active tactic determines
transition Execution monitoring Reward, and adaptation
Play 1Play 2
Play 3Role 1 Search Retrieve
Selection
ExecutionMonitoring,Adaptation
Robot 1TacticRobot 1
TacticRobot NTactic
![Page 6: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/6.jpg)
6Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Relevant Research Interests
1. Methods for pickup teams
2. Extend play-based coordination to distributed execution
3. Effective human-robot teamwork with pickup teams
![Page 7: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/7.jpg)
7Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
TraderBots: Overview
Robots are organized as an economy
Team mission is to maximize production and minimize costs
Robots exchange virtual money for tasks to maximize individual profit
System is designed to align local and global profit maximization
$
$
$$
$
![Page 8: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/8.jpg)
8Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
TraderBots: Implementation
Operator
OpTrader
Robot 1
Robot 2
Robot 3
X X
X
XTask 1 Task 2 Task 3
Task 4
Bids
Bids
BidsRoboTrader
Auction
Announce and clear auction
OpTraderGreedy Auction
Announce and clear auction
![Page 9: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/9.jpg)
9Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Relevant Research Interests
1. Human-multi-robot interaction
2. Role assignment for highly heterogeneous teams
3. Improving robustness and adaptivity
![Page 10: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/10.jpg)
10Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Speech: Overview
Communication in mobile environments Natural spoken language-
based interaction Supporting high semantic
content Communication in multi-
participant domains Human(s) and robots(s)
interacting on the same channel
![Page 11: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/11.jpg)
11Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Speech: Implementation
Sphinx ASR engine Speaker-independent recognition Speaker-adaptive capability
Phoenix semantic parser Concept extraction
RavenClaw dialogue engine Task-based representations Full mixed-initiative dialogue
Festival/Theta synthesis engine Rosetta generation engine
Portable/Wearable platforms
ROBOT (A)RavenClaw
ROBOT (B)RavenClaw
Comms Channel
. . .
Sphinx
Phoenix
Theta
Rosetta
![Page 12: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/12.jpg)
12Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Relevant Research Interests1. Developing conversational
algorithms for human-robot polylogue
2. Communicating navigational information between humans and robots in unstructured environments
3. Cooperative grounding of objects, locations, and tasks in novel environments
![Page 13: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/13.jpg)
13Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Current Status Wireless network-based (UDP) communication
protocol for human-robot interaction designed and implemented
Closed loop integration between speech-based command system, a Segway robot, and a Pioneer robot accomplished
2-D kinematic simulation tool for testing interface to Pioneer robots
Video demonstration of speech-based command of a Segway robot and a Pioneer robot
Plan for “year 1 scenario” and relevant technology development and integration to accomplish this scenario
![Page 14: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/14.jpg)
14
Treasure Hunt Scenario
First Year Objectives
![Page 15: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/15.jpg)
15Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Basic Scenario 2 Segways, 2 Pioneers, and 1 human Discover and return to base as much treasure as
possible within a 20 minute period Treasure will be identified by landmarks currently used
by the Segway soccer team
Mapper (M) Seeker (S) Handler (H) Deliverer (D) Leader (L)
Humans X X X
Segways X X X
Pioneers X X X
Gators X X X
AIBOs X
![Page 16: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/16.jpg)
16Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Detailed Scenario Goal
To combine TraderBots for negotiation and role assignment with plays for synchronization
Low-level software remains independent Challenges
How do we decide which Play to adopt? How do we handle sub-teams? How do we generate “leaders”?
![Page 17: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/17.jpg)
17Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Step 1: Form Sub-Teams
Lets form a sub-team
Fixed sub-teams for May 2005
![Page 18: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/18.jpg)
18Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Step 2: Command to Search
Team A, search area 1
1. Build map
![Page 19: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/19.jpg)
19Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Step 3: Explore and Search
1. Follow 1. Go to search area 2. Build map
![Page 20: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/20.jpg)
20Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Step 4: Search
1. Follow and search for treasure
1. Execute search pattern2. Build map
![Page 21: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/21.jpg)
21Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Step 5: Found!
We found it!
We are at <x,y>
![Page 22: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/22.jpg)
22Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Step 6: Recovery
Human aided loading
![Page 23: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/23.jpg)
23Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Step 7: Return
1. Follow2. Unload
1. Return home2. Unload
Human aided unloading
![Page 24: Faculty: Manuela Veloso, Anthony Stentz, Alex Rudnicky Brett Browning, M. Bernardine Dias Students: Thomas Harris, Brenna Argall, Gil Jones Satanjeev Banerjee](https://reader036.vdocument.in/reader036/viewer/2022062422/56649ed35503460f94be32d3/html5/thumbnails/24.jpg)
24Carnegie Mellon School of Computer Science Boeing Human-Robot Teams Project
Post Year 1 Goals Failure detection and recovery Outdoor environments Adversaries Larger more diverse teams Adaptation and learning