educ evaluation overview
TRANSCRIPT
-
7/29/2019 Educ Evaluation Overview
1/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Projects
Insect Telepresence: HCI formal techniques
Chips in museums: informal learning assessment
Mobot: long-term historical experiment
-
7/29/2019 Educ Evaluation Overview
2/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Insect Telepresence
Educational telepresence designed using
formal HCI inquiry tools.
-
7/29/2019 Educ Evaluation Overview
3/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Insect Telepresence Robot
Problem
Increase visitors engagement with andappreciation of insects in a museum terrarium atCMNH.
Approach
Provide a scalar telepresence experience withinsect-safe visual browsing
Apply HCI techniques to design and evaluate theinput device and system Cultural modeling, expert interview, baseline observation
Measure engagement indirectly by time on task
Partner with HCII, CMNH
-
7/29/2019 Educ Evaluation Overview
4/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Insect Telepresence Robot
Innovations
Asymmetric exhibit layout
Mechanical transparency
Clutched gantry lever arm FOV-relative 3 DOF joystick
-
7/29/2019 Educ Evaluation Overview
5/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Insect Telepresence Robot
-
7/29/2019 Educ Evaluation Overview
6/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Insect Telepresence Robot
Evaluation Results:
Average group size: 3
Average age of users: 19.5 years
Three age modes: 8 years, 10 years, and 35 years Average time on task of all users: 60 seconds
Average time on task of a single user: 27 seconds
Average time on task for user groups: 93 seconds
-
7/29/2019 Educ Evaluation Overview
7/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Insect Telepresence
Time on Task: from 15 seconds to 5 minutes
-
7/29/2019 Educ Evaluation Overview
8/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Robots in Museums
Educational tours in public spaces.
-
7/29/2019 Educ Evaluation Overview
9/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
A Human-Scale Museum
Edubot Problem Increase visitors engagement and
learning at secondary exhibits in
Dinosaur Hall.
Approach Lead visitors to secondary exhibits and new facts
Design a robot to share the human social space Establish long-term iterative testing over years, not days
Time on task, observation and learning evaluations
Partner with Magic Lantern, Maya, CMNH
-
7/29/2019 Educ Evaluation Overview
10/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Chips - BBC
-
7/29/2019 Educ Evaluation Overview
11/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Museum Edubot: Technical
Contributions Required Robot Competencies: Safety, navigation, longevity
Approaches
Property-based control programming
Visual landmark-based SUF (Latombe)
Visual self-docking
h/w and s/w restart diagnostics
Fault detection & communication
-
7/29/2019 Educ Evaluation Overview
12/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Museum Edubot: Technical
Contributions Required Robot Competencies: Safety, navigation, longevity
Outcome
Zero human injuries
4 years deployment, over 500 km
traversed MTBF converging beyond 1 week
Uptime: 98%
Active diagnosis approaching 100%
-
7/29/2019 Educ Evaluation Overview
13/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Museum Edubot: Iterative
Design Cycle Design Refinements
Physical Design
Morphological Transparency: designing
informative form
Interaction Design
Behavioral Transparency: affective
interaction model Shortened length of media segments
Two-way interaction, goal-based learning
-
7/29/2019 Educ Evaluation Overview
14/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Human-scale InteractionGrange, Meyer, Soto, Kunz, Willeke
Educational tours in public spaces.
-
7/29/2019 Educ Evaluation Overview
15/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Human-scale InteractionGrange, Meyer, Soto, Kunz, Willeke
Educational tours in public spaces.
-
7/29/2019 Educ Evaluation Overview
16/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Human-scale InteractionGrange, Meyer, Soto, Kunz, Willeke
Educational tours in public spaces.
-
7/29/2019 Educ Evaluation Overview
17/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Robot Components
Nomadic XR-4000 base
Pentium-scale running Linux
Vision, ultrasonic, ir, tactile
8 lead-acid deep-cyclebatteries 12V, ~25AH each
~ 5 hrs endurance
Customized interaction top LCD + DVD media system Interaction button(s)
High-quality speaker system
-
7/29/2019 Educ Evaluation Overview
18/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Chips
Carnegie Museum of Natural History
Autonomy
5 years, > 500 km navigated, auto-docking MTBF convergence at 1 week
Proactive health state identification
-
7/29/2019 Educ Evaluation Overview
19/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Sweetlips
Hall of North American
Wildlife
3 years, > 185 kmnavigated
Tour theme interaction
-
7/29/2019 Educ Evaluation Overview
20/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Joe Historybot
Heinz History Center 2 years, > 160 km
Touchscreen, quiz,
speech generation
-
7/29/2019 Educ Evaluation Overview
21/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Adam 40-80
Free-agent robot for hire
More than 30 days of work
finished Republican National Conv.
Democratic National Conv.
-
7/29/2019 Educ Evaluation Overview
22/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Analysis of Evolution
Robot Autonomy & Vision
Interaction and Education Efficacy
-
7/29/2019 Educ Evaluation Overview
23/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Robot Autonomy Trends
Convergence of safety level Redundant case-based minimal vectors
Path-level control as on-line search in safetyzones
Diagnostic Transparency Vision camera +gdb
Trends toward specificity of state identification
Undo then - Retry methods
Navigation Metrical route-level constraints on safe areas
Topological landmark-based route map
Explicit confidence measured on visual fiducials
-
7/29/2019 Educ Evaluation Overview
24/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Landmarks: Visual Fiducials
H R b t I t ti
-
7/29/2019 Educ Evaluation Overview
25/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Human-Robot Interaction
Trends
Lower soliloquoy duration
Caricatured exaggeration of awareness
Interaction bandwidth & specificity
Unidirectional bidirectional - active query
Modifying human behavior
social inclusion experiments
-
7/29/2019 Educ Evaluation Overview
26/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Human-Robot Interaction:
Awareness Chips: Push my button to start.
Joe: Hey! Hi! Hello!
Adam: Look out, here I come!
-
7/29/2019 Educ Evaluation Overview
27/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Human-Robot Interaction:
Retention Chips: 3-minute dino-talks
Sweetlips: Which tour do you want?
Joe: Learn to speak Pittsburghese!Guess how much that bell weighs?
Adam: Participate in a poll. Test your
knowledge of presidential politics!
M i I t ti
-
7/29/2019 Educ Evaluation Overview
28/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Measuring Interaction
Performance
1 Quantifiable Observation
time on task: 74% at 5 15 min. peak ages: 5 12 ; 25 34 gender: 20% more girls
-
7/29/2019 Educ Evaluation Overview
29/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Measuring Interaction
Performance
2 Educational Efficacy
.5 .9 All dinosaurs lived during same period.
.5 .7 All dinosaurs were huge animals.
.5 .8 Other animals lived on the Earth with din.
.4 .8 All dinosaurs were carnivorous.
.4 .7 Scientists agree how to assemble bones.
.4 .5 All bones in Dinosaur hall are real.
-
7/29/2019 Educ Evaluation Overview
30/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
RAVEN in the National AviaryRobot and educator collaboration
-
7/29/2019 Educ Evaluation Overview
31/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Influencing Human Behavior
-
7/29/2019 Educ Evaluation Overview
32/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Conclusion
Autonomy: general methodology leads
to robust behavior
HRI: We influence both robotic andhuman behavior with proper design
Project Closure: the shortcomings of
robotic tour guides.
Robotic A tonom : RI 16
-
7/29/2019 Educ Evaluation Overview
33/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Robotic Autonomy: RI 16-
162U
-
7/29/2019 Educ Evaluation Overview
34/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Goals
Empower students toward all robot hardware,
electronics and software
Encourage internalized student goals, both
creative and technical
Collect comprehensive weekly information for
off-line evaluation
Make the course a starting point for learning
create a rover community
give a complete robot to every graduate
-
7/29/2019 Educ Evaluation Overview
35/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Physical Design: Trikebot
Only 4 DOF
Raised camera POV
CMUcam vision system
Payload contingency
Tricycle configuration
Torsional stress limited
Back-EMF Motor control
Minimize servo torque
-
7/29/2019 Educ Evaluation Overview
36/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Trikebot Platform
Contributions Bicycle complexity goal Mechanical transparency
Hybrid design concept
Camera gaze design CMUcam vision system
Tricycle configuration
Torsional stress limited
Back-EMF speed control Minimize servo torque
http://www-2.cs.cmu.edu/~rasc/RA/Images/Day1%20Assembly.mov -
7/29/2019 Educ Evaluation Overview
37/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Trikebot Platform
Contributions Bicycle complexity goal Mechanical transparency
Hybrid design concept
Camera gaze design CMUcam vision system
Tricycle configuration
Torsional stress limited
Back-EMF speed control Minimize servo torque
-
7/29/2019 Educ Evaluation Overview
38/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Trikebot Platform
Contributions Bicycle complexity goal Mechanical transparency
Hybrid design concept
Camera gaze design CMUcam vision system
Tricycle configuration
Torsional stress limited
Back-EMF speed control Minimize servo torque
-
7/29/2019 Educ Evaluation Overview
39/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Trikebot: Snagglepuss
-
7/29/2019 Educ Evaluation Overview
40/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Trikebot: Powerpuff
-
7/29/2019 Educ Evaluation Overview
41/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Course Organization
July 1 August 16, 2002 (Repeated in 2003)
30 students: 20 Latino; 9 women; 2 NHU
teachers
Typical age: 16-18 years
3-person teams
Challenge-based curricula
Cumulative trajectory CMU University credit
Web open-source
-
7/29/2019 Educ Evaluation Overview
42/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Educational Evaluation
Team
Prof. Kevin Crowley, University of Pittsburgh Assoc. Prof. of Education and Cognitive Psychology, U. Pitt.
Learning Research and Development Center, U. Pitt.
Director, Research & Evaluation, Pgh Childrens Museum
Informal science experiences preparing students for
classroom-based science and math education
Katie Wilkinson, Psychology at LRDC
Emily Hamner, Psychology and C.S.
John DIgnazio, M.Des and journalist
-
7/29/2019 Educ Evaluation Overview
43/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Data Collected
Initial and Final Individual Surveys
Weekly Individual Surveys
Online Documentation of Code
Weekly Interviews and Footage by Onsite
Ethnographer
One Week Formal Ethnography (mid-program)
Follow-up Web-based Monthly Surveys
-
7/29/2019 Educ Evaluation Overview
44/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Students Were Engaged
Students generally liked the contests and challenges(At least one third of the students rated every
challenge or contest as their favorite, even in later
weekly surveys)
-
7/29/2019 Educ Evaluation Overview
45/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
The students rated the speakers a 4.7 out of 5
and everystudent said the speakers were an
important component.
Students Were Engaged (contd)
-
7/29/2019 Educ Evaluation Overview
46/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
What Students Learned Learning Themes coded
Mechanics Sensors, Motors, etc.
Programming Java, Debugging, Documenting, etc.
Teamwork
Communication, Importance of Teamwork Problem Solving
Patience, Perseverance, etc.
Robot Point of View
Autonomy, Integration of Hardware and Software
Self-Identification with Science and Technology
Self-Confidence, Robotics Community, etc.
The Big Four: not rare to teach, but rare to succeed
Learning themes demonstrate broad acquisition outside the field of focus robotics.
What Students Expected to Learn and
-
7/29/2019 Educ Evaluation Overview
47/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
What Students Expected to Learn and
What They Reported LearningExpected To Learn
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Team work Program ming Problem
Solving
Mechanics ID W/ Tech robot POV
PercentofStu
dents
Expected To Learn
What Students Expected to Learn and
-
7/29/2019 Educ Evaluation Overview
48/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
Teamwork Programming Problem
Solving
Mechanics ID W/ Tech robot POV
PercentofStudents
Expected To Learn Reported Learning at the End of the Class
What Students Expected to Learn andWhat They Reported Learning
Reported Learning
-
7/29/2019 Educ Evaluation Overview
49/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Reported LearningQuotes from Final Survey
To have patience.
To open source ones code for the better of robotics.
Document what one does so someone else can repeat theexperiment [and] be just or even more successful.
Developed better skills in working in groups That no matter what the obstacle we have, we can still
overcome it and solve it.
Have more confidence with myself.
Teamwork takes a lot of communication.
I learned that doing something slow is better than doing ittwice.
-
7/29/2019 Educ Evaluation Overview
50/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Reported LearningQuotes from Final Survey
To really pay attention to what I am doing and try to solve itfirst before asking for help.
Teamwork is hard especially with varying levels of skill and
different personalitiescan be rewarding only through
compromise.
Start with the basics, then make things fancier if youwantsimple is absolutely fine if it works well.
Using states in programming.
Make active decisions. Have the attitude that if I dont do it,
no one will and remember that if you choose something, you
are also choosing not to do other things because you havelimited time, energy, etc. Choose what to do with your talents
wisely and dont waste them!
Robots arent THAT complicated.
-
7/29/2019 Educ Evaluation Overview
51/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Student Struggles
Teamwork
2%
Programming60%Problem
Solving
2%
Mechanics
24%
ID W/ Tech
0%
robot POV
12%
Robots need to be tested in the same conditions as where they will perform.Teamwork is hard.
-
7/29/2019 Educ Evaluation Overview
52/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Student Breakthroughs
Teamwork
16%
Programming
33%
Problem
Solving
11%
Mechanics
24%
ID W/ Tech
7%
robot POV
9%
The big discovery was that if I try hard, by working with my teammates,
we could make a lot of things happen.
Don't ever leave anything at the end or else you will be struggling to
finish it on time.
-
7/29/2019 Educ Evaluation Overview
53/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Learning Trajectories
Statistical analysis of individual themes over
time.
Learning in terms of general to specific
knowledge transitions The Freshman engineering model
The following results all achieve statistical
significance
-
7/29/2019 Educ Evaluation Overview
54/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Mechanics and Programming
By the end of the course, students mentioned specific robot
technologies (like CMUcam) more often indicating that they had
become comfortable with the parts of the robots.
-
7/29/2019 Educ Evaluation Overview
55/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Mechanics and Programming
By the end of the course, students were much more specific inhow they talked about programming. Initially students realized
the course would teach them something about programming
but afterwards they were able to talk about the Java
programming environment or states in programming.
-
7/29/2019 Educ Evaluation Overview
56/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Teamwork
In the Initial Surveys Teamwork was mentioned in very
general terms (e.g. we will work in teams of three and
learn teamwork)
In the Final Survey Teamwork was mentioned much morefrequently and students were more likely to talk about
specific aspects of teamwork.
Teamwork leads to victory.
Teamwork is hard especially with varying levels ofskill and different personalitiescan be rewarding
only through compromise.
T k
-
7/29/2019 Educ Evaluation Overview
57/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Teamwork
P bl S l i
-
7/29/2019 Educ Evaluation Overview
58/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Problem Solving
If problem solving was mentioned in the initial surveystudents simply said they expected to learn problem solving
skills.
By the final survey students mentioned problem solvingmore frequently and specifically, often mentioning that the
course taught them patience or a new method of problem
solving.
Really pay attention to what I am doing and try to solve
it first before asking for help.
Always try hard for a long time and try all possible
ways to do something and if you cant then ask for
help.
Evidence That the Class Worked
-
7/29/2019 Educ Evaluation Overview
59/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Evidence That the Class Worked
For Girls
There were no significant differences in what girlsreported learning in the class as compared to boys.(specific mechanics)
Girls entered the class reporting less confidence withtechnology than boys but reported greater increasesin confidence than boys by the end of the class.
The only other gender difference was that girlsreported struggling more with programming thanboys.
Evidence That the Class Worked
-
7/29/2019 Educ Evaluation Overview
60/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Analysis:
Girls came in less confident and struggledmore with programming but were notdiscouraged:
Critical introductory engineering retentionissue.
Evidence That the Class Worked
For Girls
-
7/29/2019 Educ Evaluation Overview
61/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
Trikebot: Footage
-
7/29/2019 Educ Evaluation Overview
62/63
Illah Nourbakhsh | CMU Robotics Institute | HRI Summer Course
-
7/29/2019 Educ Evaluation Overview
63/63
Summary
Learning themes demonstrate broad acquisition outside
the field of focus, with statistical significance
Engineering retention issues appear mitigated by the
Robotic Autonomy curriculum Robotics has a larger role to play in K-12, informal and
college education
Educational robotics as a research field: ripe for
exploration Human-robot interaction
Robustness, diagnostic transparency and intelligence
Minimalist design and control (cost, endurance, simplicity)