with the increase of automated driving in the …wc053rq8118/me310...as cars become driverless,...
TRANSCRIPT
With the increase of automated driving in the automotive industry, the driver-car team dynamic is
evolving from a master-slave relationship, where the user simply controls the machine, to a teammate
dynamic, where the car and driver are working together. As cars become driverless, fully autonomous
vehicles, the driving experience will realistically be more of a passenger experience.
Renault, the French car manufacturer, is interested in the future of autonomous technology. They have
provided us with a project prompt that pertains to understanding and improving the future
communication between an autonomous car and the person ‘driving’ it. They feel ambient
communication can play a key role in this new team dynamic.
Last quarter, the focus of ME310 was to better understand the design space and project prompt. This
process involved need-finding, benchmarking, and initial prototyping to build the foundation for future
designs.
The goal for this winter quarter was to define requirements for a final system and envision a final
product that will be delivered in the spring, accomplished by testing a wide variety of potential
solutions. The main tools for developing these requirements are prototyping and user feedback.
Throughout the winter quarter, Team Renault designed, built, and tested a large number of widely
different prototypes diverging from one design to the next in order to ensure all avenues of the design
space were explored. These designs included almost every sense including: smell, sight, hearing, and
touch. Examples include a Dark Horse Prototype that paired smell with navigational cues (Figure 1A) and
an LED display for changes in acceleration (Figure 1B).
Figure 1: Images of Smell (A) and LED (B) prototypes
These prototypes led to the development of a series of design requirements, which were broken down
into two categories: functional and physical. Functional requirements describe what the system must
do. Key functional requirements include improving awareness of vehicle’s knowledge and increasing
trust in vehicle safety, which are crucial to improving the team dynamic between a driver and an
autonomous car. Physical requirements describe what the system must be. These physical requirements
include the final system working regardless of ambient conditions, and for a wide variety of passengers.
With requirements established, a “nugget,” or key finding, was still missing. Rounds of need-finding,
benchmarking, and discussions with the Renault liaison directed Team Renault towards a final vision. A
user survey was conducted that tried to gauge what type of information future users find useful. One of
the key findings is that passengers would like to ensure their drivers know where obstacles are around
them.
NTNU built and tested a prototype that tried to convey the intentions of a car. This prototype moved a
pedal at the passenger’s feet when the car was about to brake. This prototype was met with positive
results, as users were comforted by knowing what the car was going to do; the device allowed them to
prepare for the car’s actions.
A study by Wendy Ju of Stanford’s CDR also helped with the design development. In this study, an
autonomous car simulator highlighted obstacles in the car’s field of view. The car did not act on these
obstacles, but just highlighted them, as if to say, “don’t worry, I see that.” Ju determined that this
greatly improved trust between the passenger and the car.
This design development led to Team Renault’s nugget—improving the team dynamic between the
passenger and an autonomous car by improving a passenger’s trust in autonomous technology.
Improving trust involves providing transparency to autonomous vehicles, allowing the user to know
more about what the car knows and intends to do. A specific system that implements this vision involves
an ambient LED string that displays obstacles in the car’s field of view. An ambient foot pedal will
simultaneously convey the car’s intentions. An explicit display will accompany the ambient technology
to reinforce the car’s message. An illustration of this system can be seen Figure 2.
Figure 2: Illustration of Design Vision. The yellow lights indicate the location of objects ambiently, while the red display shows the car’s planned reaction to the objects.
In order to implement this vision, a strong project management plan is required. A timeline has been
created that outlines deliverables for the spring quarter leading up to EXPE. A budget has also been
developed to ensure Team Renault does not run out of funds. The current estimate of cost for the final
system is $4400 plus $1000 for EXPE materials, which will allow the Renault team to remain under
budget. Improved coordination with NTNU and well-established roles should help in the development of
the final system.
This winter quarter has flown by, and reflecting back on the past three months provides some key
lessons to carry into the spring quarter. First, designing for a problem of the future is not easy.
Forecasting who the users will be and what needs they might have can be frustrating. Understanding
this difficulty will help guide Team Renault’s spring design phase. Also, leveraging the ME310 network
should be a priority. ME310 projects are difficult, but with insights from the teaching team, global
partners, coaches, and corporate liaisons, the projects become more manageable.
1.0 Executive Summary ........................................................................................................................... 1
Contents ........................................................................................................................................................ 4
List of Tables ................................................................................................................................................. 8
List of Figures ................................................................................................................................................ 9
1.1 Glossary ....................................................................................................................................... 10
2.0 Context and Front Matter ............................................................................................................... 12
2.1 Need Statement .......................................................................................................................... 12
2.2 Problem Statement ..................................................................................................................... 12
2.3 Corporate Partner ....................................................................................................................... 13
2.4 The Design Team ......................................................................................................................... 14
2.4.1 Stanford Team ..................................................................................................................... 14
2.4.2 International Team.............................................................................................................. 15
2.5 Team Coaches ............................................................................................................................. 16
2.6 Corporate Liaison ........................................................................................................................ 17
3.0 Design Requirements ...................................................................................................................... 18
3.1 Functional.................................................................................................................................... 18
3.1.1 Requirements ...................................................................................................................... 18
3.1.2 Opportunities ...................................................................................................................... 19
3.1.3 Assumptions ........................................................................................................................ 19
3.1.4 Constraints .......................................................................................................................... 19
3.2 Physical........................................................................................................................................ 19
3.2.1 Requirements ...................................................................................................................... 19
3.2.2 Opportunities ...................................................................................................................... 20
3.2.3 Assumptions ........................................................................................................................ 20
3.2.4 Constraints .......................................................................................................................... 20
4.0 Design Development ....................................................................................................................... 21
4.1 Overview ..................................................................................................................................... 21
4.2 Motivation ................................................................................................................................... 21
4.3 Fall Need-finding and Benchmarking .......................................................................................... 23
4.3.1 Need-finding ....................................................................................................................... 23
4.3.2 Benchmarking ..................................................................................................................... 25
4.4 Winter Need-finding and Benchmarking - Stanford ................................................................... 25
4.4.1 New Persona ....................................................................................................................... 25
4.4.2 Matrix .................................................................................................................................. 27
4.4.3 Autonomous Passenger Survey .......................................................................................... 29
4.5 Research Studies ......................................................................................................................... 30
4.6 Critical Experience Prototype (CEP) ............................................................................................ 30
4.6.1 Goal ..................................................................................................................................... 30
4.6.2 System Overview ................................................................................................................. 30
4.6.3 Testing Results .................................................................................................................... 31
4.6.4 Requirements Learned ........................................................................................................ 31
4.7 Critical Functional Prototype ...................................................................................................... 31
4.7.1 Goal ..................................................................................................................................... 31
4.7.2 System Overview ................................................................................................................. 32
4.7.3 Test ...................................................................................................................................... 32
4.7.4 Testing Results .................................................................................................................... 33
4.7.5 Requirements Learned ........................................................................................................ 34
4.8 Dark Horse Prototype ................................................................................................................. 34
4.8.1 Goal ..................................................................................................................................... 34
4.8.2 System Overview ................................................................................................................. 35
4.8.3 Test ...................................................................................................................................... 35
4.8.4 Test Results ......................................................................................................................... 36
4.8.5 Requirements Learned ........................................................................................................ 36
4.9 Funktional Prototype .................................................................................................................. 37
4.9.1 Goal ..................................................................................................................................... 37
4.9.2 System Overview ................................................................................................................. 37
4.9.3 Testing Results .................................................................................................................... 37
4.9.4 Requirements Learned ........................................................................................................ 38
5.0 NTNU Winter Quarter Progress ...................................................................................................... 39
5.1 Need-finding and Benchmarking ................................................................................................ 39
5.2 Prototyping ................................................................................................................................. 43
5.3 Reflections on the process .......................................................................................................... 45
6.0 Design Specifications ...................................................................................................................... 46
6.1 Functional Prototype .................................................................................................................. 46
6.1.1 Goal ..................................................................................................................................... 46
6.1.2 System Overview ................................................................................................................. 46
6.1.3 Testing Results .................................................................................................................... 47
6.2 Final System Vision ..................................................................................................................... 48
6.2.1 Ambient Visual Display........................................................................................................ 48
6.2.2 Ambient Brake Pedal ........................................................................................................... 48
6.2.3 Explicit Visual Display .......................................................................................................... 49
6.2.4 Full System Requirements .................................................................................................. 49
6.2.5 Total System ........................................................................................................................ 49
7.0 Project Management ...................................................................................................................... 51
7.1 Deliverables and Milestones ....................................................................................................... 51
7.1.1 Winter Quarter Review ....................................................................................................... 51
7.2 Project Timeline .......................................................................................................................... 51
7.3 Distributed Team Management .................................................................................................. 52
7.4 Project Budget............................................................................................................................. 53
7.5 Reflections and Goals .................................................................................................................. 54
7.5.1 Matilde Bisballe ................................................................................................................... 54
7.5.2 Adam Craig .......................................................................................................................... 54
7.5.3 Jørgen Erichsen ................................................................................................................... 54
7.5.4 Caroline Flowers .................................................................................................................. 55
7.5.5 Achim Gerstenberg ............................................................................................................. 55
7.5.6 Jacob Gowan ....................................................................................................................... 55
7.5.7 Kornel Niedziela .................................................................................................................. 56
7.5.8 Carl C. Sole Semb ................................................................................................................ 56
8.0 Resources ........................................................................................................................................ 57
8.1 Components ................................................................................................................................ 57
8.2 Material Suppliers ....................................................................................................................... 57
8.3 Other Systems ............................................................................................................................. 57
8.4 Experts ........................................................................................................................................ 58
9.0 References ...................................................................................................................................... 59
Appendix A .................................................................................................................................................. 61
Appendix B .................................................................................................................................................. 65
Appendix C .................................................................................................................................................. 74
Critical Experience Prototype ............................................................................................................. 74
Critical Functional Prototype .............................................................................................................. 77
Dark Horse Prototype ......................................................................................................................... 80
Funktional Prototype .......................................................................................................................... 88
Functional Prototype .......................................................................................................................... 99
Appendix D ................................................................................................................................................ 122
Appendix E ................................................................................................................................................ 138
Appendix F ................................................................................................................................................ 146
Table 1: Functional Requirements .............................................................................................................. 18
Table 2: Physical Requirements .................................................................................................................. 19
Table 3: NTNU Prototype Requirements .................................................................................................... 44
Table 4: Spring Missions ............................................................................................................................. 51
Table 5: Distributed Team Roles ................................................................................................................. 52
Table 6: Winter Quarter Budget ................................................................................................................. 53
Table 7: Estimated Spring Budget ............................................................................................................... 54
Figure 1: Images of Smell (A) and LED (B) prototypes .................................................................................. 1
Figure 2: Illustration of Design Vision. The yellow lights indicate the location of objects ambiently, while
the red display shows the car’s planned reaction to the objects. ................................................................ 2
Figure 3: Brainstorming of Being a Teammate ........................................................................................... 22
Figure 4: Brainstorming Ways to Promote Trust, Comfort, and Confidence in Autonomous Cars ............ 23
Figure 5: Our new persona Lucas, all ready for his morning commute. ..................................................... 26
Figure 6: Design matrix of possible ambient communication and information pairings. .......................... 28
Figure 7: Results from Autonomous Car Survey of asking what passengers most wanted to know during
a ride (left), and what passengers most wanted the driver to know ......................................................... 29
Figure 8: Block Diagram of CEP ................................................................................................................... 30
Figure 9: CFP In-Car Surround Sound System Diagram ............................................................................... 32
Figure 10: Block Diagram of CFP ................................................................................................................. 32
Figure 11: CFP Sound Direction Detection Accuracy .................................................................................. 33
Figure 12: Dark Horse Wizard of Oz Block Diagram .................................................................................... 35
Figure 13: Block Diagram of Funktional Prototype 1 .................................................................................. 37
Figure 14: Block Diagram of Funktional Prototype 2 .................................................................................. 37
Figure 15: The pictures show the Kinect system in the lab ........................................................................ 39
Figure 16: When focusing on business men we created the persona Thor ............................................... 41
Figure 17: Jørgen in an old suit simulating the physical disabilities of an 80-year-old............................... 42
Figure 18: Workshop with people from TrollLABS. Leading the way through smell. ................................. 43
Figure 19: Footrest communication intention of braking, acceleration and actual braking ...................... 44
Figure 20: Photo of LED Strip for Functional Prototype ............................................................................. 46
Figure 21: Block Diagram of Functional System.......................................................................................... 47
Figure 22: Final System Vision .................................................................................................................... 48
Figure 23: Final System Block Diagram ....................................................................................................... 50
Figure 24: (Left) Demonstrating the motion tracking of the Kinect [5] ...................................................... 65
Figure 25: (Right) User gesturing to accept call with Navdy [6] ................................................................. 65
Figure 26: An example of clmtrackr, analyzing a pleasantly surprised facial expression [7] ...................... 66
Figure 27: Gazetracker eye monitoring software [8] .................................................................................. 66
Figure 28: An alarm system autonomously monitoring the safety of a house [10] ................................... 67
Figure 29: A Roomba making its rounds, autonomously cleaning [11] ...................................................... 67
Figure 30: The cockpit of a commercial airliner showing the complexity of the controls [13] ................. 68
Figure 31: Wendy Ju of Stanford CDR [14].................................................................................................. 69
Figure 32: Shad Laws, our liaison extraordinaire [15] ................................................................................ 69
Figure 33: The project path of trust as more autonomy is introduced. ..................................................... 70
Figure 34: Nancy, our ideal user for design purposes ................................................................................ 73
Figure 35: Images of Critical Functional Prototype ..................................................................................... 74
Figure 36: Image of Dark Horse Wax Prototype ......................................................................................... 81
Figure 37: Image of Dark Horse Liquid Scent Release Prototype ............................................................... 83
Figure 38: Photos of Funktional Prototype ................................................................................................. 88
Figure 39: Photos of Functional Prototype including Kinect Data and User Testing ................................ 100
Ambient Communication: Ambient communication is communication that is not explicit, and therefore
can be interpreted by the user without requiring all of their focus. Ambient communication takes the
form of a feeling more than an explicit form of communication.
Automation: Automation refers to the implementation of a routine by a robot to improve efficiency. An
automated task is one that involves no decisions to be made by the robot, this is different from
autonomy where the robot must make decisions on its own.
Car-Driver Team Dynamic: The Car-Driver team dynamic is the relationship between a car and a driver.
As autonomous cars become more prevalent, the car-driver team dynamic will change drastically
overtime.
Critical Experience Prototype (CEP): A CEP, or critical experience prototype, is a prototype that is
created to observe how the experience of the user is affected by the prototype.
Critical Function Prototype (CFP): A CFP, or critical function prototype, is a prototype that is created to
test the functionality of a given technology. This is basically a feasibility prototype.
Device: In requirements section of report, any and all methods of communicating with the user.
Driver: The definition of driver changes based on the context of its use. In fully autonomous vehicles, the “driver” can be any passenger of the car. In current technology, the driver is the passenger controlling the vehicle.
Explicit Communication: Explicit communication is direct communication where the message being
expressed is clear. E.g. a visual display that has a flashing warning or a vocal cue that tells you a specific
piece of information.
Fully-Autonomous Vehicle: A fully autonomous vehicle is defined as a vehicle that requires no human
input to complete the act of driving. There is no transition of control between a fully autonomous car
and a driver, because a fully autonomous car completes the entire driving task.
Funktional Prototype: A funktional prototype is a team firsts attempt to build an integrated system;
while it is expected to be comprised of several working subsystems, it is not expected to be pretty—the
archetype is the hummer created on the Red Green Show by duct-taping two cars together.
Haptics: Haptics refers to tactile or touch feedback to the user. This is an area of HMI research that is
currently on the rise.
Human-Machine-Interface (HMI): HMI refers to the area of research that is looking to improve how
machines and humans are able to interact. Ambient communication could be a key technology that can
improve HMIs.
Master-Slave Topology: A two-being dynamic where one gives commands (in the instance of car, a user,
through steering/pedals/etc.) which are blindly followed by second (in this case, the car itself) without
feedback or input.
Nugget: A nugget is a key finding that leads towards the vision of a final system. This nugget is
something to build off of and adds value to your team’s work.
Olfactory Fatigue: When humans smell a strong scent, their olfactory system becomes fatigued and
cannot identify other strong scents as distinctly—instead, the new smells are duller and harder than
they would normally be.
Perception System: All of the autonomous car’s sensors and detection. The perception system does not
make any decisions or act on any of the data, but rather simply measures, filters and processes what is
around it.
Semi-Autonomous Vehicle: In a semi-autonomous vehicle, the car cannot fully make decisions on its
own. Therefore, there needs to be a transition of power from the car to the user in order to complete a
drive.
Users: The definition of users changes throughout this report depending on the context. Users for fully-
autonomous cars can be any passenger of the vehicle. The users of current cars are the drivers.
The growth of autonomous vehicles has exploded; all automotive leaders have autonomous car
projects—for example, Mercedes-Benz recently had a driverless-car navigate San Francisco [1] -- and a
future with autonomous cars seems almost inevitable. Autonomous vehicles will revolutionize how
humans use an automobile. Instead of having to remain focused on the task of driving, an autonomous
vehicle can drive itself. For example, commutes to work can be spent answering emails, long road trips
can occur through the night as the “driver” is able to sleep, and being stuck in traffic can turn into a
chance to relax.
As the use of cars drastically changes, so too will the communication between the car and driver.
Currently, the car-driver team dynamic has not changed significantly since the advent of the automobile;
the car and driver travel together from point A to point B, with the driver in control at all times.
However, as the car is able to drive itself, the car-driver dynamic will, by necessity, change; instead of
the common master-slave relationship, the car and driver will become a team, working together to
accomplish a common goal. As this dynamic changes, the communication between the car and driver
must change as well. While current communication between a car and driver is mainly explicit, other
potential methods exist; one of these is the idea of ambient communication.
Ambient communication is, essentially, communication that is not explicit. Explicit communication could
include oral warnings or a flashing sign—anything that directly states what is being communicated.
Ambient communication takes a more subtle form and usually accompanies the explicit message. For
example, in human-to-human communication, the explicit message could be communicated through
words, but the tone of voice, body language, and facial expressions provide more information about the
main message in an ambient way. Ambient communication also usually manifests itself as a feeling;
instead of being something you simply hear or see, it is something you feel.
With the automotive industry moving towards fully autonomous automobiles, there exists a need for
new methods of communication between the car and driver. The car-driver team dynamic will
drastically change and the way vehicles convey information must change as well. One of these new
methods that can be explored is ambient communication to improve the driver-car team dynamic.
In order to fulfill the need for new methods of communication in order to improve the car-driver team
dynamic, there are two main questions that need to be answered:
1) What information/emotions/knowledge should be conveyed to the user?
2) How should that information be conveyed into a feeling?
Not all information is effectively communicated in an ambient way. Also, not all information needs to be
communicated to the user if he or she is not involved in every decision. Determining what information a
fully autonomous vehicle user wants to know, and ensuring this information can be communicated
effectively in an ambient way, is a major issue that needs to be addressed in the completion of this
project.
The information provided to the user also has a goal: improve the team dynamic between a driver and
an autonomous car. One way to improve this team dynamic is to allow the driver to have more trust in
the car. Without trust between team members, it is impossible to have an effective partnership.
Once it is known what information can improve trust between a driver and a car, there are many ways
this information can be conveyed into a feeling. Ambient communication can occur through a number of
various forms; haptic vibrations, light intensity, and smell are all examples of potential ambient
feedback. Determining what method of ambient communication will be most effective is a second major
task of this project.
The corporate partner for this project is Renault, an automotive manufacturer that is mostly based in
Europe. Headquartered in France, Renault is the third biggest automaker in Europe and the ninth
biggest in the world. Renault is also partnered with Nissan as part of the Renault-Nissan Alliance.
Together the Renault-Nissan Alliance is interested in the development of, among other automotive
advances, autonomous vehicles. They are currently researching aspects related to semi-autonomous
vehicles, since these will be first to market, preceding their fully autonomous brothers. Therefore, while
the area of ambient communication between a user and a fully autonomous vehicle is not currently
being researched by Renault, they are highly interested in what we will learn from this project. [2]
The enthusiastic members of the design team from the ME310 class at Stanford University can be seen
below.
Adam Craig
Status: Mechanical Engineering Graduate Student
Skills: Mechatronics, Electrical Engineering, System Integration, Embedded
Programming
Undergrad: Mechatronics Engineering at the University of Waterloo, Canada
Interests: Wearable technology, Biomechatronics, Robotics
I come to Stanford after completing 5 years of a Mechatronics Engineering from the
University of Waterloo. I have learned a lot about electronic devices and system
integration, through both work and school. However, the only design I have ever known is a very
structured engineering approach. I am taking my Master’s at Stanford in order to approach design more
creatively. My interests include mechatronic and biomechanical systems, and I love building and
tinkering with new things.
Caroline Flowers
Status: Mechanical Engineering Graduate Student
Skills: Mechanical Engineering, Biomedical Engineering, Product Design
Undergrad: Mechanical Engineering at MIT
Interests: Designs to improve everyday life, medical devices
I came to Stanford after completing a Bachelor’s degree in Mechanical Engineering at
MIT. As a Master’s (and hopefully one day PhD) student at Stanford, I’m focusing on
incorporating a more user-centric design process along with the more analytical one I
learned as an undergraduate, with current design interest focusing on medical devices, specifically
orthopedics. When not working, I can usually be found baking and then feeding everyone around me.
Jacob Gowan
Status: Mechanical Engineering Coterm Student
Skills: Mechanical Engineering, Product Realization
Undergrad: Mechanical Engineering at Stanford University
Interests: Manufacturing and design process, motorcycles
I am a first year coterminal student specializing in manufacturing engineering. At
Stanford I played football for 4 years before deciding to come back to further explore
my love of engineering. I spend my free time outside of ME 310 working on personal
projects and TAing in the PRL. On weekends I can be found napping in the sun (if
there’s no assignment due the next day), riding my motorcycle (if it is running), and making cider (if I am
feeling particularly ambitious).
Kornel Niedziela
Status: Mechanical Engineering Graduate Student
Skills: Mechatronics, Mechanical Engineering, Product Realization, Product Design,
Embedded Programming
Undergrad: Mechatronics Engineering at the University of Waterloo, Canada
Interests: Automotive, Space, Consumer Electronics
I am a Mechanical Engineering Masters student at Stanford University, but deep
down I am into robotics and mechatronics systems. I completed my Bachelor’s of
Applied Science in Mechatronics Engineering at the University of Waterloo. My goal is to further expand
my creative thinking and to explore more user centric designs over technical designs. I have two and a
half years of industry experience, but feel like there is still much to learn. I am enjoying ME310 as I am
learning about design methodology while being able to prototype and test innovative new technologies
without worrying so much about the bureaucracy involved in larger corporations.
The international team for this project attend the Norwegian University of Science and Technology
(NTNU). The awesome team members can be seen below:
Jorgen Erichsen
Status: Mechanical Engineering Graduate Student
Skills: CAD, Mechanics, Medical Devices
Undergrad: Mechanical Engineering from NTNU
Interests: Product development, Music, Tinkering with Gadgets
I was born outside the Norwegian capital, Oslo, and I am studying Mechanical
Engineering at NTNU. My specialization is in Product Development and Materials.
Previous experience includes classical mechanical engineering (statics, dynamics,
material science, etc.), and also Design Thinking, CAD, FE-analysis and some Mechatronics. Apart from
studying, I enjoy photography, biking and music.
Matilde Bisballe
Status: Engineering Design and Prototyping PhD Student
Skills: User centered design, product development
Undergrad: Engineering Design & Innovation at Technical University of Denmark
Interests: Running, Fiction Books, Interaction between people and technology
I’m a Design Engineer from the technical University of Denmark, currently writing my
PHD on problem-solving of engineering problems in the early stages of product
development. My interests are to investigate creative processes and how we can help people come up
with the best and most radical solutions, which could be by understanding the several roles and
functions of prototypes better.
Achim Gerstenberg
Status: Engineering Design and Prototyping PhD Student
Skills: Physics, Product Realization, Mechatronics
Undergrad: Experimental Physics at Bonn University and University of Duisburg-Essen
Interests: Hiking, Saxophone
I have a Master’s in experimental physics from Duisburg University in Germany.
Currently, I am a PhD candidate at TrollLABS at the Norwegian University of Science
and Technology in Trondheim. There I try to find and understand the underlying
principles for bottom-up network like project architectures applied to large-scale, complex and highly
uncertain engineering projects.
Carl C. Sole Semb
Status: Mechanical Engineering Graduate Student
Skills: Metal Workshop Experience, CAD, Arduino
Undergrad: Mechanical Engineering at NTNU
Interests: Outdoors, Engineering Design
I received my Bachelor’s degree from NTNU in Mechanical Engineering and am
currently pursuing my Master’s. I love the outdoors and being away from technology,
which allows me to have an interesting design perspective. I love having adventures
and am extremely spontaneous. My goal for this course is to apply my creativity to the design process to
help the team think outside of the box.
There are two ME310 alumni coaches for this project. Information about them can be seen below.
Bill Lukens
Occupation: Product Design Engineer at Apple
Education:
Bachelor of Science (Mechanical Engineering), Carnegie Mellon University
Master of Science (Mechanical Engineering), Stanford University
ME310: ME310 alumni, 2013-2014
Scott Steber
Occupation: Associate Director at Radicand Inc.
Education:
Bachelor of Science (Mechanical Engineering), Boston University
Master of Science (Mechanical Engineering), Stanford University
ME310: ME310 alumni, 2011-2012
The corporate liaison for this project is Shad Laws, details about him can be found below.
Shad Laws
Occupation: Innovation Projects Manager at Renault Innovation
Education:
Bachelor of Science (Mechanical Engineering), Northwestern University
Master of Science (Mechanical Engineering), Stanford University
PhD (Mechanical Engineering), Stanford University
ME310: ME310 alumni, 2003-2004
Throughout the winter quarter, the project requirements have been discovered and refined. These
requirements can be classified as functional or physical requirements.
Functional requirements describe what the system must do and how it should behave. In this section
functional requirements as well as opportunities, assumptions, and constraints are described.
Table 1: Functional Requirements
Requirement Justification Criteria/Test
Improves awareness of vehicle’s knowledge
Purpose of the system is to create a teammate dynamic by making the user understand why the vehicle makes decisions
When asked why the car will make certain decisions, the user can articulate why based on communicated information
Increases trust in vehicle safety Trust in the vehicle’s decision-making and driving abilities are the primary force behind the project; if the user trusts the vehicle, they are more likely to feel comfortable in the car
When asked how safe the user feels with the technology compared to a blind test, the former rates higher than the latter
Does not overwhelm senses If senses get saturated (for example, if too many smells are put into the cabin without clearing the air), the user might lose ability to understand communicated information, or might no longer feel comfortable in the system
There is some kind of ‘scent-clearing’ capability in the system, or majority of users tested without one respond that they do not feel overwhelmed
Non-invasive/unobtrusive If user feels the system is too much in control or they are too distracted to be able to focus on anything else, they are less likely to use the system
When given a task or allowed to focus on something else, majority of users rank the distraction from the device as minimal to non-existent on a scale of options; users do not have to stop the test because of how the device makes them feel
Clearly indicative User less likely to use the system if overly complicated
Majority of tested users can pick up on what the device is telling them and answer questions on it within five minutes, given only a broad category
Communicates Useful Knowledge
If too much information is communicated, or the information could be better communicated in another way, users will be less likely to use the device, or more easily confused
Majority of survey respondents and tested users, when asked if they want that information communicated, respond in the affirmative
Improve passenger-car team dynamic
Creating a team dynamic between user and vehicle should allow the user to feel more comfortable despite not being in control
When asked after a test, majority of tested users feel they understand the system and they are part of the driving process
Will use both ambient and explicit aspects
Multiple forms of communication can be used
Multiple types of information can be conveyed
Driver can contribute to car’s functionality/overall driving experience
‘Useful knowledge’ is global—drivers in the US would be interested in the same things as drivers in
Norway or India, for example
Experience of driving or being a passenger won’t change drastically—roads/highways/intersections
will still be essentially the same in the future
People respond differently to the same stimuli
The ambiently-communicated information must convey a feeling
Physical requirements describe what the system must be and the shape it should take. In this section
physical requirements as well as opportunities, assumptions, and constraints are described.
Table 2: Physical Requirements
Requirement Justification Criteria/Test
Works regardless of passenger’s physical condition (notably height)
Potential users could be of any size (for example, both children and adults could be in the vehicle), and height even among adults varies widely. If the system is not comfortable, the user is less likely to use it
When testing a wide variety of physical body types, majority of users are able to use the system without feeling uncomfortable
Comfortable for user If the system is not comfortable, the user is less likely to use it—for example, LEDs being too bright
Majority of users do not have complaints about the system and are capable of completing a 30 minute test without changing anything
Detection works regardless of outdoor conditions
Detection software often does not work as well in sunny areas as in shady or dark ones
90% of objects seen in the least optimal conditions (90% chosen as amount well-calibrated systems can see in sunny conditions for technology we’re currently using) [3]
Notification works regardless of outdoor conditions
Can be harder to see lights when the ambient light is bright (i.e. midday)
90% of display changes can be seen in least optimal conditions
Will use multiple senses (at the moment, sight and haptic feedback)
Object detection technology will be included as part of autonomous car technology
Object detection technology will improve along with autonomous car technology
Limited to current technology for prototyping purposes
Cannot use more power than car battery can supply
Must be completed by June 2015
The design requirements were developed throughout the winter quarter. Some of the requirements are
stated in the project prompt provided by Renault-- these requirements can be found in Appendix A.
Other requirements were learned through building prototypes and testing with users. The following
sections describe how the prototypes completed this quarter relate to the development of the design
requirements. This section also highlights how the need-finding, benchmarking, and ideation tasks
contributed to design development process.
The logic behind this project is not necessarily readily apparent when examining our project’s prompt of
use ambient communication in fully autonomous vehicles to change the relationship from a master-
slave topology to a teammate topology. Focusing on fully autonomous vehicles allows the team the
ability to work in a space that is not nearly as crowded as that of the semi-autonomous world.
Researchers are heavily looking into the semi-autonomous space while fully-autonomous is less
explored. Likewise, one can develop things well before they become necessary, having a head-start on
potential competitors.
The use of ambient communication serves the purpose of making the communication of information
less invasive. The final system will not be exclusively ambient, but the goal is to have ambient
components appropriately implemented to communicate more effectively with the passengers.
Ambiency is important because most explicit communication systems (e.g. warning lights, sounds, or
displays) often feel invasive and eventually are turned off, completely negating their effectiveness.
The teammate topology part of this project is to make the car and driver more efficient and effective
while working together. As the car is given more responsibility and agency in the relationship, the
master slave dynamic that most manufactures have designed into their cars will change. As cars are
capable of more and responsible for more, the relationship will change out of necessity or will stay the
same and be a barrier to forming trust with the fully autonomous vehicle.
To flush out these issues more fully, Team Renault spent an extensive amount of time early in the
quarter revisiting the prompt to better understand the project goals. Definitions for being in a team with
a car were created (Figure 3). To start, brainstorming what it meant to be on a team was completed;
settling on things like a division of labor and a trust in one’s teammates. From that, it was concluded
that an effective team member is, among other things reliable, does its part, and watches out for the
other team members.
Figure 3: Brainstorming of Being a Teammate
When applied to a car, one can incorporate these team member attributes in a variety of aspects:
driving in a safe, comfortable, and predictable manner; being responsive to the passengers’ needs;
explaining what is happening and why it is making certain decisions; etc. It was decided that increasing
the users’ trust, via comfort with, and confidence in the system would, be the best way to accomplish
the project goal. Ways in which a car might damage trust, comfort and confidence, were also
brainstormed (Figure 4).
Figure 4: Brainstorming Ways to Promote Trust, Comfort, and Confidence in Autonomous Cars
During the winter quarter, the team was much more oriented towards building and testing prototypes
than the fall quarter. This was necessary both to explore how one could fill the needs of the space while
leveraging the skill set of the team, and to test a wide variety of potential solutions to the project
prompt. However, while it was feasible to use the results of the fall quarter’s need-finding and the user
testing of our prototypes, additional need-finding proved necessary.
The team spent a significant portion of last quarter exploring the space through need-finding,
benchmarking, and brainstorming activities. We found that these activities were particularly necessary
for the project for multiple reasons. First, the space that the team is exploring, that of technology that
acts as a teammate toward the user, is in its infancy. Second, the fully-realized autonomous vehicles
described in the project prompt do not actually exist yet, therefore it was required to find comparable
situations and technologies to study.
The team started by considering what could be communicated and sorted these into five categories
based upon how the information changed over time.
Constant: The constant category consists of those things which are relatively constant for any car or
driver, and will not change based on current changeable conditions.
Examples: Cardinal direction of car/orientation with regards to North, time of sunrise/sunset, current
location, etc.
Car Conditions: Car conditions consists of things about the car that directly relate to its function, but
could require input/assistance from the user, or could depend on factors the car has taken into account
but has no way of avoiding or controlling.
Examples: Car diagnostics (tire pressure, low fuel, etc), car confidence in a current situation (such as
driving along a route with a large number of bars around closing time, or a route with a large number of
blind turns), etc.
Directly Controlled: Directly controlled aspects consist of those things that, in a non-fully-autonomous
car, would be in the control of the driver.
Examples: Car speed, sudden changes such as braking, route choices, etc.
Indirectly Controlled: Indirectly controlled aspects are those which the car has some control over, but
also involves interaction with cars or people—for example, while the car might be able to control the
location of traffic around the vehicle relative to it by slowing down or speeding up, or by picking an
alternative route with less traffic, the decisions of the other cars/drivers must also be taken into
consideration.
Examples: Traffic around vehicle, time to destination, entering/leaving specific zones (such as school
zones), etc.
Uncontrolled: Uncontrolled aspects are those that, while subject to change, do not accept any input
from the car about that change.
Examples: weather conditions, road conditions, etc.
To flush out how ambient communication might interact with these needs in the car the team
considered different forms of ambient communication. These modes of ambient communication were
broadly broken down by the sense or system targeted.
Vision: Lights that change color or brightness depending on state of car or displays that indicate certain
information.
Hearing: Changes in ambient noise type, volume, or location of speakers (sound vectoring).
Smell: Pre-defined smells to indicate location along an often-traveled route, type of environment
(urban, rural, suburb), or weather conditions.
Touch/Haptics: Possible solutions include a vibrating belt to indicate states or changes in the physical
state of car to show a change in the situation.
Changes in the Car: Car would modify itself based off of either the user or ongoing changes (darkening
the windows if the user appears to be sleeping, chair stabilizing itself based on turns or sudden speed
changes, etc).
This need-finding cycle formed the initial impressions of what could be possible in fully autonomous
vehicles and set the stage for further need-finding in winter quarter. A full description of our fall need-
finding can be found in Appendix B.
In the fall, exploration of what technologies that are currently being developed or used in various
industries was conducted. These technologies ranged from the aviation industry, to home robotics, and
of course, the automotive industry. These technologies actually appeared in some of our prototypes and
provided us with directions for potential future developments. A full summary of our fall benchmarking
can be found in Appendix B.
The team realized partway through the quarter that our technology-distrusting persona from the fall
quarter, Nancy, was no longer necessarily the use case that fit the problems we were most interested in
testing. While trust between the user and car was still key, someone who would feel the car was more
of a teammate than something that had to be tolerated was important. That is where the idea for
“Lucas” came from. The team sought to pursue a persona who would view the autonomous technology
not as a machine to be controlled and not trusted, but as a tool to be used and acclimated to.
Lucas is a 28-year-old up-and-coming white-collar worker who spends his weekends driving to outdoor
adventures with his girlfriend of two years whom he lives with. He views his time in all situations as a
valuable asset that is being wasted on, among other things, commutes. So while he knows he shouldn’t,
he texts and emails while driving in straight stretches or stopped at lights during his 30-minute commute
on the 101 freeway (Figure 5).
Figure 5: New persona Lucas, all ready for his morning commute.
Continuing the prototyping process through the quarter, and settling on what space to explore for the
final vision, the team drew from aspects of both of our personas—designing for someone who will not
automatically trust autonomous technology, but who is willing to take the chance with it to get some of
the advantages it would provide.
Although several different modes of communication and possible sources of valuable information to
communicate were outlined, it was realized that the list was by no means exhaustive, and it was desired
to consider every possible solution to the project prompt. So, under advice from the Renault corporate
liaison, the team sat down as a group and ideated as many separate ambient communication methods
and desirable pieces of information with both teams. Everyone then had time to rank each pairing both
on feasibility, adaptability, and usefulness (for example, using temperature of the car to communicate
outside temperature might satisfy the first two, but not the last) (Figure 6). In the figure, red indicates a
pairing that received more votes and is therefore a more viable option. Through this exercise, while
there was some differences in opinion, the team was able to lay out a smaller number of options to
consider in a final solution.
Fig
ure
6: D
esig
n m
atr
ix o
f p
oss
ible
am
bie
nt
com
mu
nic
ati
on
an
d in
form
ati
on
pa
irin
gs.
The team also realized that while, through prototyping, one could get a lot of feedback from how users
felt about distinct solutions, interviewing a lot of potential users about what they would want in an
optimal solution had not been done. It was decided to remedy this by creating a survey on
SurveyMonkey covering a wide variety of car-related topics, from car-sickness to what they would do in
a vehicle they can’t do now. After analyzing the approximately one hundred responses, it was found that
there were two particular questions that stood out to us as valuable to the teams (Figure 7).
Figure 7: Results from Autonomous Car Survey of asking what passengers most wanted to know during a ride (left), and what passengers most wanted the driver to know
The first was “What would you like to know as a passenger?” This question, seemed to have a good deal
of implicit trust in the driver. We recognized that answers like arrival time, and traffic conditions were
questions you would ask if you knew that the driver was trustworthy and safe—these were ways to
optimize your time spent in the car, not reassure you.
The more valuable input was the answers to the question “What would you like the driver to know?”
While some people answered that they would like the driver to know all of these things, the team felt
safe to interpret these results as things a responsible driver should know. The answers to this part
largely consisted of wanting to know about where other cars, imminent threats, and dangerous
conditions were. It was thought that this section provides valuable insight that the most important thing
for cars to impart onto their passengers would be the feeling that the situation is under control or, put
more plainly, the car can tell them “I got this.”
The team thinks that the sentiment expressed by that saying encapsulates the combination of
situational awareness and competency that the vehicle will need to communicate to gain the trust of its
passengers.
Though a number of studies out there provide interesting insights on the future of autonomous vehicles,
one particular study was formative for our vision: Wendy Ju’s study on how communication of an
autonomous vehicles perception of the world promotes trust between the passenger and vehicle in
simulations.
In this study, they placed people in the “autonomous” vehicle and proceeded as a normal autonomous
vehicle, with one exception. Periodically, the vehicle would point out things that it saw as it was driving.
These observations were not specific to items that the vehicle was acting on but simply acknowledging
what it could see. Even though this communication provided no piece of what the cars intended action
was, the vehicle simply pointing out (accurately) what it could see, helped to build trust with the
passenger [4]. This was done through a simulation and not an actual car.
The timing of this study was very fortuitous as the findings support what the team has come to believe is
the nugget, or key insight to build a valuable prototype off of.
The following sections describe how the prototypes built and test this winter quarter have aided the
design development process.
The goal of the CEP was to test a critical aspect of the user experience. For this prototype, the goal is to
determine how users respond to ambient vibration/haptic feedback on the torso. This ambient feedback
will be used to provide information about upcoming turns. Allowing the user to know when a turn is
about to occur, how severe the turn will be, and the direction of the turn.
The system was mounted into a car, with three main components: a potentiometer, an Arduino, and a
vibrating belt. The potentiometer was controlled by a Stanford team member and provided input to the
system. The Arduino took this input and controlled where the vibrations would occur. The vibrating belt
would be placed on the user’s torso and provide ambient feedback. The system can be seen in the block
diagram in Figure 8.
Figure 8: Block Diagram of CEP
More details regarding the implementation of this prototype including Arduino code can be seen
Appendix C.
The system was tested with five users with mixed results. Users wanted to know the information, but
the feedback was too invasive. The constant vibrations against the torso were annoying and irritating.
This prototype provided insights into how information needs to be provided to the user. Specifically, a
non-invasive form of communication is required. If the information is too invasive, it can take away from
the information being provided and be detrimental to the user experience.
The goal of the CFP was to test a critical function. After testing tactile feedback during the CEP, it was
decided that sound could be a useful medium to convey information. Other than voice systems, sound is
very ambient as humans tend to process it subconsciously for geometric location and tone.
Studies have been done on how accurately humans can detect sound location and motion [5], but the
critical question for this prototype was how well passengers can detect the sound location in a car; in
everyday driving, there are plenty of internal and external sources of noise that may throw off the
passenger. The goal was to determine if the passenger could detect the direction of sound in a real
driving environment while doing a task.
Since the nature of the design question required testing users in a driving environment, this prototype
also doubled as another CEP. The user-experience test consisted of using the same 3D sound system,
which projected a sound from a direction, and pairing it to an electronic compass so that the sound
would always come from magnetic north. The secondary goal was to test if users gained a better idea of
their absolute orientation, and if that gave them a better sense of location over time.
The system comprised of a five speaker surround sound set mounted directly around the passenger’s
head, as shown in Figure 9.
Figure 9: CFP In-Car Surround Sound System Diagram
The compass was connected to an Arduino, which sent angle values to a computer through a serial COM
port. The data was read into a basic C# program that used the irrKlang audio engine to play a song, and
then change the 3D song location according to the current car angle (Figure 10).
Figure 10: Block Diagram of CFP
The test comprised of two parts: testing a sense of cardinal direction, and testing accuracy of
determining direction of sound. First, sound was played from various speakers, and the user was told
which speaker was being used. This introduction was done so the user could become accustomed to the
different sound directions, in accordance with the study mentioned above which found that training
users before testing increased the accuracy with which they can detect sound direction [5].
After this training period, the users were driven on a pre-selected route with varying curvature, turns
and straights. At first, they were driven around with music direction coming from magnetic north
without doing any tasks. After about five minutes of acclimation, the users were given an article to read,
COM Port Compass Arduino
irrKlang 5 Channel Audio
Computer
to mimic a more authentic situation of someone working while riding in an autonomous car. Users were
then driven around for another 5-10 minutes to test the system.
The final stage of testing comprised of setting the sound to come from a fixed angle relative to the user,
and then have the passengers say which angle they heard it coming from, which was done while still
driving around. Users were again given an article to read, for the same reasons as above.
Functionally speaking, the test results were relatively decent. While reading, the users could detect the
sound direction within +/- 45 degrees, which meant that they could tell their orientation within one of
the 4 quadrants (N, S, E, W). When the users were not reading, most of the results were within +/- 20
degrees with ambient car noise and motion. The empirical results are shown in Figure 11.
Figure 11: CFP Sound Direction Detection Accuracy
The correlation between perceived and true sound direction was reasonably strong. It was noted that
there were a few trouble areas in the setup, where the acoustics caused users to misjudge sound
location--with the setup used, the sound direction directly to the user’s side was generated by a
combination of the front-side and rear-side speakers. If the users were closer to one or the other, they
tended to claim the sound was coming from that speaker rather than the side, even though both
speakers had equal volume.
The user experience testing had somewhat better results. For one, an unexpected result was users
finding the movement of sound soothing or relaxing; it was unclear if it was due to the link between
sound motion and the motion of the car or simply due to sound motion in general.
Another result was that it was not distracting, which was critical; however, it was brought up that if the
sound constantly came from a single direction (such as a long drive down a straight country road), then
it could become fatiguing. In general, users could see themselves using the system.
Additionally, some users did find that having the sound come from a direction helped them develop
better understanding of their orientation. They also expressed that the motion of the sound during the
turn was a better indicator than the absolute sound direction itself, most likely due to the dead spots in
the system mentioned above. This problem could be solved using a setup with more speakers.
This prototype strengthened the requirement that the method of conveying information needs to be
non-invasive. People found changing sound direction to be non-distracting, and therefore found it far
more pleasant than the belt from our CEP.
Other critical learning for future research and testing was that user training is very important. Training
significantly helped users understand the system better and many of the testing to date was too short
for the user to become acclimated. Even after including training time in the CFP, testing with longer
times and testing with the same person on multiple runs would have led to better results.
The goal of the Dark Horse prototype was for the team to get completely out of the norm and try
something different and crazy. After searching for a sufficiently unorthodox form of communication and
technically challenging design space, smell was chosen--more specifically, could smell be used to provide
unobtrusive navigational information?
The motivation for this area of testing came from the strong tie between memory and smell; a human’s
sense of smell is the strongest sense to induce a memory. The goal was to train a user to correlate
certain smells to percentage of route completion.
The idea is that if passengers smell the same scent halfway through their routes every single trip, then
over time they will train themselves to associate that scent with being halfway to their destination.
Afterward, every time they smell it they will instantly know they are halfway done. The motivation
behind providing distance along the route is that it was a common need for many people. People always
seem concerned with how much time is left in their journeys, especially if it is a longer ride or a boring
commute. Smell could be an unobtrusive method to convey this information.
Other unexplored options of using smell included:
Different scents for different locations (e.g. Palo Alto always smells like grass, Berkeley like
lemons, etc.)
Short scent burst when entering a new area (to alert user that scenery/location has changed)
There are many considerations to take into account when dealing with smell. The ones focused on were:
Controlling the intensity and release of the scent
Preventing cross-contamination between scents
The best form to store the scent in (e.g. solid wax, liquid, gas canister, etc.)
Handling olfactory fatigue (reduction in smell intensity after an intense smell)
A couple of functional prototypes were developed to flush out if a system would be technically
achievable. The first one consisted of resistors that melted different scented waxes to release scent; it
was quickly discovered that it would take either a significant amount time or a significant amount of
power to do so. Additionally, from a thermodynamics standpoint, repeatedly melting a solid wastes a lot
of energy.
After, that a second prototype was made that had multiple compartments filled with scented oils. These
compartments had valves that opened and closed based on the desired scent, which was channelled
into the main flow towards the user.
While the prototypes showed potential in creating a working product, at the level of quality of
prototyping, they were inadequate for use in a proper in-car test. Therefore, a Wizard of Oz setup was
utilized for user experience testing. A person in the back of the car would open a vial of scented oil and
put it behind a fan that blew at the user (Figure 12). The air entrained the scent as it passed over the
vial. It was preferred to pumping the scent directly at the user as the scent was less overwhelming. With
the fan setup the scent was more dispersed and therefore more pleasant.
Figure 12: Dark Horse Wizard of Oz Block Diagram
As stated earlier, from the CFP it was learned that training periods and longer tests were critical for
ambient communication. Therefore, the dark horse user experience testing consisted of two twenty-
five-minute drives conductive on three different days using a six total routes.
The first two days were training days, where the users were allowed to look around and correlate the
smells to progress. For additional reinforcement, an LED progress bar was shown to the users to give
them a general idea of route completion. Throughout the drive, the users were instructed to note
whenever they noticed a new smell. Different scents were released at 50%, 75%, and 85% of route
Scented Oil
Fan Passenger
completion, as well as at three minutes before arrival at the destination. The non-linear schedule was
chosen to focus on when people may care more about how far they are (having a three minute warning
to their destination is good for informing users to start wrapping up whatever they are working on,
while being notified before reaching halfway may be unnecessary). The scents released at each
checkpoint were always in the same order. In between each scent, coffee grounds were used to cleanse
the olfactory palette to prevent olfactory fatigue.
On the third day, the users were told to do a task for the whole drive and not report back when they
noticed a new scent, instead just focusing on their tasks. At the end the users were asked about their
experience. Did they like the scents? Were they distracting? Did they even notice them?
There was some interesting feedback from the users:
Some users did not associate particular smells with a given time to their destination. Instead,
they counted scents--when they had detected four different ones they knew they were close to
their destination.
They felt the experience was relaxing, and having the different smells change while reading was
soothing.
It definitely helped them understand how close the user was to their destination though they
believed there was a more effective way of conveying the information.
There were also a few unexpected observations:
Both users had an initial reaction to look up when they detected a smell. This was not intended
but could lead to using smells as a warning system.
Smells were detected more when the user was working silently. If the user was having a
conversation, some smells were missed.
There are a few key things to take away from these results. First, as anticipated, smell was a particularly
ambient form of communication. Users did have a better idea of their route completion without being
distracted by the information or having to focus all of their attention on it.
Additionally the fact that users would detect the information when they were busy working or bored,
and were able to completely ignore it when they were entertained and did not care about how far they
were is important. That means it is not too intrusive, and that it is very context dependent.
The dark horse prototype resulted in further confirming the requirement that the system being non-
invasive and unobtrusive. Users said they enjoyed the experience because the system was those two
things.
It also helped refine the team’s definition of ambient communication. It showed that training is an
integral part, and that to assess a form of ambient communication properly it takes time for the user to
become acclimated to a system. The feedback and data received was more meaningful than previous,
shorter experience prototype tests.
The goal of the Funktional prototype was to gain experience putting an entire system together—it did
not have to be polished, but it had to be complete. This stage was necessary because previous
prototypes had tested individual subsystems, instead of designing a closed loop system. The funktional
prototype combined visual and vibrational feedback in order to convey turn and speed information. A
second funktional prototype was also made that combined a visual display with an accelerometer to
have a closed system, instead of a system that requires user input.
Two different systems were used for completing a funktional prototype. The first system used three
potentiometers as input to the system. An Arduino was used to control two LED strips as well as
vibrating motors placed on the back of a seat, shown as a block diagram in Figure 13.
Figure 13: Block Diagram of Funktional Prototype 1
This first prototype provided turn information via three LEDs that would move across the strip; for
example, moving right if the car was turning right. A second LED strip was designed to provide both
current and upcoming speed information. The vibrating motors on the back of the user’s chair provided
upcoming turn information as well as large changes in acceleration.
The second prototype tried to move away from a Wizard-of-Oz type implementation. Instead of relying
on user input, this prototype was controlled via an accelerometer. The accelerometer provided data that
would be conveyed to the user via an LED strip. Once again this prototype was controlled via an Arduino,
shown as a block diagram in Figure 14.
Figure 14: Block Diagram of Funktional Prototype 2
As various accelerations are measured, the LED strip would change color from red (negative forward
acceleration), blue (no acceleration), and green (positive forward acceleration). More details about the
system implementation can be seen in Appendix C.
Because this prototype was mostly used to test system functionality, user testing was not a priority.
Therefore, each prototype was only tested with two users. The results for the first prototype were poor.
Test subjects found the system confusing and were unsure what information the LEDs were trying to
convey. However, users responded more positively to the more discrete vibrational feedback. Another
useful insight was how intuitive users found the LED strip that was used to convey turn information.
With the second prototype, most of the feedback was directed towards the functionality of the system.
Because this test was the first prototype where the control was not input by a user, there were issues
with the accuracy of the information being conveyed. However, the information from the single LED
strip was easier to process than the combined LED strip display.
There were many requirements learned from this prototype. For example, it was learned that there is a
limit to how much information one can encode in a single sense. If too much information gets encoded,
the ambient nature of the information is lost, as the user must pay more attention to the LED strip to
understand the car’s message.
Another learned requirement is that the information must be easy to learn. This was proven via the LED
strip used to indicate upcoming turns. Users found this information easy to understand because it
leveraged current sensations regarding turning. A flashing yellow light indicated severe turns which is
very similar to a blinker. Therefore, if the final system can build on sensations that users are familiar
with, the system can be easy to adopt.
This semester has been a continuously shifting journey on the way to finding a nugget to use in the final
solution. It has been challenging since the initial challenge has been so broad and futuristic, which did
not help on narrowing down to a certain focus area or helping to understand an actual use case.
Therefore the team allowed themselves to explore many different topics from motion sickness, to
working situations, and even elderly citizens. These activities have started due to a previous prototype.
All of the explorations have provided insights that, in the end, will become valuable knowledge and
background for decision making when designing and finishing the final solution.
In the following sections activities, which turned out to be particularly useful, are described.
Understanding motion sickness
As we were rounding off the previous quarter with “context awareness” and “trust issues”, we were
keen on exploring whether the issue of motion sickness had an impact on the users trust towards the
vehicle or not. We started building a setup where we could simulate driving in our workshop, and
monitoring users using a Kinect 2.0 by Microsoft. We made a small simulator using a spare widescreen
television, along with some plywood sheets and a chair (Figure 15). The simulator had room enough for
one person, who would enjoy a pre-recorded video of a car driving on a racetrack. The simulator gave us
some pointers that it is possible to make someone motion sick by having them watch a pre-recorded
video, although there were great individual differences.
Figure 15: The pictures show the Kinect system in the lab
Communication through fans and air-flow changes
We thought (and still think) that having a way to deal with motion sickness would have a huge impact on
the current car industry. From benchmarking and testing, we learnt that motion sickness develops much
quicker when the users’ internal sense of motion does not match what they observe visually. We started
prototyping how to replace the visual reference by using other senses, and thought wind would be a
good place to start. By using two fans in front of the user, we made a wizard-of-Oz prototype that used
airflow for pre-cueing the upcoming turns and accelerations.
Although we made the fans work quite well, our prototype was not comfortable to use, and gave a slight
claustrophobic feeling to some users. We had also tried making some artificial horizons by using viscous
liquids in transparent containers, but did not get any results that were conclusive enough to produce a
“nugget”.
User Studies
Continuously throughout this semester we have kept on relating the final solution to a particular use
context. This context has been changing which has resulted in the need for getting many different
insights from users. Hence we have done several user-involved activities. Some of them and their
findings are described below.
Prototype conversation on Central Station
At the central station in Trondheim we brought several prototypes to initiate a discussion on the future
autonomous car. It was interesting to see the wide differences between people being afraid of
technology and not considering an autonomous car a product for them, while others said they would
trust a car more than an unknown human driver.
Observations in the train to Værnes Airport Trondheim
In this session we observed people taking the train back and forth to Værnes Airport. We quietly
observed what people were doing while being transported and recorded the sound in the train.
People were mostly:
- Looking out of the window - Sleeping - Talking to each other - Working on a laptop
It seemed like people who had chosen the quiet zone were more introvert (either working or sleeping)
than the people in the “normal” zones.
The user studies at the central station of Trondheim, the train to Værnes, and additional information on
Norwegian business men let to the development of the persona Thor (Figure 16).
Figure 16: When focusing on business men we created the persona Thor
User studies with elderly people
The user studies with elderly people involved interviews and observations as well as trying out a full
body old suit owned by the Institute of Product Design at NTNU (See Figure 17).
Focusing on elderly people we became aware of certain needs that could be expanded to address
younger users as well. These needs concerned:
- removing complexity - keeping control - providing information on the intentions of braking and acceleration - providing information on what the car has ”seen”
Figure 17: Jørgen in an old suit simulating the physical disabilities of an 80-year-old
Workshop with TrollLabs
With the workshop with people from our research group, TrollLabs, we became aware of how easy it is
for humans to learn new rules when it comes to understanding information. For example, the
stimulation through smell was easily understood by the different participants (Figure 18).
When it comes to improving the car-experience the following topics were mentioned:
- getting feedback with disturbing the eyesight - removing less useful information from surroundings - controlling the mode of the car regarding speed and air environment
Figure 18: Workshop with people from TrollLABS. Leading the way through smell.
What should the final solution do - requirements?
When observing an intersection close to our office for several hours we got hold of the need of clear
interaction between autonomous cars other participants in traffic such as drivers of human controlled
vehicles (cars, bikes, motorcycles, trucks and busses …) but also pedestrians. The intersection has clear
right of way rules, but nevertheless the traffic flow during rush hour was not determined by traffic rules,
but by hesitant and decisive drivers and pedestrians that negotiated the order at the intersection. This
lead to a high degree of uncertainty resulting in hesitation and no one acting. Or worse, many
participants acting at the same time and sometimes leading to dangerous traffic situations. Also we
observed many stop and go movements as drivers change their decisions based on the other drivers’
behavior. This results in stopping and driving of both vehicles at the same time. Overall this observations
combined with previous findings let us to the focus of how to communicate the intentions of the future
autonomous vehicle.
Communicating intentions outside the car
To eliminate this uncertainty we tried to communicate the intention of the autonomous vehicle to its
surroundings. We wanted to communicate the presence of the autonomous vehicle and whom it had
seen as well as the intention of braking and waiting or accelerating and maintaining speed.
So far, we tested the use of the headlights of the car as indicators for stopping at a pedestrian crossing.
We turned them off as soon as it was obvious that a pedestrian wanted to cross the road. This signal
caused irritation as pedestrians are not used to this. However, they described the signal as intuitive and
helpful when using it a second time. In the same context, we tried signs with text indicating that the car
will stop and wait. This text was far less intuitive, sometimes overlooked from a distance, and even
though it was a short text, only fully recognized too late, usually while already walking in front of the car.
Communicating intentions inside the car
The footplate idea was a result of the wish to communicate intentions of the car in a non-intrusive, but
still intuitive way. Further ideating led us to an early prototype consisting of a wooden plate, two wires
and an operator with a handle sitting in the back seat, black-boxing the angle adjustment. After three
external test subjects had surprisingly good feedback, we saw the need for a more automatic system.
This second version will be tested next quarter and hopefully give the passenger an increased feeling of
the car “coming alive”. In order to do this we need to mimic the intention of the autonomous car. Since
we do not have an autonomous car we use an ultrasonic range meter attached to the brake pedal to
measure if there is the driver’s foot is hovering above the pedal. If that is the case, we interpret this as a
braking intention before the driver actually steps onto the brake and decelerates the car.
Figure 19: Footrest communication intention of braking, acceleration and actual braking
Our solution compared with the initial requirements from Renault or the ones presented in the earlier
report
Keywords from our initial functional requirements were unobtrusive, clearly indicative, communicate
useful information and easy to learn. By evaluating the first prototype of the footplate after these
requirements we got the following relevant feedback from test subjects.
Table 3: NTNU Prototype Requirements
Requirement Feedback Reflection
Unobtrusive Distraction while reading was minor. Should not be required to keep feet on the plate at all times.
By keeping the plate as an optional position of the feet, you give the passenger a choice whether or not to receive the information.
Clearly indicative Movement of plate when driver was just hovering over the gas pedal was confusing. One figured it out by himself, and another needed a 1 minute explanation.
Intuitively the movement of the plate corresponds to the movement of the gas/brake-pedal.
Communicate useful information
All subjects somehow commented that the car came alive with this additional information.
This feeling that all our subjects had was surprising, and interesting enough to look more into.
Easy to learn The first prototype had wires that needed to go on the inside of the seatbelt. The new one will presumably be perceived as a footrest, without raising questions about how to use it.
We could draw the outline of two feet on the footplate. At the same time, less is probably more in this matter.
For our group it has been very interesting and challenging to work in such a diverse group. Normally
when you work with people of same background you mostly spend time discussing analysis, where to go
from a certain finding, or other “after-process” decisions.
We have spent a lot time discussing how to approach the task, which methods to use, and how long to
work in a certain area. With that we mean that it seems as if when you work in a multidisciplinary
theme, an extra dimension of discussion is added when deciding how to approach the task. On the other
hand, the diversity comes in handy when ideating and dealing with new problems. The different
backgrounds provide knowledge and inspiration for new solutions.
Still, for a fruitful multidisciplinary team to work we believe there must be mutual acceptance of
different backgrounds, but also certain rules to follow when decision-sessions get stuck and no one is
willing to give way for other approaches than their own. This could be mediated via rules decided on
before starting up a project or a group leader, making the final decisions on where to go next.
Long distance group work is even more challenging than the multidisciplinary part of the teamwork
since ownership of ideas, knowledge sharing, and time difference provides so many barriers for the
teamwork to flow.
Working with an explorative and futuristic problem brief from Renault has been challenging. The
statement ”We heard about ambient communication…how can we use it in cars…and maybe increase
the trust relation at the same time” is so wide which has given us many difficulties on defining what
problem we were actually looking to solve.
At the same time the “prototyping-mantra” of the course has forced us to start prototyping from a very
early stage, when we did not know what to look for yet. In hindsight, we could have used our prototypes
in a more explorative, future-insight way which is also suggested by Carleton [6].
With the various design requirements, user surveys, and research studies, a nugget was found:
Improving the car-driver team dynamic via trust, gained through the user knowing more about what the
car knows and intends to do. In order to test the validity of this nugget a functional prototype was
developed.
The goal of the functional prototype was to test how well the design vision would be received. This
vision was the pairing of an ambient and explicit display conveying the perception and intention of the
autonomous car. For this prototype, the perception aspect was the main focus. This system would find
obstacles around the car and illustrate where the obstacles were to the passenger.
This prototype involved using a Kinect 2.0 sensor. The Kinect senor has a plethora of functions including
object detection, object tracking, and facial detection. One feature of the Kinect is a point cloud depth
image. This image provides a depth for every pixel within the Kinects resolution. This depth frame was
filtered and windowed in order to obtain depth data for obstacles in front of the car.
Once the Kinect data was acquired, it was mapped to an angle around the user. A 2m long Neopixel LED
strip was used to provide 180° coverage around the passenger. The LED strip only outputted a single
color and did not differentiate between objects. If an image was in the windowed frame of the Kinect, it
was displayed on the LED strip (Figure 20).
Figure 20: Photo of LED Strip for Functional Prototype
The Kinect sensor was attached to a laptop that then communicated to an Arduino via a COM port. The
Arduino drove the LED strip, shown as a block diagram in Figure 21.
Figure 21: Block Diagram of Functional System
More details about this prototype can be found in Appendix C.
This prototype was tested with three users at different times of the day--testing at different times of day
was important because daylight greatly affects both the visibility of the LED strip, and the accuracy of
the Kinect sensor. Fortunately, all users were able to effectively see the LED feedback, and the Kinect
sensor functioned in both day and night.
In terms of user feedback the results were very positive. Users both enjoyed the experience of the LEDs
and commented that seeing what the car sees allows them to trust the technology better, which was
the precise goal of the system. These results prove that adding transparency to autonomous car
technology can improve the user’s feeling of trust.
Our vision for the final system incorporates both part of the Stanford team’s Functional system and of
the NTNU team’s braking prototype. The end goal is to provide the passenger with information about
what the car’s perception system sees, as well as specific information about the car’s intensions.
The expected system will present three forms of communication to the user: a visual ambient display, an
ambient foot pedal, and an explicit visual display (Figure 22).
Figure 22: Final System Vision
The ambient visual display will extend around the passenger’s field of view. While the passengers can
look out the window and clearly see what is around them, this will assure them that the car’s perception
system sees nearby obstacles as well. This setup meets the requirement to increase the passenger’s
awareness of the vehicle’s knowledge, as well as increase their trust in vehicle safety.
The exact method of display is something to be determined (a diffused LED strip, a projected image etc).
Regardless of the final method, the system will need to detect the ambient lighting conditions and
adjust the brightness of the display accordingly, so it is not over-powering at night and visible in bright
day-light, which meets the physical requirement that the system be comfortable for the user.
At the passenger’s feet there will be a moving pedal or floor board that tilts depending on upcoming
braking or acceleration, based off of NTNU’s prototype that explored conveying vehicle intentions to the
user. From early testing this already had positive results in increasing the passenger’s trust towards the
vehicle. It also meets the requirement of increasing the passenger’s awareness of the vehicle’s
knowledge.
It was never expected that all forms of communication in the system would be ambient—as with
people, explicit communication is sometimes necessary. For that reason, we chose to strengthen the
passenger’s understanding of the vehicle’s view and intentions by presenting an explicit display. This
display will show very similar information to the ambient visual display as well the moving floor board,
but in a purely explicit and more detailed way. That way the passenger can look at a more detailed
representation if they wish, and it is a good way to train the user on what the system is doing.
It is important to note that this is not a replacement for the other two methods. Realistically a passenger
will not be constantly watching the display, so the ambient visual display will always be seen in the
passenger’s peripherals and the foot pedal will always move.
While some fulfilled requirements have been touched upon, the communication system as a whole
meets some of the other requirements too.
Since each form of communication (visual vs. motion) only conveys one piece of information, it does not
overwhelm the user’s senses. Neither is invasive or obtrusive as the light brightness will be dynamically
adjusted, and the moving floor board motion will be smooth. Users did not find the motion to be
distracting or invasive in testing.
They are also clearly indicative. In both systems, users were able to understand what the system was
telling them with ease.
Users expressed that the communicated information in both systems was useful, especially in the area
of increasing trust in the vehicle’s ability. This will improve the passenger-car team dynamic as the
passenger will henceforth feel like they can see part of the vehicle’s mind.
The communication system is, to us, the most interesting part of the project, as the other systems will
have better technology by the time autonomous cars come to market. Still, the final system will require
a few extra components to simulate information the system would receive from the autonomous driving
algorithm (Figure 23).
Figure 23: Final System Block Diagram
COM Port
Display Out
Kinect Array or LIDAR
Arduino
Kinect SDK
Computer and Software
Ultrasonic Sensor on
Brake Pedal
Ambient Visual Display
Ambient Brake Pedal Actuator
Explicit Visual Display
Perception System Communication System
A great deal was learned by the massive amount of prototyping done during the winter quarter (which
averaged out to more than one a week), and by some of the additional need-finding. While the large
number of prototypes meant we did not focus as much on any single idea, the wide variety of sensory
outputs used (hearing, smell, touch, and sight) gave us a strong perspective for how users might react to
each case.
Likewise, since we are trying to focus on what could be considered a more acquired ‘intuition’, we chose
to test fewer users, with more time devoted for each test (for example, with smell during Dark Horse,
two users underwent six tests, with only the last two involving fully ambient conditions). This change in
testing procedure allowed for what we felt are more reasonable results, and will likely be how we
continue testing during the spring quarter.
There are two major things we will have to work on during the spring quarter that we did not do as well
as during the winter: the refining process, and creating a full system instead of a ‘Wizard of Oz’ style
simulation. While we needed the wide range of prototypes to get an idea of how we wanted the final
solution to be, we missed out on the chance to practice refining one idea until it is completely solid. The
same rapid changes in plan caused us to mainly use simulations instead of fully integrated systems—
while this gave us the user feedback we needed, it will not work for a final product. However, we have
confidence in our abilities to use the resources around us to fix these gaps.
While the winter quarter was focused mainly on prototyping a wide variety of potential solutions to the
project prompt, the spring quarter will be focused on delivering one completed product by the
beginning of June at EXPE. While we have a relatively firm vision, finalizing that vision will be occurring
while the two teams meet at NTNU during spring break; the following is simply a first-round
approximation.
Table 4: Spring Missions
Mission Further Description Due Date
Travel/NTNU Collaboration Stanford team travels to Norway to do in-person prototyping and testing with the NTNU team. Two members of the Stanford team will also be traveling to Paris to consult with the Renault Human-Machine-Interface lab
March 20th-April 5th
Spring Hunting Plan Create a comprehensive plan for completing project by EXPE
April 2nd
Part X is Finished Have one (non-trivial) part of the final product completely finished Probable Part: Since our project is based so heavily on using detected information, Part X will likely involve either the object detector or a mechanism to detect the pressure on the brake and acceleration pedals.
April 23rd
Manufacturing Plans Lists of all materials, processes, schedules, quotes from outside retailers, draft of final requirements, etc.
April 23rd
Penultimate Review Largely completed hardware, with a product that should functionally stand on its own at EXPE, completed design requirement
May 21st
Expe Brochure and Poster Finished brochure and poster for the EXPE presentations
Draft: May 26th Final: May 29th
EXPE Presentations June 4th
Final Documentation June 9th
When working in teams, particularly ones that are wide-spread geographically, clearly defined roles are
important to guarantee task completion, so that no aspect falls through the cracks. Therefore, both the
Stanford and NTNU teams assigned four roles: Chief Communications Officer, Chief Documentation
Officer, Chief Financial Officer, and Chief Planning Officer.
Table 5: Distributed Team Roles
Job Stanford NTNU
Chief Communications Officer
Adam Craig Matilde Bisballe
Chief Documentation Officer
Kornel Niedziela Jørgen Erichsen
Chief Financial Officer
Jacob Gowan Carl Semb
Chief Planning Officer
Caroline Flowers Achim Gerstenberg
Communication and collaboration occurred regularly and frequently, using several methods. First, there
was a weekly skype meeting between both the NTNU and Stanford teams, usually lasting about an hour.
Second, there was a weekly/bi-weekly (depending on the amount of pivoting occurring and our
corporate liaison’s availability) between NTNU, Stanford, and our corporate liaison, also held via Skype.
Periodic check-ins about the state of projects/sharing of interesting news and developments also
occurred via a private Facebook group, limited to the eight members of the combined NTNU and
Stanford teams. Weekly emails to our coaches regarding updates and attaching documents we had
worked on during the previous week, kept them in the loop and gave them a forum for providing advice.
Document collaboration occurred through two methods. First, for things that might require
simultaneous input from multiple members of Team Renault (transcripts of team meetings,
brainstorming sessions, survey suggestions, etc.), Microsoft OneDrive was used. For sharing results and
final documentation, DropBox was used.
While in the fall, most of the need-finding and benchmarking was done cooperatively, the winter
quarter saw a divergence in areas of research and testing, with the final results being two subsystems (a
braking system and an object detection system) that are both planned to be included in the final
product, with the NTNU and Stanford teams focusing on each, respectively.
Our project continues to come in well under-budget, using less than half of our allocated winter budget.
Table 6: Winter Quarter Budget
Activity Cost
CFP $40.23
Dark Horse $489.74
FUNKtional $89.15
Functional $348.87
SUDS $374
Obtaining Car for Testing $150.91
Personalizing Team’s Loft Area $75
Total Expenditures $1490.90
Allocated Winter Budget $3000.00
Rollover Fall Budget $233.81
Rollover Winter Budget $1,742.91
Most of the budget was spent on prototyping materials: for example, during Dark Horse we needed
various kinds of scents, during Funktional and Functional we needed several kinds of electronic display
equipment, etc. This category will also likely be the biggest source of expenses during the spring quarter
as well.
The biggest concern budget-wise at the moment is obtaining a car to use for testing; while we have
been able to use a privately owned car, or rented one, this setup will not work for when a car needs to
be used for weeks in a row.
Table 7: Estimated Spring Budget
Expenditure Estimated Cost
Display Components $900
Sensing Equipment (up to 5 Kinects) $1000
EXPE $1000
Obtaining a Car $2500
SUDS $300
Total $5400
Available Budget $6250
For our group it has been very interesting and challenging to work in such a diverse group. Normally
when you work with people of same background you mostly spend time discussing analysis, where to go
from a certain finding or other “after-process” decisions.
We have experienced to spend much time discussing how to approach the task, which methods to use
and how long to work in a certain area. With that we mean that it seems as if when you work in a
multidisciplinary theme an extra dimensions of discussion is added when deciding how to approach the
task. On the other hand the diversity come in handy when ideating and dealing with new occurring
problems. Then the different backgrounds provides knowledge and inspiration for new solutions.
Still for a fruitful multidisciplinary team to work we believe there must be mutual acceptance of
different backgrounds, but also certain rules to follow when decision-sessions get stuck an no one is
willing to give way for other approaches than their own. This could be rules decided in the group before
starting up a project or a leader-role making the final decisions on where to go next.
This quarter has been a rollercoaster. There were times when I was loving ME310 and the project, and
others were I was frustrated with the design process and the problem we have been given. This quarter I
learned how hard it can be to design around a problem where there are no users. Autonomous cars are
not being driven today, therefore, trying to forecast who our users are and what their problems might
be is insanely difficult. This has been the most frustrating task for me this winter quarter.
This quarter I have also learned how cyclical the ideation process can be. At the start of the fall quarter
we were presented with the issue of trust. We began to focus elsewhere at the start of the winter
quarter. Facilitating task completion was our key need. As we continued to ideate and prototype we
circled back to focusing on trust once again. Now that we have diverged and explored as many spaces as
possible, this spring I am ready to start converging and refining a final system that will result in some
“WOWS!”.
I think working with ME310 is both very interesting and very challenging. Collaborating across such a
geographical distance is hard, although I feel that our team is both willing and able to communicate
despite different time zones and distance. For me, this quarter has been all about divergence, where we
have not been sure what to do, and what our “next big thing” will be. I think we have had a need for
stronger leadership within our team, although I feel this is getting better. Therefore, I am very excited to
see where our project is headed.
While at times it felt like we were completely lost vis-à-vis a final vision as we frolicked hither and
thither between different senses on our prototyping journey, at the same time I feel like we got a lot out
of this quarter. This quarter was also more reliant on the knowledge of those with a mechatronics
background on our team, but that will hopefully change as we move away from rapidly changing
prototyping (while they were more than willing to do the heavy lifting, the other two of us look forward
to being able to contribute as significantly again).
It’s been a process, without a doubt. We moved all over the place, had several, um, very intense
discussions, completely redid our persona (then completely redid our plan so this persona also didn’t
fit), prototyped more than we did anything else, and were accurate if not precise. But that feels like it’s
the point of winter quarter—to be out there and test everything. Yes, at times we may have scared the
teaching team with the wide directional changes, but with as large a design space as we were given, we
definitely needed the amount of divergence we did. However, despite our lack of experience refining
things during the winter quarter, I feel that it is something we can transition to without too much
difficulty. The team dynamic, both among the Stanford team and the combined Team Renault, remains
strong, and I’m excited to see where we’ll go now that we have a relatively solid vision.
About two months ago we had many ideas how to ambiently communicate. However, it was uncertain
what information is most valuable to be communicated. This resulted in a planless trying out of different
ideas with no clear vision. Therefore, we went back to needfinding and the group decided to focus on
elderly people and trying to find their pain points. At the same time I was staring out of my office
window during rush hour recognizing a lot of uncertainty between drivers at an intersection. The idea
came to mind to use the ambient communication technology that we developed earlier in someone
else’s car to communicate intentions of braking/waiting and driving. Within the group it felt like we had
consensus on this. We quickly developed the tools to measure the intentions of the driver. The hard
part, at least with the resources we have in the lab, is how to send that information to the other vehicles
and therefore we used it to communicate the driver’s intention to the passenger. Testing showed that
passengers like the system a lot. Therefore, we spend a lot of time making this system work. However, I
see this side project as a nice gimmick and we shifted away from the original and agreed upon idea,
which for me has a lot more “nugget potential”. I now figured out how to technically solve the wireless
communication problem between the cars in a fast way with enough range and I will hope to bring the
others back to that idea.
This quarter was pretty rough, not only because everyone’s schedules were significantly busier than last
quarter, but because the troubles of not having an existent user base. Though we tried to avoid
conjecture as much as possible we found it hard to design for anything without making certain
assumptions about the future technology and user. I like how we took the bold rather than the cautious
approach where this is concerned. We decided to assume what we could and apologize later if we were
too far off. We found that this made it easier to iterate more quickly and did not diminish the quality of
our work significantly.
I think that the biggest challenge for us in the coming quarter is to more effectively and efficiently.
Justify our reasons for pursuing a particular path as well as refining our nugget from rough ore, to a
highly polished product. I have great faith in my team’s ability to undertake the challenge ahead of us
and to produce the “wow” factor that we are seeking.
This quarter was definitely much busier than last in this course and others. I feel like we have lost and
found our way time and time again, trying things we may not have expected, or being shot down by our
liaison. It’s been a rough journey, but I think we may finally be onto something. There are still some
details to iron out, and an international team to convince but I think we are getting pretty close to the
point where we can commit to a goal for the end of spring quarter.
Winter quarter has been very heavy on prototyping. With the prototypes being much more hardware in
nature with less focus on mechanical, Adam and I found it hard to delegate tasks as the added overhead
of splitting up writing code, or handling the electronics didn’t make sense, even when Caroline and
Jacob asked how they could help. As a result Adam and I took most of the prototyping load. I have to
give thanks to Adam for covering one week for me when I had two midterms in that week, as I can only
imagine how much work it was. Next quarter I hope that we can better handle delegation, which is
something I will focus on, but I think we will be able to better leverage our team’s skills to refine and
manufacture the design in spring quarter.
Working with our NTNU team has also been pretty good. There were some weeks where we did not
have much to talk about in meeting, either because they were busy with other things or we were busy.
As a whole though I think we’ve been pretty decent and keeping up to date and bouncing concepts and
findings off each other. We’ve been pretty fortunate that the interdependence between us in winter
quarter was not heavy as we were diverging and trying different things. I think spring quarter will be
much more challenging in communicating with NTNU as we will have to actively work together to
converge, and not just have two separate, unrelated systems at EXPE.
This quarter has provided several challenges. Because there are no physical interaction between the two
teams it was hard to give proper feedback and throw ball with each other. I think the project would have
benefited greatly from a meeting also this quarter. It has been really fun and challenging to explore and
diverge in so many directions. The fact that we are heading back in the directions of what we were doing
before Christmas is with mixed feelings, but not necessarily a bad thing.
We were lucky to have a vast wealth of sources for materials and manufacturing in our area.
Jameco Electronics – Electrical Components
1355 Shoreway Road
Belmont, CA 94002
Jameco.com
SparkFun.com – Electrical Components
6333 Dry Creek Parkway
Niwot, CO 80503
Sparkfun.com
Alan Steel – Steel, Aluminum
505 E Bayshore Road
Redwood City, CA 94063
Aluminum REM – Recycled Aluminum
3185 De La Cruz Boulevard
Santa Clara, CA 95054
Stanford PRL – Assorted Materials
447 Santa Teresa Street
Stanford, CA 94305
productrealization.stanford.edu
TAP Plastics – Plastics
312 Castro Street
Mountain View, CA 94041
tapplastics.com
Amazon – Various Other Items
amazon.com
Microsoft Store – Kinect
microsoftstore.com
SurveyMonkey – Used to create and track user surveys
Surveymonkey.com
Walmart – Various Other Items
600 Showers Drive
Mountain View, CA 94040
Shad Laws – Renault
Work: Engineer
Wendy Ju – CDR
Work: Executive Director, Interaction Design Research & Associate Professor
Chris Gerdes – DDL
Work: Lab Director, Dynamic Design Lab & Professor in Mechanical Engineering
Sven Beiker - CARS
Work: Executive Director, Center for Automotive Research at Stanford
[1] S. Mlot, "Mercedes' Driverless Concept Car Turns Heads in SF," PC Magazine, 10 March 2015.
[Online]. Available: http://www.pcmag.com/article2/0,2817,2478085,00.asp. [Accessed 3 March
2015].
[2] R. Group, "Renault Group - About Us," Renault Group, 2015. [Online]. Available:
http://group.renault.com/en/our-company/a-group-an-alliance-and-partnerships/. [Accessed 3
March 2015].
[3] Y. Wang, X. Chen and K. Henrickson, "Pedestrian Detection With Microsoft Kinect," 2014. [Online].
Available: http://onlinepubs.trb.org/onlinepubs/conferences/2014/NATMEC/Wang-Kinect-Pres.pdf.
[4] W. Ju, Interviewee, Car/Driver Relationship Design Research. [Interview]. 17 February 2015.
[5] F. Bellott, R. Berta, A. De Gloria and M. Margarone, "Using 3D Sound to Improve the Effectiveness of
the Advanced Driver Assistance Systems," D.I.B.E., Department of Electronics and Biophysical
Engineering, University of Genova, Genova, Italy, pp. 155-163.
[6] T. Carleton and W. Cockayne, "The Power of Prototypes in Foresight Engineering," in Proceedings of
the 17th International Conference on Engineering Design, Stanford, CA, 2009.
[7] G. Gasior, 4 5 2014. [Online]. Available: http://techreport.com/news/24858/updated-kinect-motion-
sensor-coming-to-the-pc-next-year. [Accessed 2 12 2014].
[8] Navdy, "Navdy," [Online]. Available: http://www.navdy.com/. [Accessed 22 11 2014].
[9] "Face Tracker," [Online]. Available:
http://auduno.github.io/clmtrackr/examples/clm_emotiondetection.html. [Accessed 10 2014].
[10] "Gaze Tracking Library," [Online]. Available: http://sourceforge.net/projects/gazetrackinglib/.
[Accessed 10 2014].
[11] [Online]. Available: http://feelspace.cogsci.uni-osnabrueck.de/. [Accessed 10 2014].
[12] M. Brown. [Online]. Available:
http://www.pcworld.com/article/259997/protect_your_assets_a_buying_guide_to_office_security_
systems.html. [Accessed 12 2014].
[13] "The Style," 5 2009. [Online]. Available: http://yuppiecrap.com/irobot-560/. [Accessed 12 2014].
[14] A. Waytz, J. Heafner and N. Epley, "The Mind in the Machine: Anthropomorphism Increases Trust,"
Journal of Experimental Social Psychology, p. 21, 2014.
[15] [Online]. Available: http://www.gearthhacks.com/downloads/related.php?file=31263. [Accessed 12
2014].
[16] "Wendy Ju Design at Large," [Online]. Available:
https://www.flickr.com/photos/calit2/15471345782/. [Accessed 12 2014].
[17] S. Laws. [Online]. Available: https://www.linkedin.com/in/shadlaws. [Accessed 11 2014].
The project prompt submitted to ME310 by Shad Laws on behalf of Renault can be seen below.
As fully autonomous cars for commercial use do not exist yet, we chose to explore a wide variety of
sources to gain as much information about the potential future space. Choosing to focus on the
theoretical aspects of fully autonomous driving allows us to focus on enhancing the dynamic of the car
and driver as a relationship of teammates rather than the logistics of creating a semi- or fully
autonomous vehicle.
As mentioned above, our current definition of ambient is non-explicit communication, more akin to a
sense rather than a conscious intake and analysis of information (e.g. sensing approximately what time
of day it is by the color of the sky or angle of the sun versus reading a clock). In our search for examples
of ambient communication, we found that current technology exists in one of two categories: user-to-
machine communication and machine-to-user communication.
User-to-machine communication focuses on the machine interpreting non-explicit human
communication to provide a better user experience. While difficult for machines, this ability is almost
universal in the human experience, and therefore sometimes subconsciously expected by humans in
things outside of human-to-human interaction. Giving the machine ways to respond to the user’s
physical outputs allows the creation of a more natural-feeling team dynamic.
Gesture Tracking
Gesture tracking pertains to movements of appendages and tracking these to quantify movements.
Currently most gesture technology is only used explicitly but could be of use for ambient
communication. This type of tracking is already being done with both wearable and optical systems;
examples include the video game software Kinect and Navdy (explicit communication), a new method of
communicating with a smart device while driving using gestures.
Figure 24: (Left) Demonstrating the motion tracking of the Kinect [7]
Figure 25: (Right) User gesturing to accept call with Navdy [8]
Emotion Tracking
Emotion tracking software tracks facial expressions and body languages with the intention of quantifying
and translating these movements to give the machine the current emotional state of the user.
Algorithms are currently being researched that use the tracking of key features to provide an
approximation of emotion. An extremely basic version of this technology is clmtrackr, which uses
recognition of facial aspects like eye or mouth shape to hypothesize the displayed emotion.
Figure 26: An example of clmtrackr, analyzing a pleasantly surprised facial expression [9]
Eye Tracking
Using vision and EMG (Electromyography) systems, human eye movement can be quantified and
analyzed. The data may then be used to determine where the user is focusing and frequency of eye
movements. This data can be used to help the machine react to the user’s behavior in a favorable way.
Figure 27: Gazetracker eye monitoring software [10]
Ambient communication from the machine to user is a less prevalent field of research, but we believe
that there are significant opportunities to explore in this area.
Haptic Feedback
Haptic feedback uses tactile sensations to communicate with the user. Current research in this area is
not only focusing on the hardware technology (motors, magnets, and more) but also the applications of
haptic feedback (remote communication, virtual reality); an example of ambient haptic technology is a
belt that participants in a study with one point always vibrating to magnetic north [11]. After several
weeks the participants developed an extra innate sense of where north was without having to be
constantly aware of the vibration.
Benchmarking non-automotive ambient and autonomous technologies provided a good opportunity for
us to learn how these spaces develop--especially useful as semi-autonomous vehicles are in their
infancy, but other spaces have already been through an autonomous adoption period.
Autonomous Consumer Products
There are countless autonomous and semi-autonomous products that are readily available to the
average consumer. Products like the Roomba vacuum cleaner are trusted to adjust to changes in their
environment without directly needing to be informed of them by the user. Automated security systems,
particularly those that can be armed through mobile apps, are likewise trusted to respond to remote
commands. However, the trust implicit in both of these examples occurs from the situation being low-
risk, in the case of the Roomba, or easy to prove that it is working correctly, in the case of security
systems.
Figure 28: An alarm system autonomously monitoring the safety of a house [12]
Figure 29: A Roomba making its rounds, autonomously cleaning [13]
Human-Machine Interactions (HMI)
Studies [14] show that users appreciate interacting with a machine that has human-like traits, such as a
voice, a name, or something that gives it personality; it is easier for people to understand and justify the
behavior of a machine if it is slightly anthropomorphized. In general, being able to see a machine as
comparable to a human makes the user experience better and the machine more trustworthy.
Travel
Although ubiquitously autonomous travel is a very long way off, there are current examples of
technologies that could be applied or be considered analogous to autonomous vehicles. A popular
analogous technology that many automotive companies are focusing on is autopilot for airplanes,
though this example may in some ways be a false analogy, as the time frames and error margins for
autonomous driving are much smaller than those for commercial planes; if there were an error with the
controls in a plane, there could be up to several minutes before the plane would hit the ground, while in
a car there might only be a matter of seconds. Also, while the results are often much more catastrophic
with commercial airlines complications, those events are far less common. Likewise, with the training
pilots receive, if the autopilot failed the human pilots could retake control without disaster. Last, most
commercial passengers have no experience operating a plane, nor much comprehension of the logistics.
As a result, there is a high level of trust in these technologies.
Figure 30: The cockpit of a commercial airliner showing the complexity of the controls [15]
We were lucky to have very knowledgeable experts at our fingertips to talk with early in our research.
These experts helped us explore some technologies that were previously unknown to us, as well as
enlightening us to what they have witnessed in the space.
Wendy Ju – Stanford Center for Design Research (CDR)
As a professor of mechanical engineering, Wendy Ju’s research has focused on the intricacies of the
human experience in automobiles. One of the projects focuses on providing pre-haptic feedback for
turning, and is a mock up of an autonomous car. Another of her projects is measuring how people
respond in a semi-autonomous vehicle simulator. One of the insights that she has shared so far is that
people mistrust a vehicle that disobeys them, even if it is doing so to be safer or to obey the law. They
have also seen that one of the most common responses for someone who is being semi-autonomously
driven in the simulator for more than about eight minutes is to fall asleep. However, while this
automatic reaction could be a major problem for semi-autonomous driving, it would not necessarily be
so for the user of a fully autonomous car.
Figure 31: Wendy Ju of Stanford CDR [16]
We have plans to return and consult with Wendy periodically to get her guidance on our progress.
Shad Laws – Renault
In addition to being our corporate liaison, Shad has provided us with indispensable input in our project;
for example, he steered us towards exploring extreme users of ambient communication between two
living creatures, such as horse riders and people with service dogs. Shad also cautioned us of the “hey
dummy” effect; as described by him, this phenomenon is the tendency for systems to function more as
warnings at an almost insulting level, rather than being an active, positive contributor to the
user/machine relationship, which leads to people choosing not to use them. Knowing this tendency, and
to avoid it, is something that has helped to drive our design choices. Shad also provides extra sources for
expert opinion, such as a talk at the Nissan research facility in Sunnyvale to listen to a talk from a
professional pilot who works on the Google self-driving project.
Figure 32: Shad Laws, our liaison extraordinaire [17]
Although there are no commercial self-driving cars available yet to the public but there are several
precursor technologies that are being used in semi-autonomous cars
Warning Technologies
There are technologies currently on the market that, while helpful to semi-autonomous drivers, are not
in the spirit of what we are seeking to create; they are either overly explicit or employ the ‘hey dummy’
effect that Shad has described. Examples of these types of technology are blind spot detection, and lane
departure warning--functioning more as reminders that you are doing a poor job driving rather than
helping the driver work in harmony with the car.
Teammate Technologies
There are some technologies, however, that are already embodying the teammate-style relationship.
These technologies take a few good first steps towards a teammate style relationship by giving the car
some agency to impact the drive, which gives the driver confidence in the car and in themselves while
driving. Examples of this technology are self-adjusting cruise control and automatic braking. [7]
Through our benchmarking we learned that the critical part of any autonomous device (whether it be a
car, plane, or vacuum,) was trust. Trusting machines is something that is hard for people to do, even
when it can be empirically shown that machines often are better at their tasks than people are. The path
that automotive manufacturers would like to see is displayed in Figure 33. In fact they would like to
minimize the generation 3 where the public’s level of trust in their cars is projected to decrease.
Figure 33: The project path of trust as more autonomy is introduced.
To learn more about this all important trust question, we interviewed a wide range of users both around
Stanford and in Norway. We divided our users into 3 major groups: the general public, current users,
and Extreme Users.
Extreme users give us a new perspective. We chose to interview extreme users that included the
visually impaired, persons with extreme motion sickness, horse riders, and technology junkies. We
found with the people in this group we had vastly different interactions both with cars and with the
people that drive them. For instance those who are very susceptible to motion sickness need constant
awareness of their motion as well as carefully controlled atmosphere in the car to avoid being sick.
Conversely automobile enthusiasts actually enjoy the feeling of driving and as a result would not enjoy
passively sitting in a car.
Severe commuters would enjoy a more connected car that provides more opportunity for productivity
on their long drives, where as the most important thing for a control freak would be to be completely
aware of the situation and be able to have a high level of agency even in an autonomous vehicle. These
users show that especially among the fringe groups the idea of an autonomous vehicle means vastly
different things. For some it is an opportunity and convenient but for others it is terrifying or
detrimental to the experience.
Current users that we interviewed are users that are around autonomous machines. This was an
important area for us to explore. These people have extensive experience with either automation or
with situations where autonomous vehicles may be functioning. An example of this is someone who
routinely is driven around in India. The traffic is such a scary experience there that the person has to
actively distract themselves from observing the road to keep calm. Other examples include pilots and
train conductors. We learned from this group that with increased automation you tend to have less
situational awareness and as a result are less engaged in the activities. This has been the major cause of
many of the recent major airline crashes, the disconnect between the autopilot system and the pilots
leaves holes where accidents can happen. But even with those accidents the introduction of autopilot
has drastically reduced the risk of flying. We hope that this hold true for autonomous vehicles as well.
In general the public has very little conception for autonomous vehicles as they do not exist yet. Most
assume that for them the perfect ride will be the smoothest ride. Most general users also desire
increased productivity while driving while simultaneously not being surprised of the cars actions while
driving.
We found that our instincts with regard to trust really were close to the mark. Some users do not trust
autonomous technology at all such as control freaks or most general users. Some users however trust
autonomous communication too much, such as pilots and horse riders. We think that this gap of trust is
the place where ambient communication can most effectively improve the user and machine
interaction.
We also established the needs of our users. To summarize:
1) The smooth ride is the perfect ride: Passengers trust the driver more and feel safer when the ride is
smooth. If it is rough then they feel the conditions are less safe and have less trust in the situation as a
whole (driver, road conditions, etc).
2) Know the motion of the vehicle: For those who suffer from motion sickness it is important to feel the
car’s motion. As they lose these cues they become more sick due to the disconnect between body
motion and visual motion.
3) Minimal distraction when doing tasks: When passengers are able to do a task in a car (e.g. sleep,
read or browse the internet) they want to be able to do so with minimal distraction. That means they
want less signals such as warnings or rough driving. This need is motivation for a context aware system
that can detect the context of the passenger and adjust accordingly.
4) Minimal Surprises: As a passenger, nobody likes surprises such as the driver pulling out of the lane
randomly. While the action may be necessary it is still unsettling. This is motivation to evaluate
increasing situational awareness of the passenger, potentially giving them foresight about the vehicle’s
imminent actions or surroundings. This may induce better trust in the system with fewer surprises.
5) Don’t be too machine-like or pretend to be too human-like: People don’t like their technology being
cold and impersonal like a machine, but they don’t believe the technology is there yet to create a
convincing Human AI. They feel that current attempts at giving technology human characteristics are
poor and feel unnatural. This means that unless a truly realistic human personality can be achieved, we
should not attempt to make the car act like a person, such as with a voice or personality.
A multitude of potential users we could be designing for exist, divided into two main categories: those
looking for convenience and comfort, and those needing help trusting the technology. Examples of the
convenience and comfort user include, among others, the serial commuter who spends all of his or her
time in the car, the technology geek who must always have the latest and greatest toy, and the car
fanatic. However, while all potentially interesting aspects to build a persona around, we didn’t find them
truly compelling; after all, they are all the types to buy a fully autonomous car no matter how it did or
did not make them feel. Therefore, we chose to focus on the potential user who has a hard time trusting
a fully autonomous car, which was also one of the first things Shad informed us was a major potential
concern. We therefore chose the most, on occasion, irrationally nervous group of which we could think
(mothers) and came up with Nancy, our ideal user (Figure 34), using characteristics we gained from our
interviewees and other real-life sources. If we could make a vehicle that could make her comfortable,
then we presume anyone else would be comfortable in it too.
Figure 34: Nancy, our ideal user for design purposes
This appendix provides more details about the various prototypes completed each quarter. For each
prototype a component list as well as any code that was used to complete the prototyping.
Component List:
7 Vibrating Motors
1 Arduino Mega
1 10k Potentiometer
5V Power Supply
1 Solder Protoboard
7 Transmitters
Assorted Wires
Images of the prototype can be seen below:
Figure 35: Images of Critical Functional Prototype
The microcontroller used for completing this prototype was an Arduino. The code for the control of this
Arduino can be seen below:
// These constants won't change. They're used to give names
// to the pins used:
const int analogInPin = A0; // Analog input pin that the
potentiometer is attached to
int vib7 = 2;
int vib6 = 3;
int vib5 = 4;
int vib4 = 5;
int vib3 = 6;
int vib2 = 7;
int vib1 = 8;
int sensorValue = 0; // value read from the pot
double angle = 0;
void setup() {
// initialize serial communications at 9600 bps:
Serial.begin(9600);
pinMode(vib7, OUTPUT);
pinMode(vib6, OUTPUT);
pinMode(vib5, OUTPUT);
pinMode(vib4, OUTPUT);
pinMode(vib3, OUTPUT);
pinMode(vib2, OUTPUT);
pinMode(vib1, OUTPUT);
digitalWrite(vib7,LOW);
digitalWrite(vib6,LOW);
digitalWrite(vib5,LOW);
digitalWrite(vib4,LOW);
digitalWrite(vib3,LOW);
digitalWrite(vib2,LOW);
digitalWrite(vib1,LOW);
}
void loop() {
// read the analog in value:
sensorValue = analogRead(analogInPin);
angle = 0.2538*(sensorValue - 540);
if (angle < -75)
{
digitalWrite(vib7,HIGH);
digitalWrite(vib6,LOW);
digitalWrite(vib5,LOW);
digitalWrite(vib4,LOW);
digitalWrite(vib3,LOW);
digitalWrite(vib2,LOW);
digitalWrite(vib1,LOW);
} else if (angle < -45)
{
digitalWrite(vib7,LOW);
digitalWrite(vib6,HIGH);
digitalWrite(vib5,LOW);
digitalWrite(vib4,LOW);
digitalWrite(vib3,LOW);
digitalWrite(vib2,LOW);
digitalWrite(vib1,LOW);
} else if (angle < -15)
{
digitalWrite(vib7,LOW);
digitalWrite(vib6,LOW);
digitalWrite(vib5,HIGH);
digitalWrite(vib4,LOW);
digitalWrite(vib3,LOW);
digitalWrite(vib2,LOW);
digitalWrite(vib1,LOW);
} else if (angle < 15)
{
digitalWrite(vib7,LOW);
digitalWrite(vib6,LOW);
digitalWrite(vib5,LOW);
digitalWrite(vib4,HIGH);
digitalWrite(vib3,LOW);
digitalWrite(vib2,LOW);
digitalWrite(vib1,LOW);
} else if (angle < 45)
{
digitalWrite(vib7,LOW);
digitalWrite(vib6,LOW);
digitalWrite(vib5,LOW);
digitalWrite(vib4,LOW);
digitalWrite(vib3,HIGH);
digitalWrite(vib2,LOW);
digitalWrite(vib1,LOW);
} else if (angle < 75)
{
digitalWrite(vib7,LOW);
digitalWrite(vib6,LOW);
digitalWrite(vib5,LOW);
digitalWrite(vib4,LOW);
digitalWrite(vib3,LOW);
digitalWrite(vib2,HIGH);
digitalWrite(vib1,LOW);
} else {
digitalWrite(vib7,LOW);
digitalWrite(vib6,LOW);
digitalWrite(vib5,LOW);
digitalWrite(vib4,LOW);
digitalWrite(vib3,LOW);
digitalWrite(vib2,LOW);
digitalWrite(vib1,HIGH);
}
// print the results to the serial monitor:
Serial.print("sensor = " );
Serial.println(sensorValue);
delay(25);
}
Component List:
1 5.1 Surround Sound System
1 Arduino Uno
1 LSM303DLM Compass Breakout Module
5V Power Supply
1 Car 12V DC to 120V AC Inverter
Assorted Wires
Code
The following is the Arduino sketch that reads compass data and sends it to the computer on the serial
port.
#include <Wire.h>
#include <LSM303.h>
LSM303 compass;
void setup() {
Serial.begin(9600);
Wire.begin();
compass.init();
compass.enableDefault();
//Calibration Values
compass.m_min = (LSM303::vector<int16_t>){-344, -573, -379};
compass.m_max = (LSM303::vector<int16_t>){+136, -33, -202};
}
void loop() {
compass.read();
float heading = compass.heading();
Serial.println((int)heading);
delay(100);
}
The C# code that takes the data and outputs the 3D data information is below.
using System; using System.Collections.Generic; using System.Linq; using System.Text; using IrrKlang; using System.IO.Ports; using System.Threading; using System.Timers; namespace ConsoleApplication2 { class Program { static SerialPort _serialPort = new SerialPort(); static bool _continue = true; static double angle = -1; static double rockingPeriod = 5; //s static bool useSet = false; static System.Timers.Timer timer; static double timeInterval = 1; static double distance = 0.5; const float radius = 5; static void Main(string[] args) { string name; string message; StringComparer stringComparer = StringComparer.OrdinalIgnoreCase; Thread readThread = new Thread(Read); // Allow the user to set the appropriate properties. _serialPort.PortName = "COM11";
_serialPort.BaudRate = 9600; Set the read/write timeouts _serialPort.ReadTimeout = 500; _serialPort.WriteTimeout = 500; _serialPort.Open(); readThread.Start(); Console.WriteLine("Type QUIT to exit"); while (_continue) { message = Console.ReadLine(); if (stringComparer.Equals("quit", message)) { _continue = false; } else if (stringComparer.Equals("s", message)) { useSet = true; } else if (stringComparer.Equals("c", message)) { useSet = false; } else { } if (useSet == true) { try { rockingPeriod = Convert.ToDouble(message); } catch (FormatException) { Console.WriteLine("Format 2"); } } } readThread.Join(); _serialPort.Close(); } public static void Read() { int angleOffset = 167; // start the sound engine with default parameters ISoundEngine engine = new ISoundEngine(); // Now play some sound stream as music in 3d space, looped.
// We play it at position (0,0,0) in 3d space ISound music = engine.Play3D("ctd.mp3", 0, 0, 0, true); if (music != null) music.MinDistance = 5.0f; // Print some help text and start the display loop Console.Out.Write("\nPlaying streamed sound in 3D."); Console.Out.Write("\nPress ESCAPE to quit, any other key to play sound at random position.\n\n"); double posOnCircle = 0; while (_continue) { try { string message = _serialPort.ReadLine(); angle = Convert.ToInt32(message); if (useSet) angle = setAngle; angle = 180-(angleOffset - angle); } catch (TimeoutException) { Console.WriteLine("Timeout"); angle = 0; } catch (FormatException) { angle = 0; } posOnCircle = (double)angle * Math.PI / 180.0; Vector3D pos3d = new Vector3D(radius * (float)Math.Cos(posOnCircle), 0, radius * (float)Math.Sin(posOnCircle)); engine.SetListenerPosition(0, 0, 0, 0, 0, 1); if (music != null) music.Position = pos3d; } } } }
Prototype 1
Component List:
1 Ice Cube Tray
1 Arduino Uno
4 Assorted Scented Wax Cubes
12V Power Supply
6 Alligator Clips
4 10W High Power 10 ohm Resistors
4 RFP50N06 High Power Logic Level Switching MOSFET
1 Low Noise PC Fan
Assorted Wires
An image of the prototype:
Figure 36: Image of Dark Horse Wax Prototype
Prototype Code
// give it a name:
int scent1 = A0;
int scent2 = A1;
int current_scent = 0;
int num_scents = 2;
// the setup routine runs once when you press reset:
void setup() {
// initialize the digital pin as an output.
pinMode(scent1, OUTPUT);
pinMode(scent2, OUTPUT);
}
// the loop routine runs over and over again forever:
void loop() {
current_scent = current_scent + 1;
if (current_scent == num_scents+1)
{
current_scent = 1;
}
if (current_scent == 1)
{
digitalWrite(scent1,HIGH);
digitalWrite(scent2,LOW);
} else if (current_scent == 2)
{
digitalWrite(scent1,LOW);
digitalWrite(scent2,HIGH);
}
for (int i = 1; i < 60*20; i++)
{
delay(1000);
}
}
Prototype 2
Component List:
1 3 x 3 Muffin Tray
1 Arduino Uno
12V Power Supply
18 Alligator Clips
9 Solenoid Valves
9 RFP50N06 High Power Logic Level Switching MOSFET
9 Assorted Scented Oils
1 Air Mattress Pump
3ft, 5mm ID, 8mm OD PVC Tubing
10”x10”x1/4” Acrylic Sheet
Assorted Wires
An image of the prototype:
Figure 37: Image of Dark Horse Liquid Scent Release Prototype
Prototype Code
#include "Timers.h"
#define SCENT1 2
#define SCENT2 3
#define SCENT3 4
#define SCENT4 5
#define SCENT5 6
#define SCENT6 7
#define SCENT7 8
#define SCENT8 9
int scent_number = -1;
int time_open = -1;
int scent_to_close = 0;
// the setup routine runs once when you press reset:
void setup() {
// initialize the digital pin as an output.
pinMode(SCENT1,OUTPUT);
pinMode(SCENT2,OUTPUT);
pinMode(SCENT3,OUTPUT);
pinMode(SCENT4,OUTPUT);
pinMode(SCENT5,OUTPUT);
pinMode(SCENT6,OUTPUT);
pinMode(SCENT7,OUTPUT);
pinMode(SCENT8,OUTPUT);
Serial.begin(9600);
Serial.print("Enter Scent #: ");
}
// the loop routine runs over and over again forever:
void loop() {
if (scent_number == -1)
{
if (Serial.available())
{
scent_number = Serial.parseInt();
Serial.println(scent_number);
Serial.print("Open for how long [mins] (0 = no time): ");
}
} else if (time_open == -1)
{
if (Serial.available())
{
time_open = Serial.parseInt();
Serial.println(time_open);
}
}
if (time_open != -1 && time_open != -2 && time_open != -3)
{
digitalWrite(scent_number+1,HIGH);
if (time_open > 0)
{
TMRArd_InitTimer(0, (long)time_open*60*1000);
scent_to_close = scent_number;
time_open = -2;
} else {
Serial.print("Enter 1 to close scent: ");
scent_to_close = scent_number;
time_open = -3;
}
}
if (time_open == -3)
{
if (Serial.available())
{
if (Serial.parseInt() == 1)
{
Serial.println("Closing");
digitalWrite(scent_to_close+1,LOW);
Serial.print("Enter Scent #: ");
scent_number = -1;
scent_to_close = 0;
time_open = -1;
}
}
} else if (TMRArd_IsTimerExpired(0) == 1 && scent_to_close != 0)
{
digitalWrite(scent_to_close+1,LOW);
Serial.print("Closing Scent ");
Serial.println(scent_to_close);
scent_to_close = 0;
Serial.print("Enter Scent #: ");
scent_number = -1;
time_open = -1;
}
}
Wizard of Oz Prototype
Component List:
12V Power Supply
1 PC Fan
4 Assorted Scented Oils
1 Car 12V DC to 120V AC Inverter
PC to track map and results
GPS to track location
Progress Bar Code
/*
Blink
Turns on an LED on for one second, then off for one second,
repeatedly.
This example code is in the public domain.
*/
// Pin 13 has an LED connected on most Arduino boards.
// give it a name:
#define RED 7
#define YELLOW 6
#define GREEN1 5
#define GREEN2 4
#define BLUE1 3
#define BLUE2 2
int current_zone;
// the setup routine runs once when you press reset:
void setup() {
// initialize the digital pin as an output.
pinMode(RED,OUTPUT);
pinMode(YELLOW,OUTPUT);
pinMode(GREEN1,OUTPUT);
pinMode(GREEN2,OUTPUT);
pinMode(BLUE1,OUTPUT);
pinMode(BLUE2,OUTPUT);
digitalWrite(RED,HIGH);
digitalWrite(YELLOW,HIGH);
digitalWrite(GREEN1,HIGH);
digitalWrite(GREEN2,HIGH);
digitalWrite(BLUE1,HIGH);
digitalWrite(BLUE2,HIGH);
Serial.begin(9600);
Serial.print("Enter Command [0 - back, 1 - forward]: ");
current_zone = 0;
}
// the loop routine runs over and over again forever:
void loop() {
if(Serial.available())
{
int input = Serial.parseInt();
Serial.println(input);
Serial.print("Enter Command [0 - back, 1 - forward]: ");
if (input == 0)
{
current_zone = current_zone - 1;
if (current_zone == -1)
{
current_zone = 0;
}
} else {
current_zone = current_zone + 1;
if (current_zone == 5)
{
current_zone = 4;
}
}
if (current_zone == 0)
{
digitalWrite(RED,HIGH);
digitalWrite(YELLOW,HIGH);
digitalWrite(GREEN1,HIGH);
digitalWrite(GREEN2,HIGH);
digitalWrite(BLUE1,HIGH);
digitalWrite(BLUE2,HIGH);
} else if (current_zone == 1)
{
digitalWrite(RED,HIGH);
digitalWrite(YELLOW,HIGH);
digitalWrite(GREEN1,HIGH);
digitalWrite(GREEN2,HIGH);
digitalWrite(BLUE1,LOW);
digitalWrite(BLUE2,LOW);
} else if (current_zone == 2)
{
digitalWrite(RED,HIGH);
digitalWrite(YELLOW,HIGH);
digitalWrite(GREEN1,LOW);
digitalWrite(GREEN2,LOW);
digitalWrite(BLUE1,LOW);
digitalWrite(BLUE2,LOW);
} else if (current_zone == 3)
{
digitalWrite(RED,HIGH);
digitalWrite(YELLOW,LOW);
digitalWrite(GREEN1,LOW);
digitalWrite(GREEN2,LOW);
digitalWrite(BLUE1,LOW);
digitalWrite(BLUE2,LOW);
} else if (current_zone == 4)
{
digitalWrite(RED,LOW);
digitalWrite(YELLOW,LOW);
digitalWrite(GREEN1,LOW);
digitalWrite(GREEN2,LOW);
digitalWrite(BLUE1,LOW);
digitalWrite(BLUE2,LOW);
}
}
}
Component List:
0.5m Neopixel LED strip x 2
Accelerometer - MPU6050
Vibrating Motors x 3
5V – 5A Power Supply
Arduino Uno
Potentiometer x3
Assorted Connectors
Assorted Wires
Power MOSFETS x3
470uF Bulk Capacitor
Images of this prototype can be seen below:
Figure 38: Photos of Funktional Prototype
Once again an Arduino was used to control this system, the code for both prototypes can be seen below.
Potentiometer Controlled Prototype:
// NeoPixel Ring simple sketch (c) 2013 Shae Erisson
// released under the GPLv3 license to match the rest of the AdaFruit
NeoPixel library
#include <Adafruit_NeoPixel.h>
#include <avr/power.h>
// Which pin on the Arduino is connected to the NeoPixels?
// On a Trinket or Gemma we suggest changing this to 1
#define SPEED_PIN 7
#define TURN_PIN 6
#define L_VIB 5
#define C_VIB 4
#define R_VIB 3
#define SPEED_START 8
#define SPEED_END 22
#define ACCEL_END 7
#define ACCEL_START 23
#define TURN_NUM_PIXELS 3
// How many NeoPixels are attached to the Arduino?
#define NUMPIXELS 30
#define LED_MAXVAL 150
#define VIB_TIME_ON 100
#define VIB_TIME_OFF 200
Adafruit_NeoPixel pixelsSpeed = Adafruit_NeoPixel(NUMPIXELS,
SPEED_PIN, NEO_GRB + NEO_KHZ800);
Adafruit_NeoPixel pixelsTurn = Adafruit_NeoPixel(NUMPIXELS, TURN_PIN,
NEO_GRB + NEO_KHZ800);
int flasher = 0;
int flasher_sub = 1;
int counter = 0;
int counter_turn = 0;
int counter_accel = 0;
#define TURN_BUF_SIZE 5
int turn_buffer[TURN_BUF_SIZE];
int turn_buffer_index = 0;
int turn_flasher = 0;
int turn_flasher_sub = 1;
void setup() {
// End of trinket special code
Serial.begin(9600);
pixelsSpeed.begin(); // This initializes the NeoPixel library.
pixelsTurn.begin();
for (int i=0;i<NUMPIXELS;i++)
{
pixelsSpeed.setPixelColor(i,pixelsSpeed.Color(0,0,LED_MAXVAL));
pixelsTurn.setPixelColor(i,pixelsTurn.Color(0,0,0));
}
pixelsSpeed.show();
pixelsTurn.show();
pinMode(5,OUTPUT);
pinMode(4,OUTPUT);
pinMode(3,OUTPUT);
for (int i = 0; i < TURN_BUF_SIZE; i++)
{
turn_buffer[i] = analogRead(A2);
}
}
void loop() {
///// LED CODE ////
int speed_val = analogRead(A1);
int turn_val = analogRead(A2);
int accel_val = analogRead(A3);
turn_buffer[turn_buffer_index] = turn_val;
turn_buffer_index++;
if (turn_buffer_index == TURN_BUF_SIZE)
{
turn_buffer_index = 0;
}
turn_val = 0;
for (int i = 0; i<TURN_BUF_SIZE;i++)
{
turn_val = turn_val + turn_buffer[i];
}
turn_val = turn_val / float(TURN_BUF_SIZE);
speed_val = 1023 - speed_val;
turn_val = 1023 - turn_val;
accel_val = 1023 - accel_val;
//////// TURNING CODE //////////
int centre_pixel = turn_val/1023.0*30.0;
int yellow = 0;
if (turn_val > 923)
{
digitalWrite(L_VIB,HIGH);
if (counter_turn > VIB_TIME_ON + VIB_TIME_OFF)
{
counter_turn = 0;
}
if (turn_flasher_sub == 1)
{
yellow = LED_MAXVAL - turn_flasher;
turn_flasher++;
if (turn_flasher == LED_MAXVAL-15)
{
turn_flasher_sub = 0;
}
} else {
yellow = LED_MAXVAL - turn_flasher;
turn_flasher--;
if (turn_flasher == 15)
{
turn_flasher_sub = 1;
}
}
} else if (turn_val < 100)
{
digitalWrite(R_VIB,HIGH);
if (counter_turn > VIB_TIME_ON + VIB_TIME_OFF)
{
counter_turn = 0;
}
if (turn_flasher_sub == 1)
{
yellow = LED_MAXVAL - turn_flasher;
turn_flasher++;
if (turn_flasher == LED_MAXVAL-15)
{
turn_flasher_sub = 0;
}
} else {
yellow = LED_MAXVAL - turn_flasher;
turn_flasher--;
if (turn_flasher == 15)
{
turn_flasher_sub = 1;
}
}
} else {
yellow = LED_MAXVAL;
}
counter_turn++;
if (counter_turn > VIB_TIME_ON)
{
digitalWrite(L_VIB,LOW);
digitalWrite(R_VIB,LOW);
}
if (centre_pixel < TURN_NUM_PIXELS/2)
{
centre_pixel = TURN_NUM_PIXELS/2;
} else if(centre_pixel > NUMPIXELS - TURN_NUM_PIXELS/2-1)
{
centre_pixel = NUMPIXELS - TURN_NUM_PIXELS/2 - 1;
}
for (int i=0;i<NUMPIXELS;i++)
{
pixelsTurn.setPixelColor(i,pixelsTurn.Color(0,0,0));
if ((i-1) == centre_pixel || (i+1) == centre_pixel || i ==
centre_pixel)
{
pixelsTurn.setPixelColor(i,pixelsTurn.Color(yellow, yellow, 0));
}
}
///// CURRENT SPEED CODE ////////
int red;
int blue;
int green;
if (speed_val < 256){
red = 255;
blue = speed_val;
green = 0;
} else if (speed_val < 512) {
red = 512 - speed_val;
blue = 255;
green = 0;
} else if (speed_val < 768)
{
red = 0;
blue = 255;
green = speed_val - 512;
} else {
red = 0;
blue = 1023-speed_val;
green = 255;
}
for (int i=SPEED_START; i<SPEED_END+1;i++)
{
pixelsSpeed.setPixelColor(i,pixelsSpeed.Color(red,green,blue));
}
//// ACCEL CODE //////
int a_red;
int a_blue;
int a_green;
if (accel_val < 170) {
digitalWrite(C_VIB,HIGH);
if (counter_accel > VIB_TIME_ON + VIB_TIME_OFF)
{
counter_accel = 0;
}
if (flasher_sub == 1)
{
a_red = 255 - flasher;
flasher++;
if (flasher == 240)
{
flasher_sub = 0;
}
} else {
a_red = 255 - flasher;
flasher--;
if (flasher == 15)
{
flasher_sub = 1;
}
}
a_green = 0;
a_blue = 0;
} else if (accel_val < 340){
a_red = 255;
a_blue = (accel_val-170)/170.0*256.0;
a_green = 0;
} else if (accel_val < 510) {
a_red = 255 - (accel_val-340)/170.0*256.0-1;
a_blue = 255;
a_green = 0;
} else if (accel_val < 680)
{
a_red = 0;
a_blue = 255;
a_green = (accel_val-510)/170.0*256.0;
} else if (accel_val < 850){
a_red = 0;
a_blue = 255-(accel_val-850)/170.0*256.0-1;
a_green = 255;
} else {
digitalWrite(C_VIB,HIGH);
if (counter_accel > VIB_TIME_ON + VIB_TIME_OFF)
{
counter_accel = 0;
}
if (flasher_sub == 1)
{
a_green = 255 - flasher;
flasher++;
if (flasher == 240)
{
flasher_sub = 0;
}
} else {
a_green = 255 - flasher;
flasher--;
if (flasher == 15)
{
flasher_sub = 1;
}
}
a_red = 0;
a_blue = 0;
}
counter_accel++;
if (counter_accel > VIB_TIME_ON)
{
digitalWrite(C_VIB,LOW);
}
for (int i=0; i<ACCEL_END+1;i++)
{
pixelsSpeed.setPixelColor(i,pixelsSpeed.Color(a_red,a_green,a_blue));
}
for (int i=ACCEL_START; i < NUMPIXELS; i++)
{
pixelsSpeed.setPixelColor(i,pixelsSpeed.Color(a_red,a_green,a_blue));
}
//// Show Pixels ////
pixelsSpeed.show();
pixelsTurn.show();
delay(5);
}
Accelerometer Prototype:
// NeoPixel Ring simple sketch (c) 2013 Shae Erisson
// released under the GPLv3 license to match the rest of the AdaFruit
NeoPixel library
#include <Adafruit_NeoPixel.h>
#include <avr/power.h>
#include "I2Cdev.h"
#include "MPU6050.h"
// Arduino Wire library is required if I2Cdev I2CDEV_ARDUINO_WIRE
implementation
// is used in I2Cdev.h
#if I2CDEV_IMPLEMENTATION == I2CDEV_ARDUINO_WIRE
#include "Wire.h"
#endif
// class default I2C address is 0x68
// specific I2C addresses may be passed as a parameter here
// AD0 low = 0x68 (default for InvenSense evaluation board)
// AD0 high = 0x69
MPU6050 accelgyro;
int16_t ax1, ay1, az1;
int16_t gx1, gy1, gz1;
#define OUTPUT_READABLE_ACCELGYRO
// Which pin on the Arduino is connected to the NeoPixels?
// On a Trinket or Gemma we suggest changing this to 1
#define ACCEL_PIN 7
#define GYRO_PIN 6
#define L_VIB 5
#define C_VIB 4
#define R_VIB 3
#define TURN_NUM_PIXELS 3
// How many NeoPixels are attached to the Arduino?
#define NUMPIXELS 30
#define LED_MAXVAL 100
#define VIB_TIME_ON 1000
#define VIB_TIME_OFF 2000
#define MAX_ACCEL_VAL 100
#define MAX_GYRO_VAL 100
#define BUF_SIZE 50
#define P_ACCEL 3000
#define N_ACCEL -4000
#define A_OFFSET -3600
Adafruit_NeoPixel pixelsAccel = Adafruit_NeoPixel(NUMPIXELS,
ACCEL_PIN, NEO_GRB + NEO_KHZ800);
Adafruit_NeoPixel pixelsGyro = Adafruit_NeoPixel(NUMPIXELS, GYRO_PIN,
NEO_GRB + NEO_KHZ800);
int accel_vib_state = 0;
long prev_accel_vib_time = 0;
int gravity = 15515;
void setup() {
#if I2CDEV_IMPLEMENTATION == I2CDEV_ARDUINO_WIRE
Wire.begin();
#elif I2CDEV_IMPLEMENTATION == I2CDEV_BUILTIN_FASTWIRE
Fastwire::setup(400, true);
#endif
// End of trinket special code
Serial.begin(9600);
Serial.println("Initializing I2C devices...");
accelgyro.initialize();
Serial.println("Testing device connections...");
Serial.println(accelgyro.testConnection() ? "MPU6050_1 connection
successful" : "MPU6050_1 connection failed");
pixelsAccel.begin(); // This initializes the NeoPixel library.
pixelsGyro.begin();
pinMode(L_VIB,OUTPUT);
pinMode(R_VIB,OUTPUT);
pinMode(C_VIB,OUTPUT);
Serial.println("a_avg,led_val,a_red,a_green,a_blue");
}
void loop() {
// Get average accel and gyro data //
float a_avg = 0;
float g_avg = 0;
for (int i = 0; i < BUF_SIZE; i++)
{
accelgyro.getMotion6(&ax1, &ay1, &az1, &gx1, &gy1, &gz1);
delay(5);
a_avg = a_avg + ax1;
g_avg = g_avg + sqrt(gx1*gx1+gy1*gy1+gz1*gz1);
}
a_avg = a_avg/float(BUF_SIZE);
g_avg = g_avg/float(BUF_SIZE);
// LED ACCEL CODE //
int a_led_val = 0;
if (a_avg > 0)
{
a_led_val = 512.0*(a_avg-A_OFFSET)/float(P_ACCEL)+511;
} else if (a_avg < 0)
{
a_led_val = 512-(a_avg-A_OFFSET)/float(N_ACCEL)*512.0;
}
if (a_led_val > 1023)
{
a_led_val = 1023;
} else if (a_led_val < 0)
{
a_led_val = 0;
}
int a_red;
int a_blue;
int a_green;
if (a_led_val < 256){
a_red = 255;
a_blue = a_led_val;
a_green = 0;
} else if (a_led_val < 512) {
a_red = 512 - a_led_val;
a_blue = 255;
a_green = 0;
} else if (a_led_val < 768)
{
a_red = 0;
a_blue = 255;
a_green = a_led_val - 512;
} else {
a_red = 0;
a_blue = 1023-a_led_val;
a_green = 255;
}
for (int i=0; i<NUMPIXELS;i++)
{
pixelsAccel.setPixelColor(i,pixelsAccel.Color(a_red,a_green,a_blue));
}
pixelsAccel.show();
pixelsGyro.show();
// ACCEL VIB CODE //
if ((a_avg-A_OFFSET) > P_ACCEL || (a_avg-A_OFFSET) < N_ACCEL)
{
delay(3000);
if (accel_vib_state == 0 && millis() - prev_accel_vib_time >
VIB_TIME_OFF)
{
digitalWrite(C_VIB,HIGH);
prev_accel_vib_time = millis();
accel_vib_state = 1;
} else if (accel_vib_state == 1 && millis() - prev_accel_vib_time
> VIB_TIME_ON)
{
digitalWrite(C_VIB,LOW);
prev_accel_vib_time = millis();
accel_vib_state = 0;
}
} else {
digitalWrite(C_VIB,LOW);
prev_accel_vib_time = 0;
accel_vib_state = 0;
}
// GYRO LED CODE //
Serial.print(a_avg);Serial.print(", ");
Serial.print(a_led_val);Serial.print(", ");
Serial.print(a_red);Serial.print(", ");
Serial.print(a_green);Serial.print(", ");
Serial.println(a_blue);
}
Component List:
Kinect v2 Sensor
Cables to allow for Windows Utility
2m Neopixel LED strip
Arduino Uno
5V-5A Power Supply
Laptop running Windows and Kinect SDK
Mounting Materials – Cardboard, Tape
LED Diffuser – Paper Towel
USB Cable
Assorted Wires
Assorted Connectors
Some images of this prototype can be found earlier in this report. More photos can be seen below.
Figure 39: Photos of Functional Prototype including Kinect Data and User Testing
Once again an Arduino was used to control the LED strips. Code for this Arduino can be seen below. The
Arduino code was required to read the serial port as well as driving the LEDs.
#include <Adafruit_NeoPixel.h>
#include <avr/power.h>
#define PIN 8
// How many NeoPixels are attached to the Arduino?
#define NUMPIXELS 120
#define MAX_LED 100
#define NUMFRAMES 5
Adafruit_NeoPixel pixels = Adafruit_NeoPixel(NUMPIXELS, PIN, NEO_GRB +
NEO_KHZ800);
char pixel_array[NUMPIXELS][NUMFRAMES];
char pixel_sum[NUMPIXELS];
int pixel_index = -1;
// the setup routine runs once when you press reset:
void setup() {
// initialize serial communication at 9600 bits per second:
Serial.begin(115200);
// make the pushbutton's pin an input:
pixels.begin();
for(int i=0;i<NUMPIXELS;i++){
// pixels.Color takes RGB values, from 0,0,0 up to 255,255,255
pixels.setPixelColor(i, pixels.Color(0,0,0)); // Moderately bright
green color.
}
pixels.show(); // This sends the updated pixel color to the hardware.
while (1)
{
char input;
if (Serial.available())
{
input = Serial.read();
if (input == 'R' || input == 82)
{
pixel_index++;
if (pixel_index >= NUMFRAMES)
{
break;
}
for (int i = 0; i < NUMPIXELS; i++)
{
pixel_array[i][pixel_index] = 0;
}
} else if (input == 'f' || input == 102)
{
while (!Serial.available()) {}
int led_num = Serial.parseInt();
if (pixel_index != -1 && led_num >= 0 && led_num < NUMPIXELS)
{
pixel_array[led_num][pixel_index] = 1;
}
}
}
}
for (int i = 0; i < NUMPIXELS; i++)
{
pixel_sum[i] = 0;
for (int j = 0; j < NUMFRAMES; j++)
{
pixel_sum[i] = pixel_sum[i] + pixel_array[i][j];
}
}
pixel_index = -1;
}
// the loop routine runs over and over again forever:
void loop() {
// read the input pin:
char input;
if (Serial.available())
{
input = Serial.read();
if (input == 'R' || input == 82)
{
pixel_index++;
if (pixel_index >= NUMFRAMES)
{
pixel_index = 0;
}
for (int i = 0; i < NUMPIXELS; i++)
{
if (pixel_sum[i] > 3)
{
pixels.setPixelColor(i,pixels.Color(MAX_LED/2,MAX_LED/2,0));
} else {
pixels.setPixelColor(i,pixels.Color(0,0,0));
}
pixel_sum[i] = pixel_sum[i] - pixel_array[i][pixel_index];
pixel_array[i][pixel_index] = 0;
}
pixels.show();
} else if (input == 'f' || input == 102)
{
while(!Serial.available()) {}
int led_num = Serial.parseInt();
if (pixel_index != -1 && led_num >= 0 && led_num < NUMPIXELS)
{
pixel_array[led_num][pixel_index] = 1;
pixel_sum[led_num] = pixel_sum[led_num] + 1;
}
}
}
}
In order to complete this prototype it was also required to hack into the Kinect SDK. This allowed the
system to send information to the Arduino via a COM port. The code that accomplished this can be seen
below.
//--------------------------------------------------------------------
----------
// <copyright file="DepthBasics.cpp" company="Microsoft">
// Copyright (c) Microsoft Corporation. All rights reserved.
// </copyright>
//--------------------------------------------------------------------
----------
#include "stdafx.h"
#include <strsafe.h>
#include "resource.h"
#include "DepthBasics.h"
#include "SerialClass.h"
#include <string>
#include <stdio.h>
Serial* SP;
/// <summary>
/// Entry point for the application
/// </summary>
/// <param name="hInstance">handle to the application instance</param>
/// <param name="hPrevInstance">always 0</param>
/// <param name="lpCmdLine">command line arguments</param>
/// <param name="nCmdShow">whether to display minimized, maximized, or
normally</param>
/// <returns>status</returns>
int APIENTRY wWinMain(
_In_ HINSTANCE hInstance,
_In_opt_ HINSTANCE hPrevInstance,
_In_ LPWSTR lpCmdLine,
_In_ int nShowCmd
)
{
UNREFERENCED_PARAMETER(hPrevInstance);
UNREFERENCED_PARAMETER(lpCmdLine);
SP = new Serial("\\\\.\\COM15");
while (!SP->IsConnected())
{
SP = NULL;
SP = new Serial("\\\\.\\COM15");
OutputDebugStringA("We're Not Connected\n");
}
OutputDebugStringA("We're Connected");
CDepthBasics application;
application.Run(hInstance, nShowCmd);
}
/// <summary>
/// Constructor
/// </summary>
CDepthBasics::CDepthBasics() :
m_hWnd(NULL),
m_nStartTime(0),
m_nLastCounter(0),
m_nFramesSinceUpdate(0),
m_fFreq(0),
m_nNextStatusTime(0LL),
m_bSaveScreenshot(false),
m_pKinectSensor(NULL),
m_pDepthFrameReader(NULL),
m_pD2DFactory(NULL),
m_pDrawDepth(NULL),
m_pDepthRGBX(NULL)
{
LARGE_INTEGER qpf = {0};
if (QueryPerformanceFrequency(&qpf))
{
m_fFreq = double(qpf.QuadPart);
}
// create heap storage for depth pixel data in RGBX format
m_pDepthRGBX = new RGBQUAD[cDepthWidth * cDepthHeight];
}
/// <summary>
/// Destructor
/// </summary>
CDepthBasics::~CDepthBasics()
{
// clean up Direct2D renderer
if (m_pDrawDepth)
{
delete m_pDrawDepth;
m_pDrawDepth = NULL;
}
if (m_pDepthRGBX)
{
delete [] m_pDepthRGBX;
m_pDepthRGBX = NULL;
}
// clean up Direct2D
SafeRelease(m_pD2DFactory);
// done with depth frame reader
SafeRelease(m_pDepthFrameReader);
// close the Kinect Sensor
if (m_pKinectSensor)
{
m_pKinectSensor->Close();
}
SafeRelease(m_pKinectSensor);
}
/// <summary>
/// Creates the main window and begins processing
/// </summary>
/// <param name="hInstance">handle to the application instance</param>
/// <param name="nCmdShow">whether to display minimized, maximized, or
normally</param>
int CDepthBasics::Run(HINSTANCE hInstance, int nCmdShow)
{
MSG msg = {0};
WNDCLASS wc;
// Dialog custom window class
ZeroMemory(&wc, sizeof(wc));
wc.style = CS_HREDRAW | CS_VREDRAW;
wc.cbWndExtra = DLGWINDOWEXTRA;
wc.hCursor = LoadCursorW(NULL, IDC_ARROW);
wc.hIcon = LoadIconW(hInstance, MAKEINTRESOURCE(IDI_APP));
wc.lpfnWndProc = DefDlgProcW;
wc.lpszClassName = L"DepthBasicsAppDlgWndClass";
if (!RegisterClassW(&wc))
{
return 0;
}
// Create main application window
HWND hWndApp = CreateDialogParamW(
NULL,
MAKEINTRESOURCE(IDD_APP),
NULL,
(DLGPROC)CDepthBasics::MessageRouter,
reinterpret_cast<LPARAM>(this));
// Show window
ShowWindow(hWndApp, nCmdShow);
// Main message loop
while (WM_QUIT != msg.message)
{
Update();
while (PeekMessageW(&msg, NULL, 0, 0, PM_REMOVE))
{
// If a dialog message will be taken care of by the dialog
proc
if (hWndApp && IsDialogMessageW(hWndApp, &msg))
{
continue;
}
TranslateMessage(&msg);
DispatchMessageW(&msg);
}
}
return static_cast<int>(msg.wParam);
}
/// <summary>
/// Main processing function
/// </summary>
void CDepthBasics::Update()
{
if (!m_pDepthFrameReader)
{
return;
}
IDepthFrame* pDepthFrame = NULL;
HRESULT hr = m_pDepthFrameReader-
>AcquireLatestFrame(&pDepthFrame);
if (SUCCEEDED(hr))
{
INT64 nTime = 0;
IFrameDescription* pFrameDescription = NULL;
int nWidth = 0;
int nHeight = 0;
USHORT nDepthMinReliableDistance = 0;
USHORT nDepthMaxDistance = 0;
UINT nBufferSize = 0;
UINT16 *pBuffer = NULL;
hr = pDepthFrame->get_RelativeTime(&nTime);
if (SUCCEEDED(hr))
{
hr = pDepthFrame-
>get_FrameDescription(&pFrameDescription);
}
if (SUCCEEDED(hr))
{
hr = pFrameDescription->get_Width(&nWidth);
}
if (SUCCEEDED(hr))
{
hr = pFrameDescription->get_Height(&nHeight);
}
if (SUCCEEDED(hr))
{
hr = pDepthFrame-
>get_DepthMinReliableDistance(&nDepthMinReliableDistance);
}
if (SUCCEEDED(hr))
{
// In order to see the full range of depth (including
the less reliable far field depth)
// we are setting nDepthMaxDistance to the extreme
potential depth threshold
nDepthMaxDistance = USHRT_MAX;
// Note: If you wish to filter by reliable depth
distance, uncomment the following line.
//// hr = pDepthFrame-
>get_DepthMaxReliableDistance(&nDepthMaxDistance);
}
if (SUCCEEDED(hr))
{
hr = pDepthFrame->AccessUnderlyingBuffer(&nBufferSize,
&pBuffer);
}
if (SUCCEEDED(hr))
{
ProcessDepth(nTime, pBuffer, nWidth, nHeight,
nDepthMinReliableDistance, nDepthMaxDistance);
}
SafeRelease(pFrameDescription);
}
SafeRelease(pDepthFrame);
}
/// <summary>
/// Handles window messages, passes most to the class instance to
handle
/// </summary>
/// <param name="hWnd">window message is for</param>
/// <param name="uMsg">message</param>
/// <param name="wParam">message data</param>
/// <param name="lParam">additional message data</param>
/// <returns>result of message processing</returns>
LRESULT CALLBACK CDepthBasics::MessageRouter(HWND hWnd, UINT uMsg,
WPARAM wParam, LPARAM lParam)
{
CDepthBasics* pThis = NULL;
if (WM_INITDIALOG == uMsg)
{
pThis = reinterpret_cast<CDepthBasics*>(lParam);
SetWindowLongPtr(hWnd, GWLP_USERDATA,
reinterpret_cast<LONG_PTR>(pThis));
}
else
{
pThis =
reinterpret_cast<CDepthBasics*>(::GetWindowLongPtr(hWnd,
GWLP_USERDATA));
}
if (pThis)
{
return pThis->DlgProc(hWnd, uMsg, wParam, lParam);
}
return 0;
}
/// <summary>
/// Handle windows messages for the class instance
/// </summary>
/// <param name="hWnd">window message is for</param>
/// <param name="uMsg">message</param>
/// <param name="wParam">message data</param>
/// <param name="lParam">additional message data</param>
/// <returns>result of message processing</returns>
LRESULT CALLBACK CDepthBasics::DlgProc(HWND hWnd, UINT message, WPARAM
wParam, LPARAM lParam)
{
UNREFERENCED_PARAMETER(wParam);
UNREFERENCED_PARAMETER(lParam);
switch (message)
{
case WM_INITDIALOG:
{
// Bind application window handle
m_hWnd = hWnd;
// Init Direct2D
D2D1CreateFactory(D2D1_FACTORY_TYPE_SINGLE_THREADED,
&m_pD2DFactory);
// Create and initialize a new Direct2D image renderer
(take a look at ImageRenderer.h)
// We'll use this to draw the data we receive from the
Kinect to the screen
m_pDrawDepth = new ImageRenderer();
HRESULT hr = m_pDrawDepth->Initialize(GetDlgItem(m_hWnd,
IDC_VIDEOVIEW), m_pD2DFactory, cDepthWidth, cDepthHeight, cDepthWidth
* sizeof(RGBQUAD));
if (FAILED(hr))
{
SetStatusMessage(L"Failed to initialize the Direct2D
draw device.", 10000, true);
}
// Get and initialize the default Kinect sensor
InitializeDefaultSensor();
}
break;
// If the titlebar X is clicked, destroy app
case WM_CLOSE:
DestroyWindow(hWnd);
break;
case WM_DESTROY:
// Quit the main message pump
PostQuitMessage(0);
break;
// Handle button press
case WM_COMMAND:
// If it was for the screenshot control and a button
clicked event, save a screenshot next frame
if (IDC_BUTTON_SCREENSHOT == LOWORD(wParam) && BN_CLICKED
== HIWORD(wParam))
{
m_bSaveScreenshot = true;
}
break;
}
return FALSE;
}
/// <summary>
/// Initializes the default Kinect sensor
/// </summary>
/// <returns>indicates success or failure</returns>
HRESULT CDepthBasics::InitializeDefaultSensor()
{
HRESULT hr;
hr = GetDefaultKinectSensor(&m_pKinectSensor);
if (FAILED(hr))
{
return hr;
}
if (m_pKinectSensor)
{
// Initialize the Kinect and get the depth reader
IDepthFrameSource* pDepthFrameSource = NULL;
hr = m_pKinectSensor->Open();
if (SUCCEEDED(hr))
{
hr = m_pKinectSensor-
>get_DepthFrameSource(&pDepthFrameSource);
}
if (SUCCEEDED(hr))
{
hr = pDepthFrameSource->OpenReader(&m_pDepthFrameReader);
}
SafeRelease(pDepthFrameSource);
}
if (!m_pKinectSensor || FAILED(hr))
{
SetStatusMessage(L"No ready Kinect found!", 10000, true);
return E_FAIL;
}
return hr;
}
/// <summary>
/// Handle new depth data
/// <param name="nTime">timestamp of frame</param>
/// <param name="pBuffer">pointer to frame data</param>
/// <param name="nWidth">width (in pixels) of input image data</param>
/// <param name="nHeight">height (in pixels) of input image
data</param>
/// <param name="nMinDepth">minimum reliable depth</param>
/// <param name="nMaxDepth">maximum reliable depth</param>
/// </summary>
void CDepthBasics::ProcessDepth(INT64 nTime, const UINT16* pBuffer,
int nWidth, int nHeight, USHORT nMinDepth, USHORT nMaxDepth)
{
if (m_hWnd)
{
if (!m_nStartTime)
{
m_nStartTime = nTime;
}
double fps = 0.0;
LARGE_INTEGER qpcNow = {0};
if (m_fFreq)
{
if (QueryPerformanceCounter(&qpcNow))
{
if (m_nLastCounter)
{
m_nFramesSinceUpdate++;
fps = m_fFreq * m_nFramesSinceUpdate /
double(qpcNow.QuadPart - m_nLastCounter);
}
}
}
WCHAR szStatusMessage[64];
StringCchPrintf(szStatusMessage, _countof(szStatusMessage), L"
FPS = %0.2f Time = %I64d", fps, (nTime - m_nStartTime));
if (SetStatusMessage(szStatusMessage, 1000, false))
{
m_nLastCounter = qpcNow.QuadPart;
m_nFramesSinceUpdate = 0;
}
}
// Make sure we've received valid data
if (m_pDepthRGBX && pBuffer && (nWidth == cDepthWidth) && (nHeight
== cDepthHeight))
{
RGBQUAD* pRGBX = m_pDepthRGBX;
// end pixel is start + width*height - 1
const UINT16* pBufferEnd = pBuffer + (nWidth * nHeight);
UINT16 range = 100;
//UINT16 start_row = cDepthHeight/2-range+1;
//UINT16 end_row = cDepthHeight/2+range;
UINT16 start_row = 180-range+1;
UINT16 end_row = 180+range;
const UINT16* pBufferStartIndex = pBuffer + (nWidth *
(start_row-1));
const UINT16* pBufferEndIndex = pBuffer + (nWidth *
(end_row))-1;
const UINT16* pBufferStart = pBuffer;
char debugout[256] = "";
UINT col[cDepthWidth] = { 0 };
int index_counter = 0;
int minThres = 1000;
int maxThres = 6000;
while (pBuffer < pBufferEnd)
{
USHORT depth = *pBuffer;
//sprintf(debugout, "Depth: %d\n",depth);
//OutputDebugStringA(debugout);
// To convert to a byte, we're discarding the most-
significant
// rather than least-significant bits.
// We're preserving detail, although the intensity will
"wrap."
// Values outside the reliable depth range are mapped to 0
(black).
// Note: Using conditionals in this loop could degrade
performance.
// Consider using a lookup table instead when writing
production code.
BYTE intensity = static_cast<BYTE>((depth >= nMinDepth) &&
(depth <= nMaxDepth) ? (depth % 256) : 0);
if (pBuffer > pBufferStartIndex && pBuffer <=
pBufferEndIndex)
{
pRGBX->rgbRed = intensity;
pRGBX->rgbGreen = 0;
pRGBX->rgbBlue = 0;
}
else {
pRGBX->rgbRed = intensity;
pRGBX->rgbGreen = intensity;
pRGBX->rgbBlue = intensity;
}
if (pBuffer > pBufferStartIndex && pBuffer <=
pBufferEndIndex)
{
/*col[index_counter] += depth;
index_counter++;*/
/*if (depth < col[index_counter])
{
col[index_counter] = depth;
}
index_counter++;*/
if (depth > minThres && depth < maxThres)
{
col[index_counter]++;
}
index_counter++;
if (!(index_counter < cDepthWidth))
{
index_counter = 0;
}
}
++pRGBX;
++pBuffer;
}
char outgoing[8] = "R";
int dataLengthOut = 8;
//char incoming[256];
//int dataLengthIn = 256;
//int readResult = 0;
int prev_pin = 0;
SP->WriteData(outgoing, dataLengthOut);
OutputDebugStringA("\nR");
sprintf(outgoing,"");
for (int i = 0; i < cDepthWidth; i++)
{
//col[i] = col[i] / (range * 2);
if (/*col[i] > minThres && col[i] < maxThres*/ col[i]
>= 10)
{
col[i] = 1;
int angle = (120.0-120.0*i / float(cDepthWidth));
if (angle == prev_pin)
{
}
else {
sprintf(outgoing, "f%d", angle);
OutputDebugStringA(outgoing);
SP->WriteData(outgoing, dataLengthOut);
sprintf(outgoing, "");
prev_pin = angle;
}
}
else {
col[i] = 0;
}
//SP->ReadData(incoming, dataLengthIn);
//std::string test(incoming);
//sprintf(incoming,"%s\n", incoming);
//OutputDebugStringA(incoming);
}
pRGBX = m_pDepthRGBX;
pBuffer = pBufferStart;
index_counter = 0;
while (pBuffer < pBufferEnd)
{
USHORT depth = *pBuffer;
//sprintf(debugout, "Depth: %d\n",depth);
//OutputDebugStringA(debugout);
// To convert to a byte, we're discarding the most-
significant
// rather than least-significant bits.
// We're preserving detail, although the intensity
will "wrap."
// Values outside the reliable depth range are mapped
to 0 (black).
// Note: Using conditionals in this loop could degrade
performance.
// Consider using a lookup table instead when writing
production code.
BYTE intensity = static_cast<BYTE>((depth >=
nMinDepth) && (depth <= nMaxDepth) ? (depth % 256) : 0);
if (pBuffer > pBufferStartIndex && pBuffer <=
pBufferEndIndex)
{
pRGBX->rgbRed = 255 * col[index_counter];
pRGBX->rgbGreen = 0;
pRGBX->rgbBlue = 0;
index_counter++;
if (!(index_counter < cDepthWidth))
{
index_counter = 0;
}
}
else {
pRGBX->rgbRed = intensity;
pRGBX->rgbGreen = intensity;
pRGBX->rgbBlue = intensity;
}
++pRGBX;
++pBuffer;
}
// Draw the data with Direct2D
m_pDrawDepth->Draw(reinterpret_cast<BYTE*>(m_pDepthRGBX),
cDepthWidth * cDepthHeight * sizeof(RGBQUAD));
if (m_bSaveScreenshot)
{
WCHAR szScreenshotPath[MAX_PATH];
// Retrieve the path to My Photos
GetScreenshotFileName(szScreenshotPath,
_countof(szScreenshotPath));
// Write out the bitmap to disk
HRESULT hr =
SaveBitmapToFile(reinterpret_cast<BYTE*>(m_pDepthRGBX), nWidth,
nHeight, sizeof(RGBQUAD) * 8, szScreenshotPath);
WCHAR szStatusMessage[64 + MAX_PATH];
if (SUCCEEDED(hr))
{
// Set the status bar to show where the screenshot was
saved
StringCchPrintf(szStatusMessage,
_countof(szStatusMessage), L"Screenshot saved to %s",
szScreenshotPath);
}
else
{
StringCchPrintf(szStatusMessage,
_countof(szStatusMessage), L"Failed to write screenshot to %s",
szScreenshotPath);
}
SetStatusMessage(szStatusMessage, 5000, true);
// toggle off so we don't save a screenshot again next
frame
m_bSaveScreenshot = false;
}
}
}
/// <summary>
/// Set the status bar message
/// </summary>
/// <param name="szMessage">message to display</param>
/// <param name="showTimeMsec">time in milliseconds to ignore future
status messages</param>
/// <param name="bForce">force status update</param>
bool CDepthBasics::SetStatusMessage(_In_z_ WCHAR* szMessage, DWORD
nShowTimeMsec, bool bForce)
{
INT64 now = GetTickCount64();
if (m_hWnd && (bForce || (m_nNextStatusTime <= now)))
{
SetDlgItemText(m_hWnd, IDC_STATUS, szMessage);
m_nNextStatusTime = now + nShowTimeMsec;
return true;
}
return false;
}
/// <summary>
/// Get the name of the file where screenshot will be stored.
/// </summary>
/// <param name="lpszFilePath">string buffer that will receive
screenshot file name.</param>
/// <param name="nFilePathSize">number of characters in lpszFilePath
string buffer.</param>
/// <returns>
/// S_OK on success, otherwise failure code.
/// </returns>
HRESULT
CDepthBasics::GetScreenshotFileName(_Out_writes_z_(nFilePathSize)
LPWSTR lpszFilePath, UINT nFilePathSize)
{
WCHAR* pszKnownPath = NULL;
HRESULT hr = SHGetKnownFolderPath(FOLDERID_Pictures, 0, NULL,
&pszKnownPath);
if (SUCCEEDED(hr))
{
// Get the time
WCHAR szTimeString[MAX_PATH];
GetTimeFormatEx(NULL, 0, NULL, L"hh'-'mm'-'ss", szTimeString,
_countof(szTimeString));
// File name will be KinectScreenshotDepth-HH-MM-SS.bmp
StringCchPrintfW(lpszFilePath, nFilePathSize,
L"%s\\KinectScreenshot-Depth-%s.bmp", pszKnownPath, szTimeString);
}
if (pszKnownPath)
{
CoTaskMemFree(pszKnownPath);
}
return hr;
}
/// <summary>
/// Save passed in image data to disk as a bitmap
/// </summary>
/// <param name="pBitmapBits">image data to save</param>
/// <param name="lWidth">width (in pixels) of input image data</param>
/// <param name="lHeight">height (in pixels) of input image
data</param>
/// <param name="wBitsPerPixel">bits per pixel of image data</param>
/// <param name="lpszFilePath">full file path to output bitmap
to</param>
/// <returns>indicates success or failure</returns>
HRESULT CDepthBasics::SaveBitmapToFile(BYTE* pBitmapBits, LONG lWidth,
LONG lHeight, WORD wBitsPerPixel, LPCWSTR lpszFilePath)
{
DWORD dwByteCount = lWidth * lHeight * (wBitsPerPixel / 8);
BITMAPINFOHEADER bmpInfoHeader = {0};
bmpInfoHeader.biSize = sizeof(BITMAPINFOHEADER); // Size
of the header
bmpInfoHeader.biBitCount = wBitsPerPixel; // Bit
count
bmpInfoHeader.biCompression = BI_RGB; //
Standard RGB, no compression
bmpInfoHeader.biWidth = lWidth; // Width
in pixels
bmpInfoHeader.biHeight = -lHeight; // Height
in pixels, negative indicates it's stored right-side-up
bmpInfoHeader.biPlanes = 1; //
Default
bmpInfoHeader.biSizeImage = dwByteCount; // Image
size in bytes
BITMAPFILEHEADER bfh = {0};
bfh.bfType = 0x4D42;
// 'M''B', indicates bitmap
bfh.bfOffBits = bmpInfoHeader.biSize + sizeof(BITMAPFILEHEADER);
// Offset to the start of pixel data
bfh.bfSize = bfh.bfOffBits + bmpInfoHeader.biSizeImage;
// Size of image + headers
// Create the file on disk to write to
HANDLE hFile = CreateFileW(lpszFilePath, GENERIC_WRITE, 0, NULL,
CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
// Return if error opening file
if (NULL == hFile)
{
return E_ACCESSDENIED;
}
DWORD dwBytesWritten = 0;
// Write the bitmap file header
if (!WriteFile(hFile, &bfh, sizeof(bfh), &dwBytesWritten, NULL))
{
CloseHandle(hFile);
return E_FAIL;
}
// Write the bitmap info header
if (!WriteFile(hFile, &bmpInfoHeader, sizeof(bmpInfoHeader),
&dwBytesWritten, NULL))
{
CloseHandle(hFile);
return E_FAIL;
}
// Write the RGB Data
if (!WriteFile(hFile, pBitmapBits, bmpInfoHeader.biSizeImage,
&dwBytesWritten, NULL))
{
CloseHandle(hFile);
return E_FAIL;
}
// Close the file
CloseHandle(hFile);
return S_OK;
}
During Small Group Meetings or SGMs, handouts were created to illustrate weekly developments and
key findings. The handouts for this quarter can be seen below.
On March 12th, 2015, presentations for ME310 Winter Quarter took place. The presentation slides for
Team Renault can be seen below.
Team Renault’s Winter Brochure can be seen below.