supporting transactive memory systems in …
TRANSCRIPT
The Pennsylvania State University
The Graduate School
College of Information Sciences and Technology
SUPPORTING TRANSACTIVE MEMORY SYSTEMS IN DISTRIBUTED
GEOCOLLABORATION
A Thesis in
Information Sciences and Technology by
Vincent Francis Mancuso
© 2010 Vincent Francis Mancuso
Submitted in Partial Fulfillment of the Requirements
for the Degree of
Master of Science
August 2010
II
The thesis of Vincent Francis Mancuso was reviewed and approved* by the following:
Michael McNeese Professor of Information Sciences and Technology Professor in Charge of College of Information Sciences and Technology Thesis Advisor
David Hall Professor of Information Sciences and Technology
Guoray Cai Associate Professor of Information Sciences and Technology
*Signatures are on file in the Graduate School
III
ABSTRACT
As technology continues to grow and evolve the systems used to collaborate have found
new and better ways to assist distributed teams. Systems are now much more complex and can
support an indefinitely large number of collaborators, from anywhere in the world, doing any
imaginable task. While the systems keep getting more and more complex, and seemingly better
and better, humans have yet to truly adapt to these new, virtual, environments. Many of the
things that are taken for granted in face-to-face collaborations are completely absent from
distributed teamwork.
One such of this is transactive memory systems, which are simply, the knowledge of who
knows what. While these transactive memory systems are formed easily in face-to-face
communication, there has yet to be support for them in a virtual environment. In order to better
understand how transactive memory may work, this thesis explores other domains in which
distributed group work is common:
1. Awareness
2. Common Ground
3. Team Mental Models
Based on this review of other domains, this thesis proposes a set of design requirements
that may be useful in fulfilling the requirements of a transactive memory system. These design
requirements are based on a multidisciplinary research approach and provide a generic set of
heuristics that can be used in a multitude of systems. Based on these design requirements a
system for GeoCollaboration called GeoTMS was designed. This system is explicitly designed to
be a front end plug-in for other systems and datasets. It utilizes a new system architecture which
will allow programmers to easily and seamlessly integrate it into their own systems.
For usability and demonstration purposes, GeoTMS was attached to the NeoCITIES
Simulation Engine. This system allowed GeoTMS to be tested across graduate students to better
understand the interactions and usability within the interface.
This thesis summarizes the results of these studies providing an overview of the system
integration and the methods for analysis. Based on these studies, future design and research
directions are proposed for the GeoTMS Interaction Environment.
IV
TABLE OF CONTENTS
LIST OF FIGURES ................................................................................................................. VI
LIST OF TABLES ................................................................................................................... VIII
ACKNOWLEDGEMENTS ..................................................................................................... IX
Chapter 1 Introduction ............................................................................................................ 1
Introduction to Geographic Decision Making Systems ................................................... 1
Motivation ........................................................................................................................ 5
Organization of Thesis ..................................................................................................... 7
Chapter 2 Literature Review ................................................................................................... 8
Transactive Memory ........................................................................................................ 8
Alternate Perspectives on Transactive Memory ............................................................... 17 Awareness ................................................................................................................... 17 Common Ground ........................................................................................................ 31 Team Mental Models .................................................................................................. 42 Discussion ......................................................................................................................... 49
Chapter 3 GeoTMS Technical Overview ................................................................................ 52
Objectives......................................................................................................................... 52
Interface Requirements .................................................................................................... 53
GeoTMS Overview .......................................................................................................... 53
Information Hierarchy ................................................................................................ 53 Implementation ........................................................................................................... 55 Integration to External Systems .................................................................................. 65
Chapter 4 Methods and Materials ........................................................................................... 67
NeoCITIES....................................................................................................................... 67
Integrating GeoTMS to NeoCITIES ................................................................................ 69
Methods ............................................................................................................................ 76
Evaluation Process ...................................................................................................... 76 Qualitative Data Coding ............................................................................................. 78
V
Chapter 5 Results .................................................................................................................... 80
Participant Demographics ................................................................................................ 80
Survey Scales ................................................................................................................... 81
Retrospective Think Aloud .............................................................................................. 84
Discussion ........................................................................................................................ 98
Chapter 6 Conclusion .............................................................................................................. 104
Contributions .................................................................................................................... 104
Future Work ..................................................................................................................... 108
References ................................................................................................................................ 115
Appendix A IRB Materials ..................................................................................................... 131
Appendix B Study Materials ................................................................................................... 133
Appendix C GeoTMS Screen Shots ....................................................................................... 140
Appendix D NeoCITIES Scoring Model ................................................................................ 143
Appendix E Unique View Example ........................................................................................ 144
Appendix F NeoCITIES Scenario .......................................................................................... 146
VI
LIST OF FIGURES
Figure 1: Google Wave using Maps as a center of discussion ................................................. 4
Figure 2: Framework of Individual Situation Awareness, based on (Salas, et al., 1995) ........ 19
Figure 3: Google Docs Telecarets in a Spreadsheet ................................................................ 27
Figure 4: Example of Avatars being used in (a) online Poker, (b) online fantasy football, (c) online forums, and (d) online role-playing game ....................................................... 29
Figure 5: CoMotion Collaborative Visual Environment for visualizing Common Ground, from (Chuah & Roth, 2003) ............................................................................................. 37
Figure 6: 3-T Framework for Pragmatic Boundary Objects, based on (Carlile, 2004) ............ 48
Figure 7: Visualization of Information Hierarchy ................................................................... 54
Figure 8: Cross Functional Flow Chart Representation of System Layout ............................. 56
Figure 9: Class Diagram of Model Structure ........................................................................... 57
Figure 10: Screenshot of interface ........................................................................................... 60
Figure 11: Attention Circle showing the “Red Teams” current focus ..................................... 61
Figure 12: A marker tracker signaling that there is an event taking place out of the current field of vision ....................................................................................................... 62
Figure 13: The Team View, with (A) Compact View and (B) Full View ............................... 63
Figure 14: NeoCITIES System Architecture Diagram. From (Hellar & Hall, 2009) .............. 68
Figure 15: NeoCITIES 3.0 User Interface ............................................................................... 69
Figure 16: GeoTMS fully integrated into NeoCITIES ............................................................ 70
Figure 17: Team Member Unique View .................................................................................. 71
Figure 18: Event Unique View as a Pop-up on the Map ......................................................... 72
Figure 19: Event Icon Unique View ........................................................................................ 73
Figure 20: Event Status Unique View ..................................................................................... 73
Figure 21: Reporting Units Unique View ................................................................................ 74
Figure 22: Event Description Unique View ............................................................................. 74
VII
Figure 23: NeoCITIES Header ................................................................................................ 75
Figure 24: The severity of the event dictates the markers color .............................................. 76
Figure 25: Human Performance Scoring Model, Graph taken from (Hellar, 2009) ................ 96
Figure 26: Comparison of GeoTMS interface vs. NeoCITIES 3.0 for Reaction Time ............ 97
Figure 27: Additional damage done after 10 second lag in reaction time................................ 98
Figure 28: Tracking a marker as you navigate through the map (a) Marker of interest is in the center of the map (b) As you move to the right, the marker cannot be distinguished with the other marker in the area (c) As you move around and more markers appear, the one of interest gets completely lost .................................................. 99
VIII
LIST OF TABLES
Table 1: Six models for making GIS available to community organizations, from(Leitner, et al., 2002)....................................................................................................................... 2
Table 2: Overview of Transactive Memory Literature ............................................................ 13
Table 3: Overview of Concepts and Solutions in Awareness Literature ................................. 31
Table 4: Overview of Concepts and Solutions in Common Ground Literature ....................... 41
Table 5: Types of boundary objects as described in (Lee, 2007), adapted from (Star & Griesemer, 1989) .............................................................................................................. 47
Table 6: Overview of Concepts and Solutions in Team Mental Models Literature ................ 49
Table 7: Applying concepts to Group Knowledge Stock ........................................................ 50
Table 8: Applying concepts to Consensus about Knowledge Sources .................................... 50
Table 9: Applying concepts to Specialization of Expertise ..................................................... 51
Table 10: Applying concepts to Accuracy of Knowledge Identification ................................. 51
Table 11: Interface Requirements for Transactive Memory System ....................................... 53
Table 12: List of Interface Codes ............................................................................................. 78
Table 13: List of Top-Down codes .......................................................................................... 79
Table 14: Perceived Usefulness of GeoTMS Interface Components ....................................... 81
Table 15: Actual Use of Components during Simulation ........................................................ 82
Table 16: Task ability evaluation scales .................................................................................. 82
Table 17: Full breakdown of comments for each component .................................................. 84
Table 18: Full breakdown of comments under each code........................................................ 84
Table 19: Interface problems and possible solutions ............................................................... 102
Table 20: Example of breaking up a NeoCITIES description to multiple information sources .............................................................................................................................. 110
Table 21: Styles of Communication for future development ................................................... 113
IX
ACKNOWLEDGEMENTS
First and foremost I would like to thank my advisor Dr. Mike McNeese. When I first
started working on this master’s thesis I thought it was the end of the world, but through his
guidance and support, I was able to find my way. Additionally, I would like to thank the other
members of my committee--Dr. Guoray Cai and Dean Dave Hall-- for supporting my work and
giving guidance when I needed it. Through all of their help, this thesis was infinitely better than
it would have been without. I would also like to thank Dr. Susan Mohammed for her support,
advice and funding throughout the past year.
I would also like to thank my lab mates--Dev, Nathan, and Eric--for their constant
willingness to take time out of their days to talk about my project and provide me with support
and advice throughout the entire process. Additional thanks go out to the other members of the
Penn State Graduate community who agreed to participate in my study.
Thanks to Ian and Bernie who I shared many beers with at Zeno’s planning,
contemplating, and complaining about this very paper. I may not have known it at the time, but
those excursions kept me sane throughout the entire process. Furthermore, thanks to Phil, Jon,
and Todd, for giving me the advice, encouragement, support and inspiration to pursue a graduate
degree. Also, thanks to Chris and Will for responding to my instant messages at all hours, when I
needed a distraction.
I would like to thank my entire family for their love and support throughout my entire
life. I could take up pages going through everybody who made this possible, but I would like to
specifically mention--my mother, Judi, for making sure I never spiraled--no matter what the
obstacle and for being so willing to put up with my insanity over the last 25 years, my father,
Carmen, for being such an amiable, supportive and smart person, and for making sure to always
remind me that there is a time for work and a time for play and finally, to my sister, Toni, for
always making sure to ground me when I got too cocky and for finding little ways to show her
love and support.
And lastly, I would like to thank Jodi Forlizzi for taking a chance on a 19 year old frat
boy, and showing me that there is so much more to the world of HCI than just industry. Without
her guidance and the opportunity she gave me I would have never had the confidence to pursue a
graduate degree in the first place.
Chapter 1
Introduction
Introduction to Geographic Decision Making Systems
Even before the use of computers, maps have been the focal points of our collaborations.
In his book “Cognition in the Wild”, Hutchins conducted an anthropological study of a
Navigational Department on a Naval Warship (Hutchins & Lintern, 1996). During his time spent
on the ship, he saw that although many of the Navigational Officers were in the same room, they
relied on an artifact, specifically the map, as their main source of collaboration. Since then,
mapping technologies have not only become more pervasive in Navigational Tasks, but have
permeated into our everyday lives.
In 2001, Alan MacEachren estimated that 80% of all the digital data generated at the time
contained some sort of geospatial data (A. M. MacEachren & Kraak, 2001). With the rise of new
Web 2.0 Mapping technologies (i.e. Google Maps, Yahoo! Maps, Bing Maps, etc.) being freely
and readily available, one can safely assume this number has only increased. This type of data
adds new complexity to an already difficult decision making task. In (Andrienko, et al., 2007), the
authors outlined three specific features that must be considered for spatial decision problems. The
first consideration is the fact that the general nature of geographic space is much more complex
than a generic spatial plane. When dealing with geography, decisions do not only vary on the x, y
and z coordinates, but the nature of the land under them. Decisions may change based on the
biome of the area of interest, whether it is a city or rural area, and what is in or around it.
Additionally, when working with a spatial area, directions can be as simple as a line from point A
to point B, when dealing with Geographic areas the practicality of travel must be considered (i.e.
roadways, trails, mountains, etc.). A second consideration, which is common with any decision
making task, is having several persons with several different roles involved. This means that
within the GIS, collaboration, communication and flexibility become even more important. The
third and final consideration is that of tacit criteria and knowledge, this is saying that when
dealing with geospatial information, it is often difficult to verbalize or externalize the
information.
2
GeoCollaboration
As stated before, with the rise of Web 2.0 mapping technologies and these
GeoCollaboration’s being available to the masses, designers can no longer focus on just the
“experts”. This new area of Web-based GIS (WebGIS ) requires designs to no longer be focused
on making this technology available and useable to the masses (Kraak, 2004). This idea of
GeoCollaboration has been used in Emergency Response/Management (Schafer, Ganoe, &
Carroll, 2007; Wu, Zhang, Convertino, & Carroll, 2009), Military Operations (Fleming, Jordan,
Madden, Usery, & Welch, 2009) and civil planning (Drummond & French, 2008; Li, Bo, Ju, &
Guo-xue, 2009). For this new area, the map may be considered as an extension of a general
collaborative environment. Much of the research in this area focuses on bringing the technologies
and benefits of GIS systems to the community, for collaboration and planning tasks.
In (Leitner, McMaster, Elwood, McMaster, & Sheppard, 2002) they describe six models
for making GIS available to community organizations.
Type Description
Community-based (in-house) GIS An independent node located within the community organization that brings GIS to organizers and residents without requiring them to go outside o the community.
University/Community Partnerships Having students, faculty and community organization leaders work together to identify more specific goals and source of spatial data to satisfy mapping needs.
GIS Facilities in university and public libraries Facilities that creates and maintains basemaps and spatial data and makes them available for use with GIS software
“Map Rooms” A facility which creates custom maps on demand for community organizations and citizens
Internet Map Servers Pre-defined maps available to community organizations over the internet
Neighborhood GIS Centers A center where members of the neighborhood pool their expertise and resources, to provide a central facility that all affiliated community organizations could use.
Table 1: Six models for making GIS available to community organizations, from(Leitner, et al., 2002)
While making maps and mapping software available to the public is an important aspect
of GIS, the need for designing for the public must not be ignored. In the fields of Argumentation
Maps (Keßler, Rinner, & Raubal, 2005; Rinner, 2001), Collaborative Spatial Planning
3
(Dragicevic & Balram, 2004; P. Healey, 2003), and Public Participation GIS (Carver, Evans,
Kingston, & Turton, 2001; Kingston, Carver, Evans, & Turton, 2000), researchers focus on ways
that these maps can be specifically tailored to community leaders and organizers.
While many of the applications readily available on the internet focus on single user
decision making, like the naval task discussed in Hutchins, these tasks must be leveraged for
human-human collaboration. One way of looking at this area is when the humans are actually co-
present, like in the situation on the naval warship. In this area, researchers look at how maps can
be augmented with natural language interfaces and intelligent user interfaces, to allow the humans
to communicate with the map and have it respond intelligently. This concept of GeoDialogue (H.
Wang, Cai, & MacEachren, 2008) allows human-GIS communication, while still maintaining the
map as a central artifact in human-human collaboration.
GeoCollaboration on the World Wide Web
While human-GIS (-human) collaboration, sometimes referred to as same-place/face-to-
face group work within the GIS field, is an important issue for research, there is also the need for
distributed teams to collaborate over a map. This area of Distributed GeoCollaboration aims to
merge the areas of Computer Supported Cooperative Work (CSCW), Distributed Teamwork, and
GIS. This field is often referred to as Web-GIS
As GIS systems are moved to the cloud, and integrate distributed collaborations, the
function of the map stays the same, but the role itself changes (Kraak, 2004). When moving to a
distributed setting, the functionality of the map can be extended, to allow for public and private
views which allows more exploration and visual thinking. An example of this type of
extensibility can be seen in (G. Convertino, Zhao, Ganoe, Carroll, & Rosson, 2007). In this
project, they developed a Distributed GeoCollaboration system which provided a role-specific
map, tailored to the specific user, and a shared map which everybody was able to see and interact
with. This model allows for clear and distinct role definitions and knowledge and it better affords
implicit knowledge sharing. In this system they propose six necessary features of a collaborative
GIS program to support their role-based collaboration:
− Annotations – adding symbols, text notes, and free-drawing marks − Transfer – allows users to “push” annotations from their personal view to shared view − Sidebar – Overview of all the shared annotations − Telepointer – Provides action awareness among team members
4
− Role-based indication – role-based color coding to indicate actions
In addition to new interface and interaction technologies, the rise of Web 2.0, and
software-as-a-service (SAAS) architectures have added new datasets and information to further
empower GIS systems on the web. Features such as social search, collaborative publishing/web-
authoring, and mash-up embedded services provides users with new capabilities to create, share,
read, and combine geographical information with each other and other metadata generated on the
internet (Sigala, 2009).
An example of how Web 2.0 technologies are being merged with existing GIS
infrastructure to aid in online collaborations can been seen in the new Google Wave platform
(http://wave.Google.com). This system allows users to easily integrate a map into a discussion,
annotate the map, and crowd source information around the map (Figure 1).
Figure 1: Google Wave using Maps as a center of discussion (from April 2010)
Other examples of merging Web 2.0, and socially generated data with geographic data is
Twitter Maps (http://www.twittermap.tv), GeoChirp (http://www.geochirp.com), Flickr maps
(http://www.flickr.com/map/), and the Facebook Friend Map (http://apps.facebook.com/friend-
map). While these do not all integrate explicit collaborations like some of the other systems, they
provide perfect examples of how all the new information that is generated every day into
geospatial collaborations for decision making can be leveraged.
5
Motivation
Over the past decade, the MINDS lab at The Pennsylvania State University has focused
on distributed team work and macrocognitive principles that surround these teams. This research
has led to many contributions in the form of theoretical implications and scaled world simulation
design. One of the main goals of the research completed is the continued evolution of the scaled
world simulation and research platform of NeoCITIES. Since the first iteration (R. E. T. Jones,
McNeese, Connors, Jefferson, & Hall, 2004; M. D. McNeese, et al., 2005), the NeoCITIES
research platform has continued to grow and evolve, and recently has moved from being a
distributed application, to an application that is run in the cloud (Balakrishnan, Pfaff, McNeese, &
Adibhatla, 2009; Hellar & Hall, 2009).
Since NeoCITIES first implementation, the internet has become an even larger repository
of information. Today, the internet is a constantly changing organism, filled with information
generated by news reporters, bloggers, regular citizens, and even computers. From this, new
fields of research such as collaborative information search and retrieval have emerged to better
understand how people can work together in our information gathering processes.
While the research in the MINDS lab has begun moved its applications to being more
internet focused they have not begun to take advantage of all this new data and information.
Much of the research conducted focuses on how people use information given to them from
fictitious sources to make decisions. While this is sometimes the case, decision making in today’s
internet focused world, information is rarely presented to us in a nice clean form, and it is rarely
complete. Because of this, people often rely on the internet as their main source to “fill in the
gaps” and gain a better understanding of the information they have. In order to keep research
current, research platforms must be moved into the Web 2.0 world. The move has already been
made to move NeoCITIES to cloud based architecture, and now the focus is on bringing the data
into the equation.
In order to support these new streams of information, the overall architecture of
NeoCITIES had to be changed. Additionally, one of the biggest limitations of the system is the
lack of any sort of mapping component (Hellar, 2009). While designing an interface for
NeoCITIES would have been easy enough, it would also be a very limited use of the work that
would have to go into the development. Because of this, a goal was made to design a system that
would allow for data inputs from multiple sources for GeoCollaborative tasks. The goal here is to
6
design a system that does not exist, as just “throw-away-code” for one purpose, but can be
adapted and edited to numerous systems and research problems.
Rather than just moving forward with design based on our own assumptions and previous
work, it is important to consider the challenges that face distributed team’s collaborations. Along
with the challenges of being physically distributed, cultural and organizational boundaries exist
within these teams. While there have been vast amounts of work done in the field of supporting
distributed teams via computer mediated communication (CMC), the current sentiment is that
these collaborations are not and will never be as effective as face-to-face collaborations (Canney
Davison & Ward, 1999; Maznevski & Chudoba, 2000; Powell, Piccoli, & Ives, 2004; Stone &
Posey, 2008; Tutty & Klein, 2008; Warkentin, Sayeed, & Hightower, 1997).
When interacting in face-to-face environments, there are numerous attributes that people
take for granted--explicit communication, the ability to observe the orientation and movement of
another person’s body in the workspace and their actions on the objects in the workspace (J Hill
& Gutwin, 2003). These characteristics create a larger picture that affects how people work and
the bonds that they form. These characteristics do not easily transfer over to the virtual space and
are not thought of as issues that may be important in designing the tools. Therefore, system
designers often overlook some of these aspects in their design, making the virtual collaborations
clunky, inefficient and ineffective. Adding to the challenges faced is the general makeup of the
dispersed project teams. Other than the obvious differences in these teams (many people from
many different domains who speak different “languages”), organizations are committed to them
because geographically dispersed teams allow for organizations to employ more cross cultural
teams—thus enhancing the projects. However, the challenges faced need to be considered
because not only is there a barrier within the interactions, but there are also cultural and (actual)
language barriers. These barriers can and will affect the overall outcome of the teams’ work.
Even with the obvious drawbacks faced with Computer Mediated Communication within
teams, research and organizations continue to move towards the idea of distributed teams as a
critical methods for working on projects, solving problems and completing tasks because these
distributed (or virtual) teams and systems allow for new functionality and abilities that teams
who are co-located may not have.
One important theoretical construct that has yet to be adequately addressed for these
distributed teams, is that of Transactive Memory (Oshri, van Fenema, & Kotlarsky, 2008). Since
Transactive Memory has not been addressed from the system design standpoint other literatures
must be consulted in order to identify solutions and motivations for system design. Because of
7
this, in this thesis I have taken a multidisciplinary artifact-as-theory (J. Carroll & Kellogg, 1989)
approach to identifying necessary design elements for supporting and augmenting transactive
memory for distributed teams.
While one could focus on studying this concept of Transactive Memory in distributed
teams, this would become very broad and difficult to scope. As stated earlier, one interesting
domain that is becoming more rich and useful as technology progresses is that of
GeoCollaboration. The role of Transactive Memory in a GeoCollaborative Decision Making task
is a very interesting domain which has yet to be explored by research.
The goal of this thesis is to design an extendable and useful tool to help support
GeoCollaborations and the formation of effective Transactive Memory Systems in distributed
teams. Additionally from the literature review, I hope to present a multidisciplinary approach to
understanding transactive memory and how it can be supported for distributed teams.
Organization of Thesis
The remainder of this thesis will be organized as such: Chapter 2 provides a background
of the research done in the field of transactive memory, awareness, common ground and Team
Mental Models. Chapter 3 utilizes the findings from the previous chapter and organizes them into
a set of design requirements. Based on these requirements, a system for supporting Transactive
Memory systems in GeoCollaborative tasks was designed and is explained in detail. Chapter 4
will describe the methods and materials used to test the overall usability of the tool. This includes
the qualitative and quantitative methods use, as well as an overview of how the tool was
integrated into the NeoCITIES Simulation Engine for experimentation purposes. Chapter 5
presents the results that came from the study, and uses them to formulate a list of future design
considerations. Finally Chapter 6 concludes by describing on the contributions and future
research implications.
Chapter 2
Literature Review
Transactive Memory
An individual's memory can be broken down into three stages--the encoding stage--
where information is entered into memory, the storage stage--where the information is stored in
memory, and the retrieval stage-where the information is brought back. When moving
information within each of these stages, it is fairly common that one or several problems can
arise--such as incorrect encoding, forgetfulness or memory overload. In order to prevent memory
items from being incorrectly encoded or “falling out” of memory, people often rely on external
artifacts, such as notepads, calendars and to-do lists for memory storage. Although these external
artifacts can be useful in maintaining one’s own internal memory, they are only as useful as the
internal knowledge that the specific person possesses. If a person does not possess the knowledge
to understand or do something, even a very specific external memory aid may not be able to help
them accomplish their goal. Because of this limitation, people rely on a different type of memory
called transactive memory. In Transactive Memory, a person encodes the knowledge that they
have access to through other people’s internal knowledge. This means that a person has the ability
to use another person as external storage for information that they do not possess (Wegner,
Giuliano, & Hertel, 1985). Transactive memory includes two parts--the individual knowledge of a
person and the interpersonal awareness of others’ knowledge (Wegner, 1987). This awareness of
others knowledge can then be divided into three dimensions--accuracy, agreement and
complexity (Moreland, 1999). These dimensions were later expanded by (Austin, 2003).
Transactive Memory has been shown to be a critical element in Open Source Software Teams
(Chen, 2009), Emergency Response (M. P. Healey, Hodgkinson, & Teo, 2009), Collaborative
Learning Teams (Michinov & Michinov, 2009), New Product Development Teams (Akgün,
Byrne, Keskin, Lynn, & Imamoglu, 2005), and Trauma Resuscitation Teams (Sarcevic, Marsic,
Lesk, & Burd, 2008).
The majority of the research done in Instructional and Organizational Psychology agrees
that Transactive Memory is a very important aspect of effective teamwork and collaboration.
9
Several studies have shown that teams who have better Transactive Memory Systems (TMS) are
able to perform collaborative tasks quicker and with more success than teams with weaker TMS
(Ellis, 2006; Lewis, 2004; Liang, Moreland, & Argote, 1995; Littlepage, Hollingshead, Drake, &
Littlepage, 2008; Michinov & Michinov, 2009; Moreland & Myaskovsky, 2000; Wegner, Erber,
& Raymond, 1991). In many of these studies, the researchers allow these TMS to form through
collaborative training. It was shown that the teams that were trained together formed a better
TMS and thus performed better on the task than teams who did not train together. Each of these
studies showed how face-to-face interaction was a valuable component in forming TMS.
However, several questions of whether or not the TMS was what helped them better perform the
task, or if it was a result of the better (learned) group dynamics. Following this question, several
studies showed that it was the teams' TMS that was the more prevalent cause of more effective
teamwork. The research also indicated that a team with a strong TMS can outperform a team with
weak TMS even though they possessed a good group dynamic (Hollingshead, 1998a, 1998b;
Moreland & Myaskovsky, 2000; Wegner, et al., 1991). The most interesting and possibly most
significant result of the study (Moreland & Myaskovsky, 2000), indicated that face-to-face
communication was not necessary to build the TMS. For this study, rather than training the teams
together, they trained them individually. Then, before the group task began, they provided their
teammates with lists of each other's strengths and weaknesses. Even though this study showed
that face to face communication was not necessary to form the TMS, subsequent research
continued to focus on how face-to-face communication was still necessary in utilizing a TMS
(Jackson & Moreland, 2009; Lewis, 2004). In a more recent study, (Littlepage, et al., 2008)
showed that explicit communication is not a critical aspect of the utilization of a Transactive
Memory System and teams who had previously formed TMS were able to perform without the
communication. Additionally, this study was focused on more believable and relevant styles of
communication within a team than previous studies looking at communication’s utility in a TMS
(Hollingshead, 1998b).
There have been several field studies of teams in real environments to show the practical
utility of forming an effective transactive memory system. In (M. P. Healey, et al., 2009) the
authors conducted a field study to analyze the determinants of performance in teams responding
to civil emergencies in training exercises. In this study, they not only analyzed the determinants
on a team level but a multi-team level. The results showed that meta-knowledge of other’s
expertise (transactive memory) were a crucial aspect for team performance in these emergency
responses. There have been several studies conducted that followed classroom teams to analyze
10
the effects of transactive memory on their learning and performance. In (Michinov & Michinov,
2009) the authors followed 113 students in 2-3 person groups throughout the entire semester as
they participated in learning activities and produced a final report to study the three dimensions of
transactive memory. The authors showed that there was a positive relationship between
transactive memory and performance, which was based mainly on coordination and
specialization. Another learning study (Jackson & Moreland, 2009), analyzed information
systems undergraduate students as they took part in a four-part group project. Once again the
results showed that teams with a better TMS performed better than teams with a weaker TMS.
This time they showed that transactive memory was best predicted by group communication. In
addition to education, Transactive Memory has also been shown to be a major contributor to
performances in open source software development teams. In his dissertation, (Chen, 2009)
surveyed 97 open source software (OSS) project teams from a large community site to investigate
the role of Transactive Memory within their team. His study showed that a strong TMS is very
important for knowledge coordination and the communication quality of their exchanges. This
communication quality ended up being the strongest influence on team performance. In a follow
up paper, they further expanded this to show that not only knowledge coordination, but
understanding the knowledge coordination had a positive influence on the projects technical
achievements (Chen & Dietrich, 2009). This study also had interesting results that are
inconsistent with other findings saying that knowledge credibility has no impact on knowledge
coordination, but this may have been a result of the voluntary nature of the groups. Additionally
they showed that complete specialization is very detrimental to the OSS teams, and a few
“generalists” (often the project administrators) are necessary.
In order to better understand Transactive Memory, you must first step away from the
most basic definition and understand what it fully entails. In their research (Moreland, 1999)
proposed a more complex conceptualization of what makes up a Transactive Memory System.
Building on this, (Austin, 2003) used these elements and modified them by expanding the
coordination dimension into four categories:
− Group knowledge stock - the combination of the individual knowledge of each person in the group (Coordination in Moreland)
− Consensus about knowledge sources – consensus/the extent to which the group members agree about who has what knowledge (Coordination in Moreland)
− Specialization of expertise - Where each member can build a deeper knowledge base in a narrowly defined area of expertise and everybody is aware of who specializes in what (Specialization in Moreland)
11
− Accuracy of knowledge identification – the accuracy to which a member who is identified as having a specific piece of knowledge actually has that knowledge. (Credibility in Moreland)
These four dimensions can be seen in many of the research components above and may
provide explanations for the various effects that have been seen in the lack of support for building
and using a TMS in non face-to-face communication (e.g. distributed collaboration as is evident
via w.w.w interactions). In (Moreland & Myaskovsky, 2000) they showed how face-to-face
communication was not needed to improve team performance with a TMS. However, other
research that explored non face-to-face interactions (Lewis, 2004) did not find the same results.
This could be an effect of these four dimensions being met in one study and not the other. As
stated earlier, in (Moreland & Myaskovsky, 2000) the team members were provided with a list
outlining everybody’s strengths and weaknesses--creating a consensus of information between the
team members. The paper created a persistent artifact representing the knowledge stock and since
it was provided by the researchers, it could be considered accurate. In the other study, this
information would have had to be gathered by using a synchronous communication channel. This
not only could cause participants to question the reliability of the information, but since the
communication modalities were not persistent, there is a possibility of the information being lost.
These dimensions have been used as the basis of analysis for several research studies looking at
effectiveness of teamwork and TMS’s (Akgün, et al., 2005; Ilgen, Hollenbeck, Johnson, & Jundt,
2004; Kozlowski & Ilgen, 2006; Lewis, 2004). These dimensions were also used as the basis for
developing a quantitative measurement of a Transactive Memory System within a group (Lewis,
2003). This scale is based off of the initial dimensions by (Moreland, 1999) and only takes into
account specialization, credibility and coordination.
One of the dimensions--specialization of expertise--has been shown in research to be one
of the largest contributors to an effective TMS (Littlepage, et al., 2008; Michinov & Michinov,
2009). It is believed that these ability differences and specializations will only lead to group
improvements when the members of the groups become aware of different team members'
abilities. As discussed above, in a study of OSS project teams (Chen & Dietrich, 2009), the
authors showed that while specialization is important, in these types of distributed teams it is
important to have a few generalists take on the role of the project manager. By having somebody
who understands everybody’s specialty they are able to take advantage of everybody by correctly
integrating their specialized knowledge with other team members. This concept of specialization
in these types of team is further aided by the archival, asynchronous nature of the internet, and
12
Web 2.0 applications. In the past, when working in a distributed team, the only information one
would have about their partners or teammates was what they specifically shared. This requires a
person to make inferences which may or may not be correct. In today’s internet, information
about a person’s specialties and abilities is often a Google search away. Information such as their
personal web page, academic publications, and their Facebook can be useful tools in building an
awareness of each other’s specialty without any direct contact.
By having a broader range of specializations, groups can allocate a broader set of
resources and “attack a problem” from more directions than groups with a more homogeneous set
of abilities. Studies have shown that individuals tend to develop their own unique set of
specialties as they begin to recognize that other people have their own specialties (Hollingshead,
2000, 2001; Lewis, 2004; Wittenbaum, Stasser, & Merry, 1996; Wittenbaum, Vaughan, &
Stasser, 1998). This allows team members to develop a complementary and unique area of
specialization to assist in the task. This specialization also has major effects on the accuracy of
the knowledge and the consensus of the knowledge sources. If there is an agreement on the
specializations of each member of the group, then people are able to take responsibility for
specific pieces of information and allow them to rely on each other for specific domains
(Hollingshead, 2000; Lewis, 2004; Wegner, 1987). Specialization also raises more problems for
distributed teams. It is often the case that they develop their specializations prior to the
collaborations and they do not have the time to develop new specializations before they begin
their important work. Some research looks at this as more of an administrative issue (Rosen,
Furst, & Blackburn, 2007), where group coordinators need to form groups with complementary
specializations and ensure that everybody is aware of each other’s abilities. While this is a valid
solution for pre-meditated distributed teams, problems will still arise when teams must be formed
in an ad-hoc nature.
For an overview of the major contributions to the transactive memory literature see Table
2.
13
Table 2: Overview of Transactive Memory Literature Publication Study Contribution
(Wegner, et al., 1991) Memory task involving questions about their relationships and expertise in a number of knowledge categories. Subjects were either paired with their significant other or a person of the opposite sex from another couple.
When teams are assigned to each other to work together randomly memory structures can act as an aid to the transactive memory and help increase their performance. When no structure is used, teams who have a pre-built transactive memory system will outperform those without one.
(Hollingshead, 1998a) Re-examination of the experiment conducted in (Wegner, et al., 1991), but rather than varying on structure, this study varied on whether or not the groups were able to communicate information during learning.
Confirmed the results of the previous study, but also showed that communication at learning can both facilitate and inhibit the encoding of new information. When strangers communicated information they learned the words and were able to delegate the responsibility of new information and compensate for the developed TMS of the dating couples.
(Hollingshead, 1998b) Compared the effects of training participants as individuals or groups in two complex cognitive tasks. The participants would either work on two practice problems as a group of four or as individuals before performing a similar problem as a group (experiment 1), or as individuals (experiment 2).
Participants who practiced the problems as a group performed significantly better on the performance problem when working in a group, but not when they worked individually. This shows that during the performance was an effect of the transactive memory system that formed rather than a collaborative learning effect.
(Moreland, 1999) Book chapter outlining the previous work (up to this point) that has been done in transactive memory research.
Main contribution of this chapter, other than summarizing all the literature that had occurred up to this point was the initial development of the dimensions of transactive memory. The three initial dimensions that were described in this chapter were, coordination, specialization, and credibility
(Moreland & Myaskovsky, 2000)
Experiment to test whether the results of the previous study were an effect of transactive memory, or due to improved communication with another. Participants were taught to build transistor radios using kits. They would be trained as a group or individually. Some of the teams made up of those individually trained would receive information on their group member’s abilities, strengths and weaknesses prior to the performance trial.
Participants who were individually trained they provided information about one another’s skilled performed comparably to that of the groups who were trained together. Both of the group trained and the individual with information condition performed significantly better than those who were individually trained with no information. This shows that the better performance is a result of transactive memory rather than better group dynamics.
(Hollingshead, 2000) This study looked at clerical office workers in a laboratory setting to understand how individual learning in a work setting is affected by perceptions of the experience of a coworker. To test this, participants were told that they were going to complete a task that had either similar or different knowledge about a task.
The findings of this study primarily show how important specialization is to forming an effective TMS. The results showed that people learned, recalled and shared more information in their own areas of expertise when their partner had different expertise. This was the first study to investigate how perceptions of experience affect how people learn and collaborate with co-workers.
14
(Hollingshead, 2001) A laboratory based experience to understand how transactive memory systems are developed within a group. The experiment tested the effects of varying knowledge expectations on a simple learning task (similar to the studies above).
Cognitive interdependence and convergent expectations worked together to improve transactive memory among people who had no experience working together or interacting. The results show that teams that had too much diverging prior-knowledge in specific tasks were not able to form effective transactive memory systems. This goes against some of the other research in the field. The authors do say that some levels of differentiation is good.
(Austin, 2003) Moved the analysis of transactive memory out of the laboratory into a real life workgroup at a large company, in order to understand how transactive memory effects on a maturing, continuing workgroup. The data was collected via surveys of 27 different groups in a large apparel and sporting good company.
The author found that workgroups who formed a transactive memory system early on had better group performance. Though the main contribution of this research was the expansion of the three dimensions of transactive memory to four. The author described transactive memory as a measure of a groups, knowledge stock, knowledge specializations, transactive memory consensus, and transactive memory accuracy. Additionally this article proposes a new way of qualitatively analyzing a transactive memory system based on the proposed dimensions.
(Lewis, 2003) This study focused on designing and validating a field measure to analyze transactive memory systems. The author describes the development of the 15 point scale (based on the three dimensions of TM), and tested it’s validity in a laboratory settings on 124 teams. The task focused on learning how to put together a telephone.
The biggest contribution of this paper is a quantitative self report measure for analyzing a transactive memory system. Based on their study they were able to confirm the validity, and this scale has been used in numerous future studies by this and other researchers.
(Lewis, 2004) A study of 64 MBA consulting teams to see how transactive memory systems emerge and develop to affect the performance of groupwork. Each team was assigned to an organization where they expected to perform a business oriented task (i.e. develop a marketing team). Transactive Memory was analyzed based on a scale using the dimensions of transactive memory.
The findings of this experiment show that teams who have specialized knowledge initially are more likely to develop a transactive memory system. Additionally the author found that teams who had frequent face to face communication were able to build a transactive memory system, but teams who used other modalities were not. Once the TMS was formed the teams were only able to utilize it when communicating face to face. This shows that the transactive retrieval process may only be triggered when communicating in a face-to-face interaction. Finally, as with the other studies, the teams with a TMS performed better than teams without one.
15
(Littlepage, et al., 2008)
Conducted a study on the effects of specificity, communication, and ability differences in clerical staff teams. In the study, the workers were asked to complete a job-knowledge quiz and were to assign parts of the quiz to members of their groups. The worker would either allocate the questions based on the domain of knowledge or a specific quiz item. Their final performance score would be the sum of the scores of the assigned questions.
This study once again confirmed the notion that performance was better when groups had an effective transactive memory system. This paper also presents the idea of domain (broad knowledge) level vs item (specific assignment) level transactive memory systems. While groups in both of these conditions out performed those without a transactive memory system, it was found that performance was higher when allocations were made at a more specific level. This may indicate that transactive memory can lead to better performance when tasks are more specific rather than broad.
(Sarcevic, et al., 2008) This study looked at how trauma teams collaborate, communicate and utilize their transactive memory systems. In this study they videotaped 10 trauma resuscitations and transcribed each event as it happened. Based on these transcriptions they performed quantitative and qualitative research on the data to observe the teams interactions and communication patters that supported information sharing and transfer.
The authors propose a method of qualitatively studying observed transactive memory. They analyzed the information flow and identified the most common provider and seeker of information. They showed that in their collaborations inquiries and responses were the primary outlet for using transactive memory in trauma teams. Although communication is essential for TM their findings suggested that due to the nature of their job, it is crucial that information is transferred to all team members which caused members to speak frequently in short intervals to exchange information. Additionally, due to the chaotic nature of the job, some speech was poorly audible which cause members to have to speak out loud or repeat their inquires. In order to help with these concerns they propose augmenting the information sharing with an information system.
(M. P. Healey, et al., 2009)
This study focused on the role of transactive memory in multiteam systems when responding to civil emergencies. This study was conducted by using a field analyses of multiteam systems as they handled emergency response training exercises. The simulation was broken up into three different types of collaboration, collocated simulation
Previous research in multiteam systems looked at how Team Mental Models affect team performance, but the results of this study suggest that the performance of a MTS is more contingent upon the development of TMS in and between teams. They propose intra-team TM and inter-team transactive memory to show how the dynamics of a MTS works. Additionally they found that there are development differences between intra- and inter- team TM.
(Michinov & Michinov, 2009)
This study focuses on investigating the the three dimensions of TM and the performance of collaborating students. The study focused on 113 students in groups of 2 or 3 as they complete a set of learning tasks and write a final report. Between each of the tasks the students would fill out a questionnaire self-reporting measures about TM.
Results showed that coordination and specialization lead the development and use of a transactive memory system and improved the ability of the groups to collaboratively learn and perform their tasks. They also propose new methods in which instructional designers can use transactive memory theory in designing collaborative learning tasks.
16
(Chen & Dietrich, 2009)
This study takes a transactive memory theory approach to understand the knowledge coordination that happens between distributed open source software teams. The authors surveyed 61 different project teams to understand how TMS’s play into their collaborations.
This was the first study which focused on TMS in OSS teams. Their results showed that TMS played a pivotal role in teams knowledge coordination behaviors. It showed that knowing the location of knowledge that is distributed amongst the members of a team helps the team coordinate and perform better. Additionally this shows, contrary to other papers, that specialization amongst OSS developers has detrimental effects on knowledge coordination, and a few “generalists” are necessary in each group. Finally they found that credibility had no impact on the knowledge coordination of TMS. This may be a cause of the voluntary nature of the job.
(Jackson & Moreland, 2009)
This study moved the transactive memory study into the classroom to understand how it affects group performance in that domain. The main goal of this study was to determine the generizability of the results found in (Michinov & Michinov, 2009) and (Lewis, 2004). This study focused on groups in an undergraduate IS group and their abilities to analyze organizations Information System and propose changes. This study had four stages, and transactive memory was analyzed with surveys between each of the stages.
Once again, the stronger transactive memory systems were associated with better group performance. In this study they showed that communication amongst the group member was the highest predictor of a strong transactive memory system. One of the major contributions of this study that other studies have not done before is they involved a statistical analysis to weed out other possible predictors of performance. Their statistical regression showed that communication amongst group members and instructor differences (indicating specializations) were the only two significant predictors of performance.
Supporting Transactive Memory Systems
Though Transactive Memory is still a very active research area, very little research has
been done to evaluate Transactive Memory for distributed teams. In fact, the majority of current
research in Transactive Memory continues to repeat older studies and find new ways in which
Transactive Memory is used in different face to face settings.
In the research area of distributed teams, many problems begin to become apparent when
you try to draw comparisons between distributed teams and transactive memory. Aside from the
more obvious issues of difficulties of communication and face to face training for distributed
teams, it has been found that trust in distributed teams is fragile, and though is sometimes easy to
form, is even easier to destroy (Iacono & Weisband, 1997; Jarvenpaa, Knoll, & Leidner, 1998;
Jarvenpaa & Leidner, 1999; Kanawattanachai & Yoo, 2002). This lack of trust can interfere with
the formation of group knowledge stock and consensus about the knowledge sources--which can
limit the TMS effectiveness. Other barriers involved in coordination of a distributed team
17
include-- time constraints, technology constraints, leadership constraints, and cultural constraints
(Rosen, et al., 2007). These barriers can result in not being able to form an effective TMS and
failing to meet the dimensions listed above. These barriers, though difficult, are something that
must be looked at in order to find an appropriate solution. With the prevalence of distributed
teams in everyday life, this is an area that is ripe for research to start to address.
Alternate Perspectives on Transactive Memory
Since a good portion of research in the Transactive Memory domain cannot be leveraged
for distributed teams or system design, alternate perspectives on this issue need to examined in
order to move forward. Research in areas such as Computer-Supported Cooperative Work
(CSCW), Human Computer Interaction (HCI), and Human Factors, has often focused on finding
ways to support distributed teams in completing their tasks, and have already addressed many of
the issues discussed above. More specifically, the concepts of Awareness from CSCW, Common
Ground from HCI and Team Mental Models from Human Factors, have several distinct
connections to Transactive Memory, therein, concepts from these areas may be useful to explore
this new and promising area.
Awareness
When people interact in a face-to-face setting they often subconsciously build an
awareness of what is going on in and around their focus. This awareness is built by watching
what people do, listening to what people say and assessing each other’s actions. In the most
traditional sense of the word, awareness is an activity of monitoring others interacting with an
environment that encapsulates them. Building awareness is a very crucial component of any
group activity and is necessary for effective coordination. Awareness allows people to provide a
better context for their own activities, and thus are better able to tailor their actions to meet their
goals.
18
Situation Awareness
Awareness itself can take on many different forms and research areas look at it from
multiple perspectives. One of the more popular forms of awareness that is discussed in research--
Situation Awareness (SA)--is defined as "the perception of elements in the environment within a
volume of time and space, the comprehension of their meaning, and the projection of their status
in the near future" (M. R. Endsley, 1995). This definition can be further broken down into a three
level model:
− Level 1 SA (Perception) – Perceives the status, attributes, and dynamics of relevant elements in a given environment
− Level 2 SA (Comprehension) – Based on the knowledge of Level 1 elements, the decision maker forms a picture of the environment, comprehending the significance of objects and events.
− Level 3 SA (Projection) – Using knowledge of the status, dynamics of the elements and comprehension of the situation making projections of the future actions of the elements in the environment
It has been found that Situation Awareness is a significant contributor to the performance
of individuals and teams. In (Sarter & Woods, 1991) they explored how the design of aviation
controls can impact SA, and either inhibit or benefit the pilot in his/her ability to complete a task.
The aviation domain has also been looked at by (D. G. Jones & Endsley, 1996) where they were
able to show how often and what caused errors at all the levels of SA. This information was able
to inform 5 SA requirements which an aircrew would need to perceive, understand and project
(M. R. Endsley, 1999):
− Geographic SA – location of self, others around you, and points of interest in the environment
− Spatial/Temporal SA – information about your current status − System SA – information about the system you are using in the environment − Environment SA – information about the environment and it’s status − Tactical SA – information about tactical information needed to navigate the
environment
These different types of SA are not only crucial in the aircrew's ability to effectively
perform their tasks and reach their goals, but other types of environments such as emergency
response (Blandford & Wong, 2004; McGrath & McGrath, 2005), C4i systems (French &
Hutchinson, 2002; Salmon, Stanton, Walker, & Green, 2006), and surgical teams (Bardram,
Hansen, & Soegaard, 2006; Hazlehurst, McMullen, & Gorman, 2007).
19
Figure 2: Framework of Individual Situation Awareness, based on (E. Salas, Prince, Baker, &
Shrestha, 1995)
SA can also be considered in relation to a team as a whole rather than just as individuals.
This Team or Group Situation Awareness can be defined as “the sharing of a common
perspective between two or more individuals regarding current environmental events, their
meanings and projected future status” (Wellens, 1993) or “the degree to which every team
member possesses the situation awareness required for his or her responsibilities” (M. R. Endsley,
1989). In order for this to be most effective each team member should monitor part of the
environment, with only enough overlap for coordination. This loose definition was broken down
into a framework (E. Salas, et al., 1995) which used information processing, team processes, pre-
existing knowledge and dispositions, task interdependence/team characteristics and team situation
assessment. This model (Figure 2), though designed initially for an individual’s Situation
Awareness has been used as a basis for numerous (E Salas, Cooke, & Rosen, 2008)team studies
(Burke, Stagl, Salas, Pierce, & Kendall, 2006; Espinosa, Slaughter, Kraut, & Herbsleb, 2007;
Prince & Salas, 2000; Vidulich, Bolia, & Nelson, 2003).
Based on these models of Situation Awareness, a new area of research which focused on
distributed work teams began. Workspace awareness out of the research domain CSCW is a type
of SA where the situation is embedded in the workspace.
20
Workspace Awareness
In order to obtain Workspace Awareness, a person must have an up-to-the-moment
understanding of another person’s interactions within a shared workspace (Gutwin, Greenberg, &
Roseman, 1996). Workspace Awareness is a specialization of SA that is tied to a specific setting
of the shared workspace. Having and maintaining workspace awareness is very important for
virtual teams and in transitioning people back and forth from individual to shared tasks (B. Gaver,
2002).
The research in Workspace Awareness in CSCW is an outgrowth of the seminal paper
(Dourish & Bellotti, 1992) in which the authors developed a shared workspace for collaborative
writing. In this paper the authors developed two mechanisms which have been used to support
workspace awareness-- Informational Feedback—to provide explicit facilities through which
collaborators inform each other of their activities, and Role Restrictive Feedback—where explicit
support is given for specific roles in a collaborative environment. Although both of these
mechanisms had their advantages, the author proposes and provides evidence for a different and
more effective mechanism called Shared Feedback. In the Shared Feedback Approach,
information about individual activities is apparent to other participants by presenting all of the
feedback in a shared view rather than an individual view. This allows people within the
workspace to passively monitor others actions and tailor each individual decision or action they
make accordingly. The main benefit of this approach is tha it assists in collaboration by reducing
the cost of information production and allows individuals to extract on the most relevant
information and browse awareness information about the task at hand. This approach was
implemented in developing a system to assist in distributed group work called PortHoles (Dourish
& Bly, 1992). This study showed that a shared workspace system can lead to more informal
interactions, spontaneous connections, development of shared cultures and a sense of community
which may have not happened without them.
In (Gutwin & Greenberg, 2002), the authors suggest a new framework for workspace
awareness in groupware. The authors suggest that in groupware lack of awareness can cause
major problems and can often hinder the task. Not only is awareness difficult to obtain in many
current groupware systems, but once it is reached, it is even more difficult to maintain. When
working in a groupware system, the input and output device only provides a fraction of the
perceptual information that you take for granted in face-to-face interactions. The total amount of
information generated by these input and output devices is controlled by the system and can often
21
become a limiting factor. In order to try and solve these problems, the authors broke down
workspace awareness into three main levels:
− Part 1 – Awareness of the past (how when, who, where and what) and awareness of the present (who, what where)
− Part 2 – People obtain information that is produced by peoples bodies in the workspace, from workspace artifacts and from conversation and gestures
− Part 3 – Three types of activities, Management of coupling, simplification of verbal communication, coordination, anticipation and assistance.
In order to achieve all three levels, the author proposes that embodiments and expressive
artifacts are integrated into Shared Workspace systems. Embodiments are visible representations
of each person’s bodies and actions within the workspace; these can include telepointers, view
rectangles, avatars or video images. These can be used to provide information about who is in the
workspace, where they are and what they are doing. Expressive artifacts are artifacts that are able
to maximize the amount of usable awareness information produced for the group. Examples of
these are action indicators or action animations (Gutwin & Greenberg, 1998). Expressive artifacts
can be looked at as the opposite of symbolic manipulation techniques (i.e. shortcuts). These
symbolic manipulation techniques, while very powerful for single user systems, often provide
minimal feedback and can cause confusion and effect awareness in multi-user systems (Gutwin &
Greenberg, 2004). These expressive artifacts would be a modification of these shortcuts that
would provide a more detailed feedback expression on actions that the users take.
In order to implement these two types of components (Gutwin & Greenberg, 2004)
propose the use of process feedthrough, action indicators and visibility techniques in order to
make actions in a shared workspace more obvious, distinguishable and more interpretable to
others in the workspace.
Feedthrough and Action Indicators
In a typical single user system, clicking a button or taking an action will provide the user
with feedback--whether it is just a button going down or a state change within the system. This
feedback is a necessary interface element that allows the person to recognize that they have
successfully performed an action and understand the consequences of this action. While feedback
is sufficient in single-user systems, providing this information to only one person in a groupware
system limits the awareness of the other people in the workspace. In order to remedy this,
22
research has proposed the idea of (process) feedthrough (Gutwin & Greenberg, 2002, 2004; J Hill
& Gutwin, 2003), in which the feedback given off by the artifact is not only provided to the
person performing the action, but is also transferred to people around them. By providing this
information to the group, everybody can be aware and cognizant of the actions, successes and
failures of their team member’s actions so they are able to adjust their own actions accordingly
(Dix, Finlay, Abowd, & Beale, 2004). A very basic example of feedthrough would be when a
person clicks a button on their own computer, showing the effects of the button click on the
distributed team members computers also. Other implementations of feedthrough from research
include gradual interface change (Tuddenham & Robinson, 2009), shared annotations (Pinelle,
Gutwin, & Greenberg, 2003) and shared audio notifications (Abrams & Haefner, 1998;
McGookin & Brewster, 2007). In systems in which the collaborators are using significantly
diverging views, feedthrough becomes even more important (Park, Kapoor, & Leigh, 2000).
Animation is a crucial part of properly implementing feedthrough. In a real world
situation, if you are doing a task you see a incremental level of feedback as you progress. It is the
same if you are working with another person, whatever they do, you see small updates as they
progress to their goal. In a computer system, humans expect to see the same thing, and when they
see changes that just appear without anticipation, it can confuse or startle the users (Chang &
Ungar, 1995; Thomas & Calder, 2001). This problem becomes even worse when adding in long
distance internet connections. In these situations, network delay can become an issue and disrupt
people’s abilities to collaborate and see animations and visual feedback in real time. Animation
can be used to make the work that another user is doing visible to everybody else in the
workspace. For example, if you were deleting something from the workspace, and another user
did not see it get deleted, s/he would not know what you did and may not even realize it was
deleted. In this situation you could employ a fading effect so the other users can first notice the
change beginning, and would recognize that it has been deleted by another user. This idea of
tracking changes is referred to as change awareness (Tam & Greenberg, 2006), which is simply
the ability of people to track asynchronous and synchronous changes made in a collaborative
workspace by other people. Research has shown the importance of controlling this change
awareness for family calendars (Neustaedter, Brush, & Greenberg, 2007), workspace file sharing
(Whalen, Toms, & Blustein, 2008), internet wiki’s (Alshattnawi, Canals, & Molli, 2008), and
collaborative 3D environments (Hancock, Miller, Greenberg, & Carpendale, 2006).
Although network connections are getting better and better, the possible animation lag for
visual feedback is still a major problem (especially if you consider domains like emergency
23
response where the connections might be very poor). In order to help this, (Stuckel & Gutwin,
2008) proposed a solution of “local lag”, which synchronizes the visual environments of the
distributed members to prevent one client from getting ahead of another. Not only did participants
not find this distracting or frustrating, but teams who used software with local lag implemented
had improved performance.
A final technique often used as feedthrough or action indication is using sound cues to
indicate actions. The advantage of sound is that even if the object being edited is not in the users
field of vision, they are still able to perceive that it has happened. Additionally, sounds are very
expandable, and can be modified to represent different actions (W. W. Gaver, 1991; J. Hill &
Gutwin, 2004).
Furthermore, sounds can be used to inform people on their own and others interactions,
and has often been found to be faster than visual techniques, in both virtual and real world
situations (Dierker, Mertes, Hermann, Hanheide, & Sagerer, 2009; Strachan, Eslambolchilar,
Murray-Smith, Hughes, & O'Modhrain, 2005; S. Zhao, Dragicevic, Chignell, Balakrishnan, &
Baudisch, 2007). Generally, sounds provide a method of passive awareness to allow users to
monitor their progress. In a study analyzing use of a desktop system and email notifications, the
results showed that sound allowed users to be aware of updates without requiring them to shift
tasks (Iqbal & Horvitz, 2010). While these sounds can sometimes cause disruptions, users prefer
to use them because it allows them to maintain awareness without disrupting their current task
(Nees & Walker, 2009). Despite this research showing the numerous advantages of sound in
awareness, as early as 2007 it has been reported that less than 60% of system designers report
using audio in their software (Frauenberger, Stockman, & Bourguet, 2007). This may be a result
of sound adding an extra layer of complexity to the system, and may not be useful depending on
the context (i.e. if used in a noisy area). While sound has not made its way into normal
collaborative interfaces research in accessibility design has continued to focus on sound as a way
of virtual collaboration for the blind and visually challenged (McGookin & Brewster, 2007;
Sánchez, Baloian, Hassler, & Hoppe, 2003; L. Wang, Roe, Pham, & Tjondronegoro, 2008).
Information Filtration
Providing feedthrough, although necessary, may instigate other problems. Since the
amount of visual notifications begin to multiply as a function of total number of people in the
24
workspace, issues of information overload issues can crop up (Schlichter, Koch, & Bürger, 1998).
In addition, if the feedthrough is done through sound, this problem can be escalated. Also, by
making all of this information visible to everybody, certain privacy issues can arise (Alarcón,
Guerrero, & Pino, 2005; Muller, Smith, Shoher, & Goldberg, 1991).
In order to properly assist feedthrough, the ability to filter information needs to be
provided to the End-Users. Filtration can happen on either the information producer's side or the
information consumer’s side. One way of supporting this is using an outgoing filter (Schlichter, et
al., 1998) which allows users to see what others see of them, and giving users control over what
others can see from them. Additionally, a system should offer an incoming filter which allows
users to choose who and what they receive feedthrough and information for. Another possible
solution to this type of issue is using a temporal model that makes things visible over time
(Alarcón, et al., 2005). This allows the “authors” of the information or feedthrough to take their
time in filtering their information as it has already happened, rather than expecting them to have
had the foresight to filter beforehand. In (Dörner, Pipek, & Won, 2007) they propose three types
of filtration systems to be implemented into awareness systems. The first type of filter, the
privacy filter, allows for users to prevent others from seeing work that they don’t want public or
might not be final yet. The second filter, the organizational filter, allows for implementation of an
organizational policy. The third proposed filter, the interest filter, allows users to filter based on
their current interests or tasks, since not all of the information in the workspace may be relevant.
In (Ignat, Papadopoulou, Oster, & Norrie, 2008), they propose using ghost operations to reduce
these privacy concerns. A ghost operation, would obtain the action from the initial user, and then
provide masks and filters to the piece of information based on their personal privacy settings.
This system allowed for users within the distributed application to still maintain their awareness
of what is going on without breaching peoples wish for personal privacy.
Much of the current work in information filtration in these collaborative environments
focuses on automating it through computer algorithms and machine learning. The most common
example of this type of “intelligent filtration” can be seen in spam filtration systems for email. In
many new web sites this intelligent filtration, rather than being done by a machine, is
crowdsourced out to the community. This type of information filtration is referred to as
collaborative filtration. This type of filtration is a popular feature of new blogging news sites like
Slashdot (http://www.slashdot.com), Lifehacker (http://www.lifehacker.com), and Gizmodo
(http://www.gizmodo.com). In these sites, the readers are able to rate each of the comments that
appear in a specific article, and the ones with the lowest rating are filtered out by the system.
25
Users have the ability to set their filtration settings, but in a study of users habits on these
websites researchers revealed that the vast majority of registered users do not stray from the
default settings (C. A. C. Lampe, Johnston, & Resnick, 2007). In order to create a better
experience for the users, the authors here present two filtration settings to better support users,
while maintaining the spirit of the rating system. They propose adapting the system to weight
comments based on the users tendency (i.e grouping users into comment tolerance thresholds,
then filtering based on that schema), or rather than considering every comment in the system,
consider each thread as an independent entity and apply different filtrations depending on the
activity. This is a very important aspect of awareness, especially in these systems where the
filtration happens behind the scene and is invisible to the unknowing user. Users who do not
apply custom filtration rules are only be aware of a small portion of the conversation. This can
steer their views and opinions on the matters in the direction of the crowdsourced opinion.
Another way of looking at filtration is to dynamically adjust the information intensity
depending on the users set rules and restrictions. Some Internet sites do this by shading the
comments based on ratings rather than eliminating the lower rated ones completely. In (Y. Wang,
Gräther, & Prinz, 2007) the authors propose a dynamic awareness system, which uses a rule-
based inference system to adjust the notification intensity of the awareness information. They
found that by selectively highlighting certain pieces of information, it reduced information
overload, and allowed users to keep track of the relevant information as well as being aware of
the less important information. Additionally they propose that by making dynamic awareness
elements a core component of groupware systems people will be able to collaborate more
efficiently because they have continuous and flexibly.
Visibility Techniques
One very important thing to look at when information is situated in a virtual workspace is
that your field of vision is limited by technical constraints. In a real world setting, you may be
looking at a person or a computer screen, but can see what is going around outside your primary
focus. In a virtual workspace this is not the case--you can only see what you are primarily focused
on at any given point in time. In order to do this, a system needs to employ components which
can assist people in tasks where spatial manipulations and spatial relationships are important. This
can be solved using things such as radar views, mapping techniques and workspace overviews
26
(Gutwin, Roseman, & Greenberg, 1996). These types of components provide users with added
information about their interactions and the state of their actions within the workspace.
Radar views are widgets that provide a miniature overview of the entire workspace
(Gutwin, Greenberg, et al., 1996). They often provide view rectangles, which were initially
presented in (Baecker, Nastos, Posner, & Mawby, 1993), which show the area on the workspace
that each person is currently viewing. They also sometimes provide a notification of where each
person’s pointer is located. In other research, these have been referred to as workspace miniatures
(Gutwin, Roseman, et al., 1996). Not only can these radar views provide information on what
participants are doing within the workspace, they can also provide information on environmental
change. This information can assist in informing users of movement of other users, changes to
artifacts, and what people are doing. In, (Gutwin, Greenberg, et al., 1996), radar views are broken
down into three specific types of radar views:
− Portrait Radar – The simplest of all radar views. Shows each participants view outline and telepointer in a unique color and/or some other type of identification (i.e. a portrait of the person it represents).
− History Radar – This type of radar reports people’s past locations as well as their current location. This allows other users to know not only where people are currently working, but where they have been.
− Heads-Up Radar – Combine the normal and radar view in a heads up display. Often times the workspace miniature would be projected as a back layer or in front with the visibility turned down. This provides workspace awareness without requiring the user to look in other locations.
Depending on the task at hand, a different type of radar might be useful. For example, if
you are working on something that involves an extremely specific process, it is important to know
the exact steps that somebody is taking, it could be necessary to employ history radar. If you are
just worried about knowing where people are at a given point in time, then either Heads-up or
portrait radar might be better. These types were expanded to address some of the major problems
with radar views. The Extended Radar View (Tran, Yang, & Raikundalia, 2006) was made to
differentiate between where the person was viewing and where the person was working. An
example of this is seen in a word processing application, where a person is actually editing one
section, but they may have had to go back to an older section to quickly view what was stated
then.
These radar views can also be used in other means. In (J. M. Carroll, Rosson, Farooq, &
Xiao, 2009), the authors suggest that radar views can inform other users of somebody’s current
information needs, priorities and plans. This means that they do not just provide information on a
27
person’s current state, but also provide new resources, affordances and possibilities for
collaboration and coordination. It must be stressed though that radar views are only beneficial in
spatial tasks and may cause a distraction if tried to use in a non-spatial task (M. A. Nacenta,
Gutwin, Aliakseyeu, & Subramanian, 2009). More recently research has began to explore the use
of radar views in hybrid (physical and distributed) collaborations. These studies, which often
focus on tabletop groupware (Miguel A. Nacenta, Pinelle, Stuckel, & Gutwin, 2007; Reetz,
Gutwin, Stach, Nacenta, & Subramanian, 2006) and mobile devices (Aliakseyeu, Lucero, &
Martens, 2009). These studies show the utility of radar views in new ubiquitous computing
environments.
In a traditional radar view, this would give the impression that they are working where
they are just looking. To fix this, the authors propose using telecarets, which would show the
exact location where the remote authors are working. Telecarets are not only a hot issue in
research, but are also beginning to invade commercial internet applications. In the newer version
of Google Docs (http://docs.Google.com) new collaborative features have been added to show
where each of the collaborators are currently working (Figure 3).
Figure 3: Google Docs Telecarets in a Spreadsheet
Generally telecarets inform collaborators of three things, what is being edited, who is
doing the editing, and what a person’s current focus is. While this information seems like its best
utility would be in collaborative writing or editing software, other researchers have applied it to
more complex domains. The software FASTDash (Biehl, Czerwinski, Smith, & Robertson, 2007)
was designed as a visual environment to help large scale software development teams build better
awareness of the state of the project and their teammates. Since these software development
28
teams often deal with thousands upon thousands of files, simply showing a telecaret on the line
that they are working will not be useful or visible at all. As a fix for that, the system implements a
spatial representation of the codebase using a squarified tree-map and coloring and labeling
techniques. Their solution allows people to see whether a file is opened by somebody else, if it is
being edited by somebody else, if it is checked out, and whether or not any comments have been
made about the file. This system was attached to their coding environment and served as good
platform for increasing their sense of awareness and feelings of stability within the environment.
Another, more sophisticated version of telecarets that are starting to appear in more
collaborative systems are telepointers. These can be a very useful tool in augmenting deictic
references, which is simply pointing to something you are currently speaking about. In (Wong &
Gutwin, 2010) the authors compare the efficacy of collaboration between face to face and virtual
deictic reference. They showed that by supporting natural and augmented pointing and improving
the richness of the collaborative virtual environment, the gap between face to face and virtual
collaborations is a lot more narrow than previously expected. Tracking people’s movements
within a workspace is also a useful way to eliminate ambiguity and reduce the gap between what
is happening on the screen and off-screen activities. In (Fraser, McCarthy, Shaukat, & Smith,
2007), the authors propose to not only track the cursor within the workspace, but add tracking to
the users movements around the remote display and showing to the collaborators. This concept,
called display trajectory can provide remote collaborators with extra information so they can
predict to an extent what their teammates are about to interact with next. This mainly works for
large-screen displays, but could be replaced with shared eye-tracking visuals for smaller display
systems.
Another type of visibility technique that is becoming a major aspect of distributed
systems is avatars. When initially conceived, an avatar was simply a computer user’s
representation of them within a collaborative system. These avatars have become even more
popular in new collaborative systems. In practically any online environment from user forums to
poker room’s, users rely on avatars to represent themselves to the rest of the world. These avatars
can have a tremendous effect on the users sense of presence and the emotional engagement they
feel towards the task or game (Kang, 2006; Vasalou, Joinson, Bänziger, Goldie, & Pitt, 2008).
These avatars can also be used to display other information such as current status, whether the
person is available, and what they are currently doing (Gutwin, et al., 2008). Examples of
different kinds of avatars used on the world-wide-web can be seen in Figure 4.
29
Figure 4: Example of Avatars (in (a) online Poker, (b) online fantasy football, (c) online forums,
and (d) online role-playing game)
These avatars take on an even larger role in new Virtual Worlds (i.e. Second Life, World-
of-Warcraft, ect.). In these systems, rather than being a static pictographic representation of the
user, these avatars become an embodiment of the user themselves, and the characteristics of their
character within the world (Ducheneaut, Wen, Yee, & Wadley, 2009). While these avatars can
help improve people’s immersion into the environment, they must be carefully used, due to their
effects on interpersonal trust (Junglas, Johnson, Steel, Abraham, & Loughlin, 2007). While
violation of trust may not have a large effect when operating in a gaming environment, this is a
very important issue that must be considered when the task is more professionally oriented. This
may be a reason why business or serious oriented systems often rely on simple pictures as their
avatars than more creative expressions like you see in forums or games.
Awareness in Geospatial Tasks
A majority of the traditional awareness literature focuses on applications involving more
traditional work related activities, like writing and editing. While this information still serves as a
useful in other types of tasks, research has begun to try to understand how workspace and
situation awareness applies in Geospatial tasks. In (A. MacEachren, Cai, Brewer, & Chen, 2006)
the authors present a Large-Screen display holding a collaborative map which is designed to help
facilitate situation and workspace awareness. In this paper, they present two systems, HI-SPACE
and DAVE_G, both of which are designed to support co-located and distributed collaborations. In
order to support workspace awareness for the distributed collaboration they implemented a
30
system to transmit and prioritize virtual mouse events, and supporting multiple displays for users.
This is an example of how telepointers can be applicable to Geospatial tasks.
In (Schafer, et al., 2007), the authors present a GeoCollaborative Web-Portal called
BRIDGE. Within this system they outline numerous features with the goal of facilitating
workspace awareness amongst the distributed teams. The key features that were implemented
onto the map were the abilities to share annotations and selections, see as well as lead and follow
other team member’s navigations, and the ability to organize the map data using specialized
views and filtering. Other map features tailored to awareness are sharing telepointer information
and shared view of where people are looking on the map.
Radar views are also another popular feature in GeoCollaboration software. In these
types of systems, radar views are sometimes referred to as overview maps (Peterson, 2009). For
these systems the radar view, rather than being an overview of a document or workspace, is an
overview of the geographic area of interest. These radar views can be enhanced by applying a
fisheye approach to reduce problems with scale and detail changes. This fisheye radar has been
found to improve spatial collaborations and user experience and performance within the systems
(Schafer & Bowman, 2006).
Overview of Awareness
Awareness is one of the biggest issues in Collaborative Software today. Unlike some
research domains, research from awareness is beginning to appear in commercial and web-based
software packages. The main thing in designing systems to support awareness is to make the
things that are taken for granted in everyday tasks more visible in the virtual environment. While
this seems like an easy task, implementation must take account of issues such as privacy,
usability and information overload to name a few. Table 3 provides an overview of the major
issues in awareness covered in this section, and the solutions (design and otherwise) that can be
used to mediate them.
31
Table 3: Overview of Concepts and Solutions in Awareness Literature Concept Description Solutions
Situation Awareness Perception of elements and people within the environment and an understanding of their meaning and impact on the future
Information about self and others geographic location, information about current status, information about tactical information needed for navigating the environment
Workspace Awareness Specialized for of Situation Awareness which involves an awareness and understanding of the status of a shared workspace
Shared Feedback, embodiments, expressive artifacts
Feedthrough Providing feedback information on an action to everybody in a system rather than the person who initiated the action.
Shared annotations, animation, local lag, sonification
Information Filtration Selectively choosing which information a user produces or is exposed to in a shared workspace
Incoming filter, Outgoing filter, Intelligent Filtration, temporal notifications, privacy filter, organizational filter, interest filter, collaborative filtration, Dynamic intensity filtration
Visibility in Groupware Providing information about a users current focal point, attention and what they are currently working on in a shared workspace
Workspace miniature/overview, Radar views, view rectangles, telecarets, spatial telecarets, telepointers, user/eye-tracking, avatars
Geospatial Awareness Awareness when the workspace is situated in a geographical context.
Telepointers, multiple displays, map annotations, selection indicators, navigational indicators, overview map, fisheye radar
Common Ground
Common ground was initially developed based on the idea of grounding in
communication. Common ground can be defined simply as having a mutual knowledge, belief or
assumptions amongst a group of people (H. Clark & Brennan, 1991). This is a very essential
component for any group of people who are looking to effectively coordinate, communicate and
collaborate with each other. By establishing common ground, a group can increase the likelihood
that communication between parties can be understood and properly interpreted (Fussell &
Krauss, 1992; Krauss & Fussell, 1990).
There are two main aspects of common ground – knowing the information yourself, and
the awareness that other people in the group also know that information. Communicating or
32
collaborating without common ground can cause groups to misunderstand what is being said, or
for the person transmitting the information to falsely assume that what they have transmitted was
understood by the group (Blakar, 1985).
In order to form common ground, groups take part in a process called grounding--which
is the collective process where groups try to reach a mutual belief. This process can be effected
by environmental constraints, time pressures, errors, varying trust levels and ignorance, to name a
few (H. H. Clark & Wilkes-Gibbs, 1986; Greenspan, Goldberg, Weimer, & Basso, 2000). In
order to form common ground, different groups often use different techniques that are tailored
specifically to their goals and skill sets. However, it is important to note that techniques that work
over one modality do not always work over other modalities. These techniques, or constraints, are
identified as co-presence, visibility, audibility, co-temporality, simultaneity, sequentiality,
reviewability, and re-visitability. These constraints are fully defined in (Krauss & Fussell, 1990).
Using these constraints, three mechanisms were developed in which common ground can be
established:
− Direct Knowledge – created from firsthand experience with an individual in a particular environment
− Interactional Dynamics – created from interactions with a person in real time − Category Membership – created by making assumptions about others’ knowledge
based on their social categorizations
These mechanisms and constraints are easily met when a group is interacting in a face to
face situation, but do not come as easily in distributed or online interactions. In their study
(Cramton, 2001), observed geographically dispersed teams and how they were able to obtain
mutual knowledge. From the observations, the author identified five types of problems in
obtaining mutual knowledge – failure to communicate and retain contextual information,
unevenly distributed information, difficulty communicating and understanding the salience of
information, difference in speed of access to information, and difficulty interpreting the meaning
of silence. These are organized into two main types of failures—failures of information exchange
and failures of interpretation. The major issue that is identified to have caused these problems is
the lack of timely feedback. These lags disrupt the ability of senders and receivers to establish
common referents, which is a necessary aspect of any mutual knowledge system. Without this
feedback, teams members are forced to speculate why something has or hasn’t happened are more
likely to make negative associations with their partners.
33
In a follow up paper (Cramton, 2002), the author makes suggestions on how to mediate
this problem and achieve common ground/mutual knowledge in dispersed collaborations. She
recommends that team members visit each other’s locations at least once, leaders should establish
communication systems and norms, team members should resist making assumptions about the
situation and constraints of the remote users, all members should be sent the same information,
establish understandings of how remote users will check for and respond to messages, highlight
important portions of messages and analyze operating processes and improve on them when
confusion or problems occur. While these are apt solutions, they are more procedural solutions
and not always possible with fully dispersed teams. If a team is distributed across continents and
always changing members (like is the case with many distributed organizational teams), having
people visit locations is not always possible. In order to help facilitate grounding, more technical
solutions from system design literature should be consulted.
In a heavy collaborative task, two types of common ground emerges, content common
ground and process common ground (G. Convertino, et al., 2008). Content common ground, can
be considered the “know that” while process common ground is “know who”. To test these two
types of common ground, three person teams were formed to work in face to face settings. The
groups worked with paper prototypes (paper maps, post-it-notes, etc.) in an emergency planning
task. Their results showed that process common ground was usually developed early in a
collaborative task and led to more efficient work because the team focuses on action requests and
does not spend as much time focusing on what to do. Additionally when teams form a better
process common ground the need for information querying, strategizing and organization
decreases, which allows the team to focus on their task more.
Common Ground for Virtual Collaboration
In order to address this concern of finding solutions to establish common ground in
distributed teams, researchers have more recently began to look into this area. Often researchers
refer to this area as virtual co-presence, and don’t necessarily focus on the idea of meeting the
requirements listed above, but on finding solutions to establish a feeling of co-location between
people who are actually distributed across the globe. By simulating the things that are often taken
for granted in a face-to-face environment over to a virtual environment, the aspects of common
ground may just come naturally. This idea of co-presence and grounding in virtual environments
34
has been looked at in several domains, from simple shared environments to complex battle
ground simulations.
In a follow up to (G. Convertino, et al., 2008), researchers looked at how process
common ground could be supported in Computer Supported teamwork. Unlike the initial study,
this one had participants use a geo-collaboration software prototype (Gregorio Convertino,
Mentis, Rosson, Slavkovic, & Carroll, 2009). The most interesting portion of this paper was the
design rationale they used to develop the prototype. Based on prior research and common ground
they came to two major design features to implement to improve content common ground:
− Distinction between public and private (role) spaces – this allows for team to coordinate multiple role-specific views of the map, with a shared view that can be used for collaboration.
− Role-Specific Indicators – these communicate individual team member’s actions on a shared map. This also includes the pointers to highlight annotations.
These design choices were made to encourage team members to push information to the
rest of their teammates rather than waiting for information to come to them, to raise awareness of
what information is being shared, and to improve role-specific sharing. Similarly they propose
two design features to implement to help improve process common ground:
− Role-specific indicators of actions – provide a trace of the collaboration that is taking place. These are often visual aids (i.e. color-coding, labels) to support and implicit sharing process.
− Low-cost but explicit method of sharing information – A lightweight method of sharing information gives each member the ability to control their act of sharing and reinforces their goals and responsibilities within the task.
These design choices allow the team members to be aware of everybody since
everybody’s actions are shared. This also makes sure that less time is spent on checking and
management, and more time on judgment and support.
In their research (Kraut, Gergle, & Fussell, 2002) present a system for developing virtual
co-presence using what they call a shared visual workspace. The shared workspace is designed to
meet several goals in establishing co-presence, but some aspects of the system are tailored
specifically to facilitate grounding in communication. They suggest two ways in which a shared
visual space can be helpful for establishing common ground—creating efficient messages and
monitoring comprehension. In a follow up paper (Gergle, Kraut, & Fussell, 2004a) looked at
how in these shared visual spaces, that action and visual feedback can be a replacement for
efficient messages.
35
Creating Efficient Messages
Traditional theory of grounding relies on some level of communication between two
people. While in a face-to-face interaction, this is very simple to accomplish. However, when
moving to the virtual environment, this communication can become clumsy, disruptive, or in
some cases-- impossible (i.e. people who speak different languages). In order to get around this
obstacle, systems need to be tailored specifically to transmit the necessary information for
grounding. Often, it is the case that people only put in the minimal amount of effort they believe
to be necessary. This idea, known as the least collaborative effort (H. H. Clark & Wilkes-Gibbs,
1986) is often further complicated in a distributed collaboration where people have difficulty
understanding at what point is the minimum. This issue of creating effective messages has
become an even larger issue with new communication modalities and channels. Three of the more
popular new communication channels, Twitter (http://www.twitter.com), Facebook
(http://www.facebook.com), and Short Message Service (SMS), all have length caps on their
messages. In systems like Twitter (or other micro-blogging sites), some research argues that by
posting information on what is on your mind, you can increase the awareness that other people
have on your current state, this in turn may help improve common ground (D. Zhao & Rosson,
2009). Systems like Facebook have implemented other measures, such as the profile to have other
ways of forming common ground (C. Lampe, Ellison, & Steinfield, 2007). Other systems which
do not have the massive impact of Twitter and Facebook have had to find other methods of
achieving common ground.
In face-to-face collaborations, the “collaboration effort” is often augmented by awareness
of actions. In order to assist with this, systems need to facilitate and assist in creating more
efficient and effective messages. One possible solution to this, sometimes known as “action as
language” is where the system provides visual information about the teams actions which can be
used for grounding. In order to look at this idea (Gergle, et al., 2004a; Gergle, Kraut, & Fussell,
2004b), created a puzzle system for two players that had a shared virtual workspace. Their study
showed that when presented with a new style of shared visual space, the teams were able to adapt
their communication strategy based on knowing that their partner was able to see what they were
doing. This caused the participants to change their actions to provide evidence for their team
members' comprehension. Additionally, their partners were accepting of the actions they saw on
the shared visual workspace as evidence of shared knowledge. This “action as a language” was
36
able to replace the traditional communication methods and allowed teams who used it to be
become better grounded than other teams.
One of the major causes of poorly designed messages is that often text messages in
virtual worlds are perceived as being ambiguous and too general. When interacting in larger scale
virtual worlds, where there is more detail, these ambiguities can become even more of a problem.
In order to better understand the ambiguity, (Cottone, et al., 2009) conducted an ethnographic
study within a 3D Virtual world on users trying to find places to meet at a specific location. The
study allowed users to use multiple modalities of communication (face-to-face, phone and chat).
The study showed that while strategies like, looking for environmental cues, narrowing the focus
of attention and investing on cooperation were useful in limiting some of the ambiguity, they
were not able to eliminate all of it. This study is interesting because whenever the users were on
task, and talked about specific things to their goals they made progress, but often the verbosity of
their communication channel allowed them to get off task and add details that didn’t matter as
much. A possible outcome of this is if the communication channel limited their ability to discuss
things other than what was useful, their messages may become more effective, they would have
less ambiguity and they could easily form common ground.
In a more complex example focusing in the military domain, (Chuah & Roth, 2003)
developed software to be used as the “Command Post of the Future” called CoMotion (Figure 5).
CoMotion is described as a collaborative visualization environment for creating information
analysis and decision-support applications. CoMotion was designed based on a set of common
ground features that were explored in online social environments. These features were
implemented in software designed to assist command post operators.
37
Figure 5: CoMotion Collaborative Visual Environment for visualizing Common Ground, from
(Chuah & Roth, 2003)
The software was intended so the operators would spend less time transferring
information and trying to understanding each other. It also would assist them in being less
disruptive of each other’s work, and allow them to easily shift attention, tasks and resources to
manage their workload dynamically. To accomplish this, they implemented several visualization
techniques to augment common ground for distributed teams. They allowed users to explicitly
share objects--to show levels of attention people are paying to individual events, to express their
goals quickly and easily, to add interpretations to the interface, and display the history of a given
object, discussion or user. These design principles allowed users to form and maintain common
ground and convey context representing their cognition, goal interpretations, tasks and attention
foci.
Creating effective messages can become even more difficult when different native
languages come into play. In these situations people sometimes rely on mechanical translations,
like Yahoo! Babel Fish (http://babelfish.Yahoo.com) or Google Translate
38
(http://translate.Google.com). This problem was analyzed in a study analyzing three person teams
with language variance (Yamashita, Inaba, Kuzuoka, & Ishida, 2009). By analyzing how
participants from three different language communities took part in referential communication to
complete a task (tanagrams), the authors showed that although the machine translation is a good
method for breaking down language barriers it has numerous drawbacks. They showed that in
order to make use of this machine-translation the speakers must have some level of grounding
between them. Additionally, when using this machine translation, partners had trouble reaching a
mutual level of grounding in order to work effectively together. This study presents an alternate
perspective to the action as language, where the machine generates the language. In this situation
the machine is an intermediary between two people, and in this situation it becomes less effective
in assisting in the grounding process. In a similar study (Jiang & Singley, 2009) propose that
these issues are made worse due to poor awareness and communication bandwidth. To help
reduce these issues and allow bi-lingual teams to better establish common ground the authors
suggest design considerations for task-oriented, document-centric chat programs:
− Direct expression of a small number of mental/conversational states (they propose collabicons, which are simple expressions of their understanding)
− Explicit Task coordination − Integration of the object viewer and the chat window
Though these design considerations were initially targeted as collaborative editing
software, their utility should span into other domains.
Monitoring Comprehension
As with many things stated in this thesis so far, monitoring group member’s
comprehension in a face-to-face conversation is something that is taken for granted. Signs like
speech patterns, body language and direct questions can often give you a good picture of a
person's comprehension (Karabenick, 1996). In distributed systems, these types of
communication do not easily transfer over. Because of this systems need to find new ways to
allow people to monitor each other’s comprehension in a distributed or virtual environment.
One popular solution is making peoples actions and performance visible to the rest of the
group. By providing a visual notification of a partner's performance, other people are given some
knowledge of their level of comprehension on a task level (Kraut, et al., 2002). Going further, if
39
you use a shared visual space, and make people's actions transparent across the system, a group
member can then easily recognize if they comprehend by seeing if they are performing a correct
or incorrect action. By seeing the task in action, the distributed group member gets instant
feedback on whether or not the other person understood a concept or direction. The feedbacks
usefulness degrades with time therefore if there is a slight delay (i.e. a video delay), the efficacy
of the message may degrade.
Since you cannot always count on team members to monitor comprehension, it is
important that the system encourages some level of self comprehension monitoring. In a study
with students in an online learning community (Ingram, 2005), it was found that the students who
took practice in self monitoring strategies on a regular basis became more engaged in the learning
and performed better. In the learning environment, these were done with performance reviews of
work, pre-tests, and working together with other students. These can be generalized to more
generic system constraints by allowing users to review previous actions in the shared workspace,
allowing them to test actions in simulated environments, and by allowing them to review other
peoples actions in the workspace and communicate with them about what they did and what they
thought. In (Leshed, et al., 2009) the authors propose using a visual real-time feedback on the
language use to cause members to reflect upon and improve their collaborative favor. Their
results showed that automated, real time feedback can elicit positive behavioral changes and
performance increases. To visualize the information to the user they implemented simple bar
graphs and visual indicators to inform the users on their collaborative behaviors.
Another proposed method of improving comprehension in a virtual environment is by
using event adaptation (Steiner & Tomkins, 2004). The idea is to adapt the event that needs to be
comprehended based on the user's current state. The authors propose and adaptation manager
which operates on a rule based system that takes relevant information about the event and current
system state into account. The example they provide is that if the user is not currently focused on
the portion of the shared workspace in which the action is not taking place, to hold off on
displaying the action in their workspace until they are looking at that portion of the screen. This
allows the users to be aware of and comprehend what actions have been taken and what events
have happened, but can also create confusion by creating discrepancies in time of changes.
Another, better solution they present is by using widgets on the screen to draw attention to events
that are happening off the screen.
40
Common Operational Picture
A final related concept which has been explored in the military domain is that of the
Common Operational Picture (COP). The COP is often used as a description for a shared display
of relevant information among multiple people. This can be used to facilitate collaborative
planning, and create a common ground of a situation as it unfolds with various constraints
(Brewer & McNeese, 2004). On the battlefield, the Common Operational Picture can be a very
important part of ensuring that Commanders are able to make the best decisions and effectively
communicate and collaborate with their partners. In a review of the COP (M. McNeese, et al.,
2006) provide a conceptual model of the COP based on structure, representation, process and
management.
The majority of the time, when discussing the Common Operational Picture, the literature
is referring to a shared graphic picture of the unfolding battle space to show the location and
actions of everybody along with any other relevant information (Looney, 2001). In order to allow
people to make timely and effective decisions, it is important that the COP is driven by an
effective and accurate information fusion engine.
The biggest problem with current practices in the COP is that they are not designed from
a user-centric view point, and are often driven by the technology and programmers' thoughts and
constraints. By not tailoring specifically to the user, a COP can have the opposite effect of what is
desired and cause goal conflicts, unnecessary workarounds, and decimation of common ground.
This is a very important thing that needs to be considered when trying to obtain common ground.
As stated earlier, when the modality changes, the methods of grounding need to also change.
When moving from system design to system design it is difficult to use generic methods of
obtaining the COP and grounding, and the users must be considered.
The COP can also be looked at as a system attribute. In (Carley & Ren, 2001), they
define the COP as the information which is shared by everybody in a group. This provides a more
info-centric way to look at the idea of obtaining common ground and the COP. Not only do the
users need to be considered when designing the system but also the information that needs to be
transferred.
41
Overview of Common Ground
Common Ground is one of the most discussed issues in any type of collaboration. When
moving to a virtual environment, new opportunities begin to arise. In these types of collaboration,
the system which is used, is not only a modality for communication, but becomes a piece of
common ground between the collaborators. Additionally, the system is able to augment the
interactions to ensure the least-collaborative-effort is not too little, and ensure that proper
grounding does take place. Table 4 provides an overview of the major issues in common ground
covered in this section, and the solutions that can be used to mediate them.
Table 4: Overview of Concepts and Solutions in Common Ground Literature Concept Description Solutions
Common Ground Having a mutual set of knowledge, beliefs or assumptions amongst a group or team
Co-presence, visibility, audibility, co-temporality, simultaneity, sequentiality, reviewability, re-visability
Grounding A collective process in which groups or teams try to reach a mutual belief or understanding of something
timely feedback, process common ground,direct knowledge sharing, interactions, social categorizations, infrequent in person interactions
Virtual Co-Presence Establishing the feelings and affordances of co-location for distributed and virtual teams.
Public and private spaces, role specific indicators, explicit information sharing, shared workspace, message efficiency, monitoring comprehension
Message Efficiency Ensuring that the necessary amount of information is transmitted in order to achieve grounding. This is an important aspect of avoiding the pit-falls of the least collaborative effort.
Status updates, profiles, action as language, specific roles for communication channels to limit ambiguity, explicitly shared objects, attention indicators, shared annotations, history of artifacts and users in workspace, collabicons, explicit task coordination, integration of view and chat.
Comprehension Monitoring Understanding who understands or knows what within a collaborative task.
Performance and action transparency, instant feedback, self monitoring, review previous actions, test actions, visual-real-time feedback, event adaptation
Common operational Picture
A shared display of pieces of information amongst multiple people. This is used to facilitate collaborative planning, and create common ground amongst a team
Large shared display, shared geographic picture, explicit shared relevant information
42
Team Mental Models
Mental models are defined as internal representations of constructs that humans use to
make sense of the world around us (Holyoak, 1984; Johnson-Laird, 1983). These mental models
can not only be looked at on an individual level, but also a team level. A Team Mental Model is a
mental model which is shared amongst team members or a group. More specifically, they have
been defined as “team members’ shared, organized understanding and mental representation of
knowledge about key elements of a teams’ relevant environment” (S. Mohammed & Dumville,
2001). These Team Mental Models have 6 main properties – they are an emergent characteristic
of the group, they represent a tendency of individuals to categorize what they know, they reflect
organized knowledge, they can imply a variety of contexts, they reflect internalized beliefs,
assumptions and perceptions, and they exist to the extent by which they are apprehended by team
members (Klimoski & Mohammed, 1994). Having an effective Team Mental Model positively
impacts the overall performance of the team (Cannon-Bowers & Salas, 2001; Ellis, 2006;
Klimoski & Mohammed, 1994; Lim & Klein, 2006; Mathieu, Heffner, Goodwin, Salas, &
Cannon-Bowers, 2000; S. Mohammed, Ferzandi, & Hamilton, 2010; S Mohammed, Klimoski, &
Rentsch, 2000). Research has shown that Team Mental Models have a major effect on teams
ability to utilize, share and process information (Hsu, Chang, Klein, & Jiang, 2010; Hsu, Parolia,
Jiang, & Klein, 2007; Smith-Jentsch, Cannon-Bowers, Tannenbaum, & Salas, 2008).
Team Mental Models can have the biggest impact on a group in tasks where strategic
decision making is necessary. There are three main functions of a Team Mental Model--
describing, explaining, and predicting (Rouse, Cannon-Bowers, & Salas, 1992):
− Description Function – knowledge of what the system is for and what it looks like − Explanation Function – Statements about how the system works and interprets what the
system is currently doing − Prediction – what the system is likely to do.
The second two functions are usually the most emphasized because the description
function is more likely to relate to the static function of the system itself. It is important that
systems are implemented in ways to maintain these common ties between groups. It is common
for teams to create division of labor that allows them to work independently from each other,
each contributing a part to the whole, with little interaction and information sharing. While this
may be considered, at some level, a form of collaboration, and may get the job done, it has been
found that when teams do this, their Team Mental Model, with time, becomes more and more
43
divergent (Levesque, Wilson, & Wholey, 2001). To help with this divergence, the authors suggest
that clearly structured systems may be able to help to better connect team members who slowly
separate during the length of a project. Additionally, they suggest that if the team has little role
differentiation within the system, they will interact more and develop more shared mental models
in later phases of the project.
Based on a review of the literature of Team Mental Models (S. Mohammed & Dumville,
2001) discussed how a Team Mental Model with distributed task work knowledge, overlapping
team knowledge and a balance of diversity and a consensus of belief structures would result in
better team performance for teams with distinct role structure. In (S. Mohammed, et al., 2010), an
updated version of the literature review presented in, (Klimoski & Mohammed, 1994), the authors
looked at how the concept of Team Mental Models have grown in the past 15 years. They discuss
two main predictors of a Team Mental Model, Team Member Characteristics and Team
Interventions:
- Team Member Characteristics – It has been shown that teams who are made up of members with high levels of experience correlate positively with Team Mental Models and performance. Additionally teams who had a generally higher mental ability resulted in better Team Mental Models
- Team interventions – Team who take part in more team-level interventions like planning, reflexivity, leadership and training result in better Team Mental Models.
In some cases, ad-hoc teams do not have the benefit of perfect team member selection
and do not have the time for training. Therefore, other solutions must be looked into. In a case
study of communication technologies used by virtual teams in a customer support division of a
Fortune 500 company, (Suchan & Hayzak, 2001) showed that when face-to-face meetings and
training were not possible, a shared language and mental model can be build by relying on a
common database that provides all the information pertinent to their task. In this example, the
common language was an integral portion of the virtual team because their team members had a
high degree of expertise variability (i.e. the teams were made up of software engineers,
information network specialists, financial managers, etc.). To help form this common language,
the individuals in the team were not able to communicate directly, so they had to rely on this
shared database repository of information, which enabled them to review the same information
from a common perspective. Another method of helping distributed teams form effective mental
models is to design the team interactions and the system to require the setting of goals and
strategies amongst team members (Powell, et al., 2004). By requiring the team members to
actively think about and discuss their goals and strategies, they are not only taking steps to
44
improve their team performance by having a common goal set, but they will be working towards
building a Team Mental Model.
Mental Model Elicitation
One proposed way of building mental models (for both individuals and teams) is through
expert knowledge elicitation. When building a mental model, a process like knowledge elicitation
can be very helpful because of the nature of an expert’s knowledge. Knowledge which is held in
an individual’s mental model can be very difficult to capture because of its tacit nature. To
capture this knowledge, set processes in knowledge elicitation can be used to understand the full
process. This process which contains three phases--positioning, description and discussion--can
be used to build a mental model of others outside the experts. A full breakdown of the processes:
− Positioning phase – purpose of this page is to establish a context and goals for the description process. This phase has three phases:
o establishing the context o focus on one relationship at a time o Illustrate the method.
− Description Phase – guides person through the sequential development of four different descriptions of the relationship. Each description takes a different form and serves a different purpose in the transformation of knowledge. This stage has four phases:
o visual description o verbal description o textual description o graphic description
− Discussion Phase – seeks to test, understand and improve on the descriptions of the different knowledge pieces. Two phases:
o Examine individual descriptions o Compare Descriptions
This method can also be used amongst experts to investigate the causes of disparities
between team members and to resolve differences in their mental models. The experts that were
used in the study claimed to recognize the potential for this method of knowledge elicitation to
identify their inconsistencies across their mental models and resolve their issues. These results
have been verified in another study, (Madhavan & Grover, 1998) which shows that a Team
Mental Model is directly related to the ability of the team to transfer embedded knowledge into
embodied knowledge. Though these research projects focused on co-located teams, other research
45
has verified these results in regards to a virtual environment with distributed teams (Bitter-
Rijpkema, Martens, & Jochems, 2002; Sarker, Nicholson, & Joshi, 2003; Sarker, Nicholson, &
Joshi, 2005; Scozzi, Crowston, Eseryel, & Li, 2008)
Other elicitation techniques that have been suggested for mental models are Cognitive
Interviewing, Verbal Protocol analysis, Content Analysis, Visual Card Sorting, Repertory grid
technique, casual mapping, pairwise rating methods, and ordered tree technique (Langan-Fox,
Code, & Langfield-Smith, 2000). In fact, it has been claimed that a virtual environment is often
better for knowledge elicitation and building mental models based around them (Desanctis &
Monge, 1999). Though important and necessary, simply eliciting and knowing the information is
not enough. Teams need to develop effective and useful representations of the knowledge in
order to properly use their Team Mental Model (Yoo, 2007).
Another proposed elicitation technique created by (Tan, Wei, Huang, & Ng, 2000),
employed a specific dialogue technique, which helped virtual teams quickly establish a shared
understanding among team members. This technique was broken up into three separate stages--
small talk, infinite container and laser generation. The third stage is where the teams were able to
collaborate to build a Team Mental Model around the collaboration practices the team adopted in
the previous stages.
Perceptual Anchors
When working in a distributed team, perceptual anchors can be used to create a shared
experience of a problem. It has been found that whenever knowledge is perceptually anchored in
the shared environment, teams will use that knowledge for similar tasks of domains without being
prompted. This perceptually anchored knowledge can become the basis for the formation of Team
Mental Models, by encouraging team access and ability to construct shared knowledge (Jefferson,
Ferzandi, & McNeese, 2004).
In the previous study perceptual anchoring was achieved by allowing them to work
together in a JASPER video task rather than individually. Current research in distributed teams
and perceptual anchors has begun to focus on how interface elements and interactions can be used
as perceptual anchors. Perceptual Anchoring has also shown its application in GIS and
Geographic related tasks. In (Tomaszewski, MacEachren, Pezanowski, Liu, & Turton, 2006) the
authors present a Geographic Web Portal to aid in Knowledge Management during Humanitarian
46
relief logistics and GeoCollaboration. In this application the authors track and record other users
map interactions such as panning, zooming, map extent, and annotation to allow the other
collaborators to see where they have gone and what they have done. This information becomes a
perceptual anchor of what everybody has done within the workspace. This functionality was also
implemented in an early demo of applying GIS to the NeoCITIES simulation (Balakrishnan, et
al., 2009). In this study they used map annotations, annotated geo-chat and elementary GIS
analysis features to perceptually anchor distributed users to improve their information complexity
management, sense making strategies and performance.
Boundary Objects
When being used to assist people in bridging areas of expertise to collaborate more
effectively and efficiently, Team Mental Models can become boundary objects (Carlile, 2002).
These boundary objects can be used as information artifacts which can guide the collaboration
within the team. Boundary objects are able to have an effect on group collaboration in three ways
(Phelps & Reddy, 2009):
− Familiarity and trust – boundary objects that are familiar to the team can often play a more significant role within the collaboration. Additionally, the greater trust that the team has in the object, the more likely they are to utilize the object.
− Importance – The boundary object must be important within the aspect of the project. The more relationships the boundary object has with the goals of the project, the more central it becomes
− Control – When there is no clear goals within the project, the boundary object can serve as a determinant as to which pieces of information are considered valuable.
Unlike some of the other concepts in this section, boundary objects are not only the
information or mental models that reside within the team, but can be explicitly designed systems,
and thus become an embodiment of a mental model. In (Lee, 2007) the authors conducted a
thorough literature review to expand on the types of boundary objects, these can be seen in Table
5.
47
Table 5: Types of boundary objects as described in (Lee, 2007), adapted from (Star & Griesemer, 1989)
Boundary Object Description Repositories Piles of objects that are indexed in a standardized fashion.
Examples could be libraries, information databases, or directories
Ideal Type Objects that descript multiple localities, are abstract, vague and adaptable. Examples are diagrams, atlases, maps, etc
Coincident Boundaries Common objects that have similar boundaries but different contents, such as political boundaries
Standardized forms Standardized objects that serve as a method for common and consistent communication.
Prototypes Artifacts and activities that support systematic updating. These are verbal, gestural and virtual representations of a product or process
Intermediary objects Intermediate states of a project. They are representations that include all the outputs of a collaborative transformational process.
Boundary negotiating artifacts Artifacts generated by collaborators to help standardize their processes, objects and collaborations.
In some situations boundary objects are viewed as non-tangible things and/or physical
objects, but current research has shown how information technology can be implemented to
function as boundary objects (Forgues, Koskela, & Lejeune, 2009). These boundary objects can
be explicitly designed into systems to assist in bridging expertise differences and facilitate the
collaborative learning of the users in the system.
Using this boundary object theory, (Da Yang, Koolmanojwong, Brown, & Boehm, 2008)
developed a collaborative Wiki for requirements negation amongst developer teams
(WikiWinWin). In this system they implemented an “win condition” taxonomy and a general
database for communication to function as boundary objects between the negotiating party. These
helped facilitate the collaborations by creating a shared language between the users within the
system.
When dealing with distributed teams, boundaries are created, not only by personal
differences and expertises, but by the geographic disparity that occurs between the team. In these
situations the management of knowledge across the boundary becomes an important aspect of the
collaboration. These boundary objects must then serve three purposes within the system,
information transformation, translation, and transfer (Carlile, 2004). Based on this a framework
and four characteristics were developed for boundary spanning.
48
Figure 6: 3-T Framework for Pragmatic Boundary Objects, based on (Carlile, 2004)
The process shown in Figure 6 explains the necessary aspects required to support and
form a pragmatic boundary. Each of the numbers on the diagram represents a unique
characteristic of solidifying the boundary (which can operate as a Team Mental Model). The first
characteristic (1) requires the development of a common language for transferring domain
specific knowledge. The second characteristic (2) involves developing common meanings to
assist in translating the domain-specific knowledge. The third characteristic (3) establishes
common interests to assist in making trade-offs and transforming domain specific knowledge.
Finally, the fourth characteristic (4) supports an iterative approach, where the collaborators
develop a common knowledge for sharing and assessing each others’ knowledge. These are
important aspects to keep in mind when designing systems that will be used as augmented
boundary objects. The system must support developing a common language, meanings, and
knowledge to allow for better sharing and assessment of the mental model.
Overview of Team Mental Models
When interacting in a team, whether it is distributed or collocated, the efficacy of your
collaboration is determined by the similarity of the team member’s mental models. Developing
49
and taking advantage of a Team Mental Model is an extremely important task when
collaborating, especially when decision making is involved. Whenever teams are operating in
distributed or virtual environments, it is imperative that there is a Team Mental Model; else they
may end up working against each other. Table 6 provides an overview of the major issues in
Team Mental Models covered in this section, and the solutions that can be used to mediate them.
Table 6: Overview of Concepts and Solutions in Team Mental Models Literature Concept Description Solutions
Team Mental Models A mental model which is shared amongst team members or a group. This becomes the teams shared understanding and representation about the relevant knowledge of the task and environment
Limiting division of labor, information sharing, clearly structured systems, little role differentiation within the system, team interventions, shared language or information repository, common goal set
Mental Model Elicitation Eliciting the knowledge of people (or system) that have an expert level understanding of a task or domain. This allows their mental models to be captured and shared with other people
Knowledge elicitation, knowledge repositories, specific dialogue techniques
Perceptual Anchors Objects or artifacts that are used to create a shared experience of a problem. This can be the basis of the formation of a Team Mental Models
Action tracking/recording, annotations, annotated geo-chat, shared video and audio
Boundary Objects Information artifacts which can guide the collaboration of a team. Can be used to assist people in bridging areas of expertise to better collaboration amongst a distributed team.
Familiarity and trust, Repositories, prototypes, standardized forms, intermediary objects, boundary negotiating objects, language taxonomies (common language, meanings, interests),
Discussion
These last three sections of the paper have presented several concepts that have
overlapped with each other. Additionally, they have all been shown to be an integral part of
improving team performance--especially in distributed collaborations. All of these concepts also
draw parallels with the dimensions of transactive memory as presented by (Austin, 2003). These
concepts can be tied together to gain a better and deeper understanding of how a system can
utilize ideas from Awareness, Common Ground and Team Mental Models in order to help teams
develop an effective transactive memory system. The following tables show how these concepts
50
can be used in assisting and augmenting Group Knowledge Stock (Table 7), Consensus about
Knowledge Sources (Table 8), Specialization of Expertise (Table 9), and Accuracy of Knowledge
Identification (Table 10).
Table 7: Applying concepts to Group Knowledge Stock Domain Solution Explanation
Awareness Shared Workspace When all users interact within the same workspace, the information of the workspace itself becomes a shared repository of information.
Workspace Overview Using a radar view to inform everybody on who is working on what and what they are viewing.
Team Mental Models
Knowledge Elicitation Using knowledge elicitation methods to create a shared repository of information will allow information sharing and a shared understanding of what information and knowledge resides within the workspace
Shared Database Repositories
These enable all members to review the same information from a common perspective. It will allow them to build an effective Team Mental Model and be aware of the group knowledge stock.
Perceptual Anchor Like a shared workspace a perceptual anchor can help build a Team Mental Model and encourage knowledge sharing to form the group knowledge stock.
Common Ground Message Efficiency In order to properly share knowledge, messages need to be effectively transmitted across the communication channel. This can be augmented by action as language and other things such as profiles or status updates.
Table 8: Applying concepts to Consensus about Knowledge Sources Domain Solution Explanation
Team Mental Models
Team Mental Model When a team has an effective and complete Team Mental Model, there is a consensus about how the knowledge is distributed in the group. This can also mediate trust factors about who knows what
Awareness Shared Feedback If everybody is receiving the same information and is aware of it they are able to form a consensus. This is also helpful in reducing the cost of information production and allowing individuals to extract only the most useful information.
Workspace Embodiment These provide visible representations of what each person is doing in the workspace. This allows people to understand where people are getting their information from and agree upon that.
Common Ground Grounding Coming to a consensus about knowledge source can be a form of grounding. Considering the constraints of grounding will be useful in this step.
Common Operational Picture
By providing all users with a common space that presents the information then there is instant consensus about where the information is located and coming from
Comprehension Monitoring
In order for there to be a consensus on the knowledge users within the collaboration must comprehend the information and be aware of the comprehension of the others
51
Table 9: Applying concepts to Specialization of Expertise Domain Solution Explanation Awareness Expressive Artifacts Things such as action indicators for specific objects allows people in the
workspace to be aware who is working on what, which will provide information about who is able to do what. This can lead to people creating an internal model about who has what specialties.
Visibility When people are able to visibly see what everybody is viewing they can build an idea of who knows what, and what each other’s specialties are.
Team Mental Models
Boundary Objects When there is high levels of specialization it is necessary that boundary objects are used to ensure collaboration
Knowledge Elicitation In order to identify the specializations of information within the workspace, you must first have an understanding of who possesses what knowledge
Common Ground Comprehension Monitoring
When certain people have certain specialties, it is imperative that they are able to monitor the comprehension of the people who do not specialize in that area.
Table 10: Applying concepts to Accuracy of Knowledge Identification Domain Solution Explanation
Awareness Feedthrough During self reports online it is difficult for a person to for trust with the information they are being provided, especially if they never met them before. Using feedthrough the system will provide the information about people’s successes and failures. Since the system provides exact reports of what happened
Information Filtration Large amounts of information all being presented at once without filtration may cause information overload and cause the users to reject the features of the system.
Expressive Artifacts When dealing with a large workspace it is often the case that information and changes are displayed off screen. By making more dynamic and expressive artifacts people can be aware to new objects or changes that have taken place outside of their field of vision.
Common Ground Efficient Messages Using action as language to make messages efficient will allow people to see what everybody is doing, and use their actions as a basis to judge the accuracy of peoples understandings.
Comprehension Monitoring
In order to make sure the information that is being generated by people in the workspace is accurate some level of comprehension monitoring is necessary. If people misperceive others as comprehending, then take their information as the truth, the groups performance will be affective.
Team Mental Models
Boundary Objects In situations of group collaborations, the accurate knowledge can become the boundary object of their collaboration. Also, things like standardized forms, which elicit the accurate information may also be used.
Knowledge Elicitation Knowledge elicitation is an effective way of compiling accurate knowledge of what the group knows.
Obviously there is some overlap in these dimensions as well as interface elements that
may be able to solve multiple dimensions that became apparent. These “solutions” to the
dimensions of Transactive Memory can be used to develop interface components and full systems
that can help distributed teams form and utilize more effective Transactive Memory Systems, and
thus improve their performance.
Chapter 3
GeoTMS Technical Overview
Objectives
There is a clear need to research ways in which transactive memory can be utilized to
assist distributed teams in their collaborations. Though there is extensive research justifying the
necessity of a strong TMS within a team, very little research has looked at how this could be
adapted to and leveraged for distributed teams. Previous systems which explored the idea of
augmenting a TMS with a computer system, such as EWall (Keel, 2007), SLATE (John & Dister,
2009; John & Smallman, 2008), and The NeoCITIES Transactive Memory System (Adibhatla,
Shapiro, & McNeese, 2009) provide potential solutions to this problem, but they have little
empirical evidence that shows their benefits. Additionally, other researchers have approached this
issue as more of a procedural problem then a technological problem (Rosen, et al., 2007). In this
type of research they try to identify some “Best Practices” that may assist in forming a TMS for
distributed teams. While these procedures may be valid in some situations, they have not been
verified, and still continue to push for face-to-face interactions and other requirements which may
not always be feasible.
In an attempt to explore the idea of augmenting transactive memory in a computer system
“The GeoTMS Interaction Environment”, a generic front end for distributed teams in a Geo-
Collaborative task was developed. By utilizing the research outlined in the prior chapter,
solutions for augmenting the four dimensions of a TMS as proposed by (Austin, 2003; Moreland,
1999) were developed. This environment may not only be useful in better supporting transactive
memory for distributed teams in a virtual environment, but could also provide us with a more
controlled environment for analyzing the formation and effects of transactive memory on
teamwork and collaboration.
In the following sections, the research discussed in the prior chapter is tied together into a
set of concrete Interface Requirements. Based on these requirements, details on The GeoTMS
Interaction Environment will be presented.
53
Interface Requirements
Based on the research discussed in the prior chapter and the dimensions of transactive
memory as defined by (Austin, 2003; Moreland, 1999), several interface requirements become
apparent. In Table 11, a set of 7 interface requirements are proposed, as well as their links to the
dimensions of transactive memory and relevant literature. These interface requirements were
framed to be a very general set of heuristics that can be easily applied to a multitude of systems. .
Table 11: Interface Requirements for Transactive Memory System IR Requirement Dimension(s) Fulfilled 1 Shared view of the current workspace and environment 1, 2 2 Awareness of others focus (where they are looking and
where they working) within the workspace 1 ,2, 3
3 Feedthrough on others actions within the workspace 2 , 4 4 Notification on others attention and actions within the
workspace 3
5 Shared information repository 1, 2 6 Shared view of activities within the workspace 2, 3, 4 7 On demand information filtration 4 8 Awareness of Everything going on in the workspace 3, 4
In the following sections, a new interaction environment which uses these interface
requirements as its basis will be presented. It is important to note that this system is only one
interpretation of these requirements and that they can be used in several different ways to
accomplish several different goals
GeoTMS Overview
Information Hierarchy
Before programming began on the GeoTMS Interaction Environment, a simple
information hierarchy had to be developed that was effective enough to use, but simple enough
that it could be applied to different systems with ease. Since GeoTMS is primarily a Geo-
Collaborative system which usually relies on some sort of spatial data, the focal point of all the
information will be locations on a map which may represent a place or an event happening at that
54
location. For the purposes of this system the spatial data will be stored as a set of latitude and
longitude coordinates.
For each event it can be assumed that there is anywhere from one to an infinite number of
pieces of information about why that location or event is of interest the user. Not only can there
be an infinite number of information pieces, but they can be coming from several different
sources over several different modalities. An example of this could be a traffic accident at an
intersection. At an Emergency Dispatch Center, they may receive calls from on scene police
officers, bystanders who saw the situation, plus they may be receiving some sort of imagery or
video from a traffic camera at the intersection. Each of these would represent a different piece of
information with three different sources (police, bystanders and traffic camera) and two different
modalities (text/audio and image/video). A visualization of this information hierarchy can be seen
in Figure 7.
Figure 7: Visualization of Information Hierarchy
This basic structure was used as a basis for developing the data handling mechanisms of
the interaction environment, and also helped motivate the design choices for the interface
presentation.
55
Implementation
Technical Details
GeoTMS was fully implemented over a 2 month period using Adobe and Google Web
2.0 Technologies. The system was programmed primarily using Adobe Flex and ActionScript to
implement a web-based interface running on the Adobe Flash platform.
Since the system is web-based, it can be easily deployed on multiple servers and accessed
by multiple computers in varying locations. The only requirement is that the server computer has
an Apache Tomcat 6.0 web server and a proper BlazeDS configuration. To run the software on
the client computers, all that is needed is a current version of Adobe Flash and a compatible web
browser.
GeoTMS was designed to use Adobe BlazeDS as its primary communication channel but
could easily be modified to other types of communication systems. BlazeDS is an open source
project which allows flash applications on one computer to send information over the internet to
another flash application on a different computer. This allows the system to easily send
information about the user’s current focus in the workspace, actions or messages to the other
users they are collaborating with.
In order to implement the mapping elements of the interface, the Google Maps Flex
Toolkit was used. The Google Maps Flex Toolkit provides flex with all the necessary
Geographical information to display a full map, as well as the ability to plot information on the
map based on a set of latitude and longitude coordinates. The Google Maps Flex Toolkit is in its
infancy and has only been around for less than two years therefore many of the components
utilizing the maps in GeoTMS were built from the ground up.
Using these technologies, the following system provides a test bed for helping (1)
Distributed teams better collaborate and form Transactive Memory Systems in a Geospatial
workspace and (2) researchers better understand and visualize how these Transactive Memory
Systems are formed and utilized.
56
System Architecture
Since the goal of GeoTMS is for it to serve as a front-end to multiple types of systems
and data streams, it was designed independently from any sort of backend. The goal was to
provide a functional and verbose front-end which could be easily adapted to support a multitude
of back end systems.
In order to accomplish this, GeoTMS was implemented using a modified Model-View-
Controller architecture. In this architecture part of the view and controller was “outsourced” to
the external system. The overall system design and interactions can be seen in Figure 8.
Figure 8: Cross Functional Flow Chart Representation of System Layout
GeoTMS Interaction Environment
Exte
rnal
Syste
mCo
ntro
ller
View
Mod
el
External System Controller
External Controller Main Controller
Information Model
Event Model
External Model
Environment View
Information View
Team Model
Team View
Unique Views
Communication controller
57
Since this was designed to be an add-on to future or current systems, rather than clearly
defining the information within the models, they were designed to hold information as generic
“objects”. In an object-oriented language a generic object is an un-defined piece of information,
which can be later assigned a type and manipulated.
Casting and handling of this generic information will be the responsibility of programmer
defined views (Unique Views in the External System section of Figure 8). The controllers are
responsible for communicating information within the system and relaying information to and
from the external system. In the following sections, the Model, View and Controller will be more
properly defined.
Back-End
Model
The models are made up three value objects (VOs) – the Information VO, the Event VO,
and the Team VO. These objects are technical implementations of the information hierarchy as
outlined in the section, Information Hierarchy. A visual display of the VOs and their interactions
can be seen in Figure 9.
Figure 9: Class Diagram of Model Structure
+InfoKey (PK) : int-UniqueVew : object-InfoElement : object
InformationVO -EventKey (PK) : int-EventObject : object-UniqueView : object-EventName : string-lat : decimal-lng : double
EventVO
-Related To
1 0..*
-Team Key (PK) : int-TeamMember : object-UniqueView : object-Name : string-ActionList : string-StatusList : string
TeamVO
-AssignedTo
0..*
0..*
-Reacts To
0..*
0..*
58
The Event VO represents an array of common information threads or pieces happening at
a specific location. This can be an actual event that is happening (i.e. a robbery at a store), or a
discussion about a specific location (i.e. opinions on adding a new traffic light to an intersection).
Each event typically has a spatial information (a location stored as latitude and longitude), a name
or identifier, a Unique View (discussed later in this chapter), and a set of information elements
about that specific event.
The Information VO contains an information element about a specific event. The
information can be something that is explicitly expressed by the person in the team (i.e.
annotations), information provided by the system as a result of a person’s actions (i.e.
feedthough), or information defined by an external system (i.e. event description, Twitter feeds,
news feeds, pictures etc.).
The Team VO stores the relevant information about the team members who are
interacting within the workspace. Each team member is assigned a name and a unique view.
Additionally, each team member has a list of all of the actions they have completed within the
workspace (i.e. viewing a specific event) and any status updates that they have posted.
One common thing you will witness in each of the VOs is the generic objects
(TeamMember in TeamVO, InfoElement in InformationVO, and EventObject in EventVO).
These objects represent how each of these VO’s is defined in the external system. These objects
will contain more specific information tailored to the tasks and goals associated with the external
system. It is the responsibilities of the UniqueViews in each of the VOs to interpret these generic
objects and display them accordingly. These UniqueViews will be discussed in more detail in the
Interface section.
Since Adobe Flex utilizes a system whereby variables are able to be easily bounded,
whenever a change is made to one of the “generic objects” in the external system, the same
change will be apparent in the object help inside the VO.
Controller
The controller is broken up into three primary components--the main controller, the
communication controller and the external controller. The external controller, which may need to
be adapted by the programmer, handles the communication of information and data between this
system and the external system. Its primary responsibility is to populate the models with the
59
correct information and objects so that they can be manipulated by the system. The second
controller – the main controller-- is responsible for all communication that will happen within the
system on one computer. This controller is responsible for taking user actions and updating the
views for all of the team members accordingly. The main controller will remain constant and will
need no modifying for different implementations of the system. The final controller--the
communication controller-- interacts with the Adobe BlazeDS server and transfers information
from one computer to another. This controller both listens for information coming in from other
computers and pushes information to the other computers. When it receives information, it passes
information to the main controller which then updates the interface accordingly.
Interface
The interface is designed to center around the map area. This map serves as the main
“workspace” in which the team will operate within. Though the map is the most highlighted
portion of the interface, it is actually broken up into five major components--the Map
(Environment) View, the Overview Map View, the Team View, the Information View, and
Unique Views.
Figure 10 shows a full screenshot of what the interface might look like to the end-user
(Note: Interface displayed is tailored specifically to the NeoCITIES Simulation Engine).
60
Figure 10: Screenshot of interface
The following sections describe each of the views and explain their functionalities.
Map Views
The GeoTMS Interaction Environment Mapping Views is made up of two separate
components, the main map view and the overview map view. This layout allows the system to
utilize a modified Extended-Heads up Radar View (Gutwin, Greenberg, et al., 1996; Tran, et al.,
2006). This system allows the users to know both where their partners are viewing and where
they are currently working.
Both maps were implemented using the Google Maps Flex Toolkit and currently use only
the default map view but can easily be modified to show satellite imagery, topographic
information, or other custom overlays.
61
Main Map View
The Map View (B in Figure 10) is designed as the main workspace for this system. The
map allows the users to analyze the events in relation to their geographic locations, and knows
which events other people within the system are working on or viewing. The map itself is a
shared view of the workspace (IR-1 in Table 11) and a Shared Information Repository (IR-5 in
Table 11). Currently, the map only displays the default Google Maps Style map, but it can be
easily modified to show terrain or satellite images.
Each event is represented by a simple marker on the map. These markers are the same
exact ones used in common online maps. Each marker has two main functionalities. Whenever a
marker is single clicked, the system will toggle the information shown in the Information View to
the left of the map and assigns you as a “current viewer” of the event and logs that action.
Whenever you are marked the “current viewer” of an event, an attention circle with your
designated color surrounds it and appears to all users (seen in Figure 11). This serves as a heads-
up notification of where the person is currently working
When a marker is double clicked, the unique view assigned to that event is displayed in a
pop up on the map. Each marker has a color assigned to it which allows it to display relevant
information about the event with a quick glance. For the purposes of this implementation, the
color represents the total amount of damage done at that specific event. Also, if you highlight
over one of the marker, a tooltip popup--with a brief label about the event-- appears.
Another interesting functionality of the map view is the “Attention Circles”. These are
color coded notifications of what each team mate is currently viewing or working on. These
serve as an aid to increase awareness of team members' focus (IR-2 Table 11), feedthrough on
their actions (IR-3 in Table 11) and Notifications on others' attention and actions (IR-4 in Table
11).
Figure 11: Attention Circle showing the “Red Teams” current focus
62
A final functionality which was added to the map view was the MarkerTrackers. One
problem that occurs when dealing with geospatial data on a map is when new information appears
outside the current focus of the user. One solution to address this is to “zoom-to-fit” the map
around the event. However, when the map zooms out, it starts to lose some of the more granular
details that may assist in the task. Additionally constantly moving around and changing the scope
of the map could confuse and annoy the users causing a degradation in their performance.
As a solution to this, an open source add-on to the Google Maps Flex Toolkit called
MarkerTracker was implemented. As seen in Figure 12-- whenever a marker appears off screen, a
popup with an arrow pointing in its direction appears on one of the edges of the screen. The user
can either manually move the focus of the map over to its location or simply click the marker icon
and it will auto-scroll to its location. This expressive artifact provides the users with a constant
awareness of what is going on off screen and ensures that they do not ignore certain events just
because they are out of their field of vision (IR-8 in Table 11).
Figure 12: A marker tracker signaling that there is an event taking place out of the current field of vision
Overview Map View
The Overview Map View (C in Figure 10)--which is an extension of the map view--is a
visualization that allows the user to know where each of the teammates are currently focused, and
where they are focused in relation to this (IR-2 in Table 11). The location information updates in
real time as each user navigates the work space. Each of the rectangles represents a different user
and are color-coded accordingly. As each team member moves around the workspace, the zoom
level on the overview map adjusts so that all view rectangles are always visible. This display is
based off of the portrait radar views as described in (Gutwin, Greenberg, et al., 1996).
The only interaction available to the Overview Map is the zoom bar--which is located
immediately below the map. Using this, you can adjust your zoom level-- which will change the
main map and your view rectangle in the overview map.
63
Team View
The Team View (D in Figure 10) allows the user to see a detailed description of who are
currently in the workspace, what they are doing, what they have done. Also, it allows the users to
transfer information about their actions. The first thing to notice is that for each person in the
workspace, there are two separate views-- the compact Team view and the Full team view.
Figure 13: The Team View, with (A) Compact View and (B) Full View
The compact view (A in Figure 13) is a simple overview of who the person is and what
they have been doing. The two main information displays are the person’s status, and their
actions. Their status is something that each team member is responsible for setting as they interact
within the workspace. This serves as a very simple and asynchronous method of communication
which allows the users to only get the necessary information from each other.
64
At the very top of the team view, you can see a text box--which displays their most
current status. In addition, there is a button which allows them to update this to something new.
The status message in the Compact View displays the most recent status message that user has
posted. The action message is something that the system automatically populates based on their
actions within the workspace. The system currently only logs whenever the user views an event
(click the marker on the map). However, programmers can easily add other indicators for the
actions. The final aspect of the Compact view is the checkbox located in the upper right hand
corner. This allows the users to selectively filter information based on who they are interested in
(IR-7 in Table 11). Whenever you un-check the checkbox, the users view rectangle is no longer
shown on the overview map and neither are their Attention Circles shown on the main map. This
feature would be especially useful whenever the total number of users in the workspace becomes
larger.
The Full View (B in Figure 13) displays more information about each user and
information collected over time. The main portion of the Full view is the Status and Action Lists.
These act exactly the same as the action and status notifications in the compact view yet display
all the updates that have been collected over time. These operate like as a stack data structure
where the ones on the top are the most recent and the ones on the bottom are the oldest. There is
also a button (Focus In) which allows users to adjust their view to be the exact same as one or
more of the other users in the workspace. The second part of the Full View is the Unique View
(B-1 in Figure 13). This is the view which is defined by the external system and displays more
relevant information about the task at hand. The display above is an example when the system is
integrated into NeoCITIES in which it displays the available resources for each player. If no
Unique View is defined then this area would be empty.
Information View
The final view--the Information View (A in Figure 10)--is responsible for displaying the
relevant information to the Event that the user is currently viewing. In Figure 10, the boxes
displayed are Unique Views designed specifically for NeoCITIES (more on these in Chapter 4
Methods and Materials). This view changes every time the user selects a new event on the Map
View.
65
The majority of the interactions and complexity for this view will come from the Unique
Views which are designed specifically to fit the external system.
Unique Views
Unique Views--although not technical portions of the GeoTMS Interaction Environment-
- are a crucial part of its functionality. Since the goal of this project was to design a generic front
end that could be used in different systems with little effort, it was critical for the system to
display different types of information in different ways. One possible solution was to assume that
all information was a type of string thus displaying it as text. However, this assumption would
limit the system by not supporting the more rich and complex information.
In order to work around this, GeoTMS allows programmers to develop how they want to
present the information external to the system. These Unique Views are Flex Display objects
(technically referred to as MXML files) which accept the Value Object of Interest (i.e. Team VO
for a team member unique view, Information VO for an information element unique view, etc.),
and then allows the programmers to cast the generic object and display the information
accordingly (for an example Unique View please see Appendix E
Unique View Example). Since these Unique Views are basic MXML files, the programmer is
able to customize how the information is displayed and add user interactions to the external
system.
Integration to External Systems
As stated several times throughout this chapter, the GeoTMS Interaction Environment
was designed to be easily integrated into several different systems. The majority of the work for
doing this comes with the programming of the unique views. The number of necessary unique
views will depend on the external system and how many different types of information it handles.
The External Controller is programmed so that by using simple function calls, you can easily pass
all necessary information into the system.
66
Additionally, the interface is programmed to easily add extra views to the interface. This
can be seen in Figure 10 in the unlabeled box under B, which is an interface component defined
by the external system which is plugged into the GeoTMS Interaction Environment. This process
will be further described in Chapter 4 where the integration of GeoTMS to the NeoCITIES
Simulation Engine is discussed.
Chapter 4
Methods and Materials
NeoCITIES
In order to test the effectiveness and usability of the front-end GeoTMS Interaction
Environment front end, the system must be attached to an external system. For the purposes of
this study, The GeoTMS Interaction Environment was tailored to be a new and improved front
end for the NeoCities 3.0 Simulation Engine (Hellar & Hall, 2009). NeoCITIES is a human-in-
the-loop crisis management simulatino which was designed as a successor to the CITIES game
(Wellens & Ergener, 1988).
History of NeoCITIES
The NeoCITIES simulation has gone through several iterations over the past decade and
has been used to study several different theories-- all of which have been developed at The
Pennsylvania State University.
NeoCITIES was developed based on the Living Lab Framework (M. D. McNeese,
Perusich, & Rentsch, 2000). The initial designs were informed by an ethnographic study and
knowledge elicitation in emergency 911 dispatch centers (Terrell, 2006; Terrell, McNeese, &
Jefferson, 2004). This research was used in the development of the NeoCITIES 1.0 resource
allocation task (M. D. McNeese, et al., 2005). This simulation was later used as a test-bed for
studying intelligent group interfaces, information overload, and team communications. Based on
the experiences with the first NeoCITIES, NeoCITIES 2.0 was developed with a focus on how
geo-collaborative tools can impact and improve team collaboration (Balakrishnan, et al., 2009).
The newest iteration of NeoCITIES (3.0) was built using Web 2.0 technologies and was
designed to be more flexible and distributable (Hellar & Hall, 2009). The initial research using
NeoCITIES 3.0 was focused on how Information Overload affects team performance and how
interface artifacts can mitigate its effects. The major utility from NeoCITIES 3.0 came from it's
68
distributed design nature. Rather than using one application to house the interface and the
simulation engine, it was separated into two separate applications, the interface front end and the
Simulation Engine Back End.
NeoCITIES 3.0 Overview
In the NeoCITIES simulations, participants play the role of Fire, Police or Hazmat
dispatch officers. They are responsible for identifying events, and decisions on an appropriate
response and coordinating their actions with their teammates (M. D. McNeese, et al., 2005). In
the simulation, each participant plays from a personal terminal and monitors their actions. Each
participant is not able to view their teammate's screen or talk with them directly, thus simulating a
distributed team. Each terminal communicates with the main web server which runs the
simulation and informs it on the user’s actions and makes the necessary changes (This
architecture can be seen in Figure 14).
Figure 14: NeoCITIES System Architecture Diagram. From (Hellar & Hall, 2009)
Each team has its own set of resources, which each have specific tasks that they are able
to do. As time passes, the Simulation Engine passes events to the player terminals, and each
player is responsible to dispatch their available resource to try to solve the events.
Since NeoCITIES 3.0 was initially developed as a test bed for information overload, it’s
interface and design was very minimalist, and it did not make use of a map (which all previous
versions of NeoCITIES did). The full interface can be seen in Figure 15.
69
Figure 15: NeoCITIES 3.0 User Interface
The interface is made up of 5 components. The chat component (A in Figure 15), serves
as an asynchronous communication protocol where participants are able to communicate and
coordinate their actions. The dispatch window (B in Figure 15) presents the participants with the
resources which are assigned to their roles, and allows them to dispatch their resources to events.
The Team Monitor (C in Figure 15), provides information on what each team member is currently
doing, and the resources they currently have available. The unit monitor (D in Figure 15) allows
participants to check on the progress of the units they have dispatched to events. Here they will
see if the units they assigned were successful in being able to solve the event and if not they are
able to recall the units. Finally, the event tracker (E in Figure 15) displays a list of the events that
are currently active and their descriptions and current status.
Although the initial design for NeoCITIES 3.0 have served as a good vehicle for testing
information overload, Team Mental Models, situation awareness and information sharing, the
overall simplicity limits how far it can be pushed and what theories it can test.
Integrating GeoTMS to NeoCITIES
Since the general information structure of events in NeoCITIES is very similar to the
GeoTMS information structure, integrating the two was very simple. Overall, the process took
70
around 2-3 Hours, and the simplicity was a result of the flexible design of the GeoTMS
Interaction Environment. For a full image of the implementation see Figure 16.
Figure 16: GeoTMS fully integrated into NeoCITIES
In order to transform the GeoTMS Interaction Environment into a new frontend for the
NeoCITIES Simulation Engine, six Unique Views had to be created, and the NeoCITIES data
objects had to be modified to fit within the GeoTMS Information Structure.
Merging the Data Structures
The GeoTMS Information Structure contains three main structures--the player, the events
and the event/location information. Similar to this, the NeoCITIES information structure
considers the Player and the Events (other data structures are implemented in NeoCITIES but do
not pertain to implementing GeoTMS as a front end). This means that the NeoCITIES
information structure had to be extended to separate out the event information from the event
itself.
71
Since the players in NeoCITIES are basically the same as the “Team Members” in
GeoTMS, the system was able to simply pass that information in without trouble. Code was
written within the NeoCITIES controller that set the team member name, and their color (based
on their role in NeoCITIES), and passed that object into the GeoTMS External Controller.
In NeoCITIES, rather than having the Information as a separate object, all the
information is part of the Event Object. In order to separate them, in the code, the Event Logo, the
Event Descriptions, The Event Severity, and The Current Units on Scene were extracted from the
Event object and stored in a new Information Element Objects (for a visualization of each of
these see the next section). Additionally, the current version of NeoCITIES did not take
geographic information about each of the events into consideration so NeoCITIES had to be
slightly modified to handle the latitude and longitude data objects that were used for plotting the
events on the map.
Unique Views
In order to integrate these two systems 6 different Unique Views had to be created, four
for Information Elements, one for events and one for team members.
Team Member Unique View
Figure 17: Team Member Unique View
The Team Member Unique View (Figure 17) was taken directly from the Team Monitor
(C in Figure 15) in the NeoCITIES 3.0 Interface. This view, which is located at the top of each
team member full view, and depicted how many of each resource they currently had available.
This interface component was first implemented in the NeoCITIES 3.0 Interface (Hellar & Hall,
2009) and was designed as a cognitive aid to assist teammates in coordinating their resources.
72
Previous versions of this interface component included the most recent event where the player
was working, but this has been removed since the new Attention Circles provide that information.
Event Unique View
Figure 18: Event Unique View as a Pop-up on the Map
The Event Unique View (Figure 18) was taken directly from the Dispatch Panel (B in
Figure 15) NeoCITIES Interface. This is a slightly modified version from the one that is seen in
(Hellar & Hall, 2009). This view is shown as a popup window on the map with an arrow pointing
to the event that it relates to
This window allows players to dispatch resources to the selected event. Once the
resources are dispatched the window is automatically closed. Once the event is ended, this view is
no long accessible and players can no longer dispatch to the event.
73
Information Element Unique View – Event Icon
Figure 19: Event Icon Unique View
Ever since the first iteration of NeoCITIES, events were assigned an Icon as a visual aid
to correspond with their name and description. Keeping with the tradition, an Information
Element was added that contains and displays the event name and icon at the top of the
Information Element list. This unique view is consistent and does not change throughout the
simulation. For a complete overview of this Unique View and the code behind it see Appendix E
Unique View Example.
Information Element Unique View – Event Status
Figure 20: Event Status Unique View
The Event Status Unique View (Figure 20) is a modification on the Event Status column
in the Incident Tracker (E in Figure 15) from the original NeoCITIES 3.0 Interface. This unique
view contains two pieces of information, the event status and the event severity. The event status,
which is the text reading “On Route” in Figure 20 tells information about if the event is new, if
units are responding to it, if events are on scene or if the event has ended. The event severity,
which is depicted by the background color of the unique view, varies between Red and Light
Green depending on how much damage has been done at the event. The background color will
turn black if it failed and dark green if it was completed.
74
Information Element Unique View – Responding Units
Figure 21: Reporting Units Unique View
The Reporting Units Unique View (Figure 21), is based on the Incident Inspector in the
original NeoCITIES 3.0 Interface (not pictured). This provides information as to which units (all
units, not just your players) are either on scene or on route to the selected event.
This information used to be hidden in a screen which required several clicks to get to, but
is now provided on the main interface. This is a useful extension of the Team Member Unique
view, as it will provide an even deeper level of information for coordination and timing.
Information Element Unique View – Event Description
Figure 22: Event Description Unique View
75
The Event Description Unique View (Figure 22) currently displays the description of the
event which is set in the scenario. Once again this information used to be hidden away in the
Incident Inspector in the original NeoCITIES 3.0 but is now presented on the main interface.
The future vision of this component is that rather than every event has a long paragraph
and only one description that is all encompassing. Future scenarios will be designed having
multiple descriptions, each of which represents a different on scene report about what is going on
at that location.
Other Interface Elements
In order to fully integrate the two systems other small interface modifications had to be
made. Most of these changes were taken directly from the NeoCITIES 3.0 Interface.
In order to allow players to be aware of the current simulation time and which role they
are currently playing as, the header from NeoCITIES 3.0 was included (Figure 23). This provides
a logo representing the team that the player is currently assigned to, the current Simulation Time
(in 24 Hour Time) and a NeoCITIES Logo.
Figure 23: NeoCITIES Header
The second need for NeoCITIES (which the GeoTMS interface did not take into
account,) is that the players needed a way in which they could monitor their units’ progress as
they dispatched them and recall them from events. The initial thought for this was to include a
unique view for each event that handled this. However, the players would only be aware of their
units at the event they are currently viewing. As a solution to this, the Unit Monitor from the
NeoCITIES 3.0 Interface (D in Figure 15) was added below the mapping area. This allows the
players to keep track of their units and recall them if they are not able to help at an event.
76
Figure 24: The severity of the event dictates the markers color
The final interface implementation that had to be addressed was the marker colors for the
events. Since players need to be constantly aware of the severity of the events and the only way to
view this information was to click the event and look at the Event Status Unique View, something
had to be added to main view. Since the markers supported having varying colors, the severity
was used as a scale to dictate the colors of the markers (i.e. Figure 24). These colors will change
as described in the Event Status Unique View section.
Methods
Since the initial design of the GeoTMS Interaction Environment was designed based on
theory rather than user experience, the interface itself may still need to be tweaked based on user
preferences. The goal of this study was to identify things in the interface that either don’t work, or
create difficulties for the user.
Evaluation Process
One popular discount usability method, which is considered by Jakob Nielson to be “the
single most valuable usability engineering method” (Nielsen, 1993) is that of the Think Aloud
(Ericsson & Simon, 1980). In a traditional Think Aloud study, also known as a Concurrent Think
Aloud (CTA), participants are asked to verbalize their thoughts while they complete a task or
explore a system. This method has been used to test the usability of web pages (Battleson, Booth,
& Weintrop, 2001; Benbunan-Fich, 2001; Van Den Haak, de Jong, & Schellens, 2009),
Commercial Products (George, 2008; Guappone, Ash, & Sittig, 2008) and even Decision Support
Systems (Kushniruk & Patel, 1998; Trafton, et al., 2010). While an effective method, several
77
issues have been raised about how the act of speaking might have a negative effect on the users
performance or change the way that they would typically do things (Ramey, et al., 2006).
Additionally, since the NeoCITIES task requires users to make quick decisions and multi-task by
working on multiple events at once, the think aloud method might not be the best fit.
In order to get the best of the Think Aloud method without the negative effects this study
uses a method called the Retrospective Think Aloud (RTA) (Henderson, Smith, Podd, & Varela-
Alvarez, 1995; Nielsen, 1993). In this method, users are asked to complete a task, then after the
task they are asked to go back to the interface and walk the researcher through each of the steps
they took, and what they were thinking during this process. This method’s validity has been
tested using eye tracking and other methods to ensure the participants actually focus on what they
are talking about (Guan, Lee, Cuddihy, & Ramey, 2006) and has been shown to be a more
effective usability test than a CTA (Van Den Haak, De Jong, & Schellens, 2003).
The task portion of the RTA will be a simple NeoCITIES Simulation. Since this study
will only look at one user at a time all the events are Police only events. The simulation itself
lasted just over 7 min, and consisted of 7 events. This scenario was designed based on the
research and results from (Hellar, 2009) and (Obieta, 2006). For a full list of the events in the
scenario see Appendix F
NeoCITIES Scenario. To simulate a basic level of the collaborative nature of the software, the
researcher will log in as a Fire and Hazmat players on another computer.
After the task has been completed the participants are asked to fill out a quick survey
about their opinions on the importance and how often they used each interface component and
their perceptions on their abilities to complete tasks (Appendix B
Study Materials).
Following the survey the users are asked to participate in the modified RTA portion.
Since the NeoCITIES task is not as simple as a typical task that a RTA is used (i.e. browsing a
web page, setting up software, etc.) slight modifications have to be made to this process.
NeoCITIES players will often have to be redirecting their attention from event to event and
focusing on things as they become necessary, therein the process had to be modified to ensure
that participants focus on specific aspects of the interface without being distracted by other events
that are occurring. In order to do this, the participants were told before the Retrospective Think
Aloud portion to not worry about their score, and to just walk through what they were thinking
78
during the performance session. Since some of the interface elements are collaborative, the
researcher once again will simulate the Fire and Hazmat players. Since time is of no issue in this
simulation, at every interface component described in the previous chapter, the researcher asks
follow up questions to the user’s initial observations (interview guide in Appendix B
Study Materials).
During each of the Think Aloud sessions the researcher takes notes in relation to the
question on their interface guide and records the interview. At the conclusion of the interview the
participants are asked of any other interface comments or suggestions to the current design.
Qualitative Data Coding
All of the interviews were transcribed into a computer using basic transcription software
called Express Scribe (http://www.nch.com.au/scribe/). These interviews were then coded using
half top-down, half bottom- up, coding procedure, as described in (LeCompte & Schensul, 1999).
This allowed the data to speak for itself rather than forcing it to fit in a pre-defined scheme. This
coding will mainly focus on design problems and things that they liked about the interface.
The first level of coding was simply a matter of whether the participants mentioned an
interface problem (coded as a 0) or something they liked about the interface (coded as a 1).
Once the data was organized into problems or things they liked, the data was then coded
using a bottom-up design to divide the information based on the interface component they are
talking about at that given time (Table 12). Since there are times that the users are talking about
multiple components, that portion of the interview will be coded based on which component their
comment impacts the most. For example, if they talked about how something on the Main Map
distracted them from something on the incident inspector, this would be coded as a Main Map
problem, because it impacts that component the most.
Table 12: List of Interface Codes Code Number Interface Component
MM Main Map View OV Overview Map View TM Team Monitor II Incident Inspector G General Interface
79
Once the comments are separated out by which interface component they were talking
about a new coding scheme had to be formed based on the data that was collected using a top-
down coding strategy. Using the data as a compass, nine codes were created to help organize the
comments into specific areas that could be used to help better understand the system. These
codes, and an explanation of what they entail can be seen in Table 13.
Table 13: List of Top-Down codes Code Label Description INT Interactions Any issue or comment that involves an interaction with the
system and the interface itself. ET Event Tracking Any issue or comment about the participant’s ability to track
specific events throughout the scenario. IE Information Extraction Any issue or comment about whether the participant was able
to find and extract the relevant information to complete their task
GEN General Comment Any general issue or comment about the interface COM Communication Any issue or comment about communication within the
interface MAP Mapping Any issue or comment that related to the maps in the interface REP Representations Any issue or comment about how things were represented in
the interface. This mainly includes visualizations, labeling, and colors.
SI Simulation Issues Any issue or comment about the simulation and events themselves (Note: These may not be directly useful for this system, but for future NeoCITIES research)
PT Personal Tracking Any issue or comment about the participant’s ability to keep track of their progress during the simulation.
TT Team Tracking Any issue or comment about the participant’s ability to keep track of their teammates
These codes allowed for qualitative data analysis to be effectively carried out which is
discussed in the next section.
Chapter 5
Results
Participant Demographics
For this usability evaluation, data was collected from 15 student participants at the
University Park Campus at The Pennsylvania State University. Since the students were recruited
from graduate level classes in the College of Information Sciences and Technology, this resulted
in an older age than traditional NeoCITIES studies (M=32.47, SD=10.13) with all participants
being from the College of IST.
The majority of participants reported playing video games (73.3%), with the majority of
them preferring computer games (40%) over console games (20%). This should not have an
effect on the data as previous studies looking at NeoCITIES have seen little to no performance
effects based on gaming experience (Hellar, 2009).
When asked about their comfort with web 2.0 and communication technologies,
participants rated themselves to be comfortable with Google maps (73.3% responding either
comfortable or very comfortable), text messaging (73.3% responding either comfortable or very
comfortable) and Facebook (76.7% responding either comfortable or very comfortable), but not
Twitter (66.7% responding either very uncomfortable or uncomfortable).
In order to gauge the participant’s ability to multi-task, they were asked several questions
gauging their perceptions of themselves. Overall, the majority of the participants (80%) rated
themselves as either agreeing or strongly agreeing that they were good at performing more than
one task simultaneously and that they were able to multitask effectively. Participant’s perceptions
of their ability to multitask were rather high, while only a slight majority (53.3%) reported
enjoying multitasking. A more interesting result came when asked whether participants needed
technology to help them multi-task. A fairly even split of the participants agreed (40%), neither
agreed or disagreed (33.3%) and disagreed (26.6%) with the statement that they require
technology to multitask.
The final, and possibly most important demographic question, asked about participant’s
ability and experience working in distributed teams. Though a strong majority of participants
81
(80%) report that they are moderately or extremely comfortable (with none reporting any level of
uncomfortable) working in distributed teams, only a small number (33.3%) report more than
moderate experience working in a distributed team.
Survey Scales
Following the performance scenario, the participants were asked to complete a simple
survey. They were given questions to respond to about the perceived usefulness and how often
they used each of the components within the simulation. Additionally, participants were asked
questions about their abilities to perform tasks within the simulation and qualitative questions
about things they liked and things they did not like about the simulation.
A one-way repeated measures (ANOVA) was conducted to examine perceptions of
perceived usefulness of components within the GeoTMS interface. The participants were asked to
rate their perceived use on a 5-point likert scale with 5 being the highest and 1 being the lowest.
A main effect was found for component type, F(5,10)=12.105, p < .01, partial η² = 0.86, with
participants ranking the main map the most useful (M=4.33, SE=0.19) and the Team Monitor the
least (M=2.07, SE=0.28). For a full overview of the interactions between components see Table
14.
Table 14: Perceived Usefulness of GeoTMS Interface Components Perceived Usefulness
Main Map Overview Map
Incident Inspector
Team Monitor
Offsite Event
Indicator
Attention Indicators
M 4.33a 2.20b 3.73a 2.07b 3.45a 3.93a SE 0.19 0.26 0.33 0.28 0.31 0.25 Wilkes Λ = 0.14, F(5,10)=12.105, p < .01, partial η² = 0.86. Note: Means with no lower case subscript in common differ at p < .05 using Holm’s sequential bonferroni post hoc comparisons.
As a basis of comparison, participants were asked to rate how much they actually used
each of the interface components during the experiment. This used the same scale as the previous
question. Another one-way repeated measures ANOVA was conducted to examine how much the
participants used each of the components within the GeoTMS interface. Again, a main effect was
found for component type, F(5,10)=28.66, p < .001, partial η² = 0.94, with the main map being
82
used the most (M=4.93, SE=0.07) and the Team Monitor being the least (M=1.53, SE=0.24). For
a full overview of the interactions between components see Table 15.
Table 15: Actual Use of Components during Simulation Actual Use
Main Map Overview Map
Incident Inspector
Team Monitor
Offsite Event
Indicator
Attention Indicators
M 4.93a 2.00b 3.67c 1.53b 3.13bc 3.93c SE 0.07 0.39 0.35 0.24 0.38 0.33 Wilkes Λ = 0.07, F(5,10)=28.66, p < .001, partial η² = 0.94. Note: Means with no lower case subscript in common differ at p < .05 using Holm’s sequential bonferroni post hoc comparisons.
Finally, participants were asked about their abilities to perform specific tasks during the
simulation. These included questions about team awareness, incident awareness, personal
awareness and information extraction. Each of these questions were rated on a 5 point likert scale
(1- completely disagree, 5 completely agree). For a full review of the questions and results see
Table 16.
Table 16: Task ability evaluation scales Question Mean Median SD I was always aware of what my teammate was working on throughout the entire simulation.
1.60 2.00 0.63
I was always aware of where my teammates were looking throughout the entire simulation
2.13 2.00 0.92
I was aware of all incidents that happened during the simulation
3.67 4.00 1.11
I was always aware about the status of the incidents that were going on in the simulation.
2.67 3.00 0.98
I was able to keep track of all my units during the simulation.
3.07 3.00 1.10
I was able to easily find out all the necessary information for each of the incidents during the simulation.
3.40 3.00 1.24
I was able to easily extract all the necessary information to make the correct decision for each incident
3.20 3.00 1.37
When asking about team awareness, participants were asked whether they were aware of
where their teammate was working (which incident) and where they were currently looking at (on
the map). To understand if there was a difference in the ability of participant’s ability to track
viewing and working area, a paired-sample t-test was conducted. Although the difference in the
83
use of the main map/attention indicators and the overview map from the prior two measures, this
test is only approaching significance, t(14)=1.95, p = 0.08 with participants reporting knowing
where their teammates were looking (M=2.13, SD=0.92) more than where they were working
(M=1.60, SD=0.63). According to a Single-Sample T-Test, the information participants reported
in the question about awareness of where their teammates were looking was consistent with how
much they used the component in the prior measure (2.0), t(14)=0.56 p = 0.58. Unlike this scale,
participants differed in their responses about their ability to track where their teammates were
working across the last two scales. A single-sample t-test showed that in participants reports of
their actual use, participants rated the attention indicators significantly higher (3.95) than their
ability to track where their teammates were working (1.60), t(14)=14.27, p < .001.
The second set of measures-- asking about event awareness--had participants rate their
abilities to keep track of events that were active, and the status (severity) of the events.
Participants rated a higher than average awareness of events that were currently active (M=3.67,
SD=1.11) and an about average ability to keep track of the status of the events (M=2.67,
SD=0.96).
Only one question was asked for personal tracking in regards to the participant’s ability
to keep track of their units during the simulation. Participants recorded a higher than average
ability to track their units (M=3.07, SD=1.10), which a single-sample t-test showed was
consistent with participants ratings of the incident inspector (3.67) in the previous measure,
t(14)=2.13, p =0.06. The Incident Inspector may not be the only factor in this though, since the
unit tracker from previous iterations of NeoCITIES was used.
The final quantitative measure collected from the survey was two measures on
information extraction-- about their ability to find the information, and their ability to use the
information to make the correct decision. Participants rated their abilities to find (M=3.40,
SD=0.32) and use information (M=3.20, SD=0.35) about the same. A single sample t-test showed
that these were both consistent with the participants actual use of the incident inspector (3.67),
t(14)=0.84, p=0.41 (for information finding), and t(14)=1.33, p=0.21 (ability to use information).
As an extra measure, qualitative data was collected about interface elements the
participant liked and didn’t like. The most popular response for liking was something about the
main map and the components of it. Participants noted that they especially liked the smooth and
intuitive interactions with the map, as well as the offsite event indicators and attention indicators.
The lack of intuitive zooming was reported the most for the dislikes. Other popular dislikes were
complaints about hidden controls, and the colors of the events not being apparent at all times.
84
Some participants took the time to make interface suggestions, these were mainly comments
about event tracking (sound effects were requested) and some sort of legend to help them
remember what everything meant.
Retrospective Think Aloud
Once the data was coded on all three levels, the comments were separated based on
which component they applied to and then analyzed individually. Overall, there were 123 total
codes that were extracted from all the interviews conducted. From these 123, almost an even split
of positive (63) to negative (60) comments were extracted. For a full breakdown of how people
reported for each of the components, see Table 17.
Table 17: Full breakdown of comments for each component Component
G MM OV II TM Total Good 4 28 8 8 14 63 Bad 2 26 8 8 16 60
Totals 6 54 16 16 30 123
As you can see, the main map was talked about the most during the retrospective think
aloud (it also received the most good comments and the most bad comments). For the most part,
the breakdown for the types of comments was fairly similar as well--with Team Tracking, Event
Tracking and Representations being the most talked about. For a full breakdown of how people
reported for each of the codes, see Table 18.
Table 18: Full breakdown of comments under each code Top-Down Code
INT ET IE GEN TT COM MAP REP SI PT Total Good 4 8 3 10 13 7 4 8 1 1 59 Bad 9 9 2 5 9 5 4 13 2 2 60
Totals 13 17 5 15 22 12 8 21 3 3 119
85
In the following sections, I will break down the results for each component and provide
vignettes as evidence.
General Comments
Although there were not many general comments, overall, the reaction to the system was
positive. Three of the participants had prior NeoCITIES experience and explicitly commented
that they liked this interface more because it made it a more immersive environment and added a
new layer of depth to the simulation.
One of the participants with no experience with NeoCITIES before commented that the
layout allowed him to quickly get started on his task:
“I like the layout, I like the way that everything is divided, so you can actually start to determine where you need to look and where the active stuff is.”
Some of the participants felt the interface at first was, “Busy” and “Too Cluttered”. One
participant found it troubling when they started playing and there were no obvious buttons or
interactions for them to start with:
“Um, didn’t really know what to do at first, there were no obvious buttons or anything…”
This was a direct result of most of the functionality being tied to the map, which does not
have any obvious affordances for interaction.
Main Map
As discussed above, the main map was the most talked about portion of the interface by
the participants. This is understandable because the map itself takes up a large portion of the
screen and is designed to be the focal point of attention and interactions.
86
Many of the mapping elements were liked by the users. Many of the users found the
coloring scheme of the markers as intuitive and helpful in prioritization of events, but sometimes
the changing of colors was too subtle and the participants wouldn’t notice the increasing severity:
“I loved how the events changed colors to tell me how they were doing… But sometimes they changed and I didn’t even notice.”
Another feature on the map that was found to be helpful was the offsite event indicators.
Participants found these useful at allowing them to be aware of events that were not happening in
their current field of vision:
“Actually, that <offsite event indicators> worked really well, since I saw it there, I knew I wasn’t missing anything.”
Participants also noted that the thick red arrow that accompanied the event marker really
helped it stand out and be apparent to them.
The offsite event indicators caused several problems though. Since they were displayed
with a floating marker next to an arrow, some participants perceived that as a marker and an
arrow that pointed to another marker. Since they were unaware that it was only one marker
instead of two, they were confused when they scrolled over and the one marker simply
disappeared.
The biggest problem with the map came with event tracking. Even though participants
would claim that the offsite event indicators were useful in keeping track of new events, they
caused problems in keeping track of older events that you had already responded to.
Participants claimed that whenever you would move the map and caused several events
to bunch together, it caused them to have a difficult time to keep track of which ones they had
already attended to:
“They were <offsite event indicators were useful>, BUT, I had no way of distinguishing between events that I had already attended to and events that I still needed to attend to. So I take care of one event, and then go to another, but the arrow is still there…”
Additionally, if a new event popped up in a cluster of old events, the participant
sometimes would not notice that it was there, and would not react to it as quickly as they should
87
have. This was very apparent when analyzing the reaction times from the NeoCITIES data (more
on this in NeoCITIES Data).
Finally, since the user is constantly moving around the map, participants would
sometimes lose track of which marker relates to which event. For example, if there was an arrow
in the top of the screen, and the user moved up and to the right, that event would relocate to the
left of the screen. This caused the participants to become disoriented and confused as to which
one of the events they had worked on and which ones they had not worked on.
One of the common themes with the map was that it often served as more of a distraction
than an aid. When asked, some of the participants even claimed to ignore other portions of the
interface as a result of focusing too heavily on the map:
“Hmm, I didn’t really pay attention to that as much <event colors>, my thing was just to use the main map to keep track of everything. I think if I would have tried to look at the events too closely that I would have lost track of something or fallen behind.”
Another major problem on the main map was that many of the interactions were not
apparent. One participant claimed to be unaware that they were actually able to manually move
the map, and simply used the off screen event indicators to do the majority of their navigation.
Additionally, participants who were familiar with other online mapping applications, like Google
maps, instinctively wanted to use the controls that they were familiar with from their experience,
like zooming with the scroll wheel. Since there was no indication of whether they could or could
not do this many became frustrated. Similar to this, many participants would miss-double-click an
event and actually zoom in.
“Yeah, I mean it kinda made sense, but I think since this is Google maps, I instinctively want to scroll with my wheel. And like because times, I didn’t directly double click on an event, and I ended up zooming in”
Since they were unaware of this functionality, many of them would become disoriented
and not understand that they are zoomed in, and remain that way for the rest of the experiment.
Additionally, since everything was visually represented (severity, location, etc.), if the person did
not remember what something meant they were not able to easily regain their understanding. One
of the participants even proposed a solution of integrating some type of legend to the map to
ensure that people are always able to understand what is going on. Finally, the last issue with the
hidden controls was with dispatching. Most of the participants remembered or knew that by
88
double clicking the marker you would bring up the dispatch window, but a handful of them could
not remember and had to stop and ask the researcher for help during the experiment:
“Uh, I thought I knew what was going on pretty well, but the first thing I clicked on was this marker, then I panicked because I couldn’t remember how to open my dispatch window, I knew the training told me, but I just couldn’t remember and then all these events started popping up, and I was like oh crap, ok, then I just double clicked on it, and realized how to do everything.”
The final feature that was discussed on the main map with the participants was the
attention indicators. Most participants liked that they were able to see where their teammates
were currently working so they could coordinate better. Similarly, with the other features, some
participants forgot what they meant and became confused as to what it represented:
“One thing that I didn’t quite understand, when I clicked on something, the color here <the attention indicator>, I wasn’t sure if that indicated anything.”
Some participants even confused these circles as being related to the events themselves
and event made priority based decisions on their confusion:
“Um, it definitely would be useful information, um, but, the circles, are a little more confusing, for example, the fire, shows up as red, the highlighting in red, could signify urgency, and a couple times when I saw the green, I thought oh I doing good on that one, and I ignored it. Um, so that the issue. I’m not quite sure, maybe an indicator of who is there, like letters, or their name or whatever, maybe a little more useful, since I mean, initially, you have to get your head wrapped around the color scheme.”
Although this was explained in the training, some of the participants still did not
understand what they meant, therein the attention indicators may have become more detrimental
than helpful. The final issue with was that whenever two of the players would be looking at the
same event, it would show an amalgam of their colors (i.e. if fire, who is red, and police who is
blue were looking at one event, it would show up as purple). This caused some participants to be
confused, thinking there was a fourth, purple player operating in the workspace without their
knowledge:
“Oh, definitely, the only problem I had was I think it turned purple once, which I think is a result of a layover of the red and the blue and it makes sense, but my mind was looking
89
for these colors <points to the team monitor> so when I saw that, it actually confused me, more than helped me, yeah, so that helps me <points to event with one attention circle>, that confuses me <points to event with 2 attention circles>, cause I was looking for just these colors…”
This mainly was a result of a design choice to have the attention indicators be semi-
transparent rather than opaque so players could still see the streets below them.
Overview Map
Although the numbers show that the overview map had a balance of positive and negative
comments, the negative ones refer to more complex issues in comparison to the benefits relating
to the positive comments.
The majority of the positive comments about the overview map were general statements
saying that it was good and/or useful. One participant claimed that its main utility was when other
users were working in the same geographic area, rather than if they were in different areas:
“I think it would be useful so that you know when, for instance, if everyone was looking in one area, in the same area, it would make sense for one person to move over and look into something that has not been addressed. So I think it is definitely a good feature”
Participants claimed that they liked it, but when probed to explain its exact utility, they
were not able to provide much other than “to show were people are focused”. One of the more
specific benefits of it was its overall simplicity:
“Yeah… because the map is small and it can be quick and you don’t have to go anywhere to see it”
Even though some people saw the map as useful, some of the participants were
completely confused and did not understand its utility. One participant commented that the
inherent role structure of NeoCITIES did not afford this being very useful:
“Again, if the division of labor is fire, police ems, maybe not, um, because they are looking at a part of the map, where there may be events that I need to pay attention to, because the division of labor is I handle police events, and they are over there looking at an ems event or a fire event or a hazmat event, um, that’s not helpful, but, if we have
90
different geography’s we are trying to cover, I could say, oh, well he can look over there and I can stay over here…”
Additionally, some participants found that it was useful, but with its location, the felt the
time lost refocusing their attention to get the information could cause them to miss something
important:
“Uh put it <the overview map> on the main map…. See my main focus was here <points to the main map>, not here <points to the overview map>, so some how you could put this information here… A heads up display… Something other, like look here, look here, look here, I’m going all over the place, I just focused on the actual map, so I didn’t even look here <the overview map>.”
Also, some participants felt that the information being displayed on the map wasn’t
complete and either confused them or left something to be desired. Some participants did not
make the connection that the other squares represented the other users completely. And, some
participants wanted more information about which events were where on that map. Also, one
particular user expected the overview map to act like the one in Photoshop. This caused them to
become confused when it did not act the same:
“That may not actually be the best way to do it. In my mind, like an overview map, I’d rather have it static, like, if you take the Photoshop analogy, the overview, usually shows the full image and you can zoom in and zoom out using your box… Ooh, that would be interesting, if you could figure out a way to draw the box and it would zoom into that area that would be kinda cool. Like when things shift around, as for the overview map is concerned, it is used to know, hey what am I looking at, and getting an overall picture, of what is going on, and I think just focusing on just what everybody else is focusing on, you lose a little bit of that, and you lose, what you would get from a full map…”
The final major issue on the overview map was with the zoom control. This was
implemented to act like the Photoshop zooming. None of the participants made the connection,
and often did not zoom at all or just zoomed accidently (like mentioned in the previous section).
This was a result of people expecting typical map zooming, and the lack of clear labeling.
91
Incident Inspector
The incident inspector is one of the most important aspects of the interface, and probably
the simplest. If a participant expects to do well in the NeoCITIES task, they need to be aware of
all the relevant information that is needed to make the correct decision on what to dispatch. All of
this information is exclusively located in the incident inspector, so the participant is forced to use
it, or do poorly.
Of all 4 unique views in the Incident Inspector, participants claimed to use the description
the most, because it was most critical to their task and the reporting units the least:
“Yeah, I didn’t see anything wrong with this. I think that it was nice, how you can first read this (the description) then double click on it (the marker) and can decide what you need. One thing I would say is that I didn’t see the use of the reporting units.”
One of the participants thought the reporting units would be more useful if it had some
level of functionality. They suggested that adding the same recall functionality from the Unit
Monitor to that would make recalling units from events you were viewing quicker and more
intuitive.
Some of the participants, while they used the information, and thought it was really good,
thought that the location of the inspector made it inconvenient to access the information. All of
the participants who made this comment claimed that they would prefer that the incident
inspector was merged with the dispatch window so they only had to look one place:
“Uh… I don’t think so, the only thing is maybe have this <the information view> be part of the pop up <dispatch> just because then it would combine all the needed information. Cause when it was over there, I clearly stopped looking there because I was so focused on what was going on over on the map. I don’t know how well that would work, like if you had too many things popping up, that would be another problem…”
One of the major confusions with the incident inspector was with how it was represented
without clear labeling. This was especially apparent before the participants would click on an
event and the incident inspector sat there with no clear labeling and was completely blank.
Though confusing at first, this was something that was easily remedied with experience in the
system:
92
“Um, one thing that did confuse me, was that on the left, this thing sat here with nothing on it, and that was a little bit confusing. Um, but as things popped up, it made sense obviously…”
When asking participants about how much they used the incident inspector, it was
difficult to gauge their response, because they used it often, but not a lot. One participant
responded:
“Yeah, I thought that was pretty clear, I didn’t look at it a lot, but I knew what it was and I used it for every event…”
This is an understandable response, but it makes it hard to gauge its utility based on self-
report.
The final problems were simple programming issues. Some participants thought that the
text was too small for quick access, and sometimes the description box would not expand to fit
the entire descriptions, which required the participants to scroll to see all the information.
Team Monitor
Overall, the team monitor was found to be the most problematic of all the interface
components. Of all the users, nobody used this at all. Therefore, most of their comments were in
conjecture. Many of the participants thought the team monitor itself was very busy and cluttered,
which made it difficult for them to easily extract information from it.
One of the major complaints about the components within the team monitor was that
everything was listed in a crammed text only list, which was not easy for them to quickly parse in
a short amount of time. Many of the participants requested that the information that was listed in
text was displayed visually on the map instead:
“Um, in this form, I’m not sure how easily digestible it is <the history list>. If say hazmat had dispatched to this event, say you put a little hazmat icon next to it <the marker> or whatever the Google API allows you to do. So that way it’s a visual record that shows hazmat has attended to that. But if you put it on the map, then it adds a permanent artifact of what hazmat has done. In fact I thought the entire team monitor was a bit crowded with all of these lists.”
93
One thing was noted was that the current implementation completely ignores the
temporal aspect of the NeoCITIES task. Participants felt that they needed information on when
(the timing) their teammates were doing actions for it to be useful. Additionally, there were
thoughts that simply “History” was not a descriptive enough label, and they were not sure how
they should use it.
Another major interaction issue with the new Team Monitor was how it required extra
clicks to get information, and how you can only view one team member’s information at once.
Additionally, the default setting is that you are viewing you own list, which is possibly the least
useful to the participants:
“I definitely do, I actually left mine open, which is probably the least useful, cause I know what I’m doing, but it would have been nice to know what the fire and hazmat were doing during the simulation.”
This is an issue with the Flex component that was used to design this component, since it
only allows the user to have one of the players open at once.
The final interaction ability which was located in the team monitor was the “Focus In”
button, which allowed the participants to switch their view to be the same as the other players.
Only one of the participants noticed this button, and thought it was really useful, but since it was
hidden a click down within the component they didn’t notice it was there till the think aloud
portion.
The major talking point when dealing with the Team Monitor was the new
communication method using an asynchronous, “status message” system. Like the other aspects
of the Team Monitor, none of the participants used the Status Messaging system during the
experiment, so all of the comments were a conjecture. One of the participants, the only one with
Emergency response experience, did not see the new system working at all based on his
experience in dispatch centers:
“These kinds of events are difficult to communicate about what’s going on, and what’s needed. If you had completely separated dispatch centers, where I don’t think there would be communication at all… Communication would be looking over the cubical walls, and reducing that all the way down to a single line chat is terrible, you have to substitute it with, if I can’t look over to my co-worker and as them a question, um, I’m not sure if one line will help at all, and in a timely required set of events, that will be lost… If I have a compressed set of events, with high severity, and chat is anything but shouting
94
across the room, then chatting is going to fall by the wayside… I have all this stuff going on around the map on the screen… That is where I have my attention…”
The major concern with the new communication system was that it did not work with the
time sensitive nature of the NeoCITIES task. During the simulation, many of the participants felt
overwhelmed by the rapid firing events, and felt that the communication would have taken too
much time to try and update during that wave of activity. But, they felt that in a less time
sensitive environment, it may be useful:
“Only if it was a task that wasn’t time sensitive... Like if you are trying to do something in a short amount of time, then no, there is no time to look at that, but if it is something that you are doing over a long period of time, then I can see this being useful…”
Some participants felt that using a vague word like “status” made the communication
channel to generic for it to be useful. One participant proposed relabeling it to make it a more
specific use that was tailored to the task at hand:
“Yeah, because like, technically the action list, tells me what we’re actually doing, so like, maybe like requests, would be more appropriate. Something tailored more specific to the task.”
This would be more consistent with the results of the study conducted by (Hellar, 2009)
in which “Action Requests” and “Information Transfer Requests” were some of the most popular
communication strategies from the synchronous chat protocol.
Some of the participants did feel that the status messaging was an antiquate level of
communication. One participant who had prior NeoCITIES experience noted that he felt this
could help solve some of the issues with participants getting off task via the chat during
experimentation sessions:
“Yeah, I think that would be helpful, when I think about that, real time communication could get bogged down and actually cause you to do worse, but I think this would work yeah…”
Additionally, there were some thoughts that if using a synchronous chat, you would be
introducing other compounding effects into the scenario, which may cause the participant to
actually do worse:
95
“I think it’s enough. If I was really paying attention to it, status messages are probably the best thing because you the correct amount of information and not too much information. If you had too much information coming at you then you’d have even more information overload because of all the messages.”
A final benefit of this type of messaging was brought up by a gamer, who claimed it fit
with his mental model from playing only role playing games like World-of-Warcraft.
Additionally, with so many people using asynchronous communication channels like twitter and
facebook, this new style of communication may be more fitting with their prior experience than
something like a real-time chat.
Similar to the action history, there were some worries about the temporal aspect of the
status message. First of all, the communications did not include timestamps, which for the most
part makes them impossible to understand when they occurred and if you still need to respond.
Additionally since the responses were not threaded and only displayed one at a time (unless you
viewed a single persons full history list), it would be difficult for you to have a longer
conversation with somebody, or another player to jump in mid-discussion.
NeoCITIES Data
Although this usability study is not concerned with the participant’s performance, looking
at the data provided by NeoCITIES will give us extra insight into the interface and may help us
answer some of the questions that arose in the qualitative portions of this study.
Overall, participants performed quite poorly in the one simulation they took part in.
According to the Human Performance Scoring Model, they averaged below 50% on the normal
score (M=49.02%, SD=15.44%) on each of the events and on average only 8 points lower
(M=8.91, SD = 10.49) than the worst score on all the events. The Human Performance Scoring
Model can be seen in Figure 25, and a further explanation of the calculations can be found in
Appendix D
NeoCITIES Scoring Model.
96
Figure 25: Human Performance Scoring Model, Graph taken from (Hellar, 2009)
In order to better understand whether or not this is a result of the participants having little
experience, benchmarking statistics from prior NeoCITIES 3.0 experiments were obtained
(Hamilton, 2010). In these, participants averaged 56% on the normal score (M=55.93%,
SD=26.95%) on each of the events, and averaged only 8 points lower than the worst score on all
the events.
Since lack of experience does not seem to be the cause of poor results, and one can
assume that the Graduate Students who were recruited for this study are just as qualified-- if not
more than the undergraduates--for the study used for benchmarking, other factors might be at
play.
Further inspection of the data shows that even though the participants were getting poor
scores, overall they were making the correct decisions. Across all participants (N=15) for all
events (9 events per participant, 135 total events analyzed), participants dispatched the correct
unit 70 times, while only making 25 mistakes. This shows that participants knew what they were
doing, but something else had to be causing their poor scores.
One possible cause is when looking at the units dispatched to each event, 16 events (out
of 135) had absolutely no resources allocated to them, causing the participants to get the lowest
score on those events. When compared to the benchmarking study (75 events), only 4 total events
had no resources allocated.
Another possible related reason that participant’s scores were so low could correspond to
the reaction times. When using the GeoTMS interface for NeoCITIES, participant’s reaction time
between an event showing up and first dispatching to it was almost 40 seconds (M=37.60,
97
SD=25.30). In previous NeoCITIES 3.0 simulations, participants reaction times averaged around
30 seconds (First time playing: M=31.09, SD=29.24, After Experience: M=28.85, SD=22.86).
For a full graphical representation of this see Figure 26.
Figure 26: Comparison of GeoTMS interface vs. NeoCITIES 3.0 for Reaction Time
While a 10 second difference in reaction time (on average) does not seem impactful,
when considering the Human Performance Scoring Model, it does make a drastic impact on the
score. The difference between a 30 second and 40 second response time (assuming you send the
correct units), causes an extra 10-15 points of damage for a severity of 3 (relatively low) event.
This gap continues to increase as the severity increases. For a visual representation of this
difference see Figure 27.
98
Figure 27: Additional damage done after 10 second lag in reaction time
Participants claimed they had trouble keeping track of which events they had already
visited on the map, and these results are consistent with that.
Discussion
Event Tracking
Possibly the biggest issue that came from this study was the ability for users to track
events. All three portions of the study (survey, retrospective think aloud, and the data reporting
from NeoCITIES), shows significant evidence of participants abilities to track the events.
In the initial design phase this was an expected issue, especially since users may zoom in
on areas without being able to notice events popping up off screen. In order to alleviate this
problem, the off-screen event indicators were used. These were successful in raising awareness of
events that were happening off screen, but users had trouble tracking which events were which.
99
Figure 28: Tracking a marker as you navigate through the map (a) Marker of interest is in the center of the map
(b) As you move to the right, the marker cannot be distinguished with the other marker in the area (c) As you move around and more markers appear, the one of interest gets completely lost
As you see in Figure 28, when you rely on the events spatial location to remember which
events were which, confusion can occur. In Figure 28, the users are initially looking at an event,
then moves down, which “moves” the event to the upper left of the screen, then when the user
moves up, the event moves to the bottom of the screen. Since other than the color
differential(which can change), there is no way of distinguishing one event from another with a
single glance, this problem becomes worse when there are more and more events on the screen
(as seen in c in Figure 28). This may be a problem of novices interacting with the system rather
than experts. While many of the participants had lived in State College for over a year, they
would not be categorized as geographic experts, and none of them would be considered
experienced in working with GIS. Research in GIS has shown that experts are often better at
grouping information, associating data with their geographic location rather than their location on
the screen, spatial decision making and strategically moving around the map (Anderson &
Leinhardt, 2002; Wilson, Lipford, Carroll, Karr, & Najjar, 2008).
While the map was designed to be the center of the interactions and focus of the users,
more work will need to be done. Other than the “Focus In” button located on the user panel
(which was used only once), and the zoom control on the overview map (which was very
unpopular), none of the other components offered intuitive ways of interacting with the map. For
example, if a user was reading information about another users actions on an event, there was no
clear way of linking that information, spatially, to the event which it related to.
A final issue that must be addressed is the colors as representations of information on the
map. While many of the users found the colors to be intuitive, some forgot what they meant and
were very confused as to their meaning. Additionally, their change ended up being too passive for
users to track. This is especially important in a task like NeoCITIES where the color relates to
damage done. Many of the participants did not even notice when their events transitioned from
100
Green (no damage) to Red (heavy damage). If this was a real world application, more steps need
to be taken to insure that this change can be recognized.
Personal Tracking
When initially conceiving GeoTMS, much of the focus was spent on designing a system
to support better collaboration and team tracking. Because of this focus, not as much attention
was paid to ensuring that users were aware of their own actions. Many of these problems overlap
with the problems discussed in the Event Tracking section. Many of the users expressed
difficulties remembering which events they had already dispatched to, and often spent time going
back through all the events on the screen.
The action list was designed to give the users a way of keeping track of what they had
already done in the simulation. However, this proved to be too much for the users to parse in the
short amount of time they had in the simulation. Additionally, sometimes when the user would
overlap their own attention indicator with another users attention indicator to create a hybrid color
(i.e. fire and police overlap creates a purple attention indicator), which caused confusion.
Future research will be necessary to find an appropriate middle ground to display this
information in a usable fashion, without increasing the cognitive load on the users.
Team Tracking
Team Tracking was broken up into three main interface components--the attention
circles, the overview map and the team tracker.
Attention Indicators
Overall, the attention indicators were found to be very useful, though there were some
instances of confusion. Similar to personal tracking, when two users were looking at one event, a
combination of the colors was presented, which could cause users to assume there is another
person interacting within the workspace. Also, some users expressed that they incorrectly
101
assumed the circle to be an indication of the progress of the event, rather than an indication of
who is looking at the event (i.e. green circle for hazmat assumed to mean the event status is
good). Both of these are simple representational issues that should not detract from the overall
success of the attention circles. Though not brought up in the user study, further research needs to
be done to see how attention circles should be depicted when they are off screen.
Overview Map
The overview map had the most disparity in perceptions throughout the entire study.
While some of the users thought it was a very useful aspect of the interface, others thought that its
utility was very little. One possible reason for this was the style of the NeoCITIES task does not
afford a “divide and conquer” strategy that the overview map affords. Additionally its lack of
interactions really limited its utility.
Team Tracker
Of all the aspects of the interface the Team tracker was easily the least popular. While it
may have aided participants in keeping track of their team, it’s cluttered appearance and hidden
attributes made it so none of the users actively used it. Because everything was said in retrospect
about the team tracker, it is difficult to make any conclusions. Future design considerations will
have to focus on making the information more accessible and useful.
Communication
One of the most interesting aspects of this new system was the move from a synchronous
communication channel to an asynchronous one. Unfortunately, due to the experimental
constraints, this aspect of the interface was not able to be properly tested. The only major
takeaway found was that the labeling of the communication channel as “status” may have been
too general and caused users to be confused as to its utility.
102
The other aspect of communication, that was for the most part well received, was the
automatic reporting of actions, known as “action as language”. While this aspect of the system
was considered to be a good idea, some aspects of the design choices limited it’s utility. First of
all, the action list totally ignored the temporal aspect of the NeoCITIES simulation. Knowing that
somebody dispatched to an event is useless if you don’t know what time it is. Additionally, like
other aspects of the interface, in a time sensitive task, users may not have the time to parse
through the list to find the necessary information.
Implications for Design
Based on the discussion points above, and other aspects of the usability study, a set of
design fixes were developed for future iterations of GeoTMS, these can be seen in Table 19.
Table 19: Interface problems and possible solutions Problem Solution(s) Users had difficulty tracking the ever changing status of events.
Add a linear list of all the active events to the main interface to give an overview of the status of the environment. Add animations or sonifications to highlight when event status’ are changing Add extra information to the markers (like text or icons) to express more information about the event.
Users were unclear what some of the markings on the map represented (i.e. event colors, attention circles)
Add a key or legend to the map. This should be able to be hidden for expert users who do not need the information. More visual information (i.e. an icon) than just colors to represent somebody’s focus in the workspace.
Users had trouble re-finding events after they had already worked on them.
Add extra visual notifications to the map that show things such as, viewed and dispatched. These could be extra icons or text added to the marker. Other aspects of the interface should include links to the map. For example, the unit monitor and action list should allow users to “focus in” on the event that they relate to. A linear list may allow users to more easily keep track of which ones they have worked on and which ones they have not. Allow users to sort events based on which ones they have already worked on and which ones still need their attention. Add labels or unique icons to the events to differentiate them from the other ones on the screen
Users were confused about the exact utility of the overview map.
Add interaction functionality to allow users to easily scan a larger geographic area, than having to use the main map. Suggested functionalities would be dragging and dropping, and also a rubber band control to select an area to zoom to. Add information about the number of events in the
103
viewing areas. Smaller event markers could be used for this. Add alerts about units that were not useful at a specific event. Ability to minimize overview map when you don’t need it
Users did not understand how they could interact with the map (moving, zooming, etc.)
Add scroll wheel features similar to the Google Maps web app. Change the cursor to a hand when over the map to afford dragging Add the zoom control to the map like in the Google Maps web app rather than the one under the overview map. Disable double click zooming to prevent users from accidently zooming in and becoming disoriented.
Users were not sure how they should use the status messaging system
Replace with a full functional chat protocol Rethink the labeling, possibly naming it requests, or updates Add direct messaging for more direct communication
Team Monitor required too many clicks to get information
Replace accordion control with one that is always open Remove the current user from the panel since that information may not be as useful Add more visual representations of information in the team monitor to the map
Information in team monitor did not include any considerations to time.
Add timestamps to status messages Add timestamps to action history
Users had trouble tracking the status of their units after they dispatched them.
Color coding the unit monitor based on success, failure, or partial success (green for success, red for failure, yellow for partial success) Visualizing your units and their status on the map itself.
This table offers only suggested fixes, not ones that would alleviate the problems in the
GeoTMS Interaction environment. In order to continue to move towards a more usable and useful
system more usability testing will need to be done on future interface changes.
Chapter 6
Conclusion
Contributions
There are five main contributions that have arisen from this thesis: (1) The GeoTMS
interaction Environment, (2) The proposed system architecture for developing generic interfaces,
(3) Transactive Memory: Artifact as Theory, (4) Expansion of NeoCITIES to other theoretical
constructs, (5) Experimentation and Usability study.
The GeoTMS Interaction Environment
The most important and largest contribution of this thesis is the development and
implementation of the GeoTMS Interaction Environment. This system took over 6 months to
theorize, design, code and test and over 2,000 lines of code were written to develop GeoTMS
using Adobe Flex, Actionscript, EasyMVC architecture and BlazeDS.
As an interaction environment, GeoTMS provides a rich and effective way for teams to
collaborate and communicate during complex Geospatial Decision Making problems. This
system, which was built upon the theories and constructs from Awareness, Common Ground,
Team Mental Models and Transactive Memory, to allow teams to make the most of their
collaborations. As GeoTMS is designed to “make work visible,” future tests will allow
researchers to better understand the role of transactive memory, awareness, common ground and
Team Mental Models in distributed collaborations.
Due to the flexible and generic design of GeoTMS, it should prove to be a usable and
effective way to collaborate GeoSpatial data for multiple systems and datasets for several years.
The purpose of this thesis is to present a system that is not only useful in bringing NeoCITIES to
the next level, but to design a tool that other researchers can easily use and adapt to their own
problems. As stated earlier, adapting GeoTMS to NeoCITIES was an easy and quick procedure,
and once the process has been properly documented, it will only become quicker and easier. The
105
concept of the unique view will allow future system designers and researchers to put their own
spin on GeoTMS and apply it to their own problems and domains.
GeoTMS System Architecture
With new interface driven programming languages, like Adobe Flex, the organization of
system architectures and designs of systems must be reconsidered. The architecture of the
GeoTMS Interaction Environment (Figure 8) shows how Web 2.0 programming languages can be
utilized to store unique views within data structures to allow more flexible interface designs.
A major issue in research system development is their lack of utility outside of the project
they were designed for (sometimes referred to as throw-away-code). By utilizing this system
architecture, future research systems can be designed so that they are usable outside of their initial
intent. Obviously, some minor changes to the architecture would have to be made depending on
the system being built, but overall, it is generic enough that it can be applied with little work. This
architecture may allow researchers to better confirm each other’s theories and build upon each
other’s work in new domains or situations.
The concept of the Unique View should be usable in any newer Web 2.0 programming
language where interfaces are considered Dynamic Objects rather than just displays. This means
that this architecture can be used in languages other than the one presented in this thesis.
The design and execution of this architecture took time to theorize and test, and was
iterated upon several times before, during and after the programming of the GeoTMS interaction
Environment. A major goal of presenting this architecture in this thesis is to hopefully influence
future system designers and researchers to take a similar approach to this to extend the usefulness
and applications of their hard work.
Artifact as Theory
This thesis used an Artifact as Theory approach to design (J. Carroll & Kellogg, 1989).
Using concepts from awareness, common ground and Team Mental Models this artifact has
potentially “filled in the gaps” of the transactive memory literature, and found ways of
embodying transactive memory as an artifact for distributed teams. Prior to this thesis, there have
106
been very few theory-based design approaches to developing a tool for transactive memory. The
GeoTMS Interaction Environment embodies the characteristics and dimensions of Transactive
Memory in its execution.
The perspective used to develop GeoTMS as an embodiment of transactive memory took
a multidisciplinary approach in developing the principles and filling in the gaps of transactive
memory. By conducting literature reviews in the areas of HCI, CSCW, and I/O Psychology, a
wider perspective on what needs to be done to improve collaborations for virtual teams is
presented. This type of approach has not been taken before in the Transactive Memory literature
and allows us to take significant strides forward without overlapping work that has already been
done.
While these principles of design were applied to a GeoCollaborative Interaction system,
they were designed as a generic set of Theory Based Design heuristics which may be useful for
the design of future collaborative systems. Additionally, one of the hopes of this thesis is to
encourage system designers to begin to take not only a user-focused design, but a theory focused
design. The research areas of collaborations, in areas like HCI, CSCW and I/O Psychology are
very rich, and the knowledge and information can and should be used outside of their domains.
Overall, the design principles presented in this thesis will be a useful roadmap for future
designers looking to leverage transactive memory, awareness, common ground and Team Mental
Models in their distributed work.
Moving forward with NeoCITIES 3.0
One of the major issues with the NeoCITIES 3.0 Simulation is its lack of a
GeoCollaborative Map (Hellar, 2009). All of the previous versions of NeoCITIES included a map
in their designs (Balakrishnan, et al., 2009; R. E. T. Jones, et al., 2004; M. D. McNeese, et al.,
2005), and it has been a research goal of the MINDS lab at The Pennsylvania State University
College of Information Science and Technology to reintegrate that into the interface. One of the
biggest fears in integrating a map to an interface is making it the focus of attention or else it may
detract from other aspects of the interface. Rather than borrowing from previous NeoCITIES
designs, or simply “plugging” a map into the current design, the GeoTMS Interaction
Environment was designed from the ground up in order to make sure that the map was the focus.
107
Additionally, this new interface adds a new layer of complexity and depth to the
NeoCITIES simulation without taking away from its overall simplicity. In the initial versions of
NeoCITIES 3.0, Geospatial data was only loosely considered (i.e. inputting the name of a
building into the description of an event). Now with the mapping feature, future research using
NeoCITIES can test Geospatial decision making issues, rather than the simple ones that the
current version uses.
The new interface will allow for more realistic emergency situations through the
redesigned data structures (specifically separating the details out of the event object and putting
them into their own Information Element object). In the current studies conducted using the
NeoCITIES 3.0 simulation engine (including this one), events were relayed to participants using
nice and clean descriptions with all of the information they needed. In realistic situations, when
making a decision, people often take information from multiple sources and have to make
decisions on which ones are credible and useful to what they want to accomplish, sometimes
known as reality or source monitoring (Hall, 2010; Johnson, 1988; Johnson, Hashtroudi, &
Lindsay, 1993). In this thesis, the “multiple sources of information” were represented by the
unique views which displayed the description, status and name of the event. Future research can
expand upon this making a more realistic and complex decision making environment. For
example, rather than having information presented in a nice clean paragraph, future NeoCITIES
simulations using the GeoTMS environment will display several short “reports” from varying
sources, which the users will need to digest, pick which ones are credible, and make a decision on
them.
Finally, over the past 3 iterations, NeoCITIES has been a tool to test the same relative set
of macrocognition principles. This thesis shows that with very little work, NeoCITIES can be
applied to new theories and domains in addition to the ones it was born out of. From this thesis, I
hope to encourage future researchers to apply the NeoCITIES research platform to their own
work, and use it as a vehicle to better understand and refine theories of cognition, collaboration
and decision making.
Experiment Study Design and Results
The construction, design, analysis and results of the user study is a major contribution of
this thesis. The design of the study once again solidifies that the retrospective think aloud
108
protocol can be utilized for software where timing and decision making is of importance. The
experiment design also developed a new scenario, training materials, and surveys to help
understand the usability of these types of systems.
Though the scenarios from (Hellar, 2009) and other NeoCITIES 3.0 studies were still
compatible with this version for the most part, they had to be modified in order to include
geographical data. A total of 7 events were implemented into the scenario used in the study, 5 of
which were original with the rest being adapted from previous studies.
The questionnaire that was designed to analyze the usability of the system was
completely original. This questionnaire was to understand the user’s perceptions on how useful
and how much they used each of the components in the interface. Additionally questions were
designed to better understand the participant’s perceptions of their abilities to track events, their
teammates and themselves within the simulation.
The largest part of this contribution is the results that came from the experiment. The
information from the Retrospective Think Aloud and the surveys will be very instrumental in
guiding future design and development of the GeoTMS interaction environment. Additionally,
many of the conclusions on the aspects of the main map and overview map can be applied to
other Geographic Information Systems that have a focus on distributed collaboration. These
results will also be useful in guiding future research in virtual team collaborations.
Future Work
During the theorizing, development and testing phases of this thesis, numerous future
directions have become apparent. These future directions will not only lead to a better interaction
environment, but also may better our theoretical understandings of numerous concepts. Future
work and directions should focus on further developing GeoTMS and NeoCITIES and using
GeoTMS as a test bed to better understand Transactive Memory, team communication
(synchronous vs. asynchronous), and human information fusion.
109
GeoTMS Development
This thesis only presents the first iteration of the design of the GeoTMS interaction
environment. Informed by the usability study conducted here, and further investigation into the
GIS literature future iterations of GeoTMS hope to not only be more usable, but offer a richer
collaborative experience. A final direction that needs to be taken with GeoTMS development is
applying it to different problems and systems.
One thing that became apparent during this user study is that Geospatial visualization is
not a complete replacement for a list--especially in a task like NeoCITIES. Additionally, for
events or situations where the geographic information does not inform the decision maker, by
adding the map, additional complexity that may distract the decision makers rather than empower
them may have been introduced. Because of this, and the apparent individual differences in
preference of visualization, exploring a more adaptable and customizable interface may be a
future design direction. The previous interface made use of static panels, but possibly adapting
the desktop metaphor, and allowing users to choose which components were necessary at a given
point in time might allow for the tool to be even more useful. Additionally, more effort needs to
be spent developing personal awareness measures for the system. So much of the research
focused on developing components to support awareness of teammates that some of the personal
awareness tracking may have slipped through the cracks.
When designing the initial version of GeoTMS, only a fraction of the literature in the
field of GIS was consulted. Further research into this field will help guide future developments
and iterations of the GeoTMS interaction environment. New features such as map annotations
(Cai & Yu, 2009; Hopfer & MacEachren, 2007), natural language interfaces (Chintaphally,
Neumeier, McFarlane, Cothren, & Thompson, 2007; Zhang, Long, Qian, Hu, & Lv, 2007), large-
screen display (A. MacEachren, et al., 2006), to name a few, could be future features that could
be used to enhance the user experience and align the GeoTMS better with the goals and research
out of the GIS field.
A final developmental step that needs to be taken is to apply the GeoTMS interaction
environment to other systems and datasets. When initial development started, the goal was to
design a system that could be applied to numerous systems, datasets and problems, not just make
a new interface for NeoCITIES. By applying the system to new problems and sets of data, new
use cases and interface requirements may become apparent and help future iterations become
more general and applicable.
110
NeoCITIES Development
The current version of NeoCITIES, while effective for initial testing and demonstration
purposes, will have to be expanded to fully support GeoTMS and testing of transactive memory.
Features such as more focus on Geospatial decision making, new data collection mechanisms,
and greater role differentiation will need to be implemented.
The current NeoCITIES simulation engine was initially designed without consideration
for Geospatial problems. Currently, there is very low level support for Geospatial data built into
the NeoCITIES simulation engine. All it currently does is take the latitude and longitude of
events into consideration. Future development should take location into account as a major
element of the data structure. Similar to the GeoTMS information structure (Figure 7), future
developments of NeoCITIES should reform the data structures to first consider the location, then
consider the events that are currently taking place at the location. This reformulation of the
information hierarchy of NeoCITIES should allow it to function better with a mapping
component and make it a more realistic and rich simulation engine. Additionally, the current
scenario structure in NeoCITIES relies on a single description of each event. While this serves
it’s purpose for being an easy to use research platform, it’s simplicity can limit the decision
making necessary within the simulation. Future NeoCITIES scenarios should consider “multiple
descriptions”, which could relate to various on scene reports relating to different aspects of the
event. An example of this conversion can be seen in Table 20.
Table 20: Example of breaking up a NeoCITIES description to multiple information sources Current Description Multiple Sources A bomb threat was received by an instructor in the Forum Building, who panicked and pulled the fire alarm. Units are needed in the following order: FIRST to control the crowd of students running out of the building, SECOND to reset the fire alarm, and THIRD to sweep the area for bombs."
Masses of students have been reported to running out of the forum building. Send units to control the crowd Fire alarm has been pulled in the forum building. Send units to inspect the area and reset the alarm A Professor has reported a bomb threat in the forum building. Send units to sweep the area for bombs
This example shows a very simplistic view of moving from event descriptions to multiple
sources. Future developments will involve multiple types of data (i.e. a webcam image of
students rushing out rather than a text description), and also include competing information from
different sources (i.e. would you believe an old lady on a street or a reporting officer). This
addition can lead to further research in the area of Human Information fusion (discussed later in
this section).
111
Another future development with the current implementation of NeoCITIES is better data
collection. The current data collection focuses mainly on collecting information about their
actions within the simulation (dispatching or recalling units), and some collaborative aspects
(chat logs). Since GeoTMS adds several new complex layers for collaboration, new data
collection mechanisms need to be implemented. Finding ways to collect and store information
based on their actions, map movements, and viewing patterns (which events are in their field of
vision, whose views are they overlapping with, etc.). Once these data collection mechanisms’ are
implemented, research will have to be done on how they can be analyzed to be useful in
analyzing the team’s effectiveness, performance, and collaborative styles.
A final, and probably the most major change to NeoCITIES would have to be a
redefinition of the role structure. If NeoCITIES is to be used as the research platform to test
GeoTMS’s ability to leverage Transactive Memory, the current simplistic role structure will not
work. Since one of the major hurdles in forming transactive memory is varying specialization
(and recognition of everybody’s specialization) of group members, significant changes will have
to be made to NeoCITIES. The current role structure that is implemented in NeoCITIES has very
little variance in their abilities. Essentially, everybody dispatches similar units to different
situations. If NeoCITIES is wished to be used as a research platform to test Transactive Memory,
this will have to be changed. The previous version of NeoCITIES focused on dispatchers and
information managers; however, even this might not be complex enough. Research into other
simulations and new ways the roles can be structured will be necessary in order to move towards
a full tool for testing transactive memory.
Transactive Memory
The main intent of developing the GeoTMS Interaction Environment was to build a
platform to test and understand aspects of Transactive Memory and team collaborations. In many
of the studies that were referenced in the review section, the analysis of the transactive memory
system was based off of observation and survey (self-report) data. While this information is still
useful, research has shown that much of the work that goes into collaboration ends up being
invisible work that may not be able to be observed or reported (Daniels, 1987; Nardi &
ENgeström, 1999; Star & Strauss, 1999). The GeoTMS interaction environment focuses on
112
making this work “visible”, not only as an aid to the collaboration but as a way of studying and
understand how TMS are formed and utilized by teams.
Another main aspect of this thesis was how research out of CSCW, HCI and I/O
Psychology was used to “fill in the gaps” of transactive memory. Future research using GeoTMS
could be effective in understanding how awareness, common ground and Team Mental Models
interact between each other and transactive memory. Using measures that have been proven to
measure awareness (M. Endsley, Aircraft, & Hawthorne, 1988; Gutwin & Greenberg, 2002;
Taylor, 1990), common ground (Beersi, Boshuizen, Kirschner, & Van den Bossche, 2002; G.
Convertino, et al., 2008), Team Mental Models (Mathieu, et al., 2000; Ryan & O'Connor, 2009)
and transactive memory (Lewis, 2003) future research could potentially focus on developing a
model of Transactive Memory for distributed collaborations. This model could inform future
studies and system designs on how exactly transactive memory is formed and utilized in
distributed teams. This type of knowledge could be a crucial aspect of continuing to move
forward with supporting the collaboration of distributed teams.
Finally, while this thesis only focused on a few other domains to try and help support
distributed transactive memory systems, future research could continue to grow and expand this
multidisciplinary perspective on transactive memory. Areas such as distributed cognition, activity
awareness, social awareness, to name a few, are sure to have some overlap and insight on ways a
framework for transactive memory can be developed.
Communication Modality
One of the less grounded design choices made in the GeoTMS Interaction Environment
was to use an asynchronous, “status update” modality of communication rather than a more
traditional synchronous chat modality. While asynchronous communication has been found to
support collaboration in virtual environments (Ocker & Fjermestad, 2008; Phalip, Edmonds, &
Jean, 2009), some of the participants expressed that it might not be the best fit for in the
NeoCITIES simulation. While this system was not solely designed for use in the NeoCITIES
simulation, in general, it must consider time sensitive tasks.
In (den Otter & Emmitt, 2007) the authors discuss how in an effective multidisciplinary
team, team communication is often a balance of both synchronous and asynchronous modalities.
While this is very important aspect of facilitating distributed team work, it must be done very
113
carefully as when a multiple communication protocols are introduced to the team. They may start
to rival each other and hinder the effectiveness of team communication. One possible solution is
to look into using a hybrid asynchronous and synchronous communication model like the one
described in (Geyer, Silva Filho, Brownholtz, & Redmiles, 2008). While simply implementing a
chat and allowing developers to choose which style of communication style to offer would be
easy, more research and testing should be done to try and find an appropriate solution that allows
for effective communication and collaboration without distracting users from their task.
Another aspect of communication investigated in this system was that of action as
language. Since there has been little to no work done in this area as it relates to GeoCollaboration,
many of the solutions were experimental and may not actually be useful. Further investigation
into how users interact with a GeoCollaborative system may better inform future design
considerations on how to augment communication by having the system dynamically report
teammate’s actions. In this thesis, the action as language that was implemented was quite simple
(the action history list), and left many users asking for a more visual representation that didn’t
require them to scroll a list. Additionally, temporal issues (when did the action take place) were
not considered. Future research in this area will have to investigate how peoples’ actions can be
effectively and automatically communicated using map based visualization.
Table 21: Styles of Communication for future development Machine Augmented Communication Communication Channel Yes No Asynchronous I II Synchronous III IV Hybrid V VI None VII VIII
A final direction that can be taken in regards to communications on GeoTMS could lead
to a more theoretical understanding on how communication can aid in distributed
GeoCollaborative tasks. If future research is conducted investigating formation of transactive
memory systems (as discussed in the previous section), communication style could be added as
another experimental. Currently, the system uses an asynchronous and machine augmented
hybrid style of communication, but a synchronous communication channel could easily be added
for testing. All of the potential communication conditions can be seen in Table 21.
114
Human Information Fusion
While the NeoCITIES simulation served as a perfect test bed for usability analysis, it
does not take advantage of all the GeoTMS information hierarchy has to offer. The purpose of the
data structures and information hierarchy is so that the system can handle multiple inputs or
sources attaching them to a singular event or location. When merging multiple sources of
information and presenting them to the user, the human becomes a soft sensor for information
fusion.
Future development of this system could allow human collaborators to post and edit
information in a collaborative environment. Since humans, unlike machines, have a higher
probability of error, by opening up this system to a larger number of users, the GeoTMS
Interaction Environment could become a system for crowdsourcing Geospatial data. This system
could be a perfect platform for tasking, knowledge elicitation and fusion with hard sensor data
(Hall, 2010).
An example use case of this can be seen from the recent tragedy out of Haiti, where many
of the NGOs and relief efforts had trouble identifying roadways and geographic areas due to the
lack of cartographic information of the surrounding areas. While this information was not stored
in any actual geographic database, there was a good chance it was knowledge that many Haitian
immigrants had. In this situation, the GeoTMS Interaction Environment would have been a
perfect platform to allow for Haitians from all over the world to login and post their knowledge of
the area and confirm other people’s postings. Unfortunately, it is too late for this tool to be of
assistance, but it may be a useful tool in crowdsourcing cartographic and other geographical
information for future situations like this.
When depending on humans as soft sensors it is often the case that there are many more
collaborators than 3 (the number used in the NeoCITIES simulation) interacting within the
workspace. While the tools infrastructure could support a theoretical infinite number of users
within the workspace, the interface might not be as scalable. This brings up a new interface
consideration for large scale team interactions.
The area of Human Information Fusion is a very interesting and popular new field in
which there is still not a complete understanding of the collaborations that are necessary. Similar
to the other theories in this paper, the visible nature of GeoTMS may provide an excellent test bed
to better understand how humans can collaboratively act as soft sensors to help create, edit and
fuse information from multiple sources.
References
Abrams, G., & Haefner, J. (1998). SHOWME: Spear-Heading Online Work in Mathematics Education. THE Journal (Technological Horizons In Education), 25(10).
Adibhatla, V., Shapiro, A., & McNeese, M. (2009). Design and development of a Transactive memory system prototype for geocollaborative crisis management. Paper presented at the Proceedings of the HFES.
Akgün, A. E., Byrne, J., Keskin, H., Lynn, G. S., & Imamoglu, S. Z. (2005). Knowledge networks in new product development projects: a transactive memory perspective. Information & Management, 42(8), 1105-1120.
Alarcón, R. A., Guerrero, L. A., & Pino, J. A. (2005). Temporal blurring: a privacy model for OMS users. Lecture Notes in Computer Science, 3538, 417.
Aliakseyeu, D., Lucero, A., & Martens, J.-B. (2009). Users' quest for an optimized representation of a multi-device space. Personal Ubiquitous Comput., 13(8), 599-607.
Alshattnawi, S., Canals, G., & Molli, P. (2008). Concurrency awareness in a p2p wiki system. Proceedings of CTS.
Anderson, K. C., & Leinhardt, G. (2002). Maps as representations: Expert novice comparison of projection understanding. Cognition and Instruction, 20(3), 283-321.
Andrienko, G., Andrienko, N., Jankowski, P., Keim, D., Kraak, M. J., MacEachren, A., et al. (2007). Geovisual analytics for spatial decision support: Setting the research agenda. International Journal of Geographical Information Science, 21(8), 839-858.
Austin, J. R. (2003). Transactive memory in organizational groups: The effects of content, consensus, specialization, and accuracy on group performance. Journal of Applied Psychology, 88(5), 866-877.
Baecker, R. M., Nastos, D., Posner, I. R., & Mawby, K. L. (1993). The user-centered iterative design of collaborative writing software. Paper presented at the Proceedings of the INTERACT '93 and CHI '93 conference on Human factors in computing systems, Amsterdam, The Neatherlands.
Balakrishnan, B., Pfaff, M., McNeese, M. D., & Adibhatla, V. (2009). NeoCITIES Geo-tools: Assessing Impact of Perceptual Anchoring and Spatially Annotated Chat on Geo collaboration. Paper presented at the Human Factors and Ergonomics Society.
Bardram, J. E., Hansen, T. R., & Soegaard, M. (2006). AwareMedia: a shared interactive display supporting social, temporal, and spatial awareness in surgery. Paper presented at the Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work.
Battleson, B., Booth, A., & Weintrop, J. (2001). Usability testing of an academic library web site: a case study. The Journal of Academic Librarianship, 27(3), 188-198.
Beersi, P., Boshuizen, H., Kirschner, P., & Van den Bossche, P. (2002). Decision-support and Complexity in Decision Making. Paper presented at the 5th JURE conference.
116
Benbunan-Fich, R. (2001). Using protocol analysis to evaluate the usability of a commercial web site. Information & Management, 39(2), 151-163.
Biehl, J. T., Czerwinski, M., Smith, G., & Robertson, G. G. (2007). FASTDash: a visual dashboard for fostering awareness in software teams. Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems.
Bitter-Rijpkema, M., Martens, R., & Jochems, W. (2002). Supporting knowledge elicitation for learning in virtual teams. Educational Technology & Society, 5(2), 113-118.
Blakar, R. M. (1985). Towards a theory of communication in terms of preconditions: A conceptual framework and some empirical explorations. Recent advances in language, communication and social psychology. London: Lawrence Erlbaum.
Blandford, A., & Wong, W. (2004). Situation awareness in emergency medical dispatch. International Journal of Human-Computer Studies, 61(4), 421-452.
Brewer, I., & McNeese, M. (2004). Expanding concept mapping to address spatio-temporal dimensionality. Paper presented at the Proceedings of the First International Concept on Concept Mapping, Pamplona, Spain.
Burke, C., Stagl, K., Salas, E., Pierce, L., & Kendall, D. (2006). Understanding team adaptation: A conceptual analysis and model. Journal of Applied Psychology, 91(6), 1189.
Cai, G., & Yu, B. (2009). Spatial Annotation Technology for Public Deliberation. Transactions in GIS, 13(s1), 123-146.
Canney Davison, S., & Ward, K. (1999). Leading international teams. McGraw-Hill International.
Cannon-Bowers, J., & Salas, E. (2001). Reflections on shared cognition. Journal of Organizational Behavior, 22(2), 195-202.
Carley, K. M., & Ren, Y. (2001). Tradeoffs between performance and adaptability for C3I architectures. Paper presented at the Proceedings of the 2001 Command and Control Research and Technology Symposium, Annapolic, Maryland.
Carlile, P. R. (2002). A pragmatic view of knowledge and boundaries: Boundary objects in new product development. Organization science, 13(4), 442-455.
Carlile, P. R. (2004). Transferring, translating, and transforming: An integrative framework for managing knowledge across boundaries. Organization science, 555-568.
Carroll, J., & Kellogg, W. (1989). Artifact as theory-nexus: hermeneutics meets theory-based design. ACM SIGCHI Bulletin, 20(SI), 14.
Carroll, J. M., Rosson, M. B., Farooq, U., & Xiao, L. (2009). Beyond being aware. Information and Organization.
Carver, S., Evans, A., Kingston, R., & Turton, I. (2001). Public participation, GIS, and cyberdemocracy: evaluating on-line spatial decision support systems. Environment and Planning B, 28(6), 907-922.
Chang, B. W., & Ungar, D. (1995). Animation: from cartoons to the user interface. Paper presented at the Technical Report: TR-95-33
Chen, X. (2009). Knowledge coordination in open source software project teams: A transactive memory system perspective. The Univeristy of Texas at San Antonio.
117
Chen, X., & Dietrich, G. (2009). Knowledge Location, Differentiation, Credibility and Coordination in Open Source Software Development Teams. AMCIS 2009 Proceedings, 389.
Chintaphally, V., Neumeier, K., McFarlane, J., Cothren, J., & Thompson, C. (2007). Extending a natural language interface with geospatial queries. IEEE Internet Computing, 82-85.
Chuah, M. C., & Roth, S. F. (2003). Visualizing common ground. Paper presented at the Proceedings of the Seventh International Conference on Information Visualization (IV’03).
Clark, H., & Brennan, S. (1991). Grounding in communication. Perspectives on socially shared cognition, 13, 127–149.
Clark, H. H., & Wilkes-Gibbs, D. (1986). Referring as a Collaborative Process. Cognition, 22, 139.
Convertino, G., Mentis, H. M., Rosson, M. B., Carroll, J. M., Slavkovic, A., & Ganoe, C. H. (2008). Articulating common ground in cooperative work: content and process. Paper presented at the Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems.
Convertino, G., Mentis, H. M., Rosson, M. B., Slavkovic, A., & Carroll, J. M. (2009). Supporting content and process common ground in computer-supported teamwork. Paper presented at the Proceedings of the 27th international conference on Human factors in computing systems.
Convertino, G., Zhao, D., Ganoe, C., Carroll, J., & Rosson, M. (2007). A role-based multiple view approach to distributed geo-collaboration. Human-Computer Interaction. HCI Applications and Services, 561-570.
Cottone, P., Pieti, L., Schiavinato, V., Soru, D., Martinelli, M., Varotto, D., et al. (2009). “Solving” ambiguity in the virtual space: communication strategies in a collaborative virtual environment. Cognition, Technology & Work, 11(2), 151-163.
Cramton, C. D. (2001). The mutual knowledge problem and its consequences for dispersed collaboration. Organization science, 346-371.
Cramton, C. D. (2002). Finding common ground in dispersed collaboration. Organizational Dynamics, 30(4), 356-367.
Da Yang, D. W., Koolmanojwong, S., Brown, A. W., & Boehm, B. W. (2008). Wikiwinwin: A wiki based system for collaborative requirements negotiation. Paper presented at the Proceedings of the 41st Annual Hawaii International Conference on System Sciences.
Daniels, A. (1987). Invisible work. Soc. Probs., 34, 403.
den Otter, A., & Emmitt, S. (2007). Exploring effectiveness of team communication. Engineering, Construction and Architectural Management, 14(5).
Desanctis, G., & Monge, P. (1999). Introduction to the special issue: Communication processes for virtual organizations. Organization science, 693-703.
Dierker, A., Mertes, C., Hermann, T., Hanheide, M., & Sagerer, G. (2009). Mediated attention with multimodal augmented reality. Paper presented at the Proceedings of the 2009 international conference on Multimodal interfaces.
118
Dix, A., Finlay, J., Abowd, G. D., & Beale, R. (2004). Human-computer interaction: Prentice hall.
Dörner, C., Pipek, V., & Won, M. (2007). Supporting expertise awareness: finding out what others know. Paper presented at the Proceedings of the 2007 symposium on Computer human interaction for the management of information technology.
Dourish, P., & Bellotti, V. (1992). Awareness and coordination in shared workspaces. Paper presented at the Proceedings of the 1992 ACM conference on Computer-supported cooperative work, Toronto, Ontario, CA.
Dourish, P., & Bly, S. (1992). Portholes: Supporting awareness in a distributed work group. Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems, Monterey, CA USA.
Dragicevic, S., & Balram, S. (2004). A Web GIS collaborative framework to structure and manage distributed planning processes. Journal of Geographical Systems, 6(2), 133-153.
Drummond, W. J., & French, S. P. (2008). The Future of GIS in Planning: Converging Technologies and Diverging Interests. Journal of the American Planning Association, 74(2), 161-174.
Ducheneaut, N., Wen, M. H., Yee, N., & Wadley, G. (2009). Body and mind: a study of avatar personalization in three virtual worlds. Paper presented at the Proceedings of the 27th international conference on Human factors in computing systems.
Ellis, A. P. J. (2006). System breakdown: The role of mental models and transactive memory in the relationship between acute stress and team performance. Academy of Management Journal, 49(3), 576-589.
Endsley, M., Aircraft, N., & Hawthorne, C. (1988). Situation awareness global assessment technique (SAGAT). Paper presented at the Aerospace and Electronics Conference, 1988. NAECON 1988., Proceedings of the IEEE 1988 National.
Endsley, M. R. (1989). Final report: Situation awareness in an advanced strategic mission (NOR DOC 89-32). Hawthorne, CA: Northrop Corporation.
Endsley, M. R. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors: The Journal of the Human Factors and Ergonomics Society, 37(1), 32-64.
Endsley, M. R. (1999). Situation awareness in aviation systems. Handbook of aviation human factors, 257-276.
Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological review, 87(3), 215-251.
Espinosa, J., Slaughter, S., Kraut, R., & Herbsleb, J. (2007). Team knowledge and coordination in geographically distributed software development. Journal of Management Information Systems, 24(1), 135-169.
Fleming, S., Jordan, T., Madden, M., Usery, E. L., & Welch, R. (2009). GIS applications for military operations in coastal zones. ISPRS Journal of Photogrammetry and Remote Sensing, 64(2), 213-222.
Forgues, D., Koskela, L., & Lejeune, A. (2009). Information technology as boundary object for transformational learning. Journal of Information Technology in Construction.
119
Fraser, M., McCarthy, M. R., Shaukat, M., & Smith, P. (2007). Seconds matter: improving distributed coordination bytracking and visualizing display trajectories. Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems.
Frauenberger, C., Stockman, T., & Bourguet, M. L. (2007). A Survey on Common Practice In Designing Audio in the User Interface. Paper presented at the Proceedings of the British HCI Group 2007, London, England.
French, H., & Hutchinson, A. (2002). Measurement of situation awareness in a C4ISR experiment. Paper presented at the 2002 Command and Control Research and Technology Symposium.
Fussell, S. R., & Krauss, R. M. (1992). Coordination of knowledge in communication: Effects of speakers’ assumptions about what others know. Journal of Personality and Social Psychology, 62(3), 378-391.
Gaver, B. (2002). Provocative awareness. Computer Supported Cooperative Work (CSCW), 11(3), 475-493.
Gaver, W. W. (1991). Sound support for collaboration. Paper presented at the Proceedings of the second conference on European Conference on Computer-Supported Cooperative Work, Amsterdam, The Netherlands.
George, C. A. (2008). Lessons learned: usability testing a federated search product. Electronic Library, 26(1), 5-20.
Gergle, D., Kraut, R. E., & Fussell, S. R. (2004a). Action as language in a shared visual space. Paper presented at the Proceedings of the 2004 ACM conference on Computer supported cooperative work, Chicago, Illinois, USA
Gergle, D., Kraut, R. E., & Fussell, S. R. (2004b). Language efficiency and visual technology: Minimizing collaborative effort with visual information. Journal of Language and Social Psychology, 23(4), 491.
Geyer, W., Silva Filho, R., Brownholtz, B., & Redmiles, D. (2008). The Trade-Offs of Blending Synchronous and Asynchronous Communication Services to Support Contextual Collaboration. Journal of Universal Computer Science, 14(1), 4-26.
Greenspan, S., Goldberg, D., Weimer, D., & Basso, A. (2000). Interpersonal trust and common ground in electronically mediated communication. Paper presented at the Proceedings of the 2000 ACM conference on Computer supported cooperative work Philadelphia, PA USA.
Guan, Z., Lee, S., Cuddihy, E., & Ramey, J. (2006). The validity of the stimulated retrospective think-aloud method as measured by eye tracking. Paper presented at the Proceedings of the SIGCHI conference on Human Factors in computing systems, Montreal, Quebec Canada.
Guappone, K. P., Ash, J. S., & Sittig, D. F. (2008). Field evaluation of commercial Computerized Provider Order Entry systems in community hospitals. Paper presented at the AMIA Annu Symp Proc.
Gutwin, C., & Greenberg, S. (1998). Design for individuals, design for groups: tradeoffs between power and workspace awareness. Paper presented at the Proceedings of the 1998 ACM conference on Computer supported cooperative work, Seattle, WA USA.
120
Gutwin, C., & Greenberg, S. (2002). A descriptive framework of workspace awareness for real-time groupware. Computer Supported Cooperative Work (CSCW), 11(3), 411-446.
Gutwin, C., & Greenberg, S. (2004). The importance of awareness for team cognition in distributed collaboration. Team cognition: Understanding the factors that drive process and performance, 201.
Gutwin, C., Greenberg, S., Blum, R., Dyck, J., Tee, K., & McEwan, G. (2008). Supporting informal collaboration in shared-workspace groupware. Journal of Universal Computer Science, 14(9), 1411-1434.
Gutwin, C., Greenberg, S., & Roseman, M. (1996). Workspace awareness support with radar views. Paper presented at the Conference companion on Human factors in computing systems: common ground, Vancouver, British Columbia Canada.
Gutwin, C., Roseman, M., & Greenberg, S. (1996). A usability study of awareness widgets in a shared workspace groupware system. Paper presented at the Proceedings of the 1996 ACM conference on Computer supported cooperative work, Boston, MA USA.
Hall, D. (2010). Human-centered information fusion.
Hamilton, K. (2010). NeoCITIES 3.0 Performance scores and Reaction Times. In V. Mancuso (Ed.). State College, Pa.
Hancock, M., Miller, J., Greenberg, S., & Carpendale, S. (2006). Exploring visual feedback of change conflict in a distributed 3D environment. Proceedings of Advanced Visual Interfaces (AVI'06).
Hazlehurst, B., McMullen, C., & Gorman, P. (2007). Distributed cognition in the heart room: How situation awareness arises from coordinated communications during cardiac surgery. Journal of biomedical informatics, 40(5), 539-551.
Healey, M. P., Hodgkinson, G. P., & Teo, S. (2009). Responding Effectively to Civil Emergencies: The Role of Transactive Memory in the Performance of Multiteam Systems. Paper presented at the The 9th international Conference on Naturalistic Decision Making, London, UK.
Healey, P. (2003). Collaborative planning in perspective. Planning theory, 2(2), 101.
Hellar, D. B. (2009). An Investigation of Data Overload in Team-Based Distributed Cognition Systems. The Pennsylvania State University, State College, Pa.
Hellar, D. B., & Hall, D. L. (2009). NeoCITIES: an experimental test-bed for quantifying the effects of cognitive aids on team performance in C2 situations. Paper presented at the Proceedings of SPIE.
Henderson, R. D., Smith, M. C., Podd, J., & Varela-Alvarez, H. (1995). A comparison of the four prominent user-based methods for evaluating the usability of computer software. Ergonomics, 38(10), 2030-2044.
Hill, J., & Gutwin, C. (2003). Awareness support in a groupware widget toolkit. Paper presented at the Proceedings of the 2003 international ACM SIGGROUP conference on Supporting group work, Sanibel Island, FL USA.
Hill, J., & Gutwin, C. (2004). The MAUI toolkit: Groupware widgets for group awareness. Computer Supported Cooperative Work (CSCW), 13(5), 539-571.
121
Hollingshead, A. B. (1998a). Communication, learning, and retrieval in transactive memory systems. Journal of Experimental Social Psychology, 34(5), 423-442.
Hollingshead, A. B. (1998b). Group and Individual Training: The Impact of Practice on Performance. Small Group Research, 29(2), 254.
Hollingshead, A. B. (2000). Perceptions of expertise and transactive memory in work relationships. Group Processes & Intergroup Relations, 3(3), 257.
Hollingshead, A. B. (2001). Cognitive interdependence and convergent expectations in transactive memory. Journal of Personality and Social Psychology, 81(6), 1080-1089.
Holyoak, K. J. (1984). Mental models in problem solving. Tutorials in learning and memory, 193-218.
Hopfer, S., & MacEachren, A. (2007). Leveraging the potential of geospatial annotations for collaboration: a communication theory perspective. International Journal of Geographical Information Science, 21(8), 921-934.
Hsu, J. S. C., Chang, J. Y. T., Klein, G., & Jiang, J. J. (2010). Exploring the impact of team mental models on information utilization and project performance in system development. International Journal of Project Management.
Hsu, J. S. C., Parolia, N., Jiang, J. J., & Klein, G. (2007). The Impact of Team Mental Models on IS Project Teams’ Information Processing and Project Performance. Paper presented at the International Research Workshop on IT Project Management 2007.
Hutchins, E., & Lintern, G. (1996). Cognition in the Wild: MIT press Cambridge, MA.
Iacono, C. S., & Weisband, S. (1997). Developing trust in virtual teams. Paper presented at the Proceedings of the Hawaii International Conference on Systems Sciences, Hawaii USA.
Ignat, C. L., Papadopoulou, S., Oster, G., & Norrie, M. C. (2008). Providing awareness in multi-synchronous collaboration without compromising privacy. Paper presented at the Proceedings of the ACM 2008 conference on Computer supported cooperative work.
Ilgen, D. R., Hollenbeck, J. R., Johnson, M., & Jundt, D. (2004). Teams in organizations: From input-process-output models to IMOI models. Annual Review of Psychology, 56, 517-543.
Ingram, A. L. (2005). Engagement in online learning communities. Elements of quality online education: Engaging communities, 6.
Iqbal, S. T., & Horvitz, E. (2010). Notifications and awareness: a field study of alert usage and preferences. Paper presented at the Proceedings of the 2010 ACM conference on Computer supported cooperative work.
Jackson, M., & Moreland, R. L. (2009). Transactive Memory in the Classroom. Small Group Research, 40(5), 508.
Jarvenpaa, S. L., Knoll, K., & Leidner, D. E. (1998). Is anybody out there?: antecedents of trust in global virtual teams. Journal of Management Information Systems, 14(4), 64.
Jarvenpaa, S. L., & Leidner, D. E. (1999). Communication and trust in global virtual teams. Organization science, 791-815.
122
Jefferson, T., Ferzandi, L., & McNeese, M. (2004). Impact of hidden profiles on distributed cognition in spatially distributed decision-making teams. Paper presented at the Human Factors and Ergonomics Society Annual Meeting Proceedings
Jiang, H., & Singley, K. (2009). Exploring bilingual, task-oriented, document-centric chat. Paper presented at the Proceedings of the ACM 2009 international conference on Supporting group work.
John, M. S., & Dister, B. (2009). Wireless Collaboration Technology to Support Distributed Teams. Cognitive Technology Journal, 14(1).
John, M. S., & Smallman, H. S. (2008). Shared Lightweight Annotation TEchnology (SLATE) for Special Operations Forces. Macrocognition in Teams: Theories and Methodologies, 305.
Johnson-Laird, P. N. (1983). Mental models: Towards a cognitive science of language, inference and consciousness: Harvard Univ Pr.
Johnson, M. (1988). Reality monitoring: An experimental phenomenological approach. Journal of Experimental Psychology: General, 117(4), 390-394.
Johnson, M., Hashtroudi, S., & Lindsay, D. (1993). Source monitoring. Psychological Bulletin, 114, 3-3.
Jones, D. G., & Endsley, M. R. (1996). Sources of situation awareness errors in aviation. Aviation, Space, and Environmental Medicine, 67(6), 507-512.
Jones, R. E. T., McNeese, M. D., Connors, E. S., Jefferson, T., & Hall, D. L. (2004). A distributed cognition simulation involving homeland security and defense: The development of NeoCITIES. Paper presented at the Human Factors and Ergonomics Society.
Junglas, I. A., Johnson, N. A., Steel, D. J., Abraham, D. C., & Loughlin, P. M. (2007). Identity formation, learning styles and trust in virtual worlds. ACM SIGMIS Database, 38(4), 96.
Kanawattanachai, P., & Yoo, Y. (2002). Dynamic nature of trust in virtual teams. Journal of Strategic Information Systems, 11(3-4), 187-213.
Kang, S. H. (2006). The Effect of Avatar Realism on Users’ Sense of Social Presence in Emotionally Engaged Mobile-Phone Communication. Paper presented at the Seventh IEEE Workshop on Mobile Computing Systems & Applications.
Karabenick, S. A. (1996). Social influences on metacognition: Effects of colearner questioning on comprehension monitoring. Journal of educational Psychology, 88, 689-703.
Keel, P. E. (2007). EWall: A visual analytics environment for collaborative sense-making. Information Visualization, 6(1), 48-63.
Keßler, C., Rinner, C., & Raubal, M. (2005). An Argumentation Map Prototype to Support Decision-Making in Spatial Planning. Paper presented at the Proceedings of AGILE.
Kingston, R., Carver, S., Evans, A., & Turton, I. (2000). Web-based public participation geographical information systems: an aid to local environmental decision-making. Computers, Environment and Urban Systems, 24(2), 109-125.
Klimoski, R., & Mohammed, S. (1994). Team mental model: construct or metaphor? Journal of Management, 20(2), 403.
123
Kozlowski, S. W., & Ilgen, D. R. (2006). Enhancing the effectiveness of work groups and teams. Psychological Science in the Public Interest, 7(3), 77.
Kraak, M. J. (2004). The role of the map in a Web-GIS environment. Journal of Geographical Systems, 6(2), 83-93.
Krauss, R. M., & Fussell, S. R. (1990). Mutual knowledge and communicative effectiveness. Intellectual teamwork: Social and technological foundations of cooperative work, 111-146.
Kraut, R. E., Gergle, D., & Fussell, S. R. (2002). The use of visual information in shared visual spaces: Informing the development of virtual co-presence. Paper presented at the Proceedings of the 2002 ACM conference on Computer supported cooperative work, New Orleans, LA USA.
Kushniruk, A. W., & Patel, V. L. (1998). Cognitive evaluation of decision making processes and assessment of information technology in medicine. International journal of medical informatics, 51(2-3), 83-90.
Lampe, C., Ellison, N., & Steinfield, C. A. (2007). A familiar face (book): Profile elements as signals in an online social network. CHI 2007 Proceedings. Online Representation of Self.
Lampe, C. A. C., Johnston, E., & Resnick, P. (2007). Follow the reader: filtering comments on slashdot. Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems.
Langan-Fox, J., Code, S., & Langfield-Smith, K. (2000). Team mental models: Techniques, methods, and analytic approaches. Human Factors, 42(2), 242.
LeCompte, M., & Schensul, J. (1999). Analyzing & interpreting ethnographic data: Altamira Pr.
Lee, C. P. (2007). Boundary negotiating artifacts: Unbinding the routine of boundary objects and embracing chaos in collaborative work. Computer Supported Cooperative Work (CSCW), 16(3), 307-339.
Leitner, H., McMaster, R. B., Elwood, S., McMaster, S., & Sheppard, E. (2002). Models for making GIS available to community organizations: dimensions of difference and appropriateness. Community participation and geographic information systems, 37-52.
Leshed, G., Perez, D., Hancock, J. T., Cosley, D., Birnholtz, J., Lee, S., et al. (2009). Visualizing real-time language-based feedback on teamwork behavior in computer-mediated groups. Paper presented at the Proceedings of the 27th international conference on Human factors in computing systems.
Levesque, L., Wilson, J., & Wholey, D. (2001). Cognitive divergence and shared mental models in software development project teams. Journal of Organizational Behavior, 22(2), 135-144.
Lewis, K. (2003). Measuring transactive memory systems in the field: Scale development and validation. Journal of Applied Psychology, 88(4), 587-603.
Lewis, K. (2004). Knowledge and performance in knowledge-worker teams: A longitudinal study of transactive memory systems. Management science, 50(11), 1519-1533.
Li, Q. I. N., Bo, C., Ju, H. U., & Guo-xue, L. I. (2009). Application of GIS to Solid Waste Management and Planning. Environmental Science & Technology, 1.
124
Liang, D. W., Moreland, R., & Argote, L. (1995). Group versus individual training and group performance: The mediating factor of transactive memory. Personality and Social Psychology Bulletin, 21(4), 384-393.
Lim, B. C., & Klein, K. J. (2006). Team mental models and team performance: A field study of the effects of team mental model similarity and accuracy. Journal of Organizational Behavior, 27(4), 403.
Littlepage, G. E., Hollingshead, A. B., Drake, L. R., & Littlepage, A. M. (2008). Transactive memory and performance in work groups: Specificity, communication, ability differences, and work allocation. Group Dynamics: Theory, Research, and Practice, 12(3), 223-241.
Looney, C. G. (2001). Exploring fusion architecture for a common operational picture. Information fusion, 2(4), 251-260.
MacEachren, A., Cai, G., Brewer, I., & Chen, J. (2006). Supporting Map-based Geocollaboration through Natural Interfaces to Large-screen Displays. Cartographic Perspectives, 54, 4–22.
MacEachren, A. M., & Kraak, M. J. (2001). Research challenges in geovisualization. Cartography and Geographic Information Science, 28(1), 3-12.
Madhavan, R., & Grover, R. (1998). From embedded knowledge to embodied knowledge: new product development as knowledge management. The Journal of Marketing, 62(4), 1-12.
Mathieu, J. E., Heffner, T. S., Goodwin, G. F., Salas, E., & Cannon-Bowers, J. A. (2000). The influence of shared mental models on team process and performance. Journal of Applied Psychology, 85(2), 273-283.
Maznevski, M., & Chudoba, K. (2000). Bridging space over time: Global virtual team dynamics and effectiveness. Organization science, 473-492.
McGookin, D., & Brewster, S. (2007). An initial investigation into non-visual computer supported collaboration. Paper presented at the CHI '07 extended abstracts on Human factors in computing systems.
McGrath, D., & McGrath, S. (2005). Simulation and network-centric emergency response. Paper presented at the The Interservice/Industry Training, Simulation & Education Conference (I/ITSEC).
McNeese, M., Pfaff, M., Connors, E., Obieta, J., Terrell, I., & Friedenberg, M. (2006). Multiple vantage points of the Common Operational Picture: Supporting Complex Teamwork. Paper presented at the 50th Annual Meeting of the Human Factors and Ergonomics Society, San Francisco, CA.
McNeese, M. D., Bains, P., Brewer, I., Brown, C., Connors, E. S., Jefferson, T., et al. (2005). The NeoCITIES simulation: Understanding the design and experimental methodology used to develop a team emergency management simulation. Paper presented at the Human Factors and Ergonomics Society.
McNeese, M. D., Perusich, K., & Rentsch, J. R. (2000). Advancing socio-technical systems design via the living laboratory. Paper presented at the Human Factors and Ergonomics Society.
125
Michinov, N., & Michinov, E. (2009). Investigating the relationship between transactive memory and performance in collaborative learning. Learning and Instruction, 19(1), 43-54.
Mohammed, S., & Dumville, B. C. (2001). Team mental models in a team knowledge framework: Expanding theory and measurement across disciplinary boundaries. Journal of Organizational Behavior, 22(2), 89-106.
Mohammed, S., Ferzandi, L., & Hamilton, K. (2010). Metaphor No More: A 15-Year Review of the Team Mental Model Construct. Journal of Management.
Mohammed, S., Klimoski, R., & Rentsch, J. (2000). The measurement of team mental models: We have no shared schema. Organizational Research Methods, 3(2), 123.
Moreland, R. L. (1999). Transactive Memory: Learning Who Knows What in Work Groups and Organizations. Small groups: Key readings, 327.
Moreland, R. L., & Myaskovsky, L. (2000). Exploring the performance benefits of group training: Transactive memory or improved communication? Organizational Behavior and Human Decision Processes, 82(1), 117-133.
Muller, M. J., Smith, J. G., Shoher, J. Z., & Goldberg, H. (1991). Privacy, anonymity and interpersonal competition issues identified during participatory design of project management groupware! ACM SIGCHI Bulletin, 23(1), 82-87.
Nacenta, M. A., Gutwin, C., Aliakseyeu, D., & Subramanian, S. (2009). There and back again: cross-display object movement in multi-display environments. Human-Computer Interaction, 24(1), 170-229.
Nacenta, M. A., Pinelle, D., Stuckel, D., & Gutwin, C. (2007). The effects of interaction technique on coordination in tabletop groupware. Paper presented at the Proceedings of Graphics Interface 2007.
Nardi, B., & ENgeström, Y. (1999). A web on the wind: The structure of invisible work. Computer Supported Cooperative Work (CSCW), 8(1), 1-8.
Nees, M. A., & Walker, B. N. (2009). Auditory interfaces and sonification. The Universal Access Handbook, 507-521.
Neustaedter, C., Brush, A., & Greenberg, S. (2007). A digital family calendar in the home: lessons from field trials of LINC. Paper presented at the Proceedings of Graphics Interface 2007.
Nielsen, J. (1993). Usability engineering. AP Professional, Boston.
Obieta, J. (2006). Scenarios for Crisis Management Simulations. The Pennsylvania State University, State College, Pa.
Ocker, R. J., & Fjermestad, J. (2008). Communication differences in virtual design teams: findings from a multi-method analysis of high and low performing experimental teams. SIGMIS Database, 39(1), 51-67.
Oshri, I., van Fenema, P., & Kotlarsky, J. (2008). Knowledge transfer in globally distributed teams: the role of transactive memory. Information Systems Journal, 18(6), 593-616.
Park, K. S., Kapoor, A., & Leigh, J. (2000). Lessons learned from employing multiple perspectives in a collaborative virtual environment for visualizing scientific data. Paper
126
presented at the Proceedings of the third international conference on Collaborative virtual environments, San Francisco, CA USA.
Peterson, G. (2009). GIS Cartography: A Guide to Effective Map Design: CRC.
Phalip, J., Edmonds, E. A., & Jean, D. (2009). Supporting remote creative collaboration in film scoring. Paper presented at the Proceeding of the seventh ACM conference on Creativity and cognition.
Phelps, A. F., & Reddy, M. (2009). The influence of boundary objects on group collaboration in construction project teams. Paper presented at the Proceedings of the ACM 2009 international conference on Supporting group work.
Pinelle, D., Gutwin, C., & Greenberg, S. (2003). Task analysis for groupware usability evaluation: Modeling shared-workspace tasks with the mechanics of collaboration. ACM Transactions on Computer-Human Interaction (TOCHI), 10(4), 281-311.
Powell, A., Piccoli, G., & Ives, B. (2004). Virtual teams: a review of current literature and directions for future research. ACM SIGMIS Database, 35(1), 6-36.
Prince, C., & Salas, E. (2000). Team situation awareness, errors, and crew resource management: Research integration for training guidance. Situation awareness analysis and measurement, 325–347.
Ramey, J., Boren, T., Cuddihy, E., Dumas, J., Guan, Z., van den Haak, M. J., et al. (2006). Does think aloud work?: how do we know? Paper presented at the CHI '06 extended abstracts on Human factors in computing systems, Montreal, Quebec Canada.
Reetz, A., Gutwin, C., Stach, T., Nacenta, M., & Subramanian, S. (2006). Superflick: a natural and efficient technique for long-distance object placement on digital tables. Paper presented at the Proceedings of Graphics interface 2006.
Rinner, C. (2001). Argumentation maps: GIS-based discussion support for on-line planning. Environment and Planning B: Planning and Design, 28(6), 847-863.
Rosen, B., Furst, S., & Blackburn, R. (2007). Overcoming barriers to knowledge sharing in virtual teams. Organizational Dynamics, 36(3), 259-273.
Rouse, W., Cannon-Bowers, J., & Salas, E. (1992). The role of mental models in team performance in complex systems. IEEE transactions on systems, man, and cybernetics, 22(6), 1296-1308.
Ryan, S., & O'Connor, R. (2009). Development of a team measure for tacit knowledge in software development teams. Journal of Systems and Software, 82(2), 229-240.
Salas, E., Cooke, N., & Rosen, M. (2008). On teams, teamwork, and team performance: Discoveries and developments. Human Factors, 50(3), 540.
Salas, E., Prince, C., Baker, D. P., & Shrestha, L. (1995). Situation awareness in team performance: Implications for measurement and training. Human Factors: The Journal of the Human Factors and Ergonomics Society, 37(1), 123-136.
Salmon, P., Stanton, N., Walker, G., & Green, D. (2006). Situation awareness measurement: A review of applicability for C4i environments. Applied Ergonomics, 37(2), 225-238.
127
Sánchez, J., Baloian, N., Hassler, T., & Hoppe, U. (2003). AudioBattleShip: Blind learners collaboration through sound. Paper presented at the CHI'03 extended abstracts on Human factors in computing systems.
Sarcevic, A., Marsic, I., Lesk, M. E., & Burd, R. S. (2008). Transactive memory in trauma resuscitation. Paper presented at the Proceedings of the ACM 2008 conference on Computer supported cooperative work.
Sarker, S., Nicholson, D., & Joshi, K. (2003). Knowledge transfer in virtual information systems development teams: an empirical examination of key enablers. Paper presented at the Proceedings of the 36th Hawaii International Conference on System Sciences, Hawaii USA.
Sarker, S., Nicholson, D., & Joshi, K. (2005). Knowledge transfer in virtual systems development teams: An exploratory study of four key enablers. IEEE Transactions on Professional Communication, 48(2), 201-218.
Sarter, N. B., & Woods, D. D. (1991). Situation awareness: A critical but ill-defined phenomenon. The International Journal of Aviation Psychology, 1(1), 45-57.
Schafer, W. A., & Bowman, D. A. (2006). Supporting distributed spatial collaboration: An investigation of navigation and radar view techniques. GeoInformatica, 10(2), 123-158.
Schafer, W. A., Ganoe, C. H., & Carroll, J. M. (2007). Supporting community emergency management planning through a geocollaboration software architecture. Computer Supported Cooperative Work (CSCW), 16(4), 501-537.
Schlichter, J., Koch, M., & Bürger, M. (1998). Workspace awareness for distributed teams. Lecture Notes in Computer Science, 1364, 199-218.
Scozzi, B., Crowston, K., Eseryel, U. Y., & Li, Q. (2008). Shared mental models among open source software developers. Paper presented at the Proceedings of the 41st Annual Hawaii International Conference on System Sciences (HICSS 2008).
Sigala, M. (2009). Geoportals and Geocollaborative Portals: Functionality and Impacts on Travellers' Trip Panning and Decision Making Processes. Paper presented at the International CHRIE Conference-Refereed Track.
Smith-Jentsch, K. A., Cannon-Bowers, J. A., Tannenbaum, S. I., & Salas, E. (2008). Guided team self-correction: Impacts on team mental models, processes, and effectiveness. Small Group Research, 39(3), 303.
Star, S. L., & Griesemer, J. R. (1989). Institutional ecology,'translations' and boundary objects: Amateurs and professionals in Berkeley's Museum of Vertebrate Zoology, 1907-39. Social studies of science, 19(3), 387-420.
Star, S. L., & Strauss, A. (1999). Layers of silence, arenas of voice: The ecology of visible and invisible work. Computer Supported Cooperative Work (CSCW), 8(1), 9-30.
Steiner, K. E., & Tomkins, J. (2004). Narrative event adaptation in virtual environments. Paper presented at the Proceedings of the 9th international conference on Intelligent user interfaces, Funchal, Mandeira Portugal.
Stone, N., & Posey, M. (2008). Understanding coordination in computer-mediated versus face-to-face groups. Computers in Human Behavior, 24(3), 827-851.
128
Strachan, S., Eslambolchilar, P., Murray-Smith, R., Hughes, S., & O'Modhrain, S. (2005). GpsTunes: controlling navigation via audio feedback. Paper presented at the Proceedings of the 7th international conference on Human computer interaction with mobile devices & services Salzburb, Austraia.
Stuckel, D., & Gutwin, C. (2008). The effects of local lag on tightly-coupled interaction in distributed groupware. Paper presented at the Proceedings of the 2008 ACM conference on Computer supported cooperative work.
Suchan, J., & Hayzak, G. (2001). The communication characteristics of virtual teams: A case study. IEEE Transactions on Professional Communication, 44(3), 174-186.
Tam, J., & Greenberg, S. (2006). A framework for asynchronous change awareness in collaborative documents and workspaces. International Journal of Human-Computer Studies, 64(7), 583-598.
Tan, B. C. Y., Wei, K. K., Huang, W. W., & Ng, G. N. (2000). A dialogue technique to enhance electronic communication in virtualteams. IEEE Transactions on Professional Communication, 43(2), 153-165.
Taylor, R. (1990). Situational Awareness Rating Technique(SART): The development of a tool for aircrew systems design. AGARD, Situational Awareness in Aerospace Operations 17 p(SEE N 90-28972 23-53).
Terrell, I. S. (2006). Understanding 911 dispatch teams across context: Implications for theory, information technology, and practice. Citeseer.
Terrell, I. S., McNeese, M. D., & Jefferson, T. (2004). Exploring cognitive work within a 911 dispatch center: Using complementary knowledge elicitation techniques. Paper presented at the Proceedings of the Human Factors and Ergonomics Society 48th Anual Meeting.
Thomas, B. H., & Calder, P. (2001). Applying cartoon animation techniques to graphical user interfaces. ACM Transactions on Computer-Human Interaction (TOCHI), 8(3), 222.
Tomaszewski, B. M., MacEachren, A. M., Pezanowski, S., Liu, X., & Turton, I. (2006). Supporting humanitarian relief logistics operations through online geocollaborative knowledge management. Paper presented at the Proceedings of the 2006 international conference on Digital government research.
Trafton, J., Martins, S., Michel, M., Lewis, E., Wang, D., Combs, A., et al. (2010). Evaluation of the Acceptability and Usability of a Decision Support System to Encourage Safe and Effective Use of Opioid Therapy for Chronic, Noncancer Pain by Primary Care Providers. Pain Medicine.
Tran, M. H., Yang, Y., & Raikundalia, G. K. (2006). Extended radar view and modification director: awareness mechanisms for synchronous collaborative authoring. Paper presented at the Proceedings of the 7th Australasian User interface conference, Hobart, Australia.
Tuddenham, P., & Robinson, P. (2009). Territorial coordination and workspace awareness in remote tabletop collaboration. Paper presented at the Proceedings of the 27th international conference on Human factors in computing systems Boston, MA USA.
Tutty, J., & Klein, J. (2008). Computer-mediated instruction: A comparison of online and face-to-face collaboration. Educational Technology Research and Development, 56(2), 101-124.
129
Van Den Haak, M. J., De Jong, M. D. T., & Schellens, P. J. (2003). Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology, 22(5), 339-351.
Van Den Haak, M. J., de Jong, M. D. T., & Schellens, P. J. (2009). Evaluating municipal websites: A methodological comparison of three think-aloud variants. Government Information Quarterly, 26(1), 193-202.
Vasalou, A., Joinson, A., Bänziger, T., Goldie, P., & Pitt, J. (2008). Avatars in social media: Balancing accuracy, playfulness and embodied messages. International Journal of Human-Computer Studies, 66(11), 801-811.
Vidulich, M., Bolia, R., & Nelson, W. (2003). Technology, organization, and collaborative situation awareness in air battle management: Historical and theoretical perspectives. A Cognitive Approach to Situation Awareness: Theory and Application, 233-253.
Wang, H., Cai, G., & MacEachren, A. M. (2008). GeoDialogue: A Software Agent Enabling Collaborative Dialogues between a User and a Conversational GIS. Paper presented at the 20th IEEE International Conference on Tools with Artificial Intelligence.
Wang, L., Roe, P., Pham, B., & Tjondronegoro, D. (2008). An audio wiki supporting mobile collaboration. Paper presented at the Proceedings of the 2008 ACM symposium on Applied computing.
Wang, Y., Gräther, W., & Prinz, W. (2007). Suitable notification intensity: the dynamic awareness system. Paper presented at the Proceedings of the 2007 international ACM conference on Supporting group work.
Warkentin, M., Sayeed, L., & Hightower, R. (1997). Virtual teams versus face-to-face teams: An exploratory study of a web-based conference system. Decision Sciences, 28, 975-994.
Wegner, D. M. (1987). Transactive memory: A contemporary analysis of the group mind. Theories of group behavior, 185-208.
Wegner, D. M., Erber, R., & Raymond, P. (1991). Transactive memory in close relationships. Journal of Personality and Social Psychology, 61(6), 923-929.
Wegner, D. M., Giuliano, T., & Hertel, P. T. (1985). Cognitive interdependence in close relationships. Compatible and incompatible relationships, 253-276.
Wellens, A. R. (1993). Group situation awareness and distributed decision making: From military to civilian applications. Individual and group decision making: Current issues, 267-291.
Wellens, A. R., & Ergener, D. (1988). The cities game: A computer-based situation assessment task for studying distributed decision making. Simulation & Gaming, 19(3), 304.
Whalen, T., Toms, E., & Blustein, J. (2008). Information displays for managing shared files. Paper presented at the Proceedings of the 2nd ACM Symposium on Computer Human Interaction for Management of Information Technology.
Wilson, D. C., Lipford, H. R., Carroll, E., Karr, P., & Najjar, N. (2008). Charting new ground: modeling user behavior in interactive geovisualization. Paper presented at the Proceedings of the 16th ACM SIGSPATIAL international conference on Advances in geographic information systems.
Wittenbaum, G. M., Stasser, G., & Merry, C. J. (1996). Tacit coordination in anticipation of small group task completion. Journal of Experimental Social Psychology, 32(2), 129-152.
130
Wittenbaum, G. M., Vaughan, S. I., & Stasser, G. (1998). Coordination in task-performing groups. Social psychological applications to social issues, 4, 177-204.
Wong, N., & Gutwin, C. (2010). Where are you pointing?: the accuracy of deictic pointing in CVEs. Paper presented at the Proceedings of the 28th international conference on Human factors in computing systems.
Wu, A., Zhang, X., Convertino, G., & Carroll, J. M. (2009). CIVIL: support geo-collaboration with information visualization. Paper presented at the Proceedings of the ACM 2009 international conference on Supporting group work.
Yamashita, N., Inaba, R., Kuzuoka, H., & Ishida, T. (2009). Difficulties in establishing common ground in multiparty groups using machine translation. Paper presented at the Proceedings of the 27th international conference on Human factors in computing systems.
Yoo, Y. (2007). The impact of knowledge coordination on virtual team performance over time. MIS Quarterly, 31(4), 783-808.
Zhang, L., Long, Y., Qian, C., Hu, L., & Lv, G. (2007). The application of speech technology in mobile GIS. Paper presented at the Proceedings of SPIE.
Zhao, D., & Rosson, M. B. (2009). How and why people Twitter: the role that micro-blogging plays in informal communication at work. Paper presented at the Proceedings of the ACM 2009 international conference on Supporting group work.
Zhao, S., Dragicevic, P., Chignell, M., Balakrishnan, R., & Baudisch, P. (2007). Earpod: eyes-free menu selection using touch input and reactive audio feedback. Paper presented at the Proceedings of the SIGCHI conference on Human factors in computing systems, San Jose, CA USA.
Appendix A
IRB Materials
Informed Consent Form for Social Science Research The Pennsylvania State University Title of Project: Forming Effective Transactive Memory Systems in Distributed Teams Principal Investigators: Vincent Mancuso College of Information Sciences and Technology 314B IST Building University Park, PA 16802 (814) 865-6847; [email protected] Advisor: Dr. Michael McNeese College of Information Sciences and Technology 301E IST Building University Park, PA 16802 (814) 865 - 9883; [email protected] Purpose of the Study: We are conducting research of a system designed to enhance a distributed teams Transactive Memory System. This study will focus primarily on the overall usability and design of the new system in order to ensure its effectiveness for future studies. Procedures to be followed: You will be asked to take part in a experiment lasting approximately one hour. You will primarily be using the NeoCITIES simulation in which you will take on the role of an emergency response dispatcher (for either police fire and hazmat) and work together with your teammates to solve simulated crisis events. Each NeoCITIES scenario will last 10-15 min. In addition to the NeoCITIES task you will be asked to complete two surveys, one about yourself, and one about your experience with NeoCITIES, and answer a series of interview questions. The interview questions will be recorded using a digital recording device. No personal information will be attached to the files and they will be stored on a secure server that only the researcher has access to. Duration: The interview is likely to take approximately 60 min. Discomforts and Risks: There are no potential discomforts/risks involved in this study other than using an unfamiliar computer simulation, which is similar to playing a new computer based game. You will receive instruction on how to play this simulation to minimize discomfort. Benefits: There are no direct benefits to you as a participant in this study. However, with the data collected from this study we hope to continue to design a tool which may be effective in helping
ORP OFFICE USE ONLY: DO NOT REMOVE OR MODIFY IRB#33222 Doc. #1001 The Pennsylvania State University Institutional Review Board Office for Research Protections Approval Date: 03-22-2010 DWM
132
us better understand how transactive memory systems are formed and improving the usability of systems for emergency responders Statement of Confidentiality: Your participation in this research is confidential. Each participant will be assigned a number. The data will be stored and secured in a secure computer located in a locked Research Lab in IST (Room 314B, IST Building) on the Penn State University Campus. Your confidentiality will be kept to the degree permitted by the technology used. No guarantees can be made regarding the interception of data sent via the Internet by any third parties. The data will be destroyed in December 2013 (3 years following the end of the project). In the event of a publication or presentation resulting from the research, no personally identifiable information will be shared. The following may review and copy records related to this research: The Office of Human Research Protections in the U.S. Department of Health and Human Services, Penn State University’s Institutional Review Board, and Penn State University’s Office for Research Protections. Right to Ask Questions: Please contact Mr. Mancuso at (814) 865-6847 or [email protected] with questions, complaints or concerns about this research. You can also call this number if you feel this study has harmed you. If you have any questions, concerns, problems about your rights as a research participant or would like to offer input, please contact The Pennsylvania State University’s Office for Research Protections (ORP) at (814) 865-1775. The ORP cannot answer questions about research procedures. All questions about research procedures can only be answered by the research team. Voluntary Participation: Your decision to be in this research is voluntary. You can stop at any time. You do not have to answer any questions you do not want to answer. Refusal to take part in or withdrawing from this study will involve no penalty or loss of benefits you would receive otherwise. You must be 18 years of age or older to take part in this research study. If you agree to take part in this research study and the information outlined above, please sign your name and indicate the date below. _____________________________________ ______________________ Participant Signature Date _____________________________________ ______________________ Person Obtaining Consent Date
Appendix B
Study Materials
Survey 1
General Information 1. Please select the statement which best applies to you:
The majority of games I play are computer games. The majority of the games I play are console games (i.e. PS3, Xbox, Wii, etc.) I play an equal amount of computer and console games I don’t play either computer or console games
2. On average how many HOURS A WEEK ON AVERAGE do you spend playing: Real‐time strategy games (Dawn of War, starcraft, command & conquer, sim
city, Civilization, etc.) __________________________ Multiplayer Online Role player Games (World of Warcraft, Age of Conan, Star
Wars Galaxies, etc.) __________________________ Team‐based First Person Shooter Games (Call of Duty, Team Fortress, Halo, etc)
__________________________
3. If you have played NeoCITIES before, when was the most recent time (other than today) that you have played it?
Spring 2010 Fall 2009 Summer 2009 Spring 2009 Other: _______________ Have not played NeoCITIES before
4. Please rate the extent to which you agree with the following statements:
Strongly disagree
Disagree Neither agree nor disagree
Agree Strongly agree
I am good at performing more than one task simultaneously
I am able to multitask effectively I need technology to help me multitask I enjoy multitasking
134
5. Please rate the following technologies based on how comfortable you are using them (1‐
not comfortable at all, very comfortable.
1 3 4 5 Google Maps Twitter Text Messaging Facebook
6. Order the following communication methods based on how often you use them (1 being
the least and 6 being the most): Text Messaging: Twitter: Facebook: Instant Messaging: Phone: Email:
Demographic Questions: 1. Ethnicity:
Caucasian/White African American/Black Hispanic Asian/Asian American Indian Other: ___________________
2. Sex Male Female
3. Age: _____________________ 4. Academic Year
Freshmen Sophomore Junior Senior Graduate Student Other: _____________________
5. Academic Major: ___________________________
6. Please rate your comfort level working in a virtual/distributed team where you have
little face to face communication. Extremely Uncomfortable Moderately Uncomfortable Neither Comfortable nor Uncomfortable
135
Moderately Comfortable Extremely Comfortable
7. Please rate your previous experience working in a virtual/distributed team where you have little face to face communication
None Little Moderate Significant Extensive
Thank you. Please give this paper over to your researcher. You will now begin the NeoCITIES experiment
136
Survey 2
Using the diagram above as a guide, please answer the following questions using your experience with the NeoCITIES interface. Based on your experience how useful do you think each of the following components are to completing your task (1- completely useless, 5-very useful)
1 2 3 4 5 Main Map (B) Overview Map (C) Incident Inspector (A) Team Monitor (D) Off-Site Event Indicator (B-1) Attention Indicator (B-2)
B-1
B-2
137
Based on your experience how often did you use the following components are to completing your task (1- not at all, 5-all the time)
1 2 3 4 5 Main Map (B) Overview Map (C) Incident Inspector (A) Team Monitor (D) Off-Site Event Indicator (B-1) Attention Indicator (B-2)
Based on your experience please answer the following questions based on how much you agree with the statement (1- completely disagree, 5 completely agree)
1 2 3 4 5 I was always aware of what my teammate was working on throughout the entire simulation.
I was always aware of where my teammates were looking throughout the entire simulation
I was aware of all incidents that happened during the simulation
I was always aware about the status of the incidents that were going on in the simulation.
I was able to keep track of all my units during the simulation.
I was able to easily find out all the necessary information for each of the incidents during the simulation.
I was able to easily extract all the necessary information to make the correct decision for each incident
Please list 3 things that you liked about the interface ____________________________________________ ____________________________________________ ____________________________________________ Please list 3 things that you did not like about the interface ____________________________________________ ____________________________________________ ____________________________________________ Any other additional comments or suggestions?
138
Thank you for your answers, you will now be asked to answer a few interview questions.
139
Retrospective Think Aloud Guide
1) What are your thoughts on the map display that is used? a. Was it easy to understand? b. Were the controls easy to figure out? c. Did the overview display assist you in your task?
2) Was the way in which the events were displayed on the map easy to figure out? Was it useful?
a. Were you able to identify new events easily i. Advantages/Disadvantages to displaying the information
geographically b. Do the interactions with the events make sense (i.e. single click to view
information, double click to dispatch) c. Other ideas on how displaying this information could be improved
3) Did you like the modality and level of communication in which the system offered? a. Is it sufficient? b. Advantages/Disadvantages c. Improvements?
4) What did you think about how the information about events was displayed in the Incident Inspector?
5) What did you think about the system automatically reporting your actions to your teammates?
a. Notification Circles b. Action list c. Is there a privacy issue? d. Does is support better collaboration
6) Do you have any other general thoughts about the system? a. Things you liked, things you didn’t like b. Ideas on improving the overall usability of the system c. Aesthetics?
Appendix C
GeoTMS Screen Shots
Viewing an event, one event off screen
Dispatching a Unit
141
Viewing Dispatched Units in Unit Monitor
Recalling a unit
142
Attention Circles
Viewing Score
Appendix D
NeoCITIES Scoring Model
The NeoCITIES Scoring Model which is used in this experiment is the same one which
was proposed in (Wellens & Ergener, 1988) and adapted in (Hellar & Hall, 2009). The general
idea of this scoring model is that the researcher determines how severe an event is to start, and
then depending on how many resources are applied to the event, it either raises in severity or
lowers in severity. The equation for calculating the severity, or magnitude, goes as follows.
Mt = .995 * Mt-1 + .0075 * Mt-1 - .04995 * R
Mt = Magnitude at time T
Mt = Magnitude at previous timestep (t-1)
R= Number of correct resources applied
Based on this, the system is able to calculate the total damage done at that specific event,
by summing all the magnitudes together for the duration of the event. An example of how this
works can be seen in the following table:
Number Correct Resources Timestep 0 1 3 5 0 3.00 3.00 3.00 3.00 1 3.01 2.96 2.86 2.76 2 3.02 2.91 2.71 2.51 3 3.03 2.87 2.57 2.27 5 3.04 2.79 2.28 1.78 10 3.08 2.57 1.56 0.55 15 3.10 2.44 1.12 Complete 25 3.18 2.00 Complete Complete
70 3.54 Complete Complete Complete
440 8.98 Complete Complete Complete
Appendix E
Unique View Example
This Appendix will provide an example of how a Unique View is programmed. The
example Unique View being used is one that shows the name and the icon of the event (as seen
below).
Event Name and Icon Unique View
As you can see, the system accepts a generic object into the Unique View, which is given
to it by GeoTMS, and then the programmer of the view is responsible for casting it to whatever
data type they want. This organization is what allows for the flexibility of GeoTMS.
145
Code
<?xml version="1.0" encoding="utf-8"?> <mx:Canvas xmlns:mx="http://www.adobe.com/2006/mxml" width="100%" height="60" horizontalScrollPolicy="off"> <mx:Script> <![CDATA[ import edu.psu.ist.neocities.model.IncidentModel; import edu.psu.ist.transactive_memory.value.InfoElementVO; import edu.psu.ist.transactive_memory.value.EventVO; import com.pnwrain.easyCG.model.ModelFactory; import edu.psu.ist.neocities.model.ImageModel; import edu.psu.ist.neocities.value.IncidentVO; [Bindable] public var incidentVO : IncidentVO; [Bindable] public var infoElement : InfoElementVO; [Bindable] private var imageModel : ImageModel = ModelFactory.getModel("ImageModel") as ImageModel; [Bindable] private var iModel : IncidentModel = ModelFactory.getModel('IncidentModel') as IncidentModel; override public function set data(value:Object):void { super.data = value; infoElement = data as InfoElementVO; incidentVO = infoElement.infoElement as IncidentVO; } private function getIcon() : Class { return imageModel.getIncidentIcon(incidentVO.icon); } ]]> </mx:Script> <mx:HBox width="100%" height="100%"> <mx:LinkButton icon = "{getIcon()}" enabled="false" disabledColor="#000000" color="#000000" /> <mx:Text text="{incidentVO.label}" paddingTop="10" fontWeight="bold" width="100%" height="100%" textAlign="center"/> </mx:HBox> </mx:Canvas>
Appendix F
NeoCITIES Scenario
The following is an overview of the scenario that was used in the experimentation session
for GeoTMS Interaction Environment. These events were based off the research by (Hellar, 2009)
and (Obieta, 2006).
Event Name Description Dispatch Time Answer Escort Dean After a dispute with the graduate
students over the new candidacy exam format, The Dean of IST has requested an escort to his car
0:11 SWAT
Flag Disappearance Reports are coming in from several sources indicating that Flags supporting Penn State are disappearing from tailgates across campus. No suspects have been identified. Please send units to investigate
0:31 Investigator
Mystery Meat Several people at a tailgate near the Ag Arena report their box of hamburgers being stolen. Send units to interview bystanders and look for clues.
1:01 Investigator
Blue Man Strikes Again Witness claim seeing a caped man dressed entirely in blue spandex running across the roofs of the RV parked down below the baseball stadium. The individual is most likely inebriated, the use of non-lethal force is authorized
1:31 Squad Car
Crash Into Me Two of the CPS mini-trucks collided with eachother and are causing a traffic hold up. Send units to manage the traffic while they fix their vehicles.
2:01 Squad Car
Propane Disposal University Police have caught a drunk man tailgating with propane instead of charcoal. Send units to arrest him and make sure nobody else is using propane
2:31 Investigator, Squad Car
JoePa Security Please send units to escort JoePA from his house to the stadium. Although no specific threats have been issued by leering Ohio State Fans, its better to be safe than sorry.
3:01 SWAT
Street Fighting Man The Rolling Stones would not be proud. A drunken brawl has consumed the alleyway between the brewery and Tony's Big Easy. Please send units to quell the fight and take care of the injured"
3:31 Squad Car
Robbery in Progress Unknown males are robbing an academic building. Please send units to control crowds, investigate the scene and control the situation
4:01 Investigator, Squad Car,
SWAT