the issue of fidelity: what is needed in 3d military serious games?

The issue of Fidelity: What is needed in 3D Military

Serious Games?

Master Thesis

in

Educational Science & Technology

Gillian C. Visschedijk Soesterberg, April 2010 Under supervision of: Dr. Ard Lazonder, UT Faculty of Behavioural Sciences Dr. Henny Leemkuil, UT Dr. Anja van der Hulst, TNO Drs. Nathalie Vink, TNO TNO Defence, Security & Safety

The issue of fidelity: What is needed in 3D military serious games?

of 74


of 74

ACKNOWLEDGEMENTS

The thesis that lies in front of you is the result of my final project of the Master program Educational Science & Technology. This thesis it not only the end of an interesting and challenging project, but it is also a milestone in my life. I have finished my student life and have started to make myself productive in society. That is, the internship at TNO Defence, Security and Safety suited me that well that I decided to stay. Therefore, I want to take this opportunity to thank all the people who supported me coming this far. I first want to thank my colleagues of TNO. Nathalie and Anja, thank you for all the support, concerning both content as morale. I found the feedback from two different disciplines very useful, and I really appreciated all the ‘catch up hours’ with lots of tea where I could share my insecurities. Furthermore, I want to thank my roomies Dennis and Tijmen, and semi-roomy Sam for all the fun and extra feedback. I also want to thank my supervisors from the University of Twente, Ard and Hennie, for their guidance and feedback. An extra word of thanks is addressed to Ard, my first supervisor. I cannot imagine a more involved supervisor. You kept me sharp and motivated, and I always enjoyed the (huge amount of) e-mail contact, telephone calls and visits. Last, but not least, I want to thank my boyfriend Zimri and my parents. Zimri, thank you for always being there for me. Our little complain moments and laughs helped me through. Mom and dad, thank you for supporting me with all my choices and for encouraging to pursue my interests. You are the best!


of 74

SUMMARY

Background 3D serious games are becoming more and more popular because of their presumed educational gains and practical advantages compared to the more traditional training methods. The military has recognized these benefits early on, and has experienced its effectiveness for the training of cognitive skills. The main target group for such games are commanders who can train their tactical decision making skills. This type of application is also acknowledged by the Royal Netherlands Army (RNLA). Together with research institute TNO Defence, Security & Safety they have been developing various 3D serious games for squad and platoon commanders.

Goal One of the issues in the design of such 3D military serious games is fidelity. Essentially, it involves the following question: “how realistic should the game be in order to train adequately”? Full fidelity seems impossible to achieve, and attempts to approach full fidelity are typically associated with excessive development costs. Moreover, it seems that higher fidelity does not always lead to higher transfer. On the other hand, the level of fidelity should not be too low either. It is generally easier to make the right connection between the training situation and the operational situation if they resemble each other, and a higher fidelity level makes a game more acceptable to users. Considering this dilemma, it seems there should be an optimal balance between fidelity, costs, and transfer. In other words: “what level of fidelity is really necessary”? Hence, the aim of this research study was to find this optimal balance in the form of fidelity guidelines for 3D military serious games with a tactical decision making objective.

Method Three types of activities were conducted to obtain these fidelity guidelines: 1) a literature review, 2) a task analysis, and 3) two experimental studies. From the first two activities, the more general guidelines were determined. These guidelines form a framework and a direction for the optimal level of fidelity. To compose more concrete guidelines, two experimental studies were conducted considering only one element of 3D military serious games: the physical fidelity of emotional states of virtual humans. In the two experimental studies, the interaction between the modalities posture, face, and tone of voice was investigated using recognition rates and tactical decision scores. Per modality the most distinctive features derived from literature were used. In the first study, different combinations of the three modalities were compared. And in the second study, the added value of contextual information, as also present in 3D military serious games, was investigated. From the task analysis the following emotional states for military tactical decision making were said to be important and were thus subject to experimentation: angry, aggressive, scared, panic, elation and neutral.

Results The literature review showed that functional fidelity should be high, while physical fidelity might allow lower levels. Regarding the functional fidelity, this means that the feedback on actions taken by users should be realistically designed. For physical fidelity, it means that hardware as well as software can be designed in a less realistic manner. It was concluded that hardware devices like a simple desktop computer with a mouse, keyboard and/or joystick are sufficient. Considering software, it was concluded that the cues in the virtual environment which are dependent on the task have an important role. The task analysis showed that for military tactical decision making, the cues are related to environmental aspects (objects) and the behaviour of civilians and opposing forces (virtual humans). These cues should be designed in a high fidelity manner, while other aspects can be designed in a lower fidelity manner. High fidelity of these cues means that semantically they should be clear and recognizable, and they should be believable to the extent that they are accepted by users. Low fidelity of non-cues means they still need to be accepted by users.


of 74

From the two experimental studies more concrete judgments were made about the fidelity guidelines of one type of cue: the emotional states of virtual humans. The first study showed that the combination posture, facial expression, and tone of voice was significantly most effective. In addition, with respect to the second most effective combination, no significant differences were found amongst all three combinations containing two modalities. The second study showed that contextual information positively influences recognition rates and differences between combinations become smaller. Consequently, the combination posture and facial expression was as good as the combination of all three modalities together. An important conclusion, as it proves that a lower level of physical fidelity is as good as the highest level of physical fidelity in this experiment. In addition, the tactical decisions provided some useful implementation guidelines of emotional states of virtual humans in 3D military serious games.

Discussion The most straightforward answer to the most desired level of physical fidelity is the combination posture and facial expression. Yet, it was argued that leaving tone of voice out actually saves only a small amount of money, while adding tone of voice may cause a higher level of motivation and subsequently transfer. Therefore, it was concluded that if budget is really scant it is still appropriate to design emotional states of virtual humans with only facial and postural expressions. When there is a bit more room in the budget then it is advisable to add tone of voice too. This conclusion is accompanied with guidelines regarding the design of each modality and regarding the implementation of these design guidelines in 3D military serious games. Within the face, it is advisable that only the five most distinctive features are designed. These are the eyebrows, the upper/lower eyelids, the eyes, the mouth corners and the lips. For the postural expression, prototypical postures can be used. When tone of voice is added, sound samples similar to the ones used in the experiment can be applied. Concerning the implementation of the design guidelines, it was suggested that instructors need to be aware of two aspects: (1) the assessment should not merely rely on the tactical decisions students make, but also on the cues they used to make these decisions, and (2) the intensity of a perceived emotion influences the extent to which students would decide to go to a higher spectre of force. These fidelity guidelines can be applied to 3D military serious games. Yet, it should be noted that there are some issues that were not encountered in the two experimental studies, of which their influence on the fidelity guidelines are assumed to be small but at the same time remains unknown. Furthermore, for the elements other than emotional states of virtual humans, it is not possible to make concrete judgements about the optimal level of fidelity. Yet, the literature review and task analysis provide a direction of which elements in the 3D military serious games need more attention regarding a realistic design.


of 74

TABLE OF CONTENTS

1 INTRODUCTION ..........................................................................................................................8

1.1 Problem statement .................................................................................................................9

1.2 Significance ............................................................................................................................9

1.3 Thesis overview.................................................................................................................... 10

2 THEORETICAL FRAMEWORK .................................................................................................. 11

2.1 3D serious games for tactical decision making ...................................................................... 11

2.1.1 3D serious games .................................................................................................... 11

2.1.2 3D serious games with a tactical decision making objective ...................................... 12

2.1.3 Summary.................................................................................................................. 14

2.2 Relationship between fidelity and transfer ............................................................................. 15

2.2.1 Transfer ................................................................................................................... 15

2.2.2 Fidelity ..................................................................................................................... 17

2.2.3 The relationship between fidelity and transfer ........................................................... 17

2.2.4 Summary.................................................................................................................. 20

2.3 Fidelity and transfer within 3D serious games for tactical decision making............................. 21

2.3.1 Objects within the Virtual Environment ...................................................................... 22

2.3.2 AI characters within the Virtual Environment ............................................................. 24

2.3.3 Summary.................................................................................................................. 25

2.4 Summary and conclusion ..................................................................................................... 26

3 TASK ANALYSIS ....................................................................................................................... 28

3.1 Training for infantry commanders.......................................................................................... 28

3.1.1 Method ..................................................................................................................... 28

3.1.2 Results ..................................................................................................................... 29

3.2 Training for Crowd and Riot Control commanders ................................................................. 30

3.2.1 Method ..................................................................................................................... 30

3.2.2 Results ..................................................................................................................... 31

3.3 Summary and conclusion ..................................................................................................... 32

4 RESEARCH QUESTIONS .......................................................................................................... 33

4.1 Research focus .................................................................................................................... 33

4.2 Related work ........................................................................................................................ 33

4.2.1 Fidelity of emotional states ....................................................................................... 33

4.2.2 Summary and conclusion ......................................................................................... 35

4.3 Research questions .............................................................................................................. 36


of 74

5 STUDY 1: RECOGNIZING EMOTIONAL STATES OF VIRTUAL HUMANS ............................... 38

5.1 Method ................................................................................................................................. 38

5.1.1 Participants .............................................................................................................. 38

5.1.2 Materials .................................................................................................................. 38

5.1.3 Design...................................................................................................................... 40

5.1.4 Procedure ................................................................................................................ 41

5.1.5 Data analysis............................................................................................................ 42

5.2 Results ................................................................................................................................. 42

5.3 Discussion ............................................................................................................................ 43

6 STUDY 2: RECOGNIZING EMOTIONAL STATES OF VIRTUAL HUMANS IN CONTEXT ......... 45

6.1 Method ................................................................................................................................. 45

6.1.1 Participants .............................................................................................................. 45

6.1.2 Materials .................................................................................................................. 45

6.1.3 Design...................................................................................................................... 46

6.1.4 Procedure ................................................................................................................ 47

6.1.5 Data analysis............................................................................................................ 48

6.2 Results ................................................................................................................................. 48

6.3 Discussion ............................................................................................................................ 50

7 GENERAL DISCUSSION ........................................................................................................... 53

7.1 Desired level of physical fidelity of emotional states of virtual humans ................................... 53

7.2 General fidelity guidelines of 3D military serious games ........................................................ 55

7.3 Recommendations for future research .................................................................................. 56

REFERENCES ................................................................................................................................. 57

APPENDIX A: CUES FOR INFANTRY COMMANDERS................................................................... 62

APPENDIX B: CUES FOR CRC COMMANDERS............................................................................. 67

APPENDIX C: FACIAL AND POSTURAL EXPRESSIONS............................................................... 70

APPENDIX D: CONTEXT DESCRIPTIONS ...................................................................................... 73


of 74

1 INTRODUCTION

The use of games for training is becoming more and more popular. This is especially true for computer games with enabling technologies like a 3D virtual environment, artificial intelligence, and networking possibilities (Alexander, Brunyé, Sidman & Weil, 2005). The popularity of these so called ‘3D serious games’ is due to their presumed educational gains and practical advantages compared to more traditional training methods. In short, 3D serious games: • can provide an authentic context in which learners can learn-by-doing. This makes knowledge

more meaningful and therefore ‘sinks in’ better (Schank, Berman & Macpherson, 1999); • contain gaming features that are assumed to enhance motivation, which in turn can have a

positive effect on learning (Garris, Ahlers & Driskell, 2002); • allow learners to experience situations that are impossible in the real world for reasons of safety,

cost and time (Knerr, 2007); • allow preparation and rehearsal for specific tasks or situations, because users can experience the

effects of their actions (Sanchez & Smith, 2007); • can be networked to provide collective training. This is often called a multi player game (Knerr,

2007). The military has recognized these benefits early on, and experienced its effectiveness for the training of cognitive skills prior to their application in field exercises. More specifically, they say that 3D serious games can be used for improving tactical decision making, situation awareness, and communication and coordination skills, while real world training could place greater emphasis on motor skills (Knerr, 2006; Pleban & Salvetti, 2003). In practice this means that 3D serious games are especially suitable for the training of commanders.

This type of application is also acknowledged by the Royal Netherlands Army (RNLA). Together with research institute TNO Defence, Security & Safety they have developed various 3D serious games for squad and platoon commanders in fields like tactical air defence, infantry tactics and cavalry tactics (van der Hulst, Muller, Besselink, Coetsier & Roos, 2008). The results of these games are promising, and more fields are targeted. Examples include counter-IED tactics (searching for Improvised Explosive Devices) and Crowd and Riot Control (CRC) tactics for the Military Police. Figure 1 contains a screenshot from one of the games the military uses, called Virtual Battle Space 2 (VBS2). This is a 3D First Person Shooter with the ability to render and display large amounts of terrain data. It has very detailed objects such as equipment, vehicles, etc. and also virtual humans can be displayed.

Figure 1 Screenshot of VBS2


of 74

1.1 Problem statement

Although results are promising and more and more people from the RNLA have become enthusiastic about 3D serious games, some issues in the design of these games merit attention. One of these issues is fidelity.

Fidelity denotes “the degree of similarity between the training situation and the operational situation which is simulated” (Hays & Singer, 1989). For design decisions of 3D serious games, fidelity essentially involves the following question: “how realistic should the game be in order to train adequately”? This is an important question because, although a highly realistic game seems desirable, training devices will never be able to fully replicate a complete set of outside world stimuli (Andrews, Carroll & Bell, 1996). Therefore, full fidelity seems impossible to achieve. In addition, attempts to approach full fidelity are typically associated with excessive development costs. The choice to ‘buy’ as much fidelity as the budget allows is nevertheless not very efficient, especially if one realizes that higher fidelity does not always lead to higher transfer. Several studies comparing high and low fidelity training materials generally showed no significant difference in learning effects (see Feinstein & Cannon, 2002), and some studies even found that high fidelity can interfere with realizing the full training potential (Mania, Wooldridge, Coxon & Robinson, 2006; Smode, 1971 in Hays & Singer, 1989). So, aiming at the highest possible level of fidelity does not seem to be the best design decision.

On the other hand, the level of fidelity should not be too low either. One must be aware of the fact that an ineffective training can be very dangerous. Decisions made by commanders in real life can literally make a difference between life and death, so it is of vital importance that they make the right connection between the training situation and the operational situation. This is generally easier when the training situation and the operational situation resemble each other. In addition, user acceptance and student motivation could be of importance. Students will inevitably make a comparison with state-of-the-art entertainment games, and when this results into low motivation, learning outcomes are likely to lower accordingly (Lampton, Bliss, & Morris, 2002).

Considering this dilemma, it seems there should be an optimal balance between fidelity, costs, and transfer. In other words: “what level of fidelity is really necessary?” Just like Alexander et al. (2005) stated: “an important compromise must be made between physical fidelity, related costs, and training effectiveness, such that an adequate match can be made between training and real environmental elements and the logical structure of tasks” (p.5). Hence, the aim of this research study is to find this optimal balance and to formulate fidelity guidelines for 3D military serious games with a tactical decision making objective.

1.2 Significance

A more detailed look in literature shows that this dilemma is certainly not new. But even though much is written about the importance of fidelity, it still remains a somewhat vague topic with some generic suggestions like: “simulators in later learning stages require a higher fidelity than in the initial stages of learning (Hays & Singer, 1989)”. It is questionable to what extent such generic suggestions also apply to 3D military serious games, as most research has been conducted with aviation training simulations (e.g., Alessi, 2000; Noble, 2002). These studies generally focused on the more ‘hardware’ oriented flight simulators containing an exact replica of the equipment (i.e., knobs, dials, etc.) that is used in the real environment. Moreover, the skills that were targeted in these simulations are mostly psycho-motor or procedural based. In other words, a completely different situation compared to 3D serious games used for the training of cognitive skills like tactical decision making. Hence, the fidelity requirements for the latter type of training are assumed to be rather different from the level of fidelity necessary for operating the controls on a piece of equipment (Hays, 1980).

Furthermore, little is known about how to reach the optimal balance between fidelity, transfer and costs. A task analysis seems an important starting point, because it reveals the critical elements of the task one wishes to teach. Accordingly, these critical elements require higher fidelity compared to other parts. However, “no systematic guidance exists to translate task analysis information into a form


of 74

which can facilitate fidelity decisions” (Hays & Singer, 1989, p. 56). This means that empirical data is still necessary to validate these fidelity decisions. However, as the technology of 3D serious games is relatively new, there have been few empirical investigations of their effectiveness in general, and the effects of fidelity on transfer in particular (Alexander et al., 2005).

This thesis project aims to fill these gaps by means of a literature review, a task analysis in different military fields, and two empirical studies. As a result, it aspires to provide the RNLA and TNO with a suitable solution to their dilemma, and contribute to general knowledge about the relationship between fidelity, transfer and costs in 3D military serious games.

1.3 Thesis overview

The three activities mentioned previously can broadly be attributed to two phases (see schematic overview of the thesis in Figure 2).

The first phase is called the Exploration phase, in which more general fidelity guidelines are explored in Chapter 2 and 3. The two chapters provide an overview of the elements of 3D military serious games where fidelity plays a role and the kind of tasks relevant for 3D military serious games. Moreover, the results provide a basis and a framework for the empirical investigations in the second phase.

The second phase is the main part of this research study and is called the Experimentation phase. It contains a research focus, meaning that only one topic is empirically investigated. This research focus is necessary because it is far beyond the scope of this thesis project to empirically validate all fidelity decisions. The physical fidelity of emotional states of virtual humans in 3D military serious games was chosen to be this topic. The reason why this topic was chosen to be the research focus is explained in Chapter 4, as it was the result of the exploration phase. This explanation is followed by the research questions, which are answered by means of the results of two experimental studies in respectively Chapter 5 and 6.

This thesis concludes with a general discussion in Chapter 7, in which the results of the two phases are discussed in a broader perspective.

Figure 2 Schematic overview of the Master thesis

Exploration phase

Ch. 2 Theoretical Framework

Ch. 3 Task Analyses

Ch.7 General discussion

Experimentation phase Ch.4 Research

Questions

Ch.5 Study 1

Ch.6 Study 2


of 74

2 THEORETICAL FRAMEWORK

What exactly is fidelity? And how does it influence transfer in 3D (military) serious games for tactical decision making? These questions are central in this chapter. Accordingly, the aim is to provide an overview of elements of 3D serious games where fidelity plays a role and to conclude with fidelity guidelines. Note that ‘military’ is put in parentheses, as this literature review is conducted at a more general level. Still, the focus is on 3D serious games within the military field. Three sections are distinguished in order to come to the conclusion. Section 2.1 describes the context for which fidelity guidelines will be determined. It contains definitions and characteristics of 3D serious games with a tactical decision making objective. In section 2.2 the meaning of fidelity and its relationship to transfer in general is discussed. This relationship is further elaborated in section 2.3, where fidelity is considered specifically in the context of 3D serious games with a tactical decision making objective.

Method The input for writing the three sections is derived from a literature search. Frequently consulted databases were ‘PsychInfo’, ‘PiCarta’, ‘DTIC’ (Defence Technical Information Centre), ‘ERIC’ and ‘Google Scholar’. In addition, the catalogue of the University Library of the University of Twente and the digital library of TNO Defence, Security and Safety were consulted. Various queries were used, like: ‘serious games’, ‘virtual environments’, ‘simulation games’, ‘fidelity’, ‘transfer’, and ‘fidelity and 3D serious games’. The most common sources resulted from the search were journal articles, (hand)books, conference proceedings, technical reports, and research reports. Many of these are situated in the military field, predominantly originating from the U.S. Army Research Institute for the Behavioral and Social Sciences.

2.1 3D serious games for tactical decision making

The terms ‘3D serious games’ and ‘tactical decision making’ have already been used several times. But how can a 3D serious game be characterized? And how does the objective of tactical decision making further specify the characterization of 3D serious games? These two questions are answered in sections 2.2.1 and 2.2.2 respectively.

2.1.1 3D serious games

The term ‘3D serious games’ consist of characteristics related to games in general, the ‘seriousness’ of games and the 3D world of a serious game. These characteristics are discussed below, even as the presumed effectiveness of 3D serious games.

Characteristics of 3D serious games Leemkuil, de Jong and Ootes (2000) propose four main characteristics of games: (1) some goal state that must be reached, (2) constraints and rules, (3) competition, and (4) a specific context. Hays (2005) translated these characteristics in a working definition: “a game is an artificially constructed, competitive activity with a specific goal, a set of rules and constraints that is located in a specific context” (p. 15). Furthermore, he argues that the purpose of the game (e.g., enjoyment, information, instruction, etc.) helps define the goals, rules, and context of the game (Hays, 2005).

The ‘seriousness’ of games defines this purpose, as serious games are (digital) games used for purposes other than mere entertainment (Susi, Johannesson & Backlund, 2007). Michael and Chen (2006 in Rankin & Vargas, 2008) specify ‘the other purposes’ by stating that education in its various forms is the primary goal. Even more specific definitions contain references to education in specific sectors, like health, government, and military (Raybourn, 2007; Zyda, 2005). Accordingly, in


of 74

this thesis serious games are related to games with training purposes, which are situated in the military field.

Adding the three spatial dimensions to the concept of serious games provides opportunities for creating a Virtual Environment (VE). VEs allow users “to be immersed into three-dimensional digital worlds, surrounding them with tangible objects to be manipulated and venues to be traversed, which they experience from an egocentric perspective” (Stanney, 2002, p. xix). Within the military sector, the application of 3D digital worlds is crucial. It is this technology that makes it possible to provide a realistic context for training. In addition, Artificial Intelligent (AI) characters are often part of 3D serious games in the form of avatars and agents. Whereas an avatar represents the user in a game, an agent is a non-playable, and therefore autonomous character in a game (Allbeck & Badler, 2002). Within the military field, autonomous agents as well as human avatars can substitute live human role-players like tactical enemies, allied force characters, civilians, and even collections of individuals that together form a military unit such as a platoon (Tarr, Morris & Singer, 2002).

These characteristics are sometimes also attributed to simulations, as the line between games and simulations can be blurred. It is possible that games represent portions of reality, which is a distinct characteristic of simulations (Hays, 2005), and the other way around, simulations can contain gaming features (Garris et al., 2002). Well-known examples of such ‘hybrids’ are Microsoft Flight Simulator and America’s Army. Hence, other terms which are often used instead of 3D serious games are simulation-games (Rankin & Vargas, 2008) and lightweight simulators (Alexander et al., 2005).

The effectiveness of 3D serious games Evidence suggests that games can provide effective learning for a variety of learners and for several different tasks (see Hays, 2005; Garris et al., 2002). However, Hays (2005) determined that the empirical evidence indicating that games are the preferred instructional method is virtually absent in all situations. “Like any instructional activity, games should be chosen because they provide learners with interactive experiences that help them meet instructional objectives” (Hays, 2005, p. 46).

The military has already gained much experience with 3D serious games. And despite the fact that empirical evidence is scant, they strongly believe that 3D serious games can be effective for the training of cognitive skills, like tactical decision making, prior to their application in field exercises (Knerr, 2007). In the next section, the use of 3D serious games with the goal of improving tactical decision making is further elaborated.

2.1.2 3D serious games with a tactical decision making objective

How does the objective of tactical decision making further specify 3D serious games? To answer this question, it is necessary to have an understanding of the underlying processes of tactical decision making. Subsequently, it is possible to describe the features of 3D serious games which can support these processes.

Underlying processes of tactical decision making In general, decision making is the cognitive process of reaching a decision (WordNet 3.0., n.d.). Decision making takes place on different levels, from the more abstract, long-term plans to the more concrete and short-term actions. The first can be described as ‘strategic decision making’, whereas the latter is closer to ‘tactical decision making’. In the military field decision making is described as “processing environmental data in combination with situational factors to utilize combat power to accomplish a unique mission or task” (Marine Corps Institute, n.d., p.38). Accordingly, especially commanders at the lower levels (e.g., platoon or squad commanders) have to deal with tactical decision making.

Why would one train tactical decision making? The most obvious answer in the military context is to avoid (serious) accidents, and even safe lives. Therefore, it is of vital importance that commanders receive effective training in order to become expert tactical decision makers.


of 74

Recognizing this importance has led scholars to ask what characterizes an expert decision maker. Their purpose was to understand the ‘cognitive map’ that guides the process of expert decision makers. Many scientific approaches emerged, of which a relatively recent approach called Naturalistic Decision Making (NDM) seems best to describe tactical decision making in complex and dynamic real-world environments such as military missions (Klein, 1997). NDM emphasizes the idea that people make decisions based on their own previous experiences. More specifically, in order to make an effective decision, they try to understand the situation and judge its familiarity to other situations. Accordingly, the driving factor in this process is the conceptualization of the situation, which is also referred to as Situational Awareness (SA) (Endsley, 1997). It has generally been recognized that insufficient or inappropriate SA is one of the primary reasons for decision-making errors. So training students in achieving optimal SA is inherent to the training of tactical decision making. It is therefore necessary to understand the construct of SA and its role in the decision-making process. Moreover, SA is directly influenced by the level of fidelity and thus highly relevant to this thesis.

A widely accepted definition of SA is provided by Endsley (1988): “the perception of the elements in the environment within a volume of time and space, the comprehension of their meaning and the projection of their status in the near future” (p. 97). The different aspects of the definition are illustrated in Figure 3, which contains a simplified version of Endsley’s model of SA in relation to tactical decision making. The complete model also includes factors influencing SA. Discussion of these factors is beyond the scope of this thesis, for further readings see Endsley (1997).

Figure 3 Simplified model of Situational Awareness, adapted from Endsley (1997) For an infantry commander, the model exemplifies that he/she should (level 1) recognize/be aware of relevant elements (also called cues) in a particular situation, (level 2) understand what these cues in the situation mean, and (level 3) translate perception and understanding of the situation into a projection of future events likely to occur in that situation. Subsequently, the infantry commander can use this information to make a decision and act upon it. The feedback that follows from these action(s) alter the situation, so the process can start over again. The process reveals two important aspects that characterize expert decision makers. First of all, experts have a repertoire of relevant cues or patterns of cues that enable them to recognize the situation. And secondly, experts have mental models which they use to match these relevant cues or patterns of cues. In this way, they are able to understand the situation and mentally simulate possible outcomes of decisions.

Based on the underlying process and the characterization of an expert decision maker, the requirements for effective tactical decision making training can be established. Providing training scenarios lay the foundation for these requirements. According to Hartog (2009), scenarios can be characterized by a description of chronological events that represent or tell a certain story over time. In other words, scenarios contain the kinds of situations students might face in real-life. Moreover, they should be provided according to the following requirements for effective training; an effective training should contain: (1) sufficient practice, (2) varied scenarios, (3) all types of relevant cues, and (4) timely feedback. In this way, students learn to recognize typical cases and relevant cues, and build mental models through (sufficient) experience. In addition, timely feedback helps students to make the correct


of 74

associations between cues or cue patterns and appropriate actions (Cannon-Bowers & Bell, 1997). Together, these four requirements should enhance the student’s ability to accurately characterize situations and lead to greater SA. This, in turn, could lead to an improved and timely decision-making capability.

Features of 3D serious games that support tactical decision making Having specified which factors contribute to the effectiveness of this training, it is now possible to describe the designated features of 3D serious games that support these factors.

Providing sufficient opportunities for practice is a quite obvious factor that can be supported by 3D serious games. Actually, it is not necessarily a characteristic of 3D serious games itself, but more an implementation issue that implies that 3D serious games should be used repeatedly in order to maximize the effectiveness of the training.

The second factor, providing varied scenario’s, was already mentioned briefly in the previous section. The virtual environment and the presence of AI characters provide opportunities to simulate conditions similar to what the trainee might experience in the real world (i.e., dynamic environments that require rapid decisions). This makes it possible to expose a student to the situations he/she is likely to be confronted with in the real world (Pleban, Eakin, Salter & Matthews, 2001). In case of the training of infantry commanders, this means they must be trained in a variety of possible scenarios, ranging from clearing a building through performing a social patrol to reacting on an enemy ambush. The type of environment should be varied as well, for instance by alternating missions in big cities with missions in rural small villages.

In order to provide all types of relevant cues (the third factor) it would be necessary to stimulate all five senses. This is not possible within 3D serious games, because they usually support only sounds and images. Yet, sounds and images often provide the most valuable information, which means almost all relevant cues can be provided by 3D serious games. In case of the infantry commanders’ training, specific situational cues and cue patterns from various sources can be incorporated in the scenario. Examples are: audio communications from other unit leaders and the higher command, terrain and building characteristics, presence of civilians, enemy, aircraft and heavy weapons (Pleban et al., 2001).

The fourth factor, providing timely feedback, can be effectively supported by 3D serious games. Just like the relevant cues, feedback can be provided by using sounds and images. This means students need to see and hear the results of their actions in real time and as a result learn from poor decisions. For example, after deciding which kind of grenade to use, the explosion following should be characterized by that kind of grenade (i.e., much damage versus small damage capacity).

Last but not least, the importance of making quick decisions is often mentioned. This is not really a factor contributing to effective tactical decision making, but often characterizes the conditions in which tactical decision making is necessary. 3D serious games provide the possibility to include time constraints, which forces the student to make quick situational assessments and decisions under stress.

2.1.3 Summary

3D serious games contain gaming features like a specific goal, constraints and rules, competition, and a specific context. They are used for training, for which the virtual environment and artificial intelligent characters can provide a realistic context. The military experienced the effectiveness of 3D serious games for cognitive skills, like tactical decision making, prior to their application in field exercises. For this type of training, it is essential to (1) acquire a repertoire of relevant cues or patterns of cues, which makes commanders able to recognize the situation, and (2) acquire mental models which commanders can use to match relevant cues or patterns of cues, and consequently makes them able to understand the situation and mentally simulate possible outcomes of decisions. 3D serious games contain features that support the training of becoming an expert tactical decision maker. First, 3D serious games can be used repeatedly in order to provide sufficient practice. Second, varied


of 74

scenario’s can be provided in the virtual environment, if necessary with the presence of artificial intelligent characters. In this way, a student can experience the kinds of situations he is likely to encounter in the real world. Third, (almost all) relevant cues from various sources can be incorporated in the scenario through sounds and images. Fourth, timely feedback can be provided, because students can see and hear the results of their actions in real time. In this way, students can learn from poor decisions. Finally, 3D serious games can provide time constrains which characterizes the conditions in which tactical decision making is necessary.

2.2 Relationship between fidelity and transfer

This paragraph investigates the general relationship between fidelity and transfer. Before explaining this relationship in section 2.2.3, first descriptions of transfer and fidelity are provided in section 2.2.1 and 2.2.2, respectively. Next to research within the context of 3D serious games, also research within the simulation field is included. This is because fidelity research has a much longer history within the simulation field, and it was already encountered that games and simulations share many characteristics.

2.2.1 Transfer

Transfer is a complicated concept. In order to make predictions regarding the transfer of training, one must know what transfer is and what it means. Therefore, these aspects are discussed first, followed by the meaning of transfer in the context of 3D serious games for tactical decision making.

Characteristics of transfer Transfer is the ultimate goal of all games with training purposes. It is defined as the spontaneous application of knowledge, skills, and attitudes acquired during training to the environment in which they are normally used (Muchinsky, 2003). This means that game-based training is only effective if what is learned in the game can be used in the real situation. Furthermore, the higher the degree of transfer, the more successful a serious game is considered to be (Alexander et al., 2005). This statement implies transfer can be explained in terms of magnitude.

Besides magnitude, transfer can also manifest itself in different directions: positive, negative or nonexistent (Gick & Holyoak, 1987). Positive transfer occurs when learning in the training environment improves performance in the target setting. Negative transfer occurs when learning in the training environment worsens real world performance, typically because trainees apply to the real world behaviours that are appropriate only in the training environment (like actions to ‘game’ a simulator) (Alexander et al., 2005). Nonexistent transfer occurs when initial training has no effects on performance in the target setting at all.

Also the distance of transfer is often central in the discussion of describing transfer for a specific task. A distinction is generally made between near transfer and far transfer. In short, the term near transfer refers to applying what is learned to very similar circumstances, whereas far transfer refers to applying what is learned to quite different circumstances (Alessi & Trollip, 2001). It is often not possible to characterize a task as a purely near or a purely far transfer task. Therefore the two terms are often used as a continuum.

Meaning of transfer There are two mechanism explaining transfer of complex cognitive skills like tactical decision making: (1) rule-based transfer, and (2) schema-based transfer (van Merriënboer, 1997). Rule-based transfer (sometimes called procedural overlap) focuses on the familarity of situations where skills acquired in the training setting can be used in the same way as in the target setting. The mechanism emphasizes the idea that the training and target setting should share the common elements that are relevant for the task, also called the ‘cueing elements’ (Cormier, 1987). This idea is based on Thordike’s ‘theory of identical elements’ (1903 in Gick & Holyak, 1987) and


of 74

subsequent work of Singley and Anderson (1989). Rule-based transfer is most important for near transfer tasks and benefits from ‘learning-by-doing’ and extensive practice. In addition, practice must offer varied cases to work on in order to develop a broad set of domain-specific productions. Schema-based transfer focuses on the unfamilarity of situations where skills acquired in the training setting can be used in a different way in the target setting. The mechanism emphasizes the idea that cognitive schemata offer abstract generalized knowledge enabling one to understand a new situation in general terms and act according to this general understanding (van Merriënboer, 1997). It is assumed that transfer from a learned task to a new task will be more succesful if usefull cognitive schemata have been acquired during practice. In addition, the richer those cognitive schemata are, and the better they are integrated with other cognitive schemata, the more likely it is that transfer will actually occur (van Merriënboer, 1997). Schema-based transfer is most important for far transfer tasks and benefits from ‘learning-by-thinking’ (activities such as mindful abstraction, analysis, making comparisons, searching contrasts, and integrating information). Practice must promote such inductive processing by offering a high-variability of problems and examples and by explicitly provoking mindful abstraction from those problems and examples. Van Merriënboer (1997) states that both mechanisms work in tandem when performing a complex cognitive skill: “the typical expert will have domain-specific productions available to solve familiar aspects of the problem almost automatically (procedural overlap), and will have cognitive schemata (conceptual and causal models, goal-plan hierarchies, heuristics, plans) available to understand the non-familiar aspects of the problem in general terms” (van Merriënboer, 1997, p. 71). Activities contributing to the two mechanisms (‘learning-by-doing’ and ‘learning-by-thinking’) should therefore complement each other in order to reach a more complete view of the learning processes involved in the acquisition of a complex cognitive skill (van Merriënboer, 1997).

Yet, depending on the type of task, the emphasis can be on one of the two mechanisms. Predictions regarding transfer can be made if they are compared to near transfer tasks versus far transfer tasks. It seems that for very near transfer tasks, the procedural overlap between the trained task and the transfer task is very high, while schema-based transfer is of little importance here. The importance of schema-based transfer increases, and the importance of procedural overlap decreases, if the transfer tasks become more different from the original training task (van Merriënboer, 1997).

Transfer of 3D serious games with a tactical decision making objective For 3D serious games with a tactical decision making objective, it can be argued that rule-based transfer is more important than schema-based transfer. This is due to the fact that exposure to real world situations is often key to 3D serious games. Students gain experience through ‘learning-by-doing' with sufficient practice in varied (real world) scenarios, where all relevant cues and timely feedback is provided. In this way, familarity of typical situations with relevant cues is of most importance, which places the task in the continuum of near transfer to far transfer, closer to near transfer tasks. However, schema-based transfer should not be ignored, because it is impossible to provide students with all possible situations. Therefore reflection activities after playing the 3D serious game should be included to promote abstraction and generalization of what is learned. Moreover, the concrete experiences obtained in the ‘real world’ of the 3D serious game provide rich cognitive schemata and enhances the integration with other cognitive schemata. In addition, the varied scenarios make it possible to make comparisons and search for contrasts. What does this all mean in practice? In short, it means that for the training of tactical decision making in the 3D serious game itself, the aspect of sharing common elements relevant for the task is of most importance. In other words, the game should contain the cueing elements, or cues, which are also present in the real world. In addition, post-session reflection activities should be included in the training in order to strengthen the development of the necessary mental models. As the aim of this thesis is related to the design of the 3D serious game itself (and not to the debriefing session), only rule-based transfer is further discussed. Fidelity plays a key role in recognizing and understanding the


of 74

necessary cues to achieve a high degree of transfer. What this role exactly implicates, is subject in the following sections.

2.2.2 Fidelity

Fidelity was defined in Chapter 1 as “the degree of similarity between the training situation and the operational situation which is simulated” (Hays & Singer, 1989). Just like transfer, fidelity can be classified in terms of its magnitude. Researchers often use the terms low, medium and high fidelity in this respect. As these terms are rather subjective, descriptions of their meaning are necessary (Lane & Alluisi, 1992).

A closer look at the usage of the term fidelity shows that it is used for different types and concepts (Roza, 2005). When people think about the fidelity of a simulation, many of them will instantly recall the perceptual aspects of a simulator: does it give the impression of being realistic? But what about the fidelity of the model underlying a simulation? Or the fidelity of motion, content and behaviour? These examples illustrate that fidelity is not a single entity, but contains many dimensions which makes it extremely complex to measure. Many researchers therefore have identified subcategories of fidelity (see Roza, 2005). The most commonly used ones are physical and functional fidelity.

Physical fidelity refers to the degree to which the simulation looks, sounds, and feels like the operational environment in terms of the visual displays, controls, and audio. Functional fidelity refers to the degree to which the simulation acts like the operational equipment in response to the tasks performed by the trainee (Alexander et al., 2005). In other words, physical fidelity concerns the appearance or representation of the simulator, whereas functional fidelity addresses the behaviour of the simulator in terms of feedback.

Another important view on fidelity is more philosophical by nature. You can ask yourself what realism actually is, because “…much of what we know to be real is not known in any objective sense, but is believed (in the fully tentative sense of that word) to be real” (Petraglia, 1998, p. 58). So in order to interpret results in a meaningful way, one must describe what is understood by fidelity when it is measured. Hence, a distinction can be made between the objective fidelity and the perceived fidelity. The former is seen as the ‘actual’ fidelity from an engineering point of view, the latter refers to the perceptions and subjective judgments of the people using the simulation (AGARD, 1980; Lane & Alluisi, 1992).

2.2.3 The relationship between fidelity and transfer

So far, it has become clear that the (level of) similarity between the training task and the operational environment is essential for the success of a training. Accordingly, fidelity has a significant impact on the amount of transfer. However, as explained in Chapter 1, full fidelity is virtually impossible to achieve, and attempts to approach full fidelity are typically associated with high costs. This has led many scholars to ask “how much fidelity is needed to actually train the established criteria in the most efficient manner” (Hays & Singer, 1989, p. 44). Overall the studies investigating this question suggest that the relationship between fidelity and transfer is very complex. In the remainder of this section the four main issues related to this complicated relationship are discussed. These are: (1) the general relationship between the two factors, (2) the type of task, (3) the stage of learning, and (4) the influence of motivation on this relationship.

General relationship For years people believed that the higher the fidelity, the higher the transfer. This notion was predominantly driven by intuitive appeal (Noble, 2002) and implies that some sort of linear relationship exists between fidelity and transfer. However, research indicates a more complex nonlinear relationship, see Figure 4 (Alessi, 1988 in Liu & Vincenzi, 2004). This so-called ‘Alessi Hypothesis’ states that a point exists beyond which one additional unit of fidelity results in a diminished rate of


of 74

return on investment. In other words, implementing an ‘all you can afford’ level of fidelity is not always the most cost-effective approach. Moreover, many studies found that lowered fidelity actually can assist in acquiring the details of training (see Feinstein & Cannon, 2002). This means that deliberate ‘departures from reality’ can be necessary to provide the most effective training (Hays & Singer, 1989; Mania et al., 2006).

Figure 4 Illustration of Alessi Hypothesis, adapted from Liu & Vincenzi (2004)

Type of task The required amount of fidelity depends on the educational purpose of a 3D serious game. This means that different tasks require different levels of fidelity (Hays & Singer, 1989). For example, if a psychomotor task like aiming and firing a weapon is compared with a more cognitive task like the decision making process that precedes weapon use, it seems reasonable to assume that the psychomotor task requires higher levels of fidelity than the cognitive task. That is, really picking up the right weapon in the latter case doesn’t add anything to the task, which makes it actually a waste of development costs. This in contrast to the psychomotor task, where really holding the weapon does make a difference.

The notion that the necessary level of fidelity depends on the type of task corresponds with the idea of sharing the common elements relevant for the task. A good simulation or 3D serious game may accordingly require high fidelity for some parts and low fidelity for others. This is called ‘selective fidelity’ (Andrews et al., 1995). Hence, if the level of fidelity captures the critical elements (cues) of the task one wishes to teach, that level of fidelity is sufficient even if it noticeably deviates from the real world (Alexander et al., 2005). For example, if an infantry commander only uses mountains and high buildings as landmarks for navigation, then other features in the scene (e.g., trees, textures on houses, etc.) do not require high fidelity. Consequently, a detailed task analysis should be conducted for every specific task. This analysis can be used as a basis for decisions about fidelity requirements (Hays & Singer, 1989). However, as was mentioned in Chapter 1, empirical data is still needed to validate these fidelity decisions. A number of studies have therefore been conducted with the aim of establishing fidelity guidelines for a variety of tasks. These are often characterized by a description of the optimal ‘mix’ of the two types of fidelity (physical and functional) (van der Hulst, de Hoog & Wielemaker, 1999). A common distinction is made between three types of tasks: psychomotor, procedural and cognitive tasks (Farmer, Rooij, Riemersma, Jorna & Moraal, 1999). As tactical decision making clearly is a cognitive task, only conclusions related to this type of task are discussed together with an interesting finding that applies to all types of tasks. For the complete overview of the conclusions per task, see Visschedijk (2009).

Transfer Of Training (%) 100 0 1 Fidelity


of 74

Many scholars suggest that for cognitive tasks, functional fidelity is more important than physical fidelity (e.g., Alexander et al., 2005; Andrews et al., 1995; Hays & Singer, 1989). This is for example the case in business games, where it is difficult to display real world physical objects, or in fault diagnoses as studied by Rouse and Hunt (1984). They found that training for fault diagnoses of electronic networks, problem solving can be enhanced with low physical fidelity and medium/high functional fidelity PC-based simulation. It seems that in these kinds of situations, it is of most importance that “the trainees experience the results of their decisions realistically enough to learn from them” (Hays & Singer, 1989, p. 209).

Applicable to all tasks is an approach called cue dominance, which can be used to make simulators and 3D serious games more efficient (Warren & Riccio, 1985 in Korteling, van den Bosch & Emmerik, 1997). Fundamental to this approach is the idea that some cues are more relevant for task performance than others, even if they represent the same information. The cues that are most relevant are called critical cues. They can be identified by means of task analysis and empirical research, and result in an arrangement of cues in a so called cue dominance hierarchy. Research suggests that weaker cues are neglected if they are presented simultaneously with dominant cues for the same information (Warren & Riccio, 1985 in Korteling et al., 1997).

Stage of learning It has been suggested that the necessary level of fidelity does not only depend on the type of task, but also on the trainee’s stage of learning (Hays & Singer, 1989). A vast amount of evidence exists stating that novices require a lower level of fidelity than trainees who are already more familiar with the domain (e.g., Alessi, 1988 in Noble, 2002; Feinstein & Cannon, 2002; Martin & Waag, 1978). Moreover, it seems that high fidelity can hinder effective training and learning because it may overstimulate novice trainees (Martin & Waag, 1978, Smode, 1971 in Hays & Singer, 1989). These findings resulted in the idea that the training effectiveness can be increased by adjusting the level of fidelity over a sequence of training sessions (Lathan et al., 2002), also mentioned as ‘dynamic fidelity’ (Alessi, 1995).

However, it has also been suggested that for the more complex procedural and cognitive tasks, physical fidelity needs to be high in the initial stages of learning because of its contributions to schemata acquisition, whereas the more advanced trainees can do as well with a more abstract representation (Korteling et al., 1997). It can therefore be argued that for cognitive tasks, like tactical decision making, novices can be provided with less details (i.e., cues), which means a lower fidelity overall. Yet, these finite details should be represented with a high level of physical fidelity. Later on, more details can be included, but at the same time these details can be represented in a more abstract manner (lower physical fidelity).

Motivation High motivation is a distinct characteristic of games and is often used as an argument for using games as a training tool. Belanich, Sibley and Orvis (2004) examined which features contribute to motivation of a PC-based first-person-perspective game for infantry training. They found that challenge, realism, control, and opportunities for exploration motivated participants to continue playing the game. Interesting is that realism was the most commonly mentioned reason by the participants. This finding is consistent with the study from Lampton, Bliss and Morris (2002). They state that poor fidelity during training may decrease motivation, attention to details, and subsequent transfer of training. Hence, it appears that fidelity has an impact on motivation, which in turn affects initial learning, and subsequently transfer of training. More specifically, it seems that motivation is affected by the perception of fidelity, which is said to be more critical than the actual/objective fidelity (Alessi, 2000). Allessi (2000) states that transfer of training is directly and indirectly influenced by fidelity. For example, it is possible that a training device has a low objective fidelity, but is perceived as very realistic. This could result in high motivation, and subsequently more time spend on the task which could increase initial learning. This could lead to high transfer, but it is also possible that what is


of 74

learned may not transfer to the application situation if too dissimilar. And this can also occur the other way around (Alessi & Trollip, 2001).

Accordingly, the influence of motivation on the relationship between fidelity and transfer is complicated. In order to have a better understanding of this influence, the relevant underlying constructs of motivation in (3D) games can be identified. These are presence, immersion and user acceptance (Alexander et al., 2005). Immersion and presence are best described as the sense of ‘being there’ in the virtual environment (Pace, 2008). They are widely considered to be desirable elements of 3D serious games, because they encourage learners to spend longer periods of time in the learning environment, which in turn can lead to higher transfer (Pace, 2008). User acceptance refers to the degree to which a person recognizes that an experience or event is useful for training (Alexander et al., 2005). When this is the case, it is assumed that users take on a ‘training mindset’, and in addition, such acceptance may increase the time spent on the game (Alexander et al., 2005). As a result, both effects can lead to increased learning and transfer.

For games with training purposes, especially user acceptance seems to be important. It is acknowledged that a high physical fidelity makes a simulation or 3D serious game more acceptable to trainees (Pongracid, Marlow & Triggs, 1997). But as was mentioned previously, a simulator or 3D serious game can still be effective without having high physical fidelity. Moreover, it was said that high physical fidelity can also interfere in reaching the full training potential. This suggests that while fidelity is a feature that may motivate players, there is a limit where greater levels of realism may no longer increase the effectiveness of a serious game (Belanich et al., 2004).

2.2.4 Summary

The core idea of transfer is that knowledge, skills, and attitudes learned in the training environment are spontaneously applied in the real environment. This application can either be positive, negative or nil, and can be explained in terms of magnitude. Furthermore, the distance of transfer of specific tasks can be described as a continuum ranging from near transfer tasks to far transfer tasks (applying what is learned to very similar versus quite different circumstances). There are two mechanisms explaining transfer of complex cognitive skills like tactical decision making: • Rule-based transfer focuses on the same use of the same skill by emphasizing the idea of sharing

the common elements relevant for the task. This mechanism is most important for near transfer tasks and benefits from ‘learning-by-doing’.

• Schema-based transfer focuses on the different use of the same skill by emphasizing the idea that cognitive schemata offer abstract generalized knowledge enabling one to understand a new situation. This mechanism is most important for far transfer tasks and benefits from ‘learning-by-thinking’.

For the training of tactical decision making in 3D serious games, rule-based transfer is of most importance for the game itself, whereas reflection activities afterwards contribute to schema-based transfer.

Fidelity is the degree of similarity between the training situation and the operational situation which is simulated. It encompasses the appearance (physical fidelity) and behaviour in terms of feedback (functional fidelity) of the simulator. Furthermore, when measuring fidelity, one must be aware of the difference between the actual similarity (objective fidelity) and the subjective judgments of people on similarity (perceptual fidelity) between the training situation and the operational situation.

Fidelity has a significant impact on the amount of transfer. However, the relationship between fidelity and transfer is complex. There are four main issues related to this complex relationship, which are summarized in Table 1. The focus in Table 1 is on cognitive tasks, as in this thesis tactical decision making is the end goal of 3D serious games.


of 74

Table 1 Summary of the Main Issues related to the Relationship between Fidelity and Transfer Main issue Elaboration The relationship between fidelity and transfer is nonlinear

A point exists beyond which one additional unit of fidelity results in a diminished rate of return on investment. Moreover, in some cases deliberate departures from reality may provide the most effective training.

Different tasks require different degrees of fidelity

For cognitive tasks, it is suggested that functional fidelity is more important than physical fidelity. The cue dominance approach can be used to make 3D serious games more efficient.

The necessary level of fidelity depends on the trainees’ stage of learning

For cognitive tasks, novices can be provided with less details (i.e., cues), which means a lower fidelity overall. Yet, these finite details should be represented with a high level of physical fidelity. Later on, more details can be included, but at the same time these details can be represented in a more abstract manner (lower physical fidelity).

Perceived fidelity has an impact on motivation, which in turn can affect transfer

It seems that high physical fidelity makes a 3D serious game more acceptable to trainees, but there is a limit where greater levels of realism may no longer increase the effectiveness of a 3D serious game.

2.3 Fidelity and transfer within 3D serious games for tactical decision making

This section specifies the discussion on the relationship between fidelity and transfer. The context, as described in section 2.1, is now basis for the discussion. As stated in Chapter 1 and section 2.1, computer based 3D serious games can be networked to provide collective training. Furthermore, such serious games provide opportunities for creating 3D digital worlds in a Virtual Environment (VE) with Artificial Intelligent (AI) agents and avatars. These characteristics can be used to represent a realistic context for the training of tactical decision making. More specifically, this realistic context can be used to provide varied (real-word) scenarios, relevant cues through sounds and images, and real-time feedback. In addition, 3D serious games can be used repeatedly to provide sufficient practice and can include time constraints, which also contributes to the effective training of tactical decision making.

In section 2.2 it was determined that higher levels of physical and functional fidelity do not automatically lead to more effective training. Still, the question remains as to exactly how realistic the ‘realistic context for training’ should be in order to provide effective training for tactical decision making with 3D serious games. Some valuable conclusions from section 2.2 can be used as a starting point for investigating this question. First of all, it seems a reasonable conjecture that the functional fidelity of 3D serious games with a tactical decision making objective should be high. Tactical decision making clearly is a cognitive task and all research related to this type of task suggest that the feedback on actions need to represent real-life realistically (e.g., Hays & Singer, 1989; Andrews et al., 1995).

Considering the training for infantry commanders, this means for example that aiming and then firing on someone in the leg should result in a wound in the leg and that person might not be dead yet (in contrast to a shot in the chest). Another example, one can imagine that aiming a gun at a civilian in front of you should result in a scared civilian, probably trying to get away from you as quickly as possible. Especially modelling this behavioural feedback is very expensive, but because of the fact that it does influence decisions a commander makes, the feedback should be correct. It is this way that trainees can learn from poor decisions (Cannon-Bowers & Bell, 1997). So, for tactical decision making in 3D serious games it is essential to experience realistic (real-time) feedback on actions. How this feedback is then represented is an issue of physical fidelity.


of 74

The necessary level of physical fidelity seems to be more complex. It has been suggested that cognitive tasks do not need a high physical fidelity. This can be exemplified by the previously mentioned example of the training for infantry commanders. It is essential to know that the person which was shot was hit in the leg and that he is not dead yet. But the way this is presented (visually and auditory) does not really matter, as long one understands what is represented and what this means for the effectiveness of the decision. The same is true in the example of the scared civilian. It might not be necessary to put a huge amount of effort in creating a true realistic scared facial expression if the same message can be conveyed in a less realistic way.

However, the influence of motivation should not be ignored. As mentioned in section 2.2, perceived fidelity is more important than actual fidelity, and this influences motivation and subsequently transfer of training (Alessi & Trollip, 2001). There should be a certain level of realism in a way that trainees experience its relevance, but this also has its limits (Belanich et al., 2004). A good example of this latter phenomenon in the field of tactical decision making is provided by Morris and Tarr (2002 in Belanich et al., 2004). They showed that quality of decision-making actually decreased with higher levels of graphics.

Furthermore, the goal of training tactical decision making is that trainees acquire a repertoire of relevant cues or patterns of cues, and build up mental models which they can use to match these relevant cues or patterns of cues with the correct decisions. If these cues in the game look and sound different than in real-life, will they form the right repertoire and mental model which they can use in real-life situations? In other words, if the cues are represented in a low fidelity manner, will they then transfer to the operational situation? Related to that, it was conjectured that for complex cognitive tasks, novices might need a more realistic representation than advanced trainees, because that might be helpful in acquiring schemata or in other words building up the mental models (Korteling et al., 1997).

The dilemmas mentioned suggest there should be a certain balance: physical fidelity should not be too low and not too high. In order to get a more detailed insight in this dilemma, further investigation on this topic is needed. Kessler (2002) states that VEs need to be (1) interactive: constantly present the current ‘view’ of the computer-generated world and have that world quickly react to the user’s action, (2) convincing: the presentation must provide enough detail to make objects easily recognizable and enough objects to give the user the sense of being in the world, and (3) useful: it must respond to the user (i.e., navigation, grabbing objects, etc.). These aspects are affected by the physical fidelity of the hardware as well as the software of the 3D serious game. The hardware essentially deals with the interface and interaction devices, whereas the software essentially deals with the objects and AI characters within the VE. Considering the interface and interaction devices, it was concluded that simple solutions like a desktop computer and a joystick, mouse and/or keyboard are sufficient for the training of tactical decision making (Visschedijk, 2009). The necessary level of physical fidelity related to objects and AI characters in the VE is more complicated. Therefore, these aspects are discussed consecutively.

2.3.1 Objects within the Virtual Environment

The term ‘object’ in a VE can be interpreted in the broadest sense; it encompasses roads, buildings, boxes, cars, etc. It is in fact everything non-AI of which the VE is created. Many objects serve as cues when making tactical decisions. For instance, mountains can be used to make navigational decisions, a bag on an unusual spot might indicate an explosive, or a car approaching with high speed can indicate a suicide bomber. The information objects carry can be represented visually and auditory. The level of physical fidelity of these two modalities influences the way objects are interpreted and processed.

Most research has been conducted related to the visual aspects of objects. The three most important visual factors in the fidelity of a 3D environment are the degree of realism provided by the rendered 3D images, the degree of realism provided by temporal changes to these images, and the degree of realism provided by the behaviour of objects (Delgarno, Hedberg & Harper, 2002). In other


of 74

words, the appearance of the object itself matters (i.e., perspective, texture, shading, etc), and the motion of an object in terms of smoothness and behaviour.

Interpretation of objects The appearance as well as motion of objects give information that is necessary for recognizing objects the way consistent with the ideas being modelled (i.e., ‘are they realistic’?). There is an interesting anecdote from Loftin et al. (2003). Their study focussed on a checkpoint task for military guards. This is a typical tactical decision making task where trainees needed to decide if a driver may pass a fictitious military base, pull over the vehicle or ask the driver to turn around. By inspecting each vehicle, interacting with the driver and verify identification cards, the trainees could collect the necessary information (cues). In this study, Loftin et al. (2003) noticed that the limited fidelity of the vehicle models had an impact on performance: “…in the back area of one of the vehicles (the jeep model) there was a box-shaped wheel that was often misconstrued as a suspicious package. This led some participants to question drivers in what were intended to be neutral scenarios” (Loftin et al., 2003, p.11). In other words, the semantic of the object was not correctly recognized, and therefore actually decreased performance.

Processing objects Recognition is to some extent influenced by the three-dimensionality of objects, also called visual depth cues. Examples are relative size and shading (Delgarno et al., 2002). It can be argued that the more learners are confronted with realistic 3D objects, the easier they will recognize objects, and mentally represent the graphical world better. As a result, they can build up mental models more effectively. However, research has shown that this might not always be the case, because other mechanisms also influence the ability to build mental models. A good example is provided by Mania et al. (2006). They were interested in the way learners process information with different levels of visual fidelity. Their experimental scene consisted of two interconnected rooms with various objects in it. The goal of their study was to identify whether radiosity rendering (high fidelity - shadows) is associated with stronger visually induced recollections linked with the remember awareness state compared to a flat-shaded scene (low fidelity – no shadows) displayed on a stereo head-tracked Head Mounted Display (HMD). In this context, remembering refers to experiences of the past in which previous events are recreated with the awareness of reliving those events and experiences mentally. This mental imagery is significant since this is a means by which information is learned, stored and retrieved, and accordingly also crucial for SA. Interestingly, the results of their study revealed a higher proportion of recollections associated with mental imagery in the flat-shaded condition. The authors suggest that something less ‘real’ may be more attentionally demanding because of its novelty or variation from ‘real’. The additional attentional demands that the low fidelity environment places on the cognitive system may therefore enhance the memorial experiences associated with it (Mania et al., 2006). The same pattern of results was found for the interaction interface of the VE (Mania, Troscianko, Hawkes & Chalmes, 2003). They demonstrated that less ‘naturalistic’ interaction interfaces or interfaces of low interaction fidelity induce a higher amount of recollections based on mental imagery than high fidelity interaction interfaces. Another study, conducted by Tack and Colbert (2005), investigated the physical fidelity of a high-density, complex urban terrain for a navigation task. Soldier participants were required to navigate through a high-density urban city based on a mental map of the area and the mission that they developed during pre-mission rehearsal with alternative visualization tools. Next to more conventional 2D maps and aerial photographs, the tools also included four 3D virtual models at different levels of detail (see Figure 5). In each of these conditions, the VE was viewed on a desktop computer, where soldiers could virtually fly over, around, and ‘walk through’ the terrain environment of each city sector using joystick controls. It should be noted that the MFVE + textures condition and the HFVE condition were used separately, while the LFVE condition was supplemented with a 2D map, and the MFVE condition was supplemented with an aerial photo.


of 74

LFVE MFVE MFVE + textures HFVE

Figure 5 Virtual Environments with different degrees of physical fidelity in the study of Tack & Colbert (2005). LFVE = Low Fidelity Virtual Environment, MFVE = Medium Fidelity Virtual Environment, MFVE + textures = Medium Fidelity Virtual Environment with textures, and HFVE = High Fidelity Virtual Environment Results showed that in the MFVE condition, soldier participants were more likely to succeed in reaching their objective and remain on their prescribed route than when using the other visualization methods. In estimating the location of mission critical features, the medium fidelity virtual model was significantly better for bearing and distance estimations than the other visualization methods. In addition, while the 3D virtual models were most preferred by participants, the conventional 2D visualization tools (i.e., map and photo) did perform better for map orientation. Interesting is the recommendation that terrain detail could be varied along a mission route. The basic terrain model could be provided at the level of the MFVE, while at key intersections, distant visual reference cues, and likely navigation orientation and decision points, the MFVE imagery can be enhanced with photo texturing to improve recognition. This makes it not only cheaper, but soldiers will also be focused more on mission essential details.

These findings confirm the notion that higher levels of graphics can actually hinder the quality of decision-making (Morris & Tarr, 2002 in Belanich et al., 2004). And thus, as Hays & Singer (1989) also suggested earlier, deliberate departures from reality can be necessary to provide the most effective training. Yet, it is impossible to generalize the features which can be less realistic (like a flat-shaded VE is always better), because it depends on the cues relevant for a specific task. For example, if a shadow contains information which can be used to make effective decisions, such as the revelation of an enemy behind a wall, it should be included. Moreover, the Tack and Colbert (2005) study aimed at navigation training, it is not clear if the same recommendation would be beneficial for other tasks related to tactical decision making.

2.3.2 AI characters within the Virtual Environment

In many 3D serious games with a tactical decision making objective, (human) AI characters form an important part of the training. For example, the behaviour of civilians in a military game can give information about the (un)safety of the situation (e.g., quietly walking civilians suddenly start running). Just like objects, the semantic of the appearance and motion of AI characters need to be realistic to the extent that they are recognizable. However, modelling recognizable AI characters, and in particular their behaviour, is much more complex compared to object modelling. Loftin et al. (2003) experienced the difficulty of Human Behaviour Modelling (HBM) as they noted: “The poor quality of facial expressions and behaviours generated some ambiguity as what constituted ‘suspicious’ behaviour in the Jack agents. During the AAR (After Action Review) sessions, participants had to be instructed to adjust their criteria to match the lower fidelity of the agents” (Loftin et al., 2003, p. 11). In other words, the Jack agents were misinterpreted which was the result of an insufficient level of physical fidelity. Besides incorrect decision making during training with the game, this might also affect transfer of the training negatively. For example, consider the possible effect that in real-life, the trainee might think suspicious behaviour is not that crucial as a cue.


of 74

As a result of the complexity and significance of HBM, many studies have been conducted in attempts to create highly realistic AI characters (e.g., van Lent, McAlinden, Probst, Silverman, O’Brien & Cornwell, 2004; Moya, McKenzie, Nguyen, 2008). It involves huge amounts of effort and thus money, which makes the fidelity question also within this area imperative in the effort to reach efficiency. As stated before, interpretability is one requirement of AI characters. Besides that, Kenney et al. (2007) suggested that virtual humans in training environments also need to be believable and responsive. In general, responsiveness deals with functional fidelity, while believability and interpretability are related to physical fidelity and also seem to be more subjective (note that believability is related to the influence of motivation).

Few research studies have aimed to determine the necessary level of physical fidelity of virtual humans. Still, it is useful to distinguish the main factors involved as it contributes to the overview of elements in 3D serious games where fidelity plays a role.

Main factors related to physical fidelity of virtual humans Virtual humans (avatars and agents) should be realistic according to their roles, in terms of appearance, animated action, communication, mental processes, and action selection (Allbeck & Badler, 2002). The latter two aspects, mental processes and action selection, are related to the responsiveness of the virtual human and thus part of functional fidelity. It was conjectured that functional fidelity should be high; therefore much effort should be put into these aspects of HBM. It should be realized though, that this is probably the most difficult to achieve and therefore the most expensive. For physical fidelity, the first three aspects are important. This means that appearance, animated action and communication should be believable and interpretable. A human’s appearance can be characterized by its age, aesthetics, gender, body type, ethnicity, hair, skin, bone, and muscle modelling. Much attention has been given to the modelling of hair, skin (including wrinkles and textures) and clothing (including folding, wrinkling and crumpling). An agent’s appearance will influence how he or she is perceived because it is often an external indicator of status and role (Allbeck & Badler, 2002).

Movement of agents is characterized by its locomotion, body actions and facial expressions. Locomotion of agents involves issues of path planning or navigation, and collision detection and avoidance. Examples of body actions are gesture generation, grasping, reaching, lifting and posture changes. They are crucial for the believability and interpretability of agent behaviour. Especially, linking body actions to an agent’s emotional state plays an important role. This is also the case for facial expressions. In essence, the face consists of a great deal of detail in muscle, bone, and tissue (Allbeck & Badler, 2002).

Communication as part of physical fidelity does not include the inner processes like dialogue planning, turn taking and speech synthesis (which is part of functional fidelity). But it does include the way gestures, body language, facial expressions and attention are represented. These aspects can indicate what a virtual human is thinking and feeling (Allbeck & Badler, 2002). When human behaviour form an important part of the relevant cues in 3D serious games with a tactical decision making objective, then this indication is of major importance.

2.3.3 Summary

The central question is “exactly how realistic the desired ‘realistic context for training’ of 3D serious games needs to be in order to provide effective training for tactical decision making”. It seems that feedback on actions needs to represent real-life realistically. In other words, functional fidelity should be high. The necessary level of physical fidelity seems to be more complex to determine: it should not be too low (because of low motivation and building the wrong mental models) and not too high (because of unnecessary costs and information overload). A more detailed investigation related to the necessary level of physical fidelity was provided.


of 74

It was concluded that the level of physical fidelity related to the interface and interaction devices do not need to be high. That is, simple solutions like a desktop computer, joystick, keyboard and computer mouse are sufficient.

Considering objects within the VE, it was concluded that the level of physical fidelity influences the way they are interpreted and processed. Physical fidelity is related to the appearance and motion of an object. Deliberate departures from reality can be necessary to provide the most effective training. However, the semantic of objects still needs to be recognizable and generalizations of features which can be less naturalistic cannot be made, because it depends on the cues relevant for specific tasks.

The physical fidelity of AI characters (mostly virtual humans) is related to the believability and interpretability of their appearance, animated action and represented communication. In this stage, it was not possible to draw any conclusions regarding the necessary level of physical fidelity of AI characters.

2.4 Summary and conclusion

The aim of this literature review was to provide an overview of elements of 3D serious games where fidelity plays a role and to conclude with fidelity guidelines. In order to do so, three main sections were distinguished of which the core results and conclusions are presented below.

Context description It was concluded that the 3D world of a VE with possible AI characters are central to 3D serious games. They can be used effectively for the training of tactical decision making, because it provides trainees with a realistic context for training. The realistic context offers trainees the opportunity to build up a necessary repertoire of relevant (patterns of) cues and mental models. This process can be enhanced by providing sufficient practice, varied scenarios, all the relevant cues, timely feedback, and time constrains. The end goal of these activities is transfer. It was concluded that for the training of tactical decision making in 3D serious games, rule-based transfer is predominant and of most importance for the game itself, whereas reflection activities afterwards contribute to schema-based transfer. Consequently, for the design guidelines of the game, sharing the common elements relevant for the task is essential. This means that 3D serious games for tactical decision making should contain the same cues which are present in real world performance.

Relationship between fidelity and transfer In essence, fidelity determines the degree of similarity between the 3D serious game and the operational environment. Hence, it was concluded that fidelity has a large impact on the amount of transfer, because it directly influences the idea of sharing the common elements relevant for the task. A closer look at the relationship between fidelity and transfer revealed its nonlinear complexity (‘more fidelity is not always better’). Furthermore, a distinction was made between physical fidelity (appearance) and functional fidelity (behaviour in terms of feedback). It was stated that the necessary level of the two types of fidelity depends on the type of task and the stage of learning. In addition, fidelity influences motivation and subsequently transfer of training. Moreover, regarding the fidelity of cues, it was stated that fidelity influences the way cues are interpreted and processed. Several overall conclusions related to the relationship between fidelity and transfer can be drawn. The following conclusions apply for all types of training devices and tasks: • A point exists beyond which one additional unit of fidelity results in a diminishing rate of return of

investment; • The fidelity of cues relevant for a specific task need to be high, while other parts allow lower levels

of fidelity; • If fidelity needs to be high, efficiency can still be reached by means of the cue dominance

approach;


of 74

• Perceived fidelity is more important than actual/objective fidelity; • High physical fidelity makes a 3D serious game more acceptable to trainees, but there is a limit

where greater levels of realism may no longer increase the effectiveness of a serious game.

Fidelity guidelines As from now, fidelity guidelines adapted to 3D serious games with a tactical decision making objective can be established. Crucial is the conclusion that a detailed task analysis should always be conducted for every specific task. This will be the starting point for decisions about fidelity guidelines, whereupon the general (evidence-based) fidelity guidelines can be used to validate these fidelity decisions. The context-related fidelity guidelines essentially deal with the devices of the game (i.e., interface- and interaction devices) and to a greater extent to the VE of the game (i.e., objects and AI characters within the VE). An investigation related to the mentioned aspects resulted in the following fidelity guidelines for the design of 3D serious games with a tactical decision making objective: • Functional fidelity should be high, while physical fidelity allows lower levels; • Novices can be provided with less details (i.e., cues), which means a lower fidelity overall. Yet,

these finite details should be represented with a high level of physical fidelity. Later on, more details can be included, but at the same time these details can be represented in a more abstract manner (lower physical fidelity).

• There is little need for expensive interface devices, a simple desktop computer is sufficient; • Regarding input devices, relatively cheap solutions like a computer mouse, keyboard and joystick

are sufficient. • Deliberate departures from reality can be necessary to provide the most effective training. This

can be done by making the VE less naturalistic or modelling the relevant cues with high physical fidelity while the rest of the VE is modelled in a lower fidelity matter;

• In all circumstances, the semantic of the appearance and motion of objects need to be clear and recognizable;

• Next to responsiveness which is related to functional fidelity and thus need to be high, AI characters in VEs also need to be believable and interpretable.

Discussion The conclusions aid to the understanding of the relationship between fidelity and transfer within 3D serious games for tactical decision making. Moreover, they provide some valuable guidelines, which seem to differ from the more ‘old fashion’ hardware simulators. Still, the guidelines remain elusive to a certain extent. This is for a great part due to the fact that for every specific task, the relevant cues are different, which makes it difficult to make generalizations. Despite this fact, more research is still necessary in order to make the guidelines more concrete.

This thesis aims to contribute to this necessity by doing empirical research on one of the elements where fidelity plays a role. Following the fidelity guidelines, it seems best to focus this empirical research on physical fidelity of the VE. Accordingly, this can either focus on elements of objects or AI characters (virtual humans) within the VE. The choice depends on the task analysis too, which is conducted in Chapter 3.


of 74

3 TASK ANALYSIS

A task analysis is an important starting point for decisions about fidelity guidelines. It should be conducted for every specific training task, because of the diversity of relevant cues. Accordingly, a list of cues discovered by a task analysis is of most importance regarding the fidelity issue. In this chapter, a task analysis is conducted in two different military fields; for infantry tactics (section 3.1) and Crowd and Riot Control (CRC) tactics (section 3.2). The training for infantry tactics is already implemented in a 3D serious game, whereas the training for CRC tactics is currently under development. The task analysis in the two different fields should be considered as case studies, as it is impossible to perform a task analysis for all possible military tactical tasks. This chapter results in an overview of the kind of cues relevant for military tasks. Furthermore, the methods used in the task analysis can also be used in other studies.

It should be noted that one might also call this chapter a cue analysis, as this task analysis only focuses on the objectives of the training and the associated cues. It does not examine appropriate actions, likely errors, etc. as this step is not relevant in the context of this Master thesis. In addition, it should be noted that the objectives and cues are described in a way they are or will be implemented in a 3D military serious game like VBS2.

3.1 Training for infantry commanders

Many of today’s military missions take place in urban settings, like the Dutch ‘stability and reconstruction’ mission in Uruzgan. Conducting missions in urbanized terrain puts enormous cognitive demands on infantry commanders. These small unit leaders must be able to make quick decisions in a complex, uncertain and dynamic environment. To prepare commanders for such missions, VBS2 is used for a number of objectives underlying the main goal, tactical decision making. In this section these objectives and associated cues of infantry squad commanders are analyzed. First the method of the task analysis is discussed in section 3.1.1, after which the results are presented in section 3.1.2.

3.1.1 Method

The task analysis for infantry commanders is conducted in two steps: (1) a document analysis, and (2) a discussion with an operational expert from TNO. The first step resulted in a first description of the objectives and a draft version of the list with relevant cues. The second step aimed to validate the objectives and cues, which resulted in a few adjustments.

Document analysis The main documents studied were from ‘t Hart, Vink and Buiël (2008), and Hartog (2009). The book from ‘t Hart et al. (2008) contains an integrated overview of the characteristics of the urban area and the urban threat, and illustrates the characteristics with examples of their military implications. The Master thesis from Hartog (2009) contains a more concrete view on the type of cues necessary for tactical decision making in VBS2.

Discussion with an operational expert The second step in this task analysis was a discussion with an operational expert from TNO. The described objectives were approved right away and it took about one hour to discuss the list of cues in detail. The adjustments made afterwards were checked again and approved.


of 74

3.1.2 Results

The overall goal of the training for infantry commanders with VBS2 is to train infantry tactics and command and control. The overall goal contains various objectives, which can be categorized by (1) general skills relevant for any situation, and (2) specific mission preparation and rehearsal.

General skills relevant for any situation: • Plan tactical operation; • Communication with squad; • Control squad; • Coordination with chain of command; • Control movement of squad; • Effective situated/tactical decision making; • Situational awareness (which involves recognition of environmental cues).

Specific mission preparation and rehearsal: • Orientation of terrain; • Learn specific threats of the mission; • Learn relevant situational factors. Cues can be provided in the planning phase or in the execution phase of a gaming session. In the planning phase they are provided in the mission order containing a situational description. In the execution phase they are provided in the VE of VBS2 or through communication. Accordingly, different types of cues can be distinguished in these two phases. Generally, the situational description provided in the planning phase contain so-called setting-related cues and partly environment-related cues. The VE and the communication during the execution phase also contain environment-related cues, and in addition civilian behaviour-related cues and OPFOR(OPposing FORces)-related cues.

A cue provides certain information. In other words, the meaning of a cue (level 2 in the SA model) is a distinctive factor. Broadly speaking, the four cues mentioned previously can contain two kinds of meanings: 1. The first kind contains the most critical cues; they give information about the (un)safety of a

situation. There are three levels of (un)safety: • There is no threat: it is safe. For example, children playing around; • There is a latent threat: there might be (acute) threat soon. For example, there has been

recent violence in the environment; • There is an acute threat. For example, gunfire.

2. The second kind contains cues that give information about the possibilities to effectively perform a mission. They deal with: • Navigation. For example, the use of landmarks; • The ability to control threat. For example, over watch positions like a high building.

Furthermore, cues can be static and dynamic. Static does not mean the specific cue cannot change at all, but if it does, it changes rather slowly (e.g., the amount of people). Dynamic cues, in contrast, are characterized by the possibility for rapid change (e.g., from walking to running). Now that different characteristics of cues are distinguished, some general ‘groups’ can be typified: • Setting-related cues are typically part of the situational description during the planning phase.

They can give information about the (un)safety of the environment (e.g., religious valuable days) and are static.

• Environment-related cues are part of the planning phase as well as the execution phase. They can give information about the (un)safety of an environment (e.g., channelling terrain) as well as


of 74

information about the possibilities to effectively perform a mission (e.g., cover and concealment using certain buildings). In addition, they are static by nature.

• Civilian behaviour-related cues are typically part of the execution phase. They can give information about the (un)safety of an environment (e.g., gathering of civilians). They can be static (e.g., playing children and women) as well as dynamic (e.g., civilians emotion changes from ‘neutral’ to acting ‘scared’).

• OPFOR-related cues are typically part of the execution phase. They can give information about the (un)safety of an environment (e.g., gunshots). They can be static (e.g., presence of IED building materials) as well as dynamic (e.g., a car comes towards BLUFOR with high speed).

Considering the conclusions from chapter 2, stating that the fidelity issue can best focus on objects and virtual humans in the VE, especially the cues provided in the VE are essential. This means that environment-related cues, civilian behaviour-related cues and OPFOR-related cues are elaborated in a list of cues presented in Appendix A.

3.2 Training for Crowd and Riot Control commanders

Not only are today’s military missions characterized by urban settings, they are also often characterized by low-intensity conflicts and operations. Meaning military forces need to consider the civilians that live in the environment in which the forces operate. Accordingly, the risk of crowds and riots is present. Petty, Gaskins and McKenzie (2003) state that crowds of non-combatants play a large and increasing role in modern military operations, often creating substantial difficulties for the combatant forces involved. A striking quotation of Ferguson (2003 in Petty et al., 2003) illustrates possible problems: "crowds are one of the worst situations you can encounter. There is mass confusion; loss of control and communication with subordinates; potential for shooting innocent civilians, or being shot at by hostiles in the crowd; potential for an incident at the tactical level to influence operations and policy at the strategic level" (p. 3). It is the task of commanders from the Military Police to make the correct decisions in these complicated situations. VBS2 is currently under development in order to prepare commanders for these situations. In this paragraph a number of objectives and associated cues for the training of CRC commanders are expounded. The method of the task analysis is discussed in paragraph 3.2.1, after which the results are presented in paragraph 3.2.2.

3.2.1 Method

The task analysis for CRC commanders is conducted in three steps: (1) a document analysis, (2) a brainstorm with the developing project team, and (3) a walkthrough of historical events with the developing project team. The first step resulted in an initial description of the objectives and an organization of the types of cues available. In the second step the objectives were checked and completed, and a draft version of the list with relevant cues was made. Finally, in the third step the cues were checked, which resulted in a few adjustments. The reason why this task analysis is more elaborate than the task analysis conducted for the training of infantry commanders, is because there was less information available.

Document analysis The first step in this task analysis was a document analysis. The main documents studied were from van Vliet and Fennis-Bregman (2004), and Frini, Stemate, Larochelle, Toussaint and Lecocq (2008). Both reports consist of a literature review discussing characteristics and causes of crowds and riots, phases of crowd development, significant factors influencing the behaviour of individuals in a crowd, and the (possible) effects of intervention from the Military Police.


of 74

Brainstorm with the developing project team The second step in this task analysis was a brainstorm with the developing project team from TNO. The categories of cues resulted from the former step and some short movies of crowds and riots from YouTube were used as a starting point. The brainstorm took about two hours in total.

Walkthrough of historical events with the developing project team The third step in this task analysis was a walkthrough of historical events with the developing project team from TNO. Two events where analysed using multiple movies from YouTube: the student riot in Amsterdam in November 2007, and the Heysel stadium tragedy in Brussels in May 1985. The first event was a typical riot where a demonstration turned into a clash with the police. The second event was an example of football violence resulting in complete panic and people killed. Both events were extensively analysed by describing every phase using the cues from the list. In this way, the team was able to refine the cues and complete the list. Both walkthroughs took about two hours.

3.2.2 Results

The goals and objectives of the training for CRC commanders with VBS2 are quite similar to the training for infantry tactics. The overall goal is also to train tactics (but now CRC) and command and control in the field of CRC. However, specific mission preparation and rehearsal is not targeted, as environment-related cues are far less important than civilian (crowd) behaviour cues. This latter type of cue is so unpredictable that it is impossible to prepare for specific missions. So, the objectives are only related to general skills relevant for any situation: • Plan tactical operation; • Communication with team; • Control team; • Coordination with chain of command; • Control movement of team; • Effective situated/tactical decision making; • Situational awareness (which involves recognition of environmental cues). Like the training for infantry commanders, cues can be provided in the planning phase of a gaming session by means of a mission order containing a situational description, or in the execution phase of a gaming session in the VE of VBS2 or through communication. Also setting-related cues, environment-related cues and civilian (crowd) behaviour-related cues can be distinguished. OPFOR-related cues are not present; the crowd can now sometimes be seen as the ‘enemy’.

All possible cues related to these three types contain one kind of meaning: they give information about the (un)willingness to show aggression. It contains three levels: • The crowd is peaceful; there is no willingness to show aggression. For example, the people of the

crowd have neutral facial expressions. • (Part of) the crowd might become aggressive soon; there is a willingness to show aggression. For

example, the use of flags and similar clothing. • (Part of) the crowd is behaving aggressive; there is aggression. For example, vandalism. In addition, cues can also be static (e.g., sunny weather) or dynamic (e.g., transition from only an active splinter group while the main group is inactive, to a collective movement with the presence of a splinter group) of nature. The same kind of general groups as for the training for infantry commanders can be typified for CRC training.

As the VE is considered to be the central focus in the fidelity issue, the cues related to the environment and civilian behaviour is elaborated in Appendix B. In the list, the two types of cues are presented together, because there are only a few environment-related cues.


of 74

3.3 Summary and conclusion

The main aim of this chapter was to find out the type of cues relevant for the military field. Therefore, a task analysis was conducted in two different military fields: for the training of infantry commanders and for the training of CRC commanders.

Results show that the objectives of the training within the two fields are quite similar. However, for infantry commanders there are also objectives related to mission preparation and rehearsal, while these are not targeted in the training of CRC commanders. In a gaming session, the associated cues can be provided in the planning phase or in the execution phase. As the fidelity issue can best focus on objects and virtual humans in the VE, the cues presented in the execution phase are elaborated in Appendix A (infantry) en Appendix B (CRC). For infantry commanders the cues are related to environment, civilian behaviour, and OPFOR. These cues can give information about the (un)safety of a situation and the possibilities to effectively perform a mission. For CRC commanders the cues are related to the environment, and in particular to civilian behaviour. These latter cues can give information about the (un)willingness to show aggression.


of 74

4 RESEARCH QUESTIONS

By now, the fidelity guidelines and type of cues relevant for military tactics have been established. It resulted in an overview of the elements within 3D military serious games where fidelity plays a role. Now it is time to go one step further, and provide empirical evidence for these elements. As it is beyond the scope of this Master project to address all elements, a research focus is required. In this chapter, the choice for the research focus is substantiated in section 4.1, after which some related work is discussed section 4.2. Finally, the research questions are provided in section 4.3.

4.1 Research focus

In the theoretical framework it was concluded that the empirical research should focus on the physical fidelity of objects and virtual humans within the VE of 3D serious games. In the task analysis, the cues relevant for military tactical tasks in the VE were identified. These were related to environmental aspects (objects) and the behaviour of civilians and OPFOR (virtual humans). Following the idea of selective fidelity, it can be argued that these cues should be designed in a high fidelity matter, while other aspects in the VE might allow lower levels of fidelity. However, modelling the behaviour of virtual humans with a high level of fidelity is extremely complex and expensive. Therefore, the fidelity issue related to HBM is imperative in the effort to reach efficiency. This is especially true for modelling emotional states. This aspect is not only one of the most challenging and thus expensive aspects of HBM, but also captures an important part of the possible relevant cues within military fields like infantry tactics and CRC tactics. Consequently, it seems a defendable choice to focus the empirical research on the physical fidelity of emotional states of virtual humans.

4.2 Related work

Before moving on to the research questions, first some other research studies related to this research focus are discussed. The main aim of this section is to shed more light on the complexity of modelling emotional states in virtual humans, and to infer some predictions regarding the optimal level of physical fidelity.

4.2.1 Fidelity of emotional states

Humans express their emotions mainly through facial expressions, bodily movements and tone of voice (Argyle, 1988). Whether these behaviours are intentionally communicative or not, they often convey considerable information about a person, their emotional arousal, their attitudes and what they are attending to (Gratch & Marsella, 2001). For the training of tactical decision making in a 3D military serious game, this information can be effectively used to make sound decisions. For example, an infantry commander can observe the nonverbal behaviour of civilians and thereby indicate how safe they feel in the environment (i.e., are they scared, or angry with the soldiers? etc.). And within the CRC field, the emotional state of (groups within) a crowd, often related to the level of willingness to show aggression, provides the most important cues for commanders in deciding what to do (Moya et al., 2008).

A parallel can be drawn between human emotions functioning as cues in 3D military serious games with a tactical decision making objective, and VE-specific skills like walking with a joystick. It has been suggested that walking is such an over-learned and flexible skill, that it will probably not detrimentally transfer to real world activities (Knerr, Lampton, Singer, Witmer & Goldberg, 1998). Recognizing and interpreting human emotions can also be typified as an over-learned skill. Therefore, it is probably safe to suggest that the represented emotions of virtual humans will be generalized from


of 74

pre-existing mental models of emotions, which will not lead to negative transfer. In other words, one can say that trainees are very likely to be aware of the difference between represented emotional states in the game compared to real life. Accordingly, they will probably not learn the ‘wrong’ emotional states. The fact remains that emotional states of virtual humans need to be interpretable and believable. Concerning the interpretability, users should be able to recognize the intended emotional expression. And concerning believability, Shaarani and Romano (2008) stated that if a synthetically generated character fails to express the required suitable emotional expression, it will most likely break users’ suspension of belief. This could lead to low motivation (and possibly low transfer) as a result.

Attempts to reach efficiency in modelling emotional states are often related to the cue dominance approach. This approach applies within each modality (what are the most distinctive features within the face, posture and tone of voice?) and between modalities (is the face, posture or tone of voice the most distinctive modality?). The most distinctive features of every modality are discussed first, followed by the interaction between the modalities.

Facial expression It has been suggested that reproducing emotional states in facial expressions in as full detail as possible is a waste of effort and can even be counterproductive. More abstract approaches reflecting emotions in simple ways may be more appropriate (Benford, Bowers, Fahlén, Greenhalgh & Snowdon, 1995; Bartneck, 2001). Donath (2001) says that because the face is so highly expressive and we are so adapt at reading it, any level of detail in 3D facial rendering could potentially provoke the interpretation of various social messages; if these messages are unintentional, the face is arguably hindering communication rather than helping it. She therefore argues that the key is to find a balance between the information provided and the message that is sent. If minimal information is provided, a minimal message should be sent (Donath, 2001). Related to that, the most universal facial expressions are surprise, happiness, fear, anger, sadness, and disgust (Ekman, Friesen & Ellsworth, 1972 in Bartneck, 2001). In other words, these expressions contain a simple message, and are quite easy to recognize. Based on the cue dominance approach, the six expressions can be modelled relatively easily. Accordingly, it has been suggested that the eyebrows, the upper/lower eyelids, the eyes, the mouth corners and the lips are the most distinctive and essential features of a facial expression (Fabri, Moore & Hobbs, 2002).

Postural expression Based on the previous argument that recognizing and interpreting human emotions is an over-learned skill, it seems reasonable to assume that prototypical postures are appropriate. This was corroborated by Atkinson, Dittrich, Gemmell and Young (2004), who found that exaggeration improves recognition of portrayed emotions (except for the emotion ‘sad’). Consequently, many studies used prototypical postures depicting a clearly defined emotion (e.g., Coulson, 2004; Kleinsmith, De Silva & Bianchi-Berthouze, 2006).

Coulson (2004) was one of the first who did systematic research in order to identify the distinctive features of (static) postures. Examples of important posture features include head inclination, which is typical for sadness, or elbow flexion, which observers associate with the expression of anger. He found that, in general, human emotion recognition from posture is comparable to recognition rates from the voice, and some postures are recognized as effectively as facial expressions.

Vocal expression Tone of voice is less distinctive than facial and postural expression, because humans make far less use of emotional sounds (Argyle, 1988). The most obvious ones are laughs, cries, sighs and yawns. However, it is being questioned whether such vocalizations are each linked to a specific, discrete state (Russell, Bachorowski & Fernández-Dols, 2003). It has been suggested that the vocal characteristics of speech related to emotions might be easier to detect (Bartneck, 2001; Russell et al., 2003). The


of 74

most influential parameters in speech are pitch, tempo and loudness. These parameters are used to quantify different emotions (e.g., Murray, 1992 in Bartneck, 2001). However, these quantifications still remain somewhat elusive. Russell et al. (2003) concluded that the strongest single association was found between vocal acoustics and the sender’s general arousal level.

Multimodal expression Expressive behaviours of the face, voice, and body are typically studied in isolation. Although many scholars agree that recognition accuracy tends to improve with multiple modalities compared to individual modalities (e.g., Argyle, 1988; Crane, 2009), few studies investigate multimodal expression –and the ones that do merely explore the joint contribution of the face and the posture (e.g., Crane, 2009; Clavel, Plessier, Martin, Ach & Morel, 2009; Vinayagamoorthy, Brogni, Steed & Slater, 2006). Following are some results related to multimodal expression via face and posture. Ekman and Friesen (1967 in Vinayagamoorthy et al., 2006) stated that bodily cues act as a valuable indicative tool for the intensity of an emotion, and in some situations they may act as a more dominant source of information in the perception of affect. Furthermore, it has been argued that even though the face is the most expressive area of the body and the primary carrier of effect, when it is accompanied with congruent bodily cues there is less ambiguity while communicating affect (Argyle, 1988). Vinayagamoorthy et al. (2006) investigated these phenomena with virtual characters in an immersive virtual environment. They focused on two types of behavioural cues (facial expressions and posture) and two emotional states (angry and sad). Interesting is that “all the participants in the angry conditions with postural cues mentioned that body language was a primary indicator of the active character’s underlying emotional state” (p. 235). This effect was also found by a more objective measure, the participant’s psychological response. Overall, the results of the study indicate that postural cues play a vital role in the communication of affect by virtual characters in the case when the state portrayed is ‘anger’, but not when it is ’sad’. This latter effect is probably due to the ineffective modelling of sad body movement. Yet, the main interest in this study was the anger condition, because of its association with threat perception and the varieties of possible responses invoked in individuals witnessing an act of aggression. These results are therefore very relevant for tactical decision making in the military field. Clavel et al. (2009) also studied the expression of emotions in virtual characters via the face and the posture. Their first study aimed at assessing the relative contributions of the face and the posture in the perception of a single emotion. They observed that emotional congruence at the level of the body and the face facilitates the recognition of emotion categories (anger, joy, sadness, surprise, and fear) and dimensions (activation and valence) compared to face only or posture only presentations. Nevertheless, face only and posture only also yielded recognizable emotion categories. The second study considered the influence of incongruent expressions. Judgments were mainly based on the facial expressions, but were nevertheless affected by postural expressions. In such incongruent cases, posture proved to be useful for assessing the perceived activation of the emotion. In other words, it seems that facial expressions are crucial to develop a perceptual judgment, whereas postural expressions are especially used to establish a judgment about the level of activation. Furthermore, one must be aware of the fact that the distance to the virtual human representing a certain emotion plays a role. Whole-body expressions seem to be preferentially processed when the perceiver is further away from the stimulus. When the facial expression of the producer is not visible, emotional body language becomes particularly important (Gunes, Piccardi & Pantic, 2008).

4.2.2 Summary and conclusion

Humans express their emotions mainly through facial expressions, bodily movements and tone of voice. In general, it seems that combining these modalities results in the highest change of believability and interpretability of human emotions. Efficiency can be reached by means of the cue dominance approach. This approach can be used in two ways: (1) within modalities where the most distinctive features of every modality are explored, and (2) between modalities where the most


of 74

effective combinations of modalities are explored. Within modalities the following conclusions can be drawn: • Facial expression: it is best to use the most universal emotional states (i.e., only the six mentioned

expressions) with only simple adjustments of the most distinctive and essential features of facial expression (i.e., eyebrows, mouth corners, etc.).

• Postural expression: prototypical postures improve recognition of portrayed emotions. They can act as a valuable indicative tool for the intensity of an emotion in particular.

• Tone of voice: only the arousal level aids substantially the interpretation of human emotions. And between modalities the following conclusions can be drawn: • There are only a few studies which investigate multimodal expression. Moreover, in most of these

studies only the combination of face and posture were explored. • It seems that the emotion ‘anger’, but probably also ‘panic’ (which is also related to threat

perception), takes more advantage of body posture than of facial expressions. • In incongruent cases, facial expression is dominant over postural expression. • Distance to the virtual human plays a role; the further away the virtual human is, the more

important postures become. Both types of conclusions are used in the empirical investigation. The conclusions regarding the most distinctive features of every modality are used in the design of the virtual human presented in the experiments. The conclusions regarding the most effective combinations are used as a basis for the research question.

4.3 Research questions

The conclusions from the previous section provide some insight into the features of emotional states that enable recognition. However, it does not yet provide an adequate answer to how much physical fidelity is really necessary in order to recognize the emotional states of virtual humans in 3D military serious games.

To large extent, this is due to the fact that currently, no research has been conducted to investigate the interaction between the three modalities face, posture, and tone of voice in virtual humans. In addition, emotion modelling has not yet been investigated in the context of a 3D military serious game. This context contains several implications for the necessary level of physical fidelity. That is, for military tasks like infantry tactics and CRC tactics, the emotions angry, aggressive, scared, panic, elation and neutral are said to be important cues for tactical decision making (see Appendix A and B). These differ slightly from the six mentioned universal emotional states in a way that some emotions belong to the same ‘family’ (like angry-aggressive and scared-panic). Furthermore, the virtual humans are part of a 3D military serious game. Characteristics of 3D serious games as well as the fact that military tactical decision making is the most important goal of these games, may impact the way emotional states of virtual humans are interpreted and believed.

Consequently, an empirical investigation is necessary which differs from other research studies in a way that it (1) focuses on the interaction between the three modalities, (2) focuses on the cues relevant for military tactical decision making, and (3) includes contextual influences of 3D military serious games. This investigation aims to answer the following main research question:

“What is the desired level of physical fidelity of emotional states of virtual humans within the 3D world of a military serious game”?

This research question was answered using the cue dominance approach. This approach is used to investigate the interaction between the three modalities, as much research already focused on the most distinctive features of the three modalities individually. The goal of such an investigation is to discover whether it is really necessary to combine all three modalities, or if one or two modalities are


of 74

sufficient (for example, whether the same message comes through without a facial expression). This would result in a reduction of development costs. In other words, the desired level of physical fidelity is expressed in the number of cues (only one modality, two modalities or all three modalities) and the type of modality (facial expression, posture, and/or tone of voice). It can be stated that the level of physical fidelity increases with more modalities. In this study the features per modality are fixed in a way that the most distinctive features are used, as it is practically far beyond the scope of this Master project to investigate different features of modalities interacting with different combinations of modalities.

Two studies were conducted to discover the desired level of physical fidelity related to the interaction between modalities. One study to compare different combinations of the three modalities, and another study to investigate the added value of contextual information. Accordingly, the following two subquestions were formulated: 1. What is the relative effectiveness of different combinations of facial expression, posture, and tone

of voice to represent emotional states relevant for military tactical decision making in virtual humans?

2. How is this effectiveness influenced by contextual information? The first subquestion was answered through an experiment in which the different combinations of modalities were compared. For each combination, participants were shown animations of a virtual human with a certain emotional state (angry, aggressive, scared, panic, elation or neutral) and asked to describe the emotional state. Any prior knowledge about the meaning of these emotional states in a military context was irrelevant in this experiment. Hence, the answer to this subquestion formed the basis for the desired level of physical fidelity of emotional states of virtual humans. Yet, this answer is not related to the way emotions of virtual humans will be implemented in 3D military serious games. As mentioned before, there are some characteristics of 3D military serious games which might influence the way emotional states are interpreted and believed. One of the most influential characteristics is contextual information (Fabri, Moore & Hobb, 2002; Carroll & Russell, 1996). It is hypothesized that recognition of emotional states in virtual humans is easier when contextual information is included, and presumably cause smaller differences between combinations. Consequently, this hypothesis was investigated in study 2 so as to answer the second subquestion. The setup and methods of study 2 were similar to those of study 1 except that fewer combinations of modalities were involved, and each combination was extended with contextual information. That is, due to practical limitations only the most effective combinations resulting from the first study were included and participants heard a context description before they watched the animations. Now, also the meaning of emotional states of virtual humans in the situation became relevant which made it interesting to use one of the target groups for the 3D military serious games, students from the Dutch Military Police (the KMAR), as participants. These participants already had some prior knowledge and could therefore be asked to describe what their decision in the situation would be in addition to their description of the emotional state. This made it possible to go one step closer to the way emotional states of virtual humans are implemented in 3D military serious games. The goal of this second study was to discover if the influence of contextual information is that high, that differences between these most effective combinations become so small that lower levels of physical fidelity are also adequate.


of 74

5 STUDY 1: RECOGNIZING EMOTIONAL STATES OF VIRTUAL HUMANS

This chapter presents the first empirical study, to which the following subquestion is central: “what is the relative effectiveness of different combinations of facial expression, posture, and tone of voice to represent emotional states relevant for military tactical decision making in virtual humans?” An experiment has been conducted to answer this question. The research method of this experiment is described in section 5.1, after which the results are presented in section 5.2, and the discussion in section 5.3.

5.1 Method

5.1.1 Participants

A total of twenty-eight participants (n=28) took part in this study (16 male and 12 female, average age 32 years). Most of them were TNO employees, and a few were acquaintances from outside of TNO. Participation was voluntary.

5.1.2 Materials

The core material for the experimentation were animations of a virtual human representing certain emotions. In addition, an instruction form was designed to provide participants with a proper instruction before starting the experiment.

Animations The animations consisted of one standing virtual human with a neutral grey background. Every time, a certain emotional expression was represented by means of a facial expression, posture and/or tone of voice. The following six emotional expressions were indicated as relevant for 3D military serious games, and are thus key to this research study: neutral, angry, aggressive, scared, panic, and elation. Each animation lasted five seconds, from a neutral pose to the expression of an emotion and back to a neutral pose. They were designed using an Animation program called Moviestorm. This program contains standard virtual humans, designed with a reasonable amount of fidelity. Figure 6 contains a screenshot of the virtual human used in the experiment.

Figure 6 Screenshot of the virtual human used in the experiment (neutral pose)


of 74

The appearance of the virtual human selected is comparable with the virtual humans in the 3D serious games the RNLA uses. This seems an adequate choice, as there is no clear evidence that the amount of fidelity of the appearance of the virtual human influences the recognition rate of emotional states. That is, regarding postures Pasch and Poppe (2007) concluded it does (depending on the emotional state), while McDonnell, Jorg, McHuch, Newell and O’Sullivan (2008) concluded it does not.

The postures and facial expressions of the emotions were created using the prototypical expressions of the Moviestorm collection of postures, gestures, and facial expressions. Their accuracy was checked through several relevant literature studies. Accordingly, the facial expressions were checked with the distinctive features described by Fabri, Moore and Hobbs (2002) and Gunes and Piccardi (2007). The postural expressions were checked with the database of acted postures from Kleinsmith et al. (2006) and the general characteristics of postural expressions mentioned by Coulson (2004). In Appendix C all postural and facial expressions are illustrated. For tone of voice, sound samples of crowds were used. They were downloaded from the internet. At first instance, it seems a bit strange to use crowd sounds since one virtual human was plotted. Furthermore, crowd sounds could give away certain information related to the context. Yet, considering the alternatives and the way sounds are used in 3D military serious games, this seems to be the best choice. That is, methods to induce individual sounds samples still have many drawbacks considering their reliability (see Scherer, 2003). Moreover, practically it would take a huge amount of effort to make individual sound samples as reliable as possible, while in the 3D military serious game itself only crowd sounds will be used.

As a final check for attaining the right postures, facial expressions and sounds, the animations were agreed by four participants who did not take part of the main study. This was done by means of pilot-testing, in which the participants were shown six animations, one for every emotion and containing all three modalities. They were first asked to describe the emotion of each animation. Afterwards, there were told what the intended emotions were, and they were asked to judge every modality of every emotion and asked for possible improvements. The pilot test resulted in some adjustments to the animations.

Thirty animations were designed in total, consisting of five different combinations and six different emotional expressions: • Expression of emotion by means of Posture, Face and Tone of Voice; • Expression of emotion by means of Posture and Face (no voice); • Expression of emotion by means of Posture and Tone of Voice (face contains a neutral

expression); • Expression of emotion by means of Posture (no voice and face contains a neutral expression); • Expression of emotion by means Face and Tone of Voice (posture contains a neutral expression). The combinations facial expression only and tone of voice only were left out of consideration. A requirement of the representation of emotions of virtual humans in 3D military serious games is that it can be noticed from far away too. Only representing facial expressions is therefore simply insufficient (Gunes et al., 2008). Concerning tone of voice, it was argued that only the arousal level can aid substantially to the interpretability of human emotions. Valence (negative vs positive) seems to be more elusive (Russell et al., 2003). This means, for example, that anger and elation, both producing high fundamental frequency and high amplitude, can be quite easily confused. Notice that in combinations without tone of voice, there is no sound at all, while in combinations without posture or face, the expression is ‘neutral’. The reason for this is practical. In the 3D military serious game, it would be quite odd to leave out the rest of the body or to remove the face. This in contrast to voice, which can be easily removed.

Instruction form A standardized instruction was designed that addressed the following items: 1. Welcome: participants were welcomed and thanked for their time.


of 74

2. Goal research study and experiment: participants were explained about the overall goal of the research study (fidelity of 3D military serious games) and its focus (“how realistic should emotions be represented in virtual humans”?). This was followed by the explanation that the experiment aims at answering this latter question.

3. Watching the animations: participants were explained that they had to watch 30 animations lasting 5 seconds each. They were asked to describe the emotion of each animation. Then they got to see a still image of the virtual human. It was explained that this is its neutral pose and that every animation starts with this neutral pose and ends with this neutral pose. In between, an emotion can be represented (this was told with an emphasis on ‘can’). In addition, they were told that they can repeat an animation as often as they like, before moving on to the next one. And they were instructed to open every next animation by themselves.

4. Responding: participants were told to describe the emotional states in any fashion as they like (words, situation descriptions, etc.), while the experimenter would write them down.

5. Crowd sounds: participants were explained that there are sometimes sounds in the animations and that these sounds are from crowds. Participants were asked to treat these sounds as if the virtual human is one of the crowd making the sound (instead of a surrounding sound).

6. Final questions: participants were asked if they are ready to start or if they still have some final questions.

Several decisions related to this instruction form merit further explanation. First, participants were shown what the neutral pose of the virtual human looks like, as in neutral position the virtual human (just like most people in real life) looks a bit grumpy, which would bias its recognition (they are primed to find something of an emotion). Second, in line with the previous explanation related to priming, participants were told an emotion can be represented, as 5 of the 30 animations represented the emotion ‘neutral’. By putting an emphasis on ‘can’, participants knew they did not have to describe a specific emotion if they did not recognize one. Third, participants could watch an animation as often as they liked, because an animation lasted only five seconds and it was predicted that some people might need more time than others or that some people would not pay enough attention the first time they watched an animation (both predictions turned out to be true). In the 3D military serious game users cannot repeat an animation, but it is assumed that users are more immersed when they are playing the game and therefore pay more attention. Finally, it is quite instinctive to treat crowd sounds as surrounding sounds if there is just one virtual human present. Therefore, it was necessary to tell participants explicitly that the sound is also produced by the virtual human.

5.1.3 Design

The experimental design which was used is called a repeated measures design. This means that all participants got to watch all 30 animations.

The advantage of this design is that it offers opportunities to study research effects while ‘controlling’ for participants. Repeated measures designs offer greater statistical power relative to sample size. However, one must be aware of internal validity threats called ‘carryover’ effects: effects from one treatment that may extend into and affect the next treatment (Minke, 1997). In other words, later animations are easier to judge, because participants already saw comparable animations.

In order to anticipate possible order effects, participants were randomly allocated to one of four versions (A till D). Within these four versions, the sequence of the five combinations was counterbalanced. The sequence of the emotions within each combination was counterbalanced as well. The following rules were used for counterbalancing: 1. The combination P+F+V was never used in the beginning; 2. The ‘combination’ P was never used at the end; 3. The combinations were randomly mixed in a way that they are never on the same position; 4. At the very beginning, in no version the experiment started with the emotion ‘neutral’; 5. There were never two of the same emotions after another;


of 74

6. Restrictions outlined in rules four and five, the emotions within combinations were randomly sequenced.

Application of these rules lead to the four versions which are illustrated in Figure 7. The reason why combination P+F+V was never at the beginning, is because then it would be too easy to describe the subsequent ones. With ‘combination’ P this is the other way around. It would be quite easy to describe the emotions within P if a participant already saw all the other combinations. There is a fair chance that a participant would not look at these animations as if they see the postures for the first time. However, in order to still keep it as random as possible, the rest of the combinations were balanced in a way that they are never on the same position. As participants first saw the neutral pose of the virtual human in the instruction phase, it would have been quite odd to start the experiment with an animation representing a ‘neutral’ emotion. That is why the experiment never started with this emotion. In addition, two of the same emotions after another might irritate a participant, so this was also left out of consideration.

Figure 7 Four versions with different sequences in experiment 1. Combinations: P = Posture, F = Face and V = tone of Voice. Emotions: Ag = Aggressive, P = Panic, N = Neutral, E = Elation, An = Angry, and S = Scared

5.1.4 Procedure

Data collection The experiment was displayed on a 17-inch monitor of a laptop. Participants entered the experiment one at a time. This means a participant was sitting behind the laptop and the experimenter was sitting next to him/her. Participants watched and judged all 30 animations. Together with the instruction as described above, this took about 15 to 20 minutes.

Coding and scoring Participants rated the animations by means of a free response format. They could verbally respond with freely chosen labels or short expressions that in their mind best characterized the recognized emotional state. These open-ended responses are more informative and valid compared to a forced-choice methodology. Moreover, in the 3D military serious games there is no list of possible emotional states either. However, its major disadvantage is the degree of subjectivity in interpreting and

Ag P N E An S

P E An S N Ag

S Ag An P E N

E P S An Ag N

An N Ag S P E

P

P+V

Version A

P+F

P+F+V

F+V

P Ag N E An S

E An S N P Ag

N P An S Ag E

S Ag E An P N

An N P Ag S E

P+F

F+V

Version B

P+F+V

P+V

P

An P N E Ag S

E N Ag P S An

P E An S N Ag

S An Ag N E P

N E P Ag S An

P+V

P+F

Version C

P

F+V

P+F+V

S P N E An Ag

N E Ag P S An

Ag N An P E S

P E Ag S An N

An Ag N P S E

F+V

P+F+V

Version D

P+V

P

P+F


of 74

analyzing the responses. Russell (1994) explained that an open-ended format tends to yield less agreement and more use of non-emotion words (this is for example the case with ‘frustration’, which was often used to describe ‘anger’ expressions in his study). As an exact label for each emotional state is not necessary for the use of representing emotional states in 3D military serious games, responses could be coded somewhat liberally. In order to do so as accurately as possible, the judgement task was done verbally. This means the experimenter asked which emotional state the participant thinks is represented and the experimenter wrote down their answers. In this way, it was more likely the participant reported his feeling about the emotional states more precisely and nuanced.

It was mentioned that responses could be interpreted loosely. In order to keep it as objective as possible, a coding scheme was created based on the words and descriptions mentioned during the pilot-test. Afterwards, the Dutch dictionary was used to supplement the coding scheme with other synonyms.

An additional advantage of the coding scheme was that a second rater could also analyse the answers. This made the analysis more reliable. And if one of the raters hesitated whether a description of an emotion was correct, he could discuss it with the second rater until agreement was reached.

5.1.5 Data analysis

Two raters scored participants’ answers as either correct (score = 1) or incorrect (score = 0). First, the inter-rater reliability between the two coders was calculated using Cohen’s Kappa. Next, in cases where the two coders disagreed, they came to a final answer by means of a discussion.

The main goal of this study was to discover the relative effectiveness of different combinations of facial expression, posture and tone of voice. Therefore, a one-way repeated measures ANOVA analysis was conducted in order to analyse the data. With this analysis the effect of the variable ‘combination’ could be assessed. First, it was interesting to find out if differences between combinations were significant. If so, the scores of the different combinations were compared in order to find the most effective combination.

5.2 Results

Inter-rater reliability The coding of both raters matched in 86% of the occasions. The inter-rater reliability (Cohen’s Kappa) was 0.72. Both scores show that the inter-rater agreement is substantial, and consequently support the use of a free-response format.

Recognition rates of combinations Results of the one-way repeated measures ANOVA showed a significant effect of combinations, F(4,108) = 14.56 p < 0.001. In other words, if all other variables are ignored, scores were different for the combinations Posture, Posture+Face, Posture+Face+Voice, Posture+Voice and Face+Voice. These differences are represented in Table 2. Table 2 Mean Recognition Rates by Combination

Combination M SD

Posture .51 .17

Posture+Face .69 .14

Posture+Face+Voice .78 .14

Posture+Voice .64 .17

Face+Voice .58 .15


of 74

Table 2 shows that combination Posture+Face+Voice yielded the highest recognition rate (78%), whereas Posture yielded the lowest recognition rate (51%). Cross-combination differences between the second (Posture+Face, 69%), third (Posture+Voice, 64%) and fourth (Face+Voice, 58%) placed combinations were quite modest, which makes it interesting to analyze which combinations significantly differ from each other. Therefore pairwise comparisons have been conducted; their results are presented in Table 3. Table 3 Pairwise Comparisons of the Recognition Rates in All Five Combinations (Scores Represent the Mean Differences Among Combinations)

P P+F P+F+V P+V F+V

P – P+F -.17* – P+F+V -.27* -.10* – P+V -.13* .05 .14* – F+V -.07 .10 .20* .05 –

Note: Bonferroni correction has been applied to adjust for multiple comparisons. P = Posture, P+F = Posture+Voice, P+F+V = Posture+Face+Voice, P+V = Posture+Voice, F+V = Face+Voice

* p < .05 Data from Table 5 show that not all combinations differed significantly from each other. Remarkable is that combination Posture+Face+Voice significantly differed from all other combinations. This was also true for combination Posture except for its difference with combination Face+Voice. Furthermore, the second, third and fourth placed combinations did not significantly differ from each other. For combination Posture+Face and combination Posture+Voice the significance level was p = 1.00, for combination Posture+Face and combination Face+Voice p = 0.12, and for combination Posture+Voice and combination Face+Voice p = 1.00.

5.3 Discussion

The research question central in this study was “what is the relative effectiveness of different combinations of facial expression, posture, and tone of voice to represent emotional states relevant for military tactical decision making in virtual humans?” The answer to this question is presented in Figure 8, which contains a summary of the results of the experiment. Figure 8 Relative effectiveness of combinations in percentages. The combinations circled represent the non-significant differences The resulting first and last placed combinations match the expectations, as the theory suggest that the more modalities are presented the easier it is to recognize an emotion (Argyle, 1988; Crane, 2009). Considering the second, third and fourth placed combinations, it seems that especially posture plays

Relative effectiveness of combination 1. Posture, facial expression and tone of voice (78%)

2. Posture and facial expression (69%)

3. Posture and tone of voice (64%)

4. Facial expression and tone of voice (58%)

5. Posture (51%)


of 74

an important role in the recognition of the emotional states relevant for 3D military serious games. This dominance can be explained by two reasons. First, theory suggest that posture is especially valuable for the activation and intensity of an emotion (Ekman & Friesen,1967 in Vinayagamoorthy et al., 2006). Except for the emotions neutral and scared, the emotions were quite ‘big’ meaning they contained many active movements. And second, the distance of the virtual human from the perceiver probably played a role. The face from the virtual human was recognizable but still a bit far away. Theory suggests that in these cases, posture becomes more important (Gunes et al., 2008). Yet, in the 3D military serious games virtual humans are sometimes displayed even farther away than in this experiment. In addition, pairwise comparisons showed some interesting results. It was demonstrated statistically that the most effective combination is posture, facial expression, and tone of voice. This in contrast to the second best combination, as the combinations posture and facial expression, posture and tone of voice, and facial expression and tone of voice do not differ significantly from each other. Consequently, these four most effective combinations should be considered in the second study. And accordingly, to find out if contextual influences are so high that (one of) the combinations sharing the second place are as effective as the combination face, posture and tone of voice. However, unfortunately it was practically impossible to experiment with all four combinations in the second study. Therefore, only the three most effective combinations were used: Posture+Face+Voice, Posture+Face, and Posture+Voice. The decision to take these two second best combinations was motivated by the fact that (1) they are the second and third combination in rank, (2) their mutual difference is twice as small as the difference between the second and fourth combination (Posture+Face and Posture+Voice), and (3) posture seems to be a dominant cue, and this modality is missing in the Face+Voice combination. Before moving on to this second experiment, one final issue needs some further attention; the accuracy of the material. If, for example, a certain facial expression did not display the targeted emotion correctly, it would be invalid to say that posture is more dominant than facial expression. During the preparation of the material, this possible problem was taken into account as much as possible, by checking with existing evidence based literature and by means of pilot testing. However, it was impossible to guarantee the correctness of the features of the three modalities completely. At the end of the experiment, it was more clear which of the animations was properly designed and which were possibly not good enough. Accordingly, there were some serious doubts about the facial expression of the emotion scared, which was often mistaken for with the emotion sad. Yet, this was the only feature of the material which was clearly not properly designed. For the other features, the design seemed quite good.


of 74

6 STUDY 2: RECOGNIZING EMOTIONAL STATES OF VIRTUAL HUMANS IN CONTEXT

In the second experiment, the three most effective combinations from the first experiment were accompanied by contextual information. The following subquestion was at issue: “how is the relative effectiveness of different combinations of facial expression, posture, and tone of voice to represent emotional states relevant for military tactical decision making in virtual humans influenced by contextual information?” The methodology of this experiment is described in section 6.1, after which the results are presented in section 6.2 and discussed in section 6.3.

6.1 Method

6.1.1 Participants

A total of twenty-four students from the Dutch Military Police, the KMAR, (n=24) participated in this study (all male, average age at 28 years). These students are one of the target groups for the 3D military serious games where the recognition of emotional states of virtual humans plays an important role. The experiment took place in a classroom in between a training exercise. This training exercise was not part of the standard two or three year basic education, but as a regular exercise to improve their functioning. This resulted into a group of participants having different levels of prior knowledge, but they all completed at least the basic education. These students were therefore able to understand the meaning of an emotional state in a situation and thus able to describe what their decision would be.

6.1.2 Materials

The materials used in this second study were mainly the same as the ones used in the first study. That is, the animations and the instruction form were used again with some minor adjustments. These adjustments are discussed below. In addition to these materials, context descriptions and a question and answer form was developed.

Animations Based on the results from the first experiment, a minor adjustment was made. This was related to the facial expression of the emotional state ‘scared’, of which the eyes and lips were adjusted after a second check with the distinctive features derived from literature (Fabri et al., 2002; Gunes & Piccardi, 2007).

As only the emotional states of the most effective combination (Posture+Face+Voice) and the joint second best combinations (Posture+Face and Posture+Voice) were included, participants were shown 18 animations.

Instruction form The following issues differed from the first study: • Instead of telling participants they would see 30 animations for 5 seconds each which they could

replay as often as they liked, they were told they would hear 18 context descriptions followed by a five second animation that would be repeated once. The reason that all animations were shown twice is that the experiment was performed in class so participants could not individually say how many times they needed to see the animations. In addition, seeing an animation only once would have been too limited for some people for reasons mentioned earlier.


of 74

• Instead of instructing participants to start every animation themselves, this was done by the experimenter in the classroom. This phrase was accordingly removed from the instruction form.

• Instead of verbally describing the emotional states while the experimenter would write them down, participants were asked to write down their answers themselves on a question and answer form (see description question and answer form below). The questions were introduced and participants were asked to answer them as if they were a platoon commander.

• Additional instructions concerned: (1) an explanation of the three elements of the context description (see context description below), (2) an explanation that the animated virtual human represents a person from the crowd mentioned in the context description, and (3) a warning that context descriptions and animations sometimes look similar, but that there are indeed differences so they have to be alert.

Context description Before participants watched each animation, they heard a context description from the experimenter. Six context descriptions were written; one for every emotional state (see Appendix D). Every context description contained a brief account of the (1) setting, (2) assignment, and (3) penultimate event.

These first two topics are part of a standard briefing, and also of a military gaming session. The setting contained some general background information of the type of people involved, the area where the crowd had gathered, and the cause of the gathering. The assignment contained an order to the commanders, such as “control the crowd and prevent escalation”. The third topic was added to the context description because in this experiment only short animations were used of the virtual human representing a certain emotion. Participants were thus unaware of the event or immediate cause influencing the emotional state of the virtual human. Yet, in the game, these cause and effect events are present and most likely make emotion recognition easier. Therefore, a textual explanation of the penultimate event served as a substitute for these interaction effects in the game. An example of a textual explanation is: “some crowd members have just been arrested”.

All context descriptions were checked by four operational experts. Just as in the first experiment this was done by means of pilot-testing, in which participants followed the same procedure as in the actual experiment. At the end, the participants were told which target group would take part in the experiment and they were asked for possible improvements. The familiarity of the context descriptions for the target group was confirmed, and some adjustments were made based on their comments.

Question and answer form After each animation, participants had to answer two questions which appeared on a question and answer form. The questions were: “which emotion do you see and/or hear?” and “imagine the whole crowd has this emotion; what would you decide to do if you were the commander?”

The first question was of most importance because the recognition rates resulting from this question formed the basis for answering the second subquestion. The second question on the form was more informative as it aimed to provide insight in the way emotional states function as cues in a military situation. In other words, it resembled the main goal of the targeted 3D serious games: tactical decision making.

6.1.3 Design

Just like the first experiment, the experimental design which was used is called a repeated measures design. This means that all participants in the classroom have seen and heard 18 context descriptions and animations. As one group of students participated in this experiment, counterbalancing of combinations was impossible. In order to prevent order effects as much as possible, the following measures were taken:


of 74

• Some context descriptions of different emotions appeared similar. This means, for example, that the setting was the same for more than one emotion, while the penultimate event differed. In this way, it was less likely participants would notice that a context description appeared twice (see Appendix D).

• The context descriptions and animations from the P+V combination were administered first, followed by the context descriptions and animations of the P+F combination and the P+F+V combination. This sequence is based on the consecutive recognition scores from the first experiment. The reason for this, is that if the combinations that are easier to recognize were administered first, then it would be too easy to describe the other ones following.

• Identical emotional states were positioned as far from each other as possible, without using the same sequence of emotions within the three combinations after another.

The former two measures resulted in the sequence shown in Figure 9.

Figure 9 Sequence of experiment 2. Combinations: P = Posture, F = Face and V = tone of Voice. Emotions: Ag = Aggressive, P = Panic, N = Neutral, E = Elation, An = Angry, and S = Scared

6.1.4 Procedure

Data collection The experiment took place in a classroom where animations were displayed via a beamer. After the instruction, the experimenter read aloud the first context description and showed the associated animation twice. Next, participants had about two minutes to write down their answers. This procedure was repeated 18 times. In total, the experiment lasted about 40 minutes. The instructor of the training was asked to participate as well in a slightly different way. That is, he focused only on the tactical decisions of which his answers could be used as a ‘golden standard’ for the analysis of the answers of the participants to the second question. He therefore received a modified question and answer form on which answers on the first question were already given. In this way, the instructor could focus completely on the second question without making mistakes based on the wrong assessed emotions.

Coding and scoring Participants used the question and answer form to write down their answers to the two open-ended questions. For the first question, the same coding scheme as in the first experiment was used. And just as in the first experiment, two raters checked and scored the answers. For the second question, the answers from the instructor were used to score the answers. These answers all contained more than one element. In case of the emotion ‘panic’, for example, these elements were (1) stand between parties, (2) protect crowd, and (3) charge at rock throwers. The answers participants gave had to match at least one of these elements and should not contain another wrong element. Again, two raters checked and scored the answers. Preferably, the second coder would have been another instructor, as there is seldom one correct answer (the ´golden standard´ answers cannot be seen as the only correct answers). This was unfortunately not possible, so the results should be interpreted with caution. Concerning the most desired level of fidelity, the tactical decision scores are therefore treated less decisive compared to the recognition rates.

Ag P N E An S P Ag E N S An Ag E P S An N

Combination P+V

Combination P+F

Combination P+F+V


of 74

6.1.5 Data analysis

Participants’ responses to the first question were scored analogously to the first experiment. The inter-rater reliability between the two raters was calculated and after discussion in cases of disagreement, final scores were established. Subsequently, a one-way repeated measures ANOVA was performed to analyze cross-combination differences in recognition.

The answers given by the participants to the second question were analyzed the same way, with a few additional analyses. All these additional analyses aimed to provide more insight in the way emotional states function as cues in 3D military serious games. The first additional analysis concerns the percentage correctly answered tactical decisions in cases when emotions were not correctly recognized. Secondly, the differences between the three combinations were compared using tactical decision scores in cases where the emotion was correctly recognized. A one-way repeated measures ANOVA was unfortunately not possible for this analyses because of too many missing values. Therefore, simple mean calculations were conducted. Finally, overall differences between combinations were analyzed using an interpretative approach. That is, the answer statements of the three combinations were compared with one another in order to look for patterns. These results are presented in a descriptive form.

6.2 Results

Inter-rater reliability For the emotion recognition scores the coding of both raters matched in 91% of the occasions. The inter-rater reliability according to the Cohen’s Kappa measurement was 0.85. These scores show that inter-rater agreement is considerably large, and higher than in the first experiment. For the tactical decision scores the coding of both raters matched in 87% of the occasions and according to the Cohen’s Kappa measurement the inter-rater reliability was 0.73. So, also for the tactical decision scores, the inter-rater agreement is substantial.

Recognition rates of combinations Results of the one-way repeated measures ANOVA produced a significant effect of combinations, F(2,46) = 5.53 p < 0.05. In other words, if all other variables are ignored, scores differed among the three combinations. These differences are displayed in Table 4. Table 4 Mean Recognition Rates by Combination

Combination M SD




Table 4 shows that differences between combinations have decreased compared to the differences in the first experiment. It is highly questionable if the combination Posture+Face+Voice is still significantly higher than the other two combinations. To determine whether this is the case, pairwise comparisons have been conducted.

The mean difference between the combination Posture+Face+Voice and the combination Posture+Face appeared .01 with a significance level of p = 1.00. Between the combination Posture+Face+Voice and the combination Posture+Voice this difference was .10 with a significance level of p = .02. Between the combination Posture+Face and the combination Posture+Voice this difference was .08 with a significance level of p = .08. These results indicate that, indeed, the combination Posture+Face+Voice is not significantly higher than the combination Posture+Face. Yet,


of 74

it is significantly higher than the combination Posture+Voice. Furthermore, the second (Posture+Face) and third (Posture+Voice) placed combinations do not significantly differ from each other.

Tactical decision making Results of the one-way repeated measures ANOVA showed no significant effect of combinations, F(2,46) = .65 p = 0.53. In other words, scores did not differ from each other. The scores are presented in Table 5. Table 5 Mean Tactical Decision Scores by Combination (all scores considered)

Combination M SD




The non significant differences between tactical decision scores of the three combinations are at first sight strange considering the recognition rates. One of the explanations for this result is the fact that from the wrong recognized emotions, still 44% of the decisions participants would take were correct (24 of 55). It is therefore also interesting to compare the tactical decision scores of the three combinations in cases where emotions were correctly recognized. Accordingly, mean calculations were conducted of which the results are presented in Table 6. Table 6 Mean Tactical Decision Scores by Combination (only scores based on recognized emotions considered)

Combination M

Posture+Face .69

Posture+Face+Voice .68

Posture+Voice .67

Data from Table 6 show that differences between combinations have even become smaller compared to the data from Table 5. So, although it was not possible to calculate if the differences in Table 6 are significant, it highly suggests it does not. A final interpretative analyses was conducted to determine possible patterns. The only pattern encountered is related to the extent to which participants would decide to go to a higher spectre of force. That is, in some cases, participants would take a more violent approach in the combination Posture+Face+Voice compared to the other two combinations (in 13 of 144 times) and in the combinations Posture+Face+Voice and Posture+Face compared to the combination Posture+Voice (in 12 of 144 times). An example of the latter is provided in Table 7, showing some differences in actions the students decided to take in the three combinations of the emotion ‘angry’. Table 7 Example of Differences in Decisions between the three Combinations of the Angry Emotion (translated from Dutch)

Posture+Voice Posture+Face Posture+Face+Voice

Do not take or prepare for action as long as the crowd stays quiet

Prepare for a higher level of violence because I expect a possible threat

Prepare for a higher level of violence because I expect a possible threat


of 74

6.3 Discussion

The research question central to this second study was “how is the relative effectiveness of different combinations of facial expression, posture, and tone of voice to represent emotional states relevant for military tactical decision making in virtual humans influenced by contextual information?” The recognition rates were central to answering this question and the tactical decisions were used mainly to gain insight in the way emotional states function as cues in a military situation.

Figure 10 contains a summary of the results related to the recognition rates of the different combinations. Figure 10 Relative effectiveness of combinations in percentages. The combinations circled represent the non-significant differences Comparing these results with the results from the first experiment, it can be noticed that for all three combinations recognition rates were higher. This can be due to (1) the influence of contextual information which was lacking in the first experiment, and (2) the influence of the target group that was different compared to the first experiment. As the participants of the second experiment did not receive an extra training in recognizing emotions whatsoever, it is assumed that the latter influence does not play a role. It can therefore be concluded that indeed contextual information positively influences the recognition of emotional states. The results confirm the hypothesis that differences between combinations become smaller because of the contextual information. This was demonstrated by the pairwise comparisons, of which it can be concluded that the combination posture and facial expression is as good as the combination of all three modalities together. Furthermore, the combination posture and tone of voice is placed third, which suggests that facial expression is more important than tone of voice. This dominance corresponds to theory as e.g. presented by Argyle (1988), and can be explained by the fact that the emotion ‘panic’ and even more the emotion ‘scared’ rely on facial expressions. That is, when leaving these two emotions out, the combination posture and tone of voice yielded the highest recognition rate. And when leaving out the emotion scared, the three combinations did not differ significantly any more. It should be noted that it is beyond the scope of this thesis to discuss all individual emotions, while it was already determined that the six emotions are necessary cues for military tactical decision making and it is not desirable to differentiate between combinations of modalities dependent on the emotion. However, in this case, it provided an additional insight to the reason why facial expressions seem more dominant than tone of voice. Lastly, from the results it is not clear whether posture or facial expression is more important, both seem desired. Recognizing an emotion is an important condition for initial learning and transfer. Yet the tactical decision that results yields the actual indication of initial learning and subsequently transfer. For reasons mentioned earlier, it is difficult though to draw conclusions based on the tactical decisions participants made in the situation described in the context. According to the results of the tactical decision scores, all three combinations seem as effective. From the scores that considered recognized emotions only, this result was expected. That is, a tactical decision should not differ related to different combinations of modalities once an emotional state of a virtual human is recognized. From the analyses where all scores were considered, this result

Relative effectiveness of combinations 6. Posture, facial expression and tone of voice (90%)

7. Posture and facial expression (89%)

8. Posture and tone of voice (81%)


of 74

was not expected, as it seems logical that with combinations where less emotions were recognized properly, fewer tactical decisions would be correct. An important explanation for this result is related to the methodology used in the experiment. Many participants used an incomplete description to answer the question what they would decide to do. For example, many used the description “nothing”, while probably many of them actually meant “I would not take a violent approach”. Yet, this interpretation was not clear and additional actions were also needed, like “address the crowd”. Accordingly, such answers had to be coded as incorrect. The phenomenon of incomplete answers is mainly due to the fact that participants had to write down their answers, and it takes effort to write down a complete answer especially considering the fact that the experiment took place in between a training exercise. Another explanation for the non significant differences between combinations when all scores are considered, is the fact that from the wrongly recognized emotions, almost half of the tactical decisions were still correct. For example, the emotion ‘panic’ was once described as “not happy, not violent” and the associated decision was described as “run in sympathizers and drive them away using a line”. In such cases, the description of the emotion has to be rated as incorrect, while it is also not completely wrong and consequently led to an adequate decision. In other cases, the emotional state was clearly wrong, but the tactical decision was so general that it could be correct for almost all emotional states. For example, with the emotion ‘scared’ where the emotional state was once described as “angry” and the tactical decision as “line up in front of the gate”. Although these examples provide some interesting explanations to the question why tactical decisions can be correct in cases where emotional states of virtual humans are not recognized correctly, they cannot be used for the decision which combination of modalities is most desired. This is due to the fact that, if they are taken into account, negative transfer might occur as students can make a wrong connection between cue and adequate decision. Hence, both explanations confirm the reason that the results of the tactical decision scores should be treated with caution, and are in this regard more informative than decisive compared to the recognition rates. That is, the information is still useful for instructors to keep in mind, as the latter explanation shows that the assessment should not merely rely on the tactical decisions students make, but instructors also need to ask students which cues they used to get to that decision. Results of the interpretative analysis are also related to the implementation of emotional states of virtual humans in 3D military serious games. This analysis showed that in 17% of the cases, participants would decide to go to a higher spectre of force if the intensity of an emotion becomes stronger. In the example related to the emotion ‘angry’, a slightly different emphasis was noticeable in the descriptions of the emotion in the three combinations. That is, from “demonstrative, waiting” (Posture+Voice) to “angry, expectantly” (Posture+Face) to “angry, ignorance” (Posture+Face+Voice). These descriptions are all correct, but differ in intensity and consequently give rise to different decisions. For instructors, it is good to be aware of this phenomenon, and to point out that the focus is on tactical decision making instead of discussing intensity differences of emotional sates. There is not just one expression of the emotion ‘angry’, especially if one considers differences between cultures. In real life estimating someone’s emotional states is also highly subjective. It is, therefore, necessary to focus on the tactical decision in cases where the crowd is ‘angry’, no matter how intense this emotion is represented. Note that when a crowd is becoming really ‘angry’, the emotion is not called ‘angry’ any more but ‘aggressive’. And when this is the case, another decision is probably more appropriate. One final note is related to the methodology used in the experiment. It was found that the inter-rater reliability score of the recognition rates in the second experiment is reasonably higher than the inter-rater reliability in the first experiment. This is probably due to the fact that participants used the intended words for the emotions more often than in the first experiment, as the emotions occurred in context and another target group is used. This made coding less subjective and therefore easier. In the first experiment participants sometimes couldn’t find the right words to describe the emotion and used a description of a context where such an emotion could occur instead. In cases like this, the correctness of an answer was not always obvious which made coding more subjective. So, especially


of 74

in emotion recognition studies when context is included, the free response format seems to be a good choice in comparison to the more usual forced-choice format. For the tactical decision scores, the inter-rater reliability score was also considerably large. Yet, there were some drawbacks considering incomplete answer descriptions. A forced-choice format might therefore be more appropriate in experiments where participants have to write down their answers. In experiments where participants can verbally respond to the question, the free response format might be preferable as the experimenter can then ask for clarification.


of 74

7 GENERAL DISCUSSION

The main aim of this study was to establish fidelity guidelines for 3D military serious games with a tactical decision making objective. Three types of activities were conducted to obtain these fidelity guidelines: 1) a literature review, 2) a task analysis, and 3) two experimental studies. From the first two activities, the more general guidelines were established. Together they form a framework and a direction for finding an optimal level of fidelity given the nature of the task to be trained. To obtain more concrete guidelines, two experimental studies were conducted considering one element of 3D military serious games: the physical fidelity of emotional states of virtual humans. As the latter activity forms the main part of this Master project, the general discussion starts with these concrete guidelines in section 7.1. Next, the more general fidelity guidelines are discussed in section 7.2, followed by the recommendations for future research in section 7.3.

7.1 Desired level of physical fidelity of emotional states of virtual humans

The central research question of the two experimental studies was: “what is the desired level of physical fidelity of emotional states of virtual humans within the 3D world of a military serious game”? The cue dominance approach was used to answer this question, in which the interaction between the three modalities facial expression, posture and tone of voice was investigated. Accordingly, the desired level of physical fidelity is expressed in the number of cues (only one modality, two modalities or all three modalities) and the type of modality (facial expression, posture, and/or tone of voice). The following emotional states were said to be important for military tactical decision making and were therefore subject to experimentation: angry, aggressive, scared, panic, elation and neutral. Below, the results of the two studies are shortly addressed, followed by the answer to the main research question. Subsequently, the practical and theoretical implications of this answer are discussed.

First study The first study aimed at answering the following question: “what is the relative effectiveness of different combinations of facial expression, posture, and tone of voice to represent emotional states relevant for military tactical decision making in virtual humans”? Accordingly, different combinations of the three modalities were compared using recognition rates. The combination posture, facial expression and tone of voice was found to be most effective. In addition, interesting was that with respect to the second most effective combination, no significant differences were found amongst all three combinations containing two modalities.

Second study The second study elaborated on the results of the first study by answering the following question: “how is this effectiveness influenced by contextual information?” Accordingly, the added value of contextual information was investigated using recognition rates as well as tactical decision scores. It was hypothesized that context positively influences the recognition rate and that differences between combinations become smaller. Both hypotheses were confirmed, and consequently the combination posture, facial expression, and tone of voice was significantly no longer most effective. That is, the combination posture and facial expression was as good as the combination of all three modalities together. An important conclusion, as it proves that a lower level of physical fidelity is as good as the highest level of physical fidelity. Unexpectedly, the tactical decision scores showed no significant differences between combinations. Several explanations were provided, of which it was concluded that those scores are poor indicators for the most desired level of fidelity. Nevertheless, they provided some useful information for the implementation of emotional states of virtual humans in 3D military serious games. These implementation guidelines are discussed in the section ‘practical implications’.


of 74

Answer to the main research question The most desired level of physical fidelity of emotional states of virtual humans concerns transfer as well as costs. Based on the results of the two studies, the most straightforward answer to the main research question is the combination posture and facial expression. That is, when possible motivation influences are ignored, the effectiveness and consequently the expected transfer of the combination posture and facial expression is as high as the combination of all three modalities together, while it is cheaper because tone of voice is not included. Unfortunately, this is not exactly the answer TNO and the RNLA hoped for, as modelling facial or postural expressions are much more expensive than adding sounds. Leaving tone of voice out actually causes only a marginal effect on the total development costs. It is, therefore, questionable if this small cost benefit weights upon the possible benefits of enhanced immersion, which can cause a higher level of motivation and subsequently transfer. This latter hypothesis was not tested, but because of the low costs, it seems wiser to be sure that users stay motivated and to include tone of voice to 3D military serious games, than to leave tone or voice out and save only a small amount of money. Yet, if budget is really scant it is still appropriate to design emotional states of virtual humans with only facial and postural expressions. When there is a bit more room in the budget then it is advisable to add tone of voice too.

Practical implications This research study does not only provide an answer to the question which combination of modalities is most desired, but it also provides guidelines regarding the design of each modality and regarding the implementation of these design guidelines in 3D military serious games. These two aspects are discussed first, followed by a discussion of the external validity of these guidelines. Within the face, it is advisable that only the five most distinctive features are designed. These are the eyebrows, the upper/lower eyelids, the eyes, the mouth corners and the lips. For the postural expression, prototypical postures can be used, like the ones displayed in Appendix C. When tone of voice is added, sound samples similar to the ones used in the experiment can be applied. Concerning the implementation of the design guidelines, it was suggested that instructors need to be aware of two aspects: (1) the assessment should not merely rely on the tactical decisions students make, but also on the cues they used to make these decisions, and (2) the intensity of a perceived emotion influences the extent to which students decide to go to a higher spectre of force. For the former aspect, it is advised that instructors explicitly ask students in the debriefing phase of a gaming session which information they used for their decision. There are some issues regarding the external validity of these guidelines that merit further attention. First, it should be considered that in the two studies only one prototypical posture per emotion was experimented. It is not clear whether this is sufficient for crowds in 3D military serious games, as it would be a bit strange to see all humans in a crowd acting exactly the same way. The emotional states will surely be recognized, but it can influence student motivation negatively and result in decreased initial learning and subsequently lower transfer. Secondly, in the two studies not all influencing factors of 3D military serious games are taken into account. Context was assumed to be the most important one. For that purpose, this aspect was experimented. However, an aspect which has not been included in the experiments is the movement of virtual humans through the scene. This can make recognition easier, for instance, in case of running in combination with the emotion ‘panic’. For the other emotions though, this influence is assumed to be rather small. Another aspect which might influence the recognition of emotional states is interaction. In the second experiment, a textual explanation of the penultimate event served as a substitute for interaction effects in the game. This substitute is nevertheless not exactly the same, as in the game the penultimate event is more of an experience, which is represented visually and perhaps auditory. These lively ‘cause and effect’ events most likely make emotion recognition easier compared to a textual explanation. Finally, the distance of the virtual humans in the actual game should be considered. In the two studies, the distance of the user to the animated virtual human stayed more or less similar. That is, a ‘golden mean’ was chosen where the facial expression was just noticeable, as it was assumed that in 3D military serious games


of 74

virtual humans sometimes stand close by and sometimes far away. This is a reasonable assumption, but it is still worth the money to investigate whether virtual humans really come that close that facial expressions are noticeable. Accordingly, the exact influence of all the issues mentioned with regard to the external validity of the fidelity guidelines remains unknown, and therefore form an input for the recommendations for future research.

Theoretical implications The two studies revealed some useful lessons learned concerning the experimentation of fidelity. Accordingly, the cue dominance approach appeared very useful as a framework to unfold the cues of which the elements of 3D military serious games consist. In addition, it offered clear labels for what is meant by low, medium, and high fidelity by using the number of modalities as a measure for the level of physical fidelity. In this way, the subjectivity of choosing different fidelity levels for comparison is reduced, which makes the decision of what exactly to compare easier and makes generalizations from multiple fidelity studies possible. An additional lesson learned concerns the feasibility of fidelity experiments. It became clear that fidelity influences so many factors that it is impossible to vary each variable. It is, therefore, necessary to make clear decisions about the variables that are subject for comparison. In this thesis, different combinations of modalities were compared while the features of every modality were fixed. These choices are necessary to keep the experiments feasible. Starting points for making such choices are to determine what is already known in the field from other studies and to consider the cost perspective.

7.2 General fidelity guidelines of 3D military serious games

The conclusions related to the research question address only a (small) part of the fidelity issue of 3D military serious games. Considering the work that is done to make such concrete judgements, great effort is needed to do so for all elements. However, based on the literature review and task analysis, it is still possible to provide an indication of the general fidelity guidelines of 3D military serious games. One of these guidelines is that functional fidelity should be high. In other words, the feedback resulting from actions should be designed realistically. This means, for instance, that when being shot at, you actually get wounded and as such have reduced mobility, or you die and subsequently cannot participate anymore. It also means that when as CRC team you act aggressively towards a crowd, this crowd will react, for example, by turning hostile towards you. The functional reactions need to be realistic, but on the other hand, not all visualisations of elements in the VE needs to be full fidelity. In other words, physical fidelity may allow lower levels. Physical fidelity influences the hardware and software of the 3D military serious game. It was concluded that hardware devices like a simple desktop computer with a mouse, keyboard and/or joystick are sufficient. The RNLA already use these devices, and this study confirms this type of usage. Considering software, the cues in the VE that are dependent on the task play an important role. For military tactical decision making tasks, these are related to environmental aspects (objects) and the behaviour of civilians and OPFOR (virtual humans). It was suggested that these cues should be designed in a high fidelity matter, while other aspects can be designed in a lower fidelity matter. High fidelity of these cues means that their semantic should be clear and recognizable, and they should be believable to the extent they are accepted by users. For the behaviour of civilians, for example, this means that emotional states should be designed in a way they are recognizable and believable, while elements related to their appearance (e.g., wrinkles, folded clothes, etc.) do not, although such deviations from reality still need to be accepted. Accordingly, the former element is part of the necessary cues, while the latter is not.

Yet, except for emotional states of virtual humans it is still not clear how high or low fidelity every element should be designed. The experiment showed that empirical evidence is needed to establish what exactly low or high fidelity means in terms of concrete design guidelines. For every element this will be different. When exactly is a cue recognizable and believable? And how low fidelity can elements that do not belong to the necessary cues be designed, realizing that users still need to accept them? These concrete questions remain unanswered for the main part and consequently will


of 74

form an input for the recommendations for future research in the next section. Still, the literature review and task analysis provide at least a direction to which elements in the 3D military serious games need more attention regarding a realistic design.

7.3 Recommendations for future research

Great effort is needed to provide concrete fidelity guidelines for all elements of 3D military serious games. As the optimal level of fidelity differs for each specific task, it is close to a ‘mission impossible’ to have the definite solution covering the whole spectrum of tasks and cues. Yet, similar to the physical fidelity of emotional states of virtual humans, there are elements that are applicable to more than one specific task of 3D military serious games. In this section, two recommendations for future research are made that are also applicable for more than one task. In addition, a final recommendation pursues the issues regarding the external validity of the fidelity guidelines related to emotional states of virtual humans in greater depth. The RNLA currently is especially interested in using 3D military serious games for mission rehearsal training. For instance, they used the city of Tarin Kowt (Afghanistan) in VBS2 to train their soldiers before they were deployed. As described in the task analysis, the added benefits of such training are orientation of the terrain, learning the specific threats related to that mission, and learning relevant situational factors. The fidelity issue is of major importance in these types of training, as the risk of negative transfer is high. Especially the environmental elements are important in this respect. Consider, for example, the case when not all alleys are modelled correctly and the escape route a commander has in mind does not exist in real life. So, is it worth the money to model a city in complete detail, including trees, textures on houses, etc. with the risk that soldiers completely ‘believe’ the VE with possible negative transfer as a result? That is, the environment is always changing, so it is impossible to get a complete match. Or, following the recommendation of Tack and Colbert (2005), is it perhaps wiser to only match the street plan and detail mission essential objects, with the risk of lower transfer concerning other tasks than navigation? Accordingly, an interesting and critical dilemma, concerning development costs as well as effectiveness of training. The second recommendation for future research is related to the motivation of users in combination with the role of the instructor. Low fidelity objects and virtual humans still need to be believable to the extent they are accepted by users. It can be hypothesized that the instructor can play an important role in this acceptance by provoking a training mindset. For instance, at the start of a gaming session the instructor can present an example in which a decision by a commander in a certain situation resulted in an extremely dangerous situation. The instructor can thereby underline the importance of the training for tactical decision making and stress that a 3D military serious game might not be exactly the same as in real life but can still lead to useful learning experiences. This way students might recognize the relevance of the training and their acceptance of the 3D military serious game might become higher. This hypothesis can be applied for experimentation, and the exact role of the instructor described as well. Finally, it is useful to conduct several experiments with the emotional states of virtual humans in an actual 3D military serious game. Based on the issues mentioned regarding the external validity of the fidelity guidelines, the experiments should focus on: (1) measuring user acceptance in case of one posture per emotion compared to more than one posture per emotion for all virtual humans in the crowd, (2) measuring and comparing recognition rates and tactical decision scores of the three most effective combinations again, but now after playing the 3D military serious game with a crowd of virtual humans, and (3) checking whether facial expressions of virtual humans are noticeable while students are playing several scenarios in the 3D military serious game.


of 74

REFERENCES

AGARD (Advisory Group for Aerospace Research & Development) (1980). Fidelity of Simulation for Pilot Training. Advisory Report, AGARD-AR-159.

Alexander, A. L., Brunyé, T., Sidman, J. & Weil, S. A. (2005). From gaming to training: A review of studies on fidelity, immersion, presence, and buy-in and their effects on transfer in PC-based simulations and games. Woburn, MA: Aptima, Inc.

Alessi, S. (1995). Dynamic versus static fidelity in a procedural simulation. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.

Alessi, S. (2000). Simulation design for training and assessment. In H.F. O’Neil & D.H. Andrews (Eds.), Aircraft training and assessment (pp. 197-222). Mahwah, NJ: Erlbaum.

Alessi, S.M. & Trollip, S.R. (2001). Multimedia for learning: Methods and development. Boston: Allyn and Bacon.

Allbeck, J.M. & Badler, N.I. (2002). Embodied autonomous agents. In K.M. Stanney (Ed.) Handbook of virtual environments: Design, implementation, and applications (pp. 313-332). Mahwah, NJ: Lawrence Erlbaum Associates.

Andrews, D. H., Carroll, L.A. & Bell, H.H. (1996). The future of selected fidelity in training devices. Educational Technology, nov/dec, 32-36.

Argyle, M. (1988). Bodily communication. New York: Methuen & Co. Ltd. Atkinson, A. P., Dittrich, W. H., Gemmell, A. J., & Young, A. W. (2004). Emotion perception from

dynamic and static body expressions in point-light and full-light displays. Perception, 33(6), 717-746.

Bartneck, C. (2001) Affective Expressions of Machines. Proceedings of 2001 Conference on Human Factors in Computing Systems, Seattle, USA.

Belanich, J., Sibley, D.E. & Orvis, K.L. (2004). Instructional characteristics and motivational features of a PC-based game (Research Report 1822). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Benford, S.D., Bowers, J., Fahlén, L.E., Greenhalgh, C.M., Snowdon, D. (1995). User Embodiment in Collaborative Virtual Environments. Proceedings of 1995 ACM Conference on Human Factors in Computing Systems, Denver/Colorado, ACM Press.

Cannon-Bowers, J.A. & Bell, H.H. (1997). Training decision makers for complex environments: Implications of the naturalistic decision making perspective. In C.E. Zsambok and G. Klein (Eds.), Naturalistic decision making (pp. 99-110). Mahwah, NJ: Lawrence Erlbaum.

Carroll, J.M. & Russell, J.A. (1996). Do facial expressions signal specific emotions? Judging emotion from the face in context. Journal of Personality and Social Psychology, 70(2), 205-218.

Clavel, C., Plessier, J., Martin, J.C., Ach, L. & Morel, B. (2009). Combining facial and postural expressions of emotions in a virtual character. Proceedings of the 9th International Conference on Intelligent Virtual Agents (IVA'2009), Amsterdam, The Netherlands.

Cormier, S.M. & Hagman, J.D. (1987). Transfer of learning: Contemporary research and applications. San Diego: Academic Press Inc.

Coulson, (2004). Attributing emotion to static body postures: recognition accuracy, confusions, and viewpoint dependence. Journal of Nonverbal Behavior, 28(2), 117-139.

Crane, E.A. (2009). Measures of emotion: How feelings are expressed in the body and face during walking. Doctoral thesis: The University of Michigan.

Delgarno, B., Hedberg, J. & Harper, B. (2002). The contribution of 3D environments to conceptual understanding. Paper presented at the Australian Society for Computers in Learning in Tertiary Education – ASCILITE2002, Auckland, New-Zealand.

Donath, J. (2001). Mediated Faces. In M. Beynon, C.L. Nehaniv, K. Dautenhahn (Eds.) Cognitive Technology: Instruments of Mind. Cognitive Technology Conference Proceedings, Warwick, UK.


of 74

Endsley, M.R. (1988). Design and evaluation for situation awareness enhancement. Proceedings Of the Human Factors Society 32nd Annual Meeting (pp. 97-101). Santa Monica, CA: Human Factors Society.

Endsley, M.R. (1997). The role of situation awareness in naturalistic decision making. In C.E. Zsambok and G. Klein (Eds.), Naturalistic decision making (pp. 269-283). Mahwah, NJ: Lawrence Erlbaum.

Fabri, M., Moore, D.J., Hobbs, D.J. (2002). Face Value: Towards Emotionally Expressive Avatars. Proceeding of AISB Convention on Animated Expressive Characters for Social Interaction, London, UK.

Farmer, E. van Rooij, J., Riemersma, J., Jorna, P. & Moraal, J. (1999). Handbook of simulator-based training. Cambridge: University Press.

Feinstein, A.H. & Cannon, H.M. (2002). Constructs of simulation evaluation. Simulation & Gaming, 33(4), 425-440.

Frini, A., Stemate, L., Larochelle, S., Toussaint, G. And Lecocq, R. (2008). Crowd confrontation and non-lethal weapons: A literature review and conceptual model (Technical Memorandum nr. 2007-488). Valcartier: Defence R&D Canada.

Garris, R., Ahlers, R. & Driskell, J.E. (2002). Games, motivation, and learning: A research and practice model. Simulation & Gaming, 33(4), 441-467.

Gick, M.J. & Holyoak, K.J. (1987). The cognitive basis of knowledge transfer. In S.M. Cormier & J.D. Hagman (Eds.) Transfer of learning: Contemporary research and applications (pp. 9-46). London: Academic Press, Inc.

Gratch, J. & Marsella, S. (2001). Tears and fears: modeling emotions and emotional behaviors in synthetic agents. Paper presented at the Fifth International Conference on Autonomous Agents, Montreal, Canada.

Gunes, H. & Piccardi, M. (2007). Bi-modal emotion recognition from expressive face and body gestures. Journal of Network and Computer Applications 30(4) 1334–1345.

Gunes, H., Piccardi, M. & Pantic, M. (2008). From the Lab to the real world: affect recognition using multiple cues and modalities. In J.Or (Ed.) Affective computing: focus on emotion expression, synthesis, and recognition (pp. 185-218). Vienna, Austria: InTech Education and Publishing.

‘t Hart, M.H.E., Vink, N. & Buiël, E.F.T. (2008). An introduction to Urban Operations. The Hague: TNO Defence, Security and Safety.

Hartog, C. (2009). Scenario design for serious gaming: Guiding principles for the design of scenarios and curricula in military Job Oriented Training. Master thesis, Delft University of Technology.

Hays, R.T. (1980). Simulator fidelity: A concept paper (Technical Report 490). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Hays, R. T. & Singer, M. J. (1989). Simulation fidelity in training system design. New York: Springer-Verlag.

Hays, R.T. (2005). The effectiveness of instructional games: literature review (Technical Report 004). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

van der Hulst, A., de Hoog, R., & Wielemaker, J. (1999). BOOT: decision support for the selection of facilities for education and training. The Hague: FEL-99-A188.

van der Hulst, A., Muller, T., Besselink, S., Coetsier, D. & Roos, C. (2008). Bloody Serious Gaming – Experiences with Job Oriented Training. Proceedings of the 2007 Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC), Orlando.

Kenney, P., Hartholt, A., Gratch, J., Swartout, W., Traum, D., Marsella, S. & Piepol, D. (2007). Building interactive virtual humans for training environments. Proceedings of the 2007 Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC), Orlando.

Kessler, G.D. (2002). Virtual environment models. In K. M. Stanney (Ed.), Handbook of virtual environments: Design, implementation, and applications, (pp. 255-276). Mahwah, NJ: Lawrence Erlbaum Associates.

Klein, G. (1997). An overview of naturalistic decision making applications. In C.E. Zsambok and G. Klein (Eds.), Naturalistic decision making (pp. 49-59). Mahwah, NJ: Lawrence Erlbaum.


of 74

Kleinsmith, De Silva & Bianchi-Berthouze, (2006). Cross-Cultural Differences in Recognizing Affect from Body Posture. Interacting with Computers, 18(6), 1371-1389.

Knerr, B.W., Lampton, D.R., Singer, M.J., Witmer, B.G. & Goldberg, S.L. (1998). Virtual environments for dismounted soldier training and performance: Results, recommendations, and issues (Technical Report 1089). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Knerr, B.W. (2006). Current issues in the use of virtual simulations for dismounted soldier training. Paper presented at the NATO Human Factors & Medicine Panel Workshop on "Virtual Media for Military Applications", West Point, NY.

Knerr, B.W. (2007). Immersive simulation training for the dismounted soldier (Study Report 2007-01). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Korteling, J.E., van den Bosch, K. & Emmerik, M.L. (1997). Low-cost simulators 1a: Literature review, analysis of military training, and selection of task domains (TNO-report TM-97-A035). Soesterberg: TNO Human Factors Research Institute.

Lampton, D. R., Bliss, J. P., & Morris, C. S. (2002). Human performance measurement in virtual environments. In K. M. Stanney (Ed.), Handbook of virtual environments: Design, implementation, and applications, (pp. 701-720). Mahwah, NJ: Lawrence Erlbaum Associates.

Lane, N.E. & Alluisi, E.A. (2002). Fidelity and validity in distributed interactive simulation: questions and answers (Report No. IDA-D-1066). Alexandria, VA: Institute for Defense Analysis.

Lathan, C.E., Tracey, M.R., Sebrechts, M.M., Clawson, D.M. & Higgins, G.A. (2002). Using virtual environments as training simulators: Measuring transfer. In K.M. Stanney (Ed.) Handbook of virtual environments: Design, implementation, and applications (pp. 403-414). Mahwah, NJ: Lawrence Erlbaum Associates.

Leemkuil, H., de Jong, T. & Ootes, S. (2000). Review of educational use of games and simulations. Enschede: University of Twente.

van Lent, M., McAlinden, R., Probst, P., Silverman, B., O’Brien, K. & Cornwell, J. (2004). Enhancing the behavioral fidelity of synthetic entities with human behavioral models. Paper presented at the 13th Conference on Behavior Representation in Modeling and Simulation (BRIMS), SISO, May 2004.

Liu, D. & Vincenzi, D.A. (2004). Measuring simulation fidelity: A conceptual study. In D.A. Vincenzi, M. Mouloua and P.A. Hancock (Eds.) Human performance, situation awareness and automation: Current research and trends, Vol. I (p. 160-165). London: Academic Press, Inc.

Loftin, R.B., Scerbo, M.W., McKenzie, R., Catanzaro, J.M., Bailey, N.R., Phillips, M.A. & Perry, G. (2003). Training in peacekeeping operations using virtual environments. Paper presented at the RTO HFM Symposium on “Advanced Technologies for Military Training”, 2003.

Marine Corps Institute (n.d.). Tactical decision making. Washington, DC: Marine Barracks. Mania, K., Troscianko, T., Hawkes, R. & Chalmers, A. (2003). Fidelity Metrics for Virtual Environment

Simulations based on Human Judgments of Spatial Memory Awareness States. Presence, Teleoperators and Virtual Environments, 12(3), 296-310, MIT Press.

Mania, K., Wooldridge, D., Coxon, M. & Robinson, A. (2006). The effect of visual and interaction fidelity on spatial cognition in immersive virtual environments. IEEE Transactions on Visualization and Computer Graphics, 12(3), 396-404.

McDonnell, R., Jörg, S., McHugh, J., Newell, F., and O'Sullivan, C. (2008). Evaluating the emotional content of human motions on real and virtual characters. Proceedings of the 5th Symposium on Applied Perception in Graphics and Visualization, APGV '08, ACM, New York, 67-74.

Van Merriënboer, J.J.G. (1997). Training complex cognitive skills: A four component instructional design model for technical training. Englewood Cliffs, NJ: Educational Technology Publications.

Minke, A. (1997). Conducting repeated measures analyses: Experimental design considerations. Paper presented at the annual meeting of the "Southwest Educational Research Association", Austin.


of 74

Moya, L.J., McKenzie, F.D. & Nguyen, Q.H. (2008). Visualization and rule validation in human-behavior representation. Simulation & Gaming, 39(1), 101-117.

Muchinsky, P. M. (2003). Psychology applied to work (7th edition). Belmont, CA: Wadsworth/Thomson Learning.

Noble, C. (2002). The relationship between fidelity and learning in aviation training and assessment. Journal of Air Transportation, 7(30), 33-54.

Pace, S. (2008). Immersion, flow and the experience of game players. Proceedings of the SimTect2008 Conference, Melbourne.

Pasch, M. & Poppe, R. (2007). Person of puppet? The role of stimulus realism in attributing emotion to static body postures. Proceedings of the Conference on Affective Computing and Intelligent Interaction (ACII), vol. 4738, 83–94.

Petraglia, J. (1998). The real world on a short leash: the (mis)application of constructivism to the design of educational technology. Educational Technology Research and Development, 46(3), 53-65.

Petty, M.D., Gaskins, R.C. & McKenzie, F.D. (2003). Crowd modelling in military simulations: Requirements, analysis, survey, and design study. Norfolk, Virginia: Old Dominion University, Virginia Modeling, Analysis, and Simulation Center.

Pleban, R.J., Eakin, D.E., Salter, M.S. & Matthews, M.D. (2001). Training and assessment of decision-making skills in virtual environments (Research Report 1767). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Pleban, R.J., & Salvetti, J. (2003). Using virtual environments for conducting small unit dismounted mission rehearsals (Research Report 1806). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Pongracid, H., Marlow, D. & Triggs, T. (1997). Issues in cost-effectiveness and fidelity of simulation. Proceedings of the Second International SimTecT Conference, Canberra, Australia, 221-226.

Rankin, J.R. & Vargas, S.S. (2008). A review of serious games and other game categories for education. Proceedings of the SimTect2008 Conference, Melbourne.

Raybourn, E.M. (2007). Applying simulation experience design methods to creating serious game-based adaptive training systems. Interacting with Computers, 19, 206-214.

Rouse, W.B. & Hunt, R.M. (1984). Human problem solving in fault diagnosis tasks. In W.B. Rouse (Ed.), Advances in man-machine systems research (Vol. 1, pp. 195-222). Greenwich, CT: JAI Press.

Roza, Z.C. (2005). Simulation fidelity theory and practice: a unified approach to defining, specifying and measuring the realism of simulations. Doctoral thesis: Delft University of Technology, Faculty of Aerospace Engineering, Delft.

Russell, J.A., Bachorowski, J. & Fernández-Dols, J. (2003). Facial and vocal expressions of emotion. Annual Review of Psychology, 54, 329-349.

Russell, J. A. (1994). Is there universal recognition of emotion from facial expression? A review of the cross-cultural studies. Psychological Bulletin, 115(1), 102-141.

Sanchez, A. & Smith, P.A. (2007). Emerging technologies for military game-based training. Proceedings of the 2007 Spring Simulation Vol. 2, 296-301.

Schank R.C., Berman, T.R. & Macpherson, K.A. (1999). Learning by doing. In C.M. Reigeluth (Eds.), Instructional-design theories and models: a new paradigm of instructional theory (2nd edition) (pp. 161-181). Mahwah, NJ: Lawrence Erlbaum Associates.

Scherer, K.R. (2003). Vocal communication of emotion: A review of research paradigms. Speech communication, 40, 227-256.

Shaarani, A.S. & Romano, D.M. (2008). The intensity of perceived emotions in 3D virtual humans. Proceedings of the 7th international Joint Conference on Autonomous Agents and Multiagent Systems-Volume 3, Richland, USA, 1261-1264.

Singley, M.K. & Anderson, J.R. (1989). The transfer of cognitive skill. Cambridge, MA: Harvard University Press.


of 74

Stanney, K.M. (2002). Handbook of virtual environments: Design, implementation, and applications. Mahwah, NJ: Lawrence Erlbaum Associates.

Susi, T., Johannesson, M. & Backlund, P. (2007). Serious games – an overview. Technical report: HIS-IKI-TR-07-001, University of Skövde, Sweden.

Tack, D.W. & Colbert, H.J. (2005). Alternative visualization methods for high density urban operations. Toronto: Defence Research and Development Canada.

Tarr, R.W., Morris, C.S., & Singer, M.J. (2002). Low-cost PC gaming and simulation: Doctrinal survey (Technical Report 0622785). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Vinayagamoorthy, V., Brogni, A., Steed, A. & Slater, M. (2006). The role of posture in the communication of affect in an immersive virtual environment. Proceedings VRCIA ’06:ACM international conference on Virtual reality continuum and its applications, New York, 229–236

Visschedijk, G.C. (2009). Fidelity guidelines of 3D serious games for tactical decision making: Literature review. Unpublished manuscript, University of Twente, Enschede.

van Vliet, A.J. & Fennis-Bregman, M. (2004). Onderzoek naar crowd & riot control: Een overzicht (TNO-report TM-04-B002). Soesterberg: TNO Human Factors Research Institute.

WordNet 3.0. (n.d.). “Decision making”. Retrieved October 16, 2009, from Dictionary.com: http://dictionary.reference.com/browse/decision making

Zyda, M. (2005). From visual simulation to virtual reality to games. IEEE Computer Society, 38(9), 25-32.

http://dictionary.reference.com/browse/decision making


of 74

APPENDIX A: CUES FOR INFANTRY COMMANDERS

This appendix contains an overview of the cues necessary for infantry tactics in a 3D serious game like VBS2. There are three main types of cues distinguished in the virtual environment: environment-related cues, civilian behaviour-related cues, and OPFOR-related cues. Within these types, cues can either be static and dynamic. In addition, cues can contain two different kinds of meanings: cues that give information about the (un)safety of a situation (blue coloured cues) and cues that give information about the possibilities to effectively perform a mission (purple coloured cues). It should be noted that the focus of these cues is on urban operations.

Environment-related cues

Landmarks

Street signs

Channeling terrain

Open terrain

Slowing terrain

Narrow terrain

High terrain (possible over watch position BLUFOR)

Static

Obstacles on the road which are normally not there

Results of vandalism

Rubbish

Street pattern/city lay-out

Navigation

Characteristics roads/places

Artificial objects


of 74

All kinds of building shapes (possible cover and concealment position BLUFOR)

All kinds of building shapes (possible cover and concealment position OPFOR)

Doors and windows

Artificial objects in buildings (e.g., dust, garbage, etc.)

Strong/thick material (e.g., concrete walls of qala)

Weak/thin material (e.g., wooden gates)

Building lay-out

Building size

Building shape

Building objects

Building material

Sun

Rainfall/snow

Daylight

Night

Rainfall

Wind direction

Weather conditions

Audio

Visual

High buildings (possible over watch position BLUFOR)

Low buildings

High buildings (possible over watch position OPFOR)


of 74

Civilian behaviour-related cues

Static

Characte-ristics

Dynamic

Absence of people where it is normally quite busy

Usual amount of civilians

Much more people than normal

Woman, children and elderly are also present

Only men around, when normally there are also woman, children and elderly present

More armed (police)men than usual

Usual amount of armed men, civilians don’t react on them

Audio

Civilians have no collective goal (crowd or not)

Gathering of civilians with a collective goal (crowd)

Usual mumbling/ talking voices

Unusual silence

More loud than normal

Panic screaming

Voices from a certain point

Voices from everywhere

Orientation

Voices

Cohesion

Composition

Amount

Attributes


of 74

Movement

From a certain point

To a certain point

No specific orientation point (anywhere)

Non-simultaneous movement

Simultaneous movement

Running

Nervous walking, faster than normal

Usual/casual walking

Standing still

Non-verbal

Speed

Orientation

Synchronicity

Neutral

Nervous/scared

Angry

Panic

Elation

Facial expression

Neutral

Nervous/scared

Angry

Panic

Elation

Posture


of 74

OPFOR-related cues

Dynamic

Not stopping after BLUFOR warning (e.g., car keeps on driving with high speed, or a man keeps on walking)

Sound of a car coming towards BLUFOR with high speed

Static

Characte-ristics

Amount Only 1

More than 1

Attributes/artificial objects IED building materials (i.e. inner tubes, parts of an alarm

clock, etc.)

Guns or other recognizable weapons

From a certain point

From different directions

Audio Movement

Orientation

Gunshots

Artillery fire

Explosion (e.g., grenade, IED, etc.)

Aimed guns

Throwing grenades or Molotov cocktails

Movement

Weapon-related

Orientation From a certain point

From different directions

Weapon-related

Visual


of 74

APPENDIX B: CUES FOR CRC COMMANDERS

This appendix contains an overview of the cues necessary for CRC tactics in a 3D serious game like VBS2. These cues are organized according to their static versus dynamic character. The blue coloured cues are all related to the same kind of meaning: they are related to the (un)willingness to show aggression.

Statisch

Characte-ristics

Environmental factors

Causes

Composition

Cohesion

Amount

Attributes

Woman, children and elderly are also present

Especially men between 15 and 25 year old

Precense of weapons

Clothing indicating a willingness to aggression

Weather conditions

Street scene

Towards another group not present

Sharing an emotion

Rainfall (bad weather)

Confined spaces (alleys, closed rooms, etc.)

E.g.: Hooligans against one another

E.g.: Demonstrations

E.g.: Winning an election

Static

Open spaces (squares, parks etc.)

Use of flags and similar clothing

No visible cohesion

Sunny (good weather)

Agitation E.g.: Excitement at an event

< 100 people (specific amount not sure)

100-1000 people (specific amount not sure)

> 1000 people (specific amount not sure)

Towards another group present


of 74

Dynamic

Movement

Vandalism

Fleeing wherever

To a certain point (geographically)

Towards the Military police

Towards another group

Non-verbal expression

Orientation

Panic

Neutral

Angry

Aggressive

Scared

Elation

Facial expression

Running

Casual/walking

Sitting or standing still

Resisting/ inhibited in movement

Way of movement

Surrender

Normal

Angry

Provocative

Elation

Aggressive

Posture


of 74

Happy music (like band music)

Threatening music (like marching music)

Sounds of vandalism

Gunshots and other sounds of weapons

Firework

Normal

Use of slogans

Panic screams

Aggressive screams

Cheers

Silence/murmur

Choruses

Audio

Voices

Surrounding sound

Organization Movement

Collective movement with the presence of a leader

Unorganized movement

Collective movement with the presence of a splinter group

Splinter group is active, main group inactive

Throwing stones and other material from the environment

Use of (brought) weapons

Use of fireworks

Hand-to-hand fight

Barriers, fires, etc.

Use of weapons


of 74

APPENDIX C: FACIAL AND POSTURAL EXPRESSIONS

This appendix contains the facial and postural expressions from the six emotional states used in the experiments. Note that in the experiments short animations were used instead of these still images. Furthermore, the adjusted facial expression of the emotion scared is displayed.

Aggressive

Panic


of 74

Neutral

Elation


of 74

Angry

Scared


of 74

APPENDIX D: CONTEXT DESCRIPTIONS

This appendix contains the six context descriptions (one for every emotion) used in experiment 2. They were written in Dutch.

Aggressive Setting U bevindt zich in een Nederlands voetbalstadion waar zojuist een wedstrijd is begonnen tussen twee aan elkaar gewaagde voetbalclubs. Bij eerdere wedstrijden tussen de twee clubs ontstonden na afloop soms rellen tussen enkele groepen supporters. Assignment Het voorkomen dat er rellen uitbreken en dat er gevochten wordt. Penultimate event De wedstrijd is al enige tijd voorbij. Supporters van beide partijen zijn de stad in gegaan. Dan komt er een melding binnen dat beide groepen supporters zich op een groot plein aan het verzamelen zijn.

Panic Setting U bevindt zich in Den Haag waar momenteel een vredesdemonstratie plaatsvindt. Er wordt gedemonstreerd tegen de mensonwaardige omstandigheden en discriminatie van een minderheid in een Afrikaans land. Daarbij zullen zij langs de ambassade van het betreffende Afrikaanse land trekken en daar een petitie aanbieden. Er zijn zo’n 10.000 mensen op de been, waaronder ook vrouwen en kinderen. Assignment U begeleidt de demonstratie en beveiligt de ambassade van het Afrikaanse land tegen mogelijke invallen van demonstranten. Penultimate event De toezeggingen van de ambassadeur zijn rond en de kop van de menigte begint in beweging te komen. Vlak voordat zij bij het einde van de straat zijn worden zij met stenen bekogeld door een groep van 20 sympathisanten van de meerderheid van het Afrikaanse land.

Neutral Setting U bevindt zich op een militair kamp in een door oorlog geteisterd gebied. Er is de laatste tijd veel onrust en de burgers willen daarom graag een veilig heenkomen zoeken op het militaire kamp. Er komt daarom een grote groep burgers (zo’n 500) op het kamp af, bestaande uit mannen, vrouwen en kinderen. Assignment U bewaakt het militaire kamp. Penultimate event De groep is zojuist aangekomen bij de poort.


of 74

Elation Setting U bevindt zich in een grote stad in Afghanistan waar zojuist de nationale verkiezingsuitslag bekend is gemaakt. Een van de nieuwe kandidaten heeft met een kleine meerderheid gewonnen. Vele burgers gaan de straat op om te luisteren naar de overwinningstoespraak en de overwinning te vieren. De samenstelling van deze burgers is zeer gemengd: er zijn zowel jongeren als ouderen en ze komen van verschillende bevolkingsgroepen. Assignment U voorkomt ongeregeldheden (vernielingen en relletjes). Penultimate event Een menigte van zo’n 1000 man is zojuist bij een plein aangekomen waar de verkiezingswinnaar over enkele minuten een toespraak zal houden.

Angry Setting U bevindt zich in Den Haag waar momenteel een vredesdemonstratie plaatsvindt. Er wordt gedemonstreerd tegen de mensonwaardige omstandigheden en discriminatie van een minderheid in een Afrikaans land. Daarbij zullen zij langs de ambassade van het betreffende Afrikaanse land trekken en daar een petitie aanbieden. Er zijn zo’n 10.000 mensen op de been, waaronder ook vrouwen en kinderen. Assignment U begeleidt de demonstratie en beveiligt de ambassade van het Afrikaanse land tegen mogelijke invallen van demonstranten. Penultimate event De menigte is aangekomen bij de ambassade, maar de ambassadeur wil niet naar buiten komen om de petitie in ontvangst te nemen.

Scared Setting U bevindt zich op een militair kamp in een door oorlog geteisterd gebied. Er is de laatste tijd veel onrust en de burgers willen daarom graag een veilig heenkomen zoeken op het militaire kamp. Er komt daarom een grote groep burgers (zo’n 500) op het kamp af, bestaande uit mannen, vrouwen en kinderen. Assignment U bewaakt het militaire kamp. Penultimate event De algehele groep zwelt steeds meer aan en de burgers willen het kamp binnen komen.

the issue of fidelity: what is needed in 3d military serious games?

Documents