An explorative study to Business Simulation

Games: a gap between research and practice?

Sven Fortuin and Danielle Heerink

Department of Information and Computing SciencesUtrecht University, Utrecht, The Netherlands

S.E.Fortuin@students.uu.nl, D.E.M.Heerink@students.uu.nl

Abstract. Research within the field of business simulation games mainlystresses their e↵ectiveness, e�ciency and acceptance. Little research hasbeen conducted about the specific factors that enhance and facilitatelearning. Additionally, companies that develop and/or implement busi-ness simulation games as a business are often left out of investigation.These two factors hypothetically result in a misalignment between lit-erature and practice. This paper contributes to current literature byexamining the alignment of critical factors of business simulation gamesbetween literature and expert companies. A literature study was con-ducted to pose critical factors that enhance learning. Four expert com-panies were interviewed to identify factors and evaluate factors posed byliterature. Results show that expert companies apply critical factors notyet researched but also acknowledge critical factors already validated byresearch.

Keywords: Serious Gaming, Business Simulation Game, Sca↵olding, ,critical factors, Situated Cognition, Active Learning, Repetition.

1 Introduction

According to the Serious Games Association, revenues within the industry ofserious gaming vary from 2 to 10 billion dollars a year (Bohle, 2012). This billiondollar industry consists out of several markets including health-care, education,corporate games, and government & military simulations (Susi et al., 2007).

Simulation games started in the military domain, and can be traced back towar games used in ancient China (?). It took until 1950 before management sim-ulations got introduced by ex-military managers, whom wanted to transfer theirexperience with war games (Wells, 1990). Over the past ten years, managementsimulations expanded and many simulations have been developed including sim-ulations with business related concepts such as operations management (Pasinand Giroux, 2011), marketing (Simon and Gentry, 2014), financial management(Chen et al., 2010), project management (Vanhoucke et al., 2005), enterpriseresource planning (ERP) (Leger, 2006) and risk management (Taran, 2007).

Current research of individual Business Simulation Games (BSGs) mainlystresses the e↵ectiveness, e�ciency and acceptance. Unfortunately, few studies

have examined which factors of simulation games enhance learning (Garris et al.,2002; Bell and Kozlowski, 2008). However, research does suggest a close relationbetween educational simulation games & learning (Tao et al., 2009) and showsthe important role of simulation & games in education while putting learninginto context (Leemkuil et al., 2003). Theory also suggests that simulation gamescan be more e↵ective than other instructional methods (Salas et al., 2009; Tomp-son and Dass, 2000), because they simultaneously engage trainees’ a↵ective &cognitive processes (Tennyson and Jorczak, 2008). Garris et al. (2002) showedthat BSGs are important motivational and learning tools, a result shared by(Malone, 1981) whom specified the motivation to be intrinsic.

Although the domain of serious gaming and simulation games is financiallygrowing (Bohle, 2012), current research does not include companies developingand implementing BSGs as a business in their research. The exclusion of compa-nies that develop and/or implement BSGs could have resulted in a misalignmentbetween theory and practice. Considering the value BSGs can bring to learningand the growing market, it would be valuable if research and practice bene-fit from one another. Companies could benefit from proven learning concepts,whereas research could test theories & applications needed in practice. Com-bined, research and practice can provide a bigger contribution to society thanalone. The suggested misalignment should thus be avoided. To investigate thispossible misalignment, we pose the following research question: To what extentare critical factors that enhance learning aligned between companies that developand/or implement business simulation games and scientific literature?. To prop-erly answer the research question, three sub-questions will be researched andanswered:

1. What are the critical factors of business simulation games according to sci-entific literature?

2. What are the critical factors of business simulation games that can be iden-tified through case studies?

3. To what extent do companies specialised in business simulation games agreewith critical factors already validated by scientific research?

This paper contributes to current scientific literature of business simulationgames by specifically researching critical factors from the perspective of bothpractice & literature, and the (mis)alignment between the two.

The subsequent section presents the theoretical background to our study.Next we discuss the research approach in section 3, the results of which arediscussed in section 4. Section 5 compares these results, followed by a conclusionin section 6 and limitations & future research possibilities in section 7.

2 Theoretical Background

Clarke (2009) states that di↵erent terminologies are used to define business sim-ulation technologies, such as simulation games, business simulators, and micro

& macro worlds. In this paper we will use the term BSG to intend a highly com-plex man-made environment. Its objective is to o↵er students the opportunityto learn by doing, engaging them in a simulated experience of the real- worldand to immerse them in an authentic management situation (Ben-Zvi, 2010, p.62). This definition makes no distinction between BSGs played on a computerand ones that do not utilize technology (e.g. board games & role-plays), as bothtypes can create authentic management situations.

The term business simulation game comprises two concepts, namely simula-tions and games. In literature there often is confusion between simulations andgames (Lewis and Maylor, 2007). Gredler (1996) made clear distinctions betweenthese two concepts (Table 1).

Games SimulationsSetting

Students are transported to another world or environment x x

Purpose

Competition and winning xFulfilling a professional role xExecuting a professional task x

Event sequence

Typically linear xNon linear or branching x

Mechanisms that determine consequences

Set of rules (may be imaginative) x

Dynamic set of authentic causal relationship among two of morevariables

x

Participant is a component of the evolving scenario and executesthe responsibility of his or her role

x

Participant interacts with a database or sets of processes to dis-cover scientific principles, explain or predict events and confrontmisconceptions

x

Table 1. Characteristics of games and simulations (Gredler, 1996)

Both games and simulations transport the user to another world or environ-ment. Within a game the main purpose is competition and winning, whereas themain goal in a simulation is to fulfill a professional role or execute a professionaltask. In a simulation, the user interacts with a database or set of processes to dis-cover scientific principles, explain or predict events and confront misconceptions.

In a more experiential simulation, the participant can even be a component ofan evolving scenario and the rules are defined by this system and scenario. Ingames, a more linear structure is present and a predefined or imaginative set ofrules determine the consequences for the user (Gredler, 1996).

Furthermore, Leemkuil et al. (2003) also mentions that the goal of a game isto win, but phrases it as reaching a specific goal state. Additionally, the researchstated that the specific context of a game can make it more or less realistic,appealing and motivating. A simulation contains also a model of some kind, buthere trainees can insert input variables to trigger and observe an action. As aresult, simulations frequently lack an engaging context.

BSGs are both used in the corporate and educational sector. In both sectorsa wide variety of BSGs can be applied. Within the corporate sector, Collins et al.(2004) mentions a Supply Chain Management game in which agents are simu-lations of small manufactures, who must compete with each other in managinginventories and production facilities. In education, Draijer and Schenk (2004)uses a BSG to teach ERP concepts by integrating processes, ERP knowledge,best practices and other skills.

3 Research approach

As mentioned in the introduction of this paper, the research question is To whatextent are the critical factors according to companies that develop or implementbusiness simulation games and literature aligned?. To answer the research ques-tion and its related sub-questions, research is conducted that consists of threeparts.

3.1 Literature Analysis

A literature analysis on the topic of business simulation games and learninge↵ectiveness answers the first sub-question: What are the critical factors of busi-ness simulation games according to scientific literature?. The literature analysiswill be conducted using Google Scholar and emphasis will be placed on criticalfactors mentioned in multiple articles.

3.2 Case Studies

Next, we present four case studies at di↵erent companies; the results of which areused to discuss the second sub-question: What are the critical factors of businesssimulation games that can be identified through four case studies?. Each com-pany will be interviewed based on a semi-structured question list. The questionlist can be seen in Appendix A and contains open questions with room to askfor clarification. All interviews will be conducted by two persons of whom oneasks the questions, and the other makes notes. The first interview will functionas a trial, and will be used to determine if the question list needs alteration.Additionally, each interview will be recorded and relevant parts will be coded.A short description will be given of each company in the following paragraphs.

InContext Concultancy Group InContext is a Dutch consultancy firm thatspecialises in vision & strategy, change management and learning & develop-ment through BSGs. InContext serves companies world wide and everything isdeveloped specifically on customer demand. BSGs developed by InContext arenot played on a computer. During the interview we spoke with the director whosupervises and co-develops all business simulation games.

Simagine Business Simulations Simagine is a Dutch company specialisedin change management through the (international) deployment of business sim-ulation games. Simagine identifies problems that companies struggle with anddevelops generic business simulation game solutions accordingly. Currently theirsolutions are amongst others being applied in the domains of IT, business pro-cess management and enterprise architecture. BSGs developed by Simagine arenot played on a computer. During the interview we will speak with a researcherfocused on setting up a game expertise centre.

Simenco Simenco is a company that o↵ers serious gaming solutions (includingbusiness simulation games) on the topics of management development, changemanagement, training, education and business challenges. Currently their solu-tions are being applied world wide and range from fully tailor-made to generic.Simenco delivers both computer-based and non computer-based BSGs. Duringthe interview we will speak with the co-founder, who is also partially responsiblefor the development.

Business Gaming Opposed to the other three companies, the focus of BusinessGaming does not lie on the development of business simulation games. Instead,Business Gaming is a portal that brings multiple companies that focus on theimplementation and development of business games & business simulations. Fur-thermore, Business Gaming also implements existing business simulation gamesthemselves. BSGs that can be implemented by Business Gaming are all playedon computers. During the interview we will speak with the founder, who alsohas the task of game leader.

3.3 Literature validation

Finally, to answer the third sub-question To what extent do companies specialisedin business simulation games agree with critical factors already validated by sci-entific research? we ended each interview by presenting a list of seven factorswith the question to indicate the most important ones. Four of those factorswere identified through the literature study, namely: situated cognition, activelearning, sca↵olding and repetition (see section 4.1). The other three factorsworkspace tool, team building and chat facility are mentioned in literature, butnot tested & validated (Leemkuil et al., 2003). A workspace tool creates aware-ness and work flow (e.g. what has been done/what needs to be done). It can

be useful to structure the problem-solving process. Team building refers to theopportunity to play in teams within a BSG. Lastly, a chat facility lets userscommunicate with each other.

By presenting a list that contains both validated and non-validated factors,it can be analyzed whether companies specialized in BSGs have a preference forvalidated or non-validated factors. Factors validated by literature are on di↵er-ent abstraction levels, e.g. repetition is very concrete whereas situated cognitionencompasses multiple elements. For this reason, the added non-validated fac-tors are also on di↵erent abstraction levels. However, the abstraction level hasno direct influence on the importance. As we want interviewees to select theimportant factors, the di↵erent abstraction levels are of no obstruction. Thissub-question can also try to explain the existence or absence of an experiencedgap.

4 Results

4.1 Literature Study

As stated in the research approach, a literature study was conducted to retrievevalidated critical factors of business simulation games. The database GoogleScholar was queried in December 2013 & January 2014 and was searched forarticles that discuss BSGs and learning principles. Additionally, references withinthe retrieved articles were also checked for relevance. Articles were selected basedon title and abstract. After selection, the articles were completely analysed onimportance, validity and relevance. We did not restrict our search on publishingdate, since learning principles are developed throughout the years and still provetheir validation and importance. After identification of a critical factor, GoogleScholar was searched for that specific factors as indicated by search term 7 to11. The results of the literature study can be seen in Table 3

Number: Search term(s):

1 Business simulation games + critical factors2 Business simulation games + learning factors3 Simulation games + critical factors4 Simulation games + learning factors5 Business games + critical factors6 Business games + learning factors7 Situated cognition8 Sca↵olding9 Active learning10 Active instruction11 Repetitiveness + learning

Table 2. Used search terms in the Google Scholar database

Sources Factor Definition

Van Eck (2006),Choi and Hannafin (1995),Brown et al. (1989)

Situated Cognition Learning in a specific & relevantcontext

Sitzmann (2011),Sitzmann et al. (2006),Northrup (2002)

Active Learning Learning principles that rely on in-ductive learning such as explorationand elicitation

Sitzmann (2011),Garris et al. (2002)

Repetition Opportunity to play more thanonce

Alfieri et al. (2011),Hmelo-Silver and Barrows (2006),Quintana et al. (2004)

Sca↵olding Specialized instructional support tobest facilitate learning new subjects

Table 3. Summary literature results

Situated cognition Situated cognition refers to learning that occurs in mean-ingful and relevant context. It is considered as more e↵ective than learning thatoccurs outside those contexts (e.g. formal instruction (Van Eck, 2006). Situatedcognition emphasizes higher-order thinking skills over memorization of factualinformation (Choi and Hannafin, 1995). This must ease the process of translatingand applying abstract knowledge into real life situations, because the context be-comes an important part of the knowledge associated with that learning (Brownet al., 1989).

Active instruction Active learning principles rely on inductive learning. Thestudent must explore and elicit the explanation of a task for e↵ective perfor-mance. This elicitation can for instance be done by examining examples. Duringthis process the trainee is in control of making important learning decisions(Bell and Kozlowski, 2008). The opposite, passive learning, occurs when stu-dents listen to a lecture about a particular subject or watch an instructionalvideo, without questioning or discussing this subject.

The concept of active instruction was examined extensively and researchersfound proof that active instruction does enhance learning. Sitzmann (2011) sta-tistically summarized literature on the instructional e↵ectiveness for teachingwork-related skills and knowledge. Results show that if majority of the gamecontent was delivered through active, rather than passive instruction, learningfrom BSGs was enhanced. Webster and Hackley (1997) found that principle-learning is best accomplished through active involvement of the students. Thisnot only holds for learning in business simulation games but can be generalizedas an learning principle (Northrup, 2002; Sitzmann et al., 2006). Furthermore,Sitzmann (2011) found that when the comparison group learns actively and thetest group is using the simulation game guided by passive instruction, the com-parison group learned more. This claims that technology is a means for deliveringtraining and instruction, but does not have a direct e↵ect on learning (Sitzmannet al., 2006).

Repetition Sitzmann (2011) discovered that learning was also enhanced whenstudents could utilize the business simulation game as many times as desired. Wewill refer to this concept as repetition. The game cycle model from Garris et al.(2002) endorses this assumption. This model states that extensive time to engagein game play is essential for the cycle to occur, meaning that trainees must notbe restricted in their play-time. The inputs for this game cycle are instructionalcontent and game characteristics. The cycle includes user judgements like ’fun’and ’di�cult’, resulting in user behavior like ’playing further’ or ’read somemore tutorials’ and system feedback. Learning outcomes are reached when afterplaying the results are analysed and evaluated, also known as debriefing. Limitingthe user to interact with the game, will stunt the game cycle and will limit thelearning potential. This cycle does not imply that all students learn in the sameway and therefore do not all proceed through these stages (Garris et al., 2002).

Sca↵olding When a student is introduced to a new subject, there needs tobe specialized instructional support in place in order to best facilitate learning.We refer to this concept as sca↵olding (Alfieri et al., 2011). For example, itsupports students in how to do a task and to learn why it should be done thatway (Hmelo-Silver and Barrows, 2006) e.g. by displaying graphics, activatingprior knowledge or modeling an activity. Mentors who sca↵old students throughcoaching and hints are giving structure and guidance. Sca↵olding techniques willnever give the final answer (Quintana et al., 2004).

Sca↵olding is often used in combination with another factor to explain aboutlearning. For example, Garris et al. (2002) states that sca↵olding in combinationwith active engagement with the environment can provide an e↵ective learningenvironment. The importance of this combination is endorsed by Alfieri et al.(2011) stating that unassisted discovery does not enhance learning, whereas en-hanced discovery tasks requires a student to be actively engaged and has apositive e↵ect on learning outcomes. This means that discovery (active learning)itself does not have any e↵ect, but combined with sca↵olding (instruction) doesenhance learning.

4.2 Findings from interviews

As stated in the research approach, four 30 to 60 minute interviews were con-ducted at companies that develop and/or implement business simulation games.Within these interviews open questions were asked to the interviewee on theirvision of critical factors for business simulation games. A summary of those fac-tors can be seen in Table 4 (an ’x’ indicates that the company mentioned thefactor). As mentioned in section 3, the first interview was used as a trial. Thisinterview turned out successful and no alterations were made to the questionlist. As no changes were made, contents of this interview are also be used asregular results.

Situated Cognition As explained in 4.1, situated cognition refers to learningin a meaningful and relevant context. Noteworthy is that, although a factor

Factor / Company InContext Simagine Simenco Business Gaming Total

Situated Cognition x x x 3Distance to Reality x x x 3

Room for Experimentation x x 2Skillful Game Leader x x x 3

Tailor-made x x x x 4Fun to Play x x x 3Repetition x 1Competition x 1

Table 4. Summary interview results

described in 4.1, three of the interviewed companies also came up with thisfactor themselves. The CEO of InContext states ”participants need to recognizethemselves in a simulation, the context should be familiar compared to their dailyworking environment” and ”a business simulation game should be a factual andpure representation of an organisation”. Next, a researcher at Simagine states”participants should encounter the same problems in the simulation as they do intheir daily jobs”. Lastly, a developer at Simenco mentions ”participants do a lot,but what they do should be relevant, it should be about the relevant company”.

Although all three companies do not use the term ’situated cognition’ di-rectly, they do state that the context of participants’ daily jobs should be (par-tially) the same as their real life working environment.

Distance to Reality Directly related to situated cognition, is how close asimulation resembles the reality. A relevant context in a simulation should berecognized by participants as such. People have to recognize the reality in asimulation.

The same three companies that mentioned situated cognition, stated thata business simulation should not represent the reality too closely. The CEOof InContext mentions ”a business simulation game should not be too close toreality, as this will intervene with the game play”. Additionally, a developerat Simenco states ”the reality is often boring, therefore it is important to keepa certain distance between the simulation and reality”. Finally, a researcher atSimagine calls ”if a business simulation game is too close to reality, it will scarepeople from trying di↵erent approaches, because they will think that there is apredetermined correct way of doing a task”.

All three companies agree that a business simulation game should resemblethe reality, but at the same time should not be too close to reality. However, thereasons that the companies provided for the needed distance di↵er.

Room for experimentation It was shown in the previous paragraph thatSimagine opinionated that people should not be limited in their experimentationof trying di↵erent approaches. The researcher of Simagine adds ”people shouldhave freedom to experiment freely with di↵erent solutions” Simagine was not the

only company that addressed this factor, the CEO of InContext states ”businesssimulation should not be normative, people should have the room to try whatworks and what does not work”.

Both companies agree on that business simulation games should allow peopleto experiment, so that participants can find out for themselves which workingmethods are useful, and which are not.

Skillful Game Leader Three companies described that implementing andplaying a BSG requires one or multiple persons that guides the process. Thisperson does not only give instructional support or answers questions, but alsopartially determines the game play depending on the audience and goals of thecompany.

Three companies stated that quality of the game leader strongly influencedthe e↵ectiveness of a business simulation game. The CEO of InContext states”simulations can be very good, but without a proper facilitator, the learning out-come will be very disappointing” and ”the facilitator should make sure that par-ticipants do not feel attacked and feel free to experiment” and ”the translation ofsimulation to reality is a process that is achieved through instruction and guid-ance”. Furthermore, a researcher of Simagine mentions ”the role and level ofthe facilitator are very important, he or she should make sure that people actu-ally learn something”. Lastly, the founder of Business Gaming calls ”I think agame leader is very important, as he guides the entire process” and ”businesssimulation games without a game leader are less valuable”.

The fourth company did not leave this factor unaddressed. However, the opin-ion of the other three companies was not shared, a developer of Simenco states”a proper business simulation game should be playable and e↵ective without agame leader, the game should be self explanatory”.

Tailor-made All four companies that were interviewed mentioned that theytailor business simulation games specifically for a company. This factor can belinked to situated cognition, as situated cognition requires a business simulationgame to have a relevant context. A relevant context can be created by analysingthe context of a customer and creating a similar one for the business simula-tion game. The CEO of InContext mentions ”every business simulation gamethat we make is tailor-made, on specific wishes of the customer”. Additionally,a researcher of Simagine states ”we create generic business simulation games,but calibrate them based on the needs of the company”. Furthermore, a devel-oper of Simenco calls ”business simulation games for big companies are alwaystailor-made”. Finally, the founder of Business Gaming cites ”the context of abusiness simulation game is always changed based on the specific learning goalsof a company”.

As can be seen, InContext and Simenco create completely custom made busi-ness simulation games (i.e from the beginning). However, Simagine and BusinessGaming each have generic games, and customize these based on the wishes ofthe customer.

Fun to Play Three of the four companies mentioned that a BSG should befun to play. The CEO of InContext mentions ”part of the success of a businesssimulation game is determined by whether people enjoy playing it”. Additionally,the founder of Business Gaming states ”the participation rate should be high, thiscan be achieved by making the business simulation game fun to play”. Finally,a developer of Simenco calls ”a business simulation game should be fun, or elsepeople will get bored.

InContext & Business Gaming both state that business simulation gamesshould be fun to play, as this increases the chance of success. Simenco statesthat playing a game can be boring, adding fun is a way of seducing the traineeto keep on playing.

Repetition & Competition Both Repetition & Competition were only men-tioned by one company, whereas the previous discussed factors were at leastmentioned by two companies.

Noteworthy is that Repetition is a critical factor validated by literature ascan be seen in 4.1. A developer at Simenco mentioned this factor himself, as hestates: ”the more you do something, the better you become. If you do somethingten-thousand times, you will become good at it. This is a generic rule, and doesnot only apply to business simulation games”

Competition was only mentioned by the founder of Business Gaming, he cites”the element of competition increases the success rate of a business simulationgame, as people tend to work hard because they do not want to loose from theirco-workers”.

Definition of success To properly interpret the mentioned factors, the di-mension of success needs to be determined. Companies could mention di↵erentfactors due to a di↵erent definition of success. Table 5 shows the di↵erent inter-pretations of successful per company.

Company Successful when:

InContext the BSG has a lasting, positive impact on the behavior and think-ing of participants

Simagine the BSG results in a positive change of behavior of participantsand increases revenues of the customer

Simenco the customer is happyBusiness Gaming the learning e↵ect of the BSG has been validated by proper re-

searchTable 5. Definition of success per company

4.3 Literature Validation

As mentioned in the research approach, each company was given a list of sevenfactors from which they had to select the important ones. The list contained thefactors situated cognition, active learning, sca↵olding, repetition, workspace tool,team building and chat facility, of which the first four are factors validated byliterature (see 4.1). The other three factors have been touched upon by literature(Leemkuil et al., 2003), but were not validated or reproduced by new research.The results can be seen in Table 6 (an x indicates that the company deems thefactor as important).

Factor / Company InContext Simagine Simenco Business Gaming Total

Situated Cognition x x x x 4Active Learning x x x x 4

Sca↵olding x x x 3Repetition x 1

Workspace Tool 0Team Building x 1Chat Facility 0

Table 6. Summary literature validation

As can be seen in Tabel 6, situated cognition and active learning are ac-knowledged by all four companies. Sca↵olding is indicated by three companiesas important. Repetition is the only validated factor that was indicated as impor-tant only once. Additionally, the factor team building is the only non-validatedfactor that was deemed important by one company. The factors workspace tooland chat facility were not perceived as important.

5 Comparison of Results

5.1 Answer to research question

As mentioned in our research approach, to answer the research question To whatextent are critical factors that enhance learning aligned between companies thatdevelop and/or implement business simulation games and scientific literature?we stated three sub-questions:

a What are the critical factors of business simulation games according to scien-tific literature?

b What are the critical factors of business simulation games that can be iden-tified through case studies?

c To what extent do companies specialised in business simulation games agreewith critical factors already validated by scientific research?

Initially, the results of (a) and (b) suggest that there is little alignment be-tween critical factors of our literature study (a) and factors initiated by thecompanies (b). Although situated cognition was mentioned by three out of fourcompanies and repetition was mentioned once, active instruction and sca↵old-ing remain unmentioned. The results also suggest that overall companies takethe same factors into consideration, as most of the factors were mentioned byat least two companies. The mutual di↵erences can be explained by di↵erentperspectives on the definition of success (Table 5) and di↵erent experiences, asall but one of the interviewed companies mentioned that experience was theirmain source for identifying critical factors.

However, a closer look at the results indicates a smaller gap. Room for ex-perimentation can also be seen as a part of active learning, as experimentationis a way of inductive learning. Meaning that by experimentation the rules of atask will be induced. Moreover, the quality of a game leader heavily depends onhis skills to support the trainee and answer questions. This can be translated toinstructional support, also known as sca↵olding.

Additionally, the gap is slightly widened again as expert companies applycritical factors not mentioned or proved in literature, such as: tailor-made, funto play and competition. As no proper research is currently present concerningthese critical factors, conclusions can not be drawn whether these factors actuallyresult in successful BSGs.

Results from (c) show that three (situated cognition, active instruction andsca↵olding) out of four factors are acknowledged by at least three companies.Only repetition was acknowledged once. This might be explained by the fact thattwo of the interviewed companies focus on non computer-based BSGs, makingrepetition harder to accomplish. Furthermore, two factors that were unvalidatedby literature (chat facility and workspace tool) gain no support. However, theunvalidated team building factor gained one vote. No proper explanation can begiven for this without further research, other than the earlier mentioned di↵erentexperiences.

(a), (b) and the findings by sub-question 3 (c) indicate that the gap betweenliterature and practice is present but small. Although expert companies do notexplicitly mention or initiate critical factors backed by literature, they do broadlyacknowledge them and mention sub-elements of those concepts. This suggeststhat experts within the field are supposedly not fully aware of the importanceor existence of these factors & researches, but do overall agree with literature.Furthermore, research seems not to be fully aware of critical factors applied inthe field, as some are not yet researched.

5.2 Critical factors

As shown in the results, distance to reality is mentioned by three of the fourcompanies. However, each company provided a di↵erent motivation for this fac-tor. One company stated that a BSG too close to reality will intervene with thegame play. The second company mentioned that a BSG too close to reality isboring, whereas the third company noted that it would scare people from trying

di↵erent approaches. Although distance to reality is mentioned three times, thecompanies do not agree on its rationale.

Another point of interest is the discrepancy between distance to reality andsituated cognition. Three out of the four companies mentioned that both situatedcognition and closeness to reality are critical factors. However, they are alsocontradictory. Situated cognition wants to mimic the reality, whereas distanceto reality wants to keep a certain distance. The three companies mentioned thatsituated cognition and distance to reality are two di↵erent endpoints of the samespectrum. A good balance has to be found between the two in order for a BSG tobecome successful. Adobor and Daneshfar (2006) proved that participants maylose motivation to play if a simulation game is perceived as unrealistic. Thisfinding suggests that distance to reality should not be too big.

Noteworthy is the di↵erence in opinion on the factor skillful game leader.Three companies mentioned that a game leader is an important aspect of aBSG, whereas the fourth company explicitly stated that a proper BSG shouldbe playable without a person that leads the process. We could find no explanationthat could elucidate these opposite statements.

Lastly, situated cognition and active learning are factors already embeddedin the definition of BSG (section 2). The definition states that a BSG should”o↵er students the opportunity to learn by doing”, which greatly matches withactive learning. Additionally, the definitions also contains ”engaging them in asimulated experience of the real world”, which matches with situated cognition.Companies not acknowledging these two factors would technically not be makingBSGs. Fortunately, all interviewed companies indicated situated cognition andactive learning to be of importance (Table 6).

6 Conclusion

This research provides companies that develop and/or implement business simu-lation games with a detailed overview of possible critical factors that contributeto an e↵ective business simulation game. Both the interviews and literaturestudy found that situated cognition, sca↵olding & active learning increase thee↵ectiveness of business simulation games. The interviews validated the threeof the four critical factors identified by previous studies of situated cognition,active learning and sca↵olding. This study also discovered an inconsistency be-tween the interviewed companies, as one company stated that a proper businesssimulation game does not need a game leader, whereas the other three compa-nies deemed a game leader as important. Furthermore, the interviews identifiedadditional critical factors not mentioned in previous literature, namely: skill-ful game leader, distance to reality, room for experimentation, tailor-made andfun to play. Finally, results suggest a small gap between literature and practice.Expert companies can diminish the present gap by increasing their use of liter-ature as a source of critical factors. Additionally, research can reduce the gapby researching critical factors mentioned by expert companies which are not yetresearched.

7 Limitation & Future Research

Only four (Dutch) companies that perform business simulation games were in-terviewed for the presented study. Furthermore, interviews were the only meansof data gathering, no actual business simulation game was analyzed. Both thesefactors reduce the validity of this research as no proper triangulation took place.

Future research includes extending the study we present in this paper by in-terviewing several more (i.e. non-Dutch) companies that develop or implementbusiness simulation games. Research that focuses or includes other countriesmight find other factors due to various (national) di↵erences. Additionally, thefindings of this study should be validated through multiple quantitative studiesas four companies do not result in statistical significant results. The earlier iden-tified critical factors closeness to reality and presence of a game facilitator areof particular interest as they are not yet mentioned in literature. Additionally,more studies might shed more light on the reciprocal di↵erences between expertcompanies.

Ultimately, the goal is to formulate a framework or list of best practices thatassists and guides business simulation game developers in their task of develop-ing e�cient and e↵ective business simulation games. This study identified andverified multiple critical factors that could be incorporated in the aforementionedframework.

8 Acknowledgements

We want to thank the business simulation game companies InContext, Simagine,Simenco and Business Gaming for their collaboration and availability for therequired interviews.

Additionally, we want to thank our supervisors Pieter Wouters and FabianoDalpiaz, connected to the department of Information and Computing science ofUtrecht University, whom guided the process of this research.

Bibliography

Adobor, H. and Daneshfar, A. (2006). Management simulations: determiningtheir e↵ectiveness. Journal of Management Development, 25(2):151–168.

Alfieri, L., Brooks, P. J., Aldrich, N. J., and Tenenbaum, H. R. (2011). Doesdiscovery-based instruction enhance learning? Journal of Educational Psy-chology, 103(1):1.

Bell, B. S. and Kozlowski, S. W. (2008). Active learning: e↵ects of core train-ing design elements on self-regulatory processes, learning, and adaptability.Journal of Applied Psychology, 93(2):296.

Ben-Zvi, T. (2010). The e�cacy of business simulation games in creating decisionsupport systems: An experimental investigation. Decision Support Systems,49(1):61–69.

Bohle, S. (2012). Serious games now $2 to $10 billion industry. RetrievedJanuary 17, 2014, from: http://www.hypergridbusiness.com/2012/08/serious-games-now-a-multi-billion-dollar-industry/.

Brown, J. S., Collins, A., and Duguid, P. (1989). Situated cognition and theculture of learning. Educational researcher, 18(1):32–42.

Chen, Z.-H., Wu, H.-W., Li, Y.-C., and Chan, T.-W. (2010). My-investment:Simulation games to help primary students learn financial management. InProc. 18th International Conference on Computers in Education (ICCE 10),Dec, volume 10, pages 548–552.

Choi, J. I. and Hannafin, M. (1995). Situated cognition and learning environ-ments: Roles, structures, and implications for design. Educational TechnologyResearch and Development, 43(2):53–69.

Clarke, E. (2009). Learning outcomes from business simulation exercises: Chal-lenges for the implementation of learning technologies. Education+ Training,51(5/6):448–459.

Collins, J., Arunachalam, R., Sadeh, N. M., Eriksson, J., Finne, N., and Janson,S. (2004). The supply chain management game for the 2005 trading agentcompetition. Journal of Information Systems Education.

Draijer, C. and Schenk, D.-J. (2004). Best practices of business simulation withsap r/3. Journal of Information Systems Education, 15(3):244–261.

Garris, R., Ahlers, R., and Driskell, J. E. (2002). Games, motivation, and learn-ing: A research and practice model. Simulation and gaming, 33(4):441–467.

Gredler, M. E. (1996). Educational games and simulations: a technology insearch of a research paradigm. In (Eds.), D. H. J., editor, Handbook of researchfor educational communications and technology. New York, Simon & SchusterMacmillan, New York.

Hmelo-Silver, C. E. and Barrows, H. S. (2006). Goals and strategies of a problem-based learning facilitator. Interdisciplinary Journal of Problem-based Learn-ing, 1(1):21–39.

Leemkuil, H., de Jong, T., de Hoog, R., and Christoph, N. (2003). Kmquest: A collaborative internet-based simulation game. Simulation & Gaming,34(1):89–111.

Leger, P. M. (2006). Using a simulation game approach to teach ERP concepts.Montreal: HEC Montreal, Groupe de recherche en systemes d’information.

Lewis, M. A. and Maylor, H. R. (2007). Game playing and operations manage-ment education. International Journal of Production Economics, 105(1):134–149.

Malone, T. W. (1981). Toward a theory of intrinsically motivating instruction.Cognitive science, 5(4):333–369.

Northrup, P. T. (2002). Online learners’ preferences for interaction. QuarterlyReview of Distance Education, 3(2):219–226.

Pasin, F. and Giroux, H. (2011). The impact of a simulation game on operationsmanagement education. Comput. Educ., 57(1):1240–1254.

Quintana, C., Reiser, B. J., Davis, E. A., Krajcik, J., Fretz, E., Duncan, R. G.,Kyza, E., Edelson, D., and Soloway, E. (2004). A sca↵olding design frameworkfor software to support science inquiry. The Journal of the Learning Sciences,13(3):337–386.

Salas, E., Wildman, J. L., and Piccolo, R. F. (2009). Using simulation-basedtraining to enhance management education. Academy of Management Learn-ing & Education, 8(4):559–573.

Simon, R. and Gentry, J. W. (2014). Use of a simulation in a large class en-vironment for a marketing principles class: A qualitative analysis of whetherlearning objectives were met. Developments in Business Simulation and Ex-periential Learning, 38.

Sitzmann, T. (2011). A metaanalytic examination of the instructional e↵ective-ness of computerbased simulation games. Personnel Psychology, 64(2):489–528.

Sitzmann, T., Kraiger, K., Stewart, D., and Wisher, R. (2006). The comparativee↵ectiveness of webbased and classroom instruction: A metaanalysis. Person-nel Psychology, 59(3):623–664.

Susi, T., Johannesson, M., and Backlund, P. (2007). Serious games : An overview.Technical Report HS- IKI -TR-07-001, University of Skvde, School of Human-ities and Informatics.

Tao, Y. H., Cheng, C. J., and Sun, S. Y. (2009). What influences college stu-dents to continue using business simulation games? the taiwan experience.Computers & Education, 53(3):929–939.

Taran, G. (2007). Using games in software engineering education to teachrisk management. In Software Engineering Education & Training, 2007.CSEET’07. 20th Conference on, pages 211–220. IEEE.

Tennyson, R. D. and Jorczak, R. L. (2008). A conceptual framework for theempirical study of instructional games. In (Eds.), H. F. O. . R. S. P., edi-tor, Computer games and team and individual learning. Oxford, UK: Elsevier,Oxford.

Tompson, G. H. and Dass, P. (2000). Improving students self-e�cacy in strategicmanagement: The relative impact of cases and simulations. Simulation &Gaming, 31(1):22–41.

Van Eck, R. (2006). Digital game-based learning: It’s not just the digital nativeswho are restless. Educause review, 41(2):1–16.

Vanhoucke, M., Vereecke, A., Gemmel, P., Arkader, R., De Meyer, A., Klassen,R., and Whybark, C. (2005). The project scheduling game (psg): simulatingtime/cost trade-o↵s in projects. Project Management Journal, 36(1):51–59.

Webster, J. and Hackley, P. (1997). Teaching e↵ectiveness in technology-mediated distance learning. Academy of management journal, 40(2):1282–1309.

Wells, R. A. (1990). Management games and simulations in management devel-opment: An introduction. Journal of Management Development, 9(2):4–6.

Appendix(A(Questionnaire(Seminar(Serious(Gaming(

Introduction(1. What'type'of'organization'are'you?'

a. What'kind'of'activities'do'you'perform?'

b. What'products'or'services'do'you'deliver?'

'

2. Which'Business'Simulation'Games'are'in'you'portfolio?''

a. Can'you'tailorAmade'these'products?'

Critical(factors(3. What'kinds'of'factors'are'considered'during'development?'

a. Do'you'focus'on'game'or'instructional'factors?''

b. What'do'you'think'is'important'and'why?'

4. Do'you'still'adjust'the'BSG?''

a. If'yes,'on'basis'of'what?''

i. Own'insights,'client'input,'literature'etc.'ii. Collaboration'with'experts'in'the'field'of'education/training'iii. End'users'testing'

b. If'no;'why'not?'

5. When'is'according'to'you'a'BSG'successful?'

i. Customer'satisfaction'ii. End'users'learning'curve'increases''iii. Perfect'game'relative'to'wishes'client'iv. Game'was'sold'lot'

6. What'are'critical'factors'of'BSG'according'to'you'and'why?'

7. How'did'you'come'up'with'these'factors?''

i. Literature'ii. Client'whished'iii. Results'of'end'users'iv. Testing/experimenting'

8. Do'you'think'that'these'factors'are'generalizable'to'other'BSGs?''

a. Why'do'/'not?'

b. Are'these'factors'also'generalizable'to'other'educational'software?'

Success(within(instruction(9. In'what'extent'is'the'client'dependent'on'the'success'of'the'game?''

a. In'extent'of'instruction'

b. In'extent'of'availability'to'end'users'

c. In'extent'of'motivation'

10. In'which'setting'should'the'game'be'used'and'why?'i. Standalone'(BSG'is'only'form'of'instruction/training''ii. Part'of'teaching'program'(focus'on'BSG)'iii. Addition'to'teaching'program'(support'on'BSG)'iv. Are'there'any'other'objects'that'enhance'learning?'

Factors((Active(learning(The'majority'of'the'game'content'was'delivered'through'active,'rather'than'passive'instruction.'

Active'learning'principles'rely'on'inductive'learning.'The'student'must'explore'and'elicit'the'

explanation'of'a'task'for'effective'performance.'Examining'examples'can'for'instance'do'this'

elicitation.'During'this'process'the'trainee'is'in'control'of'making'important'learning'decisions.'

'

(((Scaffolding(When'a'student'is'introduced'to'a'new'subject,'there'needs'to'be'specialized'instructional'

support'in'place.'For'example,'it'supports'students'in'how'to'do'a'task'and'to'learn'why'it'should'

be'done'that'way'e.g.'by'displaying'graphics,'activating'prior'knowledge'or'modeling'an'activity.'

'

(((Repetition(Students'should'have'the'opportunity'to'play'the'game'as'many'times'as'desired.''

'

(((Situated(cognition(Situated'cognition'refers'to'learning'that'occurs'in'meaningful'and'relevant'context.'It'is'

considered'as'more'effective'than'learning'that'occurs'outside'those'contexts'(e.g.'formal'

instruction).''

'

(((Teaming(up(It'is'possible'to'work'in'teams.''

'

(((Chat(facility(A'central'chat'facility'is'always'visible.'

'

(((Workspace(tool(A'workspace'tool'creates'awareness'and'workflow'(e.g.'what'has'been'done/what'needs'to'be'

done).'It'can'be'useful'to'structure'the'problemAsolving'process'

'