Users’ Involvement in Requirements Engineering and System Success

Muneera Bano, Didar Zowghi Research Center for Human Centered Technology Design

Faculty of Engineering and Information Technology University of Technology, Sydney, Australia

Muneera.Bano@uts.edu.au, Didar.Zowghi@uts.edu.au

Abstract—Involving users in software development in general, and in Requirements Engineering (RE) in particular, has been considered for over three decades. It is axiomatically believed to contribute significantly to a successful system. However, not much attention has been paid to ascertain in which phases of software development life cycle involvement or participation of users is most beneficial. In this paper we present an investigation into the concept of users’ involvement during RE activities and explore its relationship with system success. We have conducted a systematic literature review (SLR) using guidelines of Evidence Based Software Engineering. Our SLR identified 87 empirical studies from the period of 1980 to 2012. Only 13 studies focused specifically on investigating users’ involvement in RE and 9 of these confirmed benefits of involving users in requirements analysis and 4 remain inconclusive. Effective involvement of users in RE may reduce the need for their more active involvement in the rest of software development. This paper also offers a checklist we have created from the identified factors of all 87 empirical studies that should be utilised for effective users’ involvement in RE.

Index Terms—users’ involvement, requirements engineering, software development lifecycle, systematic literature review.

I. INTRODUCTION Involving users in software development life cycle (SDLC)

has been suggested to improve the quality, accuracy and completeness of requirements [1] [2], that should result in users’ satisfaction. Users are meant to have a great deal of domain knowledge about the tasks they perform, work practices, context of the system use and their behavior and preferences. Users often are not able to articulate their needs and requirements completely as part of their knowledge about the domain is tacit and is not consciously available [2]. Articulation of this knowledge is important to achieve complete and unambiguous requirements leading to user acceptance and satisfaction. Involving users in RE phase enhances and facilitates the elicitation of this tacit knowledge [3] by getting information, feedback and also determines their participation in the rest of development lifecycle.

“Users’ involvement” has been the focus of research for over three decades and has been studied in various disciplines from different perspectives. It was intuitively accepted to play a positive key role in users’ satisfaction and ultimately to system success [1]. This axiomatic notion can be traced back to organizational management research, including group problem solving, interpersonal communication and individual motivation [1]. However, the previously published literature

reviews of over three decades on this topic in various computing disciplines have shown conflicting results ([1] [4] [5] [6] [2]). One of the major causes of this conflict is claimed to be the lack of shared understanding of this concept and its philosophical stance [4], mainly due to the divergent use of words and phrases to define this complex phenomenon. Other factors for inconsistencies in the results have been identified as: methodological problems [1], confounding effects of the terms “users’ involvement” and “users’ participation” [4] [7], and contingency factors [8] to name but a few.

The phrase “users’ involvement in software development” consists of three major concepts at a very abstract level and has many ambiguities when it comes to defining those terms individually as well as when they are combined. Firstly “Users” are not considered uniformly in all the previous studies mainly because they play different roles within the organizations [9]. Typically a user is someone who would ultimately use the system being developed or would be in some way affected by it. But what role they can have within the organization can vary depending on different methodologies. Under Participatory Design (PD) “users means the operational workers who are affected by the system, this does not include the manager”, but in Joint Application Development (JAD), “users are any non IS/non technical individuals in the organization who are affected by the system, this includes managers” [9].

Secondly, “involvement” is used inconsistently and synonymously with “participation” and “engagement” in the previous studies. Barki and Hartwick [7], make a clear distinction between users’ involvement (“a subjective psychological state reflecting the importance and personal relevance of a system to the user”) and users’ participation (“a set of behaviors or activities performed by users in the system development process”) [7]. Whereas “users’ engagement” has been used in the literature as a synonym to both concepts of involvement and participation [5].

Thirdly, software development lifecycle, comprises of various phases that include many activities and it is effected by various dynamic factors such as methodologies used, application domains where software will be situated, and technological changes [4]. Involving users in early phases of software development such as requirements engineering activities has been highly recommended and is considered to be very beneficial to the system success more so than other phases [2] [10] but there is not much empirical evidence available to support this claim.

978-1-4799-1011-3/13/$31.00 c© 2013 IEEE EmpiRE 2013, Rio de Janeiro, Brasil24

Furthermore, defining and exactly measuring system success is not addressed uniformly in the literature. Among the popular factors used for this measurement are users’ acceptance and satisfaction with the system [11]. System success is dependent on the successful outcome of the RE activities. El Emam et al [12], define RE success as: “the extent to which the outcomes of the RE process, serve the needs of, and provide a basis for ensuring the success of, all subsequent activities, individually and in aggregate, related to the software system throughout the software system's lifetime. These subsequent activities include: design, coding, testing, putting into operation, and post-deployment evolution.”

In our earlier work we have presented the results of our Systematic Literature Review (SLR) on users’ involvement in the entire software development life cycle [13]. In this paper we focus primarily on users’ involvement in RE activities. Our objective here is to analyze the impact of users’ involvement in RE phase on system success. This analysis offers the following contributions: - Identification of the studies that has explicitly specified their

focus on RE phase of SDLC - From the empirical literature, extracting factors needed, for

developing a check list to be considered while involving users in the RE phase

- Analysis of the reasons for conflicting results in empirical literature on the impact of users’ involvement on system success

- Highlighting the need for further empirical investigation using a strong mixed method research design utilizing the existing concepts of 32 years of empirical research

The paper is organized as follows; section II describes Background and Motivation for this work. Section III lists the research questions for our inquiry. Section IV summarizes the systematic literature review process we followed. Section V shows the results of our review. Section VI focuses on the results related to RE. Section VII discusses the results obtained from our review to answer our research questions. Section VIII is conclusion and IX describes our future directions.

II. BACKGROUND AND MOTIVATION The relationship between user involvement and system

success is neither simple nor direct. There are many factors that complicate and influence the successful outcome of any software system project. Those factors belong to different perspectives of user involvement, e.g. psychological, managerial, methodological, cultural and political. One of the important factors is the consideration of which phase of SDLC users are to be involved [14]. In different phases of SDLC different types and levels of participation of users are required. For example, senior management may be required throughout development, and middle management and other employees (such as Subject Matter Experts), would be required for their contribution during problem identification, requirements elicitation, design and testing [4]. System and project complexity, and uncertainty are important contributors that determine the phases of SDLC for user involvement, the required level and degree of involvement and the type of activities they carry out in each phase [14].

User involvement or participation in one stage of SDLC is said to influence the level of participation in the subsequent

stages [15]. Some researchers have stressed that involving users in early stages of SDLC is more important and beneficial than others [2] [16] [10], and stressed that after effective involvement of users in RE further involvement may not be required in subsequent phases [17].

Following the guidelines for Evidence Based Software Engineering (EBSE) [18], we have conducted a systematic literature review for analyzing various dimensions of user involvement in software development and its impact on system success [13]. Our review resulted in 87 empirical studies out of which only 13 mentioned specifically that they were investigating user involvement in Requirements phase. Surprisingly, there is a lack of empirical work to investigate the user involvement in RE and system success relationship.

Our SLR was the first ever study to aggregate 32 years of literature on user involvement [13]. To date, the impact of user involvement in RE on delivering a successful system has not been investigated or analyzed thoroughly using the guidelines for EBSE. Our review is beneficial for practitioners on enhancing their understanding of this multifaceted phenomenon and providing guidelines on different aspects of users’ involvement in requirements phase (e.g. user identification, level of involvement and perspective of involvement etc). This review is part of our ongoing empirical inquiry into the concepts of users’ involvement in modern system development. In this paper, our analysis of the available empirical research allows us to focus on answering the following questions: 1. What is the relationship between users’ involvement during RE

phase and system success? 2. Is users’ involvement during RE activities more effective

towards achieving system success than the other stages of SDLC?

3. What are the factors that have to be considered for effective involvement of users in RE activities?

III. SYSTEMATIC LITERATURE REVIEW Many researchers have conducted literature reviews to

bring together the empirical studies that have explored the user involvement and system success relationship. The most notable among them are reviews from Ives and Olsen for the empirical work conducted in 1959-1981 [1], Cavaye for 1982-1992 [4], and Hwang and Thorn for 1976-1996 [5]. The most recent of these reviews by He and King [6] summarized the results from 82 studies published till 2007 in a meta-analysis. All these reviews show conflicting results. According to Ives and Olson [1] only 36%, and Cavaye [4] only 37% of the studies are showing positive impact of users’ involvement on system success. Whereas Hwang and Thorn [5], and He and King [6] show more significant positive relation of users’ involvement on system success. None of these previously conducted reviews had the rigor of search and selection strategy of SLR proposed in the guidelines of EBSE. SLR provides a methodical and meticulous process of collecting and collating the published empirical literature for focused research questions. The guidelines are devised to produce best possible unbiased results.

25

The SLR based on the guidelines proposed in Kitchenham et. al. [18] consists of three main steps; Planning, Execution and Reporting. We followed the guidelines and carried out all the proposed phases of SLR staring from the development of protocol which outlined all the details of our primary and secondary search strategy, study selection criteria, quality assessment of finally included studies, data extraction strategy to answer our research question, data synthesis and analysis planning. The protocol was pilot tested for evaluating the completeness of our search string, and correctness of inclusion/exclusion criteria and data extraction strategy. After pilot testing the updated version of the protocol was sent to the external reviewers (including Kitchenham), and the recommended changes (especially related to the scope of the study) were accommodated in the final version of protocol. Based on our three major search terms (user, involvement, software development) we built the following search string; ON ABSTRACT ((user OR customer OR consumer OR “end user” OR end-user) AND (involv* OR participat* OR contribut*) AND (“software development” OR “software project” OR “IS” OR “information system” OR “IT” OR “information technology” OR “SDLC” OR “product development” OR “IT adoption” OR “IT diffusion”))

We executed this search string by customizing it according to the ACM Digital Library, IEEE xplore, Science Direct, Google Scholar, Citeseerx, Springerlink, and MIS Quarterly, Association of Information System electronic Library (AISeL), INFORMS online (Operation Research and Management Sciences), Journal of Management Science, Journal of Information System Research, Journal of Operations Research. We searched the DBLP1 publication profiles of some authors who were highly cited for their work on users’ involvement. They were: E. Mumford, H. Barki, J. Hartwick, M. H. Olson, J. J. Baroudi, B. Ives, G. Torkzadeh, W. J. Doll, R. Hirschheim, Khalid El Emam, L. A. Keppleman, J. D. McKeen, and S. Kujala.

We scanned references of three specific literature reviews to check for missing papers in our results. Our reason for the selection of these three reviews was based on the high quality of the journals they were published in, they had a high citation index and covered the last four decades; [1] [4] [6]. For studies that have been published in conferences and an extended version as journal articles, we only selected their journal versions. To make our review focused and manageable, we also excluded MS/PhD thesis, and papers where users’ involvement or participation was considered as only one factor among many for system success or users’ involvement was not the main focus.

We excluded a few studies that were extremely low quality and some who have plagiarized the works of others. We used the journal and conference ranking of Excellence of Research in Australia (ERA 20102,3 ) for assessing the quality of our

1 http://www.informatik.uni-trier.de/~ley/db/ 2

www.arc.gov.au/xls/era2010_journal_title_list.xls 3 www.arc.gov.au/xls/era2010_conference_list.xls

selected papers. We used MS Access for extracting data for frequency analysis and NVivo software for coding the data and performing thematic analysis to answer our research questions.

From the overall search process (from online databases as well as manual search), after discarding duplicates, applying study selection criteria and excluding non-related and low quality papers, we were left with 87 empirical studies. The details of the process execution and results are presented in [13].

IV. RESULTS OF SYSTEMATIC REVIEW Out of 87 studies, 46 have used survey as a data collection

method. The percentages of research methods utilised in the included studies are: survey 53%, case studies 23%, experiments 13%, and field studies 8%. Out of 46 survey studies, 67% show positive results, while case studies show 60%, experiments show 81%, and field studies show 86%. In our SLR, 32% studies show negative or uncertain results. This variation may have something to do with the fact that data was collected at different stages of SDLC, organisations were of different sizes, projects were developing different types of systems, could have had varying degree of complexity and the software development methodologies were varied. This analysis is depicted in Fig 1 and Fig 2.

Fig. 1. Summary of characteristics of included studies (number of studies wrt Decade, research method and era ranking) [13]

Fig 2 shows the breakdown of the frequencies of the results. The results are classified in three categories, Positive (+, showing the user involvement has positive influence in bringing about system success), Negative (-, showing that user involvement does not have positive influence on system success) and uncertain (?, the results of the studies are inconclusive). The frequencies are further refined against research methodology adopted to obtain results and the decade to which that publication belongs to give a more comprehensive picture of the results obtained. Out of 87 studies retrieved in systematic review, 59 show positive impact of users’ involvement on system success and they make up 68% of the results. 7 studies are reporting negative results whereas 21 are uncertain on the issue. Out of 59 papers showing positive results, 31 used survey research method, which is almost 53% of the positive results. Analyzing the detailed breakup shows that from 1990s onwards, the research

26

is showing more positive results. But during the first period, the trend was different.

Fig. 2. Relationship of users’ involvement and system success [13]

V. RESULTS OF USER INVOLVEMENT IN RE A large number of studies in our review were not focusing

on any particular phase of SDLC. Out of 87 empirical studies, only 13 explicitly mentioned they focused on investigating the role of user involvement in requirement analysis phase. Table I represents frequency of studies out of 87, showing positive, negative or uncertain results for the impact of users’ involvement at different stages of SDLC on system success which was explicitly mentioned in those studies. Out of 13 studies on RE, 9 reports positive impact of user involvement on system success where as 4 are uncertain about their results. Table II further divide the 13 studies related to RE studies on the basis of their characteristics related to the decade when they were published and research methodology used in the study. Once again, survey is the highly used method and 7 out of 13 studies are published after year 2000.

TABLE I. STAGES OF SDLC WHERE USERS WERE INVOLVED

Stages of SDLC Impact Number of studies Positive 9 Negative 0 Requirements Analysis Uncertain 4 Positive 9 Negative 3 Design Uncertain 6 Positive 5 Negative 1 Implementation Uncertain 1

TABLE II. STUDIES ON USER INVOLVMENT IN RE STAGE AND SYSTEM SUCCESS

Ref ERA RM Research Perspective Impact on SS

[19] A Case Study Exploring UI and SS relationship Positive [20] A Case study User representation Positive [21] C Case Study Issues in UI in SD Uncertain [22] UR Case study Exploring UI and SS relationship Uncertain [23] A* Experiment Managerial, Psychological Positive

[17] A Experiment Exploring UI and SS relationship, Psychological Positive

[24] B Experience report System development Positive

[25] A* Survey Psychological Positive

[26] A* Survey Organizational Management, System Development Positive

[27] A* Survey Contextual Situations, Psychological or Behavioural Positive

[28] A Survey System development Positive [12] B Survey Exploring UI and SS relationship Uncertain [15] C Survey Exploring UI and SS relationship Uncertain

We now answer the research questions as follows:

What is the relationship between users’ involvement during RE phase and system success?

All the studies that report positive results are of high quality (according to ERA ranking). Among studies having uncertain results, only [12] is a high quality study and the results are not conclusive, but they were investigating the relationship of user participation in RE with a variable “uncertainty” on system success. Otherwise they have supported the notion of involvement of users in early phases of SDLC to be more affective. Therefore based on available empirical literature specifically focusing on RE phase of SDLC the answer to the question seems to be: “there is a positive relationship between users’ involvement during RE and system success”. Is users’ involvement during RE activities more effective towards achieving system success than the other stages of SDLC?

From the results of our review, in design and implementation phase, the empirical studies have conflicting results (Fig II), as some studies are showing that user involvement had negative impact on system success and was causing issues rather than benefits. Although based on the results from our systematic review, it appears that users’ involvement would be most effective in RE phase but due to the small number of empirical studies available on this topic, therefore this claim would not have much foundation and remains as an open area of research. There are various factors for the conflicting results in empirical literature that are discussed in section VI. What are the factors that have to be considered for effective involvement of users in RE activities?

To answer this question, we have created a checklist based on identified factors from all 87 empirical studies of our review. For effective users’ involvement in RE, following checklist should be answered;

A. What is the Perspective for user involvement? It is very important to know the purpose of users’

involvement and what benefits it would yield. Knowing the perspective for users’ involvement, one can answer the questions of “why to involve users?” and “how to involve users”. After thematic analysis of our results, following five categories emerged.

a) Psychological perspective One of the main factors in defining system success is

users’ satisfaction, that is, considering a psychological state that comes when users perceive that they have control over the system development process [7]. Therefore involvement (where users may not be participating or engaged) actually gives users’ satisfaction by their sense of control [29]. The other psychological factors that play a role for users’ involvement are their willingness to participate, their capability which ultimately impacts their interest, and their characteristics and attitudes [4].

b) Managerial perspective The users are not to be merely involved, their

involvement and participation has to be properly managed to

27

achieve the desired results of bringing about system success. The focus of management is the identification of users or their representatives, selecting a method or technique for their “effective” involvement, and managing their participation in the required activities to ensure the achievements of objectives [30]. Managers have to take into account the stages of development where users should be involved, the allocation of financial resources to ensure higher level management commitment and support [4].

c) Political perspective The degree of power given to users during their project

involvement will undoubtedly determine their extent and degree of involvement. The level and degree of involvement can be affected by organizational or political influence especially when it comes to power of decision making and implementing changes. There can be potential conflicts between users and the development team and it is imperative that conflict resolution strategies are available when this occurs [31].

d) Cultural perspective Although our findings reveal that the concept of culture

has not been studied as widely and as frequently as other factors but it remains as one of the effective perspectives that needs to be addressed in users’ involvement. The overall purpose of users’ involvement can be different when linked to different cultural contexts, be it national, organisational or project cultures. Organizational culture influences the consideration of why and how to involve users. The two most famous cultural based practices are Scandinavian Participatory Design with strong Marxist flavour and American Joint Application Development [32].

e) Methodological perspective Depending on the role users play in the software

development process different techniques are available [9]. For informative role (where users only provide information related to the end product), interviews, focus groups, and surveys can be used. For participative role (where users are performing activities during development and have some power to influence decision making), agile methods, and Participatory Design techniques can be used. For consultative role (where users have to provide feedback or reviews), meetings can be arranged [33]. Other factors that play a role in selection of any particular method or technique are varying degree of project complexity, and available technological resources [4].

B. To what degree and level the users should be involved? Besides use of appropriate methods and techniques, and

identification of the correct users, it is crucial to know what would be the right level of users’ involvement to increase the likelihood of system success. The main difference between various methods and techniques for users’ involvement is related to their degree or level of participation which would influence the resulting system [9]. Usually it is assumed that increasing users’ involvement should make the project more successful but the benefits of users’ involvement can only be achieved if it is properly managed. The degree of the

involvement depends on various factors such as users’ expertise, their previous experience with information systems, and organizational culture and politics. The project or task uncertainty also plays a significant role in determining the impact of level of users’ involvement on system quality and success [34]. With high levels of task uncertainty, users should be highly involved to determine requirements and complexity of the system [35] [14]. However, with low task uncertainty, users’ involvement may not have significant impact on the resulting system.

Among all 13 studies focusing on RE (Table II), only [23] has experimented four modes of participations as treatment conditions during analysis phase: mute, voice, choice, voice plus choice. This study was involving users from psychological and behavioural perspective. Their results were positive demonstrating that increasing level of participation gave users a psychological sense of control and attributed to the system success.

C. Who should be involved and who should participate? For effective users’ involvement, it is very important to

identify the right people from a group of stakeholders who are to be involved or given the chance to participate. Not all the stakeholders carry equal relevance to software being developed. Not all involved users are required to participate in software development. Therefore from the involved users, often another subset is selected whom would be given the chance to actually participate in the development process [36]. Surprisingly our SLR did not find many articles that discussed this issue, however, this topic has been covered reasonably well within the RE literature though not empirically tested sufficiently, for example the work of Maccaulay [37], Robertson [38], and Alexander [39].

Before the selection process, the users are to be identified and the concept of ‘user’ is to be clearly understood as it has many interpretations. According to Eason [40], there are three categories of users: primary, secondary and tertiary. “Primary users are those likely to be frequent hands-on users of the system; secondary users are occasional users or those who use the system through an intermediary; tertiary users are those affected by the introduction of the system, or who will influence its purchase. Here, we shall assume that ‘users’ are the people, groups or companies who will interact with the software and control it directly, and those who will use the products (information, results etc) of the system”. Various types of users are to be represented to ensure appropriate selection as different users will have different influence on decision depending on their use of the future system. E.g. [33]“`first level’ users or `end users’ , who will interact directly with computer terminals to help them to perform their work, will have different interests from those users who will only utilise printed output or manage the primary users. Each type of user should influence decisions concerning design of the facilities they will use in the future.”

Once the users and their categories are identified various factors are to be considered for the selection of the users for involvement and participation e.g. stakeholders who are the most beneficial for development process, level of expertise of users, the impact of systems changes on particular group of employees within organization etc. In line to the concept of

28

stakeholder analysis in RE [41], there is further need of users’ analysis, which would require answer to the following questions [33]; “Who will be affected? (this will include people who do not use the system directly); Is it possible to identify just two or three main user categories? (based on facts such as grades of staff, possible use of system etc.); What are the characteristics of people in each user category? (e.g. age range, computing experience); What are the characteristics of the task performed by each user category? (e.g. degree of guidance, amount of human contact, sources of error); What do different users like and dislike about their jobs? How are the different users likely to react to the system? (which aspects will be seen as `costs’ and which as `benefits’ to the users?)”

D. What method and technique should be used? Different techniques, methods and approaches are used

for implementing users’ participation. E.g. PD, JAD, ETHICS, Prototyping, Agile, UCD, CSCW etc [2]. The choice of method or technique depends on the underlying philosophy for users’ participation i.e. functionalist versus the neo-humanist paradigm [4]; Functionalism: “Participation in the functionalist tradition means that involvement of users is sought in order to build functionally correct and efficient systems. Participation, then, is considered to be a means to an end: it can help provide better information on requirements, overcomes resistance, and validates design options. The objective is to create a 'better' system that is more likely to be used by potential users. If there is no apparent need to involve users (for instance, when the task to be automated is structured or when use of the system is going to be mandatory), participation may not be sought or may not be taken very seriously.” Neo-humanism: “Rather than placing the system at the centre of the development effort, people and the potential users of the system are considered the most important. Examples of methods in this tradition include ETHICS and Scandinavian approaches. Participation is seen as a way of improving understanding between users and system builders; participation is also encouraged to enable users to have input in changes to their own work environment. It is the objective of the development effort to create a superior workplace for the people working there; the end result of participation should be a human being with increased understanding, an expanded workplace, and an enriched working life.”

VI. DISCUSSION Requirements analysis phase is concerned with discovering

accurately what the users want from the systems to be developed. But it is difficult to get good quality requirements which are accurate and unambiguous. Giving users a chance to participate in the development is said to improve the quality of the system by providing a complete assessment of users’ information requirements [1] [2] [33]. Users may not be able to articulate their needs about the system explicitly as much of their understanding about their practices and work environment is tacit [2]. With properly managed users’ participation, the users would be able to tell if their requirements are correctly translated to the design or implementation phase. The issues and challenges that are faced during users’ involvement (e.g. Communication Problems, Misunderstandings, conflict etc [42], [43]) are mostly those that are to be handled in requirements analysis phase of development. According to the results of our review, there is an overall positive impact of users’ involvement

throughout SDLC on capturing users’ accurate requirements [27] [13] .

Our systematic review of literature has provided insights into some of the causes of inconsistencies in the empirical findings of the last three decades. According to the analysis of our results, the inconsistencies are due to the following factors;

A. Differences in Research Methods According to the review of Ives and Olson [1], one of the

main reasons for inconsistent results is the differences in the choice of research design by researchers in their empirical investigation. Fig I presented different methods used in investigating this complex and multi faceted phenomenon. Among them the method mostly used is survey followed by case studies. From 1980-1989 surveys are dominating the research field as 80% of the studies. But they provided very conflicting results and in second and third periods gradually other qualitative research methods are adopted to study this phenomenon. Surveys are good for collecting structured data from a large population for testing a hypothesis. But as users’ involvement is known to be a complex concept that is not fully understood, surveys may not be able to capture the rich contextual details required to understand this phenomenon [4]. Although the problem of the choice of research methodology mentioned above has long been identified (since 1984) as one of the reasons for lack of consistency in empirical findings [1], but to date we have not found a study that has utilized mixed method research approach to produce results that cover the rich and contextual details and that could also be generalizable at the same time.

B. Different definitions and understanding of the concept The studies we reviewed referred to different established

definitions of either users’ involvement or users’ participation but most of them were unclear about the concept. They used users’ involvement synonymously with ‘focus on users’, ‘consulting with end users’, contacting with system users’ and ‘participation of users’ [2].

C. Different measurement instrument Another major difference among the studies is the

instrument used for measuring users’ involvement or participation. The use of different constructs and variables for measurement has made it very difficult to consolidate the research findings or perform meta-analysis [4]. According to [44], there are five studies that provide the best measures for users’ involvement in system success: Franz and Robey [45], Baroudi et. al. [46], Robey et al. [47], and Doll and Torkzadeh [48] [49]. We believe that future research should utilize and reuse the experiences reported in these five studies

D. Data collection methods The majority of the studies have used questionnaire

surveys for their data collection followed by interviews. But the procedures they have followed differ significantly from each other on the following basis: Respondents’ sample type (managers, analysts, end users, and developers), size, type of

29

Organizations and projects, phases of SDLC when data was collected, and Instruments for measurement. These differences undoubtedly have a profound impact on the consistency of the results obtained.

E. Types of information systems It has been mentioned in various studies that the types of

systems being developed would create a different context for users’ participation and therefore have different requirements regarding various aspects of users’ involvement [4].

VII. CONCLUSION User involvement in software development requires

resources and careful management. According to various researchers, if the users are effectively involved in early phases of development their involvement may not be required in subsequent phases and hence the time and cost can be reduced. But not much empirical literature is available on the topic to provide a substantial confidence on this belief. In design and implementation phase, user involvement takes a political dimension especially where the users are part of decision making process. And if the user involvement is not carefully handled it can cause issues and problems rather than benefits. Furthermore no guidelines are available for what should be the level and degree of user involvement in requirements analysis (i.e. informative, consultative or participative [33]), and exactly what activities users are required to carry out and which of the methods would be most suitable (interviews, focus group, workshops etc).

Based on the results from 13 studies, user involvement in requirements analysis phase has positive impact on system success. But due to the small number of empirical evidence this claim needs to be further investigated and triangulated. Further research is clearly needed to investigate the relationship of the phases of SDLC that contribute along with users’ involvement to system success. A mix method research strategy should be adopted to study this multi dimensional phenomenon. It would be an illuminating study to compare the old standing concepts of users’ involvement and system success from the last four decades with the modern system development strategies. In particular, the role and extent of users’ participation in agile software development approaches should prove to be a very insightful example of the more modern concept of users’ involvement.

We found Systematic Review to be an effective choice for our exploration where the available literature is abundant. While comparing our results to the traditional literature reviews [1] [4] [6], we could see that their results were not as complete as our SLR due to the lack of rigor of systematic search and selection process as well as undefined search boundaries. Out of our 87 studies, 45 studies that are included in our results were missing in those reviews.

VIII. FUTURE WORK Our SLR on users’ involvement in software development is

the preliminary step towards our empirical inquiry into a specific case context, where we will be evaluating and observing the knowledge we have gathered from the existing

literature of the last thirty years in modern development scenarios. The results we have obtained from this SLR have set the scene and context for what is planned to follow. We have conducted two case studies of users’ involvement in collaboration with software development organizations. We are currently performing qualitative analysis of huge amount of interview and observation data gathered from these two case studies. The aim is to employ Grounded Theory throughout this longitudinal research project in order to increase our understanding of the evolution of requirement knowledge generated by users’ involvement.

REFERENCES [1] B. Ives and M. H. Olson, ‘User involvement and MIS success:

a review of research’, Management science, vol. 30, no. 5, pp. 586–603, 1984.

[2] S. Kujala, ‘User involvement: a review of the benefits and challenges’, Behaviour & information technology, vol. 22, no. 1, pp. 1–16, 2003.

[3] L. Pilat and H. Kaindl, ‘A knowledge management perspective of requirements engineering’, in Research Challenges in Information Science (RCIS), 2011 Fifth International Conference on, 2011, pp. 1–12.

[4] A. L. M. Cavaye, ‘User participation in system development revisited’, Information & Management, vol. 28, no. 5, pp. 311–323, 1995.

[5] M. I. Hwang and R. G. Thorn, ‘The effect of user engagement on system success: a meta-analytical integration of research findings’, Information & Management, vol. 35, no. 4, pp. 229–236, 1999.

[6] J. He and W. R. King, ‘The role of user participation in information systems development: Implications from a meta-analysis’, Journal of Management Information Systems, vol. 25, no. 1, pp. 301–331, 2008.

[7] H. Barki and J. Hartwick, ‘Rethinking the concept of user involvement’, MIS quarterly, vol. 13, no. 1, pp. 53–63, 1989.

[8] J. McKeen, T. Guimaraes, and J. C. Whetherbe, ‘The Relationship Between Participation and User Satisfaction of Four Contigency Factors’, MIS Quarterly, 1992.

[9] E. Carmel, R. D. Whitaker, and J. F. George, ‘PD and joint application design: a transatlantic comparison’, Communications of the ACM, vol. 36, no. 6, pp. 40–48, 1993.

[10] A. Edstrom, ‘User influence and the success of MIS projects: a contingency approach’, Human Relations, vol. 30, no. 7, pp. 589–607, 1977.

[11] W. H. DeLone and E. R. McLean, ‘Information systems success: The quest for the dependent variable’, Information systems research, vol. 3, no. 1, pp. 60–95, 1992.

[12] K. E. Emam, S. Quintin, and N. H. Madhavji, ‘User participation in the requirements engineering process: An empirical study’, Requirements Engineering, vol. 1, no. 1, pp. 4–26, 1996.

[13] M. Bano and D. Zowghi, ‘User involvement in software development and system success: a systematic literature review’, in Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering, Porto de Galinhas, Brazil, 2013, pp. 125–130.

[14] E. Kim and J. Lee, ‘An exploratory contingency model of user participation and MIS use’, Information & Management, vol. 11, no. 2, pp. 87–97, 1986.

[15] R. Palanisamy and J. L. Sushil, ‘User involvement in information systems planning leads to strategic success: an

30

empirical study’, Journal of Services Research, vol. 1, no. 2, pp. 125–157, 2001.

[16] R. W. Zmud, ‘Individual differences and MIS success: A review of the empirical literature’, Management Science, vol. 25, no. 10, pp. 966–979, 1979.

[17] J. Wu and G. M. Marakas, ‘The impact of operational user participation on perceived system implementation success: An empirical investigation’, Journal of Computer Information Systems, vol. 46, no. 5, p. 127, 2006.

[18] B.A. Kitchenham, Guidelines for performing Systematic Literature Reviews in Software Engineering Version 2.3, Keele University and University of Durham, EBSE Technical Report, 2007

[19] S. Kujala 1, ‘Effective user involvement in product development by improving the analysis of user needs’, Behaviour & Information Technology, vol. 27, no. 6, pp. 457–473, 2008.

[20] J. Niès and S. Pelayo, ‘From users involvement to users’ needs understanding: a case study’, International Journal of Medical Informatics, vol. 79, no. 4, pp. e76–e82, 2010.

[21] S. Wilson, M. Bekker, P. Johnson, and H. Johnson, ‘Helping and hindering user involvement—a tale of everyday design’, in Proceedings of the ACM SIGCHI Conference on Human factors in computing systems, 1997, pp. 178–185.

[22] B. Johansson, ‘Diffusion of Open Source ERP Systems Development: How Users Are Involved’, in Governance and Sustainability in Information Systems. Managing the Transfer and Diffusion of IT, Springer, 2011, pp. 188–203.

[23] J. E. Hunton, ‘Involving information system users in defining system requirements: The influence of procedural justice perceptions on user attitudes and performance’, Decision Sciences, vol. 27, no. 4, pp. 647–671, 1996.

[24] S. Pekkola, N. Kaarilahti, and P. Pohjola, ‘Towards formalised end-user participation in information systems development process: bridging the gap between participatory design and ISD methodologies’, in Proceedings of the ninth conference on Participatory design: Expanding boundaries in design-Volume 1, 2006, pp. 21–30.

[25] W. J. Doll and G. Torkzadeh, ‘A discrepancy model of end-user computing involvement’, Management Science, vol. 35, no. 10, pp. 1151–1171, 1989.

[26] D. Robey and D. Farrow, ‘User involvement in information system development: A conflict model and empirical test’, Management Science, vol. 28, no. 1, pp. 73–85, 1982.

[27] J. D. McKeen and T. Guimaraes, ‘Successful strategies for user participation in systems development’, Journal of Management Information Systems, pp. 133–150, 1997.

[28] S. Kujala, M. Kauppinen, L. Lehtola, and T. Kojo, ‘The role of user involvement in requirements quality and project success’, in Requirements Engineering, 2005. Proceedings. 13th IEEE International Conference on, 2005, pp. 75–84.

[29] A. M. K. Baronas and M. R. Louis, ‘Restoring a sense of control during implementation: how user involvement leads to system acceptance’, Mis Quarterly, pp. 111–124, 1988.

[30] E. L. Wagner and G. Piccoli, ‘Moving beyond user participation to achieve successful IS design’, Communications of the ACM, vol. 50, no. 12, pp. 51–55, 2007.

[31] D. Robey and D. Farrow, ‘User involvement in information system development: A conflict model and empirical test’, Management Science, vol. 28, no. 1, pp. 73–85, 1982.

[32] N. Iivari, ‘Enculturation of user involvement in software development organizations- an interpretive case study in the product development context’, in ACM International Conference Proceeding Series, 2004, vol. 82, pp. 287–296.

[33] L. Damodaran, ‘User involvement in the systems design process-a practical guide for users’, Behaviour & Information Technology, vol. 15, no. 6, pp. 363–377, 1996.

[34] R. I. Mann and H. J. Watson, ‘A contingency model for user involvement in DSS development’, MIS Quarterly, pp. 27–38, 1984.

[35] P. Tait and I. Vessey, ‘The effect of user involvement on system success: a contingency approach’, MIS quarterly, vol. 12, no. 1, pp. 91–108, 1988.

[36] M. L. Markus and J. Y. Mao, ‘Participation in development and implementation-updating an old, tired concept for today’s IS contexts’, Journal of the Association for Information Systems, vol. 5, no. 11, p. 1, 2004.

[37] L. Macaulay, ‘Cooperative requirements capture: control room 2000’, in Requirements engineering, 1994, pp. 67–85.

[38] J. Robertson and S. Robertson, ‘Requirements management: a Cinderella story’, Requirements Engineering, vol. 5, no. 2, pp. 134–136, 2000.

[39] I. F. Alexander, ‘A taxonomy of stakeholders: Human roles in system development’, International Journal of Technology and Human Interaction (IJTHI), vol. 1, no. 1, pp. 23–59, 2005.

[40] Ken Eason. 1989. Information Technology and Organizational Change (1st ed.). Taylor & Francis, Inc., Bristol, PA, USA.

[41] H. Sharp, A. Finkelstein, and G. Galal, ‘Stakeholder identification in the requirements engineering process’, in Database and Expert Systems Applications, 1999. Proceedings. Tenth International Workshop on, 1999, pp. 387–391.

[42] T. Heinbokel, S. Sonnentag, M. Frese, W. Stolte, and F. C. Brodbeck, ‘Don’t underestimate the problems of user centredness in software development projectsthere are many!’, Behaviour & Information Technology, vol. 15, no. 4, pp. 226–236, 1996.

[43] M. J. Gallivan and M. Keil, ‘The user–developer communication process: a critical case study’, Information Systems Journal, vol. 13, no. 1, pp. 37–68, 2003.

[44] H. Barki and J. Hartwick, ‘Measuring user participation, user involvement, and user attitude’, Mis Quarterly, pp. 59–82, 1994.

[45] C. R. Franz and D. Robey, ‘ORGANIZATIONAL CONTEXT, USER INVOLVEMENT, AND THE USEFULNESS OF INFORMATION SYSTEMS*’, Decision Sciences, vol. 17, no. 3, pp. 329–356, 2007.

[46] J. J. Baroudi, M. H. Olson, and B. Ives, ‘An empirical study of the impact of user involvement on system usage and information satisfaction’, Communications of the ACM, vol. 29, no. 3, pp. 232–238, 1986.

[47] D. Robey, D. L. Farrow, and C. R. Franz, ‘Group process and conflict in system development’, Management Science, vol. 35, no. 10, pp. 1172–1191, 1989.

[48] G. Torkzadeh and W. J. Doll, ‘The test-retest reliability of user involvement instruments’, Information & management, vol. 26, no. 1, pp. 21–31, 1994.

[49] W. J. Doll and G. Torkzadeh, ‘The measurement of end-user software involvement’, Omega, vol. 18, no. 4, pp. 399–406, 1990.

31