Pitfalls in Analyzing Systems in Organizations
Steven Alter School of Business and Management
University of San Francisco San Francisco, CA 94117
[email protected] 415-422-6383
© Steven Alter, 2003 1
Pitfalls in Analyzing Systems in Organizations
Abstract
Despite the availability of elaborate methods for defining data and business processes, huge amounts of time and effort
are wasted on system projects that produce disappointing results. An important contributing factor is the significant
difficulty business and IT professionals experience when they try to describe, evaluate, and/or analyze systems in
organizations even at a cursory level. Between 1997 and 2003, the author's information system courses for evening
MBAs and EMBAs have required students to write two group papers that present a business-oriented analysis of a real
world system in an organization and propose preliminary recommendations for improvements. If these working students
are representative of the types of business professionals who are involved in systems in organizations, it is plausible that
the major types of pitfalls demonstrated by their papers are representative of the pitfalls that contribute to disappointing
results with systems.
An examination of 202 group papers submitted by evening MBA and EMBA students between 1997 and 2003 revealed
pitfalls in the following categories: difficulty defining systems in organizations, difficulty identifying the information in
information systems, aversion to using measures of performance, reluctance to mention organizational and personal
issues, susceptibility to techno-hype and jargon, inadequate critical thinking, and difficulty applying abstractions and
formal methods. This paper illustrates these problems through examples from 25 of the student papers. Assuming that
typical business professionals encounter the same types of pitfalls, both MBA programs and analysis and design methods
should provide concepts and techniques that help in identifying and minimizing the related problems.
© Steven Alter, 2003 2
1. Introduction
Why is the success rate of system-related projects so abysmal? Every other year the Standish Group publishes a new
study showing that fewer than a third of IT-projects are completely successful and many are complete failures. (e.g. [1])
One frequently encounters claims that a substantial percentage of CRM projects or ERP projects or outsourcing projects
are partial or total failures. New technologies encounter surprisingly long assimilation gaps. [2] Perhaps IS, instead of
economics, should be termed the dismal science.
One of many reasons for these problems is that business professionals are often ineffective in communicating with IT
professionals and in their roles related to identifying system-related problems, determining system requirements, and
implementing systems in their organizations. An important goal of generalist MBA courses in information systems is to
impart knowledge that will help business professionals perform these roles more effectively, but the courses often
become an exercise in learning and using jargon rather than an effective way to learn how to analyze systems from a
business viewpoint. Despite the availability of elaborate methods that IT professionals can use for defining data and
business processes, little attention has been devoted to developing organized methods for business professionals.
Starting around 1992 I decided that my introductory IS courses should address these problems directly by focusing on
how business professionals can think about systems for themselves and by requiring students to write two group papers.
The first paper develops and justifies tentative recommendations for system improvements based on a preliminary
analysis of a real world system in an organization. The second paper focuses on processes through which real world
systems change or are supported. Starting in 1997 I began requesting that students submit electronic versions of their
major papers. In addition to providing a form of feedback about teaching effectiveness, this helped in the continuing
development of the work system method for understanding and analyzing systems from a business viewpoint. [3, 4, 5]
Examples from 25 of the 202 papers will be cited to illustrate different types of systems analysis pitfalls encountered by
these early-career business professionals, most of whom are not IT professionals.
© Steven Alter, 2003 3
2. Source of Data and Method for Data Analysis
To date, 202 student papers have been collected, although an unknown but relatively small number were never received
or have been lost due to a combination of syllabus variations, lack of follow-though when students failed to submit
electronic copies, and the use of five different computers at home and at work between 1997 and 2003. Over 90% of
these group papers concern a real world system in an organization to which at least one team member had access. In
other words, the vast majority of the papers were not attempts to assemble material from the Web or to analyze an
existing case study. The fact that these were group papers had a number of effects. On the one hand, students working
together sometimes clarify each other’s ideas and generate better analyses than would appear in individual papers. On the
other hand, the management of the writing and reviewing process is often haphazard in student teams. In many cases the
final papers do not hang together very well due to a combination of poor writing skills, inadequate review, and
interpersonal conflicts within the teams.
The goal of examining these papers was to identify the major types of problems encountered by business professionals
when they attempt to analyze systems from a business viewpoint after some preliminary training in thinking about
systems in organizations. Although no precise information is available about the amount of business experience of the
authors of specific papers, at least half of the students who submitted these papers had five or more years of business
experience based on their admission to USF’s PMBA or EMBA programs. Less than half of the regular MBA students
had five years of business experience.
The analysis process was informal. Based on experience teaching IS courses for over 15 years I identified what I
believed were the major categories of problems. Many of these problems were apparent in project proposals that the
student teams were required to submit before doing their research. In many cases my detailed feedback about these
proposals helped the teams recognize and avoid likely pitfalls. Although I did not keep records of my responses to the
original proposals, I believe that most of the teams would agree that my initial feedback helped them write more
successful papers.
© Steven Alter, 2003 4
I numbered the papers and then looked at each paper to identify representative examples and to find other types of
problems. Many of the 202 papers were very good and illustrated no obvious pitfalls; some of the others illustrated little
more than inattention and poor writing. The 25 papers that provide the 30 illustrative examples cited here vary in quality,
and were chosen mainly because the examples of pitfalls were understandable and possible to describe briefly. It is likely
that some of the problems would have been more severe had I not provided feedback on proposals to write the papers.
Although students knew I would keep an electronic copy and had been instructed to avoid topics that they were not
willing to discuss in class presentations on these papers, confidentiality concerns dictated that no one else could inspect
or code these papers. The examples cited here are disguised to avoid identifying specific companies. In several cases
system-related terminology was changed to assure that the company cannot be identified.
Because the source data (the papers) was a byproduct of topics and assignments that that were revised continually over
the years, and because examples would provide the best insights about specific pitfalls, no attempt was made to code
subjective observations about the papers or use quantitative procedures that might have been relevant if the source data
had been collected under more controlled circumstances.
3. Major Categories of Pitfalls
This section is the heart of this paper. It presents numerous examples illustrating pitfalls in the following categories:
difficulty defining systems in organizations, difficulty identifying the information in information systems, aversion to
using measures of performance, reluctance to mention organizational and personal issues, susceptibility to techno-hype
and jargon, inadequate critical thinking, and difficulty applying abstractions and formal methods.
It is surely possible that someone else examining the 202 papers would have responded to specific examples differently
and would have found other categories of problems. Nonetheless, the categories and examples should contribute to the IS
field’s understanding of difficulties encountered in analyzing systems, confusions and pitfalls encountered by business
professionals, and expectations about what can or should be learned from MBA courses in information systems. Most
important, this paper should contribute to the discussion of what can be done to help business professionals understand
© Steven Alter, 2003 5
systems in organizations more fully, thereby reducing the impacts of one of the many factors contributing to system
disappointments and failure.
Difficulty Defining Systems in Organizations
Every systems analysis method starts with defining the problem or opportunity and also the system that is being
examined. Although students can easily memorize the steps for analyzing a system and discuss those steps in the
abstract, it is much more challenging for them to use those steps in an open-ended situation that has not been pre-
digested in a case study. My information system courses address this challenge directly because I believe this is the most
important thing MBA and EMBA students who use IT at work every day can learn from an introductory IS course.
These courses emphasize the work system method for understanding systems in organizations. (see [3, 4, 5]). That
method’s underlying premise is that business professionals need a clear understanding of a specific work system (e.g., a
firm’s system for hiring people, finding sales prospects, designing products, manufacturing products, or developing an
annual plan) in order to contribute to the development or improvement of an information system that supports the work
system. Thus, the work system is not the software or the information system. It is the system whose performance is to be
improved as a result of the analysis. In general, the work system should be the smallest work system that exhibits the
problem or opportunity. If it is much larger, the analysis will be unnecessarily complicated. If it is smaller, the analysis
may ignore important issues that must be addressed. The work system method is organized around nine elements that
can be used to describe any work system. The first four, the components of the work system, include the business
process, participants, information, and technology within the system. Five additional elements that round out the basic
understanding of any system in an organization include the products and services produced, the customers, environment,
infrastructure, and strategy. [4]
In many cases, defining the work system is the most difficult and contentious part of the analysis, especially for student
teams with extensive business experience. A number of more experienced student teams have reported spending five or
more hours arguing about the definition of the system before finally settling on a definition, and some of those teams had
to change the definition after gathering more information during the analysis. Common difficulties in defining a work
© Steven Alter, 2003 6
system fall into a number of categories including failure to define the system clearly, viewing technology as the system,
and confusion between the system in operation and the system being built.
Failure to Define the System. Although the work system method specifically requires users to define the system being
analyzed, student teams sometimes bounce back and forth between several overlapping systems without clarifying which
one they believe the analysis is focusing on.
Example - Minimizing unnecessary expenses related to desktop printers: A large financial institution attempting to
control IT-related costs created a system for analyzing printer usage and deciding whether individual desktop printers are
justified. The student paper analyzing this situation failed to establish and maintain a consistent view of what work
system it was considering. At some points, it seemed to be discussing a work system of collecting information about
printers in use. At other points, it seemed to be discussing a system of “rightsizing”, i.e., the system of gathering
information, negotiating with managers, and helping make informed decisions about how many printers are needed. The
lack of clarity about which system was being analyzed led to confusion because these two systems had different scope,
different measures of performance, and in this situation encountered different types of problems.
Example - Tracking pharmaceutical sales representatives: A large pharmaceutical company developed an information
system for tracking physician visits by sales representatives. The student paper was sometimes confusing because it
switched back and forth between discussing the work system of providing information and service to physicians and the
work system of record keeping related to those visits. As with the printer analysis in the previous paragraph, the two
systems had different scope, different customers, different measures of performance, and in this situation encountered
different types of problems. If the work system is about tracking the visits, the measures of performance are about
whether information is gathered efficiently and completely. Whether or not the sales work is done well is beyond the
scope of the analysis. The customer in this case is the sales rep (who will use the information) and sales management. If
the work system is about serving physicians, the measures of performance are about whether the sales work is done
efficiently and successfully, and secondarily whether the information is gathered. The customers in this case are the
physicians, with managers and sales reps as secondary customers for the information that is generated.
© Steven Alter, 2003 7
Example - Sales process for expensive capital equipment: A major manufacturer began to implement what it called the
“sales coordination process” (SCP), which created selling teams consisting of everyone who had contact with certain
large customers. The selling teams were supposed to develop and track coordinated action plans. The company’s IT
group developed the “SCP network,” which was to be used to store all of the information relevant to a selling team’s
activities. The student paper had difficulty separating the effectiveness of SCP from the effectiveness of the SCP
network. Although the SCP effort had made some progress in getting teams to share information, less than a third of the
teams had produced action plans and only half of those were current. This problem was attributed to technical
shortcomings of the SCP network, although it certainly seemed possible to develop the plans outside of the network.
Example - Role of technology in an MBA program: A student paper viewed an MBA program as “a complex work
system that has in recent years become more reliant on technology as a tool to support information and aid in the
teaching practice.” The analysis looked at three representative separate courses and how technology was used in each.
The courses were unrelated and the use of technology in the courses was unrelated. The result was a set of separate
stories but no real system view of the situation. In contrast, it might have been possible to establish a work system view
of a typical student’s MBA program and then show how consistent, interrelated use of technology within various courses
could contribute to an overall experience. Alternatively, it might have been possible to establish a work system view of
the entire teaching effort.
Thinking of the Technology as the System. An important reason for using the work system approach is to recognize
that from a business viewpoint the headline is the system of doing the work rather than the technology that is being used.
In too many cases, systems that use IT are called IT systems. (e.g., “The IT System that Couldn’t Deliver,” an HBR case
study [6] discussed in [7].)
Example - Data warehouse in a finance department: A student paper focused on a data warehouse acquired to provide a
finance department and other departments convenient access to operational and accounting information captured in an
ERP system and several other information systems. The paper said the steps in the business process included extracting
data from the ERP system, filtering and aggregating the data, copying data from other information systems, and using the
© Steven Alter, 2003 8
information for financial analysis. The paper described how the technology was acquired and configured but only rarely
used in any significant way. The analysis would have been much more effective if it had started from specific finance
work systems such as closing the books each month and performing specific types of financial analysis. Starting from the
goal of improving those work systems would have led directly to identification of specific shortcomings of the data
warehouse that could have been fixed by changing its configuration. Focusing on the data warehouse as the system left
the link between finance work and operational problems in the data warehouse unclear; it was difficult to decide whether
the recommendation would solve anything, or whether the result would be a more efficient data warehouse that still
would not be used very much.
Example - Lead generation system in a software firm: A student paper attempted to analyze how a software firm’s sales
department could improve a lead generation process that used CRM software. The paper implied that the CRM software
was the system being studied. The analysis cited statistics and projections showing that the current lead generation effort
was insufficient to support the company’s sales goals. It spoke of burnout among telemarketers and argued that the lead
generation team needed better tools and information and that they lacked training. Because the work system was never
defined, it was never clear which of the following possible issues were within the scope of the system: the group
answering incoming inquiries doesn’t know how to do inbound sales; the telemarketers don’t know how to do outbound
sales; nobody knows how to use the CRM software properly; account execs (who receive the leads) don’t know how to
evaluate whether they are properly qualified or don’t know how to sell. A more effective analysis would have identified
steps in the business process and would have discussed separate performance issues for the various parts of the work
system. Based on responses to questions in the class presentation it seemed likely that some of the problems were related
to the CRM software while other parts were related to business process, personnel, and training issues.
Example - Manufacturing system at a food processing company. A food processing company uses an outdated software
product called manufacturing control system (MCS). The group paper analyzing various aspects of the company’s
manufacturing summarized the business process as follows: generating the sales forecast, recording raw material usage,
recording manufacturing yields, recording packaging completions, recording inventory data in MCS, ordering raw
material, and recording raw material receipts. In reference to raw material management it noted that the process is time
consuming, the system is inflexible (because it provides results only once a week), the system is costly (because greater
© Steven Alter, 2003 9
use of automated manufacturing would be cheaper), and the system is not reliable (because manual inventory counting is
error-prone). Unfortunately, the analysis uses the word “system” to denote both the MCS software and the system of
doing manufacturing. The analysis is ambiguous because it is sometimes unclear whether the topic is the manufacturing
software or the manufacturing.
Example - Brokerage firm’s network for financial consultants: Financial consultants working for an international
brokerage firm accessed client account information, sales material, research information, and email through customized
work stations. As the consultants became more mobile, the need to make the workstation mobile increased. The student
paper viewed the work station as the work system being studied, as revealed by describing the consultants as customers
and the business process as logging on, accessing information, and logging off. With such a limited view of the work
system, the recommendations said that technical glitches in the workstation software should be fixed but couldn’t offer
other ideas. A broader view of the way increasingly mobile financial consultants provide services for customers might
have led to a much more valuable recommendations about improving consultant efficiency and/or effectiveness.
Confusion between the System in Operation and the System Being Built. Over the years, a number of proposed
papers contained confusions about the difference between a system in operation and a system being built. This type of
confusion is revealed through business processes that include some steps about building and/or maintaining a technical
system and other steps related to using the technical system to accomplish business tasks. Both are important, but when
these separate activities are treated as a single work system the measures of performance become confused and the
recommendations may include anything from improvements in processes for doing technical system maintenance work
through improvements in performing customer-facing processes. In most cases, feedback about project proposals
clarified this issue for students, but in one paper the issue remained:
Example - Providing Web access to existing industry magazines: The analysis concerned a strategy decision related to
how a magazine publisher could use the Web to serve the customers that have previously been served using paper
periodicals. The analysis did not use system ideas effectively because it combined two systems with totally different
operational issues, measures of performance, and participants. The confusion was apparent from the way the business
© Steven Alter, 2003 10
process and participants were summarized. The business process included two parts. First, the magazine’s editorial and
marketing staff collected information, wrote product reviews, and generated revenue through sponsorships, advertising,
and subscriptions. Second, Web site users logged on, found information related to products and other aspects of their
industries, and in some instances made purchases. Perhaps because the paper combined two systems into one, it did not
discuss measures of performance and focused primarily on goals that the publisher might pursue. A more effective
analysis might have viewed the customer’s use of the Web site as a work system, and might have been more specific
about how the customer’s efforts to obtain value could be evaluated and improved. For example, that analysis might have
asked whether the customers actually want the types of information provided in magazines and whether some form of
genuine computer interaction, as opposed to data retrieval, might provide more value for some of the products reviewed
in the magazines.
Difficulty Identifying the Information in an Information System
It might seem surprising to specialists in information systems, but many of the MBA and EMBA students who wrote
these papers seemed to have relatively little propensity or ability to identify the information used or produced by the
systems they analyzed. Despite having seen demos of relational databases and despite having done several in-class
exercises involving the use of ERDs to help in specifying data requirements, many of the students seemed satisfied with
vague statements such as the information is outdated, historic data is required, and the data in the system is
manufacturing data.
Example - Financial analysis at a clothing manufacturer: The finance department of a clothing manufacturer
consolidated sales forecasts from various regions to produce a combined sales forecast for the firm. The student analysis
attempted to suggest ways to do this work more efficiently. However, from the analysis it was unclear why the forecasts
were being produced and whether the forecasts were at the SKU level or at some aggregated level. My comments to the
student team included the following: “Many of the confusions in reading the paper might have been cleared up if it had
said something like the following: ….. The forecast is by product line and extends five months into the future by month.
There are 27 product lines and the each product line is sold in 14 regions. The job Finance does is to consolidate
forecasts from each of the regions in order to create revenue forecasts for the next five months. These forecasts have
© Steven Alter, 2003 11
nothing to do with determining exactly what will be produced by SKU in the next five months because that was
determined in the annual production planning cycle. ……. The previous four or five sentences may have been
completely wrong, but something of that general form would have been a starting point toward providing enough clarity
about what the system is trying to do and why your recommendation would really create an improvement.”
Example - Production planning at a biotech manufacturing firm: A biotech manufacturing firm attempting to use an
MRP system encountered a variety of problems related to capacity planning, cost-effective purchasing, shop floor
control, and timely delivery. Among other things, the analysis called for the use of historic machine hours, scrap levels,
material consumption, cycle times and efficiencies for each product. Compared to recommendations about information
requirements in other papers, this recommendation was more specific about the types of information required. However,
with the large number of products the company produces, steep learning curves, and lumpy demand, it is not clear
whether this recommendation would truly solve the company’s problems, especially since another part of the paper says
that production orders are often generated months before they are needed. The paper did not explain whether the
recommended information currently existed in any form and how much effort, and by whom, would be needed to capture
and store it. Furthermore, the heart of the problem may have been the lack of information for several high volume
products rather than for the thousands of products the company occasionally produced.
Aversion to Using Measures of Performance
MBA and EMBA programs include extensive coverage of managerial and financial accounting, the need for
performance measurement in process improvement, and the use of balanced scorecards. Nonetheless, over the years most
student papers seemed to reflect a strong aversion to mentioning measures of performance. Even when the instructions
invited students to estimate performance indicators that were not readily available or to note that seemingly proprietary
performance data had been modified for purposes of writing a paper, students often omitted that information and
attempted to justify recommendations based totally on qualitative concerns and vague generalizations. I eventually began
to require that student papers include several tables listing estimated or actual values of important performance measures
and estimating the extent to which their recommendations would affect these values.
© Steven Alter, 2003 12
Example - Tracking of demonstration inventory for electronic equipment: A manufacturer of electronic equipment
provided demonstration models to customers but didn’t track the demonstration models effectively. The demonstration
models were often shipped through standard shipping channels without a designation that these were on loan and were
not being sold. This generated inaccurate sales, inventory, and financial data used by other information systems.
Although the analysis provided an estimate of the amount of incorrectly classified inventory, the analysis did not present
measures of performance describing other aspects of the problem, such as the amount of time and effort that is required
for reconciliations, the positive or negative impact of providing so much demonstration equipment, and any impacts on
real revenues and commissions. More attention to measures of performance in other areas might have led to a powerful
recommendation, such as a system that would charge the sales organization for the demonstration equipment. Instead,
the student team’s recommendation seemed rather toothless. It proposed creating an Equipment Request Form (ERF), a
written procedure for filling out the ERF, in-service training on the formal procedure, and education about the need for
the process.
Example - Product development at a food manufacturer: A student paper analyzed the product development process at a
food manufacturer that creates many new products each year. The paper produced the bland recommendation that a
committee should look at the product development process more closely. My feedback to the team included, “The paper
described a current product development system, but did not provide a clear statement of a problem or opportunity that
was to be analyzed. It also seemed to pull its punches on how well the product development system is operating. For
example, it might have said that developing a new product currently costs $X on average and requires Y hours of work
on average, and that these numbers could be improved to the following degree.... if we made these specific changes
…..”
Example - Reconciling local and national reservation data: Five years ago a major hotel chain had a national reservation
system and local reservation systems controlled by each of its properties. The data from the national reservation system
was downloaded periodically to the local reservation systems, which were characterized as electronic replicas of the
paper-based reservation systems that the local properties used previously. Not surprisingly, inconsistencies were
common because reservations could also be created or modified at the local properties, especially when guests left early
or extended their visits. The student team analyzed the situation and recommended that an integrated system be
© Steven Alter, 2003 13
developed (which was happening at the time). The paper mentioned that discrepancies as large as 20% sometimes
occurred, requiring as many as 300 reconciliations in a day between the local and national systems. Although the 20%
and 300 reconciliations per day represented measures of performance, the analysis never gave a clear idea of the
magnitude of the problem or of the number of people who are really involved in correcting the inconsistencies. Other
measures of performance that might have been relevant to the analysis and the justification of recommendations included
the estimated number of guests who were inconvenienced when their reservations could not be filled, the number of
rooms that went unused unnecessarily, and the amount of clerical time required to do the reconciliations.
Example - Production planning at a biotech manufacturing firm: A biotech manufacturing firm encountered a variety of
problems related to capacity planning, cost-effective purchasing, and timely delivery (mentioned earlier). Although the
paper mentioned many problems, it made no effort to quantify the extent of the problems. For example, capacity
planning was a problem, but it did not mention the extent of delays or lost orders that might have been reduced with
better capacity planning. Incorrect MRP parameters were cited as a problem, but there was no estimate of how
inaccurate the parameters were. Production planning was frustrating and time consuming due to incorrect MRP
parameters and other problems, but no measures of performance for the efficiency or quality of planning were cited.
Example - Lead generation system in a software firm: A previously mentioned student paper attempted to analyze how a
software firm’s sales department could improve a lead generation process that used CRM software. Although the paper
discussed the company’s sales goals and estimated number of leads that would be required to meet those goals, it was
quite imprecise about performance measures for the lead generation process. For example, it said that inbound phone
calls represent the bulk of the leads entering the system, not 60% or 95%, but “the bulk” of the leads. Similarly, it said
that the leads were “often inaccurate” but did not estimate what percentage of the leads were totally non-qualified and
what percentage over-estimated or under-estimated the number of seats that might be purchased. Greater specificity
about performance measures such as these would have helped in evaluating the validity of the recommendations.
© Steven Alter, 2003 14
Reluctance to Mention Organizational and Personal Issues
Although work systems rely on the human participants who do the work, MBA and EMBA students analyzing systems
tend to emphasize technology and business process problems when trying to explain performance issues and recommend
directions for performance improvement. As illustrated by the following examples, they often seem reluctant to mention
organizational and personal issues.
Example - Travel reservations at a large consulting company: A student team analyzed an ongoing change in the travel
reservations system used by a large consulting company. The goal was to provide a Web-like interface that would
encourage consultants to make reservations themselves, thereby eliminating contractors who served as an in-house travel
agency. Three years after implementation, new online capabilities were being used extensively by around 50% of
younger consultants and less than 10% of senior consultants. The analysis mentioned the relative age and billing rates of
different groups of consultants, and it called for additional encouragement and training. It did not observe that these low
adoption rates after three years indicate that the consultants did not want to use the new system and would continue
avoiding its use until it became compulsory.
Example - Tracking pharmaceutical sales representatives: A large pharmaceutical company developed an information
system for tracking physician visits by sales representatives. The student paper mentioned earlier discussed the need to
do sales work, to keep managers informed, and to do record keeping so that successor sales people would start with
usable customer information. Not mentioned in the paper was the fact that many of the sales representatives resented
spending time entering the data because they believed they were sales representatives, not data entry clerks. The
discussion following the presentation also revealed that the information entered about particular physician visits was
often fudged. To avoid annoying a physician struggling through an overloaded day, a representative might do little more
than say hello, but would still record the visit so that the information transmitted to management would look good.
Example - Time reporting system at a professional services firm: A group paper observed that the firm’s employees “are
more creatively than financially oriented. They … do not naturally understand or appreciate the importance of time
reporting nor are they held accountable for accurate time reporting. The challenge is to implement a change
© Steven Alter, 2003 15
management process that educates and creates incentives for these employees so that they gain an appreciation and
understanding of the financial importance to the business of accurate data.” Other possible interpretations are that the
company’s management is not aware of the problem or that it views accurate time reporting as far less important than
keeping customers and valued staff happy.
Example - Supporting inexperienced insurance agents: A group paper analyzing systems that support inexperienced
insurance agents noted that many new agents fail because they run out of money before producing enough new business.
The paper’s recommendations included increasing the base salary for the first few years, providing better technology,
and providing mentoring on selling and on the general challenges faced by insurance sales people. Not mentioned in the
paper, but discussed following the presentation, was the possibility that the company’s strategy was to put the new
agents on the street, see whether they could sell, and devote more resources to them only after they proved themselves.
Susceptibility to Techno-hype and Jargon
Even after the Internet bubble burst, the IS field remains rife with techno-hype and jargon. Terms such as CRM, ERP,
EAI, and data mining all set up expectations that may be unrealistic. In too many situations, textbooks do little to
puncture inflated hopes. To the contrary, they sometimes promote images that barely fit current reality. Despite my
repeated attempts to warn students about the exaggerated expectations that techno-hype and jargon sometimes engender,
a number of papers encountered problems in this area.
Example - Use of “extensible knowledge blocks” in a documentation department: A large electronic manufacturer has a
documentation department that manages contractors who write and revise technical manuals. To minimize costs,
department managers adopted the strategy of using “extensible knowledge blocks” (EKBs), electronic blocks of images
and text that could be reused and extended from one product revision to the next. Contractors were rewriting the manuals
from scratch and the EKB system was barely used. The student paper recommended that the company should set aside
additional funds for contractors to encourage them to find the relevant EKBs in a proprietary company database that had
not been made available to them. The funds for searching were necessary because the EKB database contained no
indexing or cross-reference information, making it necessary to scan through the database manually to find the EKBs
© Steven Alter, 2003 16
that were relevant to any particular new manual. My impression was that the oversold EKB concept had seduced the
student team. The real problem seemed to be the lack of a convenient way for the contractors to obtain the electronic
versions of the existing manuals. EKBs may have been an elegant approach at one time, but a feasible implementation of
that idea had not been created. Instead of focusing on the use of EKBs, the students would have done much better to
simply ask how the work system of producing those manuals might be improved.
Example - Mobile wireless panacea for medical care: A student paper argued that mobile wireless technology and PDAs
would streamline patient care in hospitals, would drastically reduce the amount of time nurses must devote to record
keeping, would reduce error rates, and would improve communication. Although some version of these benefits will
surely occur to some extent, the students seemed to be swept away with hype about a wireless future. Their paper
mentioned “being able to electronically access data anytime, anywhere.” It claimed, “there is little end-user training
required to use this technology.” It stated that the use of “standard operating systems like Microsoft’s Windows 2000” on
laptops “reduces and, in most cases, eliminates any custom application or integration development needed to access
existing network applications.” The paper ignored many difficult issues that bedevil American medical care, including
lack of nurses, difficulties with the details of different insurance schemes, ambiguities in coding of information about
medical conditions and treatments, difficulties integrating information and processes across different departments,
unwillingness of doctors and others to change traditional work practices, difficulties with privacy and security, and
difficulties dealing with downtime and bugs if all patient information is computerized.
Example - Six Sigma: A student paper tried to explain how a six sigma process including six sigma green belt and black
belt certification for employees could be used for building information systems and applied elsewhere. Although some
six sigma ideas are useful in managing projects, one might wonder whether the students understood its applicability at a
deeper level than that of slogans. The paper says, “Six Sigma is a highly disciplined process that helps businesses focus
on developing and delivering near-perfect products and services. Sigma is a statistical term that measures how far a
given process deviates from perfection. A Six Sigma process translates into approximately 3.4 defects per 1 million
opportunities.” I doubt that any IT groups, even those at capability mature model (CMM) level 5, have attained an error
rate of 3.4 per million lines, and if they have attained it, at what cost. CMM was discussed in the class, but somehow the
promise of six sigma seemed to overwhelm the discussion of CMM costs.
© Steven Alter, 2003 17
Inadequate Critical Thinking
Although some student papers make cogent arguments that use concepts and information effectively, over the years
many of the shortcomings of other student papers seemed as much due to inadequate critical thinking as to inadequate
understanding of IS concepts. This is a difficult issue at the MBA and EMBA level because the students are college
graduates who presumably have learned about critical thinking and careful writing in their high school and
undergraduate work. Unfortunately, wishful thinking doesn’t solve this widespread shortcoming in educational
background. Table 1 lists common types of pitfalls related to various aspects of critical thinking. The examples following
Table 1 illustrate how these pitfalls appeared in several papers.
Table 1: Common problems related to critical thinking
Aspect of critical thinking
Common pitfalls
Defining the problem, issue, or question
Failure to clarify exactly what problem or opportunity is being pursued. Example: “We will analyze system X” [but won’t start with a problem or opportunity that guides our analysis.]
Identifying assumptions, values, and point of view
Arguing based on values and opinions without being aware that information is lacking. Example: “People need to have upbeat personalities in order to contribute.” [but some people with pessimistic personalities may also contribute]
Gathering information and evidence
Failure to gather or consider important information. Example: “The sales training was a great success” [but sales results remained inadequate]
Evaluating the quality of the evidence
Repeating quotations from individuals or published material without evaluating the source or the underlying incentives. Example: “The company produces the highest quality products at the lowest costs.” [The CEO may have said so, but that doesn’t make it true.]
Using concepts, and frameworks to shape the information and evidence
Recitation of facts without shaping them in a way that helps in interpretation. Example: “The process has 47 steps. Step 1 is …. “ [without listing every step one can say that the process is over-structured, participants have no autonomy, and management is complacent]
Drawing inferences and interpretations
Failure to explain the significance of information presented. Example: “Company X has a tradition of excellent service.” [but information presented seems to contradict that tradition]
Searching for alternatives and possibilities
Arguing for one option without ever mentioning other possibilities. Example: “We recommend that company X buy an ERP system.” [but we seem not to have considered less expensive measures that might be more practical]
Drawing conclusions or recommendations
Substituting opinions for carefully supported recommendations Example: “We believe the manager should be replaced.’ [but we haven’t explained exactly why the facts justify that conclusion]
Presenting a complete argument
Ignoring factors that most readers might wonder about. Example: “The error rate is 22%.” [and it is also noteworthy that the system of doing
© Steven Alter, 2003 18
the work is highly error-prone.] Avoiding internal contradictions
Saying X in one location and saying something that contradicts X somewhere else. Example: The organization’s greatest resource is its people, but the resistance to the new information system sometimes resembles sabotage. [if the people are so great, perhaps they could have negotiated instead of sabotaged]
Reducing a software firm’s travel and entertainment budget: A student team noted that a software firm’s T&E
expenditures were too high and suggested that the firm move from one travel service provider to another, thereby
reducing those expenses 35% to 50%. Although the paper mentioned some shortcomings of the current travel services,
there was no analysis of specific T&E expenses and no justification that these expenses would drop by any particular
amount. For example, if the root cause was that salespeople were taking too many trips or that they were choosing
expensive travel options, merely moving to a new travel service provider without creating new internal controls might
make little difference.
Time reporting system at a professional services firm: A student paper presented internally inconsistent views of whether
that system was successful. My comments about the paper questioned its logic, saying, “On the one hand, the
information gathered is very important for billing and for long-term revenue growth. Senior management likes the
system, thinks the information is important, uses the information, etc. On the other hand, people who enter the
information fake it at least to some extent. The degree of the faking is not estimated. What is the basis for [the paper’s
statements that] the system operates at “an impressive level?” What is so impressive? What are the implications about
the degree to which senior management really cares about this information? Without direct knowledge of the situation, it
sounds to me as though they care very little about it and are quite willing to live with faked information as long as the
faking is within reason and within budget, and as long as the company is able to produce high quality results and
maintain client relationships.”
Minimizing unnecessary expenses related to desktop printers: A large financial institution attempting to control its IT-
related costs created a system for analyzing printer usage and deciding which type of printer is appropriate in any
particular situation. The student group recommended taking printers away from non-managers but the only data
presented was that the company had too many printers in relation to the number of pages it printed. A convincing
justification would have shown that savings from reducing the number of printers would have been greater than the
© Steven Alter, 2003 19
additional cost of doing work less efficiently due to the need to share printers. Furthermore, in many cases the non-
managers probably needed the printers more than managers. Allocating printers based on status and rank might have
been politically expedient, but the paper did not explain why that would attain the best business results.
Producing non-standard reports for state insurance commissions: An otherwise well argued paper about a national
insurance company’s internal inefficiencies in generating certain reports stated that “agent listings are consistently
defective with 22% to 38% wrong information in each listing.” Given that this was a regulatory compliance situation
with the data being cross-checked by the states, such a high error rate seemed implausible. More explanation to describe
the nature of the errors and what is done to fix the errors before the data is submitted to the states would have clarified
what was meant and would have eliminated the suspicion that something was wrong with the analysis.
Web-based clothing retailer: The retailer had roughly 100 orders per month and a total of $50K of sales during the
previous year, but the student paper described the retailer’s market as millions of people who use the Web. It also said
that the retailer offered thousands of semi-customized product options, more than any other retailer in the same product
category within clothing. Much of the analysis seemed disconnected from reality. Almost none of the millions of Web
users knew the Web site existed, and the fact that the retailer’s competitors did not offer as many options might be
related to the impracticality of providing quick service without maintaining an excessively large inventory of clothing in
a pre-customized state. Overall, the paper provided no convincing arguments that this company had a sustainable,
profitable business model.
Suggestion that consumer-oriented agencies offer Web-based purchasing: A paper from 1997 recommended that travel
agents provide better service to their customers by offering Web-based, ATM-like kiosks that the customers could use to
cut the time to make a reservation from 53 minutes using a conventional travel agent to 3 minutes using an ATM-like
device. My response questioned how they students arrived at the 3-minute estimate and asked why the travel agents
don’t use this type of device if it could provide such a dramatic time saving. And if this type of tool were available
through the Web, why would travelers need the travel agents?
© Steven Alter, 2003 20
Discussion of CRM systems: Instead of analyzing a system in a particular organization, a student group wanted to write
a paper about CRM systems. At various points the paper stated that:
-- CRM is “an information technology that allows companies to manage every customer request / need in one integrated
platform.”
-- CRM is “a way to consolidate all isolated IT systems, so that they can communicate with each other.”
-- CRM is “about bringing together different areas of an organization to gain insight into customer behavior and value.”
-- CRM is “an ongoing dialogue between a company and its customers.”
-- CRM is “any customer-centric automated process that holds the customer as the cornerstone of the firm.”
Although the paper cited several long cases and many sources from the Web, some of which apparently supplied the
above statements, it did not present a good analysis of CRM because it never defined what it meant by CRM.
Difficulty Applying Abstractions and Formal Methods
Many of the topics covered in business courses related to systems are abstractions based on or related to the concepts of
system, information, and process, which themselves are abstractions. Formal methods such as data flow diagrams and
entity relationship diagrams involve the creation of abstractions using particular symbols and rules. Although abstraction
and formal methods are two of the fundamental ideas of computer science, teaching abstractions and formal methods to
non-technical MBAs and EMBAs often takes a surprising amount of time and effort. The many possible reasons for
these difficulties include lack of appreciation or interest in formal methods and a propensity to prefer concrete examples
rather than abstractions (based on the Myers-Briggs psychological type of the majority of MBA students).
An example of a comparatively simple abstraction is the work system framework that forms the basis of the work system
method, various versions of which were used in the student papers. The first step in using the work system method is to
summarize the situation in terms of the nine elements. The substantial challenges of defining a system were mentioned
earlier, but even applications of the individual terms in the model are sometimes problematic. In a previously mentioned
student paper about pharmaceutical sales, the students identified physicians as the customers of the work system but
didn’t mention the sales managers as customers even though it was sales managers who benefited most directly from a
somewhat controversial information system for collecting data about physician visits. Similarly, an analysis of how a
© Steven Alter, 2003 21
small manufacturer granted credit to its customers identified commercial credit reports as the product of the system.
Those commercial credit reports were the products of organizations in the business of producing those reports, but for
the work system of granting credit, the credit reports should be treated as information used by the business process.
Formal methods that use precise notations present a deeper form of abstraction because the user is expected to apply
particular concepts to create an abstraction and then document the abstraction using a particular “grammar.” [8] A
number of researchers have studied the difficulties related to using ERDs and other conceptual modeling methods. [8].
In introductory IS courses most MBAs and EMBAs quickly grasp the way an ERD describes the logical structure of a
relational database, but without substantial practice they often have difficulty creating meaningful ERDs. The earlier
group papers in the sample contained at least five examples of DFDs and ERDs that demonstrated a poor understanding
of what the tools were attempting to accomplish. Around four years ago I decided to cover ERDs in database examples
and classroom discussions but to stop asking the students to produce ERDs in their group papers because attaining good
results would have required three or four times the amount of class time that I believed the topic deserved.
4. Discussion and Conclusions
This paper has cited 30 examples that illustrate various types of pitfalls encountered by teams of evening MBA and
EMBA students attempting to analyze real world systems in the organizations they work in. At least half the students
involved in these papers had five or more years of business experience and are therefore reasonably representative of the
types of individuals who may need to make decisions about systems in organizations or who may serve as user
representatives on committees devoted to building or maintaining systems in organizations. As a representation of the
categories and extent of the difficulties that typical business professionals encounter the examples presented here are
actually just the tip of the iceberg. All of these examples come from students who are ambitious enough to study for an
MBA, who are part way through a course in information systems, and who have received some form of feedback about
likely problems related to a proposed topic for a group paper. Typical business professionals attempting to understand a
system by themselves or attempting to collaborate with IT professionals do not have the same advantages, but will still
have to deal with the same pitfalls and perhaps others.
© Steven Alter, 2003 22
The examination of 202 papers began with a set of categories for pitfalls, and examples in all of the categories were
found. The specific categories included difficulty defining systems in organizations, difficulty identifying the
information in an information system, difficulty using measures of performance, reluctance to mention organizational
and personal issues, susceptibility to techno-hype and jargon, inadequate critical thinking, and difficulty applying
abstractions and formal methods. From the outset I was certain that I had encountered all of these difficulties in the
original proposals for topics, but I was not sure whether my feedback to the teams about likely problems would filter out
certain categories of pitfalls that therefore would not appear in the final papers. Pitfalls in each category did appear, but
my impression is that the frequency and severity of the pitfalls was somewhat reduced. Notice how only some of the
pitfalls are addressed either in typical information system courses or in systems analysis and design texts.
Difficulty defining systems in organizations: Systems analysis authors almost always say that it is necessary to define a
system before analyzing it, but this is easier said than done. It is interesting that the most experienced student teams
tended to report longer and more contentious debates about the identity and boundaries of the system they should analyze
in order to produce recommendations related to a specific real world problem or opportunity. The less experienced
student teams sometimes seemed not to realize why this was an important issue. In both cases, as the student teams
progressed with the analysis they often found that the original definition of the system was inadequate. In some cases
they even redefined the problem they were trying to solve.
Difficulty identifying the information in an information system: Information system courses and systems analysis
methods also place substantial emphasis on databases and data definitions. The findings from this study indicate that
MBA and EMBA students who can have seen demonstrations of relational databases may still tend to be vague about
data requirements and may have difficulty using ERDs to describe the data requirements of a new situation.
Difficulty using measures of performance, reluctance to mention organizational and personal issues, and susceptibility
to techno-hype and jargon: Information system courses and systems analysis methods often do not put much emphasis
on measures of performance, full engagement with human and organizational issues, and vulnerability to techno-hype
and jargon. Formal analysis methods for IS provide rigorous techniques for defining data but provide sketchy guidance
or none at all for dealing with human and organizational issues. Information systems courses and systems analysis
© Steven Alter, 2003 23
methods usually promote the use of measures of performance, but guidance about which measures of performance to
consider is often sketchy at best. (In contrast, the work system method used by the students who wrote these papers
encourages them to consider at least several measures of performance for each work system element.) Information
system courses may or may not warn students about techno-hype and the excesses of jargon; in some cases the courses
themselves seem to promote techno-hype by implying that the newest and greatest technology is important for success or
that the ability to use fancy jargon is impressive.
Inadequate critical thinking and difficulty using abstractions: Issues about critical thinking and difficulty in using
abstractions and formal methods are especially troublesome because they fall outside of the purview of the IS field, yet
are a major determinant of whether students produce logical papers with well-supported recommendations. There are
times when I wonder whether half or more of the quality of most papers is the result of good critical thinking rather than
of mastery of specific system-related topics. It is easy for professors to avoid this issue by grading through tests or other
exercises that have a demonstrably correct answer and therefore do not exercise critical thinking to the same extent. I
take my chances with relatively open-ended assignments because I believe these assignments present students with a
more realistic replica of the challenges they will face in their jobs.
Implications for Action
Assuming that the categories of pitfalls illustrated here are common, what should be done about them? The first thing to
do is to make sure that typical MBA courses about information systems address these issues. Table 2 identifies some of
the possible approaches that can be used in relation to each area of difficulty. Some of these suggestions may seem a bit
basic, and may not seem as exciting as talking about the strategic uses of information technology, the impacts of
technology on society, or the new hot technologies that may become commonplace soon. On the other hand, focusing on
basic topics such these may help the students acquire or extend what is probably most valuable for them, an ability to
participate fully in the analysis, design, implementation, and operation of systems in organizations.
Table 2: How typical MBA courses in information systems might address common pitfalls
Category of pitfall How an MBA course might address this pitfall Difficulty defining systems in organizations
• Provide exercises in defining systems. For example, give the students a situation and a problem or opportunity. Ask them to identify two possible views of what the
© Steven Alter, 2003 24
relevant system is, and to explain why one view might be preferred. • Have the students write real world case studies as projects.
Difficulty identifying the information in an information system
• Provide examples of relational databases within Microsoft Access or another DBMS. View the ERD. • Allow the students to perform transactions to modify the data on pre-existing database examples; perform queries and ask the students about the role of the database’s structure
Aversion to using measures of performance
• Provide examples of situations in which many different measures of performance might apply. Ask the students to identify several plausible measures of performance for different elements of a system, and then compare the answers. • Ask the students to identify three or more important measures of performance for work systems they have encountered in their lives.
Reluctance to mention organizational and personal issues
• Provide examples that illustrate why systems in organizations are more than just technology, information, and business process. • Provide numerous examples in which system participants might have personal incentives contrary to the goals of management • Ask the students to identify systems they encountered in which system participants might have personal incentives contrary to the goals of management
Susceptibility to techno-hype and jargon
• Provide examples showing that the same jargon term can mean many different things to different people at different times. • Provide examples that make fun of overblown jargon. One I use is called “Hyper-competitive global empowerment-speak.”
Inadequate critical thinking
• Present examples such as those in Table 1 and encourage students to think about those examples as they write their papers • Provide a refresher about critical thinking at the beginning of an MBA program or in a communication class.
Difficulty applying abstractions and formal methods
• Demonstrate abstractions and formal methods using realistic examples • Ask students to criticize examples of DFDs or ERDs that may contain incorrect descriptions (e.g., is the 1:1 relationship between professors and offices correct?) • Consider the possibility that most business professionals will not find highly abstract methods very useful in their own work.
The individual suggestions in Table 2 address particular issues but do not directly address the larger question of how
business professionals should be able to avoid these pitfalls and should be able to understand systems more completely,
perform a preliminary analysis of a system by themselves, and communicate more effectively with IT professionals.
Each of these topics should be the subject of future research because the stakes are so high in terms the unacceptable rate
of system-related disappointments and system failure.
The underlying question might be viewed as a methods and quality control issue. Assume that business professionals do
not have the types of training that IT professionals have. Assume that they are not uniformly skilled in critical thinking.
Assume that they have many unrealistic beliefs about technology. Even with all of these assumptions it might be possible
to help business professionals by providing organized methods that guide them toward a better understanding and better
analysis. Such methods might incorporate paper or computer-based tools for producing preliminary definitions of
© Steven Alter, 2003 25
systems. They might incorporate system principles in a way that would help in identifying issues and in recognizing that
changes in one part of a system could affect other parts of the system. They might incorporate system development and
system implementation principles that would help in identify likely pitfalls in the process of creating or improving a
system. Whether business professionals (or MBA or EMBA students) would seek such knowledge or methods is
debatable. However, given the importance of systems in organizations, the unacceptable rates of disappointment and
failure, and the pitfalls that business professionals currently encounter, progress in these directions is of great
importance.
References
[1] Johnson, J. K.D. Boucher, K. Connors, and J. Robinson (2001) “Collaborating on Project Success,” Software Magazine, Sponsored Supplement, February 2001, http://www.softwaremag.com/L.cfm?Doc=archive/2001feb/CollaborativeMgt.html viewed on May 6, 2003 [2] Fichman, R.G. and C. F. Kemerer (1999) “The Illusory Diffusion of Innovation: An Examination of Assimilation Gaps,” Information Systems Research, (10:3), Sept. 1999, 255-275. [3] Alter, S. (2002a) Information Systems: Foundation of E-Business, 4th ed., Upper Saddle River, NJ: Prentice-Hall, 2002. [4] Alter, S. (2002b) “The Work System Method for Understanding Information Systems and Information Systems Research”, Communications of the AIS (9:6), pp. 90-104 [5] Alter, S. (2002c) “The Collaboration Triangle”, CIO Insight, pp. 21-26. [6] Reimus, B. (1997) “The IT System that Couldn’t Deliver,” Harvard Business Review, May-June 1997 [7] Alter, S. (2003). “Pervasive Real-Time IT as a Disruptive Technology for the IS Field,” Proceedings of HICSS-36, The 36th Annual Hawaii International Conference on System Sciences. [8] Wand, Y. and R. Weber (2002) “Research Commentary: Information Systems and Conceptual Modeling – A Research Agenda,” Information Systems Research, (13:4), December 2002, pp. 363-376.
© Steven Alter, 2003 26