algorithm adoption model: what factors lead to algorithm
TRANSCRIPT
Algorithm Adoption Model: What factors lead to algorithm
adoption and use by decision-makers
JOEL DAVIS
JUNE 2020
A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF BUSINESS
AT THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT
OF THE REQUIREMENTS FOR THE DEGREE OF
DOCTOR OF BUSINESS ADMINISTRATION
UNIVERSITY OF FLORIDA
2020
© 2020 Joel Davis
3
ACKNOWLEDGMENTS
I would like to express my most profound appreciation to my dissertation chair Dr. Philip
Podsakoff. Thank you for spending the time to guide me through this patiently. This doctoral
program has been a fantastic journey.
I very much appreciate the support of Revenue Management Solutions, and the leadership team
of John Oakes, Olivier Rougie, Sebastian Fernandez, Mark Kuperman, and Jana Zschieschang
for supporting me through this endeavor. Thank you.
I would like to thank Angie Woodham, who shepherds us through the complexities of earning a
doctorate. Thanks for your time and the extra effort you put into making this a great program and
a great experience.
Finally, I would like to thank my fellow cohort members for their support and friendship. Thank
you for the words of encouragement, advice, and most of all, for listening. I am proud to be
among you.
4
Abstract
There is a long academic history describing decision-makers avoiding or under-weighting
algorithmic advice. Recent experimental studies have shown that decision makers reject using algorithms
under various conditions. Despite this, the management field has not developed the necessary theoretical
understanding and conceptual definitions that underlie algorithm adoption. In this study, semi-structured
interviews were conducted with thirty business professionals across a variety of industry backgrounds, all
of whom have had multiple occasions to consider using the advice of an algorithm to make or improve a
decision. Through an analysis of the interviews and an integration of the literature on technology adoption
and trust in automation, an algorithm adoption model (AAM) construct is defined and developed.
Algorithm adoption consists of 4 sub-dimensions: (1) input trust, (2) output trust, (3) algorithm
provenance, and (4) understandability. This study first provides clarity and definitions of algorithms and
trust through a review of the literature. It considers how factors related to technology adoption,
technology diffusion, human-human trust, and human-automation trust relate to the concept of trusting
algorithms. The conceptual positioning of the algorithm adoption model vs. existing models of
technology adoption and automation trust, and the similarities and differences between these models are
explored and discussed. Finally, the benefits, barriers, and risks of adopting algorithms into decision-
making are examined.
5
Introduction
Increasingly, decisions that were in the past only made by humans are now strongly influenced by
one or more algorithms. This evolution and growth in algorithm-related decision-making is not without
some casualties. In some cases, algorithms are substitutes for human decision-makers, and in other cases
the algorithm compliments a decision maker. As any introductory economics textbook explains, some
compliments will become more valuable as cognitive algorithms improve in performance, and some
substitutes will become less valuable. As noted by Agrawal, Gans, and Goldfarb (2018), London cab
drivers are an excellent example of the latter point. Before the introduction of sophisticated wayfinding
algorithms like Google Maps or Waze, prospective London cab drivers spent years learning the roads,
locations, and best routes through London. This daunting task ended in a test to ensure that they were able
to meet the incredibly high standards of those that drove before them. This test created a very high barrier
of entry that made the career a lucrative one. However, the introduction of step-by-step and turn-by-turn
algorithm-based solutions meant almost anyone with a car could compete with London Cabbies and
turned a decades-old tradition and industry on its head. Given the fundamental change, how do cab
drivers in London react? Do they change their standard practice and adopt the algorithm's advice, or do
they continue with their traditional approach? It is easy to see that the decision in many cases is not a
dichotomous one; in our example, the drivers can use none, some, or all the advice. It is also easy to
surmise that how much advice an individual is prepared to take from an algorithm is related to his/her
level of expertise and willingness to accept advice in general. This constant push of capabilities
enhancement, compliments, and substitutions are poised to have an enormous effect on the way we work
and live our lives in the future (Agrawal et al., 2018; Malone, 2018).
Given the importance of algorithms in our economy, and in our professional and personal lives, it
is critical to understand how human decision-makers accept or reject the outputs of algorithms. Dietvorst,
Simmons, and Massey (2015) showed that people are more apt to lose confidence in an algorithm after
seeing it err than when a human forecaster makes similar errors. Interestingly, even people who see an
algorithm outperform the human forecaster are less likely to follow its recommendations. This pattern of
6
findings builds on the seminal work of Meehl (1954), who found that linear models outperformed
psychologists' clinical approaches, even when the models were imperfect (Dawes, 1979). This research
has created an entire field of scholarly work. Much of this work supports the finding that algorithmic
combinations, even naive ones, are superior to human judgement. For example, a meta-analysis reported
by Grove, Zald, Lebow, Snitz, and Nelson (2000) found that, with few exceptions, mechanical prediction
techniques are more accurate than clinical predictions. This led these authors to conclude that, "There
seem, then, to be no barriers to a general preference for mechanical prediction where an appropriate
mechanical algorithm is available" (Grove et al., p. 26). Similar results were reported by Kuncel, Klieger,
Connelly, and Ones (2013), who showed that even when the decision-makers are experts, mechanical
combinations significantly outperformed holistic/human combinations.
So why do users adopt or fail to adopt algorithmic advice? Existing academic literature often
compares a choice to be made between accepting an algorithm’s advice or using an individual's own
judgment (Dawes, Faust, and Meehl, 1989; Dietvorst et al., 2015; Meehl, 1954). But, what happens when
the choice is not between one's self and an algorithm but rather between algorithms? Or when the
algorithm offers advice first, and then the decision-maker decides to accept or reject that advice? One
obvious avenue of research that focuses on answering these questions it the literature on trust in
automation (Lee and See, 2004; Muir, 1987). This research, however, tends to focus on specific and low-
level automation tasks such as automated factory monitoring system; not tasks that require higher levels
of cognitive work on the part of the decision-maker (Prahl and Van Swol, 2017).
Algorithm adoption is different and perhaps more complicated than human to human advice
adoption. One reason for this is that algorithms can lack accessibility. Even simple algorithms may be
hard for decision-makers to interpret and understand (Diakopoulos, 2014). This lack of accessibility may
have the effect of diminishing the decision-makers' trust in the algorithm, leading to lower rates of
adoption (Muir, 1987). Other factors of trust also play a significant role. Prior research in automation has
shown that credibility (i.e., will the machine work), is a relevant factor in automation trust. Extending this
trust effect from automation to algorithms makes sense given the findings of algorithm aversion reported
7
by Dietvorst et al. (2015) and Prahl and Van Swol (2017). Furthermore, decision-makers' prior
experiences with algorithms is likely to affect their adoption processes (McKnight, Choudhury, &
Kacmar, 2002; McKnight, Cummings, & Chervany, 1998).
Finally, it is important to note that we are not suggesting that the solution to improving human
decision-making process is simply to get decision-makers to adopt more algorithms. In some cases,
algorithms express severe bias, either because of limitations of the programming or because of the data
they are trained on (Kitchin, 2017). If individuals adopt algorithms based on many complex factors,
research such as this has a significant academic and practical interest. An essential human judgment task
is to understand the causes and remedies for this bias, in both the algorithms they use and how the
decision-makers adopt and use them. An important societal goal is understanding how and when
individuals and firms incorporate algorithms into their decision-making process, and how this can be
improved to facilitate better decision- making by reducing both human and machine bias.
Within the context of the above discussion, the purpose of the current research is to develop a
better understanding of the algorithm adoption process. In order to accomplish this goal, the following
research questions guided this study:
1. What is the meaning of the term "algorithm" and "algorithm adoption"? More specifically,
what are the defining properties of these constructs?
2. What are the factors that lead to Algorithm Adoption by decision-makers?
3. To what extent does trust in an algorithm's inputs and outputs relate to Algorithm Adoption?
4. How is Algorithm Adoption similar/different from other, related constructs such as the
Technology Acceptance Model (TAM) (Davis, 1989)?
5. What are the antecedents of Algorithm Adoption (such as personality traits, motives, the
nature of the task the algorithm is being applied to, organizational factors, etc.)?
6. What are the benefits and barriers to individuals/firms of adopting algorithms in decision-
making?
8
What is an Algorithm?
Before examining the literature, it is important to clarify what is meant by an algorithm.
Algorithms are often part of more extensive decision-making programs or tools used by organizations and
individuals. These types of interconnections make studying algorithms somewhat challenging, as their
delivery depends on other factors. Studies have shown, for example, that the interface a program uses can
impact a user’s adoption of that system. As an example, Schwartz and Cohen (2004) conducted a study
utilizing forecasting systems and found that the speed and content of the interface was an important factor
in determining the systems use. Given this, how do we know what is being evaluated, the program or
technology delivering the algorithm, or the algorithm itself? First, it is essential to realize that although
algorithms are often embedded in other technological solutions, they differ in significant ways. These
differences make the study of algorithms as an entity a critical endeavor. Indeed, according to Dourish
(2016):
Algorithms and programs are different entities, both conceptually and technically. Programs may
embody or implement algorithms (correctly or incorrectly), but, as I will elaborate, programs are
both more than algorithms (in the sense that programs include non-algorithmic material) and less
than algorithms (in the sense that algorithms are free of the material constraints implied by
reduction to particular implementations) (p. 2)
Although we agree with Dourish’s point that algorithms and programs are not the same entities,
one is still left with identifying the defining properties of algorithms. Table 1 presents the conceptual
definitions and key attributes of algorithms from a review of the literature. It is apparent from this table
that there are a few core attributes of algorithms and that these have been somewhat stable over time. For
example, virtually all of the definitions contain references to a procedure or step-by-step instructions.
Many also explicitly discuss the need for an input into that process and the need for an output for that
process. It is interesting to note that the attributes can be applied to a variety of tasks that many would not
consider algorithmic. Baking a cake is a straightforward example. A cake is baked using a set of inputs
(i.e., flour, eggs, sugar, etc.), a procedure for combining inputs and baking, and an output (the cake). For
the present study, we are interested in how humans and algorithms interact in decision making. This
naturally leads us to consider the definition of algorithms through a technology or computer science lens.
9
Table 1:
Summary of Prior Algorithm Conceptualizations
Author(s) Conceptualization Attributes
Cormen (2009) "...an algorithm is any well-defined computational procedure
that takes some value, or set of values, as input and produces
some value, or set of values, as output." (p. 1)
Procedure; Inputs;
Outputs
Cormen (2013) "...a set of steps to accomplish a task." "a computer algorithm
is a set of steps to accomplish a task that is described
precisely enough that a computer can run it." (p. 1)
Precise; Steps; Task
Accomplishment
Dourish (2016) "In computer science terms, an algorithm is an abstract,
formalized description of a computational procedure." (p. 3)
Computational
procedure
Gillespie
(2014)
"Algorithms need not be software: in the broadest sense, they
are encoded procedures for transforming input data into a
desired output, based on specified calculations. The
procedures name both a problem and the steps by which it
should be solved." (p. 1)
Procedure; Inputs;
Outputs
Goffey (2008) "Algorithms do things, and their syntax embodies a command
structure to enable this to happen." (p. 17)
Structure
Harris and
Ross (2006)
"an algorithm is a set of well-defined steps required to
accomplish some task." (p. 1)
Well-defined steps;
Task
Accomplishment
Kitchin (2017) "sets of defined steps structured to process instructions/data
to produce an output." (p. 16)
Defined steps; Input
Instructions; Output
Kowalski
(1979)
"An algorithm can be regarded as consisting of a logic
component, which specifies the knowledge to be used in
solving problems, and a control component, which
determines the problem-solving strategies by means of which
that knowledge is used." (p. 424)
Input Knowledge;
Output; Control
Steps
Lewis and
Papadimitriou
(1978)
"a precisely stated procedure or set of instructions that can be
applied in the same way to all instances of a problem." (p. 96)
Precise procedure;
Problem Solving
Mundra and
Dwivedi (2013)
"An algorithm is the step-by-step solution to a certain
problem." (p. 1)
Steps; Problem
Solving
Stephens
(2013)
"An algorithm is a recipe for performing a certain task." (p.
3)
Task
Accomplishment;
Recipe
10
The input is perhaps the most complicated part of the definition to understand, as we can consider
many scenarios where there are zero (0) or more inputs. Some might consider zero to be the lack of an
input, thereby making inputs unnecessary as part of a definition of an algorithm. However, that is not the
case, as long as the procedure has some means of using this information. A simple example showing a
computational procedure that expects as an input some value between 1-20, but also incorporates a NULL
(Missing) input may help:
IF input value is NULL
print("No Number!")
ELSE IF input value < 10
print ("Lower than 10")
ELSE
print ("10 or Higher")
ENDIF
Given the attributes from Table 1. we can ascertain that an algorithm must have some means of
acquiring input, performing some computations or testing of logical conditions on that input, selecting
various actions that it may undertake, and then providing an output. This sequencing of steps requires that
algorithms be clearly defined and have a formal procedure or a "command structure," as Goffey (2008)
calls it. Cormen (2009) says, "We can also view an algorithm as a tool for solving a well-specified
computational problem. The statement of the problem specifies in general terms the desired input/output
relationship" (p. 5).
For this review, it will be sufficient to build upon these attributes, leaning heavily of the
definition and clarity provided by Cormen (2009), and define an algorithm in the following manner:
An algorithm is any well-defined computational procedure that takes some values as inputs and
produces some values as outputs. The procedure identifies both a problem and the steps by which
it should be solved. An example of an algorithm may be a program that takes historical sales data
as input and predicts future sales as output. Another example may be an algorithm that takes as
inputs a user’s past search behavior on a website and predicts what they should be shown on the
screen (see Figure 1.)
11
Figure 1 What Are Algorithms? Adapted from Cormen (2009)
Many times when we refer to algorithms, we are referring to a procedure related to cognitive
tasks or a prediction machine (Agrawal et al., 2018). The cognitive applications and uses for algorithms
create the possibility that these will conflict with human decision-making. This potential for conflict
illustrates a critical difference between algorithms and why they are adopted or avoided versus other
technologies.
The areas of technology, programming, and computer solutions have been an active area of
research for some time. What makes algorithms interesting now, and how do they differ from the
technologies researched in the past? Perhaps one of the most significant differences is that more and more
computers can act independently of human control. This independence and the burgeoning agency is in
contrast to a program procedurally carrying out the will of human operators (Hoc, 2000). The introduction
of autonomous activity that can be viewed by humans as a program acting as an autonomous agent
(Diakopoulos, 2014; Hoc, 2000) adds additional complexity to Human-Computer Interaction (HCI) or
cooperation. Autonomy and agency also introduce a conflict with human-based judgment and decision-
making (Dietvorst et al., 2015; Logg et al., 2018; Prahl and Van Swol, 2017). Although the discounting of
algorithms has a long academic history, their prevalence in our lives makes this a going concern.
Literature Review
In the following section, I review the literature on the Technology Acceptance Model (TAM),
technology diffusion, and the role that trust plays in the acceptance process. This review provides an
overview of some of the issues that need to be considered in developing a definition of algorithm
adoption, and distinguishing it from other, related constructs.
12
Technology Acceptance Models
The information systems field has done a significant amount of research on technology
acceptance models (Marangunić and Granić, 2015; Venkatesh, Davis, and Morris, 2007). Integrating
some of the findings from research on these models provides creators or users of algorithms with
potentially valuable insights if improving acceptance is the goal. Although technology and algorithms are
not synonymous, the factors that influence users’ acceptance of technology will undoubtedly have some
overlap with the factors that influence users’ adoption of algorithms.
In one of the most influential papers in the information systems literature, Davis (1989)
introduced the Technology Acceptance Model (TAM). This systematic study of acceptance revealed why
firms are not able to fully capture the value from the technology due to barriers in acceptance. TAM
introduced two constructs to explain a user's intention to use a given technology -- perceived usefulness
(PU) and perceived ease of use (PEU). Davis (1989, p. 320) defines perceived usefulness (PU) as "the
degree to which a person believes that using a particular system would enhance his or her job
performance." The second factor, perceived ease of use (PEU), was defined as "the degree to which a
person believes using a certain system would be free of effort." (Davis, 1989, p 320). Both factors were
identified as major contributing factors related to a person’s intention to use technology. In subsequent
research (Szajna, 1996; Venkatesh and Davis, 2000), there has continued to be strong support for these
factors. In particular, perceived usefulness has been shown to be the most significant predictor in
influencing technology acceptance. Perhaps to improve adoption, algorithm designers can focus on
designing useful algorithms that improve measurable outcomes? One challenge to this approach is the
nature of many of today's algorithms, which often underperform upon their introduction and then improve
over time.
Venkatesh and Davis (2000) extended the original TAM model by including factors of social
influence such as subjective norms and image, and cognitive processes such as results demonstrability
(see Figure 2). This study validated and replicated some of the original TAM findings across four
organizations longitudinally. TAM2 explained a significant portion of the variance in intention to use
13
(60%). One of the most important additions to the original TAM model was that subjective norms have a
direct effect on the intention to use and an indirect effect through perceived usefulness. Importantly,
another finding was that a compliance-based approach to encouraging adoption is less effective over time
than a campaign of social influence when it comes to changing user's perceptions of perceived usefulness.
Finally, this study provided practical advice to implementers to focus on demonstrating the comparative
effectiveness of new technologies to increase adoption.
Figure 2 TAM2 Venkatesh and Davis (2000)
Technology Diffusion
"Diffusion is the process by which an innovation is communicated through certain channels over
time among the members of a social system." Rogers (2010, p. 5).
If algorithm adoption can at least partially be explained by the adoption of other technologies
through the technology acceptance model (Davis, 1989), what other factors can explain how algorithms
are accepted up by users? One exciting avenue in the literature is technology diffusion, which provides a
lens through which to view adoption by firms and individuals. Rogers (2010) explains that diffusion is
about innovation and communication over time within communities. Although individual adopters may
evaluate a potential innovation by trying to understand its usefulness and ease of use (Davis, 1989), they
14
first must become aware of the innovation. Organizations are similar, in that the innovation must be
introduced by, or to, the organization. Rogers (2010) differentiates between passive and active "knowing"
about an innovation. Some potential adopters hear or learn about the innovation passively, and others seek
it out. This stage of knowledge is further broken down into awareness, information acquisition on using
the knowledge, and an understanding of the underlying principles of the innovation. How might these
different stages be related to algorithm adoption?
The awareness stage is critical, because as Rogers (2010) notes, it is at this stage where adopters
may seek out additional information on the algorithm. The next stage in this model is persuasion. In the
context given by Rogers (2010), this is where a unit of decision-making, the individual or the firm, forms
an opinion of the innovation. This is not by necessity a favorable opinion. This stage seems most related
to TAM (Davis, 1989), as it is at this point in TAM that adopters are making decisions on whether or not
the technology is useful in some way.
The confirmation stage involves a decision-maker seeking reinforcement of a decision already
made. Rogers (2010) discusses anecdotal cases of innovation discontinuance after adoption. Although
Rogers (2010) discusses some of the discontinuance drivers, a more thorough model is presented in
Bhattacherjee (2001). In this research, a model of technology continuance was developed to understand
the factors that lead users to abandon or retain a particular technology. Interestingly, almost all research
based on TAM has shown that Perceived Usefulness (PU) is the most significant factor in the acceptance
of technology. As previously discussed, this suggests that any measures of algorithm adoption should
include measures of the usefulness of the algorithm. As can be seen from Figure 3, confirmation of the
performance of the technology has an indirect effect on information systems (IS) continuance through its
effect on perceived usefulness and satisfaction (Bhattacherjee, 2001). Confirmation is a cognitive belief,
based on an expectation of a given outcome and realized by the user through use of a given technology.
15
Figure 3 Technology Continuance Bhattacherjee (2001)
Trust and Algorithms
Algorithm Avoidance
Of course, another factor that may influence be an individual's willingness to adopt an algorithm
is his/her openness to receiving innovations and fairly evaluating them; a concept McKnight et al. (1998)
calls "disposition trust." Other scholars have written extensively on the desire of decision-makers to be
consistent in their decision-making, and ensuring future decisions are consistent with their prior beliefs
(Moore and Small, 2007; Tversky and Kahneman, 1975; Yaniv and Kleinberger, 2000).
In many cases, the technology tested in the various versions of TAM, TAM2, and technology
diffusion models is similar to the algorithm-driven technology we are interested in. One of the primary
differences is that algorithms can change and adapt, while the implementation and interface stay the same.
This means that inputs and outputs can evolve in a non-transparent way. This inaccessibility can cause
conflicts with a user's decision-making and judgment. One way users may react to these changes is by
changing their trust in an algorithm. Trust has been shown in multiple studies to be an essential factor in
both human-human and human-automation literature. Before moving on, it is perhaps helpful to review
the concept of trust through the literature (see Table 2) and develop a common understanding of its
relevant attributes.
16
When most technologies or algorithms historically relied on straightforward "If this, then that"
steps, the user's accessibility to how the machine thought was straightforward. Now algorithms often have
complex inputs, inaccessible procedures, and probabilistic outputs, opening them up to significantly
different interpretations of effectiveness and trustworthiness. Lee and See (2004) discuss how trust in
humans differs from trust in automation. The primary difference is that automation lacks "intentionality"
and is not part of the social exchange. As noted by Lee & See (2004):
There is symmetry to interpersonal trust, in which the trustor and trustee are each aware of the
other’s behavior, intents, and trust (Deutsch, 1960). How one is perceived by the other influences
behavior. There is no such symmetry in the trust between people and machines (p. 66)
Lee and See (2004) explore the effect of trust in human-computer interaction. Their study sets out
clear steps firms or individuals can take to increase trust in automation, including setting an appropriate
expectation for performance and training those interacting with the automation on how and when the
automation can be relied upon.
The attributes of trust discussed above can be found in various fields on the adoption of advice
from other humans, automation, or algorithms. The trust attributes most relevant for algorithms in the
present study are found in Muir (1987): expectations, competency, and responsibility. In the human judge
advisor literature, an area primarily concerned with how humans interact with other humans in giving and
receiving advice, multiple studies have been conducted to understand the implications of violating some
aspect of trust (Sniezek and Van Swol, 2001). This research has had significant implications for human-
automation and algorithm trust research. In the forecasting literature, Sanders and Manrodt (2003) discuss
the impact of complicated solutions on decision-makers, finding that complicated solutions are relied
upon less. This may not be so different from the findings of TAM (Davis, 1989), which suggest that
Perceived Ease of Use (PEU) is a critical factor in technology acceptance.
17
Table 2:
Summary of Trust Conceptualizations
Author(s) Conceptualization of Trust Key Attributes
Deutsch (1973) "The confidence that one will find what is desired from another,
rather than what is feared" (p. 148)
Confidence,
expectation
Rempel, Holmes,
and Zanna (1985)
"a generalized expectation related to the subjective probability an
individual assigns to the occurrence of some set of future events"
(p. 63)
Expectation
Cook and Wall
(1980)
(Trust) "refers, in the main, to the extent to which one is willing
to ascribe good intentions to and have confidence in the words
and actions of other people." (p. 39)
Good intentions.
confidence
Mayer, Davis,
and Schoorman
(1995)
“The willingness of a party to be vulnerable to the actions of
another party based on the expectation that the other will perform
a particular action important to the trustor, irrespective of the
ability to monitor or control that party.” (p. 712)
Willingness to be
vulnerable, based
on an expectation
Mayer et al.
(1995)
"Three characteristics of a trustee appear often in the literature:
ability, benevolence, and integrity. As a set, these three appear to
explain a major portion of trustworthiness." (p. 712)
Ability;
Benevolence;
Integrity
Lee and See
(2004)
"The attitude that an agent will help achieve an individual’s goals
in a situation characterized by uncertainty and vulnerability." (p.
54)
Uncertainty;
Vulnerability
Bhattacharya,
Devinney, and
Pillutla (1998)
"Trust reflects an aspect of predictability-that is, it is an
expectancy." (p. 461)
Predictability
Bhattacharya et
al. (1998)
"Trust cannot exist in an environment of certainty; if it did, it
would do so trivially. Therefore, trust exists in an uncertain and
risky environment." (p. 461)
Uncertainty
Castelfranchi and
Falcone (2000)
"The word "trust" is ambiguous: it denotes both the simple
trustor’s evaluation of trustee before relying on it (we will call
this "core trust"), the same plus the decision of relying on trustee
(we will call this part of the complex mental state of trust
"reliance"), and the action of trusting, depending upon trustee."
(p. 3)
Reliability,
depending on the
trustee
Muir (1987) "Trust (T) is the expectation (E), held by a member (i) of a
system, of persistence (P) of the natural (n) and moral social (m)
orders, and of technically competent performance (TCP), and of
fiduciary responsibility (FR), from a member (j) of the system,
and is related to, but not necessarily isomorphic with, objective
measures of these qualities." (p. 531)
Expectation;
Competence;
Responsibility
18
The results of five studies by Dietvorst et al. (2015) show that errors in the output of an algorithm
make people less confident, and therefore less likely to adopt the algorithm's advice. The study suggests a
certain amount of intolerance for algorithmic error, as participants chose the algorithm that erred less
frequently, even when it outperformed a human forecaster. Similarly, Prahl and Van Swol (2017) tested
participants' reactions to the consistency of the errors of an algorithm. They found that inconsistent errors
have an adverse effect on adoption. This type of response may be related to a perfection schema
(Dzindolet, Pierce, Beck, & Dawe, 2002). The perfection schema is the theory that humans have an
expectation that technology will perform flawlessly. In a series of experiments, Merritt, Unnerstall, Lee,
and Huber (2015) found that some individuals think of automation as "all or nothing” and are less likely
to forgive automation when it errs.
Despite these robust results, recent research from Logg et al. (2018) showed that at least some of
the algorithm aversion found in prior studies was due to algorithmic advice conflicting with an
individual's assessments. Soll and Mannes (2011) discuss the effect of conflict with the self and the
overweighting of one's own opinions. Overall, Logg et al. (2018) found that decision-makers do use
algorithms over human advisors and that this use improves overall decision-making.
What role does an algorithm’s accessibility or interpretability play? Poursabzi-Sangdeh,
Goldstein, Hofman, Vaughan, and Wallach (2018) designed a study to understand these factors' effects on
an individual's adoption of an algorithm. Participants took part in forecasting tasks, estimating the price of
homes. In some cases, the participants saw an algorithm with an accessible model, the coefficients for
rooms, bathrooms, square footage, etc. In the alternate case, participants only saw a "black box" model
that came up with the final home price estimate. Counter-intuitively, they found that the accessibility of
the algorithms did not impact a user's level of trust in the algorithm, nor did the ability of participants to
spot or fix algorithm errors change. These findings should cause us to think critically about how to
measure algorithm interpretability and to build a model that allows us to understand when and how it acts
in the overall schema of adoption. Algorithm interpretability may not lead directly to adoption, but it may
interact with other decision-maker factors such as domain knowledge or experience.
19
In recent research, Castelo, Bos, and Lehmann (2019) conducted several large scale experiments
to understand the role the nature of the task had on acceptance. They found that algorithms were trusted
and therefore relied on less(more) for tasks that were perceived to be more subjective(objective).
Interestingly, they also found that the trust perception users had of the tasks could be changed. A users
perception of the trustworthiness of an algorithm could be changed by providing examples of algorithms
performing well on subjective tasks. Overall their findings suggest that the effectiveness of an algorithm
is a more important factor of algorithm acceptance than an individual’s discomfort with an algorithm
performing some task.
In the human-human literature, there is at least some weighing of interpersonal factors into advice
acceptance. In much of the human/algorithm literature to date, researchers are comparing the adoption of
advice acceptance between humans and algorithms. However, many of the most powerful algorithms
decision makers use do not compete with expert opinion. The automation trust literature is perhaps most
analogous to human-algorithm trust.
Machine and People Learning with Trust
If algorithms are often better than the humans they are replacing, why are they often ignored? It
could be that some experiences are just "better" delivered by a human (Agrawal et al., 2018). Yeomans,
Shah, Mullainathan, and Kleinberg (2017) conducted an interesting study on computer vs. human
generated recommendations. The research found that subjects considered computer-generated jokes
funnier than those generated by people. However, subjects were less likely to rely on recommendations
when they knew that a computer vs. a human had generated the jokes. The researchers conducted a
follow-up study, in which rich explanations of how the joke recommender algorithm worked were given
to test participants, and a sparse recommendation was given to a control group. The results were
illustrative; the group that received a detailed explanation of the workings of the algorithm rated the
algorithm as easier to understand (M = 0.09, 95% CI = [0.01, 0.18]) vs the low information group (M = -
0.09, 95% CI = [-0.18, 0.00]), t(984) = 2.93, P = .003). This research is important in the context of how
algorithms are built and deployed. The findings on subjective vs. objective acceptance of algorithms in
20
Castelo, Bos, and Lehmann (2019) would tend to support this idea, that there are just some domains and
contexts in which the advice of an algorithm is not accepted.
Another way algorithms might be considered less trustworthy than a human advisor is when
considering the interpretably of a given algorithm (Poursabzi-Sangdeh et al., 2018.) A significant effort
has recently been made to make algorithms more interpretable (Ribeiro, Singh, & Guestrin, 2016). These
solutions however try to solve an algorithmic problem in interpretability, with more algorithms. Despite
some of these efforts, the field of machine learning is still trying to improve the robustness of these
methods as recent tests of these methods have shown a lack of consistency in the outputs (Alvarez-Melis
and Jaakkola, 2018). This reduces their usefulness in the short term because a lack of consistency in
outputs can lead to lower adoption by the general public.
Given the amount of literature discussing the relative performance benefits of algorithms vs.
humans on many tasks, and the fact that the disparity in advice taking from machines is still a relevant
topic (Meehl, 1954; Dietvorst et al., 2015), it seems unlikely that there is a simple solution to the problem
of inadequate advice adoption. There have been efforts made to remedy some of the gaps. One innovative
solution is to make the process of accepting algorithmic advice more like that of accepting advice from a
human provider. Fridman, Ding, Jenik, and Reimer (2017) detail one such solution -- train more than one
algorithm to perform a task. If the algorithms agree, provide the user with some level of confidence that
these two "arguing machines" agree. When they disagree, explain or display the results in an interpretable
way, and allow the human to arbitrate. Competing dialogues such as this are much more similar to how
humans operate today; they often receive differing views and make a decision based on these differing
views. The impact on algorithm acceptance, especially in areas where the algorithm is taking some level
of control is potentially tremendous, Fridman (2018) calls this "shared autonomy."
When machines learn from their input-algorithm-output path, this is referred to as "Supervised
Learning" (Friedman, Hastie, and Tibshirani, 2001). It works through a series of feedback loops that
reinforce "correct" outcomes. This looping and re-measuring are why many machine learning algorithms
need so much data. Learning through feedback loops is not much different than the way organizations
21
learn (Rosenberg, 1982) by doing things and measuring the outcomes. Nor is it different than the way
most of us learn lessons throughout our lives. Perhaps then one way forward is for algorithm designers to
start small, and then gradually increase the level of algorithm autonomy. This stair-stepping of algorithm
autonomy is already happening, albeit likely due to technological restrictions versus design. Most of us
will have driven in a car that self-parks or has an emergency (algorithm-generated) stopping feature
before we own a car that drives itself.
This learning by doing also brings up a serious issue. If most of our learning as humans comes
from experience, how will we learn when our environment no longer gives us this experience? A chilling
example given by Agrawal et al. (2018) is the case of Air France Flight 447 that crashed in the Atlantic in
2009. The autopilot disengaged during a severe storm, and the junior pilot was unable to manage the
emergency. The senior pilot woke up and was not able to react fast enough to save the plane or the
passengers. Although the junior pilot had several thousand hours of flying, it had earned primarily on
autopilot. The de-skilling of pilots in this way means that the pilots will get worse as algorithms get better
at handling emergencies. Contrast the Air France Flight 447 result with the US Airways Flight 1549 that
landed on the Hudson River.
One way of looking at this might be that, for 42 years, I've been making small regular deposits in
this bank of experience: education and training, and on January 15, the balance was sufficient so
that I could make a very large withdrawal. - US Airways Capt. Chesley "Sully" Sullenberger.
("Capt. Sully Worried About Airline Industry." 2009).
Thinking critically about how to train people to accept an algorithm, and when to use an
algorithm, is not always as straightforward as it seems. Technology has not progressed to the point where
an algorithm would have successfully landed US Airway 1549. Ongoing de-skilling of pilots might mean
that in the future, there may not be a Captain Sully onboard.
Concept Development
This primary goal of this dissertation is to clarify the definition of the algorithm adoption
construct and its sub-dimensions, and to explain how this construct differs from other related constructs
such as TAM. The work follows the recommendations of Podsakoff et al. (2016) (see Figure 5) to create
22
concept definitions and refine those definitions through a review of existing literature and interviews with
subject matter experts.
Figure 4 Stages for developing conceptual definitions (Podsakoff et al., 2016)
In the first stage of the construct definition process, we reviewed the literature and dictionary
definitions "algorithm," “adoption,” and "trust." Based on this literature, our preliminary definition of
algorithm adoption is as follows:
Algorithm Adoption is the act of choosing to use an algorithm's output for some purpose by
individuals or firms. Looking back at our definition of an algorithm, Input(s)→Algorithm→
Output(s), an adopter will take the outputs of the algorithm and use these outputs to perform some
type of action. Like the conceptualization of this in @rogers2010, an adopter is a unit of analysis
and may be an individual or a firm. Of course, it is possible for algorithms to be compiled
together, each taking outputs from the preceding algorithm. In this case, algorithm adoption is
determined programmatically with either deterministic or probabilistic steps. However, the
current study explicitly refers to algorithm adoption by firms and individuals and not by other
algorithms.
It is important when defining a construct to define both what it is and what it is not (MacKenzie,
2003; MacKenzie, Podsakoff, and Podsakoff, 2011). The confusion between acceptance and adoption is
one such case and requires clarification. Acceptance refers to the act or feeling of receiving willingly,
approving of, or having a favorable response to something (Merriam-Webster.com, 2020). The definition
of adoption has key phrases such as: "to take up and practice or use," "to accept formally and put into
effect" (Merriam-Webster.com, 2020). Given these straightforward definitions, it is easy to see that the
act of accepting and adopting an algorithm are temporally related. An example may illustrate this. We
may accept the fact that driverless cars are coming, and we may accept that this technology (will be
someday) better than most, if not all, human drivers. However, this level of acceptance does not mean we
will adopt, or actively use, driverless cars. To be sure, acceptance is a strong predictor or of actual use
(Davis, 1989), but it is not equivalent to it. Much of the work on algorithms, and the related concepts of
automation and cognitive algorithms, simply assumes that solutions that perform better than a human at
23
the same or similar tasks will be adopted. Most of these works ignore the genuine possibility that workers
confronted with new solutions will fail to adopt them appropriately.
So, the next step in our process was to identify the key dimensions of the algorithm adoption
construct that help explain whether it is adopted or not. In order to refine the construct definitions,
interviews were conducted with professionals who are currently working or have worked extensively,
with algorithms in decision-making contexts.
Participants
The primary source of information for this stage is interviews with current and former
professional business people. Inclusion criteria for the participants were a minimum of 1 year of
experience in a role that required or requires decision-making. Some of the subjects were known to the
researcher. Others were introduced to the researcher as part of the interview vetting process. The roles of
the participants did not necessarily involve creating algorithms used to make decisions. However, each of
the interviewee's current or previous roles would have had multiple occasions to consider using an
algorithm to assist in making decisions. The thirty interviewees (7 females, 23 males) work experience
spans 1955 and 2019, with a median work experience of twenty (20) years. Participants also discussed
their work experience in terms of using algorithms in their decision making, the median experience with
algorithms was just over ten (10) years. All participants had a college education, and seventeen (17) held
post-bachelor’s degrees. Seventeen were, or had previously been, employed at large publicly traded
company, and nine were or had previously been employed in a consulting company, and one was
employed at a non-profit. The business sectors represented by the sample include hospitality, finance and
banking, retail, manufacturing, consulting and technology among others. See Table 3 for an overview of
the interviewees.
The use of subject-matter experts as a source of information in the conceptual development stage
of construct development is well supported by the literature (Podsakoff et al., 2016). Second, because this
group extends across multiple generations, the research covers the period when algorithmic decision-
making first arose in organizations, which helps identify differences in the cohort based on age, or work
24
experience. Third, interviews allow for a depth of concept development not available using other methods
(Venkatesh, Brown, & Bala, 2013).
Table 3
Interview Participants
Interview Number
Years of
Experience
Role
1. Sergio 20 Chief Research Officer, Consulting Company
2. Craig 50 Owner/Principle Boutique Market Research. Former VP Fortune 100
3. Judy 34 Director Finance, Fortune 500 company
4. Ted 50 SVP (Various roles) Fortune 500 company
5. David 12 SVP Mid-size Analytics Consulting Co
6. Chuck 32 Chief Brand Officer- Large Fast Casual Restaurant Chain
7. William 26 CEO/President Large Restaurant Brand
8. Jake 2 Big 4 Consulting Co
9. Gordon 15 Health Care Analytics
10. Hollis 3 Data Science Analyst- Large Finance/credit card company
11. Mary 8 Director, Real Estate Services firm
12. Susan 5 Analyst, Banking Industry
13. Jennifer 2 Analyst, Energy/Oil Company
14. Alice 1 VP Analytics/Insights, Banking
15. Paul 30 Former Sales Executive
16. Darryl 2 Analyst, Home goods/products company
17. Edward 26 Finance and Investing
18. Anthony 52 Founder - Market research company
19. Charlotte 10 Director Analysis, Non-Profit
20. Travis 19 Private Consulting
21. Earl 25 Former VP- Fortune 500
22. Noah 7 Data Scientist- Tech 'Unicorn' Company
23. Matthew 30 Former CEO, Board Member (start-up and established brands)
24. Jerry 13 Director Analytics- Travel Industry
25. Emanuel 30 VP Hospitality/Hotel Industry
26. Luke 24 Senior Leader- Equipment Manufacturing
27. Cameron 8 Developer (Multiple Tech and Tech-enabled companies)
28. Miles 20 COO Analytics Company
29. Jim 20 CEO, Analytics Company
30. Jacque 24 CFO/President Analytics Company
Note: Pseudonyms have been used to identify participants, rather than their real names.
Procedure
Qualitative data was collected through semi-structured interviews related to the participant's
experiences using algorithms to assist with decision-making. The structure and definitions of an algorithm
25
and its components were used to generate ten questions (See Appendix 1 for the detailed question list).
The interviews were conducted in various business offices in the United States, England, Singapore,
Tokyo, and France and via recorded calls. All interviews were recorded and transcribed. The transcripts
and descriptive information (dates, masked contact information, gender, age, experience, and professional
role(s) of the interviewee) were input into a data table maintained by the researcher. The transcripts were
also loaded into MAXQDA software (VERBI Software, 2020) to facilitate coding. Descriptive statistics
were calculated in the R programming language (R Core Team, 2020)
Each interview was individually analyzed to identify themes or "codes" that reflect the concepts
being studied. Consistent with Saldaña’s (2009) statement that "...the qualitative analytic process is
cyclical rather than linear" (p. 58), the coding process was split into several cycles. The first coding cycle
for each transcript was done using structural codes (Guest and MacQueen, 2008). Structural codes serve
to identify areas of the transcript that are related to topics of semi-structured interviews. This approach
allowed the researcher to query the data and review similar areas of the text across all participants to aid
in the next cycles of coding.
The next phase involved In-Vivo coding. This coding technique uses the participant's own words
as codes (Corbin, Strauss, & Strauss, 2015). This technique is most closely related to and discussed as a
part of the grounded theory technique. Although the present research takes a more conceptual and broader
view, it was expected that using the participant's own words (in vivo) would provide another basis for
reviewing, restructuring, and identifying the factors related to algorithm adoption.
Finally, the last and most detailed phase was concept and pattern coding. Saldaña (2009) argues
that concept coding is appropriate for studies on theory development. This phase involved iterating over
the interviews and coding each multiple times as/until themes emerged. The codes were generated first by
individually coding interviews, and later by using the structural codes from the first coding cycle to look
for similarities and differences across the participants. Finally, as themes emerged, these pattern codes
were grouped and used to organize the prior codes.
26
At the end of each interview, participants were asked to rate their willingness to adopt an
algorithm in a verbal survey consisting of 10 items on a 5-point Likert scale ranging from 1 (not an
important part of the decision) to 5 (an important part of the decision). Although the data is ordinal, to aid
in interpreting participant's relative importance scores for each attribute, Means (𝑥‾) and the Standard
Deviations (𝜎) were derived for each factor.
Results and Discussion
The purpose of this stage of research is to help identify, define, and develop factors that lead to
algorithm adoption by decision-makers. After several rounds of coding, four key dimensions of the
algorithm adoption construct were identified: (1) Input trust (referenced by 83% of the interviewees) (2)
output trust (70%), (3) understandability (57%) and algorithm province (60%). These themes, the first-
order categories that they are comprised of, and examples of the quotes made by the interviews are
summarized in Table 4, and Table 5 provides a summary of the percentage of participants that identified
each of these themes in their interview. Finally, Table 6 provides a summary of the descriptive statistics
from the survey asking the participants in the study to rate the importance of various factors to their
willingness to adopt and use an algorithm to help you make decisions in their jobs. In the section that
follows, we define these sub-dimensions of the algorithm adoption construct.
Table 4
Summary of Attributes of Algorithm Adoption Identified by Interview Participants
Second-Order
Themes
First-Order
Categories
Exemplary Quotations
Input Trust
(Referenced by
83%)
The right
number of
inputs
"…more variables are usually better… to a point. You get to a
point where you're no longer adding value; you are just adding
noise. " (Charlotte)
"…some people feel like more is always better. So, the more
information we can shovel in here, the closer we'll get; when I
think that actually can produce a worse result. So, I actually feel
that sometimes decision-makers overdo it when it comes to what to
include." (Charlotte)
“…part of it was when we have to make decisions on our own on
whether or not to include as many variables as possible or just
27
Second-Order
Themes
First-Order
Categories
Exemplary Quotations
make executive decisions on what doesn't seem to make sense,
based on our experience." (David)
"really about focusing on the two, sometimes three, most salient
inputs that drove that output." (Earl)
Understanding
Inputs
"I have to understand the data myself. With that, I think that's... I
don't want to say necessarily feature engineering, but maybe along
those lines, right? Why are we using these particular columns,
fields, whatever? Are they directly necessary? Are they a proxy for
something else? Do we have other, better data?" (Gordon)
“feature engineering is still the art of data science. The reason we
say that is because in order for you to actually derive the maximum
benefit, you have to think about what contributes the most to your
models.” (Noah)
“qualitative research will identify if there's something that doesn’t
fit…we never replace, we always add something else in so we can
maintain that consistency.” (Anthony)
Input
Accuracy
“What I've often found is that the data they're using to put in them
has been flawed and so what is produced is the outcome of the
algorithm's calculation is inaccurate and inappropriate even though
the basic formula may be fine.” (Craig)
"but if I have garbage in or if my assumption's incorrect, I mean,
my perceived quality of the data of the output is completely
inaccurate or incorrect." (Jake)
"data is messy" (Jake)
“I think that in terms of the outputs, the part that we want to
understand better before using them is whether the data that is used
to create those outputs is accurate.” (Sergio)
“Some of our clients have issues giving us actual or let's call it
accurate costs…. in many cases what we need to do is we need to
use proxies … How far we move away from the ideal? I guess it is
going to determine whether we can trust the final output” (Sergio)
“you're feeding it bad information, poor information, which then
the algorithm's just going to continue to get worse and worse. And
then, it's really not trusted” (Emanuel)
- Inputs make
sense
"…a lot of it is what makes sense to me. So, if I think about the
business problem I'm trying to solve for, have we captured the
28
Second-Order
Themes
First-Order
Categories
Exemplary Quotations
population I think needs to be considered to derive the expected
output?" (Judy)
"Well, I think the most important thing would be whether the inputs
would make business sense or not in the specific use case that I'm
trying to solve" (Hollis)
“…it depends who is consuming the results. For example, this
product that I am currently working on I mentioned, there are a lot
of stakeholders involved. So, in this instance, everything needs to
make intuitive sense” (Alice)
“There are some things that are very logical in terms of ‘these are
the inputs that make up x’ and some things that are not. You have
things that occasionally are, what's the right word? I guess more
subtle.” (Earl)
- Input
completeness,
sufficiency
"…do we have all the inputs that we want in a perfect world?"
(Sergio)
"I like to think about specific other data sets that may exist" (Craig)
“Some of the outputs are just trash... we just don't have the right
variables in our data sets” (Darryl)
“The more we try some of these models…the predictions just don't
come out where we want them to be, the more I realize how
important it is to have the right set of attributes in your data, and
how hard it can be to collect that kind of information.” (Darryl)
“Actually, a lot of the work that I do is designing or creating the
algorithms that will allow the end client or the end-user to make
better decisions. So, defining how they're making decisions today,
what inputs might be missing to make a better decision…is the core
of what I do” (Travis)
“You have to have a good working knowledge of the data you have
and be objective about what you don't have.” (Matthew)
“The data might be insufficient to draw a reasonable conclusion.”
(Luke)
Output Trust
- (Referenced by
70%)
“Makes Sense” "It's going to be what does your gut say? Does it make sense to
you? Is it at all what you expected the result to be?" (Judy)
“is it consistent with common sense, or not?” (Craig)
29
Second-Order
Themes
First-Order
Categories
Exemplary Quotations
"I think it was more of a confirmation to them that some of the
outputs made sense and that it was the right thing to do to act upon
those things." (David)
“Does it make sense? Can you build a story from it because it
should make sense and line up with other research that's going on?"
(William)
"it's sort of the trust but verify, meaning is it reasonable to expect
the output that I'm getting? I should, I think, walk in with an
expectation of what those results are going to look like." (Gordon)
"typically with any type of algorithm, you have some type of
expected result. If it's in a range of what you would expect, then I
guess that would be an answer of ‘yes, this is accurate.’" (Susan)
“if you don't question if the way in which you constructed the
algorithm and the results that you're getting make sense and are
right for that situation, then you may get into a situation that okay,
you get the results of the algorithm and you apply it without
thinking about what may be wrong with it and what the
implications are.” (Sergio)
“…I think that's where that sense check come from just looking at
it and getting an idea of does this pass the sniff test for me?"
(Emanuel)
Error “… the downsides of using an algorithm depends on the negative
impact when the algorithm fails, right? How big of a knee jerk
reaction do business leaders have when the algorithm makes an
error”? (Jake)
Replicability “…play devil's advocate, because if you're telling me it's supposed
to do something along the way, if I'm duplicating that in those same
steps and it doesn't match what you're giving me, I want to be able
to tie that out.” (Gordon)
Understandability
(Referenced by
57%)
Explainability “It got us enough accuracy within limits of our data, so we'd know
we'd be doing basically the right decision without basically make it
so opaque that no one would trust it because they didn't understand
what was going on” (Luke)
"The last thing is that what I like personally, although I find this is
typically not done in business by people. I want someone to explain
to me the logic of the algorithm. Am I going to be as proficient as
they are, including probably in writing the algorithm? At this point
in my career, probably not. But I'm still proficient enough that I can
30
Second-Order
Themes
First-Order
Categories
Exemplary Quotations
understand whether the logic flow makes sense on the formula for
the algorithm. I want to understand enough about it to believe that
there's not some fundamental logic flaw in how they did the
calculation." (Craig)
“…saying is it something that we're going to be able to use in the
conversation with an executive, and explain, and justify versus
something that we would have a hard time explaining any kind of
relationship between the variable and the potential outcome."
(David)
"Is it something that we can explain?" (David)
"…when you're trying to explain it to people in the business maybe
they don't understand some of the terminologies that you're using,
or the steps that you use. I've seen that happen a lot here
specifically a lot of times people will ask me or other members of
my team to simplify it down to make things explainable to the
business."
(Hollis)
"At the end of the day what ended up happening was the whole
model got translated into something which was simple, decision-
based, which made sense for, you know like it was explainable to
everybody" (Hollis)
"I need to know how it works for it to be believable for me. And so
I think usefulness and explainability are critical" (Paul)
"…explainability, interpretability of outputs are huge for us."
(Darryl)
"One of the things that I feel is the hardest area in this domain is
the ability to explain predictions. The real push here is to be able to
explain the model downstream so that individuals who are being
impacted by the decision can actually go back and look at it and see
why that decision was made. And ultimately, you know, be able to
like change those, changed some of those parameters and make
sure that they can take some actions on that on that prediction."
(Noah)
"And explainability and acceptance by the user base. I'd say that's
absolutely critical and often overlooked with not just the technical
side of things, but how well it will be absorbed." (Luke)
"I think one of the things that we're always facing is how good a
model is predicatively, versus the ability to explain." (Miles)
31
Second-Order
Themes
First-Order
Categories
Exemplary Quotations
Interpretability
"… if I'm going to pass on it, I better really understand how that
can be interpreted and applied." (Craig)
"Interpretable, it's totally possible to have a look at the output of a
boosted tree, but it's possible that I can interpret the results just as
easily. At least with the output. But I don't know that that translates
into something that I can tell a story around." (Miles)
"The models are very good at identifying those and making them
parsimonious. So that happens during the model phase. But
obviously, you also want to make sure that this is aligned with
intuition you're not building the models at a pure black box."
(Noah)
"I thought it was a really important evolution that we needed to
have on the sales team to be able to talk to the clients in their terms
and get rid of that black box effect." (Earl)
"…the black box algorithm has marginal value. To me, anyway,
given my background in understanding of technology and things
like that, I need to know how it works for it to be believable for
me." (Paul)
"So, kind of taking a very transparent box approach as opposed to
black box. So, having the client understand how the algorithm
works and the benefit and the compromises of the different
elements." (Travis)
"Yeah, our predictive model to predict balances over time, I think
the impact of the business is very strong if the model is not very,
very precise, and I think most people acknowledge that so I think
we have much more flexibility there to be a little bit more black
box." (Alice)
"Well, I feel like a lot of algorithms that we've used, there's a whole
quote, unquote ‘black box’, but we have a few of them that we're
actually able to kind of able to peel it back a little bit and
understand what's going on under the hood, so I guess
interpretability is big for us with our business partners. We go to a
lot of business partners who are savvy people, who don't
necessarily have data science or analytics degrees, but they get it.
They know their stuff." (Darryl)
"…black boxes, you don't really understand what is going in, unless
you build the whole thing from scratch." (Hollis)
"But I can drive a car without knowing how to build one, right?
And I think a lot of models and algorithms are really going that
way." (Miles)
32
Second-Order
Themes
First-Order
Categories
Exemplary Quotations
Algorithm
Provenance
(Referenced by
63%)
Data
Provenance
(data quality) “If I can trace back where the data came…if I was the
business analyst, I'd probably go back to the source and see what
was happening that day and if it was accurate.” (Susan)
"Data is messy, and they come from siloed databases"
(Jake)
Process
credibility
“To me a lot of the results that I get like make sense in a way that
like not everybody else might be able to understand them, because
like when you're trying to explain it to people in the business
maybe they don't understand some of the terminologies that you're
using, or like the steps that you use.” (Hollis)
“(we used)…the very simple linear model and that allowed us,
when we got a new decision maker into the room, we could draw
this on the board really quickly they got it. Regression gave us a
straight line. It all made sense.” (Luke)
Author
Credibility
“We're still very much getting the credibility and we're trying to get
them to trust us to do our analytics.” (Alice)
“At the senior level, they don't necessarily know the algorithms.
What they really have to do is they have to have faith in the people
that actually do those.” (Craig)
“But most don't do that in business. They just take it ... They buy
into the expertise of who's sitting in front of them, and that
expertise they basically…they take blindly frankly.” (Craig)
"It is not just what the algorithm shows; it is who is providing it
within these structures. So often the evaluation, validation, and
acceptance of the algorithm's output is really anchored more in who
brings it, than what it actually is in the sense of its intellectual
integrity, or necessarily to some degree the quality of the output. I
think that is a fascinating part of this process."
(Craig)
“It's like I put my trust in who is building it.” (Matthew)
"Do you trust the person that's designing it, writing it, and
reviewing it?" (Matthew)
“If it was somebody I was working with for the first time and it
didn't pass my crap detector, I'd probably push back a little bit and
would require a higher level of convincing to understand that it is
correct and there's not a mistake in the algorithm.” (Matthew)
33
Second-Order
Themes
First-Order
Categories
Exemplary Quotations
“I think it's critical to not just get the results of the algorithms, but
to give your confidence explicitly to the users. So that they know
how much you trust your answer... I think that's absolutely vital to,
to ensure buy-in so they know how much you trust it so they're
going to get it more or less way to appropriately, as they trust you.”
(Luke)
"The advantage is it is not subjective. It's objective. If you run it
twice, you're going to get the same answer instead of just running it
and what they're feeling like that day. So that takes a lot of the
guesswork about what the motivation is behind the person making
the recommendation."
(Luke)
"You don't need explicit understanding. I would say that when I use
an algorithm or a library or something else that somebody else
wrote, I assume that they're much smarter than me and they know
what they're doing."
(Cameron)
"I found that for the most part, the people who develop algorithms
tend to do it out of necessity for their own project or their own
situation, not for ... And maybe sometimes for altruistic reasons.
Not necessarily for, "Oh, I'm in this arena and I want everybody
else to be worse at it than me."
(Cameron)
Table 5
Summary of Themes Reported by Participant
Key Attribute
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 #
% of
Interviews
Input Trust ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ 25 83%
Output Trust ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ 21 70%
Understandability ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ 17 57%
Algorithm
Provenance ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ ◼ 19 63%
Note. A block (◼) means the attribute was present in the respective interview. % of Interviews is the % of
all 30 interviews that discussed a given attribute.
34
Table 6
Descriptive Statistics for Verbal Algorithm Adoption Factor Survey.
Factor M SD
Quality of information obtained from the algorithm 4.43 0.77
Perceived usefulness to your job 4.33 0.84
Trust in the outputs of the algorithm 4.33 1.03
The interpretability of the algorithm 4.30 0.88
Past experience with using the specific algorithm 4.13 0.82
The nature of the algorithm’s inputs 4.13 0.97
Perceived ease of use 3.90 0.99
Past experiences with using other algorithms 3.70 0.95
Subjective norms of the organization 3.07 1.44
Knowing who the algorithm was developed by 2.93 1.44
Note. M and SD represent means and standard deviations, respectively. N=30.
Input trust. Recall that trust between machines and people has the key attributes of expectation,
competence, and responsibility (Muir, 1987). Additional factors such as ability, integrity, vulnerability
(Mayer et al., 1995), and uncertainty (Lee and See, 2004), can all be found in the literature on trust
between people and within organizations. In explaining why they accepted or rejected the use of an
algorithm, most interviewees (83%) mentioned some form of input trust. Although the specific
dimensions of input trust were varied, most participants (56% of all interviews) brought up issues of a
data input’s 'completeness' or 'sufficiency.' Sufficiency did not just relate to an algorithm's ability to make
better predictions, or the desire for more data, but also to what could be explained and what “made
sense.” There were in many examples a reference to the right number of variables, and sometimes a
tradeoff between an algorithm's performance needs and the needs of the decision-maker:
Keeping it to a somewhat reasonable number of variables to keep it actionable because
sometimes, I think it's that balance between going overboard from the number of variables to
supposedly get a better model or what have you and keeping it actionable. -David
Other interviewees (23%) discussed the accuracy of the inputs as being a relevant dimension in
accepting/rejecting the use of an algorithm. This relatively low percentage was somewhat surprising since
a common saying in many fields related to data analysis is "garbage in garbage out." As Jake explained,
"if I have garbage in... the quality of the data of the output is completely inaccurate or incorrect." Craig
differentiated input quality and the quality of the algorithm:
35
What I've often found is that the data they're using to put in them has been flawed and so what is
produced in the outcome of the algorithm's calculation is inaccurate and inappropriate even
though the basic formula may be fine. -Craig
Based on the participants’ comments, we defined Input Trust as the degree of trust a user has in
the data input into an algorithm by a decision-maker. It is based on the user’s confidence in the accuracy,
reliability, and timeliness of the data input into the algorithm. Input trust does not include factors related
to the algorithm’s creator or architect. Nor does it include an evaluation of the algorithm’s outputs.
Despite the high incidence of comments regarding inputs during the interview process, when
asked to score the importance of the nature of inputs on a decision to adopt an algorithms' decision,
participants scored this item in the middle of the ten items being tested (𝑥‾= 4.13, 𝜎=.97). However, it is
worth noting that the question on the survey was related to the nature of an algorithm’s inputs, not the
accuracy or quality of the inputs that most participants mentioned. Thus, it is possible that the reason for
the relatively moderate rating of the importance of this factor in terms of an algorithm adoption may have
had to do with the fact that the question did not accurately reflect participants’ trust in input quality, and
that if it had reflected this variable better, it would have been rated higher in terms of its importance.
Output trust. Most interviewees (70%) mentioned some form of output trust during the course of
the interview. This finding is generally consistent with the results of the two survey questions that asked
about the outputs of an algorithm. For example, when asked how important the quality of the outputs is to
an adoption decision, participants scored this the highest of any factor (𝑥‾= 4.43, 𝜎=.77). Similarly, when
asked how important "trust" in an algorithm's outputs is, participants ranked this tied for the second most
important factor (𝑥‾= 4.33, 𝜎=1.03). The amount of trust that users have in the output of the algorithm is
one of the main factors in whether they adopt the algorithm or not.
By far, the most frequently referenced theme to emerge from the research was the notion that an
output (result) had to "make sense" to the user. If it did not make sense, it was subject to additional
scrutiny in some cases, or it was simply rejected. The “makes sense” filter was an especially critical factor
for those interviewees whose job it was to explain the output of the algorithm to someone else. Some
interviewees expressed unease at noting that their “gut feelings” were an essential part of their decision-
36
making criteria. For example, one interviewee said, "Unfortunately, it's the worst answer I could give you,
it's gut." For the most part, participants discussed using this as a validation of some sort:
Basically, to me, an algorithm is here to validate mathematically my gut feel, right? My gut feel
tells me this right? Now I have all these inputs that are coming from those algorithms. Do those
two make sense? -Jacques
Because typically with any type of algorithm, you do have some type of expected result. If it's in
a range of what you would expect, then I guess that would be an answer of yes, this is accurate -
Susan
Taken together, these comments led us to define Output Trust as the degree of trust a user has in
the outputs of an algorithm because the outputs are considered reasonable and credible by the users, the
output meets the user’s expectations, and the algorithm is believed to have performed its task competently
and correctly. Output trust does not involve an evaluation of an algorithm creator’s competence, but
rather a user’s belief that the steps the algorithm performed we’re executed correctly.
Similar to Mayer et al. (1995), the consequences of a violation of output trust are partially
determined by factors such as the nature of the task being performed, the perception of risk to a decision-
maker, and the alternatives to using the algorithm. It is logical to extend these factors to include domain
knowledge. A simple example: A driver follows Google(TM) Maps (an algorithm-based solution), and an
alternative route is identified that purports to save 3-minutes on a 30-minute drive. The risk (total drive
time) is low, the alternatives are all available to the decision-maker, and the domain knowledge of the
decision-maker will become the most significant factor. Does this 3-minute alternative route "make
sense" given what the driver knows of this route? However, the decision is likely to change when the time
saved begins to increase. Indeed, as discussed in the next section, 47% of respondents discussed prior
knowledge and “domain knowledge” as an important consideration in deciding whether or not to accept
an algorithm’s output.
Understandability. A large number of interviewees (57%) mentioned a preference for using
models that are easy to understand when considering whether to adopt an algorithm into their decision
making. An analysis of the interviewees reveals two sub-dimensions of this desire for understandability:
Interpretability (33%), and explainability (43%). It is worth noting that participants had a hard time
37
defining precisely what these two terms represent; indeed, none of the participants were able to offer a
concise definition. They are not alone. Academic interest in both interpretability and explainability to
improve model adoption is growing, but clear concept definitions remain elusive (Gilpin et al., 2019).
Despite the lack of clear definitions of these concepts from participants, when asked how important
interpretability was in their decision to adopt an algorithm, it was the fourth-highest factor on the survey
(𝑥‾= 4.3, 𝜎=.88).
One obvious way to make models more interpretable is to keep them very simple. As has long
been discussed in the academic literature, increased adoption of even simple models can improve decision
making (Dawes, 1979). The field of computer science and statistics has a number of models, such as low
dimensional decision trees and simple linear models that may be highly interpretable (Friedman, Hastie,
and Tibshirani, 2001). Unfortunately, these simple models provide interpretability at the expense of
predictive accuracy. More sophisticated models, from highly dimensional decision trees to deep learning
models, may have vastly superior predictive accuracy but are, for the most part, "black boxes." This
tradeoff was apparent in the results of the interviews. Interviewees had strong but not always consistent
opinions of non-interpretable models. In general, interviewees rejected the use of these types of models in
their decision making:
…the black box algorithm has marginal value. To me, anyway, given my background in
understanding of technology and things like that, I need to know how it works for it to be
believable for me. -Paul
I better really understand how that can be interpreted. -Craig
Some interviewees were more forgiving of the black box, acknowledging that a better predictive
model is more important than a model's interpretability:
I think the impact of the business is very strong if the model is not very, very precise, and I think
most people acknowledge that so I think we have much more flexibility there to be a little bit
more black box. –Alice
These diverging opinions highlight an aspect of understandability that is perhaps important to
discuss here. It is relevant to consider the domain expertise of the prospective users of the algorithm. It
seems logical that users who have a significant amount of algorithm expertise, as users or creators, would
38
evaluate a model's interpretability very differently. The domain knowledge factor is perhaps one of the
reasons that experiments testing the direct effect of interpretability on algorithm acceptance do not always
show that interpretability is a significant factor (Poursabzi-Sangdeh et al., 2018.)
A closely related factor to interpretability is explainability. In computer science, explainability is
generally understood to be: can the model sufficiently explain its processes and the reasons for the output
and provide insights into the main drivers of those outputs. Data analyst D understands algorithm A by
means of the algorithm’s own explanation E (adapted from De Regt, Leonelli, and Eigner, 2009, p 23) is
one such way of clarifying this view of explainability. For clarity, this definition is the computer or
algorithm explaining itself. If explainability is the algorithm's ability to explain itself, a reasonable
question to ask here might be "explain to whom?" Like with interpretability, it seems reasonable that the
explanation one user finds sufficient, is not at all suitable for someone with a different background.
Based on these comments, Understandability is defined as the extent to which a user feels that the
algorithm is not only interpretable but is also able to explain its steps/thinking processes. To be
interpretable, an algorithm must be readily understood by the user. To be explainable, an algorithm must
be able to make clear for a human user its processes, inputs and outputs. Understandability does not
involve an evaluation of the appropriateness or fit of a given algorithm to a problem, but rather just the
user’s ability to comprehend what the algorithm is doing.
Algorithm provenance. Although the participants in the study rated the importance of knowing
the source of the algorithm relatively low in terms of their willingness to accept its output (𝑥‾= 2.93,
𝜎=1.44), an interesting pattern emerged during the coding of the interviews in relation to an algorithm, its
component pieces, and its provenance or history. Provenance is an active area of research in databases
and information retrieval (Buneman, Khanna, and Tan, 2001) and is discussed in relationship to
computational tasks by Freire, Koop, Santos, and Silva (2020). Prat and Madnick (2008) discuss
provenance as "believability" and outline several dimensions related to believability, including the
trustworthiness of the source, the reasonableness of the data, and the temporality of the data. Through the
interview process, themes related to provenance as it pertains to algorithm adoption were generated and
39
are presented in Table 4. Overall, 63% of participants identified some form of provenance as being a
factor in adoption. The difference between the ratings on the survey and the interviews may relate to the
wording of the question on the survey. The survey question asked participants whether knowing the
author of an algorithm was an important factor. It is, of course, possible to know an author and have
differing assessments of the credibility of that source. In contrast to knowing, interviewees spoke on the
credibility of an author, algorithm history, and credibility of the process through which the algorithm was
generated.
Another aspect of algorithm provenance is the credibility of the algorithms author, creator, or the
firm that produced it. No consensus was found among interview participants concerning the credibility
and motivation of the authors of algorithms that they had experience with. Some participants noted that an
algorithm’s author deliberately lied, as Ted states, “We still had people that cooked the books.” Others
did not feel like the credibility or capability of an author of an algorithm is particularly pertinent, or felt
that it is unlikely that the authors had motivations that were not aligned with their own.
I've found that for the most part, the people who develop algorithms tend to do it out of necessity
for their own project or their own situation, and maybe sometimes for altruistic reasons. Not
necessarily for, 'Oh, I'm in this arena and I want everybody else to be worse at it than me.-
Cameron
The advantage is it is not subjective. It's objective. If you run it twice, you're going to get the
same answer instead of just running it and what they're feeling like that day. So that takes a lot of
the guesswork about what the motivation is behind the person making the recommendation. -Jake
Based on these comments, Algorithm Provenance is defined as the degree of trust a user has in an
algorithm because of a user’s assessment of the credibility of the advisor that developed the algorithm.
Algorithm Provenance is based on (a) the trustworthiness (or credibility) of an algorithm’s advisor
(person or firm), (b) the advisor’s ability to clearly explain the processes and steps by which input data
are transformed to create the output, and (c) the advisor’s ability to explain the algorithm’s output.
Algorithm provenance does not refer to a computer or algorithm's ability to explain its processes, but
rather it is specific to the human advisor or agent. Algorithm Provenance does not refer to the correctness
of the algorithm’s outputs or the data used as inputs in the algorithm.
40
The aspect of provenance most prevalent in the literature relates to the sources and history of the
data being used in the algorithm. Scholars have long been concerned with ensuring that the source of data,
or its combination and manipulation, is accurate (Buneman et al., 2001). This sentiment was echoed by
the participants, several of whom spoke about making efforts to ensure the source was correct. As Gordon
said: "We want to look at what are the main sources of data across a number of different business units.
With that, we want to look at what we call sources of truth." In the present study, this concept likely has
overlap with "Input Trust" and is measured as part of that factor.
Refining the Concept of Algorithm Adoption
Based on the interviews and review of the existing literature, this paper proposes that Algorithm
Adoption is a higher-order construct with four first-order sub-dimensions (Input Trust, Output Trust,
Understandability and Algorithm Provenance) that reflect the higher-order construct (see Figure 6).
.
Figure 5. Proposed Model for the Dimensions of Algorithm Adoption.
Relationship to Other Constructs
Podsakoff et al. (2016, p. 186) have noted that when defining a construct, it is critical to
distinguish it from other, related, constructs.
It is also important to differentiate the focal concept from other, related concepts. Indeed, the very
act of defining/labeling something (saying what it is) requires distinguishing it from other things
(saying what it is not). The benefit of doing this is that it (a) helps to distinguish the attributes that
define the focal concept from the attributes that define other, related concepts; (b) diminishes the
possibility of concept proliferation; and (c) identifies the concepts that could be used in empirical
tests of the measures of the focal concept’s discriminant validity (p. 186).
41
So, in the section that follows, I discuss the similarities and differences between Algorithm
Adoption the Technology Acceptance Model (Davis, 1989) and the trust in automation model introduced
by Lee and See (2004). The similarities and differences in these models that are discussed below are
summarized in Table 7.
Comparison to TAM
The three major models based on the technology acceptance model (TAM, TAM, and UTAUT)
are presented in Figure 7, panels A, B, and C, respectively. These models share some limited traits with
the algorithm adoption model (AAM). Conceptually at least, the perceived ease of use (PEU) in TAM and
understandability (in AAM) share a common antecedent in the form of complexity. As Thompson,
Higgins, and Howell (1994) discuss, an increased level of complexity makes it harder to use a given
technology. It stands to reason that complexity also impacts the two sub-dimensions of understandability,
interpretability, and explainability, in much the same way. A more complex system is, by its nature, more
difficult to interpret and explain.
Figure 7. Evolution of TAM models. Adjusted graphic from Holden, R. J., & Karsh, B. T. (2010).
42
Table 7
Comparisons of TAM, TAM2, UTAUT, and Automation Trust Concepts with AAM. Construct Definition Similarities Differences
TAM,
TAM2,
UTAUT
(Davis 1989;
Venkatesh
and Davis
2000;
Venkatesh
2003
TAM ≈ AAM. Both models seek to explain the factors related
to behavioral intention. TAM (acceptance) and AAM
(Adoption).
TAM ≠ AAM. TAM assumes that technology acceptance is
primarily driven by a technology’s perceived ease of use
(PEU) and perceived usefulness (PE). TAM is a broad model
and thought to cover most cases of technology. AAM, on the
other hand, assumes that an individual’s trust in an algorithm's
inputs and outputs are a primary factor in adoption. The AAM
model does not seek to explain technology acceptance overall
but treats algorithms as unique.
TAM technological ≠ AAM Trust/social..As described by
Gefin (2003), “Trust is a social antecedent. Perceived ease of
use and perceived usefulness are technological antecedents.”
Perceived Ease of Use
(PEU)
“The degree to which a
person believes that using a
system would be free of
effort” - Davis (1989, p.
320)
PEU (TAM) ≈ Understandability (AAM). These constructs
share a common antecedent in the form of complexity. As
complexity increases, PEU would decrease, as would an
algorithms interpretability and explainability.
PEU (TAM) ≈ Understandability (AAM). Although not
explicated in the model, PEU must be related to an
individual’s abilities and capabilities with that technology. In
AAM, the Understanding concept shares this trait. Different
individuals will perceive different levels of interpretability and
explainability based on their familiarity with that algorithm or
algorithms like it.
PEU ≠ Understandability. An algorithm may be very easy
to use, and yet not be understandable.
Perceived Usefulness
(PU)
"the degree to which a
person believes that using a
particular system would
enhance his or her job
performance" -Davis (1989,
p. 320)
PU ≠ Output Trust. A review of the items in PEU shows that
the primary focus is on increasing productivity or the
simplicity of using technology. These are technological
factors, not trust-evaluation factors.
43
Automation
Trust
(Various)
Automation Trust ≈ AAM. Trust precedes reliance in the
majority of automation trust conceptual models and AAM.
Automation Trust ≠ AAM. AAM explicitly separates the
trust a user may have with the inputs, outputs, or provenance
of an algorithm. Existing models describe trusting beliefs in a
more general way.
Dispositional/Propensity to
Trust
"The level of trust that exists
based on one’s past
interactions with the
machine." (Merrit, 2008, p.
197)
Propensity to trust ≈ Input/output trust. Propensity to trust
should be an antecedent to input and output trust.
Perfection Automation
Schema (PAS)
The expectation that
technology should behave
perfectly.
(Dzindolet et al. 2002)
Perfection Automation Schema (PAS) ≈ Output trust.
Output trust contains an element of PAS in that output trust
requires that “algorithm is believed to have performed its task
competently and correctly.”
Perfection Automation Schema ≠ Output trust. PAS is used
in the context of human decision making vs. automation
assisted decision making. The study considered an “all or
nothing” adoption or rejection of automation advice for each
decision. The factors of output trust are “degrees.”
Performance concept
Refers to the automation’s
reliability, predictability, and
ability.
(Lee & See, 2004)
Automation Trust ≈ Output trust. The trust in automation
model includes the concept of performance, which These
roughly correlate with output trust in terms of the user's belief
that the task was performed by the algorithm competently and
correctly.
Automation Trust ≠ Output trust. Output trust does not
require that an algorithm performs in a predictable way, but
that the output is reasonable and credible, and meets
expectations.
Faith
An aspect of belief beyond
the evidence that is available
(Rempel et al. 1985)
Faith ≈ Output Trust. Output trust includes conditions that
could be related to faith, such as "the algorithm is believed to
have performed its task competently and correctly."
Faith ≠ Algorithm Provenance. Trusting can be informed
through experience, evidence, and belief. Algorithm
provenance relates to an assessment of the credibility of the
algorithms creator. It does not rely on “blind faith” in that
creator.
≈ Indicates a concept is compared to or related to its counterpart. ≠ Indicates a concept is not equal to its counterpart and will be contrasted.
44
Although not explicated in the TAM model, PEU must also consider an individual’s abilities. The
ease of use for a given technology must be related to an individual’s abilities and capabilities with that
technology. In AAM, the understanding concept shares this trait. Different individuals will perceive
different levels of interpretability and explainability based on their familiarity with that algorithm or
algorithms like it.
Putting aside the trust dimensions in AAM, which are not present in TAM, these technology
acceptance models different from the AAM in two ways. First, TAM assumes that PEU and PE primarily
drive technology acceptance. TAM is a general model and designed to cover a large number of cases of
technology. AAM, on the other hand, assumes that an individual’s trust in an algorithm’s inputs and
outputs, the algorithm’s provenance, and the algorithm’s understandability are the primary factors in
adoption. The AAM model does not seek to explain technology acceptance overall but treats algorithms
as unique.
Second, although perceived usefulness would seem to share at least some overlap with the AAM
dimension of output trust, a closer examination of the items (Table 8) used to measure perceived
usefulness (Venlatesh, 2003) makes the differences clear. As indicated in this table, items such as "using
the system would make it easier to do my job" and "Using the system increases my productivity" focus on
technological aspects that relate to improving one’s productivity/job performance, and are not consistent
with the themes of output trust, such as the credibility of the outputs or meeting a user’s expectation.
Table 8
Items used to measure Perceived Usefulness and Performance Expectancy Attribute Items
Perceived
Usefulness
(Davis 1989)
1. Using the system in my job would enable me to accomplish tasks
more quickly
2. Using the system would improve my job performance.
3. Using the system in my job would increase my productivity.
4. Using the system would enhance my effectiveness on the job.
5. Using the system would make it easier to do my job.
6. I would find the system useful in my job
Performance
Expectancy (TAM)
(Venkatesh 2003)
1. I would find the system useful in my job.
2. Using the system enables me to accomplish tasks more quickly.
3. Using the system increases my productivity.
4. If I use the system, I will increase my chances of getting a raise
Table adapted from Venkatesh (2003).
45
Comparison to Automation - Trust
One of the most influential works on trust in automation is the conceptual model of trust and
reliance developed by Lee and See (2004). This automation trust (AT) model (see Figure 8) is built on the
work of Fishbein and Ajzen (1975), which posits that an attitude is an affective evaluation of beliefs,
leading to an intention to adopt automation. In addition, the AT model proposes a feedback loop, where
interaction with the automation and updating of information continues to form beliefs, leading to an
evolution of trust. This model is similar to the AAM framework in that trust plays a key role in both
models. For example, trust precedes reliance in the Automation-Trust model, and it precedes adoption in
the AAM framework. The trust in the AT model includes the concept of performance, which refers to the
automation’s reliability, predictability, and ability. These roughly correlate with output trust, in terms of
the user’s belief that the task was performed by the algorithm competently and correctly. In addition, the
dispositional propensity to trust included in the AT Models would be expected to influence both input
trust and output trust in the AAM framework.
Figure 8. Lee and See (2004) p. 68. Conceptual model of trust and reliance
46
Despite these similarities, the treatment of trust in the Lee and See (2004) AT framework differs
from AAM in several ways. First, AAM explicitly separates the trust a user may have with the inputs,
outputs, or provenance of an algorithm, whereas the AT framework describes trusting beliefs in a more
general way. This makes sense when the goal of the model is to consider automation broadly, but does not
fit into the way algorithms are either created or used.
Second, although both the AAM and the AT framework share the goal of improving user
information to support decision making (AAM through the concept of understanding, and AT through the
concept of information display), there are also some crucial differences. For example, AAM does not
require or discuss the format of the display as an important factor, whereas AT considers it important. An
example of this might be a simple calculator. Understandability in AAM requires explainability and
interpretability. If a user enters 2 + 2 on the calculator, and it returns 4, then a user can both interpret and
explain that calculator algorithm. It is inarguable that the 4 is a type of display, of course. This display
assists users by allowing them to see that the outputs are reasonable and credible, and meets the user’s
expectations, an important component of output trust. But extending this idea, the calculator itself, the
plastic, buttons, and screen, is the wrapper around the algorithm. This wrapper is also the display. That
display, the quality of the buttons, the feel of the calculator may have a significant impact on a user’s
decision to use that particular calculator. But, the technology tool adoption decision is separate from the
adoption or acceptance of the algorithm. For the purposes of the AAM model, the function inside the
calculator is what is essential.
The perfection automation schema (PAS) introduced by Dzindolet et al. (2002) has already
received some attention in this study. An experiment by Dietvorst et al. (2015) showed that when
algorithms err, they are relied on less. The concept of PAS is related to the concept within output trust of
"belief" that a task was performed competently by an algorithm. PAS also extends somewhat further than
the AAM model, in considering whether an automation acted in some predictable way. AAM has no
correlate for a predicted outcome, but this is at least somewhat similar to the output "meeting the user’s
expectations." Prediction and expectations do not have precisely the same meaning, but neither can they
be entirely separated. A particular outcome may be both predicted to occur and expected to occur. A
47
prediction implies a probable outcome of some future event. An expectation, on the other hand, implies a
more certain outcome. If a more certain outcome fails to occur, it is likely this triggers a more negative
emotional reaction from a user than a violation of a user’s prediction.
Although not specifically a model for automation trust, Rempel, Holmes, and Zanna (1985) have
been a foundation of later work and is relevant to the concept of algorithm adoption. Rempel et al. (1985)
describe one aspect of trust, faith, as an aspect of belief beyond the evidence that is available. The act of
trusting itself is informed through experience, evidence, and belief. The belief/fair framework differs from
the conceptualization of algorithm provenance in AAM, which relates to an assessment of the credibility
of the algorithm’s creator. It does not rely on “blind faith” in that creator. It is hard to argue that the
concept of faith has no relationship to output trust, however. Specifically, output trust incorporates within
its definition, "the algorithm is believed to have performed its task competently and correctly." This belief
could indeed be generated by a process such as faith.
Algorithms, from this paper’s perspective, will live within a wide variety of contexts. Those
contexts are not fully explored here, but it is, of course, essential to develop an understanding of a user’s
adoption of a given algorithm based on both the algorithmic component, technology usability,
accessibility, and display components. Table 8 provides a summary of the critical similarities and
differences between AAM, TAM, and the automation trust literature.
Antecedents of Algorithm Adoption
One of the goals of this research is to identify some of the key antecedents to algorithm adoption.
This was achieved through a review of the literature and an analysis of the interviews. The most
prominent antecedents in the literature come from the technology acceptance model. Performance
expectancy and effort expectancy could reasonably be considered as antecedents, as they are considered
before an algorithm is used.
As noted earlier, the literature on algorithm adoption is mostly experimental in nature, and very
little has been done to define or explore the construct outside of these experiments. However, these
experiments do highlight several potential antecedents. For example, studies on algorithm errors show
that within a study, participants rely on algorithms less after they err, or when the errors are not
48
predictable (Dietvorst et al., 2015; Prahl and Van Swol, 2017). The literature on trust suggests that
violating a user's trust by showing errors reduces the reliance on algorithms. The trust literature also has
several relevant antecedents to consider. Perhaps the most relevant, Dispositional Trust is defined by
Merritt and Ilgen (2008) as, "the level of trust that exists based on one’s past interactions with the
machine" (p. 197). Although the definition of dispositional trust explicitly discusses one's experience with
a particular machine, it is sensible to consider whether an individual's experience with all machines is a
relevant factor.
Interestingly, few antecedents to a decision maker’s adoption of an algorithm were identified in
the interviews. Even when asked directly about reasons for not adopting or using algorithms, very few
participants discussed errors in other algorithms being the cause of a change in their adoption intention for
all algorithms. However, as already noted, a large number of interviewees discussed rejecting the use of a
given algorithm when the outputs did not "make sense" to them.
Domain Expertise: Another important factor discussed by a large number of participants (47% of
interviews) was the role that domain knowledge or domain expertise plays in adopting decisions
generated by an algorithm. This was often mentioned in connection with or as an explanation for the
rejection of an algorithm when it did not "make sense."
If you use the algorithm, it would tell you something ridiculous...and it gets you ridiculous
results. So, in that case, we didn't use an algorithm, we used basically instincts, past experience,
interviews with salespeople -Jake
I think, for me, this is the balance between domain expertise and quantitative knowledge
experience. Whatever the word you want to use there. The domain expertise is that ... You know,
it's funny, because this is what I say to people is: How do you feel when you take that outlier and
put it into a sentence? –Miles
It is perhaps relevant here to recall the research of Kuncel et al. (2013), who found that a
mechanical combination of results outperformed human judgment. This is very much in alignment with
earlier research by Meehl (1954). Kuncel et al. (2013) state: "While recognizing that a strong preference
for expert judgment makes a complete change in practice unlikely" (p. 1070) and go on to list several
potential remedies for the case being investigated. It is interesting to consider how the preference of
experts to override and overrule, even in the presence of evidence that is contrary, can be moderated.
49
Not all of the discussions on domain expertise were about rejecting or adopting algorithms.
Participants spoke at length about things like asking the right questions, or understanding the output
within a business context:
You know your space very, very well and you know when something's good and when
something's not very good... I can say, you know what? There is value there. Nope, there is not
value there. -Edward
One of the most sought after skills for my team is having somebody that can merge the
quantitative aspect of what we do with the business environment -Sergio
The survey asked participants to rate their willingness to adopt an algorithm based on experience
with a particular algorithm (𝑥‾= 4.13, 𝜎=.82), and based on experience with all algorithms (𝑥‾= 3.70,
𝜎=0.95). Neither factor scored very high, and it is perhaps not surprising that experience with a particular
algorithm has a stronger relationship to adoption.
Benefits and Barriers to Algorithm Adoption
The benefits of adopting algorithms into decision making seem clear. A considerable amount of
literature has been published on the appropriate adoption of algorithmic advice. This was largely echoed
throughout the interview process, with 93% of subjects describing the benefits of using algorithms as
improving decision-making quality and efficiency. In particular, interviewees describe the ability of
algorithms to find and exploit new opportunities.
Algorithms give us the capacity to really take data, information, facts, and convert it into insights
that are actionable -Craig
The biggest (benefit) is the fact that it kind of removes your personal bias -Sergio
The benefit is it can do things faster, consistently, every single time the same way. That is hugely
powerful. Humans cannot do that. -Chuck
The biggest benefit is, I think it allows scientific decision-making, and it is quick. It's very, very
quick. -Darryl
The barriers to adopting algorithms also seem clear. Algorithms can be complex (40% of
interviewees) and can lack accessibility (43% of interviewees). These issues were most often discussed in
concert with comments about interpretability and explainability. Overall complexity was a driving force
50
in a lack of adoption in decision making downstream from the interviewees. No interviewees discussed
complexity as an impediment to their use of an algorithm.
Timeframe and execution capability are definitely factors for it not being implemented. -Jerry
…they just wouldn't believe it. It was too complicated -Judy
When asked about the biggest downside of using algorithms in decision making, two main
themes emerged. The first was a general concern that using an algorithm can lead decision-makers to
ignore other potentially more critical factors, or that a lack of a clear algorithmically derived answer
would limit any decision making at all.
I've seen leaders get hamstrung to the point they can't make a decision, because the data just isn't
clear enough for their level of comfort.- Matthew
The second downside to algorithm adoption was a sort of truth bias. For example, a number of
interviewees (30%) discussed seeing algorithms being adopted, even when they should not be. There has
been a substantial amount of attention on algorithm bias in the literature and the news, and the issue is one
that deserves more attention. It could be that some decision makers become resistant to adopting
algorithms into their decision making because of their inappropriate adoption elsewhere.
Sometimes algorithms can make people who may not have full understanding or a full grasp of
what we're trying to accomplish, the objective, make them feel smarter than they might be
because the algorithm is working so hard - Chuck
I think that sometimes people may get into this automatic situation of the algorithm is always
right…stop thinking about the problem - Sergio
These barriers are difficult problems to solve, as they invite discussion into not just the algorithm,
which by its nature is simply a code executing some command, to the psychological decision-making
traits, bias, ability and perhaps even willingness of the users to adapt and change to using new solutions.
Summary and Implications
Through an examination of the literature, and input from 30 practitioner interviews, this study
examined the nature of algorithm adoption and developed conceptual definitions for several proposed
sub-dimensions of the construct. An analysis of the interview content revealed that four common themes
51
adequately covered the majority of the rational decision makers gave when describing how they decided
to accept or reject the use of an algorithm:
• Input Trust: Mentioned by 83%
• Output Trust: Mentioned by 70%
• Algorithm Provenance: Mentioned by 63%
• Understandability: Mentioned by 57%
When discussing how and when they choose to adopt an algorithm into their decision making,
many interviewers used non-quantitative approaches like "makes sense" (mentioned by 60%) or the
variety and veracity of the inputs (mentioned by 57%). With only a few exceptions, very few described
techniques to quantitatively query or test the model prior to making a decision.
Understandability and its subdimensions of interpretability and explainability were also
frequently discussed factors. Improving model interpretability is an important topic at computer science
conventions and by computer science scholars. There has been significant improvement in building
interpretability frameworks for complex algorithms, such as those popularized by Ribeiro et al. (2016).
However, these advances in improving black box models do not seem to have reached practitioners yet.
Practitioners still prefer to use algorithms that are interpretable in and of themselves. This gap may have
significant consequences. As Lipton (2018) notes, the goals of increasing interpretability and avoiding the
use of black boxes may be directly at odds with the goal of improved predictive accuracy.
The short-term goal of building trust with doctors by developing transparent models might clash
with the longer-term goal of improving health care. -Lipton (2018, p. 21)
It seems clear from the interviews and academic work to date that a user's trust in an algorithm is
evaluative. In this conceptual scheme, users of an algorithm first evaluate the inputs, outputs, and
provenance of a given algorithm against factors that are important to them and base their trust in the
algorithm on this evaluation. As discussed in the examination of the literature, studies that attempt to test
the direct effect of these factors on algorithm aversion and adoption do not always have consistent or
intuitive results (Dietvorst, 2016; Logg, Minson, & Moore, 2018; Poursabzi-Sangdeh et al., 2018; Prahl &
52
Van Swol, 2017). Given a trust-evaluation scheme, this is perhaps not surprising. A more in-depth
examination of the factors that may mediate the trust-evaluation scheme users have with algorithms
would move the field forward substantially. The concept developed here of understandability may be one
such factor.
This study also examined the relationship between the proposed construct of algorithm adoption
and other similar constructs, such as those specified in the technology acceptance model (Davis, 1989)
and the literature on automation trust. There are some similarities between the proposed algorithm
adoption model and TAM. One similarity is the TAM model’s perceived ease of use concept and the
AAM model’s understandability concept sharing a common antecedent in the form of complexity. There
are some key differences as well. The TAM model primarily considers factors of acceptance to be
technological, while the AAM model considers trust-evaluation factors and understanding. A more
nuanced example of a difference is found between the TAM factor of perceived usefulness, and the AAM
concept of output trust. To be useful to a user, one may presume that the user would have to trust that
technology, but TAM makes no assertions regarding trust in a given technology. Output trust, on the other
hand, does not require that a given technology be useful in some way. It should be evident that any future
model of algorithm adoption carefully considers and integrates the concepts found in the technology
adoption literature.
When considering the similarities between the automation trust literature and the algorithm
adoption model, it is unavoidable to see the similarities across the different concepts. Various scholars
have discussed the relevance of factors such as propensity to trust (Merrit, 2008), the perfection
automation schema (Dzindolet et al. 2002), and the concept of automation performance (Lee & See,
2004). The contribution of AAM is a model explicitly targeting algorithm adoption and incorporating
many of the prior works into a single model.
Another goal of this study was to understand the antecedents of algorithm adoption. Surprisingly,
very few of the interview subjects discussed antecedents as it relates to their decision to accept or reject
an algorithm. The literature on trust in automation would seem to suggest that factors such as
dispositional trust should be significant (Merritt & Ilgen, 2008). In addition, 43% of interviewees
53
discussed the role that domain expertise plays into algorithm adoption decisions. This factor was most
often brought up in connection with other factors of trust, such as whether to trust the output of a given
model.
The benefits of adopting algorithms into decision making seem clear. Interviewees describe using
algorithms because these improve their decision making in terms of quality and efficiency. Many describe
algorithms as a crucial part of their decision-making process. The barriers to adopting algorithms were
also evident: Algorithms can often be complicated and lack accessibility. This complexity can lead to a
lack of understandability in an algorithm and diminishes a decision makers reliance on it.
Contribution
There is a long history in the academic literature on the importance of using models to assist in
the decision-making process. These algorithm advisors are often seen as making far better predictions and
decisions than their human counterparts. However, recent experiments (Castelo et al., 2019; Dietvorst et
al., 2015) highlight certain conditions under which this advice is underutilized. These concrete examples
of algorithm aversion and adoption help us understand what happens when various algorithm-generated
advice variables such as an algorithms subjectivity vs. objectivity are changed. However, they fail to
capture the concepts at a higher level, and so they do not adequately describe the domain of algorithm
adoption. Each of the experimental factors is like looking through a narrow lens and seeing only one part
of the answer.
The current research on the concept of algorithm adoption has been insufficient, with little to no
effort in understanding conceptually how human decision-makers take advice from algorithms. What
seems to be missing is a basic understanding and definition of the factors of algorithm adoption that will
serve to advance this construct of algorithm adoption as a theory. A gap this paper seeks to partially fill.
Clear definitions are useful in a number of ways.
a. Future research can start the process of developing content valid measures of the
Algorithm Adoption construct.
b. With clear definitions, researchers can test new relationships between the newly defined
algorithm adoption construct, and other variables such as TAM and trust factors.
54
For practitioners, the overwhelming support for the sub-dimensions can be a road map. First,
decision-makers can use this to understand their decision making processes further. Practitioners can
develop a deeper understanding of how factors such as their desire for an algorithm to ‘make sense’ can
lead them to under adopt algorithms. It is also useful to consider how an algorithm's provenance, such as
the credibility of the creator, may lead decision makers to adopt an algorithm that they should not adopt.
Second, for those involved in selling or distributing algorithmically generated advice, these findings
illustrate clear guidelines as to what decision-makers look for when considering the adoption of this
advice.
Conclusion
Making decisions with algorithms is growing in importance in the workplace and at home
(Agrawal et al., 2018). Existing literature in information systems and psychology have all developed
perspectives on adoption and use of technology in various forms. Literature from the Information Systems
(IS) field has shown us a strong relationship between the usability and usefulness of a technology and the
rate or amount of adoption (Davis, 1989; Venkatesh et al., 2003). But that research does not cover the
types of interactions that human decision-makers have with algorithms. The field of psychology is
undoubtedly developing the idea of trust and the perfection schema with algorithms, and finding that
when trust is violated adoption drops (Dietvorst, 2016; Prahl & Van Swol, 2017), but does not incorporate
the factors from relevant IS research. This study improves our understanding of the factors that lead to
adoption and builds a foundation for incorporating the disparate academic work to date.
Algorithms and their use in decision-making tend not to be discussed broadly. When they are
discussed, it is usually the adverse outcomes that are possible when delegating decision making to a
machine. Things like bias and the ways in which algorithms have erred tend to make better news stories
than the many ways algorithms can enhance and improve human decision making. Algorithms can be a
powerful decision-making aid, freeing people to do the critical tasks they do best and leaving the rest to
the machine. This is only possible if we develop a much deeper understanding and appreciation for how
we accept and incorporate algorithmic advice, confront the bias both in ourselves and in algorithms and
55
build a more algorithmic inclusive and aware decision-making paradigm. The management field is at the
forefront of the disruption that algorithms will create in the business world. By choosing to understand
and adapt to the changes, practitioners and scholars can develop a deeper understanding of how best to
ride this wave of disruption, instead of being drowned by it.
56
References
Agrawal, A., Gans, J., & Goldfarb, A. (2018). Prediction Machines: The simple economics of artificial
intelligence. Harvard Business Press.
Ajzen, I. (1991). The Theory of Planned Behavior. Organizational Behavior and Human Decision
Processes, 179–211.
Alvarez-Melis, D., & Jaakkola, T. S. (2018). Towards Robust Interpretability with Self-Explaining Neural
Networks. arXiv Preprint arXiv:1806.07538.
Bhattacharya, R., Devinney, T. M., & Pillutla, M. M. (1998). A formal model of trust based on outcomes.
Academy of Management Review, 459–472.
Bhattacherjee, A. (2001). Understanding information systems continuance: An expectation-confirmation
model. MIS Quarterly, 351–370.
Buneman, P., Khanna, S., and Tan, W. C. (2001). Why and where: A characterization of data provenance.
In J. VanDenBussche and V. Vianu (Eds.), Database Theory - Icdt 2001, Proceedings (Vol.
1973, pp. 316–330). Berlin: Springer-Verlag Berlin.
Castelfranchi, C., & Falcone, R. (2000). Trust and control: A dialectic link. Applied Artificial Intelligence,
14(8), 799–823.
Castelo, N., Bos, M. W., & Lehmann, D. R. (2019). Task-Dependent Algorithm Aversion. Journal of
Marketing Research, 809–825.
Colquitt, J. A., Sabey, T. B., Rodell, J. B., & Hill, E. T. (2019). Content validation guidelines: Evaluation
criteria for definitional correspondence and definitional distinctiveness. Journal of Applied
Psychology, 1243–1265.
Cook, J., & Wall, T. (1980). New work attitude measures of trust, organizational commitment and
personal need non-fulfilment. Journal of Occupational Psychology, 39–52.
Corbin, J., Strauss, A. L., and Strauss, A. (2015). Basics of Qualitative Research. SAGE.
Cormen, T. H. (2009). Introduction to algorithms. MIT press.
Cormen, T. H. (2013). Algorithms unlocked. Cambridge, Massachusetts: The MIT Press, [2013].
57
Crano, W. D., Brewer, M. B., & Lac, A. (2014). Principles and methods of social research. Routledge.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information
technology. MIS Quarterly, 319–340.
Dawes, R. M. (1979). The robust beauty of improper linear models in decision making. American
Psychologist, 571.
Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 1668–1674.
Definition of ACCEPT. 2020. In Merriam-Webster.com. Retrieved June 5, 2020, from
https://www.merriam-webster.com/dictionary/accept.
Definition of ADOPTING. 2020. In Merriam-Webster.com. Retrieved June 5, 2020, from
https://www.merriam-webster.com/dictionary/ADOPTING.
Definition of UNDERSTAND. 2020. In Merriam-Webster.com. Retrieved June 20, 2020, from
https://www.merriam-webster.com/dictionary/UNDERSTAND.
De Regt, H. W., Leonelli, S., & Eigner, K. (2009). Scientific understanding: Philosophical perspectives.
University of Pittsburgh Pre.
Deutsch, M. (1973). The resolution of conflict. New Haven, CT: Yale.
Diakopoulos, N. (2014). Algorithmic Accountability Reporting: On the Investigation of Black Boxes.
Dietvorst, B. J. (2016). People Reject (Superior) Algorithms Because They Compare Them to Counter-
Normative Reference Points (SSRN Scholarly Paper No. ID 2881503). Rochester, NY: Social
Science Research Network.
Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid
algorithms after seeing them err. Journal of Experimental Psychology: General, 114.
Dourish, P. (2016). Algorithms and their others: Algorithmic culture in context. Big Data & Society,
2053951716665128.
Dzindolet, M. T., Pierce, L. G., Beck, H. P., & Dawe, L. A. (2002). The perceived utility of human and
automated aids in a visual detection task. Human Factors, 79–94.
February 10, Katie Couric CBS, 2009, “Capt. Sully Worried About Airline Industry.” Retrieved
June 21, 2020 (https://www.cbsnews.com/news/capt-sully-worried-about-airline-
58
industry/).
Field, A., Miles, J., & Field, Z. (2012). Discovering statistics using R. Sage publications.
Fishbein, M., & Ajzen, I. (1975). Belief, attitude, intention and behaviour: An introduction to theory and
research (Vol. 27).
Freire, J., Koop, D., Santos, E., & Silva, C. T. (2020). Provenance for Computational Tasks.
Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1). Springer
series in statistics New York, NY, USA:
Fridman, L. (2018). Human-Centered Autonomous Vehicle Systems: Principles of Effective Shared
Autonomy. arXiv:1810.01835 [Cs]. Retrieved from http://arxiv.org/abs/1810.01835
Fridman, L., Ding, L., Jenik, B., & Reimer, B. (2017). Arguing Machines: Human Supervision of Black
Box AI Systems That Make Life-Critical Decisions. arXiv:1710.04459 [Cs]. Retrieved from
http://arxiv.org/abs/1710.04459
Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1). Springer
series in statistics New York, NY, USA:
Gefen, David, Elena Karahanna, & Detmar W. Straub. "Trust and TAM in online shopping: An integrated
model." MIS quarterly 27.1 (2003): 51-90.
Gillespie, T. (2014). The relevance of algorithms. Media Technologies: Essays on Communication,
Materiality, and Society, 167.
Gilpin, L. H., Bau, D., Yuan, B. Z., Bajwa, A., Specter, M., & Kagal, L. (2019). Explaining Explanations:
An Overview of Interpretability of Machine Learning. arXiv:1806.00069 [Cs, Stat]. Retrieved
from http://arxiv.org/abs/1806.00069
Goffey, A. (2008). Algorithm. In Software studies: A Lexicon, ed. Matthew Fuller, 15-20. Cambridge,
MA: MIT Press.
Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical
prediction: A meta-analysis. Psychological Assessment, 19.
59
Guest, G., & MacQueen, K. M. (2008). Handbook for Team-based Qualitative Research. Rowman
Altamira.
Harris, S., & Ross, J. (2006). Beginning algorithms. Indianapolis, Ind. : Wiley, 2006.
Hoc, J.-M. (2000). From human Machine interaction to human Machine cooperation. Ergonomics, 833–
843.
Holden, R. J., & Karsh, B. T. (2010). The technology acceptance model: its past and its future in health
care. Journal of biomedical informatics, 159-172.
Kitchin, R. (2017). Thinking critically about and researching algorithms. Information, Communication &
Society, 14–29.
Kowalski, R. (1979). Algorithm= logic+ control. Communications of the ACM, 424–436.
Kuncel, N. R., Klieger, D. M., Connelly, B. S., & Ones, D. S. (2013). Mechanical Versus Clinical Data
Combination in Selection and Admissions Decisions: A Meta-Analysis. Journal of Applied
Psychology, 1060–1072.
Lee, J. D., & See, K. A. (2004). Trust in Automation: Designing for Appropriate Reliance. Human
Factors, 50–80.
Lewis, H. R., & Papadimitriou, C. H. (1978). The efficiency of algorithms. Scientific American, 96–109.
Lipton, Z. C. (2018). The mythos of model interpretability. Queue, 16(3), 31–57.
Logg, J., Minson, J., & Moore, D. A. (2018). Algorithm Appreciation: People Prefer Algorithmic To
Human Judgment (SSRN Scholarly Paper No. ID 2941774). Rochester, NY: Social Science
Research Network.
MacKenzie, S. B. (2003). The dangers of poor construct conceptualization. Journal of the Academy of
Marketing Science, 323–326.
MacKenzie, S. B., Podsakoff, P. M., & Podsakoff, N. P. (2011). Construct measurement and validation
procedures in MIS and behavioral research: Integrating new and existing techniques. MIS
Quarterly, 293–334.
60
Malone, T. W. (2018). How Human-Computer’Superminds’ Are Redefining the Future of Work. MIT
Sloan Management Review, 34–41.
Marangunić, N., & Granić, A. (2015). Technology acceptance model: A literature review from 1986 to
2013. Universal Access in the Information Society, 81–95.
Mayer, R. C., Davis, J. H., & Schoorman, F. D. (1995). An integrative model of organizational trust.
Academy of Management Review, 709–734.
McKnight, D. H., Choudhury, V., & Kacmar, C. (2002). Developing and validating trust measures for e-
commerce: An integrative typology. Information Systems Research, 334–359.
McKnight, D. H., Cummings, L. L., & Chervany, N. L. (1998). Initial Trust Formation in New
Organizational Relationships. Academy of Management Review, 473–490.
Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the
evidence.
Merritt, S. M., & Ilgen, D. R. (2008). Not all trust is created equal: Dispositional and history-based trust
in human-automation interactions. Human Factors, 194–210.
Merritt, S. M., Unnerstall, J. L., Lee, D., & Huber, K. (2015). Measuring Individual Differences in the
Perfect Automation Schema. Human Factors, 740–753.
Moore, D. A., & Small, D. A. (2007). Error and bias in comparative judgment: On being both better and
worse than we think we are. Journal of Personality and Social Psychology, 972.
Moore, G. C., & Benbasat, I. (1991). Development of an instrument to measure the perceptions of
adopting an information technology innovation. Information Systems Research, 192–222.
Muir, B. M. (1987). Trust between humans and machines, and the design of decision aids. International
Journal of Man-Machine Studies, 527–539.
Mundra, S. B., & Dwivedi, G. K. (2013). Design and Analysis of Algorithms. University Science Press.
Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on amazon mechanical turk.
Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2016). Recommendations for creating better
concept definitions in the organizational, behavioral, and social sciences. Organizational
Research Methods, 19(2), 159–203.
61
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018).
Manipulating and measuring model interpretability. arXiv Preprint arXiv:1802.07810.
Prahl, A., & Van Swol, L. (2017). Understanding algorithm aversion: When is advice from automation
discounted? Journal of Forecasting, 691–702.
Prat, N., & Madnick, S. (2008). Measuring data believability: A provenance approach. In Proceedings of
the 41st Annual Hawaii International Conference on System Sciences (HICSS 2008) (pp. 393–
393). IEEE.
R Core Team. (2019). R: A language and environment for statistical computing. Vienna, Austria: R
Foundation for Statistical Computing.
R Core Team. (2020). R: A language and environment for statistical computing. Vienna, Austria: R
Foundation for Statistical Computing.
Rempel, J. K., Holmes, J. G., & Zanna, M. P. (1985). Trust in close relationships. Journal of Personality
and Social Psychology, 95.
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). Model-agnostic interpretability of machine learning.
arXiv Preprint arXiv:1606.05386.
Rogers, E. M. (2010). Diffusion of innovations. Simon and Schuster.
Rosenberg, N. (1982). Inside the black box: Technology and economics. Cambridge University Press.
Saldaña, J. (2009). The coding manual for qualitative researchers. Thousand Oaks, CA: Sage
Publications Ltd.
Sanders, N. R., & Manrodt, K. B. (2003). Forecasting software in practice: Use, satisfaction, and
performance. Interfaces, 90–93.
Schriesheim, C. A., Powers, K. J., Scandura, T. A., Gardiner, C. C., & Lankau, M. J. (1993). Improving
construct measurement in management research: Comments and a quantitative approach for
assessing the theoretical content adequacy of paper-and-pencil survey-type instruments. Journal
of Management, 385–417.
Schwartz, Z., & Cohen, E. (2004). Hotel Revenue-management Forecasting: Evidence of Expert-
judgment Bias. Cornell Hotel and Restaurant Administration Quarterly, 85–98.
62
Singh, I. L., Molloy, R., & Parasuraman, R. (1993). Automation-induced "complacency": Development of
the Complacency-Potential Rating Scale. The International Journal of Aviation Psychology, 111–
122.
Sniezek, J. A., & Van Swol, L. M. (2001). Trust, confidence, and expertise in a judge-advisor system.
Organizational Behavior and Human Decision Processes, 288–307.
Soll, J. B., & Mannes, A. E. (2011). Judgmental aggregation strategies depend on whether the self is
involved. International Journal of Forecasting, 81–102.
Stephens, R. (2013). Essential algorithms: A practical approach to computer algorithms. Indianapolis, IN
: Wiley, [2013].
Szajna, B. (1996). Empirical evaluation of the revised technology acceptance model. Management
Science, 85–92.
Thompson, R. L., Higgins, C. A., & Howell, J. M. (1994). Influence of experience on personal computer
utilization: Testing a conceptual model. Journal of Management Information Systems, 167–187.
Tversky, A., & Kahneman, D. (1975). Judgment under uncertainty: Heuristics and biases. In Utility,
probability, and human decision making (pp. 141–162). Springer.
Venkatesh, V., Brown, S. A., & Bala, H. (2013). Bridging the Qualitative-Quantitative Divide: Guidelines
for Conducting Mixed Methods Research in Information Systems. MIS Quarterly, 21–54.
Venkatesh, V., & Davis, F. D. (2000). A Theoretical Extension of the Technology Acceptance Model:
Four Longitudinal Field Studies. Management Science, 186–204.
Venkatesh, V., Davis, F. D., & Morris, M. G. (2007). Dead Or Alive? The Development, Trajectory And
Future Of Technology Adoption Research. Journal of the Association for Information Systems;
Atlanta, 267–286.
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information
technology: Toward a unified view. MIS Quarterly, 425–478.
VERBI Software. (2020). MAXQDA 2020 [computer software]. Berlin, Germany: R Foundation for
Statistical Computing.
63
Yaniv, I., & Kleinberger, E. (2000). Advice taking in decision making: Egocentric discounting and
reputation formation. Organizational Behavior and Human Decision Processes, 260–281.
Yeomans, M., Shah, A. K., Mullainathan, S., & Kleinberg, J. (2017). Making sense of recommendations.
Management Science.
64
Appendix 1: Practitioner Interview Guide
Introduction:
Good morning (afternoon). Thank you for agreeing to meet with me today for an interview. The entire
interview should last approximately 45 minutes to an hour.
The purpose of this study is to understand how decision makers use algorithms in their work. For our
purposes, an algorithm is any well-defined computational procedure that takes some values as inputs and
produces some values as outputs. The procedure identifies both a problem and the steps by which it
should be solved. An example of an algorithm may be a program that takes historical sales data as an
input and makes a prediction on future sales. Another example may be an algorithm that uses your past
search behavior on a website and predicts what you should be shown on the screen.
Recording:
If it is okay with you, I will be recording our conversation. The purpose of the recording is so that I can
get all the details and review your answers in detail later, while at the same time engaging you in the
conversation. All of your comments and responses will remain confidential.
Consent Form:
(If the participant has not already completed the consent form)
Before we get started, please take a few minutes to read this consent form and sign it if you agree to
participate.
(Hand participant the consent form/preamble.)
(After participant returns consent form, turn recording device on.)
Questions
Introduction:
1. To start, I’m interested in learning a bit more about you, the organization you work for, and what you
do in your current role.
a. Tell me about your organization (e.g., products/services, numbers and types of employees
etc.
b. Tell me about your role within the organization (e.g. division/group, reporting
relationships, primary responsibilities, etc.) and what you do in your job.
c. How many years of professional experience do you have?
Key Questions
2. A good way to develop an understanding of how you use algorithms in your job is through
examples. Please tell me about a specific project where an algorithm was used successfully to
improve a decision being made? (If the following are not answered ask:)
a. What was the specific project you were working on?
b. What was the problem the algorithm was designed to address?
c. What was your role on the project?
d. Describe the algorithm? (What was the algorithm? What were the algorithms
inputs and outputs?)
e. Why did you use the algorithm to help make this decision?
65
f. What was the outcome of the use of the algorithm?
3. Thinking about the project you've described, what are the characteristics of the algorithm that
you feel helped you make successful decisions?
4. Please tell me about a time when you were given the opportunity to use an algorithm in a
decision, and you chose not to use that algorithm. (If the following are not answered, ask:)
a. What was the specific project you were working on?
b. What was your role on the project?
c. What was the problem the algorithm was designed to address?
d. Describe the algorithm? (What was the algorithm? What were the algorithms inputs
and outputs?)
e. Why did you choose to use ___ instead of the algorithm?
5. Thinking about the time you’ve described, what are the factors that you take into
consideration to reject the use of an algorithm?
6. When deciding whether or not to use a given algorithm to make decisions, what factors do
you use to judge whether the inputs to the algorithm are appropriate?
7. When deciding whether to use a given algorithms recommendation, what factors do you use
to judge whether the outputs from the algorithm are appropriate?
8. From your perspective, what are the greatest benefits of using algorithms in your job? Why do
you consider these the biggest benefits?
9. In contrast, what do you consider the biggest downsides of using algorithms in your job? Why do
you consider these the biggest downsides?
Probing Questions
That's interesting could you explain that a little more…
Let's see, you said ... just how do you mean that? What do you mean by…
How come…. Could you describe… Can you give me an example of …
Tell me about the … What do you think about…? Was this what you expected?
Could you say some more about that? What do you mean by that . . .?
10. On a scale of 1 to 5 (with 1 being “not an important part of my decision to use an algorithm” to a
5 being “an important part of my decision to use an algorithm”) please rate your willingness to
adopt and use an algorithm to help you make decisions in your job:
a. Past experience with using the specific algorithm
b. Past experiences with using other algorithms
c. Perceived ease of use
d. The nature of the algorithms inputs
e. Perceived usefulness to your job
f. Quality of information obtained from the algorithm
g. Trust in the outputs of the algorithm
h. Subjective norms of the organization
i. The interpretability of the algorithm
j. Knowing who the algorithm was developed by
losing Question