cost, accuracy, and subjective fairness in legal ... · era-of-legal-startups-is-now-in-session/...

COST, ACCURACY, AND SUBJECTIVEFAIRNESS IN LEGAL INFORMATION

TECHNOLOGY: A RESPONSE TOTECHNOLOGICAL DUE PROCESS CRITICS

JAY THORNTON*

The United States spends substantially more as a percentage of GDP on legal ser-vices than most other countries. Simultaneously, various indicators suggest this out-sized spending does not result in public perceptions of greater fairness or justice.While the digital automation of legal work offers the potential to help address thisproblematic paradigm, the legal academy’s reception of automation in law has beencritical. This Note responds to these criticisms by showing the demonstrable objec-tive and subjective fairness benefits that legal automation can achieve—all whilereducing costs.

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1822I. USE OF AUTOMATED SYSTEMS IN LAW . . . . . . . . . . . . . . . . . 1826

A. Administration of Public Benefits Systems . . . . . . . . . . 1827B. Online Dispute Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . 1829C. Wider Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1832

II. CHIEF SCHOLARLY CRITICISMS OF

AUTOMATED SYSTEMS IN LAW . . . . . . . . . . . . . . . . . . . . . . . . . . 1833A. Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1834

1. Complexity and Transparency . . . . . . . . . . . . . . . . . . . 18352. Bias Cutting Both Ways . . . . . . . . . . . . . . . . . . . . . . . . . 1839

B. In Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1840III. SOCIAL PSYCHOLOGY AND AUTOMATED DECISION

SYSTEMS IN LAW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1840A. Frameworks to Evaluate Subjective Fairness . . . . . . . . 1841B. Voice and Process Control in

Automated Decision Systems . . . . . . . . . . . . . . . . . . . . . . . . 1843C. Addressing Perceived Errors with

Limited Human Oversight . . . . . . . . . . . . . . . . . . . . . . . . . . 1846D. Fairness in Promptness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1847E. In Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1848

CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1849

* Copyright © 2016 by Jay Thornton. J.D., 2016, New York University School of Law.

1821

1822 NEW YORK UNIVERSITY LAW REVIEW [Vol. 91:1821

INTRODUCTION

Today a variety of new legal technology companies stand ready toautomate a diversity of tasks once exclusively completed by humanlawyers. Although scholarly criticisms of legal automation abound, therise of automation offers a cost-effective, accessible, accurate, andsubjectively satisfying alternative to traditional human-based legalprocesses. Such technological alternatives arise at a time when legalservices have become largely beyond the financial reach of the middleand lower classes,1 and even our nation’s wealthiest companies haveincreasingly implemented drastic measures to control apparently end-lessly mounting legal costs.2

Several studies demonstrate that the United States dedicates anenormous amount of resources to the legal market; at least one studycomparing legal outlays as a percentage of gross domestic product(GDP) has found overall legal spending in the United States to be 2.6times greater than the average level in Europe, and 5.5 times greaterthan in Japan.3 At the same time, annual satisfaction surveys con-

1 See William D. Henderson & Rachel M. Zahorsky, Paradigm Shift, 97 A.B.A. J. 40,42 (2011) (explaining that, in Connecticut, over 80% of divorces have a self-representedparty and 90% of criminal defendants are either self-represented or utilize government-provided lawyers). This situation will likely continue; in recent decades, legal fees haveincreased substantially, while middle and lower class incomes have stagnated. Id. at 46.

2 See infra note 8 and accompanying text (describing various cost-cutting methodsemployed by companies).

3 U.S. CHAMBER INSTITUTE FOR LEGAL REFORM, INTERNATIONAL COMPARISONS OF

LITIGATION COSTS: CANADA, EUROPE, JAPAN, AND THE UNITED STATES 2, add.ix (2013),http://www.instituteforlegalreform.com/uploads/sites/1/ILR_NERA_Study_International_Liability_Costs-update.pdf (comparing overall legal costs across several differentcountries). This study found U.S. liability cost—a phrase used by the study to describe thecost of claims, whether resolved through litigation or alternative dispute resolutionsystems—as a percentage of GDP to be 1.66%. Id. at 2. Other studies and scholarsestimate that U.S. legal spending is even greater, nearer to 2% of GDP. Henderson &Zahorsky, supra note 1, at 41; see also Joshua Kubicki, Make That $400 Billion for USLegal Market Size, LEGAL TRANSFORMATION INST. (Feb. 24, 2014), http://legaltransformationinstitute.com/blog/2014/2/22/make-that-400-billion-for-us (estimating the size of theU.S. legal market at above $400 billion, although direct legal services are only $274 billion);see also Jason F. Cohen, The Japanese Product Liability Law: Sending a Pro-ConsumerTsunami Through Japan’s Corporate and Judicial Worlds, 21 FORDHAM INT. L.J. 108, 123(1997) (noting that “[t]he United States has twenty-five times as many attorneys percapita” than Japan). A recent survey of corporate legal departments conducted by theSearle Center on Law, Regulation, and Economic Growth at Northwestern Universityfound that “[a]s a percent of revenue, multi-national company respondents to the surveyspend a disproportionate amount on litigation in the United States relative to theirexpenditures in foreign jurisdictions. Depending on the year, relative U.S. costs werebetween four and nine times higher than non U.S. costs (as a percent of revenue).” U.S.CHAMBER INSTITUTE FOR LEGAL REFORM, LITIGATION COST SURVEY OF MAJOR

COMPANIES (2010), www.uscourts.gov/file/document/litigation-cost-survey-major-companies.

December 2016] COST, ACCURACY, AND SUBJECTIVE FAIRNESS 1823

ducted by the World Justice Project suggest these high costs offerscant offsetting benefit. The surveys show that the accessibility andrespect given to the rule of law is fairly low as compared to othernations, in spite of our justice system’s outsized financial footprint.4The World Justice Project’s Rule of Law Index 2015 ranked theUnited States at number thirteen out of twenty-four within itsregional grouping.5 Compared to nations with high income-levels, theUnited States was ranked at number nineteen out of thirty-one.6Other sources also reflect a certain societal frustration with the legalservices sector.7 Put simply, our nation spends more as a proportion ofeconomic output on legal services than most others, but our citizenssurprisingly do not, according to these measures, perceive these out-lays to generate increased fairness or justice.

Corporate consumers of legal services have responded by struc-turally reforming how they obtain these services: by sourcing legalwork to lower cost offshore markets, shifting work to more cost-effi-cient, in-house legal departments, and most crucially, by automatingtasks previously done by human lawyers.8 Simultaneously, there has

4 See Anjanette H. Raymond & Scott J. Shackelford, Technology, Ethics, and Accessto Justice: Should an Algorithm Be Deciding Your Case?, 35 MICH. J. INT’L L. 485, 488–90(2014) (discussing the World Justice Project’s Rule of Law Index and its findings related toaccessibility to legal services and costs).

5 WORLD JUSTICE PROJECT, WORLD JUSTICE PROJECT RULE OF LAW INDEX 2015 21(2015), http://worldjusticeproject.org/sites/default/files/roli_2015_0.pdf (grouping theUnited States with other countries from the European Union, European Free TradeAssociation, and North America).

6 Id. at 22. Although the United States underperforms many of its peers on the globalstage, it is worth noting that in the global rankings, the United States was ranked atnumber nineteen out of a total of 102 countries. Id. at 20.

7 See Jeffrey Fagan, Introduction to Legitimacy and Criminal Justice Symposium, 6OHIO ST. J. CRIM. L. 123, 123 (2008) (“Surveys of public opinion over four decadesconsistently show that Americans have little confidence in the fairness or effectiveness ofthe criminal justice system and criminal law more generally.”); Rachel M. Zahorsky, It’sNot Just Money Fears Blocking Access to Legal Help; Lawyer Distrust Is Growing, A.B.A.J.: LAW SCRIBBLER (Dec. 1, 2012, 8:20 AM), http://www.abajournal.com/magazine/article/its_not_just_money_fears_blocking_access_to_legal_help_lawyer_distrust_is_g (reportingthat citizens were not only dissuaded from engaging attorneys to handle their legal claimsfor financial reasons, but because of distrust and negative views of the legal profession).

8 See Henderson & Zahorsky, supra note 1, at 44–45 (describing the increased relianceon in-house attorneys and electronic legal services to reduce costs imposed by corporatelaw firms); Law Firms: A Less Gilded Future, ECONOMIST (May 5, 2011), http://www.economist.com/node/18651114 (“The legal business has undergone not only recession butalso structural change. Ever-growing profits are no longer guaranteed. Nor, for some firms,is survival.”). Mari Sako at Oxford’s Said Business School undertook an extensive surveyand interview with the general counsels of several dozen of the largest U.K. and U.S.-basedcompanies. Her research chronicles how in-house legal departments increasingly drive costsavings by leveraging new tools such as competitive bidding processes for law firms,disaggregating and standardizing processes for their full or partial automation, use of legalprocess outsourcing firms, removing expensive lawyers from doing tasks others could do


been growth in digital alternatives, such as online legal informationrepositories or automated tools, available to otherwise legally under-served populations.9 Both top-price corporate legal consumers andlow-price individual consumers seem to be responding, at least par-tially, to the high costs and low perceived utility of traditional legalprocesses by turning to automation. These changes have inspired aboom in venture capital funding for dozens of start-ups seeking toautomate legal work.10

Despite apparently sound reasons for this growing shift, the legalacademy has displayed general hostility towards technology’s replace-ment of manual processes. Most scholars involved in this area advo-cate grafting more manual (human-based) processes, humanoversight, and other costly delays onto the automated systems thatwere created to address high legal fees and related burdens.11 Many ofthese reasoned criticisms fundamentally ignore the untenably highcosts and low user satisfaction that has motivated the substitution ofinformation technology in legal processes.12 They also tend to overem-

(“de-lawyering”), and simply not doing marginal legal work. MARI SAKO, GENERAL

COUNSEL WITH POWER? 11, 14–21 (2011), http://eureka.sbs.ox.ac.uk/4560/1/General_Counsel_with_Power.pdf. The key problem as explained by one interviewee is: “Let’s be honest.Lawyers are not good at providing either systems or process or handling volumes.” Id. at21. Many of these findings appear also in a 2014 Chief Legal Officer Survey conducted byAltman Weil of 186 chief legal officers, which chronicles how only 4% of those surveyed“are satisfied with the traditional legal service delivery model” and over two-thirds ofthose surveyed are displacing traditional legal service providers with technological tools.ALTMAN WEIL, 2014 CHIEF LEGAL OFFICER SURVEY iii, 7 (2014), http://www.altmanweil.com/CLO2014/.

9 See Henderson & Zahorsky, supra note 1, at 46–47 (discussing the structural shiftcaused by the introduction of technology in the legal marketplace).

10 See, e.g., Nicole Bradick, All Rise: The Era of Legal Startups Is Now in Session,VENTUREBEAT (Apr. 13, 2014, 8:32 AM), http://venturebeat.com/2014/04/13/all-rise-the-era-of-legal-startups-is-now-in-session/ (“In 2009, just 15 legal startups were listed onAngelList. There are now more than 400 startups and almost 1,000 investors.”). Venturefunding for legal technology in 2012 and 2013 is estimated at over $500 million. Id.

11 See infra notes 67–72 and accompanying text (detailing scholarly criticisms of ADS).12 Crawford and Schultz propose extending the purview of due process beyond those

systems used by government for adjudicating public rights and responsibilities to anysituation where a predictive algorithm may be used to sort and select individuals for anyprivate sector opportunities, suggesting that for any Big Data decisions, “the affected partymust receive an opportunity to present an argument, evidence, and corrections toprejudice.” Kate Crawford & Jason Schultz, Big Data and Due Process: Toward aFramework to Redress Predictive Privacy Harms, 55 B.C. L. REV. 93, 115–17 (2014).Kenneth Bamberger acknowledges that the “risk control [platforms] that regulationdemands simply cannot function without the data collection, analyzing, and monitoringcapacities of integrated computer technology,” yet he nevertheless advocates for “externalinstitutions that can foster accountability in technology choices” and “much more intenseregulator involvement in oversight and accountability with a threat of sanction.” KennethA. Bamberger, Technologies of Compliance: Risk and Regulation in a Digital Age, 88 TEX.L. REV. 669, 673, 726, 729 (2010). In writing about credit scores and other automated


phasize certain cases when such systems have malfunctioned13 or haveotherwise led to worrisome results.14 By amplifying a few cases whenautomated decision systems (ADS) have gone wrong—despite theirubiquitous implementation—this scholarship ignores the systematiccost, accuracy, and consistency gains that legal ADS can offer.15 ThisNote argues that the ongoing evolution in legal ADS—and ADS’inherent malleability and scalability—actually makes such systems farmore responsive than many scholars anticipate to the objective andsubjective notions of fairness that undergird legitimate legal systems.

By carefully analyzing the systematic objective and subjectivebenefits of legal ADS, which are underappreciated or ignored byscholars writing on this topic, this Note finds good reason to supportfuture legal automation. The argument proceeds in three parts. Part Idiscusses how automation is already reshaping the contours of thelegal system and how some courts are responding. Part II summarizesthe major scholarly criticisms of legal ADS and presents the objectiveadvantages of such systems in addressing longstanding challenges sur-rounding legal complexity, consistency, and bias. Part III employsempirical findings of the social psychology of procedural justice schol-arship to argue legal ADS deliver subjective fairness better than manymanual processes. Because legal systems function as much by moralsuasion as by active adjudication, accounting for these subjective fair-ness parameters will be crucial to allowing our society to realize theobjective benefits that legal ADS offer.16

ranking algorithms, Citron and Pasquale claim that “[i]f law and due process are absentfrom this field, we are essentially paving the way to a new feudal order of unaccountablereputational intermediaries.” Danielle Keats Citron & Frank Pasquale, The Scored Society:Due Process for Automated Predictions, 89 WASH. L. REV. 1, 19 (2014).

13 See Danielle Keats Citron, Technological Due Process, 85 WASH. U. L. REV. 1249,1256–57 (2008) (chronicling examples of how the public benefits administration system inColorado consistently generated improper results, how digitally generated No-Fly listfrequently detained the wrong persons, and how Federal Parent Locator Service routinelymisidentified individuals accused of delinquency on child support, among other failures).

14 See Crawford & Schultz, supra note 12, at 94–95 (discussing how Target predicted,with some degree of success, the pregnancy of certain shoppers before such individuals hadannounced it by tracking individual purchases and other data mining techniques).

15 See Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, 17RICH. J.L. & TECH. 11 (2011) (detailing how automated systems in e-Discovery likelyproduce more accurate results than massive manual review); J. Melissa Perry, Fed. Courtof Austl., iDecide: The Legal Implications of Automated Decision-Making, Speech at theCambridge Centre for Public Law Conference: Process and Substance in Public Law (Sept.15, 2014), http://www.fedcourt.gov.au/publications/judges-speeches/justice-perry/perry-j-20140915# (discussing potential for greater accuracy through using IT in law, as well asrelated risks). See Part I, infra, for judicial recognition of these points.

16 See Rebecca Hollander-Blumoff & Tom R. Tyler, Procedural Justice and the Rule ofLaw: Fostering Legitimacy in Alternative Dispute Resolution, 2011 J. DISP. RESOL. 1, 6–10


In making this argument, this Note seeks to reframe the dialoguesurrounding legal ADS in three ways. First, the objective performanceof such systems should be analyzed in terms of their systematic poten-tial to provide consistent application of the increasingly complex legaland regulatory regimes characteristic of the modern world, ratherthan on anecdotal examples of alleged errors. Second, the subjectivelegitimacy of such systems should be analyzed in light of empiricallydetermined causal factors, such as perceptions of legal fairness, ratherthan through criticisms that unquestioningly equate fairness withtraditional forms of legal practice. Finally, any evaluation of legalADS must account for our nation’s comparatively outsized spendingon legal services. Framed this way, and despite the heretofore hostilescholarly reception, the evidence suggests wider use of legal ADS canboth improve the objective and subjective fairness of our legal systemwhile simultaneously controlling costs.

IUSE OF AUTOMATED SYSTEMS IN LAW

Over two decades ago, a prescient scholar, Henry Perritt, Jr.,foresaw the potential marriage of information technology with legalpractice.17 He authored a detailed operational description for howcomprehensive computer-based “Adjudication ManagementSystem[s]” and “Rulemaking Management System[s]” could auto-mate, standardize, and streamline many legal processes to make themless costly.18 Perritt criticized the false, but persistent, perception thatthe U.S. legal system crafted only individualized outcomes instead of“assembly line justice,” or the resolution of a large volume of casesthat are often highly repetitive.19 Today, diverse parties seek to lev-erage software’s scalability and consistency advantages—recognizedseveral decades ago by Perritt—to better serve the interests of justicethrough automating manual human processes for a large number ofsimple, repetitive matters. The next sections chronicle two especiallypromising applications of ADS—the first related to public law and thesecond to private law—to legal processes previously completed by

(2011) (discussing the reasons that paying attention to subjective perceptions of proceduraljustice in designing systems can crucially affect the overall societal efficacy of suchsystems).

17 See Henry H. Perritt, Jr., The Electronic Agency and the Traditional Paradigms ofAdministrative Law, 44 ADMIN. L. REV. 79 (1992) (arguing that the adoption of computersto perform administrative law functions would advance the traditional substantive goals ofadministrative law).

18 Id. at 82–85.19 Id. at 98.


human operatives, in order to give an understanding of the currentstate of this technology and its potential.

A. Administration of Public Benefits Systems

Computers have substantially displaced human legal operatives inthe administration of various government benefit programs. Tradition-ally, public benefits, such as welfare or other large-scale governmententitlement programs, were administered by hundreds of human oper-ators who used form templates to determine the benefits a party couldreceive. Gandy and Tepperman’s sociological study of early automa-tion in social welfare organizations illustrates the transformationswrought by early technology.20 At one county’s Department of SocialServices, for example, before the implementation of the automatedsystems, agency employees would conduct site visits, interviews, andaccumulate relevant data before making their own determinationwhether an individual was eligible for various government social ser-vices.21 Following the implementation of the new automated system,human operatives went from actually calculating various income fac-tors for eligibility, to simply imputing the data on standardized param-eters for the computer to make this determination.22 The substantivedifference, albeit an early example, is that automated logical schemaaccept quantifiable data inputs to make legal determinations of eligi-bility, whereas before humans made such determinations with greaterflexibility and discretion.23

In the decades since Gandy and Tepperman’s study, legal ADShave expanded considerably in the administration of government ben-efits. Since the 1970s, the federal government has heavily subsidizedthe implementation of such systems at the state level.24 Interestquickly developed in aggregating the administration of different social

20 See generally JOHN M. GANDY & LORNE TEPPERMAN, FALSE ALARM: THE

COMPUTERIZATION OF EIGHT SOCIAL WELFARE ORGANIZATIONS (1990). This studysurveyed over one hundred social welfare organizations regarding the effects ofcomputerization upon their internal processes. Detailed site visits and interviews werecollected for a smaller subset of these organizations. See id. at 29–34 (describing how thestudy was completed).

21 Id. at 35–37 (describing the range of work-related activities completed manually).22 Id. at 47–48, 63–66.23 Id. at 21–25 (discussing the equitable and societal improvements and concerns

presented by automation); see also Terrence Maxwell, Information Federalism: History ofWelfare Information Systems 9 (The Nelson A. Rockefeller Inst. of Gov’t., Working Paper,1999), http://www.rockinst.org/pdf/workforce_welfare_and_social_services/1999-09-information_federalism_history_of_welfare_information_systems.pdf (noting that automationnarrowed discretion and potential for error in welfare determinations).

24 See id. at 5–6 (discussing early efforts by the federal government to incentivizestate’s movement toward automated systems).


benefit systems by “cross-check[ing]” information across differentagencies, such as between social service systems and the InternalRevenue Service systems.25 Government initiatives leveraged newtechnology, hoping to pursue new functionality and further integra-tion throughout the 1990s.26 Today this has resulted in a thicket ofsystems that use various standardized inputs in order to make legaldeterminations, which often serve as inputs for other decision systems.

A recent decision from California’s Court of Appeals analyzedthe objective benefits of such systems.27 Pich v. Lightbourne consid-ered allegedly erroneous benefits terminations arising from the imple-mentation of CalWIN, a unified benefit management platform with aprice tag of $744 million.28 CalWIN makes initial eligibility determina-tions, calculates the amount of benefits allowed, generates warningsregarding potential benefit termination, and, crucially, has the capa-bility to “automatically terminate benefits or impose a penalty.”29

While automated benefit administration systems had been used inthese counties previously, such systems would not automatically exe-cute an adverse decision without human involvement.30 Although theplaintiffs in Pich had their benefits reinstated after an administrativeappeal,31 they still filed suit claiming that “CalWIN had systematicprogramming flaws that caused erroneous and automatic benefit ter-minations, reductions, and delays to thousands of benefit recipi-ents.”32 In affirming the trial court’s demurrer, the appeals courtfocused its analysis on two crucial points. First, it explored whetherthe trial court incorrectly refused to enforce the State Department ofSocial Services’ “nondelegable duty . . . to ensure timely issuance ofbenefits.”33 Second, it considered whether CalWIN’s implementationand operation violated statutory timing requirements or due processprotections surrounding benefit issuance.34 These questions requiredthe court to analyze CalWIN’s operational accuracy as well as itssupervisory framework.

25 See id. at 6.26 See id. at 14–19 (discussing attempts to further integrate automated systems while

noting that complexity in regulation prevented the design of any unitary-automatedsystems and led to new single-function systems being designed for each welfare program).

27 See Pich v. Lightbourne, 164 Cal. Rptr. 3d 388 (Ct. App. 2013).28 Id. at 391–93.29 Id. at 392–93.30 Id. (“Prior to the use of CalWIN. . . eligibility continued until the caseworker took

action to discontinue benefits.”) It should be noted there was a manual override that gaveemployees the discretion to override CalWIN’s determinations if needed. Id. at 486.

31 Id. at 393.32 Id.33 Id. at 395.34 Id.


The court found that the manner in which the Department ofSocial Services “supervises the counties to ensure benefits are timelyand correctly delivered is a matter of discretion.”35 For this reason, thecourt concluded the Department did not need to intervene every timean error was made, but only if the counties’ processes were “substan-tially failing.”36 In answering this follow-on question, the court made anumber of interesting observations. First, looking at instances of alleg-edly tardy benefit distributions, the court explained that in all of thesecases the system functioned properly, “but the caseworkers hadnot.”37 Second, while the court acknowledged some improper benefitterminations, it similarly concluded that “[t]he system’s incorrectautomatic terminations are the result of staff error, not CalWINerror,” for which revisions in training procedures had already beenimplemented.38 The court determined that a few errors—resultingfrom human mistakes—in the largest benefit system in the country didnot constitute “substantial failure” under the relevant law.39 Whilesome social benefits system implementations have allegedly beenmore problematic,40 this court at least recognized the high consis-tency, accuracy, and scalability of CalWIN and the tendency of mosterroneous outcomes to be of human origin. In other words, in the eyesof this court, CalWIN demonstrated greater fidelity to principles ofobjective fairness than human operatives. This reaction contrastsmarkedly from the scholarly criticisms discussed below.

B. Online Dispute Resolution

While the above systems automate the application of explicit(although often complex) public law benefits rules, a recent flurry ofsoftware developments seek to automate a mainstay of traditionallegal practice: the resolution of potentially multifaceted and unstruc-tured disputes between parties. One of the leaders in the online dis-pute resolution field is a company called Modria, which leverages

35 Id. at 397. The court explains that governing state statutes “directed the [SocialServices] Department, when supervising the counties, to be flexible, and to aid them, asopposed to direct them, in establishing their own methods of operation.” Id. at 493.

36 Id. at 398–99.37 Id. at 400–01. The court also explained that the Department and CalWIN’s remedial

efforts to prevent future reoccurrence of late payments clearly reflected the Departmentwas properly supervising the system’s administration. Id.

38 Id. at 406–07.39 Id.40 See Gerry Smith, How a Government Computer Glitch Forced Thousands of Families

To Go Hungry, HUFFINGTON POST (Mar. 27, 2014), http://www.huffingtonpost.com/2014/03/27/food-stamp-delays-north-carolina_n_4994822.html (overviewing numerouscomputer-based benefits program glitches affecting thousands of people).


automated processes developed at eBay to resolve tens of millions ofdisputes each year.41 A Modria user inputs a dispute resolution policy,often included as part of a sales or service contract, which addressesissues such as “refunds, returns, exchanges and chargeback policies.”42

Upon the filing of a dispute by a customer, the program collects rele-vant information from the user’s software platform and from the cus-tomer making the complaint, such as customer information, productinformation, shipping data, as well as the written complaint itself.43

The presence of specified factors may be set to automatically triggercertain outcomes. For example, long-time or repeat customers withoutany history of disputes might automatically be granted whole or par-tial refunds.44 The software determines relevant dispute parametersby applying various templates that were developed by modeling ear-lier disputes resolved through the platform.45 If an automatic decisionrule is not triggered, the software scours the relevant data to identifypoint(s) of contention and suggests various options for settlement.46

Using dispute mapping, the system proposes a number of drop-downmenus whereby the terms for a potential settlement can be hammeredout through digitally structured interchanges between the parties.47

Modria claims its system permits the resolution of up to 90% of dis-putes without any costly human intervention.48 But if either party isunsatisfied with the result or cannot come to agreement through the

41 See The Modria Platform, MODRIA, http://modria.com/product/ (last visited Dec. 31,2015); see also Humayun Khan, Modria Launches Dispute Resolution Tool to Scale FormerEbay and PayPal Tech, BETAKIT (Nov. 19, 2012), http://betakit.com/modria-launches-dispute-resolution-tool-to-scale-former-ebay-and-paypal-tech/ (explaining the applicationof eBay technology to the dispute resolution platform on a pay-per-use basis).

42 How It Works, MODRIA, http://modria.com/how-it-works/ (last visited Nov. 30,2016). While premised on using private dispute resolution policy as the guiding rules forthe software, any contractual framework or other legal doctrine could be substituted intothe program instead.

43 Id.44 Id.45 See Modria Launches Global Platform for Online Conflict Resolution, STEAMFEED

(June 22, 2015), http://www.steamfeed.com/modria-launches-global-platform-for-online-conflict-resolution/ (discussing how Modria’s Fairness Engine was developed frommodeling eBay disputes).

46 See Paul Sawers, Ebay and PayPal Spin-Off Modria Launches Its Conflict-ResolutionPlatform, THENEXTWEB (Nov. 19, 2012), http://thenextweb.com/apps/2012/11/19/ebay-and-paypal-spin-off-modria-launches-its-online-conflict-resolution-platform-for-businesses/.

47 See Sarah Kessler, Ebay Spinoff Modria Is Judge Judy for Cyber Shoppers, FAST

COMPANY (Feb. 5, 2013), http://www.fastcompany.com/3005402/ebay-spinoff-modria-judge-judy-cyber-shoppers (“In some cases, the person with the complaint puts together asolution proposal using pre-populated drop-down menus. That proposal is sent to the otherparty in the dispute, who can counteroffer,. . . [f]or more complicated cases, Modria offersthe same kind of mediation and arbitration that a court would.”).

48 See How It Works, supra note 42.


digitally structured process, the platform facilitates the involvement ofan independent third party mediator or arbitrator who uses the evi-dentiary record digitally compiled in the earlier phases of the dis-pute.49 While Modria initially targeted small or multi-jurisdictionaldisputes that would otherwise be left unresolved, current plans sug-gest an intent and capability to resolve disputes with tens of thousandsof dollars or more at stake.50 The company conceptualizes its softwareas the “Fairness Engine,” a theme this Note will explore in Part III.51

Online dispute resolution platforms, like Modria, have receivedlimited and superficial attention in case law. A number of courts havesimply found that the existence of an online dispute resolution optiondoes not preclude adjudication in traditional courts, or analyze theenforceability of clauses requiring online dispute resolution underunconscionability doctrine.52 One case offers only a passing commenton how the disputants before the court previously considered using anonline dispute resolution mechanism before ultimately declining to doso.53 One of the few cases that substantively addresses the merits ofautomated dispute resolution systems involved information gatheredby Experian’s online dispute resolution tool, in the context of a claimfor inaccurately reporting a debt.54 In this case, the plaintiff com-plained that this tool unfairly “restricted her to select[ing] from a pre-filled set of dispute categories.”55 In part because Experian was volun-tarily dismissed from this case, the court concluded that it was reason-able for the defendant to rely on data collected by Experian’s system,even if its program limited the data entry options available to users.56

Taken together, the few cases in which automated dispute resolutionwas directly at issue suggest that courts tend to perceive ADS as com-parable to other kinds of alternative dispute resolution options and,

49 See id.; see also Kessler, supra note 47.50 See Frederic Lardinois, Modria Launches a “Fairness Engine” for Online Dispute

Resolution, TECHCRUNCH (Nov. 19, 2012), http://techcrunch.com/2012/11/19/modria-launches-a-fairness-engine-for-online-dispute-resolution/ (explaining Modria’s model forclaim valuation).

51 Khan, supra note 41.52 See Attaway v. Omega, 903 N.E.2d 73, 80 (Ind. Ct. App. 2009) (holding that eBay’s

dispute resolution service does not preclude adjudication in state court); Stenzel v. Dell,Inc., 2004 Me. Super. LEXIS 108, *4–8 (Me. Mar. 10, 2004) (analyzing enforceability ofclauses requiring online dispute resolution under the unconscionability doctrine in thecontext of claims for overcharging sales taxes and unfair or deceptive business practices,but upholding the provision).

53 Carmel v. Fleishman, 2005 Cal. App. Unpub. LEXIS 8082, *29 (Cal. App. 2d Dist.Sept. 7, 2005) (discussing a dispute regarding breach for real estate purchase contract).

54 Gustafson v. Experian Info. Sols., Inc., 2015 U.S. Dist. LEXIS 71280 (C.D. Cal. June2, 2015).

55 Id. at 13.56 Id. at 10–14.


therefore, not deserving of any heightened skepticism. Some adminis-trative agencies, on the other hand, such as the Ohio Board of TaxAppeals, have affirmatively embraced Modria’s technology for use intheir own adjudication processes.57

C. Wider Application

The above-discussed examples of legal ADS in all likelihood willnot be the endpoint for legal automation, but only the initial incur-sions in a vast, prolonged, and disruptive evolution. Future applica-tions of IT systems to legal processes include diverse paradigm-changing possibilities. For example, certain start-ups are working oncontracts that autonomously perform themselves.58 As anotherexample, companies increasingly rely on comprehensive softwaresuites to manage regulatory responsibilities and to ensure their opera-tions satisfy legal requirements across many jurisdictions.59 This shiftis motivated by the “increased transaction volumes, increased com-plexity in financial offerings, proliferating compliance and reportingrequirements across business lines, and the [resulting] massiveaccumulation of data” characteristic of the modern regulatory envi-ronment.60 Such concerns are reflected in the astounding growth incompliance and regulatory software spending in recent years.61

The crucial thing to realize is that despite traditionally distinctlabels applied to dispute resolution between parties (discussed in Sec-

57 See Press Release, Modria, Ohio Board of Tax Appeals (BTA) Selects ModriaResolution Center to Power New Online Case Management System (Dec. 4, 2014), http://www.prnewswire.com/news-releases/ohio-board-of-tax-appeals-bta-selects-modria-resolution-center-to-power-new-online-case-management-system-300004748.html(“Taxpayers, agents, and attorneys can now file online, gain instant access to theirelectronic case files, negotiate settlements, and take actions on cases, all from theconvenience of their computers or tablets.”).

58 See, e.g., Robert Ambrogi, Startup Uses “Internet of Things” to Enable Contracts toPerform Themselves, LAWSITES (Oct. 25, 2016), http://www.lawsitesblog.com/2016/10/startup-uses-internet-things-enable-contracts-perform.html (discussing “Clause,” a startupworking “to make contracts ‘come alive’ using a combination of data and the internet ofthings”).

59 David Cau, Governance, Risk and Compliance (GRC) Software: Business Needs andMarket Trends, DELOITTE, http://www2.deloitte.com/content/dam/Deloitte/lu/Documents/risk/lu_en_ins_governance-risk-compliance-software_05022014.pdf (last visited Sept. 16,2016) (discussing the growing use of comprehensive compliance software solutions).

60 Bamberger, supra note 12, at 685.61 See Steve Morgan, Cybersecurity Market Reaches $75 Billion in 2015; Expected to

Reach $170 Billion by 2020, FORBES (Dec. 20, 2015), http://www.forbes.com/sites/stevemorgan/2015/12/20/cybersecurity%E2%80%8B-%E2%80%8Bmarket-reaches-75-billion-in-2015%E2%80%8B%E2%80%8B-%E2%80%8Bexpected-to-reach-170-billion-by-2020/ (“The global enterprise governance, risk and compliance (GRC) market isexpected to grow from $5.8 billion in 2014 to $11.5 billion by 2019, at a CAGR of 14.6% forthe period 2014 to 2019, according to MicroMarketMonitor.”).


tion I.B), the allocation of public rights or responsibilities (discussedin Section I.A), or using digital models to predict and avoid adverselegal or regulatory action (discussed just above), the fundamental pro-cess behind each of these approaches is highly related. That is, a com-puter model is developed that accounts for the relevant legal norms—which guide either dispute resolution, the allocation of rights andresponsibilities, counseling a client, or other traditional legal activi-ties—which are applied to specific facts input from a particular situa-tion. That in one instantiation the automated model is used to resolvedisputes between parties ex post, while in another it is used to avoid adispute ex ante (by predicting likely outcomes and allowing for behav-ioral changes) means they should be seen as two sides of the sametechnological coin. The widely applicable nature of the process forinvolving ADS in legal matters suggests that future technologicaladvances will likely introduce automated systems into additional legalissues in surprising ways, while also handling increasing portions oflegal work done by humans today.

Growth in legal ADS will redefine the processes behind our legalsystem, as software overtakes roles traditionally performed by humanlegal professionals and blurs traditionally distinct legal functions. Thisblurring also suggests new cultural contours for how our society willconceptualize and interact with the law. As discussed next, unlike thecautiously neutral evaluations by courts explored above, a group ofscholars writing on “technological due process” generally seeks toresist these looming changes to our legal system, in spite of the objec-tive and subjective advantages they offer.

IICHIEF SCHOLARLY CRITICISMS OF AUTOMATED

SYSTEMS IN LAW

Scholars writing about the need to develop a novel notion of“technological due process” chronicle an array of concerns whereinformation technology and law intersect. This group’s criticisms ini-tially focused on government benefit systems, but later it extendedtheir criticisms to various privately owned and operated systems.Some of their criticisms are more persuasive than others. One persua-sive (if overstated) concern is that when taking legal requirements andturning them into computer code, “translation” errors will be made,such that the decisional framework prescribed by the legislature orregulatory agency differs from that applied by the computer code.62 Inresponse to these concerns and other potential errors in programming,

62 Citron, supra note 13, at 1261–63; Bamberger, supra note 12, at 706–10.


these scholars suggest a number of ex ante solutions, including morerigorous testing,63 releasing source code,64 more participatory automa-tion processes,65 and/or not coding agency interpretations that do nothave the full force of law.66

The second set of criticisms advocate for greater ex post humanmonitoring of automated decisions. These scholars argue comprehen-sive human oversight (presumably by a lawyer) is necessary for avariety of reasons, such as “automation bias” (wherein operators tendto blindly trust computer outputs),67 the desire for greater humaninvolvement in justice,68 discomfort with the perceived opacity ofADS,69 and the need to identify system inaccuracies.70 The most inva-sive proposals suggest a form of notice, opportunity to be heard,impartial adjudicators, and judicial review for every individualaffected by automated decisions.71 Some even argue, as noted above,that these rigorous requirements should even apply to private organi-zations, who do not meet the state actor requirement.72

A. Responses

This Note endorses the core of the first criticism, while rejectingthe notion that legal ADS are inherently less reliable, or less fair, thanhuman-based systems.

Appropriate ex ante software design and error testing cannot beargued against; they are crucial components of software design bestpractices. Government or private users of automated systems alreadyhave the contractual tools to specify appropriate levels of accuracyand reliability in the systems they build or purchase, and can make theresulting cost tradeoffs accordingly. Even at today’s standard level of

63 Citron, supra note 13, at 1310–11.64 Id. at 1308–09.65 See id. at 1280–81 (discussing risk inherent when automation processes do not

involve a high level of human participation).66 See id. at 1309–10 (noting that releasing source code may require an agency to

engage in notice-and-comment rulemaking to change coded policy).67 Id. at 1271–73.68 See, e.g., Frank Pasquale, Restoring Transparency to Automated Authority, 9 J.

TELECOMM. & HIGH TECH. L. 235, 236–39 (2011) (describing flaws that have developed inthe move to automated processes and implying the need for some form of humanintervention in order to maintain transparency).

69 Id. at 248–52.70 Citron, supra note 13, at 1279; see also Citron & Pasquale, supra note 12, at 8.71 See, e.g., Citron, supra note 13, at 1249–50 (noting that “a carefully structured

inquisitorial model of quality control can partially replace aspects of adversarial justice thatautomation renders ineffectual”).

72 See Crawford & Schultz, supra note 12, at 93–94, 127 (2014) (noting that the use ofdata analytics in the private sector should be subject to due process); see also Citron &Pasquale, supra note 12, at 6 (same).


programming accuracy, a number of courts have, as discussed above,already recognized that legal ADS can produce more accurate out-comes than manual processes. On this point, it should be noted thatone example of legal ADS gone awry used by technological due pro-cess scholars to condemn the reliability and fairness of legal ADS wasCalWIN, where, as discussed above, courts eventually concluded thathuman user error caused the problems.73 Moreover, accuracy and reli-ability should further improve as programmers and governments gainfurther experience with legal ADS. Taken together, the technologicaldue process advocates rightly emphasize the importance of proper exante design and testing, but practically, the accuracy or reliability con-cerns that motivate their criticism on these points appear overstated.74

Accepting the importance of appropriate ex ante system designleaves us to consider the other major suggestion of the technologicaldue process movement: the need for more human oversight to restrainthe inherent potential unfairness of legal ADS.75 Returning to a cen-tral theme of this Note, one wonders whether a country alreadyspending multiples more, as a percentage of GDP, than its peers onlegal services could afford extensive human oversight of every elec-tronic decision made by ADS. Such oversight would also likely imposesubstantial secondary costs on innovation. Even putting primary andsecondary cost issues aside, however, these arguments are still notpersuasive because they postulate an idealized and unrealistic visionof human capacity while ignoring the considerable subjective andobjective benefits of legal ADS. The next section discusses two waysin which the limits of human cognition favor automated decisions inlegal processes.

1. Complexity and Transparency

Contemporary regulatory regimes and legal processes havebecome so complex that individuals have little choice but to rely uponsimplifying technologies. The growing complexity of the legal systemis reflected, in addition to the legal spending figures discussed above,by numerous other indicators: an exponential growth in the volume of

73 See supra text accompanying notes 33–40 (explaining that CalWIN’s incorrectautomation terminations were the result of staff error, not system error).

74 Id.75 See Citron & Pasquale, supra note 12, at 21–22 (advocating a perplexing tier of

double licensing for users of predictive statistical methods wherein government licensesvarious licensing organizations that in turn have authority to license companies usingpredictive statistics); see also supra notes 12, 72, and accompanying text (outliningsuggestions for more human oversight).


materials available for pre-trial discovery,76 the steady expansion inthe quantity of statutory crimes,77 and the proliferation of quantitativeand metric-based regulatory requirements.78 Computers offer theability to address multi-faceted decisions that implicate diverse regula-tory bodies with a thorough consistency that human decision makerscould never realistically achieve.79 To give a specific example, therecent Dodd-Frank financial reform imposed an additional thousandpages, or so, of requirements—not including any follow-on regula-tions—onto an already highly regulated sector.80 Companies facedwith satisfying these rules report that without computer-based compli-ance software, it would not be possible to comply with such intricateand detailed regulation.81 Individual taxpayers’ rapid adoption of taxpreparation software likely is similarly motivated by the fact that evena middle class income tax return could be affected by hundreds of

76 See Michael R. Arkfeld, Proliferation of “Electronically Stored Information” (ESI)and Reimbursable Private Cloud Computing Costs, LEXISNEXIS (2011), http://www.lexisnexis.com/documents/pdf/20110721073226_large.pdf (reporting that “[c]lient [d]ata is[d]oubling [e]very [t]hree [y]ears”).

77 See John S. Baker, Jr., Revisiting the Explosive Growth of Federal Crimes, HERITAGE

FOUNDATION (June 16, 2008), http://www.heritage.org/research/reports/2008/06/revisiting-the-explosive-growth-of-federal-crimes (measuring the growth of federal criminal offensesand attempting to pinpoint the reasons for this expansion); see also Richard Heaton,Foreword to OFFICE OF PARLIAMENTARY COUNSEL, WHEN LAWS BECOME TOO COMPLEX:A REVIEW INTO THE CAUSES OF COMPLEX LEGISLATION (2013), https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/187015/GoodLaw_report_8April_AP.pdf (discussing the phenomenon of creeping growth in statutory law’scomplexity in United Kingdom and analyzing current and historical efforts to control thistendency); Michael J. Bommarito II & Daniel M. Katz, A Mathematical Approach to theStudy of the United States Code, 389 PHYSICA A 4195 (2010) (quantitatively measuring thecomplexity in the United States Code and finding “that in the recent past, the Code hasgrown in its amount of structure, interdependence, and language”).

78 See Bamberger, supra note 12, at 692–702 (describing technology products andsystems geared toward corporate risk management and compliance).

79 See James Grimmelmann, Note, Regulation by Software, 114 YALE L.J. 1719,1722–24 (2005) (exploring similarities between software and law and examining whichregulatory contexts best lend themselves to regulation by software).

80 See Over-Regulated America, ECONOMIST (Feb. 18, 2012), http://www.economist.com/node/21547789 (“The home of laissez-faire is being suffocated by excessive and badlywritten regulation.”); Ron Ashkenas, Is Dodd-Frank Too Complex to Work?, HARV. BUS.REV. (Mar. 13, 2012) (discussing the complexity of Dodd-Frank and difficulties ofimplementation).

81 See Chris Cumming, Dodd-Frank Drives Growth of Compliance Software Budgets,Survey Finds, AM. BANKER (Apr. 23, 2013), http://www.americanbanker.com/issues/178_78/dodd-frank-drives-growth-of-compliance-software-budgets-1058554-1.html. Thissame phenomenon can even be seen with something as comparatively simple as managingbenefit distribution rules, which, as chronicled above, increasingly requires automationgiven their growing complexity and overlapping areas of responsibility. See supra SectionI.A.


often interdependent rules.82 Some scholars report that lawmakersand key regulators are already behaving in ways that presume the pri-macy of legal technology by how they choose to regulate, in man-dating technical requirements as part of substantive regulation.83 In away then, legal automation may in turn enable the ongoing growth inlegislative and regulatory complexity.

Complexity-driven automation implicitly raises concerns abouttransparency and the need for human oversight of the software usedto implement highly complex laws.84 While intuitively appealing, thesecriticisms fall flat upon closer examination for a variety of reasons. Tobegin with, transparency in traditional legal processes serves the cru-cial function of ensuring fidelity to relevant legal rules and as a checkon improper human bias.85 But in the case of automated decision sys-tems, assuming the software is properly developed and tested,allowing ex post manual reevaluation of every digitally resolved issuewill not have the same salutary effects as it would for manualprocesses. Unlike a human decision maker, whose reasoning mightchange on the knowledge it will be exposed to the public in a writtenopinion ex post, automated decision systems will apply their definedlogical protocol without regard to ex post exposure.

Second, transparency for ex post review of legal ADS could exac-erbate serious potential weaknesses: If their decisional heuristic canbe modeled, workarounds can be readily developed that underminethe desired substantive results of the system.86 ADS are less able than

82 Cf. Christopher Ingraham, Charted: The Skyrocketing Complexity of the Federal TaxCode, WASH. POST WONKBLOG (Apr. 15, 2015), https://www.washingtonpost.com/news/wonk/wp/2015/04/15/charted-the-skyrocketing-complexity-of-the-federal-tax-code/ (“Backin 1940, the 1040 form had only two pages of instructions—pretty manageable, even withthe small type. But you’d have to wade through 207 pages of instructions to fullyunderstand the 2013 form.”).

83 See Bamberger, supra note 12, at 672 (explaining how legislatures increasinglypromulgate laws that seem to expect data-based and automated compliance mechanismsgiven the complexity).

84 See Pasquale, supra note 68, at 236–37 (raising concerns about the inability ofoutside observers to understand the incentives that underlie automation systems anddesignations).

85 Tom Tyler explains that “[a]uthorities benefit from openness and explanation,because it provides them an opportunity to communicate evidence that their decisionmaking is neutral.” Tom R. Tyler, Procedural Justice, Legitimacy, and the Effective Rule ofLaw, 30 CRIME & JUST. 283, 298 (2003) (discussing how the efficacy of administrative orjudicial institutions is linked to public perceptions of their fairness). Assuming properprogramming procedure and system testing, a transparent explanation to every automateddecision system participant is neither a necessary nor sufficient means to ensure neutralityof decisionmaking.

86 See, e.g., Allison Chang et al., How to Reverse-Engineer Quality Rankings, 88MACHINE LEARNING 369 (2012) (describing methods to reveal proprietary ranking models,even without transparency).


human legal decision makers to autonomously respond to gamingbehavior. For example, knowing in great detail how credit agenciesevaluate creditworthiness would permit some un-creditworthy partiesto selectively perform certain behaviors, purely for the purpose offalsely inflating their credit scores, which could result in potentiallycostly defaults.87 Absolute transparency in such a case expresslyundermines the purpose and function of such processes: to efficientlyallocate credit. This reality is acknowledged by many advocating more“transparent” human oversight of automated decision making sys-tems.88 Finally, as referenced above, automated decision systems arefrequently implemented to address legal issues of such complexity thathumans cannot readily administer them, which strongly suggests thatmaking such systems more open to ex post human review will havelimited utility.

The calls for minute human oversight of legal ADS also appeardisproportionate when one considers the degree of transparency inmanual human legal decision making today. First, as several promi-nent twentieth-century legal scholars argued, the decisions renderedby human legal decision makers often have rationales that differ fromthose expressly stated in a written opinion.89 Second, long, complex,and often technical court opinions—to say nothing of opaque jurydeterminations, which by definition give no rationale for their conclu-sions—are scarcely intelligible to most citizens.90 Finally, given thatsuch a small proportion of criminal or civil disputes are actuallyresolved through a full adjudication before a court, the terms andrationales for the resolution of a sizeable proportion of modern legaldisputes remain largely unknowable to the public, and frequentlyunintelligible to the parties. So not only does minute ex post humanreview of legal ADS not provide the same incentives for accuracy as it

87 Martin Mayer, Credit Rating Agencies in the Crosshairs, BROOKINGS INST. (Aug. 31,2010), http://www.brookings.edu/research/articles/2010/08/31-ratings-agencies-mayer(describing how credit agencies overstated the creditworthiness of some commercialinstitutions leading up to the Great Recession).

88 See Pasquale, supra note 68, at 236 (noting that individuals who fully understand anautomated system could game it).

89 See PAUL W. KAHN, THE CULTURAL STUDY OF LAW 24–25 (1999); see also WOUTER

DE BEEN, LEGAL REALISM REGAINED: SAVING REALISM FROM CRITICAL ACCLAIM 7(2008) (noting that in prior eras, “the law in the books was of only limited help inpredicting what judges would decide”).

90 See JEROME FRANK, COURTS ON TRIAL: MYTH AND REALITY IN AMERICAN JUSTICE

258 (1963) (“The conventions of judicial opinion-writing—the uncolloquial vocabulary, theuse of phrases carrying with them an air of finality . . . lend an air of thorough certainty,concealing the uncertainties inherent in the judging process. . . . [T]he average judicialopinion is so worded that, at best, only lawyers can comprehend it.”); see also AdamLiptak, Justices Are Long on Words but Short on Guidance, N.Y. TIMES (Nov. 17, 2010),http://www.nytimes.com/2010/11/18/us/18rulings.html?_r=0.


does with human decision makers, such proposals implicitly hold auto-mated decision systems to a level of transparency far beyond thatdelivered by traditional manual legal processes. Taken together, thesetwo factors undermine much of the force of these transparency-basedcriticisms, which seek to broadly restrain automation in law byrequiring minute human oversight of it.

2. Bias Cutting Both Ways

Motivating the technological due process school’s desire for rad-ical transparency and greater human oversight for automated legalprocesses is a profound fear of bias. The two kinds of bias at issue inthese criticisms differ from their usual manifestations. The first bias-related concern involves how humans rapidly become reliant on auto-mated decision making systems, such that they come to trust their out-comes to the exclusion of their own judgment and become blind tosystem failures.91 This is especially true for decisional systems thatapply complex interconnected sets of rules which users may not beable to fully understand. The second kind of bias cited by technolog-ical due process scholars involves the possibility that computer pro-grammers will make certain improper assumptions in coding legalnorms.92 These concerns at worst constitute a second-order fear giventhat readily available countermeasures exist to address them. Solu-tions addressing user automation bias include making those over-seeing automated decision systems “socially more accountable” foroutcomes, as well as introducing occurrences of “rare automation fail-ures” in relevant training procedures.93 As discussed above, propersoftware development techniques and thorough testing ex ante canprevent improper coding of laws.

Compared to these limited, readily-mitigated instances of bias,the use of automated decision systems in law can reduce the risk ofthe most insidious, pervasive, and even unintentional forms of biasthat have traditionally been the greater preoccupation of our society.Even positing a legal system where no decisionmaker consciouslydesires to ground his choices on an improper basis, extensive neuros-cience research suggests that the very mental areas used to renderlegal judgments employ neurological structures inherently susceptible

91 See J. Elin Bahner et al., Misuse of Automated Decision Aids: Complacency,Automation Bias and the Impact of Training Experience, 66 INT’L J. HUM. COMPUTER

STUD. 688–89 (2008) (discussing background research on how automated decision aids canlead to higher error rates as operators become complacent).

92 See Citron, supra note 13, at 1262 (describing instances of programmersunconsciously inserting bias into code).

93 Bahner et al., supra note 91, at 690, 696.


to unintentional bias.94 A reproducible set of empirical findings showcertain physiological features consistently correlate with subconsciousassociations that affect human decision making.95 Legal ADS, on theother hand, will only render decisions on the parameters for whichthey were programmed. The benefits of using legal ADS to mitigateaccidental human biases far outweigh the bias concerns raised by tech-nological due process advocates, which can be largely addressedthrough proven countermeasures.

B. In Sum

While the technological due process scholars are clearly correctabout the need for proper software development, testing, and usertraining, their advocacy of radical transparency and minute humanoversight misses the mark for several reasons—even if we cast asidethe crucial issue of cost savings. First, automated systems are able todeal with highly complex and multifaceted legal frameworks thathuman operatives simply cannot holistically oversee. Moreover, evenif humans had the capacity to individually review every action of com-plex legal ADS, manual oversight and human transparency in tradi-tional legal processes often fall far short of the platonic ideal ascribedto them and would not incentivize automated systems to operate withgreater accuracy. Second, properly designed and tested legal ADS, infact, offer one of the most promising prospective methods to shortcircuit both tacit and explicit human biases, offering consistent legaloutputs to all parties. In spite of these clear objective benefits, whatmay bother many critics are the potential subjective fairness concernsapparently raised by such systems. We turn to these questions next.

IIISOCIAL PSYCHOLOGY AND AUTOMATED DECISION

SYSTEMS IN LAW

Even acknowledging the objective advantages of automated deci-sion systems in law, it is unlikely those advantages can ever be realizedunless users subjectively perceive automated legal systems to be fair.96

94 Kimberly Papillon, The Court’s Brain: Neuroscience and Judicial Decision Making inCriminal Sentencing, 49 CT. REV. 48, 49 (2013) (discussing the neuroscience behindadministering criminal sanctions and noting the propensity to activate parts of the neuro-anatomy that use biases).

95 Id. at 51.96 See Tyler, supra note 85, at 284 (discussing how the efficacy of administrative or

judicial institutions is linked to public perceptions of their fairness); see also E. ALLAN

LIND & TOM R. TYLER, THE SOCIAL PSYCHOLOGY OF PROCEDURAL JUSTICE 19–21 (1988)(discussing a study on the effect of human biases on outcomes when conducting a repeatinquiry).


The body of research guiding the analysis below, known as the socialpsychology of procedural justice, consists of several decades’ worth ofstudies on what factors lead individuals to perceive legal or proceduralsystems as fair.97 A core set of findings developed in this literature areworth paying attention to because they have been consistently repro-duced in both field studies and laboratory experiments.98 This Partargues that ADS in law perform well on empirical measures of subjec-tive fairness today. Furthermore, IT’s scalability and malleabilitymeans that systems designers can make future legal ADS even moreresponsive to subjective fairness indicators in the future.99 This Partfirst evaluates the favorable performance of legal ADS on severalsocial psychology frameworks that measure subjective fairness of legalstructures, before discussing how legal ADS will likely become moreresponsive to subjective fairness drivers moving forward.

A. Frameworks to Evaluate Subjective Fairness

Several frameworks aspire to comprehensively account for themajor factors that are determinative of an individual’s perceptions oflegal fairness. Gerald Leventhal developed one of the better knownmodels in a 1980 paper that chronicled six “fairness factors.”100 These

97 It should be noted that the core hypotheses of the characteristics of proceduraljustice remain fairly constant across different cultures. Tyler, supra note 85, at 306–07(noting how the efficacy of administrative or judicial institutions is partially linked topublic perceptions of fairness).

98 See LIND & TYLER, supra note 96, at 203–06 (listing a range of studies to show thatprocedural justice effects have been validated in field and laboratory studies). For morecontemporary confirmations of the core hypotheses, see Lorraine Mazerolle et al.,Procedural Justice and Police Legitimacy: A Systematic Review of the Research Evidence, 9J. EXPERIMENTAL CRIMINOLOGY 245, 246–48 (2013), discussing several decades ofscholarship, in the context of citizens’ perceptions of the legitimacy of police action,confirming fundamental social psychology of procedural justice hypotheses, and KristinaMurphy & Tom R. Tyler, Procedural Justice and Compliance Behaviour: The MediatingRole of Emotions, 38 EUR. J. SOC. PSYCHOL. 652, 655 (2008), exploring the relationshipbetween procedural justice, emotional reactions, and later compliance with orders of legalauthorities.

99 System designers may be in a better position to understand what is needed from afairness perspective. LIND & TYLER, supra note 96, at 220 (noting that experts, in somecircumstances and fields, prefer procedures leading to greater objective fairness, butalternative procedures will be subjectively favored by the general population, leading to achoice). The literature has also chronicled the notion of “false consciousness” regardingcertain norms of fairness that are inculcated by the ruling classes for their own benefit. Seeid. at 4 (“[W]e will refer frequently to the problem of ‘false consciousness’ of proceduraljustice, by which we mean that people believe a given procedure to be structured andenacted fairly when in fact, by objective standards, it is not.”).

100 See id. at 131–32 (discussing Leventhal’s six procedural justice rules for what makesa procedure fair); see also Gerald S. Leventhal, Jurgis Karuza, Jr. & William Rick Fry,Beyond Fairness: A Theory of Allocation Preferences, in JUSTICE & SOCIAL INTERACTION


include: consistency,101 bias suppression,102 accuracy of information,103

correctability,104 representativeness,105 and ethicality.106

For the reasons discussed in Part II, users would rank properlydesigned and built legal ADS high on “consistency” and “bias sup-pression,” given that their decisional algorithms are applied identicallyin every case and decisions are made solely on the parameters encap-sulated in the coding. Disputants’ sense of fairness resulting from“accuracy of information” and “correctability,” on the other hand,have inspired key criticisms of legal ADS and will be discussed indetail below. The final two parameters of Leventhal’s framework(“representativeness” and “ethicality”) go beyond the scope of thispaper because they touch upon the substantive content of particularlegal norms rather than the technological or human mechanisms usedto apply them.

A subsequent comprehensive framework articulated by TomTyler denoted four primary factors determinative of perceptions oflegal fairness: 1) the chance for presentation and consideration of adisputant’s voice;107 2) the neutrality of decisionmaking;108 3) the

167, 195–96 (Gerold Mikula ed., 1980) (discussing the criteria that individuals use toevaluate the fairness of a procedure).

101 LIND & TYLER, supra note 96, at 131 (“For a procedure to be fair, it must be appliedconsistently across persons and across time.”).

102 Id. (“[P]rocedures are unfair if the decision maker has a vested interest in anyspecific decision. . .. [P]rocedures are [also] unfair if. . . the decision maker is so influencedby his or her prior beliefs that all points of view do not receive adequate and equalconsideration.”).

103 Id. at 132 (“[P]rocedures are perceived to be unfair if they appear to be basingdecisions on inaccurate information.”).

104 Id. (“[T]he fairness of a procedure is enhanced to the extent that it contains someprovisions for correcting bad decisions.”).

105 Id. (specifying that procedural rules should account for the “basic concerns, values,and outlook of important subgroups in the population” (quoting Gerald S. Leventhal,What Should Be Done with Equity Theory? New Approaches to the Study of Fairness inSocial Relationships, in SOCIAL EXCHANGE: ADVANCES IN THEORY AND RESEARCH 27,43–44 (Kenneth J. Gergen et al. eds., 1980))).

106 Id. (“[P]rocedural justice depends on the extent to which an allocation procedureconforms to personal standards of ethics and morality.”).

107 See Tom R. Tyler, Social Justice: Outcome and Procedure, 35 INT’L J. PSYCHOL. 117,121–22 (2000) (“People feel more fairly treated if they are allowed to participate in theresolution of their problems or conflicts by presenting their suggestions about what shouldbe done.”).

108 See id. at 122 (“People are influenced by judgments about the honesty, impartiality,and objectivity of the authorities with whom they are dealing.”).


trustworthiness of the decision maker;109 and 4) the degree to whichthe treatment of disputants is respectful and dignified.110

As with Leventhal’s framework, legal ADS receive favorable, butnot perfect, marks on Tyler’s parameters. The second and third factorsreflect favorably on legal ADS, given that the lower risk of explicit orimplicit bias for properly designed and built systems likely make themneutral and trustworthy. Moreover, as discussed above, for extremelycomplex or confusing legal regimes, parties may actually obtain singu-larly trustworthy outcomes when machines help determine legal out-puts. The first and fourth parameters, on the other hand, appear moreproblematic, as automated decision systems have traditionally beenseen as restrictive of human voice and indifferent to respectful treat-ment of individuals. Despite these concerns, many recent advances insystem design increasingly respond to these perceived shortcomings,as will be discussed below.

Taken together these frameworks demonstrate that legal ADS inlaw can deliver high perceptions of subjective fairness, even if meetingsome of the fairness standards appears more challenging. Despite this,the rapidly evolving use of IT in our society suggests that few of theseshortcomings are completely intractable. Ongoing changes in systemdesign seek to address the shortcomings highlighted above, so as toimprove subjective fairness perceptions of legal ADS over time forthe reasons discussed below.

B. Voice and Process Control in Automated Decision Systems

As suggested by Tyler’s framework, one crucial driver of subjec-tive fairness is the ability of individuals to have their voices directlyheard in the legal process. This means being able to tell their side of astory, as well as perceive that the decision maker substantivelyengages with their explanations and arguments.111 This findingemerged from an extensive set of studies comparing subjective fair-ness in inquisitorial and adversarial systems, which concluded that dis-putants generally perceived the adversarial system as more fairbecause it offered greater control over process and the introduction of

109 See id. (“Another factor shaping people’s views about the fairness of a procedure istheir assessment of the motives of the third-party authority responsible for resolving thecase.”).

110 See id. (“People value having respect shown for their rights and for their statuswithin society.”); see also Hollander-Blumoff & Tyler, supra note 16, at 12–18 (exploringthe necessary steps for more flexible, less formal, alternative dispute resolutionmechanisms and rule of law values to exist simultaneously in society).

111 See LIND & TYLER, supra note 96, at 103–06 (concluding from a group of studies that“[t]he perception that one has had an opportunity to express oneself and to have one’sviews considered by someone in power plays a critical role in fairness judgments”.


new facts.112 Subsequent empirical studies have confirmed this generalhypothesis in diverse circumstances to suggest that influence over pro-cess and the opportunity to speak, together, help determine percep-tions of legal fairness.113 At first glance, this appears problematic forlegal ADS. The very paradigm that permits ADS’s increased effi-ciency is standardizing and automating repetitive legal processes,which would inherently seem to remove disputants’ control overprocess.

While automated legal decision systems (as well as human-basedones) must limit the range of facts and legal narratives considered inorder to focus a dispute,114 recent changes in how digital technology isused mean that digitally based legal decision systems are not necessa-rily more restrictive of participant expression than their manual coun-terparts and may, in the future, even enhance it. Today’s technology isfundamentally changing how humans tell stories by offering a diver-sity of new social media, still image, video, and location-based tools toenhance expression.115 This means that people make sense of their

112 See generally id. at 21–26 (describing a study where attorneys (played by lawstudents) in the adversarial system were perceived to perform a more in-depthinvestigation when compared with their counterparts in the inquisitorial system). It isworth noting that greater subjective perceptions of fairness do not necessarily equate withgreater objective fairness. Other research suggests that the adversarial system does notgenerally produce a more thorough investigation of the facts than an inquisitorial system,except in limited circumstances. See id. at 22–25.

113 See Tom R. Tyler & Steven L. Blader, The Group Engagement Model: ProceduralJustice, Social Identity, and Cooperative Behavior, 7 PERSONALITY & SOC. PSYCHOL. REV.349, 351 (2003) (noting studies where participants who had the opportunity to speak ratedthe process as more fair even if they knew what they had to say had “little or noinfluence”); see also Mengyan Dai, James Frank & Ivan Sun, Procedural Justice DuringPolice-Citizen Encounters: The Effects of Process-based Policing on Citizen Complianceand Demeanor, 39 J. CRIM. JUST. 159, 165–66 (2011) (discussing a study wherenoncompliance in police encounters diminished by 60% when officers took citizens’opinions into consideration).

114 See OSCAR G. CHASE, LAW, CULTURE, AND RITUAL: DISPUTING SYSTEMS IN CROSS-CULTURAL CONTEXT 40–41 (2005) (“Although the kinds of information accepted asevidence vary among legal regimes, all modern systems rely on a specially filtered categoryof documents and statements through which an agreed-upon version of the past isconstructed.”).

115 See Howard Rheingold, Using Participatory Media and Public Voice to EncourageCivic Engagement, in CIVIC LIFE ONLINE: LEARNING HOW DIGITAL MEDIA CAN ENGAGE

YOUTH 97, 97 (W. Lance Bennett ed., 2008) (“[I]ncreasing numbers of young people seekto master the use of media tools to express themselves, explore their identities, andconnect with peers—to be active creators as well as consumers of culture. . ..”); AboutStoryCenter, STORYCENTER, http://www.storycenter.org/about/ (last visited Sept. 16, 2016)(describing an organization that offers workshops and other support for digital storytellingwith the goal of encouraging social change). See generally DIGITAL STORYTELLING,MEDIATIZED STORIES: SELF-REPRESENTATIONS IN NEW MEDIA (Knut Lundby ed., 2008)(exploring how storytelling has developed in recent years in light of new social digitaltechnologies).


lives through new kinds of digitally-enabled expression that can morereadily be incorporated into legal ADS.116

One good example of this evolution in digital voice can be foundwith the Modria dispute resolution platform discussed above. Thesystem is designed to maximize the intake of digital expression in sucha way as to both improve decisional accuracy, as well as maximizeusers’ subjective perceptions of fairness. This is accomplished in sev-eral ways. When a user initially flags a dispute, the system providesample space for a disputant to articulate their perceived wrong—allowing them to upload text-based or even video files.117 However,before this record is shared as the basis for a potentially counter pro-ductive free-for-all exchange between the parties, and to avoid unnec-essary ill will, the recipient of the complaint is given a chance torespond to a short-form version of the complaint.118 Only if this doesnot resolve the dispute does the process continue for the Modriasoftware to identify the issues implicated in any given dispute.119 Theinputs used by the software include not only the textual submissions ofthe disputants, but also other kinds of data harvested from the under-lying commerce platform—such as eBay—regarding the parties’ pastbehavior or disputes, quantities and timing of transaction and pay-ments, and previous communication by the parties over the plat-form.120 The key takeaway here is that Modria seeks to leverage theadvantages of standardization and automation, but in a way that takesaccount of participants’ digital voice, both by incorporating thewritten (or visual) complaints they submit, as well as their previousdigital expressions made while using the platform.121 As the humanvoice and its stories become increasingly digital, ADS in law will beable to more fully incorporate participants’ digital expression withconcomitant potential for improving subjective fairness by givingusers greater process and voice control.

116 The amazing dominance of new forms of digital narrative in our daily lives has led tothe widely-held notion that events that are not reported on social media have either nothappened at all or, even if they have, lack social relevance. Jacob Silverman, “Pics or ItDidn’t Happen” — The Mantra of the Instagram Era: How Sharing Our Every Moment onSocial Media Became the New Living, GUARDIAN (Feb. 26, 2015, 12:59 AM), http://www.theguardian.com/news/2015/feb/26/pics-or-it-didnt-happen-mantra-instagram-era-facebook-twitter.

117 Lardinois, supra note 50.118 MODRIA, supra note 42 (noting that some outcomes will be automatically approved

per the inputted policy and some disputes will require the parties to communicate directly).119 Id.120 Id.121 See Raymond & Shackelford, supra note 4, at 491–92 (describing eBay and PayPal’s

success with an automated resolution system).


C. Addressing Perceived Errors with Limited Human Oversight

Even accepting that automated decision making systems in lawoffer the potential for greater accuracy and consistency over hugevolumes of disputes,122 as Leventhal’s framework predicts, one of theconcerns that crucially motivates the technological due process move-ment regards how to catch and correct erroneous automated deci-sions.123 Even an objectively accurate and reliable system may appearsubjectively very unfair if no mechanism exists to address patentlyerroneous outcomes. This risk is amplified with automated decisionsystems, where users may feel completely trapped by the system’smechanistic operation—like when an automatic 800-number callanswering system prevents you from speaking to a human representa-tive to address a problem not included in the automated menu.

Technological due process scholars writing about this concernemphasize the importance of inserting more human oversight, namelylawyers, into the digital processes.124 In truth, human oversight andparticipation can never be fully removed from these systems. As wesaw above, both the Modria and CalWIN systems rightly includemechanisms for limited human review of automated decisions ifflagged either by internal procedures or system users.125 While ahuman backstop must remain to some degree, to realize the efficiencyand accuracy gains offered by legal ADS, administrators must make aconstant effort to reduce the proportion of issues brought before ahuman decision maker. If not, every automated dispute will eventuallybecome a manual one, largely obviating the labor-saving and accuracyadvantages of implementing IT in law.

The difficult balancing act of providing sufficient human over-sight to assure users perceive a subjectively fair process, but not somuch as to destroy the benefits of automation, does not offer any easyanswers. With this in mind, several important themes emerge aspotential best practices. First, Modria carefully models and analyzesall disputes that users have elevated for human review to identifyrepeatable areas of disagreement for the purpose of making their

122 See supra Section III.A.123 See Citron, supra note 13, at 1256–58, 1280–81 (describing failed automated

decisions and proposing possible solutions); Crawford & Schultz, supra note 12, at 121(“Various due process scholars have also conceptualized the [technological due processmovement] as a form of systemic management technique that should focus less onindividual harm and more on discovering errors, identifying their causes, andimplementing corrective actions.”).

124 Citron, supra note 13, at 1305–08 (proposing numerous methods of protecting dueprocess through better training programs for human officials and more humaninvolvement).

125 Pich v. Lightbourne, 164 Cal. Rptr. 3d 388 (Ct. App. 2013).


software responsive to a broader set of issues over time.126 Second,recent growth in platforms like Yelp or Angie’s List illuminate thesubjective fairness benefits of allowing customers to autonomouslyenter their feedback, criticisms, or complaints—even if doing so doesnot guarantee formal redress for a particular problem.127 Along theselines, permitting users to rate or otherwise comment on their exper-iences with legal ADS on a curated website, for example, could cru-cially satisfy individuals’ need to be heard, while also making theirsubstantive feedback available to improve the system’s objective func-tion. Finally, administrators of legal ADS must have highly trainedhuman representatives who deeply understand the functioning of theautomated system in order to definitively and persuasively explain thesystem’s results to dissatisfied users. Users may quickly sense poorlytrained or uninformed human overseers and therefore presume theirincompetence reflects that of the system—whether true or not.Minute human oversight and second guessing of every decision madeby legal ADS cannot be the solution because such an approach wouldinherently destroy automation’s efficiency and consistency advan-tages. On the other hand, approaches being developed at several tech-nology companies, such as careful modeling to prevent duplicativehuman consideration of similar issues, automated mechanisms forpublic user comment that do not require human involvement, andtraining for empathetic but rigorous human overseers, can all helpachieve this balance.

D. Fairness in Promptness

Though not included in the above-discussed comprehensiveframeworks, another well-documented driver of subjective fairness ispromptness and lack of delay in legal processes. While the legal pro-fession has demonstrated ongoing concern for the pernicious effects ofundue delay,128 public perceptions suggest a sustainable solution hasnot been found to this concern.129 Social psychology research confirms

126 MODRIA, supra note 42.127 New Ways to Complain: Airing Your Gripes Online Can Get You Satisfaction — or

Trouble, CONSUMER REP. (Aug. 2011), http://www.consumerreports.org/cro/money/consumer-protection/new-ways-to-complain/overview/index.htm.

128 See C.H. van Rhee, The Law’s Delay: An Introduction, in THE LAW’S DELAY:ESSAYS ON UNDUE DELAY IN CIVIL LITIGATION 1–4 (C.H. van Rhee ed., 2004) (“Eventhough there have been many more reform attempts and measures have been taken duringthe last 800 years in both Civilian and Anglo-American jurisdictions to accelerate civillitigation, complaints [of delays] are still being voiced today.”).

129 INST. FOR THE ADVANCEMENT OF THE AM. LEGAL SYS., CIVIL CASE PROCESSING IN

THE FEDERAL DISTRICT COURTS 1 (2009) (“In a recent national survey of nearly 1500experienced litigation attorneys, 69% of respondents agreed that the civil justice system


that disputants perceive a system that resolves issues more rapidly tobe fairer,130 possibly because it is more respectful of individuals’ timeand less disruptive of their lives. This research finding highlights yetanother point of tension between the fairness enhancing potential ofautomating legal processes and the technological due process school’sadvocacy for substantially more manual oversight. Beyond simplyfaster processing, the scalability of automated decision systems meansthey can readily address peaks in activity without a need to substan-tially expand, redesign, or sacrifice accuracy by rushing adjudicationor skipping process steps. The streamlining and scaling advantages ofADS permit prompt and consistent application of legal rules to dis-putes, even during periods of unexpected spikes in demand. For alegal system in which delay frustrates many participants, embracinglegal ADS can serve to reduce delay and associated subjective costs.On the other hand, requiring substantially greater human oversight oflegal ADS, as proposed by critics, would undermine automation’stimeliness advantages.

E. In Sum

Technological due process scholars raise many hypothetical fair-ness concerns about legal ADS. What they fail to consider is howautomating law can drive overall systematic improvements in percep-tions of legal fairness. Part III seeks to elucidate this potential by con-sidering legal ADS’s functioning in light of empirical socialpsychology findings that explore the very question of what character-istics make a legal system appear subjectively fair. Legal ADS inher-

takes too long. . . , and 92% agreed that the longer a case goes on, the more it costs. Thesurvey results echo findings from previous studies stretching back to the 1950s.”). Congressand the bar have been increasingly preoccupied by the excessive delays (which impose realworld economic burdens) engendered by changes in the number of civil and criminallawsuits as well as relevant procedural factors. But see Patrick E. Higginbotham, ThePresent Plight of the United States District Courts, 60 DUKE L.J. 745, 762 (2010) (arguingthat ongoing procedural reforms have effectively managed cost and delay concerns despitecontrary views held by Congress and other members of the public).

130 LIND & TYLER, supra note 96, at 87–88 (noting one study where “speed ofresolution” was found to be one of the top criteria for a user-designed dispute procedure).One area where the impact of promptness of process upon perceptions of proceduraljustice has been studied most widely involves customer complaint processes. See Amy K.Smith, Ruth N. Bolton & Janet Wagner, A Model of Customer Satisfaction with ServiceEncounters Involving Failure and Recovery, 36 J. MARKETING RES. 356, 359 (1999) (“[T]hespeed with which problems and complaints are handled has been identified as an importantdimension of procedural justice.”); see also Jeffrey G. Blodgett, Donna J. Hill & Stephen S.Tax, The Effects of Distributive, Procedural, and Interactional Justice on PostcomplaintBehavior, 73 J. RETAILING 185, 189, 201 (1997) (discussing the background principle of theprocedural justice implications of timeliness, while finding promptness did notoverwhelmingly affect negative comments or repatronage decisions in this study).


ently promote neutrality and consistency at scale in a manner whollyimpossible for masses of discrete human legal operatives to achieve.Automated systems also allow for more prompt resolutions of dis-putes, even during spikes in demand, so as to be more respectful ofindividuals’ time. Even on those fairness parameters where legal ADSperform less well, technological changes are quickly bringing improve-ment. Legal ADS companies are devising new ways to engage withusers’ digital voices and correct errors with limited human involve-ment, so as to not destroy the benefits of automation. Taken together,rather than being a threat to legal fairness or legitimacy, reference tothe characteristics experimentally determined to undergird legal fair-ness suggests that properly designed and built legal ADS can be moreresponsive to these factors than many manual alternatives.

CONCLUSION

Despite objections to the use of legal ADS by technological dueprocess scholars, such systems offer the potential to capitalize on auto-mated legal processes that can be both objectively and subjectivelymore fair than manual alternatives. With respect to objective fairness,the crushing complexity of many modern regulatory regimes and theability of automated systems to apply the same decisional frameworkwith minute consistency across huge volumes of cases suggest legalADS can offer uniquely fair outcomes—a truth readily acknowledgedby some courts. In part due to these objective reasons, legal ADSappear to rank favorably on many—but not all—empirically mea-sured parameters associated with subjective perceptions of legal fair-ness.131 Technological advances will permit legal ADS creators torapidly improve on those fairness parameters where their systems per-form less well. This suggests that legal ADS offer a sustainable andscalable way to address simultaneously our nation’s outsized spendingon legal services, its growing thicket of complex laws and regulations,and its ongoing access to justice crisis. As such, legal ADS could serveas a valuable complement to the many talented lawyers, judges, andadministrators behind today’s legal system.

Reframing the analysis of legal ADS in light of its demonstrableobjective benefits, the empirically measured drivers of subjective fair-ness, and outsized legal spending in the United States indicates,despite scholarly criticisms to the contrary, that legal automation of a

131 It should be noted that the empirically measured drivers of subjective fairnessdiscussed above correspond loosely to the World Justice Project’s parameters formeasuring the effectiveness of justice systems. See WORLD JUSTICE PROJECT, supra note 5,at 8–18.


growing portion—but not all—of legal processes offers enormoussocietal benefits. The fact that some limited glitches have inevitabilityarisen in a handful of such systems is a poor reason to restrict theirwider use. While some rightly fear the “ominous social ramificationsof a surveillance society governed by heartless algorithmicmachines,”132 human beings choose how to build these systems andhow to apply our technological capacity. Most importantly, as theabove examples sought to illustrate, the easily scalable and highly mal-leable nature of information technology means that its thoughtfulapplication in law opens totally novel horizons for crafting hybridcomputer-human processes that hew more closely to our collectiveaspirations for how the law should function. The very digital systemsthat many fear as heartless and mechanical may in many instancesactually drive human values of justice that respect the time, financiallimitations, and subjective feelings of system participants better thantraditional manual legal practices.

132 Omer Tene & Jules Polonetsky, Judged by the Tin Man: Individual Rights in the Ageof Big Data, 11 J. TELECOM. & HIGH TECH. L. 351, 352 (2013).