toward a reasonable programmer standard responsibility and negligence in software design
TRANSCRIPT
Toward A Reasonable Toward A Reasonable Programmer StandardProgrammer Standard
Responsibility and Negligence in Software Design
Overview of Presentation It’s an ethical issue – ask questions; make
comments
Why do we need a definition?
New technologies = new legal issues
Software, though not new, is applied in new areas
What to think when software causes harm?
Why a Definition?Objection: you gain nothing by stating a
definition
Objection: I know in advance that your definition is wrong
Objection: Standard of negligence doesn’t apply to software
The Law and Negligence
“Failure to be sufficiently careful in a matter in which one has a moral responsibility to exercise care...” Online Ethics Center: The Online Ethics Glossary Center. (2003, March 21).
Classic example: leaving a rake on your walkway – kills the mailman
Failing to shovel snow; doctors who leave surgery tools in a patient; etc.
When is a person negligent?
The reasonable person standard: “A phrase used to denote a hypothetical person who exercises qualities of attention, knowledge; intelligence, and judgment that society requires of its members for the protection of their own interest and the interests of others.”National Association for Court Management: Glossary of Terms.
Invokes different requirements in different societal roles – the reasonable doctor is very different from the reasonable homeowner
What is a reasonable programmer?
Test Case 1: Freeware Software is distributed freely and
with a carefully worded license
What’s the worst that can happen? Computer crashes -> data loss
Holding a programmer responsible for the data is too demanding – not everything is negligent
Test Case 2: Free Algorithm
Programmer posts implementation of a sorting algorithm
Algorithm doesn’t work on negative numbers
Company uses algorithm in air traffic control software
Moral intuition: company’s fault, not the algorithm programmer; notion of direct responsibility
Test Case 2: Ethical Analysis
Company had contractual and moral obligation to test their software sufficiently
Programmer had no intention of using algorithm in critical environment – no obligation
Conclusion: expectations of performance derive from moral and contractual obligation
Test Case 3a: Virus JunkieA programmer Q gets high off of
creating viruses
Q unwittingly unleashes virus, costing U.S. companies millions
Should Q be held responsible?
Test Case 3b: Router A properly functioning antivirus
program at company C would have stopped Q’s virus
Is C in any way responsible for the harm caused by the virus?
Test Case 4: Therac-25 Atomic Energy of Canada Limited: faulty software
leads to overdose of radiation -> death
AECL probably was ‘sufficiently careful’ – post-accident review showed many hours of testing
Contrasted with bug that fails to save internet books mark: serious consequences important to definition, as is amount of testing proportional to potential harms
Test Case 5: Patriot Missile Programming flaw -> system operates over 20 hours =
failure -> military issues patch, deployment is slow
Barracks destroyed, software patch arrives the next day
Raytheon’s fault? Testing revealed the flaw
Army’s fault? Slow deployment of patch, vague memo on proper operation, project extended beyond intended lifetime
New definition: direct causation of flaw, or indirect causation of conditions that lead to flaw
Final DefinitionUnreasonable, direct failure to be
sufficiently careful in software design and testing, or causation of such conditions leading to serious harm, wherein a programmer has a moral obligation to do so from a contractual or otherwise reasonable expectation.
Test Case 6: More VirusesLots of subtle issues, what can we
reasonably expect from each party?: Obviously, punish those who create virus OS designer let flaw propagate? Owners and designers of networks that
allow propagation? User who downloads attachment with
subject “Haven’t heard from you in a while!”?