Can We Trust Computers?

CS 301 (Spring 2007)Mark Luntzel, Niel Ngyuen, and James Cheng

Facts on Computer Errors

Error-free software is not possible F-22 Software Glitch Space Shuttle Software

Errors are often caused by more than one factor Lack of exhaustive, comprehensive testing

Errors can be reduced by following good procedures and professional practices. Denver baggage system

The Roles of People in Computer-related Problems

Computer User At home or work, users should understand the limitations of

computers and the need for proper training and responsible use (but often do not).

Computer Professional Understanding the source and consequences of computer

failures is valuable when buying, developing, or managing a complex system.

Educated Member of Society Personal decisions and political, social, and ethical decisions depend

on understanding computer risks.

Types of problems and failures

Problems for IndividualsSystem FailuresSafety-Critical Applications

What can go wrong?

Problems for IndividualsBilling Errors

Lack of tests for inconsistencies and inappropriate amounts

Database Accuracy ProblemsIncorrect information resulting in wrongful

treatment or acts

Problems for individuals

Causes:Large population.Human common sense not part of automated

processingOverconfidence in the accuracy of data from a

computerErrors in data entryInformation not updated or correctedLack of accountability for errors

System Failures

CommunicationsBusinessTransportation

Safety-Critical Applications

MilitaryPower PlantsAircraftTrainsAutomated FactoriesMedicine

Problem Causes:OverconfidenceLack of override

features.Insufficient testingSystem complexityMismanagement.

Therac-25

The Therac-25 was a software-controlled radiation-therapy machine used to treat people with cancer.Overdoses of radiation

Normal dosage is 100–200 rads.It is estimated that 13,000 and 25,000 rads were

given to six people.Three of the six people died.

- Intermission -(Next Up: Niel Ngyuen)

GENERAL REASONS FOR COMPUTER’S FAILURE

The task they are doing is inherently complex and difficult

The task is often done poorly

Complexity

Several computer systems are very large and composed of many interconnected subsystems.

Various software programs have thousands and millions of lines of codes.

Poor performance

Interaction with physical devices that do not work as expected

Incompatibility of software and hardware, or of application software and the OS

Management problems, including business and/or political pressure to get a product out quickly

Inadequate attention t potential safety risks Not planning and designing for unexpected inputs or

circumstances

Poor performance (continued)

Insufficient testingReuse of software from another system

without adequate checkingOverconfidence in softwareCarelessnessMisrepresentation; hiding problems;

inadequate response when problems are reported

Poor performance (continued)

Problems with management of the use of a system:Data-entry errorsInadequate training of usersErrors in interpreting results or outputOverconfidence in software by usersInsufficient planning for failures; no backup systems

or proceduresLack of market or legal incentive to do a better job

Key issues

OverconfidenceThe failure rate is often exaggerated

(Therac-25, the Challenger, etc)

Reuse of software without adequate testing

Professional Techniques

Software Engineering and Professional Responsibility

User Interfaces and human factorsRedundancy and self-checkingTestingTaking Responsibility

Law and Regulation

The Uniform Computer Information Transaction Act (UCITA) accepts agreements as binding contracts, letting software sellers continue to sell product with known bugs.

The FDA has regulated drugs and medical devices for decades

Professional licensing

Weakness of Law and Regulation

The approval process is extremely expensive and time-consuming.

Regulations requiring specific procedures or materials discourage or prevent the use of newer and better ones that were not thought of by people who wrote the rules.

The goal of regulation tends to get lost in details of the paperwork required.

The approval process is affected by political concerns, including influence by competitors and the incentive to be overcautious.

Failure perspective

Billing and Banking

What is the acceptable failure rate (99% or 99.9%)?

Complex systems

Should we retain some degree of human control or rely heavily on computer system to make decision in critical tasks?

Dependency on Computers

Computers offer convenience and productivity.

Tragic breakdowns in computer systems often remind us of how efficient such systems are when they are working.

Risk and Progress

Most new technologies are not safe when they are first introduced.

Risk factors must be carefully accessed.Progress must be made to correct past mistakesMore training must be offered for operating

complex system.Over years, engineers developed techniques

and procedures to increase safety. Software developers need to learn to apply their methods to software.

Solving Problem

Correctly identify the source of problems.Avoid blaming the technology and

computer system for many problems where they are irrelevant.

Make clear distinction between non-computer-related and computer-related problems so that we can improve on the latter ones.

- Intermission -(Next Up: James Cheng)

MODELS/SIMULATIONS

Criteria for evaluating modelsSuccess: Car Crash-Analysis ProgramsControversial: Climate Models/Global

Warming

What are Models?

Data/equations describing/simulating systems

Physical and non-Physical systems*Limitation on modeling: Simplification of

reality*Modeling Validity/Accuracy*

Examples of Computer Modeling

Physical and non-Physical systemsCar crash-analysis*Climate change*Population growthFiscal policies’ effects on economyAnd more…

Limitation on Modeling

Simplification of realityComputation resource limitedNot all “rules” are know/usedDifficult to numerically quantify

everything

Model Validity/Accuracy

How closely does model mimic underlying science?

Which simplification chosen? Data completeness?

How closely does simulation predict reality?

Car Crash-Analysis Programs (Part 1)

Example: NYNA3D (Lawrence Livermore National Laboratory)Divide car into grid. (10,000 – 50,000

pieces.)Each element has material property.Approx. 35hr super computer time for 40-

100 ms simulation (c.1990)

Car Crash-Analysis Programs (Part 2)

Crash-Analysis programs’ efficacy$50,000 - $800,000: Real crash test.Results closely correspond to actual crash

testsSuccess lead to other impact

modeling/simulation*

Car Crash-Analysis Programs (Part 3)

Success with crash analysis led to other simulation work:Dropping hazardous waste containers.Airplane nacelle/windshield collision with

birds.Airbag deployment predictionForecast earthquakes' effects on structures

Climate Models/Global Warming

Background info:Global temperature started rising in 1970sSharp increase in CO2 and Methane since

1950sCO2 and Methane increasing since 16,000

years agoIPCC 2001: +0.74°C in the last century

Climate’s Coupled Models

General Circulation Models + Oceanic ModelsGCM: Developed originally weather predictionUses as input:

Sun’s energy output, earth orbit, topography, sea/ice surfaces, more.

Predicts: Atmospheric temperature, solar/radiant energy I/O, precipitation, etc.

Climate Model Accuracy(Part 1)

The science? Not completely known.e.g. Cloud formation not fully understood.

Simplification used? Pretty extreme.e.g. ~200km spaced grid points.

Accuracy? Mixed results.More on this on next slide…

Climate Models’ Accuracy(Part 2)

Accurate: consensus on continued increase of temperature and sea level

Semi-accurate: +0.33°C mean surface temperature changeUpper bound of IPCC 2001

Climate Models’ Accuracy(Part 3)

Inaccurate: Predicted troposphere warming did not occur.

Inaccurate: Science magazine reports 3.3m per year since 1993. (50% higher than IPCC 2001.)

Climate Models’ Accuracy(Part 4)

Inaccurate: Greenland Ice Sheet melt rate.IPCC 2001: -44 ± 53 Gt/yr2006 Estimate (from U.S. satellites): -239 km3/year239 km3 = 239 Gt ice, approximately.

Side Note:Melting of entire ice shelf is predicted to means

6.5m increase in sea level (21 ft.)

IPCC 2001: General Predictions

Human activities likely the primary cause of warming.Note: IPPC 2007 Summary upgraded to the

language to “Very Likely”, which is 90%+ confidence level.

Continued increase in temperature.

- Intermission -(Next Up: Questions for Discussion)

Questions for Discussion

Q1: Electronic voting machines? Good? Bad? Run quickly?

Q2: How have computer errors affected you? Q3: Given all the inaccuracy and uncertainty with

climate modeling, should they be the basis for policy decisions? Or, should they be just another set of considerations, like any special interest?