1

Round-off Errors

2

Key Concepts• Round-off / Chopping Errors

• Recognize how floating point arithmetic operations can introduce and amplify round-off errors

• What can be done to reduce the effect of round-off errors

3

There are discrete points on the number lines that can be represented by our computer.

How about the space between ?

4

Implication of FP representations

• Only limited range of quantities may be represented.– Overflow and underflow

• Only a finite number of quantities within the range may be represented.– round-off errors or chopping errors

5

Round-off / Chopping Errors(Error Bounds Analysis)

Let z be a real number we want to represent in a computer, andfl(z) be the representation of z in the computer.

What is the largest possible value of ?

i.e., in the worst case, how much data are we losing due to round-off or chopping errors?

z

zflz )(

6

Chopping Errors (Error Bounds Analysis)

e

n

ennn

aaazfl

aaaaaaz

21

12121

.0)(

0,.0

Suppose the mantissa can only support n digits.

ennn

enn

nenn

enn

n

aaaaa

aa

z

zflz

aaaazflz

).0(

)0...00.0()(

).0()0...00.0()(

2121

21

2121zeroes

Suppose ß = 10 (base 10), what are the values of ai such that the errors are the largest?

Thus the absolute and relative chopping errors are

7

Chopping Errors (Error Bounds Analysis)

nezflz )(

n

z

zflz 1)(

nene

nn

nnn

aazflz

aaa

21

321

.0)(

1.0Because

nenee

ne

e

ne

enn

n

ne

ennn

ne

ennn

enn

aa

aaaaa

aaaaa

aa

z

zflz

1)1(1

21digits

2121

2121

21

1.0

100000.0

.0

.0

0...00.0)(

8

Round-off Errors (Error Bounds Analysis)

exponentbase)sign(1

0,.0 1121

e

aaaaaz enn

1n21

1n21

a2

])01...00.0().0[(

2a0).0(

)(e

n

en

n

aaa

aaa

zfl

Round down

Round up

fl(z) is the rounded value of z

9

Round-off Errors (Error Bounds Analysis) Absolute error of fl(z)

nennn

nennn

ennn

aaazflz

aaa

aaazflz

321

321

321

.0)(

.0

000.0)(

When rounding down

2

1)(.

2 11

nn aa nezflz 2

1)(

Similarly, when rounding up

12 na nezflz 2

1)(i.e.,

when

10

Round-off Errors (Error Bounds Analysis) Relative error of fl(z)

1

1

2121

2

1

)1.0()(.because)1(.2

1

)(.because)(.2

1

2

1)(2

1)(

n

n

en

e

n

ne

a

aazaa

zz

zflz

zflz

n

z

zflz 1

2

1)(

11

Summary of Error Bounds Analysis

β base n # of significant digits or # of digits in the mantissa

Regardless of chopping or round-off is used to round the numbers, the absolute errors may increase as the numbers grow in magnitude but the relative errors are bounded by the same magnitude.

Chopping Errors Round-off errors

Absolute

Relative n

z

zflz 1

2

1)(n

z

zflz 1)(

nezflz )( nezflz 2

1)(

12

Machine Epsilon

off-Round1

2

1)(eps

z

zflz n

Relative chopping error Chopping

1)(eps

z

zflz n

Relative round-off error

eps is known as the machine epsilon – the smallest number such that

1 + eps > 1

epsilon = 1;while (1 + epsilon > 1) epsilon = epsilon / 2;epsilon = epsilon * 2;

Algorithm to compute machine epsilon

13

Propagation of Errors

• Each number or value of a variable is represented with error

• These errors (Ex and Ey) are carried over to the result of every arithmetic operation (+, -, x, )⌯

• How much error is propagated to the result of each arithmetic operation?

yAT

xAT

Eyy

Exx

14

Example #1

13 1045623.01012346.0 TT yx

Assume 4 decimal mantissa with rounding are used13 104562.0101235.0 AA yx

3

3

13

101281.0)(

10128062.0

104562.0101235.0

AA

AA

yxfl

yx

(Final value after round-off)

How many types of errors and how much errors are introduced to the final value?

15

Propagated Error: (xT + yT) - (xA + yA) = Ex+Ey

1

3

3

104000.0

101235.0

1012346.0

x

A

T

E

x

x

3

1

1

103000.0

104562.0

1045623.0

y

A

T

E

y

y

Propagated Error = -0.4000x10-1 + 0.3000x10-3

Example #1

16

3

313

101281.0)(

10128062.0104562.0101235.0

AA

AA

yxfl

yx

1

33

103800.0

10128062.0101281.0)()(

AAAA yxflyx

Rounding Error:

Example #1

17

Finally, the total error is

The total error is the sum of the propagated error and rounding error

error rounding103800.0

error propagated103000.0104000.0

107770.0

101281.0101280223.0

101281.0)1045623.01012346.0(

)()(

131

1

33

313

AATT yxflyx

Example #1

18

Propagation of Errors (In General)

• Let be the operation between xT and yT

can be any of +, -, x, ⌯• Let * be the corresponding operation

carried out by the computer– Note: xA yA ≠ xA * yA

yATxAT EyyExx

19

Propagation of Errors (In General)

Error between the true result and the computed result is

(xT yT ) – (xA * yA) =

(xT yT – xA yA) + (xA yA – xA * yA)

Errors in x and y propagated by the operation

Rounding error of the result

|xA yA – xA * yA| = fl(xA yA) ≤ |xA yA | x eps

20

Analysis of Propagated ErrorsAddition and Subtraction

Addition

yx

AAyAxAAATT

EE

yxEyExyxyx

)()()()(

yx

AAyAxAAATT

EE

yxEyExyxyx

)()()()(

Subtraction

yxyxyx EEEEEE or :Note

21

Propagated Errors – Multiplication

yx

TT

yx

T

y

T

x

TT

yxyTxT

TT

yTxTTT

TT

AATTyx

yx

EE

y

E

x

Eyx

EEExEyyx

EyExyxyx

yxyx

)()(

)()(

Very small and can be neglected

22

Propagated Errors – Division

yx

y

yx

y

x

T

A

T

A

T

T

A

A

T

T

A

A

T

T

yx

yyxx

yxyx

yx

yx

yx

1

1

11

11/

if εy is small and negligible

yT

A

T

ATy

y

y

y

yy

1

23

Example #2Effects of rounding errors in arithmetic manipulations

• Assuming 4-digit decimal mantissa• Round-off in simple multiplication or division

(EXACT)3699631113333

Results by the computer:

(Result)370000

(Rounding)103700.0

10369963.0

)101110.0()103333.0(

6

6

34

24

Danger of adding/subtracting a small number to/from a large number

Possible workarounds:1) Sort the numbers by magnitude (if they have the same signs) and add the numbers in increasing order

2) Reformulate the formula algebraically

8000

(Rounding)108000.0

1080003.0

1000003.0108000.0

3.08000

4

4

44

25

Associativity not necessarily hold for floating point addition (or multiplication)

The two answers are NOT the same!

440 101325.0,101325.0,108567.0 cba

Note: In this example, if we simply sort the numbers by magnitude and add the number in increasing order, we actually get worse answer!

Better approach is analyze the problem algebraically.

?)(

?)(

cba

cba1

0

101000.0)(

108567.0)(

cba

cba

26

Subtraction of two close numbers

2

2

2

100955.0

102686.0

103641.0

The result will be normalized into 0.9550 x 101

However, note that the zero added to the end of the mantissa is not significant.

Note: 0.9550 x 101 implies the error is about ± 0.00005 x 101 but the actual error could be as big as ± 0.00005 x 102

27

Subtractive Cancellation – Subtraction of two very close numbers

0001.00001.0

105763.0

105764.04

21

421

TT

T

T

yx

y

x

The error bound is just as large as the estimation of the result!

Subtraction of nearly equal numbers are major cause of errors!

Avoid subtractive cancellation whenever possible.

28

Avoiding Subtractive Cancellations

xxxf 1)(

Is there a way to reduce the errors assuming that we are using the same number of bits to represent numbers?

Example 1: When x is large, compute

xx

xx

xxxxxf

1

11

11)(

Answer: One possible solution is via rationalization

29

Subtraction of nearly equal numbers

acba

acbbx 4when

2

4 22

Solve x2 – 26x + 1 = 0

Example 2: Compute the roots of ax2 + bx + c = 0 using

168132

42626

168132

42626

2)2(

2)1(

T

T

x

x

30

961.12168

implies that one solution is more accurate than the other one.

Example 2 (continue)

Assume 5 decimal mantissa,

25 103.10385186.0

0005.0,109.1

961.25

0005.0)1()1(

xx

0385186.0,961.25

0005.0Since

039.0961.12000.13961.25

)2()1(

)2()1(

)2()1(

TT

xx

AA

xx

EE

xx

)1()2( xx

31

with

Alternatively, a better solution is

038519.0961.25

1

16813

116813

168131681316813)2(

Ax

)1(961.251)2( xx

i.e., instead of computing

we use

as the solution for the second root

a

acbbx

2

42

acbb

c

acbba

acx

4

2

)4(2

422

Example 2 (continue)

32

• Note: This formula does NOT give more accurate result in ALL cases.

• We have to be careful when writing numerical programs.

• A prior estimation of the answer, and the corresponding error, is needed first. If the error is large, we have to use alternative methods to compute the solution.

33

Assignment 1 (Problem 1)

Assume 3 decimal mantissa with rounding(a) Evaluate f(1000) directly.(b) Evaluate f(1000) as accurate as possible using

an alternative approach.(c) Find the relative error of f(1000) in part (a) and

(b).

1

11)( Given

xxxf

34

Propagation of Errors in a Series

Let the series be m

iixS

Is there any difference between adding

(((x1 + x2) +x3) +x4) +…+xm and

(((xm + xm-1) +xm-2) +xm-3) +…+x1

35

Example:

#include <stdio.h>int main() { float sumx, x; float sumy, y; double sumz, z; int i; sumx = 0.0; sumy = 0.0; sumz = 0.0; x = 1.0; y = 0.00001; z = 0.00001;

for (i = 0; i < 100000; i++) { sumx = sumx + x; sumy = sumy + y; sumz = sumz + z; }

printf("%sumx = %f\n", sumx); printf("%sumy = %f\n", sumy); printf("%sumz = %f\n", sumz);

return 0;}

Output:sumx = 100000.000000sumy = 1.000990sumz = 0.99999999999808375506

36

Exercise

Discuss to what extent

(a + b)c = ac + bc

is violated in machine arithmetic.

37

#include <stdio.h>#include <math.h>

int main() { float x = 10, sum = 1, term = 1, temp = 0; int i = 0; while (temp != sum) { i++; term = term * x / i; temp = sum; sum = sum + term; printf("%2d %-12f %-14f\n", i, term, sum); } printf("exact value = %f\n", exp((double)x)); return 0;}

Example: Evaluate ex as ...!

...!3!2

132

n

xxxxe

nx

38

Output (when x = 10)

1 10.000000 11.000000 2 50.000000 61.000000 3 166.666672 227.666672 4 416.666687 644.333374 5 833.333374 1477.666748 6 1388.889038 2866.555664 7 1984.127197 4850.682617 8 2480.158936 7330.841797 9 2755.732178 10086.574219 10 2755.732178 12842.306641 11 2505.211182 15347.517578 12 2087.676025 17435.193359 13 1605.904541 19041.097656 14 1147.074585 20188.171875 15 764.716431 20952.888672 16 477.947754 21430.835938

17 281.145752 21711.982422 18 156.192078 21868.173828 19 82.206360 21950.380859 20 41.103180 21991.484375 21 19.572943 22011.056641 22 8.896792 22019.953125 23 3.868171 22023.822266 24 1.611738 22025.433594 25 0.644695 22026.078125 26 0.247960 22026.326172 27 0.091837 22026.417969 28 0.032799 22026.451172 29 0.011310 22026.462891 30 0.003770 22026.466797 31 0.001216 22026.468750 32 0.000380 22026.468750 exact value = 22026.465795

term sum

39

#include <stdio.h>#include <math.h>

int main() { float x = 10, sum = 1, term = 1, temp = 0; int i = 0; while (temp != sum) { i++; term = term * x / i; temp = sum; sum = sum + term; printf("%2d %-12f %-14f\n", i, term, sum); } printf("exact value = %f\n", exp((double)x)); return 0;}

Example: Evaluate ex as ...!

...!3!2

132

n

xxxxe

nx

Arithmetic operations that introduce errors

40

Output (when x = -10)

1 -10.000000 -9.000000 2 50.000000 41.000000 3 -166.666672 -125.666672 4 416.666687 291.000000 5 -833.333374 -542.333374 6 1388.889038 846.555664 7 -1984.127197 -1137.571533 8 2480.158936 1342.587402 9 -2755.732178 -1413.144775 10 2755.732178 1342.587402 11 -2505.211182 -1162.623779 12 2087.676025 925.052246 13 -1605.904541 -680.852295 …

29 -0.011310 -0.002908 30 0.003770 0.000862 31 -0.001216 -0.000354 32 0.000380 0.000026 33 -0.000115 -0.000089 34 0.000034 -0.000055 35 -0.000010 -0.000065 36 0.000003 -0.000062 37 -0.000001 -0.000063 38 0.000000 -0.000063 39 -0.000000 -0.000063 40 0.000000 -0.000063 41 -0.000000 -0.000063 42 0.000000 -0.000063 43 -0.000000 -0.000063 44 0.000000 -0.000063 45 -0.000000 -0.000063 46 0.000000 -0.000063 exact value = 0.000045

term sum

Not just incorrect answer!We get negative value!

41

Errors vs. Number of Arithmetic Operations

Assume 3-digit mantissa with rounding

(a) Evaluate y = x3 – 3x2 + 4x + 0.21 for x = 2.73

(b) Evaluate y = [(x – 3)x + 4] x + 0.21 for x = 2.73

Compare and discuss the errors obtained in part (a) and (b).

42

Summary• Round-off/chopping errors

– Analysis

• Propagation of errors in arithmetic operations– Analysis and Calculation

• How to minimize propagation of errors– Avoid adding huge number to small number– Avoid subtracting numbers that are close– Minimize the number of arithmetic operations involved