COS 323: Computing for theCOS 323: Computing for thePhysical and Social SciencesPhysical and Social Sciences

Szymon RusinkiewiczSzymon Rusinkiewicz

What’s This Course About?What’s This Course About?

• Numerical AlgorithmsNumerical Algorithms

• Analysis of DataAnalysis of Data

• SimulationSimulation

Numerical AnalysisNumerical Analysis

• Algorithms for solving numerical problemsAlgorithms for solving numerical problems– Calculus, algebra, data analysis, etc.Calculus, algebra, data analysis, etc.

– Applications in all scientific and engineering Applications in all scientific and engineering fieldsfields

• Analyze/design algorithms based on:Analyze/design algorithms based on:– Running time, memory usageRunning time, memory usage

(both asymptotic and constant factors)(both asymptotic and constant factors)

– Applicability, stability, and accuracyApplicability, stability, and accuracyfor different classes of problemsfor different classes of problems

Why Is This Hard/Interesting?Why Is This Hard/Interesting?

• ““Numbers” in computers Numbers” in computers numbers in numbers in mathmath– Limited precision and rangeLimited precision and range

• Algorithms sometimes don’t give right Algorithms sometimes don’t give right answeranswer– Iterative, randomized, approximateIterative, randomized, approximate

– UnstableUnstable

• Running time / accuracy / stability Running time / accuracy / stability tradeoffstradeoffs

Numbers in ComputersNumbers in Computers

• ““Integers”Integers”– Implemented in hardware: fastImplemented in hardware: fast

– Mostly sane, except for limited rangeMostly sane, except for limited range

• Floating pointFloating point– Implemented in most hardwareImplemented in most hardware

– Much larger rangeMuch larger range(e.g. (e.g. 223131.. 2.. 23131 for integers, vs. for integers, vs. 22127127.. 2.. 2127127 for FP) for FP)

– Lower precision (e.g. 9 digits vs. 7)Lower precision (e.g. 9 digits vs. 7)

– ““Relative” precision: actual accuracy depends Relative” precision: actual accuracy depends on sizeon size

Floating Point NumbersFloating Point Numbers

• Like scientific notation: e.g., Like scientific notation: e.g., cc is is2.997924582.99792458 101088 m/s m/s

• This has the formThis has the form(multiplier)(multiplier) (base)(base)(power)(power)

• In the computer,In the computer,– MultiplierMultiplier is called mantissa is called mantissa

– BaseBase is almost always 2 is almost always 2

– PowerPower is called exponent is called exponent

Modern Floating Point FormatsModern Floating Point Formats

• Almost all computers use IEEE 754 standardAlmost all computers use IEEE 754 standard

• ““Single precision”:Single precision”:– 24-bit mantissa, base = 2, 8-bit exponent, 1 bit sign24-bit mantissa, base = 2, 8-bit exponent, 1 bit sign

– All fits into 32 bits (!)All fits into 32 bits (!)

• ““Double precision”:Double precision”:– 53-bit mantissa, base = 2, 11-bit exponent, 1 bit sign53-bit mantissa, base = 2, 11-bit exponent, 1 bit sign

– All fits into 64 bitsAll fits into 64 bits

• Sometimes also have “extended formats”Sometimes also have “extended formats”

Other Number RepresentationsOther Number Representations

• Fixed pointFixed point– AbsoluteAbsolute accuracy doesn’t vary with magnitude accuracy doesn’t vary with magnitude

– Represent fractions to a fixed precisionRepresent fractions to a fixed precision

– Not supported directly in hardware, but can hack Not supported directly in hardware, but can hack itit

• ““Infinite precision”Infinite precision”– Integers or rationals allocated dynamicallyIntegers or rationals allocated dynamically

– Can grow up to available memoryCan grow up to available memory

– No direct support in hardware, but libraries No direct support in hardware, but libraries availableavailable

Consequences of Floating PointConsequences of Floating Point

• ““Machine epsilon”: smallest positive number Machine epsilon”: smallest positive number you can add to 1.0 and get something other you can add to 1.0 and get something other than 1.0than 1.0

• For single precision: For single precision: 10 1077

– No such number as 1.000000001No such number as 1.000000001

– Rule of thumb: “almost 7 digits of precision”Rule of thumb: “almost 7 digits of precision”

• For double: For double: 2 2 10 101616

– Rule of thumb: “not quite 16 digits of precision”Rule of thumb: “not quite 16 digits of precision”

• These are all These are all relativerelative numbers numbers

So What?So What?

• Simple example: add Simple example: add 11//1010 to itself 10 to itself 10

timestimes

Oops!Oops!

• Result: Result: 11//1010 + + 11//1010 + … + … 1 1

• Reason: 0.1 can’t be represented Reason: 0.1 can’t be represented exactly in binary floating pointexactly in binary floating point– Like Like 11//33 in decimal in decimal

• Rule of thumbRule of thumb: comparing floating point : comparing floating point numbers for equality is always wrongnumbers for equality is always wrong

More Subtle ProblemMore Subtle Problem

• Using quadratic formulaUsing quadratic formula

to solve to solve xx22 – 9999 – 9999xx + 1 = 0 + 1 = 0– Only 4 digits: single precision should be OK, right?Only 4 digits: single precision should be OK, right?

• Correct answers: 0.0001… and 9998.999…Correct answers: 0.0001… and 9998.999…

• Actual answers in single precision: 0 and 9999Actual answers in single precision: 0 and 9999– First answer is 100% off!First answer is 100% off!

– Total cancellation in numerator because Total cancellation in numerator because bb22 >> 4 >> 4acac

a

acbbx

2

42

a

acbbx

2

42

Catalog of ErrorsCatalog of Errors

• Roundoff error – caused by limitations ofRoundoff error – caused by limitations offloating-point “numbers”floating-point “numbers”

• Truncation error – caused by stopping an Truncation error – caused by stopping an approximate technique earlyapproximate technique early– e.g., too few terms of Taylor series for sin(e.g., too few terms of Taylor series for sin( ))

• Inherent error – limitation on data Inherent error – limitation on data availableavailable– GIGOGIGO

• Statistical error – too few random samplesStatistical error – too few random samples

Running TimeRunning Time

• Depending on algorithm, we’ll look at:Depending on algorithm, we’ll look at:– Convergence orderConvergence order for iterative approximate for iterative approximate

algorithmsalgorithms(e.g., an answer to precision (e.g., an answer to precision might require might require iterations proportional to 1/ iterations proportional to 1/ or 1/ or 1/ ))

– Asymptotic analysisAsymptotic analysis for noniterative for noniterative algorithmsalgorithms(e.g., inverting an (e.g., inverting an nnnn matrix requires time matrix requires time proportional to proportional to nn33))

This CourseThis Course

• Basic techniques:Basic techniques: root finding, root finding, optimization, linear systemsoptimization, linear systems

• Data analysis and modeling:Data analysis and modeling: least least squares, dimensionality reduction, squares, dimensionality reduction, visualization, statisticsvisualization, statistics

• Signal processing:Signal processing: sampling, filtering sampling, filtering

• IntegrationIntegration and differential equations and differential equations

• SimulationSimulation

MechanicsMechanics

• Programming assignmentsProgramming assignments– Typically more thought/analysis than codingTypically more thought/analysis than coding

– Some in MATLAB, some in Java or CSome in MATLAB, some in Java or C

MechanicsMechanics

• Precept time scheduled, but no regular Precept time scheduled, but no regular preceptsprecepts– Occasionally will have extra material on Occasionally will have extra material on

assignmentsassignments

– Otherwise, will hold office hoursOtherwise, will hold office hours

• Course webpageCourse webpage– (infinitely long address – find from CS dept (infinitely long address – find from CS dept

webpage)webpage)

• Contact me at Contact me at smr@princeton.edusmr@princeton.edu