data structures and algorithms analysis of algorithms richard newman
TRANSCRIPT
Data Structures and AlgorithmsAnalysis of Algorithms
Richard Newman
Players Boss/Manager/Customer
– Wants a cheap solution– Cheap = efficient
Programmer/developer– Wants to solve the problem, deliver system
Theoretician– Wants to understand
Student– Might play any or all of these roles some
day
Why Analyze Algorithms?
• Predict performance
• Compare algorithms
• Provide guarantees
• Understand theory
• Practical reason: avoid poor performance!
• Also – avoid logical/design errors
Algorithmic Success Stories• DFT
Discrete Fourier Transform Take N samples of waveform Decompose into periodic
components
• Used in DVD, JPEG, MPEG, MRI, astrophysics, ....
• Brute force: N2 steps
• FFT algorithm: N lg N steps
Algorithmic Success Stories• B-Body Simulation
Simulate gravitational interactions among N bodies
• Brute force: N2 steps
• Barnes-Hut algorithm: N lg N steps
The Challenge
Will my algorithm be able to solve problem with large practical input?
– Time– Memory– Power
Knuth (1970's) – use scientific method to understand performance
Scientific Method
Observe feature of natural world Hypothesize a model consistent with observations Predict events using hypothesis Test predictions experimentally Iterate until hypothesis and observations agree
Scientific Method Principles
Experiments must be reproducible Hypotheses must be falsifiable
Example: 3-Sum Given N distinct integers, how many
triples sum up to exactly zero
% cat 8ints.txt830 -40 -20 -10 40 0 10 5
% ./ThreeSum 8ints.txt4
3-Sum Brute Force Algo
For i=0 to N-1
For j=i+1 to N-1
For k=j+1 to N-1
If a[i] + a[j] + a[k] == 0
count++
return count
Measuring Running TimeManually Start stopwatch when starting program Stop it when program finishes Can do this in script (date)
Internally Use C library function time() Can insert calls around code of interest
– Avoid initialization, etc.
Measuring Running TimeStrategy Run program on various input sizes Measure time for each Can do this in script also Plot results
tools: http://www.opensourcetesting.org/performance.php
Measuring Running Time
What do you think the time will be for input of size 16,000?Why?
N Time (s.)
250 0.0
500 0.0
1000 0.1
2000 0.8
4000 6.4
8000 51.1
16000 ?
Data AnalysisStandard Plot Plot running time T(N) vs. input size N Use linear scales for both
Data AnalysisLog-log Plot If straight line Slope gives power
lg y = m lg x + b y = 2b xm
Hypothesis, Prediction, Validation
Hypothesis: running time 10-10 N3 Prediction: T(16,000) = 409.6 sObservation: T(16,000) = 410.8
N Time (s.)
250 0.0
500 0.0
1000 0.1
2000 0.8
4000 6.4
8000 51.1
16000 ?
Doubling HypothesisQuick way to estimate slope m in log-log plotStrategy: Double size of input each run Run program on doubled input sizes Measure time for each Take ratio of times If polynomial, should converge to power
Doubling Hypothesis
Hypothesis: running time 10-10 N3 Prediction: T(16,000) = 409.6 sObservation: T(16,000) = 410.8
N time ratio lg ratio
500 0.0 - -
1000 0.1 6.9 2.8
2000 0.8 7.7 2.9
4000 6.4 8.0 3.0
8000 51.1 8.0 3.0
16000 410.8 8.0 3.0
Doubling Hypothesis
Hypothesis: running time is about aNb
With b = lg(ratio of running times)
Caveat!!! Cannot identify logarithmic factors
How to find a? Take large input, equate time to hypothesized time with b as estimated, then solve for a
Experimental AlgorithmicsSystem Independent Effects Algorithm Input data
Experimental AlgorithmicsSystem Dependent Effects Hardware: CPU, memory, cache, ... Software: compiler, interpreter, garbage collection, ... System: OS, network, other processes
Experimental AlgorithmicsBad news Hard to get precise measurements
Good news Easier than other physical sciences! Can run huge number of experiments
Mathematical Running Time Models
Total running time = sum (cost x freq) Need to analyze program to determine set of operations over which weighted sum is computed Cost depends on machine, compiler Frequency depends on algorithm, input data
Donald Knuth1974 Turing Award
How to Estimate Constants?
*Running OS X on Macbook Pro 2.2 GHz 2 GB RAM
Operation example Time* (ns)
Integer add a + b 2.1
Integer multiply a * b 2.4
Integer divide a / b 5.4
Fp add a + b 4.6
Fp multiply a * b 4.2
Fp divide a / b 13.5
sine Math.sine(theta) 91.3
arctangent Math.atan2(x,y) 129.0
... ... ...
Experimental AlgorithmicsObservation: most primitive functions take constant time Warning: non-primitive often do not!
How many instructions as f(input size)?
int count = 0;for (int i = 1; i < N; ++i) if (a[i] == 0) count++;
Experimental Algorithmicsint count = 0;for (int i = 1; i < N; ++i) if (a[i] == 0) count++;
Operation Frequency
Var declaration 2
assignment 2
< compare N+1
== compare N
array access N
increment N to 2N
Counting Frequency - Loopsint count = 0;for (int i = 1; i < N; ++i) for (int j = i+1; j < N, ++j) if (a[i] + a[j] == 0) count++;
How many additions in loop?N-1 + N-2 + ... + 3 + 2 + 1 = (1/2) N (N-1)Exact number of other operations?Tedious and difficult....
Experimental AlgorithmicsObservation: tedious at bestStill may have noise!Approach: Simplify! Use some basic operation as proxy e.g., array accesses
int count = 0;for (int i = 1; i < N; ++i) for (int j = i+1; j < N; ++j) if (a[i] + a[j] == 0) count++;
Experimental AlgorithmicsObservation: lower order terms become less important as input size increasesStill may be important for “small” inputsApproach: Simplify! Use ~ Ignore lower order terms
– N large, they are negligible– N small, who cares?
Leading Term ApproximationExamples
Ex 1: 1/6 N3 + 20 N + 16 ~ 1/6 N3
Ex 2: 1/6 N3 + 100 N4/3 + 56 ~ 1/6 N3
Ex 3: 1/6 N3 – 1/2 N2 + 1/3 N ~ 1/6 N3
Discard lower order termse.g., N=1000, 166.67 million vs. 166.17 million
Leading Term ApproximationTechnical definition:
f(N) ~ g(N) means limit = 1 f(N)N -> inf g(N)
Bottom Lineint count = 0;for (int i = 1; i < N; ++i) for (int j = i+1; j < N, ++j) if (a[i] + a[j] == 0) count++;
How many array accesses in loop?~ N2
Use cost model and ~ notation!
Example - 3-Sumint count = 0;for (int i = 1; i < N; ++i) for (int j = i+1; j < N; ++j) for (int k = j+1; k < N; ++k) if (a[i] + a[j] + a[k] == 0) count++;
How many array accesses in loop?Execute N (N-1)(N-2)/3! Times ~ (1/6)N3
~ (1/2) N3 array accesses (3 per stmt)Use cost model and ~ notation!
Estimating Discrete Sums
Take Discrete Math (remember?)Telescope series, inductive proof
Approximate with integralDoesn't always work!
Use Maple or Wolfram Alpha
Takeaway
In principle, accurate mathematical models
In practiceFormulas can be complicatedAdvanced math might be neededAre subject to noise anywayExact models – leave to experts!
We will use approximate models
Order-of-Growth ClassesOrder
ofGrowth
Name Typical code desdription example T(2N)
T(N)
1 constant a=b+c Statement Add two numbers
1
log N logarithmic while(N>1) N=N/2
Divide in half
Binary search
~1
N linear for(i=0 to N-1) {...}
loop Find the maximum
2
N log N linearithmic See sorting Divide and conquer
mergesort ~2
N2 quadratic for(i=0 to N-1) for(j=0 to N-1) { ... }
Double loop
Check all pairs
4
N3 cubic for(i=0 to N-1) for(j=0 to N-1) for(k=0 to N-1) { ... }
Triple loop Check all triples
8
2N exponential See combinatorial search
Exhaustive search
Check all subsets
T(N)
Order-of-Growth
Definition: If f(N) ~ c g(N) for some constant c > 0, then f(N) is O(g(N))
– Ignores leading coefficient– Ignores lower order terms
Brassard notation: O(g(N)) is the set of all functions with the same order So 3-Sum algorithm is order N3
– Leading coefficient depends on hardware, compiler, etc.
Order-of-Growth
Good News! The following set of functions suffices to describe order of growth of most algorithms:
1, log N, N, N log N, N2, N3, 2N, N!
Order-of-Growth
Binary Search
Goal: Given a sorted array and a key, find the index of the key in the array Binary Search: Compare key against middle entry (of what is left)
– Too small, go left– Too big, go right– Equal, found
Binary Search Implementation
Trivial to implement? First binary search published in 1946 First bug-free version in 1962 Bug in Java's Arrays.binarySearch() discovered in 2006!http://googleresearch.blogspot.com/2006/06/extra-extra-read-all-about-it-nearly.html
Binary Search – Math Analysis
Proposition: BS uses at most 1+lg N key compares for a sorted array of size N Defn: T(N) = # key compares on sorted array of size <= N Recurrence: for N > 1, T(N) <= T(N/2) + 1 for N = 1, T(1) = 1
Binary Search – Math Analysis
Recurrence: for N > 1, T(N) <= T(N/2) + 1 for N = 1, T(1) = 1 Pf Sketch: (Assume N a power of 2)T(N) <= T(N/2) + 1
<= T(N/4) + 1 + 1<= T(N/8) + 1 + 1 + 1 ...<= T(N/N) + 1 + 1 + 1 + ... + 1= 1 + lg N
3-Sum
Version 0: N3 time, N space Version 1: N2 log N time, N space Version 2: N2 time, N space
3-Sum – N2 log N Algorithm
Algorithm– Sort the N (distinct) integers– For each pair of numbers a[i] and a[j], – Binary Search for -(a[i] + a[j])
Analysis: Order of growth is N2 log N– Step 1: N2 using insertion sort– Step 2: N2 log N with binary search
Can achieve N2 by modifying BS step
Comparing Programs Hypothesis: Version 1 is significantly faster in practice than Version 0
N Time (s)
1000 0.1
2000 0.8
4000 6.4
8000 51.1
N Time (s)
1000 0.14
2000 0.18
4000 0.34
8000 0.96
16000 3.67
32000 14.88
64000 59.16
Version 0 Version 1
Theory works well in practice!
Memory
Bit: 0 or 1 (binary digit) Byte: 8 bits (wasn't always that way) Megabyte (MB): 1 million or 220 bytes Gigabyte (GB): 1 billion or 230 bytes
NIST and networks guys Everybody else
Memory
64-bit machine: assume 8-byte pointers• Can address more memory• Pointers use more space• Some JVMs “compress” ordinary
object pointers to 4 bytes to avoid this cost
Typical Memory Usage
Type Bytes
boolean 1
byte 1
char 2
int 4
float 4
long 8
double 8
Type Bytes
char[ ] 2N + 24
int[ ] 4N + 24
double[ ] 8N + 24
Type Bytes
char[ ][ ] ~2MN
int[ ][ ] ~4MN
double[ ][ ] ~8MNPrimitive types
1-D arrays
2-D arrays
Typical Java Memory UsageObject Overhead: 16 bytesObject Reference: 8 bytesPadding: Objects use multiple of 8 bytes
Ex: Date objectpublic class Date { private int day; private int month; private int year; ...}
ObjectOverhead
day
month
year
padding
4 bytes (int)4 bytes (int)
4 bytes (int)
4 bytes (pad)
16 bytes (OH)
32 bytes total
Summary
Empirical Analysis:Execute pgm to perform experimentsAssume power law, formulate hypothesis
for running timeModel allows us to make predictions
Summary
Mathematical Analysis:Analyze algo to count freq of operationsUse tilde notation to simplify analysisModel allows us to explain behavior
Summary
Scientific MethodMathematical model is independent of
particular system, applies to machinesnot yet built
Empirical approach needed to validate theory, and to make predictions
Next – Lecture 5
Read Chapter 3 Basic data structures