CS305/503, Spring 2009Amortized Analysis

Michael Barnathan

Here’s what we’ll be learning:• Theory:

– Amortized Analysis• Java:

– TreeSet– HashSet

Traditional Asymptotic Analysis• Looks at the behavior of one operation.

– One insertion takes O(n) time…– One search takes O(log n) time…– One, one, one.

• If every operation takes the same amount of time, this is perfectly fine. We can figure out the cost of the sequence.– What is the total complexity of n operations, each taking time

proportional to O(n)?• However, this is not always the case.

– What about Vectors, which increase in size when they are filled?– Each insertion at the end takes O(1) time, until the array is full,

upon which the next insertion takes O(n) time.

Vector Doubling in Traditional Analysis

• Suppose we perform n insertions on a vector that employs the doubling strategy.

• In traditional analysis, every operation has the same cost. So what is the worst case cost of insertion into an array?– O(n), because in the worst case, we double.– This is despite the fact that most insertions take O(1) time, because

the majority do not double.• We perform n insertions, each taking O(n) time.• What is the bound upon the complexity?

– n * O(n) = O(n2).• This is clearly not a tight bound.

Amortization• Amortized analysis analyzes worst-case performance over a

sequence of operations.• It is not an average case analysis; it is an “average of worst

cases”.• Going back to Vector insertion:

– If we perform 6 insertions into a Vector of size 5, 5 of those insertions will take 1 unit of time. The sixth will take 6 units. (Both in the worst case)

– Since all 6 insertions will take 11 units of time, one insertion contributes roughly 2 time units, not on average, but in the worst case.

– 2 is a constant. We would expect constant time behavior on insert.– An individual insertion may take longer (the sixth insertion takes 6

units of time, for example), but it will make up for it by preparing subsequent insertions to run quickly (by doubling the array).

Methods of Amortization• There are three commonly employed amortized analysis

techniques.• From least to most formal:

– The Aggregate Method:• Count the number of time units across a sequence of operations and

divide by the number of operations.– The Accounting Method:

• Each operation “deposits” time, which is then used to “pay for” expensive operations.

– The Potential Method:• A “potential function” φ is defined based on the change in state brought

about by each operation and the difference in potential is added to the total cost (this difference may be negative).

• We won’t go into too much detail on this method.• Each method has its limitations.

The Aggregate Method• This is the simplest method of analysis.• Simply add the worst-case cost of each operation in a

sequence up, then divide by the total number of operations in the sequence:

• The cost of each operation is very often defined asymptotically, not as a number.

• But that’s OK; O(n) means “a linear function of n”.• So O(n) / n = O(1), O(n2) / n = O(n), and so forth.

n

iCostostAmortizedC

n

i 1)(

The Aggregate Method – Example:

Data 1 2 3Cost O(1) O(1) O(1)

So far, the amortized cost is [n * O(1)] / n = O(1).

Data 1 2 3 4 5 6 7Cost O(1) O(1) O(1) O(n) O(1) O(1) O(1)

The fourth insertion doubles the array, an O(n) operation.

Now the amortized cost is [(n-1) * O(1) + O(n)] / n = O(1) + O(1) = O(1).

Caveats

• The lack of formalism in the aggregate method has some consequences.

• Specifically, when using the aggregate method, be careful with your asymptotics!

• O(n) at the 4th insertion is very different from O(n) at the 32768th insertion!

• It is thus sometimes useful to define the elementary cost of inserting without doubling as simply “1” and to use exact numbers.

Again, with numbers.

Data 1 2 3Cost 1 1 1

So far, the amortized cost is [3 * 1] / 3 = 1.

Data 1 2 3 4 5 6 7Cost 1 1 1 4 1 1 1

The fourth insertion doubles the array, an O(n) operation.

Now the amortized cost is [6 * 1 + 4] / 7 = 10 / 7 = 1.43.

How it Converges• It turns out that this is always constant-time, no matter how

many insertions you do:• For a sequence of n operations, the algorithm will double at n,

n/2, n/4, n/8, …• The number of elements in the array at each double is the cost

of that doubling step (because it’s O(n)).• So the cost is defined by a convergent series:

• So, at worst, it will take you thrice as long to use a doubling array as a preallocated one.

• 3 * O(1) = O(1); this is still constant-time.

3...8

1

4

1

2

111

...842

n

nnnnn

CostCost of insertion

Cost of doubling

The Accounting Method• The accounting method begins by assigning each elementary

operation a cost of $1.– The cost your analysis returns, of course, is then in terms of how long

those operations take.• Each operation will then pay for:

– The actual cost of the operation, and– The future cost of keeping that element maintained (for example,

copying it in the array).• We save in advance so we have something to “spend” when

we double.• We call the saved money “the bank”.

– The bank balance never goes negative; there are no subprime loans.• This is sort of difficult because it requires us looking ahead to

see what happens when we double.

What’s the cost?• Each element we insert costs $1 immediately.• When doubling:

– We will have to move each element to a new array; each move costs $1.

– We will have to create a new element for each existing element (because we’re doubling the size). Something is eventually going to fill this spot as well. This will cost $1.

• So the total cost is $3 per insertion.– $1 for now, $2 for the future.

• This is the same answer we received using the aggregate method.

• But requires careful inspection to arrive at.

Does it work?

• Remember, the bank must never go negative.• Doubling costs $n+1: $n to copy the n

elements, $1 for the insertion that immediately follows.

• Each insertion pays $3 and costs $1, so $2 goes into the bank at each non-doubling step.

• And each doubling costs $n+1.

Yes, it does.

i 1 2 3 4 5 6

size 1 2 4 4 8 8

Deposit $3 $3 $3 $3 $3 $3

Cost $1 $2 $3 $1 $5 $1

Profit $2 $1 $0 $2 -$2 $2

Bank $2 $3 $3 $5 $3 $5

Red fields represent insertions that cause the array to double.

Potential Method

• Instead of “saving” and paying the cost later, the potential method measures the “potential difference” between two adjacent operations.

• This is defined by a potential function φ.– Φ(0) = 0.– Φ(i) ≥ 0 for all i.

• The amortized cost of operation i is determined by the actual cost plus the difference in potential:– aci = ci + [φ(i) - φ(i-1)]

• The total cost is the sum of these individual costs:

n

i i

n

i i nciic11

)]0()([)]1()([ “Telescoping”: all phi terms except n and 0 cancel; they’re both added and subtracted.

Potential Method Example• For the array doubling problem,• ci = 1 if i is not a power of 2, i otherwise. (Direct cost of inserting item i)• How did we get this? The first term counts the cost we’ve accumulated for

creating each of the first i elements. The second term subtracts the costs we’ve already paid in previous doublings (it represents the current array capacity).

• If i-1 is a power of 2,– ci = i (we’re going to double)– φ(i) = 2i - 2(i -1)= 2– φ(i-1) = 2(i-1) - (i-1) = i - 1– aci = ci + φ(i) - φ(i-1) = i + 2 - (i - 1) = i + 2 – i + 1 = 1 + 2 = 3.

• If i-1 is not a power of 2,– ceil(lg(i)) = ceil(lg(i-1)), so the potential terms cancel.– ci = 1 (no doubling)– φ(i) = 2i– φ(i-1) = 2(i-1) = 2i - 2– aci = ci + φ(i) - φ(i-1) = 1 + 2i - (2i - 2) = 1 + 2i - 2i + 2 = 1 + 2 = 3.

• So we get the same answer as in the other methods.

iii lg22)(

Careful

• The potential method appears the most rigorous of the three, but you still need to deduce the potential function from the problem.

• The other two methods are less prone to mistakes.

Sets

• A set is a data structure that can be used to store records in which the key is the same as the value, keeping all elements in the set unique.

• There are two types in Java:– TreeSets: Sorted Unique Containers.– HashSets: Unsorted Unique Containers.

• A multiset, or bag, is like a set, but without the uniqueness constraint.

Special Properties• Elements in a set class are guaranteed unique.• Attempting to insert an element that already exists will not modify the set

at all and will cause add() to return false.• Sets can be split and merged.

– You can get the entire set of elements greater than or less than a target, for example.

– Or you can merge two disjoint sets together.• This is called a union operation.

• TreeSets are implemented using Binary Search Trees in Java, providing O(log n) insertion, access, and update and guaranteeing sorted order (remember to implement Comparable in your classes).

• HashSets are implemented using hash tables, providing average-case O(1) insertion, access, and deletion, but not guaranteeing sorted order.

• They are thus appropriate data structures to use for operations such as picking out the unique words in a book and outputting them in sorted order.

Methods• Has some of the usual ones: add(), remove(), size(), addAll()…• But also some exotic ones that return elements or subsets greater than or

less than an element:– higher(Object): Returns the first element > Object.– lower(Object): Returns the first element < Object.– floor(Object): Returns the first element ≤ Object.– ceiling(Object): Returns the first element ≥ Object.– headSet(Object): Returns the whole set < Object.– tailSet(Object): Returns the whole set > Object.

• Iterating over sets:– Use java’s “for each loop”:

• Set<Type> dataset;• for (Type t : dataset)

– System.out.println(t); //Or whatever you want to do with it.

• Maps give you two sets: keySet() and values(). You can iterate over these too: Map<Key,Val> datamap; for(Key k : datamap.keySet())

Performance on a Sequence

• We covered amortized analysis and sets today.

• Next time, we will discuss graphs – the root data structure from which most others derive.

• The lesson:– Plan for the future. Plan your current actions to

make your future efforts easier.