CS503: Seventh Lecture, Fall 2008Advanced Sorting

Michael Barnathan

Here’s what we’ll be learning:• Theory:

– Shellsort.– Mergesort.– Quicksort.– Big Omega and Theta.– Analyzing recursive algorithms: the master method.– Comparison of basic and advanced sorting algorithms.

• Lab:– Working on Assignment 2 if time.

Basic Sorting Algorithms• Bubble Sort– Loop through the array twice, “bubble” elements up

the array one-by-one by swapping until each reaches where it belongs.

– Advantages:• Simplicity.• Stable.• Requires O(1) extra space.• It has a cool name.

– Disadvantages:• Asymptotic Performance: O(n^2).• Real-world performance: ~ Twice as slow as insertion sort.

Basic Sorting Algorithms

• Insertion Sort– Build a sorted “buffer” within the array. Gradually

expand the buffer by swapping elements into their appropriate places within it.

– Advantages:• Stable.• Requires O(1) extra space.• Usually the fastest basic sorting algorithm.

– Disadvantages:• Asymptotic Performance: O(n^2).• More complex than bubble and selection sort.

Basic Sorting Algorithms

• Selection Sort– Keep a buffer. Search for the minimum element

outside of the buffer and swap it with the element at the new end of the buffer (cur. end + 1).

– Advantages:• Requires O(1) extra space.• Simple to implement.

– Disadvantages:• Asymptotic Performance: O(n^2).• Not as good as insertion sort in practice.

Performance

• The O(n^2) complexity is a recurring theme.• O(n^2) isn’t that good.• We’d like to do better.– Can we?

• Obviously, or we wouldn’t be discussing other algorithms.

– But how?

• Any ideas?– It’s not really intuitive anymore.– We’re trading performance against simplicity.

Lower Bounds and Big Omega• It turns out that any sort that compares adjacent items has a lower bound of Ω(n2).• Huh? Your “O” looks a bit funny.• Big O is the most used, but there are other symbols too…

– O(n) represents an asymptotic upper bound of n.– Ω(n) represents an asymptotic lower bound of n.– Θ(n) represents both – a “tight bound”.

• Most of the time when we said “O(n)”, we actually meant Θ(n).– The bounds I’ve given you in class were tight, but I didn’t want to confuse you.

• So when we say adjacent comparison sorts are Ω(n2), we mean that they can be O(n3), O(n4), O(2n), … - but not O(n log n) or better.– This formalizes the notion of “never better than this”.

• Likewise, that means an algorithm believed to be O(n2) could technically be Θ(n).– However, a tighter O(n) bound would have to be proven for this.– Big O formalizes the notion of “never worse than this”.– We use this one the most because computer scientists are pessimists.

• If you know for sure that your algorithm will perform in time proportional to n2 and NOT proportional to n or n3, it’s best to say Θ(n2).

• If you’re not sure whether it runs in time proportional to n or n2, you can just leave it at O(n2) – someone might later improve the tightness of your upper bound.

Wait

• Don’t confuse this with best/average case performance.

• You’re still analyzing the worst case.• Using O(n) or Ω(n) reflects the idea of doing

(no worse or no better) in the worst case.• Everything still goes wrong… you just have a

wider range of performance.

Comparison

Shell Sort• Insertion sort is a good algorithm, but slow.• Why is it slow?– Elements are only swapped one position at a time.

• We can jump a bit further and perhaps gain some performance.– But we just said we can’t do better if we only compare

adjacent elements.– Ok, so don’t compare adjacent ones then.

• Shell sort is a generalization of insertion sort.– It was developed by Donald Shell in 1959.– It introduces the notion of a “gap”.

• Insertion sort is Shellsort with a gap of 1.

Why we can improve.

• If insertion sort just needs to move every element one space over, it becomes O(n).– It just shifts every element over one space on the

first swap and it’s done.– Each successive iteration would just compare the

element after the buffer to the first one inside. It wouldn’t need to move, so it would exit.

• In general, insertion sort tends to perform well when an array is almost sorted.

The Idea

• Instead of sorting one element at a time, sort over a wide range first, then narrow it down.

Gap = 5

Gap = 2

Gap = 1

Maybe it’s simpler with numbers.• [ 1 5 3 7 9 12 8 4 17 ]• Turn into a matrix with gap columns:gap = 31 5 37 9 128 4 17• Sort each column:1 4 37 5 128 9 17• Turn back into a vector: [1 4 3 7 5 12 8 9 17 ]• Repeat with smaller gaps until we reach 1.gap = 21 43 75 98 1217• [1 4 3 7 5 9 8 12 17 ]• Gap = 1 corresponds to an insertion sort, which finishes.

Shell Games• Why does it work?– Each step prepares the algorithm for the next.– The algorithm doesn’t redo work.

• If something is 5-sorted and we sort it with a gap of 3, it will then be both 5 and 3 sorted.

• Most effective if we choose a gap sequence in which each pair is relatively prime.– That is, they share no factors greater than 1.– This is so they don’t overlap; e.g., 2 and 6 would both sort 18.

• The final insertion sort (gap = 1) should be very close to linear.– But you must always end with gap = 1.

How to choose the gap sequence.

• It’s a bit of an art.• It affects the performance of the algorithm.• Usually, you use a mathematical function:– E.g. gap = 3*gap + 1

• The initial value is the largest term in this function that is less than the size of the array.

• You then reduce the gap as you sort by inverting the function (gap = (gap – 1) / 3).

The Algorithm.//A functor is an object containing a function.//In this case, GapFunctor is an interface providing nextGap() and invGap() functions.public void shellsort(Object[] arr, GapFunctor gfunc) {

int gap = 1;

//Precondition: invGap is an inverse and makes the gap smaller. Checking these is a good habit to learn.if (gfunc.invGap(gfunc.nextGap(gap)) != gap || gfunc.nextGap(gap) <= gap)

throw new IllegalArgumentException(“invGap must be an inverse and must make the gap smaller”);

while (gap < arr.length) //Compute the starting gap.gap = gfunc.nextGap(gap);

while (gap > 0) { //Since invGap is an inverse, the last gap will be 1.gap = gfunc.invGap(gap); //Make the gap smaller according to our function.//This is going through the array backwards, but is otherwise identical to our illustration.for (int endpos = gap; endpos < arr.length; endpos++) { //This is our “sorted buffer”.

Object temp = arr[endpos];int startpos;for (startpos = endpos; startpos >= gap && arr[startpos – gap] > temp; startpos -= gap)

arr[startpos] = arr[startpos - gap]; //Shift.arr[startpos] = temp; //New item in the “buffer”.

}}

}

Performance

• Shellsort is very difficult to analyze.– If you can manage to do a general theoretical

analysis of it, you have material for a publication.

• Here are some specific results, however:

Gap Function Proposed By Worst-case performance

Gap = Gap / 2 D. Shell O(n2)

Gap = (Gap + 1) * 2 – 1 Hibbard O(n3/2)

Gap = Gap / 2.2 Sedgewick? O(n7/6)

Gap.add(Gap[i]*2)Gap.add(Gap[i]*3)return Gap[i++];

Pratt O(n log2 n)

Optimal polynomial-time sort.

• Shellsort requires constant space.• It was proven that the Shellsort cannot reach

O(n log n).– But it can come pretty close.

• It is not a stable sort.• For small arrays, this may be the optimal sort.• On larger ones, Quicksort and Mergesort

outperform it.

Mergesort

• Fast as Shellsort is, it’s still polynomial.– We can do better than that.

• Now we come to the “really good” algorithms.• Mergesort.– Developed in 1945 by John von Neumann.– “Divide and conquer” algorithm.• Yes, it uses recursion.

– Java uses it to sort Collection classes.• Like Vector and LinkedList.

Mergesort: The Idea.• Similar to the idea we came up with for bubble

sort, but with a twist.• In bubble sort, we defined an array of size n as an

array of size n - 1 plus an element.• Now we define an array as 2 arrays of size n/2.• The idea behind Mergesort is to split the array in

half, sort the smaller arrays, and merge them together in a way that keeps them sorted.– Being recursive, the halves are going to get split into

quarters, and those into eighths…– Ok, so what’s the base condition? When do we stop?

Mergesort: Recursive Structure.• A 1-element array is already in sorted order.• So we use this as our base case.• The recursive step is pretty simple:– Split the list at the middle.– Merge sort the left half.– Merge sort the right half.

• The left and right halves are now sorted.• But we need to merge them together to keep them

sorted.– Merging two sorted lists isn’t too hard.– Let’s write it on the board.

Illustration

• Wikipedia has a nice example:

Merge:Object[] merge(Object[] sarr1, Object[] sarr2) {

//We’ll need space for all elements from both.Object[] ret = new Object[sarr1.length + sarr2.length]();

int arr1idx = 0;int arr2idx = 0;

while (arr1idx + arr2idx < ret.length) {//Insert the smaller element from either array.//Since both are sorted, advancing one-by-one will keep them that way.if (arr2idx >= sarr2.length || sarr1[arr1idx] < sarr2[arr2idx])

ret[arr1idx + arr2idx] = sarr1[arr1idx++];else

ret[arr1idx + arr2idx] = sarr2[arr2idx++];}

return ret;}

Mergesort Algorithm.

• Merging on two parts of one array is similar.– So you can do it in-place too.– Then you’d need to pass “leftstart”, “rightstart”, and

“rightend” indices and work on the array in-place.– You may wish to implement this variation on your

own. I’ll leave it up to you.

• We split just like we did in binary search.– Left: Start to middle.– Right: Middle + 1 to end.

Mergesort Algorithm://Overloaded for user convenience.public void mergesort(Object[] arr) {

mergesort(arr, 0, arr.length-1);}

private void mergesort(Object[] arr, int startidx, int endidx) {if (endidx <= startidx) //Base case: size 1.

return; //Already sorted.

//Some people complain that finding the middle like this //can cause an overflow if the array’s > 1 billion elements.//They’re right (if pedantic), but this is just for demonstration.int mid = (startidx + endidx) / 2;

//Reduction: split down the middle.mergesort(arr, startidx, mid); //Left half.mergesort(arr, mid + 1, endidx); //Right half.

arr = merge(arr, startidx, mid + 1, endidx); //In-place merge, but you can copy & call the other too.

}

Mergesort Performance

• This is a great algorithm.– It’s stable if you use the out-of-place merge.• The in-place merge is not stable.

– It works on sequential data structures.– It’s very fast.

• How fast? We’ll see in a moment.

The Master Method

• We need a general way to analyze recursive algorithms.

• Recursion can be modeled with recurrence relations.– These are functions defined in terms of

themselves. Like recursive functions, they have base cases and reductions.

• We’ve seen one already:– F(n) = F(n-1) + F(n-2).

Recursion -> Recurrence• You can define a recurrence on the number of elements being split

into.• There are two terms: the recursive term and the constant factor.

– The recursive term represents the call with a reduced number of elements.

– The constant term represents the work in each call.• For example, printTo(n) called printTo(n-1).

– So we would model it with T(n) = T(n-1) + O(1).– The constant term is O(1) because printing n is O(1).

• This was all the work that we did inside of the function.• If we split into two lists of size n-1 instead of one, we’d have T(n) =

2T(n-1) + O(1).• If we did something linear inside of printTo, we’d have T(n-1) + O(n).• If we split into two halves, as in mergesort, we’d have 2T(n/2)…• And so on.

Mergesort Recurrence

• We split into two subproblems of size n/2.– “Of size n/2” -> T(n/2).– “Two subproblems” -> 2*T(…).– “Two problems of size n/2” -> 2T(n/2).

• What about the other term?– Mergesort itself is O(1), but it calls merge, which is

O(n) – it looks through both arrays in sequence.• So our recurrence is T(n) = T(n/2) + O(n).• We can now use an important mathematical tool

to figure out the performance directly from this…

The Master Theorem

• If given a recurrence of the form:

T(n) = a*T(n/b) + f(n)(where f(n) represents the work done in each call).

• Then the running time of the algorithm modeled by this recursion will be Θ(nlogba)…

• *With an exception:– If f(n) is of the same polynomial order as logba,

then the running time will be Θ(nlogba * log n).

Examples:• T(n) = T(n/2) + O(1): (Binary search)– A = 1, B = 2, log2 1 = 0 O(n0 * log n).

• T(n) = 3T(n/5) + O(n):– A = 3, B = 5, log5 3 = 0.68 O(n0.68).

• T(n) = 8T(n/2) + O(1):– A = 8, B = 2, log2 8 = 3 O(n3).

• T(n) = 8T(n/2) + O(n3):– A = 8, B = 2, log2 8 = 3 O(n3 log n).

• T(n) = 2T(n/2) + O(n): (Mergesort).– A = 2, B = 2, log2 2 = 1 ?

Mergesort Performance

Θ(n log n)!• This blows the O(n2) sorts out of the water.• So mergesort:

– Is stable with O(n) extra space.– Works on sequential structures.– Has great worst (and average)-case performance.– Works really well in practice.– Is a great algorithm to use almost anywhere, as long as the O(n)

space isn’t a deterrent.• Despite this, it is not usually considered the fastest general

purpose sort.– There is one that’s faster…

Can we do better than O(n log n)?

• It has been proven (using decision trees) that O(n log n) is the optimal complexity for any sort that compares elements.– You might want to seek out the proof; it’s pretty

intuitive and not too difficult to comprehend.– But I won’t be covering it in class.

• So no, we can basically stop here.

Quicksort

• This is considered the fastest general purpose sort in the average case.

• It is also a recursive “divide and conquer” sort.• Where mergesort simply split the array,

Quicksort partitions the array.– The difference is that a partition actually splits an

unsorted array into 2 sorted arrays on each side.– Mergesort started with sorted arrays and built up.– Quicksort starts unsorted and splits down.

• Base case: array of size 1.

History and Usage• This is probably the most popular sorting

algorithm.– There are tons of variations.– Some are stable, some aren’t, some require extra

space, some don’t, some parallelize well…• It is not the simplest, however.• Java uses it to sort arrays (but not collections).• It was developed by C.A.R. Hoare in 1962.• There is an issue that prevents most people from

using the algorithm in its naïve form…– We’ll get to it.

Partitioning

• Choose a pivot element from the array– Ideal is the median, but that slows the algorithm.– Most just use the first or last element.

• Remove the pivot from the array.• Split the rest of the array into two:– Elements less than or equal to the pivot.– Elements greater than the pivot.

• Concatenate the left array, the pivot, and the right array.

Partitioning algorithm.int partition(Object[] arr, int left, int right, int pivotidx) {

//Temporarily move the pivot out of the way.swap(arr, pivotidx, right);

//Build the left list.int leftend = left;for (int curidx = left; curidx < right; curidx++)

if (arr[curidx] <= arr[pivotidx])//Swap into left list.swap(arr, curidx, leftend++);

swap(arr, leftend, right); //Move the pivot to the left end + 1.

//The rest of the data is naturally on the right side now.return leftend; //Return the new position of the pivot.

}

Quicksort• Once we partition, Quicksort is simple:

void quicksort(Object[] arr, int left, int right) {if (right <= left) //Base case: size 1.

return;

int oldpivot = right; //Or any element.int pivot = partition(arr, left, right, oldpivot);

//The array is now partitioned around the pivot.quicksort(arr, left, pivot - 1); //<= pivot.quicksort(arr, pivot + 1, right); //> pivot.

}

Quicksort Performance• On average, half of the elements will be less than the

partition, half greater.• So we are usually splitting into two subproblems, each

consisting of half the array.• Partitioning is linear.• This results in T(n) = 2T(n/2) + O(n).– Same as mergesort, same solution: O(n log n).

• In practice, Quicksort is even faster than mergesort.– The loop in partition is easy for systems to optimize.

• It also doesn’t require any extra space.• It isn’t stable, but neither is mergesort.

Quicksort: the Achilles’ Heel.• Ah, but we’re not dealing with averages. We are

interested in the worst case.– Everything ends up on one side of the pivot.

• Well, that paints a bleaker picture:– Now we have a recurrence of T(n) = T(n-1) + O(n).– This is O(n^2).

• The worst case occurs when the list is already sorted (if we choose a rightmost pivot).

• Variations on Quicksort can get around this.– “Introsort” is a popular one.

Comparison• Bubble Sort, Insertion Sort, Selection Sort

– O(n^2).– Bubble and insertion stable, selection not.– Little extra storage.– Primary advantage: simplicity.

• Shell Sort:– Between O(n log n) and O(n^2).– Highly variable with gap selection.– Generalized insertion sort.– Not stable.– Primary advantage: fairly simple, fairly good performance.

• Mergesort, Quicksort:– Optimal: O(n log n).– Mergesort either stable with O(n) memory usage or unstable in-place.– Quicksort generally unstable (stable variants exist).– Quicksort can be done in place or using extra memory.– Quicksort usually faster than Mergesort in practice.

• But both algorithms are very good.– Primary disadvantage: Difficult to implement.

Putting things in order.

• These are difficult topics. It’s ok to ask questions.• Now that we learned that O(n log n) was optimal,

we’ll learn about faster sorts.– It’s the optimal comparison sort, you see.

• The lesson:– Leverage existing work. All of the “good” algorithms

are built into the language already.• Next class: Binary trees.• Assignment due Tuesday, 9/30.– I’ll grant an extension to Thursday if you need one.