Deriving Algorithms

Ran Libeskind-HadasDepartment of Computer Science

Harvey Mudd College

Assumptions…

• Course emphasis is on preparing students to…– Design their own algorithms– Analyze and prove the correctness of those

algorithms• As opposed to what?– Emphasis on learning how a number of “classical”

algorithms work– Practice implementing existing algorithms

Assumptions…

• Course emphasis is on preparing students to…– Design their own algorithms– Analyze and prove the correctness of those

algorithms• As opposed to what?– Emphasis on learning how a number of “classical”

algorithms work– Practice implementing existing algorithms

if emphasis on design then

• Use a paradigmatic approach– Divide and conquer– Dynamic programming– Greed (and it’s dangers)– Amortized analyses– Integration of algorithms and data structures

But even in this approach…

• It’s easy to show representative algorithms rather than derive them

Why Derive?

• It’s intellectually satisfying• Provides a deeper sense of how algorithms

are designed• Helps students remember algorithms• Provides students with confidence and skills to

design their own algorithms

A Derivation-First Approach

• A paradigmatic approach to algorithms is not necessarily one that emphasizes derivations

• We are using a derivation-based approach to teaching the undergraduate algorithms course at Harvey Mudd

• Currently about 20% of the material is being taught this way; we hope to do more in the future!

The Setting…

• Sophomore/Junior level algorithms course• 75-100 students• Two 75 minute lectures per week– Repeat about four cycles of:• 15 minutes of lecture• 5 minutes of pair work

Examples

• Deriving sorting algorithms• Deriving Dijkstra’s shortest path algorithm• Deriving Kruskal’s MST algorithm• Paper describes other examples• Disclaimer: Many of these ideas are known to computer

scientists and some appear in books or scholarly articles. But algorithms courses don’t appear to be taught in this way.

Sorting

• Observation: Selection sort and mergesort are “natural” but heapsort seems like magic

• Approach: Derive heapsort from selection sort and mergesort

• Assumptions:– Students have seen selection sort and mergesort

before (e.g., CS 1)– Students have seen heaps before (e.g., CS 2)– They might even have seen heapsort, but if so it

usually hasn’t stuck!

Recall selection sort and mergesort…

42 23 5 16 47 3 8 12

Mergesort and it’s analysis

42 23 5 16 47 3 8 12

T(n) = 2 T(n/2) + nT(1) = c

5 16 23 42 3 8 12 47

merge

Q(n log n)

If dividing in two is good…

Let’s try dividing into k pieces!

What’s the recurrence relation now?

If dividing in two is good…

Let’s try dividing into k pieces!

What’s the recurrence relation now?

T(n) = k T(n/k) + (k-1)nT(n) = c

Now we have k “fingers” at the front of k lists!

Now solve this!

If dividing in two is good…

T(n) = k T(n/k) + (k-1)nT(n) = c

T(n) = (k-1) n logk n

Now we have k “fingers” at the front of k lists!

• What do you expect it to be when k = 2?• Will it ever be asymptototically better

than n log n?• What do you expect it to be when k > 2?• What happens when k = n?

When k = n

T(n) = (k-1) n logk n = = (n-1) n which is Q(n2) Have you seen this algorithm before?

Now we have n “fingers” at the front of n lists!

…

When k = n

• This is Selection sort!• But it seems a bit crazy to do all of this work to

repeatedly find the minimum!

Now we have n “fingers” at the front of n lists!

…

When k = n

…

• Insert all elements into a heap• Repeatedly remove elements

from heap

When k = n

…

• Insert all elements into a heap• Repeatedly remove elements

from heap• Set up and solve recurrence

O(n log n)

Take-Away

• Mergesort generalizes naturally from 2-way to k-way splitting

• When k = n, mergesort “degenerates” to selection sort

• But, merging more cleverly results in heapsort• Important connections established• Heapsort is derived rather than pulled from a

hat!

Dijkstra

• Observation: Dijkstra’s algorithm is often presented and proved correct, but little intuition is offered and it’s hard to remember how it works.

• Approach: Motivate Dijkstra from BFS• Assumptions: Students have seen…– BFS (and the “wave” metaphor)– Heaps

Dijkstra1000

3

5

6

7

10

9

12

Dijkstra1000

3

5

6

7

10

9

12

Dijkstra1000

3

5

6

7

10

9

12

1000

3

Dijkstra1000

3

5

6

7

10

9

12

1000

3

1000

3

8

What’s required to keep track of the wavefront?

• Deleting the item with minimum value• Updating values when they get better “offers”

Keeping track of the wave!

1000

3

10008

0

0

3

3 8

Take-Away

• BFS is “natural” but Dijkstra is much less obvious

• But Dijkstra can be derived as an efficient implementation of BFS for weighted graphs (assuming integer weights)

• Now it makes sense to prove that Dijkstra also works for arbitrary non-negative real-valued edge weights

Deriving Kruskal’s MST Algorithm

• Observation: Minimum spanning tree algorithms are typically presented and then proved correct.

• Approach: Derive intuitively reasonable MST algorithms and then show that they are correct

• Assumptions: Students have seen some elementary graph theory and it is reviewed briefly here (e.g., trees and their properties, graph cuts).

Minimum Spanning Trees• Start with motivation of the problem• Begin by assuming all edge weights are distinct• Give one result: The cheapest edge crossing a

cut must be in every MST

5

1

How would you go about choosing edges for a MST

5

1

How would you go about choosing edges for a MST

5

1

How would you go about choosing edges for a MST

5

1

What next?

How would you go about choosing edges for a MST

5

1

How would you go about choosing edges for a MST

5

1

What next?

How would you go about choosing edges for a MST

5

1

How would you go about choosing edges for a MST

5

1

Finally…

• What is the algorithm in general?• How do you prove that it’s correct?

Other derived algorithms

• Other shortest path algorithms• Other minimum spanning tree algorithms• Network flow algorithms

Outcomes

• “Standard” approach taught in 2011• “Derivation-first” approach taught in 2012– Only 5 of the 30 lectures changed

• Student course evals on 1-7 Likert Scale

2011 2012“Course promoted learning objectives” 6.52

6.80“Stimulated interest” 6.41

6.75“Learned a great deal” 6.59

6.82

Anecdotally

• A derivation-based approach appears to help students…– Feel intellectually engaged with the material– Feel a strong sense that “I could have done that!”– Remember how algorithms work and be able to

explain them to others– Derive their own algorithms!

Future work

• We are collecting data more methodically this year to better understand the impact of the derivation-based approach

• Developing a website where derivation “modules” can be contributed and borrowed

Comments and Questions