greedy algorithms dr. yingwu zhu. greedy technique constructs a solution to an optimization problem...

Greedy Algorithms

Dr. Yingwu Zhu

Greedy TechniqueConstructs a solution to an optimization problem piece by piece through a sequence of choices that are:

• Feasible: satisfying the prob. constraints

• locally optimal: the best local choice

• Irrevocable: cannot be changed on subsequent steps once made

For some problems, yields an optimal solution for every instance.For most, does not but can be useful for fast approximations.

DP vs. Greedy Algorithms

• Both to solve problems exhibiting optimal substructure

• DP– Solution to the problem assembled from the

solutions to subproblems by considering various choices

• Greedy algroithms– Greedy-choice property: the locally optimal choice

is made w/o considering results from subproblems

• However, DP could be overkill sometimes

Applications of the Greedy Strategy

• Optimal solutions:– change making for “normal” coin denominations– minimum spanning tree (MST)– single-source shortest paths (Dijkstra’s algorithm)– simple scheduling problems– Huffman codes

• Approximations:– traveling salesman problem (TSP)– knapsack problem– other combinatorial optimization problems

An Activity-Selection Problem

• Suppose A set of activities S={a1, a2,…, an}– They use resources, such as lecture hall, one lecture at

a time– Each ai, has a start time si, and finish time fi, with 0 si<

fi<.– ai and aj are compatible if [si, fi) and [sj, fj) do not

overlap• Goal: select maximum-size subset of mutually

compatible activities.• Start from dynamic programming, then greedy

algorithm, see the relation between the two.

Activity-Selection Problem

• Problem: get your money’s worth out of a carnival– Buy a wristband that lets you onto any ride– Lots of rides, each starting and ending at different

times– Your goal: ride as many rides as possible

• Another, alternative goal that we don’t solve here: maximize time spent on rides

• Welcome to the activity selection problem

Activity-Selection

• Formally:– Given a set S of n activities S ={a1, a2,…, an}

si = start time of activity ai

fi = finish time of activity ai

– Find max-size subset A of compatible activities

Assume (wlog) that f1 f2 … fn

12

34

5

6

DP Solution

• Optimal substructure?

DP solution –step 1

• Optimal substructure of activity-selection problem.– Assume that f1 … fn (otherwise, sort them by fi)

– Define Sij={ak: fi sk<fksj}, i.e., all activities starting after ai finished and ending before aj begins.

– Define two fictitious activities a0 with f0=0 and an+1 with sn+1=

• So f0 f1 … fn+1.

– Then an optimal solution including ak to Sij contains within it the optimal solution to Sik and Skj.

DP solution –step 2

• A recursive solution• Let c[i,j] be # of activities in a maximum-size

subset of mutually compatible activities in Sij. So the solution is c[0,n+1]=S0,n+1.

• C[i,j]= 0 if Sij= max{c[i,k]+c[k,j]+1} if Sij

i<k<j and akSij

Greedy Algorithms

• Greedy choice– Intuition: we should choose an activity that leaves the

resource available for as many other activities as possible– So, consider the locally optimal choice

• Select the activity ak with the earliest finish time in Si,j

• Unlike DP solution, after the local greedy choice, only one subproblem remains!

• One big question:– Is our intuition correct?– We have to prove it is safe to make the greedy choice

Justify Greedy Choice

• Theorem 16.1: consider any nonempty subproblem Sij, and let am be the activity in Sij with earliest finish time: fm=min{fk : ak Sij}, then

1. Activity am is used in some maximum-size subset of mutually compatible activities of Sij.

2. The subproblem Sim is empty, so that choosing am leaves Smj as the only one that may be nonempty.

• Proof of the theorem (p418)

Top-Down Rather Than Bottom-Up

• To solve Sij, choose am in Sij with the earliest finish time, then solve Smj, (Sim is empty)

• It is certain that optimal solution to Smj is in optimal solution to Sij.

• No need to solve Smj ahead of Sij.

• Subproblem pattern: Si,n+1.

Recursive Solutionrecursive_select(s, f, k, n) {

m = k+1

while (m < n && s[m] < f[k])

m++

if (m < n)

return {am} U recursive_select(s, f, m, n)

else

return Ø

}

Optimal Solution Properties• In DP, optimal solution depends:

– How many subproblems to divide. (2 subproblems)– How many choices to determine which subproblem to use. (j-i-1

choices) • However, the above theorem (16.1) reduces both significantly

– One subproblem (the other is sure to be empty).– One choice, i.e., the one with earliest finish time in Sij. – Moreover, top-down solving, rather than bottom-up in DP.– Pattern to the subproblems that we solve, Sm,n+1 from Sij.– Pattern to the activities that we choose. The activity with earliest

finish time.– With this local optimal, it is in fact the global optimal.

Elements of greedy strategy• Determine the optimal substructure• Develop the recursive solution• Prove one of the optimal choices is the greedy choice

yet safe• Show that all but one of subproblems are empty

after greedy choice• Develop a recursive algorithm that implements the

greedy strategy• Convert the recursive algorithm to an iterative one.

Change-Making ProblemGiven unlimited amounts of coins of denominations d1 > … > dm ,

give change for amount n with the least number of coins

Example: d1 = 25c, d2 =10c, d3 = 5c, d4 = 1c and n = 48c

Greedy solution:

Greedy solution is• optimal for any amount and “normal’’ set of denominations• may not be optimal for arbitrary coin denominations

– (4,3,1) for 6

Minimum Spanning Tree (MST), p623-628

• Spanning tree of a connected graph G: a connected acyclic subgraph of G that includes all of G’s vertices

• Minimum spanning tree of a weighted, connected graph G: a spanning tree of G of minimum total weight

Example: c

db

a

6

24

3

1

Prim’s MST algorithm (p634-636)• Start with tree T1 consisting of one (any) vertex

and “grow” tree one vertex at a time to produce MST through a series of expanding subtrees T1, T2, …, Tn

• On each iteration, construct Ti+1 from Ti by adding vertex not in Ti that is closest to those already in Ti (this is a “greedy” step!)

• Stop when all vertices are included

Prim’s MST algorithm• Start with tree T1 consisting of one (any) vertex

and “grow” tree one vertex at a time to produce MST through a series of expanding subtrees T1, T2, …, Tn

• On each iteration, construct Ti+1 from Ti by adding vertex not in Ti that is closest to those already in Ti (this is a “greedy” step!)

• Stop when all vertices are included

Prim’s algorithm

Step 0: Original graph Step 1: D is chose as an arbitrary starting node

Step 2: A is added into the MST Step 3: F is added into the MST

Prim’s algorithm

Step 4: B is added into the MST Step 5: E is added into the MST

Step 6: C is added into the MST Step 7: G is added into the MST

Notes about Prim’s algorithm• Proof by induction that this construction actually

yields MST

• Needs priority queue for locating closest fringe vertex

• Efficiency– O(n2) for weight matrix representation of graph and array

implementation of priority queue – O(m log n) for adjacency list representation of graph with n

vertices and m edges and min-heap implementation of priority queue, how to get this

O(m log n) Prim’s Alg.

• Hints– A mini-heap of size n, each vertex ordered by

mini_dist of infinity except the initial vertex– parent[n]:– n iterations of heap removal operation

• For each removal, update the mini_dist and parent of the remaining vertices in the heap

• m/n: avg. # of edges per vertex

Shortest paths – Dijkstra’s algorithmSingle Source Shortest Paths Problem: Given a weighted connected graph G, find shortest paths from source vertex sto each of the other vertices

Dijkstra’s algorithm: Similar to Prim’s MST algorithm, with a different way of computing numerical labels: Among verticesnot already in the tree, it finds vertex u with the smallest sum dv + w(v,u)

where v is a vertex for which shortest path has been already found on preceding iterations (such vertices form a tree) dv is the length of the shortest path form source to v w(v,u) is the length (weight) of edge from v to u

Example

d 4

Tree vertices Remaining verticesTree vertices Remaining vertices

a(-,0) b(a,3) c(-,∞) d(a,7) e(-,∞)

a

b4

e

37

62 5

c

a

b

d

4c

e

3

7 4

62 5

a

b

d

4c

e

3

7 4

62 5

a

b

d

4c

e

3

7 4

62 5

b(a,3) c(b,3+4) d(b,3+2) e(-,∞)

d(b,5) c(b,7) e(d,5+4)

c(b,7) e(d,9)

e(d,9)

d

a

b

d

4c

e

3

7 4

62 5

Notes on Dijkstra’s algorithm

• Doesn’t work for graphs with negative weights

• Applicable to both undirected and directed graphs

• Efficiency– O(|V|2) for graphs represented by weight matrix and array

implementation of priority queue– O(|E|log|V|) for graphs represented by adj. lists and min-heap

implementation of priority queue

• Don’t mix up Dijkstra’s algorithm with Prim’s algorithm!

Review:The Knapsack Problem

• The famous knapsack problem:– A thief breaks into a museum. Fabulous paintings,

sculptures, and jewels are everywhere. The thief has a good eye for the value of these objects, and knows that each will fetch hundreds or thousands of dollars on the clandestine art collector’s market. But, the thief has only brought a single knapsack to the scene of the robbery, and can take away only what he can carry. What items should the thief take to maximize the haul?

Review: The Knapsack Problem

• More formally, the 0-1 knapsack problem:– The thief must choose among n items, where the ith

item worth vi dollars and weighs wi pounds

– Carrying at most W pounds, maximize value• Note: assume vi, wi, and W are all integers

• “0-1” b/c each item must be taken or left in entirety

• A variation, the fractional knapsack problem:– Thief can take fractions of items– Think of items in 0-1 problem as gold ingots, in

fractional problem as buckets of gold dust

Review: The Knapsack Problem And Optimal Substructure

• Both variations exhibit optimal substructure• To show this for the 0-1 problem, consider the

most valuable load weighing at most W pounds– If we remove item j from the load, what do we

know about the remaining load?– A: remainder must be the most valuable load

weighing at most W - wj that thief could take from museum, excluding item j

Solving The Knapsack Problem• The optimal solution to the fractional knapsack

problem can be found with a greedy algorithm– How?

• The optimal solution to the 0-1 problem cannot be found with the same greedy strategy– Greedy strategy: take in order of dollars/pound– Example: 3 items weighing 10, 20, and 30 pounds,

knapsack can hold 50 pounds• Suppose item 2 is worth $100. Assign values to the other items so

that the greedy strategy will fail

The Knapsack Problem: Greedy vs. DP

• The fractional problem can be solved greedily• The 0-1 problem cannot be solved with a

greedy approach– As you have seen, however, it can be solved with

dynamic programming

Coding ProblemCoding: assignment of bit strings to alphabet characters

Codewords: bit strings assigned for characters of alphabet

Two types of codes:• fixed-length encoding (e.g., ASCII)• variable-length encoding (e,g., Morse code)

Prefix-free codes: no codeword is a prefix of another codeword

Problem: If frequencies of the character occurrences are known, what is the best binary prefix-free code?

Huffman codes• Any binary tree with edges labeled with 0’s and 1’s yields a prefix-free code

of characters assigned to its leaves

• Optimal binary tree minimizing the expected (weighted average) length of a codeword can be constructed as follows

Huffman’s algorithm

Initialize n one-node trees with alphabet characters and the tree weights with their frequencies.

Repeat the following step n-1 times: join two binary trees with smallest weights into one (as left and right subtrees) and make its weight equal the sum of the weights of the two trees.

Mark edges leading to left and right subtrees with 0’s and 1’s, respectively.

Example

character A B C D _frequency 0.35 0.1 0.2 0.2 0.15

codeword 11 100 00 01 101

average bits per character: 2.25for fixed-length encoding: 3compression ratio: (3-2.25)/3*100% =

25%

0.25

0.1

B

0.15

_

0.2

C

0.2

D

0.35

A

0.2

C

0.2

D

0.35

A

0.1

B

0.15

_

0.4

0.2

C

0.2

D

0.6

0.25

0.1

B

0.15

_

0.6

1.0

0 1

0.4

0.2

C

0.2

D0.25

0.1

B

0.15

_

0 1 0

0

1

1

0.25

0.1

B

0.15

_

0.35

A

0.4

0.2

C

0.2

D

0.35

A

0.35

A