05 analysis of algorithms: heap and quick sort

Analysis of AlgorithmsSorting

Andres Mendez-Vazquez

February 4, 2016

1 / 109

Images/ITESM.png

Outline1 Sorting problem

DefinitionClassic Complexities

2 Heap SortHeap Sort: DefinitionsHeap: Finding Parents and ChildrenMax-HeapifyBuild Max Heap: Using Max-HeapifyHeap Sort

3 Applications of Heap Data StructureHeap Sort: Exercises

4 QuicksortThe Divide and Conquer QuicksortQuicksort: Complexity AnalysisRandomized Quicksort

5 Lower Bounds of SortingLower Bounds of Sorting: Basic Concepts

2 / 109

Images/ITESM.png

3 / 109

Images/ITESM.png

Sorting Problem

InputA sequence of n numbers 〈a1, a2, ..., an〉.

OutputA permutation (reordering) 〈a′1, a′2, ..., a′n〉 such that a′1 ≤ a′2 ≤ ... ≤ a′n

4 / 109

Images/ITESM.png

Sorting Problem

InputA sequence of n numbers 〈a1, a2, ..., an〉.

OutputA permutation (reordering) 〈a′1, a′2, ..., a′n〉 such that a′1 ≤ a′2 ≤ ... ≤ a′n

4 / 109

Images/ITESM.png

5 / 109

Images/ITESM.png

Some Sorting Algorithms

Table of Sorting AlgorithmsAlgorithm Worst-case running time Expected running time

Insertion sort Θ(n2) Θ(n2)Merge sort Θ(n log n) Θ(n log n)Heapsort Θ(n log n) -Quicksort Θ(n2) Θ(n log n)(expected)

Countingsort Θ(k + n) Θ(k + n)Radix sort Θ(d(k + n)) Θ(d(k + n))Bucket sort Θ(n2) Θ(n)(average-case)

6 / 109

Images/ITESM.png

6 / 109

Images/ITESM.png

6 / 109

Images/ITESM.png

6 / 109

Images/ITESM.png

6 / 109

Images/ITESM.png

6 / 109

Images/ITESM.png

6 / 109

Images/ITESM.png

7 / 109

Images/ITESM.png

Heap Sort: Definitions

DefinitionA heap is an array object that can be viewed as a nearly complete binarytree.

0 1 2 3 4 5 6 7

8 6 3 4 5 1

8 / 109

Images/ITESM.png

Heap: Basic Attributes

Given an array A, we have that length[A]It is the size of the storing array.

heap − size[A]Tell us how many elements in the heap are stored in the array.

Thus, we have

0 ≤ heap − size[A] ≤ length[A] (1)

9 / 109

Images/ITESM.png

Thus, we have

9 / 109

Images/ITESM.png

Thus, we have

9 / 109

Images/ITESM.png

10 / 109

Images/ITESM.png

Heap Sort: Calculations given a Node i in the heap

Parent(i) - Parent NodeParent(i) =

⌊ i2⌋

Left Node Child: Left(i)Left(i) = 2i

Right Node Child: Right(i)Right(i) = 2i + 1

11 / 109

Images/ITESM.png

⌊ i2⌋

11 / 109

Images/ITESM.png

⌊ i2⌋

11 / 109

Images/ITESM.png

Max/Min Heap Properties

Given thatA [i] returns the value of the key, we have that

Max Heap propertyA [Parent(i)] ≥ A[i]

Min Heap propertyA [Parent(i)] ≤ A [i]

12 / 109

Images/ITESM.png

12 / 109

Images/ITESM.png

12 / 109

Images/ITESM.png

13 / 109

Images/ITESM.png

What we want!!!

A function to keep the property of max or min heapAfter all, remembering Kolmogorov, we are acting in a part of the arraytrying to keep certain properties

Which ONE?

Remember

14 / 109

Images/ITESM.png

What we want!!!

A function to keep the property of max or min heapAfter all, remembering Kolmogorov, we are acting in a part of the arraytrying to keep certain properties

Which ONE?

RememberSingle nodes are always min heaps or max heaps

14 / 109

Images/ITESM.png

Heap Sort: Max-HeapifyAlgorithm (preserving the heap property) when somebody violates themax/min propertyMax-Heapify(A, i)

1 l = Left(i)2 r = Right(i)3 If l ≤ heap − size [A] and A [l] > A [i]4 largest = l5 else largest = i6 If r ≤ heap − size [A] and A [r ] > A [largest]7 largest = r8 if largest 6= i9 exchange A[i] with A[largest]10 Max-Heapify(A, largest)

Figure: A trickle down algorithm

15 / 109

Images/ITESM.png

15 / 109

Images/ITESM.png

15 / 109

Images/ITESM.png

15 / 109

Images/ITESM.png

15 / 109

Images/ITESM.png

15 / 109

Images/ITESM.png

15 / 109

Images/ITESM.png

15 / 109

Images/ITESM.png

15 / 109

Images/ITESM.png

15 / 109

Images/ITESM.png

Example keeping the heap property starting at i = 1Here, you could imagine that somebody inserted a node at i = 13. If l ≤ heap − size [A] and A [l] > A [i]4 largest = l5 else largest = i6 If r ≤ heap − size [A] and A [r ] > A [largest]7 largest = r

14 7 9 3

l=2 r=3

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

4 16 10

Violating Property

14 7 9 3 8 1 1

16 / 109

Images/ITESM.png

Example keeping the heap property starting at i = 1

One of the children is chosen to be exchanged8. if largest 6= i9. exchange A[i] with A[largest]

14 7 9 3

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

4 16 10

Violating Property

Larger

14 7 9 3 8 1 1

Exchange

17 / 109

Images/ITESM.png

Example: Now i = largest

Make the excahnge and call the Max-Heapify10. Max-Heapify(A, largest)

14 7 9 3

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

416 10 14 7 9 3 8 1 1

i=2 3Violating property?

18 / 109

Images/ITESM.png

Keep going

14 7 9 3

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

416 10 14 7 9 3 8 1 1

19 / 109

Images/ITESM.png

Keep going

14 7 9 3

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

416 10 14 7 9 3 8 1 1

20 / 109

Images/ITESM.png

Keep going

14 7 9 3

l=4 r=5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

416 10 14 7 9 3 8 1 1

21 / 109

Images/ITESM.png

Keep going

i=4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

416 10 7 9 3 8 1 1

Exchange

22 / 109

Images/ITESM.png

Keep going

i=4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

416 10 7 9 3 8 1 1

Violating property?

23 / 109

Images/ITESM.png

Keep going

7 9 38

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

16 10 7 9 38 1 1

24 / 109

Images/ITESM.png

Complexity of Max-Heapify

For thisIt is possible to prove that the upper bound on the size of each children’ssubtrees is 2n

3 starting at the root (First Recursive Call!!!).

ThusIn addition, we use the idea of height from the root node (h = 0) to leaves(h = log n − 1).

25 / 109

Images/ITESM.png

For thisIt is possible to prove that the upper bound on the size of each children’ssubtrees is 2n

3 starting at the root (First Recursive Call!!!).

ThusIn addition, we use the idea of height from the root node (h = 0) to leaves(h = log n − 1).

25 / 109

Images/ITESM.png

We have that by using the nearly complete structure1 First for n = 1, we have that the size of children’s subtrees is 0 < 2

3 .2 For n = 2, we have that the size of children’s subtrees is at most

1 < 43 .

3 For n = 3, we have that the size of children’s subtrees is at most1 < 6

3 = 2.4 For n = 4, we have that the size of children’s subtrees is at most

2 < 83 .

5 etc...

26 / 109

Images/ITESM.png

1 < 43 .

2 < 83 .

5 etc...

26 / 109

Images/ITESM.png

1 < 43 .

2 < 83 .

5 etc...

26 / 109

Images/ITESM.png

1 < 43 .

2 < 83 .

5 etc...

26 / 109

Images/ITESM.png

1 < 43 .

2 < 83 .

5 etc...

26 / 109

Images/ITESM.png

Do you notice the following?

Imagine the following case

The maximum number of nodes in both children assuming a full treewith n nodes

21 + 22 + ... + 2dlog ne−2 + 2dlog ne−1 (2)

27 / 109

Images/ITESM.png

Do you notice the following?

Imagine the following case

The maximum number of nodes in both children assuming a full treewith n nodes

21 + 22 + ... + 2dlog ne−2 + 2dlog ne−1 (2)

27 / 109

Images/ITESM.png

Imagine the following special case

The maximum number of nodes in one child is equal to21 + 22 + ... + 2dlog ne−2

2 + 2dlog ne−1

28 / 109

Images/ITESM.png

Imagine the following special case

The maximum number of nodes in one child is equal to21 + 22 + ... + 2dlog ne−2

2 + 2dlog ne−1

28 / 109

Images/ITESM.png

The total number of elements in a child’s subtree

The total number of nodes in a CHILD is bounded

21 + 22 + ... + 2dlog ne−2

2 + 2dlog ne−2 = 1 + 22 + ... + 2dlog ne−3 + 2dlog ne−2

= 1− 2dlog ne−2

1− 2 + 2dlog ne−2

= 2dlog ne−2 − 1 + 2dlog ne−2

= 2× 2blog nc−2 − 1

= 2dlog ne−1 − 23 −

< 2dlog ne−1 − 23

29 / 109

Images/ITESM.png

21 + 22 + ... + 2dlog ne−2

= 1− 2dlog ne−2

1− 2 + 2dlog ne−2

= 2× 2blog nc−2 − 1

= 2dlog ne−1 − 23 −

< 2dlog ne−1 − 23

29 / 109

Images/ITESM.png

21 + 22 + ... + 2dlog ne−2

= 1− 2dlog ne−2

1− 2 + 2dlog ne−2

= 2× 2blog nc−2 − 1

= 2dlog ne−1 − 23 −

< 2dlog ne−1 − 23

29 / 109

Images/ITESM.png

21 + 22 + ... + 2dlog ne−2

= 1− 2dlog ne−2

1− 2 + 2dlog ne−2

= 2× 2blog nc−2 − 1

= 2dlog ne−1 − 23 −

< 2dlog ne−1 − 23

29 / 109

Images/ITESM.png

21 + 22 + ... + 2dlog ne−2

= 1− 2dlog ne−2

1− 2 + 2dlog ne−2

= 2× 2blog nc−2 − 1

= 2dlog ne−1 − 23 −

< 2dlog ne−1 − 23

29 / 109

Images/ITESM.png

21 + 22 + ... + 2dlog ne−2

= 1− 2dlog ne−2

1− 2 + 2dlog ne−2

= 2× 2blog nc−2 − 1

= 2dlog ne−1 − 23 −

< 2dlog ne−1 − 23

29 / 109

Images/ITESM.png

Notice the following

2dlog ne <43[2log n

When n = 2p − 1For the case that that n ≤ 2p − 1 we can use the fact thatdlog ne = p for some power of 2.

30 / 109

Images/ITESM.png

Induction to prove the previous statement

Step n = 1

2dlog 1e = 20 = 1 <43 × 2log 0 (5)

Assume is true for n

2dlog ne <43[2log n

31 / 109

Images/ITESM.png

Step n = 1

2dlog 1e = 20 = 1 <43 × 2log 0 (5)

Assume is true for n

2dlog ne <43[2log n

31 / 109

Images/ITESM.png

Now prove for n + 1

2dlog(n+1)e = 2dlog(2p−1+1)e

= 2dpe

= 2log 2p

<43[2log 2p]

= 43[2log(n+1)

32 / 109

Images/ITESM.png

Now prove for n + 1

= 2dpe

= 2log 2p

<43[2log 2p]

= 43[2log(n+1)

32 / 109

Images/ITESM.png

Now prove for n + 1

= 2dpe

= 2log 2p

<43[2log 2p]

= 43[2log(n+1)

32 / 109

Images/ITESM.png

Now prove for n + 1

= 2dpe

= 2log 2p

<43[2log 2p]

= 43[2log(n+1)

32 / 109

Images/ITESM.png

Now prove for n + 1

= 2dpe

= 2log 2p

<43[2log 2p]

= 43[2log(n+1)

32 / 109

Images/ITESM.png

Now prove for n + 1

= 2dpe

= 2log 2p

<43[2log 2p]

= 43[2log(n+1)

32 / 109

Images/ITESM.png

ThereforeWe have that

21 + 22 + ... + 2dlog ne−2

2 + 2dlog ne−2 < 2dlog ne−1 − 23

= 2dlog ne

2 − 23

<43[2log n−1

]− 2

3[2× 2log n−1 − 1

3[2log n − 1

3 [n − 1]

33 / 109

Images/ITESM.png

21 + 22 + ... + 2dlog ne−2

2 + 2dlog ne−2 < 2dlog ne−1 − 23

= 2dlog ne

2 − 23

<43[2log n−1

]− 2

3[2× 2log n−1 − 1

3[2log n − 1

3 [n − 1]

33 / 109

Images/ITESM.png

21 + 22 + ... + 2dlog ne−2

2 + 2dlog ne−2 < 2dlog ne−1 − 23

= 2dlog ne

2 − 23

<43[2log n−1

]− 2

3[2× 2log n−1 − 1

3[2log n − 1

3 [n − 1]

33 / 109

Images/ITESM.png

21 + 22 + ... + 2dlog ne−2

2 + 2dlog ne−2 < 2dlog ne−1 − 23

= 2dlog ne

2 − 23

<43[2log n−1

]− 2

3[2× 2log n−1 − 1

3[2log n − 1

3 [n − 1]

33 / 109

Images/ITESM.png

21 + 22 + ... + 2dlog ne−2

2 + 2dlog ne−2 < 2dlog ne−1 − 23

= 2dlog ne

2 − 23

<43[2log n−1

]− 2

3[2× 2log n−1 − 1

3[2log n − 1

3 [n − 1]

33 / 109

Images/ITESM.png

21 + 22 + ... + 2dlog ne−2

2 + 2dlog ne−2 < 2dlog ne−1 − 23

= 2dlog ne

2 − 23

<43[2log n−1

]− 2

3[2× 2log n−1 − 1

3[2log n − 1

3 [n − 1]

33 / 109

Images/ITESM.png

21 + 22 + ... + 2dlog ne−2

2 + 2dlog ne−2 < 2dlog ne−1 − 23

= 2dlog ne

2 − 23

<43[2log n−1

]− 2

3[2× 2log n−1 − 1

3[2log n − 1

3 [n − 1]

33 / 109

Images/ITESM.png

Complexity of Max-HeapifyKnowing that the number of nodes in any child is bounded by

2n3 (7)

Thus, given that T (n) represent the complexity of the Max-Heapify

T (n) = T (How many nodes will be touched by the recusrsion) + Θ (1)(8)

HereΘ (1) is the constant part of the algorithm before recursion.T (How many nodes will be touched by the recusrsion) =

T(∑ log2 n

2 −1i=1 3

How? 34 / 109

Images/ITESM.png

2n3 (7)

T(∑ log2 n

2 −1i=1 3

How? 34 / 109

Images/ITESM.png

2n3 (7)

T(∑ log2 n

2 −1i=1 3

How? 34 / 109

Images/ITESM.png

2n3 (7)

T(∑ log2 n

2 −1i=1 3

How? 34 / 109

Images/ITESM.png

2n3 (7)

T(∑ log2 n

2 −1i=1 3

How? 34 / 109

Images/ITESM.png

The Recursion Idea

35 / 109

Images/ITESM.png

Complexity of Max-HeapifyThus

log2 n−12 −1∑i=1

3 = 3log2 n

2 − 13− 1 − 3

12)log2 n

2 − 3

= nlog2

2 − 3

≤ n0.8

≤ 2n0.8

3≤ 2n

336 / 109

Images/ITESM.png

log2 n−12 −1∑i=1

3 = 3log2 n

2 − 13− 1 − 3

12)log2 n

2 − 3

= nlog2

2 − 3

≤ n0.8

≤ 2n0.8

3≤ 2n

336 / 109

Images/ITESM.png

log2 n−12 −1∑i=1

3 = 3log2 n

2 − 13− 1 − 3

12)log2 n

2 − 3

= nlog2

2 − 3

≤ n0.8

≤ 2n0.8

3≤ 2n

336 / 109

Images/ITESM.png

log2 n−12 −1∑i=1

3 = 3log2 n

2 − 13− 1 − 3

12)log2 n

2 − 3

= nlog2

2 − 3

≤ n0.8

≤ 2n0.8

3≤ 2n

336 / 109

Images/ITESM.png

log2 n−12 −1∑i=1

3 = 3log2 n

2 − 13− 1 − 3

12)log2 n

2 − 3

= nlog2

2 − 3

≤ n0.8

≤ 2n0.8

3≤ 2n

336 / 109

Images/ITESM.png

log2 n−12 −1∑i=1

3 = 3log2 n

2 − 13− 1 − 3

12)log2 n

2 − 3

= nlog2

2 − 3

≤ n0.8

≤ 2n0.8

3≤ 2n

336 / 109

Images/ITESM.png

Thus, if we assume that T is an increasing monotone function

T (n) = T

log2 n

2 −1∑i=1

+ Θ (1)

≤ T(2n

)+ Θ (1)

Algorithm Complexity

This is by the master the master theorem O (log2 n).

37 / 109

Images/ITESM.png

T (n) = T

log2 n

2 −1∑i=1

+ Θ (1)

≤ T(2n

)+ Θ (1)

37 / 109

Images/ITESM.png

T (n) = T

log2 n

2 −1∑i=1

+ Θ (1)

≤ T(2n

)+ Θ (1)

37 / 109

Images/ITESM.png

38 / 109

Images/ITESM.png

Heap Sort: Using Max-Heapify

Algorithm Build-Max-HeapBuild-Max-Heap(A)

1 heap − size[A] = length[A]2 for i = blength[A]/2c downto 13 Max-Heapify(A, i)

Figure: Building a Heap

39 / 109

Images/ITESM.png

Question?Why from blength[A]/2c?Look at this

4 5 6 7

8 9 10 11

Thus, the nodes blength[A]/2c+ 1, blength[A]/2c+ 2, ..., nThey are actually leaves.This can be proved by induction on n!!!

I I leave this to you.40 / 109

Images/ITESM.png

4 5 6 7

8 9 10 11

Images/ITESM.png

4 5 6 7

8 9 10 11

Images/ITESM.png

4 5 6 7

8 9 10 11

Images/ITESM.png

Question?

What about the loop invariance?Look at the Board!!!

41 / 109

Images/ITESM.png

Build Max Heap: Using Max-Heapify

Example

4 i=5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

161079 3 811

Exchange

42 / 109

Images/ITESM.png

Example

4 i=5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

16 10 79 3 811

Exchange

43 / 109

Images/ITESM.png

Example

i=4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

16 10 79 3 811

Exchange

44 / 109

Images/ITESM.png

Example

i=4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

16 10 79 3 811

Exchange

45 / 109

Images/ITESM.png

Example

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

16 10 79 3 811

Exchange

46 / 109

Images/ITESM.png

Example

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

1610 79 3 811

Exchange

47 / 109

Images/ITESM.png

Example

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

1610 79 3 811

Exchange

48 / 109

Images/ITESM.png

Example

14 7 9 3

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

16 10 7 9 3 811

49 / 109

Images/ITESM.png

Example

14 7 9 3

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

16 10 7 9 3 811

Exchange

50 / 109

Images/ITESM.png

Example

14 7 9 3

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

16 10 7 9 3 811

Exchange

51 / 109

Images/ITESM.png

Example

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

16 10 7 9 3 811

Exchange

52 / 109

Images/ITESM.png

Example

7 9 38

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

16 10 7 9 38 1 1

53 / 109

Images/ITESM.png

Height h of the Heap for Complexity of Build-Max-Heap

We can use the height of a three to derive a tight boundThe height h is the number of edges on the longest simple downwardpath from the node to a leaf.You have at most

⌉nodes at any height, where n is the total

number of nodes.

54 / 109

Images/ITESM.png

number of nodes.

54 / 109

Images/ITESM.png

number of nodes.

54 / 109

Images/ITESM.png

Cost of Building the Build-Max-Heap

Possible cost

O (n log2 n) (9)

55 / 109

Images/ITESM.png

Cost of Building the Build-Max-HeapWe have that you have

The number of nodes explored horizontally by the “for” loop can bebounded by ⌈ n

⌉(10)

The depth of the Max-Heapify is

O (h) (11)

Therefore we have the following total tighter cost

blog nc∑h=0

⌈ n2h+1

⌉O(h) = O

nblog nc∑

56 / 109

Images/ITESM.png

⌉(10)

O (h) (11)

blog nc∑h=0

⌈ n2h+1

⌉O(h) = O

nblog nc∑

56 / 109

Images/ITESM.png

⌉(10)

O (h) (11)

blog nc∑h=0

⌈ n2h+1

⌉O(h) = O

nblog nc∑

56 / 109

Images/ITESM.png

⌉(10)

O (h) (11)

blog nc∑h=0

⌈ n2h+1

⌉O(h) = O

nblog nc∑

56 / 109

Images/ITESM.png

⌉(10)

O (h) (11)

blog nc∑h=0

⌈ n2h+1

⌉O(h) = O

nblog nc∑

56 / 109

Images/ITESM.png

ThusFrom (A.8) at Cormen’s

∞∑k=0

kxk = x(1− x)2

Thus, we have that

nblog nc∑

n∞∑

h=0h(1

1− 12

= O (n)57 / 109

Images/ITESM.png

∞∑k=0

kxk = x(1− x)2

Thus, we have that

nblog nc∑

n∞∑

h=0h(1

1− 12

= O (n)57 / 109

Images/ITESM.png

∞∑k=0

kxk = x(1− x)2

Thus, we have that

nblog nc∑

n∞∑

h=0h(1

1− 12

= O (n)57 / 109

Images/ITESM.png

∞∑k=0

kxk = x(1− x)2

Thus, we have that

nblog nc∑

n∞∑

h=0h(1

1− 12

= O (n)57 / 109

Images/ITESM.png

∞∑k=0

kxk = x(1− x)2

Thus, we have that

nblog nc∑

n∞∑

h=0h(1

1− 12

= O (n)57 / 109

Images/ITESM.png

58 / 109

Images/ITESM.png

Heapsort AlgorithmHeapsort(A)

1 Build-Max-Heap(A)2 for i = length[A] downto 23 exchange A[1] with A[i]4 heap − size[A] = heap − size[A]− 15 Max-Heapify(A, 1)

Figure: Heapsort

59 / 109

Images/ITESM.png

Figure: Heapsort

59 / 109

Images/ITESM.png

Figure: Heapsort

59 / 109

Images/ITESM.png

Figure: Heapsort

59 / 109

Images/ITESM.png

Figure: Heapsort

59 / 109

Images/ITESM.png

Figure: Heapsort

59 / 109

Images/ITESM.png

Figure: Heapsort

59 / 109

Images/ITESM.png

Example: Heapsort in action! By Moving the top element to thebottom position!!!

To be exchanged

7 9 38

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

16 10 7 9 38 1 1

60 / 109

Images/ITESM.png

7 9 38

4 5 6 7

8 9 i=10

0 1 2 3 4 5 6 7 8 9 10

10 7 9 38 1 1

144 16

Max-Heapify(A,1)

61 / 109

Images/ITESM.png

7 9 38

4 5 6 7

8 9 i=10

0 1 2 3 4 5 6 7 8 9 10

10 7 9 38 1 1

14 4 16

To be exchanged

62 / 109

Images/ITESM.png

4 5 6 7

8 9 i=10

0 1 2 3 4 5 6 7 8 9 10

10 7 9 38 1 1

14 4 16

63 / 109

Images/ITESM.png

4 5 6 7

8 9 i=10

0 1 2 3 4 5 6 7 8 9 10

10 7 9 38 1 1

14 4 16

To be exchanged

64 / 109

Images/ITESM.png

4 5 6 7

8 i=9 10

0 1 2 3 4 5 6 7 8 9 10

10 7 9 38 11

144 16

To be exchangedMax-Heapify(A,1)

65 / 109

Images/ITESM.png

4 5 6 7

8 i=9 10

0 1 2 3 4 5 6 7 8 9 10

7 9 38 11

144 16

To be exchanged

66 / 109

Images/ITESM.png

4 5 6 7

8 i=9 10

0 1 2 3 4 5 6 7 8 9 10

7 9 38 11

144 16

To be exchanged

67 / 109

Images/ITESM.png

4 5 6 7

i=8 9 10

0 1 2 3 4 5 6 7 8 9 10

7 9 381 1

144 16

To be exchanged

Max-Heapify(A,1)

68 / 109

Images/ITESM.png

4 5 6 7

i=8 9 10

0 1 2 3 4 5 6 7 8 9 10

79 38 11

144 16

To be exchanged

69 / 109

Images/ITESM.png

4 5 6 7

i=8 9 10

0 1 2 3 4 5 6 7 8 9 10

79 38 11

144 16

To be exchanged

70 / 109

Images/ITESM.png

4 5 6 i=7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

7 9381 1

144 16

To be exchanged

Max-Heapify(A,1)

71 / 109

Images/ITESM.png

4 5 6 i=7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

7 938 1 1

144 16

To be exchanged

72 / 109

Images/ITESM.png

4 5 6 i=7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

7 938 1 1

144 16

To be exchanged

73 / 109

Images/ITESM.png

4 5 i=6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

7 93 811

144 16

To be exchanged

Max-Heapify(A,1)

74 / 109

Images/ITESM.png

4 5 i=6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

7 93 811

144 16

To be exchanged

75 / 109

Images/ITESM.png

4 5 i=6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

7 93 811

144 16

To be exchanged

76 / 109

Images/ITESM.png

4 i=5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

7 93 81 1

144 16

To be exchanged

Max-Heapify(A,1)

77 / 109

Images/ITESM.png

4 i=5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

7 93 811

144 1610

To be exchanged

78 / 109

Images/ITESM.png

i=4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

7 93 81 1

144 1610

To be exchangedMax-Heapify(A,1)

79 / 109

Images/ITESM.png

i=4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

7 93 811

144 1610

To be exchanged

80 / 109

Images/ITESM.png

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

7 93 81 1

144 1610

Max-Heapify(A,1)

81 / 109

Images/ITESM.png

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

7 93 81 1

144 1610

Max-Heapify(A,1)

82 / 109

Images/ITESM.png

4 5 6 7

8 9 10

0 1 2 3 4 5 6 7 8 9 10

7 93 81 1

144 1610

Max-Heapify(A,1)

83 / 109

Images/ITESM.png

CostO(n log n)

84 / 109

Images/ITESM.png

Applications of Heap Data Structure

Priority QueuesHere, Heaps can be modified to support insert(), delete() and extractmax(),decreaseKey() operations in O(logn) time

This has direct applications1 Bandwidth management:

1 Many modern protocols for Local Area Networks include the concept ofPriority Queues at the Media Access Control (MAC).

2 Discrete Event Simulations3 Schedulers4 Huffman coding5 The Real-time Optimally Adapting Meshes (ROAM)

1 It computes a dynamically changing triangulation of a terrain using twopriority queues.

85 / 109

Images/ITESM.png

85 / 109

Images/ITESM.png

85 / 109

Images/ITESM.png

85 / 109

Images/ITESM.png

85 / 109

Images/ITESM.png

85 / 109

Images/ITESM.png

Heap Sort of ArraysClearly, if the list of numbers is stored in an array!!!

86 / 109

Images/ITESM.png

87 / 109

Images/ITESM.png

Heap Sort: Exercices

From Cormen’s book6.1-16.1-46.1-76.2-56.2-66.3-36.4-26.4-36.4-4

88 / 109

Images/ITESM.png

Who invented Quicksort?

Imagine thisThe quicksort algorithm was developed in 1960 by Tony Hoare (He has apostgraduate certificate in Statistics) while in the Soviet Union, as avisiting student at Moscow State University.

Why?At that time, Hoare worked in a project on machine translation for theNational Physical Laboratory.

To doHe developed the algorithm in order to sort the words to be translated.

89 / 109

Images/ITESM.png

89 / 109

Images/ITESM.png

89 / 109

Images/ITESM.png

90 / 109

Images/ITESM.png

The Divide and Conquer Quicksort

Divide Process1 Compute the index q as part of this partitioning procedure.2 Partition (rearrange) the array A[p, ..., r ] into two (possibly empty)

sub-arrays A[p, ..., q − 1] and A[q + 1, ..., r ]1 each element of A[p, ..., q − 1] is less than or equal to A[q].2 A[q] is less than or equal to each element of A[q + 1, ..., r ].

91 / 109

Images/ITESM.png

91 / 109

Images/ITESM.png

91 / 109

Images/ITESM.png

91 / 109

Images/ITESM.png

ConquerSort the two sub-arrays A[p, ..., q − 1] and A[q + 1, ..., r ] by recursive callsto quicksort.

CombineSince the sub-arrays are sorted in place, no work is needed to combinethem: the entire array A[p, ..., r ] is now sorted.

92 / 109

Images/ITESM.png

ConquerSort the two sub-arrays A[p, ..., q − 1] and A[q + 1, ..., r ] by recursive callsto quicksort.

CombineSince the sub-arrays are sorted in place, no work is needed to combinethem: the entire array A[p, ..., r ] is now sorted.

92 / 109

Images/ITESM.png

Quicksort Algorithm

Quicksort AlgorithmQuicksort(A, p, r)

1 if p < r2 q = Partition (A, p, r)3 Quicksort(A, p, q − 1)4 Quicksort(A, q + 1, r)

93 / 109

Images/ITESM.png

Quicksort Algorithm

93 / 109

Images/ITESM.png

Quicksort Algorithm

93 / 109

Images/ITESM.png

Quicksort Algorithm

93 / 109

Images/ITESM.png

Quicksort Algorithm

93 / 109

Images/ITESM.png

Partition Algorithm

Quicksort PartitionPartition(A, p, r)

1 x = A[r ]2 i = p − 13 for j = p to r − 14 if A[j] ≤ x5 i = i + 16 exchange A[i] with A[j]7 exchange A[i + 1] with A[r ]8 return i + 1

94 / 109

Images/ITESM.png

Partition Algorithm

94 / 109

Images/ITESM.png

Partition Algorithm

94 / 109

Images/ITESM.png

Partition Algorithm

94 / 109

Images/ITESM.png

Partition Algorithm

94 / 109

Images/ITESM.png

Partition Algorithm

94 / 109

Images/ITESM.png

Partition Algorithm

94 / 109

Images/ITESM.png

Partition Algorithm

94 / 109

Images/ITESM.png

Quicksort: What is the Invariance?

Loop Invariance1 If p ≤ k ≤ i, then A[k] ≤ x.2 If i + 1 ≤ k ≤ j − 1, then A[k] > x.3 If k = r , then A[k] = x.4 UNKNOWN

Proof of the Loop InvarianceLook at the Board.

95 / 109

Images/ITESM.png

95 / 109

Images/ITESM.png

95 / 109

Images/ITESM.png

95 / 109

Images/ITESM.png

p i i+1 j-1

A[k]<x A[k]>x

95 / 109

Images/ITESM.png

p i i+1 j-1

A[k]<x A[k]>x

95 / 109

Images/ITESM.png

96 / 109

Images/ITESM.png

Quicksort: Complexity Analysis

Worst-case AnalysisPartition returns two arrays, one of size 0 and one of size n − 1. Then, wehave the recurrence:

T (n) = T (n − 1) + Θ(n) = O(n2). (12)

Best-case AnalysisPartition returns two arrays size n

2 and n2 − 1.

Then, we have the recurrence T (n) = 2T(n

+ Θ(n).

97 / 109

Images/ITESM.png

T (n) = T (n − 1) + Θ(n) = O(n2). (12)

2 and n2 − 1.

+ Θ(n).

97 / 109

Images/ITESM.png

T (n) = T (n − 1) + Θ(n) = O(n2). (12)

2 and n2 − 1.

+ Θ(n).

97 / 109

Images/ITESM.png

T (n) = T (n − 1) + Θ(n) = O(n2). (12)

2 and n2 − 1.

+ Θ(n).

97 / 109

Images/ITESM.png

However for an Unbalanced Partition!!!

Unbalanced partitioning returns a O (n log n)After certain level, the total steps are ≤ than cn!!!

98 / 109

Images/ITESM.png

Worst-case Analysis in detailFrom the recurrence: T (n) = max05q≤n−1(T (q) + T (n − q − 1)) + Θ(n)

By substitution, we can proveComplexity O(n2).

This can be proved as followsBLACKBOARD!

99 / 109

Images/ITESM.png

99 / 109

Images/ITESM.png

99 / 109

Images/ITESM.png

100 / 109

Images/ITESM.png

Remember?

The use of uniform distributionTo get the average behaivior!!!

In many casesIt is better than the worst case scenario....

ThusWe can introduce randomization in the Quicksort.

101 / 109

Images/ITESM.png

Remember?

101 / 109

Images/ITESM.png

Remember?

101 / 109

Images/ITESM.png

Randomized Quicksort

RANDOMIZED-QUICKSORT(A,p,r)Randomized-Quicksort(A, p, r)

1 if p < r2 q =Randomized-Partition(A, p, r)3 Randomized-Quicksort(A, p, q − 1)4 Randomized-Quicksort(A, q + 1, r)

RANDOMIZED-PARTITION(A,p,r)

102 / 109

Images/ITESM.png

102 / 109

Images/ITESM.png

102 / 109

Images/ITESM.png

102 / 109

Images/ITESM.png

Randomized QuicksortRANDOMIZED-QUICKSORT(A,p,r)Randomized-Quicksort(A, p, r)

RANDOMIZED-PARTITION(A,p,r)Randomized-Partition(A, p, r)

1 i =Random(p, r)2 exchange A[r ] with A[i]3 return Partition(A, p, r)

102 / 109

Images/ITESM.png

Quicksort: Randomized Quicksort

Expected running timeThe expected running time for the Randomized Quicksort algorithm arisesfrom the following lemma.

Lemma 7.1 (Cormen’s book)Let X be the number of comparisons performed in line 4 ofPARTITION algorithm over the entire execution of QUICKSORT onan n-element array.Then, the running time of QUICKSORT is O(n + X).

Now the proof of the expected running time.BLACKBOARD!

103 / 109

Images/ITESM.png

103 / 109

Images/ITESM.png

103 / 109

Images/ITESM.png

Therefore

It is possible to conclude thatThe Average Time Complexity of the Quicksort is O(n log n)

104 / 109

Images/ITESM.png

Applications

Sorting in Special EnvironmentsExample: Using Massive Parallel Stream Processors.

Multi-Objective OptimizationYes!!! Numerical Analysis using the Quick Sort!!!

Real-Time Visualization of Large Time-Varying MoleculesUse the distance of the atoms to the viewers - the Painters Algorithms!!!

105 / 109

Images/ITESM.png

Applications

105 / 109

Images/ITESM.png

Applications

105 / 109

Images/ITESM.png

106 / 109

Images/ITESM.png

Lower Bounds of Sorting: Basic Concepts

Mergesort and HeapsortThey are algorithms that sort in O(n log n).It is more we can give a sequence such that Ω(n log n).

Property“These algorithms share an interesting property: the sorted order theydetermine is based only on comparisons between the input elements. Wecall such sorting algorithms comparison sorts.”

107 / 109

Images/ITESM.png

107 / 109

Images/ITESM.png

107 / 109

Images/ITESM.png

Lower Bounds of Sorting: Theorem and Corollary

TheoremAny comparison sort algorithm requires Ω(n log n) comparisons in theworst case.

CorollaryHeapsort and Mergesort are asymptotically optimal comparison sorts.

108 / 109

Images/ITESM.png

Lower Bounds of Sorting: Theorem and Corollary

TheoremAny comparison sort algorithm requires Ω(n log n) comparisons in theworst case.

CorollaryHeapsort and Mergesort are asymptotically optimal comparison sorts.

108 / 109

Images/ITESM.png

Exercises

Cormen’s Chapter 77.1-47.2-37.2-57.4-1

109 / 109

05 analysis of algorithms: heap and quick sort

Engineering

exercises4 quicksortthe

basic concepts3

basic concepts2

basic concepts5

outputa permutation