divide and conquer - sjtugao-xf/algorithm/document/slide03... · 2016-03-10 · divide-and-conquer...
Post on 20-Jul-2020
5 Views
Preview:
TRANSCRIPT
Divide-and-ConquerApplications
Sorting Networks
Divide and Conquer∗
Xiaofeng Gao
Department of Computer Science and EngineeringShanghai Jiao Tong University, P.R.China
X033533-Algorithm: Analysis and Theory
∗
Special thanks is given to Prof. Yijia Chen for sharing his teaching materials.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 1/73
Divide-and-ConquerApplications
Sorting Networks
Outline
1 Divide-and-ConquerBasic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
2 ApplicationsSearch and SortMedianMatrix Multiplication
3 Sorting NetworksComparison NetworksZero-One PrincipleConstruction of a Sorting Network
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 2/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Outline
1 Divide-and-ConquerBasic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
2 ApplicationsSearch and SortMedianMatrix Multiplication
3 Sorting NetworksComparison NetworksZero-One PrincipleConstruction of a Sorting Network
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 3/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Divide-and-Conquer Strategy
The divide-and-conquer strategy solves a problemP by:
(1) BreakingP into subproblems that are themselves smallerinstances of the same type of problem.
(2) Recursively solving these subproblems.
(3) Appropriately combining their answers.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 4/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Key Works
The real work to implement Divide-and-Conquer strategy is donepiecemeal, where the key works lay in three different places:
(1) How to partition problem into subproblems.
(2) At the very tail end of the recursion, how to solve the smallestsubproblems outright.
(3) How to glue together the partial answers.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 5/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Outline
1 Divide-and-ConquerBasic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
2 ApplicationsSearch and SortMedianMatrix Multiplication
3 Sorting NetworksComparison NetworksZero-One PrincipleConstruction of a Sorting Network
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 6/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Johann C.F. Gauss
Johann Carl Friedrich Gauss1777 - 1855
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 7/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Johann C.F. Gauss
Johann Carl Friedrich Gauss1777 - 1855
1+ 2+ · · ·+ 100=100· (1+ 100)
2= 5050.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 7/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Multiplication for Complex Numbers
Gauss once noticed that although the product of two complexnumbers
(a + bi)(c + di) = ac− bd + (bc + ad)i
seems to involvefour real-number multiplications, it can in fact bedone with justthree: ac, bd, and(a + b)(c + d), since
bc + ad = (a + b)(c + d) − ac− bd.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 8/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Multiplication for Complex Numbers
Gauss once noticed that although the product of two complexnumbers
(a + bi)(c + di) = ac− bd + (bc + ad)i
seems to involvefour real-number multiplications, it can in fact bedone with justthree: ac, bd, and(a + b)(c + d), since
bc + ad = (a + b)(c + d) − ac− bd.
In our big-O way of thinking, reducing the number of multiplicationsfrom four to three seems wasted ingenuity. However, this modestimprovement becomesvery significant when applied recursively.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 8/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Multiplication for Integers
Supposex andy are twon-bit integers, and assume for conveniencethatn is a power of2.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 9/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Multiplication for Integers
Supposex andy are twon-bit integers, and assume for conveniencethatn is a power of2.
Lemma: ∀n ∈ N, ∃ n′ with n ≤ n′ ≤ 2n such that n′ is a power of 2.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 9/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Multiplication for Integers
Supposex andy are twon-bit integers, and assume for conveniencethatn is a power of2.
Lemma: ∀n ∈ N, ∃ n′ with n ≤ n′ ≤ 2n such that n′ is a power of 2.
As a first step toward multiplyingx andy, we split each of them intotheir left and right halves, which aren/2 bits long:
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 9/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Multiplication for Integers
Supposex andy are twon-bit integers, and assume for conveniencethatn is a power of2.
Lemma: ∀n ∈ N, ∃ n′ with n ≤ n′ ≤ 2n such that n′ is a power of 2.
As a first step toward multiplyingx andy, we split each of them intotheir left and right halves, which aren/2 bits long:
x = xL xR = 2n/2xL + xR
y = yL yR = 2n/2yL + yR.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 9/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Multiplication for Integers
Supposex andy are twon-bit integers, and assume for conveniencethatn is a power of2.
Lemma: ∀n ∈ N, ∃ n′ with n ≤ n′ ≤ 2n such that n′ is a power of 2.
As a first step toward multiplyingx andy, we split each of them intotheir left and right halves, which aren/2 bits long:
x = xL xR = 2n/2xL + xR
y = yL yR = 2n/2yL + yR.
xy = (2n/2xL + xR)(2n/2yL + yR) = 2nxLyL + 2n/2(xLyR + xRyL) + xRyR.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 9/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Multiplication for Integers
Supposex andy are twon-bit integers, and assume for conveniencethatn is a power of2.
Lemma: ∀n ∈ N, ∃ n′ with n ≤ n′ ≤ 2n such that n′ is a power of 2.
As a first step toward multiplyingx andy, we split each of them intotheir left and right halves, which aren/2 bits long:
x = xL xR = 2n/2xL + xR
y = yL yR = 2n/2yL + yR.
xy = (2n/2xL + xR)(2n/2yL + yR) = 2nxLyL + 2n/2(xLyR + xRyL) + xRyR.
The additions take linear time, as do the multiplications bypowers of2 (merely left-shifts).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 9/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Multiplication for Integers
Supposex andy are twon-bit integers, and assume for conveniencethatn is a power of2.
Lemma: ∀n ∈ N, ∃ n′ with n ≤ n′ ≤ 2n such that n′ is a power of 2.
As a first step toward multiplyingx andy, we split each of them intotheir left and right halves, which aren/2 bits long:
x = xL xR = 2n/2xL + xR
y = yL yR = 2n/2yL + yR.
xy = (2n/2xL + xR)(2n/2yL + yR) = 2nxLyL + 2n/2(xLyR + xRyL) + xRyR.
The additions take linear time, as do the multiplications bypowers of2 (merely left-shifts). The significant operations are the four n/2-bitmultiplications; these we can handle byfour recursive calls.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 9/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Our method for multiplyingn-bit numbers starts by making recursivecalls to multiply these four pairs ofn/2-bit numbers, and thenevaluates the preceding expression inO(n) time.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 10/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Our method for multiplyingn-bit numbers starts by making recursivecalls to multiply these four pairs ofn/2-bit numbers, and thenevaluates the preceding expression inO(n) time.
Writing T(n) for the overall running time onn-bit inputs, we gettherecurrence relation:
T(n) = 4T(n/2) + O(n)
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 10/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Our method for multiplyingn-bit numbers starts by making recursivecalls to multiply these four pairs ofn/2-bit numbers, and thenevaluates the preceding expression inO(n) time.
Writing T(n) for the overall running time onn-bit inputs, we gettherecurrence relation:
T(n) = 4T(n/2) + O(n)
Solution: O(n2).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 10/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Our method for multiplyingn-bit numbers starts by making recursivecalls to multiply these four pairs ofn/2-bit numbers, and thenevaluates the preceding expression inO(n) time.
Writing T(n) for the overall running time onn-bit inputs, we gettherecurrence relation:
T(n) = 4T(n/2) + O(n)
Solution: O(n2).
By Gauss’s trick, three multiplications,xLyL, xRyR, and(xL + xR)(yL + yR), suffice, as
xLyR + xRyL = (xL + xR)(yL + yR)− xLyL − xRyR.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 10/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
A divide-and-conquer algorithm for integer multiplication
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 11/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
A divide-and-conquer algorithm for integer multiplication
MULTIPLY (x,y)
Input: positive integersx andy, in binaryOutput: their product
1: n = max(size ofx, size ofy) rounded as a power of 2.2: if n = 1 then3: returnxy.4: end if5: xL, xR = leftmostn/2, rightmostn/2 bits ofx6: yL, yR = leftmostn/2, rightmostn/2 bits ofy7: P1 = MULTIPLY (xL, yL)8: P2 = MULTIPLY (xR, yR)9: P3 = MULTIPLY (xL + xR, yL + yR)
10: returnP1× 2n + (P3− P1− P2)× 2n/2 + P2.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 11/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 12/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis
The recurrence relation: T(n) = 3T(n/2) + O(n)
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 12/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis
The recurrence relation: T(n) = 3T(n/2) + O(n)
⊲ The algorithm’s recursive calls form atree structure.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 12/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis
The recurrence relation: T(n) = 3T(n/2) + O(n)
⊲ The algorithm’s recursive calls form atree structure.
⊲ At each successive level the subproblems gethalvedin size.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 12/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis
The recurrence relation: T(n) = 3T(n/2) + O(n)
⊲ The algorithm’s recursive calls form atree structure.
⊲ At each successive level the subproblems gethalvedin size.
⊲ At the (log2 n)th level, the subproblems get down to size 1, andso the recursion ends.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 12/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis
The recurrence relation: T(n) = 3T(n/2) + O(n)
⊲ The algorithm’s recursive calls form atree structure.
⊲ At each successive level the subproblems gethalvedin size.
⊲ At the (log2 n)th level, the subproblems get down to size 1, andso the recursion ends.
⊲ Theheight of the tree is log2 n.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 12/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis
The recurrence relation: T(n) = 3T(n/2) + O(n)
⊲ The algorithm’s recursive calls form atree structure.
⊲ At each successive level the subproblems gethalvedin size.
⊲ At the (log2 n)th level, the subproblems get down to size 1, andso the recursion ends.
⊲ Theheight of the tree is log2 n.
⊲ Thebranching factor is 3: each problem recursively producesthree smaller ones, with the result that at depthk in the tree thereare3k subproblems, each ofsizen/2k.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 12/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis
The recurrence relation: T(n) = 3T(n/2) + O(n)
⊲ The algorithm’s recursive calls form atree structure.
⊲ At each successive level the subproblems gethalvedin size.
⊲ At the (log2 n)th level, the subproblems get down to size 1, andso the recursion ends.
⊲ Theheight of the tree is log2 n.
⊲ Thebranching factor is 3: each problem recursively producesthree smaller ones, with the result that at depthk in the tree thereare3k subproblems, each ofsizen/2k.
For each subproblem, a linear amount of work is done in identifyingfurther subproblems and combining their answers.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 12/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis
The recurrence relation: T(n) = 3T(n/2) + O(n)
⊲ The algorithm’s recursive calls form atree structure.
⊲ At each successive level the subproblems gethalvedin size.
⊲ At the (log2 n)th level, the subproblems get down to size 1, andso the recursion ends.
⊲ Theheight of the tree is log2 n.
⊲ Thebranching factor is 3: each problem recursively producesthree smaller ones, with the result that at depthk in the tree thereare3k subproblems, each ofsizen/2k.
For each subproblem, a linear amount of work is done in identifyingfurther subproblems and combining their answers. Therefore the total
time spent at depthk in the tree is3k × O( n
2k
)
=
(
32
)k
× O(n).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 12/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis (cont’d)
3k × O( n
2k
)
=
(
32
)k
× O(n).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 13/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis (cont’d)
3k × O( n
2k
)
=
(
32
)k
× O(n).
At the very top level, whenk = 0, we needO(n).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 13/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis (cont’d)
3k × O( n
2k
)
=
(
32
)k
× O(n).
At the very top level, whenk = 0, we needO(n).
At the bottom, whenk = log2 n, it is
O(
3log2 n) = O(
nlog2 3)
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 13/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis (cont’d)
3k × O( n
2k
)
=
(
32
)k
× O(n).
At the very top level, whenk = 0, we needO(n).
At the bottom, whenk = log2 n, it is
O(
3log2 n) = O(
nlog2 3)
Between these two endpoints, the work done increasesgeometricallyfrom O(n) to O(nlog2 3), by a factor of 3/2 per level.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 13/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis (cont’d)
3k × O( n
2k
)
=
(
32
)k
× O(n).
At the very top level, whenk = 0, we needO(n).
At the bottom, whenk = log2 n, it is
O(
3log2 n) = O(
nlog2 3)
Between these two endpoints, the work done increasesgeometricallyfrom O(n) to O(nlog2 3), by a factor of 3/2 per level.
The sum of any increasing geometric series is, within a constantfactor, simply the last term of the series.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 13/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis (cont’d)
3k × O( n
2k
)
=
(
32
)k
× O(n).
At the very top level, whenk = 0, we needO(n).
At the bottom, whenk = log2 n, it is
O(
3log2 n) = O(
nlog2 3)
Between these two endpoints, the work done increasesgeometricallyfrom O(n) to O(nlog2 3), by a factor of 3/2 per level.
The sum of any increasing geometric series is, within a constantfactor, simply the last term of the series. Therefore the overall runningtime is
O(nlog2 3) ≈ O(n1.59).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 13/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
The time analysis (cont’d)
3k × O( n
2k
)
=
(
32
)k
× O(n).
At the very top level, whenk = 0, we needO(n).
At the bottom, whenk = log2 n, it is
O(
3log2 n) = O(
nlog2 3)
Between these two endpoints, the work done increasesgeometricallyfrom O(n) to O(nlog2 3), by a factor of 3/2 per level.
The sum of any increasing geometric series is, within a constantfactor, simply the last term of the series. Therefore the overall runningtime is
O(nlog2 3) ≈ O(n1.59).
We can do even better!X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 13/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Outline
1 Divide-and-ConquerBasic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
2 ApplicationsSearch and SortMedianMatrix Multiplication
3 Sorting NetworksComparison NetworksZero-One PrincipleConstruction of a Sorting Network
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 14/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Master Theorem
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 15/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Master Theorem
IfT(n) = aT(⌈n/b⌉) + O(nd)
for some constantsa > 0, b > 1, andd ≥ 0,
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 15/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Master Theorem
IfT(n) = aT(⌈n/b⌉) + O(nd)
for some constantsa > 0, b > 1, andd ≥ 0, then
T(n) =
O(nd) if d > logb aO(nd logn) if d = logb aO(nlogb a) if d < logb a.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 15/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Proof of Master Theorem
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 16/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Proof of Master Theorem
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 17/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Proof of Master Theorem
Assume thatn is a power ofb. This will not influence the final boundin any important way:n is at most a multiplicative factor ofb awayfrom some power ofb.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 17/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Proof of Master Theorem
Assume thatn is a power ofb. This will not influence the final boundin any important way:n is at most a multiplicative factor ofb awayfrom some power ofb.
Next, notice that the size of the subproblems decreases by a factor ofb with each level of recursion, and therefore reaches the basecaseafterlogb n levels. This is the height of the recursion tree.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 17/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Proof of Master Theorem
Assume thatn is a power ofb. This will not influence the final boundin any important way:n is at most a multiplicative factor ofb awayfrom some power ofb.
Next, notice that the size of the subproblems decreases by a factor ofb with each level of recursion, and therefore reaches the basecaseafterlogb n levels. This is the height of the recursion tree.
The branching factor of the recursion tree isa, so thekth level of thetree is made up ofak subproblems, each ofsizen/bk.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 17/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Proof of Master Theorem
Assume thatn is a power ofb. This will not influence the final boundin any important way:n is at most a multiplicative factor ofb awayfrom some power ofb.
Next, notice that the size of the subproblems decreases by a factor ofb with each level of recursion, and therefore reaches the basecaseafterlogb n levels. This is the height of the recursion tree.
The branching factor of the recursion tree isa, so thekth level of thetree is made up ofak subproblems, each ofsizen/bk.
The total work done at this level is
ak × O( n
bk
)d= O(nd)×
( abd
)k.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 17/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Proof of Master Theorem
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 18/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Proof of Master Theorem
The total work done islogb n∑
k=0
(
ak × O( n
bk
)d)
=
lognb
∑
k=0
(
O(nd)×( a
bd
)k)
.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 18/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Proof of Master Theorem
The total work done islogb n∑
k=0
(
ak × O( n
bk
)d)
=
lognb
∑
k=0
(
O(nd)×( a
bd
)k)
.
It’s the sum of a geometric series withratio a/bd .
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 18/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Proof of Master Theorem
The total work done islogb n∑
k=0
(
ak × O( n
bk
)d)
=
lognb
∑
k=0
(
O(nd)×( a
bd
)k)
.
It’s the sum of a geometric series withratio a/bd .
⊲ The ratio is less than 1. Then the series is decreasing, and its sumis just given by its first term,O(nd).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 18/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Proof of Master Theorem
The total work done islogb n∑
k=0
(
ak × O( n
bk
)d)
=
lognb
∑
k=0
(
O(nd)×( a
bd
)k)
.
It’s the sum of a geometric series withratio a/bd .
⊲ The ratio is less than 1. Then the series is decreasing, and its sumis just given by its first term,O(nd).
⊲ The ratio is greater than 1. The series is increasing and its sum isgiven by its last term,O(nlogb a):
nd( a
bd
)logb n= nd
(
alogb n
(blogb n)d
)
= alogb n = a(loga n)(logb a) = nlogb a.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 18/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Proof of Master Theorem
The total work done islogb n∑
k=0
(
ak × O( n
bk
)d)
=
lognb
∑
k=0
(
O(nd)×( a
bd
)k)
.
It’s the sum of a geometric series withratio a/bd .
⊲ The ratio is less than 1. Then the series is decreasing, and its sumis just given by its first term,O(nd).
⊲ The ratio is greater than 1. The series is increasing and its sum isgiven by its last term,O(nlogb a):
nd( a
bd
)logb n= nd
(
alogb n
(blogb n)d
)
= alogb n = a(loga n)(logb a) = nlogb a.
⊲ The ratio is exactly 1. In this case allO(logn) terms of the seriesare equal toO(nd).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 18/73
Divide-and-ConquerApplications
Sorting Networks
Basic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
Proof of Master Theorem
The total work done islogb n∑
k=0
(
ak × O( n
bk
)d)
=
lognb
∑
k=0
(
O(nd)×( a
bd
)k)
.
It’s the sum of a geometric series withratio a/bd .
⊲ The ratio is less than 1. Then the series is decreasing, and its sumis just given by its first term,O(nd).
⊲ The ratio is greater than 1. The series is increasing and its sum isgiven by its last term,O(nlogb a):
nd( a
bd
)logb n= nd
(
alogb n
(blogb n)d
)
= alogb n = a(loga n)(logb a) = nlogb a.
⊲ The ratio is exactly 1. In this case allO(logn) terms of the seriesare equal toO(nd).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 18/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Outline
1 Divide-and-ConquerBasic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
2 ApplicationsSearch and SortMedianMatrix Multiplication
3 Sorting NetworksComparison NetworksZero-One PrincipleConstruction of a Sorting Network
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 19/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
BinarySearch Algorithm
BINARY SEARCH(A[1 . . . n])
Input: An arrayA[1..n] in nondecreasing order and anx.Output: j if x = A[j], 1≤ j ≤ n, and 0 otherwise.
1: low← 1; high← n; j← 0;2: while low ≤ high and j = 03: mid ← ⌊(low + high)/2⌋;4: if x = A[mid] then j← mid break;5: else if x < A[mid] then high← mid − 1;6: else low← mid + 1;7: end while8: return j
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 20/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The Time Analysis
To find a keyx in A[1, · · · , n] in sorted order, we first comparex withA[n/2], and depending on the result we recurse either on the first halfof the arrayA[1, · · · , n/2− 1], or on the second halfA[n/2, · · · , n].
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 21/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The Time Analysis
To find a keyx in A[1, · · · , n] in sorted order, we first comparex withA[n/2], and depending on the result we recurse either on the first halfof the arrayA[1, · · · , n/2− 1], or on the second halfA[n/2, · · · , n].
The recurrence function is
T(n) = T(⌈n
2
⌉)
+ O(1),
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 21/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The Time Analysis
To find a keyx in A[1, · · · , n] in sorted order, we first comparex withA[n/2], and depending on the result we recurse either on the first halfof the arrayA[1, · · · , n/2− 1], or on the second halfA[n/2, · · · , n].
The recurrence function is
T(n) = T(⌈n
2
⌉)
+ O(1),
By Master Theorem,a = 1, b = 2, d = 0, and thus the running timeshould beO(logn).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 21/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
MergeSort Algorithm
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 22/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
MergeSort Algorithm
MERGESORT(a[1 . . . n])
Input: an array of numbersa[1 . . . n]Output: A sorted version of this array
1: if n > 1 then2: returnMERGE(MERGESORT(a[1 . . . ⌊n/2⌋]),3: MERGESORT(a[⌊n/2⌋+ 1 . . . n⌋]),4: else returna.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 22/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
MergeSort Algorithm (Cont.)
MERGE(x[1 . . . k], y[1 . . . ℓ])
Input: two sorted arraysx andyOutput: A sorted version of the union ofx andy
1: if k = 0 then returny[1 . . . ℓ]2: if ℓ = 0 then returnx[1 . . . k]3: if x[1] ≤ y[1] then4: returnx[1]MERGE(x[2 . . . k], y[1 . . . ℓ])5: else returny[1]MERGE(x[1 . . . k], y[2 . . . ℓ]).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 23/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The Time Analysis
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 24/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The Time Analysis
The recurrence relation:
T(n) = 2T(n/2) + O(n);
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 24/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The Time Analysis
The recurrence relation:
T(n) = 2T(n/2) + O(n);
By Master TheoremT(n) = O(n logn).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 24/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
An n logn Lower Bound for Sorting
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 25/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
An n logn Lower Bound for Sorting
Sorting algorithms can be depicted astrees.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 25/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
An n logn Lower Bound for Sorting
Sorting algorithms can be depicted astrees.
Thedepth of the tree – the number of comparisons on the longestpath from root to leaf, is exactly the worst-case time complexity of thealgorithm.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 25/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
An n logn Lower Bound for Sorting
Sorting algorithms can be depicted astrees.
Thedepth of the tree – the number of comparisons on the longestpath from root to leaf, is exactly the worst-case time complexity of thealgorithm.
Consider any such tree that sorts an array ofn elements.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 25/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
An n logn Lower Bound for Sorting
Sorting algorithms can be depicted astrees.
Thedepth of the tree – the number of comparisons on the longestpath from root to leaf, is exactly the worst-case time complexity of thealgorithm.
Consider any such tree that sorts an array ofn elements. Each of itsleaves is labeled by apermutationof 1,2, . . . , n.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 25/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
An n logn Lower Bound for Sorting
Sorting algorithms can be depicted astrees.
Thedepth of the tree – the number of comparisons on the longestpath from root to leaf, is exactly the worst-case time complexity of thealgorithm.
Consider any such tree that sorts an array ofn elements. Each of itsleaves is labeled by apermutationof 1,2, . . . , n.
every permutation must appear as the label of a leaf.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 25/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
An n logn Lower Bound for Sorting
Sorting algorithms can be depicted astrees.
Thedepth of the tree – the number of comparisons on the longestpath from root to leaf, is exactly the worst-case time complexity of thealgorithm.
Consider any such tree that sorts an array ofn elements. Each of itsleaves is labeled by apermutationof 1,2, . . . , n.
every permutation must appear as the label of a leaf.
This is a binary tree withn! leaves.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 25/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
An n logn Lower Bound for Sorting
Sorting algorithms can be depicted astrees.
Thedepth of the tree – the number of comparisons on the longestpath from root to leaf, is exactly the worst-case time complexity of thealgorithm.
Consider any such tree that sorts an array ofn elements. Each of itsleaves is labeled by apermutationof 1,2, . . . , n.
every permutation must appear as the label of a leaf.
This is a binary tree withn! leaves. Thus, the depth of our tree – andthe complexity of our algorithm – must be at least
log(n!) ≈ log(
√
π (2n + 1/3) · nn · e−n)
= Ω(n logn),
where we useStirling’s formula.X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 25/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
A Sorting Permutation Tree
An example sorts fora1, a2, a3:
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 26/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Outline
1 Divide-and-ConquerBasic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
2 ApplicationsSearch and SortMedianMatrix Multiplication
3 Sorting NetworksComparison NetworksZero-One PrincipleConstruction of a Sorting Network
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 27/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Median
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 28/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Median
Themedian of a list of numbers is its 50th percentile: half thenumbers are bigger than it, and half are smaller.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 28/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Median
Themedian of a list of numbers is its 50th percentile: half thenumbers are bigger than it, and half are smaller.
If the list haseven length, there are two choices for what the middleelement could be, in which case we pick the smaller of the two,say.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 28/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Median
Themedian of a list of numbers is its 50th percentile: half thenumbers are bigger than it, and half are smaller.
If the list haseven length, there are two choices for what the middleelement could be, in which case we pick the smaller of the two,say.
The purpose of the median is to summarize a set of numbers by asingle, typical value.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 28/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Median
Themedian of a list of numbers is its 50th percentile: half thenumbers are bigger than it, and half are smaller.
If the list haseven length, there are two choices for what the middleelement could be, in which case we pick the smaller of the two,say.
The purpose of the median is to summarize a set of numbers by asingle, typical value.
Computing the median ofn numbers is easy: just sort them. Thedrawback is that this takesO(n logn) time, whereas we would ideallylike somethinglinear.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 28/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Median
Themedian of a list of numbers is its 50th percentile: half thenumbers are bigger than it, and half are smaller.
If the list haseven length, there are two choices for what the middleelement could be, in which case we pick the smaller of the two,say.
The purpose of the median is to summarize a set of numbers by asingle, typical value.
Computing the median ofn numbers is easy: just sort them. Thedrawback is that this takesO(n logn) time, whereas we would ideallylike somethinglinear.
We have reason to be hopeful, because sorting is doing far more workthan we really need – we just want the middle element and don’tcareabout the relative ordering of the rest of them.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 28/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Selection
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 29/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Selection
Input: A list of numbersS; an integerK.Output: Thekth smallest element ofS.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 29/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
A randomized divide-and-conquer algorithm for selection
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 30/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
A randomized divide-and-conquer algorithm for selection
For any numberv, imagine splitting listS into three categories:
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 30/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
A randomized divide-and-conquer algorithm for selection
For any numberv, imagine splitting listS into three categories:
elements smaller thanv, i.e.,SL;
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 30/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
A randomized divide-and-conquer algorithm for selection
For any numberv, imagine splitting listS into three categories:
elements smaller thanv, i.e.,SL;
those equal tov, i.e.,Sv (there might be duplicates);
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 30/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
A randomized divide-and-conquer algorithm for selection
For any numberv, imagine splitting listS into three categories:
elements smaller thanv, i.e.,SL;
those equal tov, i.e.,Sv (there might be duplicates);
and those greater thanv, i.e.,SR respectively.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 30/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
A randomized divide-and-conquer algorithm for selection
For any numberv, imagine splitting listS into three categories:
elements smaller thanv, i.e.,SL;
those equal tov, i.e.,Sv (there might be duplicates);
and those greater thanv, i.e.,SR respectively.
selection(S, k) =
selection(SL, k) if k ≤ |SL|
v if |SL| < k ≤ |SL|+ |Sv|
selection(SR, k − |SL| − |Sv|) if k > |SL|+ |Sv|.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 30/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
How to choosev?
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 31/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
How to choosev?
It should be picked quickly, and it should shrink the arraysubstantially, the ideal situation being
|SL|, |SR| ≈|S|2.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 31/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
How to choosev?
It should be picked quickly, and it should shrink the arraysubstantially, the ideal situation being
|SL|, |SR| ≈|S|2.
If we could always guarantee this situation, we would get a runningtime of
T(n) = T(n/2) + O(n) = O(n).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 31/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
How to choosev?
It should be picked quickly, and it should shrink the arraysubstantially, the ideal situation being
|SL|, |SR| ≈|S|2.
If we could always guarantee this situation, we would get a runningtime of
T(n) = T(n/2) + O(n) = O(n).
But this requires pickingv to be the median, which is our ultimategoal!
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 31/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
How to choosev?
It should be picked quickly, and it should shrink the arraysubstantially, the ideal situation being
|SL|, |SR| ≈|S|2.
If we could always guarantee this situation, we would get a runningtime of
T(n) = T(n/2) + O(n) = O(n).
But this requires pickingv to be the median, which is our ultimategoal!
Instead, we follow a much simpler alternative: we pickv randomlyfrom S.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 31/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
How to choosev? (cont’d)
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 32/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
How to choosev? (cont’d)
Worst-case scenario would force our selection algorithm to perform
n + (n− 1) + (n− 2) + · · ·+n2= Θ(n2)
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 32/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
How to choosev? (cont’d)
Worst-case scenario would force our selection algorithm to perform
n + (n− 1) + (n− 2) + · · ·+n2= Θ(n2)
Best-case scenario:O(n).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 32/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
How to choosev? (cont’d)
Worst-case scenario would force our selection algorithm to perform
n + (n− 1) + (n− 2) + · · ·+n2= Θ(n2)
Best-case scenario:O(n).
Where, in this spectrum fromO(n) to Θ(n2), does the averagerunning time lie?
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 32/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
How to choosev? (cont’d)
Worst-case scenario would force our selection algorithm to perform
n + (n− 1) + (n− 2) + · · ·+n2= Θ(n2)
Best-case scenario:O(n).
Where, in this spectrum fromO(n) to Θ(n2), does the averagerunning time lie? Fortunately, it lies very close to the best-case time.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 32/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The efficiency analysis
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 33/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The efficiency analysis
v is goodif it lies within the 25th to 75th percentile of the array thatitis chosen from.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 33/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The efficiency analysis
v is goodif it lies within the 25th to 75th percentile of the array thatitis chosen from.
A randomly chosenv has a50% chance of being good,
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 33/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The efficiency analysis
v is goodif it lies within the 25th to 75th percentile of the array thatitis chosen from.
A randomly chosenv has a50% chance of being good,
Lemma: On average afair coin needs to be tossed two times before a“heads” is seen.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 33/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The efficiency analysis
v is goodif it lies within the 25th to 75th percentile of the array thatitis chosen from.
A randomly chosenv has a50% chance of being good,
Lemma: On average afair coin needs to be tossed two times before a“heads” is seen.
Proof: E := expected number of tosses before head is seen.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 33/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The efficiency analysis
v is goodif it lies within the 25th to 75th percentile of the array thatitis chosen from.
A randomly chosenv has a50% chance of being good,
Lemma: On average afair coin needs to be tossed two times before a“heads” is seen.
Proof: E := expected number of tosses before head is seen.We need at least one toss, and it’s heads, we’re done.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 33/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The efficiency analysis
v is goodif it lies within the 25th to 75th percentile of the array thatitis chosen from.
A randomly chosenv has a50% chance of being good,
Lemma: On average afair coin needs to be tossed two times before a“heads” is seen.
Proof: E := expected number of tosses before head is seen.We need at least one toss, and it’s heads, we’re done.If it’s tail (with probability 1/2), we need to repeat.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 33/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The efficiency analysis
v is goodif it lies within the 25th to 75th percentile of the array thatitis chosen from.
A randomly chosenv has a50% chance of being good,
Lemma: On average afair coin needs to be tossed two times before a“heads” is seen.
Proof: E := expected number of tosses before head is seen.We need at least one toss, and it’s heads, we’re done.If it’s tail (with probability 1/2), we need to repeat. Hence
E = 1+12
E,
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 33/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The efficiency analysis
v is goodif it lies within the 25th to 75th percentile of the array thatitis chosen from.
A randomly chosenv has a50% chance of being good,
Lemma: On average afair coin needs to be tossed two times before a“heads” is seen.
Proof: E := expected number of tosses before head is seen.We need at least one toss, and it’s heads, we’re done.If it’s tail (with probability 1/2), we need to repeat. Hence
E = 1+12
E,
whose solution isE = 2
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 33/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The efficiency analysis (cont’d)
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 34/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The efficiency analysis (cont’d)
Let T(n) be theexpected running time on an array of sizen, we get
T(n) ≤ T(3n/4) + O(n) = O(n).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 34/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Outline
1 Divide-and-ConquerBasic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
2 ApplicationsSearch and SortMedianMatrix Multiplication
3 Sorting NetworksComparison NetworksZero-One PrincipleConstruction of a Sorting Network
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 35/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The product of twon× n matricesX andY is a thirdn× n matrixZ = XY,
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 36/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The product of twon× n matricesX andY is a thirdn× n matrixZ = XY, with (i, j)th entry
Zij =
n∑
k=1
XikYkj
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 36/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The product of twon× n matricesX andY is a thirdn× n matrixZ = XY, with (i, j)th entry
Zij =
n∑
k=1
XikYkj
That is,Zij is thedot productof the ith row of X with the jth columnof Y.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 36/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The product of twon× n matricesX andY is a thirdn× n matrixZ = XY, with (i, j)th entry
Zij =
n∑
k=1
XikYkj
That is,Zij is thedot productof the ith row of X with the jth columnof Y.
In general,XY is not the same asYX; matrix multiplication is notcommutative.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 36/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
The product of twon× n matricesX andY is a thirdn× n matrixZ = XY, with (i, j)th entry
Zij =
n∑
k=1
XikYkj
That is,Zij is thedot productof the ith row of X with the jth columnof Y.
In general,XY is not the same asYX; matrix multiplication is notcommutative.
The preceding formula implies anO(n3) algorithm for matrixmultiplication.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 36/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Volker Strassen
Volker Strassen (1936 – )
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 37/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Volker Strassen
Volker Strassen (1936 – )
In 1969, the German mathematicianVolker Strassen announced asurprisingO(n2.81) algorithm.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 37/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Divide and conquer
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 38/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Divide and conquer
X =
[
A BC D
]
, Y =
[
E FG H
]
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 38/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Divide and conquer
X =
[
A BC D
]
, Y =
[
E FG H
]
Then
XY =
[
A BC D
] [
E FG H
]
=
[
AE + BG AF + BHCE + DG CF + DH
]
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 38/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Divide and conquer
X =
[
A BC D
]
, Y =
[
E FG H
]
Then
XY =
[
A BC D
] [
E FG H
]
=
[
AE + BG AF + BHCE + DG CF + DH
]
To compute the size-n productXY, recursively compute eight size-n/2productsAE, BG, AF, BH, CE, DG, CF, DH and then do someO(n2)-time addition.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 38/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Divide and conquer
X =
[
A BC D
]
, Y =
[
E FG H
]
Then
XY =
[
A BC D
] [
E FG H
]
=
[
AE + BG AF + BHCE + DG CF + DH
]
To compute the size-n productXY, recursively compute eight size-n/2productsAE, BG, AF, BH, CE, DG, CF, DH and then do someO(n2)-time addition.
The recurrence is
T(n) = 8T(n/2) + O(n2)
with solutionO(n3).X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 38/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Strassen’s trick
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 39/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Strassen’s trick
XY =
[
P5 + P4− P2 + P6 P1 + P2
P3 + P4 P1 + P5 = P3− P7
]
where
P1 = A(F − H) P5 = (A + D)(E + H)P2 = (A + B)H P6 = (B− D)(G + H)P3 = (C + D)E P7 = (A− C)(E + F)P4 = D(G− E)
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 39/73
Divide-and-ConquerApplications
Sorting Networks
Search and SortMedianMatrix Multiplication
Strassen’s trick
XY =
[
P5 + P4− P2 + P6 P1 + P2
P3 + P4 P1 + P5 = P3− P7
]
where
P1 = A(F − H) P5 = (A + D)(E + H)P2 = (A + B)H P6 = (B− D)(G + H)P3 = (C + D)E P7 = (A− C)(E + F)P4 = D(G− E)
The recurrence is
T(n) = 7T(n/2) + O(n2)
with solutionO(nlog2 7) ≈ O(n2.81).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 39/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Outline
1 Divide-and-ConquerBasic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
2 ApplicationsSearch and SortMedianMatrix Multiplication
3 Sorting NetworksComparison NetworksZero-One PrincipleConstruction of a Sorting Network
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 40/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Introduction
Previously, we examined sorting algorithms forserial computers(random-access machines, RAM’s) that allow only one operation tobe executed at a time.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 41/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Introduction
Previously, we examined sorting algorithms forserial computers(random-access machines, RAM’s) that allow only one operation tobe executed at a time.
In this section, we investigate sorting algorithms based onacomparison-networkmodel of computation, in which manycomparison operations can be performed simultaneously.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 41/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Introduction
Previously, we examined sorting algorithms forserial computers(random-access machines, RAM’s) that allow only one operation tobe executed at a time.
In this section, we investigate sorting algorithms based onacomparison-networkmodel of computation, in which manycomparison operations can be performed simultaneously.
Comparison NetworkVS RAM’s
⊲ Comparison network can only perform comparisons. (Cannotdeal with Counting Sort etc)
⊲ Comparison network run parallel operations.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 41/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Definition
A comparison network is composed solely ofwiresandcomparators.
A comparator is a device with two inputs,x andy, and two outputs,x′
andy′, that performs the following function:
x′ = minx, y, y′ = maxx, y.
Each comparator operates inO(1) time.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 42/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Wire
A wire transmits a value from place to place.
⊲ Connect the output of one comparator to the input of another;
⊲ The network input wires or output wires.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 43/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Wire
A wire transmits a value from place to place.
⊲ Connect the output of one comparator to the input of another;
⊲ The network input wires or output wires.
Assume a comparison networks containsn input wires< a1, a2, · · · , an >, through which the values to be sorted enter thenetwork, andn output wires< b1, b2, · · · , bn > , which produce theresults computed by the network.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 43/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Wire
A wire transmits a value from place to place.
⊲ Connect the output of one comparator to the input of another;
⊲ The network input wires or output wires.
Assume a comparison networks containsn input wires< a1, a2, · · · , an >, through which the values to be sorted enter thenetwork, andn output wires< b1, b2, · · · , bn > , which produce theresults computed by the network.
Draw a comparison network onn inputs as a collection ofn horizontallines with comparators stretched vertically.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 43/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
An Example
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 44/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
An Example
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 44/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
An Example
Data move from left toright.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 44/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
An Example
Data move from left toright.
Interconnections must beacyclic.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 44/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
An Example
Data move from left toright.
Interconnections must beacyclic.
If a comparator has twoinput wires withdepthsdx
anddy, then its output wirehave depthmaxdx, dy+ 1. (Initiallyis 0)
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 44/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Sorting Network
A sorting networkis a comparison network for which the outputsequence is monotonically increasing (b1 ≤ b2 ≤ · · · ≤ bn) for everyinput sequence.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 45/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Sorting Network
A sorting networkis a comparison network for which the outputsequence is monotonically increasing (b1 ≤ b2 ≤ · · · ≤ bn) for everyinput sequence.
We are discussing a family of comparison networks accordingto theinput size.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 45/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Outline
1 Divide-and-ConquerBasic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
2 ApplicationsSearch and SortMedianMatrix Multiplication
3 Sorting NetworksComparison NetworksZero-One PrincipleConstruction of a Sorting Network
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 46/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Zero-One Principle
Thezero-one principlesays that if a sorting network works correctlywhen each input is drawn from the set 0,1, then it works correctlyon arbitrary input numbers (e.g., integers, reals, or any linearlyordered set).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 47/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Zero-One Principle
Thezero-one principlesays that if a sorting network works correctlywhen each input is drawn from the set 0,1, then it works correctlyon arbitrary input numbers (e.g., integers, reals, or any linearlyordered set).
This principle allow us to focus on the operations for input sequencesconsisting solely of 0’s and 1’s.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 47/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Zero-One Principle
Thezero-one principlesays that if a sorting network works correctlywhen each input is drawn from the set 0,1, then it works correctlyon arbitrary input numbers (e.g., integers, reals, or any linearlyordered set).
This principle allow us to focus on the operations for input sequencesconsisting solely of 0’s and 1’s.
Once we have constructed a sorting network and proved that itcansort all zero-one sequence, we shall appeal to 0-1 principleto proveits correctness on arbitrary values.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 47/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Zero-One Principle
Thezero-one principlesays that if a sorting network works correctlywhen each input is drawn from the set 0,1, then it works correctlyon arbitrary input numbers (e.g., integers, reals, or any linearlyordered set).
This principle allow us to focus on the operations for input sequencesconsisting solely of 0’s and 1’s.
Once we have constructed a sorting network and proved that itcansort all zero-one sequence, we shall appeal to 0-1 principleto proveits correctness on arbitrary values.
Note: the proof of 0-1 principle relies on the notion of monotonicallyincreasing function.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 47/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Lemma
Lemma: If a comparison network transforms the input sequencea =< a1, a2, · · · , an > into the output sequenceb =< b1, b2, · · · , bn >, then for any monotonically increasingfunction f , the network transforms the input sequencef (a) =< f (a1), f (a2), · · · , f (an) > into the output sequencef (b) =< f (b1), f (b2), · · · , f (bn) >.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 48/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Proof (by Induction)
Consider a comparator whose input values arex andy. The upperoutput is minx, y while the lower output is maxx, y.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 49/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Proof (by Induction)
Consider a comparator whose input values arex andy. The upperoutput is minx, y while the lower output is maxx, y.
If we apply f (x) andf (y) as the inputs, the operation of thecomparator yields the value of upper minf (x), f (y) and lowermaxf (x), f (y).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 49/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Proof (by Induction)
Consider a comparator whose input values arex andy. The upperoutput is minx, y while the lower output is maxx, y.
If we apply f (x) andf (y) as the inputs, the operation of thecomparator yields the value of upper minf (x), f (y) and lowermaxf (x), f (y).
Sincef is monotonically increasing,x ≤ y implies f (x) ≤ f (y).Thus we have
minf (x), f (y) = f (minx, y),
maxf (x), f (y) = f (maxx, y),
which completes the proof of the claim as the base case.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 49/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Proof (Continued)
We use induction on the depth of each wire in a general comparisonnetwork to prove a stronger result than the statement of the lemma:
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 50/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Proof (Continued)
We use induction on the depth of each wire in a general comparisonnetwork to prove a stronger result than the statement of the lemma:
If a wire assumes the valueai when the input sequence isa, then itassumes the valuef (ai) when the input sequence isf (a).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 50/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Proof (Continued)
We use induction on the depth of each wire in a general comparisonnetwork to prove a stronger result than the statement of the lemma:
If a wire assumes the valueai when the input sequence isa, then itassumes the valuef (ai) when the input sequence isf (a).
Since the output wires are included in this statement, proving it willprove the lemma.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 50/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Proof (Continued)
Basis: A wire at depth 0 is an input wireai. Whenf (a) is applied tothe network, the input wire carriesf (ai).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 51/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Proof (Continued)
Basis: A wire at depth 0 is an input wireai. Whenf (a) is applied tothe network, the input wire carriesf (ai).
Induction: A wire at depthd ≥ 1 is the output of a comparator atdepthd, and the input wires to this comparator are at a depth strictlyless thand. By inductive hypothesis, if the input wires carry valuesai
andaj with input sequencea, then they carryf (ai) andf (aj) withinput sequencef (a).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 51/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Proof (Continued)
Basis: A wire at depth 0 is an input wireai. Whenf (a) is applied tothe network, the input wire carriesf (ai).
Induction: A wire at depthd ≥ 1 is the output of a comparator atdepthd, and the input wires to this comparator are at a depth strictlyless thand. By inductive hypothesis, if the input wires carry valuesai
andaj with input sequencea, then they carryf (ai) andf (aj) withinput sequencef (a).
By previous claim, the output wires of this comparator then carryf (minai, aj) andf (maxai, aj). Since the carry minai, aj andmaxai, aj when the input sequence isa, the lemma is proved.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 51/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
An Example
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 52/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Zero-One Principle
Theorem: If a comparison network withn inputs sorts all 2n possiblesequences of 0’s and 1’s correctly, then it sorts all sequences ofarbitrary numbers correctly.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 53/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Zero-One Principle
Theorem: If a comparison network withn inputs sorts all 2n possiblesequences of 0’s and 1’s correctly, then it sorts all sequences ofarbitrary numbers correctly.
Proof: (Contradiction)Suppose there exists a sequence of arbitrarynumbers that the network does not correctly sort. That is , there existsan input sequence< a1, a2, · · · , an > containing elementsai andaj,such thatai < aj, but the network placesaj beforeai in the outputsequence.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 53/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Proof (Continued)
Define a monotonically increasing functionf as
f (x) =
0 if x ≤ ai,1 if x > ai.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 54/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Proof (Continued)
Define a monotonically increasing functionf as
f (x) =
0 if x ≤ ai,1 if x > ai.
Since the network placesaj beforeai, by previous lemma, it will placef (aj) beforef (ai) in the output sequence when< f (a1), f (a2), · · · , f (an) > is input.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 54/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Proof (Continued)
Define a monotonically increasing functionf as
f (x) =
0 if x ≤ ai,1 if x > ai.
Since the network placesaj beforeai, by previous lemma, it will placef (aj) beforef (ai) in the output sequence when< f (a1), f (a2), · · · , f (an) > is input.
However, sincef (aj) = 1 andf (ai) = 0, the network fails to sort thezero-one sequence< f (a1), f (a2), · · · , f (an) > correctly.
A contradiction!
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 54/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Outline
1 Divide-and-ConquerBasic TechniqueAn Introductory Example: MultiplicationRecurrence Relations
2 ApplicationsSearch and SortMedianMatrix Multiplication
3 Sorting NetworksComparison NetworksZero-One PrincipleConstruction of a Sorting Network
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 55/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Construction of a Sorting Network
To construct a sorting network, we need three steps:
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 56/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Construction of a Sorting Network
To construct a sorting network, we need three steps:
Step 1: Construct a Bitonic Sorter⇒ to sort bitonic sequence.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 56/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Construction of a Sorting Network
To construct a sorting network, we need three steps:
Step 1: Construct a Bitonic Sorter⇒ to sort bitonic sequence.
Step 2: Construct a Merger⇒ to merge two sorted sequence.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 56/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Construction of a Sorting Network
To construct a sorting network, we need three steps:
Step 1: Construct a Bitonic Sorter⇒ to sort bitonic sequence.
Step 2: Construct a Merger⇒ to merge two sorted sequence.
Step 3: Construct a Sorter⇒ to sort an arbitrary sequence.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 56/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Step 1: Construct a Bitonic Sorter
We start fromBitonic sequence.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 57/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Step 1: Construct a Bitonic Sorter
We start fromBitonic sequence.
A Bitonic Sequence is a sequence that monotonically increases andthen monotonically decreases, or can be circularly shiftedto becomemonotonically increasing and then monotonically decreasing.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 57/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Step 1: Construct a Bitonic Sorter
We start fromBitonic sequence.
A Bitonic Sequence is a sequence that monotonically increases andthen monotonically decreases, or can be circularly shiftedto becomemonotonically increasing and then monotonically decreasing.
Examples: <1,4,6,8,3,2>, <6,9,4,2,3,5>, <9,8,3,2,4,6>
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 57/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Step 1: Construct a Bitonic Sorter
We start fromBitonic sequence.
A Bitonic Sequence is a sequence that monotonically increases andthen monotonically decreases, or can be circularly shiftedto becomemonotonically increasing and then monotonically decreasing.
Examples: <1,4,6,8,3,2>, <6,9,4,2,3,5>, <9,8,3,2,4,6>
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 57/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Step 1: Construct a Bitonic Sorter
We start fromBitonic sequence.
A Bitonic Sequence is a sequence that monotonically increases andthen monotonically decreases, or can be circularly shiftedto becomemonotonically increasing and then monotonically decreasing.
Examples: <1,4,6,8,3,2>, <6,9,4,2,3,5>, <9,8,3,2,4,6>
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 57/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Half-Cleaner
A half-cleaneris a comparison network of depth 1, in which input line
i is compared with linei +n2
for i = 1,2, · · · ,n2
(assumen is even).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 58/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Half-Cleaner
A half-cleaneris a comparison network of depth 1, in which input line
i is compared with linei +n2
for i = 1,2, · · · ,n2
(assumen is even).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 58/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Half-Cleaner (2)
When a bitonic sequence of 0’s and 1’s is applied as input to ahalf-cleaner, it produces an output sequence with smaller values in thetop-half, and larger values in the bottom-half. Both halvesare bitonic.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 59/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Half-Cleaner (2)
When a bitonic sequence of 0’s and 1’s is applied as input to ahalf-cleaner, it produces an output sequence with smaller values in thetop-half, and larger values in the bottom-half. Both halvesare bitonic.
In fact, at least one of the halves isclean, say, consists of either all 0’sor all 1’s.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 59/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Half-Cleaner Lemma
Lemma: If the input to a half-cleaner is a bitonic sequence of 0’s and1’s, then the output satisfies the following properties:
⊲ both the top half and the bottom half are bitonic;
⊲ every element in the top half is at least as small as every elementof the bottom half, and at least one half is clean.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 60/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Proof
The comparison networkHALF-CLEANER[n] compares inputsi andi + n/2 for i = 1,2, · · · , n/2. Without loss of generality, suppose thatthe input is of the form 00· · · 011· · · 100· · · 0 (the situation of11· · · 100· · · 011· · · 1 are symmetric).
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 61/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Proof
The comparison networkHALF-CLEANER[n] compares inputsi andi + n/2 for i = 1,2, · · · , n/2. Without loss of generality, suppose thatthe input is of the form 00· · · 011· · · 100· · · 0 (the situation of11· · · 100· · · 011· · · 1 are symmetric).
There are three possible cases depending upon the block ofconsecutive 0’s and 1’s in which the midpointn/2 falls, and one ofthese cases is further split into two cases. In each of the cases, thelemma holds.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 61/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Case Analysis
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 62/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Case Analysis (Cont.)
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 63/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
The Bitonic Sorter
By recursively combing half-cleaners, we can build abitonic sorter,which is a network that sorts bitonic sequences.
Half-
Cleaner[n]
Bitonic-
Sorter[n/2]
Bitonic-
Sorter[n/2]
Bitonic-Sorter[n]
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 64/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
An Example ofn = 8
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 65/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
DepthD(n)
The depthD(n) of BITONIC-SORTER[n] is given by the recurrence
D(n) =
0 if n = 1;D(n/2) + 1 if n = 2k andk ≥ 1,
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 66/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
DepthD(n)
The depthD(n) of BITONIC-SORTER[n] is given by the recurrence
D(n) =
0 if n = 1;D(n/2) + 1 if n = 2k andk ≥ 1,
Easy to see,D(n) = ln n.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 66/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
DepthD(n)
The depthD(n) of BITONIC-SORTER[n] is given by the recurrence
D(n) =
0 if n = 1;D(n/2) + 1 if n = 2k andk ≥ 1,
Easy to see,D(n) = ln n.
Thus, a zero-one bitonic sequence can be sorted byBITONIC-SORTER[n], which has a depth of lnn.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 66/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
DepthD(n)
The depthD(n) of BITONIC-SORTER[n] is given by the recurrence
D(n) =
0 if n = 1;D(n/2) + 1 if n = 2k andk ≥ 1,
Easy to see,D(n) = ln n.
Thus, a zero-one bitonic sequence can be sorted byBITONIC-SORTER[n], which has a depth of lnn.
By zero-one principle, any bitonic sequence of arbitrary numbers canbe sorted by this network.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 66/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Step 2: Construct a Merger
Merging Network can merge two sorted input sequences into onesorted output sequence.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 67/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Step 2: Construct a Merger
Merging Network can merge two sorted input sequences into onesorted output sequence.
Given two sorted sequences, if we reverse the order of the secondsequence and then concatenate the two sequences, the resultingsequence is bitonic.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 67/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Step 2: Construct a Merger
Merging Network can merge two sorted input sequences into onesorted output sequence.
Given two sorted sequences, if we reverse the order of the secondsequence and then concatenate the two sequences, the resultingsequence is bitonic.
For instance:
X = 00000111;
Y = 00001111;
YR = 11110000;
X YR = 0000011111110000.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 67/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
MERGER[n]
Given two sorted sequences< a1, a2, · · · , an/2 > and< an/2+1, an/2+2, · · · , an >, we want the effect of bitonically sortingthe sequence< a1, a2, · · · , an/2, an, an−1, · · · , an/2+1 >.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 68/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
MERGER[n]
Given two sorted sequences< a1, a2, · · · , an/2 > and< an/2+1, an/2+2, · · · , an >, we want the effect of bitonically sortingthe sequence< a1, a2, · · · , an/2, an, an−1, · · · , an/2+1 >.
Since the first half-cleaner ofBITONIC-SORTER[n] compares inputsiwith n/2+ i, for i = 1,2, · · · , n/2, we make the first stage of themerging network compare inputsi andn− i + 1.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 68/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
MERGER[n]
Given two sorted sequences< a1, a2, · · · , an/2 > and< an/2+1, an/2+2, · · · , an >, we want the effect of bitonically sortingthe sequence< a1, a2, · · · , an/2, an, an−1, · · · , an/2+1 >.
Since the first half-cleaner ofBITONIC-SORTER[n] compares inputsiwith n/2+ i, for i = 1,2, · · · , n/2, we make the first stage of themerging network compare inputsi andn− i + 1.
The order of the outputs from the bottom of the first stage ofMERGER[n] are reversed compared with the order of outputs from anordinary half-cleaner.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 68/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Comparison betweenMERGER[n] andHALF-CLEANER[n]
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 69/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
MERGER[n]
Bitonic-
Sorter[n/2]
Bitonic-
Sorter[n/2]
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 70/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Step 3: Construct a Sorter
The sorting networkSORTER[n] are composed by two copies ofSORTER[n/2] and oneMERGER[n] recursively.
Merger[n]
Sorter[n/2]
Sorter[n/2]
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 71/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
An Example withn = 8
Sorter[n/2]
Merger[n]
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 72/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Performance Analysis
The depthD(n) of SORTER[n] is the depthD(n/2) of SORTER[n/2]plus the depth lnn of MERGER[n].
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 73/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Performance Analysis
The depthD(n) of SORTER[n] is the depthD(n/2) of SORTER[n/2]plus the depth lnn of MERGER[n].
Consequently, the depth ofSORTER[n] is given by the recurrence
D(n) =
0 if n = 1;D(n/2) + ln n if k ≥ 1.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 73/73
Divide-and-ConquerApplications
Sorting Networks
Comparison NetworksZero-One PrincipleConstruction of a Sorting Network
Performance Analysis
The depthD(n) of SORTER[n] is the depthD(n/2) of SORTER[n/2]plus the depth lnn of MERGER[n].
Consequently, the depth ofSORTER[n] is given by the recurrence
D(n) =
0 if n = 1;D(n/2) + ln n if k ≥ 1.
By Master’s Theorem, the solution isD(n) = Θ(ln2 n). Thus we cansortn numbers in parallel inO(ln2 n) time.
X033533-Algorithm@SJTU Xiaofeng Gao Divide and Conquer 73/73
top related