Assignment 1• Out: This morning. Due: 23 Aug 23:59• Grace period until 24 Aug 13:00. No submission will be

accepted after the grace period• Submission: • Written report (part A + some of part B) à A1-studentID.pdf• Source codes (just the source + test cases you use for your

analysis, no object files, binaries, etc.) à main: A1-studentID.cpp / appropriate extensions for C /

C++ / java• Zip all of them in a single .zip file à A1-studentID.zip (max

200MB)• Submit the .zip file through wattle before the due date• Please follow the name format à will speed up marking

Assignment 1• There will still be tutorial questions. But, part of

the tutorial time will be available for you to ask questions about assignment 1• Drop-in session: Friday, 16 August, 8am-10am

in CSIT N115/N116• If you want to download test cases using the lab

computer:• During tutorial• Tuesday 08:00-10:00 in CSIT N111• Thursday 10:00-12:00 in CSIT N111• Friday 08:00-10:00 in CSIT N115/116

Room & time have been reserved for this class

Assignment 1• Though redeemable, you should try to work on the

assignment to get the most out of this class. Also,• We aim to release the marks for assignment 1 on the

morning of 30 Aug (census date: 31 Aug)• If you get < 55 in assignment 1, it is an indication that you

will likely be in trouble in this class. • Last but not least: Do NOT cheat !!!• Issues? Contact the teaching staff(s):

Email comp_3600_6466@anu.edu.au OR Email algorithmsatanu@gmail.com OR via piazza OR before / after / during break in lecture / tutorial time

Last WeeküFinishing off Asymptotic AnalysisüInsertion Sort analysis üComparing Algorithms• Recurrence analysisüWhat is recurrence?üWhere is it used?üSubstitution Method: Solving recurrences

Mathematics Refresher• In the class website, schedule, the notes for

29/07/19 row (you need to login to wattle first in order to access these files):• [Lev] App.A à A quick summary of logs, summation of

a sequence of numbers, etc.. • [CLRS] App.A à Provide a bit more explanation on

summation of a sequence of numbers• You can also access the above files if you go to

wattle page of this class, and then to ReferencesStrongly suggest that you become conversant in

the materials of [Lev] App.A.

Today• Finishing off Recurrence analysis• Recursion Tree: Guessing the solution form• Master’s Theorem: Finding asymptotic bound of a recurrence• Randomized Algorithms + Probabilistic Analysis• A bit about Randomized Algorithm• A bit about Probabilistic Analysis

COMP3600/6466 – Algorithms Recurrence Analysis cont.

[CLRS sec. 4.4-4.5]

Hanna Kurniawati

https://cs.anu.edu.au/courses/comp3600/

How to guess the form of the solution?Recursion Tree• Recursion Tree is a tree where • Each node represents the cost of a single sub-

problem (somewhere in the recursive function invocation)• An edge from node a to node b means the sub-

problem represented by node a calls the sub-problem represented by node b

•We usually perform two sets of summations:• Sum costs within each level to compute the per-

level cost • Sum the per-level cost to compute the total cost

Recursion Tree Usage• Usually, the total cost in recursion tree

becomes the guess for substitution method• Therefore, a bit “sloppiness” is often fine. After

all, the substitution method will verify•We can also analyze exactly the total cost

of recursion tree, we can use it as direct proof of the solution. Usually a bit involved• Master’s method, which we will cover next, is

the result of such an analysis on several classes of functions

Example: Merge Sort𝑐𝑛

𝑐𝑛

𝑐𝑛

::

Cost per level

Total cost:𝑐𝑛 log 𝑛 + 1

Examples• 𝑇 𝑛 = 3𝑇 6

7 + Θ 𝑛9 with 𝑇 1 = 1• Recurrence tree:

Examples• Computing the tree depth:

14

;𝑛 = 1

14

;=1𝑛

𝑘 =−log 𝑛−log 4

= log> 𝑛

• Cost-at-level-𝑖 : 3@𝑐 AB

>BC

• Cost at the leaves level: 𝑐3DEF7 A. • Note that at the leaves level, the cost of each node (leaf) is 𝑇 1 , which is the initial case.

• In this particular example, the initial and the rest of the cases are the same. However, in general, the initial may or may not be the same as the rest of the case. Therefore, it’s better to separate the initial cases.

• Total cost-per-level:𝑇 𝑛 = ∑@IJ

(DEF7 A)MN 3@𝑐 AB

>BC+ 𝑐𝑛DEF7 O

= ∑@IJ(DEF7 A)MN O

NP

@𝑐𝑛9 + 𝑐𝑛DEF7 O

≤ ∑@IJR ONP

@𝑐𝑛9 + 𝑐𝑛DEF7 O

≤ SAB

NMTUTV+ 𝑐𝑛DEF7 O

≤ NPWNONO

𝑐𝑛9

= 𝑂(𝑛9)• Note that since the recurrence 𝑇 𝑛 is at least Θ(𝑛9),

the above result 𝑇 𝑛 = 𝑂 𝑛9 means that 𝑇 𝑛 =Θ(𝑛9).

Examples• 𝑇 𝑛 = 𝑇 6

U + 𝑇 B6U + 𝑂 𝑛

Today• Finishing off Recurrence analysisüRecursion Tree: Guessing the solution form• Master’s Theorem: Finding asymptotic bound of a recurrence• Randomized Algorithms + Probabilistic Analysis• A bit about Randomized Algorithm• A bit about Probabilistic Analysis

Master’s Theorem• For several classes of recurrence relation,

Master’s Theorem provides a “manual”• No need to derive the recurrence tree,

someone has done it and we can use it

Master’s Theorem (def.: [CLRS] pp. 94)• Suppose we want to solve 𝑇 𝑛 = 𝑎𝑇 6

Z + 𝑓 𝑛 , 𝑎 ≥ 1, 𝑏 > 1, A

_can be A

_or A

_• Then, 𝑇 𝑛 has the following asymptotic bounds

(highly dependent on 𝑓 𝑛 )1. If 𝑓 𝑛 = 𝑂 𝑛DEFZ bMc for some constant 𝜖 > 0, then

𝑇 𝑛 = Θ 𝑛DEFZ b

2. If 𝑓 𝑛 = Θ 𝑛DEFZ b , then 𝑇 𝑛 = Θ 𝑛DEFZ b log 𝑛3. If 𝑓 𝑛 = Ω 𝑛DEFZ bWc for some constant 𝜖 > 0 and if

𝑎𝑓 6Z ≤ 𝑐𝑓 𝑛 for some constant 𝑐 < 1 and 𝑛 > 𝑛J,

then 𝑇 𝑛 = Θ 𝑓 𝑛

Examples• Can we solve the following recurrences

using Master’s Theorem? If we can, what’s the solution?• 𝑇 𝑛 = 2𝑇 A

9+ 𝑛

Here, 𝑎 = 2, 𝑏 = 2, 𝑓 𝑛 = 𝑛. Hence, 𝑛DEFZ b = 𝑛.Therefore, case-2 of Master’s Theorem applies, and 𝑇 𝑛 = Θ(𝑛 log 𝑛).

Examples• Can we solve the following recurrences

using Master’s Theorem? If we can, what’s the solution?• 𝑇 𝑛 = 2𝑇 A

9+ 𝑛

• 𝑇 𝑛 = 3𝑇 A>+ 𝑛 log 𝑛

Examples• Can we solve the following recurrences

using Master’s Theorem? If we can, what’s the solution?• 𝑇 𝑛 = 2𝑇 A

9+ 𝑛

• 𝑇 𝑛 = 3𝑇 A>+ 𝑛 log 𝑛

• 𝑇 𝑛 = 2𝑇 A9+ 𝑛 log 𝑛

TodayüFinishing off Recurrence analysisüRecursion Tree: Guessing the solution formüMaster’s Theorem: Finding asymptotic bound of a recurrence• Randomized Algorithms + Probabilistic Analysis• A bit about Randomized Algorithm• A bit about Probabilistic Analysis

COMP3600/6466 – Algorithms Randomized Algorithms and

Probabilistic Analysis[CLRS sec. 5.2, 7.1, 7.3, 7.4]

Hanna Kurniawati

https://cs.anu.edu.au/courses/comp3600/

What is Randomized Algorithm?• Recall the definition of algorithms from lecture 1:• A well-defined procedure that transforms input to output • Randomized algorithm is an algorithm whose

behavior is determined not just by its input but also by random-numbers• Behavior can varies even for the same input

Random number

Randomized Algorithms

Input Output

Use of Randomization in Algorithms• Provide better scalability (running time & space)• Abundance of witnesses• Provide a compact representation of a large space

(population)• Random sampling• Foiling an adversary• Randomization adds unpredictability on the input—output

mapping. • Can be viewed as having a distribution on many deterministic

algorithms. • Avoid pathological input that causes worst case scenario for

the algorithm

Example: RandQuick Sort (Avoid pathological cases)

RandQuickSort(A, p, r)1. If p < r2. q = RandPartition(A, p, r)3. RandQuickSort(A, p, q-1)4. RandQuickSort(A, q+1, r)

RandPartition(A, p, r)1. i = random(p, r)2. Swap A[r] with A[i] 3. Return Partition(A, p, r)

Partition(A, p, r)1. x = A[r]2. i = p-13. For j = p to r-14. If A[j] ≤ x 5. i = i + 16. Swap A[i] with A[j]7. Swap A[i+1] with A[r]8. Return i+1

RandQuickSort Illustration• Select an element of the array to be a pivot,

uniformly at random• Split the array into 2: Smaller than the pivot and

larger than the pivot• Repeat the above two steps for each sub-array

RandQuickSort Illustration• Input:

• Select a pivot uniformly at random (RandPartition)

• Set the pivot to be at the end

20 5 10 8 7 15 12 3

20 5 10 8 7 15 12 3

20 5 3 8 7 15 12 10

p r

RandQuickSort Illustration• Split the array (Partition)• i: index of the last element of the first whose value is smaller

than the pivot• j: index of the element being examined at the moment

20 5 3 8 7 15 12 10

20 5 3 8 7 15 12 10

ji

p r

ji

5 20 3 8 7 15 12 10ji

RandQuickSort Illustration

⋮

5 20 3 8 7 15 12 10ji

5 3 20 8 7 15 12 10ji

5 3 8 7 20 15 12 10ji

5 3 8 7 10 15 12 20i

TodayüFinishing off Recurrence analysisüRecursion Tree: Guessing the solution formüMaster’s Theorem: Finding asymptotic bound of a recurrence• Randomized Algorithms + Probabilistic AnalysisüA bit about Randomized Algorithm• A bit about Probabilistic Analysis

Probabilistic Analysis• Analyze the behavior of algorithms when the

input is from a probability distribution • In general, the objective is to compute the expected

properties (e.g., average running time, average space complexity) of an algorithm

• Probabilistic analysis of algorithms can be applied to non-randomized (deterministic) algorithms• Of course, also used to analyze randomized

algorithms

Probability Refresher• Probabilistic Modelling Assumptions:• Experiments with random outcome• Which outcome is likely to happen is quantifiable• Probabilistic modelling consists of 3

components:• Sample space (S): Set of all possible outcomes.• Events (𝐄): Subsets of sample space.• Probability (P): Quantify how likely an event

occurs.

Probability Refresher• Probability: A function that maps events to real

numbers satisfying the following 3 axioms:• Non-negativity: 𝑃 A ≥ 0 for any event 𝐴 ∈ 𝐄• Normalization: 𝑃 𝑆 = 1• Additivity of finite / countably infinite events: 𝑃 ⋃@IN

A 𝐴@ = ∑@INA 𝑃 𝐴@ 𝑂𝑅𝑃 ⋃@IN

R 𝐴@ = ∑@INR 𝑃 𝐴@where 𝐴@ ∈ 𝐄 are disjoint / mutually exclusive events

Probability Refresher: Example• Flipping a fair coin, head (H) or tail (T)?

S = {H, T} 𝐄 = {∅, {H}, {T}, {H,T}} and the probabilities are: P{∅} = P{H,T} = 0, P({H}) = P({T}) = 0.5For compactness of writing, we usually do not list zero probabilities (note: This does not mean they are undefined). Usually, we write the above asS = {H, T} ; P({H}) = P({T}) = 0.5

Random Variables• Random variable 𝑋 is a function that maps S to real

number• 2 types of random variable:• Discrete random variable: Range is finite/countable• Continuous random variable: Range is uncountable • Example: • S: Students of COMP3600/6466• X: The program a student is enrolled in

Suppose the programs are indexed, e.g., BAC: 0, BIT: 1, BSwEng: 2, MCompSci: 3, …X(Zoey) = 0 ; X(Vijendran) = 0 ; X(Simian) = 2 ; ….

• Y: The height of a student in cmY(Mr M) = 177.23 ; Y(Ms S) = 178.3 ; …

Characterizing Random Variables• Cumulative distribution function (cdf)

𝐹u(𝑥) = 𝑃 𝑋 ≤ 𝑥 = 𝑃 𝑠 𝑋 𝑠 ≤ 𝑥, 𝑠 ∈ 𝑆

• Discrete: Probability mass function (pmf)𝑓u 𝑥 = 𝑃 𝑋 = 𝑥 = 𝑃 𝑠 𝑋 𝑠 = 𝑥, 𝑠 ∈ 𝑆

= 𝐹u 𝑥 − 𝐹u 𝑥 − 1

• Continuous: Probability density function (pdf)

𝑓u 𝑥 =𝑑𝐹u(𝑥)𝑑𝑥 ; 𝑃 𝑎 ≤ 𝑋 ≤ 𝑏 = z

b

_𝑓u 𝑥 𝑑𝑥

Characterizing Random Variables• Cumulative distribution function (cdf)

𝐹u(𝑥) = 𝑃 𝑋 ≤ 𝑥 = 𝑃 𝑠 𝑋 𝑠 ≤ 𝑥, 𝑠 ∈ 𝑆

• Discrete: Probability mass function (pmf)𝑓u 𝑥 = 𝑃 𝑋 = 𝑥 = 𝑃 𝑠 𝑋 𝑠 = 𝑥, 𝑠 ∈ 𝑆

= 𝐹u 𝑥 − 𝐹u 𝑥 − 1

Focus in this class (at least for now)

Compact Characterization of Random Variables

• Expected value: Weighted average of possible values of X, where the weight is the probability

• Useful property: Linearity of expectation 𝐸 𝑎𝑋 + 𝑏 = 𝑎𝐸 𝑋 + 𝑏

𝐸 𝑋 =|∀~

𝑥 𝑃 𝑋 = 𝑥

𝐸 𝑔(𝑋) =|∀~

𝑔(𝑥) 𝑃 𝑔(𝑋) = 𝑥

Next Week: Finishing off Probabilistic Analysis

+ Empirical Analysis + Correctness