chapter # 3 : algorithmssite.iugaza.edu.ps/musbahshaat/files/chapter3_all_handout.pdf · ecom 2311-...

ECOM 2311- Discrete Mathematics

Chapter # 3 : Algorithms

Fall, 2013/2014

ECOM 2311- Discrete Mathematics - Ch.3 Dr. Musbah Shaat 1 / 41

Outline

1 Algorithms

2 The Growth of Functions

3 Complexity of Algorithms


AlgorithmsIntroduction

The term algorithm is a corruption of the name Al-Khowarizmi, ABUJA’FAR MOHAMMED IBN MUSA AL-KHOWARIZMI. Al-Khowarizmi,

an astronomer and mathematician, was a member of the House ofWisdom, an academy of scientists in Baghdad. The word algebra comesfrom al-jabr, part of the title of his book Kitab al-jabr w’al muquabala.



An Algorithm

is a finite sequence of precise instructions for performing a computationor for solving a problem.

Given a sequence of integers, find the largest one;

Given a set, list all its subsets;

Given a set of integers, put them in increasing order;

Given a network, find the shortest path between two vertices.

Solution Methodology

Construct a model that translates the problem into a mathematicalcontext.

To complete the solution, a method is needed that will solve thegeneral problem using the model.

Ideally, what is required is a procedure that follows a sequence ofsteps that leads to the desired answer.



Example: Describe an algorithm for finding the maximum (largest) valuein a finite sequence of integers.

Solution: We perform the following steps

Set the temporary maximum equal to the first integer in thesequence.

Compare the next integer in the sequence to the temporarymaximum, and if it is larger than the temporary maximum, set thetemporary maximum equal to this integer.

Repeat the previous step if there are more integers in the sequence.

Stop when there are no integers left in the sequence. The temporarymaximum at this point is the largest integer in the sequence.

An algorithm can also be described using a computer language. However,when that is done, only those instructions permitted in the language canbe used.



Pseudocode

Pseudocode provides an intermediate step between an English languagedescription of an algorithm and an implementation of this algorithm in aprogramming language.

A pseudocode description of the algorithm for finding the maximumelement in a finite sequence is as follows.



PROPERTIES OF ALGORITHMS

Input. An algorithm has input values from a specified set.

Output. From each set of input values an algorithm producesoutput values from a specified set. The output values are thesolution to the problem.

Definiteness. The steps of an algorithm must be defined precisely.

Correctness. An algorithm should produce the correct outputvalues for each set of input values.

Finiteness. An algorithm should produce the desired output after afinite (but perhaps large) number of steps for any input in the set.

Effectiveness. It must be possible to perform each step of analgorithm exactly and in a finite amount of time.

Generality. The procedure should be applicable for all problems ofthe desired form, not just for a particular set of input values.


AlgorithmsSearching Algorithms

Problem: locate an element x in a list of distinct elementsa1, a2, · · · , an, or determine that it is not in the list.

Solution: the location of the term in the list that equals x (that is, i isthe solution if x = ai ) and is 0 if x is not in the list.

Procedures of the Linear Search (sequential search):

Compare x and a1. When x = a1, the solution is the location of a1,namely, 1.

When x 6= a1, compare x with a2. If x = a2, the solution is thelocation of a2, namely, 2.

When x 6= a2, compare x with a3. Continue this process, comparingx successively with each term of the list until a match is found,where the solution is the location of that term, unless no matchoccurs.

If the entire list has been searched without locating x, the solution is0.



The pseudocode for the linear search algorithm is displayed as Algorithm2.



THE BINARY SEARCH: can be used when the list has termsoccurring in order of increasing size (for instance: if the terms arenumbers, they are listed from smallest to largest; if they are words, theyare listed in alphabetic order).

Procedures of the Binary search algorithm:

Compare the element to be located to the middle term of the list.

The list is then split into two smaller sublists of the same size, orwhere one of these smaller lists has one fewer term than the other.

The search continues by restricting the search to the appropriatesublist based on the comparison of the element to be located andthe middle term.

The binary search algorithm is much more efficient than the linearsearch algorithm.



Example: Search 19 in the list 1 2 3 5 6 7 8 10 12 13 15 16 18 19 20 22

Split the list into 2 subsets with 8 terms each1 2 3 5 6 7 8 10 12 13 15 16 18 19 20 22Compare 19 with the largest element of the first set19 > 10⇒ search 19 in the second set

Split the second subset into 2 smaller subsets12 13 15 16 18 19 20 22Compare 19 with 1619 > 16⇒ search 19 in the second set

Split the second subset as: 18 19 20 22Compare 19 > 19 is false ⇒ search 19 in 18 19Split the subset as : 18 19Since 19 > 18⇒ search restricted to the second list.

Finally 19 is located at the 14th element of the original list



The pseudocode for the binary search algorithm is displayed as Algorithm3.


AlgorithmsSorting

Sorting Problem: putting these elements into a list in which theelements are in increasing order. For instance, sorting the list 7, 2, 1, 4,5, 9 produces the list 1, 2, 4, 5, 7, 9. Sorting the list d, h, c, a, f (usingalphabetical order) produces the list a, c, d, f, h.

The bubble sort: is one of the simplest sorting algorithms. It puts a listinto increasing order by successively comparing adjacent elements,interchanging them if they are in the wrong order.

Procedures of the bubble sort algorithm:

Perform the basic operation, that is, interchanging a larger elementwith a smaller one following it, starting at the beginning of the list,for a full pass.

Iterate this procedure until the sort is complete.


AlgorithmsSorting

Example: Use the bubble sort to put 3, 2, 4, 1, 5 into increasing order.


AlgorithmsSorting

The insertion sort: is a simple sorting algorithm, but it is usually notthe most efficient.

Procedures of the insertion sort algorithm:

The insertion sort begins with the second element and compares thissecond element with the first element.

The insertion sort inserts it before the first element if it does notexceed the first element and after the first element if it exceeds thefirst element.

At this point, the first two elements are in the correct order. Thethird element is then compared with the first element, and if it islarger than the first element, it is compared with the second element;it is inserted into the correct position among the first three elements.

In general, in the jth step of the insertion sort, the jth element ofthe list is inserted into the correct position in the list of thepreviously sorted j - 1 elements.


AlgorithmsSorting

Example: Use the insertion sort to put 3, 2, 4, 1, 5 into increasing order.First, compare 2 and 3. As 2 is lower, we get 2,3,4,1,5.

Insert the third element 4 in the comparison. Because 2 < 4 and 3 < 4, 4remains in the third place to get 2,3,4,1,5.

To find the place of 1, we put it in the 1st place because 1 < 2 to get1,2,3,4,5

Compare the last element 5 to the rest of the elements successively, we putit in the last order as 5 > 4.


AlgorithmsGreedy Algorithms

Greedy Algorithm: goals to solve optimization problems by finding asolution to the given problem that either minimizes or maximizes thevalue of some parameter. The algorithm make what seems to be the bestchoice at each step.

Finding a route between two cities with smallest total mileage

Determining a way to encode messages using the fewest bits possible

Finding a set of fiber links between network nodes using the leastamount of fiber.

You have to prove that a greedy algorithm always finds an optimalsolution. (we call the algorithm ”greedy” whether or not it finds anoptimal solution.) To do this, we either prove that the solution is optimalor we show that there is a counterexample where the algorithm yields anonoptimal solution.



Example: Consider the problem of making n cents (67 cents forexample) change with quarters [25 cents], dimes [10 cents], nickels [5cents] and pennies [1 cent], and using the least total number of coins.

Select a quarter, leaving 42 cents.

Select a second quarter, leaving 17 cents.

Select a dime, leaving 7 cents.

Select a nickel, leaving 2 cents.

Select a penny, leaving 1 cent

Select a penny.

Remark: if we have only quarters, dimes and pennies ⇒ the change for30 cents would be made using 6 coins = 1 quarter + 5 pennies. Whereasa better solution is equal to 3 coins = 3 dimes!



Example: Suppose we have a group of proposed talks with preset startand end times. Devise a greedy algorithm to schedule as many of these

talks as possible in a lecture hall, under the assumptions that once a talkstarts, it continues until it ends, no two talks can proceed at the sametime, and a talk can begin at the same time another one ends. Assumethat talk j begins at time sj (where s stands for start) and ends at time

ej (where e stands for end).



Homework [due 29th of Oct].From the text book, Section 3.1, page 202Questions: Q2,Q10, Q14, Q20, Q34, and Q38.


The Growth of FunctionsBig-O Notation

Big-O Notation

Let f and g be functions from the set of integers or the set of realnumbers to the set of real numbers.We say that f (x) is O(g(x)) if thereare constants C and k such that|f (x)| ≤ C |g(x)|whenever x > k . [This is read as ”f (x) is big-oh of g(x).”]

C and k are called witnesses to the relationship f (x) is O(g(x)).

Approach to work with the definition of big O Notation

Select a value of k for which the size of |f (x)| can be readilyestimated when x > k .

Find a value of C for which |f (x)| ≤ C |g(x)| for x > k.



Example: Show that f (x) = x2 + 2x + 1 is O(x2).

Choose k = 1⇒ x > 1.

When x > 1⇒ x2 + 2x + 1 ≤ x2 + 2x2 + x2 = 4x2.

Consequently, we can take C = 4 and k = 1 as witnesses to showthat f (x) is O(x2). [What if k=2?]

Note that it is not necessary to use absolute values here because allfunctions in these equalities are positive when x is positive.



Observe that in the relationship ”f (x) is O(x2),” x2 can be replacedby any function with larger values than x2.

For example, f (x) is O(x3), f (x) is O(x2 + x + 7), and so on.

Note that in Example 1 we have two functions, f (x) = x2 + 2x + 1and g(x) = x2, such that f (x) is O(g(x)) and g(x) is O(f (x)).

We say that two functions f (x) and g(x) that satisfy both of thesebig-O relationships are of the same order.

It is acceptable to write f (x) ∈ O(g(x)) because O(g(x)) representsthe set of functions that are O(g(x)).

When big-O notation is used, the function g in the relationship f (x)is O(g(x)) is chosen to be as small as possible.

In subsequent discussions, we will almost always deal with functionsthat take on only positive values. All references to absolute valuescan be dropped when working with big-O estimates for suchfunctions.



Example: Show that f (x) = 7x2 is O(x3).

Choose k = 7⇒ x > 7.

Multiplying both sides of x > 7 by x2, we have 7x2 < x3.

Consequently, we can take C = 1 and k = 7 as witnesses toestablish the relationship 7x2 is O(x3).

Alternatively, when x > 1, we have 7x2 < 7x3, so that C = 7 and k= 1 are also witnesses to the relationship 7x2 is O(x3).

Big-O Estimates for Some Important Functions

Let f (x) = anxn + an−1x

n−1 + · · ·+ a1x + a0, where a0, a1, · · · , an−1, anare real numbers. Then f (x) is O(xn).


The Growth of FunctionsBig-O Estimates for Some Important Functions

Example: How can big-O notation be used to estimate the sum of thefirst n positive integers?.

Because each of the integers in the sum of the first n positiveintegers does not exceed n, it follows that

1 + 2 + · · ·+ n ≤ n + n + · · ·+ n = n.n = n2.

From this inequality it follows that 1 + 2 + 3 + + n is O(n2), takingC = 1 and k = 1 as witnesses.

Example: Give big-O estimates for the factorial function and thelogarithm of the factorial function.

A big-O estimate for n! can be obtained by noting that each term inthe product does not exceed n.

n! = 1.2.3 · · · n ≤ n.n.n. · · · n = nn

From this inequality it follows that n! is O(nn), taking C = 1 and k= 1 as witnesses.



Taking logarithms of both sides of the inequality established for n!,we obtain log n! ≤ log nn = n log n.

This implies that log n! is O(n log n), again taking C = 1 and k = 1as witnesses.

Example: use the inequality n < 2n to show that log n is O(n)..

n < 2n ⇒ log n < n

From this inequality it follows that log n is O(n), taking C = 1 andk = 1 as witnesses.

Big-O notation is used to estimate the number of operations needed tosolve a problem using a specified procedure or algorithm. The functionsused in these estimates often include the following:

1, log n, n, n log n, n2, 2n, n!


The Growth of FunctionsThe Growth of Combinations of Functions

THEOREM 2

Suppose that f1(x) is O(g1(x)) and that f2(x) is O(g2(x)). Then(f1 + f2)(x) is O(max(|g1(x)|, |g2(x)|)).

COROLLARY 1

Suppose that f1(x) and f2(x) are both O(g(x)). Then (f1 + f2)(x) isO(g(x)).

THEOREM 3

Suppose that f1(x) is O(g1(x)) and f2(x) is O(g2(x)). Then (f1f2)(x) isO(g1(x)g2(x)).



Example: Give a big-O estimate for f (n) = 3n log(n!) + (n2 + 3) log n,where n is a positive integer..

For the first part 3n log(n!), we have two subparts 3n and log(n!).

As we just mentioned, log(n!) is O(n log n) while 3n is O(n).

The product of the two subparts implies that 3n log(n!) isO(n2 log n).

For the second part (n2 + 3) log n, we have two subparts (n2 + 3)and log n.

As we mentioned, log(n) is O(n) while n2 + 3 is O(n2).

The product of the two subparts implies that (n2 + 3) log n isO(n2 log n).

Therefore, the two big-O estimates for the products shows thatf (n) = 3n log(n!) + (n2 + 3) log n is O(n2 log n).


The Growth of FunctionsBig-Omega and Big-Theta Notation

Let f and g be functions from the set of integers or the set of realnumbers to the set of real numbers.We say that f (x) is Ω(g(x)) if thereare positive constants C and k such that|f (x)| ≥ C |g(x)|whenever x > k . [This is read as ”f (x) is big-Omega of g(x).”]

There is a strong connection between big-O and big-Omega notation. Inparticular, f (x) is Ω(g(x)) if and only if g(x) is O(f (x)).

The function f (x) = 8x3 + 5x2 + 7 is Ω(g(x)), where g(x) is thefunction g(x) = x3. This is easy to see becausef (x) = 8x3 + 5x2 + 7 ≥ 8x3 for all positive real numbers x . This isequivalent to saying that g(x) = x3 is O(8x3 + 5x2 + 7)


The Growth of FunctionsBig-Omega and Big-Theta Notation

Let f and g be functions from the set of integers or the set of realnumbers to the set of real numbers. We say that f (x) is Θ(g(x)) if f (x)is O(g(x)) and f (x) is Ω(g(x)). When f (x) is Θ(g(x)) we say that f isbig-Theta of g(x), that f (x) and g(x) are of the same order.

Example: Show that 3x2 + 8x log x is Θ(x2).

Because 0 ≤ 8x log x ≤ 8x2, it follows that 3x2 + 8x log x ≤ 11x2 forx > 1. Consequently, 3x2 + 8x log x is O(x2). Clearly, x2 isO(3x2 + 8x log x). Consequently, 3x2 + 8x log x is Θ(x2).

Homework [due 29th of Oct].From the text book, Section 3.2, page 216Questions: Q2,Q4, Q15, Q26, and Q30 [Parts: a,d].


Complexity of AlgorithmsTime Complexity

How can the efficiency [computational complexity] of an algorithm beanalyzed?

The time used by a computer to solve a problem using the algorithm[time complexity].

The amount of computer memory required to implement thealgorithm [space complexity].

The time complexity of an algorithm can be expressed in terms of thenumber of operations used by the algorithm when the input has aparticular size.

The operations used to measure time complexity can be thecomparison of integers, the addition of integers, the multiplication ofintegers, the division of integers, or any other basic operation.

Time complexity is described in terms of the number of operationsrequired instead of actual computer time because of the difference intime needed for different computers to perform basic operations.



Describe the time complexity of Algorithm 1 of Section 3.1 for findingthe maximum element in a finite set of integers.?

Exactly 2(n - 1) + 1 = 2n - 1 comparisons are used. Hence, thealgorithm has time complexity O(n), measured in terms of the number ofcomparisons used.



Describe the time complexity of Algorithm 2 of Section 3.1 for the linearsearch algorithm.?

If x = ai ⇒ 2i + 1 comparisons are used.The most comparisons, 2n + 2, are required when the element is not inthe list [Worst case].Hence, a linear search requires O(n) comparisons in the worst case,because 2n + 2 is O(n).



Describe the time complexity of Algorithm 3 of Section 3.1 for the binarysearch algorithm.?

Assume that there are n = 2k elements in the list ⇒ k = log n.Each step we have i < j and x > am comparisons.At every stage we have 2k−x elements where x is the number of thestage up to k − x = 0.Hence, at most 2k + 2 = 2 log n + 2 comparisons are required which isO(log n)



AVERAGE-CASE COMPLEXITY

The average number of operations used to solve the problem over allpossible inputs of a given size is found in this type of analysis [Seeexample 4 page 221].

Although we have counted the comparisons needed to determinewhether we have reached the end of a loop, these comparisons areoften not counted. From this point on we will ignore suchcomparisons.

What is the worst-case complexity of the bubble sort in terms of thenumber of comparisons made?

We need n-1 passes. In the i th pass, there are n − i comparisons. Hence,the number of comparisons is∑n−1

i=1 (n − i) = (n−1)n2 as

∑nk=1 k = n(n+1)

2

Consequently, it has O(n2) complexity.



What is the worst-case complexity of the insertion sort in terms of thenumber of comparisons made?

— In the worst case, j comparisons are required to insert the j th elementinto the correct position. j − 1 with the ordered elements and one withitself to stop. Therefore, the number of comparisons is2 + 3 + · · ·+ n =

∑ni=2(i) = (n+1)n

2 − 1 as∑n

k=1 k = n(n+1)2

— Consequently, it has O(n2) complexity.— The most efficient sorting algorithms can sort n items in O(n log n)time, as we will show later.ECOM 2311- Discrete Mathematics - Ch.3 Dr. Musbah Shaat 37 / 41

Complexity of AlgorithmsComplexity of Matrix Multiplication

How many additions of integers and multiplications of integers are usedby Algorithm 1 to multiply two n × n matrices with integer entries?

There are n2 entries in the product of A and B.

To find each entry requires a total of n multiplications and n - 1additions.

Hence, a total of n3 multiplications and n2(n− 1) additions are used.

Multiplying two n × n matrices directly from the definition requiresO(n3) multiplications and additions.



Example: In which order should the matrices A1,A2, and A3 where A1

is 30× 20, A2 is 20× 40, and A3 is 40× 10, all with integer entries- bemultiplied to use the least number of multiplications of integers?

There are two possible ways to compute A1,A2,A3. These areA1(A2A3) and (A1A2)A3.

If A2 and A3 are first multiplied, a total of 20 · 40 · 10 = 8000multiplications of integers are used to obtain the 20× 10 matrixA2A3.

Then, to multiply A1 and A2A3 requires 30 · 20 · 10 = 6000multiplications. Hence, a total of 8000 + 6000 = 14, 000multiplications.

if A1 and A2 are first multiplied, then 30 · 20 · 40 = 24, 000multiplications are used to obtain the 30× 40 matrix A1A2.

Then, to multiply A1A2 and A3 requires 30 · 40 · 10 = 12, 000multiplications. Hence, a total of 24, 000 + 12, 000 = 36, 000multiplications.



Brute-force algorithm,

a problem is solved in the most straightforward manner based on thestatement of the problem and the definitions of terms. Brute-forcealgorithms are designed to solve problems without regard to thecomputing resources required.


Homework [due 5th of Nov].From the text book, Section 3.3, page 229Questions: Q2, Q3, Q8 and Q22[Parts a,b].

End of Chapter # 3

chapter # 3 : algorithmssite.iugaza.edu.ps/musbahshaat/files/chapter3_all_handout.pdf · ecom 2311-...

Documents