how difficult is it to solve a problem?. computability and feasibility can a specific problem be...

Computability and Feasibility

How difficult is it to solve a problem?


Can a specific problem be solved by computation?

If so, can a suitably efficient solution be designed so that the ‘solution’ is feasible?

Computability is concerned with whether a problem can or cannot be solved.

Feasibility is concerned with how efficiently a problem can be solved.

We shall look at feasibility first then consider computability.

Key Questions


Feasibility

There are two standard measures of the efficiency of an algorithm:oTime: How long the algorithm takes to execute.oSpace: How much memory the algorithm uses for

the storage of data during execution.

When more than one algorithm exists which can solve a problem there is often a trade-off between time and space efficiency. i.e. one solution may be quicker but requires more memory than another solution.


How to measure efficiency? For a particular algorithm, the amount of

space and time required usually varies depending upon the number of inputs to the algorithm – e.g. adding up a thousand numbers takes a lot longer than adding up ten numbers.

Therefore, efficiency is normally expressed as a function of the number of inputs to an algorithm (commonly referred to as ‘n’).


How to measure efficiency? Space complexity can be easily measured - we

will look briefly at this later but shall focus on time complexity for now.

Measuring time complexity is more difficult than space complexity as the same algorithm run on two different computers could take very different amounts of time to execute as a result of factors such as:oProcessor clock speed.oEfficiency of hardware implementation of

operations used e.g. add.oOther processes executing concurrently.


How to measure efficiency? We need a measure of time complexity

that focuses on the algorithm rather than the hardware that it is being executed on.

We use Big O notation to do this.


Order of Growth / Big O

The time efficiency of an algorithm is established in the following way: Identify the basic operation that is performed

by the algorithm. Establish how many times this operation is

carried out as a function of the number of inputs (n).

Consider how quickly this value grows and identify the dominant part in this (e.g. in 4n2+3 the dominant part is n2).

This is the order of complexity of the algorithm. It is written as O(g) where g is the growth rate.


Linear Time Complexity (1)

Algorithm for summing a list of numbers in a file:

What is the basic operation? Total Total + Number How many times is it carried out? It is in a loop, executed

once for each number in the file. So, if a file contained n items the basic operation would be

executed n times. The algorithm is said to have order of complexity O(n).

Total 0WHILE NOT at end of file READ Number from file Total Total + NumberEND WHILEPRINT Total


Linear Time Complexity (2)Algorithm to sum a list of numbers entered at the keyboard, user terminating list by entering 0:

What is the basic operation? Total Total + Number How many times is it carried out? It is in a loop, executed

once for each number entered plus once more for the 0 terminator.

So, if n items were entered the operation would be executed n+1 times. The rate of growth is n as 1 is a constant.

The algorithm is said to have order of complexity O(n) since this is the dominant term in n+1.

Total 0REPEAT INPUT Number Total Total + NumberUNTIL Number = 0PRINT Total


Quadratic Time ComplexityA very naive sorting algorithm (based on bubble sort) to sort n items in an array:

The basic operation is the IF command that compares two numbers to check if they are out of order.

This is inside two loops. The outer one executes n-1 times and the inner one executes n-1 times.

So the IF command is executed (n-1)*(n-1)= n2-2n+1 times. n2 is the dominant term in this expression so the algorithm

is of order of complexity O(n2).

FOR i 1 TO n-1 DO FOR j 1 TO n-1 DO IF arr[j] > arr[j + 1] THEN SWAP (arr[j], arr[j + 1]) ENDIF ENDFORENDFOR


Worst, Best and Average Case In addition to the number of inputs, the

efficiency of some algorithms depends upon the values of the inputs made to them.

For example:oThe naive implementation of the bubble sort

algorithm on the previous slide takes the same amount of time to execute for a list of length n regardless of what the input values are.

oA more sophisticated implementation will terminate as soon as this list is sorted, so will be more efficient if the input list is already partially sorted than if it is not.


Worst, Best and Average Case In this improved version of the bubble sort algorithm

on the earlier slide, the outer FOR loop that repeated n-1 times is replaced by a REPEAT loop that terminates when the inner loop passes through the entire list without having to make any swaps i.e. the list is sorted.

i 1REPEAT swapmade FALSE FOR j 1 TO n-1 DO IF arr[j] > arr[j + 1] THEN SWAP (arr[j], arr[j + 1]) swapmade TRUE ENDIF ENDFOR i i + 1UNTIL swapmade = FALSE


Worst, Best and Average Case

Initial

i=1

i=2

i=3

i=4

[1] 30 30 30 20 10

[2] 50 20 20 10 20

[3] 20 10 10 30 30

[4] 10 40 40 40 40

[5] 40 50 50 50 50

Initial

i=1

[1] 10 10

[2] 20 20

[3] 30 30

[4] 40 40

[5] 50 50

In the best case the initial list is already ordered.

The algorithm terminates after one iteration of the outer loop because no swaps are made.

n-1 (4) comparisons have been made, so the best case order of complexity is O(n).

In the worst case the initial list is far from being ordered.

The outer loop iterates the maximum n-1 (4) times.

The inner loop iterates n-1 (4) times.So, the basic operation executes

(n-1)*(n-1) = n2-2n+1 times.So the worst case order of complexity is

O (n2), the same as the more naive algorithm.


Worst, Best and Average Case For the improved sort algorithm (as for many

algorithms), there is a significant difference between best and worst case performance:


Worst, Best and Average Case Three standard measures of efficiency are

therefore made for an algorithm:oWorst case complexity is the amount of resources

that an algorithm uses to solve a problem with the least favourable set of inputs. It is the upper bound on the amount of resources consumed by the algorithm.

oBest case complexity is the amount of resources that an algorithm uses to solve a problem with the most favourable set of inputs. It is the lower bound on the amount of resources consumed by the algorithm.

oAverage case complexity is the amount of resources that an algorithm uses to solve a problem averaged over all sets of possible inputs.


Algorithm vs Problem Complexity

For many problems, more than one algorithm can provide a solution.

For example an ordered list could be searched using a linear search or a binary search algorithm.

Computer scientists therefore often consider the complexity of a problem such as searching rather than the complexity of a particular algorithm such as binary search.

The complexity of a problem is the worst case complexity of the most efficient algorithm which solves the problem.


Classes of Complexity Problems are grouped into classes based upon their

complexity. This table shows how the execution time of an

algorithm increases as the number of inputs does for logarithmic, linear, polynomial and exponential classes:

Easy to solve Almost impossible to solve

Logarithmic

O(log2 n) O(n) O(2n) O(n2) O(n3) O(2n) O(3n)

1 0 1 2 1 1 2 32 1 2 4 4 8 4 95 2.3 5 10 25 125 32 243

10 3.3 10 20 100 1000 1024 5904920 4.3 20 40 400 8000 1048576 3.49E+0950 5.6 50 100 2500 125000 1.13E+15 7.18E+23

100 6.6 100 200 10000 1000000 1.27E+30 5.15E+471000 10 1000 2000 1000000 1000000000 1.1E+301 Too Big!

10000 13.3 10000 20000 100000000 1E+12 Too Big! Too Big!100000 16.6 100000 200000 10000000000 1E+15 Too Big! Too Big!

Linear (kn) Polynomial (nk) Exponential (kn)Number of inputs (n)


Tractability The time taken to solve exponential time complexity

problems grows rapidly as the number of inputs grows. e.g. for just 50 inputs an algorithm of time complexity

O(2n) would execute the basic operation more than 1130000000000000 times.

Problems that have a polynomial (or better) time complexity are known as tractable.

Problems that can be solved but which have a worse than polynomial time complexity are known as intractable.

Intractable problems are effectively unsolvable in the general case (except for very small numbers of inputs) as a solution would take such an unreasonably long time to produce that it is unlikely to be useful.

Intractable problems may sometimes be tackled using heuristics to produce a close to optimal solution.


Example Intractable Problems Travelling salesman: Given a set of n cities and

the distances between them, find the shortest tour that connects them all and allows the salesman to visit every city, before returning home.

Bin packing: There are n objects with sizes in the range 0 to 1 which must be fitted into bins, each of which has capacity 1. What is the minimum number of bins required?

Knapsack problem: A knapsack has a fixed capacity. There are a set of objects, each of which has a size and value. What is the most valuable set of objects that can be put into the knapsack?

Other types of problem which are often intractable include scheduling and optimisation problems.


Example Intractable Problems Why are these types of problem

intractable? Because the only way to produce an

optimal solution is to try out every possible solution – a brute force approach.

And, as the number of inputs to the problem increases, the number of possible solutions increase very rapidly.


Travelling Salesman e.g. For the travelling salesman problem:

A B20

A B20

C1016

2 cities = 1 route (ABA)Length 40

3 cities = 2 routes (ABCA), (ACBA) Both length 46 (as one is the reverse of the other)

A B20

C16

D

108 22 15

4 cities = 6 routes:ABCDA length 54 ADCBA length 54ABDCA length 73 ACDBA length 73ACBDA length 55 ADBCA length 55Optimum route is 54 long.

For n cities, (n-1)*(n-2)*....* 1 routes = (n-1)! routes.For just 20 cities, 121645100408832000 routes Conclusion: Intractable!!!


Space Efficiency

We have focussed on time efficiency as speed is the aspect of complexity that is usually of most concern.

However space efficiency can also be an important factor when choosing between different algorithms that have the same purpose.

Sometimes there can be a trade-off between space and time efficiency.

A good example of this is when sorting data.


Space Efficiency Example Simple sorting algorithms such as bubble sort and

insertion sort have very poor worst case time complexity of O(n2). The data is sorted ‘in situ’ within the existing list so to sort n items only n memory locations are required. No extra memory locations are required during the sort process.

Merge sort is a much more time efficient sort algorithm with worst case time complexity O(n log2n).

However, merge sort is less space efficient. This is because it is a recursive algorithm and copies of subsections of the list are usually generated each time the algorithm recurses.

The next two slides show the merge sort algorithm and why this extra storage space is required.


Space Efficiency Example

The merge sort procedure:

If list length <= 1 then exit as list is sorted If list length > 1 thenoSplit the unsorted list into two sublists of about

half the size.oRecursively call the merge sort procedure to sort

the first of the two sublists.oRecursively call the merge sort procedure to sort

the second of the two sublists.oMerge the two sublists back into one sorted list.


Space Efficiency Example

29 49 11 18 8037 22

29 49 1137 18 80 22

29 49 1137 18 80 22

2937 49 11 18 80

37 11 4929 18 80

29 37 4911 18 22 80

18 22 29 37 4911 80

Rec

urs

ing

Un

win

din

g

Rec

urs

ion

An example execution of merge sort on a seven item list:

Additional storage space for temporary storageof lists.


Computability

We have seen that some problems admit no effective solution because they take too long to compute (intractable problems).

But are there any problems that simply cannot be solved, however much time and space we could give them?

Yes – several problems have been proven to be non-computable.

The most famous non-computable problem is the halting problem.


The Halting Problem The halting problem: Is it possible to write a program

that can tell, given a program and its inputs, whether the program will halt? (without executing the program being tested)

The answer is no, it has been proven that in the general case this problem cannot be solved.

We shall look briefly at the proof of this although A Level candidates are only required to know what the problem is and that it is not computable.

It uses a standard technique called proof by contradiction in which we:o Make a proposition.o Show that if the proposition were true, a contradiction

must occur.o Therefore we prove that the proposition must be untrue.


Proof of Non-computability Assume that the halting function does exist.

It is defined as a function h with two inputs p and i.op is the program that the halting function is

testing.o i is the input that will be made to the program p.

The function h should output 0 if p halts on the input n and 1 if p does not halt.

h ( p , i ) =0, if p halts on input i1, if p does not halt{


Proof of Non-computability

Now construct a program r(n) which calls the function h using n as both parameters i and p:

r ( n ) Halts if h ( n , n ) = 1Loops forever otherwise{


Proof of Non-computability Now use the halting function h to test if the program r

terminates with input n. So:

Recall the definition of r:

And substitute the defintion of r into h gives:

Now, if r = n we would have:

A contradiction!

h ( r , n ) =0, if r halts on input n1, if r does not halt{

r ( n ) Halts if h ( n , n ) = 1Loops forever otherwise{

h ( r , n ) = 0, if h ( n , n ) = 11, if h ( n , n ) = 0{

h ( n , n ) = 0, if h ( n , n ) = 11, if h ( n , n ) = 0{


Proof of Non-computability We have arrived at a contradiction, i.e. that

h(n,n)=0 if h(n,n)=1 and vice-versa. Since the steps we have taken in arriving at this

contradiction are sound, it follows that the original proposition, i.e. that the halting function h exists must be untrue.

Therefore the halting problem is not computable.

Other problems which have proven to be non-computable are the Entscheidungsproblem, Hilbert’s tenth problem and the Post correspondence problem.


End of Presentati

on


Dominant Term Solution

Expression Dominant Term

Order of Complexity

n + 5 n O(n)6n + n2 + 12 n2 O(n2)2n + n3 + 12 2n O(2n)

Q1) Dominant terms and order of complexity:


Most to Least Efficient SolutionQ2) Orders of complexity in order, most

efficient first:

O(1) O(log2 n) O(n) O(n2) O(n4) O(3n)


Linear Search Solution

Basic Operation: Comparison of item in list with search value.

Order of Time Complexity: O(n) Explanation: The loop is repeated once for

each item in the array, so n times for an array containing n items.

INPUT SearchValueFound FalseFOR Index 1 TO ListLength DO IF ArrayValue[Index] = SearchValue THEN Found TrueENDFOROUTPUT Found

Q3)



Why different complexities: This is because the loop terminates as soon as the item is found, if it is in the list. So the number of times the basic operation is executed depends upon where the item is in the list.

INPUT SearchValueFound FalseIndex 0REPEAT Index Index + 1 IF ArrayValue[Index] = SearchValue THEN Found TrueUNTIL Index = ListLength OR Found = TrueENDFOROUTPUT Found

Q4)



Best Case: Item is at start of list, in which case only one item needs to be examined so O(1).

Worst Case: Item is at end of list so all n items must be checked, so O(n).

INPUT SearchValueFound FalseIndex 0REPEAT Index Index + 1 IF ArrayValue[Index] = SearchValue THEN Found TrueUNTIL Index = ListLength OR Found = TrueENDFOROUTPUT Found

Q4)


Tractability Solution

Order of Complexity

Tractable?

O(3n) NO(n10) Y

O(log2 n) YO(2n) N

Q5) Which algorithms are tractable? (though not necessarily quick to solve!)


Halting Problem Solution

Q6) The significance of the Halting problem is:

o That there are some problems that we can prove cannot be solved by a computer.

o That we know that it is not possible, in the general case, to write a program that will determine if another program will halt on a given set of inputs.

how difficult is it to solve a problem?. computability and feasibility can a specific problem be...

Documents

particular algorithm

measure of time complexity

space efficiency

space complexity

measuring time complexity

order of complexity

total total numberhow

basic operation