Tonga Institute of Higher Education

Design and Analysis of Algorithms

IT 254

Lecture 8:

Complexity Theory

Complexity Introduction

• Some problems are intractable: as they grow large, we are unable to solve them in reasonable time

• What does reasonable time mean? Most of the time this means polynomial time– On an input of size n the worst-case

running time is O(nk) for some constant k– Polynomial time: O(n2), O(n3), O(1), O(n lg

n) – Not in polynomial time: O(2n), O(nn), O(n!)

Polynomial Algorithms• Are some problems solvable in polynomial

time?– Most algorithms we have looked at provide

polynomial-time solutions to some problem– In computer science, we say P is the class of

problems solvable in polynomial time• Are all problems solvable in polynomial time?

– No: Turing’s “Halting Problem” is not solvable by any computer, no matter how much time is given

– Turing’s problem tries to decide if a computer will stop on a given input or keep going. (Explained in class.)

– There are also problems that are clearly intractable (not in P), and there are also some that might be in P, but no one has figured out a good algorithm yet

Exponential Algorithms

• There are some algorithms that can only be solved in exponential time [O(n!),O(2n)]

• This is called Exp for exponential• We need to realize that all algorithms in P are

also in Exp, because the exponential class contains all polynomial time solutions as well.

• Think of this like upper bounds on algorithms

ProblemsThat can’tBe solved

Exp time

P time

NP Complete Problems

• The NP-Complete problems are an interesting class of problems whose status is unknown – No polynomial-time algorithm has been

discovered for an NP-Complete problem– But, also no lower bound has been proved for any

NP-Complete problem that says it must be bigger than polynomial

• We call this the P = NP question– The biggest open problem in CS and the person

who solves it will win a lot of prizes and a lot of money

NP Complete Example

• An example of an NP-Complete problem:– A hamiltonian cycle in an undirected graph is a

simple cycle that contains every vertex– The hamiltonian-cycle problem: given a graph

G, does it have a hamiltonian cycle?– A graph that has a hamiltonian cycle is said to

be "Hamiltonian"

Is there a Hamiltonian Cycle here?Can we think of an algorithm to find it?

Hamiltonian Cycles

• One way to check is if you list all the different permutations of the vertices of a graph and then check each permutation to see if it is a hamiltonian path.

• A permutation here means a configuration of a path that tries to go to each vertex in a cycle

• This idea unfortunately runs in O(n!) where n is the number of vertices.

• But we can check it easily. All we do is go through a permutation and see if it is Hamiltonian.

P and NP

• As mentioned, P is set of problems that can be solved in polynomial time

• NP (nondeterministic polynomial time) is the set of problems that can be solved in polynomial time by a nondeterministic computer.

• What does that mean?

Non-deterministic Computers

• Think of a non-deterministic computer as a computer that magically “guesses” a solution, then has to make sure that it is correct– If a solution exists, the computer always guesses it– One way to imagine it: a computer with an infinite

number of CPUs that can use all of them• Have one processor work on each possible solution• All processors try to make sure that their solution works• If a processor finds it has a working solution, then it’s solved

– So: NP = problems that can be checked for correctness in polynomial time,

– or in other words, if you had a magic computer, you could guess all the solutions and then just check them all to find the right one

N and NP

• Summary so far:– P = problems that can be solved in polynomial

time– NP = problems for which a solution can be

verified in polynomial time– Exp = problems that can only be solved in

Exponential time– Unknown whether P = NP (most suspect not)

• Hamiltonian-cycle problem is in NP:– Cannot be solved in polynomial time (yet)– Easy to verify solution in polynomial time

because we just need to run through a path to see if it touches all vertexes

… And NP Complete• There is also a special class of NP problems,

called NP-Complete problems. It says:– If any NP-Complete problem can be solved in

polynomial time…– …then every NP-Complete problem can be solved

in polynomial time…– …and, in fact, every problem in NP can be solved

in polynomial time (which would show P = NP)– Thus: If you can solve the Hamiltonian-cycle in

O(n100) time for any n, you’ve proved that P = NP. – Retire rich & famous.

Reducibility

• The most important part of NP-Completeness is called reducibility– We can say that a problem P can be reduced to another

problem Q if all the types of P can be “easily changed” into a type of Q

– And the solution to Q will provide a solution to the instance of P

– Thus, if P reduces to Q, then P is “no harder to solve” than Q.– The key is “easily changed” – by this, we mean can change

the problem from P to Q in polynomial time– Example: The problem of solving linear equations for an

unknown "x" can reduce to solving quadratic equations:– Instance: ax + b = 0. We can change this into 0x2 + ax + b

= 0 and then we can use the quadratic formula to solve– Changing it was easy, just add a 0x2

Reducibility

• An example of reducibility (that is not NP):– P: Given a set of Booleans, is at least one TRUE?

• Example { x1, x2, x3, x4, x5 } where x1..5 can be true or false

– Q: Given a set of integers, is their sum positive?– Can we change from P Q?

– Transformation: (x1, x2, …, xn) = (y1, y2, …, yn) where yi = 1 if xi = TRUE, yi = 0 if xi = FALSE

– Then add the y’s together, if it is greater than 0, then we also know at least one x is true.

Using Reductions

• If S is polynomial-time reducible to Q, we write this like, S p Q

• Definition of NP-Hard and NP-Complete: – If all problems R NP are reducible to S, then S is NP-

Hard– We say S is NP-Complete if S is NP-Hard and S NP– Formally, NP Complete means: R p S R NP

• If S p Q and S is NP-Complete, Q is also NP-Complete– This is the important idea you should take away today

NPP

NPC

Who cares about NP-Complete?

• Though nobody has proven that P != NP, if you prove a problem NP-Complete, most people accept that it is probably intractable

• Therefore it can be important to prove that a problem is NP-Complete– Don’t need to come up with an efficient

algorithm, because it can't be solved– Can instead work on approximation

algorithms (algorithms that come close to the solution)

Proving NP-Completeness

• What steps do we have to take to prove a problem P is NP-Complete?– Pick a known NP-Complete problem Q– Reduce Q to P, which means:

• Describe an algorithm that changes an instance of Q to an instance of P

• Prove the transformation works• Prove it runs in polynomial time

– And don’t forget, we need to prove P NP• Meaning, we have to prove that the solution

can be checked in polynomial time

NP Complete problems

• Given one NP-Complete problem, we can prove many interesting problems NP-Complete– Hamiltonian cycle– Graph coloring

• If you have a graph, can you color the graph with at most k colors so no two vertices with the same color are next to each other?

– Hamiltonian path• Does there exist a path between two vertices of a graph

that visits each vertex only once?– Knapsack problem

• If you have a bag that is a certain size, and there are different things with different sizes you can put in your bag, what is the best way to maximize what you put in

– Traveling salesman• If a salesman has to visit every city, how can he minimize

the distance that he travels totally.

Hamiltonian means Traveling Salesman

• The Hamiltonian path problem is similar to a very important problem.

• The well-known traveling salesman problem:– It’s goal is Optimization : a salesman must travel

to n cities, visiting each city exactly once and finishing where he begins. How to minimize travel time or travel distance?

– Model as complete graph with cost c(i,j) to go from city i to city j

• How would we turn this into a decision problem that can be verified in polynomial time?– Answer: ask if there exists a TSP with cost < k

Traveling Salesman

• The steps to prove TSP is NP-Complete:– Prove that TSP NP (we will not do)– Reduce the undirected hamiltonian cycle

problem to the TSP• So if we had a TSP-solver, we could use it to

solve the hamilitonian cycle problem in polynomial time

• How can we transform an instance of the hamiltonian cycle problem to an instance of the TSP?

• Can we do this in polynomial time?

Traveling Salesmen

• To transform the hamiltonian cycle problem on graph G = (V,E) into TSP, – create graph G1 = (V,E1):– G1 is a complete graph – Edges in E1 also in E have weight 0– All other edges in E1 have weight 1– TSP: is there a TSP on G1 with weight 0?– If G has a hamiltonian cycle, G1 has a cycle w/ weight

0– If G1 has cycle w/ weight 0, every edge of that cycle

has weight 0 and is thus in G. Thus G has a ham. cycle

The SAT Problem

• One of the first problems to be proved NP-Complete was satisfiability (SAT):– Given a Boolean expression on n variables, can we

assign values such that the expression is TRUE?

– Ex: ((x1 x2) ((x1 x3) x4)) x2

– Cook’s Theorem: The satisfiability problem is NP-Complete.

– It is quite complicated to prove that it is NP-Complete,

– Thus most problems today are reduced with this SAT problem and that makes them NP-Complete

Conjuctive Normal Form

• Even if the form of the Boolean expression is simplified, the problem is still NP-Complete– Literal: an occurrence of a Boolean or its

negation– A Boolean formula is in conjunctive normal form,

or CNF, if it is an AND of clauses, each of which is an OR of literals

• Ex: (x1 x2) (x1 x3 x4) (x5)

– 3-CNF: each clause has exactly 3 distinct literals• Ex: (x1 x2 x3) (x1 x3 x4) (x5 x3 x4)• Notice: true if at least one literal in each clause is true

– It is possible to reduce all Boolean equations to 3-CNF form in polynomial time

Satisifiability

• How can we transform a Boolean function into 3-CNF form? – Make a truth table and change it into normal

form. – Then we can use DeMorgan's law to change to

conjuctive normal form (product of sums).– Then we need to make sure there are only 3

literals per statement, which is not difficult, but there are extra rules to follow.

– For example, if we have (p + q), to make this into 3-CNF format we write: (p + q + r) + (p + q + ~r)

– Does this still give you the same value?• Even though it might seem easy to pick

values to make a statement true, no one has been able to do so in Polynomial time.

Summary

• What is important to know about NP completeness and complexity?– There will be times that you have problems and you

might like to use a computer to solve them. It’s good to know that you can’t solve these problems in a reasonable amount of time

• One example – scheduling classes for students is a NP complete problem. You might remember waiting weeks for the teachers to finally make up all the student’s schedules.

• If you wanted a computer to do this, you would need to realize it is a NP-hard problem (there does not exist a way to do it in polynomial time… yet)