DefinitionsExamples

Theory

UCT Algorithm Circle: Probabilistic Algorithms

Gwylim Ashley

24 February 2011

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Outline

1 Definitions

2 Examples

3 Theory

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Probabilistic algorithm

A probabilistic algorithm is one which uses a source of randominformation

The randomness used may result in randomness in the output,or the output may not be affected by it

The running time may be good in the expected case, and badin the worst case

We can model such algorithms as probabilistic turing machines

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Probabilistic algorithm

A probabilistic algorithm is one which uses a source of randominformation

The randomness used may result in randomness in the output,or the output may not be affected by it

The running time may be good in the expected case, and badin the worst case

We can model such algorithms as probabilistic turing machines

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Probabilistic algorithm

A probabilistic algorithm is one which uses a source of randominformation

The randomness used may result in randomness in the output,or the output may not be affected by it

The running time may be good in the expected case, and badin the worst case

We can model such algorithms as probabilistic turing machines

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Las Vegas algorithm

A Las Vegas algorithm is guaranteed to always return thecorrect result

The output may be different if the algorithm is run again (e.g.a non-stable sort may return different, but correct, results)

The worst-case running time may be significantly differentfrom the expected time

Example: quicksort

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Las Vegas algorithm

A Las Vegas algorithm is guaranteed to always return thecorrect result

The output may be different if the algorithm is run again (e.g.a non-stable sort may return different, but correct, results)

The worst-case running time may be significantly differentfrom the expected time

Example: quicksort

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Las Vegas algorithm

A Las Vegas algorithm is guaranteed to always return thecorrect result

The output may be different if the algorithm is run again (e.g.a non-stable sort may return different, but correct, results)

The worst-case running time may be significantly differentfrom the expected time

Example: quicksort

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Las Vegas algorithm

A Las Vegas algorithm is guaranteed to always return thecorrect result

The output may be different if the algorithm is run again (e.g.a non-stable sort may return different, but correct, results)

The worst-case running time may be significantly differentfrom the expected time

Example: quicksort

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Monte Carlo method

A Monte Carlo method is one in which the results are random;and not necessarily correct

The results may be correct some fraction of the time, or theymay be a numerical approximation to some quantity

Generally a better result can be obtained by running thealgorithm for longer, as Monte Carlo methods generally workby random sampling

The running time is usually not random

Example: Metropolis-Hastings algorithm

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Monte Carlo method

A Monte Carlo method is one in which the results are random;and not necessarily correct

The results may be correct some fraction of the time, or theymay be a numerical approximation to some quantity

Generally a better result can be obtained by running thealgorithm for longer, as Monte Carlo methods generally workby random sampling

The running time is usually not random

Example: Metropolis-Hastings algorithm

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Monte Carlo method

A Monte Carlo method is one in which the results are random;and not necessarily correct

The results may be correct some fraction of the time, or theymay be a numerical approximation to some quantity

Generally a better result can be obtained by running thealgorithm for longer, as Monte Carlo methods generally workby random sampling

The running time is usually not random

Example: Metropolis-Hastings algorithm

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Monte Carlo method

A Monte Carlo method is one in which the results are random;and not necessarily correct

The results may be correct some fraction of the time, or theymay be a numerical approximation to some quantity

Generally a better result can be obtained by running thealgorithm for longer, as Monte Carlo methods generally workby random sampling

The running time is usually not random

Example: Metropolis-Hastings algorithm

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Monte Carlo method

A Monte Carlo method is one in which the results are random;and not necessarily correct

The results may be correct some fraction of the time, or theymay be a numerical approximation to some quantity

Generally a better result can be obtained by running thealgorithm for longer, as Monte Carlo methods generally workby random sampling

The running time is usually not random

Example: Metropolis-Hastings algorithm

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Outline

1 Definitions

2 Examples

3 Theory

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Building a binary search tree

We can quickly build a binary search tree from a list by addingelements one by one, starting at the root

In the case of strange input data, this may give an unbalancedtree

If items are added in a random order, the expected height willbe small regardless of the input ordering

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Building a binary search tree

We can quickly build a binary search tree from a list by addingelements one by one, starting at the root

In the case of strange input data, this may give an unbalancedtree

If items are added in a random order, the expected height willbe small regardless of the input ordering

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Building a binary search tree

We can quickly build a binary search tree from a list by addingelements one by one, starting at the root

In the case of strange input data, this may give an unbalancedtree

If items are added in a random order, the expected height willbe small regardless of the input ordering

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Polygon triangulation

Seidel’s algorithm for polygon triangulation requires processingthe edges of a polygon sequentially and hence updating a tree

The tree will potentially become unbalanced

Randomization makes this less likely

This results in an algorithm of O(nlog∗(n))

In this case, similar to with constructing a binary search tree,randomisation is useful because it makes the worst caseperformance unlikely to occur

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Polygon triangulation

Seidel’s algorithm for polygon triangulation requires processingthe edges of a polygon sequentially and hence updating a tree

The tree will potentially become unbalanced

Randomization makes this less likely

This results in an algorithm of O(nlog∗(n))

In this case, similar to with constructing a binary search tree,randomisation is useful because it makes the worst caseperformance unlikely to occur

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Polygon triangulation

Seidel’s algorithm for polygon triangulation requires processingthe edges of a polygon sequentially and hence updating a tree

The tree will potentially become unbalanced

Randomization makes this less likely

This results in an algorithm of O(nlog∗(n))

In this case, similar to with constructing a binary search tree,randomisation is useful because it makes the worst caseperformance unlikely to occur

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Probabilistic primality testing

A number of efficient algorithms for checking primality ofnumbers are in co− RP.

For Miller-Rabin, for example, when the algorithm indicatescompositeness, it is guaranteed, and the probability of the testincorrectly identifying a composite as prime can be madeexponentially smaller by linearly increasing the running time.

This can be used to make the algorithm effectively never fail,by using a few tens of iterations.

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Probabilistic primality testing

A number of efficient algorithms for checking primality ofnumbers are in co− RP.

For Miller-Rabin, for example, when the algorithm indicatescompositeness, it is guaranteed, and the probability of the testincorrectly identifying a composite as prime can be madeexponentially smaller by linearly increasing the running time.

This can be used to make the algorithm effectively never fail,by using a few tens of iterations.

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Probabilistic primality testing

A number of efficient algorithms for checking primality ofnumbers are in co− RP.

For Miller-Rabin, for example, when the algorithm indicatescompositeness, it is guaranteed, and the probability of the testincorrectly identifying a composite as prime can be madeexponentially smaller by linearly increasing the running time.

This can be used to make the algorithm effectively never fail,by using a few tens of iterations.

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Genetic algorithms

Suppose that we want to maximise some function over someparameter space, but we don’t know anything about thefunction other than that it is (somewhat) continuous.

We can choose random points in the space, and select thebest one.

We can also make use of the fact that the function iscontinuous to search for better solutions near good solutions.

This leads to algorithms such as genetic algorithms andsimulated annealing.

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Genetic algorithms

Suppose that we want to maximise some function over someparameter space, but we don’t know anything about thefunction other than that it is (somewhat) continuous.

We can choose random points in the space, and select thebest one.

We can also make use of the fact that the function iscontinuous to search for better solutions near good solutions.

This leads to algorithms such as genetic algorithms andsimulated annealing.

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Genetic algorithms

Suppose that we want to maximise some function over someparameter space, but we don’t know anything about thefunction other than that it is (somewhat) continuous.

We can choose random points in the space, and select thebest one.

We can also make use of the fact that the function iscontinuous to search for better solutions near good solutions.

This leads to algorithms such as genetic algorithms andsimulated annealing.

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Rabin’s nearest neighbor algorithm

Suppose we have a set of points on the plane, and we want tofind the two points which have the least distance from eachother

There exist deterministic algorithms which are O(nlog(n)) tosolve this

However, there is a probabilistic algorithm which is O(n)expected time, which works by selecting a random subset ofpoints of size

√n, and hence finding an upper bound d on the

minimum distance

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Rabin’s nearest neighbor algorithm

Suppose we have a set of points on the plane, and we want tofind the two points which have the least distance from eachother

There exist deterministic algorithms which are O(nlog(n)) tosolve this

However, there is a probabilistic algorithm which is O(n)expected time, which works by selecting a random subset ofpoints of size

√n, and hence finding an upper bound d on the

minimum distance

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Outline

1 Definitions

2 Examples

3 Theory

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Probabilistic Turing machine

Similar to a non-deterministic turing machine, but the statetransition function assigns probabilities to each future state

From these probabilities, we can find the probability that theaccepting and rejecting states are reached

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Complexity classes

The following are defined with respect to probabilistic turingmachines, and refer to decision problems

ZPP: Zero-error Probabilistic Polynomial time. The class ofproblems which can be solved in expected polynomial time.

BPP: Bounded-error Probabilistic Polynomial time. The classof problems which can be solved in expected polynomial timewith a probability of error of at most 1/3.

RP: Randomized Polynomial time The class of decisionproblems for which False is always returned if the answer isFalse, and True is returned in at least 1/2 of cases if theanswer is True. The polynomial time is expected value.

co− RP: Similar to RP, with True and False switched.

QBP: Uses quantum stuff, otherwise similar to BPP.

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Complexity classes

The following are defined with respect to probabilistic turingmachines, and refer to decision problems

ZPP: Zero-error Probabilistic Polynomial time. The class ofproblems which can be solved in expected polynomial time.

BPP: Bounded-error Probabilistic Polynomial time. The classof problems which can be solved in expected polynomial timewith a probability of error of at most 1/3.

RP: Randomized Polynomial time The class of decisionproblems for which False is always returned if the answer isFalse, and True is returned in at least 1/2 of cases if theanswer is True. The polynomial time is expected value.

co− RP: Similar to RP, with True and False switched.

QBP: Uses quantum stuff, otherwise similar to BPP.

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Complexity classes

The following are defined with respect to probabilistic turingmachines, and refer to decision problems

ZPP: Zero-error Probabilistic Polynomial time. The class ofproblems which can be solved in expected polynomial time.

BPP: Bounded-error Probabilistic Polynomial time. The classof problems which can be solved in expected polynomial timewith a probability of error of at most 1/3.

RP: Randomized Polynomial time The class of decisionproblems for which False is always returned if the answer isFalse, and True is returned in at least 1/2 of cases if theanswer is True. The polynomial time is expected value.

co− RP: Similar to RP, with True and False switched.

QBP: Uses quantum stuff, otherwise similar to BPP.

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Complexity classes

The following are defined with respect to probabilistic turingmachines, and refer to decision problems

ZPP: Zero-error Probabilistic Polynomial time. The class ofproblems which can be solved in expected polynomial time.

BPP: Bounded-error Probabilistic Polynomial time. The classof problems which can be solved in expected polynomial timewith a probability of error of at most 1/3.

RP: Randomized Polynomial time The class of decisionproblems for which False is always returned if the answer isFalse, and True is returned in at least 1/2 of cases if theanswer is True. The polynomial time is expected value.

co− RP: Similar to RP, with True and False switched.

QBP: Uses quantum stuff, otherwise similar to BPP.

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Complexity classes

The following are defined with respect to probabilistic turingmachines, and refer to decision problems

ZPP: Zero-error Probabilistic Polynomial time. The class ofproblems which can be solved in expected polynomial time.

BPP: Bounded-error Probabilistic Polynomial time. The classof problems which can be solved in expected polynomial timewith a probability of error of at most 1/3.

RP: Randomized Polynomial time The class of decisionproblems for which False is always returned if the answer isFalse, and True is returned in at least 1/2 of cases if theanswer is True. The polynomial time is expected value.

co− RP: Similar to RP, with True and False switched.

QBP: Uses quantum stuff, otherwise similar to BPP.

Gwylim Ashley Probabilistic Algorithms

DefinitionsExamples

Theory

Complexity classes

The following are defined with respect to probabilistic turingmachines, and refer to decision problems

ZPP: Zero-error Probabilistic Polynomial time. The class ofproblems which can be solved in expected polynomial time.

BPP: Bounded-error Probabilistic Polynomial time. The classof problems which can be solved in expected polynomial timewith a probability of error of at most 1/3.

RP: Randomized Polynomial time The class of decisionproblems for which False is always returned if the answer isFalse, and True is returned in at least 1/2 of cases if theanswer is True. The polynomial time is expected value.

co− RP: Similar to RP, with True and False switched.

QBP: Uses quantum stuff, otherwise similar to BPP.

Gwylim Ashley Probabilistic Algorithms