ece750-txb lecture 5: veni, divisi, vici - (divide and

ECE750-TXBLecture 5: Veni,

Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

ECE750-TXB Lecture 5: Veni, Divisi, Vici(Divide and Conquer)

Todd L. [email protected]

Electrical & Computer EngineeringUniversity of Waterloo

Canada

February 14, 2007


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Divide And Conquer

BinarySearch is an (arguably degenerate) instance of abasic algorithm design pattern:

Divide And Conquer

1. If the problem is of trivial size, solve it and return.

2. Otherwise:

2.1 Divide the problem into several problems of smaller size.2.2 Conquer these smaller problems by recursively applying

the divide and conquer pattern.2.3 Combine the answers to the smaller problems into an

answer to the whole problem.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Divide and Conquer

Binary Search as Divide-and-Conquer:

1. If the problem is of size n = 1, answer is obvious —return.

2. Otherwise:I Split the array into two halves. Principle:

(x in A[l ..h]) ≡ (x in A[l ..i ]) ∨ (x in A[i + 1..j ])

I Search the two halves: for one half, call self recursively;for other half, answer is false.

I Combine the two answers: since one answer is alwaysfalse, and “x or false” is just “x ,” simply return theanswer from the half we searched.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Divide-and-Conquer Recurrences

If

1. The base cases (trivially small problems) require timeO(1), and

2. a problem of size n is split into k subproblems of sizes(n), and

3. splitting the problem into subproblems and combiningthe answers takes time f (n)

then the general form of the time recurrence is

T (n) = c + kT (s(n)) + f (n)

e.g. for binary search we had k = 1 (we only had to searchone half), s(n) = n/2, and f (n) = 0.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Strassen Matrix Multiplication

I Recall that our MatrixMultiply routine took Θ(n3)time.

I Can we do better? No one thought so until...

I A landmark paper:Volker Strassen, Gaussian Elimination is not optimal(1969). [1]

I An o(n3) divide-and-conquer approach to matrixmultiplication.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Strassen’s method

Strassen: “If A,B are matrices of order m2k+1 to bemultiplied, write

A =

[A11 A12

A21 A22

]B =

[B11 B12

B21 B22

]C =

[C11 C12

C21 C22

]where the Aik ,Bik ,Cik matrices are of order m2k ...


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Strassen’s method

“Then compute

(7 subproblems)

I = (A11 + A22)(B11 + B22)II = (A21 + A22)B11

III = A11(B12 − B22)IV = A22(−B11 + B21)V = (A11 + A12)B22

VI = (−A11 + A21)(B11 + B12)VII = (A12 − A22)(B21 + B22)

(and combine)

C11 = I + IV − V + VIIC21 = II + IVC12 = III + VC22 = I + III − II + VI


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Strassen’s method

1. Subproblems: compute 7 matrix multiplications of sizen/2.

2. Constructing the subproblems and combining theanswers is done with matrix additions/subtractions,taking Θ(n2) time.

3. Apply general divide-and-conquer recurrence withk = 7, s(n) = n/2, f (n) = Θ(n2):

T (n) = c + 7T (n/2) + Θ(n2)


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Strassen’s method

Recall that Θ(n2) means “some function f (n) for which f (n)n2

is eventually restricted to some finite positive interval[c1, c2].”Pick a value c > c2; then eventually f (n) ≤ cn2.Solve recurrence:

T (n) = c + 7T (n/2) + cn2

This will give an asymptotically correct bound, but possiblyT (n) is less than the actual time required for small n.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Strassen’s methodI Let r(ξ) = T (2ξ), and ξ = log2 n. Recurrence becomes

r(ξ) = c + 7r(ξ − 1) + c(2ξ)2

= c + 7r(ξ − 1) + c(4ξ)

I Z-transform:

Z [c] =c

1− z−1

Z [7r(ξ − 1)] = 7z−1R(z)

Z[c(4ξ)

]=

c

1− 4z−1

I Z-transform version of recurrence is:

R(z) =c

1− z−1+ 7z−1R(z) +

c

1− 4z−1

I Solve:

R(z) =c2(1− 5

2z−1)

(1− z−1)(1− 4z−1)(1− 7z−1)


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Strassen’s method

I Look at singularities: zero at z = 52 , poles at z = 1, 4, 7.

I Pole at z = 7 is asymptotically dominant:

r(ξ) ∼ c17ξ

I Change variables back: ξ = log2 n:

T (n) ∼ c17log2 n

= c1nlog2 7 = c1n

2.807···

I Strassen matrix multiplication takes Θ(n2.807···) time.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Divide-and-conquer recurrences I

I Our analysis of Strassen’s algorithm is easily generalized.

I Consider a divide-and-conquer recurrence of the form

T (n) = kT (n/s) + Θ(nd) (1)

I To obtain an asymptotic upper bound we can solve therelated recurrence

T ′(n) = kT ′(n/s) + cnd (2)

I It can be shown that T (n) � T ′(n).

I Analyzing for the case when n = sξ with ξ ∈ N, we canchange variables to turn Eqn. (2) into:

r(ξ) = kr(ξ − 1) + c(sd)ξ


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Divide-and-conquer recurrences III Z-transform:

R(z) = kz−1R(z) +c

1− sdz−1

I Solve:

R(z) =c

(1− kz−1)(1− sdz−1)

I Which is the dominant pole? Depends on the values ofk, s, d .

I Three cases: sd < k, sd = k, sd > k.

1. sd < k. Then the dominant pole is z = k. Get

r(ξ) � kξ

Since n = sξ, use ξ = logs n:

T ′(n) � k logs n = (2log k)log nlog s = (2log n)

log klog s = n

log klog s


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Divide-and-conquer recurrences III

2. sd = k. Then get a double pole at z = k. Recall that

Z−1

[kz−1

(1− kz−1)2

]= ξkξ

End up with

r(ξ) � ξkξ

T ′(n) � (logs n)k logs n � (log n)2log k·log n

log s = nlog klog s log n

= nlog sd

log s log n = nd log n

3. sd > k. Then dominant singularity is z = sd . Get

r(ξ) � (sd)ξ

T ′(n) � (sd)logs n = (2d log s)log nlog s = nd


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

‘Master’ Theorem

Theorem (‘Master’)

The solution to a divide-and-conquer recurrence of the form

T (n) = kT (dn/se) + Θ(nd)

where s > 1, is

T (n) =

Θ

(n

log klog s

)if sd < k

Θ(nd log n

)if sd = k

Θ(nd) if sd > k

I Examples:I Binary search: k=1, s=2, d=0: second case, log k

log s = 0,

so T (n) = Θ(n0 log n) = Θ(log n).I Strassen: k = 7, s = 2, d = 2: first case, log k

log s ≈ 2.807,

so T (n) = Θ(n2.807···).


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

BibliographyAbstract Data Types


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Abstract Data Types

I A basic software engineering principle:

Separate the interface (what you can do) fromthe implementation (how it is done.)

I An abstract data type is a an interface to a collection ofdata.

I There may be numerous ways to implement an ADT,each with different performance characteristics.

I An ADT consists of

1. Some types that may be required to provide operations,relations, and satisfy properties. e.g. a totally orderedset.

2. An interface: the capabilities provided by the ADT.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Abstract Data Types

I Notation for types: 〈T , f1, f2, · · · ,R1,R2, · · · 〉 indicatesa set T together with

I Some operators or functions f1, f2, . . .I Some relations R1,R2, · · · .

This is the standard notation for a structure in logic:could be

I an algebra (functions but no relations) e.g. a field〈F ,+, ∗,−, ·−1, 0, 1〉;

I a relational structure (relations but no functions) e.g. atotal order 〈T ,≤〉;

I some structure with both functions and relations e.g. anordered field 〈F ,+, ∗,−, ·−1, 0, 1,≤〉

I Often a type is required to satisfy certain axioms, orbelong to a specified class of structures, e.g. (a field, atotal order, a distance metric.)


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

ADT: Dictionary[K , V ]

Dictionary[K,V]

I Stores a set of pairs (key, value), and finds values bykey. At most one value per key is permitted.

I Types:I 〈K 〉: key type (e.g. a word).I 〈V , 0〉: value type (e.g., a definition of a word). The

value 0 is a special value used to indicate absence of adictionary entry.

I Operations:I insert(k, v): insert a key-value pair into the dictionary.

If an entry for the key is already present, it is replaced.I V find(k): if (k, v) is in the dictionary, returns v ;

otherwise returns 0.I remove(k): deletes the entry for key k, if present.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Abstract Data Types

ADTImplementationData structure

Linked list

Dictionary

55kkkkkkkkk//

))SSSSSSSSS Sorted array

Search tree

Implementation insert time find time remove time

Linked list O(n) O(n) O(n)Sorted array O(n) O(log n) O(n)Search tree O(log n) O(log n) O(log n)


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

The role of ADTs in choosing data structures

Questions to consider when choosing a data structure:

1. What type of data do you need to store?I Does it have a natural total order? e.g. integers, strings

under lexicographic ordering, database keys, etc.I Does it bear some natural partial order?I Do elements represent points or regions in some metric

space, e.g., Rn? (e.g., screen regions, boxes in athree-dimensional space, etc.)

The order relation(s) or geometric organization of yourdata may allow the use of ADTs that exploit thoseproperties to allow efficient access.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography


2. What operations are required? Do you need toI determine whether a value is in the data set (search)?I insert, modify, delete elements?I iterate through the elements (order doesn’t matter)?I iterate through the elements in some sorted order?I find the “biggest” or “smallest” element?I find elements that are “close” to some value?

These requirements can be compared with theinterfaces provided by ADTs, to decide what ADTsmight be suitable.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography


3. What is the typical mixture of operations you willperform? Will you

I insert frequently?I delete frequently?I search frequently?

Different ADT implementations may offer differentperformance characteristics. By understanding thetypical mixture of operations you can choose animplementation with the most suitable performancecharacteristics.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

ADT: Array[V]

Array[V]

I A finite sequence of cells, each containing a value.I Types :

I 〈V , 0〉: a value type, with a “default value” 0 used toinitialize cells.

I Operations :I Array(n): create an array of length n, where n ∈ N is a

positive integer. Cells are initialized to 0.I integer length(): returns the length of the arrayI get(i): returns the value of cell i . It is required that

0 ≤ i ≤ length()− 1.I set(i , v): sets the value of cell i to v . It is required that

0 ≤ i ≤ length()− 1.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

ADT: Set[V]

Set[V]

I Stores a set of values. Permits inserting, removing, andtesting whether a value is in the set. At most oneinstance of a value can be in the set.

I Types:I 〈V 〉: a value type

I Operations:I insert(v): adds v to the set, if absent. Inserting an

element already in the set causes no change.I remove(v): removes v from the set, if present;I boolean contains(v): returns true if and only if v is in

the set.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

ADT: Multiset[V]

Multiset[V]

I Stores a multiset of values (i.e., with duplicate elementspermitted.) Permits inserting, removing, and testingwhether a value is in the set. Sometimes called a bag.

I Types:I 〈V 〉: a value type

I Operations:I insert(v): adds v to the multiset.I remove(v): removes an instance of v from the multiset,

if present;I boolean contains(v): returns true if and only if v is in

the set.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Stacks, Queues, and Priority Queues

I Informally, a queue is a collection of objects “awaitingtheir turn.” e.g., customers queueing at a grocery store.The queueing policy governs “who goes next.”

I First In, First Out (FIFO): like a line at the grocerystore: the element that was added least recently goesnext.

I Last In, First Out (LIFO): the item added most recentlygoes next. A stack: like an “in-box” of work where newitems are placed on the top, and what ever is on the topof the stack gets processed next.

I Priority Queueing: items are associated with priorities;the item with the highest priority goes next.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

ADT: Queue[V]

Queue[V]

I A FIFO queue (first in, first out) of objects.I Types :

I 〈V 〉: a value type

I Operations :I insert(v): adds the object v to the end of the queueI boolean isEmpty(): returns true just when the queue is

emptyI V next(): returns and removes the value at the front of

the queue. It is an error to perform this operation whenthe queue is empty.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

ADT: Stack[V]

Stack[V]

I A LIFO (last in, first out) stack of objects.I Types :

I 〈V 〉: a value type

I Operations :I push(v): adds a value to the top of the stack.I boolean isEmpty(): returns true just when the stack

contains no elements.I V pop(): returns the value at the top of the stack. It is

an error to perform this operation when the stack isempty.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

ADT: PriorityQueue[P,V]

PriorityQueue[P,V]

I A queue of objects, each with an associated priority, inwhich an object with maximal priority is always chosennext.

I Types :I 〈P,≤〉: a priority type, with a total order ≤.I 〈V 〉: a value type

I Operations :I insert(p, v): insert a pair (p, v) where p ∈ P is a

priority, and v ∈ V is a value;I boolean isEmpty(): returns true just when the queue is

empty;I (P,V ) next(): returns and removes the object at the

front of the queue, which is guaranteed to havemaximal priority. It is an error to perform this operationon an empty queue.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Data Structure: Linked List

I Realizes a list of items, e.g., (2, 3, 5, 7, 11)

I Inserting and removing elements at the front of the listrequires O(1) time.

I Searching requires iterating through the list: O(n) time

I List can be iterated through from front to back.I Basic building block is a node that contains:

I data: A piece of data;I next: A pointer to the next node, or a null pointer if

the node is the end of the list.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Data Structure: Linked List

public class LinkedList<T> {Node<T> head; // Pointer to the first node in the list

...}

class Node<T> {T data; // Data itemNode<T> next; // Next node in the list , if any

Node(T data, Node<T> next){

data = data;next = next;

}}


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Data Structure: Linked List I

I Insert a new data element:

public void insert (T data){

head = new Node<T>(data, head);}

I Remove the front element, if any:

public T removeFirst(){

if (head == null)throw new RuntimeException(”List is empty.”);

else {T data = head.data;head = head.next;return data;

}}


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Data Structure: Linked List II

I Check if the list is empty:

public boolean isEmpty(){

return (head == null);}


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Implementing Stack[V ] with a Linked List

I The ADT Stack[V ] is naturally implemented by alinked list.

public class Stack<V> {LinkedList<V> list;

public void push(V v) { list . insert (v); }public boolean isEmpty() { return list . isEmpty(); }public V pop() { return list . removeFirst (); }}

I push(v), isEmpty(), and pop() require O(1) time.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Iterators

Iterator[V]

I An iterator is an ADT that provides traversal throughsome container. It abstracts away from the details ofhow the data are stored, and presents a simple interfacefor retrieving the elements of the container one at atime.

I Types:I V : the value type stored in the container

I Operations:I Iterator(C ): initialize the iterator to point to the first

element in the container C ;I boolean hasNext(): returns true just when there is

another element in the container;I V next(): returns the next element in the container,

and advances the iterator.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

An Iterator for a linked list

public class ListIterator <T> {Node<T> node;

ListIterator ( LinkedList<T> list){ node = list .head; }

boolean hasNext(){ return (node == null); }

T next(){

if (node == null)throw new RuntimeException(”Tried to iterate past end of list ”);

T data = node.data;node = node.next;return data;

}}


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Iterators

Example usage of an iterator:

ListIterator <Integer> iter = new ListIterator<Integer>( list );while ( iter .hasNext()){

System.out. println (”The next element is ” + iter .next ());}


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Implementing Queue[V ] with a Linked List

I Recall that a Queue[V ] ADT requires first-in first-outqueueing. But, the list as we have shown it onlysupports inserting and removing at one end.

I Simple variant of a linked list: a in addition tomaintaining a pointer to the head of the list, alsomaintain a pointer to the tail of the list.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Bidirectional Linked List

class BiNode<T> {T data; // Data itemBiNode<T> next; // Next node in the list , if anyBiNode<T> prev; // Previous node in the list , if any

BiNode(T data, BiNode<T> next){

data = data;next = next;next.prev = this;

}}


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography



Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography


public class BiLinkedList<T> {BiNode<T> head; // Pointer to the first node in the listBiNode<T> tail; // Pointer to the last node in the list

public void insert (T data){

head = new BiNode<T>(data, head);if ( tail == null)

tail = head;}


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography


public T removeLast(){

if ( tail == null)throw new RuntimeException(”List is empty.”);

else {T data = tail .data;if ( tail .prev == null){

head = null;tail = null;

}else {

tail = tail .prev ;tail .next = null;

}return data;

}}


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Implementing a Queue < V > with aBidirectional Linked List

public class Queue<V> {BiLinkedList<V> list;

public void insert (V v) { list . insert (v); }public boolean isEmpty() { return list . isEmpty(); }public V next() { return list .removeLast(); }

}

I insert(v), isEmpty() and next() all take O(1) time.


Divisi, Vici

Todd L.Veldhuizen

[email protected]

Divide andConquer

Abstract DataTypes

Bibliography

Bibliography I

[1] V. Strassen.Gaussian elimination is not optimal.Numerische Mathematik, 13:354–356, 1969. bib pdf

http://kanushu.uwaterloo.ca/~tveldhui/lib/Year/1969.complete.html#Strassen:NM:1969

http://kanushu.uwaterloo.ca/~tveldhui/documents/1969/Strassen:NM:1969.pdf

ece750-txb lecture 5: veni, divisi, vici - (divide and

Documents