ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
ECE750-TXB Lecture 5: Veni, Divisi, Vici(Divide and Conquer)
Todd L. [email protected]
Electrical & Computer EngineeringUniversity of Waterloo
Canada
February 14, 2007
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Divide And Conquer
BinarySearch is an (arguably degenerate) instance of abasic algorithm design pattern:
Divide And Conquer
1. If the problem is of trivial size, solve it and return.
2. Otherwise:
2.1 Divide the problem into several problems of smaller size.2.2 Conquer these smaller problems by recursively applying
the divide and conquer pattern.2.3 Combine the answers to the smaller problems into an
answer to the whole problem.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Divide and Conquer
Binary Search as Divide-and-Conquer:
1. If the problem is of size n = 1, answer is obvious —return.
2. Otherwise:I Split the array into two halves. Principle:
(x in A[l ..h]) ≡ (x in A[l ..i ]) ∨ (x in A[i + 1..j ])
I Search the two halves: for one half, call self recursively;for other half, answer is false.
I Combine the two answers: since one answer is alwaysfalse, and “x or false” is just “x ,” simply return theanswer from the half we searched.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Divide-and-Conquer Recurrences
If
1. The base cases (trivially small problems) require timeO(1), and
2. a problem of size n is split into k subproblems of sizes(n), and
3. splitting the problem into subproblems and combiningthe answers takes time f (n)
then the general form of the time recurrence is
T (n) = c + kT (s(n)) + f (n)
e.g. for binary search we had k = 1 (we only had to searchone half), s(n) = n/2, and f (n) = 0.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Strassen Matrix Multiplication
I Recall that our MatrixMultiply routine took Θ(n3)time.
I Can we do better? No one thought so until...
I A landmark paper:Volker Strassen, Gaussian Elimination is not optimal(1969). [1]
I An o(n3) divide-and-conquer approach to matrixmultiplication.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Strassen’s method
Strassen: “If A,B are matrices of order m2k+1 to bemultiplied, write
A =
[A11 A12
A21 A22
]B =
[B11 B12
B21 B22
]C =
[C11 C12
C21 C22
]where the Aik ,Bik ,Cik matrices are of order m2k ...
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Strassen’s method
“Then compute
(7 subproblems)
I = (A11 + A22)(B11 + B22)II = (A21 + A22)B11
III = A11(B12 − B22)IV = A22(−B11 + B21)V = (A11 + A12)B22
VI = (−A11 + A21)(B11 + B12)VII = (A12 − A22)(B21 + B22)
(and combine)
C11 = I + IV − V + VIIC21 = II + IVC12 = III + VC22 = I + III − II + VI
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Strassen’s method
1. Subproblems: compute 7 matrix multiplications of sizen/2.
2. Constructing the subproblems and combining theanswers is done with matrix additions/subtractions,taking Θ(n2) time.
3. Apply general divide-and-conquer recurrence withk = 7, s(n) = n/2, f (n) = Θ(n2):
T (n) = c + 7T (n/2) + Θ(n2)
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Strassen’s method
Recall that Θ(n2) means “some function f (n) for which f (n)n2
is eventually restricted to some finite positive interval[c1, c2].”Pick a value c > c2; then eventually f (n) ≤ cn2.Solve recurrence:
T (n) = c + 7T (n/2) + cn2
This will give an asymptotically correct bound, but possiblyT (n) is less than the actual time required for small n.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Strassen’s methodI Let r(ξ) = T (2ξ), and ξ = log2 n. Recurrence becomes
r(ξ) = c + 7r(ξ − 1) + c(2ξ)2
= c + 7r(ξ − 1) + c(4ξ)
I Z-transform:
Z [c] =c
1− z−1
Z [7r(ξ − 1)] = 7z−1R(z)
Z[c(4ξ)
]=
c
1− 4z−1
I Z-transform version of recurrence is:
R(z) =c
1− z−1+ 7z−1R(z) +
c
1− 4z−1
I Solve:
R(z) =c2(1− 5
2z−1)
(1− z−1)(1− 4z−1)(1− 7z−1)
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Strassen’s method
I Look at singularities: zero at z = 52 , poles at z = 1, 4, 7.
I Pole at z = 7 is asymptotically dominant:
r(ξ) ∼ c17ξ
I Change variables back: ξ = log2 n:
T (n) ∼ c17log2 n
= c1nlog2 7 = c1n
2.807···
I Strassen matrix multiplication takes Θ(n2.807···) time.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Divide-and-conquer recurrences I
I Our analysis of Strassen’s algorithm is easily generalized.
I Consider a divide-and-conquer recurrence of the form
T (n) = kT (n/s) + Θ(nd) (1)
I To obtain an asymptotic upper bound we can solve therelated recurrence
T ′(n) = kT ′(n/s) + cnd (2)
I It can be shown that T (n) � T ′(n).
I Analyzing for the case when n = sξ with ξ ∈ N, we canchange variables to turn Eqn. (2) into:
r(ξ) = kr(ξ − 1) + c(sd)ξ
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Divide-and-conquer recurrences III Z-transform:
R(z) = kz−1R(z) +c
1− sdz−1
I Solve:
R(z) =c
(1− kz−1)(1− sdz−1)
I Which is the dominant pole? Depends on the values ofk, s, d .
I Three cases: sd < k, sd = k, sd > k.
1. sd < k. Then the dominant pole is z = k. Get
r(ξ) � kξ
Since n = sξ, use ξ = logs n:
T ′(n) � k logs n = (2log k)log nlog s = (2log n)
log klog s = n
log klog s
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Divide-and-conquer recurrences III
2. sd = k. Then get a double pole at z = k. Recall that
Z−1
[kz−1
(1− kz−1)2
]= ξkξ
End up with
r(ξ) � ξkξ
T ′(n) � (logs n)k logs n � (log n)2log k·log n
log s = nlog klog s log n
= nlog sd
log s log n = nd log n
3. sd > k. Then dominant singularity is z = sd . Get
r(ξ) � (sd)ξ
T ′(n) � (sd)logs n = (2d log s)log nlog s = nd
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
‘Master’ Theorem
Theorem (‘Master’)
The solution to a divide-and-conquer recurrence of the form
T (n) = kT (dn/se) + Θ(nd)
where s > 1, is
T (n) =
Θ
(n
log klog s
)if sd < k
Θ(nd log n
)if sd = k
Θ(nd) if sd > k
I Examples:I Binary search: k=1, s=2, d=0: second case, log k
log s = 0,
so T (n) = Θ(n0 log n) = Θ(log n).I Strassen: k = 7, s = 2, d = 2: first case, log k
log s ≈ 2.807,
so T (n) = Θ(n2.807···).
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
BibliographyAbstract Data Types
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Abstract Data Types
I A basic software engineering principle:
Separate the interface (what you can do) fromthe implementation (how it is done.)
I An abstract data type is a an interface to a collection ofdata.
I There may be numerous ways to implement an ADT,each with different performance characteristics.
I An ADT consists of
1. Some types that may be required to provide operations,relations, and satisfy properties. e.g. a totally orderedset.
2. An interface: the capabilities provided by the ADT.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Abstract Data Types
I Notation for types: 〈T , f1, f2, · · · ,R1,R2, · · · 〉 indicatesa set T together with
I Some operators or functions f1, f2, . . .I Some relations R1,R2, · · · .
This is the standard notation for a structure in logic:could be
I an algebra (functions but no relations) e.g. a field〈F ,+, ∗,−, ·−1, 0, 1〉;
I a relational structure (relations but no functions) e.g. atotal order 〈T ,≤〉;
I some structure with both functions and relations e.g. anordered field 〈F ,+, ∗,−, ·−1, 0, 1,≤〉
I Often a type is required to satisfy certain axioms, orbelong to a specified class of structures, e.g. (a field, atotal order, a distance metric.)
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
ADT: Dictionary[K , V ]
Dictionary[K,V]
I Stores a set of pairs (key, value), and finds values bykey. At most one value per key is permitted.
I Types:I 〈K 〉: key type (e.g. a word).I 〈V , 0〉: value type (e.g., a definition of a word). The
value 0 is a special value used to indicate absence of adictionary entry.
I Operations:I insert(k, v): insert a key-value pair into the dictionary.
If an entry for the key is already present, it is replaced.I V find(k): if (k, v) is in the dictionary, returns v ;
otherwise returns 0.I remove(k): deletes the entry for key k, if present.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Abstract Data Types
ADTImplementationData structure
Linked list
Dictionary
55kkkkkkkkk//
))SSSSSSSSS Sorted array
Search tree
Implementation insert time find time remove time
Linked list O(n) O(n) O(n)Sorted array O(n) O(log n) O(n)Search tree O(log n) O(log n) O(log n)
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
The role of ADTs in choosing data structures
Questions to consider when choosing a data structure:
1. What type of data do you need to store?I Does it have a natural total order? e.g. integers, strings
under lexicographic ordering, database keys, etc.I Does it bear some natural partial order?I Do elements represent points or regions in some metric
space, e.g., Rn? (e.g., screen regions, boxes in athree-dimensional space, etc.)
The order relation(s) or geometric organization of yourdata may allow the use of ADTs that exploit thoseproperties to allow efficient access.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
The role of ADTs in choosing data structures
2. What operations are required? Do you need toI determine whether a value is in the data set (search)?I insert, modify, delete elements?I iterate through the elements (order doesn’t matter)?I iterate through the elements in some sorted order?I find the “biggest” or “smallest” element?I find elements that are “close” to some value?
These requirements can be compared with theinterfaces provided by ADTs, to decide what ADTsmight be suitable.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
The role of ADTs in choosing data structures
3. What is the typical mixture of operations you willperform? Will you
I insert frequently?I delete frequently?I search frequently?
Different ADT implementations may offer differentperformance characteristics. By understanding thetypical mixture of operations you can choose animplementation with the most suitable performancecharacteristics.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
ADT: Array[V]
Array[V]
I A finite sequence of cells, each containing a value.I Types :
I 〈V , 0〉: a value type, with a “default value” 0 used toinitialize cells.
I Operations :I Array(n): create an array of length n, where n ∈ N is a
positive integer. Cells are initialized to 0.I integer length(): returns the length of the arrayI get(i): returns the value of cell i . It is required that
0 ≤ i ≤ length()− 1.I set(i , v): sets the value of cell i to v . It is required that
0 ≤ i ≤ length()− 1.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
ADT: Set[V]
Set[V]
I Stores a set of values. Permits inserting, removing, andtesting whether a value is in the set. At most oneinstance of a value can be in the set.
I Types:I 〈V 〉: a value type
I Operations:I insert(v): adds v to the set, if absent. Inserting an
element already in the set causes no change.I remove(v): removes v from the set, if present;I boolean contains(v): returns true if and only if v is in
the set.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
ADT: Multiset[V]
Multiset[V]
I Stores a multiset of values (i.e., with duplicate elementspermitted.) Permits inserting, removing, and testingwhether a value is in the set. Sometimes called a bag.
I Types:I 〈V 〉: a value type
I Operations:I insert(v): adds v to the multiset.I remove(v): removes an instance of v from the multiset,
if present;I boolean contains(v): returns true if and only if v is in
the set.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Stacks, Queues, and Priority Queues
I Informally, a queue is a collection of objects “awaitingtheir turn.” e.g., customers queueing at a grocery store.The queueing policy governs “who goes next.”
I First In, First Out (FIFO): like a line at the grocerystore: the element that was added least recently goesnext.
I Last In, First Out (LIFO): the item added most recentlygoes next. A stack: like an “in-box” of work where newitems are placed on the top, and what ever is on the topof the stack gets processed next.
I Priority Queueing: items are associated with priorities;the item with the highest priority goes next.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
ADT: Queue[V]
Queue[V]
I A FIFO queue (first in, first out) of objects.I Types :
I 〈V 〉: a value type
I Operations :I insert(v): adds the object v to the end of the queueI boolean isEmpty(): returns true just when the queue is
emptyI V next(): returns and removes the value at the front of
the queue. It is an error to perform this operation whenthe queue is empty.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
ADT: Stack[V]
Stack[V]
I A LIFO (last in, first out) stack of objects.I Types :
I 〈V 〉: a value type
I Operations :I push(v): adds a value to the top of the stack.I boolean isEmpty(): returns true just when the stack
contains no elements.I V pop(): returns the value at the top of the stack. It is
an error to perform this operation when the stack isempty.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
ADT: PriorityQueue[P,V]
PriorityQueue[P,V]
I A queue of objects, each with an associated priority, inwhich an object with maximal priority is always chosennext.
I Types :I 〈P,≤〉: a priority type, with a total order ≤.I 〈V 〉: a value type
I Operations :I insert(p, v): insert a pair (p, v) where p ∈ P is a
priority, and v ∈ V is a value;I boolean isEmpty(): returns true just when the queue is
empty;I (P,V ) next(): returns and removes the object at the
front of the queue, which is guaranteed to havemaximal priority. It is an error to perform this operationon an empty queue.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Data Structure: Linked List
I Realizes a list of items, e.g., (2, 3, 5, 7, 11)
I Inserting and removing elements at the front of the listrequires O(1) time.
I Searching requires iterating through the list: O(n) time
I List can be iterated through from front to back.I Basic building block is a node that contains:
I data: A piece of data;I next: A pointer to the next node, or a null pointer if
the node is the end of the list.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Data Structure: Linked List
public class LinkedList<T> {Node<T> head; // Pointer to the first node in the list
...}
class Node<T> {T data; // Data itemNode<T> next; // Next node in the list , if any
Node(T data, Node<T> next){
data = data;next = next;
}}
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Data Structure: Linked List I
I Insert a new data element:
public void insert (T data){
head = new Node<T>(data, head);}
I Remove the front element, if any:
public T removeFirst(){
if (head == null)throw new RuntimeException(”List is empty.”);
else {T data = head.data;head = head.next;return data;
}}
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Data Structure: Linked List II
I Check if the list is empty:
public boolean isEmpty(){
return (head == null);}
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Implementing Stack[V ] with a Linked List
I The ADT Stack[V ] is naturally implemented by alinked list.
public class Stack<V> {LinkedList<V> list;
public void push(V v) { list . insert (v); }public boolean isEmpty() { return list . isEmpty(); }public V pop() { return list . removeFirst (); }}
I push(v), isEmpty(), and pop() require O(1) time.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Iterators
Iterator[V]
I An iterator is an ADT that provides traversal throughsome container. It abstracts away from the details ofhow the data are stored, and presents a simple interfacefor retrieving the elements of the container one at atime.
I Types:I V : the value type stored in the container
I Operations:I Iterator(C ): initialize the iterator to point to the first
element in the container C ;I boolean hasNext(): returns true just when there is
another element in the container;I V next(): returns the next element in the container,
and advances the iterator.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
An Iterator for a linked list
public class ListIterator <T> {Node<T> node;
ListIterator ( LinkedList<T> list){ node = list .head; }
boolean hasNext(){ return (node == null); }
T next(){
if (node == null)throw new RuntimeException(”Tried to iterate past end of list ”);
T data = node.data;node = node.next;return data;
}}
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Iterators
Example usage of an iterator:
ListIterator <Integer> iter = new ListIterator<Integer>( list );while ( iter .hasNext()){
System.out. println (”The next element is ” + iter .next ());}
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Implementing Queue[V ] with a Linked List
I Recall that a Queue[V ] ADT requires first-in first-outqueueing. But, the list as we have shown it onlysupports inserting and removing at one end.
I Simple variant of a linked list: a in addition tomaintaining a pointer to the head of the list, alsomaintain a pointer to the tail of the list.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Bidirectional Linked List
class BiNode<T> {T data; // Data itemBiNode<T> next; // Next node in the list , if anyBiNode<T> prev; // Previous node in the list , if any
BiNode(T data, BiNode<T> next){
data = data;next = next;next.prev = this;
}}
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Bidirectional Linked List
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Bidirectional Linked List
public class BiLinkedList<T> {BiNode<T> head; // Pointer to the first node in the listBiNode<T> tail; // Pointer to the last node in the list
public void insert (T data){
head = new BiNode<T>(data, head);if ( tail == null)
tail = head;}
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Bidirectional Linked List
public T removeLast(){
if ( tail == null)throw new RuntimeException(”List is empty.”);
else {T data = tail .data;if ( tail .prev == null){
head = null;tail = null;
}else {
tail = tail .prev ;tail .next = null;
}return data;
}}
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Implementing a Queue < V > with aBidirectional Linked List
public class Queue<V> {BiLinkedList<V> list;
public void insert (V v) { list . insert (v); }public boolean isEmpty() { return list . isEmpty(); }public V next() { return list .removeLast(); }
}
I insert(v), isEmpty() and next() all take O(1) time.
ECE750-TXBLecture 5: Veni,
Divisi, Vici
Todd L.Veldhuizen
Divide andConquer
Abstract DataTypes
Bibliography
Bibliography I
[1] V. Strassen.Gaussian elimination is not optimal.Numerische Mathematik, 13:354–356, 1969. bib pdf