multiplication of large integers -...

Heap Sort Algorithm

Heap Sort is a popular and efficient sorting algorithm in computer programming. Learning how to write the heap sort algorithm requires knowledge of two types of data structures - arrays and trees.

The initial set of numbers that we want to sort is stored in an array e.g. [10, 3, 76, 34, 23, 32] and after sorting, we get a sorted array [3,10,23,32,34,76]

Heap sort works by visualizing the elements of the array as a special kind of complete binary tree called heap.

What is a complete Binary Tree?Binary Tree

A binary tree is a tree data structure in which each parent node can have at most two children

In the above image, each element has at most two children.

Full Binary Tree

A full Binary tree is a special type of binary tree in which every parent node has either two or no children.

Complete binary tree

A complete binary tree is just like a full binary tree, but with two major differences

1. Every level must be completely filled2. All the leaf elements must lean towards the left.3. The last leaf element might not have a right sibling i.e. a complete binary tree

doesn’t have to be a full binary tree.

How to create a complete binary tree from an unsorted list (array)?

Select first element of the list to be the root node. (First level - 1 element) Put the second element as a left child of the root node and the third element as a

right child. (Second level - 2 elements) Put next two elements as children of left node of second level. Again, put the

next two elements as children of right node of second level (3rd level - 4 elements).

Keep repeating till you reach the last element.

Relationship between array indexes and tree elementsComplete binary tree has an interesting property that we can use to find the children and parents of any node.

If the index of any element in the array is i, the element in the index 2i+1 will become the left child and element in 2i+2 index will become the right child. Also, the parent of any element at index i is given by the lower bound of (i-1)/2.

Let’s test it out,

Left child of 1 (index 0)

= element in (2*0+1) index

= element in 1 index

= 12

Right child of 1



= 9

Similarly,

Left child of 12 (index 1)



= 5

Right child of 12



= 6

Let us also confirm that the rules holds for finding parent of any node

Parent of 9 (position 2)

= (2-1)/2

= ½

= 0.5

~ 0 index

= 1

Parent of 12 (position 1)

= (1-1)/2

= 0 index

= 1

Understanding this mapping of array indexes to tree positions is critical to understanding how the Heap Data Structure works and how it is used to implement Heap Sort.

What is Heap Data Structure ?Heap is a special tree-based data structure. A binary tree is said to follow a heap data structure if

it is a complete binary tree All nodes in the tree follow the property that they are greater than their children

i.e. the largest element is at the root and both its children and smaller than the root and so on. Such a heap is called a max-heap. If instead all nodes are smaller than their children, it is called a min-heap

Following example diagram shows Max-Heap and Min-Heap.

How to “heapify” a treeStarting from a complete binary tree, we can modify it to become a Max-Heap by running a function called heapify on all the non-leaf elements of the heap.

Since heapfiy uses recursion, it can be difficult to grasp. So let’s first think about how you would heapify a tree with just three elements.

heapify(array)

Root = array[0]

Largest = largest( array[0] , array [2*0 + 1]. array[2*0+2])

if(Root != Largest)

Swap(Root, Largest)

The example above shows two scenarios - one in which the root is the largest element and we don’t need to do anything. And another in which root had larger element as a child and we needed to swap to maintain max-heap property.

If you’re worked with recursive algorithms before, you’ve probably identified that this must be the base case.

Now let’s think of another scenario in which there are more than one levels.

The top element isn’t a max-heap but all the sub-trees are max-heaps.

To maintain the max-heap property for the entire tree, we will have to keep pushing 2 downwards until it reaches its correct position.

Thus, to maintain the max-heap property in a tree where both sub-trees are max-heaps, we need to run heapify on the root element repeatedly until it is larger than its children or it becomes a leaf node.

We can combine both these conditions in one heapify function as

void heapify(int arr[], int n, int i)

{

int largest = i;

int l = 2*i + 1;

int r = 2*i + 2;

if (l < n && arr[l] > arr[largest])

largest = l;

if (right < n && arr[r] > arr[largest])

largest = r;

if (largest != i)

{

swap(arr[i], arr[largest]);

// Recursively heapify the affected sub-tree

heapify(arr, n, largest);

}

}

This function works for both the base case and for a tree of any size. We can thus move the root element to the correct position to maintain the max-heap status for any tree size as long as the sub-trees are max-heaps.

Build max-heap

To build a max-heap from any tree, we can thus start heapifying each sub-tree from the bottom up and end up with a max-heap after the function is applied on all the elements including the root element.

In the case of complete tree, the first index of non-leaf node is given by n/2 - 1. All other nodes after that are leaf-nodes and thus don’t need to be heapified.

So, we can build a maximum heap as

// Build heap (rearrange array)

for (int i = n / 2 - 1; i >= 0; i--)

heapify(arr, n, i);

As show in the above diagram, we start by heapifying the lowest smallest trees and gradually move up until we reach the root element.

If you’ve understood everything till here, congratulations, you are on your way to mastering the Heap sort.

Procedures to follow for Heapsort1. Since the tree satisfies Max-Heap property, then the largest item is stored at the

root node.2. Remove the root element and put at the end of the array (nth position) Put the

last item of the tree (heap) at the vacant place.3. Reduce the size of the heap by 1 and heapify the root element again so that we

have highest element at root.4. The process is repeated until all the items of the list is sorted.

The code below shows the operation.

for (int i=n-1; i>=0; i--)

{

// Move current root to end

swap(arr[0], arr[i]);

// call max heapify on the reduced heap

heapify(arr, i, 0);

}

PerformanceHeap Sort has O(nlogn) time complexities for all the cases ( best case, average case and worst case).

Let us understand the reason why. The height of a complete binary tree containing n elements is log(n)

As we have seen earlier, to fully heapify an element whose subtrees are already max-heaps, we need to keep comparing the element with its left and right children and pushing it downwards until it reaches a point where both its children are smaller than it.

In the worst case scenario, we will need to move an element from the root to the leaf node making a multiple of log(n) comparisons and swaps.

During the build_max_heap stage, we do that for n/2 elements so the worst case complexity of the build_heap step is n/2*log(n) ~ nlogn.

During the sorting step, we exchange the root element with the last element and heapify the root element. For each element, this again takes logn worst time because we might have to bring the element all the way from the root to the leaf. Since we repeat this n times, the heap_sort step is also nlogn.

Also since the build_max_heap and heap_sort steps are executed one after another, the algorithmic complexity is not multiplied and it remains in the order of nlogn.

Also it performs sorting in O(1) space complexity. Comparing with Quick Sort, it has better worst case ( O(nlogn) ). Quick Sort has complexity O(n^2) for worst case. But in other cases, Quick Sort is fast. Introsort is an alternative to heapsort that combines

quicksort and heapsort to retain advantages of both: worst case speed of heapsort and average speed of quicksort.

Application of Heap SortSystems concerned with security and embedded system such as Linux Kernel uses Heap Sort because of the O(n log n) upper bound on Heapsort's running time and constant O(1)upper bound on its auxiliary storage.

Although Heap Sort has O(n log n) time complexity even for worst case, it doesn’t have more applications ( compared to other sorting algorithms like Quick Sort, Merge Sort ). However, its underlying data structure, heap, can be efficiently used if we want to extract smallest (or largest) from the list of items without the overhead of keeping the remaining items in the sorted order. For e.g Priority Queues.

Heap Sort Implementation in different Programming LanguagesC++ Implementation// C++ program for implementation of Heap Sort

#include <iostream>

using namespace std;

void heapify(int arr[], int n, int i)

{

// Find largest among root, left child and right child

int largest = i;

int l = 2*i + 1;

int r = 2*i + 2;

if (l < n && arr[l] > arr[largest])

largest = l;

if (right < n && arr[r] > arr[largest])

largest = r;

// Swap and continue heapifying if root is not largest

if (largest != i)

{

swap(arr[i], arr[largest]);

heapify(arr, n, largest);

}

}

// main function to do heap sort

void heapSort(int arr[], int n)

{

// Build max heap

for (int i = n / 2 - 1; i >= 0; i--)

heapify(arr, n, i);

// Heap sort

for (int i=n-1; i>=0; i--)

{

swap(arr[0], arr[i]);

// Heapify root element to get highest element at root again

heapify(arr, i, 0);

}

}

void printArray(int arr[], int n)

{

for (int i=0; i<n; ++i)

cout << arr[i] << " ";

cout << "\n";

}

int main()

{

int arr[] = {1,12,9,5,6,10};

int n = sizeof(arr)/sizeof(arr[0]);

heapSort(arr, n);

cout << "Sorted array is \n";

printArray(arr, n);

}

Multiplication of Large IntegersSome applications, notably modern cryptography, require manipulation of inte-gers that are over 100 decimal digits long. Since such integers are too long to fit in a single word of a modern computer, they require special treatment.

Multiplication of Large Integers and Strassen’s Matrix MultiplicationIn this section, we examine two surprising algorithms for seemingly straightfor-ward tasks: multiplying two integers and multiplying two square matrices. Both achieve a better asymptotic efficiency by ingenious application of the divide-and-conquer technique.

Multiplication of Large Integers

Some applications, notably modern cryptography, require manipulation of inte-gers that are over 100 decimal digits long. Since such integers are too long to fit in a single word of a modern computer, they require special treatment. This practi-cal need supports investigations of algorithms for efficient manipulation of large integers. In this section, we outline an interesting algorithm for multiplying such numbers. Obviously, if we use the conventional pen-and-pencil algorithm for mul-tiplying twon-digit integers, each of the n digits of the first number is multiplied by each of the n digits of the second number for the total of n2 digit multiplications. (If one of the numbers has fewer digits than the other, we can pad the shorter number with leading zeros to equalize their lengths.) Though it might appear that it would be impossible to design an algorithm with fewer than n2 digit multiplica-tions, this turns out not to be the case. The miracle of divide-and-conquer comes to the rescue to accomplish this feat.

To demonstrate the basic idea of the algorithm, let us start with a case of two-digit integers, say, 23 and 14. These numbers can be represented as follows:

23 = 2 . 101 + 3 . 100 and 14 = 1 . 101 + 4 . 100.

Now let us multiply them:

23 ∗ 14 = (2 . 101 + 3 . 100) ∗ (1 . 101 + 4 . 100)

= (2 ∗ 1)102 + (2 ∗ 4 + 3 ∗ 1)101 + (3 ∗ 4)100.

The last formula yields the correct answer of 322, of course, but it uses the same four digit multiplications as the pen-and-pencil algorithm. Fortunately, we can compute the middle term with just one digit multiplication by taking advantage of the products 2 ∗ 1 and 3 ∗ 4 that need to be computed anyway:

2 ∗ 4 + 3 ∗ 1 = (2 + 3) ∗ (1 + 4) − 2 ∗ 1 − 3 ∗ 4.

Of course, there is nothing special about the numbers we just multiplied. For any pair of two-digit numbers a = a1a0 and b = b1b0, their product c can be computed by the formula

c = a ∗ b = c2102 + c1101 + c0,

where

c2 = a1 ∗ b1 is the product of their first digits,

c0 = a0 ∗ b0 is the product of their second digits,

c1 = (a1 + a0) ∗ (b1 + b0) − (c2 + c0) is the product of the sum of the a’s digits and the sum of theb’s digits minus the sum of c2 and c0.

Now we apply this trick to multiplying two n-digit integers a and b where n is a positive even number. Let us divide both numbers in the middle—after all, we promised to take advantage of the divide-and-conquer technique. We denote the first half of the a’s digits by a1 and the second half bya0; for b, the notations are b1 and b0, respectively. In these notations, a = a1a0 implies that a =a110n/2 + a0 and b = b1b0 implies that b = b110n/2 + b0. Therefore, taking advantage of the same trick we used for two-digit numbers, we get

= a ∗ b = (a110n/2 + a0) ∗ (b110n/2 + b0)

(a1 ∗ b1)10n + (a1 ∗ b0 + a0 ∗ b1)10n/2 + (a0 ∗ b0)

c210n + c110n/2 + c0,

where

c2 = a1 ∗ b1 is the product of their first halves,

c0 = a0 ∗ b0 is the product of their second halves,

c1 = (a1 + a0) ∗ (b1 + b0) − (c2 + c0) is the product of the sum of the

a’s halves and the sum of the b’s halves minus the sum of c2 and c0.

If n/2 is even, we can apply the same method for computing the products c2, c0, and c1. Thus, if nis a power of 2, we have a recursive algorithm for computing the product of two n-digit integers. In its pure form, the recursion is stopped when n becomes 1. It can also be stopped when we deem nsmall enough to multiply the numbers of that size directly.

How many digit multiplications does this algorithm make? Since multiplica-tion of n-digit numbers requires three multiplications of n/2-digit numbers, the recurrence for the number of multiplications M(n) is

M(n) = 3M(n/2) for n > 1, M(1) = 1.

Solving it by backward substitutions for n = 2k yields

M(2k) = 3M(2k−1) = 3[3M(2k −2)] = 32M(2k−2) = . . . = 3iM(2

k−i) = . . . = 3

kM(2

k−k) = 3

k.

Since k = log2 n,

M(n) = 3log2 n = nlog2 3 ≈ n1.585

.

(On the last step, we took advantage of the following property of logarithms:

alogb c = c

logb a.)

But what about additions and subtractions? Have we not decreased the num-ber of multiplications by requiring more of those operations? Let A(n) be the number of digit additions and subtractions executed by the above algorithm in multiplying two n-digit decimal integers. Besides 3A(n/2) of these operations needed to compute the three products of n/2-digit numbers, the above formulas

5.4 Multiplication of Large Integers and Strassen’s Matrix Multiplication

require five additions and one subtraction. Hence, we have the recurrence

A(n) = 3A(n/2) + cn for n > 1, A(1) = 1.

Applying the Master Theorem, which was stated in the beginning of the chapter, we obtain A(n) ∈ (nlog2 3), which means that the total number of additions and subtractions have the same asymptotic order of growth as the number of multipli-cations.

The asymptotic advantage of this algorithm notwithstanding, how practical is it? The answer depends, of course, on the computer system and program quality implementing the algorithm, which might explain the rather wide disparity of reported results. On some machines, the divide-and-conquer algorithm has been reported to outperform the conventional method on numbers only 8 decimal digits long and to run more than twice faster with numbers over 300 decimal digits long—the area of particular importance for modern cryptography. Whatever this outperformance “crossover point” happens to be on a particular machine, it is worth switching to the conventional algorithm after the

multiplicands become smaller than the crossover point. Finally, if you program in an object-oriented language such as Java, C++, or Smalltalk, you should also be aware that these languages have special classes for dealing with large integers.

Discovered by 23-year-old Russian mathematician Anatoly Karatsuba in 1960, the divide-and-conquer algorithm proved wrong the then-prevailing opinion that the time efficiency of any integer multiplication algorithm must be in (n2). The discovery encouraged researchers to look for even (asymptotically) faster algorithms for this and other algebraic problems. We will see such an algorithm in the next section.

multiplication of large integers -...

Documents