algorithms and data structures

Algorithms and data structuresAlgorithms and data structures

Protected by http://creativecommons.org/licenses/by-nc-sa/3.0/hr/22.04.23

Creative CommonsCreative Commons

You are free to:You are free to: shareshare — copy and redistribute the material in any medium or format — copy and redistribute the material in any medium or format adaptadapt — remix, transform, and build upon the material — remix, transform, and build upon the material

Under the following terms: Under the following terms: Attribution Attribution — You must give— You must give appropriate credit, appropriate credit, provide a link to the license, and provide a link to the license, and

indicate if changes were madeindicate if changes were made. You may do so in any reasonable manner, but not . You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. in any way that suggests the licensor endorses you or your use.

NonCommercial NonCommercial — You may not use the material for— You may not use the material for commercial purposes commercial purposes. . ShareAlike ShareAlike — If you remix, transform, or build upon the material, you must — If you remix, transform, or build upon the material, you must

distribute your contributions under the same license as the original.distribute your contributions under the same license as the original.

No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

Text copied from http://creativecommons.org/licenses/by-nc-sa/3.0/

22.04.23Algorithms and data structures, FER

Notices: You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation. No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.

2 / 36

Sorting algorithmsSorting algorithms

22.04.23

Algorithms and data structures, FER 22.04.23

AlgorithmsAlgorithms

Selected for illustration:Selected for illustration:Selection sortSelection sortBubble sortBubble sortInsertion sortInsertion sortShell sortShell sortMerge sortMerge sortQuick sortQuick sortHeap sort - later!Heap sort - later!

Sortovi (Sorting)http://www.solidware.com/sort/

4 / 36


Selection sortSelection sort

Find the smallest element inFind the smallest element in the array and swap it withthe array and swap it with the first array elementthe first array element Repeat it with the rest of the array, while decreasing the unsorted portion of the Repeat it with the rest of the array, while decreasing the unsorted portion of the

arrayarray

66 44 11 88 77 55 33 2211 44 66 88 77 55 33 2211 22 66 88 77 55 33 4411 22 33 88 77 55 66 4411 22 33 44 77 55 66 8811 22 33 44 55 77 66 8811 22 33 44 55 66 77 8811 22 33 44 55 66 77 8811 22 33 44 55 66 77 88

5 / 36


Algorithm and complexityAlgorithm and complexity

Implementation Implementation - 2 - 2 loopsloops:: The outer loop determines the range of the sorted portion of the arrayThe outer loop determines the range of the sorted portion of the array The inner loop finds the minimum array elementThe inner loop finds the minimum array element

void SelectionSort (int A [], int N) {

int i, j, min;

for (i = 0; i < N; i++) {

min = i;

for (j = i+1; j < N; j++) {

if (A[j] < A[min]) min = j;

}

Swap(&A[i], &A[min]);

}

}

O(n-i-1)O(n-i-1)

1

0

( 1)n

i

O n i

6 / 36


Analysis of the execution timeAnalysis of the execution time

Comparison prevails; there are less swapsComparison prevails; there are less swaps The execution time does not depend on the initial arrangement, but The execution time does not depend on the initial arrangement, but

on the number of array elementson the number of array elements O(nO(n22)) – roughly – roughly nn22/2/2 comparisons and comparisons and nn swaps in an average swaps in an average

and in a worst caseand in a worst case Execution is not essentially faster if the input elements are closer to their Execution is not essentially faster if the input elements are closer to their

final placementsfinal placements Acceleration: Acceleration:

Start sStart sortortinging simultaneously from both ends simultaneously from both ends The worst case: reversely sorted sequenceThe worst case: reversely sorted sequence

7 / 36


Bubble sortBubble sort

The main idea: swapping of neighbouring elements if The main idea: swapping of neighbouring elements if they are they are in wrong in wrong sequencesequence Start from the array beginning and progress towards the endStart from the array beginning and progress towards the end Swap 2 elements if the first is larger than the second oneSwap 2 elements if the first is larger than the second one Acceleration: ifAcceleration: if no swap has occurredno swap has occurred while while traversing traversing throughthrough the whole the whole

array, the array is sortedarray, the array is sorted

8 / 36


Bubble sort - exampleBubble sort - example

6 4 1 8 7 5 3 24 6 1 8 7 5 3 24 1 6 8 7 5 3 24 1 6 8 7 5 3 24 1 6 7 8 5 3 24 1 6 7 5 8 3 24 1 6 7 5 3 8 24 1 6 7 5 3 2 8

4 1 6 7 5 3 2 81 4 6 7 5 3 2 81 4 6 7 5 3 2 81 4 6 7 5 3 2 81 4 6 5 7 3 2 81 4 6 5 3 7 2 81 4 6 5 3 2 7 8

1 4 6 5 3 2 7 81 4 6 5 3 2 7 81 4 6 5 3 2 7 81 4 5 6 3 2 7 81 4 5 3 6 2 7 81 4 5 3 2 6 7 8

1 4 5 3 2 6 7 81 4 5 3 2 6 7 81 4 5 3 2 6 7 81 4 3 5 2 6 7 81 4 3 2 5 6 7 8

1 4 3 2 5 6 7 81 4 3 2 5 6 7 81 3 4 2 5 6 7 81 3 2 4 5 6 7 8

1 3 2 4 5 6 7 81 3 2 4 5 6 7 81 2 3 4 5 6 7 8

1 2 3 4 5 6 7 81 2 3 4 5 6 7 8

1. traversal 2. traversal 3. traversal 4. traversal

5. traversal 6. traversal 7. traversal

9 / 36


AlgorithmAlgorithm

void BubbleSort (int A [], int N) {

int i, j;

for (i = 0; i < N-1; i++) {

for (j = 0; j < N-1-i; j++) {

if (A[j+1] < A[j])

Swap(&A[j], &A[j+1]);

}

}

}

O(n-1-i)O(n-1-i)

2

0

( 1)n

i

O n i

10 / 36


Improved bubble sortImproved bubble sort

If in aIf in a traversaltraversal no swapping has occurredno swapping has occurred, , the array is sortedthe array is sorted

void BubbleSort (int A [], int N) {

int i, j, SwapOccurred;

for (i = 0, SwapOccurred = 1; SwapOccurred; i++) {

SwapOccurred = 0;

for (j = 0; j < N-1-i; j++) {

if (A[j+1] < A[j]) {

Swap(&A[j], &A[j+1]);

SwapOccurred = 1;

}

}

}

}

11 / 36



O(nO(n22)) – – roughly roughly nn22/2/2 comparisons andcomparisons and nn22/2/2 swaps in an average swaps in an average and a worst caseand a worst case If the input elements are close to their final positions, sorting can be quickly If the input elements are close to their final positions, sorting can be quickly

completedcompleted The worst caseThe worst case: : reversely sorted sequencereversely sorted sequence The best caseThe best case: : already sorted sequencealready sorted sequence

The position of elements is essential for efficiencyThe position of elements is essential for efficiency Large elements at theLarge elements at the beginning of the beginning of the array are not a problem array are not a problem – – they they

quickly move towards the end - quickly move towards the end - hareshares Small elements at the end of array are a problem - they slowly progress Small elements at the end of array are a problem - they slowly progress

towards the begintowards the beginningning - - turtlesturtles

12 / 36

Algorithms and data structures, FER 22.04.2322.04.23

Insertion sortInsertion sort

The ideaThe idea: : there are two parts ofthere are two parts of an an arrayarray: : the sorted and the non-sorted the sorted and the non-sorted oneone In each step, the first element from the non-sorted partIn each step, the first element from the non-sorted part is inserted into is inserted into the the

sorted part at sorted part at the right placethe right place The way the cards are (usually) sorted in card gamesThe way the cards are (usually) sorted in card games

<=x<=x >x>x xx non-sortednon-sorted

<=x<=x >x>xxx non-sortednon-sorted

13 / 36


Insertion sort - exampleInsertion sort - example

66 44 11 88 77 55 33 2244 66 11 88 77 55 33 2211 44 66 88 77 55 33 2211 44 66 88 77 55 33 2211 44 66 77 88 55 33 2211 44 55 66 77 88 33 2211 33 44 55 66 77 88 2211 22 33 44 55 66 77 88

66 44 11 88 77 55 33 22

14 / 36


Algorithm and complexityAlgorithm and complexity

Implementation Implementation - 2 - 2 loopsloops:: The outer loop determines the range of the sorted portion of the arrayThe outer loop determines the range of the sorted portion of the array The inner loop inserts an element into the sorted part and shifts the rest of The inner loop inserts an element into the sorted part and shifts the rest of

elementselementsvoid InsertionSort (int A [], int N) {

int i, j;

int aux;

for (i = 1; i < N; i++) {

aux = A[i];

for (j = i; j >= 1 && A[j-1] > aux; j--)

A[j] = A[j-1];

A[j] = aux;

}

}

O(i)O(i)

1

0

( ( ))n

i

O O i

15 / 36



O(nO(n22)) – roughly – roughly nn22/4/4 comparisons and comparisons and nn22/4/4 swaps in an average swaps in an average case and case and double that muchdouble that much in in a a worst caseworst case

When the input elements are close to their final positions, it is When the input elements are close to their final positions, it is quickly completedquickly completed

Sorting is stableSorting is stable Elements with the same key value are not swappedElements with the same key value are not swapped if if aa i i bb bear the same key value and bear the same key value and aa was positioned before was positioned before bb, after a , after a

stable sort stable sort aa shall maintain its relative position to shall maintain its relative position to bb Best and worst cases?Best and worst cases?

The worst case: a reversely sorted arrayThe worst case: a reversely sorted array The best case: an already sorted arrayThe best case: an already sorted array

16 / 36


Shell sortShell sort

The oldest among quick sorting algoritms, a modified insertion sortThe oldest among quick sorting algoritms, a modified insertion sort author: Donald Shellauthor: Donald Shell

Idea:Idea: For a For a kk-sorted array -sorted array AA it is valid it is valid A[i] A[i] A [i + k], A [i + k], i, i+k i, i+k where where i i andand

i+k i+k are regular indicesare regular indices If an array is If an array is kk-sorted and then additionally -sorted and then additionally tt-sorted (-sorted (t<kt<k), it remains also ), it remains also kk--

sortedsorted Completely sorted array is Completely sorted array is 11-sorted-sorted

Generally, an incremental sequence of numbersGenerally, an incremental sequence of numbers hh11, h, h22, h, h33, … ,h, … ,ht t is used in is used in reverse orderreverse order Choice of the sequence is crucial for the algorithm efficiencyChoice of the sequence is crucial for the algorithm efficiency Animation: http://www.cis.fiu.edu/~weiss/Shellsort.htmlAnimation: http://www.cis.fiu.edu/~weiss/Shellsort.html

17 / 36


Shell sort – exampleShell sort – example

6 4 1 8 7 5 3 2

6 4 1 8 7 5 3 2

6 4 1 8 7 5 3 2

6 4 1 2 7 5 3 8

1 4 6 2 7 5 3 8

1 2 6 4 7 5 3 8

1 2 6 4 7 5 3 8

1 2 6 4 7 5 3 8

1 2 3 4 6 5 7 8

1 2 3 4 6 5 7 8

1 2 3 4 6 5 7 8

1 2 3 4 6 5 7 8

1 2 3 4 6 5 7 8

1 2 3 4 6 5 7 8

1 2 3 4 5 6 7 8

1 2 3 4 5 6 7 8

1 2 3 4 5 6 7 8

step= 4 step = 2 step = 1

18 / 36


AlgorithmAlgorithm

void ShellSort (int A [], int N) {

int i, j, step, aux;

for (step = N / 2; step > 0; step /= 2) {

for (i = step; i < N; i++) {

aux = A [i];

for (j = i; j >= step && A[j-step] > aux; j -= step) {

A [j] = A [j - step];

}

A [j] = aux;

}

}

}

19 / 36


Complexity analysisComplexity analysis

The average execution time has been an open (unsolved) problem for The average execution time has been an open (unsolved) problem for a long timea long time Worst case Worst case O(nO(n22))

Hibbard’s sequence: {1, 3, 7, …, 2Hibbard’s sequence: {1, 3, 7, …, 2kk -1} results in the worst case with -1} results in the worst case with O(nO(n3/23/2)) The average The average O(nO(n5/45/4)) has been determined by simulation; there is no proof for has been determined by simulation; there is no proof for

it yet!it yet! Sedgwick’s sequence: {1, 5, 19, 41, 109,…}, namely Sedgwick’s sequence: {1, 5, 19, 41, 109,…}, namely 9*49*4ii - 9*2 - 9*2ii + 1 + 1

alternating with alternating with 44ii - 3*2 - 3*2ii +1 +1 Worst case Worst case O(nO(n4/34/3)), and the average is , and the average is O(nO(n7/67/6))

It is not known whether a better sequence can be foundIt is not known whether a better sequence can be found A simple algorithm with an extremely complicated complexity analysisA simple algorithm with an extremely complicated complexity analysis

20 / 36


Comparison of sorting algorithms with complexityComparison of sorting algorithms with complexity O(nO(n22))

BubbleBubble

SelectionSelection

InsertionInsertion

ShellShell

21 / 36


MergesortMergesort

Divide-and-conquerDivide-and-conquer strategy is applied, with recursion strategy is applied, with recursion author: John von Neumann, in year 1945author: John von Neumann, in year 1945 Idea:Idea:

Un-sorted sequence is divided into two approximately equal partsUn-sorted sequence is divided into two approximately equal parts Each sub-array is sorted recursively, until the sub-array is reduced to a Each sub-array is sorted recursively, until the sub-array is reduced to a

single elementsingle element– This single element array is sorted!This single element array is sorted!

Two sorted sub-arrays are merged into a sorted arrayTwo sorted sub-arrays are merged into a sorted array– From two sorted arrays (A and B) a third array (C) is formedFrom two sorted arrays (A and B) a third array (C) is formed

By branching, By branching, loglog22 n n levels are created, and in each level a process of levels are created, and in each level a process of complexity complexity O(n)O(n) is performedis performed The execution time is The execution time is O(n logO(n log22 n) n)

22 / 36


MergesortMergesort - example - example

3131 2424 4747 11 66 7878 1212 6565

3131 2424 4747 11

3131 2424

3131 2424

2424 3131

4747 11

4747 11

11 4747

11 2424 3131 4747

66 7878 1212 6565

66 7878 1212 6565

66 7878 1212 6565

66 7878 1212 6565

66 1212 6565 7878

11 66 1212 2424 3131 4747 6565 7878

23 / 36


AlgorithmAlgorithm

Exercise: write the function Exercise: write the function MergeMerge The function merges the left and the right (approximate) half into a sorted The function merges the left and the right (approximate) half into a sorted

arrayarray

void MSort(int A [], int AuxArray[], int left, int right) {

int middle;

if (left < right) {

middle = left + (right - left) / 2;

MSort (A, AuxArray, left, middle);

MSort (A, AuxArray, middle + 1, right);

Merge (A, AuxArray, left, middle + 1, right);

}

}

24 / 36


RemarksRemarks

Price of faster sorting: memoryPrice of faster sorting: memory An auxiliary array is created!An auxiliary array is created!

Rarely used for sorting in core memory Rarely used for sorting in core memory Increased demand for additional memory and copyingIncreased demand for additional memory and copying

This is the principal algorithm for sorting using external memoryThis is the principal algorithm for sorting using external memory Complexity in the average and the worst case: Complexity in the average and the worst case: O(nlogO(nlog22n)n)

NoNott faster if the input sequence is already sorted! faster if the input sequence is already sorted!

25 / 36


QuicksortQuicksort

The fastest algorithm known so farThe fastest algorithm known so far Recursion: “divide-and-conquer“Recursion: “divide-and-conquer“

http://euler.slu.edu/~goldwasser/demos/quicksort/http://euler.slu.edu/~goldwasser/demos/quicksort/ http://www.cs.queensu.ca/home/cisc121/2004f/lecturenotes/malamb/SortingDemos/QuickSortDemo.htmlhttp://www.cs.queensu.ca/home/cisc121/2004f/lecturenotes/malamb/SortingDemos/QuickSortDemo.html

4 steps – 4 steps – quicksort (S)quicksort (S) If the number of members of array If the number of members of array SS equals 0 or 1, return to the calling equals 0 or 1, return to the calling

programprogram Choose any member Choose any member vv in the array in the array S.S. Let it be the pivot. Let it be the pivot. Distribute the remaining members of the array Distribute the remaining members of the array S, S \ {S, S \ {vv}} into two into two disjoint disjoint

sets:sets:– SS11 = { x = { x S \ { S \ {vv} } x x vv} } (everything less than pivot move to the left)(everything less than pivot move to the left)

– SS22 = { x = { x S \ { S \ {vv} } x x vv}} (everything bigger than pivot, move to the right(everything bigger than pivot, move to the right)) Return the array constructed from Return the array constructed from {quicksort (S{quicksort (S11), ), vv, quicksort (S, quicksort (S22)})}

26 / 36


Pivot selectionPivot selection

Not uniquely determinedNot uniquely determined Not uniquely determinedNot uniquely determined either what to do with array members equal to the either what to do with array members equal to the

pivotpivot The question of algorithm implementationThe question of algorithm implementation A part of good implementation is an efficient solution to this question, see:A part of good implementation is an efficient solution to this question, see:

– Weiss: "Data Structures and Algorithm Analysis in C". Weiss: "Data Structures and Algorithm Analysis in C".

Possible methods for pivot selection:Possible methods for pivot selection: Estimation of the median, based on 3 elements (the first element, the last element, the Estimation of the median, based on 3 elements (the first element, the last element, the

element in the middle of the array)element in the middle of the array)– At the estimation of the median, these elements are immediately sortedAt the estimation of the median, these elements are immediately sorted

Other possibilities: randomly chosen element, the first element, the last elementOther possibilities: randomly chosen element, the first element, the last element E.g. E.g. tthe array: 8 1 4 9 6 3 5 2 7 0he array: 8 1 4 9 6 3 5 2 7 0

– pivot = med3 (8, 6, 0) = 6pivot = med3 (8, 6, 0) = 6 What would be the worst pivot?What would be the worst pivot? What is the actual median?What is the actual median?

27 / 36


QuicksortQuicksort - example - example

8 1 4 9 6 3 5 2 7 0

^ ^ ^

0 1 4 9 6 3 5 2 7 8

0 1 4 9 7 3 5 2 6 8

i-> <-j 0 1 4 9 7 3 5 2 6 8

i j

0 1 4 2 7 3 5 9 6 8

Selection of the pivot

Pivot penultimate

i i j are bypassing

i j

0 1 4 2 5 3 7 9 6 8

0 1 4 2 5 3 7 9 6 8

j i

0 1 4 2 5 3 7 9 6 8

0 1 4 2 5 3 7 9 6 8

i

0 1 4 2 5 3 6 9 7 8

0 1 4 2 5 3

Selection of pivot

9 7 8

Selection of pivot

6

Returning the pivot to the position i

28 / 36


Algorithm complexityAlgorithm complexity

Average execution time is Average execution time is O(n log n)O(n log n) Sorting is very quick, mostly due to very efficient inner loopSorting is very quick, mostly due to very efficient inner loop

The worst case is The worst case is O(nO(n22)) For a wrong choice of pivot (min or max member), For a wrong choice of pivot (min or max member), nn partitions are obtained partitions are obtained

and for each of them the execution timeand for each of them the execution time isis O(n)O(n) It can be arranged that the probability of such a case exponentially It can be arranged that the probability of such a case exponentially

decreasesdecreases

29 / 36


Comparison of sorts with complexityComparison of sorts with complexity O(nlogn)O(nlogn)

HeapHeap

MergeMerge

QuickQuick

30 / 36


Sorting proceduresSorting procedures

Sorting of about a million of records is not rare in practiceSorting of about a million of records is not rare in practice If a single execution of a loop takes 1 If a single execution of a loop takes 1 s:s:

Classical sort would need about 10Classical sort would need about 1066 s (i.e. more than 11 days) s (i.e. more than 11 days)Quick sort takes about 20 sQuick sort takes about 20 s

The solution should not be always attempted in acquisition of faster The solution should not be always attempted in acquisition of faster and more expensive computersand more expensive computersInvestment in development and application of better algorithms Investment in development and application of better algorithms

may pay offmay pay off

31 / 36


Indirect sortingIndirect sorting

For sorting of large data structures it would not be efficient to swap For sorting of large data structures it would not be efficient to swap many recordsmany records Examples for such structuresExamples for such structures

– Student’s ID, family name, name, address, enrolled courses and Student’s ID, family name, name, address, enrolled courses and gradesgrades

If the data are sorted e.g. by the ID, then a separate array of IDs If the data are sorted e.g. by the ID, then a separate array of IDs is formed with adjoined pointers to the rest of the data. is formed with adjoined pointers to the rest of the data.

Only the formed array is sorted (using any appropriate algorithm)Only the formed array is sorted (using any appropriate algorithm)

32 / 36


ComparisonComparison

Sorting Sorting algorithmalgorithm

bestbest averageaverage worstworst stablestable methodmethod

selection sort O(n2) O(n2) O(n2) no selection

insertion sort O(n) O(n2) O(n2) yes insertion

bubble sort O(n) - O(n2) yes swapping

shell sort - - ? no insertion

merge sort O(nlogn) O(nlogn) O(nlogn) yes merging

quick sort O(nlogn) O(nlogn) O(n2) no division

heap sort O(nlogn) O(nlogn) O(nlogn) no selection

33 / 36


Animations of algorithmsAnimations of algorithms

http://www.geocities.com/SiliconValley/Network/1854/Sort1.htmlhttp://www.geocities.com/SiliconValley/Network/1854/Sort1.html http://www.solidware.com/sort/http://www.solidware.com/sort/ http://www.cs.hope.edu/~dershem/ccaa/animator/Animator.htmlhttp://www.cs.hope.edu/~dershem/ccaa/animator/Animator.html http://cg.scs.carleton.ca/~morin/misc/sortalg/http://cg.scs.carleton.ca/~morin/misc/sortalg/ http://homepages.dcc.ufmg.br/~dorgival/applets/SortingPoints/SortingPoints.htmlhttp://homepages.dcc.ufmg.br/~dorgival/applets/SortingPoints/SortingPoints.html http://www.cis.fiu.edu/~weiss/Shellsort.htmlhttp://www.cis.fiu.edu/~weiss/Shellsort.html http://www.educypedia.be/education/mathematicsjavasorting.htmhttp://www.educypedia.be/education/mathematicsjavasorting.htm

Google...Google...

34 / 36


ExercisesExercises

Write a program to find the Write a program to find the kkthth largest member in an integer array of largest member in an integer array of nn members. members.a)a) Sort the input array in descending sequence and print out the Sort the input array in descending sequence and print out the

member with index member with index k-1k-1..b)b) Enter Enter kk array members, sort them in descending sequence. array members, sort them in descending sequence.

Enter the remaining array members. If a member is smaller than Enter the remaining array members. If a member is smaller than the one with index the one with index k-1k-1, ignore it, if it is larger insert it into the , ignore it, if it is larger insert it into the appropriate place, and throw out the array member that would appropriate place, and throw out the array member that would obtain the index obtain the index k k now.now.

Apply various sorting algorithms and determine the Apply various sorting algorithms and determine the corresponding a priori execution times and measure the a corresponding a priori execution times and measure the a posteriori execution times.posteriori execution times.

35 / 36


ExercisesExercises

Write a program to merge two sequential sorted files into a third Write a program to merge two sequential sorted files into a third sorted sequential filesorted sequential file. .

UpariDatotekeUpariDatoteke (MergeFiles) (MergeFiles)

36 / 36

algorithms and data structures

Documents

data structuresprotected

fersorting algorithms

array elementrepeat

license permits

intended use

number of array elementson2

n swaps

neighbouring elements