Clustering Technology

Clustering Schematic

Cluster Components

• Cluster hardware (processor, main memory, hard disk, …)

• Cluster network (Fast Ethernet, Gigabit Ethernet, Myrinet, …)

• Cluster Software (operating system, programming environment, …)

Cluster Operating System Characteristics

• Manageability: An absolute necessity is remote and intuitive system administration; this is often associated with a Single System Image (SSI) which can be realized on different levels, ranging from a high-level set of special scripts down to real state-sharing on the OS level.

• Stability: The most important characteristics are robustness against crashing processes, failure recovery by dynamic reconfiguration, and usability under heavy load.

• Performance: The performance critical parts of the OS, such as memory management, process and thread scheduler, file I/O and communication protocols should work in as efficiently as possible.

• Extensibility: The OS should allow the easy integration of cluster-specific extensions, which will most likely be related to the inter-node cooperation. A good example for this is the MOSIX system that is based on Linux.

• Scalability: The scalability of a cluster is mainly influenced by the provision of the contained nodes, which is dominated by the performance characteristics of the interconnect.

• Support: Many intelligent and technically superior approaches in computing failed due to the lack of support in its various aspects: which tools, hardware drivers and middleware environments are available.

• Heterogeneity: Clusters provide a dynamic and evolving environment in that they can be extended or updated with standard hardware just as the user needs to or can afford. Therefore, a cluster environment does not necessarily consist of homogenous hardware.

Cluster Solution

• Hardware of Cluster nodes is typical PC’s (this selection reduces the cost per performance ratio).

• Fast Ethernet was used as Cluster interconnection (17 nodes connected by Fast Ethernet infrastructure).

• Linux was choosed as Cluster OS to provide our needs as much as possible (we can recompile and tune the kernel to meet our needs).

• Using VMware software for virtualizing our computing resources.

• Message Passing Interface was selected as the parallel programming environment.

Configuring Cluster

• Configuring the Cluster nodes (network configuration, packages installation, …).

• Optimizing and securing the Linux OS to extract the maximum utilization from Cluster resources.

• Cluster administration (Samba service, ssh, rlogin, rcp, administration scripts and …).

Algorithm Identification

Integer Factorization

Sieving

Trial Division

QS Algorithm

MPQS Algorithm

SIQS Algorithm

Algorithm Complexity Improvement

QS MPQS SIQS NFS

OptimizingSerial Implementation

Optimizing Serial Implementation

• Algorithm level optimizations (the most important step for optimizing serial codes is to reduce the complexity of algorithm maximally).

• Code level optimizations (in this phase we use

• Compiler level optimizations

Algorithm Optimization

• In computation-intensive software programs, we will often find that 99% of the CPU time is used in the innermost loop.

• Identifying the most critical part of your software is therefore necessary if you want to improve the speed of computation (by profilers).

• Study the algorithm used in the critical part of your code and see if it can be improved.

Innermost loop(Conventional Sieving)

Pentium4 Memory Access Times

An Optimized Sieving Approach (1)

An Optimized Sieving Approach (2)

Code level optimizationtechniques

• Loop unrolling (Unrolling amortizes the branch overhead, since it eliminates branches and some of the code to manage induction variables. Unrolling allows you to aggressively schedule (or pipeline) the loop to hide latencies).

• Function inlining (We can instruct the compiler to insert the code of a function into the code of its callers, to the point where actually the call is to be made. inlining method reduces the function-call overhead. In a compiler, inlining a function exposes more opportunity for optimization).

• gcc inline assembly (Assembly routines written as inline functions. They are handy, speedy and very much useful).

• Builtin gcc functions (__builtin_prefetch) (This function is used to minimize cache-miss latency by moving data into a cache before it is accessed).

• Using “unsigned int” type only (Use 32-bit integers instead of integers with smaller sizes (16-bit or 8-bit) to reduce the machine cycles needed).

• Division-free arithmetic (change division to use multiplication by reciprocals).

• Release allocated memory blocks.• and …

Loop unrolling (code level opt.)

gcc compiler optimizations

Parallel Algorithm Design

Parallel Algorithm Design Methodology

• Partitioning (Domain decomposition or Functional Decomposition)

• Communication

• Agglomeration

• Mapping

Methodical Design (1)

• Partitioning: The computation that is to be performed and the data operated on by this computation are decomposed into small tasks. Practical issues such as the number of processors in the target computer are ignored, and attention is focused on recognizing opportunities for parallel execution.

• Communication: The communication required to coordinate task execution is determined, and appropriate communication structures and algorithms are defined.

• Agglomeration: The task and communication structures defined in the first two stages of a design are evaluated with respect to performance requirements and implementation costs. If necessary, tasks are combined into larger tasks to improve

performance or to reduce development costs. • Mapping: Each task is assigned to a processor in a manner that attempts to

satisfy the competing goals of maximizing processor utilization and minimizing communication costs. Mapping can be specified statically or determined at runtime by load-balancing algorithms.

Methodical Design (2)

Load Balancing Mechanism

• For load balancing Master/Slave mechanism was used.

• Master node sends the initial data and assign the jobs to slave nodes.

Data Decomposition Algorithm

• Using SPMD model (SPMD program that creates exactly one task per processor).

• We can sieve with multiple polynomials in SIQS.• To generate these polynomials, we must first

compute ‘a’ factors.• Sieving with separated ‘a’ values can be done

independently on different processors.• Thus we need to build ‘a’ values on several

tasks without any coordination.• Duplicated ‘a’ factor conclude to weak

concurrency.

Data Decomposition Algorithm (1)(Initialization Data)

Data Decomposition Algorithm (2)(Determining the number

and size of ‘a’ value’s factors)

Data Decomposition Algorithm (2)(Determining the number

and size of ‘a’ value’s factors)

Data Decomposition Algorithm (3)(Computing the factors of ‘a’ values)

Data Decomposition Algorithm (3)(Computing the factors of ‘a’ values)

Master Node Algorithm

Slave Node Algorithm

Double Large PrimeVariation effects

Master Node Algorithm (1)(Improved version)

Master Node Algorithm (2) (Improved version)

Slave Node Algorithm (Improved version)

Cluster Benchmarks

Performance Evaluation

• Speedup: Amdahl’s law gives the ideal speedup Sp:

• Efficiency: The efficiency, p, of a p-node computation with speed-up Sp is given by:

Total Execution Time

Sieving Execution Time

Total Speedup

Sieving Speedup

Total Efficiency

Sieving Efficiency

Sources of inefficiency

• Static load balancing (overestimate our needs and waste Cluster resources).

• Communication overhead (Ethernet protocol and tcp/ip stack operations).

• Load imbalance.• Non parallelized stages of the application such

as Linear algebra stage (Admahl’s law).• Inefficient software and hardware technologies

(MPI overhead and OS inefficiencies).

Conclusions and Future works

• The distributed computing by Cluster technology is an open problem yet. The Mosix project is one of the most successful solutions that builds an single system image and concrete Linux kernels to simulate one single system for running processes.

• But Mosix and other similar solutions not work optimally for every desired application.

• In fact supplying a SSI at the operating system level, while a definite boon in terms of manageability, drastically inhibits scalability.

• The availability of the source code in conjunction with the possibility to extend (and thus modify) the operating system on this base. This property has a negative influence on the stability and manageability of the system: over time, many variants of the operating system will develop, and the different extensions may conflict when there is no single supplier.

Conclusions and Future works

• In this thesis our goal was to present an global approach to run computational applications on distributed systems optimally.

• Algorithm optimization is the major step in presented approach. In the computation-intensive software programs, we must in first step identify the most critical part of the software and try to optimize this algorithm as much as possible (most efficient step).

• In the second phase we implement the best optimized algorithm serially to exploit the maximum usage of local resources. In fact we do distributed computing when local computation cost is bigger than remote execution).

• In the last stage, parallelize the serial algorithm optimally. You must focus on most computational part of the serial algorithm and try to parallelize this part efficiently.

Conclusions and Future works

• The performance benchmarks shows that our strategy works well, and therefore we can apply this method to every other application similarly.

• Using NFS algorithm for RSA key cracking.

• Parallelize linear algebra stage for factorization of very large numbers (greater than 120 digits).