1998 - thesis jl pacherie parallel perators

A Pattern System for A Pattern System for Enumeration Processing Enumeration Processing

Expression and ParallelisationExpression and ParallelisationJean-Lin Pacherie

ContentsContents• Introduction

• Conclusion

• Design Model for Processing• Pattern System

• Impact of the Data Driven Distribution (DDD) • Services for Data Parallelism

IntroductionIntroduction

Motivations

Pattern and Parallelism

Ideas and Thesis

IntroductionIntroduction

Design Model for Processing

Pattern System

Impact of the DDD

Fundamental Services for Data Parallelism

Conclusion

MotivationsMotivations

Parallel Programming, Why?

Decrease computation duration

Increase data size

Deriving sequential applications (DDD)

MotivationsMotivations

Parallel Programming, Why?

Parallel Programming, How?

Direct parallel programming

MotivationsMotivations

Parallel Programming, Why?

The solution should conforms with usual programming environment

Constraints for the solution

Standard compilation tools

No language extensions

Parallel Programming, How?

MotivationsMotivations

Parallel Programming, Why?

Constraints for the solution

Parallel Programming, How?

Available tools

Abstraction

Encapsulation

Polymorphism

Software architecture with Patterns

MotivationsMotivations

Parallel Programming, Why?

Constraints for the solution

Parallel Programming, How?

Available tools

Evolution

Encapsulating parallelism in a sequential OOL [ Jézéquel 91 ]

Method & Framework PALADIN [ Guidec 95 ]

Reusable expression of methods [ Pacherie 97 ]

RequirementsRequirementsExperts and Reusability

Beyond the software component reusability : Pattern

Where is expressed the knowledge in an application?

What kind of tools for reusability?

Library

OO Library

ComponentDesign

FrameworkFramework

Method

Pattern

Pattern Language

Architecture

Knowledge and ReusabilityKnowledge and Reusability

Evolution

Reusability level

RequirementsRequirementsKnowledge and Reusability

Beyond the software components reusability

Pattern

[ Gamma 94 ]

Express architecture to parallelize

Express common architecture involved in parallel mechanism

RequirementRequirementKnowledge and ReusabilityData Parallelism

Context : Data collection processing

1) Cut-out

From iteration start until iteration end loop action ( current item ) next iterationend

Collection :Aggregation of homogeneous items

RequirementsRequirementsKnowledge and ReusabilityData Parallelism

Increasing Datasize

Decreasing computationduration

1) Cut out

2) Layout

3) // Processing

RequirementsRequirementsKnowledge and ReusabilityData ParallelismData Driven Distribution

Parallel Code

Collection

Sequential Code ?

Code Proc 2Code Proc 2

Code Proc 1Code Proc 1

Code Proc 3Code Proc 3

Data LayoutOwner Compute ruleRemote Data RefreshingSPMD

Principles

ThesisThesis

Design patterns for data parallelism programming

Identify and organize the design pattern dedicatedto data collection processing

Apply the DDD mechanism to these patterns:Parallelization at the software architecture level

Use the design patterns of serialUse the design patterns of serialapplications to introduce parallelismapplications to introduce parallelism

Design Model for EnumerationDesign Model for EnumerationProcessingProcessing

Operator Design Pattern

Consequences

Introduction

Design Model for ProcessingDesign Model for Processing

Pattern System

Impact of the DDD

Fundamental Services for Data Parallelism

Conclusion

Towards a pattern...Towards a pattern...

Description of behavior

Behavior vs.. Implementation

Instaciation delegation trough the Factory Method pattern

AbstractClient

FactoryMethod : AbstractProduct

AnOperation

AbstractProduct

Service

…product := FactoryMethodproduct.service...

ConcreteClient

FactoryMethod : ConcreteProduct

ConcreteProduct

Service

!ConcreteProduct!Result

AbstractSolver

make_matrix : Matrix

solve

Matrix

Item

ConcreteSolver

make_matrix : SparseMatrix

SparseMatrix

Item

Towards a pattern...Towards a pattern...

Description of behavior

Behavior vs. Implementation

Instaciation delegation trough the Factory Method pattern

Factory Pattern

Extracting iteration domain through the Iterator pattern

ArrayIteratorArray

ConcreteClient

Independence from data structures

AbstractClient

Collection Iterator

collection := make_collectioniteration := collection.make_iterator from iteration.start until iteration.exhausted loop … iteration.nextend

Towards a pattern...Towards a pattern...

Description of behavior

AbstractClient

Collection Iterator

ArrayIteratorArray

ConcreteClient

Extracting iteration domain through the Iterator patternIndependence from data structures

Towards a pattern...Towards a pattern...

Description of behavior

Factory Pattern

Extracting control structure

AbstractClient

Collection Operator

collection := make_collectioniteration := collection.make_iterator from iteration.start until iteration.exhausted loop … iteration.nextend

Iterator

Extracting iteration domain through the Iterator pattern

Towards a pattern...Towards a pattern...

Description of behavior

AbstractClient

Collection Operator

collection := make_collectioniteration := collection.make_iterator operator.attach(iteration)operator.run

Iterator Operator

Extracting control structure

Extracting iteration domain through the Iterator pattern

Towards a pattern...Towards a pattern...

Description of behavior

ConsequencesConsequences

Separation of the responsibilitiesCollection, Iterator and Operators

The modularity encourage reusability [Meyer 97]

Redefining implementationUsing the factory method pattern

Data Collections

Processing operators

Iteration domains

Client Product

Application Collection

Operator

IteratorCollection

Application

ConsequencesConsequences

Separation of the responsibilities

Processing ???!!

ProcessingProcessing

StructuralStructural OperatorOperator

CommandCommand RequestRequest

EnumerativeEnumerative AtomicAtomic

Collections Enumerations

No results With results

List of items One item

Redefining implementation

ConsequencesConsequences

Separation of the responsibilities

Operators ?

Enumeration Free Monoïd [ A* , ( ) , . ]

Operator Morphism toward [ P , e , + ]

••••[A*, ( ) , . ]

••••[P*, ( ) , . ]

+ + + +[ P , e , + ]

RValuation +Valuation +

List of A

List of P

. +

f : A P

Redefining implementation

ConsequencesConsequences

Separation of the responsibilities

Operators !

MAP MAP A,B

CROSS CROSS A,B,C

FILTER FILTER A

REDUCE REDUCE A

FORALL FORALL A

BMFBMF

Eval_Morphism Eval_Morphism A,P

Iterator Iterator A

Morphism Morphism A,P

Homomorphism Homomorphism A,B

Operator_C Operator_C A,B

Command & RequestCommand & Request

Iterator Iterator B Morphism Morphism

A,P

Operator_NC Operator_NC A,P

Redefining implementation

ConsequencesConsequences

Separation of the responsibilities

Pattern SystemPattern System

Designing collections and iterators

Expressing Operators

Using Enumerations

Introduction

Design Model for Processing

Pattern SystemPattern System

Impact of the DDD

Fundamental Services for Data Parallelism

Conclusion

MethodMethodIdentification of significant characteristics

Accessor

CapacityIndexed

Hashed

LinearIndexed

Dictionary Bounded

Dynamic

FIFO

FILO

Classification with familiesRelations between characteristics

Specification of abstractions (Collections & Iterators)

A

B

C

a

b

c Abstr

action

MethodMethodIdentification of significant characteristicsClassification with familiesRelations between characteristics

Application to collectionsApplication to collectionsdeferred class ACCESSORfeature General definitions end

deferred class ACCESSORfeature General definitions end

FamilyFamilyFamilies: Abstract classes

Characteristic

deferred class INDEXED [ITEM,INDEX] inherit ACCESSORfeature read ( i : INDEX ) : ITEM is deferred end write ( i : INDEX ; e : ITEM ) is deferred endend

deferred class INDEXED [ITEM,INDEX] inherit ACCESSORfeature read ( i : INDEX ) : ITEM is deferred end write ( i : INDEX ; e : ITEM ) is deferred endend

Characteristics: Progressive specialization

deferred class COLLECTION [ ITEM ] inherit ACCESSOR CAPACITY ../.. -- Other familiesfeature ../..end

deferred class COLLECTION [ ITEM ] inherit ACCESSOR CAPACITY ../.. -- Other familiesfeature ../..end

collection

AccesseurAccesseur

capacitycapacity

Collections: Abstract classes using families and specializationusing characteristics

Mix-inMix-in

Indexed

Indexed_Iterator

Writable_Iterator

2Way_Iterator

Indexed

Indexed

Writable

Writable

Two_Way

Two_Way

CharacteristicsCharacteristics

accessors

Iterator

FamiliesFamilies

Application to iteratorsApplication to iterators

• Similar approach

• Characteristics: from the collections, Iterator’s own

• Designed with multiple inheritance

The type constraints are expressed using characteristics combination and not using collection types

Remark :

Collection

Building an OperatorBuilding an OperatorOperator properties • Items type

• First order function

• Second order function

• Type of enumeration

Implementation • Generic type

• Virtual method (deferred)

• Classification (BMF, etc.)

• Iterator type

Pattern for IteratorsPattern for IteratorsComposable operator and Iterator

Compositions

Factorization

static

dynamic

Multiplexer Demultiplexer

Impact of the DDDImpact of the DDD

Deriving the operator pattern

Equivalence Constraint

Introduction

Design Model for Processing

Pattern System

Impact of the DDD Impact of the DDD

Fundamental Services for Data Parallelism

Conclusion

Deriving the Operator patternDeriving the Operator patternReminder : Operator structure

Redefinition of collections : data distributionParallelisation of processing

AbstractClient

AbstractCollection OperatorIterator

collection := make_collectioniteration := collection.make_iterator operator.attach(iteration)operator.run

ConcreteOperator

ParaOperator

ConcreteIterator

SpreadIterator

SpreadCollection

Enumeration of local data

Enumeration of remote data

Conformity constraintsSpread collections and Local collectionsSequential and Parallel OperatorsSequential and Parallel execution

Deriving the Operator patternDeriving the Operator patternReminder : Operator structure

DerivationFactory method : select distributed services

Invocation : local computations

ParaOperator

Spread_Collection

ParaClient

make_collection : Spread_Collection

make_operation : ParaOperator

SeqClient

make_collection : ACollection

AnOperation

make_operation : AnOperatorConcreteOperator

Collection

collection := make_collectionoperator := make_operator(collection) operator.run

make_operation ( collection) : AnOperator isdo !!Result Result.attach(Collection.items)end

make_operation ( collection) : ParaOperator isdo !!Result Result.attach(Collection.local_items)end

Deriving the Operator patternDeriving the Operator patternReminder : Operator structure Conformity constraints

Parallel Evaluation SchemeParallel Evaluation Scheme

[A*, () , . ]

[ P , e , + ]

F

[ A**, () , . ][ A**, () , . ]DistDist

[ P*, () , . ]

F*

Aggregation

• Morphism (f,e,+)• Distribution

Data :

Intermediate :

• Computation of the local contributions

Result :

• Aggregation of the local contributions

Equivalence ConstraintEquivalence ConstraintIs there a problem???

Equivalence ConstraintAggregation order of local contributions

Particular situationsPropriety of morphismsCommutative operator

Morphism

+F ( ) F ( )

+{F ( ) F ( )} F ( )+

F ( ) F ( ) F ( )// Evaluation

Aggregation

Order of aggregation = initial order

Is there a problem???

Equivalence ConstraintEquivalence Constraint

F ( ) F ( ) F ( )//Evaluation

Inner composition rule is commutative

& : Commutative

&F ( ) F ( )

&{F ( ) F ( )} F ( )&Aggregation

Order of aggregation =/= initial order

& F ( ) F ( )=?=

Particular situationsPropriety of morphisms

Is there a problem???

Equivalence ConstraintEquivalence Constraint

General situationAn operator defines a partial order

The distribution layout implements this order

Warning : collaboration between operator and layout manager

Particular situationsIs there a problem???

Equivalence ConstraintEquivalence Constraint

Fundamental Services for Data Fundamental Services for Data ParallelismParallelism

Spreading of the Collections:

Spread Layout

Model of Memory and Communication

Introduction

Design Model for Processing

Pattern System

Impact of the DDD

Fundamental Services for Data ParallelismFundamental Services for Data Parallelism

Conclusion

Spreading of the CollectionsSpreading of the Collections

Proc

Collection

P1

P0 P2Instance

P1Instance

P2

InstanceP0

ConformityUbiquity of spreaded collections - Local Collection

ConformitySeparation of concerns

Spread model

Invocation

Id

Key

Owner

Layout

Proc 1 Proc 2

Layout

Dynamic DistributionDynamic DistributionLocal SharingLocal Sharing

Spreading of the CollectionsSpreading of the Collections

Memory model

Shared Memory

Local Storage

Shared Collections

ConformitySeparation of concerns

Spreading of the CollectionsSpreading of the Collections

Spread model

Local memoryLocal memory

Local memory

Local memory

Local copy Original

Distributed Collection

Memory model

ConformitySeparation of concerns

Spreading of the CollectionsSpreading of the Collections

Spread model

Redefinition of the accessors

Granularity

Fundamental Behavior

• read

• write

Use case• sequential (SPMD)

• parallel

Interpretation• local

• remote (distributed, shared, etc.)

• atomic

• enumerative

ConformitySeparation of concerns

Spreading of the CollectionsSpreading of the Collections

Structure

DIST_MEMORY DIST_MEMORY

SHD_MEMORY SHD_MEMORY SHD_COLLECTIONSHD_COLLECTION

DIST_COLLECTIONDIST_COLLECTION

MEMORY_MODEL MEMORY_MODEL SPREAD_COLLECTIONSPREAD_COLLECTION

SVM_LIBSVM_LIB

POMPOM PVMPVM MPIMPI

MP_APIMP_API

SHD_APISHD_API

Redefinition of the accessors

ConformitySeparation of concerns

Spreading of the CollectionsSpreading of the Collections

ConclusionConclusion

Experimentation

Assessment

Perspectives

Introduction

Design Model for Processing

Pattern System

Impact of the DDD

Fundamental Services for Data Parallelism

ConclusionConclusion

ExperimentationExperimentation

Linear Algebra

Genome

Forest Simulation

Contract Intel SSD

Collaboration D. Lavenier

Collaboration M.-A. MoravieLaboratory of Biometrics Lyon

N-Body Application derivation

Shared Memory API User, GC extensionsParagon, Origin2000, PowerChallenge

Origin2000, Clusters

Paragon

Clusters (Myrinet, ATM),Origin2000

Distributed Memory API User (POM) Origin2000, Myrinet, etc.

Theme Context Plate form

NoW

AssessmentAssessment

Pattern system for enumeration processing

Application of the patterns for data parallelism

Patterns dedicated to data parallelism

Method to design collections, iterators and operators

PerspectivesPerspectives

Research :

Industry :

Integration of research in the area of functional programming

Validation and conformity of parallel code

Dealing with data dependencies

A pattern system for data collection processing

Library for data collection processing (C++ & STL, Java & JGL, etc.)

Toward an automatic production of the code for parallel patterns