1998 - thesis jl pacherie parallel perators
TRANSCRIPT
A Pattern System for A Pattern System for Enumeration Processing Enumeration Processing
Expression and ParallelisationExpression and ParallelisationJean-Lin Pacherie
ContentsContents• Introduction
• Conclusion
• Design Model for Processing• Pattern System
• Impact of the Data Driven Distribution (DDD) • Services for Data Parallelism
IntroductionIntroduction
Motivations
Pattern and Parallelism
Ideas and Thesis
IntroductionIntroduction
Design Model for Processing
Pattern System
Impact of the DDD
Fundamental Services for Data Parallelism
Conclusion
MotivationsMotivations
Parallel Programming, Why?
Decrease computation duration
Increase data size
Deriving sequential applications (DDD)
MotivationsMotivations
Parallel Programming, Why?
Parallel Programming, How?
Direct parallel programming
MotivationsMotivations
Parallel Programming, Why?
The solution should conforms with usual programming environment
Constraints for the solution
Standard compilation tools
No language extensions
Parallel Programming, How?
MotivationsMotivations
Parallel Programming, Why?
Constraints for the solution
Parallel Programming, How?
Available tools
Abstraction
Encapsulation
Polymorphism
Software architecture with Patterns
MotivationsMotivations
Parallel Programming, Why?
Constraints for the solution
Parallel Programming, How?
Available tools
Evolution
Encapsulating parallelism in a sequential OOL [ Jézéquel 91 ]
Method & Framework PALADIN [ Guidec 95 ]
Reusable expression of methods [ Pacherie 97 ]
RequirementsRequirementsExperts and Reusability
Beyond the software component reusability : Pattern
Where is expressed the knowledge in an application?
What kind of tools for reusability?
Library
OO Library
ComponentDesign
FrameworkFramework
Method
Pattern
Pattern Language
Architecture
Knowledge and ReusabilityKnowledge and Reusability
Evolution
Reusability level
RequirementsRequirementsKnowledge and Reusability
Beyond the software components reusability
Pattern
[ Gamma 94 ]
Express architecture to parallelize
Express common architecture involved in parallel mechanism
RequirementRequirementKnowledge and ReusabilityData Parallelism
Context : Data collection processing
1) Cut-out
From iteration start until iteration end loop action ( current item ) next iterationend
Collection :Aggregation of homogeneous items
RequirementsRequirementsKnowledge and ReusabilityData Parallelism
Increasing Datasize
Decreasing computationduration
1) Cut out
2) Layout
3) // Processing
RequirementsRequirementsKnowledge and ReusabilityData ParallelismData Driven Distribution
Parallel Code
Collection
Sequential Code ?
Code Proc 2Code Proc 2
Code Proc 1Code Proc 1
Code Proc 3Code Proc 3
Data LayoutOwner Compute ruleRemote Data RefreshingSPMD
Principles
ThesisThesis
Design patterns for data parallelism programming
Identify and organize the design pattern dedicatedto data collection processing
Apply the DDD mechanism to these patterns:Parallelization at the software architecture level
Use the design patterns of serialUse the design patterns of serialapplications to introduce parallelismapplications to introduce parallelism
Design Model for EnumerationDesign Model for EnumerationProcessingProcessing
Operator Design Pattern
Consequences
Introduction
Design Model for ProcessingDesign Model for Processing
Pattern System
Impact of the DDD
Fundamental Services for Data Parallelism
Conclusion
Towards a pattern...Towards a pattern...
Description of behavior
Behavior vs.. Implementation
Instaciation delegation trough the Factory Method pattern
AbstractClient
FactoryMethod : AbstractProduct
AnOperation
AbstractProduct
Service
…product := FactoryMethodproduct.service...
ConcreteClient
FactoryMethod : ConcreteProduct
ConcreteProduct
Service
!ConcreteProduct!Result
AbstractSolver
make_matrix : Matrix
solve
Matrix
Item
ConcreteSolver
make_matrix : SparseMatrix
SparseMatrix
Item
Towards a pattern...Towards a pattern...
Description of behavior
Behavior vs. Implementation
Instaciation delegation trough the Factory Method pattern
Factory Pattern
Extracting iteration domain through the Iterator pattern
ArrayIteratorArray
ConcreteClient
Independence from data structures
AbstractClient
Collection Iterator
collection := make_collectioniteration := collection.make_iterator from iteration.start until iteration.exhausted loop … iteration.nextend
Towards a pattern...Towards a pattern...
Description of behavior
AbstractClient
Collection Iterator
ArrayIteratorArray
ConcreteClient
Extracting iteration domain through the Iterator patternIndependence from data structures
Towards a pattern...Towards a pattern...
Description of behavior
Factory Pattern
Extracting control structure
AbstractClient
Collection Operator
collection := make_collectioniteration := collection.make_iterator from iteration.start until iteration.exhausted loop … iteration.nextend
Iterator
Extracting iteration domain through the Iterator pattern
Towards a pattern...Towards a pattern...
Description of behavior
AbstractClient
Collection Operator
collection := make_collectioniteration := collection.make_iterator operator.attach(iteration)operator.run
Iterator Operator
Extracting control structure
Extracting iteration domain through the Iterator pattern
Towards a pattern...Towards a pattern...
Description of behavior
ConsequencesConsequences
Separation of the responsibilitiesCollection, Iterator and Operators
The modularity encourage reusability [Meyer 97]
Redefining implementationUsing the factory method pattern
Data Collections
Processing operators
Iteration domains
Client Product
Application Collection
Operator
IteratorCollection
Application
ConsequencesConsequences
Separation of the responsibilities
Processing ???!!
ProcessingProcessing
StructuralStructural OperatorOperator
CommandCommand RequestRequest
EnumerativeEnumerative AtomicAtomic
Collections Enumerations
No results With results
List of items One item
Redefining implementation
ConsequencesConsequences
Separation of the responsibilities
Operators ?
Enumeration Free Monoïd [ A* , ( ) , . ]
Operator Morphism toward [ P , e , + ]
••••[A*, ( ) , . ]
••••[P*, ( ) , . ]
+ + + +[ P , e , + ]
RValuation +Valuation +
List of A
List of P
. +
f : A P
Redefining implementation
ConsequencesConsequences
Separation of the responsibilities
Operators !
MAP MAP A,B
CROSS CROSS A,B,C
FILTER FILTER A
REDUCE REDUCE A
FORALL FORALL A
BMFBMF
Eval_Morphism Eval_Morphism A,P
Iterator Iterator A
Morphism Morphism A,P
Homomorphism Homomorphism A,B
Operator_C Operator_C A,B
Command & RequestCommand & Request
Iterator Iterator B Morphism Morphism
A,P
Operator_NC Operator_NC A,P
Redefining implementation
ConsequencesConsequences
Separation of the responsibilities
Pattern SystemPattern System
Designing collections and iterators
Expressing Operators
Using Enumerations
Introduction
Design Model for Processing
Pattern SystemPattern System
Impact of the DDD
Fundamental Services for Data Parallelism
Conclusion
MethodMethodIdentification of significant characteristics
Accessor
CapacityIndexed
Hashed
LinearIndexed
Dictionary Bounded
Dynamic
FIFO
FILO
Classification with familiesRelations between characteristics
Specification of abstractions (Collections & Iterators)
A
B
C
a
b
c Abstr
action
MethodMethodIdentification of significant characteristicsClassification with familiesRelations between characteristics
Application to collectionsApplication to collectionsdeferred class ACCESSORfeature General definitions end
deferred class ACCESSORfeature General definitions end
FamilyFamilyFamilies: Abstract classes
Characteristic
deferred class INDEXED [ITEM,INDEX] inherit ACCESSORfeature read ( i : INDEX ) : ITEM is deferred end write ( i : INDEX ; e : ITEM ) is deferred endend
deferred class INDEXED [ITEM,INDEX] inherit ACCESSORfeature read ( i : INDEX ) : ITEM is deferred end write ( i : INDEX ; e : ITEM ) is deferred endend
Characteristics: Progressive specialization
deferred class COLLECTION [ ITEM ] inherit ACCESSOR CAPACITY ../.. -- Other familiesfeature ../..end
deferred class COLLECTION [ ITEM ] inherit ACCESSOR CAPACITY ../.. -- Other familiesfeature ../..end
collection
AccesseurAccesseur
capacitycapacity
Collections: Abstract classes using families and specializationusing characteristics
Mix-inMix-in
Indexed
Indexed_Iterator
Writable_Iterator
2Way_Iterator
Indexed
Indexed
Writable
Writable
Two_Way
Two_Way
CharacteristicsCharacteristics
accessors
Iterator
FamiliesFamilies
Application to iteratorsApplication to iterators
• Similar approach
• Characteristics: from the collections, Iterator’s own
• Designed with multiple inheritance
The type constraints are expressed using characteristics combination and not using collection types
Remark :
Collection
Building an OperatorBuilding an OperatorOperator properties • Items type
• First order function
• Second order function
• Type of enumeration
Implementation • Generic type
• Virtual method (deferred)
• Classification (BMF, etc.)
• Iterator type
Pattern for IteratorsPattern for IteratorsComposable operator and Iterator
Compositions
Factorization
static
dynamic
Multiplexer Demultiplexer
Impact of the DDDImpact of the DDD
Deriving the operator pattern
Equivalence Constraint
Introduction
Design Model for Processing
Pattern System
Impact of the DDD Impact of the DDD
Fundamental Services for Data Parallelism
Conclusion
Deriving the Operator patternDeriving the Operator patternReminder : Operator structure
Redefinition of collections : data distributionParallelisation of processing
AbstractClient
AbstractCollection OperatorIterator
collection := make_collectioniteration := collection.make_iterator operator.attach(iteration)operator.run
ConcreteOperator
ParaOperator
ConcreteIterator
SpreadIterator
SpreadCollection
Enumeration of local data
Enumeration of remote data
Conformity constraintsSpread collections and Local collectionsSequential and Parallel OperatorsSequential and Parallel execution
Deriving the Operator patternDeriving the Operator patternReminder : Operator structure
DerivationFactory method : select distributed services
Invocation : local computations
ParaOperator
Spread_Collection
ParaClient
make_collection : Spread_Collection
make_operation : ParaOperator
SeqClient
make_collection : ACollection
AnOperation
make_operation : AnOperatorConcreteOperator
Collection
collection := make_collectionoperator := make_operator(collection) operator.run
make_operation ( collection) : AnOperator isdo !!Result Result.attach(Collection.items)end
make_operation ( collection) : ParaOperator isdo !!Result Result.attach(Collection.local_items)end
Deriving the Operator patternDeriving the Operator patternReminder : Operator structure Conformity constraints
Parallel Evaluation SchemeParallel Evaluation Scheme
[A*, () , . ]
[ P , e , + ]
F
[ A**, () , . ][ A**, () , . ]DistDist
[ P*, () , . ]
F*
Aggregation
• Morphism (f,e,+)• Distribution
Data :
Intermediate :
• Computation of the local contributions
Result :
• Aggregation of the local contributions
Equivalence ConstraintEquivalence ConstraintIs there a problem???
Equivalence ConstraintAggregation order of local contributions
Particular situationsPropriety of morphismsCommutative operator
Morphism
+F ( ) F ( )
+{F ( ) F ( )} F ( )+
F ( ) F ( ) F ( )// Evaluation
Aggregation
Order of aggregation = initial order
Is there a problem???
Equivalence ConstraintEquivalence Constraint
F ( ) F ( ) F ( )//Evaluation
Inner composition rule is commutative
& : Commutative
&F ( ) F ( )
&{F ( ) F ( )} F ( )&Aggregation
Order of aggregation =/= initial order
& F ( ) F ( )=?=
Particular situationsPropriety of morphisms
Is there a problem???
Equivalence ConstraintEquivalence Constraint
General situationAn operator defines a partial order
The distribution layout implements this order
Warning : collaboration between operator and layout manager
Particular situationsIs there a problem???
Equivalence ConstraintEquivalence Constraint
Fundamental Services for Data Fundamental Services for Data ParallelismParallelism
Spreading of the Collections:
Spread Layout
Model of Memory and Communication
Introduction
Design Model for Processing
Pattern System
Impact of the DDD
Fundamental Services for Data ParallelismFundamental Services for Data Parallelism
Conclusion
Spreading of the CollectionsSpreading of the Collections
Proc
Collection
P1
P0 P2Instance
P1Instance
P2
InstanceP0
ConformityUbiquity of spreaded collections - Local Collection
ConformitySeparation of concerns
Spread model
Invocation
Id
Key
Owner
Layout
Proc 1 Proc 2
Layout
Dynamic DistributionDynamic DistributionLocal SharingLocal Sharing
Spreading of the CollectionsSpreading of the Collections
Memory model
Shared Memory
Local Storage
Shared Collections
ConformitySeparation of concerns
Spreading of the CollectionsSpreading of the Collections
Spread model
Local memoryLocal memory
Local memory
Local memory
Local copy Original
Distributed Collection
Memory model
ConformitySeparation of concerns
Spreading of the CollectionsSpreading of the Collections
Spread model
Redefinition of the accessors
Granularity
Fundamental Behavior
• read
• write
Use case• sequential (SPMD)
• parallel
Interpretation• local
• remote (distributed, shared, etc.)
• atomic
• enumerative
ConformitySeparation of concerns
Spreading of the CollectionsSpreading of the Collections
Structure
DIST_MEMORY DIST_MEMORY
SHD_MEMORY SHD_MEMORY SHD_COLLECTIONSHD_COLLECTION
DIST_COLLECTIONDIST_COLLECTION
MEMORY_MODEL MEMORY_MODEL SPREAD_COLLECTIONSPREAD_COLLECTION
SVM_LIBSVM_LIB
POMPOM PVMPVM MPIMPI
MP_APIMP_API
SHD_APISHD_API
Redefinition of the accessors
ConformitySeparation of concerns
Spreading of the CollectionsSpreading of the Collections
ConclusionConclusion
Experimentation
Assessment
Perspectives
Introduction
Design Model for Processing
Pattern System
Impact of the DDD
Fundamental Services for Data Parallelism
ConclusionConclusion
ExperimentationExperimentation
Linear Algebra
Genome
Forest Simulation
Contract Intel SSD
Collaboration D. Lavenier
Collaboration M.-A. MoravieLaboratory of Biometrics Lyon
N-Body Application derivation
Shared Memory API User, GC extensionsParagon, Origin2000, PowerChallenge
Origin2000, Clusters
Paragon
Clusters (Myrinet, ATM),Origin2000
Distributed Memory API User (POM) Origin2000, Myrinet, etc.
Theme Context Plate form
NoW
AssessmentAssessment
Pattern system for enumeration processing
Application of the patterns for data parallelism
Patterns dedicated to data parallelism
Method to design collections, iterators and operators
PerspectivesPerspectives
Research :
Industry :
Integration of research in the area of functional programming
Validation and conformity of parallel code
Dealing with data dependencies
A pattern system for data collection processing
Library for data collection processing (C++ & STL, Java & JGL, etc.)
Toward an automatic production of the code for parallel patterns