university of catania, engineering faculty department of computer science and telecommunications...

17
University of Catania, Engineering Faculty University of Catania, Engineering Faculty Department of Computer Science and Telecommunication Department of Computer Science and Telecommunication Report about the Development of Report about the Development of Scheduling Algorithm in the Resource Broke Scheduling Algorithm in the Resource Broke S. Cavalieri and S. Monfort S. Cavalieri and S. Monfort

Upload: luke-pitts

Post on 21-Jan-2016

216 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

University of Catania, Engineering FacultyUniversity of Catania, Engineering Faculty

Department of Computer Science and TelecommunicationsDepartment of Computer Science and Telecommunications

Report about the Development of Report about the Development of

Scheduling Algorithm in the Resource BrokerScheduling Algorithm in the Resource Broker

S. Cavalieri and S. MonforteS. Cavalieri and S. Monforte

Page 2: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the implementatiimplementationon

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

Page 3: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the implementatiimplementationon

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

Page 4: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the implementatiimplementationon

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

Current GRIS/GIIS Information ArchitectureCurrent GRIS/GIIS Information Architecture

Information Services (i.e. grid-info-host, globus-version etc.) on GRISes publish Information Services (i.e. grid-info-host, globus-version etc.) on GRISes publish LDIF objects representing their resources in LDAP directories.LDIF objects representing their resources in LDAP directories.

None of these informations are sent to the GIIS.None of these informations are sent to the GIIS. The GIIS maintains a directory (grid-info-site-regdir) as an index of registered The GIIS maintains a directory (grid-info-site-regdir) as an index of registered

GRISes (reg.GRISes (reg.<port><port>..<hn><hn>.conf).conf)

GIISGIIS

GRIS gridct1.ct.infn.itGRIS gridct1.ct.infn.it

GRIS gridct2.ct.infn.itGRIS gridct2.ct.infn.it

GRIS gridct3.ct.infn.itGRIS gridct3.ct.infn.it

globus-versionglobus-version

grid-info-hostgrid-info-host

globus-gram-schedulerglobus-gram-scheduler

globus-versionglobus-version

grid-info-hostgrid-info-host

globus-gram-schedulerglobus-gram-scheduler

globus-versionglobus-version

grid-info-hostgrid-info-host

globus-gram-schedulerglobus-gram-scheduler

grid-info-site-regdirgrid-info-site-regdir

reg.2163.gridct1.ct.infn.it.confreg.2163.gridct1.ct.infn.it.conf

reg.2163.gridct2.ct.infn.it.confreg.2163.gridct2.ct.infn.it.conf

reg.2163gridct3.ct.infn.it.confreg.2163gridct3.ct.infn.it.conf

Page 5: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

The Resource BrokerThe Resource Broker

Implementation of the scheduling functions performed by Implementation of the scheduling functions performed by the Resource Broker, is based on the following architecture:the Resource Broker, is based on the following architecture:

GIISGIIS

GRISGRIS

GRISGRIS

GRISGRIS

Resource Resource BrokerBroker

Resource Resource BrokerBroker

Resources Published Resources Published in LDAP treesin LDAP trees

ClassAd Description of ClassAd Description of GRISes ResourcesGRISes Resources

JDLJDLJDLJDL

ClassAd Description of ClassAd Description of Job RequirementsJob Requirements

Best Matching GRISBest Matching GRIS

IntroductionIntroduction

State of the State of the implementatiimplementationon

Page 6: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the ImplementatiImplementationon

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

Implemented Functions Implemented Functions

Search of all the available resources. Through the analysis Search of all the available resources. Through the analysis of the content of the directory “grid-info-site-regdir” where of the content of the directory “grid-info-site-regdir” where GIIS maintains the list of GRIS referring to it, information GIIS maintains the list of GRIS referring to it, information useful to localisation and access to each GRIS is obtained.useful to localisation and access to each GRIS is obtained.

For each GRIS that can be reached from the GIIS, For each GRIS that can be reached from the GIIS, information published are retrieved through LDAP.information published are retrieved through LDAP.

Collected resources are converted in terms of ClassAds Collected resources are converted in terms of ClassAds and maintained by an ad hoc data structure. and maintained by an ad hoc data structure.

Page 7: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the ImplementatiImplementationon

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

Implemented Functions Implemented Functions

Two objects have been developed in order to provide an Two objects have been developed in order to provide an abstraction over the resources published in a LDAP abstraction over the resources published in a LDAP directory tree of a GRIS as well as regarding the directory tree of a GRIS as well as regarding the architecture designed for connecting GRISes and GIIS. architecture designed for connecting GRISes and GIIS.

A GRISResourceWrapper is instantiated for each GRIS A GRISResourceWrapper is instantiated for each GRIS supplying the GIISResourceWrapper with a ClassAd View supplying the GIISResourceWrapper with a ClassAd View of its resources.of its resources.

The GIISResourceWrapper provides for a coherent and The GIISResourceWrapper provides for a coherent and easy to query view over the entire site resources, ready to easy to query view over the entire site resources, ready to be used for matching.be used for matching.

Page 8: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the ImplementatiImplementationon

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

Design IssuesDesign Issues

What is the best design for the GRISResourceWrapper ?What is the best design for the GRISResourceWrapper ?

How many ClassAds should be mantained for each How many ClassAds should be mantained for each GRISResourceWrapper ?GRISResourceWrapper ?

Which attributes should be mantained ? Which attributes should be mantained ?

How and when a GRISResourceWrapper should be How and when a GRISResourceWrapper should be updated ? updated ?

Page 9: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the ImplementatiImplementationon

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

What is the best design for the What is the best design for the GRISResourceWrapper ?GRISResourceWrapper ?

Three Solutions:Three Solutions:

One ClassAd, One LDAP EntryOne ClassAd, One LDAP Entry

One ClassAd for all the LDAP EntriesOne ClassAd for all the LDAP Entries

Nested ClassAd (not discussed in the presentation)Nested ClassAd (not discussed in the presentation)

Page 10: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the ImplementatiImplementationon

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

One ClassAd, One LDAP EntryOne ClassAd, One LDAP Entry… …

A GRISResourceWrapper maintains a collection of ClassAds, one for A GRISResourceWrapper maintains a collection of ClassAds, one for each entry of the LDAP directory, which contains the resources each entry of the LDAP directory, which contains the resources published by a globus information service.published by a globus information service.

[ [ ……

schedulertype=”condor”schedulertype=”condor”

……]]

[ [ ……

schedulertype=”condor”schedulertype=”condor”

……]]

globus-gram-schedulerglobus-gram-scheduler

grid-info-hostgrid-info-host

[ …[ …

ostype=”linux”ostype=”linux”

……] ]

[ …[ …

ostype=”linux”ostype=”linux”

……] ]

globus-versionglobus-version

[ …[ …

releaseDate=2000/04/11releaseDate=2000/04/11

……] ]

[ …[ …

releaseDate=2000/04/11releaseDate=2000/04/11

……] ] ClassAd View of ClassAd View of ResourcesResources

Page 11: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the ImplementatiImplementationon

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

One ClassAd, One LDAP EntryOne ClassAd, One LDAP Entry… …

The ClassAd expression of the job requirements may contain attributes The ClassAd expression of the job requirements may contain attributes published by different information services, thus owned by different published by different information services, thus owned by different ClassAd of the collection the GRISResourceWrapper stores. ClassAd of the collection the GRISResourceWrapper stores.

[… […

requirements= requirements= other.ostype==”linux” && other.ostype==”linux” && other.schedulertype==”condorother.schedulertype==”condor

……]]

[… […

requirements= requirements= other.ostype==”linux” && other.ostype==”linux” && other.schedulertype==”condorother.schedulertype==”condor

……]]

JobAdJobAd

[ [ ……

schedulertype=”condor”schedulertype=”condor”

……]]

[ [ ……

schedulertype=”condor”schedulertype=”condor”

……]]

[ …[ …

ostype=”linux”ostype=”linux”

……] ]

[ …[ …

ostype=”linux”ostype=”linux”

……] ]

[ …[ …

releaseDate=2000/04/11releaseDate=2000/04/11

……] ]

[ …[ …

releaseDate=2000/04/11releaseDate=2000/04/11

……] ]

GRISGRIS

ResourceAdsResourceAds

Page 12: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the ImplementatiImplementationon

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

One ClassAd, One LDAP EntryOne ClassAd, One LDAP Entry… …

Since the MatchClassAd specialization performs a bilateral match Since the MatchClassAd specialization performs a bilateral match between two and only two ClassAds, it will return FALSE for every between two and only two ClassAds, it will return FALSE for every ClassAd describing the GRIS resources.ClassAd describing the GRIS resources.

GIISGIIS

Resource Resource BrokerBroker

Resource Resource BrokerBroker ????

[… […

requirements= requirements= other.ostype==”linux” && other.ostype==”linux” && other.schedulertype==”condorother.schedulertype==”condor

……]]

[… […

requirements= requirements= other.ostype==”linux” && other.ostype==”linux” && other.schedulertype==”condorother.schedulertype==”condor

……]]

JobAdJobAd

[ [ ……

schedulertype=”condor”schedulertype=”condor”

……]]

[ [ ……

schedulertype=”condor”schedulertype=”condor”

……]]

[ …[ …

ostype=”linux”ostype=”linux”

……] ]

[ …[ …

ostype=”linux”ostype=”linux”

……] ]

[ …[ …

releaseDate=2000/04/11releaseDate=2000/04/11

……] ]

[ …[ …

releaseDate=2000/04/11releaseDate=2000/04/11

……] ]

GRISGRIS

ResourceAdsResourceAds

No Such GRISNo Such GRISNo Such GRISNo Such GRIS

Page 13: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the ImplementatiImplementationon

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

All in OneAll in One… …

The second approach is based on the use of just one ClassAd describing The second approach is based on the use of just one ClassAd describing all the resources published by a given GRIS. all the resources published by a given GRIS.

[ [ ……

releaseDate=2000/04/11releaseDate=2000/04/11

schedulertype=”condor”schedulertype=”condor”

ostype=”linux”ostype=”linux”

……]]

[ [ ……

releaseDate=2000/04/11releaseDate=2000/04/11

schedulertype=”condor”schedulertype=”condor”

ostype=”linux”ostype=”linux”

……]]

ResourceAdResourceAd

globus-versionglobus-version

grid-info-hostgrid-info-host

globus-gram-schedulerglobus-gram-scheduler

Page 14: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the ImplementatiImplementationon

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

All in OneAll in One… …

On the other hand, it should be kept in mind that two different entries of On the other hand, it should be kept in mind that two different entries of the LDAP directory might contain one or more attributes with the same the LDAP directory might contain one or more attributes with the same name, but different values.name, but different values.

Therefore, any new value for an attribute already inserted in the Therefore, any new value for an attribute already inserted in the ClassAd must appended, thus creating a list of values. ClassAd must appended, thus creating a list of values.

grid-info-hostgrid-info-host

globus-gram-schedulerglobus-gram-schedulerCONDORCONDOR

[ [ ……releaseDate=2000/04/11releaseDate=2000/04/11schedulertype={”condor”,”lsf”}schedulertype={”condor”,”lsf”}ostype=”linux”ostype=”linux”……]]

[ [ ……releaseDate=2000/04/11releaseDate=2000/04/11schedulertype={”condor”,”lsf”}schedulertype={”condor”,”lsf”}ostype=”linux”ostype=”linux”……]]

ResourceAdResourceAd

globus-versionglobus-version

globus-gram-schedulerglobus-gram-schedulerLSFLSF

Page 15: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the ImplementatiImplementationon

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

All in OneAll in One… …

Nevertheless,Nevertheless, the match between the ResourceAd and the the match between the ResourceAd and the JobAd does not work yet. JobAd does not work yet.

GIISGIIS

Resource Resource BrokerBroker

Resource Resource BrokerBroker ????

GRISGRIS

No Such GRISNo Such GRISNo Such GRISNo Such GRIS

[… […

requirements= requirements= other.ostype==”linux” && other.ostype==”linux” && other.schedulertype==”condorother.schedulertype==”condor

……]]

[… […

requirements= requirements= other.ostype==”linux” && other.ostype==”linux” && other.schedulertype==”condorother.schedulertype==”condor

……]]

JobAdJobAd

ResourceAdResourceAd

[ [ ……releaseDate=2000/04/11releaseDate=2000/04/11schedulertype={”condor”,”lsf”}schedulertype={”condor”,”lsf”}ostype=”linux”ostype=”linux”……]]

[ [ ……releaseDate=2000/04/11releaseDate=2000/04/11schedulertype={”condor”,”lsf”}schedulertype={”condor”,”lsf”}ostype=”linux”ostype=”linux”……]]

Page 16: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the ImplementatiImplementationon

Design Issues Design Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

All in OneAll in One… …

The problem is within the description of the job requirements in The problem is within the description of the job requirements in ClassAd, that is, the operator == does not work when applied to ClassAd, that is, the operator == does not work when applied to an attribute whose value is a list. The member function must be an attribute whose value is a list. The member function must be rather used.rather used.

GIISGIIS

Resource Resource BrokerBroker

Resource Resource BrokerBroker ????

GRISGRIS

Match OkMatch OkMatch OkMatch Ok

[… […

requirements= other.ostype==”linux” && requirements= other.ostype==”linux” &&

Member(“condor”,other.schedulertype)Member(“condor”,other.schedulertype)

……]]

[… […

requirements= other.ostype==”linux” && requirements= other.ostype==”linux” &&

Member(“condor”,other.schedulertype)Member(“condor”,other.schedulertype)

……]]

JobAdJobAd

ResourceAdResourceAd

[ [ ……releaseDate=2000/04/11releaseDate=2000/04/11schedulertype={”condor”,”lsf”}schedulertype={”condor”,”lsf”}ostype=”linux”ostype=”linux”……]]

[ [ ……releaseDate=2000/04/11releaseDate=2000/04/11schedulertype={”condor”,”lsf”}schedulertype={”condor”,”lsf”}ostype=”linux”ostype=”linux”……]]

It should be pointed out that the JDL user must know which attribute It should be pointed out that the JDL user must know which attribute requires the member function for a feasible matching. requires the member function for a feasible matching.

Page 17: University of Catania, Engineering Faculty Department of Computer Science and Telecommunications Report about the Development of Scheduling Algorithm in

IntroductionIntroduction

State of the State of the ImplementatiImplementationon

Design IssuesDesign Issues

One ClassAd One ClassAd One LDAP One LDAP EntryEntry

All in OneAll in One

Final Final RemarksRemarks

Final RemarksFinal Remarks In the author’s point of view the best solution is to use only one In the author’s point of view the best solution is to use only one

ClassAd representing a subset of the more meaningful attributes, ClassAd representing a subset of the more meaningful attributes, rather than the entire attributes published.rather than the entire attributes published.

Problem to be discussed: what information is needed for Problem to be discussed: what information is needed for the Broker and who has to publish itthe Broker and who has to publish it