efficiency analysis of decentralized grid scheduling with job migration and replication

Efficiency Analysis of Decentralized

Grid Scheduling with Job Migration

and Replication

International Conference on Ubiquitous Information Management and Communication (ACM ICUIMC 2013)

January 17–19, 2012, Kota Kinabalu, Malaysia

[email protected], [email protected]

Rzhanov Institute of Semiconductor Physics Siberian Branch of Russian Academy of Sciences

Siberian State University of Telecommunications and Information Sciences

Mikhail Kurnosov, Alexey Paznikov

Семинар «Вычислительные системы», Новосибирск, 29 апреля 2011 г.Михаил Курносов, Алексей Пазников

Decentralized Grid Scheduling

Семинар «Вычислительные системы», Новосибирск, 29 апреля 2011 г.Михаил Курносов, Алексей Пазников 2M. Kurnosov, A. Paznikov. Efficiency Analysis of Decentralized Grid Scheduling …

Subsystem1

EC1 EC2 ECn1

…

RMS

Local jobs

Subsystem3

EC1 EC2 ЭМn3

…

RMS

Subsystem2

EC1 EC2 ECn2

…

RMS

Subsystem4

EC1 EC2 ECn4

…

RMS

Local jobs

Local jobs Local jobsJob

r – number of processes,

z1, z2, …, zk – stage-in files sizes,

h1, h2, …, hk – subsystems with stage-in files.

ACM ICUIMC, Kota Kinabalu, Malaysia, January 17-19


Scheduling of Parallel MPI-programs

in Geographically-distributed Computer Systems

Семинар «Вычислительные системы», Новосибирск, 29 апреля 2011 г.Михаил Курносов, Алексей Пазников 3

Centralized

scheduling

• GridWay

• GrADS

• AppLeS

• Condor-G

• Nimrod/G

• WMS

Decentralized

scheduling

• GBroker

• (Korneev, 1985)*,

(Monakhov, 2000)**

Korneev V.V. Architecture of Computer Systems with Programmable Structure. Novosibirsk, 1985.

Monakhov O.G., Monakhova E.A. Parallel Systems with Distributed Memory. Resource and Task Management. Novosibirsk, 2001.

*

**

M. Kurnosov, A. Paznikov. Efficiency Analysis of Decentralized Grid Scheduling … ACM ICUIMC, Kota Kinabalu, Malaysia, January 17-19


Decentralized Scheduling of Parallel Programs


Scheduler1

EC1 EC2 ECn1

…

RMS

EC1 EC2 ECn3

…

RMS

Scheduler3

Scheduler2

EC1 EC2 ECn2

…

RMS

EC1 EC2 ECn4

…

RMS

Scheduler4

Local jobs Local jobs

Local jobs Local jobs

Subsystem1

Subsystem3

Subsystem2

Subsystem4



Algorithms of Decentralized Scheduling


i j1

j2

ju

Scheduler Scheduler

Scheduler

Scheduler

The algorithms are proposed:

• Locally optimal scheduling (LOS),

• Based on job replication (RS),

• Based on job migration (MS),

• Based on job migration and

job replication (MRS).

Each of the algorithms describes

functioning of the scheduler i after

submitting a job to its queue.

Step 0: tij, cj, sj, qj and nj are obtained from the monitoring system, then

the set S(i) = {j | nj ≥ r, j ∈ L(i) ∪ {i}} is built up.

L(i) ={ j1, j2, …, ju }


Семинар «Вычислительные системы», Новосибирск, 29 апреля 2011 г.

Scheduleri

Михаил Курносов, Алексей Пазников 6

Locally Optimal Scheduling (LOS)

EC1 EC2 ECni

…

RMS

EC1 EC2 ECn3

…

RMS

Scheduler2

Scheduler1

EC1 EC2 ECn2

…

RMS

EC1 EC2 ECn3

…

RMS

Scheduler3

Job

Local jobs

Local jobs

Local jobs

Local jobs

Subsystemi

Subsystem3

Subsystem2

Subsystem4

++

=

−

−

,

,

)(

max

max

1

max

1

max

t

t

w

w

c

c

t

t

jFj

jjjif cj < r or qj > 0,

else.

∑=

=

k

l

llj zjhtt1

),,(

cj – number of non-used EC on subsystem j;

qj – number of jobs in the queue;

wj = qj / nj – number of jobs in the queue per one EC

tj – time of staging-in to the subsystem j

Step 1. Subsystem j* = argmin{F(j)} is chosed



Scheduleri


Locally Optimal Scheduling (LOS)

EC1 EC2 ECni

…

RMS

EC1 EC2 ECn3

…

RMS

Scheduler2

Scheduler1

EC1 EC2 ECn2

…

RMS

EC1 EC2 ECn3

…

RMS

Scheduler3

Job

Local jobs

Local jobs

Local jobs

Local jobs

Subsystemi

Subsystem2

Subsystem1

Subsystem3

Step 2. Job is submitted to the local RMS

of subsystem j*.



Scheduleri


Scheduling Based on Job Replication (RS)

EC1 EC2 ECni

…

RMS

EC1 EC2 ECn3

…

RMS

Scheduler2

Scheduler1

EC1 EC2 ECn2

…

RMS

EC1 EC2 ECn3

…

RMS

Scheduler3

Job

Local jobs

Local jobs

Local jobs

Local jobs

Subsystemi

Subsystem2

Subsystem1

Subsystem3

Step 1. m subsystem j*1, j*2, …, j*m ∈ S(i)

with the least function F(j) value are chosen

cj – number of non-used EC on subsystem j;

qj – number of jobs in the queue;

wj = qj / nj – number of jobs in the queue per one EC

tj – time of staging-in to the subsystem j

++

=

−

−

,

,

)(

max

max

1

max

1

max

t

t

w

w

c

c

t

t

jFj

jjjif cj < r or qj > 0,

else.

∑=

=

k

l

llj zjhtt1

),,(



Scheduleri


Scheduling Based on Job Replication (RS)

EC1 EC2 ECni

…

RMS

EC1 EC2 ECn3

…

RMS

Scheduler2

Scheduler1

EC1 EC2 ECn2

…

RMS

EC1 EC2 ECn3

…

RMS

Scheduler3

Local jobs

Local jobs

Local jobs

Local jobs

Subsystemi

Subsystem2

Subsystem1

Subsystem3

Step 2. Job are submitted to the queues of

subsystems j*1, j*2, …, j*m.


JobJob


Scheduleri


Scheduling Based on Job Migration (MS)

EC1 EC2 ECni

…

RMS

EC1 EC2 ECn3

…

RMS

Scheduler2

Scheduler1

EC1 EC2 ECn2

…

RMS

EC1 EC2 ECn3

…

RMS

Scheduler3

Job

Local jobs

Local jobs

Local jobs

Local jobs

Subsystemi

Subsystem2

Subsystem1

Subsystem3

Step 1. With the interval ∆1 locally optimal subsystem

j' search procedure is performed:

j'= argmin{F(j)}

Step 2. If F(j*) – F(j') > ε, then migration is

performed from subsystem j to the subsystem j'.



Scheduleri


Scheduling Based on Combined Approach (MRS)

EC1 EC2 ECni

…

RMS

EC1 EC2 ECn3

…

RMS

Scheduler2

Scheduler1

EC1 EC2 ECn2

…

RMS

EC1 EC2 ECn3

…

RMS

Scheduler3

Job

Local jobs

Local jobs

Local jobs

Local jobs

Subsystemi

Subsystem2

Subsystem1

Subsystem3

The algorithm is based on combination of two

approaches – replication to the multiple subsystem

and migration between the subsystems.



GBroker Functional Structure


– software, developed in CPCT SibSUTIS

– third-party software

Local RMS

Globus Toolkit DCSMon

GBroker

NetMon

GClient



Multicluster Computer System Testbed


GBroker

NetMon DCSMon

Globus Toolkit 5.0

GBroker

NetMon DCSMon

Globus Toolkit 5.0

GBroker

NetMon DCSMon

Globus Toolkit 5.0

GBroker

NetMon DCSMon

Globus Toolkit 5.0

GBroker

NetMon DCSMon

Globus Toolkit 5.0

GBroker

NetMon DCSMon

Globus Toolkit 5.0

GridWay 5.6.1


MPI-programs (SPEC MPI2007):

• Weather Research and Forecasting (WRF) – simulation of climate

processes.

• The Parallel Ocean Program (POP) – simulation of processes in the

ocean.

• LAMMPS – molecular dynamics simulation tools.

• RAxML – bioinformatics modeling.

• Tachyon – graphical rendering.

Jobs flows:

• Simple flows with flux level λλλλ.

• The jobs are chosen randomly with uniform distribution.

• Rank r of parallel program is chosen from the set {1, 2, 4, 8}.

• Stage-in files were on the subsystem Xeon80.


Experiments Organization

Семинар «Вычислительные системы», Новосибирск, 29 апреля 2011 г.Михаил Курносов, Алексей Пазников 14M. Kurnosov, A. Paznikov. Efficiency Analysis of Decentralized Grid Scheduling … ACM ICUIMC, Kota Kinabalu, Malaysia, January 17-19

• System bandwidth B

B = M / τ

• Mean queue waiting time W

• Mean job service time T

Notations:

• tk – job submitting time to the scheduler k ∈ {1, 2, …, M}.

• – time of job k beginning execution.

• – time of job k finishing execution.


Indices of Efficiency of Scheduling Algorithms


kt′

kt ′′

∑=

−′=

M

k

kk ttM

W1

)(1

∑=

−′′=

M

k

kk ttM

T1

)(1


Flow of M = 200 jobs was submitted to the subsystem Xeon80


Efficiency comparison of algorithms LOS and RS


System bandwidth B



Efficiency comparison of algorithms LOS and RS


Mean queue waiting time W Mean job service time T



Efficiency Comparison LOS, MS, MRS


Flow of M = 200 jobs was submitted to the subsystem Xeon80

System bandwidth B



Efficiency Comparison LOS, MS, MRS




а б в

г д е

а – full graph, б – D2(6; 2, 3), в – ring, г – mesh, д – 2D-torus, е – star


Schedulers Logical Structures


Flows of M = 50 jobs were submitted to all the subsystem (algorithm MS)




System bandwidth B



Efficiency comparison of schedulers GBroker and GridWay


GBroker GBroker GBroker GBroker

GBroker GBroker

NetMon DCSMon

Globus Toolkit 5.0

NetMon DCSMon

Globus Toolkit 5.0

NetMon DCSMon

Globus Toolkit 5.0

NetMon DCSMon

Globus Toolkit 5.0

NetMon DCSMon

Globus Toolkit 5.0

GridWay 5.6.1

NetMon DCSMon

Globus Toolkit 5.0

GridWay configuration(recommended by developers)

SCHEDULING_INTERVAL (interval of scheduling) = 5 s;

DISCOVERY_INTERVAL (interval of resource discovery) = 100 s;

MONITORING_INTERVAL (interval of resource state monitoring) = 2 s;

POLL_INTERVAL (interval of job state query) = 10 s;

DISPATCH_CHUNK (number of jobs, dispatched simultaneously) = 15.





System bandwidth B


Algorithms usage recommendations

• Small stage-in files ⇒ LOS, MS, RS, MRS.

• Large stage-in files ⇒ LOS and MS.

• Low rate of job flows or small stage-in files ⇒ RS and MRS

LOS – locally optimal scheduling,

RS – based on job replication,

MS – based on job migration,

MRS – based on job migration and job replication


Recommendations


Conclusion

• Job migration helps increase system bandwidth.

• More system bandwidth (algorithm MS) of decentralized scheduling as

compared with centralized scheduling (GridWay).

• Usage of non-full graphs with small diameter (e.g. ND-torus, D2-graphs) as

logical structures of decentralized schedulers do not leads to job service

efficiency degradation.

• Decentralized algorithms are independent from the number of subsystems

and allows to robustness job flows service.

Future Works

• Algorithms of scheduling Grid workflows

• Forecasting methods in network and subsystems monitoring

• Generation of optimal local neighborhood structures


Conclusions and Future Works


Thank you for your attention!

Семинар «Вычислительные системы», Новосибирск, 29 апреля 2011 г.Михаил Курносов, Алексей Пазников Семинар «Вычислительные системы», Новосибирск, 29 апреля 2011 г.Михаил Курносов, Алексей Пазников Семинар «Вычислительные системы», Новосибирск, 29 апреля 2011 г.Михаил Курносов, Алексей Пазников 28M. Kurnosov, A. Paznikov. Efficiency Analysis of Decentralized Grid Scheduling … ACM ICUIMC, Kota Kinabalu, Malaysia, January 17-19

jjjj

jjj

j

nqsw

w

w

c

c

t

t

t

t

iF

/)(

,

,

)(

max

1

max

1

max

max1

+=

++

=

−

−

if ci ≥ r и qi = 0,

else

jjj

jjj

j

nqw

w

w

c

c

t

t

t

t

jF

/

,

,

)(

max

1

max

1

max

max3

=

++

=

−

−

if ci ≥ r and qi = 0,

else

jjj

jj

j

nqw

w

w

t

t

t

t

iF

/

,

,

)(

maxmax

max2

=

+

=


if ci ≥ r и qi = 0,

else

Subsystem Ranking


1 – LOS, F1; 2 – LOS, F2; 3 – LOS, F3


Алгоритм локально-оптимальной диспетчеризации.

Пропускная способность системы


B, jobs/s

λ, s-1


1 – LOS, F1; 2 – LOS, F2; 3 – LOS, F3

ба


Алгоритм локально-оптимальной диспетчеризации. Среднее время

ожидания в очереди и среднее время обслуживания задачи

Семинар «Вычислительные системы», Новосибирск, 29 апреля 2011 г.Михаил Курносов, Алексей Пазников Семинар «Вычислительные системы», Новосибирск, 29 апреля 2011 г.Михаил Курносов, Алексей Пазников 31

λ, s-1 λ, s-1

W, s T, s



1 – forecasting by NetMon; 2 – time of GridFTP


Forecasting of File Transfer Time


Time of transfer file with size

m = 100 Мб between subsystems

Xeon80 и L400l1

Time of transfer file with size

m = 1 Гб between subsystems

Xeon80 и L400l1


efficiency analysis of decentralized grid scheduling with job migration and replication

Documents