karma with proactive parallel suite 12/01/2009 air france, sophia antipolis

KARMA with ProActive Parallel Suite12/01/2009

Air France, Sophia Antipolis

Solutions and Services for Accelerating your Applications

ProActive Parallel Suite

Speed-up applications by distributing them over a set of resources

The result of 9 years of R&D at INRIA

KARMA performances constraints• KARMA processes computation needs refer to :

- HEAVY PROCESS (Batch long time)- LIGHT PROCESS (Interactive short time)

• Processes are expressed according to the following features:- Triggering mode (start and kill)- Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative)- Data volume (light, medium, heavy)

• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization )- Fault error management

• Open system- Interoperability (upper and lower stream)- API (language, completeness, reliability)- Tools (friendliness, eclipse integration ability,..)

• Parallelization- Multiple processes capability (functional)- Parallelization programming capability (technical).

ProActive Technology Integration

Simple and Seamless integration

Installation : one directory Deployment : one XML file (GCM Deployment)

Possible automated agent-based deployment

Support for common standards Connection/Acquisition: Ssh, Rsh, LSF, PBS, SGE,… Communication: RMI, RMI/SSH, HTTP, Ibis,… Export any activity as a WebService J2EE (in and out)

Validated on common JDKs 1.5, 1.6 and 1.7

Activities are monitored through standard JMX Compliant with common monitoring tools (TIBCO Hawk)



• Processes are expressed according to the following features:- Triggering mode (start and kill)- Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative)- Data volume (light, medium, heavy)


• Open system- Interoperability (upper and lower stream)- API (language, completeness, reliability)- Tools (friendliness, eclipse integration ability,..)




• Processes are expressed according to the following features:- Triggering mode (start and kill)- Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) - Data volume (light, medium, heavy)


• Open system Interoperability (upper and lower stream)- API (language, completeness, reliability)- Tools (friendliness, eclipse integration ability,..)


ProActive Scheduling

Managed environment for distributed computation

Scheduler / Resource Manager Overview

• Multi-platform Graphical Client (RCP)

• File-based or LDAP authentication

• Static Workflow Job Scheduling, Native and Java tasks, Retry on Error, Priority Policy, Configuration Scripts,…

• Dynamic and Static node sources, Resource Selection by script, Monitoring and Control GUI,…

• ProActive Deployment capabilities : Desktops, Clusters, ProActive P2P,…





• Open system Interoperability (upper and lower stream)- API (language, completeness, reliability)- Tools (friendliness, eclipse integration ability,..)






• Open system Interoperability (upper and lower stream)- API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)


Parallelism with ProActive Scheduling

Task 1(input 1) Task 2(input 2)

Task 3(res1,res2)

res1 res2

Task 1(input 1) Task 2(input 2) Task N(input N)…

Task 1(input 1)

Functional // Technical //

Computations are defined as Jobs made of Tasks

Bag of Tasks

Tasks Flow

ProActive application



• Processes are expressed according to the following features:- Triggering mode (start and kill) Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) - Data volume (light, medium, heavy)



• Parallelization Multiple processes capability (functional) Parallelization programming capability (technical).

Tasks can be defined in Java or as Native Application

Dynamic Task creation through Java APIJavaTask aTask = new JavaTask();aTask.setName("task 1");aTask.setRerunnable(2);aTask.setRestartOnError(RestartMode.ELSEWHERE);aTask.addArgument("foo",new Boolean(true));aTask.setExecClassName("com.activeeon.Compute");aTask.setPreScript("/path/to/script_file");

Can attach pre/post script to a task Any JSR-223 supported language (JS, Python, Ruby,…) Used for configuration and data transfer

Data are handled separately

Defining Tasks

Jobs can be defined in Java or XML

Dynamic job creation through Java APITaskFlowJob job = new TaskFlowJob();

job.setName("job_name");job.setPriority(JobPriority.NORMAL);job.setCancelOnError(true);job.addTask(new NativeTask(cmd));

Can easily develop tools that generate and submit jobs Specialized Automated

Static description in XML descriptor Can be defined by any non specialized user

Defining Jobs



• Processes are expressed according to the following features:- Triggering mode (start and kill) Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) - Data volume (light, medium, heavy)





HEAVY PROCESS (Batch long time) LIGHT PROCESS (Interactive short time)

• Processes are expressed according to the following features:- Triggering mode (start and kill) Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)


• Open system Interoperability (upper and lower stream) API (language, completeness, reliability) Tools (friendliness, eclipse integration ability,..)


Executing Jobs

Submission through… Java API Graphical interface Command line

Recovery is fully configurable with User/System level distinction

1. Host failure : automatically re-dispatched with needed input Number of retries bounded by the administrator

2. Task failure : can be either… Automatically re-dispatched with needed input

Number of retries bounded by the user Aborted (Task) : exception is transmitted as task result Aborted (Job) : job is fully canceled

Scheduler itself is recoverable from underlying database



• Processes are expressed according to the following features:- Triggering mode (start and kill) Parallelization capability (functional / technical) - Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)






• Processes are expressed according to the following features: Triggering mode (start and kill) Parallelization capability (functional / technical) Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)

• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory)- Strategy / Policy (Arbitration resources and tasks request prioritization ) Fault error management



Monitoring Jobs

System and ProActive level monitoring Through standard JMX Provide generic tool for monitoring and reporting Compliant with common monitoring tool

Monitoring Jobs

System and ProActive level monitoring Through standard JMX Provide generic tool for monitoring and reporting Compliant with common monitoring tool

Scheduler level monitoring On-going porting from MVC to JMX



• Processes are expressed according to the following features: Triggering mode (start and kill) Parallelization capability (functional / technical) Re-run ability (fault tolerance / recovery)- Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)






• Processes are expressed according to the following features: Triggering mode (start and kill) Parallelization capability (functional / technical) Re-run ability (fault tolerance / recovery) Performance (elapse time)- Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)




Controlling Jobs Fine grain control through GUI and API

Job level (User)

Scheduler level(Admin)




• Manage resources consumption- Availability- Resources optimization (CPU, Core, Memory) Strategy / Policy (Arbitration resources and tasks request prioritization ) Fault error management



Specializing the Scheduler

Business Level specialization and automation

ActiveEon provides business specific interface for the Scheduler Bio-technology (Gold, MapRead) Finance MatLab/Scilab




User-level hooks to (very) easily specialize the Scheduler

Plug-in for RCP client e.g. ActiveEon’s Gold module

Result preview mechanism Can preview result in the scheduler GUI (textual/graphical) Can attach any external application to a task for result display




User-level hooks to (very) easily specialize the Scheduler

Plugin for RCP client e.g. ActiveEon’s Gold module

Result preview mechanism Can preview result in the scheduler GUI (textual/graphical) Can attach any external application to a task for result display



• Processes are expressed according to the following features: Triggering mode (start and kill) Parallelization capability (functional / technical) Re-run ability (fault tolerance / recovery) Performance (elapse time) Type (interactive, schedule, event, exceptional, administrative) Data volume (light, medium, heavy)




ProActive Resource Manager

Rely on ProActive Programming features GCM deployment standard Agent based deployment Support for many LAN / WAN / Grid / Cloud standards…

MultiCPU/Core Cluster

Desktop Grid

Clusters / Grille Clouds / RAS

Straightforward Upgrade

ProActive Resource Manager

Rely on ProActive Programming features GCM deployment standard Agent based deployment Support for many LAN / WAN / Grid / Cloud standards…

Deal with resources heterogeneity with Selection Scripts User defined : no need for system reconfiguration Any JSR-223 supported language (JS, Python, Ruby,…)

Can rely on dynamic decision or on static host description Optimized (probabilistic selection of candidates)




• Manage resources consumption Availability Resources optimization (CPU, Core, Memory) Strategy / Policy (Arbitration resources and tasks request prioritization ) Fault error management



Thank you for your attention

Demo

karma with proactive parallel suite 12/01/2009 air france, sophia antipolis

Documents

set of resources

eclipse integration

lower streamapi language

reliabilitytools friendliness

triggering mode

following features

directory deployment

problem solution karma