main sphinx design concepts
DESCRIPTION
Main Sphinx Design Concepts. There are two primary design components which comprise Sphinx The Database Warehouse The Control Process The Database Warehouse is the information depot for all of Sphinx All information is stored/retrieved via the database (or possibly multiple databases) - PowerPoint PPT PresentationTRANSCRIPT
Main Sphinx Design Concepts
• There are two primary design components which comprise Sphinx• The Database Warehouse• The Control Process
• The Database Warehouse is the information depot for all of Sphinx• All information is stored/retrieved via the database (or possibly
multiple databases)• There is no direct communication between different internal
Sphinx modules• The API for all internal modules is SQL along with the
database schema
• The Control Process is the main point of activity• It is a loop which calls different Sphinx modules• All modules in the control process operate on stateful entities in
the database, changing the state of an entity from one to another state
• Sphinx operates as a Finite State Machine!• Uses data mining principles in the design:
• The API for all modules is SQL along the database schema• Modules mine the database for information and then take
some action.
More Details
• The components are currently split (packaged) in the following way:• Sphinx Client
• Clarens Web Service (XML-RPC) + VDT-Client• Sends a Chimera DAX to Sphinx Server • Receives individual “concrete Jobs” from Sphinx Server• Submits “Job” (= “micro DAG”) to Condor-G/DAGMan• “micro DAG” (for now…could use Grid-shell or equivalent in the
future)• Job to create environment• Job to move input data locally (if required)• Job to run application• Job to move output data (if required)• Job to register data in RLS• Job to clean-up environment
• Sphinx Server• Clarens Web Service (XML-RPC)• Receives a Chimera DAX• Returns individual concrete “Jobs” (=“micro DAG”) • Provides heuristic algorithms and Satisfies constraints for:
• policies, • input data existence, • required resources• grid weather
Version 1 (current configuration):Sphinx Server with Sphinx Submission
Client
Sphinx Server
VDT Client
VDT Server Site
Monitoring Service
Resource
Repl. Loc. Serv.
Condor-G/DAGMan
Multi-threadedControl Process
Job Execution Planner
DAG Reducer
DAG Tracker
Job Predictor
DAG Data Repl. Planner
Job Admission Control
Message Interface
DAG Predictor
DAG Admission Control
Policies
Accounting
Resource Prop.
Resource Hier.
Resource Dist.
Distance
Entity-Name
Entity Hier.
Job-Entity Prop.
Job
DAG
Rel. Data
Rel. Appl.
Job Tracking
Grid Weather
Data Warehouse
Schedule
Prediction
Data Management
Information Gatherer
Sphinx ClientMessage Interface
Job Submitter
Job Tracker
Message
indicates current place-holder implementation
Version 1: Sphinx Control Process – Simple Finite State Machine
DAG: unreducedJOB: unpredicted
DAG: unpredictedJOB: unpredicted
DAG: unacceptedJOB: unaccepted
DAG: unpredictedJOB: unaccepted
DAG: unacceptedJOB: unplanned
DAG: unplannedJOB: unplanned
yes
noAwait Mesg.
Reduce DAG
Predict Job
no
no
yes
yes
yes
noPredict DAG
Admit Job
Admit DAG
no
no
yes
yes
DAG: unfinishedJOB: unplanned
DAG: unfinishedJOB: unsent
DAG: removeJOB: remove
DAG: unfinishedJOB: remove
yes
noPlan Dag
Plan Job
Submit Job
no
no
yes
yes
yes
noRemove Job
Remove DAG
Sphinx Server
VDT Client
VDT Server Site
Monitoring Service
Resource
Repl. Loc. Serv.
Condor-G/DAGMan
Multi-threadedControl Process
Job Execution Planner
DAG Reducer
DAG Tracker
Job Predictor
DAG Data Repl. Planner
Job Admission Control
Message Interface
DAG Predictor
DAG Admission Control
Policies
Accounting
Resource Prop.
Resource Hier.
Resource Dist.
Distance
Entity-Name
Entity Hier.
Job-Entity Prop.
Job
DAG
Rel. Data
Rel. Appl.
Job Tracking
Grid Weather
Data Warehouse
Schedule
Prediction
Data Management
Information Gatherer
Sphinx ClientMessage Interface
Job Submitter
Job Tracker
Message
Examples of how planning modules can be moved inside/outside the Sphinx
Server/Client
Sphinx Server/VDT Client
UI Machine
VDT Server Site
Monitoring Service
Resource
Multi-threadedControl Process
Job Execution Planner
DAG Reducer
DAG Tracker
Job Predictor
DAG Data Repl. Planner
Job Admission Control
Job Submitter
Message Interface
DAG Predictor
DAG Admission Control
Job Tracker
Data Management
Information Gatherer
Repl. Loc. Serv.C
on
dor-
G/D
AG
Man
Policies
Accounting
Resource Prop.
Resource Hier.
Resource Dist.
Distance
Entity-Name
Entity Hier.
Job-Entity Prop.
Job
DAG
Rel. Data
Rel. Appl.
Job Tracking
Grid Weather
Data Warehouse
Schedule
Prediction
Sphinx ClientMessage Interface
Message
Examples of how planning modules can be moved inside/outside the Sphinx
Server/Client
Sphinx Server/VDT Client
VO SpecificExternal Planning Service
VDT Server Site
Monitoring Service
Resource
Multi-threadedControl Process
DAG Reducer
DAG Tracker
Job Predictor
DAG Data Repl. Planner
Job Admission Control
Job Submitter
Message Interface
DAG Predictor
DAG Admission Control
Job Tracker
Data Management
Information Gatherer
Repl. Loc. Serv.C
on
dor-
G/D
AG
Man
Policies
Accounting
Resource Prop.
Resource Hier.
Resource Dist.
Distance
Entity-Name
Entity Hier.
Job-Entity Prop.
Job
DAG
Rel. Data
Rel. Appl.
Job Tracking
Grid Weather
Data Warehouse
Schedule
Prediction
Sphinx ClientMessage Interface
Message
Job Execution Planner
Examples of how planning modules can be moved inside/outside the Sphinx
Server/Client