phenix off-line computing system pep the phenix event processor the phenix collaboration presented...

PHENIX Off-line Computing System

PEPPEPThe PHENIX Event ProcessorThe PHENIX Event Processor

The PHENIX Collaborationpresented by

S. P. Sorensen

University of Tennessee

at CHEP97

The PEP Team:The PEP Team:Han Geng, Qihang Liu, Han Geng, Qihang Liu, David P. MorrisonDavid P. Morrison, , Li Qun, Li Qun, Victor PerevoztchikovVictor Perevoztchikov, Kyle Pope, , Kyle Pope,

Mark Pollark, Soren Sorensen and Mark Pollark, Soren Sorensen and Yan ZhaoYan Zhao


Outline

PHENIX’s Computing Model PHENIX Requirements Types of Analysis Frameworks Domains and Implementation Choices Interprocess Communication User Modules Data Structure Management RHIC Off-line Software Collaboration


PHENIX Computing Model

Level ? Trigger

Online Monitoring

Event Reconstruction

Data Mining

RHIC Data Analysis

Routing Layer

HDSS:Hierarchical Data Storage System [raw]

HDSS [particles]

HDSS: [Filtered

data]

HDSS: [Data Analysis

results]

Central Data Base

Local Data Analysis

Local Data Base

Local Data StorageSimulation

Local Data Base

HDSS

Data Mining

Local Data Analysis

Central Data Base

PHENIX Counting House

RHIC Computing Center

Remote Sites Remote Sites

Regional Computing Center


PHENIX’s CPU and Data Storage Requirements

CPU LocationGflops kSPECInt92

Event Reconstruction 70 175 RHICData Mining 10 30 RHICData Analysis (Data Scaning) 20 60 RHIC & JapanPhysics Models 25 75 JapanDetector Simulation 25 75 JapanTotal 150 415

Type of data TB/year Storage MediumDisk Robot

Raw Data 350 1 12Data Summary Level 175 3 8micro DST's 100 5 10Simulated Data 150 1 5Physics Model Data 10 1 5Off-line Database 0.1 0.1Total 785 11 40

CPU

Data Production and Storage


PHENIX Analysis Framework Requirement I

Environment Can operate in a distributed,

heterogeneous environment Can operate in a single node environment Can operate both on the highest

performance chips and the best price/performance chips


PHENIX Analysis Framework Requirement II

Configurability Can be configured to do all aspects of the

PHENIX Data analysis Will allow the creation of

• a small, ``tight'' executable for the single user

• a large comprehensive set of executables for the large-scale tasks like Event Reconstruction

Can operate in both batch and interactive mode with same functionality

Will allow already existing software (PISA/PISORP) to be incorporated


PHENIX Analysis Framework Requirement III

Data Access Will allow the user access to all of

PHENIX data through a uniform API Will allow the user to perform data queries Will allow full access to both a global,

central database and local data bases Will allow the user to graphical view the

data Will allow access specifically to all data in

the online/DAQ system injected into the routing layer


PHENIX Analysis Framework Requirement IV

User Interface GUI and batch interface shall provide

identical functionality GUI should be tailorable by the user GUI-batch mode switchable at run time GUI can run on a small, cheap computers

(PC, Mac, etc..) User Interface the same in all running

modes except for a few mode dependent menus. (a la Mac menus)


PHENIX Analysis Framework Requirement V

I/O Will provide I/O from/to disk, tape,

HSMS, memory and the PHENIX routing layer

Visualization Will provide interface to HEP visualization

tools: PAW, ROOT, .. Will allow all event data and as much

static data as possible to be viewed superimposed on the PHENIX geometry


Classical Type ofAnalysis Framework

MainInitializationBookkeepingetc.

EventLoop

Module

Module

Module

Co-located

ControlCore vs. User Core vs. User

Run time vs.Compile time

Run time vs.Compile time

ExamplesGEANT: UserPISA: CoreROOT: Core

PEP1: Core, CompilePEP2: User, CompileSTAF: User, RunROOT: User, Run

PEP: CompileSTAF: CompileROOT: Run


“Modern” Type of Analysis Framework

MainEventLoop

Module

Module

ModuleIndi

rect

ion

Indi

rect

ion

Indirection: Software Bus, Communication Layer

•Advantages of indirection:•Insulates Modules (Language …)•Allows distributed architecture•Insulates GUI from Modules


PEP’s LayeredArchitecture

Control Messages

Controller

Event Data Messages

DatabaseEvent Data Store

DataOpera-tor A

DataOpera-tor B

I/O Tool

Data-base Tool

(Graphical) User Interface

This is a typical example of an implementation ofthe “Toaster” Model.

UserLayer

StandardInterface

StandardInterface


SDC Toaster Model


PEP Domains(Shlaer-Mellor OOA)

Controller Data Operators

GUIDatabase

Tool

Interprocess Communication

Data Monitoring

Data Visualization

Tool

In/Out Tool

Error Handling

Architecture

GNU C++ Unix Cernlibs Fortran

Data Structure

Management

PVM OnX DSPACK

C++ STL

CORBA TCL/TK

APPLICATION DOMAINS

SERVICE DOMAINS

ARCHITECTURAL DOMAIN

IMPLEMENTATION DOMAINS


PEP Implementation Choices I

GUI: TCL/TK • Allows both GUI and Macro interface

• Available on many platforms Interprocess Communication: PVM

• Parallel Virtual Machine

• Between TCP/IP and CORBA in complexity, flexibility and overhead

Data Structure Management: DSPACK • Created by Ryszard Zybert et al. for NA49

Database: ORACLE • Robust and fast for simple data structures

• Does the job for off-line

• But an OODB might be better in the long run


PEP Functional Units II

Input & Output Tool • Provides all I/O of data structures (as

opposed to control data which flows through the GUI)

Data Base Tool • Provides local access to either the central

data base or a local, cache data base Central Data Base Server

• Provides access to the central data base at RHIC. Runs on the RHIC cluster.

Visualizer • Provides all graphics services like histogram

display, event display and graphical data interaction

• Provides monitoring of all PHENIX specific aspects of the data flow


PEP Control and Data Flow

User Interface

Process Controller

Main Event Data Flow

Data Operators Data Tools

DatabaseManager

Input

Output

DataMonitor

Calibration

LocalEv. Rec. I

LocalEv. Rec. II

GlobalEv. Rec.

Con

trol

Dat

a F

low

Con

trol

Dat

a F

low

SecondaryData Flow


PEP Functional Units I

Controller • The ``brain'' of PEP

• Starts and stops all other processes

• Controls all communication between processes

• Monitors the status of all processes

• Allocates resources Graphical User Interface (GUI)

• Provides for all interaction between PEP and the user

Data Operators • Accept data structures as input, perform

operations on them and provide data structures as output

• Contains 95\% of all user code


PEP Implementation Choices II

Visualization • Spectra: DSPACK & PAW --> DSS

• Event display: OnX. OS: UNIX & NT

• UNIX: SGI, HP, DEC, IBM, Sun

• NT: On Pentium Pro Languages: C, C++

• Most of core PEP implemented in C

• A bit of FORTRAN for interface to Event Generators and for GEANT


PEP V2 (PEPSI)Single Co-located Configuration


PEP V2Simple Configuration


PEP V2Production System Configuration


PEP Standard Message Passing “Handshake” Procedure

Controller

Input DataOperator

12

3 4

5

6

7 8

Message # From To Message Action by Receiver1 Dop Cont Idle Controller locates idle Input2 Contr Input Process Input read and prepares DS3 Input Contr Done & p Controller prepares message4 Contr Dop Process & p DOP state change & message prep5 Dop Input Deliver & p Insert DS in PVM buffer and submit6 Input Dop DS Buffer Unpack DS7 Input Contr Idle Update proxy8 Dop Contr Processing Update proxy


PEP Controller Internal Objects

Controller

Data Operator A

PVM

PVM

Proxies

Dop A

Dop B

Data Operator B

PVM

CheckIn

PostOffice

Broker

Mailman

ConfigurationFile

Pep Stub

User Code User Code

Pep Stub

PVM


PEP Controller Main Loop

do while ( ActiveOperators )

CheckIn !Checks for mail and !sends it to the postoffice

do while ( new mail in PostOffice )

MailMan gets mail with generic address (Dop) MailMan asks Broker for best specific address (Dop.A.1)

If ( Specific address exists ) MailMan delivers mail to Proxy Proxy Object sends mail

to real object Original mail in Postoffice

is destroyed else Original mail is kept end if

end do

Audit

end do


PEP Data OperatorSimple State Model

IDLE

RECEIVING EVENT

PROCESSING EVENT [event]

DELIVERING EVENT

INITIALIZING [initg]

FINISHING [finis]


PEP Data OperatorRealistic State Model

IDLE

RECEIVING EVENT

PROCESSING EVENT [event]

DELIVERING EVENT

INITIALIZING [initg]

FINISHING [finiG]

SPAWNING

PRE-NATAL

FINISH RUN [finiR]

DEAD

FINISH RUN [finiR]

INITIALIZE RUN [InitR]


Pep User Module “Hooks”

Initialize DOP

JobInit

Loop over Runs

RunInit

Loop over Events Event

RunFinis

JobFinis

End of DOP

Error

PEP Data Operator Core User Module Program Units


PEP WAN Access to the Central Database

Controller

Data Ope- rator

Database Tool

Local Data- base

Remote Database Access

Local Database

Remote Database Access Database

Tool ORACLE Interface [proC]

Central ORACLE Database

Off-site Location

RHIC Computing Facility

ORACLE table

DSPACK Object

DSPACK Object

Controller

Data Ope- rator Database

Tool

ORACLE Interface [proC] ORACLE

table

PHENIX Event Reconstruction Facility

DSPACK Object


PEP Physics Module Types

EVENT GENERATOR [Gen]

HIT GENERATOR [Hit]

SIGNAL GENERATOR [Sig]

FEE SIMULATOR [Fee]

LEVEL 1 TRIGGER [Tl1]

LEVEL 2 TRIGGER [Tl2]

CALIBRATOR [Cal]

LOCAL RECONST. [ReA]

GLOBAL RECONST. [ReC]

EVENT BUILDER [Evb]

DsGen

DsHit

DsSig

DsGenI

DsHitI

DsSigI

DsFeeI

DsFee

DsTl1

DsTl1I

DsTl2I

DsTl2

DsEvb

DsCal

DsReB

DsReC

DsPar

DsEvbI

DsCalI

DsReAI

DsReCI

DsParI

PAIR RECONSTRUCTION [Par]Input Data Structure

Module Type

Event Data Structure

ANALYSIS [Ana]

N Times

DsAnaI

SUBSYSTEM RECONST. [ReB]

DsReBIDsReA

N Times

N Times


PEP Module Types I(Simulation)

Event Generator [Gen]• Generates simulated events in the form of a

list of particles created at the vertex Hit Generator [Hit]

• Accepts the Gen output and generates hits in all sensitive volumes. (GEANT Application)

Signal Generator [Sig] • Accepts the Hit output, sums hits over all

readout volumes and calculates the final physics output signal in each read-out channel

FEE Simulator [Fee] • Simulates the response of the front-end

electronics to provide the digital signals in each channel. Typical effects: digitization, discriminator walk, pedestals etc.


PEP Module Types II(Trigger)

Level 1 Trigger Simulator [Tl1] • Simulates all aspects of the data handling in

the level 1 trigger Level 2 Trigger Simulator [Tl2]

• Simulates all aspects of the data handling in the level 2 trigger

Level 3 Trigger Simulator [Tl3] • Simulates all aspects of the data handling in

the level 3 trigger


PEP Module Types III(Initial Off-line)

Calibrator [Cal] • Accepts the raw data as input an either

calculates calibrations constants or performs all calibrations and corrections of the raw data.

Local Event Reconstruction [ReA] • Performs all event reconstruction at the

detector level which an be done in a stand-alone mode. Examples: Cluster finding in EMC, Hit finding in the Drift Chambers

Sub-system Event Reconstruction [ReB] • Performs all event reconstruction at the sub-

system level, which is done based on the input of information from another detector in the same sub-system. Example: Finding muon tracks in the muon tracking systems based on a list of identified muons from the muon identifier.


PEP Module Types IV(Late Off-line)

Global Event Reconstruction [ReC] • Performs all event reconstruction at the

global PHENIX level, where information from other sub-systems is necessary. Example: Particle identification

Data Mining [DaM] • Perform all function related to data mining:

preparation of data in storage system and search of data.

Analyzer [Ana] • Performs subsequent data analysis.


PEP Current Status

PEP has been frozen as of Mar 1997 RHIC Software Collaboration

PHENIX has decided to collaborate with STAR and the other RHIC experiments on Core Off-line Software

Straw man model• Analysis Framework: STAF

• Display: ROOT

• Database: ORACLE

• Event Data Storage: Outcome of Computational Grand Challenge Project on Data Organization

phenix off-line computing system pep the phenix event processor the phenix collaboration presented...

Documents