6/21/2007 PHENIX - MJL 1

cosmic raymuons

Mike Leitch, LANLRun Coordinator

Run7 Report

PHENIX 2007 Collaboration Meeting

Boulder, 11-13 July 2007reaction plane

6/21/2007 PHENIX - MJL 2

Run7 Chronology• Originally hoped to start with Nov 1st cool-down

and PHENIX planned for:– AuAu – 15 wks (1.1 nb-1), pp – 10 wks (32 pb-1)

• Delayed by continuing resolution & meager FY06 level to match• But PHENIX started anyhow:

– Cosmic rays starting Jan 9th

– “Run7 is on” Feb 1st

– Start of cool-down Feb 8th

– Cryo problems Feb 24th

– Rings cold & with beam Mar 13th

– Start of Physics Mar 27th

• End of Run7 Jun 26th

• 13 wks of physics (Mar 27th – Jun 26th)

cosmics

6/21/2007 PHENIX - MJL 3

• 3/27 "Start of physics" 1st run 227016• 4/2 23:59 Change to BBCLL1(>1 tubes) 1st run after change 228040

for main MB trigger • 4/4 14:15 scaler update change 1st run after change 228040

from 5 to 30 seconds • 4/25 HBD West out 1st run after this 231210

(during maintenance day) • 5/11 20:18 last level-2 config change 1st run after 234416

(E/p > 0.35 instead of > 0.5) • 5/21 9:30 am "++" magnetic field 1st run after this started

235889 - 5/23 10:00 am

• 5/23 10:00 go to "-+" magnet config 1st run after change 236133(during maintenance day)

• 6/4 14:22 fixed PC-E channels finally about 237717 • 6/6 5pm low-energy run last run before this 237831

- 6/7 4pm (after maintenance day) 1st run after it 238022 • 6/12 major DC LV repair after 238681

West-south sectors 6,7,8 • 6/12 12:05 am (west) converter run includes runs between

238682 - 6/14 8:00 am and 238854 (inclusive)

run 238943 is after it is out • 6/26 11:00 end of run last run 240121 • 6/29 2:00 end of run party....

Important Run7 Dates and Run Numbers

6/21/2007 PHENIX - MJL 4

PHENIX Physics Goals for Run7• Increase statistical & systematic

precision of rare signals in AuAu, e.g. J/, jet correlations, etc

• Increase reach in pT, especially with PID from new TOF-West detector (pT > 8 GeV/c)– Identified particle spectra – Identified leading particles in

jets• Factor of two or more

improvement in Reaction Plane resolution - valuable to many signals– v2 for J/, - new– electrons, hadrons – extended

• Low-mass lepton pairs with the HBD

simulatio

n

Submitted to PRL

6/21/2007 PHENIX - MJL 5

TOF-W

RXNP

HBD

MPC-N

Four New Detector Subsystems for Run7

6/21/2007 PHENIX - MJL 6

Centrality

RXNP

MPC

BBC

< c

os

2(

so

uth -

n

ort

h )

>PHENIX Reaction Plane Measurements Now

from MPC, RXNP & BBC

RP

vv

expt2true

2

< c

os

2(

A+

B -

tr

ue

) >

< cos 2(A – B ) >

RXN

BBCMPC

MPC

Reaction Plane Detector

6/21/2007 PHENIX - MJL 7

TOF-West Particle Identification

6/21/2007 PHENIX - MJL 8

5.12 Billion Min-bias events

813 b-1

(~3.4 x Run4)

6/21/2007 PHENIX - MJL 9

All Run7 % Last week (6/19-26) %PHENIX

BUP

CA-D luminosity

3260 b-1 482 b-1

PHENIX ZDC luminosity

Uptime

20.46B ZDC/9.8b= 2088 b-1

64.4%3.41B ZDC/9.8b

= 348 b-172.2%

PHENIX BBC luminosityVtx cut*

6.27B BBCLL1/(0.92*6.847b)

= 995 b-1

47.7%960M BBCLL1/(0.92*6.847b)

= 152 b-1

43.7% 70%

PHENIX live luminosityLive time

5.12B BBCLL1/(0.92*6.847b)

=813 b-1

81.7%854M BBCLL1/(0.92*6.847b)

=136 b-1

89.5%

PHENIX up*live time

59.5%*81.7% 48.6% 72.2%*89.5% 64.6% 60%

Overall 24.9% 28.2%

The PHENIX Luminosity Story

*Vertex eff slightly worse after “+-” “-+” magnetic field change (cause of lower vtx eff for last week than whole run)

6/21/2007 PHENIX - MJL 10

beginningof store

endof store

from yellow beam

from blue beam

Effects of Stochastic Cooling at PHENIX

Improvement in integrated luminosity for PHENIX with Stochastic Cooling (SC)

16%

6/21/2007 PHENIX - MJL 11

Fraction of Time at Collision

6/21/2007 PHENIX - MJL 12

Luminosity/Day vs Day

6/21/2007 PHENIX - MJL 13

PHENIX Triggering & fast analysis for Run7

Level-1 Triggering just Minimum-Bias (+UPC)• 5 kHz event rate through DAQ• only small fraction of integrated luminosity missed due to rates above the DAQ limit in the 1st ½ hour or so of storesLevel-2 filtering in ATP’s producing selected, parallel sample of enriched events for fast processing• J/ ; J/ ee; high-pT level-2 triggers

• 10% of raw data size• sent to CCF in France via GRIDFTP• analyzed in France for fast results

10% of min-bias events sent to Vanderbilt for fast analysis

• check analysis/calibrations, prepare for full analysis pass• provide events needed for mixing in level-2 analysis

Counting house machines for fast HBD analysis

6/21/2007 PHENIX - MJL 14

PHENIX Data Acquisition (DAQ)

DAQ Advances:• level-2 code speedup• additional buffer box• fix SEB memory leaks• more compact HBD data format• etc

Present DAQ performance:• up to 5 khz event rate & 700 Mb/sec throughput

6/21/2007 PHENIX - MJL 15

(plot from Tom Throwe)

6/21/2007 PHENIX - MJL 16

cosmics

PHENIX Shakedown/Cosmic Ray Running(Jan & Feb)

Need Cosmic Ray background measurement for W physics• Cosmic/W estimated to be 1/1 for p>=10 GeV, but estimate is unreliable – need measurement!Also Cosmic ray in EMCal looks much like high-pT photon

6/21/2007 PHENIX - MJL 17

PRL98, 232301 (2007)

Statistical & systematic precision of new Run7 AuAu data will improve this picture – But will also benefit enormously from new dAu data that can constrain the cold nuclear matter baseline accurately

J/’s in AuAu Collisions at PHENIX

PRL96, 012304 (2006)

6/21/2007 PHENIX - MJL 18

Fro

m T

ony

Fra

wle

y, h

eavy

PW

G,

Jan

4th

Expected Results for J/ flow from Run7

6/21/2007 PHENIX - MJL 19

for ~8% of present Run7 integrated luminosity(~16,000 J/ for present luminosity sum)

J/

unlike-sign pairslike-sign pairs (randoms)

thanks to Catherine Silvestre

6/21/2007 PHENIX - MJL 20

J/ e+e-

about 15% of level-2 data

thanks to Ermias Atomssa& Raphael Granier de Cassagnac

6/21/2007 PHENIX - MJL 21

0’s & ‘sthanks to Justin Franz

from about 16% of present data

0

extended range in pT

6/21/2007 PHENIX - MJL 22

• A “hadron-blind” detector to detect and track electrons near the vertex.”

• Dalitz rejection via opening angle – Identify electrons in field free

region– Veto signal electrons with

partner• HBD: a novel detector

concept: – windowless CF4 Cherenkov

detector – 50 cm radiator length– CsI reflective photocathode – Triple GEM with pad readout

• The HBD will improve our S/Bby a factor of ~100

Hadron Blind Detector (HBD)

signal electron

Cherenkov blobs

partner positronneeded for rejection e+

e-

pair opening angle

Submitted to PRL

6/21/2007 PHENIX - MJL 23

electrons

hadrons

Hadron Blind Detector (HBD)West side of HBD taken out Apr 25th for repairs• HV trips with large stored energy damaged detector• Now being refurbished with new GEM’s & fixed HV • ½ of the East side of HBD still in

6/21/2007 PHENIX - MJL 24

PHENIX Issues & Failures• HV problems

– Slow or dead servers– SA1 screen slow, or dead– Non-standard subsystem HV control – HBD, ToF-W, ToF-E– Unclear subsystem standing orders

• LV problems– Connections or burned out LV modules

• DAQ front end– HBD event size and zero suppression– DC large packets– MPC fiber split, muTr & EmCal new formats done

• SEB memory leaks• Buffer Boxes

– Confusion about Box7 (LVL-2) & its transfers– Weird transfer impact on DAQ speed when trigger rate high– Several disk failures

• Event builder & ATP’s– Did not reach near 5khz event rate until last few weeks of run!– Only one expert, many changes– Will performance survive for Run8 and Run9??– Trigger/level-2 configuration complicated – few know how to do it– Diagnostics weak for DAQ operator

6/21/2007 PHENIX - MJL 25

PHENIX Issues & Failures - 2

• muTr HV trips & dead channels (capacitor removal in progress)• Level-2 analysis slow starting; partly because of slow DAQ level-2 start

caused by struggles to get DAQ to run fast• SA2 under utilized• Communication in English difficult for some young subsystem specialists

and shifters– recommend one person in subsystem team be fluent in English

• Never had a chance to start new mandatory shift assignment scheme because of run start uncertainties

• HBD data size– Already implemented more compact format, but will (hopefully) have

many more good channels next year (~24/6)– Drop to fewer than 12 time samples?– Reduce large tail in event size with light baffling?– More sophisticated zero-suppression scheme

• Haggerty-less operation next year – a challenge to us all

6/21/2007 PHENIX - MJL 26

Other Issues• Water cooling

– Availability of chiller & evaporator status to shift crews so they diagnose cooling problems quickly & call for help

– Coupling of magnet and electronics cooling• Air conditioning

– Humidity swings in IR, can they be minimized to alleviate HV problems?– Get all four air conditioners to run (most of the time)

• Flammable gas detectors– Many recalibrations or head replacements– New technology that is stable at levels being monitored?

• Fire Alarms & power shutdowns– HSSD smoke detectors?– Fix light bulb coupling to PHENIX power shutoff system

• Magnet trips – CMI mostly• Access to MCR logbooks (readonly) for PHENIX shift crews

– New xterm for acnlinxx access?• Communications between workers in IR and PHENIX control room for

debugging problems quickly and efficiently (old radios not allowed)?– POTS with long cords or wireless phones?

6/21/2007 PHENIX - MJL 27

HV trips 161 HBD,muTr,DC,PCserver/MF 10massive trips 9

Front end feed 60 muTr AMU& glink

Back end EB,BB,RC 34 EB,BBSEB,CorbaOncs Cleanup

Magnet trips 14 CMICooling Water,A/C 8 or more…Power Off Fire Alarms 3 HSSD smoke detectors

& Light bulbs

Rough Survey of PHENIX Problems in Last 2 weeks of Run7

6/21/2007 PHENIX - MJL 28

Concluding Remarks• Over 3 times Run4 luminosity (~813b/241b)

PLUS new capabilities from new detectors– E.g. 3-4 times better Reaction plane resolution– Full /K/p PID up to 9 GeV/c– major advances in our physics should result

• Analysis in progress already thanks to level-2 filtering & mature analysis model

• HBD a very state-of-the-art detector, but lessons learned in Run7 should allow full operation in Run8 & beyond

• Excellent stores when they could be delivered, but too many breakdowns– Stochastic cooling - a step towards the future– Low energy (9.2 GeV) running feasible

6/21/2007 PHENIX - MJL 29

Backup

6/21/2007 PHENIX - MJL 30

charged multiplicity in RXN (a.u.)

centrality selections are givenby BBC charged multiplicity

z-vertex selections by BBC[-40,40] cm

[-30, -20] cm[-5,5] cm

[20,30] cm

total[~500]

[500~1500][1500~]

RXN south charged mult. (a.u.)

RX

N n

ort

h c

ha

rge

d m

ult

. (a

.u.)

south (rad)

north (rad)

< c

os

2(

sou

th -

n

ort

h )

>

mid-central collisions

PHENIX Reaction Plane Detector (RXNP)200GeV Au+Au collisionsRUN7 - 2007/Mar

6/21/2007 PHENIX - MJL 31

Luminosity fraction captured by PHENIX DAQ saturation in red

Fraction of luminosity for late April stores (at bottom) 97.8% for 4.8 khz DAQ rate

• for ½ hr turn-on time

6 min 21 min 18 min 16 min 32 min 26 min

0 min

starts at10 amFri Apr 27th

6/21/2007 PHENIX - MJL 32

date time store cooled b-1/hrZdc

(pk)

FM

(x10-3)

5/15 2:10 8776 no 0.9804 34.97 28.04

5/30 2:10 8875 no 0.8789 32.07 27.41

5/30 23:50 8879 no 0.9746 34.85 27.97

5/19 14:47 8805 yes 1.1524 35.19 32.75

5/25 16:30 8849 yes 0.9874 30.34 32.54

6/1 8:00 8886 yes 1.0530 33.24 31.68

Average Improvement with SC:

03.016.135.081.27

57.032.32

Improvement in integrated luminosity for PHENIX with Stochastic Cooling (SC)

Store 8879(no cooling)

Store 8886(cooling on)

6/21/2007 PHENIX - MJL 33

T. Roser’s vertex shape

Differentiation of vertex into distinct peaks due to Stochastic Cooling

ZDC vertex (ZDC Wide)BBC vertex (BBC no vertex cut)BBC vertex (BBC ±30 cm vertex cut)

6/21/2007 PHENIX - MJL 34

HPSS To

Data flow

rftpexp

phenix

ccphenix

srm/dcache@RCF

srm/dcache10 TB

Online filtered PRDF (21TB)

DST, 7 MB/s, srmcpdccp???

sps disk4.1 TB

DB snapshot, scp or bbftp

bqsworkerssrmcp

srmcp

HPSS TB

(database server)

Input prdf flowOutput plan AOutput plan B

Aggregation not done

Offline filtered PRDF

6/21/2007 PHENIX - MJL 35

PRDFsRaw data filesGridFTP to VU30 MBytes/sec

11

33

44

FDT45

MB/s

22

Vanderbilt Farm1600 CPUs, 80 TBytes disk

PRDFs nanoDSTs

ReconstructionReconstruction200 jobs/cycle200 jobs/cycle

45TB and 200 CPUsAvailable for Run7 Reconstruction

4.4TB of buffer disk space

Dedicated GridFTP ServerDedicated GridFTP ServerFirebirdFirebird

18 hours/job

FDT45

MB/s

770 GBytesper cycle

nanoDSTsReco output to RCFGridFTP 23 MB/sec

RCFRHIC computing facility

6/21/2007 PHENIX - MJL 36

HBD Status • We believe we have determined the main cause

of the sparking problems with the both detectors:

• Normal GEM spark would cause Lecroy HV to trip• Mesh also trips leaving large stored energy on filter

capacitor• As GEM voltage goes to zero, large DV develops

across gap between top GEM and mesh, ultimately resulting in a large spark

• This spark induces sparks in other GEMs (massive trips) by propagation of scintillation light

There is however a danger in the “One source of all troubles” theory and we are still looking for more possible causes of the problems we had during the run

from T.Hemmick

6/21/2007 PHENIX - MJL 37

Pock Marks

• Human eye could see the holes.• Holes smaller than pock parks.• Pocks almost certainly from meshGEM

from T.Hemmick

6/21/2007 PHENIX - MJL 38

Low Energy Running at PHENIX• 9.19 GeV/nucleon• PHENIX RXNP trigger

– timed in at full energy– expect ~96% efficiency (compared to

~15% for BBC)– ZDC trigger should be very inefficient

due to large Fermi motion wrt longitudinal momentum

EvB

bufferbox

• Wednesday afternoon could run our DAQ with the “blue low-energy clock”• But Thursday morning, when beam was in the rings, we could not• Presumably due to glitches in the clock (associated with other clock events that were not there on Wednesday??)