pixel systemtest workshop, 08-02-2007, kerstin lantzsch joachim schultes university of wuppertal dcs...

Pixel Systemtest Workshop,Pixel Systemtest Workshop, 08-02-2007, Kerstin Lantzsch 08-02-2007, Kerstin Lantzsch

Joachim SchultesUniversity of Wuppertal

DCS status and experience in the DCS status and experience in the system testsystem test

Pixel Systemtest Workshop,Pixel Systemtest Workshop, 08-02-2007, Kerstin 08-02-2007, Kerstin LantzschLantzsch

OutlineOutline

1. short description of system1. short description of system• HWHW• SWSW• system overviewsystem overview

2. General Experience + Special Issues2. General Experience + Special Issues

3. conclusion3. conclusion


BBM

DCS-PCsCAN-Open protocol TCP/IP

Environm.

SC-OLink

BOC

Optoboard

Door

Cover

T

Data

DataVPin VISet

RegulatorStation

Wiener

Module

LV-PP4

T

HV VDD VDDA

Iseg

HV-PP4

T

Interlock

System

Dis

tanc

e fr

om in

tera

ctio

n po

int

Sensors

The DCS Hardware The DCS Hardware

CAN-Open protocol

VVDC

BBIM


HWHW

HV:HV:

2 Iseg crates (1 channel/Detector-module, 1 iseg module/sector)2 Iseg crates (1 channel/Detector-module, 1 iseg module/sector)• 16 modules EHS16 modules EHS• 8 modules EHS + EHQ8 modules EHS + EHQ

LV:LV:

WIENER ( 2 channels/sector)WIENER ( 2 channels/sector)• 4 crates, 12 channels each 4 crates, 12 channels each

PP4 (½ ELMB/sector)PP4 (½ ELMB/sector)• 4 crates à 3 ELMBs à 4 Wiener channels4 crates à 3 ELMBs à 4 Wiener channels

Regulators (1 board/sector)Regulators (1 board/sector)• 2 crates with 12 Boards each (+ 1 „old“ crate, 2 Boards)2 crates with 12 Boards each (+ 1 „old“ crate, 2 Boards)

Optolink:Optolink:

SC-OLink (1 complex channel/sector)SC-OLink (1 complex channel/sector)• 2 crates à 4 ELMBs, 4 complex channels per ELMB (Viset, Vpin, Vvdc)2 crates à 4 ELMBs, 4 complex channels per ELMB (Viset, Vpin, Vvdc)


HW, IIHW, II

Environment:Environment:

BBM BBM • 1 crate à 3 ELMBs (2NTC, 1HS)1 crate à 3 ELMBs (2NTC, 1HS)

Interlock:Interlock:

BBIM (½ ELMB plug/sector T_mod + 1/2 ELMB plug/6 T_opto (+ T_PP2))BBIM (½ ELMB plug/sector T_mod + 1/2 ELMB plug/6 T_opto (+ T_PP2))• 2 crates à 4 ELMBs à 4 plugs2 crates à 4 ELMBs à 4 plugs

BOC-I-BOX (5 racks -> up to 2 crates -> 12 BOCs -> 1-4 sectors)BOC-I-BOX (5 racks -> up to 2 crates -> 12 BOCs -> 1-4 sectors)• 1 crate à 1 ELMB1 crate à 1 ELMB

PP1-BOX PP1-BOX • 1 crate à 1 ELMB1 crate à 1 ELMB

LULU• 2 LU à 4 ELMBs (3 Tmod, 1 FPGA (TOpto, TPP2, IPP1, IBOC, IDSS))2 LU à 4 ELMBs (3 Tmod, 1 FPGA (TOpto, TPP2, IPP1, IBOC, IDSS))

IDB-xIDB-x• 1 IDB-SC for 5 SC-OLink crates (5 * 4 * 4 = 80 ELMB channels)1 IDB-SC for 5 SC-OLink crates (5 * 4 * 4 = 80 ELMB channels)• 1 IDB-LV for 6 Wiener-crates (6 * 12 = 72 ELMB channels)1 IDB-LV for 6 Wiener-crates (6 * 12 = 72 ELMB channels)• 1 IDB-HV for 5 iseg slots (5*2*8 = 80 ELMB Channels)1 IDB-HV for 5 iseg slots (5*2*8 = 80 ELMB Channels)


SWSW

FITs: functional orderSIT: geographical orderFSM: hierarchical

DDC: DAQ-DCS communicationFSM commands + set voltages + Temps


overview of systems in SR1, Ioverview of systems in SR1, I

dcs2dcs2dcs1dcs1 dcs3dcs3_DriverConnection_DriverConnection

Peak (iseg)Peak (iseg)

Iseg OPCServerIseg OPCServer

PVSS: OPCClient (num 3)PVSS: OPCClient (num 3)(-data dcs3 –event dcs3)(-data dcs3 –event dcs3)

_DriverConnection_DriverConnection

KVASER (ELMBs)KVASER (ELMBs)

CANopen OPCServerCANopen OPCServer


WIENER OPCServerWIENER OPCServer


UserInterfaceUserInterface(-data dcs3 –event dcs3)(-data dcs3 –event dcs3)

(SR1EnvironmentCooling_Disc.pnl)(SR1EnvironmentCooling_Disc.pnl)

network drive S: ->network drive S: ->

D:\PVSS_Project_UpdateD:\PVSS_Project_Update

(Subprojects)(Subprojects)

network drive S: -> network drive S: ->

\\dcs1.pixel.cern.ch\\\dcs1.pixel.cern.ch\

PVSS_Project_UpdatePVSS_Project_Update

_FW _FW [21][21]

Data-, Event- ManagerData-, Event- Manager

User InterfaceUser Interface

Control Manager Control Manager -num 11 (FSM), 20 (Cooling)-num 11 (FSM), 20 (Cooling)

Subprojects on S:\...Subprojects on S:\...

dist -> [3], [40]dist -> [3], [40]

Sysname:Sysname: ATLPIX_FSM_dcs1ATLPIX_FSM_dcs1

network drive S: -> network drive S: ->



_FW_DDC _FW_DDC [33][33]

Data, Event, UIData, Event, UI

DIMDIM

Ctrl 12 (DDC Handling)Ctrl 12 (DDC Handling)


dist -> [3], [21]dist -> [3], [21]

Sysname:Sysname: ATLPIX_DDC_dcs3ATLPIX_DDC_dcs3

ATLASPixDCS ATLASPixDCS [3][3]

Data, Event, UIData, Event, UI

Archiving (RDB)Archiving (RDB)

Ctrl 10 (ELMB Watchdog)Ctrl 10 (ELMB Watchdog)


dist -> [40]dist -> [40]

Sysname:Sysname: ATLPIX_3ATLPIX_3


overview of systems in SR1, IIoverview of systems in SR1, II

control3control3

_FSM _FSM [26][26]


User Interface (DEN)User Interface (DEN)

dist -> [3], [21]dist -> [3], [21]


Sysname:Sysname: ATLPIX_FSM_CONTROL3ATLPIX_FSM_CONTROL3

dcs4dcs4

_CAN-ELMB_FW _CAN-ELMB_FW [4][4]

KVASER (ELMBs)KVASER (ELMBs)

CANopen OPCServerCANopen OPCServer

PVSS: OPCClient (num 7)PVSS: OPCClient (num 7)


User Interface (DEN)User Interface (DEN)

Control ManagerControl Manager

dist -> [3]dist -> [3]

Sysname:Sysname: ATLPIX_4ATLPIX_4

ServiceTestServiceTest

ServiceTest ServiceTest [99][99]


ServiceTest-MainPanel (GE)ServiceTest-MainPanel (GE)

dist -> [3]dist -> [3]

Subprojects localSubprojects local

Sysname:Sysname: ATLPIX_ServiceTestATLPIX_ServiceTest

_RemoteUI _RemoteUI

Remote User InterfaceRemote User Interface

ATLASPixDCS (dcs3): ATLASPixDCS (dcs3): SR1Menu.pnlSR1Menu.pnl

nezwork drive S: -> nezwork drive S: ->




PP2PP2

many problems with PP2 crates for endcapmany problems with PP2 crates for endcap

• new hardware slower, new hardware slower, was not taken into account in PVSSwas not taken into account in PVSS (FIT and FSM), (FIT and FSM),commands were sent while Controller was still busycommands were sent while Controller was still busyno reliable execution of commandsno reliable execution of commands-> introduction of very conservative -> introduction of very conservative delaysdelays

• no ADC reading (no feedback about Vset) no ADC reading (no feedback about Vset) -> missing -15V supply for controller, need PS monitored by PVSS-> missing -15V supply for controller, need PS monitored by PVSS

• problems in ELMB/FPGA code -> already solved by firmware upgradeproblems in ELMB/FPGA code -> already solved by firmware upgrade• move trimmer to inhibit/uninhibit channel, move trimmer to inhibit/uninhibit channel, • GU when moving trimmer of Board1, Ch1, GU when moving trimmer of Board1, Ch1, • sending inhibit for one channel caused another channel to be inhibitedsending inhibit for one channel caused another channel to be inhibited• Inhibiting/Uninhibiting of Channels 1 when reading T1/T2 after powering up PP2Inhibiting/Uninhibiting of Channels 1 when reading T1/T2 after powering up PP2

• „„Reset“ (i.e. set all Trimmers to Min) does not work properly in SR1Reset“ (i.e. set all Trimmers to Min) does not work properly in SR1-> solved with new firmware version-> solved with new firmware versionprobably was not recognized more clearly, because most of the time PP2 in SR1 was operated probably was not recognized more clearly, because most of the time PP2 in SR1 was operated in the way that trimmersteps were in the way that trimmersteps were notnot calculated directly, but trimmer was set via moving to calculated directly, but trimmer was set via moving to minimumminimum

• no Imon, because new crates were already defined, and monitoring with PDO3 was not no Imon, because new crates were already defined, and monitoring with PDO3 was not debugged previously with real HWdebugged previously with real HW-> changing to PDO3 now done for PP2_S21, PP2_S22-> changing to PDO3 now done for PP2_S21, PP2_S22


PP2PP2

New Communication Mechanism:New Communication Mechanism:

Acknowledgment between each node: FPGA <-> ELMB <-> PVSSAcknowledgment between each node: FPGA <-> ELMB <-> PVSS

• Commands are acknowledgedCommands are acknowledged (no chance of loosing command without noticing between FPGA and PVSS)(no chance of loosing command without noticing between FPGA and PVSS)

• only as much delay as necessaryonly as much delay as necessary • 1000 ms -> 10 ms (Inhibit)1000 ms -> 10 ms (Inhibit)• 3000 ms -> 200 ms (100 Trimmer Steps)3000 ms -> 200 ms (100 Trimmer Steps)


PP2PP2

Also:Also:

• Watchdog between ELMB and FPGAWatchdog between ELMB and FPGA• FPGA can reset ELMBFPGA can reset ELMB• FPGA failure is recognized by ELMBFPGA failure is recognized by ELMB

• Reduction of number of commands sent by PVSS to ELMBReduction of number of commands sent by PVSS to ELMB• Readout will be stopped/started for commands by ELMB Readout will be stopped/started for commands by ELMB • PDO will be sent to signal routine modePDO will be sent to signal routine mode

(still makes sense to stop routine before long sequence of commands)(still makes sense to stop routine before long sequence of commands)

• New CommandsNew Commands• Firmware version of FPGAFirmware version of FPGA• Firmware version of ELMBFirmware version of ELMB• multiple commands reasonable? e.g. „inhibit_Board“? „inhibit_Modules“ ...?multiple commands reasonable? e.g. „inhibit_Board“? „inhibit_Modules“ ...?

can be implemented on a short timescale in ELMB codecan be implemented on a short timescale in ELMB code

• Number of ADC bits for Monitoring values will be increased (8 bit now)Number of ADC bits for Monitoring values will be increased (8 bit now)• faster reading of monitoring values possiblefaster reading of monitoring values possible


isegiseg

some iseg channels were not switching off, when using „SR1DirectLink“ some iseg channels were not switching off, when using „SR1DirectLink“ (while switching off one disk at a time and switching on worked)(while switching off one disk at a time and switching on worked)

• FSM: order of switching on introduced delays (-> PP2) before switching on iseg-FSM: order of switching on introduced delays (-> PP2) before switching on iseg-channels -> not so many commands were sent simultaneouslychannels -> not so many commands were sent simultaneously

• introducing delay between commands for modules also helped (but not 100%)introducing delay between commands for modules also helped (but not 100%)

problem was the way PVSS accessed the opc items:problem was the way PVSS accessed the opc items:• 1 item for each iseg module with one bit for each channel1 item for each iseg module with one bit for each channel• this item is „split“ by OPC client and „distributed“ to the DPEs this item is „split“ by OPC client and „distributed“ to the DPEs

(trans.type bitstring, DPE bool)(trans.type bitstring, DPE bool)• old values for first bits caused channels to be turned on againold values for first bits caused channels to be turned on again

using now opc item for iseg channel instead of module, and „write only“using now opc item for iseg channel instead of module, and „write only“• trans.type bool for DPE booltrans.type bool for DPE bool• no more problems seen till now (switching off 4 times, first week of january)no more problems seen till now (switching off 4 times, first week of january)


FSMFSM

many problems seen in system test were problems of underlying layer many problems seen in system test were problems of underlying layer (PP2, iseg)(PP2, iseg)

missing functionality:missing functionality:• order for switching on/off should be flexibleorder for switching on/off should be flexible• also for example switching on without HV should be possiblealso for example switching on without HV should be possible• command for reading in of (default) values (->SIT)command for reading in of (default) values (->SIT)• take into account HV for state for modularity 1 (done for 6/7)take into account HV for state for modularity 1 (done for 6/7)• overall status of FSM („idle“, „active“) instead of only on ROgroup leveloverall status of FSM („idle“, „active“) instead of only on ROgroup level• SW-interlock implemented but not yet testedSW-interlock implemented but not yet tested


SITSIT

•provides mapping for FSM provides mapping for FSM

•improved xml parsing (mapping and configuration)improved xml parsing (mapping and configuration)

•reading in of configuration filesreading in of configuration files


DDCDDC

• supports all FSM commandssupports all FSM commands• SWITCH_ONSWITCH_ON• SWITCH_OFFSWITCH_OFF• RECOVERRECOVER• RESETRESET• ENABLEENABLE• DISABLEDISABLE

• Setting of VoltagesSetting of Voltages• SET_VDDASET_VDDA• SET_VDDDSET_VDDD• SET_HVSET_HV• SET_VISetSET_VISet• SET_VPinSET_VPin• SET_VVDCSET_VVDC

• Publishing of values:Publishing of values:• Set Voltages, measured Voltage, measured CurrentSet Voltages, measured Voltage, measured Current• T_Module, T_OptoT_Module, T_Opto

• improved monitoring of necessary processes in work (dim, dist, ctl)improved monitoring of necessary processes in work (dim, dist, ctl)

• boc monitoring in work (DT)boc monitoring in work (DT)

• message transfer (MT) todomessage transfer (MT) todo


Archiving, TrendingArchiving, Trending

PVSS access to Oracle database?PVSS access to Oracle database?

trending very painful for large amount of DPEs or longer timespantrending very painful for large amount of DPEs or longer timespan(trying to access 24 T_Opto-DPEs of ~2 weeks before almost impossible with PVSS (trying to access 24 T_Opto-DPEs of ~2 weeks before almost impossible with PVSS trending, but also problems with recent trends)trending, but also problems with recent trends)

-> PVSS dataviewer-> PVSS dataviewer

PVSS Data Viewer

What's new :

Improved data export Possibility to filter values depending on their PVSS status word Improved data extracting (faster and can now display deleted elements)

To do : Un-/Aliasing of DPEs Additional information on requests Different display options

(https://twiki.cern.ch/twiki/bin/view/Atlas/PvssDataViewer)

Olivier Pisano


Temperature reading (BBIM/BBM)Temperature reading (BBIM/BBM)

BBMs were defined as standard ELMBBBMs were defined as standard ELMB• Temperatures had to be converted from Voltages manuallyTemperatures had to be converted from Voltages manually• new ELMB-type „BBM_NTC“new ELMB-type „BBM_NTC“

Problem with Temp-conversion in OPC-server (BBIM, BBM)Problem with Temp-conversion in OPC-server (BBIM, BBM)• if disconnected (Vref = VNTC) value cannot be evaluated by OPCserverif disconnected (Vref = VNTC) value cannot be evaluated by OPCserver

no check possibleno check possible• if value ok again, does not get updated immediately inside PVSSif value ok again, does not get updated immediately inside PVSS

wait sometimes minutes for up-to-date valuewait sometimes minutes for up-to-date value(looks like combination of „division by zero“ in OPC-Server and PVSS-smoothing)(looks like combination of „division by zero“ in OPC-Server and PVSS-smoothing)GQ gets actual value, but not practicableGQ gets actual value, but not practicable

• no smoothing, Vref = const 2.5V seems to improve this: update with next SYNCno smoothing, Vref = const 2.5V seems to improve this: update with next SYNC• archiving? archiving?

• amount of data without smoothing amount of data without smoothing • change of value propagated to archive if smoothing applied there?change of value propagated to archive if smoothing applied there?


conclusionconclusion

• iseg problem seems solvediseg problem seems solved• PP2: many updates in the next few days scheduledPP2: many updates in the next few days scheduled• FSM: much additional functionality desiredFSM: much additional functionality desired

pixel systemtest workshop, 08-02-2007, kerstin lantzsch joachim schultes university of wuppertal dcs...

Documents

smoothing change of

vntc value

uptodate value

actual value

pvsssmoothing gq

pvss data viewerwhats

pvss status word improved

check possibleif value