doe review – august 2010

Software QASafety Systems

atSLAC

Enzo CarroneControls Department – Safety Systems

SLAC National Accelerator Laboratory

Assessment of the- Natural Phenomena Hazards, - Quality Assurance- Work Planning and Control- Safety Software, and - Control of Hazardous Energy Programs

DOE Review – August 2010

Safety Software includes:

-Safety System Software: it performs a safety function as part of a structure, system, or component and is cited in either (a) a DOE approved documented safety analysis or (b) an approved hazard analysis.

Safety Software

Courtesy of Carl MazzolaDOE ES&H, Office of Quality Assurance Programs

Safety Software includes:

- Safety and Hazard Analysis Software and Design Software: used to classify, design, or analyze nuclear facilities. This software is not part of a Structure, System, or Component (SSC) but helps to ensure the proper accident or hazards analysis of nuclear facilities or an SSC that performs a safety function.

Safety Software

Safety Software includes:

- Safety Management and Administrative Controls Software – it performs a hazard control function in support of nuclear facility or radiological safety management programs or technical safety requirements or other software that performs a control function necessary to provide adequate protection from nuclear facility or radiological hazards.

Safety Software

Level A:

-Software failure that could compromise a limiting condition for operations;

-Software failure that could cause a reduction in the safety margin for a safety SSC that is cited in DOE approved documented safety analysis;

-Software failure that could cause a reduction in the safety margin for other systems […];

-Software failure that could result in non-conservative safety analysis, design or misclassification of facilities or SSCs

Description of Grading Levels

Level B:

-Includes safety software applications that do not meet Level A criteria but meet one or more of the following criteria:

-Safety management databases used to aid in decision making whose failure could impact safety SSC operation.- Software failure that could result in incorrect analysis, design, monitoring, alarming, or recording of hazardous exposures to workers or the public. - Software failure that could comprise the defense in depth capability for the nuclear facility.

Description of Grading Levels

Level C:

- Includes safety software applications that do not meet Level B criteria but meet one or more of the following criteria:

-Software failure that could cause a potential violation of regulatory permitting requirements.-Software failure that could affect environment, safety, health monitoring or alarming systems. - Software failure that could affect the safe operation of an SSC

Description of Grading Levels

Functional Area: Safety-Related Software Applications Criteria (NQA-1-2004)

Findings:

SS.1.12-P2-009 A SLAC-wide safety software inventory has not been identified, documented, and maintained.

SS.1.13-P2-010 Graded approach for implementation of software requirements is not complete or formalized for all three types of safety software.

Functional Area: Safety Instrumented System Criteria (ANSI/ISA 84.01)

Observation:

SS.2.12-P3-006 Requirements associated with use of Safety Integrity Levels for Safety Instrumented Systems are not fully implemented per ANSI/ISA-84.00.01-2004.

What we have now

CCR Equipment

LW CPU +I/O LE CPU +I/O

LI20 I/O MCC I/O

Note: Only Chain A Shown

What we are building (CCR Upgrade)

1.Software project management2.Software risk management3.Software configuration management4.Procurement & vendor management5.Software requirements identification & management6.Software design & implementation7.Software safety design8.Verification & validation9.Problem reporting & corrective action10.Training of personnel in the design, development, use & evaluation of safety software

10 Required SQA Work Activities

Software Configuration Control

Pilz Allen-BradleyUse CVS for Version Control

of SoftwareYES YES

Manage check-in/out of CVS with procedures

YES YES

Track & Check Checksum YES YES

Software Download is password protected

YES

Download over network?No; local connection

only possibleNo; not allowed

Download to wrong CPU across ProfiNet network?

N/A N/A

Protection against wrong safety program load (chain A

vs B)N/A

No; isolated networks and different CPU names/IP addresses even if on same network

Hardware configuration is loaded; safety modules have hardware DIP switches. Hardware

configuration error causes fail-safe shutdown

Current Architecture Proposed Architecture

No; not allowed. Local serial connection only allowed.

YES

YES

Siemens SAFETY

YES

YES

Siemens has two levels of password protection – one for the safety hardware setup and another for the safety program.

CVS

Change (and risk) Management

Safety Systems at SLAC

Change Control Board (CCB)

• Reviews change requests submitted by Project Managers;

• Authorizes new projects approving Project Initiation Documents (PID);

• Acts as a consulting body to the Section Leader (e.g. for acceptance of follow-up to reviews);

• Maintains, reviews and approves corrective actions and requests from customers (using a tracking database).

Program Governance ModelProjects are managed through a matrix structure internal to the Section.

CCR Relocation – An Organizational Perspective E. Carrone

Project Initiation and Design Review

Lifecycle

Engineering Work Order Quality Tracking Sheet (EWOQ)

Project QA Process Example

Review Process

Review Process

• Minor Modifications: adding or moving an emergency off button, BSOIC, or Ion Chamber, equivalent device substitutions such as upgraded annunciator panels, or minor logic changes that improve performance but are not changes in the logic specification;

• Medium Changes: redesigns of stopper, BTM, BSOIC, PIC Chassis, or power supply interface chassis, or minor changes in PPS logic specification;

• Large Changes: new PPS zones, new BCS regions, complete PPS rebuilds or significant logic modification.

Future upgrades

MCC

Linac Sector PPS’

CCR Linac Supervisory

I/O

BSY+

PEP-X

SSRL

+

+

???+

Note: Only Chain A Shown

Cyber Security

Specifications and Certification

• Finite State Machine;

• MatLab, Simulink, Stateflow.

My most pressing questions:How to streamline the process? Can we take credit for an automatic, extensive software-based test?Where does cyber security fit?

ZoneEntry

Enabled

Ignition: Disabled

RFWarning

Timer

RFPermit

CMD: Set RF Permit

RF Timer CompleteCMD: Set

Beam Permit

Ignition: Disabled

E-Stop

CMD: Set Zone Entry

E-StopSecure Loop Fault,Ignition Disabled

E-Stop,Entry Loop

Fault

CMD: SetZone Entry

CMD: Set Zone Entry

OFF

Fault -Crash

ReadyFor Beam

RADWarning

Timer

Radiation timer Complete

CMD: Set Zone Entry

E-StopSecure Loop Fault,Ignition Disabled

E-StopSecure Loop Fault,Ignition Disabled

E-StopSecure Loop Fault,Ignition Disabled

CMD: Set RF Permit

CMD: Set RF Permit

The Bottom Line

“In God we trust, all others bring data.”

- W. Edwards Deming