C ModAlcator

−

Alcator C-Mod Digital Plasma ControlSystemPresented by:

S. Wolfe, J. Stillerman, M. Ferrara, T. Fredian, I. Hutchinson

APS DPP05

Denver, CO

Oct. 26, 2005

Abstract

A new digital plasma control system (DPCS) has been implemented for AlcatorC-Mod. The new system was put into service at the start of the 2005 run campaignand has been in routine operation since. The system consists of two 64-input,16-output cPCI digitizers attached to a rack-mounted single-CPU Linux server,which performs both the I/O and the computation. During initial operation, thesystem was set up to directly emulate the original C-Mod ”Hybrid” MIMO linearcontrol system. Compatibility with the previous control system allows the existinguser interface software and data structures to be used with the new hardware. Thecontrol program is written in IDL and runs under standard Linux. Interrupts aredisabled during the plasma pulses to achieve real-time operation. A synchronousloop is executed with a nominal cycle rate of 10 kHz. Emulation of the originallinear control algorithms requires 50 µsec per iteration, with the time evenly splitbetween I/O and computation, so rates of about 20 KHz are achievable. Reliablevertical position control has been demonstrated with cycle rates as low as 5 KHz.Additional computations, including non-linear algorithms and adaptive response,are implemented as optional procedure calls within the main real-time loop.

(MS WORD TEMPLATE for Submission in Fusion Engineering and Design)

• The diagnostics provide information on the magnetic field inside the machine and on the plasma.

• The multiplication by the time dependent ‘A’ matrix (96 inputs x 16 outputs) computes the ‘observers’, that is the 16 physical quantities of interest.

• The pre-programmed target waveforms are subtracted from the observers.

• The PID controllers act on the error signals with time dependent gains.

• The controller matrix (‘M’) is multiplied by the output of the PID to produce the correct action on the relevant actuators, and 16 pre-programmed feed-forward waveforms are added. This allows for both open loop control signals, and reduces the needed gains in the PID feedback stage.

The hybrid’s analog outputs as well as the values of the intermediate steps in the computation are digitized using CAMAC for use in debugging and

diagnosis of the control system operation. Care must be taken that hybrid computed values stay within the hardware voltage ranges throughout all of the steps of the computation. The real-time node (INTEL 8044) plays out pre-programmed scenarios, stepping the waveforms and matrices through their programmed values. In the hybrid, digitally programmed analog circuits to perform the matrix operations and produce the feedback signals. There are also provisions in the hardware for adaptive control, but the difficulties and risks of programming and debugging the real-time computer in 8044 assembler prevented us from pursuing it.

Communication between the data system computers and the hybrid control computer was done over a bitbus interface connected to a micro-vax computer. There are significant maintenance and performance issues with this interface and these obsolete computers. Bitbus is no longer a supported standard and micro-vaxes have not been manufactured for many years. We could have pursued bitbus interfaces for more modern computers, but this seemed

Fig. 1. Alcator C-Mod control system. Hybrid Digital/Analog computer (top). All Digital system (bottom)

.

C ModAlcator

− C-Mod Linear Multi-Input-Multi-OutputControl Loop

• Diagnostic sensors measure fields, density, currents, . . .• Estimators of parameters of interest obtained by multiplication of

sensor signals by “A-matrix” of linear observers

• Pre-programmed Target Waveforms (optionally scaled by plasmacurrent) subtracted from estimated values

• Error signals are multiplied by (time-dependent) pre-programmedPID Gains

• Output of PID stage is multiplied by “M-matrix” of Controllers toproduce demand signal for actuators (power supplies, gas valves,

. . . )

• Pre-programmed Feed-forward “V-waveforms” added to actuatordemands

.

C ModAlcator

− “Hybrid” analog-digital control system inuse from 1991-2004

• Collaboration between MIT and EPFL (Lausanne)

• Implements strictly (piecewise) linear MIMO control system

• Analog computer with digital programming based on DAC cards

• Bitbus interface� No longer a supported standard

� Obsolete Micro-Vax interface card (Q-bus)

• Intel 8044 real-time node (programmed in assembler) updatesmatrices and target waveforms

.

C ModAlcator

− DPCS all-digital real-time control systemreplaced the old Hybrid system at thebeginning of the 2005 Campaign

Design Requirements:

• Drop-in replacement for Hybrid� Run old shots

� Operate transparently on same data structures as Hybrid

� Programmed using existing GUI (PCS)

• Remove constraints imposed by analog hardware

• Allow implementation of non-linear control schemes

• Provide for increased number of Inputs (96 → 128)and Outputs (16 → 32)

.

C ModAlcator

− Control System Configuration Description,Programming, Results Stored in MDSplus

New Data Structures added to support DPCS

• Data-Driven System

• Control and DataAcquisition Actions

controlled by

MDSplus

DISPATCHER

• Communication byMdsIp servers over

Ethernet link

.

C ModAlcator

− DPCS Testing and Migration

• DPCS Run in parallel with Hybrid during 2003-2004 campaign� Outputs not connected to actuators

� Intermediate (computed) signals, outputs compared to

corresponding Hybrid signals

� Observed discrepancies associated with offsets, noise, and voltage

limits of Hybrid analog hardware

• DPCS Control implemented at start of 2005 campaign� Initial power system testing carried out under DPCS control

� First plasma operation 4-Feb-2005

? Successful startup established after 8 attempts

(compare to 6 tries on first day of 2003-2004 Campaign)

? Re-runs of old discharges indicate good reproducibility

� Hybrid run in parallel (disconnected from actuators) for first

months, eventually disabled

.

C ModAlcator

− DPCS Implementation (Platform)

• Xeon single processor server (3.2GHz, 2GB)

• PCI/cPCI Bus extender (SBS)

• Two 64-input, 16-output cPCI digitizer cards (D-Tacq)

• Trigger and 1 MHz clock signals from CAMAC (MPB, Jorway)

• Standard Linux Operating System (RedHat 9)� Diskless boot (PXE, Root over NFS)

� Interrupts disabled during real-time operation

? Kernel call from low-latency digitizer driver library

? No disk i/o, page and swap, network access

? Memory locked down after one dummy pass

• Standard Ethernet Network Interface

.

C ModAlcator

− DPCS Implementation (Software)

• Coded in IDL (Research Systems Inc.)

• High level language with efficient Vector/Matrix operations

• MDSplus interface

• Emulation of Hybrid operation in ∼ 50 lines of code

• Real-time single synchronous loop driven by digitizer clocking

• Deterministic performance (with interrupts disabled)

.

C ModAlcator

− Software Architecture (1)

• Server process initialization� One-time initialization at boot (or restart)

� Initialize cPCI interface

� Initialize IDL process, communication via FIFO buffer (named

pipe)

• Per-shot Initialization (INIT phase)� Read in data structures from MDSplus tree

� Setup parameters for real-time loop

(Real-time parameters passed using IDL Pointer/Heap Variable

mechanism)

� Initialize diagnostic buffers

.

C ModAlcator

− Software Architecture (2):Real-time procedure

• Run one dummy cycle to pagefault all code and data into memory

• Lock down memory

• Disable interrupts

• Loop waiting for trigger (timeout on MAX LOOPS)

• Loop until MAX STEPS� Read inputs and set outputs from previous calculation

� Do control calculations

� Save inputs, outputs, and intermediate results in memory

• Enable Interrupts

• Reset system time

.

C ModAlcator

− Software Architecture (3): Post-pulse

• Store buffers saved in real-time into MDSplus

• Clean up Heap Memory

• Check and report errors from real-time step

.

C ModAlcator

− Additional computations incorporated inreal-time loop

• Callable user-supplied modulesspecified by name in DPCS tree

• Init, Real-time, Cleanup, Storemethods invoked via

IDL call proceduremechanism

• Parameters passed using IDL pointers,allowing standardized calling syntax

• Flexible calling mechanism costs ∼ 2µsec per invocation in real-time routine• Presently implemented examples include Baseline Subtraction, TAE frequency

.

C ModAlcator

− Plasma Control User Interface (PCS)

• GUI designed for original Hybrid Control System• New DPCS System operates on same data structures• Maintains access to 10+ years

of legacy shot programming,operator experience

(MS WORD TEMPLATE for Submission in Fusion Engineering and Design)

to be a dead-end approach. Instead we decided to build an all-digital replacement for the control system.

2. PCS User Interface Alcator C-Mod is a pulsed experiment, operating for about 10 seconds on a 10 to 20 minute cycle. Pulses are designed either offline or between shots using a program called PCS. The discharge is broken into a set of separately programmable sequential segments. Within each segment the user defines the quantities of interest (observers) and their target values, the gains of the controllers, the feed-forward waveforms and other relevant parameters. PCS provides user-friendly graphical interfaces; many of the quantities are directly drawn on screen. All of the control quantities are stored in MDSplus[4], and loaded into the hardware during the initialization phase of the experiment pulse. Entire pulses, segment definitions or individual observers can be copied from one pulse to another. Figure 2 shows some of the PCS user interfaces.

Fig.2. PCS – User interface for Plasma Control System.

3. DPCS implementation

The DPCS system replaces the hybrid computer described above with CompactPCI digitizer cards and a general-purpose computer. (See Fig. 1 – bottom) The control algorithm is written in IDL[5] and the computer

runs an off the shelf version of Linux. All calculations in the system are done in floating point arithmetic, eliminating signal saturation problems that were present in the hybrid. More fundamentally, the computations performed by the hybrid were fixed, linear observers, PID feedback, etc, whereas the digital system can do arbitrary calculations between the input and output stages. 3.1 Hardware

The system is comprised of two CPCI data acquisition cards, a CPCI to PCI bus extender and a XEON Server. Each I/O card has 64 16-bit inputs and 16 analog outputs, totalling 128 inputs and 32 outputs. Cards with more channels or additional cards can be added should the need arise. The original hybrid control computer had a total of 96 inputs and 16 outputs. The server is a standard Intel® Xeon™ 3.20GHz with 2 Gigabytes of memory. It can easily be replaced or upgraded since it is completely standard. The computer is running the RedHat[6] Linux distribution, booted diskless over NFS. The system boots using dhcp and PXE. A local disk is present for paging and swapping when not in real-time mode. The PCI/CPCI extender card transparently makes the CPCI cards appear as PCI peripherals on the host computer. The one in use is a 32 bit 33 MHz card but 64 bit 66 MHz cards are available should this prove to be a bottleneck in the future. 3.2 Software The DPCS system is written in IDL, a commercially produced interpreted language from Research Systems Inc. This provides a simple interface mechanism to load the inputs of the system from MDSplus, and to store the results into MDSplus. The code that performs the computation of the control loop is less than 50 lines of IDL. This is accomplished by using the language’s built in vector and matrix operations. Input and output is done by calling a small set of glue routines, written in C, which in turn call into the peripheral card’s API. After some one-time initialization, the software reads commands from a named pipe, telling it to setup for the next pulse, operate the real-time loop, or store the results from the last real-time operation. The pulsed nature of the experiment means that for

.

C ModAlcator

− Physics Operator Specifies PlasmaParameters, Control Laws

• System is data-driven, all parameters reside in MDSplus trees(HYBRID, PCS)

• Static Configuration data in HYBRID tree

• PCS GUI stores volatile setup values to PCS Sub-tree

• Shot specification� Draw target waveforms

� Cut and paste

� Numeric Entry

� Specify Observer algorithms

� Specify PID gains (time-dependent)

� Specify Controller algorithms (actuator response)

• Recall old shots or portions thereof (segments)

.

C ModAlcator

− User Transparency provided by parallelMDSplus tree structure

• Initial DPCS Implementation� Physics Operator Interface (PCS

GUI) modifies PCS Sub-tree

� DPCS Sub-tree structures referencecorresponding PCS Sub-tree nodesusing MDSplus Expressions

� DPCS software (DPCS INIT)references nodes in DPCS Sub-tree

• New Capabilities appear only in\DPCS CONFIG and DPCS Sub-tree

Future GUI will modify DPCS tree directly,

provide for importation of old PCS data

.

C ModAlcator

− Real-time Performance

• Nominal 100µsec cycle time(10 kHz)

• Basic Hybrid Control emulationtakes

.

C ModAlcator

− Vertical Control maintained for ∆t ≤ 200µsec

-10000

-50000

500010000 Zcur Feedback Error (A-m)

0.00.20.40.60.81.0 Ip (MA)

0.00.51.01.52.0 Elongation

0.0 0.5 1.0 1.5Time (sec)

0.00.5

1.0

1.5 Decay Index N/Ncrit

tcyc=100 µsectcyc=200 µsectcyc=400 µsec

Increasing cycle to 400µsec results in disruption.

.

C ModAlcator

− Turning Interrupts Off providesdeterminant real-time behavior

With interrupts enabled, latency can exceed 10 msec

(nominal 100 µsec cycle)

0.0001 0.0010 0.0100Actual cycle time (sec)

10-1

100

101

102

103

104

105

106 Interrupts EnabledInterrupts Disabled

.

C ModAlcator

− In-shot baseline subtraction implemented

Small (

.

C ModAlcator

− DPCS provided real-time frequency forActive MHD Spectroscopy experiments

Requirements:

• Calculate and output demand signal (VCO input) for TAE dampingmeasurements (see Posters by Snipes & Sears, this session)

• Waveform superposes a sweep over calculated fTAE ∝ BT/√

n

Implementation:

• Output assigned (\\HYBRID::TOP.DPCS CONFIG to new OUTPUT 17 channel(analog fiber optic link to AMHD amplifier)

• Setup parameters in MAGNETICS tree, filled in by userDPCS calculated values stored to same Tree Branch during STORE cycle

• Custom IDL modules (DPCS fTAE) for INIT, REAL-TIME, STORE, CLEANUPoperations invoked using call procedure mechanism (based on entry inDPCS::\TOP.LOADABLES.PROCEDURES:TEST PROS)

.

C ModAlcator

− Active MHD Spectroscopy frequencyfollows calculated fTAE(t)

.

C ModAlcator

− DPCS has operated routinely over the2005 Experimental Campaign

• Over 2000 plasmas successfully produced

• One shot abandoned because DPCS INIT hung during INIT phase

• No real-time control failures� Zero missed cycles at standard 100µsec sample rate� Essentially all plasma conditions accessed under old Hybrid

control have been reproduced (or exceeded) under DPCS control

• New features implemented

Initial implementation of the C-Mod DigitalPlasma Control System has been successful!

.

C ModAlcator

− Future Developments - Nonlinear control

• Feedback control of plasma parameters which do not have linearestimators,e.g. qψ

• Replace linear observers with improved non-linear algorithms

• Extend range of controlled parameters, incorporate new sensors,actuators

e.g. Control ICRF power to feedback on neutron rate, for burncontrol simulation experiments

• Move toward real-time MHD equilibrium modeling/control (suchas rtEFIT)

.

C ModAlcator

− Future Developments: Adaptive ControlSchemes

• Mitigation of power supply voltage and current limits, non-linearities

• Adaptive response to off-normal or unplanned behavior� Digital “Fizzle Detector” – Benign power system rampdown on

loss of plasma event

� Alternate (graceful exit) scenarios in response to

? Plasma parameters out of desired range

? Faulting of auxiliary systems (heating, fueling, . . . )

� Sense and adapt to non-responsive or out-of-range sensor signal

� Real-time coil stress calculator/limiter

.

C ModAlcator

− Future Development – Tools

• Develop New Physics Operator Interface GUI� Take advantage of increased number of inputs and outputs

� Eliminate constraints associated with Hybrid hardware

� Provide operator interface to new functionality, including ability

to specify and set parameters for user-provided callable routines

� Provide importation feature to incorporate old PCS shot setupsinto DPCS tree framework

• Continue development of Alcasim Simulation Code (M. Ferrara)� MatLab/Simulink Model of tokamak plasma, power systems,

diagnostic response

� Coupled to DPCS software through FIFO buffer

� Facilitates off-line testing for scenario, algorithm development

.

C ModAlcator

− Future Development – Architecture

• Enhance DPCS beyond single-cycle, single-processor implementation� Allow slower control loops where appropriate and necessary while

maintaining fast control loop for vertical stability, etc.� Run additional calculations on multiple processors

? SMP configuration with shared memory

? Networked CPUs

� Allow for external processing of diagnostic data to provide new

control system inputs (analog or digital)