C ModAlcator

−

Alcator C-Mod Digital Plasma ControlSystemPresented by:

S. Wolfe, J. Stillerman, M. Ferrara, T. Fredian, I. Hutchinson

APS DPP05

Denver, CO

Oct. 26, 2005

Abstract

A new digital plasma control system (DPCS) has been implemented for AlcatorC-Mod. The new system was put into service at the start of the 2005 run campaignand has been in routine operation since. The system consists of two 64-input,16-output cPCI digitizers attached to a rack-mounted single-CPU Linux server,which performs both the I/O and the computation. During initial operation, thesystem was set up to directly emulate the original C-Mod ”Hybrid” MIMO linearcontrol system. Compatibility with the previous control system allows the existinguser interface software and data structures to be used with the new hardware. Thecontrol program is written in IDL and runs under standard Linux. Interrupts aredisabled during the plasma pulses to achieve real-time operation. A synchronousloop is executed with a nominal cycle rate of 10 kHz. Emulation of the originallinear control algorithms requires 50 µsec per iteration, with the time evenly splitbetween I/O and computation, so rates of about 20 KHz are achievable. Reliablevertical position control has been demonstrated with cycle rates as low as 5 KHz.Additional computations, including non-linear algorithms and adaptive response,are implemented as optional procedure calls within the main real-time loop.

(MS WORD TEMPLATE for Submission in Fusion Engineering and Design)

• The diagnostics provide information on the magnetic field inside the machine and on the plasma.

• The multiplication by the time dependent ‘A’ matrix (96 inputs x 16 outputs) computes the ‘observers’, that is the 16 physical quantities of interest.

• The pre-programmed target waveforms are subtracted from the observers.

• The PID controllers act on the error signals with time dependent gains.

• The controller matrix (‘M’) is multiplied by the output of the PID to produce the correct action on the relevant actuators, and 16 pre-programmed feed-forward waveforms are added. This allows for both open loop control signals, and reduces the needed gains in the PID feedback stage.

The hybrid’s analog outputs as well as the values of the intermediate steps in the computation are digitized using CAMAC for use in debugging and

diagnosis of the control system operation. Care must be taken that hybrid computed values stay within the hardware voltage ranges throughout all of the steps of the computation. The real-time node (INTEL 8044) plays out pre-programmed scenarios, stepping the waveforms and matrices through their programmed values. In the hybrid, digitally programmed analog circuits to perform the matrix operations and produce the feedback signals. There are also provisions in the hardware for adaptive control, but the difficulties and risks of programming and debugging the real-time computer in 8044 assembler prevented us from pursuing it.

Communication between the data system computers and the hybrid control computer was done over a bitbus interface connected to a micro-vax computer. There are significant maintenance and performance issues with this interface and these obsolete computers. Bitbus is no longer a supported standard and micro-vaxes have not been manufactured for many years. We could have pursued bitbus interfaces for more modern computers, but this seemed

Fig. 1. Alcator C-Mod control system. Hybrid Digital/Analog computer (top). All Digital system (bottom)

.

C ModAlcator

− C-Mod Linear Multi-Input-Multi-OutputControl Loop

• Diagnostic sensors measure fields, density, currents, . . .

• Estimators of parameters of interest obtained by multiplication of

sensor signals by “A-matrix” of linear observers

• Pre-programmed Target Waveforms (optionally scaled by plasma

current) subtracted from estimated values

• Error signals are multiplied by (time-dependent) pre-programmed

PID Gains

• Output of PID stage is multiplied by “M-matrix” of Controllers to

produce demand signal for actuators (power supplies, gas valves,

. . . )

• Pre-programmed Feed-forward “V-waveforms” added to actuator

demands

.

C ModAlcator

− “Hybrid” analog-digital control system inuse from 1991-2004

• Collaboration between MIT and EPFL (Lausanne)

• Implements strictly (piecewise) linear MIMO control system

• Analog computer with digital programming based on DAC cards

• Bitbus interface

� No longer a supported standard

� Obsolete Micro-Vax interface card (Q-bus)

• Intel 8044 real-time node (programmed in assembler) updates

matrices and target waveforms

.

C ModAlcator

− DPCS all-digital real-time control systemreplaced the old Hybrid system at thebeginning of the 2005 Campaign

Design Requirements:

• Drop-in replacement for Hybrid

� Run old shots

� Operate transparently on same data structures as Hybrid

� Programmed using existing GUI (PCS)

• Remove constraints imposed by analog hardware

• Allow implementation of non-linear control schemes

• Provide for increased number of Inputs (96 → 128)

and Outputs (16 → 32)

.

C ModAlcator

− Control System Configuration Description,Programming, Results Stored in MDSplus

New Data Structures added to support DPCS

• Data-Driven System

• Control and Data

Acquisition Actions

controlled by

MDSplus

DISPATCHER

• Communication by

MdsIp servers over

Ethernet link

.

C ModAlcator

− DPCS Testing and Migration

• DPCS Run in parallel with Hybrid during 2003-2004 campaign

� Outputs not connected to actuators

� Intermediate (computed) signals, outputs compared to

corresponding Hybrid signals

� Observed discrepancies associated with offsets, noise, and voltage

limits of Hybrid analog hardware

• DPCS Control implemented at start of 2005 campaign

� Initial power system testing carried out under DPCS control

� First plasma operation 4-Feb-2005

? Successful startup established after 8 attempts

(compare to 6 tries on first day of 2003-2004 Campaign)

? Re-runs of old discharges indicate good reproducibility

� Hybrid run in parallel (disconnected from actuators) for first

months, eventually disabled

.

C ModAlcator

− DPCS Implementation (Platform)

• Xeon single processor server (3.2GHz, 2GB)

• PCI/cPCI Bus extender (SBS)

• Two 64-input, 16-output cPCI digitizer cards (D-Tacq)

• Trigger and 1 MHz clock signals from CAMAC (MPB, Jorway)

• Standard Linux Operating System (RedHat 9)

� Diskless boot (PXE, Root over NFS)

� Interrupts disabled during real-time operation

? Kernel call from low-latency digitizer driver library

? No disk i/o, page and swap, network access

? Memory locked down after one dummy pass

• Standard Ethernet Network Interface

.

C ModAlcator

− DPCS Implementation (Software)

• Coded in IDL (Research Systems Inc.)

• High level language with efficient Vector/Matrix operations

• MDSplus interface

• Emulation of Hybrid operation in ∼ 50 lines of code

• Real-time single synchronous loop driven by digitizer clocking

• Deterministic performance (with interrupts disabled)

.

C ModAlcator

− Software Architecture (1)

• Server process initialization

� One-time initialization at boot (or restart)

� Initialize cPCI interface

� Initialize IDL process, communication via FIFO buffer (named

pipe)

• Per-shot Initialization (INIT phase)

� Read in data structures from MDSplus tree

� Setup parameters for real-time loop

(Real-time parameters passed using IDL Pointer/Heap Variable

mechanism)

� Initialize diagnostic buffers

.

C ModAlcator

− Software Architecture (2):Real-time procedure

• Run one dummy cycle to pagefault all code and data into memory

• Lock down memory

• Disable interrupts

• Loop waiting for trigger (timeout on MAX LOOPS)

• Loop until MAX STEPS� Read inputs and set outputs from previous calculation

� Do control calculations

� Save inputs, outputs, and intermediate results in memory

• Enable Interrupts

• Reset system time

.

C ModAlcator

− Software Architecture (3): Post-pulse

• Store buffers saved in real-time into MDSplus

• Clean up Heap Memory

• Check and report errors from real-time step

.

C ModAlcator

− Additional computations incorporated inreal-time loop

• Callable user-supplied modules

specified by name in DPCS tree

• Init, Real-time, Cleanup, Store

methods invoked via

IDL call proceduremechanism

• Parameters passed using IDL pointers,allowing standardized calling syntax

• Flexible calling mechanism costs ∼ 2µsec per invocation in real-time routine

• Presently implemented examples include Baseline Subtraction, TAE frequency

.

C ModAlcator

− Plasma Control User Interface (PCS)

• GUI designed for original Hybrid Control System

• New DPCS System operates on same data structures

• Maintains access to 10+ yearsof legacy shot programming,operator experience

(MS WORD TEMPLATE for Submission in Fusion Engineering and Design)

to be a dead-end approach. Instead we decided to build an all-digital replacement for the control system.

2. PCS User Interface Alcator C-Mod is a pulsed experiment, operating for about 10 seconds on a 10 to 20 minute cycle. Pulses are designed either offline or between shots using a program called PCS. The discharge is broken into a set of separately programmable sequential segments. Within each segment the user defines the quantities of interest (observers) and their target values, the gains of the controllers, the feed-forward waveforms and other relevant parameters. PCS provides user-friendly graphical interfaces; many of the quantities are directly drawn on screen. All of the control quantities are stored in MDSplus[4], and loaded into the hardware during the initialization phase of the experiment pulse. Entire pulses, segment definitions or individual observers can be copied from one pulse to another. Figure 2 shows some of the PCS user interfaces.

Fig.2. PCS – User interface for Plasma Control System.

3. DPCS implementation

The DPCS system replaces the hybrid computer described above with CompactPCI digitizer cards and a general-purpose computer. (See Fig. 1 – bottom) The control algorithm is written in IDL[5] and the computer

runs an off the shelf version of Linux. All calculations in the system are done in floating point arithmetic, eliminating signal saturation problems that were present in the hybrid. More fundamentally, the computations performed by the hybrid were fixed, linear observers, PID feedback, etc, whereas the digital system can do arbitrary calculations between the input and output stages. 3.1 Hardware

The system is comprised of two CPCI data acquisition cards, a CPCI to PCI bus extender and a XEON Server. Each I/O card has 64 16-bit inputs and 16 analog outputs, totalling 128 inputs and 32 outputs. Cards with more channels or additional cards can be added should the need arise. The original hybrid control computer had a total of 96 inputs and 16 outputs. The server is a standard Intel® Xeon™ 3.20GHz with 2 Gigabytes of memory. It can easily be replaced or upgraded since it is completely standard. The computer is running the RedHat[6] Linux distribution, booted diskless over NFS. The system boots using dhcp and PXE. A local disk is present for paging and swapping when not in real-time mode. The PCI/CPCI extender card transparently makes the CPCI cards appear as PCI peripherals on the host computer. The one in use is a 32 bit 33 MHz card but 64 bit 66 MHz cards are available should this prove to be a bottleneck in the future. 3.2 Software The DPCS system is written in IDL, a commercially produced interpreted language from Research Systems Inc. This provides a simple interface mechanism to load the inputs of the system from MDSplus, and to store the results into MDSplus. The code that performs the computation of the control loop is less than 50 lines of IDL. This is accomplished by using the language’s built in vector and matrix operations. Input and output is done by calling a small set of glue routines, written in C, which in turn call into the peripheral card’s API. After some one-time initialization, the software reads commands from a named pipe, telling it to setup for the next pulse, operate the real-time loop, or store the results from the last real-time operation. The pulsed nature of the experiment means that for

.

C ModAlcator

− Physics Operator Specifies PlasmaParameters, Control Laws

• System is data-driven, all parameters reside in MDSplus trees

(HYBRID, PCS)

• Static Configuration data in HYBRID tree

• PCS GUI stores volatile setup values to PCS Sub-tree

• Shot specification

� Draw target waveforms

� Cut and paste

� Numeric Entry

� Specify Observer algorithms

� Specify PID gains (time-dependent)

� Specify Controller algorithms (actuator response)

• Recall old shots or portions thereof (segments)

.

C ModAlcator

− User Transparency provided by parallelMDSplus tree structure

• Initial DPCS Implementation

� Physics Operator Interface (PCS

GUI) modifies PCS Sub-tree

� DPCS Sub-tree structures reference

corresponding PCS Sub-tree nodes

using MDSplus Expressions

� DPCS software (DPCS INIT)references nodes in DPCS Sub-tree

• New Capabilities appear only in

\DPCS CONFIG and DPCS Sub-tree

Future GUI will modify DPCS tree directly,

provide for importation of old PCS data

.

C ModAlcator

− Real-time Performance

• Nominal 100µsec cycle time

(10 kHz)

• Basic Hybrid Control emulation

takes <∼ 50µsec

� ∼ 25µsec I/O

� ∼ 25µsec Computation

� About 1/2 cycle time available

for additional computations

• Maximum feasible cycle time

set by vertical stability control

requirement

• Control loop robust to up to 10%

lost samples

(MS WORD TEMPLATE for Submission in Fusion Engineering and Design)

most of the time, the plasma control system does not need to operate in real-time mode. We only need real-time response during the relatively infrequent plasma pulses. The solution chosen is to run standard linux on the control computer, and to disable interrupts when real-time behaviour is needed[7]. With interrupts disabled the behaviour of the system is deterministic, with timing variations between iterations of 1 to 2 µs.

With interrupts disabled extra care must be taken with respect to certain aspects of the code. The code and data must be loaded into memory and locked down to prevent any access to the hard drive or network. The system can not be externally told to abort the real-time loop. Instead it operates for a fixed number of iterations (fixed amount of time) and then exits. Diagnostic print statements and symbolic debuggers will not operate. Structural debugging is done with interrupts enabled. When operating in real-time mode, diagnostic information is recorded in memory to be stored after interrupts are re-enabled. Weighed against these difficulties are the advantages of being able to run a standard operating system. It is easily upgradeable, configurable, and even replaceable. Standard copies of the data system tools can be used to both load the inputs to the system, and write the results of the operation. We are not locked into a proprietary hard to change configuration. Many real-time operating systems guarantee millisecond or even tens of millisecond response times. Our system operates at 10 KHz, well beyond these specifications. 3.3 Performance

The performance of the digital system was extensively tested during the 2003-2004 experimental campaign. DPCS was operated in parallel with the hybrid control system but was disconnected from any actuators. The outputs of the two systems were compared, and the discrepancies between the systems accounted for. Most of these were related to either offsets or noise in the hybrid system or saturation in the analog signal paths.

An external clock attached to the data acquisition cards controls the synchronous real-time loop. At the start of each iteration, the computer waits for the next available set of input values. When the inputs are read, the outputs from the previous computation are written to the output cards. The intermediate and output control signals are then computed, in a way that accounts for

the actual time at which the input values were sampled. With the final hardware and software implementation, the time needed for the input of 128 channels, the computations and the output of 32 channels, is 48 µs. At the nominal 10 kHz cycle rate, over 50 µs is available for additional computation or increased update rate. (See Figure 3)

Fig. 3. DPCS 100 µs cycle time. Note that more than ½

the time (bottom) is available.

The new control system was put on line at the start of the 2005 campaign, with the hybrid operating in parallel and offline for comparison purposes. DPCS has fully satisfied the expectations on its operation. Dedicated run time was spent characterizing the minimum update rates required to maintain control of the vertical position of the plasma. At the nominal speed of 10kHz, DPCS has an equivalent bandwidth below 5kHz, while the EFC coils, which compensate for the vertical instability of our plasmas, have an analog bandwidth of about 1.5kHz. We tried reducing the DPCS cycle time for a moderately unstable plasma, with elongation 1.65 and current 0.8MA. The plasma was successfully controlled at 10 and 5kHz, but disrupted at 2.5kHz. Figure 4 shows the vertical position error signal for three pulses. In the top two traces the system was operated at 10kHz and 5kHz and control is maintained. At 2.5kHz, the control seems to compensate for the error at .1 seconds and .25 seconds, but finally loses control at .6 seconds, as shown in the bottom trace. At the 5kHz cycle rate the headroom for extra computation is about 150µs.

.

C ModAlcator

− Vertical Control maintained for ∆t ≤ 200µsec

-10000

-50000

500010000 Zcur Feedback Error (A-m)

0.00.20.40.60.81.0 Ip (MA)

0.00.51.01.52.0 Elongation

0.0 0.5 1.0 1.5Time (sec)

0.00.5

1.0

1.5 Decay Index N/Ncrit

tcyc=100 µsectcyc=200 µsectcyc=400 µsec

Increasing cycle to 400µsec results in disruption.

.

C ModAlcator

− Turning Interrupts Off providesdeterminant real-time behavior

With interrupts enabled, latency can exceed 10 msec

(nominal 100 µsec cycle)

0.0001 0.0010 0.0100Actual cycle time (sec)

10-1

100

101

102

103

104

105

106

Interrupts EnabledInterrupts Disabled

.

C ModAlcator

− In-shot baseline subtraction implemented

Small (<∼ 1% full scale) variationsin diagnostic baseline offsetsobserved (shot-to-shot, day-to-day)• Possible contributor to startup

variability• Uncompensated baselines

required introduction of ad hocoffsets and “tweaks” in PCSshot programming

• Long term drifts could impactcontrol reproducibility

BP20_BC Baseline Variation

0 1000 2000 3000Shots (Feb-Sep 2005)

-0.15

-0.10

-0.05

0.00

0.05

Bas

elin

e O

ffset

(V)

In-shot baseline compensation incorporated into DPCS

• Input values averaged over pre-set (per channel) interval early in each shot

• Average baseline values subtracted from inputs at subsequent times

• Detected baseline values output to MDSplus tree during STORE cycle

• About 10µsec impact on minimum control cycle time (principally duringaveraging interval)

• PCS Observer “offset” terms eliminated

.

C ModAlcator

− DPCS provided real-time frequency forActive MHD Spectroscopy experiments

Requirements:

• Calculate and output demand signal (VCO input) for TAE dampingmeasurements (see Posters by Snipes & Sears, this session)

• Waveform superposes a sweep over calculated fTAE ∝ BT/√

n

Implementation:

• Output assigned (\\HYBRID::TOP.DPCS CONFIG to new OUTPUT 17 channel(analog fiber optic link to AMHD amplifier)

• Setup parameters in MAGNETICS tree, filled in by userDPCS calculated values stored to same Tree Branch during STORE cycle

• Custom IDL modules (DPCS fTAE) for INIT, REAL-TIME, STORE, CLEANUPoperations invoked using call procedure mechanism (based on entry inDPCS::\TOP.LOADABLES.PROCEDURES:TEST PROS)

.

C ModAlcator

− Active MHD Spectroscopy frequencyfollows calculated fTAE(t)

.

C ModAlcator

− DPCS has operated routinely over the2005 Experimental Campaign

• Over 2000 plasmas successfully produced

• One shot abandoned because DPCS INIT hung during INIT phase

• No real-time control failures

� Zero missed cycles at standard 100µsec sample rate

� Essentially all plasma conditions accessed under old Hybrid

control have been reproduced (or exceeded) under DPCS control

• New features implemented

Initial implementation of the C-Mod DigitalPlasma Control System has been successful!

.

C ModAlcator

− Future Developments - Nonlinear control

• Feedback control of plasma parameters which do not have linear

estimators,e.g. qψ

• Replace linear observers with improved non-linear algorithms

• Extend range of controlled parameters, incorporate new sensors,

actuators

e.g. Control ICRF power to feedback on neutron rate, for burn

control simulation experiments

• Move toward real-time MHD equilibrium modeling/control (such

as rtEFIT)

.

C ModAlcator

− Future Developments: Adaptive ControlSchemes

• Mitigation of power supply voltage and current limits, non-

linearities

• Adaptive response to off-normal or unplanned behavior

� Digital “Fizzle Detector” – Benign power system rampdown on

loss of plasma event

� Alternate (graceful exit) scenarios in response to

? Plasma parameters out of desired range

? Faulting of auxiliary systems (heating, fueling, . . . )

� Sense and adapt to non-responsive or out-of-range sensor signal

� Real-time coil stress calculator/limiter

.

C ModAlcator

− Future Development – Tools

• Develop New Physics Operator Interface GUI

� Take advantage of increased number of inputs and outputs

� Eliminate constraints associated with Hybrid hardware

� Provide operator interface to new functionality, including ability

to specify and set parameters for user-provided callable routines

� Provide importation feature to incorporate old PCS shot setups

into DPCS tree framework

• Continue development of Alcasim Simulation Code (M. Ferrara)

� MatLab/Simulink Model of tokamak plasma, power systems,

diagnostic response

� Coupled to DPCS software through FIFO buffer

� Facilitates off-line testing for scenario, algorithm development

.

C ModAlcator

− Future Development – Architecture

• Enhance DPCS beyond single-cycle, single-processor implementation

� Allow slower control loops where appropriate and necessary while

maintaining fast control loop for vertical stability, etc.� Run additional calculations on multiple processors

? SMP configuration with shared memory

? Networked CPUs

� Allow for external processing of diagnostic data to provide new

control system inputs (analog or digital)