U.S. Govt

_f _'_

A

303

PRECEDING PAGE BLANK NOT FILMED

https://ntrs.nasa.gov/search.jsp?R=19920002801 2020-07-23T05:39:21+00:00Z

U.5. Gov I

U5

!-

ILlI--U5>-U5

<I--.<r_LLI

i

<U.I

ELII--v1>-

F--¢Z:ELI

XuJIJJ

m

F-!

-.I<ELICZ:,--I<Z0F-,<ok:EL!n_0

C)LJL.

ILlZ¢Z:,<ELI..J

UlZOU5

LLI.J

'J1r_F--.

V1ilili

F-<-1-

t/1LLIOk:::)...I

LL

Z,<I/1I.L/L/1U'JLL!

uJ LJre LJ

I-- _n

ELIj.. 0•.- r,,,"1" u.

rv, LU<I: z

I.-- u.iu'l ..I

w Zu 0inn

tJ5 t/IU5<: .j

ca _J

I l I

(

304

U.S. Gov t

,.<,n,,

ELIu.IZ z

305

U.S. Gov f

z

_ ___=,='__=_ _ _=_

_=_ .,=, eg__. _=, .. =,

l I I I

3O6

U.S. Gov t

°,A

E_I.-

E

4,J

E_

EI.-

0

tL

t_

307

U-5. Gov i

0 m m_

SI I I l I i

308

Fr

i

_L

1

I

i!

U.S. Gov "t

t/l

l-¢x:

UJI--

>-

<¢I--<¢

IJJ

m

<¢

I./1Um

h-,

<_ZO

LLI,¢OI--I,/1I,M:3Zm

I--ZOUZOm

t41Z

Xi11

o'1QOo'1

o'1P4

I

X¢ZOI1_t/Im

t41,<¢¢1141<CI,/1

I--ee'¢/1i,M141:3

IJ,I IJJI-- I--Z u_uJ >-U u_

U Z

<{0LU I"141 --uJ ;[

re 0

-r-

" IJ,i_Ju. I--

z'<uJ__0>- I--

I

E

309

U.S. Govl

AU'l

I--me,

m

I-_J,<LLIm,"

L,/,IV'_U::--0"I"1_

,<z,'_0

Z0

×_

,,A/

2_-C

LuJ;

_CI.-C

m

_J

\

U

310

U.S.Govt

>-

F-

311

U_ Gov'!

AU1

I--rr'

I--

_U

1

I--_J<CILl

_U

u_

_o_._I-- _.n U

u'l

u_

l, °ia._

b _1..."_ _I

Z0I--

×<_

_v

0w.,1#"_l%.._I

/

LLi

<_

c_v

1

IOI

>-

t_

o_ _

312

I--I.UZ

ILl"rI--LII

z0F'-,e'f" _/'1

__v

_Z

Z

0UZ>-

l.U

I1

0

0

i11U0

o.

!--0U

z0I--

!--

i i i %,,_

o

Z

wZ0

Z0I--

I--

0Uwo

ilW

I--

!

z

U.S. Gov't

gJl,-U'I>.

: <I:I--<I:t'1IJJ

,..I<I:

..!Z Z

,,=,=

Z0 z

LM M,.IU

UJ _',,,' 0 0

UIlli---1

U

___!

U1

I--1

U

I11

0

I--1

> .jul ul

_ !- a.i.- 0 Z

1

313

U.S. Gov !

r-,__ :) _jI--

,<::)_o

zl

__ .., •Z _____ o_. _ _

LLJ

L_ n,

.--I

Z0I-raa

I-

I--

>.

0

"i"u

0

I-n_

I--

>-

C

314

_u4

i

-ii

ZU_0 z

WLIJ.I..4

U.S.Gov t

: ,=., W __

o = ,. _= =,,=, ="-_U ''='# == N _. U_G

,.., _._ z_ ,.>,:_ _=,:. =,=, ,-...o ._= . ,I I | I II

_-- tN

U.S. Gov!

i I

m

::_ 316

U.S.Govt

317

U.S. Govf

e,,.IJJl.-l,,J.

LU

l I l l

I.¢I I,o p-,

m

r

U.S. Govt

UJI-,

>-

<I--<I:£::)UJ

<C

;

U ,"

Z U I-- uu )._. L/1_- <

-- <I: I-w i- Z(/1 V'l ..- ,--

C_ uuuu •Z OZ

m

Z Z _0 <w O

_ 0

_Z _

_Z _ w

_w < 0 =u O =O _ Z _ _

1z_ _ _ __< _ 0 _m

_ 0 _ _< -

_ _ _O =

UU =OZ _ ,3<

CO

319

U.S.Gov!

Z0 I- >"

Z ,_, a.. Z .w_ Z Z --

:I:0 u_ :r: _ -- _- "I"--' U I-- _" Z I--

I-- Z >" '_'Z_ ::D _ _0 a. uu

z_ o

DO _uuu=: X_ x z"I""I" ' ' ' '_- _- , ,

O_

:_ 320

U.S. Gov't

::1¢:321

,°

r_I-

E

EI--

O

M.

E

O'_

U.S.Govf

O-l-e,.OZ

!-

Zr,.

uJ ::3-r- I--_J< O

< zuu <_2:

o

ego

,,=, =_ _. ..>z , , <O <

| I |

O_

E3i--e,,-,,.,..

Eq_

>,

_3q_E

q_

k..-O4._

t,.

r_L

,r'--_

(1.

O

322

U.S.Gov t

F-

E

¢11

E

I I I I

323

21st ANNUALSYMPOSIUM PROCEEDINGS

z

"Flight Test XXl. . .The Sky's No Longer the Limit"

6-10 August 1990Garden Grove, California

r

Sponsored by the Los Angeles Chapter

324

Perhaps one of the most powerfulsymbols of the United States'technological prowess is the MissionControl Center (MCC) at the Lyndon B.Johnson Space Center in Houston, Texas.The rooms at Mission Control have beenwitness to major milestones in thehistory of American technology such asthe first lunar landing, the rescue ofSkylab, and the first launch of the SpaceShuttle. When Mission Control was firstactivated in the early 1960's, it was trulya technological marvel. This facilityhowever has received only modestupgrades since the Apollo program.Until recently it maintained a singlemainframe based architecture that

displayed data and left the job of dataanalysis to human beings. The displaytechnology utilized in this system wasmonochrome and primarily displayedtext information only with limitedgraphics (picture 1). An example displayof 250 communication parameters isshown in picture 2.

The system processed incoming dataand displayed it to the flight controllers,however it performed few functions toturn raw data into information. The jobof turning data into information uponwhich flight decisions could be madewas performed by the flight controllers.In some cases, where additionalcomputational support was required,small offline personal computers wereadded to the complex. Flight controllersvisually copied data off th_ consoledisplay screens, and manually enteredthe data nto the small personalcomputers where off line analysis couldbe performed.

Although, this system wastechnorogically outdated, it containedyears of customizing efforts and servedNASA well through the early SpaceShuttle Program. Several factors arenow driving NASA to change thearchitecture of Mission Control toaccommodate advanced automation.First is the requirement to support anincreased flight rate without majorgrowth in the number of personnelassigned to flight control duties. We areattempting to fly more missions with thesame staff to control operational costs:

Real Time Data Acquisition For Expert Systemsin Unix Workstations at Space Shuttle Mission Control

John F. Muratore, Troy A. Heindel and Terri B. MurphyNational Aeronautics and Space Administration

Arthur N. Rasmussen and MArk GnabasikMITRE Corporation

Robert Z. McFarlandUnisys Corporation

Samuel A. BaileyDual and Associates

NASA is using automation to expand thecapabilities of individuals so they canaccomplish more work. This concept of"more work for the same dollar is verydifferent from trying to automate a

factory where the desire is to replacehumans with robotics and do the samework for less dollars." In MissionControl, the goal is to support thehuman operator, and not to eliminatethe human.

r.

ORIGINAL PAGE ISOF POOR QUALrrY

Picture I - Space Shuttle Mission Control

0_t tl:lS_IZ:li Of,DieT O=llizzt=li ll"r[ IDA Ol ill _ liIGlit _l:lSiil:ll ul IT t I I leTS $Pl Sl I I II

UL _=J M UL. _ SG.[DT Z C[]_7_i C_D

• _ s5 -Zo_i TORS _Zi. T SVNC BiT KU 11 I,*Bt.

x[St • _Fr / w=_t 5v_c r_ I C3_ T_ m_E_St /-113t EST / _ 5V_C T_ a C_ _ C_nc-_=, LC_ CX mr pot,_R D 'sc_mc[ S _ ¢xo

pM [_ _S *NT MODE M!U,_ mA_g _0 .-D_¢_,t_r tO- S[AnC_ o , cool: gN _ _1_= PNm

_N_ _-[L LL_ D_TEtT 0 'O/L _E HI ¢)¢ liSi Oi._ G_O r_c_ D ¢00[ ON _ IO! O_

8-_O1D[

_ltC--TEC-- - - Dm,lLil,l¢ [l_ _ _tm

......... ; ........... " I:! ;, i o=. ',:

• ICOI I_l NUN ill _.It lS_ I 131i llt_Ji mC_A IL_ IMU t.II DD*4 I l_Itill:gt_l:q_ iMI/_l

_/i. mcnl _l P'l I lull lit Di_ I tlil_/lll I1.1

Picture 2 - Typical Mainframe Display

5.1-1

325

A second major concern is loss ofcorporate knowledge due to the uniquebimodal age distribution of NASA.Hiring freezes between the Apollo andShuttle programs have resulted in twoprimary groups in NASA. Approxi-mately, half of NASA consists of Apolloveterans within 5 years of retirement.The other half consist of personnelunder _S with Shuttle-only experience.NASA Considers it highly desirable to

capture the corporate knowledge of theApollo veterans in knowledge basedsystems before they retire. Because themainframe complex is primarily orientedto data display, it is a poor environmentfor capturing and utilizing knowledge.

These factors have resulted in aggressiveefforts by NASA's Mission OperationsDirectorate to utilize a distributedsystem of Unix (TM) engineering-classworkstations to run a mix of online realtime expert systems and traditionalautomation to allow flight controllers toperform more tasks andto capture thecorporate knowledge of seniorpersonnel. Starting with the first flightof the Space Shuttle after theChallenger accident, this effort, namedthe Real Time Data System (RTDS), hasplayed an increasingly significant role inthe flight-critical decision makingprocess.

APPLICATIONS EXAMPLES

The application o!,these techniques hasresulted in a new look and feel toMission Control. Picture 3 shows antelemetry-animated schematic of theShuttle's communication and trackingsystem. This display contains all of theinformation contained on the

traditional monochrome text displayshown in picture 2. The display utilizescolor graphics to organize theinformation into a schematic. It alsocontains rules which draw inferencesabout the systems performance andoperation from the telemetry.Previously, a major part of an operator'straining was to learn how to look atcomplex displays of digital data andbuild a mental model of the system.Only after this training was complete,could an operator be trained to evaluatethe situation and make recom-mendations. Utilizing the RTDSapproach allows the operator to utilizethe expertise of senior operatorscaptured in the display program to builda mental model of the system and jumpto learning how to evaluate the system.

This effort has also resulted in dramaticnew and unexpected capabilities. For

example, fl!ght controllers who monitorthe Shuttle s Remote Manipulator

System (RMS) traditionally determinedthe position of the robotarm" by

Picture 3 - Telemetry-AnimatedCon_munlcatlons Schematic OnWorkstation

observing digital readouts of the anglesof each of the arms joints. A combina-tion of offline tools and mentalgymnastics allowed operators todetermine the arm's position and adviseastronauts on operation. Picture 4shows an RTDS application whichacquires real time telemetry of the arm'sangles and animates a view of theShuttle showing the arm's position. Thisapplication not only lowers the flightcontroller's workload, but also allowsthe controller to visually monitor forpotential collisions of the Shuttle andpayloads.

Picture 4 - Remote Manipulator SystemWorkstation Display

5.1-2

326

ORiGiNAL PAGE ISOF POOR QUALn'Y

ORIGINAL P_,GE iS

OF, POOR QUALITY

In another example, flight controllers

typically monitored the performance ofthe Shuttle s flight instrumentsbywatching digital displays of instrumenttelemetry describing the Shuttle's roll,pitch andyaw attitude angles or thebearing to the landing site. Picture Sshows an RTDS display which acquiresthe real time telemetry on the flightinstruments and displays an emulationof the flight instruments on a work-station screen.

Picture 5 - Workstation Emulation of

Shuttle Flight Instruments

RTDS has also been applied to real timedata sources other than telemetry. Oneof the most significant applications hasbeen in the evaluation of weather data.The Flight Director is the NASAindividual in Mission Control with finalauthority on all flight decisions. One ofthe most difficult tasks for the FlightDirector during landing is the selectionof runways based on winds. Because theShuttle during landing is essentially avery large glider, it has very criticalcrosswind limitg. Traditionally, duringlandings the Weather Officer wouldread out wind values from the landingsite and the Flight Director was requiredto determine if the winds were withinlimits by consulting a paper crosswindgraph. This could get quite hectic asthere are several runways and the windsat Edwards Air Force Base arenotoriously variable. In RTDS we builtan application which receives the windreports electronically from the landingsites, computes crosswind components,applies flight rules to determine if windsare within limits, and displays the resultsgraphically to the Flight Director in realtime. This application dramaticallylowers the Flight Director's workloadand enhances the safety of flight.

In developing RTDS, NASA met severalchallenges in the use of Unix (TM)workstations to operate in real time andto provide real time expert systems andcolor graphics in mission-criticalenvironments. All of these applicationsrequired access to real time data. Theremainder of this paper explains thetechniques developed to acquire realtime data under Unix and supply it toexpert systems. The techniquesdescribed in this paper have not onlybeen used on Space Shuttle but also onaeronautics applications. Systems havebeen developed usingRTDS for aircraftflight test operations by NASA s DrydenFlight Research Facility (reference 1) formonitoring the X-29 and by the AirForce Flight Test Center (reference 2) for

monitoring the F-15 short takeoff andlanding demonstrator aircraft.

HARD AND SOFT REAL TIMECONSTRAINTS

The most important item to understandin developing real time Unixapplications is the difference betweenhard and soft real time constraints. Asdefined by Stankovic (Reference 3) realtime systems are those where thecorrectness of a computation is afunction of both the result of thecomputation and the time at which thecomputation is delivered. Hard realtime systems are those where thefunct!on is completely failed if thecomputation does not always meet thetime constraint. Soft real time systemsare those where system performance isdegraded if the time constraint is notalways met, but can be fulfilled if theconditions are met with a responsedistribution. A typical hard real timesystem is an aircraft flight control systemwhere loss of control occurs if the systemdoes not meet time constraints. An

example of a soft real time s_/stemwould be an airline reservaUon systemwhere slow response results in degradedoperations but not system failure.

In RTDS we had to meet both hard andsoft real time constraints. The hard realtime constraint occurred in theacquisition of real time data intoworkstations. The soft real timeconstraints were in the performance offault detection algorithms, faultdetection rule based expert systems anddata displays. Understanding thedifference in these types of constraints isthe key to successful implementation.

Unix Data Acquisition is A Hard RealTime Constraint

In RTDS we tapped into Space Shuttletelemetry as soon as the data reachedthe Mission Control (Diagram 1).Telemetry is a uniquely structured data

I327

mLB m

3 r

I

I mr_m_

I

stream. In order to minimize hard realtime processing in the Unix workstationsthe basic telemetry processingroles ofbit, frame, and subframe synchroniza-tion, and parameter extraction(decommutation) were performed in adedicated telemetry processor (LoralInstrumentation ADS-100 or System500). The extracted parameters(approximately 5,000 16-bit words persecond) were communicated to theworkstations (Concurrent 6600) via aDirect Memory Access (DMA) interface.The DMA is based on the DigitalEquipment Corporation (DEC) DR-11Wstandard using an Ikon Corporationinterface board installed in theMasscomp. The telemetry processorbuffers the incoming telemetry data in a1,000 word First In First Out (FIFO)buffer. The buffer unloads over theinterface when polled by theworkstation. The hard real timeconstraint was that the FIFO bufferwould lose data (overflow) and requirea reset if it was not polled before itfilled. The Unix workstation had to pollthe system sufficiently to drain thebuffer before it overflowed.

Unix has some well-kn0wn problemsthat hinder its use in hard real timeapplications. Unix normally does notprovide the capability to assign hard

priorities to tasks. This makes itimpossible to require that a task executeat a given rate, such aspolling atelemetry processor at four times asecond. This is complicated by the factthat in most Unix implementations thekernel cannot be pre-empted. Thus,even if an external device, such as a_.elemetry processor, needs service, thekernel may block its servicing.Additionally,,most Unix schedulers,reduce a task s switching priority inproportion to the processor time used.

328

This is a problem when data acquisitionis to be performed continuously over aseveral day mission, Most Uniximplementations perform taskswapping; in addition, virtual memoryversions also perform on-demandpaging. If a task has critical code or datapaged to secondary memo_ or if theentire task is swapped out, then thetime to recover critical code and datamay violate real time constraints. Unixalso typically buffers all disk transactionsin buffer cache in primary memory. Thisintroduces an uncontrolled factor incritical disk I/O tasks such as data

logging. Concurrent Corporation's RealTime Unix (RTU -TM Reference 4) has

eCific features to allow the user toal with these constraints. In specific

RTU allows tasks to lock a memorysegment, and to circumvent the normalchanges in switching priority by *specifyinga fixed real time switchingpriority. RTU provides contiguouslyallocated disk files that allow the user tobyRass the clisk buffer cache mechanismand perform direct I/O to disk. RTUprovides kernel pre-emption withinspecified time constraints.

Even with these capabilities, dataacquisition systems and real timeapplications under Unix must be - _carefully structured. Specifically, thetasks performing data acquisition mustbe isolated from applications processingso that increasing the applications loaddoes not prevent the data acquisitionfrom meeting the hard real timeconstraints. In order to deal with thisproblem, a technique was developed totake advantage of multiprocessingcapabilities (Diagram 2).

_/_.._/_ _ e DATA ACQUISITION SOFTWARE

In this technique a single "stripped-down', task manages the DMAcontroller, instructing it to fill differentsediments of a ringbuffer in sequence.This "Ring Stuffer TM was identified as thehighest priority real time task in the_stem. A second task, called the

Buffer Stuffer', reads the ring and

performs decoding steps toplace datainto time homogeneous buffers inshared memory that can be accessed inparallel by many applications.

The two stuffers interact in several ways.The buffer stuffer is set to a real timec_riority so that its switching priority

oes not degrade with accumulated CPUtime, but at a lower priority than thering stuffer to prevent interfering withthe hard real time performance of theringstuffer. The Ring and Bufferstuffers communicate their position inthe ringthrough shared memory. If abufferbeing accessed by the BufferStuffer is about to be overwritten by theRing Stuffer, then the Ring Stuffer stillperforms DMA to meet the real timeconstraint and prevent overflow of thetelemetry processor FIFO. The data istransferred into a "bit bucket', a sparebuffer dedicated for this purpose.Although this mechanism can lose data,acquired data is not contaminated.Whenever data is lost, tasks are notifiedby flags in shared memory. When twoprocessors are available in aworkstation, the two stuffers are run inparallel on separate CPU's to preventcontention.

This dual-stuffer technique was alsoimplemented in a 80386 PersonalComputer running the Lynx (TM) realtime operating system to acquireweather data for the Flight DirectorWinds application. The weather datawas acquired over an asynchronousserial line and placed in a ring by onetask, and organized for evaluation intime homogeneous buffers by anothertask.

Shared memory is vital for interprocesscommunication i_nthis approach. The

Unix AT&T System V shared memoryimplementation is very good for realtime data acquisition and distribution.It is important to perform the dataacquisition task as a service and makeacquired data available to allapplications by shared memory. Ifshared memory is not available, theneach application would have to performsignificant data acquisition tasks andthis would severely limit the number ofsimultaneous applications. Sharedmemory was also an importanttroubleshooting tool for the dataacquisition software. Normaldebugging techniques typically involvemessages written into files or to theterminal. This assumes that the

debugging techniques will not impactthe proper operation of the process.However, in a real time environment theuse of such techniques significantlyaffects the timing characteristics of thesystem and cannot be used. Instead,RTDS logs information about the dataacquisition (such as pointers, number ofbytes transferred, overflow flags) toshared memory. This allowed us to buildgraphic monitors which can be observedi-n real time to troubleshoot data

acquisition problems.

Several other RTU features are critical tothe performance of RTDS. Theseincluded the ability of an application todelay for time periods less than 1second. This function was requiredbecause high priority tasks such as thebuffer stuffer had to delay for incomingdata and the standard Unix minimum

sleep of 1 second was too long to meetrate constraints. RTUprovides amechanism allowing delays as short as 1millisecond. RTU also provided a time ofday clock (for rate calculations) andsignals for trapping floating pointerrors. Floating point error traps arecritical because noise in data can causefloating point errors and applicationsmust trap and handle these errors. Realtime tasks that perform continuouscyclic display also need the capability topoll for user input without holding.Conventional Unix terminal drivers

provide the O NDELAY option to allowa single application to read thekeyboard without delay. This howeverdoes not provide a mechanism forcontrolling which applications shouldreceive the keyboard input. An XWindows approach would be themodern solution to this problem and isbeing used by RTDS in its newestapplications. At the time we startedRTDS however, X Windows was notavailable on our equipment, so we usedmouse button signals with signalhandlers to provide inputs toapplications. The mouse signals did notrequire polling, so cyclic displays wouldnot delay for user input. AI/oftheapplications received any mouse inputso that the handlers were required todetermine from the cursor location ifthe inputs were intended for theirapplication.

This technique has been highlysuccessful in processing real time data.During a recent mission, a workstationrunning a moderate applications loadran for over 3 days without an overflowoccurring. With a heavily loadedworkstation we experience an overflowevery 10-12 hours. This vulnerabilityoccurs because the ring stuffer runs as aUnix application. It context switchesfrom application state to kernel state

5.1-5329

wheneveritexecutes the DMA Con-troller driver. If another applicationrequests a kernel service while the ringstuffer is in applications state, thecontext switch of the ring stuffer can bedelayed. To minimize the effects of thiscase we modified the telemetryprocessor board to automatically resetitself when an overflow occurs. This isnot completely satisfactory as data lossoccurs during the reset and we arecurrently developing a version of theDMA driver that performs all of the ringstuffer activity in the kernel.

RTDS is an interesting statement on thepower of current engineeringworkstations. The Apollo MissionControl Center used for the lunarlandings in 1969 processed less than

1,000 parameters a second in the largemainframes of the time. RTDS processes4,000 parameters a second in a singleworkstation.

Data Display and Expert Systems are SoftReal Time Tasks

When we started RTDS we thouc_ht ofdata display and computation alongtraditional mainframe-based terms.Specifically, we expected all data to bedisplayed and all computations to runon every data sample. After some earlyexperimentation it became clear that itwould not be possible to display dataand run rule based expert systems in ahard constrained fashion. The nature ofrule based expert systems makes itdifficult to guarantee hard real-timeperformance. In rule based expertsystems, computational load variesbased on the number of rules fired andthis varies with circumstances beingmonitored.Examining the actual monitoring tasksperformed by flight controllers revealsthat although data is displayed at once,t)er second, it is not monitored at thatrate. Flight controllers are themselvesmultitasking and monitor severalscreens, event lights, and voi_e loops aswell as using other materials such asproced,ures and schematics. The humanmonitor is a "soft" real timeimplementation.

We also found that the tasks could bestructured so that only key detectionlogic was being evaluatedevery second.Supporting logic was only activatedwhen primary logic detected a problem.This significantly improved our real timeperformance. In specific, a rule-basedexpert system monitoring the Shuttle'scommunications system utilizedapproximately S00 rules. These wereinitially implemented in CLIPS, a rulebased expert system tool developed bythe Mission Planning and AnalysisDivision at Johnson Space Center. Thistool does not have any special real-time

support. By structuring the rules intophases and enforcing certain

precedence, we found thatapproximately 100 rules were requiredto capture the key detection logic. Inthe Unix environment, CLIPS was able tofire these 100 rules approximately 2 to 3times a second. When the key logic rulesdetected problems, additional rulesfired, which slowed down the systemand caused only momentary violation ofreal time constraints.

It is important to realize that the Shuttlesystems do not normally operate withcontinuous failures being introduced.One of the goals of the expert systems isto provide expert evaluation whenfailures occur. If the system slows downwhen the failure occurs, but is still ableto provide the expertise, then it ismeeting its desired function.

When we performed tests on the)erformance of the fault detection, weound that a large percentage of the

processing was meeting the once-a-second constraint and all of theprocessing was being performed on a 2to 3 second cycle. This was notdetectable by flight Controllers lookingat the RTDS displays.

We also found that RTDS displaystelemetry 3 to 4 seconds ahead of themainframe complex. This is because theRTDS architecture minimizes thenumber of data transfers betweenprocessors. Because the mainframeperforms such a large number ofcomputations, it requires extensiveminicomputer preprocessing to meetreal time constraints. This introducessignificant delay in the mainframesystem.

On several occasions during actualShuttle flight, RTDS has detectedproblems and brought them to theattention of flight controllers beforethey noticed the problems on theconventional displays. In several cases,the mainframe displays have beencompletely removed from the Controlcenter andthe controllers rely entirelyon the workstation based displays.

SUPPORTTECHNIQUES FOR REAL TIMEEXPERT SYSTEMS

In developing the real time dataacquisition support for expert systemswe utilized several critical techniques.The most important technique is that ofthe time homogeneous buffers. If taskautomation or a rule based expertsystem is to combine several differentpieces of sampled information andmake a decision on them, then the timerelationship of these samples must beknown. Typically we try to only combine

33(_ 1-6

datafromt:hesamedataacquisitionsamplingcycleormajorframe.Amajorframeisthetimeperiodinasam.pieddatasystemwhenallmeasurementsaresampledatleastonce.On Space Shuttlethe maior frame is sampled andtransmitted once per second. Allmeasurements from a given majorframe represent a time homogeneousdataset bounded by the sampling rate.

An example of the importance of thistype of relationship is detecting a failedreaction control system jet thruster onthe Space Shuttle. There are twoconditions that we want to detect. A jetthat does not fire in response tocommand is considered failed off. A jetthat does not turn off when thecommand is removed is failed on. So wehave two rules :

a. Ifjet command ison and jetchamber pressure is low,then jet is notfiring andfailed off.

b. If jet command is off and jetchamber pressure is high,then jet isfiring andfailed on.

If the jet command telemetry parameteris not from the same sampling period asthe jet chamber pressure command, twothings can happen which cause a normaljet firing to be misdiagnosed as a failure.If the command measurement leads theresponse measurement, then the firstrule will be satisfied indicating that thejet is not responding to commands. If

the command lags the response, thenthe second rule will be satisfied

indicating that the jet is firing without acommand. It is only when the commandand response are from the same framethat a normal firing will be properlyevaluated.

In RTDS we placed data into timehomogeneous buffers in shared memoryon major frame boundaries. We usefour buffers on a round robin basis. TheBuffer Stuffer places telemetryparameters in the round robin buffers innamed locations where they can beextracted by applications using standardlibrary routines. Whenever a parameteris placed in the buffer it is marked asvalid for that major frame. The stufferalso searches for the frame markers inthe telemetry stream. When a majorframe marker is detected, the bufferstuffer closes the buffer being updatedand makes it available to applicationsfor reading. Flags are set in sharedmemory to indicate the most recentlyupdated buffer. After releasing thecompleted buffer to applications, theBuffer Stuffer then opens the nextround robin buffer. Before starting tofill this new buffer, data from the lastmajor frame period is copied into thenew buffer. All data is marked as invalid

5.1-7

for the new major frame when it iscopied forward. As each parameter isprocessed in the new major frame, theparameter status is updated to valid. Bycopying forward the most recentlyreceived data, we ensure thatapplications always have access to themost recently received data (withappropriately marked validity), even ifthe data is not received in a given majorframe due to errors in transmission. Byswitching buffers and making themavailable to applications at major frameboundaries, we maintain the timehomogeneity of the original sampleddata stream.

In order to maintain major frame time-homogeneity in the applications, it wasnecessary to ensure that once the dataacquisition library starts to pull data

from a buffer, that all of the parametersare pulled from only that one buffer(major frame). This must happen whileUnix is switching applications, pagingand swapping. It make take more than 1second for an application to complete itscomputation cycle between dataacquisitions. The four round-robinbuffer design gives an application threemajor frame times to complete anacquisition before the buffer isoverwritten.

This major frame buffer techniquemaintains the data time characteristicsfor automated monitoring and expertsystems. Alternative approaches havebeen used in other telemetry computersystems which lose this critical timerelationship information. An alternativetechnique that has been implemented inat least one major NASA system and twonew commercial off-the-shelf telemetrymonitoring systems is called the CurrentValue Table (CVT). In CVT, telemetrydata is acquired and the most currentvalues of parameters are placed in asingle table without regard to the majorframe. When an application requestsdata, the CVT ships out the most currentvalue received for the requestedparameters. Because the requests occurasynchronously with the dataacquisition, it is almost certain that datafrom multiple major frames are in thesame data request. This technique maybe acceptable for low rate data displaysand limited automated monitoring butis not acceptable for advancedautomation using rule-based systems.

Major Frame Buffers For Logging andDistribution

The major frame buffers are also apowerful structure for logging data. InRTDS, the buffer stuffer logs allparameters to disk in contiguous files.RTDS has an "instant replay" modewhere real time data acquisition isstopped and a "replay stuffer" is used tostuff data into the major frame buffers.

331

Inthiswayalloftherealtimeapplicationscanbeusedinplaybacks.ThereplaystufferhasacontrolpanelwhichisfashionedafteraconventionalVideocassetteRecordercontrolpanel.Wechosethisinterfacebecauseitwasfamiliartoalmosteveryone and it has allof the functionality needed. With theVCR control panel users can playbackdata, view in fast forward, "rewind, orShuttle between set points. Speed ofthe playback can be adjusted for slow-motion analysis.

This capabilityhas turned out to beessential for three reasons. First, thecapability was essential for debuggingthe automation applications. Datasignatures captured during actual flightor simulations can be replayed time andtime again to work out bugs inautomation and expert systems as wellas to perform regression testing.Second, as a real time tool this capabilityhas enabled operators to significantlycut the time required to view playbackdata. This capability was useddramatically after the pad engineshutdown and countdown abort on thefirst STS-34 launch attempt. RTDSenabled engineers and managers toreplay the shutdown within minutes totroubleshoot the cause. This is a bigimprovement over the current playbacksystems which can take from 30 minutesto 5 hours to retrieve playback data. Infact, the director of Mission Operationhas stated that RTDS paid for its entiredevelopment in those few minutes.third, there has been an unexpected

training benefit from the playbackcapability. Flight controllers can recordsimulations and then play them back attheir convenience for training. Severaltraining objectives in flight controllercertification are now met by thistechnique. This saves the large costsassociated with meeting trainingobjectives by a full-up simulations withthe entire simulator, control center, andflight team in place.

The major frame buffer is also naturalformat for distributing data over localarea networks. In RTDS we distributemajor frame buffers over Ethernet (TM).

This enables RTDS to provide remotetelemetry monitoring and softwarecheckout, even on computers whichcould not normally support real timedata acquisition. The User DatagramProtocol (UDP) subset of TCP/IP was usedto provide a connectionlessunacknowledged data transfer. Thisallowed the transmitting workstation tobe unaffected by the receiverworkstation if the receiver was unableto keep up with the transmitter. Thiscapability has allowed us to conductoperational demonstrations whereflight controiTers-=mor_i_tored data out oftheir offices. We sent the data to the

experts, rather than sending the expertsto the control center. This techniquewill become more important as NASApursues long term missions such as SpaceStation where it becomes less feasible totie experts down to a central location.

Data Quality of Frames and IndividualParameters

In order for expert systems and taskautomation to use real time data, it isnecessary to determine the quality ofthe data. There are two measures ofdata quality, the quality of a majorframeand the validity status ofindividual parameters.

In order to determine the validity of aframe of data, we utilized the fact thateach major frame of telemetry is isdivided into a number of smaller frames,called minor frames. Each minor framecontains an identifying counter. InShuttle there are 100 minor frames permajor frame and one major frame persecond. Parameters are spread acrossthe minor frarnes_ in telemetry systems,data can be interrupted due to radio-frequency noise on the space-to-groundlink. Typically noise is of short durationand will only affect one or two minorframes.

In order to inform applications whennoise was present, the minor framecounter is transferred via DMA to theworkstation. The Buffer Stuffer

examines each frame counter to ensurethat all 100 frames are received insequence for a given major frame. ThisQuality parameter is expressed as anumber from 0 to 100 indicating arough percentage of the quality of thedata. The Quality parameter is placed inshared memory, with one qualityestimate for each major frame buffer.Applications receive the buffer Qualityvalue whenever they request data froma buffer. In this way applications canchose to display or discard data based onoverall data quality. In many datadisplay tasks in RTDS we chose to displayall of the data even in high noise (lowQuality) conditions. In critical taskautomation, we discard all data thatdoes not have a 100 percent quality toprevent erroneous results.

There are approximately 32,000arameters in the Space Shuttle systemut only approximately half of them are

being downlinked at any one time. Thisis due to the restriction of the downlinkbandwidth of the telemetry system andrecognizes the fact that certain data isnot required during all flight phases. Forexample, engine data is only neededduring launch. If during a major frame,data is not in the specific telemetryformat, then RTDS marks the individualparameter status as invalid. When an

applicationattemptstoacquiretheparameter,theapplicationreceivesboththevalueandthestatusoftheparameterfromthebuffer.

IndividualparametervaliditystatusisanimportanttechniquethathasbeenoverlookedinseveralNASAandcommercial systems and caused seriousarchitectural problems when retrofittedinto these systems. This function mustbe providedby the data acquisitionsubsystem. Without a system solution,each application must contain sufficientformat definition information toindependently assess validity. This is asevere software maintenance and leavesthe possibility that applications mightfunction improperly due to stale data.

Calibration and Conversion

In typical flight vehicle telemetrysystems data originates from one of twomajor sources. Some data is acquireddirectly from sensors and other data isthe result of computations in theonboard computer. Calibration is thefunction of calculating engineering unitvalues (temperature, pressure, etc..)from sensor telemetry. Conversion is theprocess of transforming values from theword formats of the flight vehiclecomputer into the word formats of the

round computer. Both of thesenctions must be performed if an

expert system is to use telemetry.

Typical sensors convert some physicalquantity (e.g. temperature, pressure)into an analog value proportional over aspecified range (Diagram 4). Sensoroutput voltages are normally amplifiedand then converted to a digitalvalue.These values are usually called ' counts."The sensor value in counts (8, 10, 11,12and 16 bit are popular sizes) is placed ina serial stream for transmission to the

ground. On the ground, the computersystem converts these to a numberrepresenting a physical quantity. This isbecause it is much easier for humans and

expert systems to reason about physicalquantities than 'counts." This is donewith a calibration curve of the form:

y = A(0) + A(1)*JX + A(2)*X 2 + A(3)*X _ +.

wher_ the counts are supplied as the Xvalues and the A(N) values are thecoefficients. In RTDS we use the Shuttleprogram standard fifth orderpolynomials. The coefficients for thispolynomial are stored in shared memoryso they can be viewed and altered andto assure they are common to all

applications. Acquired, data is placed inthe shared memory in counts andwhen applications request data, theshared memory is examined to obtainand apply the calibration curve.

5.1-9

CAUBRAnON ¢ONCEPn

Flight vehicle computers historicallyhave unique architectures due to thedemands on weight, size, powerconsumption and reliability in hostileenvironments. In fact,the Space Stationprogram is the first NASA mannedprogram to specify a hardened standardcommercial-off-the-shelf architecturefor a flight computer (80386). Becauseflight computers tend to be unique, the)usually have unique floating pointformats (Diagram S). Unix workstationsusually use IEEE standard floating pointformats or manufacturers variants. Thedata acquisition subsystem must be ableto convert between these differentnumber representation subsystems ifthe display user or expert system is tointerpret the data. In RTDSwe keep theparameter conversion information inshared memory together with thecalibration curve. Data is kept in theflight vehicle form in the sharedmemory. When an application requestsdata, the shared memory is used toselect and apply the appropriateconversions. Twelve different types ofconversions are required on SpaceShuttle. Override modes for bothcalibration and conversion are providedin the library calls so that userapplications can acquire raw datadirectly as it was downlinked from thevehicle.

Noise Filtering

Even when the quality and individualparameter validity mechanisms are used,there still is the possibility of gettingincorrect data into a real time expertsystem. There ace two basic sources oferror. First, communications noise maybe of very short duration so as to onlyaffect a small number of bits. If thenoise doesn't affect a frame counter or

frame synchronization marker, then it isdifficult to determine (from a datacommunications standpoint) that anerror has occurred. Some telemetrysystems use parity bits on individualparameters and frames or a forward-error-correction technique but these arenot available on Shuttle. The second

333

S_TtLE

o ,

SHUTTLE FLOATING pOINT FORMAT VS.

IEEE STANDARD

(IBM BIT NUMBERING CONVENTION)

,lee --

source of noise is the sensor itself.Sensors are nonideal devices and can benoisy. In RTDS we applied "noise-filtering" techniques to minimize theeffects of these errors on applications.

The first technique was applicable todiscrete (binary 1 or 0) values, such asswitch settings and valve positions.Whenever one of these items changed,it would not be provided to the expertsystem unless the change was presentfor a specified number of seconds (N).This N-count noise filter is a techniquewhich has been used successfully in theonboard automation of the SpaceShuttle. The problem with this filter isthat a "chattering" sensor is notdetected if it changes state faster or atthe same rate as the noise filter.Operationally, the N-count algorithmworked well for our applications.

The second technique was applicable tonumbers. A numeric value maytakemany values. In a slowly changingsituation the same N Count algorithmused for discretes can be used. Butwhere values change rapidly, comparingupdates to the last value can result in noupdates being made available to theexpert system because the last valuenever stabilizes. For fast changingnumeric values we used reasonablenesstests. The expert systems andautomation would reject values thatwere not in a specified reasonable rangefor that parameter.

In some cases, these checks wereperformed within the expert system orautomation. But in many cases the samenoise-filtered value was required byseveral applications (automated faultdetection, displays etc...). In these cases,we wanted a single authoritative copyfor all applications. We used anothershared memory buffer to which thenoise filtering routines could write. This"signal buffer" was accessible by allapplications by requesting parameternames through a library routine. This"signal buffer' did not represent amajor frame time-homogeneous buffer.

5.1-t0

334

Because noise filters are set at differentvalues for each parameter, there was noway to maintain time homogeneity" Themechanism however worked very wellas both a repository for best-estimate"values for low dynamics tasks and ageneral purpose mechanism forcommunicating results betweenapplications. Many applications usedthis mechanism.

FLIGHT TEST- THE SKY'S NO LONGERTHE LIMIT

It is appropriate that this conference'stitle includes the statement that the skyis no longer the limit for flight test.RTDS and similar systems will be used inthe next few years by NASA to performthe most extensive set of space flighttechnology tests since the early 1960's.NASA's manifest for the next few yearscontains several missions which willflight test new technologies such astethers, aerobrakes, and free-flyingtelerobots in near-earth orbit. All ofthese missions require advancedtelemetry monitoring and visualizationtechniques similar to those developed inthe RTDS project.

The Tethersat will be flown on STS-46,currently scheduled for late 1991(picture 6). This will be the first attemptto ever use a large scale tether betweentwo orbiting objects. This flight willexplore the motion and electrodynamicsof tethers in orbit.

Picture 6 - Tethersat

ORiGiNAL PAGE tS

OF POOR QUALITY

m

Also in late 1991, STS-49 will testelements of the Flight TeleroboticServicer (FTS picture 7). The FTS is beingdeveloped by Goddard Space FlightCenter as a tool to assist in the assemblyof the Space Station. It can operate infree-flying mode or attached on the endof the RMS. The STS-49 flight wil beused to conduct flight experiments onprototype hardware elements. The firstactual flight of the complete FTS will beperformed on STS-72 in late 1993. Thiswill qualify the equipment for use inSpace Station assembly in 1995.

Picture 7 - Flight Tele-Robotlc Servicer

In late 1994 the Aero-Assist FlightExperiment (AAFE) will be flown on STS-82 This flight will deploy an unmannedfree-flying vehicle which will fly aaerobraking profile (picture 8) todemonstrate the feasibility of usingatmospheric aerodynamic braking toalter orbits. This technology isconsidered crucial for capturing vehiclesreturning to earth from the moon infuture lunar exploration programs_

An Aeroa.ssist Flight Experiment (AFE) mission scenario calls

for the AFE to be deployed from the cargo bay orthe Space

Shuttle. A solid rockel motor (SR M) will accelerate the vehicle

tO 33,800 feet per second simulating the speed a! which a

spacecraft trave]s in geosynch ronous orbi! A fief the burn, the

SR M is jettlsoned A dip through the Earth's upper atmosphere

is expected to slow the AFE so it can rendezvous with and be

retrieved by the Shuttle. Back in the cargo bay, I he AFE will be

retu rned to Earth for in depth anal ysis

Picture 8 - Aero-Assist Flight Experiment

CONCLUSIONS

The advances in workstations and realtime Unix have enabled smallprogramming teams to implement realtime telemetry systems that have mademajor improvements in NASA space andaeronautics mission operations. Severalreal time adjustments must be made toUnix and applications properlystructured to meet ard and soft real timedemands. Many monitoring problemsare actually soft real time problems andthus can be implemented using currentworkstations and expert systemtechnology. New techniques in dataacquisition have been developed toensure that the correctness of the expertsystem recommendations is not affectedby the data acquisition process. Thesemechanisms are general and can beapplied to any real time expert system.Although, these techniques are moretraditionally associated with real timesystems or process control rather thanexpert systems, they must be applied forreal time expert systems to provideuseful information in mission criticalenvironments.

References1. Mackall, McBride, and Cohen,"Overview of the NASA Ames-DrydenIntegrated Test Facility", NASA TM-101720, 1990

2. Jones, Madison, and Flanders, "RealTime Discrete Monitoring", AIAA FifthBiannual Flight Test Conference, 1990,AIAA-90-1311

3. Stankovic and Ramamritham, "HardReal Time Systems - A Tutorial",Computer Society Press (IEEE),Washington DC, 1988

4. "Understanding Real Time Unix",Concurrent Computer Corporation 1989

Acknowledgements

The development of RTDS at SpaceShuttle Mission Control has been a teameffort. Fliqht controllers Michael

Dingier, Jon Redding, RonaldMontgomery and Joe Hughes eachdeveloped specific expert systems intheir discipline areas. Cheryl Whittaker,Deborah Horton, and Erick Kindreddeveloped significant applications. BrianKowalski provided invaluable assistancewith the first version of the DMA driver.The development of the X-29 systemwas performed by Dale Mackall,Dorothea Cohen, and Dick Simon fromNASA's Ames-Dryden Flight ResearchFacility. The F-15 STOL system was

O_:Gi_,_.L PAGE tS

OF POOR QUALITY5,1-11

335

developedbyateamheadedbyRobinMadisonoftheAirForceFlightTestCenter. Funding for RTDS was providedby NASA's Office of Aeronautics andExploration Technology, The SpaceShuttle Advanced Development Effortand the Space Station FreedomAdvancedDevelopment Program.

The United States Government does notendorse any products and mention ofproducts in this article does notconstitute an endorsement by theUnited States Government.

5.1-12336