ROBERT E. JENKINS and JOHN B. GARRISON

AUTOMATIC CONTOURING FOR TWO­DIMENSIONAL ECHOCARDIOGRAPHY

A . new: noninvasive technique has recently been developed to improve the visual diagnosis of cardIac dIsorders and to obtain subtle information on regional cardiac performance.

INTRODUCTION Nearly half of the 500,000 people who have heart

attacks each year experience no warning symptoms before their first attack. About a fifth report no discomfort attributable to a heart problem during or after the attack. Thus, identification of ischemic heart disease (caused by oxygen starvation of por­tions of the heart muscle) may occur only at the time of death or after sufficient heart damage has occurred to limit the treatment that can be adminis­tered safely and effectively. Therefore, there is great motivation to develop a reliable, safe, and convenient method to screen large segments of the population in order to detect early stages of heart disease.

Real-time, phased-array, two-dimensional ultra­sonic echocardiographic systems have been developed recently. They provide a noninvasive technique that defines the geometry and dynamics of cardiac structure (e.g., the walls of the heart chambers) and allows the rapid collection of image data on cardiac structure and function. These systems are immediately useful for visually diagnos­ing numerous cardiac disorders. In addition, their output contains subtle information on several re­gional cardiac performance parameters, such as variations in the rate of change of cardiac dimen­sions. These parameters are not readily quantifiable even by skilled observers unless they can extract ac­curate numerical information from the pictorial display provided by a typical two-dimensional echo system.

This article summarizes recent research that pro­vides contours of the outer and inner heart wall as a function of time, and reviews use of the contour data in a program to develop improved methods for the early detection and diagnosis of heart disease by noninvasive methods. .

MANUAL CONTOURING SYSTEM A collaboration between APL and the Car­

diovascular Division of the Johns Hopkins Medical Institutions (JHMI) on the development of clinical applications of two-dimensional echocardiographic data has been under way since 1975. 1 The contour­ing method developed, using data from an ex­perimental contouring terminal at APL, has been shown to be a useful medical technique. During the

Volume 1, Number 2, 1980

past year, a production version of the manual con­touring terminal was built for clinical use at JHMI by the Fleet Systems Department at APL for delivery in June 1980.

The contouring terminal is a PDP-l1/34 com­puter-aided system that generates, stores, plays back, and modifies operator-controlled contours that define the inner and outer heart walls as seen on two-dimensional echocardiographic images. Its purpose is for quantitative study of heart-wall mo­tion.

U sing the data from a video tape recorded on the echocardiographic system, the terminal provides a freeze-frame, instant-replay picture of selected heart cross sections on a cathode-ray screen. An operator using a digitizing tablet outlines the inner and outer walls of the heart with two computer­generated curves that appear on the same screen. The word "contour" is used to refer to either the inner or outer curve of the combined set.

In a session typically lasting several hours, the operator processes a series of echocardiographic pictures, usually covering one or more cardiac cycles. At the beginning of each session, the operator enters information for identifying, pro­cessing, and archiving the data, and calibrates the system using calibration marks appearing on the ultrasonic image.

Each inner or outer contour is composed of 128 individual points. The operator first manually posi­tions two sets of 16 control points to outline the in­ner and outer walls of the heart image. After the control points are positioned, the computer spline­interpolates between them and fills in to form a total of 128 points, equally spaced about the center of the contour. The contours are superimposed ' on the echocardiographic image for operator approval and, if necessary, readjustment. When he obtains a satisfactory match, the operator enters the data on disk file and moves to the next image. These data are sent to APL for processing and storage via a telephone link between the terminal computer and the APL's 3033 computer.

AUTOMATIC CONTOURING

As part of a grant from the W. W. Smith Charitable Trust, APL was funded to investigate the possibility of automating the contouring pro-

139

cess. A candidate method was developed and a pilot study was performed to establish its feasibili­ty. The remainder of the article discusses this method.

To automate the current manual contouring op­eration, it is necessary to convert the sequence of pictures used by the operator into numbers so that a computer can determine how the contour lines should be moved during the cardiac cycle to follow the heart-wall motion.

Figure 1, depicting a cross section of the left ven­tricle, shows a typical echo image obtained with a two-dimensional cardiac ultrasound system. The left side of the figure shows the output of the echo system; the right side shows the same picture on the contouring terminal display with the (operator­defined) contours of the inner and outer walls of the ventricle superimposed. These pictures are ob­tained at a rate of 30 per second in typical ultra­sound systems and are recorded on standard TV video tape systems.

A picture is generated on a TV screen by con­trolling the brightness of a large number of in­dividual points on the screen. A conventional TV picture consists of about 500 horizontal lines, each

The ultrasonic system being used at JHMI to obtain the echocardiography data used by the contouring terminal is a Varian V-3000 ultrasonograph. The in­strument is a diagnostic tool that uses ultrasonic echoes to produce real-time, two-dimensional images of the heart and other organs.

The ultrasound signals are generated and the echoes received by a phased array transducer that permits rapid electronic scanning in a plane. The thickness and beamwidth of the scan plane are about 0.5 and 0.3 em, respectively . The beam is stepped through 128 dif­ferent positions covering an 85 0 angle at a rate of 30 full scans per second. The transducer is hand-held on the chest and can be tilted or rotated to portray dif­ferent "slices" through the heart.

The echoes are then displayed and recorded under control of an operator, who may view the fan-shaped cross-sectional image using either Polaroid or 90 mm film, or use a built-in TV camera to provide video to a tape recorder to record the session. The video tape recordings are the input to the contouring system described in the main article.

line having about 500 points whose brightness can be different. The horizontal "scan lines" are clear­ly visible on standard TV systems. The individual points along each line are generally not visible because they are blurred as a result of the limited bandwidth. Anyone such picture can be defined by an array of 250,000 numbers (500 by 500) that give the numerical value of the brightness at each point. The individual points in the array are referred to as "pixels.' ,

Standard TV pictures are transmitted and re­ceived as time-varying voltages that contain the pic­ture information plus horizontal and vertical syn­chronization signals to synchronize the receiver with the transmitter. To convert a TV picture to a set of numbers, the voltage signal must be passed through an analog-to-digital converter.

In contrast to the familiar saying that "a picture is worth a thousand words," one TV picture is worth 250,000 (digital) words. Because 30 complete pictures are presented every second, their conver­sion into tables of numbers requires the transfor­mation of voltage into numbers at a rate of 7 ,500,000 per second. Conversions (either way) at these rates are not difficult, but general-purpose computers can neither provide nor accept numbers at such high rates. Thus, although computers can easily handle many millions of numbers, special in­terface equipment is required to handle the high rates required for digitizing TV pictures and for presenting visually a picture that has been compu ter -processed.

We will' now consider a sequential pair of echo pictures during the cardiac cycle. These pictures are separated in time by 1/30 s; during this time, the walls of the heart have moved a small but signifi­cant amount. With a heart rate of 60 beats per minute, one beat requires 1 s. Therefore, 30 TV pictures are available per beat. The automatic tracking system must measure the change in wall position from picture to picture through the entire sequence.

We have found in the development and use of the manual system that tracking 16 control points

Fig. 1-A typical two-dimensional echocardiogram without and with superimposed contours of the inner and outer walls of the left ventricle.

140 Johns Hopkins A PL Technical Digest

on the inner and outer contours of the heart wall is usually adequate to define the cardiac motion. To initiate tracking, the operator will define, on the first picture in the sequence, the contour he wishes the system to track through subsequent pictures in exactly the same way as is done for manual con­touring.

Figure 2 is an example of a digitized echo picture with one of the 16 control points used to define a contour. Each of these control points will be tracked separately, using a correlation technique based on the following assumptions:

As the heart beats, we assume that, for a short time interval, the tissue in a relatively small neighborhood about each individual point is trans­lating without rotation. We also postulate that, in a short time interval, the tissue in that small neigh­borhood will not change its appearance and small­scale features very much as it changes position. That is, the small-scale image features in the vicini­ty of each point will keep their approximate shape but will translate with time.

These are reasonable assumptions. We have care­fully studied video playbacks of sample cases and have noted that, in addition to tissue edges, which are not always apparent, there are a number of spots of various sizes and brightness that move with the tissue but change in appearance with time. These produce a "glinting" effect as the tissue moves. Watching these image features on stop ac­tion frame by frame, one can see that the changes are fairly slow compared to the scan time of 1/30.s. However, the rate of change is fast enough to make automatic tracking a challenge.

We follow the translation of an individual point with the correlation technique as we move from picture to picture. A small square centered on an individual control point is selected from the current image field. This defines the local feature, and we call this square the "feature window." We then select a somewhat larger square, centered on the same picture coordinates, from the next image; this defines the "search window." The small square is moved within the search area, comparing the two images until the position of maximum correlation is found. This defines the new position of the control point in the new field and thus measures its mo­tion. This field becomes the current field, and it then defines the local features for each control point. We move to the third field and repeat the procedure.

This process is similar to so-called "map match­ing" techniques, except that for the echocardio­gram the "map" is always changing (hence, decor­relating) with time. The solution is to make the latest image field define the local features for each point. In this way, we are faced with only the smallest possible decorrelation of the image as we search for the translation that has taken place.

Obviously, this technique is not "edge tracking," although edges are certainly not ignored in the

Volume 1, N umber 2, 1980

.. . . . .. . . ... .

..

iiHHiiiiiliijq'iii ...... ~ ::;::: :. :: :iiijH::;;:iliiimnniii: ;: ::,:- ::ii.i.i,Z:,:,

...

· :;1 :i).::·:·;: :i';·: :;· ;· · ..... :.: . . ..

Fig. 2-An example of a digitized echo picture showing one contour control point and its feature window.

method. A well-defined edge in the vicinity of a point is as good as any other clear feature and will improve the correlation. However, local correlation does not depend on the presence of a well-defined edge at each contour point. It only requires that there is some structure to the image that is not changing too rapidly in shape from frame to frame.

There is a trade-off in selecting the size of the correlation window. If the area is too small (small­er than the size of the local features), the process becomes dominated by the small-scale noise in the image and the correlation does not have a well~ defined peak. If the area is too large, it can become dominated by features that are far removed from the point of interest and may even be moving in a different direction. This trade-off was one of the areas explored early in the study. We settled on an 11 by 11 pixel feature window and a 19 by 19 pixel search window.

PILOT STUDY To investigate the use of correlation techniques

in automatic contouring, the following steps are re­quired:

The echocardiograph images on standard video tape must be digitized to obtain the numerical representation of each picture. To describe each picture completely, about 250,000 8-bit numbers are required. One cardiac cycle, which lasts about 1 s, therefore requires about 7.5 million numbers. Because it is difficult to enter the numbers into a computer in real time, a block of pictures is trans­ferred from the video tape to an analog disk re­corder. Such recorders can usually store 10 to 20 s worth of data (Le., 300 to 600 pictures). Any selected picture can be displayed continuously or

141

the set can be played back at variable rates. The stop action permits a more leisurely digitization and storage of each picture.

Although the principles behind the tracking tech­nique are easy to understand, the numbers of arith­metic operations and bit manipulations are rather large. For example, using an 11 by 11 array of pix­els around each control point and assuming that the motion from picture to picture is no more than three units in any direction, it is necessary to use a 19 by 19 array as a search window. This results in about 40,000 arithmetic operations for each of the 32 control points as they move from picture to pic­ture.

We performed the calculations on the IBM 3033 central computer using the interactive APL lan­guage. We were able to use a readily available Tektronix-4015 graphics terminal in a gray level mode to display the digital images. This permitted us to use the powerful APL language for the track­ing operation, display the resulting contours graph­ically over the two-dimensional images, and obtain hard copy of the display immediately.

Figure 3 shows the results of processing a block of pictures in an automatic mode. In this experi­ment, we used 20 consecutive high-quality echo pic­tures. The starting contours (Fig. 3a) were gener­ated manually. Figures 3b through 3j show the positions of these points as they are automatically moved by using the correlation tracking technique. Only the point positions at every other picture are shown; the tracking was done picture to picture. The automatic results were compared to the manual contours that were defined by a skilled echocardiographer on a duplicate set of the same pictures.

The results of the pilot study provided convinc-

(a) (b)

(f) (g)

(c)

(h)

ing evidence that this technique has a high proba­bility of being successful.

MEDICAL APPLICATIONS OF THE CONTOURING DATA

Under grants from various sources, the JHMI collaborating personnel, primarily Drs. J. L. Weiss and M. L. Weisfeldt, have been carrying out clini­cal research using the data produced by computer­aided contouring at APL.

Experiments have been conducted at JHMI to validate the accuracy of two-dimensional echocardi­ography in determining heart volumes. A study was made of the accuracy of a ventricular heart volume determination in the dog heart preparation (isolated and working in a plastic tank). This study showed that, without mathematical or major geometric assumptions, two-dimensional echocardiography can be used to determine left ventricular heart volume on a moment-to-moment basis in a remark­ably accurate fashion in a beating heart prepara­tion. A specially designed "volumetric chamber" preparation, developed by the JHMI Department of Biomedical Engineering, was used as a standard for comparison. This validation of accurate volume determination was an essential step in the long-term program. Further validation studies were done that used fewer cross-sectional images. It is important that a few cross-sectional slices provide accuracy comparable to a large number of slices because the time required to obtain the additional data can make the technique cumbersome and not very easi­ly applicable to the clinical situation. Using dog data, volumes were calculated using different numbers of slices. It was found that even with thicker and fewer slices (approximately 1.25 cm

(d) (e)

( i) (j)

Fig. 3-Correlation tracking of initial contour (Fig. 3a), established by echocardiographer, through a sequence of 20 consecutive fields using 16 points on the endocardial and epicardial boundaries. Figures 3b through 3j show, at alter­nate fields, the new computer-determined positions of the control points. It is apparent that the movement of the ven­tricle walls is being followed.

142 Johns Hopkins A PL Technical Digest

thick) the volume estimations correlate very closely with those that were determined by the technique used as a standard for comparison.

The ability of two-dimensional echocardiography to evaluate heart function by measuring the amount of heart-wall thickening during contraction was assessed and compared with results obtained by standard angiographic techniques (where a catheter in the heart injects a contrast dye for x­ray photography). Angiography is a surgical pro­cedure that carries some risk, whereas echocar­diology is noninvasive and, therefore, benign. It was found that the contouring technique of evalu­ating regional thickening in patients with coronary disease and in normal subjects correlates very close­ly with regional malfunction determined by cardiac angiography. Data from the contouring terminal at APL for 34 normal and abnormal patients were used in this study. An important finding of the study was that heart-wall thickening obtained by the two-dimensional echocardiographic contouring data identified the same diseased segments as observed by the standard method of cineangi­ography. Thus, the technique holds promise as an accurate noninvasive quantification of regional heart-wall malfunction in man.

Using the contoured echocardiographic data, it was found that some patients with acute myocar­dial infarction (heart attack) undergo severe bulging or dilatation of the infarcted (or dead) zone at an . alarming rate. This heretofore poorly understood complication of acute myocardial in­farction, known as regional cardiac dilatation, may be an early and potentially lethal consequence of heart attacks . It is difficult to observe by conven­tional methods but is readily recognized by two­dimensional echocardiography using the computer­aided contouring technique.

Since publication of the study on the first 28 pa­tients,2,3 longer follow-ups on these patients have been obtained. Others have been added to the study protocol. The contour data are being ana­lyzed with several new questions in mind: (a) What

Volume J, Number 2, J980

is the role of bulging of the local dead heart tissue (infarct) in the development of left ventricular ab­normal dilatations (aneurysm)? (b) Do the unin­jured zones of the heart undergo hypertrophy (or thickening), and if so, what is the function of these zones? (c) What is the outlook for patients with in­farct expansion? (d) Does the regional dilatation, which occurs so soon after a heart attack, result in later complications such as irregular heartbeats and/or death? Future studies are planned for much larger patient populations to determine whether current drug therapies, such as nitroglycerine or propranolol, or future therapies can prevent these complications following a heart attack.

In these research projects, the manual contouring terminal has been an essential component. The next stage of development is an attempt to automate the contouring process itself in order to reduce the time required for patient processing. With the manual system, a trained medical technician needs about two minutes to contour a single image. This trans­lates to hours of grueling work to process even a few cardiac cycles. If we can automate the process and speed it up by an order of magnitude, the sys­tem will have greatly enhanced clinical applicabil­ity. This is the current thrust of APL's research.

REFERENCES

IJ. B. Garrison, J. L. Weiss, W. L. Manghan, o. M . Tuck, W. H . Guier, and N. J . Fortuin, "Quantifying Regional Wall Motion and Thickening in Two-Dimensional Echocardiography with a Computer­Aided Contouring System," Proc. Computers in Cardiology (1977), IEEE, pp. 25-35,(1977).

2L. W . Eaton, J . L. Weiss , B. H . Bulkley, J. B. Garrison, and M. L. Weissfeldt, "Regional Cardiac Dilation after Acute Myocardial Infarc­tion, " N. Engl. 1. Med. 300, pp. 57-62 (1979).

3 J . T . Willison, "Echocardiography after Acute Myocardial Infarction (editorial), " N. Engl. 1. Med. 300, pp. 87-88 (1979).

ACKNOWLEDGMENTS - This work is supported by two grants from the W . W. Smith Charitable Trust. The major work in building the new contouring terminal was performed in the F3C group, chiefly by M. Granger, R. Grossman, and S. Kahng. The clinical research at JHMI has been carried out under the direct supervision of Dr. James L. Weiss, who is one of the originators of the collaborative program on echocardiography.

143