field testing protocols for evaluation of 3d imaging focal plane array ladar systems

8/7/2019 Field Testing Protocols for Evaluation of 3D Imaging Focal Plane Array Ladar Systems

1/14

Field Testing Protocols for Evaluation of 3D

Imaging Focal Plane Array Ladar Systems

Brian Miles, Jay Land, Andrew Hoffman, William Humbert, Brian Smith,

Andrew Howard, Joe Cox, Mike Foster, Dave Onuffer, Sammie ThompsonAdvanced Guidance Division

Munitions Directorate

Air Force Research Lab

Eglin AFB, Florida 32542

Tom Ramrath, Clarke Harris, and Paul Freedman

Fastmetrix, Inc.

1 AbstractHerein are discussed five straightforward field tests that are appropriate for evaluation of the performance of focal

plane array (FPA) based ladar systems capable of generating high-resolution 3D imagery. The tests assess systemlevel performance using traditional imaging targets and ladar specific targets. In addition, the tests allow

comparisons to be made between the predicted performance of a ladar system and the actual performance. Analysis

of actual field test ladar data is included based on appropriateness and availability of data. In the first test, range

resolution is examined when the target is obscured by camouflage; the intent is to provide two pulse returns within

the same instantaneous field of view (IFOV) and determine the source of the range report from different pixels

within the range image with the emphasis on determining performance based on the pulse detection approach that isimplemented. The second series of tests evaluates the lateral and range resolution of the FPA using standard

modulation transfer function (MTF) and statistical approaches. The third test (Sect. 3.4) involves a moving target to

introduce a dynamic version of the previous spatial frequency dependent tests. The fourth test (Sect. 3.5) assesses

the system range performance as a function of received signal, essentially determining the performance of thesystem as signal-to-noise ratio (SNR) is varied. The fifth test (Sect. 3.6) assesses the uniformity of the range

resolution and range accuracy of the FPA.

2 IntroductionThree-dimensional time-of-flight FPA ladar systems are currently being developed for a variety of military and

commercial uses. Performance assessment of these systems begins during individual component bench level tests

and continues through operational testing of fielded systems. The purpose of this paper is to describe a set of tests

and test data analysis that can be performed to assess the system performance at an intermediate point in the productdevelopment evolution mentioned above. The tests described next can be executed once the components have been

assembled to form a fully functional, stand-alone 3-D ladar system. Typically, these systems are at a breadboard

level of development where developers are looking to assess system functionality and basic performance. Test data

collected should aid developers in debugging current systems and should provide benchmarks for further system

refinement. It should be noted, however, that these tests are examples of candidate tests that have been executed at

Eglin AFB and are meant to provide examples for use and to stimulate further discussion rather than to be an all

encompassing definitive test set. The tests presented here can be used with time-of-flight, phase detection and

coherent ladar approaches.

2.1 Test Philosophy and FPA Evaluation ChallengesIdeally, the performance of the FPA could be perfectly separated from the total system performance, but the reality

is that significant amounts of supporting electronics are necessary for overall system function and complicate

analysis and determination of the location of performance limiting factors. For example, access to specific signals

internal to the FPA may not be possible since the desired probe points are often buried deep within the sensor chip

assembly (SCA) and are totally inaccessible. Furthermore, 3-D imaging FPAs are augmented with extensive

additional hardware to allow the systems to function as complete imaging ladar systems. This paper attempts to

Laser Radar Technology and Applications VII, Gary W. Kamerman, Editor,Proceedings of SPIE Vol. 4723 (2002) 2002 SPIE 0277-786X/02/$15.00 43


2/14

offer tests that provide as much information as possible to support an assessment of the FPA performance. In doing

so, care must be taken to note the limitations of the tests and the difficulty in extracting information about the FPAs

while avoiding influences from other elements of the system on the desired measurement. Lastly, the tests provide a

means to evaluate system level parameters that can only be tested in a relevant field environment.

2.2 Outdoor Range

The outdoor range at Eglin AFB was used for these tests, and consists of a primary 17-degree arc. The range iscontrolled for 700 metersladar system nominal ocular hazard distance (NOHD) must be


3/14

Figure 2. Clear aperture resolution panel (left) and rear panel adjustable laser radar resolution target.

3 3-D Imaging Tests3.1 Definitions of TermsPrior to discussing the individual tests, there are definitions of terms that need to be established for consistency. Thedefinitions presented here are exemplar and useful for this paper. Resolution, accuracy and precision definitions do

not appear to be consistently uniform within the community at this point; thus, the definitions below are provided toclarify the authors intent when using these terms.

1. Range Accuracy: The difference between the range reported and the true range.

2. Range Resolution within a pixel (Intra IFOV): The minimum range separation between two pulses

received by the same detector that can be resolved by the range processing circuitry.

3. Range Resolution pixel to pixel (Inter IFOV): The minimum distance that targets in different pixels can be

offset in range and still be differentiated in range by the measurement system.

3.2 Intra IFOV Resolution Investigation - Camouflage Net Test3.2.1 PurposeIt is useful to determine a system's ability to separate two return pulses within an IFOV. This situation arises when

two or more objects are within the same IFOV, but are located at different distances from the ladar system. Thereturn pulse is, in general, distorted by the interaction of the laser pulse with the target. Specifically, the temporal

profile (optical signal vs. time) of a single IFOV ladar return at the entrance pupil is the convolution of the outgoing

Figure 3. Black and white uniformity reflectance panels, camouflage in front of white panel.

temporal laser pulse profile and the shape/reflectance function of the IFOV, ignoring atmospheric effects. Forexample, if the IFOV sees two equally sized normally oriented planar surfaces within the IFOV separated by

distance D and the near object has twice the monostatic reflectivity of the far object, the temporal return will be

Proc. SPIE Vol. 4723 45


4/14

two gaussian peaks, the first twice the height of the second and separated by 2D/c in time where c is the speed of

light. Systems with a second pulse or last pulse detection mechanism should be able to selectively image the object

pixels corresponding to the near object or the far object. For systems with no second pulse detection, it is

informative to determine how pulse detection methods operate in the presence of a 2nd

pulse.

3.2.2 MethodIn order to investigate the effect of multiple objects within an IFOV on range imaging, an 8 by 8 foot section ofcamouflage netting was strung across a wood frame and positioned at various fixed distances in front of a black

panel. The black rear panel has a lower reflectance than the camouflage netting, resulting in a smaller secondary

return peak given an equal subdivision of the IFOV between the camouflage and rear panel. The test can then be

repeated using a white panel in the background, and, for the same equal subdivision of the IFOV, the return from the

white panel will be greater than that from the camouflage. This approach will likely lead to interesting results based

on the type of peak or pulse processing electronics used in the system.

Three-dimensional imagery will be collected at four camouflage/panel separation distances (1 foot, 2 feet, 3 feet and

4 feet), effectively changing the return pulse temporal pair spacing. To ensure that the IFOV of each pixel is

subdivided, household screening of differing reflectivities (black, silver, gray) can be substituted for the camouflage.Since a screen has many mesh periods over an IFOV, each IFOV should effectively have two returns.

3.2.3 Evaluation

Evaluation will consist of determining which object each range report is derived from and whether the camouflageor target can be selectively imaged via pulse processing control. Each pixels range estimate should be color coded

to indicate if it corresponds to the target panel, camouflage net or another range. For the systems that have a 2nd

pulse detection capability, pixel range reports need to be identified based on whether the reported range is the 1st

or

2nd

pulse. The data will be examined to determine whether the differential between the intensity of the 1st

and 2nd

pulses affects the systems capability to correctly report the range to the 1st

and/or 2nd

pulse.

3.3 Resolution Tests with Ladar Panels3.3.1 Lateral Resolution

3.3.1.1 PurposeThe purpose of this test is to evaluate the lateral, or transverse-spatial, resolution of the ladar system using

traditional MTF analysis methods. The performance and capability of laser radar systems is directly linked to both

lateral imaging and range imaging capabilities combined; range resolution is addressed in the next section.

3.3.1.2 MethodDuring the lateral resolution test the panel shown in Figure 4 was imaged under static conditions. The lateral

resolution was determined/evaluated using the ladar systems intensity image of this panel. The raised bars of the

panel are gray and the background region between and surrounding the bars is black. One hundred frames were

imaged and were averaged for lateral resolution processing. The intensity plots for a single row were extracted from

an intensity image and averaged with all additional rows from the same row of bars.

3.3.1.3 Moving Resolution Panel DescriptionRefer to Figure 4 for a diagram of the layout of the target panel. The target panel has overall dimensions of 8 by 8

feet, and holds 3 identical rows of resolution elements (bars) at varying standoff distances (6 inches, 12 inches and18 inches) from the back plane. Three identical bars make up a bar set, and there are five bar sets per row. Within a

bar set, the bar width and spacing are the same. The size of the bars in a set and the bar-to-bar spacing within a set

decrease from left to right, effectively creating a target of increasing spatial frequency. The spacing of the bars was

determined using 0.09 mrad as the highest spatial frequency, consistent with a high-resolution ladar FPA and 1.44

mrad for the lowest spatial frequency. The resolution element widths were determined based on a range of 257 m to

the target. Refer to Table 1 for a complete list of the target specifications.

Proc. SPIE Vol. 472346


5/14

Figure 4. Layout of moving target lateral and range resolution panel.

(a) (b)

Figure 5. Photograph (a) and intensity image (b) of lateral and range resolution panel mounted on truck;

white panel is in the background.

Table 1. Target specifications at 257 m.

3.3.1.4 EvaluationIt is assumed that the spatial resolution in the transverse and vertical spatial dimensions are identical based on the

circularly symmetric character of the optical system and unity aspect ratio FPA detectors in this example. Lateral

resolution will be determined by evaluating the transverse intensity modulation as a function of the targets spatial

Bar-to-Bar Bar-to-Bar Bar-to-Bar Bar width Moving Bar Freq. Moving Bar Freq.

angular spacing (mrad) spatial freq. (1/mrad) spacing (m) (m) f (Hz) @ 5mph f (Hz) @ 20mph

0.09 11.13 2.571 1.286 0.87 3.48

0.18 5.57 1.287 0.643 1.74 6.95

0.36 2.78 0.642 0.321 3.49 13.92

0.72 1.39 0.321 0.161 6.98 27.84

1.44 0.7 0.162 0.081 13.85 55.29



6/14

frequency. The lateral resolution will be assessed by determining the MTF of the measured intensity data. The

MTF is calculated from equation 3.1.

IMAGE f

OBJECT ff

FFT ( = )

FFT ( = )MTF( = )

= (3.1)

In practice, this is the product of the component MTFs including a sampling MTF, the FPA MTF and the optics

MTF. Figure 5 shows a photograph along with a corresponding intensity image of the lateral resolution test panel

described above. The bright regions to the left and right of the resolution panel in Figure 5b are due to saturationcaused by the unpainted stainless steel sides of the truck. The ladar used to obtain the intensity image has a 0.2

mrad pixel IFOV, so only the two lowest frequency bar patterns are clearly visibility. Figure 6 shows an average of10 rows of intensity data of the lower set of bars on the panel. Figure 7a shows the normalized spectrum

Figure 6. One dimensional slice through lateral resolution panel intensity image.

of the ideal bar pattern model along with that of the actual intensity data for the second set of bars. As described inequation 2.1 the MTF value at the fundamental frequency can be obtained by taking the ratio of these spectra at the

(a) (b)

Figure 7. (a) FFT of 0.7 mrad bar target intensity (solid line) and FFT of 0.7 mrad bar target model intensity,

and (b) MTF evaluated at the fundamental frequency for the first three bar spacings (1.4, 0.7, and 0.36mrad).



7/14

fundamental frequency, which is signified in Figure 7a by the two points at 1.39 mrad-1

. Figure 7b shows that the

fundamental MTF falls to 0.5 at a spatial frequency of approximately 1.9 mrad-1

, which corresponds to a bar spacing

of 0.52 mrad. No useful intensity information is found in the regions containing the two highest bar frequencies,

which are well beyond the detector defined Nyquist limit of 2.5 mrad-1

.

3.3.2 Range Resolution and Range Accuracy Test

3.3.2.1 Purpose:Range resolution and range accuracy determine how close the ladar measured estimate of the objects shape

corresponds to the true object shape. Range resolution is critical to the generation of high 3-D spatial fidelity

images, where ladar systems are often capable of better than 30 cm range resolution.

3.3.2.2 Method:

The white uniform panel, whose range image is shown in Figure 8, was imaged 100 times in succession, providing

range data that can be statistically analyzed on a pixel-by-pixel basis.

Figure 8. Range image of white reflectance panel showing FOV and region where statistics are derived.

a. b.

Figure 9. (a) Mean range to white panel for all 1024 pixels (b) image of mean range to white panel for all

1024 pixels.

3.3.2.3 Evaluation:The range data from the panel was used to generate a 100-sample distribution of range values for each pixel. The



8/14

mean range of the 100 sample set of 32x32 images is shown in trace format and gray scale format in Figure 9a and

Figure 9b respectively. The mean range value for each pixel that images the panel was compared against the

surveyed range to determine the range accuracy of the ladar system. The surveyed range was estimated by the

surveyor to be accurate to within 2-3 cm. The four corners of the white reflectance panel were surveyed and a plane

fit to the four survey points to form the true range data set. An example range accuracy plot is shown in Figure 10,where larger range errors are illustrated by darker colors. The range error data set is further evaluated in a latersection on FPA uniformity.

Figure 10. Range accuracy for all 1024 pixels.

Figure 11. Trace of standard deviation of all 1024 pixels in FPA when imaging white reflectance panel.

Range resolution for each pixel is defined for this paper as twice the standard deviation of the distribution of range

values. The plot of standard deviations for each pixel in the FPA is shown in Figure 11, and indicates an average

range resolution of approximately 30 cm. Range resolution for spatially separated pixels will be defined as the sum

of their range sample standard deviations. The range resolution for adjacent pixels takes into account the non-

uniformity of the range resolution across the array, and determines how well two spatially separated points can be

resolved in range.



9/14

Figure 12. Vertical rectangular regions indicate areas of top surface and back plane. A single point within

each region was used for the histogram below.

Figure 13. Range histogram of resolution panel front plane and adjustable back plane.

In Figure 13, two histograms provide graphic insight into the spatially separated pixel resolution concept. The left

histogram corresponds to 100 range measurements of a front plane pixel on the vertical resolution panel (Figure 2)

while the right histogram is derived from 100 measurements of a single pixel on the adjustable rear back plane.

3.4 Moving Target Tests

3.4.1 PurposeThe objective of this portion of the field test is to assess the 3-D imaging performance of tested systems when used

to image an object moving transverse to the range direction. This will help to determine whether artifacts develop

when using the systems in dynamic scenarios. Systems that perform frame averaging, require multiple frames for a

range image or have long integration times are examples of systems that may exhibit temporally dependent

performance.



10/14

3.4.2 Method

The approach is to perform 3-D imaging of a moving target that has varying transverse and longitudinal spatial

frequency. Due to problems with the example ladar system, the data for the moving target was too poor to be

useful.

3.4.2.1 ProcessThe target panel was mounted on the side of a 5 ton panel truck, and data was collected at three predeterminedspeeds: 0 mph, 5 mph and 20 mph. Initially, the truck was parked with the panel in the ladar FOV to allow for

collection of static data for baseline purposes and to be used for the lateral resolution tests previously discussed. For

the non-static portion of the test, the truck moved from left to right on Road 2 while the system operated as shown in

Figure 14. Table 1 has the frequencies (f) for the various bar sets at both speeds. This was calculated by simply

dividing each speed by the bar-to-bar spacing.

Figure 14. Illustration of non-static target panel test at C-3.

3.4.3 Evaluation

Three sets of range images were provided, one for each vehicle speed. The lateral spatial resolution will be

determined based on the methods of section 3.3.1. Dynamic range resolution will be defined as the bar spacing forwhich the range modulation between the bar and the back panel drops to 50% of its peak static (non-moving) value.

3.5 System Performance as a Function of Return SignalIdeal characterization of ladar system performance would test the systems 3-D imaging capability as a function of

the SNR. This requires the ability to measure the signal output after the first stage of amplification (the primary

contributor to the noise figure at low photon fluxes) with a high bandwidth oscilloscope. Due to the probable

inaccessibility of these outputs in each of the unit cells of the FPA of the assembled system, more indirect methods

of assessing system performance must be used. Specifically, two candidate approaches to assessing systemperformance as a function of return signal strength are suggested and are described in Sections (3.5.3, 3.5.4). This

technique was tested and found to be reasonable, but the candidate laser radar was unavailable for evaluation via this

test prior to this paper submission.

Bldg. 2067Bldg. 2066

257 m

Road 1

Road 2 Truck

Target Panel

Indoor Range

Door



11/14

3.5.1 Background on Return Signal Estimation

Both of the approaches rely on an estimate of the return signal present at the detector. The return signal for a single

detector can be written as follows:

2

t t det ep rec

laser

E R A A Ti

f

= (3.2)

where laseri is the portion of the total current generated by laser reflectance; recT is the transmission of the receiveroptics at the wavelength of the laser; tE is the laser irradiance on target; tR is the target diffuse reflectance; detA is

the detector area; epA is the area of the entrance pupil of the receiver; f is the effective focal length of the receiver;

and is the detector current responsivity (amps/watt) at the laser wavelength. The equation above is for an on

axis (good for small FOVs), lambertian target with a fully illuminated detector IFOV and short path lengths (no

appreciable atmospheric absorption). In the case where the IFOV is underfilled,t

E is defined as the beam power

divided by the IFOV area on target. The laser flux during the course of a single pulse is assumed to dominate solar

flux during the same interval. tE will be measured using a calibrated photometer located at the target; tR of the

target panels will be measured in the lab. The rest of the variables are known system parameters. An estimate forthe return signal can be calculated, thus allowing for an investigation of ladar imaging performance versus return

signal strength without knowing the systems noise figure. A brief description of the measurement of the irradiance

on target,t

E , follows in Section 3.5.2

3.5.2 Target Irradiance Measurement

Irradiance measurements were conducted remotely with a sensor suite located on the adjustable ladar resolution

panel target and control suite located at the indoor range facility. Specifically, the sensor was located in the corner

of one of the central apertures in the resolution panel, thereby providing a point measurement of the irradiance on

the resolution panel. The sensor head and mount is visible in the lower left corner of an aperture in Figure 2. The

sensor suite consists of an ILX Lightwave OMM-6810B Optical Multimeter with Lightwave sensor headappropriately chosen for the ladar wavelength. A two-inch diameter positive lens was used to enhance collection

efficiency. The collection optic was solar shielded to mitigate the effects of ambient radiation and the entire suite

was housed in a weatherproof plexi-glass container. The sensor suite also consisted of an ICS Electronics GPIB

controller and Maxstream 2.4 GHz wireless serial modem transceiver. The serial modem transceiver was paired

with a duplicate transceiver at the control suite. A block diagram of the sensor and communication setup is shownin Figure 15.

Figure 15. Concept diagram of irradiance measurement instrumentation.



12/14

The control suite consists of a standard PC and a Maxstream 2.4GHz wireless modem. The two modems and GPIB

controller provide seamless GPIB connectivity from the PC to the optical multimeter. From the control suite an

operator collocated with the ladar system under test will be able to command the multimeter remotely and acquire

irradiance data measurements. LabView software installed on the PC will record irradiance measurements from theoptical multimeter. Data collection will be conducted on an interval that enables accurate irradiance measurementsto be compared with the ladar images extracted during the test. Several approaches to controlling the received signalstrength while holding the background signal constant are presented below.

3.5.3 Controlling Return Signal Strength: Method 1 Neutral Density Filters

3.5.3.1 PurposeThe purpose of this approach is to reduce the laser signal reaching the detector by introducing neutral density filters

into the optical path. The neutral density (ND) filters serve to attenuate the optical signal based on the transmission

characteristics of the filter. The advantage of the approach is its simplicity; the disadvantage is that several large

ND filters must be purchased. The primary approach will be the implementation of ND filters on transmit.

3.5.3.2 MethodThe ND filters can be used to attenuate either the laser output or the receiver input. If laser output attenuation isused, the ND filter must be carefully chosen and aligned properly. The laser peak power and average power mustnot exceed the ND filters damage threshold at the wavelength used. Also, the ND filter must not be placed so that a

retro-reflection from the filter feeds back into the laser cavity; a small tilt often suffices. The ND filter must havesmall enough wedge angle to avoid steering the beam out of the FOV; this effect is likely very small for flash

illuminated systems, but can be appreciable for small IFOV scanned systems.

ND filters can also be used in the receive path to attenuate the signal. In this case, a multiplicative attenuation factor

is included in the numerator of equation 3.2. The ND filter must be sized to the entrance pupil and its wave front

derived ray aberrations must be much less than one detector diameter. A benefit of this approach is that the received

peak energy densities will be much less than the ND filter damage thresholds. The disadvantage is that this

technique filters solar return as well.

In either case, the signal will be gradually attenuated through the use of successive ND filters. At each attenuationlevel, the 3D image data will be recorded based on the systems reported range to a target of known distance. As the

signal decreases, the 3-D imaging capability of the system will diminish and eventually fail.

3.5.3.3 EvaluationThe evaluation of system performance as a function of diminishing return signal is straightforward. The recorded

range data will be analyzed statistically based on a known true range to target. Plots of both range accuracy and

range resolution (including error bars) versus return signal strength will be generated to assess performance.

3.5.4 Controlling Return Signal Strength: Method 2 Varying Target Reflectivity and Distance

3.5.4.1 PurposeAs in Section 3.5.3.1, the purpose of this approach is to control the amount of signal returned to the system.

Reducing the target reflectivity can also lessen the signal reaching the detector. In this approach, several targets arelocated at a common distance and imaged sequentially or within the same FOV to measure signals that can vary by

40:1 at the same range. Moving the target set to progressively further distances lowers the signal by the square of

the distance, allowing for additional adjustability. With a 10:1 range of distances, a 4000:1 ratio in signal can be

achieved. In this approach the target needs to overfill the IFOV so that target reflectivity and power on target areinvariant with distance.

3.5.4.2 Method

This method of return signal control is an alternate choice to the ND filter approach, however, if necessary, there are

several candidate target panels that can be used to perform this test. The common characteristic of each is they willhave diffuse coatings of white, black or gray, and their reflectance will be measured using a spectral reflectometer.

The systems will be required to image the panels at various distances (on the order of 100 to 350 meters) to adjust

the amount of return signal. The 3D image data will be recorded at each of the positions for further evaluation.



13/14


14/14

Figure 17. Range error for 32 by 32 pixel array averaged over 100 frames.

4 SummaryFive tests were presented that can be included in a field investigation of FPA ladar system performance. Thecamouflage netting test was used to better understand a systems performance with a spatially and range subdivided

IFOV. The intent is that for a given system the pulse processing techniques should be varied to study the resultingrange performance. Both black and white rear panels were suggested to help engineer the first and second ladar

return pulse amplitudes. The range resolution for specific pixels was readily calculated based on statistical

principles by calculating the standard deviation of a 100 sample measurement set for each pixel. The lateral spatial

resolution test used a three bar target and in our example indicated a system MTF limit of 1.9 mrad-1

, consisting of

optics, sampling and detector MTFs, limited by the detector Nyquist limit of 2.5 mrad-1

. The range resolution

defined as twice the standard deviation and calculated to be ~30cm was fairly consistent across the array. Statistics

from single spatially separated pixels from the top and back plane of the adjustable ladar resolution panel illustrate

the pixel-to-pixel resolution concept defined as the sum of the respective range standard deviations. The moving

target test was developed to assess the effect of a dynamic target on the range and accuracy performance of FPA

ladar systems. The range resolution and accuracy are affected by SNR and are evaluated as a function of thereceived optical signal strength to determine system performance at long ranges or for low reflectivity targets. TheSNR test is likely the most involved as shown here because of the additional hardware used in this case. It is likelythat simpler measurement schemes can be employed for specific cases. Examples of ways to adjust the signal

strength were presented. A last test was performed to evaluate the FPAs uniformity in range resolution and range

accuracy across the FPA.

1Boreman, Glenn D.; Modulation Transfer Function in Optical and Electro-Optical Systems, Tutorial Texts in

Optical Engineering, Volume TT52, SPIE Press, pg 83, Bellingham, WA, 2001.


field testing protocols for evaluation of 3d imaging focal plane array ladar systems

Documents