Topic 4: Sensor Processing

David L. Hall

Topic Objectives

• Introduce the input side of data fusion processing

• Provide a brief survey of sensor types

• Describe how a generic sensor works• Provide a basis for analysis of

sensors for your selected application

Sensors: The Input Side of Data Fusion

Classes of Input • Traditional “hard” sensors

– Observation of physical phenomena via physical sensors

• “Soft” sensors– Humans as observer/reporters– Direct input (e.g., via cell phone)– Indirect input (web logs, etc.)

• Emerging sensors– Emulation of biological sensors via new nano-scale

technologies (e.g., electronic noses)

The rapid evolution of sensor technology provides opportunities and challenges:Nano and micro-scale sensors include the capability for self-location, wireless communication in self-organizing networks, and web-based serves; sensors maybe mobile and provide extensive on-sensor computations

Examples of New Sensors: Intel Mote

• 32 bit processor• Blue tooth wireless comm.

http://www.intel.com/research/exploratory/motes.htm

Smart Dust Vision

www.sice.umkc.edu/~leeyu/Udic/Groups/SensorNetworksPresentationFINAL_0909.ppt

Examples of New Platforms for Sensor Deployment

> 2.7 B Cell Phones

New space-based environmental sensors

Military platforms

10,000 surveillance cameras in London

Smart factories & buildings

Examples of Military Sensor Platforms

MILITARYPLATFORM

ROLE SENSOR FUNCTIONS TYPICAL SENSORS

Air-to-Air Combat Detect, track and I/D aircraft(friend-foe and type I/D)

Engage hostile aircraft andverify kill

Multi-mode radar IRST, TV IFF ESM

Ground Attack Search, acquire, I/D hostileground targets (mobile, fixed)

Engage targets: hand-offaircraft-to-weapon sensors

Terrain-following radar Imaging/mapping radar Forward looking IR ESM

Anti-Air Warfare Conduct surveillance formilitary ATC hostile targetdetection, I/D

I/D, track and engage hostileaircraft (CAP/SAM)

Air search radars Fire control radars ESM IRST IFF

Surface, Sub-Surface Warfare

Conduct surface/sub-surfacesurveillance for hostile ship-sub-detection, I/D

Coordinate air, surface, sub-surface engagements

Surface search radar Hull-mounted sonar Towed-array sonar ESM

Examples continued

Representative Observable-Sensor PairingsDetectable/measureable emission characteristics

Representative sensors

Radio Frequency Radar warning receivers; Electronic intelligence; Communications intelligence

Infra-red (IR) emissions & contrast Infra-red imagery

Acoustic/seismic Acoustic sensors; Seismometers; Sonar

Optical contrast TV imagery; Direct view optics; Optical augmentation sensors

Radar cross section (RCS) RADAR; Millimeter Wave Radar

Electro-optical (E-O) E-O sensors; LIDAR

Mechanical/structural vibrations Laser-based sensors; accelerometers

The Frequency Spectrum of Sensors

Representative Sensor Characteristics

SENSOR CHARACTERISTIC DESCRIPTION

Detection Performance Detection characteristics (false alarm rate, detection probabilities and ranges) for acalibrated target characteristic in a given noise background

Spatial/Temporal Resolution Ability to distinguish between two or more targets in space or time

Spatial Coverage Spatial volume covered by the sensor, for scanning sensors ( this may be describedby the instantaneous field-of-view, the scan pattern volume and the total field-of-regard achievable by moving the scan pattern)

Detection/Tracking Modes Search and tracking modes performed: Staring or scanning Single or multiple target tracking Single- or multi-mode (track-while-scan/stare)

Target Revisit Rate Rate at which a given target is revisited by the sensor to perform a samplemeasurement (staring sensors are continuous)

Measurement Accuracy Statistical accuracy of sensor measurements

Measurement Dimensionality Number of measurement variables (range, range rate, and spectral features)between target categories

Hard/Soft Data Reporting Sensor outputs are provided either as hard-decision (threshold) reports or as pre-processed reports with quantitative measures of evidence for possible decisionhypothesis

Detection/Track Reporting Sensor reports each individual target detection or maintains a time-sequencerepresentation (track) of the target’s behavior

Inside a Generic Sensor

Steering Control

Adaptive Control

Heuristic Data

Signal Conditioning

- Translation- Analog/Digital- Digital/Analog- Detection

Signal Processing

- Filtering- Transforms- Thresholding- Storage- Special Algorithms

Information Processing/

Decision Making

- Look-up Tables- Bit Map- Heuristics- Declaration Matrix Map

Output Processing

- Buffering- Data Conversion - Coordinate Transforms- Smoothing- Filtering

ENERGY EMISSION SENSOR ELEMENTS

SENSOR GUIDANCE/CONTROL

image

y(t)

• tasking• control data• tip-off information• environmental data

Sensor Input

Time FunctionEnergy

Emission

Input Energy• intentional• unintentional• jamming

Single Sensor Target Detection and Parameter Estimation

SIGNAL PARAMETRIZATION and

DETECTION

determine a presence of signal and represent via

attribute vectorExamples

• Feature extraction - Peak detection - Shape characterization - Statistic summaries• Detection logic - Hypothesis testing - Thresholding• Tracking/geolocation - Position estimation - Kinematic estimation• Attribute estimation

Image

Time

X(t) Time Series Processing

CANONICALTRANSFORMATION

natural Hilbert space representation of

signals

Examples• Fourier• Wavelet• Gabor• Walsh

etc.

• • •

SIGNAL INFORMATION

ANALYSIS

characterize and analyze a signal

Examples• Time series - State space - ARMA - HOS• Matched filter• Range-Doppler• Ambiguity function

DATABASE

• Target characteristics• Environment/sensor models• Signal models

Sensor Selection Matrix (Steinberg)

Radar• Direct Beam• SAR

IR Imagery• Passive• Augmented

TV Imagery• Passive• Augmented

EW Sensors• RWR• ESM

Acoustic/ Seismic

Direct View Optics• Passive• Augmented

Optical Augmentation Sensors

E-O Tracking• IRSTR• LADAR

SENSORTYPE

DetectionRange

DetectionTime

TargetID

RangeMeasurement

Probability of

Detection

Vulnerability to

Detection

Vulnerability to

CM

RELATIVEPERFORMANCE

Poor

Fair

Good

* Steinberg, DFS-87.

Radar: Radio Detection & Ranging

Basic Measurements:Basic Measurements:• Radar cross-section• Frequency • Time

Derived Measurements:Derived Measurements:• Range• Azimuth• Elevation• Velocity• Target Size/Shape• Signature

State DeterminationState Determination• Position, velocity, identity

– Limited target I/D from radar cross-section signature– Target size/shape may be obtained in limited cases

1. Skolnick, M.I., Ed., Radar Handbook , McGraw-Hill, NY, 1970.

2. Barton, D.L., Modern Radar System Analysis , Artech House, Norwood, MA, 1988.

3. Stein, A., "Bistatic Radar Applications in Passive Systems," Supplement to the January 1990 Journal

of Electronic Defense , Horizon House Microwave, Norwood, Massachusetts.

SPY-1 Phased Array Radar

LIDAR: Light Detection and Ranging

Basic Measurements:Basic Measurements:• Optical intensity• Range• Angular information (direction)

Derived Measurements:Derived Measurements:• Visual signature

State DeterminationState Determination• Location

LIDAR is the visual light equivalent of RADAR.

Cruickshank, J.M., and R.C. Harney, Eds., Laser Radar Technology and Applications ,

SPIE Proceedings, Vol. 663, Quebec City, Canada, June 1986.

ELINT: Electronic Intelligence

ELINTDEFINITION: Electronic intelligence – derived from observations of

radar, other non-communication emitters (e.g., NAVequipment)

TACTICAL ELINT: Type Radar/SEI Operations Timeline Location

TECHNICAL ELINT: Reverse Engineering of Design Level of technology Weaknesses to ECM Parametric data for database development

- Power- Antenna- Modulation

PROBLEMS: Collection Series Separating Signal-of-Interest (SOI) Statistical Analysis

Electronic Intelligence Sensors

Basic Measurements:Basic Measurements:• Amplitude• Frequency • Time

Derived Measurements:Derived Measurements:• Received signal-to-noise ratio• Polarization• Pulse shape• Pulse repetition interval• Radio frequency

State DeterminationState Determination• Identity

– General and specific emitter identity– Analysis yields radar design characteristics

Wiley, R.G., Electronic Intelligence: The Analysis of Radar Signals , Artech House, Norwood, MA, 1988

ESM: Electronic Support Measures

Basic Measurements:Basic Measurements:• RF Intensity/Amplitude• Frequency• Range/Time of Arrival• Angular information (direction)

Derived Measurements:Derived Measurements:• Signal signature

State DeterminationState Determination• Location• Type of emitter

.

EW Design Engineers Handbook 1989/1990 , Supplement to the January 1990 Journal of Electronic

Defense , Horizon House Microwave, Norwood, Massachusetts.

AN/ALR-606 Electronic Support Measures (ESM) Receiver (Northrop Grumman)

Communications Intelligence

Human Skills:• Equipment operator• Linguistics• Gisting

Analysis Tools:• Transforms• Graphics• Statistics• Database

Hardware:• Receivers• Turners• Demodulation• Filters

Tactical IntelligenceTechnical Intelligence

IR: Infrared Warning

Basic Measurements:Basic Measurements:• IR intensity• Detection

Derived Measurements:Derived Measurements:• Direction• Intensity

• Spectral Characteristics

State DeterminationState Determination• Identity• Location

Warning of heat sources approaching.Buser, R.G., and F.B. Warren, Eds., Sensors and Sensor Fusion,

Proceedings of the SPIE , Vol. 782, Orlando, Florida, May 1987.

Word to the wise

Sutton Coldfield Observer,Sutton Coldfield Observer, SUTTON COLDFIELD, ENGLAND: SUTTON COLDFIELD, ENGLAND:

“Even when the helicopter is flying between 800 and 1,000 feet”, say police intelligence, their heat seeking sensors can differentiate between a fleeing suspect and a heat producing device. The officers who rushed to the scene of the robbery at a builders site called for helicopter assistance, and the high tech sensor led them to the door of local, Barry Silvester. Detectives burst into the house to discover that the sensor had led them straight to a steaming hot compost heap in the back garden.

Synthetic Aperture Radar

http://www.nnsa.doe.gov/na-20/synthetic_aperture.shtmlhttp://www.nnsa.doe.gov/na-20/synthetic_aperture.shtml

Basic Measurements:Basic Measurements:• Coherent radar cross-section

Derived Measurements:Derived Measurements:• Target/platform shape• Target size• Aim point (direction)

State DeterminationState Determination• Identity

Enhanced radar resolution via coherent processing ofdata collected by a stable moving antenna.

1. Hovanessian, S.A., Introduction to Synthetic Array and Imaging Radars , Artech House, Norwood,

Massachusetts, 1980.

2. Fitch, J.P., Synthetic Aperture Radar, Springer-Verlag, NY, 1988.

3. Hovanessian, S.A., Introduction to Sensor Systems , Artech House, Norwood, Massachusetts, 1988.

Electro-Optical

1. Blouke, M.N., and D. Pophal, Eds., Optical Sensors and Electronic Photography,

Proceedings of the SPIE , Vol. 1071, January 16-18 1989, Los Angeles, California.

Basic Measurements:Basic Measurements:• Picture elements (pixel)• Picture intensity and color

Derived Measurements:Derived Measurements:• Location• Size• Shape

State DeterminationState Determination• Position• Identity

- Can be used to obtain environmental data

Good for size/shape determination, but requires significant processing time.

Representative Classification Results for FLIR and TV Imagery

TRACKED TRUCK CLUTTER CLUTTER TRACKED

TRUCK CLUTTERTRACKED

FLIR IMAGE: TRUE CLASSES FLIR CLASSIFICATIONS

PIXEL-LEVEL FUSION

TRUCK CLUTTERTRACKED TRACKED

CLUTTER TRACKED

TV IMAGE: TRUE CLASSES

TV CLASSIFICATIONS

FEATURE-LEVEL FUSION

Example: Self-Locating/

Self-Calibrating Acoustic SensorNODE PROTOTYPE

BreadboardProcessor

BatterySensor Platform

Node ComponentsNode Components• Acoustic array

• Seismic/tiltmeter array

• GPS, flux-gate compass/magnetometer

• Temp/humidity, solar, soil sensors

• PC/DSP breadboard processor

• Wireless Ethernet networking

FunctionsFunctions• NLOS acoustic/seismic detection, bearing estimation, localization

• Environmental data fusion for real- time data confidence formulation

• Low-cost research tool for proofing concepts and sensors

Input/OutputInput/Output• Acoustic and seismic waves, environmental data, commands via network is input

• Digital files of detection, confidences, position tracks, I/D, predictions of performance in current environment are outputs

Examples of Emerging Sensors

Every soldier a sensor; 4/29/2004Aviation Week

Use of Insect sensing organs (SPIE May 2003)

Plants as observers and sensors

NASA E-Nose (2004)

Signal Propagation

Both the natural physical environment and hostile attempts to degrade sensor effectiveness via countermeasures present complex problemsin the application of data fusion techniques.

Propagation

True SignalDegraded

Received Signal

AtmosphericNoise

Jamming

Multi-path

Attenuation

Jamming and Deception

Random signals may be added to the environment to decreasethe signal-to-noise ratio (SNR). This is called JAMMING.

Specific signals may be added to the environment to createfalse alarms at the receiver. This is called DECEPTION.

Normal Signal Noise

JAMMING

DECEPTION

Camouflage & Deception in nature

http://www.i-am-bored.com/bored_link.cfm?link_id=31398

Examples of on-line photos of animal camouflage and mimicry

http://www.oceanlight.com/lightbox.php?x=camoflage__fish_behavior__fish__animal

http://rainforests.mongabay.com/0306.htm

http://www.nytimes.com/2008/02/19/science/19camo.html?8br

Sensor Analysis Overview

KEY QUESTIONS:KEY QUESTIONS:• What can/should be sensed?• What sensors are available?• Physics of emissions, transmission,

reflection, scattering, detection• CM/CCM environment?• How does clutter noise, multi-path,

interference affect P(D), P(fa)?• Sensor Performance?• Mission constraints on sensing,

data communication?

PurposePurpose:: Select the best suite of sensors to detect, locate, characterize, and identify critical targets/entities

Key Factors:Key Factors:• Sensor selection/availability• Accuracy vs timeliness vs computer resources• Extent to which sensors are Smart• Scheduling, utilization of sensors• Contextual sensor performance

- Environmental effects- Countermeasures

• Sensor control Parameters• Stealth requirements

Analysis Tools and Techniques:Analysis Tools and Techniques:• Static threat vs observability analysis• Engagement timeline analysis• Observation modeling

- Observation prediction- Observation statistics

• Covariance error analysis• Monte Carlo simulation

• Emerging network centric operations– Ubiquitous connectivity– Soldier and civilian hosted sensors– Emerging “every soldier/civilian a sensor” concept

• New types of hybrid “hard/soft” sensors– Sensors carried/worn by soldiers/civilians– Sensors that monitor the condition of people (e.g., using the human body as a

sensor for complex chem/bio phenomena and monitoring the body’s responses)– Self-reporting via soldier as “continuous commentator” – Use of animals and plants as sensors– Emergent social phenomena – mining soldiers chat & blogs for emerging insights

and unconscious insights

• Need for research in– Development of a framework for fusing report-level data from human observers– Mapping and modeling the data flow for fusion involving humans as “soft

sensors” combined with conventional sensors

Mixed input environmentsTraditional and new “hard” and “soft” sensing

General Issues

• What is the general framework for human observing/reporting?

– What is the role of the human as both provider and consumer of information?– What is the process for human observations (e.g., do humans provide

information on a volunteer or ad hoc basis; can or should humans be tasked; is/should there be modulation/filtering/censuring of observations;

– Should human observers be characterized (e.g., self-report of expertise; system “grading” of human observers, etc) – consider Accuweather example

– What is the role of indirect reporting (e.g., blogging; news reports, chat, etc)

• How should knowledge and reports be solicited?– How should knowledge be solicited (e.g., via structured entry forms; via

prompted questions; etc.– What language/structure should be used (free-form text, specialized & restricted

language;– How should uncertainty be represented (e.g., via fuzzy terms, confidence factors,

subjective probabilities– How should 2nd order uncertainty be represented

Issues continued

• What is the framework for human reporting uncertainty representation/fusion with traditional sensors?

• How does “soft” data and web mined data flow into the traditional JDL model components? – Soft data provides contextual information (e.g., human judged

relationships) & input directly into higher levels of JDL processing

– Issues of legacy, out-of-sequence reports, significant differences in time scales require more sophisticated data management functions

Topic 4 Assignments• Preview the on-line topic 4 materials• Writing Assignment 4: Write a 2-page paper describing the

challenges and issues of searching for the Loch Ness Monster (see on-line questions) and web resources.

• Discussion 2: On-line discussion of the concept of camouflage in nature (and for human activities); what strategies are used to affect “observability”; can you find interesting examples on the web?

• Team Assignment (T-3): For your selected application; provide a summary table of the anticipated information sources and sensors (e.g., provide a list of sensors and/or sources; for each sensor or source identify the type of data (what is observed or reported), the data rate (e.g., reports, signals, images per second, minute or day), any information about the data format, and information about the characteristics of the sensor/source (e.g., reliability, capability of observing in various environments, etc.)

Data Fusion Tip of the Week

There’s no substitute for a good sensor that is appropriate to the situation or threat of interest: No amount of fusion of multiple poor sensors can substitute for an effective sensor!