wpafb 2010

TNO Defence, Security and Safety

INVIS Integrated Night Vision Surveillance and Observation System

Lex Toet & Sjaak Kriekaard

TNO Defence, Security and Safety2

Overview

Rationale of image fusion

Color the Night: real-time sample-based natural color fusion

Previous implementations:

– Gecko = Visual + NIR

– Viper = II + LWIR

Latest development : Triclobs = Visual + NIR + LWIR

INVIS system = Triclobs + Synthetic Scene Generator +

Image Enhancement Capabilities

Example nighttime movies

• Intensified imagery: see in the dark, context but no targetsIntensified imagery: see in the dark, context but no targets

• Thermal imagery: see through smoke, targets but no contextThermal imagery: see through smoke, targets but no context

• Fused imagery: Fused imagery: - increased information content - increased information content - best of both worlds: both targets and context- best of both worlds: both targets and context - all weather capability- all weather capability - increased system robustness- increased system robustness - decreased false alarm rate- decreased false alarm rate

Rationale of Image Fusion :

Color-the-night

• deduces a natural color mapping from a multiband night sensor image + daytime reference image

• color mapping can be applied to other multiband images with similar content, resulting in a multiband nighttime image with natural daytime colors

• Method is:• Fast, easy to implement in real-time applications• Results in stable object colors (invariant under panning,

zooming etc).

• Natural colors lead to:• Easier and better interpretation of scene• Faster scene recognition• Better discimination & recognition of materials and objects

TNO Defence, Security and Safety5

Sample Based Color Transfer

color photograph visual NIR

vis + NIR recolored

TNO Defence, Security and Safety6

Color the Night: Applying Daytime Colors to Nighttime Imagery

ThermalImagery

Intensified Imagery

Color Reference Image

Resulting Imagery

TNO Defence, Security and Safety7

Visual Thermal

Grayscalefused

Color fusedimage

TNO Defence, Security and Safety8

Gecko: portable real time Visual + NIR fusion

TNO Defence, Security and Safety9

Visual + NIR Fusion: Gecko principle

TNO Defence, Security and Safety10

Gecko: Visual + NIR

< 700 nm

> 700 nm

TNO Defence, Security and Safety11

Visual Thermal

RG Gecko

NVG photo

Gecko system

TNO Defence, Security and Safety12

Viper: Portable Real time Visual + Thermal Fusion

eyepinhole

TNO Defence, Security and Safety13

Viper system: principle

TNO Defence, Security and Safety14

Image Intensifier Thermal camera

Intevac NightVista E1100 FLIR A10

Viper: portable real time Visual + Thermal fusion

TNO Defence, Security and Safety15

Viper: combined Visual + Thermal Imager

TNO Defence, Security and Safety16

Viper: Visual + Thermal

TNO Defence, Security and Safety17

Visual Thermal

RG Viper

Photo

Viper system

Gecko bands

NV

G >

700nm

NV

G <

700nm

Daytime image

Gecko image

Th

emal

NV

G

Viper bandsViper image

Daytime image

The INVIS Integrated Night Vision Surveillance and Observation System

Dynamic fusion of Visual + NIR + LWIR

Real-time natural color mapping

Real-time image enhancement

Synthetic image generation

TNO Defence, Security and Safety21

INVIS

sensor suitelayout

TNO Defence, Security and Safety22

Photonis Intensified Camera Unit (ICU PP3000L)

XenICs Gobi 384 infrared microbolometer

INVIS sensors

TNO Defence, Security and Safety23

Viewing window

displays

LWIR

Visual

NIR GPS - compass

Data storage

Network connection

TNO Defence, Security and Safety25

INVIS

mounted on all terrainmobileplatform

TNO Defence, Security and Safety26

Visual NIR LWIR

RGB Photo

Color the Night DNR LACE

Color the Night

Visual NIR LWIR

TNO Defence, Security and Safety28

Color the Night

Visual NIR LWIR

TNO Defence, Security and Safety29

Color the Night

Visual NIR LWIR

TNO Defence, Security and Safety30

Color the Night

Visual NIR LWIR

TNO Defence, Security and Safety31

Color the Night

Visual NIR LWIR

TNO Defence, Security and Safety32

Color the Night

Visual NIR LWIR

TNO Defence, Security and Safety33

Color the Night

Visual NIR LWIR

TNO Defence, Security and Safety34

Color the Night

Visual NIR LWIR

Image enhancement techniques

• Temporal noise reduction• Scene-based non-uniformity correction• Super-resolution reconstruction• Local adaptive contrast enhancement• Color-the-Night image fusion

SBNUC DSR

Color-the-Night

LACE

DSR

DSR

Resizing the TIR imageTIR

VISUAL

NIR

Noise reductionby Dynamic Super Resolution

• Advantages• Decreased noise (improve SNR ratio)• Increased detection range• Increased situational awareness

Fixed pattern noise reductionby Scene-Based Non-Uniformity Correction

• Advantages

• Low maintenance (hands-off operation)

• Clean pictures (performance maximized)

• Reduced sensor size / weight / cost (simpler design)

Resolution enhancementby Dynamic Super Resolution

• Advantages• Increased sharpness • Increased situational awareness• Increased recognition and identification

Local Adaptive Contrast Enhancement (LACE)• Advantages

• Increased situational awareness• Increased recognition and identification

in the total dynamic range

Noise reduction

NIR VIS LWIR

Noise reduction for I2

NormalDSR1DSR1+low LACE

NIR image

Noise Reduction and LACE on I2 images

Noise Reduction and LACE on LWIR images

weak LACE vs strong LACE

Original images, RGB fusion Original images, Color-the-Night fusion

Enhanced images, RGB fusion Enhanced images, Color-the-Night fusion

No enhancement DSR1 + SBNUC

DSR1 + SBNUC; LACE before CtN DSR1 + SBNUC; LACE after CtN

Color-the-Night & LACE

No LACE Low LACE before CtN Low LACE after CtN

High LACE before CtN High LACE after CtN

Conclusions on Image Enhancement :

• Temporal Noise reduction works for all imagery• Scene Based Non Uniformity Correction (SBNUC) is useful for LWIR

• Super-resolution reconstruction has added value for LWIR• images are resolution limited

• Super-resolution reconstruction has limited value for I2

• Images are noise limited

• Enhancement algorithms need accurate motion estimation• works for small moving objects• not for scenery dominated by smoke or large moving objects

• LACE can be applied on all imagery• noisy imagery requires noise reduction

Relation between Image Enhancement and Image Fusion

• Enhanced images can be used as input for Color-the-Night fusion

• The results have similar appearance and a significantly improved SNR

• LACE applied after tone mapping keeps the appearance similar

• LACE applied before tone mapping may enhance more details

• but is more expensive and may disturb color mapping!

• Effect of LACE applied before fusion is reduced by color mapping

TNO Defence, Security and Safety50

Synthetic Scene Generation

TNO Defence, Security and Safety51

TNO Defence, Security and Safety52

Corresponding LWIR and synthetic imagesin 1st person perspective

LWIR Synthetic

TNO Defence, Security and Safety53

Corresponding LWIR and synthetic imagesin 3rd person perspective

LWIR Synthetic

TNO Defence, Security and Safety54

LWIR synthetic synthetic

Switching walls on --- off

TNO Defence, Security and Safety55

LWIR synthetic synthetic

Switching vegetation on --- off

TNO Defence, Security and Safety56

LWIR Material view

TNO Defence, Security and Safety57

Visual NIR LWIR

RGB

TNO Defence, Security and Safety58

RGB Photo

Color the Night DNR LACE

TNO Defence, Security and Safety59

RGB Synthetic scene

Color the Night DNR LACE

TNO Defence, Security and Safety60

Visual NIR LWIR

RGB

TNO Defence, Security and Safety61

Photo

Color the Night DNR LACE

RGB

TNO Defence, Security and Safety62

Synthetic scene

Color the Night DNR LACE

RGB

June 2010: Data Collection Experiment on Military Base in the Netherlands

Triclobs on scootmobile

The Targets

Multispectral Pinkus Target

Real-time Scene Display

Visual NIR

LWIR

The End