dsc image processing pipeline - github pagesmatthiasbock.github.io/hardware/rollei...

1 1

Confidential

DSC Image Processing Pipeline

NOVATEK Microelectronics Corp.

IM

2

Confidential NT96432/96200 Block Diagram

CPU

JPEG EngineImagePipeline

SensorInterface

LCDController

OSD

Scaler

Memory CardController

USB 2.0

UART

Static MemoryController

GPIO / ADC /PWM Boot ROM

NOR SerialFlash

TFT LCD

PC

SD / MMC/NAND Flash

Button /Buzzer /

Battery Detect

SPI

SIF / I2C

DRAMController

SDRAM

CMOS Sensor

3

ConfidentialImage Processing Inside DSC Controller

R GG B

R GG B

R GG B

X GG B

RGRG

What’s we have fromimage sensor

Colorinterpolation

B B B B BG G G G G

R R R R RR R R R RR R R R RR R R R R

Colorcorrection

B B B B BG G G G G

R R R R RR R R R RR R R R RR R R R R

Edgeenhancement

B B B B BG G G G G

R R R R RR R R R RR R R R RR R R R R

Gammacorrection

B B B B BG G G G G

R R R R RR R R R RR R R R RR R R R R

Colortransform

Cr Cr Cr Cr CrCb Cb Cb Cb Cb

Y Y Y Y YY Y Y Y YY Y Y Y YY Y Y Y Y

What’s we feed toimage compression

Only one color componentper pixel position

Now, we have three colorsfor each pixel

Right color is reproduced,but image is still blur

Edge sharpening is usedto recover edge strength

To fit human perception &display characteristics

Compression needs YCbCrimage input

4

Confidential Image Processing Samples

5

Confidential Image Pipeline

DRAM Storage USB

SensorInterface (SIE)

ImageProcessing

Engine (IPE)

ImageManipulationEngine (IME)

Image DisplayInterface (IDE)

Sensor/TG/AFE JPEG Codec

6

Confidential Sensor Interface Engine (SIE)

OB Detection

Color Gain & Offset Line Cull Gamma

Lens ShadingCorrection

Defect PixelCorrection

Horizontal Down SampleActive Crop

Histogram

Form Dram orAFE

To Dram orIIPE

Color Accumulation

7

Confidential SIE Feature List Optical Black Detection & Black Level Clamping

Remove sensor dark current accumulation Lens Shading Compensation Blemish Concealment Line Cull

Raw image vertical subsampling Raw Image SubSampling

Raw image horizontal subsampling Color Offset & Gain

AWB color gain Gamma LUT 1 Color Accumulation

AE & AWB analysis Histogram Accumulation Instant color or luminance distribution

8

Confidential Color Accumulation(CA_ST_X, CA_ST_Y)

CA_WHSZ

CA_WVSZ

(0, 0)

CA_RSLT0

(W-1, H-1)

CA_RSLT1 CA_RSLT2 CA_RSLT3 CA_RSLT4 CA_RSLT5 CA_RSLT6 CA_RSLT7

CA_RSLT8 CA_RSLT9 CA_RSLT10 CA_RSLT11 CA_RSLT12 CA_RSLT13 CA_RSLT14 CA_RSLT15

CA_RSLT16 CA_RSLT17 CA_RSLT18 CA_RSLT19 CA_RSLT20 CA_RSLT21 CA_RSLT22 CA_RSLT23

CA_RSLT24 CA_RSLT25 CA_RSLT26 CA_RSLT27 CA_RSLT28 CA_RSLT29 CA_RSLT30 CA_RSLT31

CA_RSLT32 CA_RSLT33 CA_RSLT34 CA_RSLT35 CA_RSLT36 CA_RSLT37 CA_RSLT38 CA_RSLT39

CA_RSLT40 CA_RSLT41 CA_RSLT42 CA_RSLT43 CA_RSLT44 CA_RSLT45 CA_RSLT46 CA_RSLT47

CA_RSLT48 CA_RSLT49 CA_RSLT50 CA_RSLT51 CA_RSLT52 CA_RSLT53 CA_RSLT54 CA_RSLT55

CA_RSLT56 CA_RSLT57 CA_RSLT58 CA_RSLT59 CA_RSLT60 CA_RSLT61 CA_RSLT62 CA_RSLT63

CA_WHSZ CA_WHSZ CA_WHSZ CA_WHSZ CA_WHSZ CA_WHSZ CA_WHSZ

CA_WVSZ

CA_WVSZ

CA_WVSZ

CA_WVSZ

CA_WVSZ

CA_WVSZ

CA_WVSZ

• NT96431 supports 8x8 = 64 color accumulation windowsto facilitate AE and AWB analysis

• Programmable starting point and window size

9

Confidential Histogram Accumulation

GR

BGB

R

Y

HST_MODE(R/G/B/Y)

CA_SMP

(CA_STX, CA_STY)(2^(CA_WHSZ+1)) * 8

(2^(CA_WVSZ+1)) * 8

HST_X, HST_Y, HST_DX, HST_DY,HST_NML, HST_RSLT[0~31]

32 bins histogram assists AE and imagedistribution display on the fly

10

Confidential AE/AWB

Color Analysis

Calculateluminance & AE

Algorithm

Color Gain & OffsetAdjustment

Input image Output image

Windowed coloraverage

Exposure time & AFE gainadjustment and/or turn on

LED or strobe light

Sensor & AFE

AWB Algorithm

Color offset & gainadjustment

11

Confidential AE/AWB Flow

Sensor/TG/AFE

Color Gain IPE IMEGamma

Awb/ae data

Awb algorithm

Ae dataAe algorithm

Pre-processing

SIE

12

Confidential Lens Shading Compensation

Programmable lens shading profile, can fit any convexlens shading defect

13

Confidential Color Shading Correction: 2 Approaches

1. Circular & Linear Color Shading Compensation Embedded in image pipeline No penalty for shot-to-shot time Useful for regular color shading

2. Blockwise Color Shading Correction Need mask picture Slight delay for shot-to-shot time Only support still capture Useful for irregular color shading

NT96432 Provides 2 HW Approaches for Color Shading Correction!!

14

ConfidentialThree Remedies for Defect Pixels

1. SIE direct defect pixel concealment (HW)• Need pixel address from DRAM• Embedded in pipeline

2. Noise reduction (HW)• Tunable edge sensitive 2D median filter• Embedded in pipeline

3. Software defect pixel concealment• 20us/pixel, for still image defect pixel• Need pixel address stored in DRAM

DSP provides three remedies fordefect pixel concealment

15

ConfidentialCCD Input Format at Different Mode

R Gr BGb R Gr BGb

R Gr BGb R Gr BGb

R Gr BGb R Gr BGb

R Gr BGb R Gr BGb

R Gr

BGb

R Gr

BGb

R Gr

BGb

R Gr

BGb

R Gr

BGb

R Gr

BGb

R Gr

BGb

R Gr

BGb

R Gr BGb R Gr BGb

R Gr BGb R Gr BGb

R Gr BGb R Gr BGb

R Gr BGb R Gr BGb

Xn,Yn

Xn+1,Yn

Xn,Yn+Ysmp

Xn+1,Yn+Ysmp

Preview Mode

Movie Mode

R Gr

BGb

R Gr

BGb

R Gr

BGb

R Gr

BGb

R Gr

BGb

R Gr

BGb

R Gr

BGb

R Gr

BGb

Still Mode

Xn+4,Yn

Xn+5,Yn

Xn+4,Yn+Ysmp

Xn+5,Yn+Ysmp

NT96432 supports Sony & Sharp movie CCDPanasonic movie CCD will be supported in our next chip

16

Confidential Image Processing Engine

Color Gain &Offset

Color Interpolation

Noise Reduction Gamma

Color Correction

Edge Enhancement

VariationAccumulation

Color Control

Color SpaceTransform

Form Dram orSIE

To Dram orIME

Edge Extraction

17

ConfidentialAF Windows (Variation Accumulation)

(VACC_STX, VACC_STY)

VACC_HS

VACC_VSZ

(0, 0)

VACC_HS VACC_HS

VACC_VSZ

VACC_VSZ

VACC0 VACC1 VACC2

VACC3 VACC4 VACC5

VACC6 VACC7 VACC8

(W-1, H-1)

• NT96431 supports 3x3 = 9 focusing windowsto facilitate AF analysis

• Programmable starting point and window size

18

Confidential Noise Reduction

Edge sensitive 2D filtering

Programmable filtering control

Usable for preview, still capture & video clip

Capable of canceling spot noise under low light condition

19

Confidential Color Effect

Negative Sepia Solarize

20

ConfidentialProprietary Color Control Technology

Increase RED saturation Increase Green hue

Increase BLUE intensity

H = 0

H = 60 H = 120

H = 180

H = 240H = 300

21

Confidential Color Control Image Samples

Original Green tone Skin tone enhanced

22

Confidential DFS Image Comparison

Exposure time = 8 sec

DSP DFS OFF DSP DFS ON

DSP DFS removes FPN and spot noise!

23

ConfidentialNonlinear False Color Reduction(1)

• There are nonlinear false color in high light area, it is very bad when it occurs in our skin. We can reduce it, and get a satisfied image.

0%50%

100%

24

ConfidentialNonlinear False Color Reduction (2)

• Improve the tone response of each color channel, and get great detail.

0%

50%

100%

25

ConfidentialImage Manipulation Engine (IME)

• Dual scaler outputs for single input image• Y channel can be bicubic interpolation• A forward/inverse color space transform is designed

for RGB YCbCr transform• Input and output ping pong buffering mechanism

From DRAMor IPE

H-LPF

H-LPF

H&V Bi-cubic/Bilinear

Interpolation

H&V BilinearInterpolation

Inverse/Forward Color

SpaceTransform

To DRAM

To DRAM

26

Confidential

Image Processing Flow

27

Confidential Image Display Engine (IDE)

• 2 video & 2 OSD buffers to support various display needs• Support picture-in-picture• Support dithering for low bit-depth LCD panel

OSD2

OSD1

VDO2

VDO1

Blending

Gamma/Contrast/

Saturation/Brightness

Dithering LCD/TG

CCIR-601

CCIR-656

TV-Endoder

LCD

From DRAM

From DRAM

From DRAM

From DRAM

28

Confidential Preview

DRAM Storage USB

SensorInterface (SIE)

ImageProcessing

Engine (IPE)

ImageManipulationEngine (IME)

Image DisplayInterface (IDE)

Sensor/TG/AFE JPEG Codec

29

Confidential Still Capture

DRAM Storage USB

SensorInterface (SIE)

ImageProcessing

Engine (IPE)

ImageManipulationEngine (IME)

Image DisplayInterface (IDE)

Sensor/TG/AFE JPEG Codec

30

Confidential Video Recording

DRAM Storage USB

SensorInterface (SIE)

ImageProcessing

Engine (IPE)

ImageManipulationEngine (IME)

Image DisplayInterface (IDE)

Sensor/TG/AFE JPEG Codec

Audio Codec

31

Confidential Still & Video Playback

DRAM Storage USB

SensorInterface (SIE)

ImageProcessing

Engine (IPE)

ImageManipulationEngine (IME)

Image DisplayInterface (IDE)

Sensor/TG/AFE JPEG Codec

Audio Codec

32

Confidential Dark Frame Subtraction

DSP supports HW dark frame subtraction!

DRAM Storage USB

SensorInterface (SIE)

ImageProcessing

Engine (IPE)

ImageManipulationEngine (IME)

Image DisplayInterface (IDE)

Sensor/TG/AFE JPEG Codec

DMA Operation

Dark frame

Normal frame

• Dark frame subtraction(DFS) can remove sensorfixed pattern noise

• DFS can not remove noise completely withoutfurther accompanied processing