dsc image processing pipeline - github pagesmatthiasbock.github.io/hardware/rollei...
TRANSCRIPT
1 1
Confidential
DSC Image Processing Pipeline
NOVATEK Microelectronics Corp.
IM
2
Confidential NT96432/96200 Block Diagram
CPU
JPEG EngineImagePipeline
SensorInterface
LCDController
OSD
Scaler
Memory CardController
USB 2.0
UART
Static MemoryController
GPIO / ADC /PWM Boot ROM
NOR SerialFlash
TFT LCD
PC
SD / MMC/NAND Flash
Button /Buzzer /
Battery Detect
SPI
SIF / I2C
DRAMController
SDRAM
CMOS Sensor
3
ConfidentialImage Processing Inside DSC Controller
R GG B
R GG B
R GG B
X GG B
RGRG
What’s we have fromimage sensor
Colorinterpolation
B B B B BG G G G G
R R R R RR R R R RR R R R RR R R R R
Colorcorrection
B B B B BG G G G G
R R R R RR R R R RR R R R RR R R R R
Edgeenhancement
B B B B BG G G G G
R R R R RR R R R RR R R R RR R R R R
Gammacorrection
B B B B BG G G G G
R R R R RR R R R RR R R R RR R R R R
Colortransform
Cr Cr Cr Cr CrCb Cb Cb Cb Cb
Y Y Y Y YY Y Y Y YY Y Y Y YY Y Y Y Y
What’s we feed toimage compression
Only one color componentper pixel position
Now, we have three colorsfor each pixel
Right color is reproduced,but image is still blur
Edge sharpening is usedto recover edge strength
To fit human perception &display characteristics
Compression needs YCbCrimage input
4
Confidential Image Processing Samples
5
Confidential Image Pipeline
DRAM Storage USB
SensorInterface (SIE)
ImageProcessing
Engine (IPE)
ImageManipulationEngine (IME)
Image DisplayInterface (IDE)
Sensor/TG/AFE JPEG Codec
6
Confidential Sensor Interface Engine (SIE)
OB Detection
Color Gain & Offset Line Cull Gamma
Lens ShadingCorrection
Defect PixelCorrection
Horizontal Down SampleActive Crop
Histogram
Form Dram orAFE
To Dram orIIPE
Color Accumulation
7
Confidential SIE Feature List Optical Black Detection & Black Level Clamping
Remove sensor dark current accumulation Lens Shading Compensation Blemish Concealment Line Cull
Raw image vertical subsampling Raw Image SubSampling
Raw image horizontal subsampling Color Offset & Gain
AWB color gain Gamma LUT 1 Color Accumulation
AE & AWB analysis Histogram Accumulation Instant color or luminance distribution
8
Confidential Color Accumulation(CA_ST_X, CA_ST_Y)
CA_WHSZ
CA_WVSZ
(0, 0)
CA_RSLT0
(W-1, H-1)
CA_RSLT1 CA_RSLT2 CA_RSLT3 CA_RSLT4 CA_RSLT5 CA_RSLT6 CA_RSLT7
CA_RSLT8 CA_RSLT9 CA_RSLT10 CA_RSLT11 CA_RSLT12 CA_RSLT13 CA_RSLT14 CA_RSLT15
CA_RSLT16 CA_RSLT17 CA_RSLT18 CA_RSLT19 CA_RSLT20 CA_RSLT21 CA_RSLT22 CA_RSLT23
CA_RSLT24 CA_RSLT25 CA_RSLT26 CA_RSLT27 CA_RSLT28 CA_RSLT29 CA_RSLT30 CA_RSLT31
CA_RSLT32 CA_RSLT33 CA_RSLT34 CA_RSLT35 CA_RSLT36 CA_RSLT37 CA_RSLT38 CA_RSLT39
CA_RSLT40 CA_RSLT41 CA_RSLT42 CA_RSLT43 CA_RSLT44 CA_RSLT45 CA_RSLT46 CA_RSLT47
CA_RSLT48 CA_RSLT49 CA_RSLT50 CA_RSLT51 CA_RSLT52 CA_RSLT53 CA_RSLT54 CA_RSLT55
CA_RSLT56 CA_RSLT57 CA_RSLT58 CA_RSLT59 CA_RSLT60 CA_RSLT61 CA_RSLT62 CA_RSLT63
CA_WHSZ CA_WHSZ CA_WHSZ CA_WHSZ CA_WHSZ CA_WHSZ CA_WHSZ
CA_WVSZ
CA_WVSZ
CA_WVSZ
CA_WVSZ
CA_WVSZ
CA_WVSZ
CA_WVSZ
• NT96431 supports 8x8 = 64 color accumulation windowsto facilitate AE and AWB analysis
• Programmable starting point and window size
9
Confidential Histogram Accumulation
GR
BGB
R
Y
HST_MODE(R/G/B/Y)
CA_SMP
(CA_STX, CA_STY)(2^(CA_WHSZ+1)) * 8
(2^(CA_WVSZ+1)) * 8
HST_X, HST_Y, HST_DX, HST_DY,HST_NML, HST_RSLT[0~31]
32 bins histogram assists AE and imagedistribution display on the fly
10
Confidential AE/AWB
Color Analysis
Calculateluminance & AE
Algorithm
Color Gain & OffsetAdjustment
Input image Output image
Windowed coloraverage
Exposure time & AFE gainadjustment and/or turn on
LED or strobe light
Sensor & AFE
AWB Algorithm
Color offset & gainadjustment
11
Confidential AE/AWB Flow
Sensor/TG/AFE
Color Gain IPE IMEGamma
Awb/ae data
Awb algorithm
Ae dataAe algorithm
Pre-processing
SIE
12
Confidential Lens Shading Compensation
Programmable lens shading profile, can fit any convexlens shading defect
13
Confidential Color Shading Correction: 2 Approaches
1. Circular & Linear Color Shading Compensation Embedded in image pipeline No penalty for shot-to-shot time Useful for regular color shading
2. Blockwise Color Shading Correction Need mask picture Slight delay for shot-to-shot time Only support still capture Useful for irregular color shading
NT96432 Provides 2 HW Approaches for Color Shading Correction!!
14
ConfidentialThree Remedies for Defect Pixels
1. SIE direct defect pixel concealment (HW)• Need pixel address from DRAM• Embedded in pipeline
2. Noise reduction (HW)• Tunable edge sensitive 2D median filter• Embedded in pipeline
3. Software defect pixel concealment• 20us/pixel, for still image defect pixel• Need pixel address stored in DRAM
DSP provides three remedies fordefect pixel concealment
15
ConfidentialCCD Input Format at Different Mode
R Gr BGb R Gr BGb
R Gr BGb R Gr BGb
R Gr BGb R Gr BGb
R Gr BGb R Gr BGb
R Gr
BGb
R Gr
BGb
R Gr
BGb
R Gr
BGb
R Gr
BGb
R Gr
BGb
R Gr
BGb
R Gr
BGb
R Gr BGb R Gr BGb
R Gr BGb R Gr BGb
R Gr BGb R Gr BGb
R Gr BGb R Gr BGb
Xn,Yn
Xn+1,Yn
Xn,Yn+Ysmp
Xn+1,Yn+Ysmp
Preview Mode
Movie Mode
R Gr
BGb
R Gr
BGb
R Gr
BGb
R Gr
BGb
R Gr
BGb
R Gr
BGb
R Gr
BGb
R Gr
BGb
Still Mode
Xn+4,Yn
Xn+5,Yn
Xn+4,Yn+Ysmp
Xn+5,Yn+Ysmp
NT96432 supports Sony & Sharp movie CCDPanasonic movie CCD will be supported in our next chip
16
Confidential Image Processing Engine
Color Gain &Offset
Color Interpolation
Noise Reduction Gamma
Color Correction
Edge Enhancement
VariationAccumulation
Color Control
Color SpaceTransform
Form Dram orSIE
To Dram orIME
Edge Extraction
17
ConfidentialAF Windows (Variation Accumulation)
(VACC_STX, VACC_STY)
VACC_HS
VACC_VSZ
(0, 0)
VACC_HS VACC_HS
VACC_VSZ
VACC_VSZ
VACC0 VACC1 VACC2
VACC3 VACC4 VACC5
VACC6 VACC7 VACC8
(W-1, H-1)
• NT96431 supports 3x3 = 9 focusing windowsto facilitate AF analysis
• Programmable starting point and window size
18
Confidential Noise Reduction
Edge sensitive 2D filtering
Programmable filtering control
Usable for preview, still capture & video clip
Capable of canceling spot noise under low light condition
19
Confidential Color Effect
Negative Sepia Solarize
20
ConfidentialProprietary Color Control Technology
Increase RED saturation Increase Green hue
Increase BLUE intensity
H = 0
H = 60 H = 120
H = 180
H = 240H = 300
21
Confidential Color Control Image Samples
Original Green tone Skin tone enhanced
22
Confidential DFS Image Comparison
Exposure time = 8 sec
DSP DFS OFF DSP DFS ON
DSP DFS removes FPN and spot noise!
23
ConfidentialNonlinear False Color Reduction(1)
• There are nonlinear false color in high light area, it is very bad when it occurs in our skin. We can reduce it, and get a satisfied image.
0%50%
100%
24
ConfidentialNonlinear False Color Reduction (2)
• Improve the tone response of each color channel, and get great detail.
0%
50%
100%
25
ConfidentialImage Manipulation Engine (IME)
• Dual scaler outputs for single input image• Y channel can be bicubic interpolation• A forward/inverse color space transform is designed
for RGB YCbCr transform• Input and output ping pong buffering mechanism
From DRAMor IPE
H-LPF
H-LPF
H&V Bi-cubic/Bilinear
Interpolation
H&V BilinearInterpolation
Inverse/Forward Color
SpaceTransform
To DRAM
To DRAM
26
Confidential
Image Processing Flow
27
Confidential Image Display Engine (IDE)
• 2 video & 2 OSD buffers to support various display needs• Support picture-in-picture• Support dithering for low bit-depth LCD panel
OSD2
OSD1
VDO2
VDO1
Blending
Gamma/Contrast/
Saturation/Brightness
Dithering LCD/TG
CCIR-601
CCIR-656
TV-Endoder
LCD
From DRAM
From DRAM
From DRAM
From DRAM
28
Confidential Preview
DRAM Storage USB
SensorInterface (SIE)
ImageProcessing
Engine (IPE)
ImageManipulationEngine (IME)
Image DisplayInterface (IDE)
Sensor/TG/AFE JPEG Codec
29
Confidential Still Capture
DRAM Storage USB
SensorInterface (SIE)
ImageProcessing
Engine (IPE)
ImageManipulationEngine (IME)
Image DisplayInterface (IDE)
Sensor/TG/AFE JPEG Codec
30
Confidential Video Recording
DRAM Storage USB
SensorInterface (SIE)
ImageProcessing
Engine (IPE)
ImageManipulationEngine (IME)
Image DisplayInterface (IDE)
Sensor/TG/AFE JPEG Codec
Audio Codec
31
Confidential Still & Video Playback
DRAM Storage USB
SensorInterface (SIE)
ImageProcessing
Engine (IPE)
ImageManipulationEngine (IME)
Image DisplayInterface (IDE)
Sensor/TG/AFE JPEG Codec
Audio Codec
32
Confidential Dark Frame Subtraction
DSP supports HW dark frame subtraction!
DRAM Storage USB
SensorInterface (SIE)
ImageProcessing
Engine (IPE)
ImageManipulationEngine (IME)
Image DisplayInterface (IDE)
Sensor/TG/AFE JPEG Codec
DMA Operation
Dark frame
Normal frame
• Dark frame subtraction(DFS) can remove sensorfixed pattern noise
• DFS can not remove noise completely withoutfurther accompanied processing