Distortion CorrectionECE 6276 Project Review

Team 5:Basit MemonFoti KacaniJason HaedtJin Joo LeePeter Karasev

2 ECE 6276 Final Project Team 5

7/14/2009

Outline

• Motivation

• Components

• Component Optimization

• Current Results

• Plans for Catapult C

• Schedule

3 ECE 6276 Final Project Team 5

7/14/2009

Objective

Given a distorted image with known size and known lens distortion parameter, generate an undistorted image.

4 ECE 6276 Final Project Team 5

7/14/2009

Motivation – Why?

• The formation of undistorted images can be described by a series of matrix multiplications

• Distortion makes it very difficult to calibrate a camera to measure geometry (depth, size, orientation, etc)

• Many applications in image processing and computer vision like structure estimation, image mosaicing, and ultimately vision-based control.

5 ECE 6276 Final Project Team 5

7/14/2009

Motivation contd..

Application: Measure motion and geometryProblem: Known geometry in the scene is warped, relationship between 3D and 2D points is nonlinear. Solution: Undo the distortion, so x2D = A * X3D

6 ECE 6276 Final Project Team 5

7/14/2009

Literature Review (I)

K.T. Gribbon, C.T. Johnston, and D.G. Bailey, “A Real-time FPGA Implementation of a Barrel Distortion Correction Algorithm with Bilinear Interpolation”

•Focus on reducing hardware complexity.

•Uses LUTs to store mapping data.

•No quantitative results provided.

•Logic resource utilization on RC-100 is 51%

7 ECE 6276 Final Project Team 5

7/14/2009

Literature Review (II)

Qiang, L.; Allinson, N.M.;” FPGA Implementation of Pipelined Architecture for Optical Imaging Distortion Correction”, Signal Processing Systems Design and Implementation, 2006. SIPS '06.

•Same algorithm as previous one

•Implementation on a Xilinx FPGA XCS3 1000-4 uses 75% of the hardware multipliers.

•Residual error of the undistorted image was 1.5% of the distorted image.

8 ECE 6276 Final Project Team 5

7/14/2009

Literature Review (III)

Hany Farid & Alin C. Popescu, “Blind Removal of Lens Distortion”, Journal of the Optical Society of America 2001 

•For removal of distortion in absence of any calibration

data.•Uses polyspectral analysis to detect higher order correlations in frequency domain which are proportional to the distortion.

•Computationally intensive.

•No quantitative results.•Accuracy is not comparable to those based on known distortion parameters.

9 ECE 6276 Final Project Team 5

7/14/2009

Components to Achieve ObjectiveComponent 1: Matlab forward distortion function

– Verifies correctness of undistortion algorithm

Component 2: Data ordering test bench – Order the C++ input stream from MATLAB generated data

Component 3: Undistortion lookup table or lookup function– Initial prototype in MATLAB

Component 4: Least squares interpolation lookup function– Initial prototype in MATLAB– Compare different techniques such as LUT vs NEAREST NEIGHBOR

Component 5: Verification structure – Compare original C++ result to undisorted image in MATLAB

MATLAB C++ Catapult C MATLAB

10 ECE 6276 Final Project Team 5

7/14/2009

Component Optimization Questions• What size buffers do we need to compare against previous frames?

• For undistortion function can we compute them dynamically or do we need a pre-defined LUT?

• For finding fast least squares / linear system solver, compare speed cost vs. nearest neighbor and effect on output error

11 ECE 6276 Final Project Team 5

7/14/2009

Matlab Demo of Algorithm - Original

12 ECE 6276 Final Project Team 5

7/14/2009

Matlab Demo of Algorithm - Distorted

13 ECE 6276 Final Project Team 5

7/14/2009

Matlab Demo of Algorithm - Recovered

14 ECE 6276 Final Project Team 5

7/14/2009

Design Goals

• Limited Buffers

• 8 bits per sub-pixel (24 bits total)

• Resolution (up to 640x480)

• Concerned with geometry

• Area, Throughput, Latency

•Target a low-cost implementation that handles consumer video application pixel clocks of 165 MHz (Apprx 6 ns cycle time).

15 ECE 6276 Final Project Team 5

7/14/2009

Test Vectors

The Line Test

Original Image

Distorted Image

Recovered Image

16 ECE 6276 Final Project Team 5

7/14/2009

Plans for Catapult C Code

• Use C/C++ and Algorithmic C data types to describe synthesizable hardware• Architectures (type of hardware interface (streaming buffers) )• Constraints (Throughput, area, latency)•RTL generation and verification • Optimizations

-Pipelining-Parallelism-Loop Unrolling-Scheduling-Streaming buffers/ Read & Write

17 ECE 6276 Final Project Team 5

7/14/2009

Project Timeline

Tasks Owner Description Week 1 Week 2 Week 3Algorithm Research ALL Research different papers on barrel distortion

MATLAB Implementation PeterMock up distortion algorithm using MATLAB and use image

processing libraries for ease of use

MATLAB/C++ Testbench Jason Take MATLAB results and convert them into C++ structure to

be used in C++ testbenchC++ Code Porting Jin Joo Port MATLAB implementation of undistortion to C++

Presentation 1 ALL Initial project reportCatapult C Porting Foti,Memon Port to supported CATAPULT datatypes

Catapult C Synthesis Jason See results of CATAPULT algorithmPlans for Optimization Foti, Memon Look at ways to optimize CATAPULT code

Presentation 2 ALLCompleted Catapult Code, Synthesis Results, Steps for

OptimizationCatapult C Optimization Jin Joo Complete optimization suggestions

Catapult C Synthesis of Optimization Jason See results of CATAPULT algorithmDemo ALL Show CATAPULT demo and final HDL

Presentation 3 ALL Final report out

18 ECE 6276 Final Project Team 5

7/14/2009

Project Risks

Risk 1:

Slowness of interpolation plan: the key goal of undistortion is “geometrical accuracy” which hold even if noise is injected. To mitigate this risk we have a forward mapping algorithm that is fast compared to any interpolation method but at the cost of missing pixels near the edges.

Risk 2:

Not enough storage space to keep a lookup table of coordinates. To mitigate this risk we could compute coordinates on the fly at the cost of math operations.

Risk 3:

Cannot find least squares method for FPGA. To mitigate this risk we could do nearest neighbor interpolation.  

19 ECE 6276 Final Project Team 5

7/14/2009

Current Status

•Straight lines are recovered with minimal injection of missing points or noise through our algorithm.

•Add more quantitative results on the geometrical accuracy of recovered images than presented in previous results.

•Generalize the coordinate mapping to make the implementation more robust to varying  distortion models and hence supporting more cameras “on the fly.”

20 ECE 6276 Final Project Team 5

7/14/2009

References

• Yi Ma, Stefano Soatto, et al., “An Invitation to 3-D Vision”

•Richard Hartley, Andrew Zisserman, “Multiple View Geometry in Computer Vision”

•Edward M. Mikhail, James S. Bethel, J. Chris McGlone, “Introduction to Modern Photogrammetry”

21 ECE 6276 Final Project Team 5

7/14/2009

Questions?

?