ece532 final project demo disparity map generation on a fpga using stereoscopic cameras ece532 final...
TRANSCRIPT
![Page 1: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/1.jpg)
ECE532 Final Project Demo
Disparity Map Generation on a FPGA Using Stereoscopic Cameras
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
![Page 2: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/2.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Depth Detection
• Depth information applied in automotive collision avoidance, XBOX Kinect, robot vision
• Input stereoscopic image pair compared to produce disparity map
• Disparity map contains depth information at each pixel location
![Page 3: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/3.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Disparity Map to Depth
• Disparity value is (a – b)
• Given:f – focal length
d – camera distance
• Depth = z+f = fd/(a-b)
![Page 4: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/4.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Disparity Map to Depth
• Goal:
– Produce a disparity map for depth sensing using stereoscopic cameras
– Implement in hardware:• Adaptive Window Sizing• Sum of Absolute Differences Pixel Matching• Disparity Map Generation• Output Disparity Map Filtering
![Page 5: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/5.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Disparity Map Algorithm
Variation Window Sizing• Areas of less pixel variation receives larger
window• Areas of more pixel variations receives smaller
window• Window size information to be used for S.A.D
phase
Variation Window Sizing
S.A.D Pixel Matching
Disparity Map
Generation
Disparity Map Filter
![Page 6: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/6.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Disparity Map Algorithm
Sum of Absolute Differences• Simple cost function for pixel matching between
reference and search frames• Other cost functions such as sum of square
differences
Variation Window Sizing
S.A.D Pixel Matching
Disparity Map
Generation
Disparity Map Filter
![Page 7: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/7.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Disparity Map Algorithm
Disparity Map Generation• Determines disparity value, pixel coordinate
difference of matching pixels in reference and search frames
• Stores in memory as disparity map
Variation Window Sizing
S.A.D Pixel Matching
Disparity Map
Generation
Disparity Map Filter
![Page 8: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/8.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Disparity Map Algorithm
Disparity Map Filter• A filter can be applied to remove outliers
Variation Window Sizing
S.A.D Pixel Matching
Disparity Map
Generation
Disparity Map Filter
![Page 9: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/9.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Disparity Map Algorithm
![Page 10: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/10.jpg)
![Page 11: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/11.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Final Hardware System
• Video In IP
• Disparity Map Frontend
• Disparity Map Backend
• HDMI Out IP
• MicroBlaze
![Page 12: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/12.jpg)
Spartan 6 FPGA
disparity_map_frontend
asynch fifo (reference image)
asynch fifo (search image)
pixel conversion
pixel conversion
BRAM BRAM
disparity calculation
block
disparity_map_backend
async fifo (disparity as grayscale pixel)
address calculation for AXI burst write
Master Burst Read Interface
Microblaze softprocessor
AXI Interconnect
Memory Controller
Master Burst Write Interface
HDMI out IP
video in IP
VmodCam Reference IP
async fifo (CAM A input)
async fifo (CAM B input)
Master Burst Write Interface
DDR2 SDRAM
VModCam Stereoscopic
Camera
Monitor
Legend
Custom IP
Existing IP
Peripheral Hardware
![Page 13: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/13.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Design Process
• MATLAB implementation
• MicroBlaze C Program
• Hardware modules divided among team members
![Page 14: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/14.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Results
• MATLAB produced disparity map
![Page 15: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/15.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Results
• Hardware produced disparity map
![Page 16: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/16.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Performance Comparison
System Frame Size Time
MATLAB 100x200 ~5 minutes(>1hr for 640x480)
MicroBlaze C Program
640x480 ~20 minutes
Hardware System 640x480 2 - 8.5 seconds
![Page 17: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/17.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
What We Learned…
• Simulation is faster than synthesis
• How to use EDK/SDK Tools and ModelSim
• Communicating about interfaces between blocks is key
![Page 18: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/18.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Demo!
![Page 19: ECE532 Final Project Demo Disparity Map Generation on a FPGA Using Stereoscopic Cameras ECE532 Final Project Demo Team 3 – Alim, Muhammad, Yu Ting](https://reader035.vdocument.in/reader035/viewer/2022062802/56649e9c5503460f94b9d01b/html5/thumbnails/19.jpg)
ECE532 Final Project DemoTeam 3 – Alim, Muhammad, Yu Ting
Thank You!
• Questions/Comments