FPGA-based Platform for FPGA-based Platform for Real-Time Stereo VisionReal-Time Stereo Vision

FPGA-based Platform for FPGA-based Platform for Real-Time Stereo VisionReal-Time Stereo Vision

Sergiy Zhelnakov, Pil Woo (Peter) Chun,Sergiy Zhelnakov, Pil Woo (Peter) Chun,Valeri KirischianValeri Kirischian

Supervisor: Dr. Lev KirischianSupervisor: Dr. Lev Kirischian

Reconfigurable Embedded Systems LabReconfigurable Embedded Systems Lab

Ryerson University, Toronto, ONRyerson University, Toronto, ON

MotivationMotivationMotivationMotivation

Areas of application:Areas of application:– Real-time Stereo-Vision Systems Real-time Stereo-Vision Systems – Telematic Systems: Remote Control of Telematic Systems: Remote Control of

Manipulators in Hazardous AreasManipulators in Hazardous Areas– Virtual Reality Systems and SimulatorsVirtual Reality Systems and Simulators– UAV Navigation SystemsUAV Navigation Systems– Telemedicine Telemedicine – Surveillance / Security SystemsSurveillance / Security Systems

ObjectivesObjectivesObjectivesObjectives

Development of the Run-Time Reconfigurable Development of the Run-Time Reconfigurable PlatformPlatform for implementation, testing and real- for implementation, testing and real-time verification of algorithms for stereo-vision time verification of algorithms for stereo-vision stereo-image recognition, and visualization of stereo-image recognition, and visualization of 3D images3D images

Implementation and testImplementation and test of real-time stereo- of real-time stereo-video processing algorithms (e.g. Edge video processing algorithms (e.g. Edge Detection in moving objects)Detection in moving objects)

SpecificationSpecificationSpecificationSpecification Functional specification:Functional specification:

– The system performs:The system performs: image capture from image capture from two color cameras, stereo video two color cameras, stereo video visualization with shutter glasses, image visualization with shutter glasses, image processing (edge detection)processing (edge detection)

Technical specificationTechnical specification– Input:Input: video data with video data with

spatial resolution 640 x 480 pixelsspatial resolution 640 x 480 pixels frame rate: 30 fpsframe rate: 30 fps color, 8-bit resolutioncolor, 8-bit resolution

– Output:Output: standard SVGA standard SVGA resolution: 640 x 480 pixelsresolution: 640 x 480 pixels frame rate: not less than 60 fpsframe rate: not less than 60 fps

Fastest process:

40nS per pixel output

Platform Components and LinksPlatform Components and LinksPlatform Components and LinksPlatform Components and Links

Stereo-Image Capturing &

Video Pre-processing ModuleVideo Processing Module

Multi-Channel Post-processing

Video-output Module

CRT Monitor LCD Projector

Shutter Glasses

Run-time Reconfigurable Multi-Stream Video Processor

Implemented AlgorithmsImplemented Algorithms Implemented AlgorithmsImplemented Algorithms Color matching: (Bayer pattern) – at 2x30 Color matching: (Bayer pattern) – at 2x30

frames/sec and 640x480 resolutionframes/sec and 640x480 resolution

Implemented AlgorithmsImplemented AlgorithmsImplemented AlgorithmsImplemented Algorithms Color matchingColor matching

Implemented AlgorithmsImplemented AlgorithmsImplemented AlgorithmsImplemented Algorithms

Edge detection (Robert Cross) backgroundEdge detection (Robert Cross) background512 pixels

480 pixels

a b

c d

Max (|d-a|, |b-c|) ,where

Implemented AlgorithmsImplemented Algorithms Implemented AlgorithmsImplemented Algorithms

Edge detection (Robert Cross)Edge detection (Robert Cross)

A2 B2 C2 D2 E2 F2 …………………………… Z2Y2X2

512 pixels

P C camera DATA

P : Previous camera dataC : Current camera data

Temporary Storage

a1 b1 c1 d1 e1 f1 ……………………… z1y1x1

512 pixels

Block RAM 2/1 (previous row data)

Calculation = C-b1 or P-a1

………

Block RAM 1/2(current row data)

Platform Assembly and Platform Assembly and Implementation ResultsImplementation Results

Implementation resultsImplementation results– Image capture and Image capture and

visualization on the visualization on the FPGA based FPGA based Reconfigurable Reconfigurable Functional Unit (RFU) Functional Unit (RFU) (1st stage - XCV50E; (1st stage - XCV50E; 2nd - XC2V1000)2nd - XC2V1000)

– Edge detection Edge detection algorithm (Robert algorithm (Robert Cross) implementedCross) implemented

SummarySummarySummarySummary The Run-Time Reconfigurable Platform was The Run-Time Reconfigurable Platform was

developed for different Stereo-Vision applications.developed for different Stereo-Vision applications.

All components of the Platform were tested by All components of the Platform were tested by implementation of real-time stereo-image capture, implementation of real-time stereo-image capture, image processing and visualization on stereo-image processing and visualization on stereo-video output display systemvideo output display system

Perspectives of the platform development: stereo-Perspectives of the platform development: stereo-panoramic vision systempanoramic vision system