institute of information sciences and technology towards a visual notation for pipelining in a...

Institute of InformationSciences and Technology

Towards a Visual Notation for PipeliningTowards a Visual Notation for Pipelining in a Visualin a Visual Programming Language for Programming Language for

Programming FPGAsProgramming FPGAs

Chris Johnston

Supervisors: Donald Bailey, Paul Lyons

Institute of Information Sciences and Technology

Massey University, Palmerston North

New Zealand

CHINZ July 2006 2


OutlineOutline

Background to the problemReal time image processing using FPGAs

Overview of VERTIPHVisual Environment for Real Time Image Processing in Hardware

Visual notations for pipeliningPipelining is required to meet timing constraintsRepresenting pipelining in a VPL

Summary

CHINZ July 2006 3


Programmable hardwareConfigurable logic blocks (arbitrary logic functions)

Programmable interconnects (routing switches)

I/O

RAM

(Multipliers)

(Processor cores)

FPGA ArchitectureFPGA Architecture

CLB

RAMI/O

CLB

CLB

CLB

CHINZ July 2006 4


Image processingImage processing

Extracting information from or improving an Image

CHINZ July 2006 5


Why FPGAs?Why FPGAs?

Image processing operationsdata intensivecomputationally expensive

To do more workserial processors have to go fasterFPGAs can process information in parallel

FPGAsbuild custom hardware for each operation can be reconfigured as needs change

CHINZ July 2006 6


Hardware design is different from software design

Issues With Programming FPGAsIssues With Programming FPGAs

Hardware Software

Parallelism All circuits active Parallelism through threads

Resource sharing

Need hardware arbitration

Only one process running

Speed Use long pipelines, trade off with throughput

Get a faster processor!

(!)

CHINZ July 2006 7


ParallelismParallelismTemporal and spatial parallelism

Functionblock

Inputdata

Outputdata

Functionblock

Re

gis

ter

Re

gis

ter

Functionblock

Re

gis

ter

Functionblock

Re

gis

ter

Temporal Spatial

Inputdata

Functionblock

Re

gis

ter

Re

gis

ter

Outputdata

Functionblock

A pipeline

Functionblock

Inputdata

Outputdata

Functionblock

Parallel operations

Hybrid

CHINZ July 2006 8


Hardware Design Entry Hardware Design Entry

x y

C

z

Schematic

Handel-C code

unsigned 1 x,y,z,C;par { C = x & y; z = x ^ y;}

HLLs(High-level languages)

VHDL codeentity HA is port( x,y:in bit; z,C:out bit );End HA;architecture BEHAVIOURAL of HA isbegin C<=x and y; z<=x xor y;end HA;

HDLs(Hardware description languages)

Hard to reuseNot algorithmic

VerboseLow level

Very flexible

Like softwareHigher level

Have to work to a design model

Increasing abstraction

CHINZ July 2006 9


Resource & scheduling

Frame Pixel Line Pixel

VERTIPHVERTIPHThree Views of the System

Architecture view

If

else

If

While

Computational view

CHINZ July 2006 10


VERTIPHVERTIPH

Shows components, data and control flow

Allows for encapsulation of data structures and processes

Architecture view

Three Views of the System

Barrel Correction Keyboard Interface

Frame Buffer Manager

RAM1

RAM2

Bilinear Interpolation Video DriverCamera Interface

Control flowData flow

(Architectural)

CHINZ July 2006 11


Barrel Correction Keyboard Interface

Frame Buffer Manager

RAM1

RAM2

Bilinear Interpolation Video DriverCamera Interface

Control flowData flow

Shows components, data and control flow

Allows for encapsulation of data structures and processes




Architecture view

If

else

If

While

Computational view

CHINZ July 2006 12




Frame Pixel Line PixelLocal and Global scheduling

Avoids resource conflicts

Shows when operations can run

Architecture view

If

else

If

While

Computational view

(Resource & Scheduling)

CHINZ July 2006 14


While (true)While (true)If (videoScanX == VisibleCols)

xc * xc sx

xc x

y + 1 y

sy+2y+1 sy

xadd

sqrd

x

sx

sx

k

y

Kru1

Interpolated LUT magcorrectx

correcty

Operatorsxadd: x + 1sqrd: sx + 2x + 1kru: (sx + sy) * kcorrectx: mag * xcorrecty: mag * y

If (videoScanX == VisibleCols)xc * xc sx

xc x

y + 1 y

sy+2y+1 sy

Operatorsxadd: x + 1sqrd: sx + 2x + 1kru: (sx + sy) * kcorrectx: mag * xcorrecty: mag * y

-3



If

else

If

While

Computational view


Architecture view

Clock cycle

(Computational)

CHINZ July 2006 15


Computational viewshows concurrency, pipelining and latency

no obvious way to show a pipeline

Sequential

Parallel

If

else

If

While

Computational view


A B C

Time

B

C

Concurrency

CHINZ July 2006 16


illustrates the dataflow

spacially inefficient

shows that A, B & C run in separate clock cycles

VERTIPHVERTIPH Pipelining notationsPipelining notationsDiagonal Gantt

A

B

C

CHINZ July 2006 17


shows when all processors are active

VERTIPHVERTIPH Pipelining notationsPipelining notationsStaggered Gantt

A

B

C

A

B

A

CHINZ July 2006 18


branching makes diagrams more complex

cannot express multicycle pipelines

VERTIPHVERTIPH Pipelining notationsPipelining notationsStaggered Gantt with Conditional Branching

A

C

B

B1

A

B

B1

A

CHINZ July 2006 19


VERTIPHVERTIPH

A series of processors perform successive phases of a taskprocessor passes data “down the line” when its phase is complete

early processors accept more data while later ones handle early data

A slider to change data interarrival times

Notation is compact, but hides processor concurrency

Pipelining notationsPipelining notationsSequential Pipeline

A B

C

C1

D E F

CHINZ July 2006 20


Pipelining notationsPipelining notations

Combines the benefits of Sequential and Gantt

Bars indicate: phase, throughput & time when processors have valid data.

VERTIPHVERTIPHSequential Pipeline With Staggered Bars

A B

C

C1

D E F

CHINZ July 2006 21


A B

C

C1

D E F

A B

C

C1

D E F

Pipelining notationsPipelining notationsVERTIPHVERTIPH

Detailed bars show repeated hardware

Staggered Sequential Pipeline With Staggered Bars

CHINZ July 2006 22


SummarySummary

Staggered Gantt used too much screen space

could not describe all options

Sequential was not expressive enough

Adding Staggered Barsgives an indication of data throughput

Using Detailed BarsBalance between complexity and expressiveness.

Shows hardware, phase and throughput

CHINZ July 2006 23


FIN

institute of information sciences and technology towards a visual notation for pipelining in a...

Documents