real time neuromorphic camera architecture … · neuromorphic camera using an inexpensive camera...

International Journal of Electronics and Electrical Engineering Vol. 5, No. 3, June 2017

©2017 Int. J. Electron. Electr. Eng. 230doi: 10.18178/ijeee.5.3.230-234

Real Time Neuromorphic Camera Architecture

Implemented with Quadratic Emphasis in an

FPGA

Elizabeth Fonseca Chavez1, Mario A. Ibarra-Carrillo

2, Julio C. Sosa

1, and Ruben Ortega-Gonzalez

1

1 Polytechnic/Escom, City of México, México

2 UNAM/Telecomm City of Mexico, Mexico

Email: [email protected], [email protected], {jcsosa, rortegag}@ipn.mx

Abstract—A neuromorphic camera has one neuromorphic

vision sensor which was designed to detect an approaching

object in real time. This sensor is the “Dynamic Vision

Sensor” (DVS) which works like a retina, but on a silicon

chip. The DVS responds to changes in intensity under a

wide range of lighting conditions and offers the possibility of

fast, computationally efficient visual processing for

navigation in mobile robotics. Reichardt proposed a model

of fly s vision such as the correlation-based “Elementary

Motion Detector” (EMD). As a proposal, we present a new

architecture for the EMD such that increases sensitivity. We

call this new design the "EMD with emphasis". Because the

"EMD with emphasis" requires more arithmetic operations

than the classic EMD, it is implemented in a pipeline for its

operation at a frequency of 25 MHz on a FPGA. The

experimental results show that the performance of the

proposed model is better than that of the classic model. The

tests were successful for motion detection for size, contrast,

as well as for movement: horizontal, expansion and rotation.

Our system works with 30 fps and, a very low latency,

dynamic range of 56dB, a resolution of 640x480. There are

two special features: the first implies that the system

operates with inexpensive cameras. The second

characteristic has to do with the way the VHDL is used, ie,

the system is easily transportable between FPGA branches.

Index Terms—neuromorphic, Reichardt, elementary motion

detector, EMD, DVS, FPGA

I. INTRODUCTION

In recent years, the neuromorphic cameras have

become more and more common in machine vision

oriented tasks. One neuromorphic camera has one vision

sensor was designed to detect an approaching object in

real time; this sensor is the Dynamic Vision Sensor

(DVS), it solves these problems by using patented

technology that works like the human retina. Dynamic

Vision Sensors are biologically inspired vision systems

that asynchronously generate events upon relative light

intensity changes [1]. The DVS is a silicon Chip, it offers

the possibility of fast, computationally efficient visual

processing for navigation in mobile robotics. A silicon

retina camera responds to changes in intensity under a

Manuscript received July 29, 2016; revised April 2, 2017.

wide range of lighting conditions. The output signals of

the sensor is a stream of events that encode the positions

and the precise timestamps of the intensity changes. Then,

an “Address-event Representation Protocol” (AER)

allows communication between the DVS and a

microprocessor-based system [2].

Vision systems under biological inspiration, usually

are constructed with analog electronic circuits that mimic

such neuro-biological architectures present in the nervous

system [3]. For example, the fly has extraordinary

abilities to detect motion. Reichardt proposed the basic

model of the fly s vision nervous system (1956) such as

the correlation-based Elementary Motion Detector (EMD)

[4]. An EMD is driven by the combination of two

neighboring photoreceptors. This combination transforms

the motion of an object into two separate signals by a

delay. There are others articles with modified model of

Reichardt for to improve detection of horizontal

movement and reduce system noise. Placing an additional

process to each adjacent photoreceptors: log-normal filter

[5], high pass filter [6]-[8], low pass filter [6], [8] and [7],

logarithmic filter [9], delay exponential [5], and

automatic gain adaptable [8]. Most validates these

modified with Matlab model.

There are another way to validate their modified

models and some authors use the FPGA to store massive

data and have more processing power, as shown by the

articles [8], [9].

This document is aimed at implementing a

neuromorphic camera using an inexpensive camera and

developing a specific application processor. Thus, the

present work is divided into three parts. The first part is

the proposal of a new architecture for Reichardt EMD.

The second part consists of simulation and comparative

testing between classical architecture and the proposed

architecture. Simulations are to quantify and validate the

capabilities of the new model. The last part describes the

implementation of the new architecture in FPGA and

testing in real time. The camera used has a 640x480

frame size, delivery, according to the data sheet, up to 30

frames per second. The FPGA will perform its

calculations with integer data of 5 bits.

The scope of the developments presented below

involve an architecture in which the sensors act along a


©2017 Int. J. Electron. Electr. Eng. 231

horizontal line. A model that makes detecting both,

horizontally and vertically, is not implemented, this is a

simplification that allows us to clarify the developments

in this Article.

II. EMD MODEL

A. Classical Model

The fly's vision system comprises several thousands of

photoreceptors. These sensors are connected to an array

of motion sensing neurons. Reichardt [3] proposed the

model which approximates the behavior of such neurons.

In this model, he called Elemental Motion Detector or

EMD. This model is based on the correlation. Suppose

this, A1 and A2 are two neighboring photoreceptors and

representing the input signals, left and right. B1 and B2

representing the corresponding delayed signals [9]. The

EMD consists of two correlators as shown in Fig. 1. The

left correlator calculates the product A2B1, which means

motion detection right. In turn, the right correlator

calculates the A1B2 product, which means the detection

of movement to the left. Both correlators are combined

by a difference between their effects (A2*B1 - A1*B2).

Thus, a positive result indicates movement to the right

while a negative result implies movement to the left.

Figure 1. Classical EMD model

B. Modified Emphasis EMD

Figure 2. Emphasis EMD model

The classic EMD model when tested in the MATLAB,

showed little sensitivity to certain experimental pattern.

Therefore, it was decided to add a quadratic emphasis on

motion detection model thus achieved an improvement in

sensitivity. The Fig. 2 shows the architecture with

emphasis.

III. THE SIMULATIONS

To start, Fig. 3a shows the image obtained from

classical EMD, when exposed to a pattern that moves

leftward camera. Meanwhile, Fig. 3b shows the image

obtained by the EMD with emphasis being exposed to the

same pattern: we clarify that the pattern was moved

manually and trying to maintain the same speed in both

tests. The reader can see the difference in the texture of

the images obtained.

Figure 3. (a) Resulting images from the classical EMD model. (b) Resulting images from the emphasis EMD model

Because the images shown in Fig. 3 depend on an

empirical assessment, it is necessary to define a method

to quantify the quality of the two EMD.

Then, an experiment is performed to quantify the

reliability of both architectures. The experiment consists

of two stages. The first stage consists of expose the

camera to a pattern that shifts leftward. The second step is

to evaluate the percentage of pixels that indicate a

movement to the left and must also evaluate the amount

of pixels that erroneously indicate a move to the right.

The experiment was performed 100 times, this is, 100

frames with a pattern moving to the left of the camera

were taken.

Fig. 4 shows graphs obtained from the experiment

performed on the classical model of EMD. The Fig. 4a

indicates the relationship between the percentage of

pixels detected as indicators of movement to the left, and

the index picture analyzed. The Fig. 4b shows the

percentage of pixels erroneously detected as indicating

movement to the right, and the index picture analyzed.



Figure 4. About of classical EMD model: (a) Relationship between the percentage of pixels that indicate movement to the left and the index

picture analyzed. (b) Percentage of pixels erroneously detected as

indicating movement to the right, and the index picture analyzed

Taking averages of the graphs of Fig. 4, it is obtained

that the real left detected is 88%. So then, the percentage

of falsely detected left is 12%. So we can conclude that

the movement was correctly detected.

Fig. 5 shows graphs obtained from the experiment

performed on the model of EMD with emphasis. The Fig.

5a indicates the relationship between the percentage of

pixels detected as indicators of movement to the left, and

the index picture analyzed. The Fig. 5b shows the

percentage of pixels erroneously detected as indicating

movement to the right, and the index picture analyzed.

Taking averages of the graphs of Fig. 5, it is obtained

that the real right detected is 98 %. So then, the

percentage of falsely detected is 2%. So we can conclude

that not only was correctly detected, a better detection

was also performed.

Figure 5. About the EMD model with emphasis: (a) Relationship between the percentage of percentage of pixels that indicate movement

to the left and the index picture analyzed. (b) Percentage of pixels

erroneously detected as indicating movement to the right, and the index picture analyzed

IV. IMPLEMENTATION ON FPGA

The architecture model involves the construction of

two blocks of RAM. So while a memory block stores the

image newly captured by the camera, the other block

stores a previous image. This second image is the one that

represents the delay defined by Reichardt EMD. The

memory blocks also have the characteristic of having

dedicated writing ports and having reading dedicated

ports. Both memory blocks are connected to a block that

is responsible for implementing the correlator EMD with

emphasis.

Figure 6. Internal structure of the EMD with emphasis

The block that implements the EMD with emphasis

operates in pipeline mode, as shown in Fig. 6. It is

possible for the reader to see that the four products,

required by the EMD with emphasis, are performed in a

single pipeline stage, that is, four multiplications in

parallel, which is possible thanks to the FPGA. In the

next stages of the pipeline other EMD operations are

performed.

The FPGA Cyclone IV used is mounted in the

development system DE2-115. The Cyclone IV has the

ID, EP4CE115 and contains 114,480 logic elements

(LEs), the largest offered in the Cyclone IV E series, up

to 3.9-Mbits of RAM, and 266 multipliers. The camera

used is the OV7670, widespread in the local market. This

camera can works up 30 FPS, dynamic range of 56dB,

format YUV of 8 bits. Fig. 7 shows the elements to

implement and probe the new architecture of

neuromorphic camera. The Fig. 7 does not show the

monitor which is used to observe what sees the

neuromorphic camera.

Figure 7. Elements to construct a neuromorphic camera and elements to test it

V. RESULTS

A. Real Time Experimental Results

In this section one experiments are proposed to test the

system implemented in the FPGA. The experiment

consists of performing three tests using high-contrast

patterns: patterns in black and white. For these tests the

results delivered by the FPGA implementation of the

classic EMD against the results of the implementation in

FPGA with an emphasis EMD are compared.

Figure 8. Test patterns and results of EMD classic camera and with emphasis.

Fig. 8 shows the probes and results: (a) Test Pattern to

test sensitivity to rotation. (b) Result of the classic EMD

to rotation. (c) Result of EMD with emphasis to rotation.

(d) Test Pattern to test sensitivity to lateral displacement.

(e) Result of the classic EMD to lateral displacement. (f)

Result of EMD with emphasis to temporal displacement.

(g) Test Pattern to test sensitivity to the approach and

departure. (h) Result of the classic EMD to the approach

and departure. (i) Result of EMD with emphasis to the

approach and departure.

B. Discussion

The result of simulation with MATLAB shows

improved motion detection, with emphasis model. The

test was done with webcam and with reproduction of

video. The simulation results show that using MATLAB

neuromorphic camera based on EMD with emphasis

delivers a better quality picture. This can be seen by

comparing the graph of Fig. 4 with the graph of Fig. 5. In

both figures, the reader can see that the camera may be

based on an EMD with emphasis delivery fewer false

detections.

The results of tests on FPGA show that the EMD with

emphasis delivery sharper images and more defined

edges. The reader may notice some differences in

position of the elements shown in the figure. This is

because the images were photographed directly from the

monitor. In the case of used to “zoom in” and “zoom out”

pattern, both EMD showed a slight insensitivity.

VI. CONCLUSION

The neuromorphic camera needs a dynamic vision

sensor to detect a movement, the Reichardt correlator

provides low computational cost and efficient solution for

motion detection, but for better performance we modified

the Reichardt model (the “quadratic model emphasis”) for

to replace with success the DVS. The experimental result

shows that the performance of the proposed model is

better than the classic model. Our system has a latency of

0.2 microseconds because the EMD with emphasis was

implemented in a pipeline. Lower costs, low storage, no

redundant data, and resolution of 640x480. The detection

is good for the Reichardt classic model, but with the

modified Reichardt the detection is better. The tests were

successful for motion detection for size, contrast, as well

as for movement: horizontal, expansion and rotation. We

have a neuromorphic camera ready for use, with open

source. The future work will be to reduction of bits of

capture, reduction of noise using a filter, and to create a

test module in the FPGA.

REFERENCES

[1] T. Delbruck, V. Villanueva, and L. Longinotti, “Integration of

dynamic vision sensor with inertial measurement unit for

electronically stabilized event-based vision,” in Proc. IEEE International Symposium on Circuits and Systems, 2014, pp. 2636-

2639.

[2] A. Rios-Navarro, E. Cerezuela-Escudero, M. Dominguez-Morales, A. Jimenez-Fernandez, G. Jimenez-Moreno, and A. Linares-

Barranco, “Real-time motor rotation frequency detection with event-based visual and spike-based auditory AER sensory

integration for FPGA,” in Proc. International Conference on



Event-based Control, Communication, and Signal Processing, 2015, pp. 1-6.

[3] C. Mead, “Neuromorphic engineering: Overview and potential,”

in Proc. IEEE International Joint Conference on Neural Networks, 2005.

[4] W. Reichardt, M. Egelhaaf, and R. W. Schlögl, “Movement detectors provide sufficient information for local computation of

2-D velocity field,” Naturwissenschaften, vol. 75, no. 6, pp. 313-

315, 1988. [5] D. C. O'Carroll, P. D. Barnett, and K. Nordström, “Computational

models reveal non-linearity in integration of local motion signals by insect motion detectors viewing natural scenes,” in Proc.

Seventh International Conference on Intelligent Sensors, Sensor

Networks and Information Processing, 2011, pp. 131-136. [6] A. Fabrice and F. Nicolas, “Optic flow sensors for robots:

Elementary motion detectors based on FPGA,” in Proc. IEEE Workshop on Signal Processing Systems Design and

Implementation, 2005, pp. 182-187.

[7] F. Aubépart and N. Franceschini, “Bio-inspired optic flow sensors based on FPGA: Application to micro-air-vehicles,”

Microprocessors and Microsystems, vol. 31, no. 6, pp. 408-419, 2007.

[8] J. Plett, A. Bahl, M. Buss, K. Kühnlenz, and A. Borst, “Bio-

inspired visual ego-rotation sensor for MAVs,” Biological Cybernetics, vol. 106, no. 1, pp. 51-63, 2012.

Elizabeth Fonseca-Chávez, Mechanical

Electrical Engineer graduated from FES-C

UNAM, Master in engineering systems of the Faculty of Engineering of the UNAM. She is

Assistant Professor in Telecommunications and Computation Engineering Department of

the Faculty of Engineering, UNAM. Her areas

of interest: Digital Systems Design, FPGA y CPLDs. She is an IEEE member.

[email protected]

Mario A. Ibarra-Carrillo, Mechanical Electrical Engineer graduated from the Faculty

of Engineering, UNAM, Master in

Engineering with a concentration in Digital Signal Processing, graduated from the

Division of Graduate Studies of the Faculty of Engineering of the UNAM. He is Professor

Associated in Telecommunications

Engineering Department of the Faculty of Engineering, UNAM. His areas of interest:

Digital Signal Processing, DSP, Digital Systems Design, FPGA and CPLDs. He is an IEEE member. [email protected]

Julio C. Sosa received the Engineering degree

in Electronic Engineering in 1997 from ITLAC, Mich. Mexico. The M. Sc. in

Electrical Engineering, in 2000 from

CINVESTAV-IPN, Mexico and the Ph.D. in Technology of Information, Communication

and Computation in 2007 from the University of Valencia, Spain. He is Titular Professor at

Instituto Politécnico Nacional - ESCOM,

México D. F. since 1998. His current research interests are computer architecture, signal and image processing, sensor

networks and embedded systems. [email protected]

Rubén Ortega-González received the B.Sc. degree in electrical engineering from the

Instituto Politécnico Nacional, Mexico City, Mexico, 1999 and the Ph.D. degree in

Electronics Engineering from the Universidad

Politécnica de Valencia, Valencia, Spain, in 2012. He has been professor in Escuela

Superior de Computo, Instituto Politécnico Nacional since 1995. His main research fields

are in modeling and control of power

converters applied to the distributed generation in microgrids, and digital signal processing.



[9] T. Zhang, H. Wu, A. Borst, K. Kuhnlenz, and M. Buss, “An FPGA implementation of insect-inspired motion detector for high-

speed vision systems,” in Proc. IEEE International Conference on

Robotics and Automation, 2008, pp. 335-340.

mailto:[email protected]

mailto:[email protected]

real time neuromorphic camera architecture … · neuromorphic camera using an inexpensive camera...

Documents