hybrid pipeline structure for self-organizing learning array yinyin liu 1, ding mingwei 2, janusz a....

Hybrid Pipeline Structure for Hybrid Pipeline Structure for Self-Organizing Learning ArraySelf-Organizing Learning Array

Yinyin Liu1, Ding Mingwei2 , Janusz A. Starzyk1,

1 School of Electrical Engineering & Computer ScienceOhio University, USA

2 Ross University

ISNN 2007: The 4th International Symposium on Neural Networks

2

OutlineOutline

•RC systems design of SOLAR

•Dimensionality reduction

•Input selection, weighting

•Pipeline structure

• Experimental results

• Conclusions

Broca’sarea

Parsopercularis

Motor cortex Somatosensory cortex

Sensory associativecortex

PrimaryAuditory cortex

Wernicke’sarea

Visual associativecortex

Visualcortex

3

• “…Perhaps the last frontier of science – its ultimate challenge- is to understand the biological basis of consciousness and the mental process by which we perceive, act, learn and remember..” from Principles of Neural Science by E. R. Kandel et al.

E. R. Kandel won Nobel Price in 2000 for his work on physiological basis of memory storage in neurons.

• “…The question of intelligence is the last great terrestrial frontier of science...” from Jeff Hawkins On Intelligence. Jeff Hawkins founded the Redwood Neuroscience Institute devoted to brain research. He co-founded Palm Computing and Handspring Inc.

Intelligence

AI’s holy grailFrom Pattie Maes MIT Media Lab

4

How can we design intelligence?How can we design intelligence?

• We need to know how

• We need means to implement it

• We need resources to build and sustain its operation

5From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006

Resources – Evolution of Electronics

6By Gordon E. MooreBy Gordon E. Moore

7

8From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006

Clock Speed (doubles every 2.7 years)

9From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006

10

OutlineOutline

•RC systems design of SOLAR

•Dimensionality reduction

•Input selection, weighting

•Pipeline structure

• Experimental results

• ConclusionsBroca’sarea

Parsopercularis

Motor cortex Somatosensory cortex

Sensory associativecortex

PrimaryAuditory cortex

Wernicke’sarea

Visual associativecortex

Visualcortex

11

Traditional ANN HardwareTraditional ANN HardwareTraditional ANN HardwareTraditional ANN Hardware

– Limited routing resource.

– Quadratic relationship between the routing and the number of neuron makes classical ANNs wire dominated.

input

output

information flow

hidden

Interconnect is Interconnect is 70% of chip area70% of chip area

12

Biological Neural NetworksBiological Neural Networks Biological Neural NetworksBiological Neural Networks

Cell body

From IFC’s webpage Dowling, 1998, p. 17

13

Sparse StructureSparse Structure

• 1012 neurons in human brain are sparsely connected

• On average, each neuron is connected to other neurons through about 104 synapses

• Sparse structure enables efficient computation and saves energy and cost

14

Why should we care?Why should we care?

Source: SEMATECHSource: SEMATECH

15

0%

20%

40%

60%

80%

100%

1999

2002

2005

2008

2011

2014

% Area Memory

% Area ReusedLogic

% Area New Logic

Percent of die area that must be occupied by memory to maintain SOC design productivity

Design Productivity Gap Design Productivity Gap Low-Value Designs? Low-Value Designs?

Source = Japanese system-LSI industry

16

OutlineOutline

•RC systems design of SOLAR

•Dimensionality reduction

•Input selection, weighting

•Pipeline structure

• Experimental results

• ConclusionsBroca’sarea

Parsopercularis

Motor cortex Somatosensory cortex

Sensory associativecortex

PrimaryAuditory cortex

Wernicke’sarea

Visual associativecortex

Visualcortex

17

SOLAR System DesignSOLAR System Design

• SOLAR Introduction Entropy based self-

organization

– data-driven

– Local connection Dynamical reconfiguration Local and sparse

interconnections Online inputs selection Feature neurons and

merging neurons Pattern recognition,

classification

18

Pipeline OverviewPipeline Overview

node computing ability → “soft” connections

Four modes

1. Idle2. Read3. Process4. Write

19

Pipeline Signal Flow 1Pipeline Signal Flow 1

20

Pipeline Signal Flow 2Pipeline Signal Flow 2

21

Pipeline Signal Flow 3Pipeline Signal Flow 3

22

Node OperationsNode Operations

Implemented with Xilinx picoBlaze

Runs at higher frequency

23

OutlineOutline

•RC systems design of SOLAR

•Dimensionality reduction

•Input selection, weighting

•Pipeline structure

• Experimental resultsExperimental results

• ConclusionsBroca’sarea

Parsopercularis

Motor cortex Somatosensory cortex

Sensory associativecortex

PrimaryAuditory cortex

Wernicke’sarea

Visual associativecortex

Visualcortex

24

Em(x) Simulation ResultsEm(x) Simulation Results

25

Iris Data ProcessingIris Data Processing

4x7 array processing Iris data

Linear growth of HW cost

26

Chip LayoutChip Layout

27

XILINX

XILINX

VIRTEX XCV 1000

VIRTEX XCV 1000

Hardware DevelopmentHardware Development

28

Future WorkFuture Work- System SOLAR- System SOLAR

29

Conclusions & Future workConclusions & Future work

• Sparse coding building in sparsely connected networks

• WTA scheme: local competition accomplish the global competition using primary and secondary layers –efficient hardware implementation

• OTA scheme: local competition produces neuronal activity reduction

• OTA – redundant coding: more reliable and robust

• WTA & OTA: learning memory for developing machine intelligence

Future work:

• Introducing temporal sequence learning

• Building motor pathway on such learning memory

• Combining with goal-creation pathway to build intelligent machine