Using tags toImprove Diversity of

Sparse Associative Memories

Stephen Larroque

with Ehsan Sedgh Gooya, Vincent Gripon, Dominique Pastor

An alternative to the size-diversity trade-off forneuromorphic devices

23rd March 2015COGNITIVE 2015

Long-term storage?

2

Long-term storage?

• Addressed memory models in computers :

3

Long-term storage?

• Addressed memory models in computers :

4

Index Value

00000100 Anabelle

00000101 Anais

... ...

111111111 Zoey

Long-term storage?

• Addressed memory models in computers :

5

Index Value

00000100 Anabelle

00000101 Anais

... ...

111111111 Zoey

Partial/noised query : 0000010_ ?

Long-term storage?

• Addressed memory models in computers :

6

Index Value

00000100 Anabelle

00000101 Anais

... ...

111111111 Zoey

Partial/noised query : 0000010_ ?

Corruption ->

Long-term storage?

• Addressed memory models in computers :

7

Index Value

00000100 Anabelle

00000101 Anais

... ...

111111111 Zoey

Partial/noised query : 0000010_ ?

Corruption ->

Long-term storage?

• Addressed memory models in computers :

• Connectionist solutions :– Associative memories (Hebbian rule)– Deep neural networks (McCulloch-Pitts neurons)8

Index Value

00000100 Anabelle

00000101 Anais

... ...

111111111 Zoey

Partial/noised query : 0000010_ ?

Corruption ->

Associative memories

9

Associative memories

• Hopfield (1982)

10

(Courtesy ofR. Rojas)

Associative memories

• Hopfield (1982)

• Willshaw (non-holographic associative memory, 1969)

11

(Courtesy ofR. Rojas)

Associative memories

• Hopfield (1982)

• Willshaw (non-holographic associative memory, 1969)

• Cliques network (2011)

12

(Courtesy ofR. Rojas)

Cliques network, storing

• Hebbian rule

• Message ≡ clique

13

Cliques network, retrieval

14

Cliques network, retrieval

15

Cliques network, retrieval

16

Iterative !

Cliques network, storing more

17

000000000

000000000

000000000

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix

Cliques network, storing more

18

100000000

100000000

100000000

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix

Cliques network, storing more

19

100000000

101000000

101000000

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix

Cliques network, storing more

20

100000100

101000001

101000000

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix

Cliques network, storing more

21

100000101

101000101

101000100

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix

Size-diversity trade-off

22

100000101

101000101

101000100

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix

Problem : a non-learnt message is retrievable !

Fake memory :(network’s size toosmall for diversity)

Let’s color this graph!

23

200000304

102000403

103000400

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix

Let’s color this graph!

24

200000304

102000403

103000400

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrix

Let’s color this graph!

26

200000304

102000403

103000400

Part 1 2 3

1

2

3

... BGW ...Unit

BGW...

symm

etry

Adjacency matrixFake memory avoided by tag disambiguation

Performance

27

• Diversity/density for various tags limits against the error rate :

χ = 16, c = 8, L = 64, erasure rate = 0,5 (half of the query is erased)

Conclusion and future works• Tags : a viable alternative to the diversity-size trade-

off for fixed-size networks(e.g., neuromorphic devices)

• A tentative explanation of synapses heterogeneity : brain may use an affinity system to co-sustain synapses with similar parameters.« Neurons that fire together, wire together, and with a strong affinity. »

• Next :– Noisy scenario (unreliable tags)– pertinence of memories, variable resiliency : all items may

not need to be stored with equal resiliency. Try to refresh tags on access ? (« spacing effect » ?) 28

– Nonholographic associative memory, D. J. Willshaw, O. P. Buneman, and H. C. Longuet-Higgins, Nature, vol. 222(5197), June 1969, pp. 960–962

– Neural networks and physical systems with emergent collective computational abilities, J. J. Hopfield, Proceedings of the national academy of sciences, vol. 79, no. 8, 1982, pp. 2554–2558.

– Sparse neural networks with large learning diversity, V. Gripon and C. Berrou, Neural Networks, IEEE Transactions on, vol. 22, no. 7, 2011, pp. 1087–1096.

Thanks and a few references slideshare.net/LRQ3000

Vincent GRIPON Dominique PASTOR

ERC grant agreement n° 290901

EhsanSEDGH GOOYA

Thank you! slideshare.net/LRQ3000

Bonus Slides

Neural constraints• Energetic parsimony

• Material resources parsimony

• Noise robustness

• Simple processing rules (analog?)

=> Feed-forwoard ANNs : synaptic weights as floats are too sensitive(learning = adjust weights)and capacity ~ sub-linear in number of nodes.

35

(Aiello & Wheeler, 1995)

Cliques network

• Brain = information encoder

• Fully graphical model

• Associative, recurrent network with binary weights, integer output, capacity ~ n² :– Network : set of clusters– Cluster : set of fanals– Fanal : graph nodes

(microcolumn?)

36

(Gripon-Berrou Neural Network, 2011)

Capacity vs Diversity

• Diversity = number of messages possibly leant/stored

• Capacity = whole learnt information

• « From a cognitive point of view, it is better to learn (and possibly combine) 1000 messages of 10 characters than to learn 10 messages of 1000 characters »

37

(C. Berrou & V. Gripon, 2010)

Tagged network: colors as layers

38

Thrifty code

39

Disambiguation by tags

40

Disambiguation by tags - 2

41

Disambiguation by tags - 3

42

Analysis of error rate

43

• New error type : lost unit error

Analysis of error rate

44

• New error type : lost unit error

Analysis of error rate

45

• New error type : lost unit error

Analysis of error rate

46

• New error type : lost unit error

Analysis of error rate

47

• New error type : lost unit error

Lost forRed clique !

Analysis of error rate

48

• New error type : lost unit error

=> When a clique lose one node, because all edges have been overwritten by other tags of newer cliques, it becomes unretrievable.

=> only dependent on storage process !

•

Lost forRed clique !

Theoretical lost unit error

49

• With :

• Approximation : messages are i.i.d. variables

M = total messages ; c = clique orderχ = total graph parts ; L = units per part

Proba to overwrite one edgewhen storing one new clique

(1 chance over network’s size)

Clique size(repeat for all edges of each new message)

(need to overwrite all

edges of one unit to lose it)

What about other error types?

50

• Real error rate (red) against a composition of errors from each possible type (green) : lost unit error is a good predictor

What about other error types?

51

• Theoretical lost unit error (black) against real (blue)

Efficiency?

52

• Efficiency  = B (amount of info stored) Q (material used)

• Clique network :

• Tagged network :

=> Tagged network use more material, proportionally to the number of tags !

Performance - 2

53

• Performance when accounting the efficiency :

THE END