EpisodicMemory(EpMem)IJCAI2016

NateDerbinsky

Agenda

• Bigpicture• Basicusage• Exampleagents

July11,2016 EpisodicMemory(EpMem) 2

Soar9

July11,2016 EpisodicMemory(EpMem) 3

SymbolicLong-TermMemories

SymbolicWorkingMemory

Procedural

Decision

Procedure

ChunkingReinforcementLearning

Action

Semantic

SemanticLearning

Episodic

EpisodicLearning

Spatial Visual SystemObject-basedcontinuousmetricspace

Perception

SoarBasicFunctions

1. Input fromenvironment2. Elaboratecurrentsituation:parallelrules3. Proposeoperatorsviaacceptable preferences4. Evaluateoperatorsviapreferences:Numericindifferentpreference5. Selectoperator6. Applyoperator:Modifyinternaldatastructures:parallelrules7. Output tomotorsystem[andaccesstolong-termmemories]

OperatorDecisionInput Output

StateElaboration

OperatorProposal

OperatorEvaluation

OperatorElaboration

OperatorApplication

July11,2016 EpisodicMemory(EpMem) 4

EpisodicMemory:BigPicture

Episodicmemoryisaweak learningmechanism– Automaticallycaptures,stores,andtemporallyindexesagentstate

– Supportscontent-addressableagentinterfacetoautobiographicalpriorexperience

July11,2016 EpisodicMemory(EpMem) 5

ArchitecturalIntegration

July11,2016 6EpisodicMemory(EpMem)

ArchitecturalIntegration

July11,2016 7EpisodicMemory(EpMem)

ArchitecturalIntegration

July11,2016 8EpisodicMemory(EpMem)

ArchitecturalIntegration

July11,2016 9EpisodicMemory(EpMem)

epmem “link”availableoneach

state!

BasicUsage

• Episodic-memoryrepresentation• Storingknowledge• Retrievingknowledge

July11,2016 EpisodicMemory(EpMem) 10

Episodic-MemoryRepresentation

Similartoworkingmemory:symbolictriples– Structureswithinanepisodeareconnected;separateepisodesaredisconnected

July11,2016 EpisodicMemory(EpMem) 11

(<id0> ^epmem <id4> ^io <id1> ^reward-link <id5> ^smem <id3> ^superstate nil ^svs <id2> ^type state)

(<id1> ^input-link <id7> ^output-link <id6>)

(<id2> ^command <id9> ^spatial-scene <id8>)

(<id3> ^command <id11> ^result <id10>)(<id4> ^command <id13> ^present-id 1 ^result <id12>)(<id8> ^id world)

ID_0

ID_1

io

ID_2

svs

ID_3

smem

ID_4

epmem

ID_5

reward-link

nil

superstate

state

type

ID_6

output-link

ID_7

input-link

ID_8

spatial-scene

ID_9

command

ID_10

result

ID_11

command

ID_12

result

ID_13

command

1

present-id

world

id

StoringKnowledge• AutomaticstoragerequiresEpMem tobeenabled (seeslide12)

• Storagecapturesthetopstateofworkingmemory

• Events triggerstorageofnewepisodes– epmem --set trigger << dc output >>

• dc:decisioncycle(default)• output:newaugmentation ofoutput-link

• Storagetakesplaceattheendofaphase– epmem --set phase << output selection >>

• output isdefault• selectionmaybeuseful forin-the-headagents

July11,2016 EpisodicMemory(EpMem) 12

RetrievingKnowledgeCue-Based

Findtheepisode thatbestmatchesacueandaddittoworkingmemory

TemporalProgressionReplace thecurrentlyretrievedepisodewiththenext/previously encodedepisode

Non-Cue-BasedAddanepisode toworkingmemoryfromepisode #

CommonConstraints:– RequiresthatEpMem isenabled– Onlyoneperstateperdecision– Processedduringphase– Onlyre-processed ifWMchangestocommands– Meta-data(status,etc)automatically cleaned bythearchitecture

July11,2016 EpisodicMemory(EpMem) 13

Cue-BasedRetrieval:Syntax(<epmem> ^command <cmd>)(<cmd> ^query <q>

^neg-query <nq>)

• Theneg-query isoptional• Cuesmustbeacyclic• The<q> and<nq> identifiersformtherootsofepisodesub-graphcues– query representsdesiredstructures– neg-query representsundesiredstructures

July11,2016 EpisodicMemory(EpMem) 14

Cue-BasedRetrieval:CueSemantics

ValuesofcueWMEsareinterpretedbytype– Constant:exactmatch– Long-TermID:exactmatch,stop– Short-TermID:wildcard(butmustbeidentifier)

Cuematchingwillreturnthemostrecentgraph-matchedepisode,orthemostrecentnon-graph-matchedcandidateepisodewiththemaximalepisodescore(w.r.t.root-to-leafpath)

July11,2016 EpisodicMemory(EpMem) 15

Cue-BasedRetrieval:ExampleResult

July11,2016 EpisodicMemory(EpMem) 16

ID_0

ID_1

io

ID_2

svs

ID_3

smem

ID_4

epmem

ID_5

reward-link

nil

superstate

state

type

ID_6

output-link

ID_7

input-link

ID_8

spatial-scene

ID_9

command

ID_10

result

ID_11

command

ID_12

result

ID_13

command

2

present-id

world

id

ID_14

success

ID_15

mapping

ID_16

retrieved

1

cue-size

1

match-cardinality

1

match-score

1

normalized-match-score

1

graph-match

1

memory-id

2

present-id query

nil

superstate

ID_17

io

ID_18

svs

ID_19

smem

ID_20

epmem

ID_21

reward-link

nil

superstate

state

type

ID_22

output-link

ID_23

input-link

ID_24

spatial-scene

ID_25

command

ID_26

result

ID_27

command

ID_28

result

ID_29

command

1

present-id

world

id

ID_30

query

nil

superstate

Cue-BasedRetrievalOptionalModifiers

(<cmd> ^before time-id)(<cmd> ^after time-id)(<cmd> ^prohibit time-id1 time-id2 …)

Hardconstraintsontheepisodesthatcanberetrieved.

July11,2016 EpisodicMemory(EpMem) 17

TemporalProgression

(<cmd> ^next <new-id>)(<cmd> ^previous <new-id>)

Retrievesthenext/previousepisode,temporally,withrespecttothelastthatwasretrieved

July11,2016 EpisodicMemory(EpMem) 18

EpMem Task:VirtualSensingepmem-virtual-sensing.soar

1. Producearandom numberinWMEpMemautomaticallyrecordsthisepisode

2. RemovethenumberfromWMWritetothetrace(forlaterverification)

3. QueryepisodicmemoryWhendidIlastseearandomnumber?

4. ReasonabouttheretrievedepisodeExtractandprintthenumber

July11,2016 19EpisodicMemory(EpMem)

Eaters!

July11,2016 EpisodicMemory(EpMem) 20