the sneps approach to cognitive robotics

S.C. Shapiro

cse@buff

The SNePS Approach to Cognitive Robotics

Stuart C. Shapiro

Department of Computer Science and Engineering

and Center for Cognitive Science

University at Buffalo

shapiro@cse.buffalo.edu

S.C. Shapiro

cse@buff

Outline• Introduction• Intensional Representation & Propositions• SNePS Connectives and Quantifiers• SNeRE Acting Constructs• Example Plans • Representation and Use of Indexicals• A Personal Sense of Time• Summary

S.C. Shapiro

cse@buff

• A computational cognitive agent that can:– Understand and communicate in English; – Discuss specific, generic, and “rule-like” information;– Reason;– Discuss acts and plans;– Sense;– Act;– Remember and report what it has sensed and done.

S.C. Shapiro

cse@buff

Embodied Cassie

• A computational cognitive agent– Embodied in hardware– or Software-Simulated– Based on SNePS and GLAIR.

S.C. Shapiro

cse@buff

SNePS• Knowledge Representation and Reasoning

– Propositions as Terms

• SNIP: SNePS Inference Package– Specialized connectives and quantifiers

• SNeBR: SNePS Belief Revision

• SNeRE: SNePS Rational Engine

• Interface Languages– SNePSUL: Lisp-Like– SNePSLOG: Logic-Like– GATN for Fragments of English.

S.C. Shapiro

cse@buff

GLAIR Architecture

Knowledge Level

Perceptuo-Motor Level

Sensory-Actuator Level

ActionPercepts

Grounded Layered Architecture with Integrated Reasoning

S.C. Shapiro

cse@buff

Interaction with CassieEnglish

(Statement, Question, Command)

(Current) Set of Beliefs[SNePS]

(Updated) Setof Beliefs[SNePS]

Actions[SNeRE]

(New Belief)[SNePS]

English sentence expressingnew belief answering question reporting actions

Answer[SNIP]

GATN Parser

GATN Generator

ReasoningClarification DialogueLooking in World

Reasoning

S.C. Shapiro

cse@buff

Cassie, the BlocksWorld Robot

S.C. Shapiro

cse@buff

Cassie, the FEVAHR

S.C. Shapiro

cse@buff

FEVAHR/Cassie in the Lab

S.C. Shapiro

cse@buff

FEVAHRWorld Simulation

S.C. Shapiro

cse@buff

UXO Remediation Cassie

CassieCorner flag

NonUXO object

Corner flag

Batterymeter

Corner flag

Drop-off zone

Safe zone

RechargingStation

S.C. Shapiro

cse@buff

Crystal Space Environment

S.C. Shapiro

cse@buff

S.C. Shapiro

cse@buff

Entities, Terms, Symbols, Objects

• Cassie’s mental entity: a person named Bill

• SNePS term: B5• Object in world:

S.C. Shapiro

cse@buff

Intensional Representation

Intensional entities are distincteven if coreferential.

“The morning star is the evening star.”

“George IV wondered if Scott was the author of Waverly.”

S.C. Shapiro

cse@buff

McCarthy’s Telephone Number Problem

Mary's telephone number is Mike's telephone number.

I understand that Mike's telephone number is Mary's telephone number.

Pat knew Mike's telephone number.

I understand that Pat knew Mike's telephone number.

Pat dialed Mike's telephone number.

I understand that Pat dialed Mike's telephone number.

S.C. Shapiro

cse@buff

Answering the Telephone Number Problem

Did Pat dial Mary's telephone number?

Yes, Pat dialed Mary's telephone number.

Did Pat know Mary's telephone number?

I don't know.

S.C. Shapiro

cse@buff

Representing Propositions

Propositions must befirst-class entities of the domain

Represented by terms.

S.C. Shapiro

cse@buff

Discussing Propositions

That Bill is sweet is Mary's favorite proposition.

I understand that Mary's favorite proposition is that Bill is sweet.

Mike believes Mary's favorite proposition.

I understand that Mike believes that Bill is sweet.

S.C. Shapiro

cse@buff

S.C. Shapiro

cse@buff

Logic for NLU &Commonsense Reasoning

Either Pat is a man or Pat is a woman or Pat is a robot.

I understand that Pat is a robot or Pat is a woman or Pat is a man.

Pat is a woman.

I understand that Pat is a woman.

What is Pat?

Pat is a woman and Pat is not a robotand Pat is not a man.

S.C. Shapiro

cse@buff

Representation in FOPL?

Man(Pat) Woman(Pat) Robot(Pat)

S.C. Shapiro

cse@buff

but don’t want inclusive or

S.C. Shapiro

cse@buff

but don’t want inclusive or

Man(Pat) Woman(Pat) Robot(Pat)+ +

TSo don’t want exclusive or either

S.C. Shapiro

cse@buff

andor(i, j){P1, ..., Pn}

True iff at least i, and at most j of the Pi are True

S.C. Shapiro

cse@buff

thresh

thresh(i, j){P1, ..., Pn}

True iff either fewer than i,

or more than j

of the Pi are True

Note: thresh(i, j) ~andor(i, j)

S.C. Shapiro

cse@buff

or-entailment

{P1, ..., Pn} v=> {Q1, ..., Qn}

True iff for all i, j Pi Qj

S.C. Shapiro

cse@buff

and-entailment

{P1, ..., Pn} &=> {Q1, ..., Qn}

True iff for all j

P1 &…& Pn Qj

S.C. Shapiro

cse@buff

Numerical entailment

{P1, ..., Pn} i=> {Q1, ..., Qn}

True iff for all j

andor(i, n){P1, …, Pn } Qj

S.C. Shapiro

cse@buff

Universal Quantifier

all(ū)({R1(ū),..., Rn(ū)} &=> {C1(ū),..., Cm(ū)})

Every ā that satisfies

R1(ū)&…& Rn(ū)

also satisfies

C1(ū),..., Cm(ū)})

S.C. Shapiro

cse@buff

Numerical Quantifiers

nexists(i,j,k)(x)

({P1(x),..., Pn(x)}: {Q(x)})}

There are k individuals that satisfy

P1(x) ... Pn(x)

and, of them, at least i and at most j also satisfy

S.C. Shapiro

cse@buff

S.C. Shapiro

cse@buff

MENTAL ACTS

• Believe(proposition)• Disbelieve(proposition)

S.C. Shapiro

cse@buff

Act Selection

• Do-One({act1 ... actn})

• Snif(if(condition, act),

if(condition, act)

[else(act)])

S.C. Shapiro

cse@buff

Act Iteration• Do-All({act1 ... actn})• Sniterate(if(condition, act),

if(condition, act),

[else(act)])• Snsequence(act1, ..., actn)• Cascade(act1, ..., actn)• P-Do-All({act1, ..., act2})

S.C. Shapiro

cse@buff

Entity Iteration

WithSome(var, suchthat,

do, [else])

WithAll(var, suchthat,

do, [else])

WithSome+(var, suchthat,

do, [else])

WithNew(vars, thatare, suchthat,

do, [else])

S.C. Shapiro

cse@buff

Proposition/Act Transformers

• Achieve(proposition)• ActPlan(act, plan)• GoalPlan(proposition, act)• Precondition(act, proposition)• Effect(act, proposition)• WhenDo(proposition, act)• WheneverDo(proposition, act)• IfDo(proposition, act)

S.C. Shapiro

cse@buff

S.C. Shapiro

cse@buff

Conditional Plans If a block is on a support then a plan to achieve

that the support is clear is to pick up the block and then put the block on the table.

all(x, y) ({Block(x), Support(y), On(x, y)} &=> {GoalPlan(Clear(y), Snsequence(Pickup(x), Put(x, Table)))})(STRIPS-like representation: No times)

S.C. Shapiro

cse@buff

Use of Conditional Plan

GoalPlan(Clear(B),

Snsequence(Pickup(A),

Put(A, Table)))Remember (cache) derived propositions.

S.C. Shapiro

cse@buff

Use of Conditional Plan

GoalPlan(Clear(B),

Snsequence(Pickup(A),

Put(A, Table)))???SNeBR to the rescue!

S.C. Shapiro

cse@buff

A FEVAHR Acting Ruleall(a, o) ({Agent(a), Thing(o)}

&=> {Precondition(Follow(a, o), Near(a, o)),

GoalPlan(Near(a, o), Goto(a, o)),

Precondition(Goto(a, o), Lookat(a, o)),

ActPlan(Lookat(a, o), Find(a, o))})

Uses a temporal model.

S.C. Shapiro

cse@buff

Acting According to the Rule

S.C. Shapiro

cse@buff

I found a red robot.I am looking at a red robot.

Follow a red robot.

S.C. Shapiro

cse@buff

I went to a red robot.I am near a red robot.I am following a red robot.

I found a red robot.I am looking at a red robot.

Follow a red robot.

S.C. Shapiro

cse@buff

A Plan for Blowing up UXOsall(a)(Agent(a) =>ActPlan(Blowup(a, UXOs), Act(a,Cascade(SearchforUxo(a),

WithSome+(obj, Near(a, obj), WithNew({ch ex},

{Charge(ch), Explosion(ex)}, Possess(a, ch), Cascade(Place(a, ch, obj), Hide(a),

Waitfor(a, ex), SearchforUxo(a))),

goto(a, SafeZone))))))

S.C. Shapiro

cse@buff

S.C. Shapiro

cse@buff

Representation and Use of Indexicals

• Words whose meanings are determined by occasion of use

• E.g. I, you, now, then, here, there

• Deictic Center <*I, *YOU, *NOW>

• *I: SNePS term representing Cassie

• *YOU: person Cassie is talking with

• *NOW: current time.

S.C. Shapiro

cse@buff

Analysis of Indexicals(in input)

• First person pronouns: *YOU• Second person pronouns: *I• “here”: location of *YOU• Present/Past relative to *NOW.

S.C. Shapiro

cse@buff

Generation of Indexicals

• *I: First person pronouns

• *YOU: Second person pronouns

• *NOW: used to determine tense and aspect.

S.C. Shapiro

cse@buff

Come here.

Use of Indexicals 1

S.C. Shapiro

cse@buff

Come here.I came to you, Stu.I am near you.

Use of Indexicals 2

S.C. Shapiro

cse@buff

Who am I?Your name is ‘Stu’and you are a person.

Who have you talked to?

I am talking to you.Talk to Bill.

I am talking to you, Bill.Come here.

Use of Indexicals 3

S.C. Shapiro

cse@buff

Come here.

I found you.I am looking at you.

Use of Indexicals 4

S.C. Shapiro

cse@buff

Come here.

I came to you.I am near you.

I found you.I am looking at you.

Use of Indexicals 5

S.C. Shapiro

cse@buff

Who am I?

I talked to Stuand I am talking to you.

Your name is ‘Bill’and you are a person.

Who are you?I am the FEVAHRand my name is ‘Cassie’.

Who have you talked to?

Use of Indexicals 6

S.C. Shapiro

cse@buff

S.C. Shapiro

cse@buff

A Personal Sense of Time

• *NOW contains SNePS term representing current time.

• *NOW moves when Cassie acts or perceives a change of state.

S.C. Shapiro

cse@buff

Representation of Time

action object

agentact

eventtime

!!before after before after

?????????????

S.C. Shapiro

cse@buff

Movement of Timet1 t2!before after t3!before after

NOW NOW NOW

S.C. Shapiro

cse@buff

Performing a Punctual Actt1 t3!before after

NOW NOW

t2!before after

S.C. Shapiro

cse@buff

Performing a Durative Actt1

!before after t2

supintsubint

S.C. Shapiro

cse@buff

The Pacemaker

• PML process periodically increments variable COUNT.

• *COUNT = some PML integer.

• Reset to 0 when NOW moves.

• Provides bodily “feel” of passing time.

S.C. Shapiro

cse@buff

Quantizing Time

Cannot conceptualize fine distinctions in time intervals.

So quantize, e.g. into half orders of magnitude (Hobbs, 2000).

S.C. Shapiro

cse@buff

Movement of Time with Pacemaker

NOW COUNT n

t1 t2q

!before after

time duration!

S.C. Shapiro

cse@buff

The Problem of the Fleeting Now

How can you reason about “now”

if it never stands still?

S.C. Shapiro

cse@buff

Fleeting Now Example 1

12:15:00: “Is John having lunch now?”

12:15:02: Agent walks to John’s office.

12:17:00: Agent sees John at his desk, eating.

12:19:00: Agent reports “yes”.

Appropriate granularity.

S.C. Shapiro

cse@buff

Fleeting Now Example 2

12:15:00: “Is John having lunch now?”

Agent knows John is at home without a phone.

Agent contemplates driving to John’s home.

Don’t bother---inappropriate granularity.

S.C. Shapiro

cse@buff

The Vagueness of “now”

I’m now giving a talk.I’m now supervising PhD students.I’m now visiting Paris.I’m now living in Buffalo.The agent is now walking to John’s office.The agent is now seeing if John is eating lunch.

Multiple now’s at different granularities.

S.C. Shapiro

cse@buff

NOW-MTF

Semi-lattice of times, all of which contain *NOW,any of which could be meant by “now”Finite---only conceptualized times of conceptualized states

Maximal Temporal Frame based on *NOW

S.C. Shapiro

cse@buff

Moving NOW with MTF

S.C. Shapiro

cse@buff

Typical Durations

“If the walk light is on now, cross the street.”Relevant duration is typical duration of walk lights.

“Is John having lunch now?”Relevant duration is typical duration of lunch.

Use quantized typical durations when updating NOW-MTFs.

S.C. Shapiro

cse@buff

Using Appropriate Granularity

Lunch time

Lunch? Lunch!

S.C. Shapiro

cse@buff

S.C. Shapiro

cse@buff

• A computational cognitive agent/robot

• That can communicate in natural language.

S.C. Shapiro

cse@buff

Intensional Representation& Propositions

• SNePS terms represent mental entities.

• May assert that two entities are coreferential.

• Relations/acts may be declared transparent.

• Propositions are first-class entities.

S.C. Shapiro

cse@buff

SNePS Connectives and Quantifiers

• Designed logical connectivesand rules of inference

More appropriate for NLU and Commonsense reasoningthan in standard FOPC.

S.C. Shapiro

cse@buff

SNeRE Acting Constructs

• Separate, but Coordinated

Syntax and Semantics

For Acting and for Reasoning

S.C. Shapiro

cse@buff

Representation and Useof Indexicals

• Use of Deictic Center for parser to interpret indexicals as current referents

• And for generator to generate indexicals from current referents.

S.C. Shapiro

cse@buff

A Personal Sense of Time

• *NOW is current time.

• Updated when Cassie acts

or perceives a change of state.

• Points into MTF to support vagueness of “now”.

S.C. Shapiro

cse@buff

For More Information

• Personnel

• Manual

• Tutorial

• Bibliography

• ftp’able SNePS source code

• etc.

• http://www.cse.buffalo.edu/sneps/

the sneps approach to cognitive robotics

Documents

15-494 cognitive robotics 03/30/09 1 manipulation by pushing...

cse@buffalo s.c. shapiro knowledge representation and...

meta-morphogenesis theory as background to cognitive...

challenge 2 cognitive systems and robotics€¦ · 5...

the glair architecture for cognitive robotics

how robotics and cognitive automation will transform the...

6th international cognitive robotics...

cognitive robotics - intranet...

introduction to robotics cognitive robotics 2007/2008 period...

a visual imagination approach to cognitive robotics

cassie as a self-aware sneps/glair agent

notes on cognitive robotics 1machine psychology...

reinventing the reinvented shakey in sneps

[h2020] new horizon of european robotics and cognitive...

cognitive robotics: robustness through model-based … ·...

inconsistency tolerance in sneps

cognitive robotics slam with lasers - politecnico di...

cognitive and neural aspects in robotics with applications

cognitive robotics in industrial...

gpu computing for cognitive robotics