raciocínio e percepção espacial: uma abordagem lógicapsantos/slidesmc5.pdf · raciocínio e...
TRANSCRIPT
Raciocínio e Percepção Espacial: Uma abordagemlógica
Paulo Santos
FEI - São Paulo
September 10, 2010
Paulo Santos ( FEI - São Paulo ) September 10, 2010 1 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 2 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 3 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 4 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 5 / 136
Where is FEI?
S. Paulo, SPFEI Campus
FEI is the largest engineering school in Brazil, with over 8,000studentsit is already a regional centre of scientific development for theautomotive industry and started investing intensively to becomealso a regional centre for intelligent robotics.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 6 / 136
Where is FEI?
S. Paulo, SPFEI Campus
FEI is the largest engineering school in Brazil, with over 8,000studentsit is already a regional centre of scientific development for theautomotive industry and started investing intensively to becomealso a regional centre for intelligent robotics.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 6 / 136
Where is FEI?
S. Paulo, SPFEI Campus
FEI is the largest engineering school in Brazil, with over 8,000studentsit is already a regional centre of scientific development for theautomotive industry and started investing intensively to becomealso a regional centre for intelligent robotics.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 6 / 136
Where is FEI?
S. Paulo, SPFEI Campus
FEI is the largest engineering school in Brazil, with over 8,000studentsit is already a regional centre of scientific development for theautomotive industry and started investing intensively to becomealso a regional centre for intelligent robotics.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 6 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 7 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 8 / 136
Introduction and motivation
reasoning about space is ubiquitous: it is necessary in situationsfrom tying a shoe-lace to urban traffic navigation;automation of spatial reasoning has led to the development of anumber of application domains:
I geographical information systems (GIS)I roboticsI commonsense reasoningI natural language processingI virtual world modelling and animationI medical analysis and diagnosis systemsI computer vision
Paulo Santos ( FEI - São Paulo ) September 10, 2010 9 / 136
Introduction and motivation
reasoning about space is ubiquitous: it is necessary in situationsfrom tying a shoe-lace to urban traffic navigation;automation of spatial reasoning has led to the development of anumber of application domains:
I geographical information systems (GIS)I roboticsI commonsense reasoningI natural language processingI virtual world modelling and animationI medical analysis and diagnosis systemsI computer vision
Paulo Santos ( FEI - São Paulo ) September 10, 2010 9 / 136
Introduction and motivation
We acquire knowledge about spatial relationships mainly in two ways:sensory processing: intensively studied in Computer Vision andRobotics;being told, or reading, about spatial arrangements (high-levelreasoning). This is the kind of information processing we’reconcerned about here.
In other words, we’ll be talking about qualitative reasoning, in contrastto numerical processing.We will also present computer vision systems whose aim is to bridgethe gap between sensory processing and high-level reasoning:Cognitive Vision systems.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 10 / 136
Introduction and motivation
We acquire knowledge about spatial relationships mainly in two ways:sensory processing: intensively studied in Computer Vision andRobotics;being told, or reading, about spatial arrangements (high-levelreasoning). This is the kind of information processing we’reconcerned about here.
In other words, we’ll be talking about qualitative reasoning, in contrastto numerical processing.We will also present computer vision systems whose aim is to bridgethe gap between sensory processing and high-level reasoning:Cognitive Vision systems.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 10 / 136
Introduction and motivation
We acquire knowledge about spatial relationships mainly in two ways:sensory processing: intensively studied in Computer Vision andRobotics;being told, or reading, about spatial arrangements (high-levelreasoning). This is the kind of information processing we’reconcerned about here.
In other words, we’ll be talking about qualitative reasoning, in contrastto numerical processing.We will also present computer vision systems whose aim is to bridgethe gap between sensory processing and high-level reasoning:Cognitive Vision systems.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 10 / 136
Aim and inspiration for this tutorial
We follow the believe expressed in Takeo Kanade KeynoteLecture (given at IJCAI 2003): it is now the time to combineComputer Vision with Relational models and Reasoning.This tutorial presents:
I tools and methodology of QSR;I an overview of major QSR calculi;I an overview of Cognitive Vision Systems;I examples of QSR systems for Cognitive Vision.
Aim: make a brief overview of these areas, presenting the contextand foundations to kick start new projects.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 11 / 136
Aim and inspiration for this tutorial
We follow the believe expressed in Takeo Kanade KeynoteLecture (given at IJCAI 2003): it is now the time to combineComputer Vision with Relational models and Reasoning.This tutorial presents:
I tools and methodology of QSR;I an overview of major QSR calculi;I an overview of Cognitive Vision Systems;I examples of QSR systems for Cognitive Vision.
Aim: make a brief overview of these areas, presenting the contextand foundations to kick start new projects.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 11 / 136
Aim and inspiration for this tutorial
We follow the believe expressed in Takeo Kanade KeynoteLecture (given at IJCAI 2003): it is now the time to combineComputer Vision with Relational models and Reasoning.This tutorial presents:
I tools and methodology of QSR;I an overview of major QSR calculi;I an overview of Cognitive Vision Systems;I examples of QSR systems for Cognitive Vision.
Aim: make a brief overview of these areas, presenting the contextand foundations to kick start new projects.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 11 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 12 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 13 / 136
Why a “logic-based approach”?
has a semanticslarge variety of distinct logics for different kinds of reasoningvariety of inference mechanismsprovides a tool kit from which it is possible to characterise thecalculideveloping relational calculi is well understood
Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136
Why a “logic-based approach”?
has a semanticslarge variety of distinct logics for different kinds of reasoningvariety of inference mechanismsprovides a tool kit from which it is possible to characterise thecalculideveloping relational calculi is well understood
Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136
Why a “logic-based approach”?
has a semanticslarge variety of distinct logics for different kinds of reasoningvariety of inference mechanismsprovides a tool kit from which it is possible to characterise thecalculideveloping relational calculi is well understood
Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136
Why a “logic-based approach”?
has a semanticslarge variety of distinct logics for different kinds of reasoningvariety of inference mechanismsprovides a tool kit from which it is possible to characterise thecalculideveloping relational calculi is well understood
Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136
Why a “logic-based approach”?
has a semanticslarge variety of distinct logics for different kinds of reasoningvariety of inference mechanismsprovides a tool kit from which it is possible to characterise thecalculideveloping relational calculi is well understood
Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136
Why a “logic-based approach”?
has a semanticslarge variety of distinct logics for different kinds of reasoningvariety of inference mechanismsprovides a tool kit from which it is possible to characterise thecalculideveloping relational calculi is well understood
Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136
Logical Reasoning
Every man is mortal.Socrates is a man.Ergo: Socrates is mortal.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 15 / 136
Logical Reasoning
Every man is mortal.Socrates is a man.Ergo: Socrates is mortal.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 15 / 136
Logical Reasoning
Every man is mortal.Socrates is a man.Ergo: Socrates is mortal.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 15 / 136
Logical Reasoning
Every man is mortal.Socrates is a man.Ergo: Socrates is mortal.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 15 / 136
Logical Reasoning
∀x m(x)→ mo(x).
m(Socrates).Ergo: mo(Socrates).
MODUS PONENS: A→B, AB
Paulo Santos ( FEI - São Paulo ) September 10, 2010 16 / 136
Logical Reasoning
∀x m(x)→ mo(x).
m(Socrates).Ergo: mo(Socrates).
MODUS PONENS: A→B, AB
Paulo Santos ( FEI - São Paulo ) September 10, 2010 16 / 136
Logic programming
m(X ) :− mo(X ).
m(s).?- m(X )
?- X = s
Computational processes: resolution, model checking, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 17 / 136
Logic programming
m(X ) :− mo(X ).
m(s).?- m(X )
?- X = s
Computational processes: resolution, model checking, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 17 / 136
Logic programming
m(X ) :− mo(X ).
m(s).?- m(X )
?- X = s
Computational processes: resolution, model checking, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 17 / 136
Logic programming
m(X ) :− mo(X ).
m(s).?- m(X )
?- X = s
Computational processes: resolution, model checking, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 17 / 136
Logical Reasoning
Deduction: Inferring logical truthsI All the beans from this bag are white.I These beans are from this bag.I Ergo, these beans are white. (result)I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusionsI I took one bean from this bag and it is whiteI Ergo, all the beans from this bag are white.I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examplesI one bean from this bag is white;I another bean from this bag is white;I another bean from this bag is white;I ...I Ergo, all the beans from this bag are white.I Progol, HR, Claudien, FOIL, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136
Logical Reasoning
Deduction: Inferring logical truthsI All the beans from this bag are white.I These beans are from this bag.I Ergo, these beans are white. (result)I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusionsI I took one bean from this bag and it is whiteI Ergo, all the beans from this bag are white.I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examplesI one bean from this bag is white;I another bean from this bag is white;I another bean from this bag is white;I ...I Ergo, all the beans from this bag are white.I Progol, HR, Claudien, FOIL, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136
Logical Reasoning
Deduction: Inferring logical truthsI All the beans from this bag are white.I These beans are from this bag.I Ergo, these beans are white. (result)I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusionsI I took one bean from this bag and it is whiteI Ergo, all the beans from this bag are white.I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examplesI one bean from this bag is white;I another bean from this bag is white;I another bean from this bag is white;I ...I Ergo, all the beans from this bag are white.I Progol, HR, Claudien, FOIL, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136
Logical Reasoning
Deduction: Inferring logical truthsI All the beans from this bag are white.I These beans are from this bag.I Ergo, these beans are white. (result)I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusionsI I took one bean from this bag and it is whiteI Ergo, all the beans from this bag are white.I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examplesI one bean from this bag is white;I another bean from this bag is white;I another bean from this bag is white;I ...I Ergo, all the beans from this bag are white.I Progol, HR, Claudien, FOIL, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136
Logical Reasoning
Deduction: Inferring logical truthsI All the beans from this bag are white.I These beans are from this bag.I Ergo, these beans are white. (result)I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusionsI I took one bean from this bag and it is whiteI Ergo, all the beans from this bag are white.I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examplesI one bean from this bag is white;I another bean from this bag is white;I another bean from this bag is white;I ...I Ergo, all the beans from this bag are white.I Progol, HR, Claudien, FOIL, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136
Logical Reasoning
Deduction: Inferring logical truthsI All the beans from this bag are white.I These beans are from this bag.I Ergo, these beans are white. (result)I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusionsI I took one bean from this bag and it is whiteI Ergo, all the beans from this bag are white.I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examplesI one bean from this bag is white;I another bean from this bag is white;I another bean from this bag is white;I ...I Ergo, all the beans from this bag are white.I Progol, HR, Claudien, FOIL, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136
Logical Reasoning
Deduction: Inferring logical truthsI All the beans from this bag are white.I These beans are from this bag.I Ergo, these beans are white. (result)I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusionsI I took one bean from this bag and it is whiteI Ergo, all the beans from this bag are white.I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examplesI one bean from this bag is white;I another bean from this bag is white;I another bean from this bag is white;I ...I Ergo, all the beans from this bag are white.I Progol, HR, Claudien, FOIL, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136
Logical Reasoning
Deduction: Inferring logical truthsI All the beans from this bag are white.I These beans are from this bag.I Ergo, these beans are white. (result)I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusionsI I took one bean from this bag and it is whiteI Ergo, all the beans from this bag are white.I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examplesI one bean from this bag is white;I another bean from this bag is white;I another bean from this bag is white;I ...I Ergo, all the beans from this bag are white.I Progol, HR, Claudien, FOIL, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136
Logical Reasoning
Deduction: Inferring logical truthsI All the beans from this bag are white.I These beans are from this bag.I Ergo, these beans are white. (result)I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusionsI I took one bean from this bag and it is whiteI Ergo, all the beans from this bag are white.I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examplesI one bean from this bag is white;I another bean from this bag is white;I another bean from this bag is white;I ...I Ergo, all the beans from this bag are white.I Progol, HR, Claudien, FOIL, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136
Logical Reasoning
Deduction: Inferring logical truthsI All the beans from this bag are white.I These beans are from this bag.I Ergo, these beans are white. (result)I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusionsI I took one bean from this bag and it is whiteI Ergo, all the beans from this bag are white.I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examplesI one bean from this bag is white;I another bean from this bag is white;I another bean from this bag is white;I ...I Ergo, all the beans from this bag are white.I Progol, HR, Claudien, FOIL, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136
Logical Reasoning
Deduction: Inferring logical truthsI All the beans from this bag are white.I These beans are from this bag.I Ergo, these beans are white. (result)I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusionsI I took one bean from this bag and it is whiteI Ergo, all the beans from this bag are white.I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examplesI one bean from this bag is white;I another bean from this bag is white;I another bean from this bag is white;I ...I Ergo, all the beans from this bag are white.I Progol, HR, Claudien, FOIL, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136
Logical Reasoning
Deduction: Inferring logical truthsI All the beans from this bag are white.I These beans are from this bag.I Ergo, these beans are white. (result)I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusionsI I took one bean from this bag and it is whiteI Ergo, all the beans from this bag are white.I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examplesI one bean from this bag is white;I another bean from this bag is white;I another bean from this bag is white;I ...I Ergo, all the beans from this bag are white.I Progol, HR, Claudien, FOIL, ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136
Knowledge Representation and Reasoning
the logic formalisation of reasoning processes, capable of inferringknowledge from representations of the world;the construction of a medium for efficient computation, in whichthe formal representation provides the means to organisedomain knowledge allowing for efficient (and consistent)queries, updates and revisions of the knowledge base;the rigorous treatment of ontological commitments, which providethe base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actionsI nature of knowledge about temporal entitiesI belief changeI vaguenessI spatial entitiesI Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993
Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136
Knowledge Representation and Reasoning
the logic formalisation of reasoning processes, capable of inferringknowledge from representations of the world;the construction of a medium for efficient computation, in whichthe formal representation provides the means to organisedomain knowledge allowing for efficient (and consistent)queries, updates and revisions of the knowledge base;the rigorous treatment of ontological commitments, which providethe base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actionsI nature of knowledge about temporal entitiesI belief changeI vaguenessI spatial entitiesI Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993
Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136
Knowledge Representation and Reasoning
the logic formalisation of reasoning processes, capable of inferringknowledge from representations of the world;the construction of a medium for efficient computation, in whichthe formal representation provides the means to organisedomain knowledge allowing for efficient (and consistent)queries, updates and revisions of the knowledge base;the rigorous treatment of ontological commitments, which providethe base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actionsI nature of knowledge about temporal entitiesI belief changeI vaguenessI spatial entitiesI Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993
Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136
Knowledge Representation and Reasoning
the logic formalisation of reasoning processes, capable of inferringknowledge from representations of the world;the construction of a medium for efficient computation, in whichthe formal representation provides the means to organisedomain knowledge allowing for efficient (and consistent)queries, updates and revisions of the knowledge base;the rigorous treatment of ontological commitments, which providethe base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actionsI nature of knowledge about temporal entitiesI belief changeI vaguenessI spatial entitiesI Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993
Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136
Knowledge Representation and Reasoning
the logic formalisation of reasoning processes, capable of inferringknowledge from representations of the world;the construction of a medium for efficient computation, in whichthe formal representation provides the means to organisedomain knowledge allowing for efficient (and consistent)queries, updates and revisions of the knowledge base;the rigorous treatment of ontological commitments, which providethe base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actionsI nature of knowledge about temporal entitiesI belief changeI vaguenessI spatial entitiesI Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993
Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136
Knowledge Representation and Reasoning
the logic formalisation of reasoning processes, capable of inferringknowledge from representations of the world;the construction of a medium for efficient computation, in whichthe formal representation provides the means to organisedomain knowledge allowing for efficient (and consistent)queries, updates and revisions of the knowledge base;the rigorous treatment of ontological commitments, which providethe base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actionsI nature of knowledge about temporal entitiesI belief changeI vaguenessI spatial entitiesI Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993
Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136
Knowledge Representation and Reasoning
the logic formalisation of reasoning processes, capable of inferringknowledge from representations of the world;the construction of a medium for efficient computation, in whichthe formal representation provides the means to organisedomain knowledge allowing for efficient (and consistent)queries, updates and revisions of the knowledge base;the rigorous treatment of ontological commitments, which providethe base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actionsI nature of knowledge about temporal entitiesI belief changeI vaguenessI spatial entitiesI Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993
Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136
Knowledge Representation and Reasoning
the logic formalisation of reasoning processes, capable of inferringknowledge from representations of the world;the construction of a medium for efficient computation, in whichthe formal representation provides the means to organisedomain knowledge allowing for efficient (and consistent)queries, updates and revisions of the knowledge base;the rigorous treatment of ontological commitments, which providethe base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actionsI nature of knowledge about temporal entitiesI belief changeI vaguenessI spatial entitiesI Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993
Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136
Knowledge Representation and Reasoning
the logic formalisation of reasoning processes, capable of inferringknowledge from representations of the world;the construction of a medium for efficient computation, in whichthe formal representation provides the means to organisedomain knowledge allowing for efficient (and consistent)queries, updates and revisions of the knowledge base;the rigorous treatment of ontological commitments, which providethe base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actionsI nature of knowledge about temporal entitiesI belief changeI vaguenessI spatial entitiesI Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993
Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 20 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 21 / 136
What is QSR?
The formal representation of (qualitative) spatial knowledge in terms ofsome basic entities and primitive relations in order to allow meaningfuland, sometimes, efficient inference methods about space.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 22 / 136
Spatial Reasoning
“The basic stories we know best are small stories of events in space:The wind blows clouds through the sky, a child throws a rock, a motherpours milk into a glass, a whale swims through the water. Thesestories constitute our world.” (M.Turner, The Literary Mind)
Paulo Santos ( FEI - São Paulo ) September 10, 2010 23 / 136
Philosophical origins of QSR
the fundamental principles of Geometry were first investigated inancient Greece by Thales circa 600 B.C.)the laws of valid argument in terms of logical modes of inferencewere studied separately by early Greek philosophers:analytic geometry (Descartes 1637)19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolicgeometry (1829)
I Cantor’s point-set topology (1845-1918):I Poincaré’s algebraic topology
Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136
Philosophical origins of QSR
the fundamental principles of Geometry were first investigated inancient Greece by Thales circa 600 B.C.)the laws of valid argument in terms of logical modes of inferencewere studied separately by early Greek philosophers:analytic geometry (Descartes 1637)19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolicgeometry (1829)
I Cantor’s point-set topology (1845-1918):I Poincaré’s algebraic topology
Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136
Philosophical origins of QSR
the fundamental principles of Geometry were first investigated inancient Greece by Thales circa 600 B.C.)the laws of valid argument in terms of logical modes of inferencewere studied separately by early Greek philosophers:analytic geometry (Descartes 1637)19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolicgeometry (1829)
I Cantor’s point-set topology (1845-1918):I Poincaré’s algebraic topology
Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136
Philosophical origins of QSR
the fundamental principles of Geometry were first investigated inancient Greece by Thales circa 600 B.C.)the laws of valid argument in terms of logical modes of inferencewere studied separately by early Greek philosophers:analytic geometry (Descartes 1637)19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolicgeometry (1829)
I Cantor’s point-set topology (1845-1918):I Poincaré’s algebraic topology
Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136
Philosophical origins of QSR
the fundamental principles of Geometry were first investigated inancient Greece by Thales circa 600 B.C.)the laws of valid argument in terms of logical modes of inferencewere studied separately by early Greek philosophers:analytic geometry (Descartes 1637)19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolicgeometry (1829)
I Cantor’s point-set topology (1845-1918):I Poincaré’s algebraic topology
Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136
Philosophical origins of QSR
the fundamental principles of Geometry were first investigated inancient Greece by Thales circa 600 B.C.)the laws of valid argument in terms of logical modes of inferencewere studied separately by early Greek philosophers:analytic geometry (Descartes 1637)19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolicgeometry (1829)
I Cantor’s point-set topology (1845-1918):I Poincaré’s algebraic topology
Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136
Philosophical origins of QSR
the fundamental principles of Geometry were first investigated inancient Greece by Thales circa 600 B.C.)the laws of valid argument in terms of logical modes of inferencewere studied separately by early Greek philosophers:analytic geometry (Descartes 1637)19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolicgeometry (1829)
I Cantor’s point-set topology (1845-1918):I Poincaré’s algebraic topology
Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136
Philosophical origins of QSR
the fundamental principles of Geometry were first investigated inancient Greece by Thales circa 600 B.C.)the laws of valid argument in terms of logical modes of inferencewere studied separately by early Greek philosophers:analytic geometry (Descartes 1637)19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolicgeometry (1829)
I Cantor’s point-set topology (1845-1918):I Poincaré’s algebraic topology
Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136
Philosophical origins of QSR
the fundamental principles of Geometry were first investigated inancient Greece by Thales circa 600 B.C.)the laws of valid argument in terms of logical modes of inferencewere studied separately by early Greek philosophers:analytic geometry (Descartes 1637)19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolicgeometry (1829)
I Cantor’s point-set topology (1845-1918):I Poincaré’s algebraic topology
Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136
Philosophical origins of QSR: 20th century
Bertrand Russell and Alfred Whitehead: principles of logic tophenomenological theories, describing the world as it is perceivedthrough sense data;Whitehead (1920): a theory of the perceived world should have asbasic entities the very ’phenomena’: integral objects or events
I geometry becomes concerned with relationships between regionsoccupied by bodies and dynamical laws with qualitative rules aboutworld events
Paulo Santos ( FEI - São Paulo ) September 10, 2010 25 / 136
Philosophical origins of QSR: 20th century
Bertrand Russell and Alfred Whitehead: principles of logic tophenomenological theories, describing the world as it is perceivedthrough sense data;Whitehead (1920): a theory of the perceived world should have asbasic entities the very ’phenomena’: integral objects or events
I geometry becomes concerned with relationships between regionsoccupied by bodies and dynamical laws with qualitative rules aboutworld events
Paulo Santos ( FEI - São Paulo ) September 10, 2010 25 / 136
Philosophical origins of QSR: 20th century
Bertrand Russell and Alfred Whitehead: principles of logic tophenomenological theories, describing the world as it is perceivedthrough sense data;Whitehead (1920): a theory of the perceived world should have asbasic entities the very ’phenomena’: integral objects or events
I geometry becomes concerned with relationships between regionsoccupied by bodies and dynamical laws with qualitative rules aboutworld events
Paulo Santos ( FEI - São Paulo ) September 10, 2010 25 / 136
Origins of QSR in Artificial Intelligence
phenomenological theories: closer to human reasoningwould it be possible to automate them ?at least for some particular domains?the construction of formal theories about the qualitativerelationships between basic spatial (“phenomenological”) theoriesis the main goal of qualitative spatial reasoning in AI.
but perhaps we should talk about time before talking aboutspace...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 26 / 136
Origins of QSR in Artificial Intelligence
phenomenological theories: closer to human reasoningwould it be possible to automate them ?at least for some particular domains?the construction of formal theories about the qualitativerelationships between basic spatial (“phenomenological”) theoriesis the main goal of qualitative spatial reasoning in AI.
but perhaps we should talk about time before talking aboutspace...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 26 / 136
Origins of QSR in Artificial Intelligence
phenomenological theories: closer to human reasoningwould it be possible to automate them ?at least for some particular domains?the construction of formal theories about the qualitativerelationships between basic spatial (“phenomenological”) theoriesis the main goal of qualitative spatial reasoning in AI.
but perhaps we should talk about time before talking aboutspace...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 26 / 136
Origins of QSR in Artificial Intelligence
phenomenological theories: closer to human reasoningwould it be possible to automate them ?at least for some particular domains?the construction of formal theories about the qualitativerelationships between basic spatial (“phenomenological”) theoriesis the main goal of qualitative spatial reasoning in AI.
but perhaps we should talk about time before talking aboutspace...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 26 / 136
Origins of QSR in Artificial Intelligence
phenomenological theories: closer to human reasoningwould it be possible to automate them ?at least for some particular domains?the construction of formal theories about the qualitativerelationships between basic spatial (“phenomenological”) theoriesis the main goal of qualitative spatial reasoning in AI.
but perhaps we should talk about time before talking aboutspace...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 26 / 136
Allen’s interval calculus
Figure: adapted from M. Ragni, Reasoning in Dynamic Environments, KI2006
13 relationsany two intervals expressed by one and only one relation (JEPD)temporal reasoning: derive facts about temporal intervals
Paulo Santos ( FEI - São Paulo ) September 10, 2010 27 / 136
Allen’s interval calculus: applications
represent activities and temporal knowledgeplanning and schedulingtemporal databasesnatural language understanding
Paulo Santos ( FEI - São Paulo ) September 10, 2010 28 / 136
Tools of temporal reasoning
Conceptual Neighbourhood Diagrams: graphs representing intheir vertices relations on some specific objects; whereas theiredges represent continuous transitions between these relations.
I Continuous transitions: in between adjacent vertices of the graphthere is no other relations that the entities in the relation’sarguments can assume.
Composition (or transitivity) tables: given two relations on anyobjects a, b, ad c (e.g., R1(a,b) and R2(b, c)), the compositiontable entry for R1(a,b) and R2(b, c) gives the minimal set ofdisjunctions R3(b, c) of the possible relations between a and c.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 29 / 136
Tools of temporal reasoning
Conceptual Neighbourhood Diagrams: graphs representing intheir vertices relations on some specific objects; whereas theiredges represent continuous transitions between these relations.
I Continuous transitions: in between adjacent vertices of the graphthere is no other relations that the entities in the relation’sarguments can assume.
Composition (or transitivity) tables: given two relations on anyobjects a, b, ad c (e.g., R1(a,b) and R2(b, c)), the compositiontable entry for R1(a,b) and R2(b, c) gives the minimal set ofdisjunctions R3(b, c) of the possible relations between a and c.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 29 / 136
Tools of temporal reasoning
Conceptual Neighbourhood Diagrams: graphs representing intheir vertices relations on some specific objects; whereas theiredges represent continuous transitions between these relations.
I Continuous transitions: in between adjacent vertices of the graphthere is no other relations that the entities in the relation’sarguments can assume.
Composition (or transitivity) tables: given two relations on anyobjects a, b, ad c (e.g., R1(a,b) and R2(b, c)), the compositiontable entry for R1(a,b) and R2(b, c) gives the minimal set ofdisjunctions R3(b, c) of the possible relations between a and c.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 29 / 136
Conceptual Neighbourhood Diagram: illustration
Figure: adapted from M. Ragni, Reasoning in Dynamic Environments, KI2006
Paulo Santos ( FEI - São Paulo ) September 10, 2010 30 / 136
Transitivity table
< > d di ...before (<) < no info <; o; m; d;
s< ...
after (>) no info > >; oi; mi;d; f
> ...
during (d) < > no info. d ...contains(di)
<; o; m; di;fi
>; oi; di;mi; si
o; oi; dur;con; =
di ...
overlaps(o)
< >; oi; di;mi; si
o; d; s <; o; m ...
overlappedby (oi)
<; o; m; di;fi
> oi; d; f >; oi; mi;di;si
...
meets (m) < >; oi; mi;di; si
o; d; s < ...
met-by(mi)
<; o; m; di;fi
> d <; o; m; di;fi
...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 31 / 136
Basic elements in a spatial theory
space: absolute or relative; global or local;basic entities: points, regions, directions, bodies, shapes, things,sense-data, ...primitive relations: meet, between, connect, part-of, ...
I the set of relations is usually Jointly Exhaustive and PairwiseDisjoints (JEPD)
formal tools: axiomatic (deriving axioms and proving theorems),algebraic (encode knowledge with operators and equations),purely logical (design of a spatial logic)
Paulo Santos ( FEI - São Paulo ) September 10, 2010 32 / 136
Tools of QSR
Conceptual Neighbourhood DiagramsComposition (or transitivity) tables
Paulo Santos ( FEI - São Paulo ) September 10, 2010 33 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 34 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 35 / 136
Region Connection Calculus (RCC)
many sorted, first order logic axiomatisation of spatial regionsbased on a primitive binary relation about the connection betweentwo regions (C/2).C(x , y) (“x is connected to y”), i.e., the topological closures of xand y share at least one point
∀x C(x , x);∀xy C(x , y)→ C(y , x);∀xyz (C(z, x)↔ C(z, y))→ x = y .
Paulo Santos ( FEI - São Paulo ) September 10, 2010 36 / 136
Region Connection Calculus
P(x , y), “x is part of y”;O(x , y), “x overlaps y”;PP(x , y), “x is proper part of y”;Pi/2 and PPi/2 are inverse relations of P/2 and PP/2, resp;DC(x , y), “x is disconnected from y”;EQ(x , y), “x is equal to y”;PO(x , y), “x partially overlaps y”;EC(x , y), “x is externally connected to y”;TPP(x , y), “x is tangencial proper part of y”;NTPP(x , y), “x is non-tangential proper part of y”;TPPi/2 NTPPi/2 are inverses of TPP/2 and NTPP/2
{DC(x , y), EQ(x , y), PO(x , y), EC(x , y), TPP(x , y), NTPP(x , y),TPPi/2, NTPPi/2 } is a JEPD set known as RCC8.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 37 / 136
RCC: conceptual neighbourhood diagram
x
y
x
x y
x
y
x
y
x
y
y yx
xy
DC EC PO
TPP
TPPi
EQNTPP
NTPPi
Paulo Santos ( FEI - São Paulo ) September 10, 2010 38 / 136
Region Connection Calculus
((DC(x , y)↔ ¬C(x , y))). (1)((P(x , y)↔ (∀z(C(z, x)→ C(z, y))))). (2)((PP(x , y)↔ (P(x , y) ∧ ¬P(y , x)))). (3)((EQ(x , y)↔ (P(x , y) ∧ P(y , x)))). (4)((O(x , y)↔ (∃z(P(z, x) ∧ P(z, y))))). (5)((PO(x , y)↔ (O(x , y) ∧ ¬P(x , y) ∧ ¬P(y , x)))). (6)((DR(x , y)↔ −O(x , y))). (7)((EC(x , y)↔ (C(x , y) ∧ ¬O(x , y)))). (8)((TPP(x , y)↔ (PP(x , y) ∧ (∃z(EC(z, x) ∧ EC(z, y)))))). (9)((NTPP(x , y)↔ (PP(x , y) ∧ ¬(∃z(EC(z, x) ∧ EC(z, y)))))). (10)((Pi(x , y)↔ P(y , x))). (11)((PPi(x , y)↔ PP(y , x))). (12)((TPPi(x , y)↔ TPP(y , x))). (13)((NTPPi(x , y)↔ NTPP(y , x))). (14)Paulo Santos ( FEI - São Paulo ) September 10, 2010 39 / 136
RCC: Transitivity table
DC EC PO TPP ...DC no info. DR,PO,
PPDR,PO,PP
DR,PO,PP ...
EC DR,PO,PPi
DR,PO,TPP
DR,PO,PP
EC,PO,PP
...
PO DR,PO,PPi
DR,PO,PPi
no info. PO,PP ...
TPP DC DR DR,PO,PP
PP ...
NTPP DC DC DR,PO,PP
NTPP ...
TPPi DR,PO,PPi
EC,PO,PPi
PO,PPi PO,TPP ...
NTPPi DR,PO,PPi
PO,PPi PO,PPi PO,PPi ...
EQ DC EC PO TPP ...
Paulo Santos ( FEI - São Paulo ) September 10, 2010 40 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 41 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 42 / 136
Lines of Sight Calculus
Represent the relative positions between pairs of (non-overlapping)convex bodies
Figure: Randell et a. From Images to Bodies: Modelling and Exploiting SpatialOcclusion and Motion Parallax. IJCAI 2001
Paulo Santos ( FEI - São Paulo ) September 10, 2010 43 / 136
Lines of Sight Calculus
PHI
BA
JFI
BA
EHI
B A
FI
BA
HI
AB
JH
BA
PH
AB
JF
AB
EH
A B
F
AB
AB
AB
JHI
BA
HA
B
C
JC
Paulo Santos ( FEI - São Paulo ) September 10, 2010 44 / 136
Lines of Sight Calculus
14 relations:I C(x , y), is clear from;I JC(x , y), is just clear from;I PH(x , y), partially hides;I PHI(x , y), is partially hidden by;I JH(x , y), just hides;I JHI(x , y), is just hidden;I H(x , y), hides;I HI(x , y), is hidden by;I EH(x , y), exactly hides;I EHI(x , y), is exactly hidden;I F (x , y), is in front of;I FI(x , y), has y in front of it;I JF (x , y), is just in front of;I JFI(x , y), has y just in front of it;
Paulo Santos ( FEI - São Paulo ) September 10, 2010 45 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 46 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 47 / 136
Region Occlusion Calculus (ROC)
Region Occlusion Calculus (ROC) is an extension of RCC torepresent the various possibilities of interposition (occlusion)between arbitrary shaped bodies.Two functions: region and image.
I region: maps a physical body to is occupancy region.I image maps a physical body to its bi-dimensional projection from a
particular viewpoint.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 48 / 136
Region Occlusion Calculus (ROC)
ROC primitive relations: C/2 and TotallyOccludes(x , y , ν) (“body xtotally ocludes body y from the viewpoint ν”).
∀x∀ν¬TotallyOccludes(x , x , ν)∀x∀y∀z∀νTotallyOccludes(x , y , ν) ∧ TotallyOccludes(y , z, ν))→
TotallyOccludes(x , z, ν)
Paulo Santos ( FEI - São Paulo ) September 10, 2010 49 / 136
Region Occlusion Calculus
The following axioms introduce RCC8 in ROC:.
((TotallyOccludes(x , y , ν) ∧ P(region(z), region(y)))→TotallyOccludes(x , z, ν))
(TotallyOccludes(x , y , ν)→ ∀z(P(region(z), region(y)))→¬TotallyOccludes(z, x , ν))
(TotallyOccludes(x , y , ν)→ ∀z∀u(P(region(z), region(x))∧P(region(u), region(y)))→ ¬TotallyOccludes(u, z, ν))
∃y∃z(P(region(y), region(x)) ∧ P(region(z), region(x))∧TotallyOccludes(y , z, ν))
(TotallyOccludes(x , y , ν)→ P(image(y , ν), image(x , ν)))
Paulo Santos ( FEI - São Paulo ) September 10, 2010 50 / 136
Region Occlusion Calculus
We can introduce a weaker notion of occlusion Occludes/3:
Occludes(x , y , ν)↔ ∃z∃u(P(region(z), region(x))
∧ P(region(u), region(y))∧TotallyOccludes(z,u, ν))
Paulo Santos ( FEI - São Paulo ) September 10, 2010 51 / 136
Region Occlusion Calculus
Non occlusion, partial occlusion, mutual occlusion:
PartiallyOccludes(x , y , ν)↔ Occludes(x , y , ν)∧¬TotallyOccludes(x , y , ν) ∧ ¬Occludes(y , x , ν)
MutuallyOccludes(x , y , ν)↔ Occludes(x , y , ν) ∧Occludes(y , x , ν)NonOccludes(x , y , ν)↔ ¬Occludes(x , y , ν) ∧ ¬Occludes(y , x , ν)NonOccludes(x , y , ν)→ DR(image(x , ν), image(y , ν))PartiallyOccludes(x , y , ν)→
(PO(image(x , ν), image(y , ν)) ∨ PP(image(x , ν), image(y , ν)))MutuallyOccludes(x , y , ν)→
(PO(image(x , ν), image(y , ν)) ∨ P(image(x , ν), image(y , ν))∨PI(image(x , ν), image(y , ν)))
Paulo Santos ( FEI - São Paulo ) September 10, 2010 52 / 136
Region Occlusion Calculus
MutuallyOccludesPO
NonOccludesEC
NonOccludesDC
TotallyOccludesTPPI
PartiallyOccludesPO!1
PartiallyOccludesTPP
TotallyOccludesTPPI !1
TotallyOccludesEQ
PartiallyOccludesPO
TotallyOccludesNTPPI
TotallyOccludesNTPPI !1
PartiallyOccludesTPP!1
MutuallyOccludesTPP !1
TotallyOccludesEQ
PartiallyOccludesNTPP
MutuallyOccludesEQ
MutuallyOccludesNTPP
MutuallyOccludesTPP
MutuallyOccludesNTPP!1
!1
!1PartiallyOccludesNTPP
Paulo Santos ( FEI - São Paulo ) September 10, 2010 53 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 54 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 55 / 136
Cardinal Direction Calculus
Cardinal Direction Calculus (CDC) is a formalism for reasoningabout the directions between spatial objects9 base relations: north, south, east , west , northeast , northwest ,southeast , southwest e EQ (EQ(x , y) means “x is at the samedirection as y”).Main goal of CDC is to infer facts about the relative direction oftwo objects A and B from the known directions between A and C(A 6= C and B 6= C).
I E.g., from north(A,B) and northeast(B,C), the task is to calculatethe possible directions between A e C.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 56 / 136
Cardinal Direction Calculus
Cardinal Direction Calculus (CDC) is a formalism for reasoningabout the directions between spatial objects9 base relations: north, south, east , west , northeast , northwest ,southeast , southwest e EQ (EQ(x , y) means “x is at the samedirection as y”).Main goal of CDC is to infer facts about the relative direction oftwo objects A and B from the known directions between A and C(A 6= C and B 6= C).
I E.g., from north(A,B) and northeast(B,C), the task is to calculatethe possible directions between A e C.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 56 / 136
Cardinal Direction Calculus
Figure: Adapted from SparQ User Manual v0.7Paulo Santos ( FEI - São Paulo ) September 10, 2010 57 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 58 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 59 / 136
Double Cross Calculus
is a calculus that defines the direction of a point with respect to adirected line segment15 ternary relations on pointsrepresents every distinct relation between the directions left-rightand front-back (e.g.left-front, left-back, left-line, left -perpendicular,straight-front, ...)motivation: qualitative description of paths
Paulo Santos ( FEI - São Paulo ) September 10, 2010 60 / 136
Double Cross Calculus
is a calculus that defines the direction of a point with respect to adirected line segment15 ternary relations on pointsrepresents every distinct relation between the directions left-rightand front-back (e.g.left-front, left-back, left-line, left -perpendicular,straight-front, ...)motivation: qualitative description of paths
Paulo Santos ( FEI - São Paulo ) September 10, 2010 60 / 136
Double Cross Calculus
is a calculus that defines the direction of a point with respect to adirected line segment15 ternary relations on pointsrepresents every distinct relation between the directions left-rightand front-back (e.g.left-front, left-back, left-line, left -perpendicular,straight-front, ...)motivation: qualitative description of paths
Paulo Santos ( FEI - São Paulo ) September 10, 2010 60 / 136
Double Cross Calculus
is a calculus that defines the direction of a point with respect to adirected line segment15 ternary relations on pointsrepresents every distinct relation between the directions left-rightand front-back (e.g.left-front, left-back, left-line, left -perpendicular,straight-front, ...)motivation: qualitative description of paths
Paulo Santos ( FEI - São Paulo ) September 10, 2010 60 / 136
Double Cross Calculus
Figure: Adapted from SparQ User Manual v0.7
Paulo Santos ( FEI - São Paulo ) September 10, 2010 61 / 136
Double Cross Calculus
Paulo Santos ( FEI - São Paulo ) September 10, 2010 62 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 63 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 64 / 136
Other Calculi
size and distance: are defined either in absolute scale (whereasthe relations < or >) are introduced in the usual way), or inrelative terms, where the relative connection between threeobjects is used to define the relations of proximity (near than, farthan) and equidistance . Size and distance calculi are usuallycoupled with other calculi to extend their expressivity,shapes: one of the least understood areas of QSR. In general,shape is defined by means of a number of primitives such asinterior or boundary.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 65 / 136
Other Calculi
default: not much has been done wrt default theories aboutspace; Shanahan formalises a pre-condition about spatialoccupancy of objects assuming that space is empty by default.spatial change: base to the development of spatial change is thework of Galton, where both time instants and intervals areincluded. Two predicates are used to this end: HoldsT representsa spatial state that is true at a time instant, whereas HoldsIrepresents true stated during a time interval. From this, 8 distinctkinds of transitions between pairs of states are defined in order torepresent the relation of two states in time.Qualitative Trajectory Calculi, Line Segments, Dipole Calculi, andmany others!
Paulo Santos ( FEI - São Paulo ) September 10, 2010 66 / 136
SparQ
toolbox for QSR in applicationsI reference implementations of QSRI typical QSR proceduresI uniform interface
http://www.sfbtr8.uni-bremen.de/project/r3/sparq/
Paulo Santos ( FEI - São Paulo ) September 10, 2010 67 / 136
Applications of QSR
qualitative simulation of physical systemssyntax and semantics of visual programming languagesdatabases integrationGISreal time event recognitionrobotics
Paulo Santos ( FEI - São Paulo ) September 10, 2010 68 / 136
Applications of QSR
qualitative simulation of physical systemssyntax and semantics of visual programming languagesdatabases integrationGISreal time event recognitionrobotics
Paulo Santos ( FEI - São Paulo ) September 10, 2010 68 / 136
Applications of QSR
qualitative simulation of physical systemssyntax and semantics of visual programming languagesdatabases integrationGISreal time event recognitionrobotics
Paulo Santos ( FEI - São Paulo ) September 10, 2010 68 / 136
Applications of QSR
qualitative simulation of physical systemssyntax and semantics of visual programming languagesdatabases integrationGISreal time event recognitionrobotics
Paulo Santos ( FEI - São Paulo ) September 10, 2010 68 / 136
Applications of QSR
qualitative simulation of physical systemssyntax and semantics of visual programming languagesdatabases integrationGISreal time event recognitionrobotics
Paulo Santos ( FEI - São Paulo ) September 10, 2010 68 / 136
Applications of QSR
qualitative simulation of physical systemssyntax and semantics of visual programming languagesdatabases integrationGISreal time event recognitionrobotics
Paulo Santos ( FEI - São Paulo ) September 10, 2010 68 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 69 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 70 / 136
Tractability and computability
efficiency is inversely proportional to expressivityfirst order formulation of mereotopology is not decidable
I find a decidable subset (e.g. RCC8)I look for tractable subsets
Paulo Santos ( FEI - São Paulo ) September 10, 2010 71 / 136
Tractability and computability
efficiency is inversely proportional to expressivityfirst order formulation of mereotopology is not decidable
I find a decidable subset (e.g. RCC8)I look for tractable subsets
Paulo Santos ( FEI - São Paulo ) September 10, 2010 71 / 136
Tractability and computability
efficiency is inversely proportional to expressivityfirst order formulation of mereotopology is not decidable
I find a decidable subset (e.g. RCC8)I look for tractable subsets
Paulo Santos ( FEI - São Paulo ) September 10, 2010 71 / 136
Tractability and computability
efficiency is inversely proportional to expressivityfirst order formulation of mereotopology is not decidable
I find a decidable subset (e.g. RCC8)I look for tractable subsets
Paulo Santos ( FEI - São Paulo ) September 10, 2010 71 / 136
Tractability and computability
efficiency is inversely proportional to expressivityfirst order formulation of mereotopology is not decidable
I find a decidable subset (e.g. RCC8)I look for tractable subsets
Paulo Santos ( FEI - São Paulo ) September 10, 2010 71 / 136
Tractability and computability
Tractable subsets of QSR formalism is the subject of a number ofpapers. In particular (Renz e Cohn 2008) presents the followingingredients for finding tractable subsets of such formalism:
a method to prove that a given subset if tractablea method to suggest possible tractable subsetsin order to prove that a given set of relations is tractable, it issufficient to prove that the inclusion of any new relation makes theset intractable
J. Renz, Qualitative Spatial and Temporal Reasoning: EfficientAlgorithms for Everyone, in: Proc (IJCAI-07)
Paulo Santos ( FEI - São Paulo ) September 10, 2010 72 / 136
Tractability and computability
Tractable subsets of QSR formalism is the subject of a number ofpapers. In particular (Renz e Cohn 2008) presents the followingingredients for finding tractable subsets of such formalism:
a method to prove that a given subset if tractablea method to suggest possible tractable subsetsin order to prove that a given set of relations is tractable, it issufficient to prove that the inclusion of any new relation makes theset intractable
J. Renz, Qualitative Spatial and Temporal Reasoning: EfficientAlgorithms for Everyone, in: Proc (IJCAI-07)
Paulo Santos ( FEI - São Paulo ) September 10, 2010 72 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 73 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 74 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 75 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 76 / 136
Our knowledge of the external world
(...) what is actually given in sense is much less than mostpeople would naturally suppose, and (...) much of what at firstsight seems to be given is really inferred. This appliesespecially in regard to our space-perceptions. For instance,we unconsciously infer the “real” size and shape of a visibleobject from its apparent size and shape, according to itsdistance and our point of view. (...). Thus, the first step in theanalyses of data, namely, the discovery of what is really givenin sense, is full of difficulty.
[B. Russell (1914), Our Knowledge ofthe External World, pp.75-76]
Although these ideas are at the foundations of QSR, their applicationin scene analysis is still work in progress.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 77 / 136
Our knowledge of the external world
(...) what is actually given in sense is much less than mostpeople would naturally suppose, and (...) much of what at firstsight seems to be given is really inferred. This appliesespecially in regard to our space-perceptions. For instance,we unconsciously infer the “real” size and shape of a visibleobject from its apparent size and shape, according to itsdistance and our point of view. (...). Thus, the first step in theanalyses of data, namely, the discovery of what is really givenin sense, is full of difficulty.
[B. Russell (1914), Our Knowledge ofthe External World, pp.75-76]
Although these ideas are at the foundations of QSR, their applicationin scene analysis is still work in progress.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 77 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 78 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 79 / 136
What is CogVis?from ECvision network
A Cognitive Vision System can achieve the four levels of genericvisual functionality: Detection, Localisation, Recognition,Understanding (role, context, purpose)and exhibits purposive goal-directed behaviour, is adaptive tounforeseen changes, and can anticipate the occurrence ofobjects and events. This is achieved through:
I Learning semantic knowledge (form, function and behaviours)I Retention of knowledge (about the cognitive system, its
environment, and the relationship with the environment)I Deliberation about objects and events, including the cognitive
system itself.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 80 / 136
Some approaches for CogVis
VITRA system- Visual Translator: Integration of computer visionand natural language processingALVEN system - textual description of all heart dynamics viaX-Ray image sequences.Brand’s visual understanding through causal analysisSiskind’s systems - Event classification from camera input usingforce dynamicsLeeds traffic interaction - Modelling traffic interaction using learntqualitative spatio-temporal relations and variable length MarkovmodelsLeeds Protocol learning - Combining continuous and symbolicmodels to learn games from observationFEI contributions - Spatial reasoning image analysis
Paulo Santos ( FEI - São Paulo ) September 10, 2010 81 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 82 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 83 / 136
Nagel’s Image Sequence Evaluation
Requirements and purpose:exhaustive internal representation for all tasks and experimentalconditions it is expected to handlesituation: suitable intermediate representation during theevaluation of image sequencesderive scene-specific conceptual descriptions from imagesequences, based on general assumptions about motion as thecause of observable changes
Image sequence evaluation is based on the idea that the flow of imagesequences reflects (to some extent) the coherency of the conceptualworld.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 84 / 136
Nagel’s Image Sequence Evaluation
Requirements and purpose:exhaustive internal representation for all tasks and experimentalconditions it is expected to handlesituation: suitable intermediate representation during theevaluation of image sequencesderive scene-specific conceptual descriptions from imagesequences, based on general assumptions about motion as thecause of observable changes
Image sequence evaluation is based on the idea that the flow of imagesequences reflects (to some extent) the coherency of the conceptualworld.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 84 / 136
Nagel’s Image Sequence Evaluation
Requirements and purpose:exhaustive internal representation for all tasks and experimentalconditions it is expected to handlesituation: suitable intermediate representation during theevaluation of image sequencesderive scene-specific conceptual descriptions from imagesequences, based on general assumptions about motion as thecause of observable changes
Image sequence evaluation is based on the idea that the flow of imagesequences reflects (to some extent) the coherency of the conceptualworld.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 84 / 136
Nagel’s Image Sequence Evaluation
Requirements and purpose:exhaustive internal representation for all tasks and experimentalconditions it is expected to handlesituation: suitable intermediate representation during theevaluation of image sequencesderive scene-specific conceptual descriptions from imagesequences, based on general assumptions about motion as thecause of observable changes
Image sequence evaluation is based on the idea that the flow of imagesequences reflects (to some extent) the coherency of the conceptualworld.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 84 / 136
Nagel’s Image Sequence Evaluation
Intermediate levels of description:change: any deviation of sensor signal which significantly differsfrom noiseevent: any change which has been defined a priori as a primitivefor the construction of more complex descriptionsverb: describe activitieshistory: extended sequence of activities
Paulo Santos ( FEI - São Paulo ) September 10, 2010 85 / 136
Some early systems
Morio (Nagel 1983), Naos (Neumann 1986), Epex (Nagel 1987),CityTour (Herzog 1986)usually describing traffic scenes/situations from an static viewpoint(using motion verbs such as overtake, approach, move, beside,recedebut also to describe a walking person (Hogg 1983), heart motion(Tsotos 1980) and so on
Paulo Santos ( FEI - São Paulo ) September 10, 2010 86 / 136
Some early systems
Morio (Nagel 1983), Naos (Neumann 1986), Epex (Nagel 1987),CityTour (Herzog 1986)usually describing traffic scenes/situations from an static viewpoint(using motion verbs such as overtake, approach, move, beside,recedebut also to describe a walking person (Hogg 1983), heart motion(Tsotos 1980) and so on
Paulo Santos ( FEI - São Paulo ) September 10, 2010 86 / 136
Some early systems
Morio (Nagel 1983), Naos (Neumann 1986), Epex (Nagel 1987),CityTour (Herzog 1986)usually describing traffic scenes/situations from an static viewpoint(using motion verbs such as overtake, approach, move, beside,recedebut also to describe a walking person (Hogg 1983), heart motion(Tsotos 1980) and so on
Paulo Santos ( FEI - São Paulo ) September 10, 2010 86 / 136
Project Vitra: Visual Translator
long term project: 80’s and 90’sdesign and construction of integrated knowledge-based systemscapable of translating visual information into natural languagedescriptionsgeneral goals:
I extend the scope of scene analysis beyond the level of objectrecognition
I explicit description of spatial configurations by means of spatialrelations
I interpretation of object movementI automatic recognition of goals and plans of observed agents
specific goals:I answering queries about traffic situationsI generating reports of football gamesI communicating with a mobile robot
Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136
Project Vitra: Visual Translator
long term project: 80’s and 90’sdesign and construction of integrated knowledge-based systemscapable of translating visual information into natural languagedescriptionsgeneral goals:
I extend the scope of scene analysis beyond the level of objectrecognition
I explicit description of spatial configurations by means of spatialrelations
I interpretation of object movementI automatic recognition of goals and plans of observed agents
specific goals:I answering queries about traffic situationsI generating reports of football gamesI communicating with a mobile robot
Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136
Project Vitra: Visual Translator
long term project: 80’s and 90’sdesign and construction of integrated knowledge-based systemscapable of translating visual information into natural languagedescriptionsgeneral goals:
I extend the scope of scene analysis beyond the level of objectrecognition
I explicit description of spatial configurations by means of spatialrelations
I interpretation of object movementI automatic recognition of goals and plans of observed agents
specific goals:I answering queries about traffic situationsI generating reports of football gamesI communicating with a mobile robot
Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136
Project Vitra: Visual Translator
long term project: 80’s and 90’sdesign and construction of integrated knowledge-based systemscapable of translating visual information into natural languagedescriptionsgeneral goals:
I extend the scope of scene analysis beyond the level of objectrecognition
I explicit description of spatial configurations by means of spatialrelations
I interpretation of object movementI automatic recognition of goals and plans of observed agents
specific goals:I answering queries about traffic situationsI generating reports of football gamesI communicating with a mobile robot
Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136
Project Vitra: Visual Translator
long term project: 80’s and 90’sdesign and construction of integrated knowledge-based systemscapable of translating visual information into natural languagedescriptionsgeneral goals:
I extend the scope of scene analysis beyond the level of objectrecognition
I explicit description of spatial configurations by means of spatialrelations
I interpretation of object movementI automatic recognition of goals and plans of observed agents
specific goals:I answering queries about traffic situationsI generating reports of football gamesI communicating with a mobile robot
Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136
Project Vitra: Visual Translator
long term project: 80’s and 90’sdesign and construction of integrated knowledge-based systemscapable of translating visual information into natural languagedescriptionsgeneral goals:
I extend the scope of scene analysis beyond the level of objectrecognition
I explicit description of spatial configurations by means of spatialrelations
I interpretation of object movementI automatic recognition of goals and plans of observed agents
specific goals:I answering queries about traffic situationsI generating reports of football gamesI communicating with a mobile robot
Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136
Project Vitra: Visual Translator
long term project: 80’s and 90’sdesign and construction of integrated knowledge-based systemscapable of translating visual information into natural languagedescriptionsgeneral goals:
I extend the scope of scene analysis beyond the level of objectrecognition
I explicit description of spatial configurations by means of spatialrelations
I interpretation of object movementI automatic recognition of goals and plans of observed agents
specific goals:I answering queries about traffic situationsI generating reports of football gamesI communicating with a mobile robot
Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136
Project Vitra: Visual Translator
long term project: 80’s and 90’sdesign and construction of integrated knowledge-based systemscapable of translating visual information into natural languagedescriptionsgeneral goals:
I extend the scope of scene analysis beyond the level of objectrecognition
I explicit description of spatial configurations by means of spatialrelations
I interpretation of object movementI automatic recognition of goals and plans of observed agents
specific goals:I answering queries about traffic situationsI generating reports of football gamesI communicating with a mobile robot
Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136
Project Vitra: Visual Translator
long term project: 80’s and 90’sdesign and construction of integrated knowledge-based systemscapable of translating visual information into natural languagedescriptionsgeneral goals:
I extend the scope of scene analysis beyond the level of objectrecognition
I explicit description of spatial configurations by means of spatialrelations
I interpretation of object movementI automatic recognition of goals and plans of observed agents
specific goals:I answering queries about traffic situationsI generating reports of football gamesI communicating with a mobile robot
Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136
Project Vitra: Visual Translator
Overview of the system:from raw data, image analysis generates a geometricalrepresentation of the scene, with object’s locations through timethe geometric description is interpreted by the cognitive level (orhigh-level scene analysis)this high-level analysis extracts:
I spatial relationsI interesting motion eventsI presumed intentions, plans and plan interactions between
agents
the final, linguistic, level transform the conceptual descriptions intoutterances
Paulo Santos ( FEI - São Paulo ) September 10, 2010 88 / 136
Project Vitra: Visual Translator
Overview of the system:from raw data, image analysis generates a geometricalrepresentation of the scene, with object’s locations through timethe geometric description is interpreted by the cognitive level (orhigh-level scene analysis)this high-level analysis extracts:
I spatial relationsI interesting motion eventsI presumed intentions, plans and plan interactions between
agents
the final, linguistic, level transform the conceptual descriptions intoutterances
Paulo Santos ( FEI - São Paulo ) September 10, 2010 88 / 136
Project Vitra: Visual Translator
Overview of the system:from raw data, image analysis generates a geometricalrepresentation of the scene, with object’s locations through timethe geometric description is interpreted by the cognitive level (orhigh-level scene analysis)this high-level analysis extracts:
I spatial relationsI interesting motion eventsI presumed intentions, plans and plan interactions between
agents
the final, linguistic, level transform the conceptual descriptions intoutterances
Paulo Santos ( FEI - São Paulo ) September 10, 2010 88 / 136
Project Vitra: Visual Translator
Overview of the system:from raw data, image analysis generates a geometricalrepresentation of the scene, with object’s locations through timethe geometric description is interpreted by the cognitive level (orhigh-level scene analysis)this high-level analysis extracts:
I spatial relationsI interesting motion eventsI presumed intentions, plans and plan interactions between
agents
the final, linguistic, level transform the conceptual descriptions intoutterances
Paulo Santos ( FEI - São Paulo ) September 10, 2010 88 / 136
Project Vitra: Visual Translator
Overview of the system:from raw data, image analysis generates a geometricalrepresentation of the scene, with object’s locations through timethe geometric description is interpreted by the cognitive level (orhigh-level scene analysis)this high-level analysis extracts:
I spatial relationsI interesting motion eventsI presumed intentions, plans and plan interactions between
agents
the final, linguistic, level transform the conceptual descriptions intoutterances
Paulo Santos ( FEI - São Paulo ) September 10, 2010 88 / 136
Alven (1984)
natural language description of ventricular wall motiondata extracted from X-ray image sequencesknowledge organisation: ontologyAnalyse pre-operative and post-operative marker films to evaluatethe efficacy of surgeryanalysis using both quantitative and qualitative representationsmajor issues:
I understanding visual motion from image tokens over timeI reasoning about spatio-temporal relationships
Paulo Santos ( FEI - São Paulo ) September 10, 2010 89 / 136
Alven (1984)
natural language description of ventricular wall motiondata extracted from X-ray image sequencesknowledge organisation: ontologyAnalyse pre-operative and post-operative marker films to evaluatethe efficacy of surgeryanalysis using both quantitative and qualitative representationsmajor issues:
I understanding visual motion from image tokens over timeI reasoning about spatio-temporal relationships
Paulo Santos ( FEI - São Paulo ) September 10, 2010 89 / 136
Alven (1984)
natural language description of ventricular wall motiondata extracted from X-ray image sequencesknowledge organisation: ontologyAnalyse pre-operative and post-operative marker films to evaluatethe efficacy of surgeryanalysis using both quantitative and qualitative representationsmajor issues:
I understanding visual motion from image tokens over timeI reasoning about spatio-temporal relationships
Paulo Santos ( FEI - São Paulo ) September 10, 2010 89 / 136
Alven (1984)
natural language description of ventricular wall motiondata extracted from X-ray image sequencesknowledge organisation: ontologyAnalyse pre-operative and post-operative marker films to evaluatethe efficacy of surgeryanalysis using both quantitative and qualitative representationsmajor issues:
I understanding visual motion from image tokens over timeI reasoning about spatio-temporal relationships
Paulo Santos ( FEI - São Paulo ) September 10, 2010 89 / 136
Alven (1984)
natural language description of ventricular wall motiondata extracted from X-ray image sequencesknowledge organisation: ontologyAnalyse pre-operative and post-operative marker films to evaluatethe efficacy of surgeryanalysis using both quantitative and qualitative representationsmajor issues:
I understanding visual motion from image tokens over timeI reasoning about spatio-temporal relationships
Paulo Santos ( FEI - São Paulo ) September 10, 2010 89 / 136
Alven (1984)
natural language description of ventricular wall motiondata extracted from X-ray image sequencesknowledge organisation: ontologyAnalyse pre-operative and post-operative marker films to evaluatethe efficacy of surgeryanalysis using both quantitative and qualitative representationsmajor issues:
I understanding visual motion from image tokens over timeI reasoning about spatio-temporal relationships
Paulo Santos ( FEI - São Paulo ) September 10, 2010 89 / 136
Alven
Paulo Santos ( FEI - São Paulo ) September 10, 2010 90 / 136
Alven’s knowledge base
Set of classes organised using relations aboutIS_A: generalisation/specialisations (taxonomy)PART _OF : part/whole (partonomy)temporal precedence
Should satisfy:1 motion classes should be sufficient to express the domain2 image tokens are connected to general knowledge in the leaves of
a PART _OF hierarchy of motion concepts
Paulo Santos ( FEI - São Paulo ) September 10, 2010 91 / 136
Alven’s knowledge base
Set of classes organised using relations aboutIS_A: generalisation/specialisations (taxonomy)PART _OF : part/whole (partonomy)temporal precedence
Should satisfy:1 motion classes should be sufficient to express the domain2 image tokens are connected to general knowledge in the leaves of
a PART _OF hierarchy of motion concepts
Paulo Santos ( FEI - São Paulo ) September 10, 2010 91 / 136
Alven’s control structure
extract tokens from the input signal, instantiating then in theleaves of the PART_OF hierarchyfollows this hierarchy to activated hypotheses that are aggregatesof the input tokensthis set of hypotheses is specialised by going down one level inthe IS_A hierarchyeach hypothesis is matched with other data instances: matchingleads to further specialisations, failure leads to the selection ofother hypothesesthe best hypotheses generate a set of predictionspredictions are mapped back to the image level and are used as aguidance to the token extraction procedure
Paulo Santos ( FEI - São Paulo ) September 10, 2010 92 / 136
Alven’s control structure
extract tokens from the input signal, instantiating then in theleaves of the PART_OF hierarchyfollows this hierarchy to activated hypotheses that are aggregatesof the input tokensthis set of hypotheses is specialised by going down one level inthe IS_A hierarchyeach hypothesis is matched with other data instances: matchingleads to further specialisations, failure leads to the selection ofother hypothesesthe best hypotheses generate a set of predictionspredictions are mapped back to the image level and are used as aguidance to the token extraction procedure
Paulo Santos ( FEI - São Paulo ) September 10, 2010 92 / 136
Alven’s control structure
extract tokens from the input signal, instantiating then in theleaves of the PART_OF hierarchyfollows this hierarchy to activated hypotheses that are aggregatesof the input tokensthis set of hypotheses is specialised by going down one level inthe IS_A hierarchyeach hypothesis is matched with other data instances: matchingleads to further specialisations, failure leads to the selection ofother hypothesesthe best hypotheses generate a set of predictionspredictions are mapped back to the image level and are used as aguidance to the token extraction procedure
Paulo Santos ( FEI - São Paulo ) September 10, 2010 92 / 136
Alven’s control structure
extract tokens from the input signal, instantiating then in theleaves of the PART_OF hierarchyfollows this hierarchy to activated hypotheses that are aggregatesof the input tokensthis set of hypotheses is specialised by going down one level inthe IS_A hierarchyeach hypothesis is matched with other data instances: matchingleads to further specialisations, failure leads to the selection ofother hypothesesthe best hypotheses generate a set of predictionspredictions are mapped back to the image level and are used as aguidance to the token extraction procedure
Paulo Santos ( FEI - São Paulo ) September 10, 2010 92 / 136
Alven’s control structure
extract tokens from the input signal, instantiating then in theleaves of the PART_OF hierarchyfollows this hierarchy to activated hypotheses that are aggregatesof the input tokensthis set of hypotheses is specialised by going down one level inthe IS_A hierarchyeach hypothesis is matched with other data instances: matchingleads to further specialisations, failure leads to the selection ofother hypothesesthe best hypotheses generate a set of predictionspredictions are mapped back to the image level and are used as aguidance to the token extraction procedure
Paulo Santos ( FEI - São Paulo ) September 10, 2010 92 / 136
Alven’s control structure
extract tokens from the input signal, instantiating then in theleaves of the PART_OF hierarchyfollows this hierarchy to activated hypotheses that are aggregatesof the input tokensthis set of hypotheses is specialised by going down one level inthe IS_A hierarchyeach hypothesis is matched with other data instances: matchingleads to further specialisations, failure leads to the selection ofother hypothesesthe best hypotheses generate a set of predictionspredictions are mapped back to the image level and are used as aguidance to the token extraction procedure
Paulo Santos ( FEI - São Paulo ) September 10, 2010 92 / 136
Brand’s visual understanding through causal analysis
describe the causal structure from imagesknowledge about physical causality is used in the interpretationsceneshypothesis: a small core set of qualitative rules accounts for mostof what humans ordinarily seeit is possible to make useful inferences with qualitative knowledgeabout connectivity and free spacequalitative rules of connectivity, friction, attachment andpenetrationinterpretation of simple mechanical machinespurpose
I AnalyseI diagnosisI predictionI inspection
Paulo Santos ( FEI - São Paulo ) September 10, 2010 93 / 136
Brand’s visual understanding through causal analysis
describe the causal structure from imagesknowledge about physical causality is used in the interpretationsceneshypothesis: a small core set of qualitative rules accounts for mostof what humans ordinarily seeit is possible to make useful inferences with qualitative knowledgeabout connectivity and free spacequalitative rules of connectivity, friction, attachment andpenetrationinterpretation of simple mechanical machinespurpose
I AnalyseI diagnosisI predictionI inspection
Paulo Santos ( FEI - São Paulo ) September 10, 2010 93 / 136
Brand’s visual understanding through causal analysis
describe the causal structure from imagesknowledge about physical causality is used in the interpretationsceneshypothesis: a small core set of qualitative rules accounts for mostof what humans ordinarily seeit is possible to make useful inferences with qualitative knowledgeabout connectivity and free spacequalitative rules of connectivity, friction, attachment andpenetrationinterpretation of simple mechanical machinespurpose
I AnalyseI diagnosisI predictionI inspection
Paulo Santos ( FEI - São Paulo ) September 10, 2010 93 / 136
Brand’s visual understanding through causal analysis
describe the causal structure from imagesknowledge about physical causality is used in the interpretationsceneshypothesis: a small core set of qualitative rules accounts for mostof what humans ordinarily seeit is possible to make useful inferences with qualitative knowledgeabout connectivity and free spacequalitative rules of connectivity, friction, attachment andpenetrationinterpretation of simple mechanical machinespurpose
I AnalyseI diagnosisI predictionI inspection
Paulo Santos ( FEI - São Paulo ) September 10, 2010 93 / 136
Brand’s visual understanding through causal analysis
describe the causal structure from imagesknowledge about physical causality is used in the interpretationsceneshypothesis: a small core set of qualitative rules accounts for mostof what humans ordinarily seeit is possible to make useful inferences with qualitative knowledgeabout connectivity and free spacequalitative rules of connectivity, friction, attachment andpenetrationinterpretation of simple mechanical machinespurpose
I AnalyseI diagnosisI predictionI inspection
Paulo Santos ( FEI - São Paulo ) September 10, 2010 93 / 136
Brand’s visual understanding through causal analysis
describe the causal structure from imagesknowledge about physical causality is used in the interpretationsceneshypothesis: a small core set of qualitative rules accounts for mostof what humans ordinarily seeit is possible to make useful inferences with qualitative knowledgeabout connectivity and free spacequalitative rules of connectivity, friction, attachment andpenetrationinterpretation of simple mechanical machinespurpose
I AnalyseI diagnosisI predictionI inspection
Paulo Santos ( FEI - São Paulo ) September 10, 2010 93 / 136
Brand’s visual understanding through causal analysis
describe the causal structure from imagesknowledge about physical causality is used in the interpretationsceneshypothesis: a small core set of qualitative rules accounts for mostof what humans ordinarily seeit is possible to make useful inferences with qualitative knowledgeabout connectivity and free spacequalitative rules of connectivity, friction, attachment andpenetrationinterpretation of simple mechanical machinespurpose
I AnalyseI diagnosisI predictionI inspection
Paulo Santos ( FEI - São Paulo ) September 10, 2010 93 / 136
Brand’s SPROCKET (1997)
Paulo Santos ( FEI - São Paulo ) September 10, 2010 94 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 95 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 96 / 136
Siskind’s Event classification from camera input usingforce dynamics
similar to Brand’s but on image sequencesforce dynamics: SUPPORTED, ATTACHED, PICKUP, PUTDOWN,CONTACTmore recently: supervised learning of visual event definitions fromvideo
Paulo Santos ( FEI - São Paulo ) September 10, 2010 97 / 136
Siskind’s Event classification from camera input usingforce dynamics
similar to Brand’s but on image sequencesforce dynamics: SUPPORTED, ATTACHED, PICKUP, PUTDOWN,CONTACTmore recently: supervised learning of visual event definitions fromvideo
Paulo Santos ( FEI - São Paulo ) September 10, 2010 97 / 136
Siskind’s Event classification from camera input usingforce dynamics
similar to Brand’s but on image sequencesforce dynamics: SUPPORTED, ATTACHED, PICKUP, PUTDOWN,CONTACTmore recently: supervised learning of visual event definitions fromvideo
Paulo Santos ( FEI - São Paulo ) September 10, 2010 97 / 136
Siskind’s Leonard System
Paulo Santos ( FEI - São Paulo ) September 10, 2010 98 / 136
Leeds Traffic Interaction
apply QSR to dynamic scene analysislearning events from the observation of traffic scenesrecognising these events and generate predictionsa scene is interpreted from the comparison of its temporaldevelopment with a (previously learned) transition diagramrepresenting change in the domain.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 99 / 136
CND of relative positions
Paulo Santos ( FEI - São Paulo ) September 10, 2010 100 / 136
Leeds Traffic Interaction
Paulo Santos ( FEI - São Paulo ) September 10, 2010 101 / 136
Protocol learning
A system for learning protocol behaviour from computer visiondata using ILPUnsupervised continuous learning of perceptual categories andunsupervised symbolic learning of protocols (as sets of Hornclauses);
Paulo Santos ( FEI - São Paulo ) September 10, 2010 102 / 136
Protocol Learning
Off-the-shelf vision systemI Attention: identifies key frames with no motion preceded by a
number of frames with notion;I Statistical classifier: assigns different classes to clusters of features
Off-the-shelf ILP system: ProgolI Generalises a set of positive only examples according to user
defined mode declarationsI Mode declarations restrict the possible form of the proposed
generalisations
Paulo Santos ( FEI - São Paulo ) September 10, 2010 103 / 136
Protocol Learning
Paulo Santos ( FEI - São Paulo ) September 10, 2010 104 / 136
Protocol Learning
Inducing axioms of ordering and equivalence, without knowingabout numbersBuilding equivalence classes to cope with over clusteringsound and completeness of the agent actually playing the game[AIJ 05]
Paulo Santos ( FEI - São Paulo ) September 10, 2010 105 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 106 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 107 / 136
Example of QSR from Vision: Depth Profile Calculus
How much knowledge about a robot’s environment can be fromvision alone?How can we construct the knowledge about objects in the worldfrom sensor data of the robot?Construction of a qualitative spatial reasoning (QSR) systembased on sensor dataUse abduction for sensor data assimilation and deduction forpredictions
Paulo Santos ( FEI - São Paulo ) September 10, 2010 108 / 136
Example of QSR from Vision: Depth Profile Calculus
How much knowledge about a robot’s environment can be fromvision alone?How can we construct the knowledge about objects in the worldfrom sensor data of the robot?Construction of a qualitative spatial reasoning (QSR) systembased on sensor dataUse abduction for sensor data assimilation and deduction forpredictions
Paulo Santos ( FEI - São Paulo ) September 10, 2010 108 / 136
Example of QSR from Vision: Depth Profile Calculus
How much knowledge about a robot’s environment can be fromvision alone?How can we construct the knowledge about objects in the worldfrom sensor data of the robot?Construction of a qualitative spatial reasoning (QSR) systembased on sensor dataUse abduction for sensor data assimilation and deduction forpredictions
Paulo Santos ( FEI - São Paulo ) September 10, 2010 108 / 136
Example of QSR from Vision: Depth Profile Calculus
How much knowledge about a robot’s environment can be fromvision alone?How can we construct the knowledge about objects in the worldfrom sensor data of the robot?Construction of a qualitative spatial reasoning (QSR) systembased on sensor dataUse abduction for sensor data assimilation and deduction forpredictions
Paulo Santos ( FEI - São Paulo ) September 10, 2010 108 / 136
Simplified environment
Paulo Santos ( FEI - São Paulo ) September 10, 2010 109 / 136
Spatial Reasoning about Robot Sensor Data
Attributes:I Distance, disparity, size;I Changes in the sensor data;
Representation:I Depth profiles and time points;I Displacement between regions;I Mapping function between images and objects
Paulo Santos ( FEI - São Paulo ) September 10, 2010 110 / 136
Assimilating changes
Axioms of the system:
< Dynamic spatial rel > ← < desc. sensor transition >< Dynamicspatialrel . > ← < obj − obs relation >
Paulo Santos ( FEI - São Paulo ) September 10, 2010 111 / 136
Depth Profiles
Paulo Santos ( FEI - São Paulo ) September 10, 2010 112 / 136
The model
Extract one horizontal depth profile of each scene from the visualdata;Objects in the scenes are represented as peaks;Axiomatise relations on the depth and size of these profiles aswell as displacements;
Paulo Santos ( FEI - São Paulo ) September 10, 2010 113 / 136
Depth Profile Calculus
A theory about displacement, size and depth;27 base relations;Large and complex conceptual neighbourhood diagrams andcomposition tables;
Paulo Santos ( FEI - São Paulo ) September 10, 2010 114 / 136
Depth Profile Calculus
Paulo Santos ( FEI - São Paulo ) September 10, 2010 115 / 136
DPC example
Paulo Santos ( FEI - São Paulo ) September 10, 2010 116 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 117 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 118 / 136
Reasoning about Shadows in Robotics
making explicit the knowledge contained in cast shadowsuse it to reason about the robot environmentComputer vision however has largely been filtering out castshadows as noise
Paulo Santos ( FEI - São Paulo ) September 10, 2010 119 / 136
Reasoning about Shadows in Robotics
making explicit the knowledge contained in cast shadowsuse it to reason about the robot environmentComputer vision however has largely been filtering out castshadows as noise
Paulo Santos ( FEI - São Paulo ) September 10, 2010 119 / 136
Reasoning about Shadows in Robotics
making explicit the knowledge contained in cast shadowsuse it to reason about the robot environmentComputer vision however has largely been filtering out castshadows as noise
Paulo Santos ( FEI - São Paulo ) September 10, 2010 119 / 136
Illusory Motion from Shadows
Paulo Santos ( FEI - São Paulo ) September 10, 2010 120 / 136
Perception of shadows
“no luminous body ever sees the shadows that it generates” [daVinci, Notebooks of Leonardo Da Vinci. Project Gutenberg (1888)]from the light source viewpoint shadows are occluded by theircastersWe model observer-caster-shadow within qualitative spatialreasoning: ROC + an axiom about shadow :
Shadow(s,o,Scr ,L)↔ PO(r(s), r(Scr))∧TotallyOccludes(o, s,L)∧
¬∃o′TotallyOccludes(o′,o,L).
“a shadow is totally occluded by its caster from the lightsourceviewpoint”
Paulo Santos ( FEI - São Paulo ) September 10, 2010 121 / 136
Perception of shadows
“no luminous body ever sees the shadows that it generates” [daVinci, Notebooks of Leonardo Da Vinci. Project Gutenberg (1888)]from the light source viewpoint shadows are occluded by theircastersWe model observer-caster-shadow within qualitative spatialreasoning: ROC + an axiom about shadow :
Shadow(s,o,Scr ,L)↔ PO(r(s), r(Scr))∧TotallyOccludes(o, s,L)∧
¬∃o′TotallyOccludes(o′,o,L).
“a shadow is totally occluded by its caster from the lightsourceviewpoint”
Paulo Santos ( FEI - São Paulo ) September 10, 2010 121 / 136
Perception of shadows
“no luminous body ever sees the shadows that it generates” [daVinci, Notebooks of Leonardo Da Vinci. Project Gutenberg (1888)]from the light source viewpoint shadows are occluded by theircastersWe model observer-caster-shadow within qualitative spatialreasoning: ROC + an axiom about shadow :
Shadow(s,o,Scr ,L)↔ PO(r(s), r(Scr))∧TotallyOccludes(o, s,L)∧
¬∃o′TotallyOccludes(o′,o,L).
“a shadow is totally occluded by its caster from the lightsourceviewpoint”
Paulo Santos ( FEI - São Paulo ) September 10, 2010 121 / 136
Perception of shadows
“no luminous body ever sees the shadows that it generates” [daVinci, Notebooks of Leonardo Da Vinci. Project Gutenberg (1888)]from the light source viewpoint shadows are occluded by theircastersWe model observer-caster-shadow within qualitative spatialreasoning: ROC + an axiom about shadow :
Shadow(s,o,Scr ,L)↔ PO(r(s), r(Scr))∧TotallyOccludes(o, s,L)∧
¬∃o′TotallyOccludes(o′,o,L).
“a shadow is totally occluded by its caster from the lightsourceviewpoint”
Paulo Santos ( FEI - São Paulo ) September 10, 2010 121 / 136
Qualitative regions for self-localisation
*L
o2
3
4
5
4
3
2
11S
Paulo Santos ( FEI - São Paulo ) September 10, 2010 122 / 136
In practice
Qualitative robot self-localisationrelative depth from the observation of shadowsthreshold finding from qualitative regions
Paulo Santos ( FEI - São Paulo ) September 10, 2010 123 / 136
In practice
*L
o2
3
4
5
4
3
2
11S
located(Region 1, ν,o, s)← Is_a_Shadow(s,o) ∧NonOccludesDC(o, s, v) ∧ v 6= o;
located(Region 2, ν,o, s)← Is_a_Shadow(s,o) ∧NonOccludesEC(o, s, v) ∧ v 6= o;
located(Region 3, ν,o, s)← Is_a_Shadow(s,o) ∧PartiallyOccludesPO(o, s, v) ∧ v 6= o;
located(Region 4, ν,o, s)← Is_a_Shadow(s,o) ∧TotallyOccludesTPPI(o, s, v) ∧ v 6= o;
located(Region 5, ν,o, s)← Is_a_Shadow(s,o) ∧TotallyOccludesNTPPI(o, s, v) ∧ v 6= o.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 124 / 136
Outline I1 Preface
Where/what is FEI?
2 PART I: The Big PictureIntroduction and motivationAutomated Reasoning 101Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial ReasoningRegion Connection CalculusLines of Sight CalculusRegion Occlusion CalculusCardinal Direction CalculusDouble Cross CalculusOther calculiTractability and computability
4 Coffee Break20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 125 / 136
Outline II5 Part III: Cognitive Vision
FoundationsCognitive Vision at a glanceEarly systemsModern systems (from 2000)
6 Part IV: QSR in CogVisDepth Profile CalculusReasoning about Shadows in RoboticsThe future: Probabilistic Logic Encoding of Spatial Domains
7 Conclusion
Paulo Santos ( FEI - São Paulo ) September 10, 2010 126 / 136
Probabilistic Logic Encoding of Spatial Domains
incorporate incomplete sensor data and domain knowledge in aprobabilistic logic settingexplore inferences about space
Paulo Santos ( FEI - São Paulo ) September 10, 2010 127 / 136
Traffic Scenario
Paulo Santos ( FEI - São Paulo ) September 10, 2010 128 / 136
Sensor model
P(OnTwoWayRoad |SensedTwoWayRoad) = 0.99P(OnOneWayRoad |SensedOneWayRoad) = 0.99P(DashedDivider |SensedDashedDivider) = 0.93P(SolidDivider |SensedDashedDivider) = 0.07P(DashedDivider |SensedSolidDivider) = 0.20P(SolidDivider |SensedSolidDivider) = 0.80
Paulo Santos ( FEI - São Paulo ) September 10, 2010 129 / 136
Taxonomy of concepts
Lane = GoingUp ∪ GoingDown ,Divider = DashedDivider SolidDivider ,Vehicle = OnOneWayRoad ∪ OnTwoWayRoad ,disjoint(Vehicle, Divider, Lane) ,disjoint(GoingUp, GoingDown) ,disjoint(DashedDivider, SolidDivider) ,disjoint(OnOneWayRoad, OnTwoWayRoad) ,
Paulo Santos ( FEI - São Paulo ) September 10, 2010 130 / 136
Hard Constraints
GoingUp ⊂ ∀ri .(GoingUp ∪ ¬Lane),
GoingDown ⊂ ∀le.(GoingDown ∪ ¬Lane),
DashedDivider ⊂ ∃ri .Lane u ∃le.Lane,SolidDivider ⊂(¬∃ri .Lane ∪ ¬∃le.Lane) ∪ (∃cdc.GoingUp u ∃cdc.GoingDown)
OnTwoWayRoad ⊂ ∃cdc.OneWayNorth u ∃cdc.OneWaySouth
Paulo Santos ( FEI - São Paulo ) September 10, 2010 131 / 136
Generate a Bayesian Net out of it
Paulo Santos ( FEI - São Paulo ) September 10, 2010 132 / 136
Answer queries such as
On which lane are we? ArgmaxliP(v : Onli) : li is the lane withmaximum probability of being the vehicle.Which driving directions does each lane permit?∀i : P(li : GoingDown): for each lane li, the probability of being aGoingUp lane
Paulo Santos ( FEI - São Paulo ) September 10, 2010 133 / 136
Conclusion
We’ve presented a brief overview of QSR and CogVisDiscussed some early work on CogVis systemspresented some new work based mainly at UoL and FEIMuch work remains to be done!
Paulo Santos ( FEI - São Paulo ) September 10, 2010 134 / 136
Conclusion
We’ve presented a brief overview of QSR and CogVisDiscussed some early work on CogVis systemspresented some new work based mainly at UoL and FEIMuch work remains to be done!
Paulo Santos ( FEI - São Paulo ) September 10, 2010 134 / 136
Conclusion
We’ve presented a brief overview of QSR and CogVisDiscussed some early work on CogVis systemspresented some new work based mainly at UoL and FEIMuch work remains to be done!
Paulo Santos ( FEI - São Paulo ) September 10, 2010 134 / 136
Conclusion
We’ve presented a brief overview of QSR and CogVisDiscussed some early work on CogVis systemspresented some new work based mainly at UoL and FEIMuch work remains to be done!
Paulo Santos ( FEI - São Paulo ) September 10, 2010 134 / 136
Guide lines
A. Cohn, Slides of the tutorial: Knowledge Representation andReasoning for Computer Vision: Qualitative Spatio-temporalreasoningA. Cohn and J. Renz, Qualitative Spatial Representation andReasoning, Handbook of Knowledge Representation, 2008B. Bennett, Logical representations for automated reasoningabout spatial relationships, PhD thesis, School of Computing,University of Leeds, UK.P. Santos, Raciocínio e Percepção espacial: uma abordagemlógica. Working notes of CBA 2010 tutorial. Tutoriais do XVIIICongresso Brasileiro de Automática, 2010.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 135 / 136
Thanks!
This author was partially supported by:FAPESP LogProb project:2008/03995-5CNPq bolsa PQ
Paulo Santos ( FEI - São Paulo ) September 10, 2010 136 / 136