Mental Development and Mental Development and Representation Building through Representation Building through

Motivated LearningMotivated Learning

Janusz A. Starzyk, Ohio University, USA, Pawel Raif, Silesian University of Technology, Poland,

Ah-Hwee Tan, Nanyang Technological University, Singapore

2010 International Joint Conference on Neural Networks, Barcelona

• Embodied Intelligence (EI)• Embodiment of Mind• Computational Approaches to

Machine Learning• How to Motivate a Machine• Motivated Learning (ML)• Building representation through

motivated learning– ML agent in „Normal” vs. „Graded”

Environment– ML agent vs. RL agent in „Graded”

Environment• Future work

OutlineOutline

Traditional AITraditional AI Embodied IntelligenceEmbodied Intelligence

• Abstract intelligence– attempt to simulate

“highest” human faculties:• language, discursive reason,

mathematics, abstract problem solving

• Environment model– Condition for problem solving

in abstract way– “brain in a vat”

• Embodiment– knowledge is implicit in the fact

that we have a body• embodiment supports brain

development

• Intelligence develops through interaction with environment– Situated in environment– Environment is its best model

Embodied IntelligenceEmbodied Intelligence

• Mechanism: biological, mechanical or virtual agentwith embodied sensors and actuators

• EI acts on environment and perceives its actions• Environment hostility: is persistent and stimulates EI to act• Hostility: direct aggression, pain, scarce resources, etc• EI learns so it must have associative self-organizing memory• Knowledge is acquired by EI

Definition

Embodied Intelligence (EI) is a mechanism that learns how to minimize hostility of its environment

IntelligenceIntelligence

An intelligent agent learns how

to survive in a hostile environment.

Embodiment

Actuators

Sensors

Intelligence core

channel

channel

Embodiment

Sensors

Intelligence core

Environment

channel

channelActuators

Embodiment

Actuators

Sensors

Intelligence core

channel

channel

Embodiment

Sensors

Intelligence core

Environment

channel

channelActuators

Embodiment of a MindEmbodiment of a Mind Embodiment: is a part of environment under

control of the mind It contains intelligence core and sensory motor

interfaces to interact with environment It is necessary for development of intelligence It is not necessarily constant

Changes in embodiment modify brain’s self-determination

Brain learns its own body’s dynamics

Self-awareness is a result of identification with own embodiment

Embodiment can be extended by using tools and machines

Successful operation is a function of correct perception of environment and own embodiment

Embodiment of MindEmbodiment of Mind

Computational Computational ApprApproachoaches to es to Machine LearningMachine Learning

Machine Learning Supervised Unsupervised Reinforced

problems with Complex environments

lack of motivation

Motivated Learning Definition Need for benchmarks

How to Motivate a Machine ?How to Motivate a Machine ?

A fundamental question is how to motivate an agent to do anything, and in particular, to enhance its own complexity?

What drives an agent to explore the environment build representations and learn effective actions?

What makes it successful learner in changing environments?

How to Motivate a Machine ?How to Motivate a Machine ?

Although artificial curiosity helps to explore the environment, it leads to learning without a specific purpose.

We suggest that the hostility of the environment, required

for EI, is the most effective motivational factor. Both are needed - hostility of the environment and

intelligence that learns how to reduce the pain.

Fig. englishteachermexico.wordpress.com/

Motivated LearningMotivated Learning Definition*: Motivated learning (ML) is pain

based motivation, goal creation and learning in embodied agent.

It uses externally defined pain signals. Machine is rewarded for minimizing the

primitive pain signals. Machine creates abstract goals based on the

primitive pain signals. It receives internal rewards for satisfying its

abstract goals. ML applies to EI working in a hostile

environment.

*J. A. Starzyk, Motivation in Embodied Intelligence, Frontiers in Robotics,

Automation and Control, I-Tech Education and Publishing, Oct. 2008, pp. 83-110.

NeuralNeural self-organizing self-organizing structures in MLstructures in ML

Goal creation schemean abstract pain is introduced by solving lower level pain

Motivations and selection of a goalWTA competition selects motivationanother WTA selects implementationa primitive pain is directly sensedthresholded curiosity based pain

Building representation

through motivated learning

Experiments…

Base Base Task SpecificationTask Specification

•Environment Environment consist of six different categories of

resources. Five of them have limited availability. One, the most abstract resource is inexhaustible.

Food Bank Office

SandboxGrocery

School

The least abstract

The most abstract

Agent uses resources performing proper actions. There are 36 possible actions but only six of them are meaningful and at a given situation (environment’s and agent’s state) there is usually one best action to perform.

The problem is: determine which action should be performed renewing in time the most needed resource.

Meaningful sensory-motor pairs and their effect on the environment:

Base Base Experiment - Task SpecificationExperiment - Task Specification

Id SENSORY MOTOR INCREASES DECREASES PAIR Id0 Food Eat Sugar level Food supplies 01 Grocery Buy Food supplies Money at hand 72 Bank Withdraw Money at hand Spending

limits14

3 Office Work Spending limits

Job opportunities

21

4 School Study Job opportunities

Mental state 28

5 Sandbox Play Mental state - 36

How How toto simulate complexity and simulate complexity and hostility of environmenthostility of environment

Food

Bank

Office

Feast

Grocery

School

1

2

1. ComplexityDifferent resources are available in the environment. Agent should learn dependencies between resources and its actions to operate properly.

2. HostilityFunction which describes the probability of finding resources in the environment.

Mild environment Harsh environment

Base Experiment ResultsBase Experiment Results

2

RL agent (left side) can learn dependencies between only few basic resources.

In contrast ML agent is able to learn dependencies between all resources.

In a harsh environmentML agent is able to control its

environment (and limit its ‘primitive pain’) but

RL agent cannot

RL ML

1

ML agent in „Normal” vs. „Graded” EnvironmentML agent in „Normal” vs. „Graded” Environment

Two kinds of environments - “normal” (1) and “graded” (2). “Graded” environment corresponds to gradual development and representation building Simulations in four environments with:

6, 10, 14 and 18 different hierarchy levels each one representing different resource.

1

Time

Resources …

Time

Resources

2…

ML agent learns more effectively in the ”graded” environments with gradually increasing complexity.

In a complex environment this difference becomes more significant.

“gradual” learning is beneficial to mental development

ML agent in „Normal” vs. „Graded” EnvironmentML agent in „Normal” vs. „Graded” Environment

ML agent vs. RL agent in „Graded” Environment.ML agent vs. RL agent in „Graded” Environment.

The second group of experiments compares effectiveness of ML and RL based agents.

In this simulation we have used “graded” environments with gradually increasing complexity.

We simulated environments with:6, 10, 14, 18 levels of hierarchy.

Time

Resources

…

6 levels of hierarchy Initially ML agent experiences similar

primitive pain signal Pp as RL agent. ML agent converges quickly to a stable

performance.

10 levels of hierarchy Initially RL agent experiences lower

primitive pain signal Pp than ML agent. RL agent’s pain increases when

environment is more hostile.

ML agent vs. RL agent in „Graded” Environment.ML agent vs. RL agent in „Graded” Environment.

14 levels of hierarchyML agent keeps learning while RL agent exploits early knowledgeIn effect, RL doesn’t learn all

dependencies it time to survive

18 levels of hierarchySimilar results to 10 and 14 levels

ML agent vs. RL agent in „Graded” Environment.ML agent vs. RL agent in „Graded” Environment.

RLRL actionstate

reward

Future workFuture work

actionstate

GCGCreward

GOALS (motivations)

RL

References:References:

• Starzyk J.A., Raif P., Ah-Hwee Tan, Motivated Learning as an Extension of Reinforcement Learning, Fourth International Conference on Cognitive Systems, CogSys 2010, ETH Zurich, January 2010.

• Starzyk J.A., Raif P., Motivated Learning Based on Goal Creation in Cognitive Systems, Thirteenth International Conference on Cognitive and Neural Systems, Boston University, May 2009.

• J. A. Starzyk, Motivation in Embodied Intelligence, Frontiers in Robotics, Automation and Control, I-Tech Education and Publishing, Oct. 2008, pp. 83-110.

Questions?