Cognitive informatics:Cognitive informatics: HITs, DREAMs & Perfect BabiesHITs, DREAMs & Perfect Babies

Włodzisław DuchDepartment of Informatics,

Nicolaus Copernicus University

Department of Computer Science, School of Computer Engineering, Nanyang Technological University,

SingaporeGoogle: Duch

A*STAR Cognitive Science Symposium, Sept. 2005

PlanPlan

1. Cognitive Science at foundations of science.2. Cognitive architectures.3. Engineering vision. 4. Humanized Interfaces (HITs).5. DREAMs.6. Perfect Babies.7. In the year 2015 ...8. Bye bye

ToruToruńń

ToruToruńń

Nicolaus Copernicus: born in 1473 in ToruNicolaus Copernicus: born in 1473 in Toruńń

Cognitive science: foundation for Cognitive science: foundation for understandingunderstanding

How do we known anything? Franz-Joseph Gall (1758-1828) discovered 26 "organs" on the surface of the brain which affect the contour of the skull, including a "murder organ" present in murderers. Phrenology become a widely accepted theory and cranioscopy was even more popular than psychological testing today, psychograph machines were sold till 1937. Thousands of observations supported phrenology!

Are we any wiser now?

Belief in miracle cures: is it only a placebo effect? Is homeopathy effective? No, but it is profitable … Is Traditional Chinese Medicine effective? Don’t know.

Be open, but skeptical! Examine your cognition!

Can we trust our senses? Our Can we trust our senses? Our minds? minds?

Harry McGurk and John MacDonald, “Hearing lips and seeing voices”, Nature 264, 746-748 (1976)

We all know visual illusions, but know much less about cognitive illusions.

N. Nicholls, Bulletin of the American Meteorological Society (1999).“These illusions, and ways to avoid them impacting on decision making, have been studied in the fields of law, medicine, and business. The relevance of some of these illusions to climate prediction is discussed here. The optimal use of climate predictions requires providers of forecasts to understand these difficulties and to make adjustments for them in the way forecasts are prepared and disseminated.”

Cognitive math and physics Cognitive math and physics G. Lakoff & R. E. Nunez, “Where mathematics comes from: How the Embodied Mind Brings Mathematics into Being”. Basic Books 2000Check Wikipedia article on Cognitive Mathematics.

How can a number express a concept? How can mathematical formulas and equations express general ideas that occur outside of mathematics, ideas like recurrence, change, proportions, self-regulating processes, and so on? How can "abstract" mathematics be understood? What cognitive mechanisms are used in mathematical understanding?“Can we understand the meaning of Euler’s formula ei

There is no agreement on the meaning of quantum mechanics.Can cognitive approach solve the problems in understanding physics?

Cognitive everythingCognitive everythingHas memory always worked as it does now? Ancient texts suggest significant differences.Dragon sightings in XI century, UFO in XX’s.Important for understanding history, ancient beliefs and religions, but also witness’ reliability, and even one’s own trustworthiness (false memory syndromes).

Blanke O, Landis T, Spinelli L, Seeck M, Out-of-body experience and autoscopy of neurological origin. Brain 127: 243-258, 2004. Fairly common experiences few people talk about.

Extensive brain imaging studies show disintegration of personal/extrapersonal space due to dysfunction of temporoparietal junction.

M. Persinger, "Neuropsychological Base of God Beliefs" (1987). ShaktiLight, transcranial magnetic stimulation device, creates in many people epiphanies, out-of-body experiences etc.

Few more CS applicationsFew more CS applicationsAnimal sociobiology is a result of specific brain organization forming a basis of moral behavior. Understanding origins of morality and ethics led to a conferenceon neuroethics at Stanford University in 2002. Empathy, the basis of compassion, is closely related to our ability to imitate and understand other people using mirror neurons.

Neuroaesthetics tries to understand what and why seems to be beautiful to our brains, why art and music are common in all human societies, and what can we learn about the brain from artist’s experiments.

S. Zeki, Inner Vision: An Exploration of Art and the Brain (1999).Neuroaesthetics Institute (UCL), UC Berkeley Department, Harvard’s Institute for Music and Brain Science … journals, conferences, courses.

Cognitive revolutionCognitive revolution

In many fields of science and art we are going through

Cognitive Revolution!

How about business and engineering?

CS & BusinessCS & BusinessNeuromarketing: brain imaging tells us more than we know ourselves about our minds …

Recent fMRI proof of the effectiveness of branding: people like Pepsi but choose Cola.

Understanding human decision-making is of primary importance for business and financial world.

Gerald Zaltman, How Customers Think: Essential Insights into the Mind of the Market (Harvard Business School Press), 2003.

Cognitive Approach to Understanding How-and Why-Customers Buy – great introduction to CS for business people.

Cognitive InformaticsCognitive InformaticsCognitive = the mental process by which knowledge is acquired. Informatics = The use of science, computer science, information and other technologies to provide data, information and knowledge to the individual and the organization.

School of Informatics, University of Edinburgh: "The study of the structure, behavior, and interactions of natural and engineered computational systems", representation, processing, and communication of information, including cognitive and social aspects.

Central notion: transformation of information, by computation or communication, in organisms or in computing artifacts.

Cognitive Informatics: AI, Cog Sci, computing … focusing on the way humans understand and solve problems.

Human-Computer InteractionHuman-Computer InteractionHuman factors: how to build a Human-Computer Interface that would be easy to use? How to test for information overload?

EPIC (Executive Process Interaction Control), human multitask simulator by D.E. Kieras (Michigan University, 1997).

Although quite useful to explain skill acquisition it does not capture human executive processes correctly.

Cognitive robotics & complex Cognitive robotics & complex devicesdevices

Robots need artificial minds, cognitive and affective control. In fact all complex devices need artificial minds to communicate with us effectively. Smart phones will soon have hundreds of functions, but the complexity of their use should be hidden from us.Human-Computer Interaction becomes central engineering problem.

Humanoid roboticsHumanoid roboticsRobots need artificial minds, cognitive and affective control. Toys – AIBO family is quite advanced, over 100 words, face/voice recognition, 6 weeks to rise, self-charging.

Qrio: Predicts its next movement in real time, shifts center of gravity in anticipation, very complex motor control, but little cognitive functions.

Wakamaru: recognizes faces, orients itself towards people and greets them, recognizes 10.000 words but does not understand much.

Artificial minds in robots and complex devices are still a dream …

Most advanced humanoid robots:Sony Qrio, standing-up, dancing, running, directing orchestra …

Mistsubishi-heavy Wakamaru, first commercially sold household robot (Sept 2005)!

Honda P3 Honda Asimo

Neuro-Robotics: using robots to Neuro-Robotics: using robots to investigate the braininvestigate the brain

Retina-likeVision system: 2 cameras

Anthropomorphic robot arm:8 d.o.f, 16 (8+8) proprioceptive sensors

Biomechatronic hand:10 d.o.f16 proprioceptive sensors135 tactile sensors

Anthropomorphic neck & head:7 d.o.f7 proprioceptive sensors

Age

learning of the sensory effects of motor commands

(S1,M1)

learning of the sensory effects of motor commands

(S1,M1)

learning of a multimodal

representation of hand movement

(PP)

learning of a multimodal

representation of hand movement

(PP)

learning of goal directed sequences

(PT)

learning of goal directed sequences

(PT)

learning of reach, grasp and

manipulation for simple objects

(PMd)

learning of reach, grasp and

manipulation for simple objects

(PMd)

learning of reach and grasp of various objects

with subsequent manipulation (PMv)

learning of reach and grasp of various objects

with subsequent manipulation (PMv)

Five “primitive learning steps” related to “five types of cortical connections”of the multi-network architecture forming “five learning modules”

Validating a step-wise learning theory for grasping and manipulation

EU IST-FET PALOMA Project – developmental approach

Artificial MindsArtificial MindsAM: software and robotic agents that humans can talk to & relate to in a similar way as they relate to other humans.

AMs need: some perception, inner representation of the world, language abilities, situated cognition, behavioral control.Cognitive architecture: the highest-level controller, responsible for executive functions (corresponding to frontal lobes in the brain).

Functions: recognition (patterns, situations, events) and categorization, different types of memory, use of associative recall, decision making, conflict resolution, improve with experience (learn), select information (pay attention to relevant inputs), anticipate, predict and monitor events, plan and solve problems, reason and maintain beliefs, search for additional knowledge and communicate with specialized agents that may find it ... Only a few such large-scale architectures exist.

Cognitive architecture: Cognitive architecture: SOARSOAR

Behavior = Architecture x Content

Symbolic, rule-based architectures based on theory of cognition.A. Newell, Unified theories of cognition (Harvard University Press, 1990).SOAR: architecture based on universal theory of cognition, more than 25 years of development, community of over 100 researchers. • NL-Soar, a theory of human natural language (NL)

comprehension and generation, explained many results from psycholinguistics.

• SCA, a theory of symbolic concept learning.• NOVA, a theory of visual attention matching human data.• NTD-Soar, a computational theory of the perceptual, cognitive,

and motor actions performed by the NASA Test Director (NTD)• Instructo-Soar, a computational theory of how people learn

through interactive instruction … and many others.

Steve = Soar Training ExpertSteve = Soar Training Expertfor Virtual Environments. for Virtual Environments.

STEVE, autonomous educational agent based on SOAR, helping students to learn how to operate and maintain complex equipment.

Virtual 3D environment – graphics design relatively easy. Hard: understanding student’s mind; use of natural language, understanding questions; monitoring performance of students in virtual space.Used in Intelligent Forces project. New type of intelligent tutor but not easy to create … SOAR-EPIC combinations

Cognitive architecture: Cognitive architecture: ACT-RACT-R

The Atomic Components of Thought, by J.R. Anderson & C. Lebiere (LEA 1998)

ACT-R, a cognitive architecture by John Anderson (CMU), has over 50 active groups in the world, including NASA, Naval Research, Sandia Labs ACT-R is a framework for creating models that incorporate the ACT-R's view of cognition and add new assumptions about the particular task. These assumptions can be tested by comparing the results of the model with experiments, for example time, accuracy or brain imaging data.ACT-R has been used successfully to create models in domains such as: learning and memory, problem solving and decision making, language and communication, perception and attention, cognitive development etc.Applications in human-computer interaction to produce user models that can assess different computer interfaces, education (cognitive tutoring systems) to "guess" the difficulties that students may have and provide focused help, computer-generated cognitive agents that inhabit training environments, neuropsychology, to interpret FMRI data.

Brain-inspired architecturesBrain-inspired architecturesG. Edelman (Neurosciences Institute) & collaborators, created a series of Darwin automata, brain-based devices, “physical devices whose behavior is controlled by a simulated nervous system”.

(i) The device must engage in a behavioral task. (ii) The device’s behavior must be controlled by a simulated

nervous system having a design that reflects the brain’s architecture and dynamics.

(iii) The device’s behavior is modified by a reward or value system that signals the salience of environmental cues to its nervous system.

(iv) The device must be situated in the real world.

Darwin VII consists of: a mobile base equipped with a CCD camera and IR sensor for vision, microphones for hearing, conductivity sensors for taste, and effectors for movement of its base, of its head, and of a gripping manipulator having one degree-of-freedom; 53K mean firing +phase neurons, 1.7 M synapses, 28 brain areas.

Engineering visionEngineering vision1. What would be the biggest engineering achievement?

To see results of your research used by almost everyone on Earth.

2. What is it that everyone is using? Portable phones and pilots to control TVs and other devices, getting smarter but also more complex and difficult to use every year.

3. What is the stumbling block in developing the new generation of even more useful smart phones that can do all we ask for? Their complexity, the difficulty to use all their functions fully, the need for tedious programming, in short human-machine communication.

4. What is the solution? Humanized InTerfaces (HITs) that would communicate with us in a natural way, ask minimum number of questions if the commands are ambiguous and do what we ask for: control household devices, help us to remember, communicate, access information and services, advice, educate, play word games ...

Cognitive informatics Cognitive informatics againagain

• Creation of such HITs with Artificial Minds interfaces is the greatest challenge to computer engineering.

• All other layers of software became more or less standardized, from BIOS hardware level to the graphics API for Windows user interfaces.

• Creation of an extensible platform for natural perception, language processing and behavioral modeling is the single most important subject left in computer engineering.

• This requires a concentrated effort of many people in an area that should be best called “cognitive informatics”: understanding how humans perceive, create their inner world, communicate and act, and creating artifacts that behave as “artificial minds”, that understand and interact with us in similar way as people do.

• Cognitive architectures created so far are good beginning, but are not sufficiently flexible to model many tasks.

Humanized InTerfaces (HIT)Humanized InTerfaces (HIT)

C2I = Center for Computational Intelligence, SCE NTU Flagship project, principal investigators:Wlodzislaw Duch & Michel Pasquier+ 15 other staff members so far …

HIT: definition and goalsHIT: definition and goalsHIT is a computer/phone interface that can interact in a natural way with the user, accept natural input in form of:

• speech and sound commands; text commands; • visual input, reading text (OCR), recognizing gestures, lip movement;

HIT should have a robust understanding of user intentions for selected applications.

HIT should respond and behave in a natural way. It may have a form of simulated talking head user can relate to, an android head, or a robotic pet.

Major goals of the HIT project: • develop modular extensible software/hardware platform for HITs; • create interactive word games, information retrieval and other

applications on PCs;• extend HIT functionality adding new interactivity & behavior;• move it to portable devices (PDAs/phones) & broadband services.

HIT: motivationHIT: motivation• HIT software/hardware/services may find their way to a billion

portable devices/phones in a few years time. The value of telephone ringtones alone in 2003 was 5 bln S$. New telephone functions include: camera, speech recognition, on-line translation, interactive games and educational software.

• Complexity of devices: a small fraction of the functions of electronic devices, such as PDAs, phones, cameras, or new TVs is used, new humanized interfaces that will help users are needed.

• Many applications in education, entertainment, services; talking heads connected to knowledge bases are already used in E-commerce.

• Creating HITs is a great computer engineering challenge, like building a rocket, it requires integration of many technologies and research themes, move research to a higher level. 17 SCE staff members expressed their interest and formulated HIT subprojects.

• A test-bed is urgently needed to experiment with such technologies.

HIT: state of the artHIT: state of the artHIT may draw from results of many large frontier programs, such as:Microsoft Research, offering free speech recognition/synthesis tools and publishing work on Attentional User Interface (AUI) project.DARPA’s Cognitive Information Processing Technology (call 6/2003). European Union’s Cognition Unit (started 10/2004) programs that have a goal to create artificial cognitive systems with human-like qualities.

Intel has projects in natural interfaces, providing free libraries for speech, vision, machine learning methods and anticipatory computing.

Talking heads already answer questions on Web pages for car, telecom, banks, pharmaceutical & other companies.Animated personal assistants work as memory enhancements and information sources, news, weather, show times, reviews, sports access...Services answering questions in natural languages are coming: AskJeeves and 82ask give answers (human) to any question!

But ... HITs are not yet robust, are still very primitive in all respects, with limited interaction with the user, poor learning abilities, no anticipation ...

HIT related areasHIT related areas

HIT projectsHIT projects

T-T-S synthesisT-T-S synthesis

Speech recognitionSpeech recognition

Talking headsTalking heads

BehavioralBehavioralmodelsmodels

GraphicsGraphics

Cognitive ArchitecturesCognitive Architectures

Cognitive Cognitive sciencescience

AIAI

A-MindsA-MindsLingu-botsLingu-bots

KnowledgeKnowledgemodelingmodelingInfo-retrievalInfo-retrieval

VR avatarsVR avatars

RoboticsRobotics

Brain modelsBrain models

Affective Affective computingcomputing

EpisodicEpisodicMemoryMemorySemantic Semantic

memorymemory

WorkingWorkingMemoryMemory

LearningLearning

HIT: proposed approachHIT: proposed approachProposed platform:

core functions: limitedspeech, graphics, and natural language processing (NLP)

+ extended functions:perceptual, cognitive, affective, specialized agents, behavioral.

Challenge and opportunity is to build modular platform for HIT on a PC, with 3D graphical head, robust speech recognition, memory, reasoning + cognitive abilities, and move it to new phones/broadband services.Uniqueness: nothing like that exists, requires a large-scale effort, integration and extension of many existing projects; collaboration with telecom and software industry, great student training.

Natural input modules

Cognitive functions

Affectivefunctions

Web/text/databases interface

Behavior control

Control of devices

Graphicaltalking head

Text to speechNLP

functions

Specializedagents

HIT sub-areasHIT sub-areasCore functions (5 experts) include HIT platform design/integration, robust limited vocabulary speech recognition, basic vision and learning; object and face tracking, synthesis of auditory and visual sensory signals, basic responses, control and talking head graphics.

Extended functions (13 experts) include synchronization of lip motion, analyzing prosody in speech, control of facial expressions, emotion analysis from face video, cognitive and affective functions, natural language analysis and context identification, cognitive learning, reasoning and theories of mind (user model), learning rules for behavior, integration with robot control, visual understanding, and hardware architecture design for selected applications.

Applications (5 experts) will initially include creation of specialized agents, implementing trivia / quiz/ educational and entertainment games, word games, 20 questions game, medical information presentation, office/ household automation, multi-lingual message (SMS, chat) understanding, an intelligent tutor to promote learning using appropriate gestures & more.

HIT benchmarksHIT benchmarksFirst year: we should be able to demonstrate basic HIT platform with:

• 3D graphic head with control over rich facial movements coupled with• visual object and face tracking coordinated with basic responses; • robust limited vocabulary speech recognition for playing word games;• speech synthesis and copycatting; • following simple spoken commands; • interface with WWW search agents finding and presenting information; • full specifications for adding and interacting with HIT functions.

Second year: initial applications and extensions of the HIT platform, work on the natural language processing and cognitive modeling: • demonstrate HIT playing the 20 questions game, selecting most

informative questions that help to define the subject uniquely; • playing educational word games implementing trivia and quizzes;• medical information presentation system answering simple questions; • use of HIT in the NTU virtual campus; • organize an internal competition for new applications of HIT;• preliminary designs for moving the HIT platform to portable devices.

HIT extensions & managementHIT extensions & managementThird year and later: Improve existing platform and applications, move them to portable devices, add new functions needed for specific applications.For example, auditory and visual scene analysis, fusion of sensory signals, testing of perceptual modules inspired by neuroscience, applications of HIT for programming household devices, tutor programs to teach problem solving for some local courses.

A competition for adding such new functions and applications to the HIT platform will be made at NTU; this requires creation of test data, scenarios of use, providing specification for the interface, selection/sponsoring of the best proposed approach, and integration in the HIT.

Parallel project submitted to A*STAR: Developmental Robot-Embedded Artificial Mind (DREAM), aimed at controlling real android head, focused on comparison of general cognitive architectures; includes collaboration with 3 leading groups in cognitive modeling (Berkeley, Michigan and Memphis University), plus KAIST (Korea) on natural perception.

Word gamesWord gamesWord games that were popular before computer games took over. Word games are essential to the development of analytical thinking skills. Until recently computer technology was not sufficient to play such games.

The 20 question game may be the next great challenge for AI, much easier for computers than the Turing test; a World Championship with human and software players?

Finding most informative questions requires understanding of the world.

Performance of various models of semantic memory and episodic memory may be tested in this game in a realistic, difficult application.

Asking questions to understand precisely what the user has in mind is critical for search engines and many other applications.

Creating large-scale semantic memory is a great challenge: ontologies, dictionaries (Wordnet), encyclopedias, collaborative projects (Concept Net) … movie

Humanized interface

Store

Applications, eg. 20 questions game

Query

Semantic memory

Parser

Part of speech tagger& phrase extractor

On line dictionaries

Manual

verification

Puzzle generatorPuzzle generatorSemantic memory may invents a large number of word puzzles that the avatar presents. The application selects a random concept from all concepts in the memory and searches for a minimal set of features necessary to uniquely define it. If many subsets are sufficient for unique definition one of them is selected randomly.

It is an Amphibian, it is orange and has black spots. How do you call this animal?

A Salamander.

It has charm, it has spin, and it has charge. What is it?

If you do not know, ask Google!Quark page comes at the top …

Developmental Robot-Embedded Developmental Robot-Embedded Artificial Mind (DREAM)Artificial Mind (DREAM)

Ng Geok See, Wlodzislaw Duch, Michel Pasquier, Quek Hiok Chai, Abdul Wahab, Shi Daming,John Heng

Centre for Computational IntelligenceNanyang Technological University

Objectives and Motivations• To develop a robotic head endowed with complex cognitive

processor that recognizes and interacts with humans using natural means of communication.

• To facilitate integration of research areas in perception (signal processing, computer vision), real-time control, natural language processing and cognitive modeling.

• To create new applications for educational games, office assistants, chatterbot interfaces, etc.

• To provide training of graduate students in a new field of high potential importance for Singapore economy.

• This project is at the heart of cognitive robotics, a very important field not yet pursued in Singapore.

Related work

Android receptionists Inkha in the King’s College London

Although functionality of these receptionists is very limited these projects have been very popular, including a note in “Nature”.

The Carnegie-Mellon University receptionist Valerie.

DREAM architecture DREAM architecture

Natural input

modules Cognitive functions

Affectivefunctions

Web/text/databases interface

Behavior control

Control of devices

Talking head

Text to speechNLP

functions

Specializedagents

DREAM is concentrated on the cognitive functions + real time control, we plan to adopt software from the HIT project for perception, NLP, and other functions.

Research PartnersResearch PartnersMain goal: integration of perception with cognitive architectures, tests of 4

architectures and creation of new architecture for android head control.

• Brain Science Research Center of the Korea Advanced Institute of Science and Technology headed by Soo-Young Lee. Need his expertise for perception and brain-inspired computational models.

• SOAR by John Laird, University of Michigan. His experience will be helpful in interfacing SOAR with natural perception and motor control of our android.

• The “Conscious” Software Research Group headed by Stan Franklin, Institute of Intelligent Systems, University of Memphis; we will use his Intelligent Distribution Agent (IDA) for solving problems that require complex reasoning.

• Shruti by Lokendra Shastri, International Computer Science Institute, Berkeley; we will use Shruti for understanding natural language commands and for formation of episodic memories as a result of interaction with users.

Some applicationsSome applications• DREAM will serve as a application platform for human-computer

interactions, for example in projects where evolution of language and development of mind through natural interactions with people is investigated (at present text-based interfaces are used).

• Interface to word games, such as trivia games, educational quizzes, games requiring reasoning, and the 20 question games.

• Natural interface to information resources, such as structured internet databases (MIT Start system), encyclopedias, or transportation.

• Receptionist, science museums, office mates ...

• An interface to control household devices. • Taking care of old people, reminding them of their schedule, talking to

them to enhance their memories and collect their life stories (in collaboration with Alzheimer clinic in Bad Aibling, Bayern, Germany).

• Creating alter-ego, by mimicking behavior of people, understanding their stories (forming episodic memories) and asking additional questions to gather more information.

• Creating realistic instincts for robotics pets.

Emovere: Emovere: A Neuro-Cognitive Computational A Neuro-Cognitive Computational

Framework for Research on EmotionsFramework for Research on Emotions

David Cho, Quek Hiok Chai, W. Duch, Ng Geok See, A. Wahab,

John Taylor (King College) and Looi Chee Kit, (NIE/LSL)

MotivationsMotivations• Human can express emotions easily, but from computational point

of view emotions are not easy to understand.

• Emotions are an important factor in intelligent behavior, including problem solving, they can help to focus attention on correct reasoning.

• Understanding on how to capture real emotions in artificial system is a challenging problem.

• Research on computational approaches to emotions is a state-of-the-art basic research topic.

• HIT, DREAM and intelligent tutor projects will benefit from affective computing.

What is Emotion?What is Emotion?

“The core of an emotion is readiness to act in a certain way; it is an urgency, or prioritization, of some goals and plans rather than others. Emotions can interrupt ongoing action; also they prioritize certain kinds of social interaction, prompting, for instance, cooperation or conflict.”

From N.H. Frijda, The Emotions, Cambridge, 1986.

ObjectivesObjectives• Neurocognitive computational framework for emotion

– Encoding of Emotional Tags– Encoding of Episodic Memory using Emotional Tags and

Emotional Expressions.– Information flow between memory modules, computational

model similar to the brain info flow

• Fusion of Multiple Modal Inputs– Facial, gesture, body posture, prosody and lexical content in

speech.

• Application– Implementation and validation of the model in

an Intelligent Tutoring System.

Proposed Neuro-Cognitive FrameworkProposed Neuro-Cognitive Framework

Emotions using visual cues: face, gesture, and body posture.

• Emotional face processing– Feedforward sweep through primary visual cortices ending up in

associative cortices.– Projections at various levels of the visual (primary) and the

associative cortices to the amygdala.– Activation of the prefrontal cortex initiate a re-prioritization of the

salience of this face within the prefrontal cortex area.– Amygdala generates or simulates a motor response providing

effectively a simulation of the other person’s emotional state.

• Emotional gesture processing is similar to face processing.• Emotional body posture understanding:

– Body movements accompany specific emotions. – Coding schemata for the analysis of body movements and

postures will be investigated.

Emotions using auditory cues: Emotions using auditory cues: linguistic and prosodylinguistic and prosody

• Speech carries a significant amount of information about the emotional state of the speaker in the form of its prosody or paralinguistic content.

– Temporal recurrent spiking networks have already been used in identification of prosodic attitudes, but only using fundamental frequency, still 6 attitudes were distinguished with 82% accuracy.

– Primary and high level auditory cortices are involved in the extraction and perceptual processing of various prosodic cues.

– The amygdala and the pre-frontal cortex appears to be responsible for translating these prosodic cues into emotional information regarding the speech source.

Affect-based Cognitive Skill Instruction in Affect-based Cognitive Skill Instruction in an Intelligent Tutoring Systeman Intelligent Tutoring System

• Intelligent Tutoring Systems (ITS)

– Integrating characteristics proper of human tutoring into ITS performance.

– Providing the student with a more personalized and friendly environment for learning according to his/her needs and progress.

– A platform to extend the emotional modeling to real life experiments with affect-driven instruction.

– Will provide a reference for the use of affect in intelligent tutoring systems.

IDoCareIDoCare: : IInfant nfant DDeveleveloopment and pment and CareCare for development of perfect babies!for development of perfect babies!

W. Duch, D.L. Maskell, M.B. Pasquier, B. Schmidt, A. Wahab School of Computer Engineering, Nanyang Technological University

Problem: about 5-10% of all children have a developmental disability that causes problems in their speech and language development.Identification of congenital hearing loss in USA is at 2½ years of age!

Solution: permanent monitoring of babies in the crib, stimulation, recording and analysis of their responses, providing guideline for their perceptual and cognitive development, calling an expert help if needed. Key sensors: suction response (basic method in developmental psychology), motion detectors, auditory and visual monitoring.

Potential: market for baby monitors (Sony, BT...) is billions of $; so far they only let parents to hear or see the baby and play ambient music.

IDoCareIDoCare intelligent crib intelligent cribRevolutionary enhancement of baby monitors: intelligent crib with wireless suction, motion detector and audio/visual monitoring, plus software for early diagnostics of developmental problems.

Hardware: embedding pressure and temperature sensors in telemetric pacifier, for monitoring and feedback of baby's reactions to stimuli.Software: signal analysis and blind source separation; interpretation of baby’s responses, selection of stimuli and comments for parents. Home applications: monitoring, diagnostics, preventive actions by enhancement of perceptual discrimination by giving rewards for solving perceptual problems.

Children love to be stimulated, and IDoCare will be the first active environment that will allow them to influence what they see and hear.

Active learning may gently pressure baby’s brain to develop perceptual and cognitive skills to their full potential achieved now by very few.

Telemetric pacifier

Control unit la-la … la-ra-ra… sound sequences

Database of speech sounds A/D converter receiver

Speaker

Audiovisual device (reward)

D/A converter

Wireless communication

Non-volatile memory

Database of reward patterns

RAM

Phases, Resources & AlliancesPhases, Resources & Alliances• Phase 1 (2008): Build wireless pacifier for monitoring suction, temperature and heartbeat. Use it in baby monitors; create a spin-off company to market it.Collect data from many children and develop algorithms to interpret it. • Phase 2 (2011): Build intelligent crib with many sensors;collect data and develop algorithms for interpretation, including early diagnosis of hearing and speech impairments;Stimulate and challenge babies using perceptual/cognitive games.

• Phase 3 (2015): IDoCare hi-tech cribs in every house; dyslexia and other problems largely a thing of the past; follow-up studies show that children have perfect ear, speak foreign language with no accent, and have IQ>150; Nobel prize in medicine awarded to our team in 2016. Collaboration: KKH & Cincinnati Children’s Hospital Research

Foundation, Dept. of Biomedical Informatics.

SummarySummaryThese were only a few selected cognitive informatics projects initiated recently. Other projects include:

Brain as Complex System (BRACS) global brain simulation (EU project just submitted, coordinated by J.G. Taylor),

Brain-Inspired Model of Skill Learning: From Conscious Cognition to Subconscious Actions.

Understanding creativity and implementing it in artificial systems.

The future of art, science, business and engineering is cognitive !