content, multimedia tools and markets linguistic applications philippe gelin interface technologies...

Content, Multimedia Tools and MarketsLinguistic Applications Philippe Gelin

Interface technologies Interface technologies and cognitive systemsand cognitive systems

Granada 16th December 2002Granada 16th December 2002

Philippe GelinPhilippe GelinEuropean Commission - Information SocietyEuropean Commission - Information Society


Content

PART I : European Research Area

PART II : Interface Technologies

PART III : Cognitive Systems

PART IV : FP6 Instruments


PART I

The European Research Area (ERA) and

its Sixth Framework Programme (FP6)


FP6

The Treaty establishing the European Community provides for the creation of multiannual research and technological development

The Sixth Framework Programme (FP6) is the European Unions main instrument for the funding of research in Europe within 2002-2006.

The overall budget for FP6 is 17.5 billion euro.


ERA

In the past, collaborative actions have been initiated at European and Community level,

but now is the time to bring our endeavours together and to build a research and innovation equivalent of the "common market" for goods and services.

That structure is called the European Research Area (ERA) and is regrouping all Community supports for the better coordination of research activities and the convergence of research and innovation policies, at national and EU levels.


ERA


From ERA to IST

1. Focusing and integrating Community research 1.1. Priority thematic areas of research 1.1.1. Life sciences, genomics and biotechnology for health (i) Advanced genomics and its applications for health. (ii) Combating major diseases. 1.1.2. Information Society Technologies (3.625 billion Euro)1.1.3. Nano-technologies and nano-sciences, knowledge-based multifunctional materials, and new production processes and devices 1.1.4. Aeronautics and space 1.1.5. Food quality and safety 1.1.6. Sustainable development, global change and ecosystems (i) Sustainable Energy Systems (ii) Sustainable surface transport (iii) Global change and ecosystems 1.1.7. Citizens and governance in a knowledge-based society 1.2. Specific activities covering a wider field of research 1.2.1. Policy support and anticipating scientific and technological needs 1.2.2. Horizontal research activities involving SMEs 1.2.3. Specific measures in support of international cooperation 2. Strengthening the foundations of the European Research Area 2.1. Support for the coordination of activities 2.2. Support for the coherent development of policies


IST Programme 2002-2006

Instantiate the European challenges (ERA), Initiatives

Trans national partnerships Critical mass of skills

and adequate resources hence ... Focus, Excellence, Impact and RoI


IST Vision

IST today …………………………..The IST in FP6 vision

PC based ………………………………. “Our surrounding” is the interface “Writing and reading”…………….…. …Use all senses, intuitive “Word” based information search……….Context-based knowledge handling Low bandwidth, separate networks…. ….Infinite bandwidth, convergence, .. Mobile telephony (voice)………….. ……Mobile/Wireless full multimedia Micro scale………………………………Nano-scale Silicon based……………………………..+ new materials e-Services just emerging……………….. Wide adoption (eHealth, Learning,..) < 10% of world population on-line….. …World-wide adoption


ISTIST

Communication ComputingSoftware

ComponentsMicrosystems

Technologies for major economicand societal challenges

Future andEmerging Technologies

www.cordis.lu/ist/fp6/fp6.htm

Knowledge andInterface Technologies


Call 2

Advanced displays Optical, opto-electronic, & photonic

functional components Open development platforms for

software and services Cognitive systems (25ME) Embedded systems Applications and services for the

mobile user and worker Cross-media content for leisure and

entertainment (55ME) GRID-based Systems for solving

complex problems Improving Risk management

Strategic Objectives in 2002-2003 WP

Call 1

Pushing the limits of CMOS, preparing for post-CMOS

Micro and nano-systems Broadband for all Mobile and wireless systems beyond 3G Towards a global dependability and

security framework Multimodal interfaces (65ME) Semantic-based knowledge systems Networked audiovisual systems and

home platforms Networked businesses and governments eSafety of road and air transports e Health Technology-enhanced learning and

access to cultural heritage (65ME)

80% of budget is pre-allocatedRemainder flexible, to be allocated after evaluation

Allocation only for the first 2 years: to be revised for 2005-2006

!!PROVISIONALPROVISIONAL

!!!PROVISIONALPROVISIONALPROVISIONAL


PART IIPART II

Interface Technologies in FP 6

scope and focus in 2003


Outline

Why interface technologies? IST Vision

What research? Focus of first calls in FP6

Where are we today? Ongoing activities


Vision

Building the knowledge society for all

Allow people to use the surrounding hi-tech easily and in a way that is consonant to humans rather than to computers

Facilitate interpersonal communication anywhere, anytime, beyond linguistic and cultural barriers

Assist people and augment their abilities when interacting and communicating


Needs

Bring people to the foreground – build technologies for the background

Our surrounding is the interface: everywhere, embedded in everyday objects (e.g: clothes, furniture, vehicles, smart materials, …)

Exploring interfaces beyond the PC, screens & keyboard. (brake the chains…)

Enabling multi-sensorial dialogues supported by ubiquitous computing & networking

Multi-linguistics, multi-cultural interactions


Known Challenges

Strongly multidisciplinary with many component technologies

Significant system level integration issues

Research resulting in autonomous (self-learning, self-organising), adaptive (time varying) systems that can work with partial and uncertain information


2003-2004 FOCUS

Multimodal Interfaces (in call 1) Objective: To develop natural and natural and

adaptive multimodal interfacesadaptive multimodal interfaces, that respond intelligently to speech and language, vision, gesture, haptics and other senses.

Focus 1Focus 1 Multimodal interactionMultimodal interaction

Focus 2Focus 2 Multilingual communicationMultilingual communication


Multimodal Interaction

Interaction between and among humans and the virtual and physical environment

intuitive multimodal interfaces that are autonomous and capable of learning and adapting to the user environment in dynamically changing contexts. They should recognise emotive user reaction and feature robust dialogue capability with unconstrained speech and language input.

Human-to-human: technology mediated communication

Human-to-things: virtual and physical

Human-to-self: health, well being

Human-to-content: information retrieval/browsing

Device-to-device: human mediated device communication

Human-to-embodied robots


Multilingual Systems

facilitating translation for unrestricted inputs, especially for spontaneous (unrestricted) or ill-formed (speech) inputs, in task-oriented settings

Unrestricted spontaneous speech-to-speech translation in task-oriented settings

Statistical/mixed approaches to translation

Adaptation to task/user, learning

Robustness


Basic Research

Basic research, component technologies

Examples:• accurate vision

• gesture tracking

• speech and audio processing

• language technologies

• affective computing

• machine learning

• autonomous systems

• fusion of multiple channels


System Level Integration

Proof of concept in challenging application domains, including:

wearable interfaces and smart clothes

intelligent rooms and interfaces for collaborative working tools

cross-cultural communications

usability issues and evaluation


Shared Infrastructure

Data: large amounts of multimodal data, synchronisation and IPR issues

Metrology, technology evaluation, usability

Infrastructural knowledge: machine learning, applied mathematics

Best practice and standards

Tools and platforms

Socio-economic issues (e.g. human factors)


Current Status


Ongoing Activities

Collection, processing and browsing of multimodal meeting data systems that enable recording, structuring, browsing and querying

of an archive of multi-modal recordings of meetings.

Multimodal, multicultural, multilingual communication integration of multiple communications modes - vision, speech

and object manipulation - by combining the physical and virtual worlds to provide support for multi-cultural communication and problem solving

Speech-to-speech translation development of speech-to-speech translation and its integration in

automatic in e-commerce and e-service environments

data collection for speech-to-speech translation


Ongoing Activities

Preparing future multisensorial interaction research providing technological baselines, comparative evaluations, and

assessment of prospects of core technologies for speech-to-speech translation, the detection and expressions of emotional states, and technologies for children’s speech.

Automatic animation of human models design and development of a virtual person animation system in

controlled environments, which enables the modeling, analysis and simulation of human motion.

Recognition of the user’s emotional state human computer interaction that can interpret its users' attitude or

emotional state in terms of their speech and/or their facial gestures and expressions.

See www.hltcentral.org for more information


PART IIIPART III

Cognitive Systems in FP6

scope and focus


Outline

why cognitive systems?

IST vision

what research?

focus of call

where are we starting from?

current activities


VisionAmbient intelligence

“… ability of computationally empowered devices to interconnect with each other and with us …”

sensors provide a window from world of interconnected computation into real physical world

using sensors, these devices will sense the world around us and respond by interacting with the world or by communicating with us

they need to ‘see’, ‘hear’, …. ‘understand’ and ‘act’

Ambient Intelligence requires perceptual systems capable of cognition


Objective

To construct physically instantiated or embodied systems that can :

perceive, understand (the semantics of information conveyed

through their perceptual input) learning (structured memory) interact with their environment, (CCI & CHI

communication) evolve in order to achieve human-like performance in

activities requiring context-(situation and task) specific knowledge.


Focus 2002-2003

Research will aim at realizing:

complete systems with real-time performance and/or bounded rationality, have well developed memory capacities (e.g. short

term, long term, iconic, associative) with efficient representation,

and that acquire representations as needed to realise performance goals.

to perform goal-directed tasks

The emphasis is on closing the loop in realistic test cases.


Known Challenges

A main target of this research is interdisciplinarity:

computer vision, natural language understanding, robotics, artificial intelligence, mathematics and cognitive neuroscience

and its impact on overall system design.


Outcomes

Direct Applications:– in image recognition, in behavioural interpretation,

in video annotation, speech recognition, in automatic categorisation and classification, in goal-specification and decision support,…

– can serve to demonstrate and measure progress but applications NOT main target of research

Methods & approaches for constructing robust & adaptive systems capable of cognition


Ongoing Activities

cognitive vision systems 2000 – 2002 robust image recognition

from application-specific to generalised solutions

from focus on low-level processing & robustness of individual components to systems approach where every component (incl. high-level cognitive functionalities) has a role to play in assuring robust behaviour of system

longer-term, largely academic, interdisciplinary effort

8 RTD projects & 1 Network: www.ecvision.info


Related Topics

Multimodal Interfaces, call 1 intuitive multimodal interfaces ... autonomous and capable of learning and adapting … in dynamically changing contexts

Beyond Robotics, call 1 focus on open-ended lifelong learning systems

Disappearing Computer, call 1 focus on open architectures allowing arbitrary

combinations of ‘building blocks’ for ‘universal application’

Embedded Systems, call 2focus on networked systems, distributed control


PART IVPART IV

Implementation


Outline

Schedule New Instruments Old Instruments Applied to IFT & CS. Expected results Learning from EoI “Do’s and don’ts” Conclusions


FP6 ROADMAP 21 February 2001 Commission decides on first proposal for Framework Programme March 2001Informal presentation to the Ministers of Research and Education during the Uppsala

Informal Council (1 - 3 March) May 2001 Adoption by the Commission of proposals regarding the specific programmes and the

implementation of the Framework programme September 2001 Adoption by the Commission of proposals for decisions on the rules for participation

and dissemination of results 26 June 2001 Council discusses Commision proposal July/Nov. 2001 First reading by the European Parliament 12/2001- 01/2002 Common position of the Research Council 11 March 2002 Council adopt final text 15 May 2002 The European Parliament approves the Sixth Framework Programme 3 June 2002 Council of Ministers formally adopted the Sixth Framework Programme 12 June 2002 Parliament approves specific programmes 19 June 2002 FP6 rules of participation compromise approved by ITRE 3 July 2002 FP6 rules of participation compromise approved by EP September 2002 Specific Programmes has been adopted (October) End 09/2002 Rules for Participation to be formally adopted by Council and Parliament November 2002 Conference marking the launch of the Sixth Framework Programme 12/2002 First calls for proposals end 2003 Second calls for proposals


Publication: 2nd half December 2002

Call deadline: 24 April 2003

Evaluation: weeks of 12th and 19thMay 2003

IP/NoE hearings: week of 9th June 2003

Negotiations: from July 2003

Commission Decision: from October 2003

Project start: January 2004

Call 1 Timetable !!PROVISIONALPROVISIONAL



Overview of FP6 instruments

Integrated Projects (IP)

Networks of Excellence (NoE) Article 169 (joint implementation of national

programmes)

Specific Targeted Research Projects (STReP’s)

Co-ordination actions (CA)

Specific Support Actions (SSA)

„New“

„Old“


Use of New Instruments


Why new instruments ?

Adapt to the size of the challenge The philosophy of both instruments is to move from

multiple project funding to the funding of coherent programmes of research activities,

Flexibility and responsibility leaving the highest degree possible of autonomy and

flexibility to European research consortia.


Expected to address the objectives within a holistic approach enabling,

where justified, competition within and across projects

A major target is to integrate actors and disciplines = build a community

An IP should be THE project in the target area

“critical mass”

Integrated Projects


Integrated Projects

Core activities genuine research work “engineering” tasks (esp. methods & tools) system integration & validation (“total system”

approach) along with

promotion & dissemination of results training, awareness & best practice (researchers,

integrators, launching users) cooperation & exchanges with related national

and international efforts (incl. standards bodies) socio-economic impact & consequences


Example of an IP

Knowledge & Data bases

Gesture & Facial Recognition

Intelligent Virtual Worlds

Taste Smell

Body Sensors Displays Actuators

Speech Output

Speech

Intelligent Assistants & Virtual HumansPhysical Virtual World

Virtual & Augmented Realities

Interface Layer

Integrated projectCurrent projects

Area: Multi-sensorial interfaces

Haptic devices


0 6 12 18 24 30 36 42 48 Months

Reporting

Accepted costs

Reporting Detailed work plan

Accepted costs Adjusted advance

Reporting

Detailed work planAccepted costs

Adjusted advance

Reporting

Accepted costs Detailed work plan

Adjusted advance

Detailed work plan

Initial advance

IP - payment & report schedule

Example of 4 year contract


Networks of Excellence

Spread excellence Xfertilisation among disciplines Bring together the best research in

constituent disciplines, integrate resources, (e.g. image, video, multimodal,…)

Create a NEW research space (ERA) Have a lasting nature Define joint activities


Networks of Excellence

Joint programme of activities: Integrating activities

coordinated programming of the partners’ activitiessharing of tools/facilities/ infrastructuresmanaging the knowledge portfoliostaff mobility and exchanges

Joint research activitiesundertaking research on topics

supporting the network’s goals Spreading excellence

training of researchers and other key staffdissemination and technology transfer to industry

“virtual” centres of excellence

New or additional

to ongoing efforts


Each call will fix the founding per researcher, based on the characteristics of the field of research concerned

Computer science less expensive than nuclear research

The number of researchers implied in the integration

PhD or more than 4 years of researchExisting head count at time of proposal.Table of Found Per researcher issued at each call

@ max the cost incurred to implement the JPA @ max 25% of the existing value proposed to

be integrated

NoE - grant


Use of Old Instruments


Specific targeted research projects

Similar to Old instrument, except for: joint and several financial liability costing / funding adapted to grant scheme rates / activities: research (50%),

demonstration (35%), combined research & demonstration

Activities focussed RTD and / or demonstration;

management Partnership

more limited than for IPs; in terms of breadth, time, scope


Coordination & specific support actions

Coordination Actions Instrument for ad-hoc co-operation between

organisations (no durable integration !), similar to FP5 thematic networks

Activities include expert groups, seminars, studies, staff exchanges, project definition, etc

Can cover up to 100% of eligible costs, …

Specific Support Actions For conferences, working groups, fact finding, technology

transfer, operational support and dissemination Can cover up to 100% of eligible costs, and be carried

out by one participant Call for Grants: conferences, workshops or exhibitions


Article 169

EC funding to support jointly executed member state programmes most powerful tool for integration

However, long-winded implementation: to generate a proposal requires a co-initiative

by national programmes and the Commission Council and Parliament would have to

approve


Expectation

Within Multimodal Interface&

Cognitive systems


Expected OutcomeCalls 1 and 2

Multimodal Interfaces and Cognitive Systems

2/3rd of funds devoted new instruments

6-10 new instruments

8-12 old instruments




Interface Technology Tools

IPs are expected to address the objectives within a holistic approach enabling, where justified, competition within and across projects.

NoEs should aim at lowering barriers between hitherto split communities and disciplines and advance knowledge in the field. They should help establish and reinforce shared infrastructures, including for training and evaluation, annotation standards and appropriate usability metrics and benchmarks.

STREPs are expected to bootstrap research in identifiable or emerging sub-domains and to prepare associated communities.


Cognitive System tools

Integrated Projects are expected to leverage these communities to integrate methods and insights towards the objective of realizing entire systems and to promote community building.

NoEs will provide a channel for fostering foundational research, for developing and maintaining common resources, specifically, of open systems and training environments to study learning and evolving systems.


EoI Analysis


Analysis of submitted EOIs14 ~evenly populated clusters

Infrastructures, resources and basic technologies Interface and Artefact Design and Ergonomic, Human Factors and User

Acceptance Cognitive Science for Ambient Intelligence Improvement of Human Senses and Perception Speech and Vision Based Intelligent Interfaces Interfaces Based on Recognition and Synthesis of Emotion, Facial Expression

and Gesture Multilingual Access and Dialogue Intelligent Wearable Devices and Smart Cloths Interfaces for Mobile Devices and Services Intelligent Rooms with Support for Collaborative Creative Processes and Virtual

Environments Multisensory interfacing with complex information spaces and virtual

manipulation of information objects Virtual Reality and Environments Critical Interfaces for Safer Control of Vehicles and Devices, including Training

for these Environments Physical Interfaces, Haptics, Mechatronics and Robotics


Analysis of Submitted EOIs

150+ EoIs in Interface technologies (incl. Cognitive Systems)

70% IPs, 30% NoEs

60% of submitted IPs “defective”

lack of required ambition, scope and integration some could be part of other IPs or a STRP

30% of the submitted NoEs “defective” lack the main characteristics of this instrument

e.g. joint activity programme, lasting integration of existing research capacity

some more appropriate as coordination actions


EoI per Country


Content Narrow scope, little or no EU and ERA dimension Aims too general, lack of focus (e.g. programmes

instead of projects) Lack of innovation, not beyond current state of art

Planning Links missing between objectives & workplan Milestones missing or too general Risk factors not addressed, no contingency plans

Management Management structure vague or inadequate Weak dissemination, exploitation plans

EoI Shortcomings


EoI Partnership Size

Nino Varile /


How To: Partnership Highly competent and reliable partners

Complementarity: cover all areas you need

Duplication of competence:

acceptable for IPs depending on project needs

necessary for NoEs

At least 3 of which 2 EU/NAS or associated states

Ratio between Industry/SME/NAS/Academia participation depending solely on project needs


Do’s and Don’ts

Do carefully study the evaluation criteria

relevance, excellence, ……

Do respond to all the evaluation criteria, not just the scientific & technical ones

Do remember that proposals are evaluated by peers = “no-nonsense”

Don’t mirror the workprogramme: put your own ideas in the proposal

Don’t participate in a multitude of competing proposals, select one and make it win


Do pay extra attention to using the full range of activities allowable for the new instruments

Don’t create artificial IPs (=inflated STREPs)

Do pay due attention to co-ordination of large projects

Do give realistic cost/resource estimates good return on investment

Remember that 50 MEUR ~ 1,200 person/years!

Do’s and Don’ts


Reminder

FP6 is about more ambitious endeavours, in a simpler procedural environment

IPs should aim at breakthroughs in the target area and yield new technologies, products and services

NoEs are to structure and shape research areas (interdisciplinary !)

STREP’s are more limited in scope but must go well beyond the state of the art


IST Workprogramme 2003-2004 Call for proposals text Call 1 (Official Journal)

Submission details (deadline, address...) Brochure “The 6FP in Brief”

FP6 and Specific Programmes, instruments, proposal to contract (flowchart), contractual matters, etc

Guides for proposers Call 1 How to submit, references,..., proposal forms Instrument-specific annexes

Evaluation Manual “High level” description of ground rules for evaluation

Guidelines for Evaluators Call 1 Evaluation process, evaluation forms

IST Infopack


Part A Form A1 - General information Form A2 - Information on participants Form A3 - Cost breakdown (cost and grant

requested as per activity / participant)

Part B Aspects relating to evaluation criteria Other aspects (ethics, safety, gender) Overall workplan of project

IPs, NOEs additionally provide 18 month Implementation plan/JPA (detailed)

Proposal content

No anonymity

Mats Ljungqvist /


Evaluation Process

Peer-review system (many experts needed !) 2-step evaluation (not in Call 1) Hearings of applicants (likely in Call 1)

Evaluation & selection criteria



IInntteeggrraatteedd PPrroojjeeccttss((IIPP))

NNeettwwoorrkkss ooffEExxcceelllleennccee ((NNooEE))

TTaarrggeetttteedd RReesseeaarrcchhpprroojjeeccttss ((SSTTRREEPPSS))

RReelleevvaannccee ttoo oobbjjeeccttiivveessooff tthhee pprrooggrraammmmee



PPootteennttiiaall iimmppaacctt PPootteennttiiaall iimmppaacctt PPootteennttiiaall iimmppaaccttSScciieennttiiffiicc && tteecchhnniiccaalleexxcceelllleennccee

EExxcceelllleennccee ooff tthheeppaarrttiicciippaannttss

SScciieennttiiffiicc && tteecchhnniiccaalleexxcceelllleennccee

QQuuaalliittyy ooff ccoonnssoorrttiiuumm DDeeggrreeee ooff iinntteeggrraattiioonnaanndd tthhee jjooiinntt pprrooggrraammmmeeooff aaccttiivviittiieess

QQuuaalliittyy ooff ccoonnssoorrttiiuumm

QQuuaalliittyy ooff pprroojjeeccttmmaannaaggeemmeenntt

OOrrggaanniissaattiioonn aannddmmaannaaggeemmeenntt

QQuuaalliittyy ooff pprroojjeeccttmmaannaaggeemmeenntt

MMoobbiilliissaattiioonn ooffrreessoouurrcceess

--- MMoobbiilliissaattiioonn ooffrreessoouurrcceess

Higher thresholds on partnership and management.

Higher weights on impact and excellence.

Mats Ljungqvist /


Stay in Touch!www.cordis.lu/fp6/www.cordis.lu/rtd2002/www.cordis.lu/ist/www.cordis.lu/ist/fp6/fp6.htmemail: [email protected] ftp://ftp.cordis.lu/pub/ist/docs/

istagscenarios2010.pdf

www.cordis.lu/experts/fp6_candidature.htmexperts!!


Contact info

Official address:

European Commission, EUFO 0170 Jean Monnet Building,Rue Alcide de Gasperi,L-2920 Luxembourg.

Express Courier & visitors:

European CommissionEUFO 0170Euroforum, Building,10, rue Robert StumperL-2557 Luxembourg

Tel.: +352 - 4301 - 33423, Sec.: +352 - 4301 - 34178. Fax : +352 - 4301 - 34999 e-mail: [email protected]


Thank You

content, multimedia tools and markets linguistic applications philippe gelin interface technologies...

Documents

multimedia tools

community research

european research area

funding of research

convergence of research

european community

horizontal research

roi slide