introduction - ghum.kuleuven.be · • emoticons • pictographs for people with disabilities....

Introduction

Augmentative and Alternative Communication

• Assist people with Intellectual Disabilities (ID)• Allow them to communicate or use the Internet• Increase life quality by reducing social isolation• Picture-based communication systems are

a form of AAC technology

Introduction

Picture-based Communication

Three types:• Universal pictographs• Emoticons• Pictographs for people with disabilities

Beta pictographsSclera pictographs

hond (dog) ik (I)

fruit-persen (squeeze-fruit)

gebraden-kip (roasted-chicken)

hond (dog) ik (I)

hond-eten-geven(dog-food-feed)

afwas-proper-terugbrengen

(dishes-clean-return)

Introduction

Pictograph Sets

Introduction

Problem Definition

• The internet and social media websites play an important role in our lives– However, they are not accessible for people with ID– Reading and writing difficulties, no access to pictographs– There is a risk of e-exclusion

• Solution: Text2Picto and Picto2Text

• Enabling internet access for people with ID• Specific applications for these users

– Chat and e-mail client with the Picto engines– Easy-to-Read News – Text-to-Speech– Games

• www.wai-not.be

The Users

WAI-NOT

• European project: Beginning of March 2014 – End of March 2017• Goal: Improve the living conditions of people with Intellectual

Disabilities– Natural Language Processing tools can bring independency to them– Ex. Social media websites, email

• Accessibility Layer based on three key technologies:– Tech 1: Text and content simplifier (Simplext): English & Spanish– Tech 2: Text-to-speech functionalities: Dutch, English & Spanish– Tech 3: Text-to-Pictograph and Pictograph-to-Text translation: Dutch,

English & Spanish

The Users

Able to Include

• Our partners (Belgium, UK, Spain) tested the tools in three “user scenarios”:– Labour integration: email client, word processor, pdf reader– Mobility: guidance systems for people with ID– Leisure within information society: Facebook, Twitter, WhatsApp,…

Text2Picto

Introduction

Text

Mijn kat eet vis.My cat eats fish.

Mi gato come pez.

Sequence of pictographs

Sclera

Beta

I eat straberries.(Spelling error made on purpose)

I eat straberries .I eat strawberries .

pronoun verb noun punctuationI (to) eat strawberry .

06832033 01168468 04696316 n/a

ik.png voeding.png aardbei.png n/a

TokenizationSpelling correction

Part-of-speech taggingLemmatization

Find synset IDs

Find pictographs

• “Language-independent” design of the Text2Picto tool:– Same architecture for Dutch, English, and Spanish– The system can easily be extended toward other languages

• Users with ID and their coaches gave us feedback on the original Text2Picto tool: difficulties, needs, wishes,…

• We are now improving the Text2Picto tool (case study: Dutch)– Syntactic simplification for pictograph translation – Automated spelling correction for users with ID

Picto2Text

Introduction

Text

Mijn kat eet vis.My cat eats fish.

Mi gato come pez.

Sequence of pictographs

Sclera

Beta

mijn.png kat.png eten.png vis.png my…

catkittencats

the catsthe kitten

a cat…

eatate

consumeconsuming

eatingeats

…

fishguppy

the fish…

• Challenge: a pictograph corresponds to multiple words in a given language, withmultiple inflected forms (number, tense,…)

• Again: “Language-independent” design of the Picto2Text tool:– Same architecture for Dutch, English, and Spanish– The system can easily be extended toward other languages

Picto2Text

Composing pictograph messages

• The static hierarchy of pictographs– A three-level category system– Based on topic counts, frequency of use,…

• The dynamic predictor of pictographs– Can be compared to Google’s autocomplete function for text– Uses n-gram information and semantic associations

Contact: leen.sevens@kuleuven.be

Picto demo & publications: http://picto.ccl.kuleuven.be