beating common sense into interactive applications henry lieberman, hugo liu, push singh, barbara...

Beating Common Sense into Interactive Applications

Henry Lieberman, Hugo Liu, Push Singh, Barbara BarryAI Magazine, Winter 2004

As (mis-)interpreted by Peter Clarkfor Boeing KR Group

Introduction

• Claim: Commonsense applications are closer than you think

• Problems with CommonSense (CS) applications:– Even large KBs have sparse coverage

– Inference is unreliable

Their Common Sense KB: Open Mind Common Sense (OMCS)

• 750k NL assertions from 15k contributors

• ConceptNet: A semantic net built from these– 20 link types

Against Question-Answering…• Question answering is a bad CS domain:

– User expects a direct answer to all his/her questions

– System has to be right (almost) all of the time

– Got to be fast (few seconds)

• Alternative: intelligent interfaces– Assists user when it can

– “fail soft” - user can ignore it if he/she wants

– But: Is yet another paperclip?

1. ARIA: Annotation and Retrieval Integration Agent

• Helps annotate photos, and find photos– Similar to Thesaurus search

– Photos are annotated with keywords

– a. People, places and events are recognized in text

– b. Use the semantic net to find “close” photos to text

– Text also adds to the net (system learns)

• “My sister’s name is Mary” → “Joe –sister→Mary”

2. Detecting Moods (“affect”) in Text

• Approach:– Mood keyword (e.g., “sad”)

→ mine a “small society of linguistic models of effect” from the KB (=?)

• Applications:– Empathy Buddy: (purpose=?)

– Summarizing a collection of reviews about a topic

“My wife left me; she took the kids and the dog”

3. Cinematic Commonsense:Video Capture and Editing

• Videographer shoots, adds NL annotations– E.g., “a street artist is painting a painting”

• Send annotations to KB for elaboration– “after painting, you clean the brushes”– “during painting, you might get paint on your hands”

• Elaborations– suggest new shots for the videographer– Also are stored for improved retrieval – Can help order shots into temporal/causal

• (isn’t temporal ordering already done?)

• But: need more complex story understanding to create effective suggestions for the filmmaker.

4. Common Sense Storytelling: StoryIllustrator• Continuosly retreive photos relevant to user’s typing

• Use Yahoo image search, not annotations, for Web images

• CSK for query expansion, E.g., “baby” ↔ “milk”

5. Common Sense Storytelling:OMAdventure

• E.g., in kitchen→ what do you find in kitchen?→ Other associated locations?

• E.g., oven→ what can you do with an

oven?

• Hence oven, cooking are “moves” for user. Associated locations are “exits”.

• User can add objects (e.g., “blender”) → extend KB (“blenders are in kitchens”)

• Generates dungeons-and-dragons game on the fly

7. Common Sense Storytelling:StoryFighter

• System and user take turns to contribute lines to a story to get from A to B, e.g.,– “John is sleepy” (start)

– “John is in prison” (end)

• Must avoid “taboo” words (e.g., “arrest”)• CSK deduces consequences of an event

– “If you commit a crime, you might go to jail”

• CSK also picks obvious taboo words

8. Topic Spotting

• Task: Given speech, identify situation • E.g., “fries”, “lunch”, “Styrofoam” → “eating in a fast-

food restaurant”• Use Bayesian inference + ConceptNet• Used in collaborative storytelling with kids

– Computer starts the story

– Kid continues

– Computer can’t fully understand kid’s speech, but can at least identify the topic → generate plausible continuation

• E.g., “bedroom” → “Jane’s parents walked into the bedroom while she was hiding under the bed”

9. Globuddy: A Tourist Phrasebook

• Type in your situation– “I’ve just been arrested”

• It retrieves and translates associated CS (?)– “If you are arrested, you

should call a lawyer”

– “Bail is a payment that allows an accused person to get out of jail until a trial”

10. Predictive typing/phrase completion

• E.g., for a cellphone keyboard

• Use ConceptNet to find next word that “makes sense”– E.g., “train st” → “train

station”

11. Search: GOOSE and Reformulator as Google adjuncts

• Infer user’s search goals and add keywords,e,g,:– “my cat is sick” → “Did you mean to look for veterinarians?”

• Currently interactive. Later, will suggest better query.

12. Semantic Web

• Given user’s goals, find services that might accomplish subgoals, e.g.,– “Schedule a doctor’s appt” → look up directory of doctors,

check reputation, geographic lookup, lookup schedules, etc.

13. Knowledge Acquisition

• Criticism: Many OpenMind sentences are decontextualized– “At a wedding, bride and groom exchange rings” is

culturally specific

• → develop a prompt-based interface to have user’s make context explicit.

Reflections• Logic: What inferences are possible• Commonsense: What inferences are plausible• Qn: How well does OpenMind support this? E.g.,

– “People live in houses”

– “Things fall down rather than up”

– “Acid irritates skin”

Same for our own database…

“There is a rocket” +

= ?

Reflections (cont): Limitations

• Spottiness of subject coverage in OpenMind• Inference is unreliable → reluctant to use it

– Need new inference methods

– E.g., “interleave context-sensitive inference with retrieval in a breadth-first manner”

• CS suggestions may be distracting– But trials suggest otherwise (people tolerate wrong but

plausible suggestions better than stupid ones)

Some additional thoughts…

• Domain-specific vs. domain-general applications– Domain-specific – how much CS is needed?

• CycSecure

• Oil exploration

• etc.

– Domain-general – still need task-specific algorithm

– Unusual to find a domain- and task-general application

• “Scenario completion” is a good task– newswire, incident reports, etc.