automatic extraction of human activity knowledge from

May 19, 2010

Automatic Extraction of

Human Activity Knowledge

from Method-Describing Web Articles

Jihee Ryu, Yuchul Jung, Kyung-min Kim,

Sung-Hyon Myaeng

IR & NLP Lab

Computer Science Department

KAIST

Motivation

People need helps around their activities (context)

Intelligent systems need to provide customized services

© 2010 IR&NLP Lab. All rights reserved.4

User information

user

Loosen lug nuts on tire

Install spare tire

Call AAA

Explain the situation

Motivation

User feedback

Realizing that the bank balance is

too low to call the emergency

road service, she decides to do it

by herself and clicks on “Change

a Flat Tire When You Are a

Woman Alone” (a kind of

feedback)

5

Change a Tire like a Real

Woman

Change a Flat Tire When You

Are a Woman Alone

1. Set your emergency brake.

2. Loosen lug nuts on tire.

3. Place jack at the most solid

point on the car.

4. Remove loosened lug nuts.

5. Install spare tire.

6. Reinstall lug nuts.

7. Let the jack down.

Motivation

To understand what the user is up to, we need to infer goal activity from the contextual info

know actions to be taken for the goal

Activity knowledge is not available in large quantity Most automatically acquired KB are entity-oriented

e.g. concept hierarchy or ontology

Activity knowledge is dynamic in nature Difficult to acquire knowledge about daily lives of people

Past attempts Linguistic knowledge (e.g. WordNet)

Narrowly defined area of human activities

Comprehensive KB is labor intensive (e.g. CYC)

Often mixed with other types of general knowledge (e.g. ConceptNet)


Common Sense Knowledge

Open Mind Common Sense (OMCS) (2002)

More than 729,000 raw sentences representing

commonsense knowledge

Collected from the general public through a template-

based Web interface

EventNet (2005)

Based on OMCS, about 30,000 temporal relation

instances extracted from commonsense events

Utilized as a basis for an intelligent household system

Knowledge is skewed toward household activities


Activity Extraction

Shah and Gupta (2005)

Based on Open Mind Indoor Common Sense (OMICS)

Composed of task steps to populate an initial robot knowledge

base

For default household task plans

A simple activity extraction method

Part-of-speech (POS) tagger is applied

The first verb and noun phrase extracted as the action and its

object

Used as a baseline for our experiment


Activity Modeling

Perkowitz et al. (2004)

To sense and model activities from the real world

Scale to a much larger class of activities than before

Activity extraction

Sequence of objects that are physically involved in an activity

and their probabilities

Probabilistic translations from terms that represent an activity

name to terms that represent the objects involved

E.g. drinking tea teacup, teabag, …


Use of Web Resources

Copyright © 2010 IRNLP Lab. 12

e-how (http://www.ehow.com/)

A collection of method-describing articles

Title How to Make Omelet Soup

Step 1 Place the water or canned chicken broth in a large saucepan.

Boil the sweet yellow onion for several minutes.

Step 2 Add the powdered chicken broth along with the canned

mushrooms.

Boil the soup for a few more minutes, and then add the chopped

green onion.

Step 3 Drop the eggs into the simmering broth a few minutes before

you're ready to serve the omelet soup.

http://www.ehow.com/

e-how Statistics

Category # Articles Percentage

Arts & Entertainment 68,165 6.7%

Business 31,846 3.1%

Careers & Work 39,291 3.9%

Cars 30,900 3.1%

Computers 47,450 4.7%

Culture & Society 26,508 2.6%

Education 30,677 3.0%

Electronics 18,876 1.9%

Fashion, Style & Personal Care 49,270 4.9%

Food & Drink 75,842 7.5%

Health 122,152 12.1%

Hobbies, Games & Toys 74,216 7.3%

Holidays & Celebrations 22,632 2.2%

Home & Garden 102,843 10.2%

Internet 24,938 2.5%

Legal 9,805 1.0%

Parenting 19,427 1.9%

Parties & Entertaining 8,874 0.9%

Personal Finance 41,086 4.1%

Pets 30,017 3.0%

Relationships & Family 25,220 2.5%

Sports & Fitness 74,930 7.4%

Travel 29,359 2.9%

Weddings 8,449 0.8%

Total 1,012,773 100.0%

Review

Aug. 26, 2009

(1 million articles)

May 19, 2010

(1.5 million articles)

Our Work

Observation

24 categories cover a variety of task domains.

A sequence of actions associated with a goal (how to …)

can be considered a solution to the task.

KB Construction

Title Goal

Steps in a how-to instruction Actions

Form a hierarchical activity network


Task=goal=activity

Activity model = (G, A, I)G : Goal

A : Action sequences

I : Ingredients (objects, location, time)

Activity Extraction Model


Title) How to Make Omelet Soup

Step 1) Place the water or canned chicken

broth in a large saucepan.

Boil the sweet yellow onion for

several minutes.

Step 2) Add the powdered chicken broth

along with the canned mushrooms.

Boil the soup for a few more

minutes, and then add the chopped

green onion.

Step 3) Drop the eggs into the simmering

broth a few minutes before you're

ready to serve the omelet soup.

(boil, sweet yellow onion)

(add, powdered chicken broth)

(boil, soup)

water chicken broth

onionsoup eggs

Action Sequences

Goal

(place, water) (place, canned chicken broth)

(drop, eggs)

(add, chopped green onion)

Ingredients

Make Omelet Soup

how-to article

Activity Knowledge Base (recap)

Three Major Elements

Activity (or goal)

Extracted from the title of a how-to article

Action Sequences

Specific steps to be taken for the given task in the article

Ingredients (location, time, object)

Entities associated with action steps

Network of goals, actions, and ingredients [JWS

2010]

Actions can be linked to multiple goals

Normalization


Preprocessing

Sentence boundary detection

Identification of imperative sentences

Parsing Using Stanford NLP library

E.g. loosen the nut (VP (VB loosen) (NP (DT the) (NN nut))

Dependency tree generation Simplify parse trees

Eliminate adverbial phrase, determiners, and articles

E.g. also contact them using a phone contact them using phone

Convert a parse tree to a dependency structure

E.g. (VP (VBP start) (PRT (RP up)) (NP (NN car)))

prt(start-1, up-2)

dobj(start-1, car-3)


Target Extraction

To extract (Goal, Actions, Ingredients) from each article

Syntactic Pattern-based Method

Obvious patterns high precision

But coverage is limited.

CRF-based Method

For more complete coverage


Syntactic Pattern-based Method

Copyright © 2010 IRNLP Lab. 20

Pattern discovery

Mask words generate dependency relation driven patterns

E.g. prt(start-1, up-2) & dobj(start-1, car-3) prt(a, b) & dobj(a, c)

Identify frequent patterns (f 3)

E.g. prt(a, b) & dobj(a, c) verb(„a b‟) / ingredient(c, „a b‟)

Compute confidence for each pattern

verb(„a b‟) / ingredient(c, „a b‟) [confidence: 95%]

184 patterns were generated (confidence > 85%)

Instance generation

Extract (action, ingredient) instances based on pattern matching

E.g. check out the engine (Action: check out, Ingredient: engine)

CRF-based Method

To extract targets not matching known patterns Increase coverage

General and expressive modeling technique to label objects

Training data

Take up to 5 sentence instances extracted by applying each of the selected patterns


none verb ingredient ingredient

You/PRP remove/VBP timing/VBG belt/NN .

none

Feature Generation

Feature candidates

POS with order

E.g. you tighten nut you(PR1) / tighten(VB1) / nut(NN1)

Dependency type for word pairs

E.g. you tighten nut

you(nsubjB1) / tighten(nsubjA1, dobjA1) / nut(dobjB1)

Phrase chunk with order (Open NLP tool)

E.g. you tighten nut

you(NP1) / tighten(VP1) / nut(NP2)

POS and dependency features were used finally.


Evaluation Overview

Purpose

Measure performance of proposed automatic extraction

methods

Method

A manually constructed test collection

2400 eHow articles from 24 domains were randomly chosen.

10 graduate and undergraduate students annotated the data for

actions and ingredients

A tool was provided.

2 judgments for each sentence

Comparison with baselines (previous approaches)


Annotation Tool

© 2010 IR&NLP Lab. All rights reserved.

guidelinetab

exampletab

workbenchtab

reviewtab

currently activated document

verb form ingredient item (noun clause)

verb form ingredient item (noun phrase)

range button save button

25

Result

Comparisons with baselines

Baseline 1

Extract every first verb and first noun phrase as an action

Baseline 2

Extract every first verb and every noun phrase under 'object' and

'substance' categories in WordNet


MethodAverage

Accuracy

Average

Coverage

Baseline 1 (based on Shah and Gupta) 0.7866 0.9821

Baseline 2 (based on M. Perkowiz et al) 0.5432 0.9897

Syntactic Pattern-based Method 0.9130 0.5660

CRF-based Method 0.8192 0.9499

Pattern-based & CRF-based 0.8261 0.9501

Discussion

Limitations

Prepositional phrases are not processed

Loss of essential information about ingredients

E.g. check for dates on the inside of the wheels *(check for, dates)

Multi-word expressions arenot handled properly.

E.g. pay attention to words of hymn *(pay, attention)

Limited verb scope handling


Conclusion

Proposed a method for constructing a large-scale

human activity KB (~.5 activities & 3.2M actions) by

using how-to articles on the Web

Need further applications of NLP techniques for

context-providing sentences, verb scoping, &

prepositional phrases

The constructed KB is to be complemented with

mined experiences from blogs [ACL 2010]


Jihee Ryu ([email protected])

@zzihee5 http://profile.zzihee.net

Yuchul Jung ([email protected])

@enth77 http://ir.kaist.ac.kr/~jyc

Kyung-min Kim ([email protected])

@kimdarwin http://ir.kaist.ac.kr

Sung-Hyon Myaeng ([email protected])

@myaeng http://ir.kaist.ac.kr

IR&NLP Lab

@irnlp http://ir.kaist.ac.kr

Action Normalization

Construct an equivalence class of actions with a representative

name clustering

Mapped

into same

cluster

Fixa flat tire

(pump, brake pedal)

(pump,brake foot

pedal)

(check,equipments)

(jack up, car)

(raise,vehicle)

(take,spare tire)

Changea flat tire

Goals

Actions

(a)

(b)

(c)

contextual

Similarity

additional

Descriptor

synonyms

Reference Banko, M., Cafarella M. J., Soderland, S., Broadhead, M., and Etzioni, O. Open Information

Extraction from the Web. In Proceedings of the 20th International Joint Conference on Artificial

Intelligence (IJCAI 2007), pp. 2670–2676, 2007. San Francisco, CA, Morgan Kaufmann Publishers

Inc.

Cai, G. and Xue, Y. Activity-oriented Context-aware Adaptation Assisting Mobile Geo-spatial

Activities. In Proceedings of the 2006 International Conference on Intelligent User Interfaces (IUI

2006). pp. 354–356, New York, NY, 2006. ACM.

Cios, K., Swiniarski, R., Pedrycz, W., and Kurgan L. Unsupervised Learning: Association Rules. In

Data Mining: A Knowledge Discovery Approach, chapter 10, pp. 289–306. Springer-Verlag New

York, Inc., Secaucus, NJ, 2007.

de Marneffe, M. and Manning, C. D. Stanford typed dependencies manual, September 2008.

Diab, M. T. and Krishna, M. Unsupervised Classification of Verb Noun Multi-Word Expression

Tokens. In Gelbukh, A. (ed.) Proceedings of 10th International Conference on Intelligent Text

Processing and Computational Linguistics (CICLing 2009), pp. 98–110, Berlin, Heidelberg, 2009.

Springer.

Espinosa, J., and Lieberman, H. EventNet: Inferring Temporal Relations Between Commonsense

Events. In Gelbukh, A., de Albornoz, A., and Terashima, H. (eds.) Proceedings of the 4th Mexican

International Conference on Artificial Intelligence (MICAI 2005), pp. 61–69, Berlin, Heidelberg,

2005. Springer.

Gupta, R. and Kochenderfer, M. J. Common Sense Data Acquisition for Indoor Mobile Robots. In

Proceedings of the 19th National Conference on Artificial intelligence (AAAI 2004), pp. 605–610,

2004. AAAI Press / The MIT Press.© 2010 IR&NLP Lab. All rights reserved.34

Reference Jung, Y., Ryu, J., Kim, K., and Myaeng, S. Automatic Construction of a Large-Scale Situation

Ontology by Mining How-to Instructions from the Web. Journal of Web Semantics,

8(2):10.1016/j.websem.2010.04.006, 2010.

Langheinrich, M., Mattern, F., Römer, K., and Vogt, H. First Steps Towards an Event-Based

Infrastructure for Smart Things. In Ubiquitous Computing Workshop (PACT 2000), 2000.

Liu, H., and Singh, P. ConceptNet: A Practical Commonsense Reasoning Toolkit. BT Technology

Journal, 22(4):211–226, 2004.

Luther, M., Fukazawa, Y., Souville, B., Fujii, K., Naganuma, T., Wagner, M., and Kurakake, S.

Classification-based Situational Reasoning for Task-oriented Mobile Service Recommendation. In

Proceedings of the 17th European Conference on Artificial Intelligence (ECAI 2006) Workshop on

Contexts and Ontologies, pp. 15–19, 2006.

Maekawa, T., Yanagisawa. Y., Sakurai, Y., Kishino, Y., Kamei, K., and Okadome, T. Web Searching

for Daily Living. In Proceedings of the 32nd international ACM Special Interest Group on

Information Retrieval conference (SIGIR 2009). pp. 27–34, New York, NY, 2009. ACM.

Morgan, B. and Singh, P. Elaborating Sensor Data using Temporal and Spatial Commonsense

Reasoning. In Proceedings of the International Workshop on Wearable and Implantable Body

Sensor Networks (BSN 2006), pp. 187–190, Washington, DC, 2006. IEEE Computer Society.

Perkowitz, M., Philipose, M., Fishkin, K., and Patterson, D. J. Mining Models of Human Activities

from the Web. In Proceedings of the 13th International World Wide Web Conference (WWW 2004),

pp. 573–582, New York, NY, 2004. ACM.


Reference Ravi, S. and Pasca, M. Using Structured Text for Large-Scale Attribute Extraction. In Proceedings

of the 17th ACM Conference on Information and Knowledge Management (CIKM 2008), pp. 1183-

1192, New York, NY, 2008. ACM.

Reisinger, J. and Pasca, M. Low-Cost Supervision for Multiple-Source Attribute Extraction. In

Gelbukh, A. (ed.) Proceedings of 10th International Conference on Intelligent Text Processing and

Computational Linguistics (CICLing 2009), pp. 382–393, Berlin, Heidelberg, 2009. Springer.

Shah, C., and Gupta, R. Building Plans for Household Tasks from Distributed Knowledge. In

Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI 2005)

Workshop on Modeling Natural Action Selection, 2005.

Singh, P., Lin, T., Mueller, E. T., Lim, G., Perkins, T., and Zhu, W. L. Open Mind Common Sense:

Knowledge Acquisition from the General Public. In Meersman, R. and Tari, Z. (eds.), Proceedings

of CoopIS/DOA/ ODBASE 2002, pp. 1223–1237, Berlin, Heidelberg, 2002. Springer.

Tokunaga, K., Kazama, J. and Torisawa, K. Automatic Discovery of Attribute Words from Web

Documents. In Proceedings of the 2nd International Joint Conference on Natural Language

Processing (IJCNLP 2005), pp. 106-118, 2005.
