5/11/98 1
Untangling Text Data Mining
Stanford Digital Libraries SeminarMay 11, 1998
Marti Hearst UC Berkeley SIMS
www.sims.berkeley.edu/~hearst
5/11/98 2
Caveat Emptor:
I do information access. I do not do text data mining (yet).
This talk is an attemptto explore the relationshipbetween the two.
5/11/98 3
Talk Outline Definitions
– What is Data Mining?– What is Information Access?– What is Text Data Mining?
Empirical Computational Linguistics Real text data mining tasks Conclusions and Future Directions
5/11/98 4
The Knowledge Discovery from Data Process (KDD)KDD: The non-trivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data. (Fayyad, Shapiro, & Smyth, CACM 96)
Note: data mining is just one step in the process
5/11/98 5
What is Data Mining? (Fayyad & Uthurusamy 96, Fayyad 97)
Fitting models to or determining patterns from very large datasets.
A “regime” which enables people to interact effectively with massive data stores.
Deriving new information from data.– finding patterns across large datasets– discovering heretofore unknown
information
5/11/98 6
What is Data Mining? Potential point of confusion:
– The extracting ore from rock metaphor does not really apply to the practice of data mining
– If it did, then standard database queries would fit under the rubric of data mining• Find all employee records in which employee
earns $300/month less than their managers– In practice, DM refers to:
• finding patterns across large datasets• discovering heretofore unknown information
5/11/98 8
Why Data Mining? Because the data is there. Because current DBMS technology
does not support data analysis. Because
– larger disks– faster cpus– high-powered visualization – networked information
are becoming widely available.
5/11/98 9
DM Touchstone Applications
(CACM 39 (11) Special Issue) Finding patterns across data sets:
– Reports on changes in retail sales• to improve sales
– Patterns of sizes of TV audiences• for marketing
– Patterns in NBA play
• to alter, and so improve, performance – Deviations in standard phone calling behavior
• to detect fraud• for marketing
5/11/98 10
DM Touchstone Applications
(CACM 39 (11) Special Issue) Separating signal from noise:
– Classifying faint astronomical objects
– Finding genes within DNA sequences
– Discovering novel tectonic activity
5/11/98 12
What’s new here?
Sounds like statistical modeling or machine learning.
Main Difference: scale and availability (Fayyad 97)
– Datasets too large for classical analysis– Increased opportunity for access
• end user is often not a statistician– New issues in sampling
5/11/98 13
Statistician’s Viewpoint(David Hand 97)
What’s new about DM?– Returns statisticians to their empirical
roots• exploration rather than modeling
– Hypothesis testing may be irrelevant• given the large data sizes everything is
significant– Data was collected for some other
purpose than what it is being analyzed for now
5/11/98 15
Talk Outline Definitions
– What is Data Mining?– What is Information Access?– What is Text Data Mining?
Empirical Computational Linguistics Real text data mining tasks Conclusions and Future Directions
5/11/98 17
Information Access(Information Retrieval more broadly
construed)
Problem:– Huge amounts of online textual information
Goal: – Build systems to help people discover, create
use, re-use, and understand information
Approach:– Leverage off of users’ smarts
– Combine stats, text analysis, user interfaces
5/11/98 18
Information RetrievalA restricted form of Information
Access The system has available only pre-
existing, “canned” text passages. Its response is limited to selecting from
these passages and presenting them to the user.
It must select, say, 10 or 20 passages out of millions!
5/11/98 19
Needles in Haystacks
The emphasis in IR (and standard DB) is in answering ad hoc queries.
5/11/98 22
IA vs. KDD ProcessQuery/Information Need
Match query against transformed
data
Show results ranked in relevance order
5/11/98 24
Talk Outline Definitions
– What is Data Mining?– What is Information Access?– What is Text Data Mining?
Empirical Computational Linguistics Real text data mining tasks Conclusions and Future Directions
5/11/98 25
What is Text Data Mining?
Peoples’ first thought: – Make it easier to find things on the Web.– But this is information retrieval!
The metaphor of extracting ore from rock:– Does make sense for extracting
documents of interest from a huge pile.– But does not reflect notions of DM in
practice:• finding patterns across large collections• discovering heretofore unknown
information
5/11/98 26
Real Text DM
What would finding a pattern across a large text collection really look like?
5/11/98 27
From: “The Internet Diary of the man who cracked the Bible Code” Brendan McKay, Yahoo Internet Life, www.zdnet.com/yil (William Gates, agitator, leader)
Bill Gates + MS-DOS in the Bible!
5/11/98 28
From: “The Internet Diary of the man who cracked the Bible Code”Brendan McKay, Yahoo Internet Life, www.zdnet.com/yil
5/11/98 29
Real Text DM
The point:– Discovering heretofore unknown
information is not what we usually do with text.
– (If it weren’t known, it could not have been written by someone!)
However:– There is a field whose goal is to learn
about patterns in text for its own sake ...
5/11/98 30
Observation
Research that exploits patterns in text does so mainly in the service of
computational linguistics, rather than for learning about and exploring text
collections.
5/11/98 31
Talk Outline Definitions Empirical Computational Linguistics
– Special and important properties of text– Relationship to TDM– Examples of TDM as CL
Real text data mining tasks Conclusions and Future Directions
5/11/98 32
Recent Trends in NLP (CL)
Previously: AI, full understanding Current: Corpus-based, Statistical
• ACL proceedings: from 3 corpus-based papers in 1991 to at least half in 1996
• Stat NLP was tried long ago (Z. Harris)
Simple Often Wins• Echoes results in IR
Interesting direction: • Statistics + Linguistics (Klavans & Resnik 96)
5/11/98 33
Text Analysis (CL) Tasks
Word Sense Disambiguation Automatic Lexicon Augmentation Discourse Analysis Parsing
• Phrase Identification• Phrase Attachments• Predicate/Argument Structure• Scope of Conjunctions• ...
5/11/98 34
Why Text is Tough
– Abstract concepts difficult to represent (AI-Complete)
– “Countless” combinations of subtle, abstract relationships among concepts
– Many ways to represent similar conceptsspace ship, flying saucer, UFO, figment of imagination
– Concepts are difficult to visualize– High dimensionality
Tens or hundreds of thousands of features
5/11/98 35
Why Text is Tough
Language is:– ambiguous (many different meanings
for the same words and phrases)– different combinations imply different
meanings
5/11/98 36
Why Text is Tough
I saw Pathfinder on Mars with a telescope. Pathfinder photographed Mars. The Pathfinder photograph mars our
perception of a lifeless planet. The Pathfinder photograph from Ford has
arrived. The Pathfinder forded the river without
marring its paint job.
5/11/98 37
Why Text is Easy
Highly redundant in bulk Just about any simple
algorithm can get “good” results for coarse tasks– Pull out “important” phrases– Find “meaningfully” related words– Create summary from document– Major problem: Evaluation
5/11/98 38
Stupid Text Tricks
– Coarse IR, Clustering• Don’t need dimension reduction (except
stopwords)
• Don’t need morphological analysis• Don’t need word sense disambiguation
– Partial parsing:• Simple, greedy transformation rules• Cascading finite state machines
– Categorization• Assume independence
5/11/98 39
Text “Data Cleaning”
Pre-process text as follows: Tokenization Morphological Analysis
(Stemming)inflectional, derivational, or crude IR methods
Part-of-Speech TaggingI/Pro see/VP Pathfinder/PN on/P Mars/PN ...
Phrase Boundary Identification[Subj I] [VP saw] [DO Pathfinder] [PP on Mars] [PP
with a telescope].
5/11/98 40
CCL Methodology
Describe here the standard methodology for corpus-based computational linguistics algorithms
5/11/98 41
CCL Examples
Place here examples of the kinds of output generated for computational linguistics applications
5/11/98 42
Inducing MetaData for Documents
Assigning bibliographic metadata – author, genre, time, region
Subject/Topic assignments– category labels: MeSH, LoC, ACM keywords
Information Extraction (MUC)– MUC: terrorist incidents
• who did the bombing• where did the bombing take place• what weapon(s) were used• when did it happen
5/11/98 43
Inducing MetaData for Collections
Indexes Hierarchical Categorization Overviews of Connectivity
• hyperlinks• co-citation links
Overviews of Subject Matter• 2D• 3D• dynamic
5/11/98 44
A Main Point:
Empirical CL is usually not helpful for improving Information Access.
However, it can produce– metadata– overviews– associations
that are indirectly useful for IA.
5/11/98 45
Talk Outline Definitions Empirical Computational Linguistics Real text data mining tasks
– TDM not using text– TDM using text
Conclusions and Future Directions
5/11/98 46
TDM using Metadata (instead of Text)
(Dagan, Feldman, and Hirsh, SDAIR ‘96)– Data:
• Reuter’s newswire (22,000 articles, late 1980s)• Categories: commodities, time, countries,
people, and topic– Goals:
• distributions of categories across time (trends)• distributions of categories between collections• category co-occurrence (e.g., topic|country)
– Interactive Interface:• lists, pie charts, 2D line plots
5/11/98 47
Combining Text with Metadata
(images, hyperlinks)
Examples– Text + Links to find “authority pages”
(Kleinberg at Cornell, Page at Stanford)
– Usage + Time + Links to study evolution of web and information use (Pitkow et al. at PARC)
– Images + Text to improve image search
5/11/98 48
Talk Outline Definitions The New Empirical Computational
Linguistics Real text data mining tasks
– TDM not using text– TDM using text
Conclusions and Future Directions
5/11/98 49
Ore-Filled Text Collections
Newspaper/Newswire Medical Articles
– Patterns associated with symptoms, drugs
Patent Law– Recent Study Justifying Scientific
Funding– Hypotheses for New Inventions
“Corporate Memory”
5/11/98 50
True Text Data Mining:Don Swanson’s Medical
Work Given
– medical titles and abstracts– a problem (incurable rare disease)– some medical expertise
find causal links among titles– symptoms– drugs– results
5/11/98 51
Swanson Example (1991) Problem: Migraine headaches (M)
– stress associated with M– stress leads to loss of magnesium– calcium channel blockers prevent some M– magnesium is a natural calcium channel blocker– spreading cortical depression (SCD)implicated in M– high levels of magnesium inhibit SCD– M patients have high platelet aggregability– magnesium can suppress platelet aggregability
All extracted from medical journal titles
5/11/98 52
Swanson’s TDM
Two of his hypotheses have received some experimental verification.
His technique– Only partially automated– Required medical expertise
Few people are working on this.
5/11/98 53
Text Collection Overviews
Clusters/Unsupervised Overviews– Chalmers: BEAD, Networks of Words– Lin,Chen: Kohonen Feature Maps– Xerox PARC: Local Clusters– Pacific Northwest: ThemeScapes– Rennison: Galaxy of News
5/11/98 54
Text Overviews
– Huge 2D maps may be inappropriate focus for information retrieval • can’t see what documents are about• documents forced into one position in
semantic space• space difficult to browse for IR purposes
– Perhaps more suited for pattern discovery• problem: often only one view on the
space
5/11/98 55
Talk Outline Definitions The New Empirical Computational
Linguistics Real text data mining tasks
– TDM not using text– TDM using text
Conclusions and Future Directions
5/11/98 56
Conclusions
Currently, what might be construed as Text Data Mining is really Computational Linguistics– Text is tricky to process, but rich and abundant (now)– There are many CL tools available
Data Mining directly from text – tells us about language– produces meta-information that may be useful for
information access
5/11/98 57
Conclusions, continued Information Access != Text Data Mining
– IA = finding needle in haystack– TDM = finding patterns or discovering new information
However, Information Access may potentially be served by Text Data Mining techniques:– automated metadata assignment– collection overviews
The synthesis of ideas from TDM and IA: – Perhaps a new field of exploratory data analysis over
text!
5/11/98 58
Promising Research Directions
Text Data Mining Problems:– Patterns within sets of documents:
• What is the latest in this field?• How is this field related to that field?
– Chains of evidence embedded in text:• What drugs have been tested for this
symptom?• What effects did this funding have on that field?
– Human use of information over time, • How does information diffuse across the web?