clustering the short stories of edgar allan poe using galois lattices

8/4/2019 Clustering the Short Stories of Edgar Allan Poe Using Galois Lattices

1/20

Clustering the Short Stories of

Edgar Allan Poe Using Galois Lattices

Joint Statistical Meetings, 2007

Roger Bilisoly, Ph.D.Department of Mathematical Sciences

Central Connecticut State UniversityNew Britain, Connecticut


2/20

Why Edgar Allan Poe?

Poe wrote in many styles Horror, Detective, Satire, Science Fiction

Poes styles are distinctive Hopefully easier for a computer to detect

Only ~70 short stories His novel, short stories and poems all fit in one

volume of about 1000 pages

Poe freely available on Web Lived 1809-49, so all works are out of copyright

Project Gutenberg has Poe (and many other authors)


3/20

Two Goals: 1) Cluster Poes stories2) Be able to explain clusters in terms of language

Text similarity is used in information retrieval (IR) to helpmatch queries and texts Can be used to measure distances between two texts

There are many ways to do this Vector Space Models: term-document matrix allows geometric

approach in high dimensional spaces (e.g., R20,000) Probabilistic Models: find maxi P(Texti | Query) Language Models: find max i P(Query | Language Modeli) And so forth. See Grossman and Frieder (2004)

These approaches are well tested Search engines are profitable and competitive Understanding these approaches require a good

knowledge of mathematics and probability


4/20

Vector Space Model uses

a term-document matrix

Each row has word counts for a particular word

Each column has word counts for a particular storyColumns can be viewed as vectors

Cos(Angle between two column vectors) is a text similarity measure

Just over20,000 rows

for Poes

short stories

Note allthe 0s


5/20

Zipfs Law: Word Freq c/Rank

(Applied to Poes Short Stories)

The Freq=24401

Of Freq=14622 And Freq=9559

8715 words where Freq=1Examples: acceleration,goats, loyal, theft,

This type ofword distribution

implies mostly 0sand low counts in a

term-documentmatrix


6/20

Clustering by Formal Concept Theory:Based on Incidence Matrices

Here rows represent authors (objects) & columns represent attributes

Formal concepts have the form {{objects}, {attributes}}

Idea: look for maximal submatrices that contain all 1s

0101001Whitman

1010100Eliot

0110100Dickens

1001100Stowe

0101011Poe

FemaleMaleUKUSANovelistShortStories

Poet

1 = author has attribute, 0 = author doesnt have attribute


7/20

Examples of Concepts {{Poe}, {Poet, Short Stories, USA, Male}} {{Poe, Whitman}, {Poet, USA, Male}} {{Stowe, Eliot}, {Novelist, Female}} {{Stowe, Dickens, Eliot}, {Novelist}} {{Poe, Dickens, Whitman}, {Male}}

0101001Whitman

1010100Eliot

0110100Dickens

1001100Stowe

0101011Poe

FemaleMaleUKUSANovelistShort

Stories

Poet

Example: Poe and Whitman share exactly Poets, USA & Male attributes

Moreover, Poets, USA & Male are all attributes of exactly Poe and Whitman.


8/20

Concepts in a Galois Lattice

Poe, Short Story Stowe Dickens Eliot

Whitman, Poet

American Male Female English

Novelist

We see that Dickens is a Male and English (go upwards)and All English are Novelists (go upwards again)

We see that Males are Whitman, and Dickens (go downwards),

as well as Poe (go downwards again)

These concepts are only true for the incidence

matrix we used. Adding further authors orattributes can change the formal concepts.


9/20

Goal: Find Galois lattice for Poe stories (as objects)

and word groups (as attributes)

Word groups are easy to understand

E.g., colors: black, dark, white, red, color, green, A thematic thesaurus lists related words, e.g., deathis

related to: dying, demise, killing, to kill, to decease, toexpire, the end,

These obtained from WordNet 2.1, an open source lexicaldatabase

Can produce a word group-document matrix Dimensionality much lower, so matrix is not sparse Word groups easier to link to literary ideas

For incidence matrix, let 1 = story in top 25%,0 otherwise

Rates of word groups are much higher than individualword rates, so sparseness not a problem

Other percentagestried, too


10/20

Word Groups Used in Analysis ofPoes Short Stories

7 word groups

Death: death, corpse, dead, murder, died,

Body: eyes, head, hand, body, feet, heart,

Sensation: see, seen, saw, heard, view, Religious: soul, god, spirit, heaven, moral,

Fear: horror, terror, fear, horrible, anxiety,

Family: family, wife, mother, daughter, Colors: black, dark, white, red, color, green, ...


11/20

Problem: Many word measures are

functions of sample size

We have seen that lexical measures such as mean

frequency and vocabulary size, as well as [modelparameter estimates] all depend on the sample size. p 24 of Word Frequency Distributionsby R. Harald Baayen

One solution used in corpus linguistics: take equal sized

samples from texts However, stories are whole entities

An approximate solution is to consider sets of storiesclose in size

Story sets used here have maximum difference in size < 33% Three groups: 2001 to 3000 words, 3001 to 4200 words and

4201 to 6000 words. (Includes 44 of 70 short stories)

Top 25% for each word group labeled 1, otherwise 0.


12/20

Galois Lattice for Poes Stories

of Length 2001-3000 Words

DeathMorella

Tell-Tale HeartMasque Red Death

BodyMorella

Tell-Tale HeartCask Amontillado

SensationTell-Tale Heart

Furniture4 Beasts in One

ReligiousMorella

EleonoraImp Perverse

FearMorella

Tell-Tale HeartImp Perverse

FamilyMorella

Eleonora3 Sundays

ColorsFurniture

Masque Red DeathEleonora

Death, ColorsMasque Red Death

Colors, SensationFurniture

Religious, FearMorella

Imp of the Perverse

Religious, FamilyMorella

Eleonora

Death, Body, FearMorella

Tell-Tale Heart

Death, Body,Fear, Sensation

Tell-Tale Heart

Religious, FamilyFear, Death, Body

Morella

Religious, Family, ColorsEleonora


13/20

Morella appears in 5 word groups;

Tell-Tale Heart appears in 4.

The Tell-Tale Heart is about a man who kills his older

roommate, hides the body under the floor, the police visit, andhe cracks and shows them the body. This story ranks 3rd (of 12) in Death (5.12 per K), 1st in Body (18.18 per K),

1st in Sensation (12.58 per K), 1st in Fear (9.32 per K) This story is, in fact, considered one of Poes iconic tales.

Morella is the narrators wife who is obsessed with mysticism,dies during childbirth, the daughter grows up to resembleMorella more and more, and upon baptism she cries I amhere, and dies.

Poe has several stories about wives who die (Berenice, Ligeia, The OvalPortrait, The Oblong Box and Eleonora who dies before she and narratorcan marry). So the plot is one Poe has explored several times.

Ranks 1st in Death (8.94 per K), 3rd in Body (13.18 per K), 1st in Religious (8.47per K), 2nd in Fear (5.65 per K), and 2nd in Family (5.18 per K)


14/20

Concepts for Stories with

3001-4200 words'Death''Metzengerstein''The Colloquy of Monas and Una''The Facts in the Case of M Valdemar''The Black Cat''A Tale of the Ragged Mountains'

'Body''The Devil in the Belfry''The Facts in the Case of M Valdemar''A Predicament'

'The Man That Was Used Up''The Angel of the Odd'

'Sensation''The Man of the Crowd''A Predicament''The Man That Was Used Up''A Tale of the Ragged Mountains''MS Found in a Bottle'

'Religious''The Colloquy of Monas and Una''Mesmeric Revelation''The Angel of the Odd''The Black Cat'

'Never Bet the Devil Your Head'

'Fear''Metzengerstein''Berenice'

'The Black Cat''A Tale of the Ragged Mountains''MS Found in a Bottle'

'Family''Metzengerstein''Berenice''The Black Cat'

'Never Bet the Devil Your Head''Diddling'

'Colors''Metzengerstein''The Devil in the Belfry''Berenice'

'The Black Cat''MS Found in a Bottle'

'Death' 'Body''The Facts in the Case of M Valdemar'

'Death' 'Sensation' 'Fear''A Tale of the Ragged Mountains'


15/20

'Sensation' 'Fear' 'Colors'

'MS Found in a Bottle'

'Death' 'Religious' 'Fear' 'Family' 'Colors''The Black Cat'

'Religious' 'Family''The Black Cat'

'Never Bet the Devil Your Head'

'Fear' 'Family' 'Colors''Metzengerstein''Berenice''The Black Cat'

'Fear' 'Colors''Metzengerstein''Berenice''The Black Cat''MS Found in a Bottle'

'Death' 'Religious''The Colloquy of Monas and Una''The Black Cat'

'Death' 'Fear''Metzengerstein''The Black Cat''A Tale of the Ragged Mountains'

'Death' 'Fear' 'Family' 'Colors''Metzengerstein''The Black Cat'

'Body' 'Sensation''A Predicament''The Man That Was Used Up'

'Body' 'Religious'

'The Angel of the Odd'

'Body' 'Colors''The Devil in the Belfry'

'Sensation' 'Fear''A Tale of the Ragged Mountains''MS Found in a Bottle'

Concepts for Stories with

3001-4200 words (cont.)


16/20

Galois Lattice for Poes Stories

of Length 4201-6000 Words

DeathLoss of Breath

Premature BurialThou Art the Man

BodyAssignation

Loss of BreathKing Pest

SensationAssignation

Landors CottageDomain Arnheim

ReligiousAssignation

Bon-BonDomain Arnheim

FearKing Pest

Premature Burial1002nd Tale

FamilyOblong Box1002nd Tale

Thou Art Man

ColorsLandors Cottage

1002nd TaleBon-Bon

Death, BodyLoss of Breath

Death, FearPremature Burial

Family, DeathThou Art the Man

Fear, BodyKing Pest

Religious,Sensation

AssignationDomain Arnheim

Colors, SensationLandors Cottage

Colors, ReligiousBon-Bon

Body, Religious,Sensation

The Assignation

Family, Fear, Colors1002nd Tale ofScheherezade


17/20

References

Word Frequency Distributions R. Harald Baayen

Corpus Linguistics: Investigating Language Structure and Use Biber, Conrad and Reppen

The Use of Color Words by Edgar Allan Poe PMLA, 45(2), Wilson Clough

Concept Data Analysis: Theory and Applications Claudio Corpineto and Giovanni Romano

Word Index to Poes Fiction Edited by Burton Pollin

The Last Haunting of Edgar Allan Poe: An Identification of Poe Preferences

contained in the The Beale Papers Robert Wald Text Mining: Predictive Methods for Analyzing Unstructured Information

Weiss, Indurkhya, Zhang and Damerau


18/20

Numbering for Poes Short Stories

1. The Unparalleled Adventures of One Hans Pfaall2. The Gold Bug3. Four Beasts in One

4. The Murders in the Rue Morgue

5. The Mystery of Marie Rogt6. The Balloon-Hoax7. MS. Found in a Bottle8. The Oval Portrait

9. The Purloined Letter10. The Thousand-and-Second Tale of Scheherezade

11. A Descent into the Maelstrm12. Von Kempelen and his Discovery13. Mesmeric Revelation

14. The Facts in the Case of M Valdemar

15. The Black Cat16. The Fall of the House of Usher17. Silence -- a Fable18. The Masque of the Red Death19. The Cask of Amontillado

20. The Imp of the Perverse21. The Island of the Fay22. The Assignation

23. The Pit and the Pendulum24. The Premature Burial

25. The Domain of Arnheim26. Landor's Cottage27. William Wilson

28. The Tell-Tale Heart29. Berenice30. Eleonora31. Ligeia32. Morella

33. A Tale of the Ragged Mountains34. The Spectacles

35. King Pest

36. Three Sundays in a Week37. The Devil in the Belfry

38. Lionizing39. X-ing a Paragrab

40. Metzengerstein41. The System of Doctor Tarr and Professor Fether42. How to Write a Blackwood article43. A Predicament

44. Mystification45. Diddling46. The Angel of the Odd47. Mellonta Tauta48. The Duc de L'Omlette

49. The Oblong Box

50. Loss of Breath51. The Man That Was Used Up

52. The Business Man53. The Landscape Garden

54. Maelzel's Chess-Player55. The Power of Words56. The Colloquy of Monas and Una

57. The Conversation of Eiros and Charmion58. Shadow -- A Parable

59. Philosophy of Furniture

60. A Tale of Jerusalem61. The Sphinx

62. Hop Frog63. The Man of the Crowd64. Never Bet the Devil Your Head65. Thou Art the Man

66. Why the Little Frenchman Wears his Hand in a Sling67. Bon-Bon

68. Some words with a Mummy69. Literary Life of Thingum Bob Esq.

70. Morning on the Wissahiccon


19/20

Word Variety Keeps Growing

Plot of

# Tokens/# Typesas function ofsize of sample


20/20

Inclusiveness of Concept Lattices

This is for a

Galois latticethat includes all70 short stories

clustering the short stories of edgar allan poe using galois lattices

Documents