natural language processing with...

Natural Language Processing with Python

CS372: Spring, 2021

Lecture 3Accessing Text Corpora and

Lexical Resources

Jong C. ParkSchool of Computing

Korea Advanced Institute of Science and Technology

ACCESSING TEXT CORPORA AND LEXICAL RESOURCESAccessing Text CorporaConditional Frequency DistributionsMore Python: Reusing CodeLexical ResourcesWordNet

CS372: NLP with Python 22021-03-09

Questions• What are some useful text corpora and lexical

resources, and how can we access them with Python?

• Which Python constructs are most helpful for this work?

• How do we avoid repeating ourselves when writing Python code?

2021-03-09 CS372: NLP with Python 3

Introduction

Gutenberg Corpus Web and Chat Text Brown Corpus Reuters Corpus Inaugural Address Corpus Annotated Text Corpora Corpora in Other Languages Text Corpus Structure Loading Your Own Corpus


Accessing Text Corpora

The Project Gutenberg electronic text archive • contains some 25,000 electronic books• http://www.gutenberg.org/.


Gutenberg Corpus


Gutenberg Corpus


Gutenberg Corpus

Average sentence length and lexical diversityappear to be characteristics of particular authors.


Gutenberg Corpus

The sents() function divides the text into its sentences, which are lists of words.

Most NLTK corpus readers include a variety of access methods in addition to words(), raw(), and sents().

NLTK’s collection of web text includes • content from a Firefox discussion forum; • conversations overheard in New York; • the movie script of Pirates of the Carribean; • personal advertisements; and • wine reviews.


Web and Chat Text


Web and Chat Text

A corpus of instant messaging chat sessions:• originally collected by the Naval Postgraduate

School (nps) for research on automatic detection of Internet predators;

• contains over 10,000 posts, anonymized by replacing usernames with generic names of the form “UserNNN”, and manually edited to remove any other identifying information;


Web and Chat Text

• organized into 15 files, where each file contains several hundred posts collected on a given data, for an age-specific chatroom (teens, 20s, 30s, 40s, plus a generic adults chatroom).


Web and Chat Text

The Brown Corpus• the first million-word electronic corpus of

English;• created in 1961 at Brown University;• contains text from 500 sources; and• the sources have been categorized by genre,

such as news, editorial, and so on.


Brown Corpus

http://icame.uib.no/brown/bcm-los.htmlfor a complete list.


Brown Corpus

We can access the corpus as a list of words or a list of sentences. • We may optionally specify particular

categories or files to read.


Brown Corpus

It is a convenient resource for studying systematic differences between genres, a kind of linguistic inquiry known as stylistics.


Brown Corpus

Is there any other selection of words that one can try for similar stylistics?


Brown Corpus

Computing counts for each genre of interest. • Use NLTK’s support for conditional frequency

distributions.

What kind of observations can we make?

The Reuters Corpus • It contains 10,788 news documents totaling

1.3 million words.• The documents are classified into 90 topics,

and grouped into two sets, “training”/“test”.• For example, the text with fileid ‘test/14826’ is

a document drawn from the test set. • The split is for training and testing algorithms

that automatically detect the topic of a document.


Reuters Corpus


Reuters Corpus

Unlike the Brown Corpus, categories in the Reuters Corpus overlap with each other.


Reuters Corpus

Why?

We can specify the words or sentences we want in terms of files or categories.


Reuters Corpus

The Inaugural Address Corpus• a collection of 55 texts, one for each

presidential address;• its time dimension is an interesting property.


Inaugural Address Corpus



‘2021-Biden.txt’ is not yet available.

Looking at how the words America and citizen are used over time.



Many text corpora containing linguistic annotations represent part-of-speech tags, named entities, syntactic structures, semantic roles, and so forth. • Consult http://www.nltk.org/data for

information about downloading them.


Annotated Text Corpora

NLTK comes with corpora for many languages, though in some cases we need to learn how to manipulate character encodings in Python.


Corpora in Other Languages

the “Floresta Sinta(c)tica Corpus http://www.linguateca.pt/Floresta/(cf. http://nltk.googlecode.com/svn/trunk/doc/howto/portuguese_en.html)

the CESS-ESP Treebank, with 6030 parsed sentences

bangla, hindi, marathi, telugu

The corpus, udhr, contains the Universal Declaration of Human Rights in over 300 languages. • The fields include information about the

character encoding used in the file, such as UTF8 or Latin1.





Words having five or fewer letters account forabout 80% of Ibibio text, 60% of German text,and 25% of Inuktitut text.

Common structures


Text Corpus Structure

There is a difference between some of the corpus access methods:



Load your own collection of text files.


Loading Your Own Corpus

your own path to replace /usr/share/dict

Another example


Loading Your Own Corpus

corpus reader for corpora that consist of parenthesis-delineated parse trees

your own path to replace /corpora/penntreebank/parsed/mrg/wsj”

Conditions and Events Counting Words by Genre Plotting and Tabulating Distributions Generating Random Text with Bigrams


Conditional Frequency Distributions

While a frequency distribution counts observable events, a conditional frequency distribution needs to pair each event with a condition.


Conditions and Events


Counting Words by Genre


Plotting and Tabulating Distributions

1,638 words of the English text have nine or fewer letters.

Create a table of bigrams using a conditional frequency distribution.


Generating Random Text with Bigrams


Generating Random Text with Bigrams

Example 2-1. Generating random text

Accessing Text Corpora• Gutenberg Corpus• Web and Chat Text• Brown Corpus• Reuters Corpus• Inaugural Address Corpus• Annotated Text Corpora• Corpora in Other Languages• Text Corpus Structure• Loading Your Own Corpus


Summary (1/2)

Conditional Frequency Distributions• Conditions and Events• Counting Words by Genre• Plotting and Tabulating Distributions• Generating Random Text with Bigrams


Summary (2/2)

natural language processing with...

Documents