nlp: overview, non-dnn approaches, computational linguistics

NLP: overview,non-DNN approaches,

computational linguisticsJonathan K. Kummerfeld

Postdoctoral Research Fellow, CSE

Part 1:Introduction

Who am I?Introduction

NLP Overview CL Overview

Researcher in NLP

First paper, 2008, on detecting non-compositional verb-particle constructions,e.g.“ferret out” and “hand in”

Since then, topics including:- Parsing - Dialogue - Crowdsourcing - Cybercrime analysis- Playing Diplomacy

3

What is this lecture about?

An overview of:

• Natural Language Processing (NLP)

• Computational Linguistics(CL)

IntroductionNLP Overview

CL Overview

Note: For NLP,Lectures 2 and 9 covered what This lecture covers how

4

Recap: NLP in Conversational AIIntroduction

NLP Overview CL OverviewReview: Conversational Flow

Automated Speech Recognition

(ASR)

Spoken language (i.e., sound)

utterance

“Please pay Dr. Leach $1000.”

Natural language understanding

Intent Classification:transfer_money

Slot mapping:recipient: “Dr. Leach”amount: “$1000”

Business Logic:

Deduct $1000 from accountAdd $1000 to recipient account

Response Generation

Template Responses

“OK, I gave Dr. Leach $1000.”

“Sorry fam, you don’t have

enough cash”

Text-to-speech (TTS)

5

Recap: Other NLP ApplicationsIntroduction

NLP Overview CL OverviewApplications: Text Classification

10

www.wired.com

Applications: Sentiment/Opinion

12

6

Applications: Machine Translation

11

Part 2:Natural Language

Processing

Example task: Sentiment AnalysisIntroduction

NLP OverviewCL Overview

+

+

-

Example from Bo Pang, Lillian Lee, Shivakumar Vaithyanathan (2002), via Greg Durrett’s Lecture Slides for CS378 at UT Austin8

This movie was great! Would watch again

The movie was gross and overwrought, but I liked it

This movie was not really very enjoyable

Key components of NLP SystemsIntroduction


9

Disclaimer: These slides focus on widely used forms of supervised learning.

There is an enormous range of other methods that don't fit neatly into this framework. Take some AI / ML classes and you’ll learn about them!

Key components of NLP SystemsIntroduction


10

Data Examples of the language phenomena we want our system to handle

Model A function that maps (input, output) pairs to scores

Inference Method A way to make a prediction for an example given a Model

Learning Method A way to update a given and an Model Data Inference

Method

Rule-based MethodsIntroduction


11

Model

Inference Method

Learning Method

DataNone in theory, but rules are hard to write without examples

People edit the rules

A set of rules If the review contains “good” return positive

Code to run the rules if ‘enjoyable’ in review: return True






+

+

-


Linear ModelsIntroduction


13

Model

Inference Method

Learning Method

Data

A set of features and weights

“liked”: +1.2,“gross”: -1.4,“gross AND liked”: +0.3, …

A set of examples with their true labels

++-




A range of methods - this is Machine Learning if prediction is wrong:

for each feature used: feature += sign of answer

A range of methods prediction = sign[sum features in the review]



+



-I liked the start, but overall it was too gross

Non-Linear ModelsIntroduction


15

Model

Inference Method

Learning Method

Data

A range of methods - this is Machine Learning


++-




A set of weights and structure of the neural network

Language Preprocessing: Why?

• Curating and cleaning data is critically important• Statistical models, classical machine learning, and deep learning are

all based on the same type of data as inputs

“Do my taxes.”

transfer

Withdraw

Fill_form

Delete_form

Hello

?6

Method ComparisonIntroduction


16

Model

Inference Method

Learning Method

DataNone in theory, but rules are hard to write without examples

A set of rules

Code to run the rules

People edit the rules


A set of features and weights

A range of methods

A set of weights and structure of the neural network

A range of methods - this is Machine Learning

Rule-based Linear Non-Linear

Data typesIntroduction


17

Model

Inference Method

Learning Method

Data(input, output)

(sentence, sentiment)

(document, summary)

(paragraph, argument structure)

Input - Various sized pieces of text

ExampleOutput

A set of labels

Structured

Free text

Data sizesIntroduction


18

Model

Inference Method

Learning Method

Data

Language modeling: 1 million - 30 trillion+ words

Translation: 60 million words from each of 21 languages

Sentiment: 12,000 sentences

Sentence Structure: 40,000 sentences

Models are functions that take an (input, label) pair and return a score


CL Overview

19

Model

Inference Method

Learning Method

Data def model(text, label): . . . do stuff . . . . . . return score

Inputs

Scores with label 1

Scores with label 2



CL Overview

20

Model

Inference Method

Learning Method

Data

Inputs

Scores with label 1

Scores with label 2

score1 = a1x + k1

score2 = a2x + k2



CL Overview

21

Model

Inference Method

Learning Method

Data

Input 1

score1 = a1x1 + b1x2 + k1

score2 = a2x1 + b2x2 + k2 Input 2Scores with label 1 ???

Scores with label 2 ???



CL Overview

22

Model

Inference Method

Learning Method

Data

Input 1

score1 = a1x1 + b1x2 + k1

score2 = a2x1 + b2x2 + k2 Input 2Scores with label 1 ???

Scores with label 2 ???

Input 3



CL Overview

23

Model

Inference Method

Learning Method

Data def model(text, label): . . . do stuff . . . . . . return score

x1 = 1 if ‘good’ in text 0 otherwise{

Check every output option?Introduction


24

Model

Inference Method

Learning Method

DataUp to thousands of labels - all good.

Exponential set of labels - hmmm Translation:all possible sentences

Parsing:2(words squared)

Mark Liberman, http://languagelog.ldc.upenn.edu/nll/?p=17711

Algorithms to the rescue!Introduction


25

Model

Inference Method

Learning Method

Data Greedy Search

A* Search

Dynamic Programming

Best(word1) Best(word2 given word1) …

Core idea: Apply the model, see what it gets wrong, update it to fix the mistake


CL Overview

26

Model

Inference Method

Learning Method

Data

Inputs

Scores with label 1

Scores with label 2



CL Overview

27

Model

Inference Method

Learning Method

Data score1 = a1 x1 + b1 x2 + . . .

How many examples at once? 1, a few, all

How much should a, b, etc change? +/-1, vary based on error, vary based on previous updates



CL Overview

28

Model

Inference Method

Learning Method

Data score1 = a1 x1 + b1 x2 + . . .

Key property for linear models: convexity

No longer true for neural networks :(

End of Part 2Introduction


29

Model

Inference Method

Learning Method

Data

Questions?

Part 3:Computational Linguistics

Sub-Areas of Computational Linguistics

1. Natural Language Processing / Human Language TechnologyAll the things you’ve seen so far.

2. Computational PsycholinguisticsStudying humans, e.g., how do we learn language?

3. Digital LinguisticsUsing computation to support language documentation and linguistic research.

4. Other ApplicationsSocial science research that uses language, e.g., literary theory, political science

Introduction NLP Overview CL Overview

Partially based on Steve Abney’s Ling 441 notes31

Computational PsycholinguisticsIntroduction


32

Using computational methods to study the way people learn and use language.

https://penntoday.upenn.edu/2016-03-17/latest-news/eye-tracking-study-penn-linguist-reveals-inner-workings-human-mind

Digital LinguisticsIntroduction


33

Using computational methods to support language documentation and Linguistics research.

https://archive.mpi.nl/tla/elanBy Noahedits - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=84815849

Literary TheoryIntroduction


34

15,099 English novels published between 1700 and 1899

Models to understand pronoun references and identify

sentence structure

Political ScienceIntroduction


35

24,236 press releases

Model to identify topics in text

Computational SociolinguisticsIntroduction


36

Using computational methods to study the influence of society and language on each other.

LGBTQ 1986 - 2015

Thanks!What questions do you have?

nlp: overview, non-dnn approaches, computational linguistics

Documents