chapter 7 natural language processing. natural language processing is a branch of ai whose goal is...
Post on 21-Dec-2015
222 views
TRANSCRIPT
Natural Language Natural Language ProcessingProcessing
Natural language processing is a branch of AI Natural language processing is a branch of AI whose goal is to facilitate communication whose goal is to facilitate communication between humans and computers using written between humans and computers using written (monologue) or oral (dialogue) of the human (monologue) or oral (dialogue) of the human languagelanguage
Natural language processing consist of two Natural language processing consist of two main areas :main areas : Natural language understanding : Natural language understanding :
to make computers understand instruction given in natural to make computers understand instruction given in natural languagelanguage
Natural language generation : Natural language generation : to make computers generate natural languageto make computers generate natural language
Study of LanguageStudy of Language Language: Language:
Written: Long-term record of knowledge from one generation Written: Long-term record of knowledge from one generation to anotherto another
Spoken: primary mean of coordinating day-to-day behavior Spoken: primary mean of coordinating day-to-day behavior with otherswith others
Natural (eg. Malay, English) vs. Artificial (Java, Prolog, Coding)Natural (eg. Malay, English) vs. Artificial (Java, Prolog, Coding) CommunicationCommunication
Use sign / natural language/ body languageUse sign / natural language/ body language Sender and ReceiverSender and Receiver
Studied in several disciplines:Studied in several disciplines: Linguist: structure of languageLinguist: structure of language PsychoLinguists: the process of human language production PsychoLinguists: the process of human language production
and comprehensionand comprehension Philosopher: how words can mean anything & how they Philosopher: how words can mean anything & how they
identify object in the world, what it means to have belief, identify object in the world, what it means to have belief, goals and intention, cognitive capabilities relate to languagegoals and intention, cognitive capabilities relate to language
CL: to develop a computational theory of Language (using CL: to develop a computational theory of Language (using the notions of algorithm & data structure from CS)the notions of algorithm & data structure from CS)
Application of NLUApplication of NLU
It represents It represents the meaning of sentences in the meaning of sentences in some representation languagesome representation language that can be that can be used later for further processing : used later for further processing : applicationsapplications
Text-based applicationsText-based applications Written text processing (books,newpaper, Written text processing (books,newpaper,
reports, manual, email, sms) = reading-based reports, manual, email, sms) = reading-based taskstasks
Searching/finding from database of textSearching/finding from database of text Extracting information from textExtracting information from text Translating documents (MT)Translating documents (MT) Summarizing texts for certain purposeSummarizing texts for certain purpose Story understandingStory understanding
Application of NLUApplication of NLU
Dialogue-based applicationsDialogue-based applications – involve human- – involve human-machine communication (spoken / machine communication (spoken / keyboard/mouse/ recognizer)keyboard/mouse/ recognizer) Q&A systems, eg. Query databaseQ&A systems, eg. Query database Automated customer service (phone)Automated customer service (phone) Tutoring systems (interaction with students)Tutoring systems (interaction with students) Spoken language control of machineSpoken language control of machine General cooperative problem-solving systemGeneral cooperative problem-solving system
Speech recognition <> Language understanding Speech recognition <> Language understanding system (only identify the word spoken from a system (only identify the word spoken from a given speech signal, not – how words are used to given speech signal, not – how words are used to communicate)communicate)
Discuss ELIZA systemDiscuss ELIZA system
ELIZA systemELIZA system Mid-1960s, MIT, a Therapist (system) & patient Mid-1960s, MIT, a Therapist (system) & patient
(user), Weizenbaum, 1966(user), Weizenbaum, 1966 Algorithm:Algorithm:
Has a Dbase of particular words (keywords)Has a Dbase of particular words (keywords) For each keyword -> store an integer, a pattern to match For each keyword -> store an integer, a pattern to match
against the input and a specification of the outputagainst the input and a specification of the output Given Sentence(S), find a keyword in S whose pattern Given Sentence(S), find a keyword in S whose pattern
matches Smatches S If > 1 keyword, pick the one with highest integer valueIf > 1 keyword, pick the one with highest integer value Use the output specification that is associated with this Use the output specification that is associated with this
keyword to generate next sentencekeyword to generate next sentence If there are No keywords, generate an innocuous If there are No keywords, generate an innocuous
continuation statement, eg: Tell me more, Go on. (figure continuation statement, eg: Tell me more, Go on. (figure 1.2, 1.3 Allen)1.2, 1.3 Allen)
Flow of Language Flow of Language AnalysisAnalysis
Natural language understanding follows the following stagesNatural language understanding follows the following stages Parsing Parsing
Involves the analysis of the syntactic structure of Involves the analysis of the syntactic structure of sentences. Parsing determines that a sentence follows the sentences. Parsing determines that a sentence follows the syntactic rules of the language. The output of the parsing syntactic rules of the language. The output of the parsing stage is a parse treestage is a parse tree
Semantic interpretation Semantic interpretation Involves the production of a representation (propositions, Involves the production of a representation (propositions,
conceptual graphs, frames) of the meaning of a sentenceconceptual graphs, frames) of the meaning of a sentence Incorporation of world knowledgeIncorporation of world knowledge
Involves the generation of an expanded representation of Involves the generation of an expanded representation of the sentence’s meaning for the complete understanding of the sentence’s meaning for the complete understanding of the sentencethe sentence
The output produced could then be used by application systems The output produced could then be used by application systems such as the database query handler, expert system interface, such as the database query handler, expert system interface, translator and HCI systems.translator and HCI systems.
Flow of Language Flow of Language AnalysisAnalysis
1.1. ParsingParsing
Sentence : Ahmad kicked the ballSentence : Ahmad kicked the ball
sentence
noun phrase verb phrase
noun
Ahmad
verb noun phrase
kicked determiner noun
the ball
Stages of Language Stages of Language AnalysisAnalysis
2. Semantic Interpretation2. Semantic Interpretation
Eg. Eg. Sentence : Ahmad kicked the ballSentence : Ahmad kicked the ball
ahmad ballkicked
Flow of Language Flow of Language AnalysisAnalysis
3. Incorporation of World Knowledge3. Incorporation of World Knowledge
Sentence : Ahmad kicked the ballSentence : Ahmad kicked the ball
ahmad ballkicked
football fieldat
human
is a
leg
haskicked
two
number
The Different Levels of The Different Levels of Language AnalysisLanguage Analysis
Phonetics/phonology Knowledge (K)-Phonetics/phonology Knowledge (K)- how words are related to the sounds that how words are related to the sounds that realize them realize them
Morphology KMorphology K– how words are constructed – how words are constructed from more basic meaning units called from more basic meaning units called morphememorpheme, the primitive unit of meaning in a , the primitive unit of meaning in a languagelanguage
Syntactic KSyntactic K– how words can be put together – how words can be put together to form correct sentences and determines to form correct sentences and determines what structural role each word plays in the what structural role each word plays in the sentence and what phrases are subparts (eg. sentence and what phrases are subparts (eg. POS) of what other phrasesPOS) of what other phrases
Levels of Language Analysis Levels of Language Analysis cont.cont.
Semantic KSemantic K– what words mean (lexical semantics) – what words mean (lexical semantics) and how these meanings combine in sentences to and how these meanings combine in sentences to form larger meaning, eg. sentence meanings form larger meaning, eg. sentence meanings (compositional semantic). Study of (compositional semantic). Study of context-context-independent meaningindependent meaning
Pragmatic KPragmatic K – concern how sentences are used in – concern how sentences are used in different situations and how use affects the different situations and how use affects the interpretation of the sentence (kind of polite and interpretation of the sentence (kind of polite and indirect language); indirect language); Context-dependent meaningContext-dependent meaning..
Discourse K-Discourse K- how the immediately preceding how the immediately preceding sentences affect the interpretation of the next sentences affect the interpretation of the next sentence. (pronoun and temporal aspects of sentence. (pronoun and temporal aspects of information conveyed)information conveyed)
World KWorld K – includes the general knowledge about the – includes the general knowledge about the structure of the world that language users must have structure of the world that language users must have in order to eg. Maintain a conversation. Includes what in order to eg. Maintain a conversation. Includes what each language user must know about the other user’s each language user must know about the other user’s beliefs and goals (discourse model)beliefs and goals (discourse model)
Morphological AnalysisMorphological Analysis The construction of words from more basic The construction of words from more basic
componentscomponents Large vocabulary system has a problem in Large vocabulary system has a problem in
representing lexiconrepresenting lexicon Reasons:Reasons:
A large number of words. Word can be formed A large number of words. Word can be formed in 2 ways:in 2 ways:
Inflectional form : go+es/ne = goes/gone (v -> v)Inflectional form : go+es/ne = goes/gone (v -> v) Derivational form : friend + ly = friendly (n -> adj)Derivational form : friend + ly = friendly (n -> adj)
Open Class words (noun, verb, adj, adv) & Open Class words (noun, verb, adj, adv) & Closed class words (articles, pronouns, Closed class words (articles, pronouns, prepositions)prepositions)
One SolutionOne Solution
Preprocess the input sentence into a Preprocess the input sentence into a sequence of sequence of morphemesmorphemes
A word may consist of a single A word may consist of a single morpheme, but often a word consists morpheme, but often a word consists of of a root form plus an affixa root form plus an affix
ExampleExample The word: The word: goesgoes
Root word : Root word : gogo Suffix : Suffix : eses (plural, present tense) (plural, present tense) Without pre-processing, a lexicon needs to list Without pre-processing, a lexicon needs to list
all the form of go, including: all the form of go, including: went, going, gonewent, going, gone With preprocessing, there would be ONE With preprocessing, there would be ONE
morpheme go that may combine with suffixes morpheme go that may combine with suffixes such as such as –ing, -es, –ing, -es, and and –en;–en; and ONE entry for and ONE entry for the irregular form: the irregular form: wentwent. Thus, the lexicon . Thus, the lexicon would only need to store TWO entries (would only need to store TWO entries (gogo and and wentwent) rather than FOUR.) rather than FOUR.
Other examples: Other examples: eaten, happiesteaten, happiest Some word cannot be decomposed into a Some word cannot be decomposed into a
root form and a suffix. Example is the root form and a suffix. Example is the word word seedseed
Finite State Transducer Finite State Transducer (FST)(FST)
A lexicon would have to encode what A lexicon would have to encode what forms are allowed with each rootforms are allowed with each root
One famous model is based on FSTsOne famous model is based on FSTs This model is like the This model is like the Finite State Finite State
MachinesMachines except that they except that they produce produce anan output given an inputoutput given an input
FST cont.FST cont.
An arc is labeled with a pair of symbolsAn arc is labeled with a pair of symbols For eg:For eg:
An arc labeled An arc labeled i:yi:y ; could only be followed if the ; could only be followed if the current input is the letter current input is the letter ii and the output is the and the output is the letter letter yy
FST can be used to concisely represents the FST can be used to concisely represents the lexicon and to transform the surface form lexicon and to transform the surface form of words into a sequence of morphemes.of words into a sequence of morphemes.
Show examples in Allen, pg 71-72Show examples in Allen, pg 71-72
FST cont.FST cont.
Arcs labeled by a single letter have that Arcs labeled by a single letter have that letter as both input and outputletter as both input and output
FST accepts the appropriate forms and FST accepts the appropriate forms and outputs the desired sequence of morphemesoutputs the desired sequence of morphemes
The entire lexicon can be encoded as an FST The entire lexicon can be encoded as an FST that encodes all the legal words and that encodes all the legal words and transforms them into morphemic sequencestransforms them into morphemic sequences
The different suffixes need only be defined The different suffixes need only be defined once, and all root forms that allow that suffix once, and all root forms that allow that suffix can point to the same nodecan point to the same node
Syntactic AnalysisSyntactic Analysis
Syntactic analysis involves analyzing the Syntactic analysis involves analyzing the structure of a sentence. This would require structure of a sentence. This would require checking whether the sentence is formed checking whether the sentence is formed according to a set of syntactic rules – grammaraccording to a set of syntactic rules – grammar
Parsing is an activity that takes a sentence as Parsing is an activity that takes a sentence as a set of linguistic token (words) and checks the a set of linguistic token (words) and checks the ordering of the tokens against a grammar. If ordering of the tokens against a grammar. If the sentence is derived from the grammar the sentence is derived from the grammar then parsing yields a parse tree of the then parsing yields a parse tree of the sentencesentence
Parsing using context free Parsing using context free grammarsgrammars
A context free grammar comprises rules that are A context free grammar comprises rules that are made up of two types of symbols – terminals and made up of two types of symbols – terminals and non terminalsnon terminals
Non – terminalsNon – terminals Terms that describe higher-level linguistic concepts such Terms that describe higher-level linguistic concepts such
as sentence, noun phrase verb phase. Non terminals as sentence, noun phrase verb phase. Non terminals need to be further expanded as they may contain other need to be further expanded as they may contain other non terminals and terminalsnon terminals and terminals
Terminals Terminals Terms that are usually individual words. Terminals Terms that are usually individual words. Terminals
cannot be further expanded. They never appear on the cannot be further expanded. They never appear on the right of a ruleright of a rule
Parsing using context free Parsing using context free grammarsgrammars
Parsing of a sentence begins with the non-terminals Parsing of a sentence begins with the non-terminals symbol symbol sentencesentence at the top of the parse tree at the top of the parse tree
Parsing progresses by way of substitutions according Parsing progresses by way of substitutions according to the rules of the grammar. to the rules of the grammar.
A legal substitution replaces the left-side of a rule A legal substitution replaces the left-side of a rule with the non-terminal (and terminal) symbols of the with the non-terminal (and terminal) symbols of the right side of the rule. In this case, higher level non-right side of the rule. In this case, higher level non-terminal symbols are replaces by lower level non-terminal symbols are replaces by lower level non-terminal symbols or terminals.terminal symbols or terminals.
Parsing is terminated when all the lower nodes of the Parsing is terminated when all the lower nodes of the parse tree comprise terminals, i.e. individual words.parse tree comprise terminals, i.e. individual words.
If the order of the terminals in the parse tree is the If the order of the terminals in the parse tree is the same as that of the original sentence when it is said same as that of the original sentence when it is said the sentence follows the rules of the language, i.e. is the sentence follows the rules of the language, i.e. is a legal sentencea legal sentence
Parsing a Natural Language Parsing a Natural Language SentenceSentence
Consider the grammar :Consider the grammar :1.1. sentence -> noun_phrase verb_phrasesentence -> noun_phrase verb_phrase2.2. noun_phrase -> nounnoun_phrase -> noun3.3. noun_phrase -> article nounnoun_phrase -> article noun4.4. verb_phrase -> verb verb_phrase -> verb 5.5. verb_phrase -> verb noun_phraseverb_phrase -> verb noun_phrase6.6. article -> thearticle -> the7.7. article -> aarticle -> a8.8. noun -> mannoun -> man9.9. noun ->carnoun ->car10.10. verb -> droveverb -> drove
Step 1Step 1
The man drove a The man drove a carcar sentence
noun_phrase verb_phrase
The man drove a car
Step 2Step 2
The man drove a The man drove a carcar sentence
noun_phrase verb_phrase
The man drove a car
article noun verb noun_phrase
Step 3Step 3
The man drove a The man drove a carcar
article
sentence
noun_phrase verb_phrase
The man drove a car
noun verb noun_phrase
article noun
Step 4Step 4
The man drove a The man drove a carcar
article
sentence
noun_phrase verb_phrase
noun verb noun_phrase
article noun
the man adrove car
Parsing a Natural Language Parsing a Natural Language SentenceSentence Derivation of the sentence “the man drove a car” Derivation of the sentence “the man drove a car”
according to the given grammaraccording to the given grammar
STRINGSTRING APPLY RULE #APPLY RULE #
1. 1. sentencesentence 11
2. 2. noun_phrasenoun_phrase verb_phrase verb_phrase 33
3. article noun 3. article noun verb_phraseverb_phrase 55
4. article noun verb 4. article noun verb noun_phrasenoun_phrase
33
5. 5. articlearticle noun verb article noun verb article nounnoun
66
6. 6. thethe nounnoun verb article noun verb article noun 88
7. 7. thethe manman verbverb article noun article noun 1010
8. 8. thethe manman drovedrove articlearticle noun noun 77
9. 9. thethe manman drovedrove thethe nounnoun 99
9. 9. thethe manman drovedrove thethe carcar Sentence parsedSentence parsed
Representation & Representation & UnderstandingUnderstanding
A crucial component of A crucial component of understanding involves computing a understanding involves computing a representation of the meaning of representation of the meaning of sentences and texts. (Reason: sentences and texts. (Reason: Senses & ambiguity)Senses & ambiguity)
Representations and Representations and UnderstandingUnderstanding
Computing a representation of the meaning of Computing a representation of the meaning of sentences and texts (Notion of representation)sentences and texts (Notion of representation)
Why can’t use the sentence itself as a representation Why can’t use the sentence itself as a representation of its meaning? Most words have multiple meanings of its meaning? Most words have multiple meanings ((SensesSenses). eg. Cook, bank, still (verb or noun), ). eg. Cook, bank, still (verb or noun), I made her duck.I made her duck. I saw a man in the park with a telescopeI saw a man in the park with a telescope
Thus, Thus, ambiguityambiguity inhibit system from making the inhibit system from making the appropriate inferences needed to model appropriate inferences needed to model understanding (need to resolve or disambiguate: eg. understanding (need to resolve or disambiguate: eg. Use Lexical disambiguation: POS, word-sense Use Lexical disambiguation: POS, word-sense disambiguation, ontology)disambiguation, ontology)
A program must explicitly consider each senses of a A program must explicitly consider each senses of a word to understand a sentenceword to understand a sentence
Represent meaning: must have a more Represent meaning: must have a more precise languageprecise language
Mathematics & Logic and the use of Mathematics & Logic and the use of formally specified representation formally specified representation languages (formal language) – notion of an languages (formal language) – notion of an atomic symbolatomic symbol
Useful representation languages have 2 Useful representation languages have 2 properties:properties: Precise and unambiguousPrecise and unambiguous Capture the intuitive structure of the natural Capture the intuitive structure of the natural
language sentences that it representslanguage sentences that it represents
RepresentationRepresentation SyntaxSyntax – indicates the way that words in the – indicates the way that words in the
sentence are related to each othersentence are related to each other The structure illustrates how the words are The structure illustrates how the words are
grouped together into phrases, what words grouped together into phrases, what words modify what other words and what words are of modify what other words and what words are of central importance in the sentencecentral importance in the sentence
It may identify the types relationships that exist It may identify the types relationships that exist between phrases and can store information about between phrases and can store information about the particular sentence structure that may be the particular sentence structure that may be needed for later processingneeded for later processing
Eg: 1. John sold the book to MaryEg: 1. John sold the book to Mary 2. The book was sold to Mary by John2. The book was sold to Mary by John
Representation cont.Representation cont.
Sentence Structure does not reflect Sentence Structure does not reflect its meaning (although have the same its meaning (although have the same syntactic structure, eg. the catch)syntactic structure, eg. the catch)
The intended meaning of a sentence The intended meaning of a sentence depends on depends on the situationthe situation in which in which the sentence is produced.the sentence is produced.
Context independent (the logical Context independent (the logical form,LF) vs. Context dependentform,LF) vs. Context dependent
Semantic Analysis:Semantic Analysis:The Logical Form, LFThe Logical Form, LF
LF = encodes possible word senses and identifies LF = encodes possible word senses and identifies the semantic relationships between the words the semantic relationships between the words and phrasesand phrases
Many of the relationships are captured using an Many of the relationships are captured using an abstract set of semantic relationships between abstract set of semantic relationships between the verb and its NPthe verb and its NP
Context Independent Context Independent Eg: Selling event, John is the seller, the book is Eg: Selling event, John is the seller, the book is
the object being sold and Mary is the buyer.the object being sold and Mary is the buyer. These roles are instances of the abstract These roles are instances of the abstract
semantic roles: AGENT, THEME and TO-POSS semantic roles: AGENT, THEME and TO-POSS (final possessor), respectively.(final possessor), respectively.
Show another example: invite - the ballShow another example: invite - the ball
The Final Meaning The Final Meaning RepresentationRepresentation
The final representation: a The final representation: a general Knowledge general Knowledge Representations languageRepresentations language, which is the system , which is the system uses to represent and reason about its application uses to represent and reason about its application domaindomain
The goal of contextual interpretation is to take a The goal of contextual interpretation is to take a representation of the structure of a sentence and representation of the structure of a sentence and its logical form, and to map this into some its logical form, and to map this into some expression in the KR that allow the system to expression in the KR that allow the system to perform the appropriate task in the domain.perform the appropriate task in the domain.
This is the language in which all the specific This is the language in which all the specific knowledge based on the application is representedknowledge based on the application is represented
Use FOPC, Semantic NetworkUse FOPC, Semantic Network Eg: Q-A application – a Q might map to a DB; Story Eg: Q-A application – a Q might map to a DB; Story
Understanding application – a sentence might map Understanding application – a sentence might map into a set of expressions that represent the into a set of expressions that represent the situation that the sentence describes.situation that the sentence describes.
Discourse & Pragmatic Discourse & Pragmatic AnalysisAnalysis
Context DependentContext Dependent Discourse Structure TheoryDiscourse Structure Theory
Discourse RelationsDiscourse Relations Discourse ModelDiscourse Model Discourse StructureDiscourse Structure
World KnowledgeWorld Knowledge Domain SpecificDomain Specific CorpusCorpus
DiscussionDiscussion
Use the following sentences to Use the following sentences to understand (to describes) the understand (to describes) the distinction between syntax, distinction between syntax, semantics and pragmatics:semantics and pragmatics: Language is one of the fundamental Language is one of the fundamental
aspects of human behavior and is a aspects of human behavior and is a crucial component of our lives.crucial component of our lives.
Green frogs have large noses.Green frogs have large noses. Green ideas have large noses.Green ideas have large noses. Large have green ideas noses.Large have green ideas noses.
Discuss the following Discuss the following sentences (ambiguity)sentences (ambiguity)
1. I made her duck. (5 meanings)1. I made her duck. (5 meanings)
2. I saw a man in the park with a 2. I saw a man in the park with a telescope. (2 meanings)telescope. (2 meanings)
Make your own ambiguous sentencesMake your own ambiguous sentences
BibliographyBibliography
ACL (Association for CL) / EACLACL (Association for CL) / EACL COLING (int conference of CL)COLING (int conference of CL) Applied NLPApplied NLP Workshop on Human Language TechnologyWorkshop on Human Language Technology Journal: CL & NLEJournal: CL & NLE IEEE ICASSP: Acoustic, Speech and Signal IEEE ICASSP: Acoustic, Speech and Signal
ProcessingProcessing IEEE Transactions on Pattern Analysis and IEEE Transactions on Pattern Analysis and
Machine IntelligenceMachine Intelligence IJCAI: Int Joint Conference on AIIJCAI: Int Joint Conference on AI Journal: AI, Computational Intelligence, Cognitive Journal: AI, Computational Intelligence, Cognitive
ScienceScience