text classification 1 - sameer singhsameersingh.org/courses/statnlp/wi17/slides/lecture-0112... ·...

TextClassification1

Prof.SameerSinghCS295:STATISTICALNLP

WINTER2017

January12,2017

BasedonslidesfromNathanSchneider,NoahSmith,DanKleinandeveryoneelsetheycopiedfrom.

TextClassification1

CS295:STATISTICALNLP(WINTER2017) 2

IntroductiontoTextClassification

NaiveBayesClassification

CourseProjects

TextClassification




CourseProjects

SentimentAnalysis


Filledwithhorrificdialogue,laughablecharacters,alaughableplot,adreallynointerestingstakesduringthisfilm,"StarWarsEpisodeI:ThePhantomMenace"isnotatallwhatIwantedfromafilmthatissupposedtobethehugeopeningtothesegueintothefantasticOriginalTrilogy.Thepositivesincludethescore,thesound…

OtherExamples


• Reviewsoffilms,restaurants,products:positivevs.negative• Amazonreviewsdata,IMDBreviewsdata

• Library-likesubjects(e.g.,theDeweydecimalsystem)• Newsstories:politicsvs.sportsvs.businessvs.technology...

• 20newsgroupdata• Authorattributes:identity,politicalstance,gender,age,...• Email:spamvs.not

• Gmail:important,promotion,updates,socialmedia,…• Whatisthereadinglevelofapieceoftext?

• Automaticgraders?• Howinfluentialwillascientificpaperbe?• Advertisementrecommendations…• Willapieceofproposedlegislationpass?

• Identifythepresidentialcandidatefromspeeches• Postrecommendations/Fakenewsdetection

• Canmajorlyinfluencetheworld!

FormalSetup


Classification

SupervisedLearning

TrainingAlgorithm

Evaluation:ContingencyTable


Accuracy


Problem

• Classimbalancehurts..• Gettingoneclassrightmattersmorethantheother(retrieval)

PrecisionandRecall


>2Classes?


Macro-averagedMeasures

Micro-averagedMeasures

StatisticalSignificance


McNemar’s Test,Psychometrika, (1947)MoretestsinSmithbook,appendixB

TextClassification




CourseProjects

ClassificationusingJointProb


NaïveBayesClassifier


Twoassumptions

• Wordorderingdoesnotmatter(BagofWords)

NaïveBayesClassifier


Twoassumptions

• Wordorderingdoesnotmatter (BagofWords)• Wordsareindependentgivencategory

EstimationofParameters


ProblemwithNaïveBayes


LinearModels


NaïveBayesasaLinearModel


TextClassification




CourseProjects

GroupProjects


• Idealteamsizeis3• Absolutemaximumof4• <3ifIapprove(ongoingwork)

GroupsfortheProject

• Firsttworeportsareveryshort(1page)• Finalreportmattersthemost

SubmitFourReports

• Outputisanyphraseorsentence,definitely!• Inputisanyphraseorsentence

• Outputisasequenceorstructure(yes!)• Classification:onlyifoverwordsorphrases

• Outputislinguisticclasses/structures(yes!)

HowdoIknowit’sNLP?

ScopeofWork


• NewTask/Data• NewMethod/Models• NewApplicationofExistingMethodtoExistingTask

Novelty

• Youdonothavemuchtime!• Aimtohavethewholepipelinedonesoon• Keepthe“scale”ofthedatasmall,sub-sampleifneeded• Bettertohaveacompletefinishedreport

• thangrandideasthatdidnotwork

Butnottoomuch!

• Youdonothavetocodeeverything• Exploitexistingcode,datasets,libraries,webservices• Donotreinventallthewheels!

Reuse

Example1:What’stheword..


What’sthewordforsomeoneusingpretentiouswords? lexiphanic

MachineLearning(LSTM)

definitionofawordfromthedictionary theworditself

ThiscanbeacoolTwitterbot!

• Accuracyofguessingtheword,usingdefinitionsfromdifferentdictionary?

• Baselines:Google,reversedictionary.org,…Evaluation

Example2:SQuAD


https://rajpurkar.github.io/SQuAD-explorer/

Tesla was the fourth of five children. He had an older brother named Dane and three sisters, Milka, Angelina and Marica. Dane was killed in a horse-riding accident when Nikola was five. In 1861, Tesla attended the "Lower" or "Primary" School in Smiljan where he studied German, arithmetic, and religion. In 1862, the Tesla family moved to Gospić, Austrian Empire, where Tesla's father worked as a pastor. Nikola completed "Lower" or "Primary" School, followed by the "Lower Real Gymnasium" or "Normal School."

How many siblings did Tesla have?fourWhat was Tesla’s brother’s name?DaneWhat happened to Dane?killed in a horse-riding accident

DatasetsandPapers


• SearchKaggle,Quora,etc forlargetextdatasets• SeerecentpapersinNLPforreleaseddatasets• Lookfor“sharedtasks”,“challenges”,workshops• Linkstosomeexistingdatasetscomingtowebsitesoon

Data

• NLPConferences:ACL,EMNLP,NAACL• MLConferences:NIPS,ICML,ICLR,AAAI• Datafocusedvenues:TREC/TAC,SemEval,CONLL• Workshopsattheseconferences:interestingdirections• Morepaperscomingsoontothewebsite

Papers

WritingthePitch


• Teamnameandmembers• Singlesentencedescriptionforeachmember

• (approximately)whattheywilldo• Singlesentenceonwhatmakesyourteamdiverse

Team

• MotivationandProblemDescription• Plannedapproach:tentative• Evaluation:usually,mostimportant

Project

• If1or2,meetmebefore/on January17(o.w.noneed)• Everygrouphastomeetafterwardstodiscusstheproject

Appointment

Upcoming…


• Homework1isup!• Nextlectureswillcontinuewithmoredetails• SignupfortheKaggle account(@uci.edu email)• Due:January26,2017

Homework

• ProjectpitchisdueJanuary23,2017!• Startassemblingteamsnow!(usePiazza)• Startlookingatpapers,data,etc.forideas

Project

text classification 1 - sameer singhsameersingh.org/courses/statnlp/wi17/slides/lecture-0112... ·...

Documents