PowerPoint Presentation

Albert GattCorpora and Statistical Methods Part 2Preliminaries: Hypothesis testing and the binomial distributionPermutationsSuppose we have the 5 words {the, dog, ate, a, bone}How many permutations (possible orderings) are there of these words?the dog ate a bonedog the ate a bone

E.g. there are 5! = 120 ways of permuting 5 words.

Binomial coefficientSlight variation:How many different choices of three words are there out of these 5?This is known as an n choose k problem, in our case: 5 choose 3

For our problem, this gives us 10 ways of choosing three items out of 5

Bernoulli trialsA Bernoulli (or binomial) trial is like a coin flip. Features: There are two possible outcomes (not necessarily with the same likelihood), e.g. success/failure or 1/0.If the situation is repeated, then the likelihoods of the two outcomes are stable.

Sampling with/out replacementSuppose were interested in the probability of pulling out a function word from a corpus of 100 words.we pull out words one by one without putting them back

Is this a Bernoulli trial?we have a notion of success/failure: w is either a function word (success) or not (failure)but our chances arent the same across trials: they diminish since we sample without replacement

Cutting cornersIf the sample (e.g. the corpus) is large enough, then we can assume a Bernoulli situation even if we sample without replacement.Suppose our corpus has 52 million wordsSuccess = pulling out a function wordSuppose there are 13 million function wordsFirst trial: p(success) = .25Second trial: p(success) = 12,999,999/51,999,999 = .249On very large samples, the chances remain relatively stable even without replacement.Binomial probabilities - ILet represent the probability of success on a Bernoulli trial (e.g. our simple word game on a large corpus).Then, p(failure) = 1 - Problem: What are the chances of achieving success 3 times out of 5 trials?Assumption: each trial is independent of every other. (Is this assumption reasonable?)

Binomial probabilities - IIHow many ways are there of getting success three times out of 5?Several: SSSFF, SFSFS, SFSSF, To estimate the number of possible ways of getting k outcomes from n possibilities, we use the binomial coefficient:

Binomial probabilities - III5 choose 3 gives 10.Given independence, each of these sequences is equally likely.

Whats the probability of a sequence?its an AND problem (multiplication rule)P(SSSFF) = (1- )(1 ) = 3(1- )2P(SFSFS) = (1- ) (1- ) = 3(1- )2(they all come out the same)

Binomial probabilities - IVThe binomial distribution states that:given n Bernoulli trials, with probability of success on each trial, the probability of getting exactly k successes is:

probability of each successNumber of different ways of getting k successesprobability of k successes out of nExpected value and varianceExpected value:

where is our probability of success

Expected value of X over n trialsVariance of X over n trialsUsing the t-test for collocation discoveryThe logic of hypothesis testingThe typical scenario in hypothesis testing compares two hypotheses:The research hypothesisA null hypothesis

The idea is to set up our experiment (study, etc) in such a way that:If we show the null hypothesis to be false thenwe can affirm our research hypothesis with a certain degree of confidenceH0 for collocation studiesThere is no real association between w1 and w2, i.e. occurrence of is no more likely than chance.

More formally:H0: P(w1 & w2) = P(w1)P(w2)i.e. P(w1) and P(w2) are independentSome more on hypothesis testingOur research hypothesis (H1): are strong collocatesP(w1 & w2) > P(w1)P(w2)

A null hypothesis H0P(w1 & w2) = P(w1)P(w2)

How do we know whether our results are sufficient to affirm H1?I.e. how big is our risk of wrongly falsifying H0?The notion of significanceWe generally fix a level of confidence in advance.

In many disciplines, were happy with being 95% confident that the result we obtain is correct.So we have a 5% chance of error.Therefore, we state our results at p = 0.05 The probability of wrongly rejecting H0 is 5% (0.05)

Tests for significanceMany of the tests we use involve:having a prior notion of what the mean/variance of a population is, according to H0computing the mean/variance on our sample of the populationchecking whether the sample mean/variance is different from the sample predicted by H0, at 95% confidence.The t-test: strategyobtain mean (x) and variance (s2) for a sampleH0: sample is drawn from a population with mean and variance 2estimate the t value: this compares the sample mean/variance to the expected (population) mean/variance under H0check if any difference found is significant enough to reject H0Computing tcalculate difference between sample mean and expected population meanscale the difference by the variance

Assumption: population is normally distributed.If t is big enough, we reject H0. The magnitude of t given our sample size N is simply looked up in a table. Tables tell us what the level of significance is (p-value, or likelihood of making a Type 1 error, wrongly rejecting H0).

Example: new companiesWe think of our corpus as a series of bigrams, and each sample we take is an indicator variable (Bernoulli trial):value = 1 if a bigram is new companiesvalue = 0 otherwise

Compute P(new) and P(companies) using standard MLE.

H0: P(new companies) = P(new)P(companies)

Example continuedWe have computed the likelihood of our bigram of interest under H0.Since this is a Bernoulli Trial, this is also our expected mean.

We then compute the actual sample probability of (new companies).Compute t and check significanceUses of the t-testOften used to rank candidate collocations, rather than compute significance.Stop word lists must be used, else all bigrams will be significant.e.g. M&S report 824 out of 831 bigrams that pass the significance test. Reason: language is just not randomregularities mean that if the corpus is large enough, all bigrams will occur together regularly and often enough to be significant.Kilgarriff (2005): Any null hypothesis will be rejected on a large enough corpus.Extending the t-test to compare samplesVariation on the original problem:what co-occurrence relations are best to distinguish between two words, w1 and w1 that are near-synonyms?e.g. strong vs. powerful

Strategy:find all bigrams and e.g. strong tea, strong supportcheck, for each w1, if it occurs significantly more often with w2, versus w2.

NB. This is a two-sample t-testTwo-sample t-test: detailsH0: For any w1, the probabilities of and is the same.i.e. (expected difference) = 0

Strategy:extract sample of and assume they are independentcompute mean and SD for each samplecompute tcheck for significance: is the magnitude of the difference large enough?

Formula:

Simplifying under binomial assumptionsOn large samples, variance in the binomial distribution approaches the mean. I.e.:

(similarly for the other sample mean)

Therefore:

Concrete example: strong vs. powerful (M&S, p. 167); NY Times

Words occurring significantly more often with powerful than strongWords occurring significantly more often with strong than powerfulCriticisms of the t-testAssumes that the probabilities are normally distributed. This is probably not the case in linguistic data, where probabilities tend to be very large or very small.

Alternative: chi-squared test (2)compare differences between expected and observed frequencies (e.g. of bigrams)The chi-square testExampleImagine were interested in whether poor performance is a good collocation. H0: frequency of poor performance is no different from the expected frequency if each word occurs independently.

Find frequencies of bigrams containing poor, performance and poor performance.compare actual to expected frequenciescheck if the value is high enough to reject H0

Example continuedf(w1= poor)F(w1 =/= poor)f(w2=performance)15(poor performance)1,230(bad performance)F(w2 =/= performance)3,580(poor people)12,000(all other bigrams)

OBSERVED FREQUENCIESExpected frequencies need to be computed for each cell:E.g. expected value for cell (1,1) poor performance:

Computing the valueThe chi-squared value is the sum of differences of observed and expected frequencies, scaled by expected frequencies.

Value is once again looked up in a table to check if degree of confidence (p-value) is acceptable.If so, we conclude that the dependency between w1 and w2 is significant.

More applications of this statisticKilgarriff and Rose 1998 use chi-square as a measure of corpus similaritydraw up an n (row)*2 (column) tablecolumns correspond to corporarows correspond to individual typescompare the difference in counts between corporaH0: corpora are drawn from the same underlying linguistic population (e.g. register or variety)corpora will be highly similar if the ratio of counts for each word is roughly constant. This uses lexical variation to compute corpus-similarity.Limitations of t-test and chi-squareNot easily interpretablea large chi-square or t value suggests a large differencebut makes more sense as a comparative measure, rather than in absolute terms

t-test is problematic because of the normality assumption

chi-square doesnt work very well for small frequencies (by convention, we dont calculate it if the expected value for any of the cells is less than 5)but n-grams will often be infrequent!Likelihood ratios for collocation discoveryRationaleA likelihood ratio is the ratio of two probabilitiesindicates how much more likely one hypothesis is compared to another

Notation:c1 = C(w1)c2 = C(w2)c12 = C()

Hypotheses:H0: P(w2|w1) = p = P(w2|w1)H1: P(w2|w1) = p1P(w2|w1) = p2p1 =/= p2

H0H1P(w2|w1)P(w2|w1)Prob. that c12 bigrams out of c1 are Prob. that c2 - c12 out of N- c1 bigrams are )Computing the likelihood ratio

Computing the likelihood ratioThe likelihood (odds) that a hypothesis H is correct is L(H).

Computing the Likelihood ratioWe usually compute the log of the ratio:

Usually expressed as:

because, for v. large samples, this is roughly equivalent to a 2 value

Interpreting the ratioSuppose that the likelihood ratio for some bigram is x. This says:If we make the hypothesis that w2 is somehow dependent on w1, then we expect it to occur x times more than its actual base rate of occurrence would predict.This ratio is also better for sparse data.we can use the estimate as an approximate chi-square value even when expected frequencies are small.Concrete example: bigrams involving powerful (M&S, p. 174)

Source: NY Times corpus (N=14.3m)

Note: sparse data can still have a high log likelihood value!

Interpreting -2 log l as chi-squared allows us to reject H0, even for small samples (e.g. powerful cudgels)Relative frequency ratiosAn extension of the same logic of a likelihood ratioused to compare collocations across corpora

Let be our bigram of interest. Let C1 and C2 be two corpora:p1 = P() in C1p2 = P() in C2.

r= p1/p2 gives an indication of the relative likelihood of in C1 and C2.

Example applicationManning and Schutze (p.176) compare:C1: NY Times texts from 1990C2: NY Times texts from 1989

Bigram occurs 44 times in C2, but only 2 times in C1, so r = 0.03

The big difference is due to 1989 papers dealing more with the fall of the Berlin Wall.

SummaryWeve now considered two forms of hypothesis testing:t-testchi-squareAlso, log-likelihood ratios as measures of relative probability under different hypotheses.Next, we begin to look at the problem of lexical acquisition.ReferencesM. Lapata, S. McDonald & F. Keller (1999). Determinants of Adjective-Noun plausibility. Proceedings of the 9th Conference of the European Chapter of the Association for Computational Linguistics, EACL-99

A. Kilgarriff (2005). Language is never, ever, ever random. Corpus Linguistics and Linguistic Theory 1(2): 263

Church, K. and Hanks, P. (1990). Word association norms, mutual information and lexicography. Computational Linguistics 16(1).