tests jean-yves le boudec. contents 1.the neyman pearson framework 2.likelihood ratio tests 3.anova...
TRANSCRIPT
Tests
Jean-Yves Le Boudec
Contents
1. The Neyman Pearson framework
2. Likelihood Ratio Tests
3. ANOVA
4. Asymptotic Results
5. Other Tests
2
Tests
Tests are used to give a binary answer to hypotheses of a statistical natureEx: is A better than B?
Ex: does this data come from a normal distribution ?
Ex: does factor n influence the result ?
3
Example: Non Paired Data
Is red better than blue ?
For data set (a) answer is clear (by inspection of confidence interval) no test required
4
Is this data normal ?
5
5.1 The Neyman-Pearson Framework
6
Example: Non Paired Data
7
Is red better than blue ?
Critical Region, Size and Power
8
Example : Paired Data
9
Power
10
11
Grey Zone
12
p-value of a test
13
14
Tests are just tests
15
Test versus Confidence Intervals
If you can have a confidence interval, use it instead of a test
16
2. Likelihood Ratio Test
A special case of Neyman-Pearson
A Systematic Method to define tests, of general applicability
17
18
19
A Classical Test: Student Test
The model :
The hypotheses :
20
21
22
Here it is the same as a Conf. Interval
23
The “Simple Goodness of Fit” Test
Model
Hypotheses
24
1. compute likelihood ratio statistic
25
2. compute p-value
26
Mendel’s Peas
P= 0.92 ± 0.05 => Accept H0
27
3 ANOVA
Often used as “Magic Tool”
Important to understand the underlying assumptions
ModelData comes from iid normal sample with unknown means and same variance
Hypotheses
28
29
30
The ANOVA Theorem
We build a likelihood ratio statistic test
The assumption that data is normal and variance is the same allows an explicit computation
it becomes a least square problem = a geometrical problem
we need to compute orthogonal projections on M and M0
31
The ANOVA Theorem
32
Geometrical InterpretationAccept H0 if SS2 is small
The theorem tells us what “small” means in a statistical sense
33
34
ANOVA Output: Network Monitoring
35
The Fisher-F distribution
36
37
Compare Test to Confidence Intervals
For non paired data, we cannot simply compute the differenceHowever CI is sufficient for parameter set 1
Tests disambiguate parameter sets 2 and 3
38
Test the assumptions of the test…Need to test the assumptions
Normal In each group: qqplot…
Same variance
39
40
4 Asymptotic Results
41
2 x Likelihood ratio statistic
42
The chi-square distribution
43
Asymptotic Result
Applicable when central limit theorem holds
If applicable, radically simpleCompute likelihood ratio statistic
Inspect and find the order p (nb of dimensions that H1 adds to H0)
This is equivalent to 2 optimization subproblems
lrs = = max likelihood under H1 - max likelihood under H0
The p-value is
44
Composite Goodness of Fit Test
We want to test the hypothesis that an iid sample has a distribution that comes from a given parametric family
45
Apply the Generic Method
Compute likelihood ratio statistic
Compute p-value
Either use MC or the large n asymptotic
46
47
Is it normal ?
48
49
50
Mendel’s Peas
P= 0.92 ± 0.05 => Accept H0
51
Test of Independence
Model
Hypotheses
52
Apply the generic method
53
54
5 Other TestsSimple Goodness of Fit
Model: iid data
Hypotheses: H0 common distrib has cdf F()H1 common distrib is anything
Kolmogorov-Smirnov: under H0, the distribution of
is independent of F()
55
56
Anderson-Darling
An alternative to K-S, less sensitive to “outliers”
57
58
59
Jarque Bera test of normality (Chapter 4)
Based on Kurtosis and SkewnessShould be 0 for normal distribution
60
61
Robust TestsMedian Test
Model : iid sample
Hypotheses
62
Median Test
63
Wilcoxon Signed Rank Test
64
Wilcoxon Rank Sum Test
Model: Xi and Yj independent samples, each is iid
Hypotheses: H0 both have same distributionH1 the distributions differ by a location shift
65
Wilcoxon Rank Sum Test
66
Turning Point
67
Questions
What is the critical region of a test ?
What is a type 1 error ? Type 2 ? The size of a test ?
What is the p-value of a test ?
68
Questions
What are the hypotheses for ANOVA ?
How do you compute a p-value by Monte Carlo simulation ?
A Monte Carlo simulation returns p = 0; what can we conclude ?
What is a likelihood ratio statistic test ? What can we say about its p-value ?
69
We have data X_1,…,X_m and Y_1, …,Y_m. Explain how we can compute the p-value of a test that compares the variance of the two samples ?
We have a collection of random variables X[i,j] that corresponds to the result of the ith simulation when the machine uses configuration j. How can you test whether the configuration plays a role or not ?
70