Testing point null hypothesis, a discussion by Amira Mziou

Testing a point null hypothesis: IPE

Testing a point null hypothesis: The
Irreconcilability of P-values and Evidence
Authors: James O.Berger & Thomas Sellke
Source: Journal of the American Statistical Association 1987

Presented by: MZIOU Amira

Reading Seminar in Statistical Classics: C.P Robert
Univ Paris Dauphine

February 


Outline

1 Introduction
Statistical Hypothesis Test
P-value and Evidence in Statistics


Outline

1 Introduction

2 Problematic


Outline

1 Introduction

2 Problematic

3 Posterior Probability, Bayes Factor and Posterior ODDS


Outline

1 Introduction

2 Problematic


4 Irreconcilability and Conﬂit between P-value and Pr(H0 x)

2


Outline

1 Introduction

2 Problematic



5 Various lower bounds on Posterior Probabilities

2


Outline

1 Introduction

2 Problematic




6 Solution

2


Outline

1 Introduction

2 Problematic




6 Solution

7 Conclusion

Introduction

1 Introduction

2 Problematic




6 Solution

7 Conclusion

3

Introduction


4

Introduction


Let X be a characteristic of the population whose distribution
depends on an unknown parameter θ. We want to make a
decision about the value of this parameter θ from a sample.

4

Introduction



Question :How to decide on a population from the examination
of a sample from this population ?

4

Introduction



Question :How to decide on a population from the examination
of a sample from this population ?
Deﬁnition

A statistical hypothesis test is a method of making decisions
using data, whether from a controlled experiment or an
observational study.

4

Introduction


There are six steps to do a Statistical Hypothesis Test :

5

Introduction


1 State hypothesis

5

Introduction


1 State hypothesis

2 Identify test statistic

5

Introduction


1 State hypothesis

3 Specify signiﬁcance level

5

Introduction


1 State hypothesis

4 State decision rule

5

Introduction


1 State hypothesis

5 Collect data and perform calculations

5

Introduction


1 State hypothesis

5 Collect data and perform calculations
6 Make statistical decision : reject or accept H0

5

Introduction

1 Introduction

2 Problematic




6 Solution

7 Conclusion

6

Introduction

P-Value and Evidence in statistics

7

Introduction


P-value
Statistical hypothesis tests answer the question : Assuming that
the null hypothesis is true, what is the probability of observing a
value for the test statistic that is at least as extreme as the value
that was actually observed ?

7

Introduction


P-value
That probability is known as the P-value

7

Introduction


P-value
That probability is known as the P-value

Statistical evidence
A set or collection of numbers that prove a theory or story to be
true.

Introduction

Example
Let x = (x1 , x2,... , xn ) a sample of X = (X1 , X2,... , Xn ) where the Xi are
iid N (θ, σ2 )
It’s desired to test the null hypothesis :

H0 : θ = θ0 VS H1 : θ = θ0

A classical test is based on consideration of test statistic T (X ) where
large values of T (X ) cause doubt on H0 .
The P-value of observed data x is then

p = Prθ=θ0 (T (X ) ≥ t = T (x )) (1)

√
T (X ) = n|X − θ0 |/σ (2)

8

Problematic

1 Introduction

2 Problematic




6 Solution

7 Conclusion

9

Problematic

Problematic
Most statisticians prefer use of P-values feeling it to be important to
indicate how strong the evidense against H0

10

Problematic

Problematic

Given the observed value, is it likely that H0 is true

10

Problematic

Problematic

i.e how high is P (H0x ) : the posterior probability of H0 giving x ?

10

Problematic

Problematic


Irreconcilability
By a Bayesian analysis with a ﬁxed prior and for values of t chosen to yield a
given ﬁxed p, the posterior probability of H0 is greater than p.

10

Problematic

Problematic


Irreconcilability
=⇒ P-values can be highly misleading measures of the evidence
provided by the data against the null hypothesis

10

Problematic

Problematic


Irreconcilability
=⇒ P-values can be highly misleading measures of the evidence
provided by the data against the null hypothesis

Relationship between the P-value and conditional and Bayesian
measures of evidence against the null hypothesis.

10

Posterior Probability, Bayes Factor and Posterior ODDS

1 Introduction

2 Problematic




6 Solution

7 Conclusion

11


Posterior probability
X has density f (x θ) , θ ∈ Θ

12


π0 :Prior probability of H0 (θ = θ0 ) θ0 ∈ Θ

12


π1 :Prior probability of H1 (θ = θ0 ) : π1 = 1 − π0

12



We suppose that the mass on H1 is spread out according to a density g (θ).

12




The marginal density of X is :

m(x ) = f (x /θ0 )π0 + (1 − π0 )mg (x ) (3)

where mg (x ) = f (x /θ)g (θ)d θ

12




The marginal density of X is :

m(x ) = f (x /θ0 )π0 + (1 − π0 )mg (x ) (3)

where mg (x ) = f (x /θ)g (θ)d θ

Assuming that f (x /θ) > 0, the posterior probability of H0 is given by :

π0 1 − π0 mg (x ) −1
Pr (H0 x ) = f (x θ0 ) × = [1 + × ] (4)
m (x ) π0 f (x /θ0 )
12


Posterior ODDS Ratio

Deﬁntion

The Odds Ratio is a measure of effect size, describing the strength of
association or non-independence between two binary data values. It is used
as a descriptive statistic, and plays an important role in logistic regression.

13



Deﬁntion


The posterior ODDS ratio of H0 to H1 is :

13



Deﬁntion


The posterior ODDS ratio of H0 to H1 is :

Pr (H0 /x ) π0 f (x /θ0 )
PosteriorODDSRatio = = × (5)
1 − Pr (H0 /x ) 1 − π0 mg ( x )
Prior ODDS Ratio Bayes Factor Bg (x )

13


Remark

f (x /θ0 )
• The Bayes factor Bg (x ) = does not involve the prior
mg (x )
probabilities of the hypotheses. It’s the evidence reported by the
data alone.It can be interpreted as the likelihood ratio where the
likelihood of H1 is calculated with respect to g (θ)

Irreconcilability and Conﬂit between P-value and Pr(H0 x)

1 Introduction

2 Problematic




6 Solution

7 Conclusion

15


Astronomer’s Example

16


Astronomer’s Example

An astronomer how learned that many statistical users rejected
null normal hypothesis at the 5% level when t = 1, 96 was
observed. He decides to do an experiment to verify the validity
of rejecting H0 when t = 1, 96 by testing normal hypothesis. He
supposes that about half of the point nulls are false and half are
true. When he concentrates attention on the subset in which t is
between 1, 96 and 2 , he discovers that 22% of null hypotheses
are true.

16


Example(continued)
2
Again X ∼ N (θ, σ2 ), X ∼ N (θ, σ )
n


Example(continued)
2
Again X ∼ N (θ, σ2 ), X ∼ N (θ, σ )
n

Suppose that π0 is arbitrary and g is N (θ0 , σ2 ) ,
We have that

mg (x ) ∼ N (θ0 , σ2 (1 + n−1 ))

Thus
f (x /θ0 ) 1
Bg (x ) = mg (x )
= (1 + n) 2 exp{− 1 t 2 /(1 + n−1 )}
2

and
1
P (H0 x ) = [1 + 1−π0 ]−1 =[1 + 1−π0 (1 + n)− 2 × exp[ 1 t 2 /(1 + n−1 )]]−1
π B π 2
0 g 0

17


Jeffrey’s Bayesian Analysis
1
Let π0 = 2
and g is N (θ0 , σ2 )
1
=⇒ Pr (H0 x ) = (1 + (1 + n)− 2 × exp[ 1 t 2 (1 + n−1 )])−1
2

18


1
Let π0 = 2
1
=⇒ Pr (H0 x ) = (1 + (1 + n)− 2 × exp[ 1 t 2 (1 + n−1 )])−1
2

F IGURE : Pr(H0 x ) for Jeffreys Type prior

If n = 50 and t = 1, 96 : classically we reject H0 at signiﬁcance level
p = 0, 05 , but Pr(H0 ) = 0, 52 which indicates the evidence favors H0 .

18


1
Let π0 = 2
1
=⇒ Pr (H0 x ) = (1 + (1 + n)− 2 × exp[ 1 t 2 (1 + n−1 )])−1
2

F IGURE : Pr(H0 x ) for Jeffreys Type prior

If n = 50 and t = 1, 96 : classically we reject H0 at signiﬁcance level
p = 0, 05 , but Pr(H0 ) = 0, 52 which indicates the evidence favors H0 .
=⇒ the conﬂict between p and Pr (H0 x ).
18


Bad choice of priors

19


Bad choice of priors
To prevent having a non-Bayesian reality, working with
lowers bounds on Pr (H0 x ) (translate into lower bounds on
Bg ), and raisonable densities’ classes can be considered to
be objective.

19

Various lower bounds on Posterior Probabilities

1 Introduction

2 Problematic




6 Solution

7 Conclusion

20


Lower bounds

21


Lower bounds
This section examine some lower bounds on Pr (H0 x ) when g (θ) the
distribution of θ given that H1 is true is alowed to vary within some class
of distributions G.
1 GA : all distributions
2 GS : all distributions symmetric about θ0
3 GUS : all unimodal distributions symmetric about θ0
4 GNOR :all N (θ0 , τ2 )
Letting

Pr (H0 x , G)= infg ∈G Pr (H0 x )

B (x , G) = infg ∈G Bg (x ) =⇒ B (x , G) = f (x θ0 )/supg ∈G mg (x )

Pr (H0 x , G) = [1 + 1−π0 × B (x ,G) ]−1
π
1
0

21


GA : all distribution

1
F IGURE : Comparison of P-values and Pr (H0 x , GA ) when π0 = 2

22


GS : all distributions symmetric about θ0

1
F IGURE : Comparison of P-values and Pr (H0 x , GS ) when π0 = 2

23


GUS : all unimodal distributions symmetric about
θ0

1
F IGURE : Comparison of P-values and Pr (H0 x , GUS ) when π0 = 2

24


GNOR :all N (θ0 , τ2 )

1
F IGURE : Comparison of P-values and Pr (H0 x , GNOR ) when π0 = 2

25


Comparaison of the lower bounds B(x,G) for the
four G’s considered

F IGURE : Values of B (x , G) in the normal example for different choices
of G.

26

Solution

1 Introduction

2 Problematic




6 Solution

7 Conclusion

27

Solution

Solution
Again B (x , G) can be considered to be a reasonable lower bound on the comparative
likelihood measure of evidence against H0 (under an unimodal symmetric prior on H1 ).

28

Solution

Solution
Again B (x , G) can be considered to be a reasonable lower bound on the comparative
likelihood measure of evidence against H0 (under an unimodal symmetric prior on H1 ).
Replacing t by (t − 1)+ , the lower bound on Bayes factor is quite similar to the p-value
obtained by (t − 1)+ .

F IGURE : Comparison of P-values and B (x , GUS )

28

Conclusion

1 Introduction

2 Problematic




6 Solution

7 Conclusion

29

Conclusion

P-values can be dangerous to quantify evidence against a
point null hypothesis because they lead to talk about
posterior distribution without going through Bayesian
Analysis.
What should a statistician desiring a point null hypothesis
do ?
Lower bound of Pr (H0 x ) can be argued to be useful to
test evidence : if the lower bound is large we know not to
reject H0 but if the lower bound is small we still do not know
if H0 can be rejected.
For normal distribution, replacing t by (t − 1)+ ,
p-value= 0, 05 can conﬁrms the rejection of H0

30

Conclusion

REFERENCES
JAMES O.BERGER, THOMAS SELKE, The
Irreconcilability of P-value and Evidence, JOURNAL OF
AMERICAN STATISTICAL ASSOCIATION (March 1987).
MATAN GAVISH, A Note on Berger and Selke
LINDLEY, D.V, The Use of Prior Probability Distributions in
Statistical Inference and Decision, UNIVERSITY OF
CALIFORNIA (1961)
LINDLEY, D.V, A Statistical Paradox, BIOMETRIKA(1957)
CHRISTIAN P.ROBERT,Faut-il Accepter ou rejeter les
p-values ? , PRIX DU STATISTICIEN DE LANGUE
FRANCAISE
WIKIPEDIA
31

Conclusion

THANK YOU FOR YOUR ATTENTION

32

Testing point null hypothesis, a discussion by Amira Mziou

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Testing point null hypothesis, a discussion by Amira Mziou

Similar to Testing point null hypothesis, a discussion by Amira Mziou (20)

More from Christian Robert

More from Christian Robert (20)

Recently uploaded

Recently uploaded (20)

Testing point null hypothesis, a discussion by Amira Mziou