The document describes several bootstrap methods for estimating parameters from sample data when the underlying distribution is unknown. It outlines the bootstrap procedure, which involves resampling the original data with replacement to create bootstrap samples and estimating the parameter from each resample. Three methods for calculating the bootstrap distribution are described: direct theoretical calculation, simulation-based resampling, and Bayesian approaches. The document also provides an example of using the bootstrap to estimate the median from a sample.

1. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Bootstrap Methods:
Another Look at the Jackknife
Marco Brandi
TSI-EuroBayes Student
University Paris Dauphine
26 November 2012 / Reading Seminar on Classics
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

2. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
"To pull oneself up by one is bootstrap"
Rudolph Erich Raspe
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

3. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
OUTLINE
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

4. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

5. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
PRESENTING THE PROBLEM
X = (X1 , . . . , Xn )
Xi ∼ F with F completely unspecified
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

6. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
PRESENTING THE PROBLEM
X = (X1 , . . . , Xn )
Xi ∼ F with F completely unspecified
GOAL
⇓
Given R(X, F ) estimate R on the basis of x = (x1 , . . . , xn )
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

7. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
INTRODUCTION JACKKNIFE METHOD
θ(F ) parameter of interest and t(X) its estimator
R(X, F ) = t(X) − θ(F )
ˆ
t(X)−Bias(t)−θ(F )
R(X, F ) =
ˆ(t))1/2
(Var
ˆ ˆ
Bias(t) and Var (t) are obtained recomputing t(·) n times , each
time removing one component of X
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

8. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BOOTSTRAP METHOD
BOOTSTRAP METHOD
at x1 , x2 , . . . , xn put mass 1/n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

9. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BOOTSTRAP METHOD
BOOTSTRAP METHOD
at x1 , x2 , . . . , xn put mass 1/n
ˆ
F is the sample probability distribution
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

10. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BOOTSTRAP METHOD
BOOTSTRAP METHOD
at x1 , x2 , . . . , xn put mass 1/n
ˆ
F is the sample probability distribution
Xi∗ = xi∗ ˆ
Xi∗ ∼ F i = 1, . . . , n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

11. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BOOTSTRAP METHOD
BOOTSTRAP METHOD
at x1 , x2 , . . . , xn put mass 1/n
ˆ
F is the sample probability distribution
Xi∗ = xi∗ ˆ
Xi∗ ∼ F i = 1, . . . , n
X∗ boostrap sample
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

12. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BOOTSTRAP METHOD
BOOTSTRAP METHOD
at x1 , x2 , . . . , xn put mass 1/n
ˆ
F is the sample probability distribution
Xi∗ = xi∗ ˆ
Xi∗ ∼ F i = 1, . . . , n
X∗ boostrap sample
R∗ ˆ
= R(X∗ , F )
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

13. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
SIMPLE EXAMPLE
Dichotomous Example
θ(F ) = Pr {X = 1} ¯
R(X, F ) = X − θ(F )
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

14. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
SIMPLE EXAMPLE
Dichotomous Example
θ(F ) = Pr {X = 1} ¯
R(X, F ) = X − θ(F )
ˆ
Xi∗ = 1 x = θ(F )
¯
Xi∗ =0 1−x ¯
⇓
ˆ ¯
R ∗ = R(X∗ , F ) = X ∗ − x
¯
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

15. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
SIMPLE EXAMPLE
Dichotomous Example
θ(F ) = Pr {X = 1} ¯
R(X, F ) = X − θ(F )
ˆ
Xi∗ = 1 x = θ(F )
¯
Xi∗ =0 1−x ¯
⇓
ˆ ¯
R ∗ = R(X∗ , F ) = X ∗ − x
¯
¯
E∗ (X ∗ − x ) = 0
¯ ¯
Var∗ (X ∗ − x ) = x (1 − x )/n
¯ ¯ ¯
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

16. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
PROBLEM
The complexity on the bootstrap procedure is to calculate
the bootstrap distribution
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

17. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
PROBLEM
The complexity on the bootstrap procedure is to calculate
the bootstrap distribution
⇓
3 methods of calculation are possible
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

18. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

19. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

20. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Method 1
Direct theoretical calculation
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

21. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 1ST STEP
Initializing the procedure
θ(F ) indicate the median of F
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

22. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 1ST STEP
Initializing the procedure
θ(F ) indicate the median of F
t(X) = X(m)
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

23. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 1ST STEP
Initializing the procedure
θ(F ) indicate the median of F
t(X) = X(m)
X(1) ≤ X(2) ≤ · · · ≤ X(n) n = 2m − 1
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

24. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 1ST STEP
Initializing the procedure
θ(F ) indicate the median of F
t(X) = X(m)
X(1) ≤ X(2) ≤ · · · ≤ X(n) n = 2m − 1
R(X, F ) = t(X) − θ(F )
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

25. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 2ST STEP
Formalazing the procedure
X∗ = x∗
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

26. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 2ST STEP
Formalazing the procedure
X∗ = x∗
Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn )
∗ ∗ ∗
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

27. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 2ST STEP
Formalazing the procedure
X∗ = x∗
Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn )
∗ ∗ ∗
R∗ ˆ
= R(X∗ , F ) = X(m) − x(m)
∗
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

28. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
ESTIMATING THE MEDIAN 2ST STEP
Formalazing the procedure
X∗ = x∗
Ni∗ = #{Xi∗ = xi } N∗ = (N1 , N1 , . . . .Nn )
∗ ∗ ∗
R∗ ˆ
= R(X∗ , F ) = X(m) − x(m)
∗
l −1
Pr∗ {R ∗ = x(l) − x(m) } =Pr {Bin(n, ) ≤ m − 1}−
n (1)
l
−Pr {Bin(n, ) ≤ m − 1}
n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

29. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
RESULTS(1)
for n = 15 and m = 8
l 2 or 14 3 or 13 4 or 12 5 or 11 6 or 10 7 or 9 8
(1) .0003 .0040 .0212 .0627 .1249 .1832 .2073
15
Use E∗ (R ∗ )2 = l=1 [x(l) − x(8) ]2 Pr∗ R ∗ = x(l) − x(8)
as an estimate of EF R 2 = EF [t(X) − θ(F )]2
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

30. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
RESULTS(2)
Results for bootstrap
limn→∞ nE∗ (R ∗ )2 = 1/4f 2 (θ)
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

31. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
RESULTS(2)
Results for bootstrap
limn→∞ nE∗ (R ∗ )2 = 1/4f 2 (θ)
Results for the standard jackknife
2
limn→∞ nVarˆ(R) = (1/4f 2 (θ)) χ2
2
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

32. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

33. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
METHOD 2 - MONTE CARLO APPROXIMATION
Repeat X∗ B times
x∗1 , x∗2 , . . . , x∗B
ˆ ˆ ˆ
R(x∗1 , F ), R(x∗2 , F ), . . . , R(x∗B , F )
is taken as an approximation of the boostrap distribution
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

34. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(1)
Xi ∼ Pois(2) i = 1, . . . , 15
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

35. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(1)
Xi ∼ Pois(2) i = 1, . . . , 15
Histogram of bootstrap mean t(X) = E [X]
0.8
Density
0.4
0.0
0.5 1.0 1.5 2.0 2.5 3.0 3.5
Bootstrap estimation of mean
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

36. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(1)
Xi ∼ Pois(2) i = 1, . . . , 15
Histogram of bootstrap mean t(X) = E [X]
B = 10000
0.8
n◦ of bootstrap samples
Density
0.4
0.0
0.5 1.0 1.5 2.0 2.5 3.0 3.5
Bootstrap estimation of mean
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

37. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(1)
Xi ∼ Pois(2) i = 1, . . . , 15
Histogram of bootstrap mean t(X) = E [X]
B = 10000
0.8
n◦ of bootstrap samples
Density
mean = 1.9341
0.4
0.0
0.5 1.0 1.5 2.0 2.5 3.0 3.5
Bootstrap estimation of mean
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

38. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(1)
Xi ∼ Pois(2) i = 1, . . . , 15
Histogram of bootstrap mean t(X) = E [X]
B = 10000
0.8
n◦ of bootstrap samples
Density
mean = 1.9341
0.4
se = 0.382
0.0
0.5 1.0 1.5 2.0 2.5 3.0 3.5
Bootstrap estimation of mean
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

39. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(2)
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

40. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(2)
Histogram of bootstrap variance t(X) = V [X]
0.4
Density
0.2
0.0
0 1 2 3 4 5
Bootstrap estimation of variance
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

41. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(2)
Histogram of bootstrap variance t(X) = V [X]
B = 10000
n◦ of bootstrap samples
0.4
Density
0.2
0.0
0 1 2 3 4 5
Bootstrap estimation of variance
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

42. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(2)
Histogram of bootstrap variance t(X) = V [X]
B = 10000
n◦ of bootstrap samples
0.4
Density
mean = 2.191
0.2
0.0
0 1 2 3 4 5
Bootstrap estimation of variance
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

43. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
EXAMPLE(2)
Histogram of bootstrap variance t(X) = V [X]
B = 10000
n◦ of bootstrap samples
0.4
Density
mean = 2.191
se = 0.649
0.2
0.0
0 1 2 3 4 5
Bootstrap estimation of variance
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

44. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
R CODE
## s i m u l a t i o n poisson data
s e t . seed ( 5 9 2 )
x= r p o i s ( 1 5 , lambda =2)
B=10000
## c r e a t e t h e b o o t s t r a p f u n c t i o n
b o o t s t r a p <− f u n c t i o n ( data , nboot , t h e t a , . . . )
{
z <− l i s t ( )
datab <−
m a t r i x ( sample ( data , s i z e = l e n g t h ( data ) ∗nboot , r e p l a c e =TRUE) , nrow=nboot )
e s t b <− a p p l y ( datab , 1 , t h e t a , . . . )
e s t <− t h e t a ( data , . . . )
z$ e s t <− e s t
z$ d i s t n <− e s t b
z$ b i a s <− mean ( e s t b)−e s t
z$se <− sd ( e s t b )
z
}
## E s t i m a t i n g t h e mean
X1= b o o t s t r a p ( x , B , t h e t a =mean )
h i s t ( X1$ d i s t n , main= " Histogram o f b o o t s t r a p mean " , prob=T ,
x l a b = " B o o t s t r a p e s t i m a t i o n o f mean " )
mean ( X1$ d i s t n )
X1$se
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

45. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

46. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
METHOD 3 - RELATIONSHIP WITH THE JACKKNIFE
Pi∗ = Ni∗ /n P∗ = (P1 , P2 , . . . , Pn )
∗ ∗ ∗
E∗ P∗ = e/n Cov∗ P∗ = I/n2 − e e/n3
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

47. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
USING TAYLOR EXPANSION
ˆ
R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

48. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
USING TAYLOR EXPANSION
ˆ
R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n
1
R(P∗ ) = R(e/n) + (P∗ − e/n)U + (P∗ − e/n)V(P∗ − e/n)
2
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

49. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
USING TAYLOR EXPANSION
ˆ
R(P∗ ) = R(X∗ , F ) evaluate in P∗ = e/n
1
R(P∗ ) = R(e/n) + (P∗ − e/n)U + (P∗ − e/n)V(P∗ − e/n)
2
. . .
 .
.

. . .
. . . .
 ∂R(P∗ )  . .
∂ 2 R(P∗ )
U =  ∂P ∗  V = .
. .
.

 i  ∂Pi∗ ∂Pj∗
.
 
. .
. .
. .
.
. P∗ =e/n . . . P∗ =e/n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

50. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
DERIVATION OF BOOTSTRAP EXPECTATION AND
VARIANCE
P∗
R(P∗ ) = R n ∗
i=1 Pi
eU = 0 eV = −nU eVe = 0
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

51. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
DERIVATION OF BOOTSTRAP EXPECTATION AND
VARIANCE
P∗
R(P∗ ) = R n ∗
i=1 Pi
eU = 0 eV = −nU eVe = 0
1 1 ¯
E∗ R(P∗ ) = R(e/n) + tr V I/n2 − e e/n3 = R(e/n) + V
2 2n
n
Var∗ R(P∗ ) = U I/n2 − e e/n3 U = Ui2 /n2
i=1
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

52. INTRODUCTION
DESCRIPTION OF METHODS
METHOD 1
BOOTSTRAP IN REGRESSION MODELS
METHOD 2
BAYESIAN BOOTSTRAP
METHOD 3
DISCUSSION
BAG OF LITTLE BOOTSTRAP
RESULTS
ˆ
BiasF θ(F ) ≈ 1 ¯
2n V
ˆ n 2 2
VarF θ(F ) ≈ i=1 Ui /n
The results agree with those given by Jaeckel’s infinitesimal
jackknife
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

53. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

54. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
REGRESSION MODELS
Xi = gi (β) + i i ∼F i = 1, . . . , n
Having observed X = x we compute the estimate of β
n
2
ˆ
β = minβ ˆ
xi − gi β
i=1
ˆ 1 ˆ
F : mass at ˆi = xi − gi β
n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

55. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BOOTSTRAP SAMPLE
Xi∗ = gi β +
ˆ ∗ ∗ ˆ
∼F
i i
n
2
ˆ
β ∗ : minβ xi∗ − gi β
ˆ
i=1
β ∗1 , β ∗2 , β ∗3 , . . . , β ∗B
ˆ ˆ ˆ ˆ
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

56. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
LINEAR MODEL
gi (β) = ci β CC=G
β = G−1 C X has mean β and covariance matrix σF G−1
ˆ 2
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

57. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
LINEAR MODEL
gi (β) = ci β CC=G
β = G−1 C X has mean β and covariance matrix σF G−1
ˆ 2
ˆ
β ∗ = G−1 C X∗ has boostrap mean and variance
E∗ β ∗ = β
ˆ ˆ Cov∗ β ∗ = σ 2 G−1
ˆ ˆ
2
n ˆ
where σ 2 =
ˆ i=1 xi − g β /n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

58. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
JACKKNIFE IN LINEAR REGRESSION
Applying the infinitesimal jackknife in a linear regression model,
Hinkley derive the approximation of
n
Cov β ≈ G−1
ˆ ci ci ˆ2 G−1
i
i=1
Jackknife methods ignore that the errors i are assumed to
have the same distribution for every value of i
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

59. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

60. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
DEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin
1981)
Bayesian Bootstrap
In bootstrap we consider sample cdf is population cdf
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

61. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
DEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin
1981)
Bayesian Bootstrap
In bootstrap we consider sample cdf is population cdf
Each BB replications generates a posterior probability for
each xi
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

62. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
DEFINITION OF BAYESIAN BOOTSTRAP (D. Rubin
1981)
Bayesian Bootstrap
In bootstrap we consider sample cdf is population cdf
Each BB replications generates a posterior probability for
each xi
1
The posterior probability of each xi is centered at n but has
variability
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

63. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BB REPLICATION
BB replication
(n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

64. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BB REPLICATION
BB replication
(n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1
gl = u(l) − u(l−1)
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

65. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BB REPLICATION
BB replication
(n − 1) Unif (0, 1) u(0) = 0 e u(n) = 1
gl = u(l) − u(l−1)
Attach the vector (g1 , . . . , gn ) to the data X
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

66. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
CONCEPTUAL DIFFERENCE
Bayesian Bootstrap
Simulates the posterior distribution of the parameter
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

67. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
CONCEPTUAL DIFFERENCE
Bayesian Bootstrap
Simulates the posterior distribution of the parameter
Classical Bootstrap
Simulates the estimated sampling distribution of a statistic
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

68. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BB EXAMPLE
Dichotomous Example
The parameter is θ = Pr {Xi = 1} and let n1 number of Xi = 1
Call P1 the sum of the n1 probabilities assigned to the xi = 1
(g1 , . . . , gn ) ∼ Dirichlet(1, . . . , 1) ⇒ P1 ∼ Beta(n1 , n − n1 )
Note: Beta(n1 , n − n1 ) is the posterior distribution when the
prior is P(θ) ∝ [θ(1 − θ)]−1
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

69. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

70. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
INFERENCES PROBLEMS
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

71. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
INFERENCES PROBLEMS
Is it possible that all the values of X have been observed?
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

72. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
INFERENCES PROBLEMS
Is it possible that all the values of X have been observed?
Is it reasonable to assume a priori independent
parameters, constrained only to sum to 1, for these values?
Using the gap to simulate the posterior distributions of
parameters may no longer work
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

73. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
INFERENCES PROBLEMS
Is it possible that all the values of X have been observed?
Is it reasonable to assume a priori independent
parameters, constrained only to sum to 1, for these values?
Using the gap to simulate the posterior distributions of
parameters may no longer work
so..
BB and bootstrap cannot avoid the sensitivity of inference to
model assumptions
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

74. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
CONCLUSION
Knowledge of the context of a data set may make the
incorporation of reasonable model constraints obvious and
bootstrap may be useful in particular contexts
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

75. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
CONCLUSION
Knowledge of the context of a data set may make the
incorporation of reasonable model constraints obvious and
bootstrap may be useful in particular contexts
In general
"There are no general data analytic panaceas that
allow us to pull ourselves up by our bootstraps"
Donald Rubin
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

76. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
Outline
1 INTRODUCTION
2 DESCRIPTION OF METHODS
METHOD 1
METHOD 2
METHOD 3
3 BOOTSTRAP IN REGRESSION MODELS
4 BAYESIAN BOOTSTRAP
5 DISCUSSION
6 BAG OF LITTLE BOOTSTRAP
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

77. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BLB (M. Jordan 2012)
When n gets large computational cost is large
Expected numbers of distinct points in a resample is ∼ 0.632n
BLB Procedure
Divide the dataset in s subset of dimension b, with b < n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

78. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BLB (M. Jordan 2012)
When n gets large computational cost is large
Expected numbers of distinct points in a resample is ∼ 0.632n
BLB Procedure
Divide the dataset in s subset of dimension b, with b < n
From each subset we draw r samples with replacement of
dimension n
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

79. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BLB (M. Jordan 2012)
When n gets large computational cost is large
Expected numbers of distinct points in a resample is ∼ 0.632n
BLB Procedure
Divide the dataset in s subset of dimension b, with b < n
From each subset we draw r samples with replacement of
dimension n
Compute for each subset the estimator quality assessment
(e.g the bias) indicated with ξ
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

80. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
BLB IMAGE
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

81. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
FINALLY...
if we choose b = n0.6 ad we have a dataset of 1TB, the
subsamples contains at most 3981 distinct points and have size
at most 4GB
Like the bootstrap
Share bootstrap’s consistency
Automatic : without knowledge of the internals θ
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

82. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
FINALLY...
if we choose b = n0.6 ad we have a dataset of 1TB, the
subsamples contains at most 3981 distinct points and have size
at most 4GB
Like the bootstrap
Share bootstrap’s consistency
Automatic : without knowledge of the internals θ
Beyond the bootstrap
Can explicity control b
Generally faster than the bootstrap and requires less total
computation
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

83. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
References I
B. Efron.
Bootstrap Methods: Another Look at the Jackknife.
The Annals of Statistics, Vol. 7, No. 1, (Jan. 1979), pp. 1-26.
D.B. Rubin.
The Bayesian Bootstrap.
The Annals of Statistics, Vol. 9, No.1, pp. 130-134.
M. Jordan.
The Big Data Bootstrap.
Proceedings of the 29th International Conference on
Machine Learning (ICML).
Marco Brandi Bootstrap Methods: Another Look at the Jackknife

84. INTRODUCTION
DESCRIPTION OF METHODS
BOOTSTRAP IN REGRESSION MODELS
BAYESIAN BOOTSTRAP
DISCUSSION
BAG OF LITTLE BOOTSTRAP
THANK YOU
FOR
YOUR ATTENTION
Marco Brandi Bootstrap Methods: Another Look at the Jackknife