# A Geometric Note on a Type of Multiple Testing-07-24-2015

16 May 2016
1 sur 22

### A Geometric Note on a Type of Multiple Testing-07-24-2015

• 1. A Geometric Note on a Type of Multiple Testing Dipak K Dey, Junfeng Liu, Nalini Ravishanker, Edwards Qiang Zhang (07-24-2015) ABSTRACT. For a collection of subjects, the within-subject replicate measurements are usually modeled as subject-speciﬁc mean (zero and/or non-zero) plus random noises. For the problem of selecting a set of potentially signiﬁcant subjects (likely with non-zero means) out of all subjects, we study some new aspects of the elegant false discovery rate (FDR) control procedure proposed by Benjamini and Hochberg (1995). 1 Introduction We present the collected measurements as yi,j = µi + ϵi,j, ϵi,j ∼ N(0, σ2 i ), j = 1, . . . , m, i = 1, . . . , n, where n is the total number of subjects and m is the sample size (number of replicates) for each subject. A type of conﬁdence interval for each subject mean (µi) could be constructed as µi ∈ ¯yi ± Ctm−1,1− α 2 ˆσm−1/ √ m, i = 1, . . . , n. where, 1 − α is the prescribed conﬁdence level, subject-speciﬁc variance is estimated as ˆσ2 m−1 = 1 m−1 ∑m j=1(yi,j − ¯yi)2 involving subject-speciﬁc mean estimator ¯yi = 1 m ∑m j=1 yi,j, tm−1,1− α 2 is the 1 − α 2 quantile of the central Students’ t-distribution with degrees of freedom m − 1, and C is a cross-the-board tuning parameter. Simply employing the following rule | √ m¯yi ˆσm−1 | > Ctm−1,1−α 2 → reject µi = 0 | √ m¯yi ˆσm−1 | ≤ Ctm−1,1−α 2 → accept µi = 0 (1) relates to checking out t-statistic based p-value, pi = 1 − F(| √ m¯yi ˆσm−1 |), where F is the probability distribution function for certain random variable. For instance, F could be the probability distri- bution of |Tm−1| (Tm−1 is the central Students’ t-statistic with degrees of freedom m − 1). Rule (1) becomes into 1
• 2. pi < 1 − Fm−1(Ctm−1,1− α 2 ) → reject µi = 0, pi ≥ 1 − Fm−1(Ctm−1,1−α 2 ) → accept µi = 0. We ﬁrst look at an example where three groups have subject mean (indexed by i = 1,. . .,n = 100 for each group) proﬁles µi = 0 (Group 1), µi = 0.01(1 + sin10i n ) (Group 2) and µi = 0.10(1 + sin10i n ) (Group 3), respectively. The within-subject variation σ = 1. Under rule (1), the rejection proportion proﬁles (α = 0.10, m varies from 6 to 10, C = 1 + j 20 (j = 1, . . . , 16)) are plotted in Figure 1. Groups 1 and 2 have similar rejection proportion proﬁles since these two subject mean proﬁles are substantially close to each other. Thus, the resultant false discovery rate in this manner is roughly 1/2 when we combine these two groups (H0 = Group 1 (zero mean), Ha = Group 2 (non-zero mean)) under any values of m and C. 2 New perspectives Built upon the ordered-p-value set ({p(j), 1 ≤ j ≤ n}) with p-values (indexed by rank j) being arranged from the smallest to the largest, the elegant false discovery rate (FDR) control procedure (Benjamini and Hochberg, 1995) would reject all subjects with rank ≤ max{j : p(j) ≤ j n q, 1 ≤ j ≤ n}, (2) where n is the total number of hypotheses (subjects) with H0 and Ha combined. If rejections are found then the instant FDR is calculated as the proportion of wrong rejections out of all rejections (H0 and Ha combined). If no rejections are found then the instant FDR is deﬁned as 0. The so-called FDR which is deﬁned as the expectation of the instant FDRs is controlled at π0q, where π0 is the proportion of H0 hypotheses (subjects) out of all hypotheses. For illustration purposes, we set the subject mean function (Ha) as f(u, x) = 0.08u(1 + |sin(6x)|u ), x ∈ [0, 1], u = 1, 2, . . . . (3) The subject means under H0 are implemented through setting u = 0 at x = i n0 (i = 1, . . . , n0), where n0 is the number of subjects (hypotheses) under H0. The subject means under Ha are implemented 2
• 3. through setting x = i n1 (i = 1, . . . , n1), where n1 is the number of subjects (hypotheses) under Ha. Under any numerical simulation conﬁguration (subject group size (n0, n1), within-subject variation (σ), within-subject replicate/sample size (m)), separating Ha subjects from H0 is expected to be easier as we increase Ha subject mean proﬁle to ∞. We take a look at the resultant speciﬁcity proﬁles and ﬁnd they approach to a limit (regulated by q) as Ha mean proﬁle increases. Such a limit is achieved exactly once Ha mean proﬁle is suﬃciently large. We are thus motivated to take a geometric view by juxtaposing the ordered-p-value proﬁles (H0 and Ha) along with an overriding adaptive hypothesis rejection cut-oﬀ route (indexed by subjects, H0 and H1 combined) for sequential p-value check. In Figure 2, the ordered-p-value proﬁle under H0 roughly resembles a straight line connecting points (π1,0) and (1,1). As ordered-p-value proﬁle under Ha approaches to the bottom (mean proﬁle increases), the rejected hypothesis set includes all Ha and those H0 subjects with p-value located from D to B (Rule (2)). The limiting speciﬁcity is subsequently calculated. Along the cut route (the solid line spanning from (0,0) to (x1,y1)) in Figure 2, each check point j∗ ∈ {1, . . . , n(= n0 + n1)} corresponds to a number (n0(j∗ )) of p-values (≤ j∗ n q, under H0) and another number (n1(j∗ )) of p-values (≤ j∗ n q, under Ha) (Figure 3). All those hypotheses linked to these n0(j∗ )+n1(j∗ ) p-values will be rejected as long as n0(j∗ ) + n1(j∗ ) ≥ j∗ . However, any check point (j∗ ) along the cut route (Figure 2) which is beyond that one (j∗ B) corresponding to point B would not be able to collect a suﬃcient number of hypotheses (H0 and Ha combined) such that n0(j∗ ) + n1(j∗ ) ≥ j∗ . The set {j∗ − n0(j∗ ) : 1 ≤ j∗ ≤ j∗ B} roughly formulates a no-rejection region boundary prescribed for Ha hypotheses (the bold dash line, Figure 3), i.e., there will be no discovery (rejection) unless the ordered-p-value proﬁle under Ha ever crosses this boundary from upper portion (“NO REJECTION region”, Figure 3)) to the lower portion. When there is such a crossing, geometric arguments show that the instant FDR is always around π0q no matter where the crossing point is located along the no-rejection boundary. Numerical simulation would disclose some operating characteristics under diﬀerent speciﬁcations on experimental factors (e.g., within-subject variation (σ), within-subject sample size (m), Ha subject mean proﬁle, population size (n0 + n1), H0 proportion (π0 = n0/(n0 + n1)), etc.). Moreover, we also try applying a quadratic cut route reject all subjects with rank ≤ max{j : p(j) ≤ ( j n )2 q, 1 ≤ j ≤ n}, (4) 3
• 4. We summarize some observations. • In Figure 2, the intersection (B) between H0 p-value proﬁle ( y = (x − π1)/π0) and linear cut route (y = xq) has location (x1,y1) with x1 = (1 − π0)/(1 − qπ0), the intersection (C) between H0 p-value proﬁle and quadratic cut route (y = x2 q) has location (x2,y2) with x2 = (1 − √ 1 − 4π0(1 − π0)q)/(2qπ0). • From Figure 3, when the probability of discovery= 1, FDR=pFDR (positive false discov- ery rate)= π0q (constant) no matter where the ordered-p-value proﬁle (Ha) crosses the no- rejection boundary. The no-rejection boundary function g(x) = qx/(1 − qπ0) (0 ≤ x ≤ π1, under linear cut) and g(x) = (1−2qπ0x)− √ 1−4qπ0x 2qπ2 0 (0 ≤ x ≤ π1, under quadratic cut). The relationship between instant FDR(=pFDR) and no-rejection boundary function (g(x)) is pFDR= π0g(x)/(x + π0g(x)) (0 ≤ x ≤ π1). • In Figure 4, at each q, the instant FDR(=pFDR) increases with the location (x ∈ (0, π1), the x-axis) where the ordered-p-value proﬁle (Ha) crosses the no-rejection boundary. When q=1, FDR= π0 for any cut routine (linear, quadratic). • In Figure 5, under linear cut, when probability of discovery is less than one (e.g., ordered- p-value proﬁles are close between H0 and Ha), pFDR>FDR and FDR= π0q. pFDR is less sensitive to q compared to FDR. This is relevant to the observation in Figure 1 (Groups 1 and 2). In Figure 5, under quadratic cut, the FDR is much less than that under linear cut case. When Ha mean proﬁles are close to zero, the pFDR is more volatile than linear cut case. • Under linear cut, the speciﬁcity approaches to (1−q)/(1−qπ0) as µ increases. Under quadratic cut, the speciﬁcity approaches to 1 π0 − 1−(1−4qπ0(1−π0))1/2 2qπ2 0 as µ increases. See Figures 5, 6, 7, 10, 11, 12. • As ordered-p-value proﬁle under Ha decreases (mean proﬁle increases), the numbers of discov- eries becomes very small. The number increases with Ha subject mean proﬁle. The expected number of discoveries under linear cut is higher than that under quadratic cut. The diﬀerence is larger as π0 gets larger. See Figures 8, 9. 4
• 5. • As n increases, the limiting speciﬁcity proﬁle approaches to the aforementioned calculated curve more closely. See the left panels in Figures 5, 10. • As π0 decreases, the limiting speciﬁcity proﬁle approaches to the aforementioned calculated curve more closely. See the left panels in Figures 10, 11. • The speciﬁcity under linear cut is lower than that under quadratic cut and the diﬀerence lessens as π0 decreases. The sensitivity under linear cut is higher than that under quadratic cut. The probability of discovery under linear cut is higher than that under quadratic cut. • We consider an unrealistic case where H0 ordered-p-value proﬁle is not random: {i/n0, i = 1, . . . , n0}. The FDR and pFDR is less than π0q when the Ha mean proﬁle is close to zero (Figure 13). • When σ (homogeneous among subjects) increases, the resultant cluster of proﬁles (collected from mean proﬁle set) behaves similarly to a sub-cluster of proﬁles (collected from mean proﬁle set with small values) with small σ (Figure 14 ). • When σ is heterogeneous across subjects (roughly independent of subject mean), the proba- bility of discovery tend to be larger (closer to one) than that under homogeneous σ case when the mean proﬁle is close to zero. The pFDR under heterogeneous σ is closer to FDR compared to the case with homogeneous σ. All other proﬁles (sensitivity, speciﬁcity) are similar between these two cases (homogeneous and heterogeneous σ) (Figure 15 ). • If all Ha p-values are ≤ p, we reject all p-values ≤ p. The false rejection rate ≤ π0q amounts to p ≤ π1q π1+(1−q)π0 (Figures 16, 17). 3 A note on p-value We numerically study the ranking of p-values through setting set size (n), Ha subject mean pro- ﬁle (µ) and noise variance (σ2 ) and others. Stochastic p-value rankings from both H0 and Ha 5
• 6. (Figures 18. Although the subject means are clearly ordered across domain [0, 1]) and the within- subject variation is moderate (=1) or minor (=1/100), the rankings of p-values are substantially ﬂuctuating around a trend. The degree of shuﬄing seems to be similar between two cases (σ =1 and 1/100). The p-values are individually calculated for each subject without considering the overall model structure (e.g., mean proﬁle function, homogeneous variation, etc.). Each p-value is associ- ated with a probability function, Pr(Tm−1 > √ m¯xm/ˆσm), where ¯xm and ˆσm are independent of each other. This pair of statistics (sample standard deviation, sample mean) is also used to estimate the population coeﬃcient of variation (σ/µ). The stochastic ˆσm has an substantial shuﬄing impact on the ranking of ¯xm. For instance, given another subject *, the comparison between √ m¯xm/ˆσm and √ m¯x∗ m/ˆσ∗ m may be confused by the stochastic relative magnitude between ˆσm and ˆσ∗ m. The distribution of estimate of coeﬃcient of variation is available (e.g., Hendricks and Robey (1936), Vangel (1996)). Even pairwise comparison between any two subject means is generally complicated under certain circumstances and numerical investigation is usually needed (e.g., Hsu (1938)). References [1] W.A. Hendricks, K.W. Robey (1936). The sampling distribution of the coeﬃcient of variation. The Annals of Mathematical Statistics 7(3): 129-132. [2] P.L. Hsu (1938). Contribution to the theory of “Student’s” t-test as applied to the problem of two samples. Statistical Research Memoirs 2: 1-24. [3] Y. Benjamini and Y. Hochberg (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society (B) 57: 289-300. [4] M.G. Vangel (1996). Conﬁdence intervals for a normal coeﬃcient of variation. The American Statistician 50(1): 21-26. 4 APPENDIX 6
• 7. 0.0 0.2 0.4 0.6 0.8 1.0 −0.10.10.30.5 Subject means Subject population Subjectmean (n=100 per group) Group 1 2 3 6 7 8 9 10 0.000.050.100.15 Rejection proportion m (replicates) Rejectionproportion (Group 1) 6 7 8 9 10 0.000.050.100.15 Rejection proportion m (replicates) Rejectionproportion (Group 2) 6 7 8 9 10 0.000.050.100.15 Rejection proportion m (replicates) Rejectionproportion (Group 3) Figure 1: The rejection proportion proﬁles arising from applying testing rule (1) (α = 0.10). Three groups (1,2,3) have subject mean (subject index i= 1, . . . , 100) proﬁles µi = 0, µi = 0.01(1 + sin10i n ) and µi = 0.10(1+sin10i n ), respectively (the top-left panel). The within-subject variation (σ)= 1. The tuning parameter (C) in rule (1) = 1+ j 20 (j = 1, . . . , 16) with resultant rejection proportion proﬁles (with m spanning from 6 to 10) located from top to bottom in each panel (top-right, bottom-left, bottom-right). 7
• 8. Geometry of false discovery rate control B(x1,y1) C(x2,y2) A D Specificity=AB/AD (linear cut) Specificity=AC/AD (quadratic cut) π1 π0 q p−value H0 Ha Figure 2: The bold dash line represents the ordered p-values from Ha with large positive means (π1 = 0.7). The bold dot line represents the ordered p-values from H0 (π0 = 0.3). The solid lines represent the linear and quadratic cut routes (x-axis is the ordered p-value index, y-axis is the threshold for H0 rejection). Under Benjamini-Hochberg (1995) FDR control procedure, speciﬁcity approaches to its limit as the alternative means increase. The intersection points between the linear and quadratic cut routes and H0 ordered p-value proﬁle are the ﬁnal p-value cut-oﬀ point for rejecting H0, which are labeled as B (location=(x1,y1)) and C (location=(x2,y2)), respectively. The speciﬁcities are calculated. 8
• 9. FDR control (linear cut) π1 π0 (NO REJECTION region) (Ha) q p−value FDR control (quadratic cut) π1 π0 (NO REJECTION region) (Ha) q p−value Figure 3: The left panel shows the geometry of Benjamini-Hochberg FDR control procedure (1995). The bold solid line represents the linear cut route (x-axis is the ordered p-value index, y-axis is the threshold for H0 rejection). The bold dot line represents the ordered p-value proﬁle under H0 (group size ∝ π0). The bold dash line represents the no-rejection region boundary for the ordered p-values from Ha (group size ∝ π1). In the horizontal direction, the distance between the bold dash and the solid lines equals the distance between the bold dot line and the point which separates the two regions labeled by “π1” and “π0”, respectively). The right panel shows the geometry of FDR control procedure under quadratic cut route. 9
• 11. 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Linear cut q Proportions (n0,n1)=(900,100) FDR pFDR Sensitivity Specificity Pr(discovery) 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Quadratic cut q Proportions (n0,n1)=(900,100) Figure 5: The FDR, pFDR, speciﬁcity and sensitivity proﬁles under linear and quadratic cut routes (n0 = 900 (H0),n1 = 100 (Ha)). σ = 1, m = 6 and Ha subject mean proﬁle = 0.08u(1 + |sin(6x)|u ), u = 1, . . . , 35. 11
• 12. 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Linear cut q Proportions (n0,n1)=(500,500) FDR pFDR Sensitivity Specificity Pr(discovery) 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Quadratic cut q Proportions (n0,n1)=(500,500) Figure 6: The FDR, pFDR, speciﬁcity and sensitivity proﬁles under linear and quadratic cut routes (n0 = 500 (H0),n1 = 500 (Ha)). σ = 1, m = 6 and Ha subject mean proﬁle = 0.08u(1 + |sin(6x)|u ), u = 1, . . . , 35. 12
• 13. 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Linear cut q Proportions (n0,n1)=(100,900) FDR pFDR Sensitivity Specificity Pr(discovery) 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Quadratic cut q Proportions (n0,n1)=(100,900) Figure 7: The FDR, pFDR, speciﬁcity and sensitivity proﬁles under linear and quadratic cut routes (n0 = 100 (H0),n1 = 900 (Ha)). σ = 1, m = 6 and Ha subject mean proﬁle = 0.08u(1 + |sin(6x)|u ), u = 1, . . . , 35. 13
• 14. 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 Linear cut q Numberofdiscoveries (n0,n1)=(90,10) 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 Quadratic cut q Numberofdiscoveries (n0,n1)=(90,10) Figure 8: The number of discoveries under linear and quadratic cut routes (n0 = 90 (H0),n1 = 10 (Ha)). σ = 1, m = 6 and Ha subject mean proﬁle = 0.08u(1 + |sin(6x)|u ), u = 1, . . . , 35. 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 Linear cut q Numberofdiscoveries (n0,n1)=(10,90) 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 Quadratic cut q Numberofdiscoveries (n0,n1)=(10,90) Figure 9: The number of discoveries under linear and quadratic cut routes (n0 = 10 (H0),n1 = 90 (Ha)). σ = 1, m = 6 and Ha subject mean proﬁle = 0.08u(1 + |sin(6x)|u ), u = 1, . . . , 35. 14
• 15. 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Linear cut q Proportions (n0,n1)=(90,10) FDR pFDR Sensitivity Specificity Pr(discovery) 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Quadratic cut q Proportions (n0,n1)=(90,10) Figure 10: The FDR, pFDR, speciﬁcity and sensitivity proﬁles under linear and quadratic cut routes (n0 = 90 (H0),n1 = 10 (Ha)). σ = 1, m = 6 and Ha subject mean proﬁle = 0.08u(1 + |sin(6x)|u ), u = 1, . . . , 35. 15
• 16. 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Linear cut q Proportions (n0,n1)=(50,50) FDR pFDR Sensitivity Specificity Pr(discovery) 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Quadratic cut q Proportions (n0,n1)=(50,50) Figure 11: The FDR, pFDR, speciﬁcity and sensitivity proﬁles under linear and quadratic cut routes (n0 = 50 (H0),n1 = 50 (Ha)). σ = 1, m = 6 and Ha subject mean proﬁle = 0.08u(1 + |sin(6x)|u ), u = 1, . . . , 35. 16
• 17. 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Linear cut q Proportions (n0,n1)=(10,90) FDR pFDR Sensitivity Specificity Pr(discovery) 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Quadratic cut q Proportions (n0,n1)=(10,90) Figure 12: The FDR, pFDR, speciﬁcity and sensitivity proﬁles under linear and quadratic cut routes (n0 = 10 (H0),n1 = 90 (Ha)). σ = 1, m = 6 and Ha subject mean proﬁle = 0.08u(1 + |sin(6x)|u ), u = 1, . . . , 35. 17
• 18. 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Linear cut (p non−random) q Proportions (n0,n1)=(900,100) 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Quadratic cut (p non−random) q Proportions (n0,n1)=(900,100) Figure 13: The FDR under linear and quadratic cut routes with ordered H0 p-values forming a non- random equal-partition of [0, 1]. σ = 1, m = 6 and Ha subject mean proﬁle = 0.08u(1 + |sin(6x)|u ), u = 1, . . . , 35. 18
• 19. 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Linear cut q Proportions (n0,n1)=(90,10), σ increased FDR pFDR Sensitivity Specificity Pr(discovery) 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Quadratic cut q Proportions (n0,n1)=(90,10), σ increased Figure 14: The FDR, pFDR, speciﬁcity and sensitivity proﬁles under linear and quadratic cut routes (n0 = 90 (H0),n1 = 10 (Ha)). σ = 10, m = 6 and Ha subject mean proﬁle = 0.08u(1 + |sin(6x)|u ), u = 1, . . . , 35. 19
• 20. 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Linear cut q Proportions (n0,n1)=(90,10), σ diverse FDR pFDR Sensitivity Specificity Pr(discovery) 0.0 0.2 0.4 0.6 0.8 1.0 0.00.20.40.60.81.0 Linear cut q Proportions (n0,n1)=(50,50), σ diverse FDR pFDR Sensitivity Specificity Pr(discovery) Figure 15: The FDR, pFDR, speciﬁcity and sensitivity proﬁles under linear and quadratic cut routes (n0 = 90 (H0),n1 = 10 (Ha)). σ is heterogeneous among subjects, m = 6 and Ha subject mean proﬁle = 0.08u(1 + |sin(6x)|u ), u = 1, . . . , 35. Subject variation= 2|cos(1000i)| (i = 1, . . . , n0) (under H0) and subject variation= 2|cos(1000i)| (i = 1, . . . , n1) (under Ha). 20
• 21. Histogram (class=5) p−value Frequency n0 n1 =5x 104 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 A H0 Ha p Histogram (class=5) p−value Frequency n0 n1 =5x 104 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 B H0 Ha p Figure 16: Histogram of p-values. σ = 1, m = 6 and Ha subject mean proﬁle = 0.08u(1+|sin(6x)|u ), u = 1, 5 (A,B). Histogram (class=5) p−value Frequency n0 n1 =5x 104 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 C H0 Ha p Histogram (class=5) p−value Frequency n0 n1 =5x 104 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 D H0 Ha p Figure 17: Histogram of p-values. σ = 1, m = 6 and Ha subject mean proﬁle = 0.08u(1+|sin(6x)|u ), u = 10, 20 (C,D). 21
• 22. 0 5 10 15 20 −0.20.00.20.40.60.81.0 Lag Autocorrelation Autocorrelation (rank residual) 0 5 10 15 20 −0.20.00.20.40.60.81.0 Lag Autocorrelation Autocorrelation (rank) 0.0 0.5 1.0 1.5 0.00.20.40.60.81.0 Rank (p−value) Subject(index=i/n) Rank(p−value) (f(i)=i/n,σ=1/100,m=6,n=100) Rank fit Rank Mean 0 5 10 15 20 25 30 0.00.20.40.60.81.0 Lag Autocorrelation Autocorrelation (rank residual) 0 5 10 15 20 25 30 0.00.20.40.60.81.0 Lag Autocorrelation Autocorrelation (rank) 0.0 0.5 1.0 1.5 0.00.20.40.60.81.0 Rank (p−value) Subject(index=i/n) Rank(p−value) (f(i)=i/n,σ=1/100,m=6,n=1000) Rank fit Rank Mean Figure 18: Rankings of p-values. Subject mean proﬁle is modeled as i/n (i = 1, . . . , n). σ = 1/100. 22