On a solution to selective estimation using post selection confidence intervals.

1. Selective Correlations-
NotVoodoo
Malach Lab 8.5.2014
Jonathan Rosenblatt- WIS

2. Acknowledgments
• Part of PhD dissertation under guidance of
Prof.Yoav Benjamini
• Prof. Russ Poldrack
• Ms. Neomi Singer, Mr. Omri Perez, Prof.
Talma Hendler

3. Outline
• Tom et al.,
“The Neural Basis of Loss Aversion in
Decision-Making Under Risk.”
Science 315 (January 26, 2007)
• Selection Bias- Problem & Remedy.
• Revisiting Tom et al.
• Discussion.

4. Tom et al. (2007)

5. Reported Correlations

6. EnterVul
• Vul, Edward, Christine Harris, Piotr
Winkielman, and Harold Pashler.
“Puzzlingly High Correlations in fMRI
Studies of Emotion, Personality, and
Social Cognition.” Perspectives on
Psychological Science 4, no. 3 (May 1, 2009):
274–290.

7. And The People Rejoice
• Diener, Ed.“Editor’s Introduction toVul et Al. (2009) and Comments.” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 272–273.
• Fiedler, Klaus.“Voodoo Correlations Are Everywhere—Not Only in Neuroscience.” Perspectives on Psychological Science 6, no. 2 (March 1, 2011)
• Jabbi et al.“Response to ‘Voodoo Correlations in Social Neuroscience’ byVul et Al.–summary Information for the Press.” Accessed July 30, 2013.
• Kriegeskorte et al.“EverythingYou Never Wanted to Know about Circular Analysis, but Were Afraid to Ask.” Journal of Cerebral Blood Flow & Metabolism 30, no. 9
(September 2010): 1551–1557.
• Lazar, Nicole A.“Discussion of ‘Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition’ byVul et Al. (2009).” Perspectives on
Psychological Science 4, no. 3 (May 1, 2009): 308–309.
• Lieberman, Matthew D., Elliot T. Berkman, and Tor D.Wager.“Correlations in Social Neuroscience Aren’tVoodoo: Commentary onVul et Al. (2009).” Perspectives on
Psychological Science 4, no. 3 (May 1, 2009): 299–307. doi:10.1111/j.1745-6924.2009.01128.x.
• Lindquist, Martin A., and Andrew Gelman.“Correlations and Multiple Comparisons in Functional Imaging:A Statistical Perspective (Commentary onVul et Al., 2009).”
Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 310–313. doi:10.1111/j.1745-6924.2009.01130.x.
• Nichols,Thomas E., and Jean-Baptist Poline.“Commentary onVul et Al.’s (2009) ‘Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social
Cognition.’” Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 291–293. doi:10.1111/j.1745-6924.2009.01126.x.
• Poldrack, Russell A., and Jeanette A. Mumford.“Independence in ROI Analysis:Where Is theVoodoo?” Social Cognitive and Affective Neuroscience 4, no. 2 (June 1,
2009): 208–213. doi:10.1093/scan/nsp011.
• Vul, Edward, Christine Harris, Piotr Winkielman, and Harold Pashler.“Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition.”
Perspectives on Psychological Science 4, no. 3 (May 1, 2009): 274–290. doi:10.1111/j.1745-6924.2009.01125.x.
• “Reply to Comments on ‘Puzzlingly High Correlations in fMRI Studies of Emotion, Personality, and Social Cognition.’” Perspectives on Psychological Science 4, no. 3
(May 1, 2009): 319–324. doi:10.1111/j.1745-6924.2009.01132.x.
• Vul, Edward, and Nancy Kanwisher.“Begging the Question:The Non-Independence Error in fMRI Data Analysis.” Foundational Issues for Human Brain Mapping
(2010): 71–91.
• Yarkoni,Tal.“Big Correlations in Little Studies: Inflated fMRI Correlations Reflect Low Statistical Power—Commentary onVul et Al. (2009).” Perspectives on
Psychological Science 4, no. 3 (May 1, 2009): 294–298. doi:10.1111/j.1745-6924.2009.01127.x.

8. The Usual Suspects
• Multiplicity control
• Small samples => underpowered
• Reporting standards

9. n.subjects: 12 n.subjects: 29 n.subjects: 47 n.subjects: 64 n.subjects: 82 n.subjects: 100
●
●
● ● ● ● ●
●
● ● ● ● ● ● ●
●
● ● ● ● ● ● ●
●
● ● ● ●
●
●
●
●
● ● ● ● ●
●
●
●
●
●
● ● ● ● ●
●
●
●
● ●
● ●
●
●
●
●
●
● ● ●
●
●
●
●
●
● ● ● ● ●
●
●
●
●
● ●
●
●
●
●
●
●
● ● ● ●
●
●
●
●
●
● ● ● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
●
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
0.4
0.6
0.8
1.0
resels:5000resels:10000resels:3e+05
0.000.250.500.751.000.000.250.500.751.000.000.250.500.751.000.000.250.500.751.000.000.250.500.751.000.000.250.500.751.00
True Correlation
MeanofSelectedCorrelations
Selection Bias

10. Cureton, Edward (1950)
• When a validity coefficient is computed
from the same data used in making an item
analysis, this coefficient cannot be
interpreted uncritically.And, contrary to
many statements in the literature, it cannot
be interpreted “with caution” either.There
is one clear interpretation for all such
validity coefficients.This interpretation is–
“Baloney”

11. Selective Estimation
• A.k.a.“circular inference”,“double dipping”,
“voodoo correlations”,...
• Estimation with quality guarantees following
a parameter selection stage.
• Relation to selective testing.

12. Ingredients of Estimation
• Point/interval?
• Error criterion?
• Algorithms with uniform error bounds?

13. False Coverage statement Rate
• An error criterion for selective interval
estimation.
• R= selected parameters
• V= false coverage statements
• FCP=V/R and 0 if none are selected.
• FCR=E(FCP)

14. FCR of Nominal CIs
• Parameters selected: R=3 (big dots)
• Parameters not covered:V=2 (grey bars)
• FCP=V/R= 0.66
• Desired FCR= 1-confidence level= 0.1
• Nominal CIs do not control the FCR.
● ●
●
● ● ●
● ● ●
●
● ● ● ●
●
●
●
● ●
●
1 2 3 4 5 6 7 8 9 10 12 14 16 18 20
0
● ● ●● ● ●

15. FCR Adjusted CIs
• Benjamini,Yekutieli (2005)
• Motivation: conservative nominal CIs.
• In practice:

16. Conditional CIs
• Weinstein, Fithian, Benjamini (2013)
• Motivation: invert acceptance region of
conditional distribution.
• In practice: selectiveCI R package.

17. (non)Uniqueness of Acceptance Region
• Conditional Shortest Length (CSR):
Short interval, but indifferent to sign
ambiguity.
• Conditional Modified Pratt (CMP):
Best sign determination, while no larger
than r times the CSR.
• Conditional Quasi Conventional (CQC):
Shortest interval with penalty for sign flip

18. Properties

19. Remarks
• “Simple”: varying the value of a selected i’th
estimator in its selectable range, does not
change R.
• FCR adjusted CIs are B-H selection duals.
Duality does not hold in general.
• Some selection rules are very hard to
condition on.

20. Applying FCR Adjusted CIs
“Confidence Calibration Plot”

21. Applying CQC CIs
“Confidence Calibration Plot”

22. Conditional or FCR Adjusted?

23. “If the functional contrast is
demonstrably independent of the
effects to be estimated for the
selected data, then the same data
may be used for effect estimation.
Otherwise, independent data are
required to render the effect
estimate independent.”
Kriegeskorte (2013)

24. ●
●●●●●●●●
● ●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●
0.00
0.25
0.50
0.75
1.00
0.00 0.25 0.50 0.75 1.00
FullAndSplit
SplitAndFull2
Split.count
●
●
●
●
●
0
10000
20000
30000
40000
Spatial Agreement of a Half Split

25. CI Agreement with Half Split

26. CI Agreement with Half Split
• In less than 0.1% of voxels, splitting will
provide strong sign determination and
CQC will not.
• In more than 50% of possible splits, 1/3 of
jointly selected voxels will have opposing
strong sign determination.
• Conclusion: CQC has higher probability of
catching the right effect sign.

27. Summary
• Selective estimation in social-neuroscience:
Acknowledged but untreated.
• FCR controlling CIs as a general remedy.
• Better than splitting small samples.

28. Open Problems
• Conditioning
• Inference on aggregates
• Other Error measures

29. Thank you
<jonathan.rosenblatt@weizmann.ac.il>