Presented 11/2/13 at National Association of Science Writers 2013 in Gainesville, Florida, for the panel "Rising Above the Noise: Using Statistics-Based Reporting." Tips for journalists.

3. How many relationships did you test?
What is your overall false-positive rate?
How did you control for multiple comparisons?
Are these the unadjusted p-values?
Thought question: How do we provide context to our readers? With all the
false-positive studies and non-replicated findings, they’re experiencing
whiplash and losing trust in the media.

5. P-hacking: When researchers desperately (and perhaps unconsciously)
beat the data until they yield p < 0.05.
Overfitting and Double-Dipping: How robust are your findings? How sensitive are
they to these conditions? Were they tested in an independent sample?
Fishing: Did you determine your sample size in advance? Or did you keep
adding new subjects until you got a significant result?
Mining: What other variables did you test that you didn’t report here?
Cherry picking: How did you select these particular subsamples for analysis?
Just-so stories: What is the proposed mechanism here? What led you to test this
relationship in the first place? Would you have been surprised if the results had gone
in the opposite direction?

7. Were there any nonlinear effects?
Were there any interaction effects?
Did you find anything surprising that didn’t make it
into the final paper?
Thought question: Raw data are increasingly
available. Could we use that data to develop our
own graphics and summaries?

9. Check out:
ONA (Online News Association): journalists.org
IRE (Investigative Reporters and Editors) and NICAR (National
Institute for Computer-Assisted Reporting): ire.org
Knight Lab, especially Timeline.js: projects.knightlab.com
My fun xkcd timeline!
Tableau Online software (which makes
awesome online interactive graphics -- no stats or
programming experience necessary):
tableausoftware.com

11. What is the effect size?
In what ways might these subjects not be representative of the
population?
Is there any reason to think there’s a causal factor here?
It’s statistically significant, but is it practically
significant? If your results were adopted, how would the world
change? If everyone got that drug/vitamin/intervention, how many
more deaths/colds/divorces would we prevent?
Thought question: Journals are tightening up their standards, but
poor analyses -- and false results -- are still legion. Should we trust
peer review, even for good journals? How do we decide what
studies to report on?