1. Leading opinion
Negative results: why do they need to be published?
Peter Sandercock
This short narrative review article defines ‘negative results’ There are many scientific reasons for publishing negative or
and cites several ethical and scientific reasons why such neutral (i.e. uninformative) studies; chiefly, they contain valu-
studies should be made publicly available. able information, which should become part of the scientific
record on the subject under study. Systematic reviews are an
Key words: clinical trial, methodology, publication bias, stroke
essential part of the research cycle (5). When scientists plan
units, systematic reviews, treatment
new research studies (clinical trials or observational studies in
humans or experiments on laboratory animals), the first step
What do I mean by ‘negative results’? The term applies to should be a systematic review of the evidence (5). Such a
studies conducted both in human and in animal subjects and review may reveal that the question has already been answered
encompasses three different types of result: reliably, or it may indicate that a further study is justified. For
• truly inconclusive with ‘no evidence of effect’, generally example, the UK Medical Research Council and the UK Health
because the study was too small and inadequately powered Technology Appraisal Programme require that new applica-
(several of the small studies included in the Cochrane system- tions for clinical trial funding should have performed (or at
atic review of stroke units are in this category) (1); least cite) an up-to-date systematic review of the subject to
• a well-conducted study, which is sufficiently large to provide ensure that the new research really is justified. If during the
‘clear evidence of no effect’, i.e. that any effect is too small to be course of a clinical study, a large negative or neutral study is
worthwhile pursuing either clinically or in further research published, this might require the trial steering committee or
(the Clots in Legs Or Stockings after Stroke (CLOTS) trial data monitoring committee to pause for thought and consider
of graded compression stocking for deep vein thrombosis whether the study should continue or be modified in some
prevention is a good example) (2); or way. At the end of any clinical study, the results should pref-
• clear evidence of harm when benefit had been expected. erably be presented in context of all the available evidence.
Unfortunately, many such ‘negative’ yet still important For clinical trials, this is a requirement of the Consolidated
studies in man (3) and in animals (4) remain unpublished. Standards of Reporting Trials (CONSORT) guidelines on
Why should negative studies be published? The most publication of randomized clinical trials (6).
important reason is ethics. If human subjects have given Systematic reviews can put a small but strikingly positive
consent to participate in a clinical research study, be it a treat- study into context. For example, when a strikingly positive
ment trial, or an observational study, they have done so in the small study is viewed within the totality of the evidence avail-
clear understanding that the research results will in some way able, it becomes evident that it is a freak ‘lucky’ result arising
be of benefit to other people and contribute to scientific from the play of chance, and not a reliable estimate of the true
advance. Furthermore, these human subjects have exposed effect (4,7). The availability of the ‘negative’ studies then
themselves to risk and inconvenience by participating in the ensures that the one small positive (but outlying) study is
study, and the justification for doing that ‘good deed’ should interpreted appropriately. There are numerous examples from
be that the author makes the data publicly available and the literature where small clinical trials or small studies of
ensures that it is put to good use. Although animals do not give genetic associations produce striking positive results (often
consent to participate in research, we still have an ethical duty leading to a high-profile paper in a major journal), which are
to make the best use of the data from any animals used in such then not subsequently replicated when larger, more reliable
research. (neutral or negative) studies are published (8–10).
To be reliable, systematic reviews need to include all rel-
Correspondence: Peter Sandercock, Division of Clinical Neurosciences, evant randomized trials. The Stroke Unit Trialists’ Collabora-
University of Edinburgh, Western General Hospital, Bramwell Dott tive systematic review of organized inpatient stroke care (1)
Building, Edinburgh EH4 2XU, UK.
Email: peter.sandercock@ed.ac.uk
very clearly demonstrates the importance of making negative
Twitter: @IST_3 trial results available for inclusion in systematic reviews. Four-
teen of the 16 trials comparing a stroke unit with general
Conflict of interest: None declared.
medical ward care were ‘negative’ for the effect on death (they
DOI: 10.1111/j.1747-4949.2011.00723.x were all underpowered to detect a moderate but clinically
© 2011 The Author.
32 International Journal of Stroke © 2011 World Stroke Organization Vol 7, January 2012, 32–33

2. P. Sandercock Leading opinion
important difference), and four were never published. of Edinburgh Datashare site is an example http://datashare.
However, the overall estimate of the effect from all the trials is.ed.ac.uk/).
together is that stroke unit care reduces the odds of death by In summary, if a study involves informed consent in
17% (95% confidence interval 4–29%) (P = 0·01), a result humans or the use of whole animals, and has been satisfacto-
which has helped support the introduction of stroke unit care rily conducted, it should appear in the publicly available sci-
into practice worldwide. What would have happened if all of entific record, irrespective of its overall conclusions.
the apparently ‘negative’ studies had remained unpublished?
We might well have ‘lost’ one of the most significantly benefi-
cial interventions for stroke! Acknowledgements
There are many obstacles to publication of ‘neutral or nega- I would like to thank William Whiteley, Charles Warlow, and
tive’ studies; the first is the author. A study that does not have Norberto Cabral for helpful comments.
a big fat juicy P-value does not stir the author’s adrenalin
sufficiently to write the paper in the first place. It may not be
in the commercial interest of the sponsors of negative research References
for the results to be published. Journals like clear stories with
1 Stroke Unit Trialists’ Collaboration. Organised inpatient (stroke unit)
definite results to increase their sales. Negative studies do not
care for stroke. Cochrane Database of Systematic Reviews 2007; Issue 4.
do much for journal income! There have been reports that Art. No.: CD000197. DOI: 10.1002/14651858.CD000197.pub2.
some journals would only publish results from studies that are 2 Dennis M, Sandercock PA, Reid J et al. Effectiveness of thigh-length
statistically significant at the P < 0·05 level. This is clearly graduated compression stockings to reduce the risk of deep vein
absurd, but I suspect that the practice still continues (perhaps thrombosis after stroke (CLOTS trial 1): a multicentre, randomised
controlled trial. Lancet 2009; 373:1958–65.
rather more covertly these days). In the fast-moving world of
3 Gibson L, Brazzelli M, Thomas B, Sandercock P. A systematic review
the Internet and social media, where only the most strikingly of clinical trials of pharmacological interventions for acute ischaemic
positive results gain their ‘15 minutes of fame’, it is difficult to stroke (1955–2008) that were completed, but not published in full.
ensure that neutral or negative results from well-conducted Trials 2010; 11:43. http://www.trialsjournal.com/content/11/1/43
research would reach the public domain. 4 Sena ES, van der Worp HB, Bath PMW, Howells DW, Macleod MR.
Publication bias in reports of animal stroke studies leads to major
There are a number of possible solutions. Journal editors
overstatement of efficacy. PLoS Biol 2010; 8:e1000344.
could (perhaps) be persuaded to prioritize publishing well- 5 Bath PM, Gray LJ. Systematic reviews as a tool for planning and
conducted negative research over poorly conducted positive interpreting trials. Int J Stroke 2009; 4:23–7.
research. There are easier alternatives: although the Journal 6 Moher D, Hopewell S, Schulz KF et al. CONSORT 2010 explanation
of Negative Results in Biomedicine (http://www.jnrbm.com/) and elaboration: updated guidelines for reporting parallel group ran-
domised trials. BMJ 2010; 340:c869.
provides a rather specific destination, there are now many
7 Collins R, MacMahon S. Reliable assessment of the effects of treatment
open-access publishing journals. Furthermore, several grant- on mortality and major morbidity, I: clinical trials. [Review] [74 refs].
giving bodies, such as the UK Medical Research Council and Lancet 2001; 357:373–80.
the Wellcome Trust, expect data from research they have 8 Ioannidis JPA. Contradicted and initially stronger effects in highly
funded to be published in an open-access format (applications cited clinical research. JAMA 2005; 294:218–28.
9 Ioannidis JP. Why most discovered true associations are inflated.
for research grants now need to incorporate anticipated costs
Epidemiology 2008; 19:640–8.
for such publications). For authors without such funding, 10 Ioannidis JPA, Panagiotou OA. Comparison of effect sizes associated
many institutions now make datasets accumulated by their with biomarkers reported in highly cited individual articles and in
scientists freely available via open data repositories (University subsequent meta-analyses. JAMA 2011; 305:2200–10.
© 2011 The Author.
International Journal of Stroke © 2011 World Stroke Organization Vol 7, January 2012, 32–33 33