Cullen: Why most published research is wrong
•Télécharger en tant que PPT, PDF•
4 j'aime•1,541 vues
Question everything! Louise cullen examines the minefield of published research and importance of reading around topics, not articles.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Cullen: Why most published research is wrong
Similaire à Cullen: Why most published research is wrong (20)
Plus de SMACC Conference
Plus de SMACC Conference (20)
Dernier
Dernier (20)
Cullen: Why most published research is wrong
- 1. “Why most published research is wrong.” Louise Cullen (Clinician researcher)
- 3. “It is everyone’s responsibility to find out how to ask questions systematically, find answers from searching the literature, critically appraise the literature and apply the results to practice.” Rinaldo Bellomo
- 4. “It is everyone’s responsibility to find out how to ask questions systematically, find answers from searching the literature, critically appraise the literature and apply the results to practice.” Rinaldo Bellomo
- 7. 40 ingredients associate with cancer Most single studies showed implausibly large effects.
- 8. The p value
- 9. The p value Observed size of Effect
- 10. p=0.01
- 11. p=0.01 There is a 1% chance of results as extreme as these would occur when there is really no difference occurring in the experiment.
- 12. 1000 hypotheses
- 18. Problems with the study itself.
- 19. Wrong question
- 20. Wrong Theory
- 22. 2 ACS: Trial and community populations Circulation. 115(19):2549-69, 2007 May 15.
- 23. n=2
- 24. Wrong design
- 26. • Greater the flexibility in – designs – definitions – outcomes – analytical modes
- 27. • Greater the flexibility in – designs – definitions – outcomes – analytical modes • Hotter a scientific field with more teams involved.
- 28. Wrong Endpoints
- 29. Ad and high dose Ad Ca++ in cardiac arrest COX-2 inhibitors Milrinone
- 31. Methodology Statistical hypothesis inference testing
- 33. Interpretation
- 39. • “a little significance” • “a definite trend is evident” • “a clear tendency” • “almost achieved significance”
- 40. • “a little significance” • “a definite trend is evident” • “a clear tendency” • “almost achieved significance” The data is practically meaningless
- 42. • “In my experience” • “In case after case” • “In a series of cases” • “It is generally believed that..” • “A highly significant area for exploratory study” • Once • Twice • Three times • A couple of others think so too • A totally useless topic in my underpowered study…….
- 44. Why? Incentives
- 45. Pharma
- 46. Pharma
- 48. Why? Ethical practice of researchers
- 50. Don’t believe in the review process
- 52. • 2004 “original articles” in NEJM – 363 tested an established therapy – 146 (40%) reversed that practice – 138 (38%) reaffirmed it
- 53. What can you do about it?
- 54. Read more than the title!
- 55. Reporting Framework CONSORT (http://bit.ly/14qUNEF) – Standards for reporting of trials STARD – Standards for the Reporting of diagnostic accuracy studies
- 56. Biases
- 59. Be sceptical!
Notes de l'éditeur
- When Chris asked me to do this I thought – are you for real Chris? Why all published research is wrong….Do you really think I can cover all of this in 20mins?
- Sounds easy? Right? Great – but how…
- Sounds easy? Right? Great – but how…
- Where this all come from. Open access article by John in 2005. In here John – began the discussion about issues relating to problems with studies, analyses, their publication and reporting
- Ioannidis randomly selected 50 ingredients PubMed queries for recent studies that evaluated relationship with cancer.
- Effect size shrunk with meta-analyses.
- Ronald Fisher (a UK statistician) introduced the P value in the 1920s. He did not mean it to be a definitive test He intended it simply as an informal way to judge whether evidence was significant in the old-fashioned sense: i.e. that it was worth a second look. The idea was to run an experiment, then see if the results were consistent with what random chance may produce. Researchers would first set up a ‘null hypothesis” that they wanted to disprove – such as there being no correlation or no difference between 2 groups. Next hey would play devils advocate and, assuming that this null hypothesis was in fact true, calculate the chances of getting results at least as extreme as what was actually observed. This probability s the P value
- Most look at this and say that there is a 1% chance of the findings/results being wrong. But this is wrong. The P value cannot say this – you also need to know the odds that the real effect was there in the first place. The P value actually means that there is a 1% chance of results as extreme as these would occur when there is really no difference occurring in the experiment – e.g. that a drug has no effect.
- Most look at this and say that there is a 1% chance of the findings/results being wrong. But this is wrong. The P value cannot say this – you also need to know the odds that the real effect was there in the first place. The P value actually means that there is a 1% chance of results as extreme as these would occur when there is really no difference occurring in the experiment – e.g. that a drug has no effect.
- Consider 1000 hypotheses, of which only 10% are true.
- Random error makes a hypothesis that is really false – look true. These = FP. Medicine accepts that this happens in the order of 1 in 20 times, so in 1000 hypotheses (where 100 are TP) this means there are 45 FPs
- If there are 100 TP and 45 FP then almost a third of the results that look positive would be wrong.
- But it is worse than that. There is another type of error – and that is False negatives. Where there is a true effect, but it is misinterpreted as a false one. Say 20% of the true finding fail to be detected (and this figure is difficult to quantity). That is in this case - 20 cases. Now researches see 125 hypotheses as true (where 45 are not true)
- A growing number cannot be replicated, because many studies may have not found a real result in the first place. Perhaps we should only be looking at P values <0.005
- A growing number cannot be replicated, because many studies may have not found a real result in the first place. Perhaps we should only be looking at P values <0.005
- Start at the beginning: Studies themselves
- Interesting but not really relevant?
- Steroids for traumatic brain injury Others Medical example – Investigations into H2 blockers in PUD Before realised that PUD related to H bactor pylori
- Selective ACS – pts over 75 excluded from most studies Underpowered
- The National Registry of Myocardial Infarction (NRMI) Large, US observational registry Collects baseline data, procedural, therapeutic and outcome data on discharge >1million NSTEACs
- More lilkey that the findigns are false with Small size
- More lilkey that the findigns are false
- More lilkey that the findigns are false
- Not clinically relevant endpoints (Ad/Vasoproessin/ high dose Ad)
- Others – Milronone – Congestive heart failure – inc cardiac contractility – inc mortaility.
- Since publication in 1990, results from the National Acute Spinal Cord Injury Study II (NASCIS II) trial have changed the way patients suffering an acute spinal cord injury (SCI) are treated. though well-designed and well-executed, both NASCIS II and III failed to demonstrate improvement in primary outcome measures as a result of the administration of methylprednisolone. Post-hoc comparisons, although interesting, did not provide compelling data to establish a new standard of care in the treatment of patients with acute SCI. Evidence of the drug's efficacy and impact is weak and may only represent random events.
- Renamed
- In the late 1940s before there was a polio vaccine Health authorities noted that polio cases increased with ice cream and soft drink consumption. Eliminating such treats were part of the advice given to combat the spread of the diseases. Polio was more common in summer, when people eat more icecream Hence a Correlation vs causation
- Spin
- Incentives Egos, cudos, Academics – KPIs, output, promotions, grants Tied to remuneration – QH contracts - KPI
- Pharma – All bad 75% of all US research funded by Pharma
- Pharma – All bad 75% of all US research funded by Pharma Surgeon – cannot recommend surgery Interventional cardiologist - stenting
- UQ RORT: One of Australia's leading universities is investigating new concerns of possible academic misconduct by two former academics. The University of Queensland's Dr Caroline Barwood and Professor Bruce Murdoch published a peer-reviewed paper in the prestigious European Journal of Neurology, heralding a major breakthrough in the treatment of Parkinson's disease. theuniversity made the unusual admission that it could find no data or evidence that the research was ever conducted. Before the article was retracted, the study's apparent success led to a number of grants. Ten months after the allegations of academic misconduct were first raised and one month after the investigation was referred to the CMC, the university accepted part of a $300,000, five-year research fellowship on behalf of Dr Barwood.
- Incentives Egos, cudos, Academics – KPIs, output, promotions, grants Tied to remuneration – QH contracts - KPI
- Don’t believe open access – where if I pay the $ to many journal they will simply accept my paper!
- Neg trials hard to publish High impact journals – ‘1st’ of something – often exaggerated.
- Where we end up at is this>>> What are we doing now that is harmful to our patients?
- Don’t just skim article Not just abstract and conclusions Learn a little about stats Don’t be fooled by high quality journal Know/review the literature/topic. Not just article
- I don’t recommend that you go home tonight and try a “booty call” with your partner Perhaps read the article and find out the details.
- Consistent and transparent
- Be aware of your own biases – especially confirmation biases. Stop searching for information to confirm your own views. Read broadly. Don’t believe a single articles findings – look for bodies of work around a topic.
- Who do you believe and what do you believe? Do you believe who speaks loudest?
- Sit and contemplate your position…………Put the patient (not the patients leg) at the forefront of your focus. Be sceptical!