Workshop for Family Medicine Residents at the University of Calgary on Evidence-Based Medicine, the PICO approach to critical appraisal, and the need for skepticism

1. From PICO to EBM: Skeptical about Evidence

2. Looking for Evidence:
What do I look for? P-I-C-O (P-E-C-O)
• Taxonomy used in EBM to help formulate questions
and translate clinical problem into structured question
and identify key concepts
• Elements:
Patient: Which patient population?
Intervention: What intervention or evidence is
involved?
Comparison: What was the comparator?
Outcome: What are the outcomes in the study?

3. Looking for Evidence:
What field does my question fall into?
• Therapy/Prevention – drug interventions, alt.
methods of service delivery, etc?
• Diagnosis/Screening – new test valid and
reliable versus gold standard?
• Etiology/Cause/Harm – exposure to harmful agent
related to development of illness or disease?
• Prognosis – what happens if disease is
caught at an early stage?

4. Finding stuff
Where do I look?
• Ovid Medline / Pubmed
• Cochrane Library: Cochrane reviews, DARE reviews
• ACP PIER (via StatRef) shows specific evidence sources for its
recommendations
• Clinical Evidence (synthesized evidence—check references)
• ACP Journal Club (or a relevant Evidence-based digest, such as Evidence-
based Child Health, Evidence-based Eye Care, etc.)
• Clinical Practice Guidelines— current, from reputable source, with
good levels of evidence e.g. Alberta Towards Optimized Practice (TOP)
• Quick & dirty: Google Scholar
• Are your search skills better than Google’s?

5. Anatomy of a Research Paper
• Abstract
• Introduction
• Methods
• Results
• Discussion

6. Hierarchy of Evidence
Adapted from Will Olmstadt, Evidence-Based Medicine: The Basics (https://becker.wustl.edu/sites/default/files/EBM_Std.pdf)
Secondary
research
Prim ary/
original
researc
h

7. Appraising a Research Paper

8. Are the results valid?
• Is the research question focused?
Patient
Intervention
Comparison
Outcome
• Was the method used appropriate?
• How was it conducted?
Appraising a research paper

9. What are the results?
• How was data collected and analyzed?
• Are results significant?
• How precise are the results?
• How are they presented?
Appraising a research paper

10. Are results relevant?
• Will the results help my work with patients?
• Can I apply these results to my own practice?
• Is my local setting different?
• Are these findings applicable to my patients?
• Are findings detailed enough to be applied?
• Were all outcomes considered?
Appraising a research paper

11. Role of Statistics
• “… Critical appraisal should not… be regarded as just another
aspect of medical statistics. This misconception often leads to an
overestimation of the level of statistical knowledge required for
critical appraisal.”
(Ajetunmobi, 2002)
• However, for quantitative papers, a little
statistical knowledge can help a lot!
Appraising a research paper

12. Is it significant?

13. Disclosure:
The effect size of my statistical knowledge is weak

14. Not good with numbers?

15. Skepticism & realism ≠ cynicism & pessimism
Robert Fulghum
Be skeptical

16. Important concepts
Pre-test probability
• Lots of examples out there
• Centre for EBM
• http://www.cebm.net/pre-test-probability/
• http://www.cebm.net/likelihood-ratios/

17. Important concepts
Principles of a screening test
• PSA testing
• Much attention to test accuracy
• http://www.bmj.com/rapid-
response/2011/10/31/multi-level-likelihood-ratios-
and-post-test-probabilities-prostate-cancer-
• …but is early detection a benefit?

18. How good is the evidence overall?
• Ioannidis JPA. Why most published research
findings are false.
• PLoS Med. Public Library of Science;
2005;2(8):e124
• Most cited paper on PLoS
• https://www.youtube.com/watch?v=42QuXLu
cH3Q
• Same points in easy to digest video
• p < 0.05 so how many are wrong?

19. How good is the evidence overall?
• Are RCTs the ultimate?
• Ioannidis JP, Haidich AB, Lau J. Any casualties
in the clash of randomised and observational
evidence?
• BMJ. England; 2001. p. 879–80
• Observational studies are refuted less often

20. Who me? Biased?
• Pat Croskerry on Cognitive Bias
• Zwaan et al, 2016
• Is bias in the eye of the beholder?

21. Correlation vs causation
TED talk by Kenneth Cukier
www.ted.com/talks/kenneth_cukier_big_data_is_
better_data

22. More info:
David Topps – topps@ucalgary.ca
Additional resources:
• OLab workbook
• http://demo.openlabyrinth.ca/renderLabyrinth/index/1081
• > 7.61* people email me to post resources
• 61.1% of statistics are made up on the spot