An introduction to conducting a systematic literature review for social scientists and health researchers presented by Luke van Rhoon Health Behaviour Change Research Group, School of Psychology, NUI Galway November 2020
An introduction to conducting a systematic literature review for social scientists and health researchers
1. An introduction to conducting a systematic
literature review for social scientists and
health researchers
Luke Van Rhoon
Health Behaviour Change Research Group,
School of Psychology, NUIG
4th Nov 2020 10:00am -12:30pm
5. Learning Outcomes
By the end of this session you should be able to:
• Describe the key features of a systematic review
• Outline the pros and cons of systematic reviews
• Develop a systematic review research question
• List the steps in carrying out a systematic review
• Choose an appropriate method to synthesise results
Quantitative Meta-analysis
Narrative Review
Qualitative Meta-synthesis
What is a systematic
review?
How to conduct a
systematic review?
How to synthesise and
report the results of a
systematic review?
Session Overview
7. 1. A type of literature review
- Google Search
- Papers recommended by colleagues
- Papers discussed in class/seminars
- Snowballing
How do you normally identify literature in
your area?
9. 2. Differs from a traditional literature review
Kysh, Lynn (2013): Difference between a systematic review and a literature review. figshare.
https://doi.org/10.6084/m9.figshare.766364.v1 Retrieved: 13:36, Dec 05, 2017 (GMT)
10. 3. The importance of method
• A systematic review uses explicit and replicable methods to identify,
critically appraise, and synthesize relevant studies
• These methods are reported so that other researchers could use
these methods to identify the same studies
11. Can you differentiate a systematic from a non-systematic review
just from the abstract?
3. The importance of method
12. The current paper will review existing evidence on PA rates, correlates,
consequences, and interventions in older adults with MS, and then
highlight important areas for future research on PA and health aging with
MS.
13. The purpose of this systematic review and meta-analysis is to evaluate
the evidence of the modifiable, individual level psychosocial
constructs associated with PA participation in people with MS..
14. 4. Related but different terms
Meta-Analysis: A method of synthesising the results of the
studies included in a systematic review
15.
16. 4. Related but different terms
Meta-Analysis: A means of synthesising the results of a
systematic review
Cochrane Review: A type of systematic review following the
guidelines of the Cochrane collaboration
17.
18. Systematic Reviews: Pros and Cons
Rigorous, unbiased and reliable
Informative as summarise large body of research
Aid decision making, policy making, clinical decisions
19. • Working together to provide the best
evidence for health care
• Archie Cochrane (1909-1988), a British
epidemiologist, who advocated the use of
RCTs to reliably inform healthcare practice
• Cochrane Handbook for Systematic
Reviews of Interventions
• Gold standard
20.
21. Systematic Reviews: Pros and Cons
Rigorous, unbiased and reliable
Informative as summarise large body of research
Aid decision making, policy making, clinical decisions
Provide an answer only to one specific question
Results may be disappointing
“More research needed in the area"
22. More systematic theory based research is required to understand
the precipitating and perpetuating factors associated with exercise
dependence, as well as effective treatment regimes. 22
23. Conducting a Systematic Review: Pros and Cons
Highly cited
Requirement of professional qualifications
(e.g. Stage 2 Professional Health Psychology Training)
Required for funding applications
Develops skills in searching for and working with existing literature
Useful skill for your CV
Time-consuming
Requires a lot of reading
24. Time to complete and publish review: 67.3 weeks
Number of reviewers: 5
Number of studies found: 27 to 92 020
Funded reviews take longer to complete and publish
and involve more authors
Conclusions: Systematic reviews take much time and
require large amounts of human resources. Existing
computing and informatics technology should be applied to
decrease this time and resource burden.
Top Tip 1: Be Realistic
25. Rapid reviews
Rapid reviews – provide timely and cost-effective evidence
Urgent evidence needed to inform policy (e.g. Ebola crisis,
humanitarian disasters etc)
Spectrum of methods to expedite reviews:
Searching:
Focused, specific topics, specific outcomes
Limit searches to English
Limited grey literature searching/author contact
Screening/extraction/quality appraisal:
Single reviewer with subset verified by second reviewer
Synthesis:
Narrative > meta-analysis
Limitations acknowledged, cautious conclusions
Tricco et al 2015; Tricco et al 2017
28. Carrying out a Systematic Review: 8 Steps
1. Define review question
2. Develop inclusion/exclusion criteria
3. Plan search terms/strategy
4. Conduct search
5. Apply inclusion/exclusion criteria to assess relevance
6. Review included studies
Extract data and assess risk of bias/quality of studies
7. Synthesise included studies
Meta-analysis/qualitative meta-synthesis/narrative review
8. Write report and publish
How to conduct a
systematic review
How to synthesise and
report the results of a
systematic review
29. Systematic Review Protocol
The plan or set of steps to be followed in a systematic review
Includes:
Rationale for the review
Research question
Methods to locate, select, and critically appraise studies
Methods to collect data from included studies
Methods to synthesise data from included studies
30.
31. Systematic Review Protocol
The plan or set of steps to be followed in a systematic review
Includes:
Rationale for the review
Research question
Methods to locate, select, and critically appraise studies
Methods to collect data from included studies
Methods to synthesise data from included studies
Why register a
systematic review
protocol?
35. 1. Define the Review Question
• Conduct scoping review
• Define terms precisely
• Choose a narrow and well specified question
• Specific review criteria:
Population Intervention Comparison Outcome (PICO)
Sample, Phenomenon of Interest, Design, Evaluation, Research type
(SPIDER)
36. Scope
Technology-driven Type 2 Diabetes Prevention Interventions (DPIs)
Define terms
Technology-driven, Type 2 Diabetes, Behaviour Change Techniques, Digital
Features
Narrow and well specified question
To determine the effectiveness of technology-driven DPIs in producing clinically
significant weight loss and improvements in additional outcomes linked to the
onset of T2D; and
Identify the BCTs and digital features most frequently used in effective
interventions
Specific review criteria
P: Adults (≥18 years) at risk of developing T2D (define criteria)
I: Active Intervention arm
C: Usual Care, non-digital DPI (single-arm studies were included in this review)
O: Weight loss, fasting glucose, glycated haemoglbin (A1c)
36
39. 1. Define the Review Question
Top Tip 3: Be Specific
Top Tip 4: Know the Literature
40. 2. Develop Inclusion and Exclusion Criteria
List of criteria to set out the papers to be included and excluded
41. 3. Plan Search Strategy
Identifying papers for a systematic review
• Electronic databases
Platforms: EBSCO, OVID (search interface and features)
Databases: CINAHL, EMBASE, Medline, PsycInfo
42. Name Summary Coverage Comments
CINAHL Cumulative Index to Nursing and
Allied Health Literature
1937
Nursing, biomedicine, health sciences librarianship,
alternative/complementary medicine, consumer health and other
allied health disciplines.
Health care books, nursing dissertations, selected conference
proceedings, standards of practice, educational software,
audiovisual materials and Evidence-Based Care Sheets
English Language
EMBASE Excerpta Medica dataBASE
Biomedical and pharmacological
database
European focus
1947
Drug research, pharmacology, pharmacy, pharmacoeconomics,
pharmaceutics and toxicology, clinical and experimental human
medicine, basic biological research, health policy and management,
public, occcupational and environmental health, substance
dependence and abuse, psychiatry, forensic science and biomedical
engineering and instrumentation
Selective Coverage: nursing, dentistry, veterinary medicine,
psychology and alternative medicine.
Approx 30%
duplication with
Medline
Medline Biomedical database produced
by the US National Library of
Medicine
US Focus
1946
Biomedicine and life sciences, bioengineering, public health, clinical
care and plant and animal science.
3,000 medical, dental and nursing journals
PsycINFO Psychological database produced
by the American Psychological
Society
1597
Comprehensive coverage from
the 1880s
All areas of psychology: applied, experimental, developmental,
educational, behavioural and animal psychology and psychological
disorders. Relevant items from related disciplines (eg. sociology,
linguistics, education, law, physiology, business, psychiatry and
anthropology) and dissertations are also included
Publications from more than 50 countries and 29 languages
43. 3. Plan Search Strategy
Identifying papers for a systematic review
• Electronic databases
Platforms: EBSCO, OVID (search interface and features)
Databases: CINAHL, EMBASE, Medline, PsycInfo
Meet with Academic Librarian for advice
• Hand search key journals
E.g. Past five years of journals in your area
• Forward and backward citation searches
• Unpublished or “Grey Literature”
E.g. conference abstracts, contact authors in the field
44. 3. Plan Search Strategy
More comprehensive review
Reduces risk of publication bias
Unpublished data difficult to source
Located studies may be unrepresentative
Lower quality in unpublished studies
Unpublished studies include or exclude?
File drawer problem:
Published trials showed
a 9% larger intervention
effect than grey trials
(Hopewell et al., 2007)
Support for inclusion of
grey literature in
systematic reviews:
Review Authors: 86%
Journal Editors: 69%
(from Cochrane Handbook)
46. 3. Plan Search Terms
Top Tip 5: Learn from Existing Reviews Top Tip 6: Use Available Expertise
47. 3. Plan Search Terms
• Use terms that will accurately identify relevant material
• Medical Subject Heading (MeSH terms)
National Library of Medicine's controlled vocabulary thesaurus
• Combine terms; AND, NOT, OR, NEAR, ADJ
• Test out in electronic databases
Modify and refine terms
48. 3. Plan Search Terms
Downloaded from: https://www.linkedin.com/pulse/boolean-logic-marketers-explained-part-1-
fab-capodicasa
49.
50. 4. Conduct the Search
• 100s or 1000s of hits
• Bibliographic Software to manage references
• Record where references were identified
51.
52. 5. Apply Inclusion and Exclusion Criteria: Screening
Screen
titles
Screen
abstracts
Obtain
full
article
Screen
full
article
53. 5. Apply Inclusion and Exclusion Criteria: Screening
• Keep good records and a paper-trail of inclusion/exclusion and reasons
Create inclusion/exclusion table/bibliographic database folders
Create inclusion/exclusion flow diagram
• Assessment of eligibility of studies should be done by at least two people
independently
Second reviewer for a percentage if all not feasible
Calculate the kappa statistic to measure agreement
0.40 - 0.59 is fair
0.60 - 0.74 is good,
> 0.75 is excellent (Orwin 1994).
Top Tip 7: Work in a Team
54.
55.
56.
57.
58. Top Tip 8: Be Organised
Articles Included
37
Studies Included
19
62. 6. Review Studies - Extract Study Data
• Systematically record data from each paper
• Record key information relevant to answering your review question
• Create a table to ensure info is accurately and consistently extracted
• Extraction of data should be done by at least two people independently
67. 6. Review Studies – Critical Appraisal
• Quality check-list and scales exist with inclusion cut-off scores
e.g. Downs and Black checklist for non randomised studies
• Recent expert groups have discouraged the use of summary
quality scales (The Cochrane Collaboration, 2009)
Due to the difficulties in assigning weights to items
Limited validity of these measures
• Recommend not to exclude studies based on critical appraisal
but provide information for discussion of the confidence in the
results
76. Carrying out a Systematic Review: 8 Steps
1. Define review question
2. Develop inclusion/exclusion criteria
3. Plan search terms/strategy
4. Conduct search
5. Apply inclusion/exclusion criteria to assess relevance
6. Review included studies
Extract data and assess risk of bias/quality of studies
7. Synthesise included studies
Meta-analysis/narrative review/qualitative meta-synthesis
8. Write report and publish
How to synthesise and
report the results of a
systematic review?
77. 7. Synthesise Included Studies
Quantitative Meta-Analysis
• Statistical approach to integrate results across studies by obtaining
an average effect size
• “The analysis of the analyses”
• Often weighted by sample size
Effect sizes based on larger sample sizes receive
greater weight than those from smaller samples
• Also used to:
Estimate the variability between effect sizes across studies
Identify moderator variables
Limited to similar studies:
“Because of their limited number and varied
designs, populations and interventions, meta-
analytic techniques were deemed inappropriate.
Instead, randomized controlled trials are
discussed in a narrative review”.”
78. 7. Synthesise Included Studies
Quantitative Meta-Analysis
References
Field A. & Gillett, R. (2010). How to do a meta-analysis. Br J Math Stat Psychol, 63:
665–694.
Hedges L. & Vevea J. (1998) Fixed-and random-effects models in meta-analysis. Psychol
Methods, 3: 486–504.
Example
Mc Sharry, J., Moss‐Morris, R., & Kendrick, T. (2011). Illness perceptions and glycaemic
control in diabetes: a systematic review with meta‐analysis. Diabetic Medicine, 28(11),
1300-1310.
Cochrane logo
82. 7. Synthesise Included Studies
Narrative Review
•Summarise the types of studies and patterns of results giving
consideration to risk of bias
•Use headings and sub-headings as appropriate
•Try to explain differences between study findings
•Used for broad research questions when included studies differ
83. 7. Synthesise Included Studies
Narrative Review
References
J. Popay, H. Roberts, A. Sowden, M. Petticrew, L. Arai, M. Rodgers et al. (2006).
Guidance on the conduct of narrative synthesis in systematic reviews. University of
Lancaster, UK (2006).
Dixon-Woods , M., Agarwal, S., Jones, D., Young, B. & Sutton, A. (2005). Synthesising
qualitative and quantitative evidence: a review of possible methods. J Health Serv Res
Policy 2005, 10:45-53.
Example
Hynes, L., Byrne, M., Dinneen, S. F., McGuire, B. E., O'Donnell, M., & Mc Sharry, J.
(2014). Barriers and facilitators associated with attendance at hospital diabetes clinics
among young adults (15–30 years) with type 1 diabetes mellitus: a systematic review.
Pediatric diabetes.
87. Setting Facilitators No. of
Studies
Barriers No. of
Studies
Transition Transition programme 4 Lack of collaboration between
paediatric and adult clinics
6
Support and information during
transition
5 Unsupported transition 6
Adult diabetes clinic Continuity of staff in adult clinic 4 Challenges to relationship
development
3
Valued clinic characteristics and
resources
4 Low perceived value of attendance 2
Clear procedure for appointment
making & breaking
1 Difficulties communicating with
service
4
Conflicting schedules 3
Facilitators & Barriers
88. 8. Write Report and Publish
• PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses)
Provides a guide the reporting key methodology information
• Summarise results and answer your review question
• Implications: Further research, clinical practice, policy
• Limitations: Of the primary studies and of the review methodology.
89. 8. Write Report and Publish
• Identify relevant journal
Contact editor if unsure
Some journals only accept invited reviews
• Tailor write-up to specific journal and target audience
• Check word-count and maximum number of references
90. How to Synthesise and Report the
Results of a Systematic Review
Any Questions?
91. Learning Outcomes
By the end of this session you should be able to:
• Describe the key features of a systematic review
• Outline the pros and cons of systematic reviews
• Develop a systematic review research question
• List the steps in carrying out a systematic review
• Choose an appropriate method to synthesise results
Quantitative Meta-analysis
Narrative Review
Qualitative Meta-synthesis
What is a systematic
review?
How to conduct a
systematic review?
How to synthesise and
report the results of a
systematic review?
Session Overview
92. https://nuighealthpsyc
hology.wordpress.com/
2014/10/
Top Tip 1: Be Realistic
Top Tip 2: Register Your Protocol
Top Tip 8: Be Organised
Top Tip 5: Learn from Existing
Reviews
Top Tip 3: Be Specific
Top Tip 4: Know the Literature
TOP TIPS
Top Tip 6: Use Available Expertise
Top Tip 7: Work in a Team
93. Systematic Review Methods:
•Higgins, J. P. T. (2011). Green S. Cochrane handbook for systematic reviews of interventions version 5.1.
0. The Cochrane Collaboration, 5(0).
•Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009). Preferred reporting items for systematic
reviews and meta-analyses: the PRISMA statement. Annals of Internal Medicine, 151(4), 264-269.
•Hopewell, S., McDonald, S., Clarke, M., & Egger, M. (2007). Grey literature in meta-analyses of
randomized trials of health care interventions. Cochrane Database Syst Rev, 2(2)
•Katrak, P., Bialocerkowski, A. E., Massy-Westropp, N., Kumar, V. S., & Grimmer, K. A. (2004). A systematic
review of the content of critical appraisal tools. BMC Medical Research Methodology, 4(1), 22.
•Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., ... & Stewart, L. A. (2015).
Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration
and explanation. BMJ,349, g7647.
•Tricco et al 2015. A scoping review of rapid review methods, BMC Medicine, 13: 224.
•Tricco et al 2017.Rapid reviews to strengthen health policy and systems: a practical guide. 2017.
Sample Reviews:
•Mc Sharry, J., Moss‐Morris, R., & Kendrick, T. (2011). Illness perceptions and glycaemic control in
diabetes: a systematic review with meta‐analysis. Diabetic Medicine, 28(11), 1300-1310.
•Hynes, L., Byrne, M., Dinneen, S. F., McGuire, B. E., O'Donnell, M., & Mc Sharry, J. (2014). Barriers and
facilitators associated with attendance at hospital diabetes clinics among young adults (15–30 years) with
type 1 diabetes mellitus: a systematic review. Pediatric Diabetes.
•McSharry, J., McGowan, L., Farmer, A. J., & French, D. P. (2016). Perceptions and experiences of taking
oral medications for the treatment of Type 2 diabetes mellitus: a systematic review and meta‐synthesis of
References
94. Synthesis Methods:
• Popay, H. Roberts, A. Sowden, M. Petticrew, L. Arai, M. Rodgers et al. (2006). Guidance on the conduct
of narrative synthesis in systematic reviews. University of Lancaster, UK (2006).
• Dixon-Woods , M., Agarwal, S., Jones, D., Young, B. & Sutton, A. (2005). Synthesising qualitative and
quantitative evidence: a review of possible methods. J Health Serv Res Policy 10, 45-53.
• Noblit, G.W., Hare, R.D. (1988). Meta-ethnography: Synthesizing Qualitative Studies. Newbury Park:
Sage Publications; 1988.
• Booth, A., Noyes, J., Fleming, K., Gerhardus, A., Wahlster, P., Van Der Wilt, G.J., Moztgemba, K., Refolo,
P., Sacchini, D., Tummers, M., Rehfuess, E. (2016) Guidance on choosing qualitative evidence synthesis
methods for use in health technology assessments of complex interventions [Online]. Available from:
http://www.integrate-hta.eu/downloads/
• Tong, A., Flemming, K., McInnes, E., Oliver, S., & Craig, J. (2012). Enhancing transparency in reporting
the synthesis of qualitative research: ENTREQ. BMC medical research methodology, 12(1), 181.
• Field A. & Gillett, R. (2010). How to do a meta-analysis. Br J Math Stat Psychol, 63, 665–694.
• Hedges L. & Vevea J. (1998) Fixed-and random-effects models in meta-analysis. Psychol Methods, 3,
486–504.
• http://libguides.library.nuigalway.ie/systematicreviews
• Cochrane Collaboration Website: http://www.cochrane.org/
• PROSPERO: http://www.prospero.com/
• Andy Field: A Bluffer’s Guide to Meta-analysis: http://www.statisticshell.com/docs/meta.pdf
• http://nuighealthpsychology.wordpress.com/2014/10/13/5-tips-on-conducting-a-systematic-review/
References