Interested in a tool to appraise all types of public health evidence? Do you appraise public health evidence? Are you interested in a single tool that can appraise many different types of evidence and study designs? Join us for a webinar to learn about the MetaQAT. This tool was developed by Public Health Ontario. How can the MetaQAT help you? Many different types of evidence and study designs may be used to answer a single public health question. The MetaQAT provides a single process that can be used to appraise these different types of evidence. This supports the use of evidence to formulate recommendations and develop public health actions. The MetaQAT consists of a four domain appraisal framework that includes relevance, reliability, validity, and applicability. A suite of design-specific companion tools are included to provide further guidance to assess validity of common designs. This webinar includes an overview of the MetaQAT by its developer, Dr. Laura Rosella, followed by a presentation from Dr. Catherine Bornbaum, who used the MetaQAT in conducting a systematic review. To see the summary statement of this tool developed by NCCMT, click here: http://www.nccmt.ca/resources/search/243 The National Collaborating Centre for Methods and Tools is funded by the Public Health Agency of Canada and affiliated with McMaster University. The views expressed herein do not necessarily represent the views of the Public Health Agency of Canada. NCCMT is one of six National Collaborating Centres (NCCs) for Public Health. The Centres promote and improve the use of scientific research and other knowledge to strengthen public health practices and policies in Canada.

1. Follow us @nccmt Suivez-nous @ccnmo
Funded by the Public Health Agency of Canada | Affiliated with McMaster University
Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The
views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..
MetaQAT
Presenters:
Laura Rosella, PhD, MHsc
Catherine Bornbaum, PhD
March 21, 2017 1:00 – 2:30 PM ET

2. Follow us @nccmt Suivez-nous @ccnmo
2
Housekeeping
Use Chat to post comments and/or
questions during the webinar
• ‘Send’ questions to All (not
privately to ‘Host’)
Connection issues
• Recommend using a wired
Internet connection (vs.
wireless),
• WebEx 24/7 help line
• 1-866-229-3239
Participant Side
Panel in WebEx
Chat

3. Follow us @nccmt Suivez-nous @ccnmo
3
After Today
The PowerPoint presentation (in English and French)
and English audio recording will be made available.
These resources are available at:
PowerPoint:
https://www.slideshare.net/NCCMT/presentations
Audio Recording:
https://www.youtube.com/user/nccmt/videos

4. Follow us @nccmt Suivez-nous @ccnmo
4
How many people are watching
today’s session with you?
Poll Question #1
A. Just me
B. 1-3
C. 4-5
D. 6-10
E. >10

5. Follow us @nccmt Suivez-nous @ccnmo
Your profession?
Put a √ on your answer (or RSVP via email)
/
Epidemiologist Management (director,
supervisor, etc.)
Allied health
professionals (nurse,
dietician, dental
hygenist, etc.)
Librarian Physician / Dentist Other
5

6. Follow us @nccmt Suivez-nous @ccnmo
MetaQAT
http://www.nccmt.ca/resources/search/243
Episode 32
6

7. NCC
Infectious
Diseases
Winnipeg, MB
NCC
Methods
and Tools
Hamilton, ON
NCC Healthy
Public Policy
Montreal, QC
NCC
Determinants
of Health
Antigonish, NS
NCC
Aboriginal
Health
Prince George, BC
NCC
Environmental
Health
Vancouver, BC
7

8. Registry of Methods and Tools
Online Learning
Opportunities
WorkshopsMultimedia
Public Health+
Networking and
Outreach
NCCMT Products and Services
8

9. Follow us @nccmt Suivez-nous @ccnmo
9
Poll Question #2
How familiar are you with the
method or tool we are discussing
today?
A. I am not familiar with the method or tool
B. I have heard of the method or tool
C. I have used the method or tool

10. Follow us @nccmt Suivez-nous @ccnmo
10
Presenter
Laura Rosella, PhD, MHsc
Canada Research Chair in Population
Health Analytics
Assistant Professor, Dalla Lana School of
Public Health, University
of Toronto
Scientist, Public Health Ontario
Adjunct Scientist, Institute for Clinical
Evaluative Sciences

11. Follow us @nccmt Suivez-nous @ccnmo
11
Presenter
Catherine Bornbaum, PhD
Research Associate, Dalla
Lana School of Public Health,
University of Toronto
Adjunct Research Professor,
Western University

12. A public health approach to appraising
evidence:
Meta-tool for Quality Appraisal
(MetaQAT)
Laura Rosella, PhD
Dalla Lana School of Public Health
Public Health Ontario
March 21, 2017

13. Project team
• Laura Rosella, Scientist
• Beata Pach, Manager, Library Services
• Carolyn Bowman, Senior Research Coordinator
• Sarah Morgan, Library Technician
• Tiffany Fitzpatrick, Epidemiologist
• Vivek Goel, Scientist
13

14. Objectives
1. Rationale and development of the PHO MetaQAT tool
2. Orientation of the tool and how to use it
14

15. 15
Idea/request for
review topic
Formulation of specific review
question
Literature search
Application of inclusion &
exclusion criteria
Quality appraisal of included
studies
Data extraction and analysis
Synthesis or
conclusions
Will start directly here
for review of one study
Some iteration may
be required
Can be done at the same time
i.e. when documenting appraisal you can
also add desired data extraction fields

16. “Critical appraisal is the process of carefully and systematically
examining research to judge its trustworthiness, and its value
and relevance in a particular context.”
• Judiciously assessing quality of evidence is challenging
• Requires expertise (preferably methods and content)
• If we are attempting to inform an evidence base that will inform public
health decisions – it needs to be a judicious and transparent process
• Transparency is critical – not only will it enable us to better support our
recommendations but also challenge our own assumptions to ensure
the science is as strong as possible
• The “why” is often as important as the outcome
Quality (critical) appraisal
Source: Burls A. What is critical appraisal? 2nd ed. [Internet]. Newmarket, UK: Hayward Group; 2009 [cited 2015 Mar 2]. Available from
http://www.bandolier.org.uk/painres/download/whatis/What_is_critical_appraisal.pdf
16

17. Appraisal should be about understanding
• Both strengths and weaknesses
• We need to make decisions, important to use evidence to
fullest extent possible
• Understand how evidence can be used appropriately
17

18. Subjective is not a bad word
• Judgments are made based on appraisal principles
• Using the same principles, similar issues will be identified
• No one right answer, that doesn’t mean anything goes!
• Transparency is key: another person should be able to
understand why you made your assessment
18

19. Transparency
• Importance of documentation – if others know what you
appraised and why, they can ‘appraise your appraisal’
• Transparency is key: another person should be able to
understand why you made your assessment
• Think of it as a way that one can audit your analysis of the
evidence – akin to auditing code following a statistical analysis
19

20. Internal Validity
• Elwood: measure of how easily a difference in an outcome
between two groups can be attributed to the effects of an
exposure or intervention
• L. Green: “the essence of rigor”
External Validity
• The process of generalizing the findings of the study to the
population from which the sample was drawn (or even beyond
that population to a more universal statement)
• Without internal validity a study cannot have external validity
• L. Green: “ the essence of relevance”
20
Internal and external validity

21. Internal validity can be influenced by...
• Study design
• Measurement properties of the variables
• Study recruitment/response rates
• Selection pressures
• Sampling strategy
• Setting
• Investigators/funding
• Among others factors…
• Preferred term: RISK OF BIAS (RoB)
21

22. 22

23. 23
www.riskofbias.info

24. 24

25. Risk of Bias says nothing about the intended use of
the evidence
• Application of the evidence is important in public health
• Context matters – where the evidence was generated, where
it is going to be applied
• Importance of different factors when appraising evidence
should be guided by the intended application
25

26. Contemporary view of “evidence-based”
• Evidence-informed
• The idea that evidence (in the scientific sense) informs practice but that
other factors are important
• Weight of evidence versus strength of evidence
• EBM traditionally focuses on strength of evidence (i.e. RCT) and
discounts indirect evidence from other sources
• Larry Green: “ If We Want More Evidence Based Practice, We
Need More Practice Based Evidence”
26

27. Evidence Informed Public Health (EIPH)
• “public health endeavor in which there is an informed, explicit,
and judicious use of evidence that has been derived from any
of a variety of science and social science research and
evaluation methods”
• Acknowledges the many factors, beyond simply the evidence,
influence decision-making
• “EIPH is a complex, multi-disciplinary process that occurs
within dynamic and ever-changing communities and
encompasses different sectors of society”
http://www.nccmt.ca/uploads/media/media/0001/01/b331668f85bc6357f262944f0aca38c14c89c5a4.pdf
27

28. Beware the evidence pyramid

29. A Hierarchy of Strength of Evidence for Treatment/
Intervention Decisions (Quantitative Research)
29
Relative Strength of
Evidence (with 1 being
the strongest)
Type of Evidence
1 • systematic review of randomized controlled trials
2 • systematic review of observational studies addressing
population health important outcomes
3 • single randomized trial
4 • single observational study addressing population
important outcomes
5 • physiologic and epidemiological study (e.g., study of
infection prevalence, prevalence of cardiovascular risk
factors)
http://www.nccmt.ca/uploads/media/media/0001/01/b331668f85bc6357f262944f0aca38c14c89c5a4.pdf

30. What’s better?
• A large and rigorous prospective cohort study or a small poorly
conducted randomized control trial?
• A systematic review of dozens of well-conducted case control
studies or one large randomized control trial?
• Large and rigorous randomized controlled trial done in Japan
or a time-series analysis in Calgary? (assuming you live in
Toronto)
30
Critical appraisal and the evidence hierarchy

31. What’s better?
• A large and rigorous prospective cohort study or a small poorly
conducted randomized control trial?
• A systematic review of dozens of well-conducted case control
studies or one large randomized control trial?
• Large and rigorous randomized controlled trial done in Japan or a
time-series analysis in Calgary? (assuming you live in Toronto)
• A well conducted study that’s ‘lower’ on the evidence hierarchy may
actually be better...every study needs to be interpreted and assessed
in its own right
31
Critical appraisal and the evidence hierarchy

32. 32
Points to remember:
1. Overall quality appraisal ≠ internal validity
only
2. Risk of bias assessment speaks mainly to
internal validity
3. The study’s quality is only one consideration
in the weight of evidence

33. Skills and knowledge required for quality appraisal
• Knowledge of topic area
• Knowledge and experience with the specific methods used
• Knowledge of intended application
• Experience with appraisal
• Excellent documentation skills - record why decisions were
made
• Note: These traits can be covered by a team
33

34. TOOLS, TOOLS, EVERYWHERE
34

35. A systematic review of 121 published critical appraisal
tools
• 87% of critical appraisal tools were specific to a research
design with most tools having been developed for
experimental studies
• 49% of the critical appraisal tools summarized the quality
appraisal into a numeric summary score
• Few critical appraisal tools had documented evidence of
validity of their items, or reliability of use
• “We found no gold standard critical appraisal tool for any
type of study design”
35
Source: Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA. A systematic review of the content of critical appraisal tools. BMC
Med Res Methodol. 2004 [cited 2014 Aug 7];4:22. Available from: http://www.biomedcentral.com/content/pdf/1471-2288-4-22.pdf

36. Suggested benefits of appraisal tools frameworks
• Checklists that ensure that we don’t overlook important
considerations (i.e. Aide-mémoire)
• CAN limit what we do look for depending what’s in the tool
• Standardize the approaching to reviewing the evidence
• This can be easily challenged because many of the judgements are
subjective – so I'd rather know why you judged something a particular
way versus what your judgement was
36

37. A comparison of tools
Voss and Rehfuess
QUALITY APPRAISAL IN SYSTEMATIC REVIEWS OF PUBLIC
HEALTH INTERVENTIONS: AN EMPIRICAL STUDY ON THE
IMPACT OF CHOICE OF TOOL ON META-ANALYSIS
Recommendations:
1) Testing of a broader set of QATs on a more up-to-date
systematic review of a public health intervention covering a
wide range of epidemiological study designs
2) Research into the development of a reliable QAT with broad
applicability across study designs.
37
Source: Voss PH, Rehfuess EA. Quality appraisal in systematic reviews of public health interventions: an empirical study on
the impact of choice of tool on meta-analysis. J Epidemiol Community Health. 2013;67(1):98-104.

38. Features of a Gold Standard QAT
Voss and Rehfuess
1) Instead a set of companion QATs comprising common components and study
design-specific components could initially be developed for standard designs
and then expanded to address more complex designs.
2) An overall summary score may lead to information loss and oversimplification.
Instead, the authors propose a careful quantification of domains (rather than
individual questions) in a weighted checklist.
3) Should address both internal and external validity with a clear distinction
between these.
4) Much more effort should be invested into the development and testing of
appropriate and specific questions and answer categories; a manual can help
with their correct interpretation.
5) Subject-specific adaptation of selected questions makes quality appraisal more
precise and reliable than a one size fits all QAT.
38
Source: Voss PH, Rehfuess EA. Quality appraisal in systematic reviews of public health interventions: an empirical study on
the impact of choice of tool on meta-analysis. J Epidemiol Community Health. 2013;67(1):98-104.

39. METAQAT PROCESS
39

40. MetaQAT project stages
40
Stage 1: Gather
information
Stage 2: Put
information
together
Stage 3a: Draft
the meta-tool
Stage 3b:
Refine draft
meta-tool
Stage 4: Pilot
testing
Stage 5:
Validation

41. Stage 1: Gather information
• Searched for existing tools
• Databases
• Grey literature
• Guidance and background documents on critical appraisal in public health
(NICE, ECDC, NCCMT, Cochrane)
• Websites
• Universities with public health programs
• NCCMT repository of critical appraisal tools
• Consult with experts
• Consult with PHO staff
41

42. Summary on tools to guide appraisal
• Many tools exist (most not validated)
• Most from evidence-based medicine perspective
• Internal validity (or risk of bias) focus
• Many use numeric scales or scores
• Not valid, not recommended by Cochrane Collaboration
• Many tools appropriate for specific contexts (topic, project,
study design)
• Something more flexible was required
42

43. What kind of tool?
• One appropriate for our context
• Various types of reviews
• Rapid reviews, evidence briefs, systematic reviews, etc.
• Various types of content
• Chronic disease, infectious disease, environmental health, health promotion,
injury prevention, etc.
• Strong focus on the applicability of the evidence
• Not just the quality
• Not reinventing the wheel
• Build on existing content
43

44. What tool for PHO?
• Many tools appropriate for specific contexts (topic, project,
study design)
• Resources available to find tools
• NCCMT tool repository
• University SA library
• PHO needs:
• Compatible with all of the types of evidence we use
• Compatible with all of the types of projects we do
• Consistent with current best practice
• No numeric scores; acceptable to field
• Promotes a transparent process
• Outcome of stage 1: not a critical appraisal tool (too limited)
44

45. Stage 2: Putting the information together
• Group tools by study design
• Compare the questions between tools
• Similarities across tools within each design
• Compare these similarities across study designs
• Core process is similar across tools
• Coverage of process matches OPHLA guide
• Generic, applies to all evidence types
• Generic framework emerges
• All questions can be grouped into four categories
• Still need study specific detail for high level of rigour
• Not a critical appraisal tool, a quality assessment process
45

46. Stage 3a: Develop draft meta-tool
• Develop framework and select companion tools
• Companion tools – pragmatic approach
• Widespread use in public health literature
• Preference to tools with documented validation
• Compromise – reporting tools widely used
• Ease of use
• Revise based on scientific advice
46

47. Stage 3b: Refine draft meta-tool
• Feedback sessions
• Staff selected for feedback sessions
• Sample article assessed, interpretations of questions
• Revisions made for clarity, reorganization
• Revised version presented to third group
• Revisions made
• Tool approved for pilot testing and further development and
evaluation
47

48. Stage 4: Pilot testing
• Introduced to teams for trial use in specific projects
• Variety of topics and types of projects
• Training methods developed
• Introduction to tool as project reached appraisal stage
• Evaluation
• Fit of tool for projects and staff, requirements for full implementation
48

49. Selected pilot testing outcomes
INTEGRATION OF DIVERSE GROUPS OF EVIDENCE
• Applied to projects with a heterogeneous mix of evidence
• The flexibility of MetaQAT allowed for the integrated
assessment of evidence across methods and designs
• Facilitated screening decisions
• Broadened scope of appraisal promoted appropriate
consideration of context versus internal validity alone
• Maintains consistent process of appraisal across a group of
evidence that includes grey literature and surveillance
literature
49

50. Stage 5: Validation
• Two groups of appraisers
• One group used MetaQAT, other group appraised strengths and
weaknesses of several studies
• Written appraisals analysed
• Content analysis
• Both groups identified similar issues
• MetaQAT group identified public health relevance
50

51. Meta-tool
• The “meta” concept is not a single tool/framework (traditional
sense), it is a process guide which orients the user to the
appropriate use of several appraisal tools, and places them
within a larger framework to guide their use
• A tool of tools: Appraisal framework that incorporates existing
tools for design-specific risk of bias assessment
• Documentation – records the appraisal process for
transparency
51

52. MetaQAT structure
52
Relevancy
Companion
tools
AMSTAR
CASP
TREND
CONSORT
Mixed Methods
AGREE II
Navigation Guide
PHO Guide for Grey
Literature
Reliability
Validity
Applicability
Appraisal Framework
Rosella, L. Bowman, C., Pach, B., Morgan, S., Fitzpatrick, T., Goel, V. The development and validation of a meta-tool for
quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health. 2016;136:57-65.

53. MetaQAT structure
Domain Idea
Relevancy
• Determine if the item being appraised is sufficiently related to your topic or
research question.
• In a structured systematic review, relevancy can also be considered in the
inclusion and exclusion criteria.
Reliability
• Evaluate the completeness of reporting; complete reporting is necessary in
order to conduct a thorough examination of quality.
• Lack of detail regarding the conduct of the study or report may be
indicative of lower quality information.
Validity
• The likelihood and magnitude of error or bias in a study.
• “Risk of Bias” or “Internal Validity”
Applicability
• Consider how the evidence might be applied to public health practice.
• Also can consider more broadly what can be learned from the evidence and
how you can apply that knowledge to public health decision making.
53
Rosella, L. Bowman, C., Pach, B., Morgan, S., Fitzpatrick, T., Goel, V. The development and validation of a meta-tool
for quality appraisal of public health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health.
The form uses the term “study” to refer to the work being appraised and is meant to include all study types, including research published as grey
literature, as syntheses, and any other type of evidence being considered.

54. MetaQAT appraisal form
54
• Domain
• Main question
• Prompting questions
• Guide thinking about
the main question
• Stimulate thought, not
limit
• Written answer
• Optional check boxes

55. MetaQAT appraisal form
• Written answer format
• Written summary of key points
• Tick box options available BUT
• Written summary is the answer to the
question (many formats available)
55

56. MetaQAT appraisal form
• Branch point in process – to design-specific companion tools IF DESIRED
• In validity domain, question b) Is the research methodology free from bias?
• Design-specific detail via companion tools IF DESIRED
56
• Can also create
context-specific
prompts

57. Summarizing the appraisal of an item
• Textual summary of key strengths and weaknesses by domain
and question
• This information will provide a good understanding of the quality of the
item
• E.g. which is more helpful? 8/10 or “the study group was small but
characteristics were very similar to our local population”
• We are working on guidance to help with summary/synthesis language
• “The use of scales for assessing quality or risk of bias is
explicitly discouraged in Cochrane reviews”
57
Source: Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions. Version 5.1.0
[Internet]. London: Cochrane Collaboration; 2011 [cited 2015 Feb 10]. Section 8.3.3, Quality scales and
Cochrane reviews. Available from: http://handbook.cochrane.org

58. MetaQAT user guide
• The MetaQAT document includes relevant background
information and user guidance – please read! (url to follow)
• For more details:
Rosella L, Bowman C, Pach B, Morgan S, Fitzpatrick T, Goel V. The
development and validation of a meta-tool for quality appraisal
of public health evidence: Meta Quality Appraisal Tool
(MetaQAT). Public Health. 2016;136:57-65. Available from:
http://www.publichealthjrnl.com/article/S0033-3506(15)00437-
0/abstract
58

59. Hard work is hard
• The process of systematically appraising evidence and
documenting this process is difficult and can be time
consuming but these challenges may exist regardless of the
tool/framework
59

60. 60
https://www.publichealthontario.ca/en/ServicesAndTools/Pages/Critical-Appraisal-Tool.aspx
Screen shots to follow

61. 61

62. 62

63. 63

64. References
• Rosella L, Bowman C, Pach B, Morgan S, Fitzpatrick T, Goel V. The
development and validation of a meta-tool for quality appraisal of public
health evidence: Meta Quality Appraisal Tool (MetaQAT). Public Health.
2016;136:57-65. Available from:
http://www.publichealthjrnl.com/article/S0033-3506(15)00437-0/abstract
• Voss PH, Rehfuess EA. Quality appraisal in systematic reviews of public
health interventions: an empirical study on the impact of choice of tool on
meta-analysis. J Epidemiol Community Health. 2013;67:98-104.
• Heller RF, Verma A, Gemmell I, Harrison R, Hart J, Edwards R. Critical
appraisal for public health: A new checklist. Public Health. 2008;122:92-98.
• Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of
clinical trials for meta-analysis. JAMA. 1999;282(11)1054-1060.
• Katrak P, Bialocerkowski AE, Nassy-Westropp N, Kumar VSS, Grimmer KA. A
systematic review of the content of critical appraisal tools. BMC Med Res
Methodol. 2004 [cited 2013 Dec 16];4(22): . Available from:
http://www.biomedcentral.com/1471-2288/4/22
64

65. Information management
• Excel version of form is available as template
65

66. 66

67. 67

68. 68

69. 69
NCCMT Spotlight on Methods and Tools:
Using the MetaQAT for Quality Appraisal in a
Systematic Review
Catherine Bornbaum, PhD
Managing Director, Population Health Analytics Research Program
Research Associate, Dalla Lana School of Public Health, University of Toronto
Adjunct Research Professor, Health & Rehabilitation Sciences, Western University

70. Background
• Failure to optimize the use of research evidence may result in
reduced quality of care1, inefficient use of resources2,3, and poorer
health outcomes for individuals and communities4
• Given the challenges with effectively translating knowledge
between researchers, practitioners, and decision-makers5,
knowledge translation experts advocated for an intermediary
person to mitigate challenges in knowledge sharing6,7
• Knowledge brokers work collaboratively with key stakeholders to
facilitate the transfer and exchange of evidence and context
• Key roles:
• Knowledge manager, linkage agent, capacity builder
70

71. To understand how knowledge brokers operate in health-related practice,
and assess if use of a knowledge broker could facilitate effective
knowledge translation initiatives.
Objectives:
1. To improve conceptualization of the knowledge brokering role in
health-related settings; and
• Test theory of knowledge brokering in practice (confirmatory
thematic analysis8)
2. To determine whether knowledge brokers contribute to effective
knowledge translation in health related settings
• Explore outcome data for evidence of changes in knowledge, skill,
policy and practice9
• Critically appraise evidence using the Meta Quality Appraisal Tool
(MetaQAT)
71
Goal

72. Inclusion and Exclusion Criteria
• Inclusion Criteria
• Reports on an actual application of knowledge brokering in practice
• Health-related setting (e.g., public health, clinical setting,
rehabilitation)
• Document available in English
• Exclusion Criteria
• Reports on the theoretical assumptions about the practice of
knowledge brokering only
• Study protocols
72

73. Method
Search strategy
Data Sources
• Databases
• MEDLINE; EMBASE; PsychINFO; CINAHL; SCOPUS; SocINDEX; Health
Business Elite
• Grey literature
• Knowledge translation networks: Canadian Foundation for Healthcare
Improvement, National Collaborating Centre for Methods and Tools
• Health-focused organizations: CIHR, Canadian Public Health Information,
Health Evidence, Ontario Public Health Unit, World Health Organization
• Hand searching
• E.g., Implementation Science, BMC Health Service Research, etc.
• Reference lists
73

74. Method
74
IdentificationScreeningEligibilityIncluded
Records identified
through databases
(n = 7,022)
Records identified
through other
sources (n = 340)
Records identified
through databases
(n = 502)
Records identified
through other
sources (n = 71)
January 2014 November 2014
Records after
duplicates removed
(n = 6,363)
Records screened
(n = 227)
Full-text articles
assessed for
eligibility (n = 46)
Full text articles
excluded
(n = 181)
Records
excluded
(n = 6,136)
Articles included in synthesis (n = 26)
Unique projects included in synthesis (n = 20)
Articles included in synthesis (n = 3)
Unique projects included in synthesis (n = 3)
Total articles (n = 29)
Total unique projects (n = 23)
Records after
duplicates removed
(n = 573)
Records screened
(n = 13)
Full-text articles
assessed for
eligibility (n = 12)
Records
excluded
(n = 560)
Full text articles
excluded
(n = 1)

75. MetaQAT USER INSIGHTS
• Adapted MetaQAT forms to study context prior to conducting
critical appraisal
• Modified “hint questions” where appropriate
• Adapted for Excel
Method
Critical appraisal
• Why we chose the MetaQAT to conduct critical appraisal
• Permitted analysis of diverse research designs (e.g., qualitative, quantitative,
grey literature) and health-related settings
• Four step critical appraisal framework to assess: relevancy, reliability, validity
and applicability
• Research design-specific companion tools for quantitative, qualitative and
mixed method research design appraisal
75

76. Conducting Critical Appraisal
Using an Excel-based template to organize findings
76
Additional companion
tools required?
Assessments of Relevancy,
Validity, Reliability and
Applicability

77. Conducting Critical Appraisal
Completing companion tools
77
Where applicable, complete the companion tool before completing the MetaQAT

78. 78
Presenting MetaQAT Results
PowerPoint format
Project Relevance Reliability Validity Applicability
A16,17 Relevant
setting and
population
Insufficient description
of population,
intervention, outcomes
Potential bias in design,
sampling, outcomes
Insufficient
description of
context/outcomes
B18 Relevant
setting and
population
Insufficient description
of population and
intervention
Potential bias in sampling,
outcomes (unsystematic
reflection)
Difficult to generalize
C19 Relevant
setting and
population
No description of
analytic methods
Potential bias in sampling,
outcomes (no KB
perspective)
Insufficient
description of
context/outcomes
D20-22 Relevant
setting and
population
No significant concerns No significant concerns No significant
concerns
E23 Relevant
setting and
population
Insufficient description
of population,
intervention, outcomes
Potential bias in design,
sampling, data sources
Insufficient data to
support applicability
F24,25 Relevant
setting and
population
Insufficient description
of population and
intervention
Potential bias in design,
data sources, exclusion
criteria
Insufficient
description of KB
role/outcomes
G26-28 Relevant
setting and
population
No significant concerns Chance cannot be ruled
out (no CIs presented)
No significant
concerns

79. 79
Presenting MetaQAT Results
Manuscript Format

80. Interested in our findings?
80
Review available from:
https://implementationscience.biomedcentral.com/articles/10.1186/s13012-015-0351-9

81. • When conducting a meta-synthesis, quality appraisal tool needs to be
broad enough to account for various design-specific concerns
• Although designed for use in public health contexts, MetaQAT is
applicable to diverse health-related contexts
• E.g., clinical, rehabilitation, health policy settings
• The “Applicability” appraisal was highly useful for assessing utility of
knowledge brokering strategies – a key component of our review
• Overall, MetaQAT was the only tool capable of meeting our broad needs
relative to methodological and contextual factors
81
Lessons Learned

82. Public Health Ontario Library Services:
https://www.publichealthontario.ca/en/ServicesAndTools/Pages
/Critical-Appraisal-Tool.aspx
82
Ready to give it a try?

83. Acknowledgements
• We wish to acknowledge the Canadian Institutes of Health Research for
their support of this research program (KAL-129895)
• We wish to thank Public Health Ontario Research Librarians, Allison
McArthur and Domna Kapetanos for their assistance with the design
and conduct of the search strategy
83

84. Thank you!
Want to connect? Feel free to reach out:
Catherine Bornbaum, PhD
catherine.bornbaum@utoronto.ca
@CathBornbaum
84

85. References
1. Jernberg T, Johanson P, Held C, Svennblad B, Lindbäck J, Wallentin L et al.: Association between
adoption of evidence-based treatment and survival for patients with ST-elevation myocardial
infarction. Journal of the American Medical Association 2011, 305: 1677-1684.
2. David D, Davis M, Jadad A, Perrier L, Rath D, Ryan D et al. (Eds):The case for knowledge translation:
Shortening the journey from evidence to effect. In British Medical Journal 2003, 327: 33-35.
3. Madon T, Hofman K, Kupfer L, Glass R: Public health: Implementation science. Science 2007, 318:
1728-1729.
4. Chalmers I (Eds):If evidence-informed policy works in practice, does it matter if it doesn't work in
theory? In Evidence & Policy: A journal of Research, Debate and Practice 2005, 1: 227-242.
5. Graham I, Logan J, Harrison M, Straus S, Tetroe J, Caswell W et al. (Eds):Lost in knowledge
translation: Time for a map? In Journal of Continuing Education in the Health Professions 2006, 26:
13-24.
6. Dobbins M, DeCorby K, Twiddy T (Eds):A knowledge transfer strategy for public health decision
makers. In Worldviews on Evidence-Based Nursing 2004, 1: 120-128.
7. Ward V, House A, Hamer S: Knowledge brokering: The missing link in the evidence to action chain?
Evidence & Policy 2009, 5: 267-279.
8. Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, et al.: Guidance on the conduct of
narrative synthesis in systematic reviews: A product from the ESRC methods programme. 2006.
9. Kujbida G, Stratton J. Effective knowledge translation tactics for increasing the use of health status
and surveillance data. 2014. Retrieved from: www.peelregion.ca/health/library/pdf/effective-kt.pdf
85

86. Follow us @nccmt Suivez-nous @ccnmo
86
Your Comments/Questions
• Use Chat to post comments
and/or questions
• ‘Send’ questions to All (not
privately to ‘Host’)
Chat
Participant Side
Panel in WebEx

87. Follow us @nccmt Suivez-nous @ccnmo
87
Poll Question #3
Could this method or tool be useful
in practice?
A. Very useful
B. Somewhat useful
C. Not at all useful
D. Don’t know

88. Follow us @nccmt Suivez-nous @ccnmo
88
Your Feedback is Important
Please take a few minutes to share your thoughts
on today’s webinar.
Your comments and suggestions help to improve
the resources we offer and plan future webinars.
The short survey is available at:
https://nccmt.co1.qualtrics.com/SE/?SID=SV_23n
CMSgc5sz9QJD

89. Follow us @nccmt Suivez-nous @ccnmo
89
Poll Question #4
What are your next steps? (Check all
that apply)
A. Access the method/tool referenced in the
presentation
B. Read the NCCMT summary about the
method/tool described today
C. Consider using the method/tool in
practice
D. Tell a colleague about the method/tool

90. Follow us @nccmt Suivez-nous @ccnmo
90
Join us for our next webinar
Spotlight on Methods and Tools: Knowledge
Translation and Implementation Video Series
Date: Wednesday April 5, 2017
Time: 1:00 – 2:30pm EST
Interested in a tool to support planning program
development? Do you plan program implementation? Are you
interested in an overview of the key concepts regarding
implementation of evidence-based practices? Join us for a
webinar to learn about the Knowledge Translation and
Implementation Video Series.
Register at: https://health-evidence.webex.com/health-
evidence/onstage/g.php?MTID=e4a0468604363b8e349ce95
b2277da0f6

91. Follow us @nccmt Suivez-nous @ccnmo
Funded by the Public Health Agency of Canada | Affiliated with McMaster University
Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The
views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..
For more information about the
National Collaborating Centre
for Methods and Tools:
NCCMT website www.nccmt.ca
Contact: nccmt@mcmaster.ca