Quality Assessment of Community Evidence (QACE) Tools (March 2020)
•Télécharger en tant que PPTX, PDF•
0 j'aime•355 vues
Join us in launching the NCCMT’s new tools for assessing the quality of community evidence. These tools can be applied to evidence for local health issues and context, such as surveillance data and community health status reports, as well as evidence for community and political preferences and actions, such as the expressed needs and interests of community members and support from public or government officials.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Quality Assessment of Community Evidence (QACE) Tools (March 2020)
Similaire à Quality Assessment of Community Evidence (QACE) Tools (March 2020) (20)
Plus de The National Collaborating Centre for Methods and Tools
Plus de The National Collaborating Centre for Methods and Tools (20)
Dernier
Dernier (20)
Quality Assessment of Community Evidence (QACE) Tools (March 2020)
- 1. Quality Assessment of Community Evidence (QACE) Tools Today’s webinar will be starting at 1:00 pm (EST).
- 2. Housekeeping • Recently, we have been experiencing connection issues during our webinars. • To reconnect: 1. Hoover over the bottom of the screen until a toolbar appears 2. Click on the audio button 3. Click play (button should now be blue) 2
- 3. Housekeeping • Use Chat to post comments and/or questions during the webinar – ‘Send’ questions to All (not privately to ‘Host’) • Connection issues – Recommend using a wired Internet connection (vs. wireless), • WebEx 24/7 help line – 1-866-229-3239 3 Chat
- 4. After Today The PowerPoint presentation (in English and French) and English audio recording will be made available. These resources are available at: • PowerPoint: https://www.slideshare.net/NCCMT/presentations • Audio Recording: https://www.youtube.com/user/nccmt/videos 4
- 5. How many people are watching today’s session with you? A. Just me B. 1-3 C. 4-5 D. 6-10 E. >10 5 Poll Question #1
- 6. Have you visited the NCCMT website or used its resources before? A. Yes B. No 6 Poll Question #2
- 7. If you stated YES on the previous question, how many times have you used the NCCMT’s resources? A. Once B. 2-3 times C. 4-10 times D. 10+ times 7 Poll Question #3
- 9. Registry of Methods and Tools Online Learning Opportunities WorkshopsMultimedia Public Health+ Networking and Outreach 9
- 10. Quality Assessment of Community Evidence (QACE) Tools Susan Snelling, PhD
- 11. Outline • Evidence-informed decision making in public health • Community evidence in public health • Quality assessment of community evidence – New tool(s): QACE • What next? 11
- 12. The NCCMT: Curating evidence, sharing knowledge, building skills • Our goal is to support the use of the best available evidence in public health practice and policy to optimize health across Canada • We support public health professionals in finding and using high quality, up-to-date methods and tools to build capacity for evidence-informed public health. 12
- 13. What is Evidence-Informed Public Health? • The process of distilling and disseminating the best available evidence from research, context and experience, and using that evidence to inform and improve public health practice, programs and policy. 13
- 14. What is Evidence? 14 Community Health Issues, Local Context e.g. surveillance data or community health status reports Public Health Expertise Community and Political Preferences and Actions e.g. newspaper stories or community focus groups Research e.g. qualitative or quantitative research Resources e.g. financial or human resources
- 15. “Community Evidence” Evidence for local health issues and context includes: • Magnitude of the health issue in the local setting, based on surveillance data and community health status reports • Significance and importance of the issue in comparison to other community health concerns Evidence on community and political preferences and actions includes: • Needs, interests and intervention preferences of community members, stakeholder organizations and specific consumer groups • Support or opposition from the public/government officials • Current political climate (organizational, local, regional, provincial, federal) 15
- 16. “Quality Assessment” Evidence-informed decision making (EIDM) in public health uses the best-available evidence to inform decisions. 16
- 17. Quality Assessment of Community Evidence • There are existing tools to appraise the quality of research evidence (e.g., AGREE II; CASP; etc.). But research evidence is not the only influence on decisions. • How can the quality of community-based evidence be assessed? – Quality Assessment of Community Evidence (QACE) tools were developed to meet this need. 17
- 18. QACE Development Process of Development • Academic and grey literature review, looking for existing tools • Focus group sessions with Ontario public health professionals • Other feedback from librarians, other public health professionals 18
- 20. 20
- 21. QACE Tools: www.nccmt.ca/qace The Quality Assessment of Community Evidence (QACE)Tools were developed to assess the quality of evidence for: A) B) The QACE tools help practitioners answer the question: “Is the quality of this evidence about community needs and preferences good enough to influence decision making?” 21
- 22. 22
- 23. Decision: Should our community implement an email vaccination reminder program? • Find research evidence • Assess quality, retaining the best-available evidence 23 Research e.g. qualitative or quantitative research
- 24. Decision: Should our community implement an email vaccination reminder program? • Find research evidence • Assess quality, retaining the best-available evidence Emailed reminders appear to increase uptake, compared to no reminders 24 Research e.g. qualitative or quantitative research
- 25. Decision: Should our community implement an email vaccination reminder program? 25 Community health issues, local context: QACE Tool A • Find evidence of local community need and context • Assess quality, retaining the best-available evidence • E.g., health status data on vaccination rates • E.g., data on local email access
- 26. 26 Is the quality of this evidence about local health issues and local context good enough to influence decision making?” QACE Tool A Relevant Meaningful Does this source address my topic of interest? Is this indicator relevant to my topic? Does this source allow me to determine the significance of this issue compared to other issues? Applicable Transferable Does this source include data available at the regional/local level? How important is regional/local level data for my topic of interest? Summary of your assessment: (e.g., quality, gaps and limitations)
- 27. 27 Is the quality of this evidence about local health issues and local context good enough to influence decision making? QACE Tool A Trustworthy Methodologically sound What methods were used? Were those appropriate methods for the topic? To what extent did the methods reduce the risk of bias? Are there conflicts of interest that could introduce bias into the evidence? Transparent Cognizant of research evidence Does this source draw a conclusion? Is the conclusion based on evidence? To what extent is the basis for that conclusion transparent? To what extent does the conclusion align with other available evidence (“triangulation”)? What might account for any differences? Richness/ Saturation/ Adequacy of data Are there gaps in this data source? How significant are those gaps to a complete understanding of the issue? Summary of your assessment: (e.g., quality, gaps and limitations)
- 28. 28 Is the quality of this evidence about local health issues and local context good enough to influence decision making? QACE Tool A Equity-Informed Representative of community What is the level of analysis and reporting (e.g., census area or smaller)? Does this source provide data on the health status of specific groups in the community? Where are the population health data gaps? Engaging stakeholders Intersectional Inclusive Does this source include all groups, including disadvantaged groups? Were any population groups excluded from data collection (e.g., people without telephones, no permanent housing, etc.)? Where are the population health data gaps? Culturally safe; ethical data collection To what extent was the evidence collected in an ethical and culturally safe way? Did the source abide by Tri-Council policies on ethical data collection? Were OCAP™ principles observed, if applicable? Were communities consulted about whether and how they wanted to provide data? Were communities involved in the interpretation and sharing of the findings? Summary of your assessment: (e.g., quality, gaps and limitations)
- 29. 29 Community and political preferences and actions: QACE Tool B • Find evidence of local community preferences • Assess quality, retaining the best-available evidence • E.g., focus group data on support for email reminders among parents? • E.g., other similar programs to learn from? Decision: Should our community implement an email vaccination reminder program?
- 30. Is the quality of this evidence about community and political preferences and actions good enough to influence decision making? 30 QACE Tool B Relevant Meaningful Does this source outline a clear question that is being answered? Does this source address my topic of interest? Applicable Transferable How applicable and transferable is this evidence to my community and local context? How similar is the context of this source to my setting? Summary of your assessment: (e.g., quality, gaps and limitations)
- 31. Is the quality of this evidence about community and political preferences and actions good enough to influence decision making? 31 QACE Tool B Trustworthy Methodologically sound What methods were used? Were those appropriate methods for the topic? To what extent did the methods reduce the risk of bias? (A separate quality assessment tool such as the GRADE-CERQual can help answer this question.) Are there conflicts of interest that could introduce bias into the evidence? Transparent Cognizant of research evidence Does this source draw a conclusion? Is the conclusion based on evidence? To what extent is the basis for that conclusion transparent? To what extent does the conclusion align with other available evidence (“triangulation”)? What might account for any differences? Richness/ Saturation/ Adequacy of data To what extent is this evidence adequate for understanding all perspectives on the issue? How significant are any gaps to a complete understanding of the issue? Summary of your assessment: (e.g., quality, gaps and limitations)
- 32. Is the quality of this evidence about community and political preferences and actions good enough to influence decision making? 32 QACE Tool B Equity-Informed Representative of community How representative is this source of my community, including disadvantaged groups? Have all affected community groups been considered? Whose perspectives are not heard? Engaging stakeholders Intersectional Inclusive How were affected communities engaged in the evidence gathering process? Whose perspectives are not being considered? How participatory was the engagement of members of affected community groups? Where does the community engagement fall on the IAP2 spectrum: inform, consult, involve, collaborate, empower? Culturally safe; ethical data collection To what extent was the evidence collected in an ethical and culturally safe way? Did the source abide by Tri-Council policies on ethical data collection? Were OCAP™ principles observed, if applicable? Were communities consulted about whether and how they wanted to provide data? Were communities involved in the interpretation and sharing of the findings? Summary of your assessment: (e.g., quality, gaps and limitations)
- 33. 33 Community Health Issues, Local Context e.g. surveillance data or community health status reports Public Health Expertise Community and Political Preferences and Actions e.g. newspaper stories or community focus groups Research e.g. qualitative or quantitative research Resources e.g. financial or human resources Decision: Should our community implement an email vaccination reminder program?
- 34. 34
- 35. We’ve done quality assessment…now what? QACE in context 35
- 36. Stay in Touch! Susan Snelling: snellin@mcmaster.ca Website: nccmt.ca Twitter: @nccmt Email: nccmt@mcmaster.ca 36
- 37. Your Comments / Questions • Use Chat to post comments and/or questions – ‘Send’ questions to All (not privately to ‘Host’) 37 Chat Participant Side Panel in WebEx
- 38. Webinar Feedback Your responses will be completely anonymous to other Webinar participants. Please indicate your level of agreement with the following: 1. Participating in this webinar increased my knowledge and understanding of this tool. 2. How likely are you to read the NCCMT user’s guide on the tool from today’s webinar in your own practice. 3. Which of the following statements apply to your experience with the webinar today (select all that apply): The webinar was relevant to me and my public health practice. The webinar was effectively facilitated The webinar had opportunities to participate The webinar was easy to follow along The webinar met my expectations 4. Can we contact you in the future to discuss how NCCMT can improve its webinar series? Yes No 5. If yes, Please provide your name and email address: Name: ____________________________________________ E-mail: ____________________________________________ 38 Strongly agree Agree Undecided Disagree Strongly Disagree Strongly agree Agree Undecided Disagree Strongly Disagree
- 39. Thank you! For more information about the NationalCollaborating Centre for Methods andTools: NCCMT website: www.nccmt.ca Contact: nccmt@mcmaster.ca 39 Funded by the Public Health Agency of Canada | Hosted by McMaster University Production of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The views expressed here do not necessarily reflect the views of the Public Health Agency of Canada.
Notes de l'éditeur
- FYI: Presentation Guidelines 8 minutes + 2 minutes for discussion Max. 10 slides including introduction, conflict of interest and end slide
- NCCs are a unique structure in Canada – government-funded but apolitical Collective goal to improve the health of all Canadians by sharing knowledge, providing training, identifying gaps in knowledge and practice Each NCC focuses on a different theme in public health and hosted by different organization, but all national in scope At NCCMT we champion the use of evidence in public health decision making We help to compile the latest research and evidence for what is working in public health practice (e.g. Health Evidence) We work with practitioners to help develop more knowledge and skills for using different types of evidence when making decisions in their own work We support and help build confidence in the public health workforce so that practitioners know they are making informed decisions
- NCCMT offers a products and services to help apply research evidence in decision making This presentation today is going to provide an overview of the Online Learning Opportunities that NCCMT offers.
- Start with context for the tools: EIDM and community evidence. Then I’ll introduce the NCCMT’s new tools for assessing the quality of community evidence. The QACE tools will help you answer the question: “Is the quality of this evidence about local community needs and preferences good enough to influence decision making?” Finally, I’ll have some suggestions for how QACE may fit into an overall approach to EIDM. Welcome questions through chat…I may hold until a pause to answer.
- .
- These are all examples of Community Evidence By its nature, community-level evidence is contextual and local, which makes it different in some ways from published research evidence which may not have a local flavour, and can still be very relevant to decision-making even without a local component. Community evidence can be identified through population health status data, community consultations, newspaper reports, letters and lobbying to public health and politicians, and many other avenues. These are the kinds of evidence that the QACE tool are designed for. Using community evidence in decision-making is probably not new to you, so this idea is not what’s new here. Public health practitioners have always looked to bring in contextual information into decision-making. But in public health we have not always been rigorous about the quality of the community evidence that is or should be influential. It’s when we bring concepts of quality assessment or critical appraisal in that we started to see a gap in the available tools.
- When I talk about quality assessment, recall that Evidence-informed decision making (EIDM) in public health uses the best-available evidence to inform decisions. You can’t know what’s best if you don’t assess quality. How can we know what is ‘good enough’ quality? Not all community evidence is of sufficient quality that we should base our decisions on it. Sometimes the loudest community voice is not the only one we should listen to. Although community evidence is different from research evidence in some ways, there is no reason to have a lower standard of quality for community evidence than for research evidence…it’s just that what constitutes quality may differ. That’s what the QACE tools are designed to address.
- Those tools don’t work very well for community evidence.
- What constitutes high-quality evidence about a local community: You want to know that the evidence is RELEVANT to the decision you need to make and to your community context. You want to know that the evidence is TRUSTWORTHY, using methods that reduce bias and are transparent. You want to know that the evidence is EQUITY-INFORMED, to guide decisions that are sensitive to equity concerns and will not negatively affect disadvantaged groups. Each of these three dimensions inform the quality of community evidence and should be assessed before being used to influence decisions. The QACE Tools provide questions that can guide the quality assessment of community evidence. A collection of supplementary tools and resources can also help you delve more deeply into these aspects of quality assessment.
- These tools can be applied to evidence for local health issues and context, such as surveillance data and community health status reports, as well as evidence for community and political preferences and actions, such as the expressed needs and interests of community members and support from public or government officials
- Fillable PDF, two sided. Determine which questions apply – flexible. Not all apply. Then consider each question and summarize your response – maybe as a team! You’ll repeat the process for each source of community evidence you are thinking about using.
- I’m going to use this example…but note it’s just an example and I’m not claiming to have pulled together all the evidence on this topic. We’re not going to answer this question definitively in this webinar. Let’s say someone comes to you and says “I heard about this email reminder idea at a conference, or I read about it in online. I think we should consider doing this here! But I want you to help.” When you have a question, a decision to make about what to do, we generally suggest starting with evidence from research.
- If an intervention truly doesn’t work, you probably don’t need to look at the community preference evidence. Be efficient in your decision-making! But in this case, Health Evidence has a recent systematic review that suggests this could be effective. And in terms of quality, it’s moderate. So we might feel it’s worth at least exploring as an option. Not so fast! What other evidence might influence our decision?
- You might look to local health status data on vaccination rates, because it your local rates are already high, you may not need this intervention, despite it being effective. And another source of evidence might be data on how much of the population has email access – because if it’s limited, this might not be an intervention that would work well for your community.
- Keep in mind that the QACE tool helps with quality assessment, not analyzing the content of the evidence. You find local health status data on vaccination rates. First are questions about relevance. If you only have data on HPV vaccination, but not other kinds of immunization, you might say those data are not super relevant for the decision you need to make. So if the questions is, is the quality good enough to influence decision-making, you might already say there are some limitations in the quality for your purpose. Or if we’re thinking about evidence on how much of the population has email access, you might find data only for your province, not at a local level. But that might not be a serious flaw if you have no reason to believe your area is any different from the province.
- Still thinking about the data on vaccination rates, for quality assessment, you want to understand how those data were collected, and whether the methods were appropriate. Thinking about data on email access, the data may come from an internet provider…so you might question the quality of the data because there could be some bias, or they could have drawn a conclusion that isn’t really supported by the evidence, or there are other sources that say something very different. These questions are designed to help you navigate the trustworthiness of the evidence, and identify any unanswered questions.
- Finally, in public health, quality includes whether evidence is equity-sensitive. For example, does the evidence on vaccination rates include everyone in the community? Might people without a health care provider be missing? That would be a limitation on the quality of the evidence. Or when considering data on email access, if your source has somehow acquired data that really should not be used for tis purpose, that’s a quality problem. It may not apply so much in the examples I’m using, but I am sure you can think of evidence that has been acquired without community agreement, and especially for sensitive topics, you would not want to rely on ethically suspect research to back up your decisions!
- That was tool A, focused on pop health and surveillance kinds of evidence. Now Tool B is similar in many of the questions, but is designed for community-based data such as focus groups, or environmental scans that could tell you what is going on in the community. So keeping with our example maybe you have some data from a focus group you did to identify the priorities of parents, and part of that was about immunization. Or in another example of evidence, you may have some insight from a program you’ve done that sent email coaching to people who are quitting smoking. It’s not the same as tis issue but it might have some relevant info about how this could work in your community for immunization.
- Again we assess the quality of our information in terms first of relevance. If you did a focus group you did to identify the priorities of parents, and there was one point about immunization, that is somewhat relevant to your decision. If the focus group was done in another community, then you would be asking, how similar is that community to mine?
- Second we assess the methodological quality. And if you have a focus group or something similar, you will benefit from using another methods tool to really assess the quality, these questions just get you started. Are the conclusions transparent? What if the focus group report says we should use email for everything, but the evidence isn’t really clear on why that would work. Or if the report cite a lot of anecdotal experiences with vaccines, but skips over other sources, these would all be quality limitations. It’s not that we expect community evidence to align perfectly with research or other sources, but if it is saying something different, we want to ask if there’s a methodological reason for that. And then we want to ask about completeness, and whose voices are missing? Which leads into equity-informed questions
- Again, if we have focus group evidence, or evidence from a different program that also used email reminders, we want to assess the extent to which equity issues were addressed. Here we think about community engagement as part of what goes into good quality community evidence. Especially if we were thinking about a very politically charged or sensitive topic, we want to ask whether the evidence is helping us understand the perspectives of those most affected. The evidence doesn’t have to be perfect, but is it good enough that we would want to use it in our decision-making? Once complete, you would do this for each other source you are considering. Not only will you have quality assessment information for your community evidence, but you will also identify the limitations of the evidence you have and know about the gaps in evidence. You might determine that the community evidence you have is not of very good quality, and so you would be cautious about basing too much of your decision on it. You might then decide that you need to carry out a focus group would generate better quality community evidence. Or you might discover, for example, “we don’t know anything about the views of local physicians on this topic…maybe we should consult with them”.
- Evidence-informed decision making in public health uses the best-available evidence to inform decisions. In any given public health situation, the different factors may be weighted differently in making a final decision, because of the content and the quality of that evidence. You’ll notice that there’s a piece of the puzzle we haven’t touched on: Resources! Even if we decide that an emailed reminder system has evidence to support it, through research, local need and community support, we still need to consider the resource implications before we make our decision. It’s not quite ready yet, but we are developing a tool to guide the consideration of resources evidence too!
- If you need more…resources that speak to elements of quality in these types of evidence are on the second page of each tool. Each QACE tool can only cover so much…depending on the type of evidence you have, you may need to delve a little deeper to truly understand how good the quality is. Particularly with terms of methodology, you will likely benefit from using other tools specifically designed to assess methodological quality of your type of evidence.
- As I am sure you know from experience, just saying ‘We have good quality evidence that shows we should be doing X’ is not enough to make sure that happens. There are resources that can support you in the synthesize, adapt and implementation steps also, to increase the chances that the evidence you have gathered will actually have an influence in the way you hope it will from an EIDM standpoint. In Brief: Synthesize across all sources Describe quality of all evidence; Some dimensions weighted more heavily than others because of their quality (based on QACE or other tools) Unanswered questions; More evidence needed? Adapt: How to adapt the evidence for your local context Implementation: Recommended actions