Indicators of study quality in systematic reviews of qualitative research to inform public health policy making Presented at the 2nd European Conference on Qualitative Research for Policy Making, 26-27 May 2011, Belfast, UK Organised by Merlien Institute

1. Qualitative research and policy making
May 2011. Belfast
Ruth Garside
Senior Research Fellow
PenTAG, Peninsula Medical School, University of Exeter
Ruth. Garside@pms.ac.uk

2. Session plan 75 mins
Background to our experience: 10 mins
Group work: What does good quality research and its
reporting look like? 10mins
Checklist examples from Public Health Policy Making
in England 10 mins
Break out practical session and feedback: 35 mins
Round up (and a suggestion!) (10 mins)

3. Our experience
Working with CPHE NICE in the UK
Producing systematic reviews of qualitative research
Tendency to assume similar concerns as for quantitative
research

4. What makes qualitative research
“good quality”?

5. p.8

11. Alternative criteria
Wallace A, Croucher K, Quilagars D, Baldwin S.
Meeting the challenge: developing systematic
reviewing in social policy. Policy and Politics 2004;
32(4):455-470.

12. Question Is the research question clear? E/D
1 Theoretical Is the theoretical or ideological perspective of the author (or funder) D
perspective explicit?
2 Has this influenced the study design, methods, or research findings?
3 Study design Is the study design appropriate to answer the question? E
4 Context Is the context or setting adequately described? E
5 Sampling Is the sample adequate to explore the range of subjects and E
6 settings?
Has it been drawn from an appropriate population?
7 Data collection Was the data collection adequately described? E
8 Was it rigorously conducted to ensure confidence in the findings?
9 Data analysis Was there evidence that the data analysis was rigorously conducted E
to ensure confidence in the findings?
10 Reflexivity Are the findings substantiated by the data and has consideration D
been given to any limitations of the methods or data that may have
affected the results?
11 Generalisibility Do any claims to generalisibility follow logically and theoretically from D
the data?
12 Ethics Have ethical issues been addressed and confidentiality respected? D
13 Question Is the research question clear? E
14 Theoretical Is the theoretical or ideological perspective of the author (or funder) D
perspective explicit?
15 Has this influenced the study design, methods, or research findings?
16 Study design Is the study design appropriate to answer the question? E

13. Small group work
Are there any challenges with using the
checklists provided?
Do you agree that the questions they are asking
will assess “quality”? Why?/Why not?
What would you use?

14. My preferred approach
Technical aspects
Trustworthiness
Theoretical considerations
Practical considerations
Adapted from: Popay J, Using Qualitative Research to Inform Policy and Practice. ONS, Cardiff: April 2008.

15. 1. Technical aspects
Y/P/N Comments
1. Is the research question(s) clear?
2. Is the research question(s) suited to qual. enquiry?
Are the following clearly described?
3. Context
4. Sampling
5. Data collection
6. Analysis
Adapted from: Dixon-Woods M, Shaw RL, Agarwal S, Smith JA. The problem of appraising qualitative
research. Qual Saf Health Care 2004; 13:233-225

16. 2. Trustworthiness
For example:
Are the design and execution appropriate to the
research question?
What evidence of reflexivity is there?
Do the voices of the participants come through?
Are alternative interpretations, theories etc explored?
How well supported by the data are any conclusions?
Are ethical considerations given appropriate thought?
etc.

17. 3. Theoretical considerations
For example:
Does the report connect to a wider body of knowledge
or existing theoretical framework; and, if so
Is this appropriate (e.g. not uncritical verification);
Does the paper develop explanatory concepts for the
findings
etc.

18. 4. Practical considerations
Not “is this research valid?” but rather “what is this
research valid for?”
For example
Does this study usefully contribute to the policy
question?
Does this study provide evidence relevant to the policy
setting?
Does this study usefully contribute to the review?
Aguinaldo JP. Rethinking Validity in Qualitative Research from a Social Constructionist Perspective:
From "Is this valid research?" to "What is this research valid for?". The Qualitative Report 2004; 9(1):127-136.

19. Presented at the 2nd European
conference on Qualitative Research for
Policy Making, 26 -27 May 2011, Belfast
For more information
Please visit: http://www.merlien.org