Reviewing evidence: the added value of a systematic approach to yogurt and weight
•
1 like•1,155 views
As in any evaluation of an intervention, determining the health effects of foods requires careful consideration of all the available evidence using an unbiased and transparent approach. For weight management outcomes, the evidence comes not only from randomized controlled trials, but also from a range of observational study designs. When attempting to gather the evidence on a question such as this, several challenges present. Often, unearthing the studies is the first major hurdle, as many relevant studies may not be readily available in the published literature. Another challenge is how to bring together evidence from a wide range of observational and interventional study types, whilst systematically and objectively identifying differences between studies in the quality of their methodology. These differences must be acknowledged, because they will affect the impact of the study results. Systematic reviews are commonly used to evaluate and synthesise the body of evidence on a topic, and are guided by an a priori protocol. They are well-established in evidence-based medicine and health care, social and economic evaluations and can be a useful tool in revealing and/or overcoming the challenges described above. A recent systematic review of the consumption of yoghurt for weight management provides a good example of the application of the protocol-driven approach. This example demonstrates how systematic review methods can bring objectivity and transparency to a review, and shows how challenges, such as heterogeneity between studies and confounding factors, can be identified. It also demonstrates the usefulness of recommendations for the design and reporting of futures studies in the topic area.
Recommended
More Related Content
Viewers also liked
More from Yogurt in Nutrition #YINI
More from Yogurt in Nutrition #YINI (20)
Recently uploaded
Recently uploaded (20)
Reviewing evidence: the added value of a systematic approach to yogurt and weight
- 1. Providing Consultancy & Research in Health Economics Dr Jacqui Eales Reviewing evidence: the added value of a systematic approach to yoghurt and weight
- 2. Disclosure: Dr Jacqui Eales The work carried out in the example was funded by Danone Institute International, conducted by an independent organisation, York Health Economics Consortium and completed in February 2015
- 3. Does an intervention work? ...what are the effects? For who?...
- 4. Good reviews are challenging to do… Problem
- 6. yogurt consumption & weight management An overview of the available evidence
- 7. Bone Health Weight Management and Nutritional Health Metabolic Health Cardiovascular Health Gastrointestinal Health Cancer Diabetes Other First… a scoping review
- 8. on a range of outcomes around weight management… Review question in the general, apparently healthy, adult population? What are the effects of yoghurt (containing the symbiotic cultures S. thermophilius and L. delbrueckii subsp. Bulgaricus)…
- 9. yoghurt, yaourt, yoghourt, jogourt streptococcus thermophilus lactobacillus delbrueckii Search strategy
- 10. Bone Health Weight Management and Nutritional Health Metabolic Health Cardiovascular Health er a priori inclusion criteria + multiple reviewers 22 Relevance assessment
- 11. Minimising subjectivity… ...using standardised and tailored quality assessment tools Quality assessment ? ? ? ? ? – + + + + + + + + + + + + Risk of Bias A B C D E F Mean Difference IV, Random, 95% CI -4 -2 0 2 4 ours yoghurt Favours control Study 1 Study 2 Study 3
- 12. Added value of reporting on all data Analysing the evidence % CI 33] 23] 11] 23] ce ? ? ? ? ? – + + + + + + + + + + + + Risk of Bias A B C D E F Mean Difference IV, Random, 95% CI -4 -2 0 2 4 Favours yoghurt Favours control 95% CI 1.23] .11] 26] nce + + + + + + + + + + + + Risk of Bias A B C D E F Mean Difference IV, Random, 95% CI -4 -2 0 2 4 Favours yoghurt Favours control
- 13. Final report IV, Random, 95% CI 0.30 [-1.73, 2.33] -1.40 [-4.03, 1.23] -1.64 [-3.17, -0.11] -0.99 [-2.21, 0.23] Mean Difference ? ? ? ? ? – + + + + + + + + + + + + Risk of Bias A B C D E F Mean Difference IV, Random, 95% CI -4 -2 0 2 4 Favours yoghurt Favours control Study 1 Study 2 Study 3
- 14. Reviewing evidence… difficult to be robust Added value of systematic reviews: comprehensive subjectivity transparent combined strength of evidence
- 15. Providing Consultancy & Research in Health Economics http://tinyurl.com/yhec-facebook http://twitter.com/YHEC1 http://www.minerva-network.com/ http://tinyurl.com/YHEC-LinkedIn Thank you Systematic Review of Yoghurt for Weight Management Outcomes. Eales, J.; Edwards, M.; King S.; Wood, H.; Glanville, J. February 2015 Dr Jacqui Eales jacqui.eales@york.ac.uk www.yhec.co.uk
- 16. Flow chart: from scoping search results to included studies
Editor's Notes
- In this talk I will give an overview of how systematic reviewing methods provide one of the best ways we can understand the evidence base on a topic. Some of you may be familiar with systematic review approach, but many others will not, and this forum provides an opportunity to clarify what SRs can offer, as a reviewing tool A different approach General picture of the evidence Added value of a SR Strength of evidence Add in questions
- Notes: Disclosure slide- a requirement for all the EB conference speakers
- In our various sciencfic endeavours, we often want to know how well something works, or what its effects are, and in the health sciences, for what populations an intervention is effective. We could answer these questions by going ahead and undertaking good quality primary research-and this is often the go-to option for many researchers … however, the first step should always be to find out if the question has already been answered… For example, has the research already been undertaken and the question answered in a high quality study by someone else? Have several research groups undertaken well-conducted studies and come up with similar results? In this case, we could save ourselves time, effort and money, by instead, concentrating our primary research effort on another question. Or perhaps there is a pool of published studies showing conflicting results.. In this scenario, further, quality primary research is required, and we should go ahead and conduct our research accordingly. Clearly, a thorough, unbiasd review and evaluation is needed to ascertain what the current evidence base is
- However, this is easier said than done! There are lots of challenges involved in collating and assessing current evidence! Many people simply don’t realise the depth of these challenges when starting out to write a review of evidence. Firstly, the research studies which are relevant to the topic can be hard to find (or it take a long time to do so)- how do we search for the evidence without wasting too much time? Some studies which are found by the search may not actually be relevant to the topic- we need some way of selecting the relevant ones, without being subjective about it And some of the relevant studies may have used poor quality methodology and therefore their results subject to bias or inaccuracies- need some way of determining this, again, without being subjective about it The relevant studies may include lots of different study designs (RCTs, single arm studies, cross-sectional studies, case studies)- and its often difficult to combine and report the evidence across these different study types So what does the reviewer do in light of these challenges?
- Well, these problems can be tackled by taking a systematic approach to looking at the evidence. Systematic reviewing is a methodology well-known to many in the medical and pharmacological sciences but is less well known outside of these spheres. We wanted to give this talk to raise awareness of the approach, and its relevance to the field of nutritional science. Systematic Reviews follow a standard procedure which ensures that they are carried out to highly standardised, rigorous and transparent methods. Notes: How can this reviewing method overcome the problems we have just pointed out? If you’re having difficulty in finding studies...The method use a pre-defined, pre-tested systematic way of searching databases that maximises the chances of capturing all available evidence, in minimum time If you’re concerned about subjectivity creeping in... They require all decisions abot which studies to include to be undertaken to pre-defined inclusion criteria. The quality of the studies’ methodology is also assessed using specifically designed quality assessment tools. And multiple reviewers will make these assessments, minimising the chances of subjectivity. What about combining data across studies?....Studies are assessed for their combinability- and pooled in meta-analysis or brought together in a structured narrative if not. Guiding all of this is a protocol, set out a priori and providing a plan for the review. Having this protocol maximises the transparency of the approach, and is not a standard proceedure in non-systematic reviews.
- So, SRs give a “comprehensive picture” of all the available evidence, adding value in its transparent, objective approach. Also of added value is the combination of data from different study types- often reviews can exclude study types of “lower quality evidence”- observational studies. The SR approach does not exclude a study on the basis of its risk of bias, but reports all evidence found, and reports the quality of it. But how exactly can this reviewing method overcome the problems I pointed out a couple of slides ago? The recent review of yoghurt consumption and weight management is a good example of this different approach to evaluating evidence. I’m going to walk through the key stages of the systematic review method, using our example to demonstrate how challenges associated with producing an objective review are addressed. Notes: The review was commissioned by DII, and undertaken by an independent organisation, York Health Economics Consortium, who specialising in providing systematic reviewing and health economics services . The report from the review was finalised in February 2015
- A common pre-curser to a systematic review is a scoping review, undertaken when you want to assess the characteristics of a wide evidence base (where many studies are published for different outcomes), before settling on which part of it to focus on. Scoping can help planning of the subsequent SR, in terms of time and resources, and of course, whether it is worth doing in the first place (is there evidence out there). In the scoping review for our example, we searched the literature to see how many studies were published on yoghurt consumption and a range of health outcomes, summarised here. As you can see, weight management and nutritional health held the highest numbers of studies. There were also a large proportion of RCTs in this chunk of studies, indicating that this area may provide studies of this highest quality research design. Note: population was wider- included children and elderly CHANGE PICTURE
- Having decided on the outcome likely to contain the most studies and of higher quality design, the next step is to set the review question- it seems obvious, and tedious (!), but it’s really important to get the wording right as guides the following parts of the review. Three important terms are the: Intervention, the treatment in studies- yoghurt (defined according to codex alimentarius- international standardised definition of foodstuffs) Outcomes being measured for evidence of effect- weight management outcomes- BMI, body weight, %fat. These are surrogate markers of body composition. Population to whom the intervention is applied- general adult population, with no obvious diseases The question is clearly set out in a protocol for the review- so there cant be any post-hoc mind changing. The protocol sets out a plan for the review, in all the stages I am about to describe, and is agreed on by all members of the review team. This systematic review, like most others, was conducted by multidisciplinary team, who all contribute in their own specialisms to the protocol and review. This contrasts with a non-systematic or standard review, where the is no requirement for a protocol and undertaken by only one or two researchers.
- So, how do you go about trying to find the studies in your topic area? Systematic reviews use a pre-defined, pre-tested systematic way of searching databases that maximises the chances of capturing all available evidence, in minimum time. Standard or traditional reviews rarely achieve the numbers of studies that systematic reviews do, usually because they do not have technical support to design search strategies that SRs do. In the yoghurt and weight management review, information specialists designed a searching strategy that used a list of search terms based on the intervention element of the review question- (different synonyms of yoghurt). In fact, we used the searching strategy used for the scoping review, and updated it to include any more recent studies published since the scoping review. In other systematic reviews, where a larger amount of evidence exists, search terms relating to more elements of the question (outcomes and population elements) are often added. The search was trialed and then implemented in a range of databases including journals related to nutrition, health and weight management, aiming to get the maximal number of relevant studies mixed in with the minimum number of non-relevant ones. Grey literature searches (such as proceedings of relevant conferences) were also undertaken. Subject experts were consulted to ensure that the databases being searched and the search terms themselves were appropriate and comprehensive. In the final report, the search strategy (sources and the search terms used) were fully provided, so it can be updated in the future.
- As I mentioned towards the beginning of my talk, there is opportunity for subjectivity to creep in at the stage where studies are assessed for their relevance to the topic..... SRs require all decisions about which studies to include to be undertaken to pre-defined inclusion criteria. And multiple reviewers will make these assessments, minimising the chances of subjectivity. In the example review, we went from over 18,000 returned search hits to 22 relevant studies To make this an efficient process, and to minimise subjectivity, two reviewers independently assessed the studies for relevance according to pre-defined inclusion criteria set out in the protocol. This was done over a number of stages to maximise efficiency: by assessing titles, abstracts only and finally, the full text of studies. Inclusion criteria were, again, based on the elements of the question, we only included studies that assessed yoghurt consumed by a non-diseased, adult population, that measured weight management outcomes. We also specified that the studies must also have some kind of comparator group (group of adults that consumed either no yoghurt or a control/placebo product). As you will be aware, having this group is vital for providing comparative evidence of an effect in the outcome measure. In fact, the inclusion criteria we used were a bit more prescriptive than I have just described, but I have skipped over the details, because they are quite tedious! Note: Title and abstract was by 2 reviewers, full text by one reviewer and was checked by another..
- The next step is to assess the methodological quality of the studies- identifying any risks of bias within the study design. In our example, one reviewer undertook the assessment (using standardised checklists or specifically designed tools to detect differences in the way that studies were undertaken). A second reviewer checked the quality assessment, with any disagreements discussed and if necessary a third reviewer brought in until consensus was reached. It’s important to underline again that studies at high risk of bias are not discarded at this point, but the risk of bias assessment is used in later stages where the data is analysed, combined or discussed. Meta-analysis is capable of determining how risk of bias this can affect study results, using sensitivity analyses- which can be very informative. It is useful to know the range of the evidence quality, to identify particular criteria which are not being reached and to use the information to make recommendations for future study conduct Notes Study-type specific tools were defined a priori (where possible) in the review protocol. Quality assessment of the internal and external validity of eligible studies was undertaken using the Cochrane Risk of Bias tool for RCTs, controlled trials and cross-over trials, the Centre for Reviews and Dissemination (CRD) for cohort studies (5) and the Newcastle-Ottawa Quality Assessment Scale adapted for cross-sectional studies.
- What about combining data across studies? ....in SRs, studies are assessed for their combinability. Studies which can be combined are pooled in meta-analysis or brought together in a structured narrative if not. Though other reviews can of course undertake meta-analysis, they generally lack procedures for minimising bias and subjectivity, so there is a strong possibility that the studies they include are not representative of the actual evidence base. In our review of yoghurt and weight management outcomes, one researcher extracted the data and information from the full document of each of the studies into a spreadsheet, and a second researcher checked the extraction. Where appropriate, data was combined in forest plots. The two plots shown here demonstrate how the pooling of results from multiple studies of low risk of bias can reveal the combined strength of the evidence base. Green squares represent RCTs, size of box indicates the study size and the length of the arms, the CI for the effect. Squares over to the left indicate an effect of yoghurt, and to the right indicate an effect of the control for weight management. The overall effect is represented by the black diamond at the bottom of the plot. The upper forest plot indicates greater body weight loss in participants consuming yoghurt compared with the comparator group but the effect size was not significant. The lower forest plot has excluded the study with a high risk of bias, the summary effect size reached statistical signficance- you can see the diamond is fully within the “favours yoghurt” domain. This demonstrates the value of using a standardised and transparently reported method of quality assessment. Being able to pinpoint studies of low quality, justifies the removal of these studies from meta-analyses, where their results can cloud the data. Where meta-analysis of data was not possible, a structured narrative report was written. This gives a written description of the study results within the context of other studies, and enables “triangulation” (or not!) of supporting evidence. In this review, the outcome effect reported by cross sectional studies that yoghurt consumers had a significantly lower BMI, and significantly smaller WC and lower risk/prevalence of overweight/obesity than low- or non-consumers is an example of evidence from an observational study type supporting evidence on another weight management outcome (body weight) derived from meta-analyses of RCTs. Notes: The potential for study data to be pooled in meta-analysis was assessed, based on the comparability of their populations, interventions, comparators and outcomes. In both standard analysis and sensitivity analysis, heterogeneity was low for both the fixed-effect and random-effects models (14% and 0% respectively). The summary effect sizes were similar in the fixed-effect model (Z = -1.80; p=0.07 and Z = -2.34; p=0.02 respectively). When required information was not presented in a document, the study authors were contacted and requested to supply that data. Only some of the RCTs provided data suitable for meta-analysis. Lack of data reporting was a factor preventing the meta-analysis of cohort and cross-sectional studies. It is also possible to combine data from study arms reported by different study types, however, this was not the case for this review.
- Guiding all of this is a protocol, set out a priori and providing a plan for the review. Having this protocol maximises the transparency of the approach, and is not a standard procedure in non-systematic reviews. The entirety of the work involved in the review, including the methods and rationale behind the methods used to search for, appraise and combine studies is presented in a final report.
- So although there are many challenges when undertaking a review of evidence, the methods followed by SRs aim to overcome them. They try to include all available evidence in well-designed search strategies. They aim to reduce subjectivity and maximise transparency by setting out an apriori protocol and using multiple reviewers and checking stages. Employing an objective party to undertake a review is one way to add an extra layer of transparency to the process. Finally, by combining and reporting data of high quality, SRs can justify sensitivity analysis and use triangulation of evidence from different study types in narrative syntheses to support conclusions.
- Extra info not to include Studies not reporting the strains of yoghurt in the intervention- not 100% sure it is the type of yoghurt we are interested in Many different comparators: Comparators should be similar between studies- should be no yoghurt or a placebo if not possible, or two separate interventions, each acting as a control Outcome measurement not by clinicians- Measurement of outcomes should be undertaken by clinicians Lack of long term studies- most less than a year High risk of bias (partly due to the above, also reporting quality) Confounding factors- e.g. socioeconomic status High heterogeneity of studies- difficult to compare between them- heterogeneity in study design, populations, treatments, controlsand outcomes– this is par for the course when including a wide range of studies, and also in a topic like this- so need to take the individual study results into account separately, as well as a combining narrative Publication bias- usually a funnel plot can demonstrate publication bias, but not enough studies in this case