- Background – the answer to this question is unknown.
- Methods - We searched for evidence, did some word-heavy methodology (that I don’t understand and gloss over) and used the GRADE system (which I’ve forgotten too many times).
- Results – Some RCTs were found but they all looked at different things and the level of evidence was poor. We did find X and Y (but putting X and Y together the conclusion may or not make sense e.g. we found a higher rate of amputations but no reduction in mobility).
- Conclusions – A repeat of the main results.
What follows in the body of the paper is generally something I don’t really understand, other than the introduction, the discussion and maybe a Forest plot in-between.
The reason for this rant is that we’re re-considering whether we should start using subglottic suction (we don’t currently) and there’s this meta-analysis that’s just been published. In a fit of enthusiasm I thought I might try and de-code the methods….
The protocol for the review was published on PROSPERO, which is a database of health related systematic reviews. The advantages of using PROSPERO are that someone else can see if a review on a topic has already been done / is underway, and a comparison can be made between what was planned and what was produced. The entry for this meta-analysis can be found here. It all looks pretty lingo-free other than the ‘risk of bias assessment’ and ‘strategy for data synthesis’ sections:
Risk of bias assessment:
The tool they used for the review was the ‘Cochrane collaboration’s tool for assessing the risk of bias’. There’s a paper about that tool here, and the results of that are in figure 2 of the paper. Looks straightforward, at least in theory.
The protocol then talks about using MINORS grading for non-RCTs, however the protocol also says RCTs only will be included – not sure what that’s about (I’ve checked and the paper does only include RCTs).
It then reports that the GRADE system will be used in the article. The GRADE system doesn’t look at an individual study, but instead looks at all the available evidence about a particular outcome. There’s a link to more information here but essentially it’s a guided judgement call about the strength of the evidence with four possible outcomes. In the paper, table 2 is where to find the GRADE scores for each of the outcomes to do with subglottic suction.
Essentially therefore, they will use two standard scoring tools to look at each paper (Cochrane) and then each outcome (GRADE).
This part explains the majority of the systematic review part of the study. The meta-analysis is described in the next section….
Strategy for data synthesis:
Not the most readable. It starts with “We will provide a narrative synthesis of the findings from the included studies, structured around the type of intervention, target population characteristics, type of outcome and intervention content.” which could be translated to they will produce a written summary of the studies! (I remain jealous of people who can write ‘fluent science’ though, and the fact that the authors are Chinese doesn’t make me feel any better).
They will use risk ratios for dichotomous outcomes. A dichotomous outcome is when A OR B happens. There is no C outcome, or A AND B. If you code A and B as 0 and 1 it becomes a binary outcome (pedantic stats pub trivia!). Risk ratio is the ratio of how often an event occurs in one group compared to another (i.e. pneumonia in those getting subglottic suction vs those not).
For non-dichotomous outcomes, they will use something called standardised mean differences. This method standardises results to correct for differences in the way they were measured, and is done by dividing the difference in means by the standard deviation of the measurements. For example, a study compares travelling speed down the M1 vs the M25. They use mph as the unit and find the mean difference is 10mph. To standardise that, 10 is divided by the SD of all the speed measurements. Doing that would allow comparison between a different study that used km/hr as the unit of measurement. For more info about standardised means click here.
The next section talks about combining results from several studies to calculate an improved result where possible (making it a meta-analysis rather than a systematic review). To do this they plan on using a random effects meta-analysis.
There are two methods of doing a meta-analysis – fixed effect and random effect. A fixed effect model assumes that the effect of the intervention is the same in all studies, and studies only come out with different results because they are talking different samples (i.e. they are different by chance). The alternative is a random effects model. This assumes that the outcome from the intervention is not the same in different trials – this could be because the population is different, the intervention differed slightly, or the method of measurement and analysis was different etc. The different results seen between studies are in part because of chance (different samples), but also because of these differences (referred to as trial heterogeneity).
To quantify the effect of chance and hererogenity, something called an 'I squared' statistic is used. The result comes out as a number which is the percentage of the differences between results that are due to heterogeneity. I think (although am not sure) the difference between fixed and random effect meta-analysis in practice is down to how you use the stats software.
The punchline is that it’s worth looking at whether the use of fixed or random effect seems valid for a particular meta-analysis, and to remember that if a fixed effects model is used the final result is interpreted as the effect of the intervention, whereas if the random effects is used it is an average effect across similar but different populations.
The next sentence is “we will conduct sensitivity analysis based on study quality”. I’m not sure what this means (and the methods bit of the paper doesn’t make it clearer), but I think it means that if they find the studies identified by the literature review aren’t what they hoped that might alter how they do the meta-analysis i.e. they will do the best they can with what they can find??? If anyone can explain this please do so in the comments.
Finally, they talk about doing stratified meta-analysis to explore heterogeneity in effect estimates. This means that they will repeat the meta-analysis including, for example only the high-quality studies, or certain patient groups etc. We see this quite a bit and not just in meta-analysis, for example in ARDS studies where a meta-analysis doesn’t show a difference so the result is repeated looking at only patients with severe (as was) ARDS.
In the last sentence, they drop in that they’ll assess evidence of publication bias. They don’t say how, but frankly at this point I’m thankful (in the paper they say they looked at a funnel plot – for more info about that click here).
If you’ve read the paper another technique appears in the method - ‘trial sequential analysis’. If you’ve got this far you will probably agree that TSA should be left until a later post. I do sometimes worry that trial design will get so complicated that only certain academics will be able to understand it. Whilst the research would be ‘scientifically faultless’ there would be something lost if as clinicians we can’t understand how the results were reached.
The original question (if you remember!) was whether we should we start using sub-glottic suction? Based on this I’m not convinced we should. We don’t see a lot of early VAP in our unit (we’re non-neurosurgical). Even if ventilator days are reduced, the difference looks to be small (1 day), and there is only a moderate GRADE quality associated with the finding. There doesn’t seem to be any other benefit.
Congratulations if you’ve come to the end of this – shame on you if you skipped the middle part. Please use the comments below to correct anything I might have got wrong – I think it’s all OK but if you know otherwise….