Draft synthesis: Kidney outcomes with incretin receptor… — submissions
The 11 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
11 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Regulatory assessment documents were not searched
The respondent states that regulatory assessment reports frequently contain analyses that never appear in journals, and that a search limited to bibliographic databases will miss them.
The respondent proposes that regulatory documents be searched for every review in the series.
The secretariat accepts this submission in part. Regulatory assessment documents are searched. The proposal that they be treated as equivalent to a full study report is declined, because the level of detail varies and is often insufficient for risk-of-bias assessment.
Published regulatory assessment documents are now searched as a named source, are reported as a distinct evidence class in the included-studies table, and contribute to the assessment while being excluded from pooled estimates where risk of bias could not be assessed from the document.
Overlapping primary studies across included reviews are counted more than once
The draft is an overview of reviews and several included reviews share primary studies. The respondent states that the overview reports the total number of participants across reviews as though the sets were disjoint.
The respondent proposes that overlap be assessed and reported, and that no total be given without correcting for it.
The secretariat accepts this submission. An inflated participant total overstates the evidence base by an amount the reader cannot see.
Overlap between included reviews is now assessed at primary-study level and reported in a citation matrix, and participant totals are reported for the union of primary studies rather than as a sum across reviews.
A single-trial result is presented in the visual form of a pooled estimate
An outcome contributed by one trial is presented in the same forest plot format as outcomes contributed by several. The respondent states that the format carries an implication of replication that a single trial does not have.
The respondent proposes that single-trial outcomes be presented differently and labelled as unreplicated.
The secretariat accepts this submission. The form of a graphic is part of what it asserts.
Outcomes contributed by a single trial are now reported without a pooled diamond, labelled as unreplicated, and downgraded for imprecision or inconsistency according to the framework rather than presented as a synthesis.
Registered trials that never reported are not counted anywhere in the review
The respondent states that the screening flow accounts for records retrieved and excluded but does not record registered trials identified with no posted result, which are neither included nor excluded and simply disappear.
The respondent proposes that they be counted and reported as a category, with the proportion of the registered evidence base they represent.
The secretariat accepts this submission. A review that cannot say how much of the evidence base is unreported cannot say how much weight its own estimate deserves.
Registered trials without posted results are now identified, counted and reported as a distinct category in the screening flow, with the proportion of registered participants they represent stated in the limitations.
The choice of effect measure is not justified and changes the appearance of the result
The draft reports relative effects for benefits and absolute effects for harms. The respondent states that the combination flatters the intervention and that the choice should be justified or made uniform.
The respondent proposes that both relative and absolute effects be reported for every outcome.
The secretariat accepts this submission in part. Both measures are reported for every outcome where the baseline risk needed for the absolute effect can be stated. Where it cannot, the relative effect is reported alone with the reason.
Every outcome now reports the relative effect and, where an assumed baseline risk can be stated and sourced, the corresponding absolute effect, with the baseline risk and its source given in the same row.
Subgroup findings are reported that were not registered in the protocol
The draft reports differences between subgroups that do not appear in the registered protocol. The respondent states that unregistered subgroup analysis is hypothesis-generating and that the draft presents it in the same form as the pre-specified results.
The respondent proposes that unregistered analyses be removed.
The secretariat accepts this submission in part. The analyses are retained and relabelled rather than removed, because removing an analysis that was conducted leaves no record that it was.
Every subgroup analysis is now labelled as pre-specified or post hoc against the registered protocol, post hoc analyses are reported in a separate subsection without a certainty rating, and the protocol version against which the labelling was made is stated.
The conclusion is stated more strongly than the certainty rating supports
The draft rates the anchor outcome at low certainty and then states in the conclusion that the intervention is effective. The respondent states that the two sentences cannot both be true as written.
The respondent proposes that conclusion language be tied mechanically to the certainty rating, so that a low rating cannot produce an unqualified claim.
The secretariat accepts this submission. The mismatch is the failure mode the certainty framework exists to prevent.
Conclusion wording is now drawn from a fixed set of formulations tied to the certainty rating, so that a low certainty rating produces a statement that the evidence may suggest an effect and that the estimate is likely to change with further research.
Results at materially different follow-up durations are pooled
Contributing trials report the outcome at durations ranging across more than a year. The draft pools them into a single estimate. The respondent states that a mean at one duration and a mean at another are not estimates of the same quantity when the effect is still changing.
The respondent proposes that estimates be reported by duration band.
The secretariat accepts this submission. Pooling across durations assumes a plateau the contributing trials do not demonstrate.
Estimates are now reported by duration band, with the number of contributing trials and participants in each band stated, and no estimate is pooled across bands.
A sponsor trial meeting the eligibility criteria was excluded
The submission is made on behalf of the sponsor. It identifies a completed trial of the sponsor's compound that meets the stated eligibility criteria and does not appear in the included set, and supplies the trial report and the registry record.
The sponsor asks that the trial be included and the estimate recomputed. No view is expressed on the direction the recomputation should take.
The secretariat accepts this submission in part. The trial does meet the criteria and has been included. The recomputed estimate is materially unchanged, which the response states explicitly so that the outcome of the correction is on the record.
The trial is added to the included set, the estimate and the certainty rating have been recomputed, and the screening record now states why the trial was missed, which was a database indexing gap rather than a screening judgement. The submission is identified as an industry submission.
Point estimates are given without an interval
Several estimates in the draft appear as single figures. The respondent states that a point estimate without an interval invites a precision the underlying data do not support, and that the effect is worst where the estimate is drawn from a small contributing set.
The respondent proposes that no point estimate appear anywhere in the document set without its interval, including in summary tables and in the abstract.
The secretariat accepts this submission in part. Intervals are added wherever the source reports one. The proposal is declined for figures the source published without an interval, because the Institute will not compute an interval a source did not report.
Every estimate now carries its interval where the source reported one, and where it did not, the estimate is annotated as reported without an interval rather than left to appear as a precise figure.
Doses differing several-fold are pooled without examining dose-response
Contributing trials administer doses that differ by a factor of several. The draft pools them and reports one estimate. The respondent states that where a dose-response relationship exists the pooled figure corresponds to no dose that anyone receives.
The respondent proposes that estimates be reported by dose and that dose-response be examined where the data allow.
The secretariat accepts this submission. A pooled estimate across doses is an estimate for an average dose that no protocol specifies.
Estimates are now reported by dose group, a dose-response examination is reported where three or more dose levels contribute, and the pooled across-dose estimate is removed rather than retained alongside.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.