Draft synthesis: Acute pancreatitis with incretin-based… — submissions
The 10 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
10 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
The document is unreadable without specialist training
The respondent, a trustee of a patient organisation, states that the draft is written for a reader who already understands certainty grading, and that the people most affected by the subject matter will not reach the assessment at all.
The respondent proposes a plain-language summary at the head of every document, written to the same standard of accuracy as the document itself and not as a promotional abstract.
The secretariat accepts this submission in part. A plain-language summary is added. The proposal that it replace the technical abstract is declined, because the abstract is the part of the document other assessors read and cite.
Every document now opens with a plain-language summary of not more than 150 words, placed above the technical abstract and carrying the same certainty language, so that the two cannot diverge.
Baseline imbalance in a small contributing trial is not remarked on
The respondent identifies a small contributing trial with a baseline difference in the outcome variable, and states that the difference is large enough to account for a share of the reported effect.
The respondent asks that the trial be excluded or that the imbalance be addressed.
The secretariat notes this submission. The imbalance is recorded in the risk-of-bias assessment for that trial, in the domain concerning the randomisation process, where the respondent may not have looked.
No amendment arises. The judgement and the supporting figures are already reported at domain level for that study, and a sensitivity analysis omitting it is reported in the results, which does not change the direction of the estimate.
The review question is drawn too broadly to be answerable
The respondent states that the population as registered spans groups in which the intervention would be expected to behave differently, and that a single estimate across them is uninformative.
The respondent proposes that the question be split and the review re-registered.
The secretariat does not accept this submission. The breadth was registered before screening, the pre-specified subgroups address the variation the respondent identifies, and re-registering after the results are known would convert a pre-specified analysis into a post hoc one.
The question stands as registered. The point is recorded in the limitations, and the assessment committee has noted it for the protocol of the next version of this review, which will be registered before the search is run. The submission remains published in full.
Statistical heterogeneity is treated as though it measured clinical heterogeneity
The draft reports a heterogeneity statistic and proceeds to pool where it is low. The respondent states that a low statistic in a small set of trials is uninformative, and that clinical and methodological similarity should be assessed before any statistic is consulted.
The respondent proposes that the decision to pool be justified on clinical grounds first and that the statistic be reported as a description rather than used as a threshold.
The secretariat accepts this submission in part. The decision to pool is now made on clinical and methodological grounds and stated as such. The statistic continues to be reported, because readers expect it and its absence would be read as concealment.
The decision to pool is now justified in prose against the population, intervention, comparator and outcome before any statistic is presented, and the heterogeneity statistic is reported with its confidence interval and a note of the number of contributing studies.
The document should not describe uses outside the approved indication
The submission is made on behalf of a marketing-authorisation holder. It states that the draft describes uses of the compound that fall outside the approved indication, that such uses are not supported by the sponsor, and that describing them may be read as legitimising them.
The sponsor asks that the sections concerned be removed, or in the alternative that they carry a prominent statement that the sponsor does not support such use.
The secretariat does not accept this submission. The Institute records what is supplied and what is used, because a reader encountering a compound outside a regulated route is the reader most in need of an assessment of the evidence for it.
The sections remain. Every such section already states the regulatory status of the use described and states that the evidence for it is assessed separately from the evidence for the approved indication. The submission is published in full and identified as an industry submission.
An indirect comparison is presented without an assessment of transitivity
The draft compares two interventions through a common comparator. The respondent states that the trials contributing to each side differ in background therapy and in baseline severity, and that the resulting estimate assumes a similarity the draft does not test.
The respondent proposes that transitivity be assessed explicitly and reported before any indirect estimate is presented.
The secretariat accepts this submission. An indirect estimate presented without a transitivity assessment invites a reader to treat an assumption as a result.
Transitivity is now assessed against a stated list of effect modifiers and reported before any indirect estimate, and where the assessment fails the contrast is reported as unestimable with the reason rather than estimated with a caveat.
Risk-of-bias judgements are reported as an overall rating without the domains
The draft reports a single overall risk-of-bias judgement per study. The respondent states that the overall judgement conceals which domain drove it, and that a reader assessing whether the concern applies to their question needs the domain.
The respondent proposes a domain-level table with the supporting quotation for each judgement.
The secretariat accepts this submission. The overall judgement is a summary of the domains and publishing only the summary makes it uncheckable.
Risk of bias is now reported at domain level for every included study, with the text on which each judgement was based quoted and referenced, and the overall judgement derived from the domains rather than asserted alongside them.
The document set should be published in translation
The respondent notes that the assessments concern compounds supplied internationally and that publishing only in English restricts access to the assessment to readers who work in it.
The respondent proposes machine translation of the document set as an interim measure, with human review of the certainty language.
The secretariat does not accept this submission, and records that the underlying point is sound and that the proposed remedy is the difficulty.
A translation whose certainty language has drifted is a different assessment carrying the Institute's name, and the Institute cannot review translations it does not have the capacity to review. The documents remain in English. The submission is published in full because the access problem it identifies is real and unresolved.
Quantitative claims are reproduced without the method that produced them
Several figures in the draft are quoted from sources that determined them by different methods. A figure obtained by one determination and a figure obtained by another are not comparable, and the draft places them in the same sentence without distinguishing them.
The respondent, an analytical chemist, proposes that every quantitative claim carry the method that produced it at the point of use rather than in the reference.
The secretariat accepts this submission. Placing two figures side by side is an implicit claim that they are the same kind of quantity, and in the cases identified they were not.
Every quantitative claim now carries the determination that produced it at the point of use, and figures obtained by non-comparable methods are no longer presented in the same row or sentence.
A superseded version should remain reachable from the version that replaced it
The respondent states that the draft supersedes an earlier document and that a reader who cited the earlier version has no way to reach it from the new one, which makes it impossible to see what changed.
The respondent asks that every version carry a link both to what it supersedes and to what supersedes it.
The secretariat notes this submission. The corrections and versioning policy already requires bidirectional version links and every superseded document is retained at its own address.
No amendment arises. The requirement is stated in the corrections and versioning policy and the amendment log of this document links to the version it replaced. The respondent is correct that the link was absent from the draft page furnished for consultation, which was a defect of the consultation copy and not of the policy.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.