Draft synthesis: Mitochondria-targeted peptides in… — submissions
The 12 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
12 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Statistical heterogeneity is treated as though it measured clinical heterogeneity
The draft reports a heterogeneity statistic and proceeds to pool where it is low. The respondent states that a low statistic in a small set of trials is uninformative, and that clinical and methodological similarity should be assessed before any statistic is consulted.
The respondent proposes that the decision to pool be justified on clinical grounds first and that the statistic be reported as a description rather than used as a threshold.
The secretariat accepts this submission in part. The decision to pool is now made on clinical and methodological grounds and stated as such. The statistic continues to be reported, because readers expect it and its absence would be read as concealment.
The decision to pool is now justified in prose against the population, intervention, comparator and outcome before any statistic is presented, and the heterogeneity statistic is reported with its confidence interval and a note of the number of contributing studies.
A superseded version should remain reachable from the version that replaced it
The respondent states that the draft supersedes an earlier document and that a reader who cited the earlier version has no way to reach it from the new one, which makes it impossible to see what changed.
The respondent asks that every version carry a link both to what it supersedes and to what supersedes it.
The secretariat notes this submission. The corrections and versioning policy already requires bidirectional version links and every superseded document is retained at its own address.
No amendment arises. The requirement is stated in the corrections and versioning policy and the amendment log of this document links to the version it replaced. The respondent is correct that the link was absent from the draft page furnished for consultation, which was a defect of the consultation copy and not of the policy.
Overlapping primary studies across included reviews are counted more than once
The draft is an overview of reviews and several included reviews share primary studies. The respondent states that the overview reports the total number of participants across reviews as though the sets were disjoint.
The respondent proposes that overlap be assessed and reported, and that no total be given without correcting for it.
The secretariat accepts this submission. An inflated participant total overstates the evidence base by an amount the reader cannot see.
Overlap between included reviews is now assessed at primary-study level and reported in a citation matrix, and participant totals are reported for the union of primary studies rather than as a sum across reviews.
The review does not say when it will be updated or what would trigger an update
The respondent states that a synthesis in a field where trials report frequently is out of date from the day it is published, and that a reader has no way to know whether the version in front of them is current.
The respondent proposes a stated trigger for an update as well as a scheduled review date.
The secretariat accepts this submission. A review date alone tells a reader when the document will be looked at, not whether it should already have been.
Every synthesis now records a scheduled review date and a stated update trigger, being the reporting of a trial that would materially change the contributing evidence, and the registered trials capable of triggering an update are listed by name.
The document set should be published in translation
The respondent notes that the assessments concern compounds supplied internationally and that publishing only in English restricts access to the assessment to readers who work in it.
The respondent proposes machine translation of the document set as an interim measure, with human review of the certainty language.
The secretariat does not accept this submission, and records that the underlying point is sound and that the proposed remedy is the difficulty.
A translation whose certainty language has drifted is a different assessment carrying the Institute's name, and the Institute cannot review translations it does not have the capacity to review. The documents remain in English. The submission is published in full because the access problem it identifies is real and unresolved.
Subgroup findings are reported that were not registered in the protocol
The draft reports differences between subgroups that do not appear in the registered protocol. The respondent states that unregistered subgroup analysis is hypothesis-generating and that the draft presents it in the same form as the pre-specified results.
The respondent proposes that unregistered analyses be removed.
The secretariat accepts this submission in part. The analyses are retained and relabelled rather than removed, because removing an analysis that was conducted leaves no record that it was.
Every subgroup analysis is now labelled as pre-specified or post hoc against the registered protocol, post hoc analyses are reported in a separate subsection without a certainty rating, and the protocol version against which the labelling was made is stated.
Declared interests should appear on the document rather than on a separate page
The draft links to a central conflicts register. The respondent argues that a reader assessing whether to rely on a particular document should not have to leave it to find out who assessed it and what they declared.
The respondent proposes that the interests of every named contributor to a document be printed on that document.
The secretariat notes this submission and records that the draft already provides for it, which the respondent could reasonably have missed because the provision sits in an appendix.
Every document carries the declared interests of its named contributors in its front matter, and the central register exists so that a reader can see a person across all documents rather than one at a time. No amendment arises; the provision has been moved from the appendix into the body of the methodology document so that it is findable.
Studies using different outcome definitions are pooled into one estimate
The contributing studies define the outcome in at least two ways, and the draft pools them. The respondent states that the resulting estimate is an average across definitions rather than an estimate of any one quantity.
The respondent proposes that studies be pooled only within a definition, and that the definitions be reported separately with their own certainty ratings.
The secretariat accepts this submission. Pooling across definitions produces a figure with no referent, and the draft did it without saying so.
Studies are now pooled only within an outcome definition, each definition is reported with its own estimate and certainty rating, and the summary of findings states which definition each row concerns.
The review protocol is described but its registration record is not linked
The respondent asks for the prospective registration record of the review protocol so that the registered outcomes can be compared with the reported ones.
The respondent states that without it the claim of prospective registration cannot be checked.
The secretariat accepts this submission in part. The protocol is published in full on the Institute site with its version history, which permits the comparison the respondent asks for. Where an external registration identifier is not held by the Institute, the review says so rather than supplying one.
Each synthesis now links its own protocol with the version in force at screening, and states explicitly where an external registration identifier is not held, so that the absence is a recorded fact.
The same concept is given three different names in one document
The draft refers to the same quantity as a response rate, a responder proportion and a categorical outcome in different sections. The respondent, who works in health-technology assessment, states that a reader cannot tell whether the three refer to one thing or to three.
The respondent proposes that the glossary term be used at every occurrence and that the glossary entry be linked at first use in each section rather than only at first use in the document.
The secretariat accepts this submission. The variation was stylistic and its cost to the reader exceeds any benefit.
A single term is now used throughout for each defined concept, and the glossary entry is linked at first use within each section rather than once per document.
A sponsor trial meeting the eligibility criteria was excluded
The submission is made on behalf of the sponsor. It identifies a completed trial of the sponsor's compound that meets the stated eligibility criteria and does not appear in the included set, and supplies the trial report and the registry record.
The sponsor asks that the trial be included and the estimate recomputed. No view is expressed on the direction the recomputation should take.
The secretariat accepts this submission in part. The trial does meet the criteria and has been included. The recomputed estimate is materially unchanged, which the response states explicitly so that the outcome of the correction is on the record.
The trial is added to the included set, the estimate and the certainty rating have been recomputed, and the screening record now states why the trial was missed, which was a database indexing gap rather than a screening judgement. The submission is identified as an industry submission.
The document is unreadable without specialist training
The respondent, a trustee of a patient organisation, states that the draft is written for a reader who already understands certainty grading, and that the people most affected by the subject matter will not reach the assessment at all.
The respondent proposes a plain-language summary at the head of every document, written to the same standard of accuracy as the document itself and not as a promotional abstract.
The secretariat accepts this submission in part. A plain-language summary is added. The proposal that it replace the technical abstract is declined, because the abstract is the part of the document other assessors read and cite.
Every document now opens with a plain-language summary of not more than 150 words, placed above the technical abstract and carrying the same certainty language, so that the two cannot diverge.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.