Draft analytical review: surveys of unregulated peptide supply — submissions
The 14 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
14 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Absence of evidence is presented in a form a reader will take as negative evidence
Where the Institute has identified no study, the draft states that no evidence was found. In several places that sentence sits immediately after a paragraph describing an adverse outcome, and the juxtaposition invites the reading that the compound was studied and found wanting.
The respondent proposes a standing formulation, used identically wherever the situation arises, distinguishing an outcome that was studied and not demonstrated from an outcome that has not been studied at all.
The secretariat accepts this submission. The two states are different, they support different decisions, and the draft rendered them in language a reader could not reliably separate.
A standing formulation has been adopted and is applied wherever an evidence gap is recorded, distinguishing an outcome assessed and not demonstrated from an outcome not assessed. The formulation is identical at every occurrence so that it can be recognised at a glance.
The same concept is given three different names in one document
The draft refers to the same quantity as a response rate, a responder proportion and a categorical outcome in different sections. The respondent, who works in health-technology assessment, states that a reader cannot tell whether the three refer to one thing or to three.
The respondent proposes that the glossary term be used at every occurrence and that the glossary entry be linked at first use in each section rather than only at first use in the document.
The secretariat accepts this submission. The variation was stylistic and its cost to the reader exceeds any benefit.
A single term is now used throughout for each defined concept, and the glossary entry is linked at first use within each section rather than once per document.
Two included studies do not meet the registered eligibility criteria
The respondent identifies two studies in the included set whose duration falls below the minimum stated in the protocol, and one excluded study that appears to meet every criterion.
The respondent proposes that the three be reassessed and that the outcome of the reassessment be recorded whichever way it goes.
The secretariat accepts this submission in part. On reassessment, one of the two included studies does fall below the minimum duration and has been removed. The second reports a duration that met the criterion in the protocol version registered at the time. The excluded study was excluded for a reason not clearly recorded, which was a documentation failure.
One study has been removed from the included set and the estimate recomputed, the exclusion reason for the third study has been corrected in the excluded-studies table, and the screening decisions are now recorded against the protocol version in force at the time of screening.
Doses differing several-fold are pooled without examining dose-response
Contributing trials administer doses that differ by a factor of several. The draft pools them and reports one estimate. The respondent states that where a dose-response relationship exists the pooled figure corresponds to no dose that anyone receives.
The respondent proposes that estimates be reported by dose and that dose-response be examined where the data allow.
The secretariat accepts this submission. A pooled estimate across doses is an estimate for an average dose that no protocol specifies.
Estimates are now reported by dose group, a dose-response examination is reported where three or more dose levels contribute, and the pooled across-dose estimate is removed rather than retained alongside.
The document is unreadable without specialist training
The respondent, a trustee of a patient organisation, states that the draft is written for a reader who already understands certainty grading, and that the people most affected by the subject matter will not reach the assessment at all.
The respondent proposes a plain-language summary at the head of every document, written to the same standard of accuracy as the document itself and not as a promotional abstract.
The secretariat accepts this submission in part. A plain-language summary is added. The proposal that it replace the technical abstract is declined, because the abstract is the part of the document other assessors read and cite.
Every document now opens with a plain-language summary of not more than 150 words, placed above the technical abstract and carrying the same certainty language, so that the two cannot diverge.
The conclusion is stated more strongly than the certainty rating supports
The draft rates the anchor outcome at low certainty and then states in the conclusion that the intervention is effective. The respondent states that the two sentences cannot both be true as written.
The respondent proposes that conclusion language be tied mechanically to the certainty rating, so that a low rating cannot produce an unqualified claim.
The secretariat accepts this submission. The mismatch is the failure mode the certainty framework exists to prevent.
Conclusion wording is now drawn from a fixed set of formulations tied to the certainty rating, so that a low certainty rating produces a statement that the evidence may suggest an effect and that the estimate is likely to change with further research.
The document should state what a reader ought to do
The draft assesses evidence and stops. The respondent, a practising clinician, states that a reader arriving at the document with a decision to make is left to convert an assessment into an action without help, and proposes that each document close with a recommendation.
The respondent argues that other evidence bodies issue recommendations and that declining to do so transfers the difficult part of the work to the reader.
The secretariat does not accept this submission, and records that the point is a reasonable one rather than a misunderstanding.
The Institute assesses evidence and does not issue recommendations, because a recommendation embeds values and a resource context that the Institute does not hold and cannot state. That constitutional limit is published on the methodology page and is not varied by consultation. The submission remains published in full.
Statistical heterogeneity is treated as though it measured clinical heterogeneity
The draft reports a heterogeneity statistic and proceeds to pool where it is low. The respondent states that a low statistic in a small set of trials is uninformative, and that clinical and methodological similarity should be assessed before any statistic is consulted.
The respondent proposes that the decision to pool be justified on clinical grounds first and that the statistic be reported as a description rather than used as a threshold.
The secretariat accepts this submission in part. The decision to pool is now made on clinical and methodological grounds and stated as such. The statistic continues to be reported, because readers expect it and its absence would be read as concealment.
The decision to pool is now justified in prose against the population, intervention, comparator and outcome before any statistic is presented, and the heterogeneity statistic is reported with its confidence interval and a note of the number of contributing studies.
The review question is drawn too broadly to be answerable
The respondent states that the population as registered spans groups in which the intervention would be expected to behave differently, and that a single estimate across them is uninformative.
The respondent proposes that the question be split and the review re-registered.
The secretariat does not accept this submission. The breadth was registered before screening, the pre-specified subgroups address the variation the respondent identifies, and re-registering after the results are known would convert a pre-specified analysis into a post hoc one.
The question stands as registered. The point is recorded in the limitations, and the assessment committee has noted it for the protocol of the next version of this review, which will be registered before the search is run. The submission remains published in full.
The same concern is used to downgrade in two domains
The draft downgrades for both indirectness and imprecision citing the same feature of the contributing trials. The respondent states that this double-counts a single concern and produces a rating lower than the framework supports.
The respondent proposes that the rating be raised by one level.
The secretariat accepts this submission in part. The double count is real in one of the two outcomes identified and not in the other, where the two domains rest on different features.
The rating for the first outcome has been raised by one level and the domain reasoning restated, and the reasoning for the second has been rewritten to make clear that the two downgrades rest on different features of the evidence.
Analytical surveys are treated as evidence about suppliers when they establish only what was in a sample
Several included surveys purchased material anonymously and analysed it. The respondent states that where the chain from a named supplier to the analysed vial cannot be documented, the result describes a sample and not a supplier.
The respondent proposes that provenance be an explicit eligibility dimension, with surveys reported separately according to whether it could be established.
The secretariat accepts this submission. Moderate certainty evidence from the included surveys supports statements about material circulating in a market; it does not support statements about any named supplier's output.
Provenance is now assessed for every included survey and reported as a study characteristic, surveys without establishable provenance are reported separately rather than pooled with those that have it, and no supplier-level inference is drawn from a sample-level result.
A sortable table implies a comparison the underlying data do not support
The draft presents a sortable table whose columns are drawn from sources of differing quality. The respondent states that sorting on such a column produces an ordering that looks like a ranking and is not one.
The respondent proposes that sorting be disabled on any column whose values are not commensurable.
The secretariat notes this submission and records that the point is correct in principle.
No amendment arises here because every sortable table in the document set already carries a standing statement above it that the ordering is not a ranking and that the values in each column are commensurable only where the column header says so. The proposal to disable sorting was considered and not adopted, because a reader who cannot sort a table generally sorts it elsewhere and without the statement.
Efficacy outcomes are rated for certainty and harms are not
The draft assigns certainty ratings to the efficacy outcomes and reports harms narratively without ratings. The respondent states that the asymmetry implies harms are less amenable to assessment when they are simply less well measured.
The respondent proposes that harms carry certainty ratings on the same scale, with the reasons for downgrading stated.
The secretariat accepts this submission. Rating one side of the balance and not the other produces a document that cannot be used to weigh them.
Every reported harm now carries a certainty rating on the same scale as the efficacy outcomes, with the downgrade reasons stated, and discontinuation for adverse events appears in the summary of findings rather than in an annex.
Patient-reported outcomes are collected by the trials and not reported by the review
The respondent, a trustee of a patient organisation, states that several contributing trials measured quality of life and function and that the review reports neither, having selected outcomes on the basis of what could be pooled.
The respondent proposes that these outcomes be reported narratively where they cannot be pooled.
The secretariat accepts this submission in part. The outcomes are reported narratively. The proposal to pool them is declined because the instruments used are not comparable and pooling would produce a figure with no interpretation.
Patient-reported outcomes measured by any contributing trial are now reported narratively by instrument, with the instrument named and its minimum important difference stated where one is published, and the absence of a pooled estimate explained.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.