Draft search-strategy template for the synthesis series — submissions
The 19 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
19 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
The search date is not on the face of the document
The draft carries a publication date and a review date but not the date on which the evidence was last searched. Those are three different dates and only the third tells a reader how current the assessment is. A document published in one quarter may rest on a search run two quarters earlier, and nothing on the page allows that gap to be measured.
The respondent proposes that the search date be printed adjacent to every certainty rating rather than in the methods section, on the ground that a reader who acts on a rating is unlikely to have read the methods section first.
The secretariat accepts this submission. The distinction between publication, review and search dates is real and the draft did not make it visible where it mattered.
The search date is now printed adjacent to every certainty rating and is carried in the document metadata, so that the interval between the search and the reading is visible without reference to the methods section.
The document should describe how an assessment becomes a decision
The respondent, employed by a health-technology assessment body, states that the methodology stops at the certainty rating and that the step from a rating to a decision is where most disagreement actually occurs.
The respondent proposes an evidence-to-decision framework.
The secretariat notes this submission and records the point as correct about the field and not about this document.
No amendment arises. The Institute assesses evidence and does not make decisions, because a decision requires values and a resource context the Institute does not hold. The boundary is stated on the methodology page and the submission has prompted it to be stated at the head of the document rather than in the section on scope.
Registry records without posted results are neither screened nor counted
The respondent states that the screening procedure operates on bibliographic records and that a registered trial without a publication falls outside it entirely.
The respondent proposes that registry records be screened as records in their own right and reported as a separate category.
The secretariat accepts this submission. A procedure that cannot see an unreported trial cannot measure how much of the evidence base is unreported.
Registry records are now screened as records in their own right, and those meeting the eligibility criteria without a posted result are reported as unreported studies with their registered outcome measures and their completion dates.
It is not clear which provisions bind the assessment committee
The respondent states that the document mixes requirements with descriptions of current practice in the same voice, so that a reader cannot tell which departures would be a breach and which would be a change of habit.
The respondent proposes that binding provisions be distinguished typographically and listed.
The secretariat accepts this submission. A rule indistinguishable from a description is not enforceable and does not reassure.
Binding provisions are now stated in a fixed form, are listed together in an annex, and a departure from any of them must be recorded in the document it affects with the reason, while descriptive passages are marked as descriptions of practice.
Declared interests should appear on the document rather than on a separate page
The draft links to a central conflicts register. The respondent argues that a reader assessing whether to rely on a particular document should not have to leave it to find out who assessed it and what they declared.
The respondent proposes that the interests of every named contributor to a document be printed on that document.
The secretariat notes this submission and records that the draft already provides for it, which the respondent could reasonably have missed because the provision sits in an appendix.
Every document carries the declared interests of its named contributors in its front matter, and the central register exists so that a reader can see a person across all documents rather than one at a time. No amendment arises; the provision has been moved from the appendix into the body of the methodology document so that it is findable.
The document should state what a reader ought to do
The draft assesses evidence and stops. The respondent, a practising clinician, states that a reader arriving at the document with a decision to make is left to convert an assessment into an action without help, and proposes that each document close with a recommendation.
The respondent argues that other evidence bodies issue recommendations and that declining to do so transfers the difficult part of the work to the reader.
The secretariat does not accept this submission, and records that the point is a reasonable one rather than a misunderstanding.
The Institute assesses evidence and does not issue recommendations, because a recommendation embeds values and a resource context that the Institute does not hold and cannot state. That constitutional limit is published on the methodology page and is not varied by consultation. The submission remains published in full.
Two factual descriptions of the sponsor's programme are inaccurate
The submission is made on behalf of the marketing-authorisation holder and is confined to two matters of fact. The draft describes a trial as terminated where the sponsor closed it at a pre-specified interim analysis, and gives a dose in a unit that does not match the approved labelling.
Supporting documentation, comprising the published trial report and the current summary of product characteristics, accompanied the submission. No view is expressed on the certainty ratings, which the sponsor considers a matter for the assessment committee.
The secretariat accepts this submission. Both points are matters of fact, both were checkable against documents the Institute holds, and both were wrong in the draft.
The trial status now reads as completed at a pre-specified interim analysis, with the analysis identified, and the dose is stated in the unit used in the approved labelling. The submission is identified as an industry submission on its face in accordance with the conflicts policy.
Absence of evidence is presented in a form a reader will take as negative evidence
Where the Institute has identified no study, the draft states that no evidence was found. In several places that sentence sits immediately after a paragraph describing an adverse outcome, and the juxtaposition invites the reading that the compound was studied and found wanting.
The respondent proposes a standing formulation, used identically wherever the situation arises, distinguishing an outcome that was studied and not demonstrated from an outcome that has not been studied at all.
The secretariat accepts this submission. The two states are different, they support different decisions, and the draft rendered them in language a reader could not reliably separate.
A standing formulation has been adopted and is applied wherever an evidence gap is recorded, distinguishing an outcome assessed and not demonstrated from an outcome not assessed. The formulation is identical at every occurrence so that it can be recognised at a glance.
References should carry a persistent identifier for every cited source
Several references in the draft carry a journal, a year and a volume but no persistent identifier. The respondent, who works in a library setting, states that retrieval of such a reference is materially slower and that identifiers should be supplied throughout.
The respondent asks in the alternative that where an identifier exists but is not carried, the omission be explained rather than left as a gap the reader must interpret.
The secretariat accepts the second limb of this submission and declines the first. Identifiers are supplied wherever the Institute holds one. Where the Institute does not hold an identifier it will not supply one, because a reconstructed identifier that resolves to the wrong record is a worse defect than an absent one.
Every reference without a persistent identifier now carries an explicit statement that the identifier is not held by the Institute, so that its absence is a recorded fact rather than an apparent oversight.
The document set should be published in translation
The respondent notes that the assessments concern compounds supplied internationally and that publishing only in English restricts access to the assessment to readers who work in it.
The respondent proposes machine translation of the document set as an interim measure, with human review of the certainty language.
The secretariat does not accept this submission, and records that the underlying point is sound and that the proposed remedy is the difficulty.
A translation whose certainty language has drifted is a different assessment carrying the Institute's name, and the Institute cannot review translations it does not have the capacity to review. The documents remain in English. The submission is published in full because the access problem it identifies is real and unresolved.
Nothing prevents a conclusion stronger than its certainty rating
The respondent states that the framework rates certainty and then leaves conclusion wording to the author, so that a low rating and a confident conclusion can coexist in one document.
The respondent proposes that conclusion wording be drawn from a fixed set tied to the rating.
The secretariat accepts this submission. The coupling is the mechanism by which a rating changes what a reader takes away.
Conclusion wording is now drawn from a fixed set of formulations tied to the certainty rating, the mapping is published in the methodology document, and a document whose conclusion does not match its rating cannot be ratified.
The rating conflates absence of evidence with conflicting evidence
The framework applies a single very low certainty rating to two situations a reader will act on differently. In the first, several studies exist and disagree. In the second, no study exists at all. The same badge in both places gives the same signal for two different states of knowledge.
The respondent proposes that the rating be accompanied by a standing formulation identifying which of the two applies, identical wherever it appears so that it can be recognised at a glance.
The secretariat accepts this submission in full. The distinction is real, it is decision-relevant, and the draft did not make it.
A standing formulation has been adopted and is applied to every low and very low certainty rating in the document set. The rating scale itself is unchanged, so that Institute ratings remain readable against published assessments using the same scale.
The same concept is given three different names in one document
The draft refers to the same quantity as a response rate, a responder proportion and a categorical outcome in different sections. The respondent, who works in health-technology assessment, states that a reader cannot tell whether the three refer to one thing or to three.
The respondent proposes that the glossary term be used at every occurrence and that the glossary entry be linked at first use in each section rather than only at first use in the document.
The secretariat accepts this submission. The variation was stylistic and its cost to the reader exceeds any benefit.
A single term is now used throughout for each defined concept, and the glossary entry is linked at first use within each section rather than once per document.
A superseded version should remain reachable from the version that replaced it
The respondent states that the draft supersedes an earlier document and that a reader who cited the earlier version has no way to reach it from the new one, which makes it impossible to see what changed.
The respondent asks that every version carry a link both to what it supersedes and to what supersedes it.
The secretariat notes this submission. The corrections and versioning policy already requires bidirectional version links and every superseded document is retained at its own address.
No amendment arises. The requirement is stated in the corrections and versioning policy and the amendment log of this document links to the version it replaced. The respondent is correct that the link was absent from the draft page furnished for consultation, which was a defect of the consultation copy and not of the policy.
Protocol amendments are permitted without a requirement to record when they were made
The respondent states that the value of a registered protocol lies in the sequence, and that an amendment without a date cannot be shown to have preceded the results.
The respondent proposes that every amendment be dated and that its position relative to screening and analysis be recorded.
The secretariat accepts this submission. An undated amendment removes the property that registration exists to provide.
Every protocol amendment is now dated, is recorded against the stage of the review in progress at the time, and appears in the published protocol history, so that a reader can see which analyses were specified before the results were known.
The document is unreadable without specialist training
The respondent, a trustee of a patient organisation, states that the draft is written for a reader who already understands certainty grading, and that the people most affected by the subject matter will not reach the assessment at all.
The respondent proposes a plain-language summary at the head of every document, written to the same standard of accuracy as the document itself and not as a promotional abstract.
The secretariat accepts this submission in part. A plain-language summary is added. The proposal that it replace the technical abstract is declined, because the abstract is the part of the document other assessors read and cite.
Every document now opens with a plain-language summary of not more than 150 words, placed above the technical abstract and carrying the same certainty language, so that the two cannot diverge.
Automated assistance in screening should be disclosed and validated
The respondent asks whether any automated tool is used in screening or data extraction, and states that if one is, its performance should be reported in the same way as a human screener's agreement.
The respondent proposes that automated assistance be prohibited.
The secretariat accepts this submission in part. Disclosure and validation are adopted. A prohibition is not, because the alternative to a validated tool is not a human but a smaller search.
The methodology now requires that any automated assistance in screening or extraction be disclosed in the review, that its output be verified by a human for every included record, and that its measured performance against human screening be reported.
The database set omits sources relevant to the compounds in scope
The respondent, an information specialist, states that regional databases index trials of several compounds in the Institute's scope that are not indexed in the listed sources.
The respondent proposes that the named regional databases be added to the standard set.
The secretariat accepts this submission in part. Two of the named sources are added to the standard set. The remainder are added as conditional sources searched where the review question concerns a compound developed or registered in the relevant region.
The standard database set is extended, the conditional sources are named together with the trigger for searching them, and every review states which sources were searched and which were not, with the reason.
Quantitative claims are reproduced without the method that produced them
Several figures in the draft are quoted from sources that determined them by different methods. A figure obtained by one determination and a figure obtained by another are not comparable, and the draft places them in the same sentence without distinguishing them.
The respondent, an analytical chemist, proposes that every quantitative claim carry the method that produced it at the point of use rather than in the reference.
The secretariat accepts this submission. Placing two figures side by side is an implicit claim that they are the same kind of quantity, and in the cases identified they were not.
Every quantitative claim now carries the determination that produced it at the point of use, and figures obtained by non-comparable methods are no longer presented in the same row or sentence.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.