Draft terms for the Institute’s calculators — submissions
The 18 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
18 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
The Institute publishes no telephone number and no messaging channel
The respondent states that a body inviting submissions ordinarily provides a channel for questions about the process, and that the written address alone is a high barrier for a first-time respondent.
The respondent asks that a route for procedural questions be provided.
The secretariat notes this submission. The written route is the only route and that is a deliberate constitutional position rather than an omission.
No amendment arises. The Institute holds no telephone number and uses no messaging channel, because a submission made in writing is a document that can be published with its response and an oral exchange is not. Guidance on making a submission has been expanded so that the written route requires no prior enquiry to use.
An industry submission should be identifiable without reading the declared interest
The respondent states that a declared interest printed below a submission is read after the submission, and that a reader who has already absorbed the argument reads the declaration as an afterthought.
The respondent proposes that a submission made on behalf of a company be labelled at the head of the submission as well as in the declaration.
The secretariat accepts this submission. The order in which a reader encounters the two facts changes how the second is weighed.
A submission made on behalf of a company is now identified as an industry submission on its face, at the head of the submission and in the disposition table, in addition to the declared interest published in full beneath it.
The same concept is given three different names in one document
The draft refers to the same quantity as a response rate, a responder proportion and a categorical outcome in different sections. The respondent, who works in health-technology assessment, states that a reader cannot tell whether the three refer to one thing or to three.
The respondent proposes that the glossary term be used at every occurrence and that the glossary entry be linked at first use in each section rather than only at first use in the document.
The secretariat accepts this submission. The variation was stylistic and its cost to the reader exceeds any benefit.
A single term is now used throughout for each defined concept, and the glossary entry is linked at first use within each section rather than once per document.
The position of a contributor who withdraws consent is not stated
The respondent states that consent to be named is obtained at submission and that the policy is silent on what happens if it is later withdrawn, which leaves both the Institute and the contributor without a rule.
The respondent proposes that the position be stated in advance and be the same for everyone.
The secretariat accepts this submission. A rule made at the point of the first withdrawal is a rule made about a particular person.
The policy now states that a contributor may withdraw from the register at any time and that the register entry is removed, while a signed submission already published remains published, because the consultation record is a record of what was said and by whom. The position is stated at the point consent is given.
Whether external reviewers are paid is not stated
The respondent states that payment or its absence bears on who is able to review and on what the Institute may expect of them, and that the policy does not address it.
The respondent proposes that the position be stated and that any payment be disclosed at the level of the individual document.
The secretariat accepts this submission. Payment is an interest and the policy treats interests as facts to be published.
The policy now states whether external review is remunerated, records any payment made in respect of a document in that document's front matter, and treats a payment received from the Institute as a declarable interest in the same way as any other.
A sortable table implies a comparison the underlying data do not support
The draft presents a sortable table whose columns are drawn from sources of differing quality. The respondent states that sorting on such a column produces an ordering that looks like a ranking and is not one.
The respondent proposes that sorting be disabled on any column whose values are not commensurable.
The secretariat notes this submission and records that the point is correct in principle.
No amendment arises here because every sortable table in the document set already carries a standing statement above it that the ordering is not a ranking and that the values in each column are commensurable only where the column header says so. The proposal to disable sorting was considered and not adopted, because a reader who cannot sort a table generally sorts it elsewhere and without the statement.
The document should state what a reader ought to do
The draft assesses evidence and stops. The respondent, a practising clinician, states that a reader arriving at the document with a decision to make is left to convert an assessment into an action without help, and proposes that each document close with a recommendation.
The respondent argues that other evidence bodies issue recommendations and that declining to do so transfers the difficult part of the work to the reader.
The secretariat does not accept this submission, and records that the point is a reasonable one rather than a misunderstanding.
The Institute assesses evidence and does not issue recommendations, because a recommendation embeds values and a resource context that the Institute does not hold and cannot state. That constitutional limit is published on the methodology page and is not varied by consultation. The submission remains published in full.
A party criticised in an assessment should have a right of reply
The respondent states that a supplier or sponsor whose documentation is assessed adversely has no route to respond other than a comment period that may not be open.
The respondent proposes a standing right of reply published alongside the assessment.
The secretariat accepts this submission in part. A route to submit a correction of fact is opened at any time and is not dependent on a consultation window. A general right of reply published alongside the assessment is not adopted, because it would give an assessed party the last word on the Institute's own document.
A standing correction route is now published, corrections of fact are assessed and applied at any time with the change recorded in the amendment log, and a submission of substance received outside a window is held and published in the next comment period on that document.
The terms on which the documents may be reused are not stated
The respondent, a librarian, states that the site carries no statement of reuse terms and that an assessment which cannot be quoted with confidence will be paraphrased instead, which is worse for accuracy.
The respondent proposes an explicit reuse statement.
The secretariat accepts this submission. An unstated position on reuse is a position that will be guessed at.
The reuse terms are now stated explicitly on the legal page and in the document footer, permitting quotation and reproduction with attribution and the document version, and requiring that any reproduction carry the certainty rating that accompanied the quoted statement.
The secretariat answers submissions and is not itself named
The respondent states that responses are attributed to the secretariat as a body, and that a reader cannot tell who wrote a response or whether that person had a relevant interest.
The respondent proposes that responses be signed individually.
The secretariat accepts this submission in part. Responses are attributed to the named members of the secretariat responsible for the document rather than to individuals per response, and the interests of those members are published.
Every comment period now names the secretariat members responsible for it, with their declared interests, in the consultation front matter, so that the responses can be read against the interests of the people who wrote them.
The disclosure requirements exceed those of comparable bodies
The submission is made on behalf of a manufacturer. It states that the per-page disclosure requirement is more extensive than that applied by comparable assessment bodies and that the repetition may be read as implying a relationship of greater significance than exists.
The submission does not object to disclosure and objects to its placement and frequency.
The secretariat notes this submission and records that the point about repetition is a fair one.
No amendment arises. Per-page disclosure was consulted on separately and retained, on the ground that a reader arriving at a single page from a search engine does not see a disclosure held elsewhere. The wording has been standardised so that the repeated statement is identical at every occurrence and is not read as a series of different claims. The submission is identified as an industry submission.
The document should not describe uses outside the approved indication
The submission is made on behalf of a marketing-authorisation holder. It states that the draft describes uses of the compound that fall outside the approved indication, that such uses are not supported by the sponsor, and that describing them may be read as legitimising them.
The sponsor asks that the sections concerned be removed, or in the alternative that they carry a prominent statement that the sponsor does not support such use.
The secretariat does not accept this submission. The Institute records what is supplied and what is used, because a reader encountering a compound outside a regulated route is the reader most in need of an assessment of the evidence for it.
The sections remain. Every such section already states the regulatory status of the use described and states that the evidence for it is assessed separately from the evidence for the approved indication. The submission is published in full and identified as an industry submission.
The document is unreadable without specialist training
The respondent, a trustee of a patient organisation, states that the draft is written for a reader who already understands certainty grading, and that the people most affected by the subject matter will not reach the assessment at all.
The respondent proposes a plain-language summary at the head of every document, written to the same standard of accuracy as the document itself and not as a promotional abstract.
The secretariat accepts this submission in part. A plain-language summary is added. The proposal that it replace the technical abstract is declined, because the abstract is the part of the document other assessors read and cite.
Every document now opens with a plain-language summary of not more than 150 words, placed above the technical abstract and carrying the same certainty language, so that the two cannot diverge.
What the site collects about a reader is not stated in terms a reader can check
The respondent states that a body assessing evidence should be checkable in its own conduct, and that a general privacy statement does not permit a reader to verify what is collected.
The respondent proposes that the statement enumerate what is collected and that it be verifiable from the page itself.
The secretariat accepts this submission. An unverifiable statement about data is exactly the kind of claim the Institute declines to accept from others.
The statement now enumerates what is collected and what is not, and is written so that a reader can verify it from the page source, which is the only form of the statement that can be checked without trusting the Institute.
Recusal should follow automatically from any declared financial interest
The respondent states that case-by-case recusal decisions require the body making them to judge its own members, and that an automatic rule removes the judgement and with it the appearance of partiality.
The respondent proposes automatic recusal on any declared financial interest in a named company.
The secretariat does not accept this submission. An automatic rule of that breadth would recuse most of the people competent to assess the evidence, and the Institute would be left with assessors chosen for their distance from the field.
Recusal remains determined against the specific document, with the decision, its reasons and the person recused published in the document front matter. The submission is published in full because it identifies a genuine tension the Institute has resolved in one direction rather than removed.
Declared interests should appear on the document rather than on a separate page
The draft links to a central conflicts register. The respondent argues that a reader assessing whether to rely on a particular document should not have to leave it to find out who assessed it and what they declared.
The respondent proposes that the interests of every named contributor to a document be printed on that document.
The secretariat notes this submission and records that the draft already provides for it, which the respondent could reasonably have missed because the provision sits in an appendix.
Every document carries the declared interests of its named contributors in its front matter, and the central register exists so that a reader can see a person across all documents rather than one at a time. No amendment arises; the provision has been moved from the appendix into the body of the methodology document so that it is findable.
A superseded version should remain reachable from the version that replaced it
The respondent states that the draft supersedes an earlier document and that a reader who cited the earlier version has no way to reach it from the new one, which makes it impossible to see what changed.
The respondent asks that every version carry a link both to what it supersedes and to what supersedes it.
The secretariat notes this submission. The corrections and versioning policy already requires bidirectional version links and every superseded document is retained at its own address.
No amendment arises. The requirement is stated in the corrections and versioning policy and the amendment log of this document links to the version it replaced. The respondent is correct that the link was absent from the draft page furnished for consultation, which was a defect of the consultation copy and not of the policy.
Absence of evidence is presented in a form a reader will take as negative evidence
Where the Institute has identified no study, the draft states that no evidence was found. In several places that sentence sits immediately after a paragraph describing an adverse outcome, and the juxtaposition invites the reading that the compound was studied and found wanting.
The respondent proposes a standing formulation, used identically wherever the situation arises, distinguishing an outcome that was studied and not demonstrated from an outcome that has not been studied at all.
The secretariat accepts this submission. The two states are different, they support different decisions, and the draft rendered them in language a reader could not reliably separate.
A standing formulation has been adopted and is applied wherever an evidence gap is recorded, distinguishing an outcome assessed and not demonstrated from an outcome not assessed. The formulation is identical at every occurrence so that it can be recognised at a glance.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.