Draft print and archival format for Institute documents — submissions
The 18 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
18 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
The threshold between a correction and a new version is drawn too high
The respondent states that under the draft a change to a certainty rating could be applied as a correction without a version increment, and that a reader holding a citation to the earlier text would not know the rating had changed.
The respondent proposes that any change to a rating or an estimate force a new version.
The secretariat accepts this submission in part. A change to a rating, an estimate or a conclusion now forces a version increment. Typographical and reference corrections continue to be applied as corrections with a log entry.
The versioning threshold is now stated by category: a change to a certainty rating, an effect estimate or a conclusion produces a new version at a new address, and every other change is a logged correction that states the previous wording.
Declared interests should appear on the document rather than on a separate page
The draft links to a central conflicts register. The respondent argues that a reader assessing whether to rely on a particular document should not have to leave it to find out who assessed it and what they declared.
The respondent proposes that the interests of every named contributor to a document be printed on that document.
The secretariat notes this submission and records that the draft already provides for it, which the respondent could reasonably have missed because the provision sits in an appendix.
Every document carries the declared interests of its named contributors in its front matter, and the central register exists so that a reader can see a person across all documents rather than one at a time. No amendment arises; the provision has been moved from the appendix into the body of the methodology document so that it is findable.
There is no route to challenge a disposition
The respondent states that a submission not accepted receives a reason and nothing further, and that a respondent who considers the reason mistaken has nowhere to go.
The respondent proposes a review of disposition by a person who did not make it.
The secretariat accepts this submission. A disposition that cannot be challenged is a decision without a check.
A disposition may now be referred for review to a member of the assessment committee who did not author the response, once per submission, with the referral and its outcome published alongside the original disposition whether or not it changes.
What the site collects about a reader is not stated in terms a reader can check
The respondent states that a body assessing evidence should be checkable in its own conduct, and that a general privacy statement does not permit a reader to verify what is collected.
The respondent proposes that the statement enumerate what is collected and that it be verifiable from the page itself.
The secretariat accepts this submission. An unverifiable statement about data is exactly the kind of claim the Institute declines to accept from others.
The statement now enumerates what is collected and what is not, and is written so that a reader can verify it from the page source, which is the only form of the statement that can be checked without trusting the Institute.
Two factual descriptions of the sponsor's programme are inaccurate
The submission is made on behalf of the marketing-authorisation holder and is confined to two matters of fact. The draft describes a trial as terminated where the sponsor closed it at a pre-specified interim analysis, and gives a dose in a unit that does not match the approved labelling.
Supporting documentation, comprising the published trial report and the current summary of product characteristics, accompanied the submission. No view is expressed on the certainty ratings, which the sponsor considers a matter for the assessment committee.
The secretariat accepts this submission. Both points are matters of fact, both were checkable against documents the Institute holds, and both were wrong in the draft.
The trial status now reads as completed at a pre-specified interim analysis, with the analysis identified, and the dose is stated in the unit used in the approved labelling. The submission is identified as an industry submission on its face in accordance with the conflicts policy.
The secretariat answers submissions and is not itself named
The respondent states that responses are attributed to the secretariat as a body, and that a reader cannot tell who wrote a response or whether that person had a relevant interest.
The respondent proposes that responses be signed individually.
The secretariat accepts this submission in part. Responses are attributed to the named members of the secretariat responsible for the document rather than to individuals per response, and the interests of those members are published.
Every comment period now names the secretariat members responsible for it, with their declared interests, in the consultation front matter, so that the responses can be read against the interests of the people who wrote them.
References should carry a persistent identifier for every cited source
Several references in the draft carry a journal, a year and a volume but no persistent identifier. The respondent, who works in a library setting, states that retrieval of such a reference is materially slower and that identifiers should be supplied throughout.
The respondent asks in the alternative that where an identifier exists but is not carried, the omission be explained rather than left as a gap the reader must interpret.
The secretariat accepts the second limb of this submission and declines the first. Identifiers are supplied wherever the Institute holds one. Where the Institute does not hold an identifier it will not supply one, because a reconstructed identifier that resolves to the wrong record is a worse defect than an absent one.
Every reference without a persistent identifier now carries an explicit statement that the identifier is not held by the Institute, so that its absence is a recorded fact rather than an apparent oversight.
A party criticised in an assessment should have a right of reply
The respondent states that a supplier or sponsor whose documentation is assessed adversely has no route to respond other than a comment period that may not be open.
The respondent proposes a standing right of reply published alongside the assessment.
The secretariat accepts this submission in part. A route to submit a correction of fact is opened at any time and is not dependent on a consultation window. A general right of reply published alongside the assessment is not adopted, because it would give an assessed party the last word on the Institute's own document.
A standing correction route is now published, corrections of fact are assessed and applied at any time with the change recorded in the amendment log, and a submission of substance received outside a window is held and published in the next comment period on that document.
Quantitative claims are reproduced without the method that produced them
Several figures in the draft are quoted from sources that determined them by different methods. A figure obtained by one determination and a figure obtained by another are not comparable, and the draft places them in the same sentence without distinguishing them.
The respondent, an analytical chemist, proposes that every quantitative claim carry the method that produced it at the point of use rather than in the reference.
The secretariat accepts this submission. Placing two figures side by side is an implicit claim that they are the same kind of quantity, and in the cases identified they were not.
Every quantitative claim now carries the determination that produced it at the point of use, and figures obtained by non-comparable methods are no longer presented in the same row or sentence.
The document should state what a reader ought to do
The draft assesses evidence and stops. The respondent, a practising clinician, states that a reader arriving at the document with a decision to make is left to convert an assessment into an action without help, and proposes that each document close with a recommendation.
The respondent argues that other evidence bodies issue recommendations and that declining to do so transfers the difficult part of the work to the reader.
The secretariat does not accept this submission, and records that the point is a reasonable one rather than a misunderstanding.
The Institute assesses evidence and does not issue recommendations, because a recommendation embeds values and a resource context that the Institute does not hold and cannot state. That constitutional limit is published on the methodology page and is not varied by consultation. The submission remains published in full.
The document is unreadable without specialist training
The respondent, a trustee of a patient organisation, states that the draft is written for a reader who already understands certainty grading, and that the people most affected by the subject matter will not reach the assessment at all.
The respondent proposes a plain-language summary at the head of every document, written to the same standard of accuracy as the document itself and not as a promotional abstract.
The secretariat accepts this submission in part. A plain-language summary is added. The proposal that it replace the technical abstract is declined, because the abstract is the part of the document other assessors read and cite.
Every document now opens with a plain-language summary of not more than 150 words, placed above the technical abstract and carrying the same certainty language, so that the two cannot diverge.
Absence of evidence is presented in a form a reader will take as negative evidence
Where the Institute has identified no study, the draft states that no evidence was found. In several places that sentence sits immediately after a paragraph describing an adverse outcome, and the juxtaposition invites the reading that the compound was studied and found wanting.
The respondent proposes a standing formulation, used identically wherever the situation arises, distinguishing an outcome that was studied and not demonstrated from an outcome that has not been studied at all.
The secretariat accepts this submission. The two states are different, they support different decisions, and the draft rendered them in language a reader could not reliably separate.
A standing formulation has been adopted and is applied wherever an evidence gap is recorded, distinguishing an outcome assessed and not demonstrated from an outcome not assessed. The formulation is identical at every occurrence so that it can be recognised at a glance.
Anonymous submissions should be accepted where the respondent risks employment consequences
The respondent states that people in a position to identify a defect in a manufacturer's documentation are frequently employed by that manufacturer, and that requiring a signature excludes exactly those submissions.
The respondent proposes an anonymous route with verification held by the secretariat and not published.
The secretariat does not accept this submission, and records that the exclusion it identifies is real.
The Institute publishes no anonymous submission, because a submission whose author is not published cannot be weighed against a declared interest and the declaration is the mechanism the whole surface rests on. Material bearing on a factual defect may instead be submitted to the evidence log, where it is assessed on the document and the submitter is named only with consent. The submission remains published in full.
Where a document is generated rather than authored, the reader should be told
The respondent states that some records in the document set are constructed by crossing existing records, that the prose in them is templated, and that nothing on the page distinguishes them from an authored assessment.
The respondent proposes a standing notice on every generated record.
The secretariat accepts this submission. The distinction is material to how much independent judgement a reader should assume.
Every generated record now carries a notice stating that its prose frame is templated and that its included studies, summary of findings and certainty rating are computed from the underlying records, so that the reader can see which parts were authored and which were derived.
The terms on which the documents may be reused are not stated
The respondent, a librarian, states that the site carries no statement of reuse terms and that an assessment which cannot be quoted with confidence will be paraphrased instead, which is worse for accuracy.
The respondent proposes an explicit reuse statement.
The secretariat accepts this submission. An unstated position on reuse is a position that will be guessed at.
The reuse terms are now stated explicitly on the legal page and in the document footer, permitting quotation and reproduction with attribution and the document version, and requiring that any reproduction carry the certainty rating that accompanied the quoted statement.
Recusal should follow automatically from any declared financial interest
The respondent states that case-by-case recusal decisions require the body making them to judge its own members, and that an automatic rule removes the judgement and with it the appearance of partiality.
The respondent proposes automatic recusal on any declared financial interest in a named company.
The secretariat does not accept this submission. An automatic rule of that breadth would recuse most of the people competent to assess the evidence, and the Institute would be left with assessors chosen for their distance from the field.
Recusal remains determined against the specific document, with the decision, its reasons and the person recused published in the document front matter. The submission is published in full because it identifies a genuine tension the Institute has resolved in one direction rather than removed.
Disagreement within the assessment committee is not visible
The respondent states that ratified documents present a single position and that a reader cannot tell whether a rating was unanimous or contested.
The respondent proposes that a dissenting view be recorded where a member asks for it to be.
The secretariat accepts this submission. A document that conceals a division presents more certainty about its own judgement than it holds.
A ratification record is now published with every document stating whether the assessment was unanimous, and a member may record a dissenting statement which is published with the document rather than in minutes.
The document should not describe uses outside the approved indication
The submission is made on behalf of a marketing-authorisation holder. It states that the draft describes uses of the compound that fall outside the approved indication, that such uses are not supported by the sponsor, and that describing them may be read as legitimising them.
The sponsor asks that the sections concerned be removed, or in the alternative that they carry a prominent statement that the sponsor does not support such use.
The secretariat does not accept this submission. The Institute records what is supplied and what is used, because a reader encountering a compound outside a regulated route is the reader most in need of an assessment of the evidence for it.
The sections remain. Every such section already states the regulatory status of the use described and states that the evidence for it is assessed separately from the evidence for the approved indication. The submission is published in full and identified as an industry submission.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.