Draft corrections and versioning policy — submissions
The 19 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
19 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
What the site collects about a reader is not stated in terms a reader can check
The respondent states that a body assessing evidence should be checkable in its own conduct, and that a general privacy statement does not permit a reader to verify what is collected.
The respondent proposes that the statement enumerate what is collected and that it be verifiable from the page itself.
The secretariat accepts this submission. An unverifiable statement about data is exactly the kind of claim the Institute declines to accept from others.
The statement now enumerates what is collected and what is not, and is written so that a reader can verify it from the page source, which is the only form of the statement that can be checked without trusting the Institute.
Point estimates are given without an interval
Several estimates in the draft appear as single figures. The respondent states that a point estimate without an interval invites a precision the underlying data do not support, and that the effect is worst where the estimate is drawn from a small contributing set.
The respondent proposes that no point estimate appear anywhere in the document set without its interval, including in summary tables and in the abstract.
The secretariat accepts this submission in part. Intervals are added wherever the source reports one. The proposal is declined for figures the source published without an interval, because the Institute will not compute an interval a source did not report.
Every estimate now carries its interval where the source reported one, and where it did not, the estimate is annotated as reported without an interval rather than left to appear as a precise figure.
Anonymous submissions should be accepted where the respondent risks employment consequences
The respondent states that people in a position to identify a defect in a manufacturer's documentation are frequently employed by that manufacturer, and that requiring a signature excludes exactly those submissions.
The respondent proposes an anonymous route with verification held by the secretariat and not published.
The secretariat does not accept this submission, and records that the exclusion it identifies is real.
The Institute publishes no anonymous submission, because a submission whose author is not published cannot be weighed against a declared interest and the declaration is the mechanism the whole surface rests on. Material bearing on a factual defect may instead be submitted to the evidence log, where it is assessed on the document and the submitter is named only with consent. The submission remains published in full.
The search date is not on the face of the document
The draft carries a publication date and a review date but not the date on which the evidence was last searched. Those are three different dates and only the third tells a reader how current the assessment is. A document published in one quarter may rest on a search run two quarters earlier, and nothing on the page allows that gap to be measured.
The respondent proposes that the search date be printed adjacent to every certainty rating rather than in the methods section, on the ground that a reader who acts on a rating is unlikely to have read the methods section first.
The secretariat accepts this submission. The distinction between publication, review and search dates is real and the draft did not make it visible where it mattered.
The search date is now printed adjacent to every certainty rating and is carried in the document metadata, so that the interval between the search and the reading is visible without reference to the methods section.
Whether external reviewers are paid is not stated
The respondent states that payment or its absence bears on who is able to review and on what the Institute may expect of them, and that the policy does not address it.
The respondent proposes that the position be stated and that any payment be disclosed at the level of the individual document.
The secretariat accepts this submission. Payment is an interest and the policy treats interests as facts to be published.
The policy now states whether external review is remunerated, records any payment made in respect of a document in that document's front matter, and treats a payment received from the Institute as a declarable interest in the same way as any other.
The terms on which the documents may be reused are not stated
The respondent, a librarian, states that the site carries no statement of reuse terms and that an assessment which cannot be quoted with confidence will be paraphrased instead, which is worse for accuracy.
The respondent proposes an explicit reuse statement.
The secretariat accepts this submission. An unstated position on reuse is a position that will be guessed at.
The reuse terms are now stated explicitly on the legal page and in the document footer, permitting quotation and reproduction with attribution and the document version, and requiring that any reproduction carry the certainty rating that accompanied the quoted statement.
The position of a contributor who withdraws consent is not stated
The respondent states that consent to be named is obtained at submission and that the policy is silent on what happens if it is later withdrawn, which leaves both the Institute and the contributor without a rule.
The respondent proposes that the position be stated in advance and be the same for everyone.
The secretariat accepts this submission. A rule made at the point of the first withdrawal is a rule made about a particular person.
The policy now states that a contributor may withdraw from the register at any time and that the register entry is removed, while a signed submission already published remains published, because the consultation record is a record of what was said and by whom. The position is stated at the point consent is given.
A sortable table implies a comparison the underlying data do not support
The draft presents a sortable table whose columns are drawn from sources of differing quality. The respondent states that sorting on such a column produces an ordering that looks like a ranking and is not one.
The respondent proposes that sorting be disabled on any column whose values are not commensurable.
The secretariat notes this submission and records that the point is correct in principle.
No amendment arises here because every sortable table in the document set already carries a standing statement above it that the ordering is not a ranking and that the values in each column are commensurable only where the column header says so. The proposal to disable sorting was considered and not adopted, because a reader who cannot sort a table generally sorts it elsewhere and without the statement.
References should carry a persistent identifier for every cited source
Several references in the draft carry a journal, a year and a volume but no persistent identifier. The respondent, who works in a library setting, states that retrieval of such a reference is materially slower and that identifiers should be supplied throughout.
The respondent asks in the alternative that where an identifier exists but is not carried, the omission be explained rather than left as a gap the reader must interpret.
The secretariat accepts the second limb of this submission and declines the first. Identifiers are supplied wherever the Institute holds one. Where the Institute does not hold an identifier it will not supply one, because a reconstructed identifier that resolves to the wrong record is a worse defect than an absent one.
Every reference without a persistent identifier now carries an explicit statement that the identifier is not held by the Institute, so that its absence is a recorded fact rather than an apparent oversight.
Disagreement within the assessment committee is not visible
The respondent states that ratified documents present a single position and that a reader cannot tell whether a rating was unanimous or contested.
The respondent proposes that a dissenting view be recorded where a member asks for it to be.
The secretariat accepts this submission. A document that conceals a division presents more certainty about its own judgement than it holds.
A ratification record is now published with every document stating whether the assessment was unanimous, and a member may record a dissenting statement which is published with the document rather than in minutes.
The threshold between a correction and a new version is drawn too high
The respondent states that under the draft a change to a certainty rating could be applied as a correction without a version increment, and that a reader holding a citation to the earlier text would not know the rating had changed.
The respondent proposes that any change to a rating or an estimate force a new version.
The secretariat accepts this submission in part. A change to a rating, an estimate or a conclusion now forces a version increment. Typographical and reference corrections continue to be applied as corrections with a log entry.
The versioning threshold is now stated by category: a change to a certainty rating, an effect estimate or a conclusion produces a new version at a new address, and every other change is a logged correction that states the previous wording.
Where a document is generated rather than authored, the reader should be told
The respondent states that some records in the document set are constructed by crossing existing records, that the prose in them is templated, and that nothing on the page distinguishes them from an authored assessment.
The respondent proposes a standing notice on every generated record.
The secretariat accepts this submission. The distinction is material to how much independent judgement a reader should assume.
Every generated record now carries a notice stating that its prose frame is templated and that its included studies, summary of findings and certainty rating are computed from the underlying records, so that the reader can see which parts were authored and which were derived.
The same concept is given three different names in one document
The draft refers to the same quantity as a response rate, a responder proportion and a categorical outcome in different sections. The respondent, who works in health-technology assessment, states that a reader cannot tell whether the three refer to one thing or to three.
The respondent proposes that the glossary term be used at every occurrence and that the glossary entry be linked at first use in each section rather than only at first use in the document.
The secretariat accepts this submission. The variation was stylistic and its cost to the reader exceeds any benefit.
A single term is now used throughout for each defined concept, and the glossary entry is linked at first use within each section rather than once per document.
The document set should be published in translation
The respondent notes that the assessments concern compounds supplied internationally and that publishing only in English restricts access to the assessment to readers who work in it.
The respondent proposes machine translation of the document set as an interim measure, with human review of the certainty language.
The secretariat does not accept this submission, and records that the underlying point is sound and that the proposed remedy is the difficulty.
A translation whose certainty language has drifted is a different assessment carrying the Institute's name, and the Institute cannot review translations it does not have the capacity to review. The documents remain in English. The submission is published in full because the access problem it identifies is real and unresolved.
Funding should not be accepted from any party the Institute assesses
The respondent states that disclosure does not neutralise an incentive, and that accepting funding from an assessed party creates one whether or not it is acted on.
The respondent proposes an outright prohibition on funding from any assessed supplier or manufacturer.
The secretariat accepts this submission in part. Funding from an assessed party is not accepted for assessment work. A general prohibition covering all activity is not adopted, because it would extend to unrestricted contributions the Institute can accept without any assessment consequence.
The policy now prohibits funding from an assessed party for any activity bearing on that party's assessment, requires that any other relationship be published on the funding page and on every page it could bear on, and records the sums involved rather than the fact of a relationship alone.
Absence of evidence is presented in a form a reader will take as negative evidence
Where the Institute has identified no study, the draft states that no evidence was found. In several places that sentence sits immediately after a paragraph describing an adverse outcome, and the juxtaposition invites the reading that the compound was studied and found wanting.
The respondent proposes a standing formulation, used identically wherever the situation arises, distinguishing an outcome that was studied and not demonstrated from an outcome that has not been studied at all.
The secretariat accepts this submission. The two states are different, they support different decisions, and the draft rendered them in language a reader could not reliably separate.
A standing formulation has been adopted and is applied wherever an evidence gap is recorded, distinguishing an outcome assessed and not demonstrated from an outcome not assessed. The formulation is identical at every occurrence so that it can be recognised at a glance.
The materiality threshold for a declarable holding is set too high
The respondent states that the threshold in the draft excludes holdings a reader would regard as relevant, and that the cost of declaring a small holding is negligible against the cost of a reader discovering an undeclared one.
The respondent proposes that all holdings be declared regardless of size.
The secretariat accepts this submission in part. The threshold is lowered. Declaration of every holding without threshold is not adopted, because an undifferentiated list obscures the holdings that matter.
The materiality threshold is lowered and is now expressed as a proportion of the declarer's holdings as well as an absolute figure, so that a holding material to the person is declarable whether or not it is large in absolute terms.
The policy does not say what happens to a document the Institute is asked to remove
The respondent states that assessments of named parties will attract removal requests, and that a policy silent on them will be applied inconsistently under pressure.
The respondent proposes that the grounds on which a document may be withdrawn be stated in advance and be exhaustive.
The secretariat accepts this submission. A policy written after the first request will be written under the conditions least suited to writing it.
The grounds for withdrawal are now stated exhaustively, being a factual error that cannot be corrected in place and a legal obligation, a withdrawn document is replaced by a notice recording the fact and the ground rather than removed, and every withdrawal is recorded in the corrections log.
The document is unreadable without specialist training
The respondent, a trustee of a patient organisation, states that the draft is written for a reader who already understands certainty grading, and that the people most affected by the subject matter will not reach the assessment at all.
The respondent proposes a plain-language summary at the head of every document, written to the same standard of accuracy as the document itself and not as a promotional abstract.
The secretariat accepts this submission in part. A plain-language summary is added. The proposal that it replace the technical abstract is declined, because the abstract is the part of the document other assessors read and cite.
Every document now opens with a plain-language summary of not more than 150 words, placed above the technical abstract and carrying the same certainty language, so that the two cannot diverge.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.