Draft certainty index — submissions
The 15 submissions received, published in full with declared interests and secretariat responses.
§2Submissions and responses
15 submissions were received. Each is published in full below with its declared interest, the secretariat response and the disposition. The Institute publishes submissions it did not accept in the same form as those it did.
Absence of evidence is presented in a form a reader will take as negative evidence
Where the Institute has identified no study, the draft states that no evidence was found. In several places that sentence sits immediately after a paragraph describing an adverse outcome, and the juxtaposition invites the reading that the compound was studied and found wanting.
The respondent proposes a standing formulation, used identically wherever the situation arises, distinguishing an outcome that was studied and not demonstrated from an outcome that has not been studied at all.
The secretariat accepts this submission. The two states are different, they support different decisions, and the draft rendered them in language a reader could not reliably separate.
A standing formulation has been adopted and is applied wherever an evidence gap is recorded, distinguishing an outcome assessed and not demonstrated from an outcome not assessed. The formulation is identical at every occurrence so that it can be recognised at a glance.
Two factual descriptions of the sponsor's programme are inaccurate
The submission is made on behalf of the marketing-authorisation holder and is confined to two matters of fact. The draft describes a trial as terminated where the sponsor closed it at a pre-specified interim analysis, and gives a dose in a unit that does not match the approved labelling.
Supporting documentation, comprising the published trial report and the current summary of product characteristics, accompanied the submission. No view is expressed on the certainty ratings, which the sponsor considers a matter for the assessment committee.
The secretariat accepts this submission. Both points are matters of fact, both were checkable against documents the Institute holds, and both were wrong in the draft.
The trial status now reads as completed at a pre-specified interim analysis, with the analysis identified, and the dose is stated in the unit used in the approved labelling. The submission is identified as an industry submission on its face in accordance with the conflicts policy.
References should carry a persistent identifier for every cited source
Several references in the draft carry a journal, a year and a volume but no persistent identifier. The respondent, who works in a library setting, states that retrieval of such a reference is materially slower and that identifiers should be supplied throughout.
The respondent asks in the alternative that where an identifier exists but is not carried, the omission be explained rather than left as a gap the reader must interpret.
The secretariat accepts the second limb of this submission and declines the first. Identifiers are supplied wherever the Institute holds one. Where the Institute does not hold an identifier it will not supply one, because a reconstructed identifier that resolves to the wrong record is a worse defect than an absent one.
Every reference without a persistent identifier now carries an explicit statement that the identifier is not held by the Institute, so that its absence is a recorded fact rather than an apparent oversight.
The front matter delays the reader from reaching the assessment
The respondent states that a reader must pass through identifiers, dates, contributor lists and declarations before reaching any content, and that the most important sentence in the document is the furthest from the top.
The respondent proposes that the front matter be moved to the end.
The secretariat accepts this submission in part. The assessment statement is moved to the top. The front matter is not moved to the end, because it is what allows the assessment to be checked and a document whose provenance is at the bottom will be quoted without it.
Every document now opens with its assessment statement and its certainty rating, with the identifiers, dates and declarations following immediately beneath rather than above, so that the first thing read is the finding and the second is its provenance.
The anchor outcome for an indication is not the outcome a decision turns on
The respondent, a practising clinician, states that for two indications in the draft set the anchor is the outcome most often measured rather than the outcome that would determine whether to treat.
The respondent proposes that the anchors be changed and gives the alternatives.
The secretariat accepts this submission in part. One anchor is changed. The other is retained because no contributing trial measures the alternative, and an anchor no study reports produces a series of documents that can report nothing.
One anchor outcome is changed to the outcome proposed, and the second is retained with the alternative recorded as the preferred anchor and listed as an evidence gap, so that the reason for the compromise is on the record.
Indications for which compounds are supplied are absent from the set
The respondent states that the indication set is drawn from approved and investigated uses, and that several uses for which compounds are widely supplied have no entry, so no document can address them.
The respondent proposes that such uses be added as indications with no anchor outcome and an explicit statement of the evidence position.
The secretariat accepts this submission. An indication set that omits the uses a reader is most likely to encounter cannot serve that reader.
Uses observed in supply are now included in the indication set, marked as having no approved status and no anchor outcome, and each carries an explicit statement that no trial evidence was identified where that is the case.
An identifier that encodes the document type will have to change when the type does
The respondent states that the proposed identifier embeds the series and the type, and that a document reclassified after publication would then require a new identifier, breaking every citation to it.
The respondent proposes that the identifier carry no semantic content and that type be a field rather than part of the identifier.
The secretariat accepts this submission. An identifier that has to change is not an identifier.
The identifier is now opaque and permanent from first publication, the series and type are recorded as fields that may change, and a reclassified document keeps the identifier it was issued with.
The same quantity is presented differently in two series
The respondent states that a certainty rating is displayed one way in the monograph series and another in the synthesis series, and that a reader moving between them cannot be sure the two mean the same thing.
The respondent proposes a single presentation across all series.
The secretariat accepts this submission. A difference in presentation is read as a difference in meaning.
Certainty ratings are now presented identically across every series, using the same badge, the same wording and the same accompanying formulation, and the presentation is specified in the series structure document rather than left to each series.
The index shows ratings without the reasons behind them
The respondent states that a list of low ratings without their causes tells a reader that the evidence is weak but not in what respect, and that the respect is what determines whether new research would help.
The respondent proposes that the downgrade domains appear in the index row.
The secretariat accepts this submission. The reason is more actionable than the rating.
Every index row now carries the downgrade domains alongside the rating, and the index can be filtered by domain so that a reader can see, for example, every outcome downgraded for imprecision across the document set.
The registry is published only as pages and not in a form that can be processed
The respondent, a graduate student, states that examining the document set as a whole requires extracting fields from rendered pages, and that a structured record would make the registry usable for research about the Institute itself.
The respondent proposes that the registry fields be published in a structured form alongside the pages.
The secretariat accepts this submission. A body that asks others to publish checkable records should publish one.
The registry fields are now emitted in a structured form alongside the rendered pages, carrying the identifier, series, type, version, dates, certainty rating and status for every document, and the structured record is regenerated with the site so that it cannot drift from the pages.
An index organised by certainty invites comparison between ratings that are not comparable
The respondent states that a rating assigned to an analytical question and a rating assigned to a clinical outcome sit in the same band in the index, and that placing them together implies a comparability the framework does not claim.
The respondent proposes that the index be organised by domain first and certainty second.
The secretariat accepts this submission in part. The index is grouped by domain. It continues to offer a certainty view, because the whole purpose of the index is to show how much of the output rests on weak evidence.
The index is now grouped by domain with certainty as the ordering within each group, and a standing statement records that ratings in different domains are not directly comparable and states why.
The search date is not on the face of the document
The draft carries a publication date and a review date but not the date on which the evidence was last searched. Those are three different dates and only the third tells a reader how current the assessment is. A document published in one quarter may rest on a search run two quarters earlier, and nothing on the page allows that gap to be measured.
The respondent proposes that the search date be printed adjacent to every certainty rating rather than in the methods section, on the ground that a reader who acts on a rating is unlikely to have read the methods section first.
The secretariat accepts this submission. The distinction between publication, review and search dates is real and the draft did not make it visible where it mattered.
The search date is now printed adjacent to every certainty rating and is carried in the document metadata, so that the interval between the search and the reading is visible without reference to the methods section.
A surrogate anchor should not be permitted at all
The respondent states that permitting a surrogate anchor allows a series of documents to be built on a relationship that has not been demonstrated for the population in question.
The respondent proposes that surrogates be excluded from the anchor set.
The secretariat accepts this submission in part. A surrogate anchor is permitted only where the validation evidence is stated and rated. It is not excluded, because for several indications the alternative is to have no anchor and therefore no series.
A surrogate may now serve as an anchor only where the document states the evidence for the surrogate relationship in the population concerned and rates that evidence separately, and every document with a surrogate anchor carries the rating on its face.
A withdrawn document should not simply disappear
The respondent states that a reader arriving from an external citation at a withdrawn document needs to know that it existed, that it was withdrawn and why, and that a redirect to an index gives none of that.
The respondent proposes a notice at the original address.
The secretariat accepts this submission. A silent removal makes the citation that led there unresolvable.
A withdrawn document is now replaced at its own address by a notice recording its identifier, its title, the date and ground of withdrawal, and a link to whatever supersedes it, and the withdrawal is recorded in the corrections log.
The default ordering of a comparison table carries an implication
The respondent states that whatever column a table is sorted on by default will be read as the column that matters, and that the draft sorts on a score.
The respondent proposes an alphabetical default.
The secretariat accepts this submission. A default is a claim whether or not it is intended as one.
The default ordering is now alphabetical, a standing statement above every comparison table records that the ordering is not a ranking, and the sort state is reset on reload so that a shared link does not carry an ordering the sender chose.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.