Public comment period · §3
Draft revision of the Institute’s certainty assessment… — disposition
The disposition of every submission and the amendments that resulted.
§3Disposition and amendments
§3.1Disposition table
Table 1. Every submission with its disposition. Each row links to the submission in full.
| Reference | Respondent | Point raised | Disposition |
|---|---|---|---|
| DRAFT-CERTAINTY-FR/009 | Dr Hortensia Larsson-Ekwueme | It is not clear which provisions bind the assessment committee | Accepted |
| DRAFT-CERTAINTY-FR/004 | Dr Katarzyna Oppenheimer-Ade | The database set omits sources relevant to the compounds in scope | Accepted in part |
| DRAFT-CERTAINTY-FR/001 | Dr Bertrand Steenkamp-Ferreira | The search date is not on the face of the document | Accepted |
| DRAFT-CERTAINTY-FR/013 | Dr Katarzyna Oppenheimer-Ade | The document should state what a reader ought to do | Not accepted |
| DRAFT-CERTAINTY-FR/006 | Dr Kamila Ubertini | Automated assistance in screening should be disclosed and validated | Accepted in part |
| DRAFT-CERTAINTY-FR/015 | Dr Mordecai Glendinning-Uche | Quantitative claims are reproduced without the method that produced them | Accepted |
| DRAFT-CERTAINTY-FR/003 | Dr Ivo Quintanilha | Registry records without posted results are neither screened nor counted | Accepted |
| DRAFT-CERTAINTY-FR/008 | Dr Vasilisa Immelmann | The document should describe how an assessment becomes a decision | Noted, no amendment |
| DRAFT-CERTAINTY-FR/012 | Dr Bertrand Steenkamp-Ferreira | The document set should be published in translation | Not accepted |
| DRAFT-CERTAINTY-FR/002 | Dr Anselm Ashworth-Danquah | The treatment of sponsor-conducted evidence is unworkable where all the evidence is sponsor-conducted | Accepted in part |
| DRAFT-CERTAINTY-FR/005 | Dr Oswin Yorkstone | Declared interests should appear on the document rather than on a separate page | Noted, no amendment |
| DRAFT-CERTAINTY-FR/011 | Dr Anselm Ashworth-Danquah | A superseded version should remain reachable from the version that replaced it | Noted, no amendment |
| DRAFT-CERTAINTY-FR/007 | Dr Ludmila Castellanos-Reid | Indirectness is defined so broadly that any evidence could be downgraded under it | Accepted |
| DRAFT-CERTAINTY-FR/014 | Dr Ivo Quintanilha | The rating conflates absence of evidence with conflicting evidence | Accepted |
| DRAFT-CERTAINTY-FR/010 | Dr Ottoline Oppenheimer-Ade | The conditions for upgrading observational evidence are too permissive | Accepted in part |
| 15 submissions in total. | |||
§3.2Summary by disposition
Table 2. Counts by disposition, with the meaning of each.
| Disposition | Count | Meaning |
|---|---|---|
| Accepted | 6 | The submission is accepted and the draft is amended as proposed. The amendment is recorded in the amendment log of the document it changed and is traceable to the numbered submission. |
| Accepted in part | 4 | Part of the submission is accepted and part is not. The secretariat response states which part is which and on what ground the remainder was declined. |
| Noted, no amendment | 3 | The submission raises a point the Institute accepts but that does not require a change to the draft, most often because the draft already states it elsewhere. The location is given in the response. |
| Not accepted | 2 | The submission is not accepted. The secretariat response gives the reason. A submission that is not accepted remains published in full; the Institute does not remove a submission because it disagrees with it. |
§3.3Resulting amendments
- It is not clear which provisions bind the assessment committee — arising from DRAFT-CERTAINTY-FR/009. Binding provisions are now stated in a fixed form, are listed together in an annex, and a departure from any of them must be recorded in the document it affects with the reason, while descriptive passages are marked as descriptions of practice.
- The database set omits sources relevant to the compounds in scope — arising from DRAFT-CERTAINTY-FR/004. The standard database set is extended, the conditional sources are named together with the trigger for searching them, and every review states which sources were searched and which were not, with the reason.
- The search date is not on the face of the document — arising from DRAFT-CERTAINTY-FR/001. The search date is now printed adjacent to every certainty rating and is carried in the document metadata, so that the interval between the search and the reading is visible without reference to the methods section.
- Automated assistance in screening should be disclosed and validated — arising from DRAFT-CERTAINTY-FR/006. The methodology now requires that any automated assistance in screening or extraction be disclosed in the review, that its output be verified by a human for every included record, and that its measured performance against human screening be reported.
- Quantitative claims are reproduced without the method that produced them — arising from DRAFT-CERTAINTY-FR/015. Every quantitative claim now carries the determination that produced it at the point of use, and figures obtained by non-comparable methods are no longer presented in the same row or sentence.
- Registry records without posted results are neither screened nor counted — arising from DRAFT-CERTAINTY-FR/003. Registry records are now screened as records in their own right, and those meeting the eligibility criteria without a posted result are reported as unreported studies with their registered outcome measures and their completion dates.
- The treatment of sponsor-conducted evidence is unworkable where all the evidence is… — arising from DRAFT-CERTAINTY-FR/002. Sponsor concentration is now reported with every certainty rating as the number of independent sponsors contributing evidence, and a downgrade may not be made on sponsor identity alone but must name the feature of the trial on which it rests.
- Indirectness is defined so broadly that any evidence could be downgraded under it — arising from DRAFT-CERTAINTY-FR/007. An indirectness downgrade now requires the assessor to name the specific difference and to state the mechanism by which it would be expected to change the effect, and the statement is published with the rating so that it can be disputed.
- The rating conflates absence of evidence with conflicting evidence — arising from DRAFT-CERTAINTY-FR/014. A standing formulation has been adopted and is applied to every low and very low certainty rating in the document set. The rating scale itself is unchanged, so that Institute ratings remain readable against published assessments using the same scale.
- The conditions for upgrading observational evidence are too permissive — arising from DRAFT-CERTAINTY-FR/010. An upgrade for a large effect now requires an explicit statement of the confounding structure that would be needed to produce the observed effect and a reason for regarding it as implausible, and the statement is published with the rating.
References cited on this page
References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.
- International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute
Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.
Compound Evidence Institute · CEI-CP-001/3 · https://compoundevidence.com/comment-periods/draft-certainty-framework-v2/disposition/ · retrieved 30 July 2026