Draft methodology document, version 4 — disposition

§3Disposition and amendments

§3.1Disposition table

Table 1. Every submission with its disposition. Each row links to the submission in full.

§3.2Summary by disposition

Table 2. Counts by disposition, with the meaning of each.

§3.3Resulting amendments

1. The treatment of sponsor-conducted evidence is unworkable where all the evidence is… — arising from DRAFT-METHODOLOGY-/001. Sponsor concentration is now reported with every certainty rating as the number of independent sponsors contributing evidence, and a downgrade may not be made on sponsor identity alone but must name the feature of the trial on which it rests.
2. Quantitative claims are reproduced without the method that produced them — arising from DRAFT-METHODOLOGY-/002. Every quantitative claim now carries the determination that produced it at the point of use, and figures obtained by non-comparable methods are no longer presented in the same row or sentence.
3. The framework does not say when a concern warrants one level and when it warrants two — arising from DRAFT-METHODOLOGY-/003. Worked criteria are now published for imprecision, inconsistency and indirectness. For risk of bias and publication bias the judgement is left to the assessors with a requirement to state the reason, because a criterion expressed in advance would be applied…
4. Imprecision is judged against a relative threshold where an absolute one is what matters — arising from DRAFT-METHODOLOGY-/004. Imprecision is now judged on the absolute scale wherever a baseline risk is available from the contributing trials, with the baseline and its source recorded in the domain note.
5. References should carry a persistent identifier for every cited source — arising from DRAFT-METHODOLOGY-/005. Every reference without a persistent identifier now carries an explicit statement that the identifier is not held by the Institute, so that its absence is a recorded fact rather than an apparent oversight.
6. The rating conflates absence of evidence with conflicting evidence — arising from DRAFT-METHODOLOGY-/006. A standing formulation has been adopted and is applied to every low and very low certainty rating in the document set. The rating scale itself is unchanged, so that Institute ratings remain readable against published assessments using the same scale.
7. The treatment of imprecision does not work where the event did not occur — arising from DRAFT-METHODOLOGY-/007. The framework now states the rule for zero-event outcomes: the rating is determined by the total exposure and the frequency that exposure could have detected, and the resulting statement records that the evidence is uninformative rather than reassuring.
8. Interoperability with published certainty guidance must be preserved in any revision — arising from DRAFT-METHODOLOGY-/008. The framework states at first use that the Institute rating scale is the four-level scale used in published guidance, and the distinction between absent and conflicting evidence is carried in an accompanying formulation rather than as a fifth level.
9. The framework does not say what new evidence would change a rating — arising from DRAFT-METHODOLOGY-/009. Every rating now carries a statement of what would change it, and the search date against which it was made. A trial meeting the stated description triggers reassessment ahead of the ordinary cycle.
10. The framework should say when a rating must not be issued at all — arising from DRAFT-METHODOLOGY-/010. Where no eligible study exists, no certainty rating is issued, no domain assessment is rendered, and the outcome is recorded as not assessed with the reason stated. The rule is applied retrospectively across the series.
11. Nothing states how many assessors rate a body of evidence — arising from DRAFT-METHODOLOGY-/013. Ratings are made independently by two assessors and adjudicated by a third where they differ. The method and the fact of any adjudication are now recorded with the rating.
12. The risk-of-bias instrument is not named, so a judgement cannot be reproduced — arising from DRAFT-METHODOLOGY-/014. The instrument and version are now named in the framework and recorded on every assessment, so that a judgement can be checked against the instrument that produced it.
13. It is not clear which provisions bind the assessment committee — arising from DRAFT-METHODOLOGY-/015. Binding provisions are now stated in a fixed form, are listed together in an annex, and a departure from any of them must be recorded in the document it affects with the reason, while descriptive passages are marked as descriptions of practice.
14. The conditions for upgrading observational evidence are too permissive — arising from DRAFT-METHODOLOGY-/016. An upgrade for a large effect now requires an explicit statement of the confounding structure that would be needed to produce the observed effect and a reason for regarding it as implausible, and the statement is published with the rating.
15. Preprints and conference abstracts are excluded categorically — arising from DRAFT-METHODOLOGY-/018. Preprints and conference abstracts are now screened, listed and reported as a distinct evidence class with their status stated, contribute to the narrative assessment, and are excluded from pooled estimates with the reason recorded rather than excluded at…
16. Screening is described in terms that two people would apply differently — arising from DRAFT-METHODOLOGY-/019. Screening criteria are now operationalised as tests that can be applied without further judgement, screening is performed in duplicate, and the agreement observed together with the resolution of disagreements is reported in every review.
17. Automated assistance in screening should be disclosed and validated — arising from DRAFT-METHODOLOGY-/020. The methodology now requires that any automated assistance in screening or extraction be disclosed in the review, that its output be verified by a human for every included record, and that its measured performance against human screening be reported.
18. Indirectness is defined so broadly that any evidence could be downgraded under it — arising from DRAFT-METHODOLOGY-/021. An indirectness downgrade now requires the assessor to name the specific difference and to state the mechanism by which it would be expected to change the effect, and the statement is published with the rating so that it can be disputed.

References cited on this page

References are numbered in order of first citation in this document. Each superscript in the text links to its entry below.

1. International Organization for Standardization. ISO/IEC 17025:2017 General Requirements for the Competence of Testing and Calibration Laboratories. ISO/IEC Standard 2017;3rd edition. identifier not held by the Institute

Identifiers are reproduced only where the Institute holds them. Where a digital object identifier or PubMed identifier is not shown, the Institute has recorded the journal and year and has not constructed an identifier.