Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
ATTRIBUTES OF GOOD PRACTICE GUIDELINES 67 Pretesting Pretesting a set of guidelines on members of the intended user group (for example, practitioners or patients) using a real organization or a set of prototypical cases is desirable. (See also the discussions of reliability/reproducibility and clarity, below.) Description of methods, settings, and results of any pretests of the guidelines should be described. The Forum has been given authority to pretest guidelines, and the committee believes it should exercise that authority. RELIABILITY/REPRODUCIBILITY As conventionally used in a research context, reliability is linked to the measuring, diagnosing, or scoring of some phenomenon such as intelligence or bacterial infection.5 In the context of guidelines, the committee uses the concept to refer to the ability of some method or process to produce consistent results across time or across users, or both. In strictly technical terms, levels of reliability dictate possible (achievable) levels of validity; that is, qualitative and quantitative instruments and tools cannot be valid if they are not reliable. One kind of reliability is methodological. Ideally, if another group of qualified individuals using the same evidence, assumptions, and methods for guidelines development (for example, the same rules for literature review) were to develop guidelines, then they should produce essentially the same statements. In practice, such replications are almost unknown given the expense of the process,6 but discussion of previous trials of the methodology (for different conditions) and any resulting revisions may be useful. Likewise, review of the guidelines by an outside panel can help in assessing reliability. (Recall that independent review is also important to an assessment of validity, a fact that underscores the link between reliability and validity.) 5 The committee discussed how two common methodological concepts, sensitivity and specificity, applied to practice guidelines. For medical review and other evaluation criteria, these two related terms are fairly straightforward. Sensitivity and specificity refer, respectively, to a high "true positive rate" in detecting deficient or inappropriate care and a high "true negative rate" in passing over cases of adequate care. The concepts can be operationalized by requiring some evidence, drawn, for example, from pretesting of the review criteria on "prototype" cases or through pilot-testing in a specific organization. As described in Chapter 10 of the Medicare quality report, case-finding screens have often been found to be deficient on these two attributes. The committee concluded that these attributes need to be considered for evaluation instruments but do not add anything to the assessment of practice guidelines. 6 One effort at replication has been undertaken by those involved with the RAND Corporation's work to develop appropriateness indicators (Chassin, 1990).