Telemedicine A Guide to Assessing Telecommunications for Health Care (1996) / Chapter Skim
Currently Skimming:
6 A FRAMEWORK FOR PLANNING AND IMPROVING EVALUATIONS OF TELEMEDICINE
6 A FRAMEWORK FOR PLANNING AND IMPROVING EVALUATIONS OF TELEMEDICINE
Pages 137-161
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 137... ...
Continued improvement in the field will depend on agreement by those interested in telemedicine that it is important to invest in systematic evaluation of telemedicine's effects on the quality, accessibility, cost, and acceptability of health care. The evaluation framework presented in this chapter attempts to relate broadly accepted strategies ot health services research and evaluation research in general to some of the challenges and problems in evaluating telemedicine that have been described in preceding chapters.
|
From page 138... ...
Third, the benefits and costs of specific telemedicine applications should be compared with those of current practice or reasonable alternatives to current practice. Careful comparison is the core of evaluation.
|
From page 139... ...
This list reflects several decades' worth of work in many disciplines to create scientifically respectable evaluation strategies that are also useful to decisionmakers and feasible to implement (see, e.g., Suchman, 1967; Weiss, 1972; NAS, 1978; Cook and Campbell, 1979; Sechrest, 1979; OTA, 1980a; Rutman, 1980; Wortman, 1981; Tufte, 1983, 1990; Rossi and Freeman, 1989; Mohr, 1988; Flagle, 1990; Wholey et al., 1994~. Although this report was not intended to be a how-to-do-it manual, or to duplicate existing texts, the discussion below briefly discusses the above steps.
|
From page 140... ...
Even if program objectives are relatively clear, other considerations such as evaluation feasibility and anticipated concerns of possible future funders may influence the choice of specific evaluation questions. Government agencies, private foundations, and vendors will usually have interests related to public policies or market strategies that go beyond those of specific demonstration sites.
|
From page 141... ...
to formulate an umbrella framework for project evaluation that is intended to make it easier to aggregate conclusions from individual evaluations. Setting Priorities As is true for any activity, resources for evaluating telemedicine applications are limited, and funding for an evaluation may compete with funding for the services to be evaluated.
|
From page 142... ...
Second, what are the costs of a strategy relative to its potential to provide answers to the evaluation questions? Third, is the cost of the evaluation strategy reasonable in relation to the potential costs and benefits of the application or program to be evaluated?
|
From page 143... ...
Although general methodological and statistical principles exist to guide a multiplicity of evaluation tasks, no "one size fits all" evaluation plan exists. For example, if an evaluation is an early "test of concept" to determine the basic technical and procedural feasibility of a telemedicine application (e.g., home health monitoring)
|
From page 144... ...
These elements include Project description and research questionts) Strateglc 0 electives Clinical objectives Business plan or project management plan Level and perspective of evaluation Research design and analysis plan characteristics of experimental and comparison groups technical, clinical, and administrative processes measurable outcomes .
|
From page 145... ...
Strategic and Clinical Objectives The strategic objectives in an evaluation plan state how the telemedicine project is intended to affect the organization's or sponsor's goals and how the evaluation strategy relates to those objectives. These goals might include improving health services in rural areas, keeping deployed soldiers in the field, reducing expenses for government-funded medical care, or strengthening an organization's competitive position.
|
From page 146... ...
. Depending on the results, later strategic objectives might relate more to the patient care mission or to reinforcing the institution's position in focal, regional, and broader health care markets.
|
From page 147... ...
Or, more specifically, how do particular telemedicine applications compare to other policy options, such as area health education centers, direct subsidies for rural hospitals, and educational loan programs linked to practice in underserved areas? If the evaluation results look positive at this level, decisionmakers may support broad adoption and diffusion of the technology.
|
From page 148... ...
Business or Project Management Plan The committee concluded that a significant weakness of many demonstration projects and their evaluations has been the lack of a business plan that sets forth how the implementation and evaluation of the project are designed to provide information that decisionmakers can use to decide whether the test application is financially sustainable as an ongoing program. It is likely that the demise of many telemedicine programs can be attributed to an incomplete understanding of the business case for establishing and maintaining a telemedicine program and an inadequate appreciation of the costs involved.
|
From page 149... ...
If it is clear that the project is being evaluated as a possible component of its parent organization's overall business plan, then the project business plan usually would include a multiyear summary of the income statement and cash flow projections, with more detailed monthly projections for the first year and quarterly projections for later years. Each should be backed by documentation of assumptions, for example, about revenue sources.
|
From page 150... ...
The starting point for identifying such characteristics is the basic research question for the project, which will suggest a series of additional questions drawn from past research, judgment, and experience about other independent factors or variables that may intensify, block, or confound the relationship between the experimental and dependent variables. These factors usually include but are not limited to patient characteristics (e.g., age, sex, race, severity of illness)
|
From page 151... ...
Clinical processes are the way medical services are to be provided as part of the telemedicine project. Often, they are precisely set forth in a clinical protocol that identifies specific activities, their order and timing, responsible personnel, circumstances that trigger different protocols, and appropriate clinical documentation.
|
From page 152... ...
To cite another case, if military clinicians try to use telemedicine services but find that the clinical protocols are irritating, the equipment does not work, or the consultants are not scheduled appropriately, an evaluation needs to document this and, if possible, suggest how the problem could be resolved. Event or problem logs kept by project personnel may be used to record (for later analysis)
|
From page 153... ...
Depending on the objectives, circumstances, and resources, an evaluation may involve a range of immediate, intermediate, and fong-term outcome measures, as discussed further in Chapter 7. Sensitivity Analysis Because the committee believed that the fast pace of change and other uncertainties surrounding telemedicine applications were particular challenges, it highlighted one element of an analysis plansensitivity analyses as a distinct item in the evaluation framework.
|
From page 154... ...
Documentation of Methods and Results In reviewing evaluations of telemedicine applications, the committee was often frustrated by the incomplete or casual documentation of the methods employed and the specific findings. One result was to diminish the utility and credibility of the reports.
|
From page 155... ...
Another key principle emphasizes close relationships between customers and suppliers, for example, patients and providers or providers and suppliers of equipment or services. The application of this principle to the design and evaluation of telemedicine applications would address one of the human factor problems identified in Chapter 3: inadequate assessment of and attention to user needs.
|
From page 156... ...
Likewise, it has stressed the value of developing a business plan that explicitly states how the evaluation will provide information to help decisionmakers determine whether a telemedicine application is useful, consistent with their strategic plan, and sustainable beyond the initial evaluation stage. The fast pace of change and other uncertainties surrounding telemedicine applications argue strongly for sensitivity analyses to explore how conclusions may change if values of key variables or assumptions change.
|
From page 157... ...
limitations of various common research designs. Internal valiclity focuses on the funciamental question: "Dic!
|
From page 159... ...
because of adverse effects and therapeutic ineffectiveness suggested that "rates reported in randomized clinical trials may not give an accurate reflection of the tolerability or effectiveness of therapy in the general population" under ordinary conditions (Andrade et al., 1995~. From a practical perspective, traditional, tightly controlled RCTs suffer several handicaps: they tend to be expensive, time-consuming, complex to plan and administer, and ethically or practically unsuitable for some research questions.
|
From page 160... ...
Instead, they measure relevant patient characteristics using severity assessment tools and statistically adjust for differences in experimental and comparison groups. Further, they accommodate departures from rigid treatment protocols by carefully monitoring and measuring actual treatments and then incorporating these data in the statistical analysis.
|
From page 161... ...
Another stream of work on alternatives or supplements to the RCT has emphasized nonexperimental research based on the retrospective analysis of large databases that have often been compiled for other purposes (Roos et al., 1982; Moses, 1990; Hannan et al., 1992; NAHDO, 1993~. Until telemedicine applications become much more common and routine and are assigned codes to identify them, large databases are unlikely to be useful sources of data on telemedicine applications.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.