Findings and Recommendations
For some applications of telemedicine, more rigorous evaluations will support claims about their value and will encourage their more widespread use. For other applications, better evaluation may discourage adoption, at least until technologies and infrastructures improve or other circumstances change. That is to be expected. The purpose of evaluation—and the purpose of this report—is not to endorse telemedicine but to endorse the development and use of good information for decisionmaking.
The committee recognized that telemedicine applications—like other health services and technologies—will diffuse in some measure despite limited systematic assessment of their benefits and costs. This diffusion may also be marked by too much attention to the more glamorous but not necessarily more cost-effective technologies, although strong incentives to control costs may be weakening tendencies in this direction. Conversely, telemedicine applications may also languish for lack of good evidence documenting their relative value compared to alternative services or for lack of evaluation research identifying the obstacles standing in the way of useful and sustainable programs.
This final chapter builds on the preceding seven chapters. It begins by summarizing the technical and human factors and the policy context that may affect decisions about telemedicine. It next
reviews challenges in evaluating telemedicine. The chapter then presents the evaluation principles set forth by the committee and a summary of the committee's evaluation framework and related recommendations and conclusions.
The Technical, Human, And Policy Context for Telemedicine Evaluations
Telecommunications and information technologies are evolving to provide and support medical care at a distance. Some of these technologies involve incremental improvements in the way familiar tools, such as the telephone, are used; others, such as telesurgery, involve devices and procedures that are still experimental.
The committee found general consensus about technical, behavioral, and policy factors that contribute to the modest implementation and documented success to date of the more technologically advanced forms of telemedicine. On the technical side, those responsible for deploying, sustaining, and managing information and telecommunications systems and programs face an often confusing array of constantly changing hardware and software options, many of which are not tailored to health care users. Assessing the utility of advanced technologies can be difficult, particularly given the frequent need to consider options in combination and not just individually. New systems generally have to be patched together with existing or legacy systems that cannot be immediately replaced. Although many groups are working to develop hardware and software standards, it remains frustrating and difficult to put together systems in which the components operate predictably and smoothly together and function in different settings without extensive adaptation.
The limited adoption of telemedicine also appears to stem from a variety of human factors. Research on factors affecting the acceptance of telemedicine is sparse, but the committee heard considerable consensus about practical, socioeconomic, and system constraints related to
- meager evidence for clinicians that an application will benefit them in their day-to-day practice;
- inadequate assessment of practitioner and community needs by those promoting telemedicine;
- practical difficulties in incorporating telemedicine into daily
- practice and existing physical space, even for informed and amenable clinicians;
- limited, although growing, clinician familiarity with information and telecommunications technologies and, thus, lack of a critical mass of users able to share or document their expertise within and across organizations; and
- uncertainties and even fear about how telemedicine will affect clinicians and organizations in a period characterized by increased competition, structural realignments, and surpluses of some categories of health professionals.
In addition, both state and federal policies have prompted considerable debate and anxiety for those promoting or considering clinical applications of telemedicine. Through most of telemedicine's brief history, Medicare and other payer policies have been viewed as a major problem because they have generally proscribed payment for fee-for-service consultations provided by telephone or otherwise without "face-to-face" contact between patients and clinicians. (Exceptions exist for radiology and similar services that have traditionally not involved such contact.) Behind this refusal lies a fear of excessive and inappropriate use of such consultations. Now, with the continuing shift to per case and capitated payment schemes, the search is increasingly for the least costly means of achieving health objectives or for a reassessment of the acceptable trade-offs between expected costs and expected benefits. As the fear of overuse diminishes, however, concerns about underuse of appropriate services are growing.
Attention has also focused on licensure, malpractice, and confidentiality and privacy—issues that have been largely the purview of state policymakers. Varied and restrictive state licensure laws have attracted attention from those interested in or worried about the interstate practice of telemedicine, and differences in state laws about medical liability create additional anxieties. A range of advances in information and telecommunications technologies are intensifying concerns about the inadequacies of state and federal laws to protect the privacy and confidentiality of personal medical information. Significant support exists for some degree of federal action to regularize protections for sensitive medical data in any form, electronic or otherwise, although disagreement about the extent of these protections blocked legislation through mid-1996.
At the same time that some public policies have posed problems for telemedicine, other policies have been devised specifically to encourage telemedicine. Such policies include demonstration project funding, technical assistance, and infrastructure development that supports not only health care but also educational, governmental, and other purposes.
Challenges And Progress In Evaluating Telemedicine
In facing the call for evidence of effectiveness and cost-effectiveness, telemedicine is not alone. Health care services generally are being subjected to increased scrutiny following more than two decades of escalation of health care costs, concern about wide variations in clinical practice, and questions about the appropriate use of both established and new technologies. This scrutiny has reinforced the drive to develop and implement computer-based patient records and associated electronic information systems, which are key to a wide range of evaluation and performance monitoring efforts in health care.
Challenges Facing Evaluators
Based on its review of efforts to evaluate telemedicine, the committee identified several major challenges faced by those evaluating telemedicine. These difficulties also characterize some other applications of information and telecommunications technologies and, thus, are not unique to telemedicine. The combination of challenges is, nonetheless, formidable and calls for a long view rather than a fixation on immediate results.
First, the rapid advance of information and telecommunications technologies makes evaluations vulnerable to obsolescence as key hardware and software components move from being state of the art to being out-of-date. This prospect sometimes discourages or delays investments in technologies; once an investment is made, it may also discourage more rigorous—and often more expensive—research designs, including experimental clinical trials and quasi-experimental clinical studies. At the least, the potential for rapid change in telemedicine technologies puts a premium on careful assessment of
the prospects for reasonably analyzing the sensitivity of results to potential changes in cost, transmission speed, or other variables.
Second, most telemedicine applications depend on technical and human infrastructures that are complex, incomplete, and sometimes unwieldy. Until those infrastructures become more ubiquitous and user-friendly (e.g., flexible, easy-to-use work stations located where clinicians work), evaluations of costs and acceptability will often prove disappointing to proponents of the programs and applications that depend on this common infrastructure.
Third, telemedicine is not a single, homogeneous technology but a family of quite diverse technologies that must be evaluated accordingly, that is, through discrete evaluations of specific applications. Some of these individual evaluations may be aggregated to provide broader perspectives on related applications or purposes, for example, health information management, patient care management, or access to appropriate health care for remote populations. Overall, however, the multiplicity of technologies, clinical uses and users, economic circumstances, and relevant comparison groups or interventions encompassed by telemedicine applications may limit such aggregation of results.
Fourth, the dazzling array of advanced information and telecommunications technologies can distract managers and evaluators from the task of identifying the least costly and most practical ways of achieving defined quality, access, or cost objectives. A preoccupation with glamorous technologies may also interfere with efforts to distinguish the conditions under which a telemedicine application is likely to become a sustainable element of day-to-day health care delivery in an environment dominated by cost concerns.
Fifth, evaluators in rural and even urban sites have found it particularly difficult to design and then recruit appropriate comparison groups, to generate a sufficient number of cases from both experimental and comparison sites for reliable comparisons, and to assure compliance with the research protocol when multiple institutions and investigators are involved. Initial efforts to combine data from multiple sites have experienced some difficulty but may, over time, become more feasible.
Sixth, providing or supporting medical care at a distance may require an unusual level of cooperation among institutions and individuals not bound by common organizational affiliation and governance
structure. Although cooperation cannot be taken for granted even within a single institution (e.g., an academic medical center), collaboration generally is more demanding to create and maintain when more independent actors are involved.
In addition to these factors, several more general challenges may also complicate evaluations of clinical telemedicine. One of these is the restructuring of the nation's health care delivery system, which has brought with it shifts in institutional missions and priorities and more instability for health personnel, research funding, and project management. Another challenge is the increasing role of for-profit enterprises that must provide shareholders a return on their investments and that are less free to allocate resources to support health services and evaluative research, access for disadvantaged groups, and similar purposes that do not add to corporate profits. At the state and federal level, policymakers are cutting budgets (or budget growth) in many areas, and they may be reluctant to shift even modest resources from the core activities of grant programs to evaluation research on their actual consequences.
Further, as some of the technological bases of telemedicine become more familiar, the evaluation questions for clinical applications of telemedicine are getting harder in some respects. Although it is still appropriate to ask "Can it be done?" or "Is it safe?" for some applications, the questions are increasingly the more difficult ones: How well can it done? Under what circumstances? At what cost? With what balance of benefits and risks to particular patients or populations? In comparison to what specific alternatives?
Again, these are questions that all health services face. Telemedicine is not unique.
Progress in Improving Telemedicine Evaluations
The committee found that a number of government and private organizations have clearly recognized the limited evaluative base for telemedicine applications and, as a consequence, have supported individual evaluation projects and more general work to define more rigorous and practical evaluation strategies. In addition to the National Library of Medicine, which funded this study (with a contribution from the Health Care Financing Administration), several agencies of the U.S. Department of Health and Human Services and various Department of Defense units have developed telemedicine
evaluation initiatives. A federal Joint Working Group on Telemedicine is also working to strengthen and coordinate evaluation efforts.
In some respects, the military and the veterans' health systems offer particular opportunities for systematic evaluations of telemedicine. These systems have large, defined populations; integrated health care delivery and financing systems; salaried, full-time personnel; freedom from state regulation; integrated medical records; and easier access to follow-up data on patients. Although not all of these characteristics (e.g., freedom from state regulation) are completely matched by private managed care plans and integrated health systems, some lessons learned from military and veterans' system evaluations may be more transferable than those derived from the fee-for-service environments.
Framework For Evaluating Telemedicine
The committee's framework for evaluating telemedicine has several components. The first presents a set of basic evaluation principles. The second presents the case for careful evaluation planning to establish objectives and priorities well in advance of implementation. The third describes the key elements of an evaluation, and the fourth outlines the primary questions about quality, access, cost, and patient and clinician perceptions that will form the starting point for most evaluations of specific clinical applications of telemedicine.
In Chapter 6, the committee set forth several principles to guide the design—and the use—of its evaluation framework for clinical applications of telemedicine. These principles called for evaluation to be treated as an integral part of program design, implementation, and redesign and to be understood as a cumulative and forward-looking process for building knowledge rather than as an isolated exercise in assessing individual projects. Other principles put the emphasis on identifying the least costly and most practical ways of achieving desired results rather than investigating the most exciting or advanced telemedicine options. Potential benefits and costs should be broadly defined to encourage an assessment of overall effects (including unanticipated or unwanted effects) on all significant parties.
A key principle stressed that comparison is the core of evaluation.
Planning for Evaluation
Evaluation planning tailors general evaluation concepts and methods to fit specific circumstances and concerns. Any systematic evaluation, even those that involve retrospective assessments of established programs, must define objectives that reflect the perspectives of the evaluation's sponsors and the concerns of its target audiences. Because more applications and evaluation questions exist than do resources available to answer them, sponsors and planners of single or multiple evaluations must set priorities among potential applications and questions.
A critical planning step is the precise specification of the telemedicine application to be evaluated, the alternatives to which it will be compared, the outcomes of interest, the expected relationships between interventions and outcomes, and other factors that might affect these relationships. These specifications will help evaluators devise an evaluation strategy, although choices among research design and measurement options will also be shaped by practical considerations related to such matters as financial and human resources, timetable, and organizational relationships.
The committee understood that no "one size fits all" evaluation plan exists or can be devised to fit the array of objectives, settings, clinical conditions, populations, and technologies that characterize telemedicine. Moreover, a project that serves as an early "test of concept" or demonstration of basic technical and procedural feasibility for a new application will generally call for a different research strategy than one intended to help decisionmakers determine whether to adopt a more developed application as part of its routine operations. In addition, somewhat different evaluation strategies may be appropriate depending on whether the purpose is to inform decisions at the clinical or patient care level, the level of institutional strategy, and the system or societal level.
Nonetheless, diversity is not an excuse for divergence from basic standards for evaluation research. It does, however, challenge the principle of cumulative research. Thus, the committee encourages those sponsoring and funding a number of different projects to consider how the project evaluations might be designed to reinforce and
supplement each other despite differences in the objectives, applications, and other characteristics of the projects.
Elements of an Evaluation
Drawing on the committee's own experiences in various health care fields, its review of telemedicine applications and evaluations, and its understanding of the general evaluation literature, the group set forth basic elements that should be included in the planning, conduct, and reporting of telemedicine evaluations. The elements presented in Box 8.1 constitute a basic foundation upon which evaluations of specific applications could build. The framework emphasizes the careful description and monitoring of characteristics of the test and control sites (e.g., organizational structure, financial environment, provider characteristics and relationships), patient characteristics, infrastructure elements, and care processes.
Many elements in this framework can be found in most evaluation handbooks, regardless of the topic. Two aspects of this framework for evaluating telemedicine, however, are less common. The first is the emphasis on both documenting how the technical infrastructure and the clinical processes of care were intended to operate and tracking what actually does occur. This is crucial if evaluators who find negative results are to determine, for example, whether the hypothesis linking independent and dependent variables is untenable or whether the hypothesis was not actually tested because the application was not implemented as intended. By tracking what actually happened, evaluators also may achieve a fuller understanding of critical success factors or the factors that, if changed, might improve results.
A second aspect of the committee's evaluation framework that warrants emphasis is its inclusion of a business plan. The intent is to underscore the importance of practical evaluations of telemedicine, particularly for applications that are beyond the "test of concept" stage. A business plan explicitly states how the evaluation will provide information to help decisionmakers determine whether a telemedicine application is useful, consistent with the overall strategic plan, and sustainable beyond the test phase. For most private organizations (whether for-profit or not-for-profit) and for some public organizations (e.g., public hospitals), the key elements of the typical business plan are financial projections of start-up and operating
Project description and research question(s): the application or program to be evaluated and the basic questions to be answered by the evaluation.
Strategic objectives: how the project is intended to serve the sponsor or parent organization's purposes.
Clinical objectives: how the telemedicine project is intended to affect individual or population health by changing the quality, accessibility, or cost of care.
Business plan or project management plan: a formal statement of how the evaluation will help decisionmakers judge whether and when the application will be a financially and otherwise sustainable enterprise or, less formally, what the project's management, work plan, schedule, and budget will be.
Level and perspective of evaluation: whether the focus of the research question(s) and objectives is clinical, institutional, societal, or some combination.
Research design and analysis plan: the strategy and steps for developing valid comparative information and analyzing it.
Experimental and comparison groups: characteristics of (a) the group or groups that will be involved in testing the target telemedicine application and (b) the group or groups that will receive alternative services for purposes of comparison.
Technical, clinical, and administrative processes: as planned and actually implemented, the communications and information systems, the methods for providing medical care, and the supportive organizational processes.
Measurable outcomes: the variables and the data to be collected to determine whether the project is meeting its clinical and strategic objectives.
Sensitivity analysis: the inclusion of techniques to assess to what extent conclusions may change if assumptions or values of key variables changed.
Documentation: the explicit reporting of the methods employed in the evaluation and the findings so that others can determine how the results were established.
costs, income and cash flow, and a break-even analysis. For public organizations that depend on government appropriations and that do not generate significant revenues, the business plan would still include some estimate of start-up and operating costs, but projections would link expected net costs against expected budgets.
Interpreting the category broadly, the evaluation literature in telemedicine is weighted toward nonexperimental studies with occasional attempts at quasi-experimental or even experimental research designs. The committee particularly encourages researchers to look beyond nonexperimental designs (e.g., simple before-and-after studies of a single intervention with no comparison case) to more rigorous experimental and quasi-experimental designs, including those that attempt to control some important threats to validity through statistical adjustments when random assignment of participants, homogeneous populations, or strict treatment protocols are not feasible. Sophisticated computer-based patient information systems are making such quasi-experimental designs more practical and robust. Random assignment of a large enough number of patients for meaningful comparisons may be realistic mainly in integrated governmental health systems, such as those operated by the Department of Defense and Department of Veterans Affairs. The committee commends researchers and research sponsors that attempt collaborative trials of telemedicine involving independent institutions.
Budgetary constraints notwithstanding, the committee encourages all federal agencies providing grants for demonstration projects to strengthen the provisions for formal evaluation of individual projects and encourages agencies to provide technical support and fund innovative research and methodology development activities. Further, given the relative sparsity of evaluations of telemedicine, the committee urges those sponsoring and funding a number of different projects to consider how their project evaluations might be designed to reinforce and supplement each other despite differences in the objectives, applications, and other characteristics of the projects. The efforts of the federal Joint Working Group on Telemedicine and of individual agencies such as the Office of Rural Health Policy to provide general direction and technical guidance to federally funded telemedicine projects are important steps in this direction.
In the private sector, the committee likewise encourages organizations considering telemedicine to build evaluation into their program plans. They can also demand more complete and relevant documentation of costs and promised benefits. Further, the committee recognizes the role of peer-reviewed publications in encouraging more systematic evaluation and better reporting of results.
Evaluating Quality, Access, Cost, and Acceptance
From project to project and application to application, specific questions and evaluation criteria will vary because clinical telemedicine varies so much in the patient problems addressed, the specific outcomes sought, the diagnostic and therapeutic strategies employed, and information and telecommunications infrastructure required. For example, the dimensions of access relevant for teleradiology can be expected to differ from those considered for telepsychiatry. The health outcomes and cost data relevant for chronically ill homebound patients will not match those relevant for emergency medicine, although some measures of patient and clinician acceptance may be virtually the same. Some evaluations may appropriately stop short of directly assessing health outcomes and consider only the adequacy of the information (e.g., image resolution for mammograms) available from the application compared to the alternative.
Insofar as possible, evaluators should begin with some sense of what constitutes favorable or unfavorable (or acceptable or unacceptable) outcomes for a telemedicine application in a particular context. Again, the specification may vary from evaluation to evaluation. Superior outcomes at equivalent cost may be the goal for some evaluations, whereas others may look for reduced cost with equivalent quality. For evaluations that are beyond the "test of concept" phase, a central question will often be: Do the quality, access, cost, and other results suggest whether and how the telemedicine program can be sustained beyond the evaluation stage?
Given the large number of possible quality, access, cost, and acceptability measures for different clinical applications of telemedicine and the difficulty of stipulating many of them in the abstract, the committee did not present application-specific measures and criteria. Instead, they identified several sets of basic questions to guide the selection of evaluation criteria or measures for particular telemedicine evaluation projects (see Box 8.2).
Although the questions about quality, access, cost, and patient and clinician perceptions are presented sequentially, their interactions and interrelationships also warrant evaluation. This is most obvious in cost-effectiveness analyses that relate the costs of an application to its benefits. It is also plausible that the timeliness of care—an element of access as defined here—may have important
NOTE: Each question assumes an analysis of results will control for severity of illness, comorbidities, demographic characteristics, and other relevant factors.
consequences for quality through earlier detection and better management of clinical problems. Patient satisfaction is frequently considered in evaluations of the quality of care. Satisfaction or acceptance data are, however, both important in their own right (to the extent that successful telemedicine depends on patient and clinician acceptance) and insufficient as the sole measure of quality.
Proper interpretation of patient outcomes data requires good information on patient characteristics, in particular, the severity of their health problem and any comorbid conditions. The methods for adjusting for differences in patient severity and other patient factors are not completely satisfactory and remain the subject of some disagreement, but evaluators should make an effort to identify and adjust for differences in patient characteristics. Such adjustments are also important in evaluating cost comparisons.
Although significant conceptual and practical challenges remain, the last two decades have seen substantial strides forward in both measurement methodologies and data collection. Progress is particularly notable in the areas of quality assessment and health outcomes measurement. Public and private utilization databases and clinical information systems have grown in both size and scope, as has an appreciation of the complexities of assuring that the information is reliable, valid, complete, and secure from inappropriate disclosure
or use. The computer-based patient record is becoming a reality in many institutions. Nonetheless, for telemedicine and other services and technologies, the health system is still some distance from an integrated, longitudinal record that would allow patient outcomes to be compared and tracked across different settings, providers (including different managed care plans), time periods, and episodes of illness.
The committee found, in sum, that telemedicine is similar in most respects to other technologies for which better evidence of effectiveness is also being demanded. Telemedicine, however, has some special characteristics—shared with information technologies generally—that warrant particular notice from evaluators and decisionmakers. Most notably, telemedicine is not a single technology or a discrete set of related technologies; it is, rather, a large and very heterogeneous collection of clinical practices, technologies, and organizational arrangements. In addition, widespread adoption of effective telemedicine applications depends on a complex, broadly distributed technical and human infrastructure that is only partly in place and is being profoundly affected by rapid changes in health care, information, and communications systems. The difficulties encountered during more than two decades of work to implement integrated information systems suggests the importance of persistence and realism for those working to demonstrate telemedicine's promise.
Special challenges notwithstanding, more rigorous and systematic evaluation is as necessary for telemedicine as it is for other technologies. Decisionmakers still do not have good enough information comparing the effects of telemedicine applications to those of alternative health care strategies for quality, access, cost, and acceptability. Decisionmakers also lack good analyses of the infrastructure implications and requirements for sustaining telemedicine past an initial "test of concept" period. The evaluation and implementation of telemedicine projects will benefit from a careful project and evaluation plan and, when appropriate, a business plan that pays close attention to this infrastructure, to project management fundamentals, and to the relationship between the project/business plan and the mission and strategic plan of the parent institution. Although
individual telemedicine projects and evaluation approaches will vary, general adoption of an evaluation framework that includes the elements specified here would strengthen the rigor and cumulative value of telemedicine evaluations and results.
As noted at the beginning of this chapter, more rigorous evaluations of some telemedicine applications will produce positive findings that will, in turn, encourage wider adoption of these applications. In other cases, the results may be disappointing, yet they may also stimulate further technical innovation and more attention to user needs and circumstances. Thus, even negative results can be viewed as opportunities. The task for evaluators is not to justify telemedicine as such but to provide the credible and relevant information that people need to make immediate decisions and plans for the future. The framework presented here is offered in that constructive spirit.