National Academies Press: OpenBook

Vital Signs: Core Metrics for Health and Health Care Progress (2015)

Chapter: 3 Improving the Impact of Measurement

« Previous: 2 Health and Health Care Measurement in America
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

3

Improving the Impact of Measurement

When the health landscape is replete with measures that are too numerous, poorly designed, of limited comparability, and sporadically accessible or applicable, the result is a dilution of focus and an overly burdensome set of requirements and processes that run counter to the basic aim of measuring what matters most. Responding to this challenge means developing a more focused system of measurement; bolstering those measures most critical to understanding and improving health; and downgrading or eliminating measures that are redundant, inaccessible, inaccurate, or impracticable. In setting out to identify a core measure set, the Committee explored the ways in which targeting core priorities can accelerate change, identified criteria for a core measure, developed criteria for a core measure set overall, and considered lessons learned from examples of existing sentinel measurement efforts.

Complexity and rapid proliferation present a significant challenge for health and health care measurement. While health care has the capacity to test and measure almost countless aspects of a patient’s condition, careful consideration is necessary to avoid a strategy that is costly, dangerous, and inefficient for the patient. Similarly, the rapid proliferation of measurement activities within the health system without thoughtful consideration and planning for priorities, focus, and coordination fails to capture a meaningful, actionable picture of the U.S. health system.

A core set of measures centered on what matters most could be utilized through a variety of pathways, leveraging multiple stakeholders and stakeholder coalitions. In the context of the broad range of health determinants and the various policy and program levers at work, often wielded

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

images

FIGURE 3-1 Core measures as levers for enhancing the impacts of the key determinants of health.

by semiautonomous, siloed stakeholders, the natural tendency is toward fragmented intentions, focus, and activities. Figure 3-1 outlines several potential pathways through which core measures, individually and as a set, could accelerate progress, acting to sharpen the focus of programs and policies in shaping the intersecting impacts of the key determinants of health (McGinnis, 1985; McGinnis et al., 1997, 2002).

A parsimonious, standardized set of measures collected regularly and consistently across the nation could enhance the ability of health care leaders and the public to track progress toward shared goals and to work in collaboration to achieve standardization and interoperability in measurement and data systems. If the same set were implemented at the national, state, local, and organizational levels, these benefits would be multiplied as a result of the enhanced ability to make comparisons and determine best practices. While each of the measures in a candidate core measure set could be used for a variety of purposes, the set as a whole would have specific applicability for measuring health, with each measure offering complementary and mutually supportive pathways to improvement.

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

CHANGE AS A PRODUCT OF TARGETING THE CORE ISSUES

Measurement in health care is a tool for improvement, not an end point or a solution in itself. The diversity of the current range of health measures is a reflection of the wide variety of purposes and targets within health care that have the potential to be empirically assessed and systematically monitored or compared as a route to improvement (see Chapter 2). A core measure set is not intended to replace the full range of measures in use today. Rather, a core set can raise the profile of the most compelling health challenges facing the nation; draw attention to issues and actions that can trigger broader-scale system improvement; provide a platform for harmonizing efforts to monitor national, state, local, and institutional progress in health and health care; create opportunities for alignment and the resolution of redundancies in areas in which measurement is burdensome; and guide the creation of a more robust multilevel data infrastructure. Focusing attention primarily on the results of core interest but not prescribing the precise measures to be used to ensure progress toward those outcomes will encourage organizational attention to focus on the most important issues, limit the formal reporting burden, and ensure that other performance improvement measures are tailored to local needs and interests.

Focusing Attention

As discussed in Chapter 2, focusing attention on key outcomes with the potential for broad improvement can enable the orientation of measurement efforts around the outcomes that matter most, reduce the propagation and required reporting of secondary measures, and thereby help reduce the burden of measurement. A core set of performance measures draws attention to the high-priority issues most important to improving health, improving care, lowering costs, and engaging people. Too large a number of measures could distract attention, and thereby dilute the consideration of any particular metric, whereas a parsimonious core measure set can focus attention on the highest priority targets for improving health and health care. Often, the large number of measures used by an institution or community represents inefficiency in the collection of data, driven in part by competing reporting requirements around similar concepts. For example, payers—including private payers, Medicare, and Medicaid—currently are using different measures in their payment incentive programs (Lee et al., 2010). However, clinicians generally do not provide different types of care to patients based on the health plan in which they are enrolled (Baker, 1999; Glied and Zivin, 2002). Multiple sets of different measures may work at cross-purposes by dividing providers’ attention and thereby limiting their ability to significantly improve care in the measured areas.

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

Realigning System Operations

Core measures can encourage consideration of broad, interacting forces and reorientation of the interplay between health systems and leadership to enable decision making aligned with the goal of improving health outcomes as efficiently as possible. Measures provide a window into the performance of complex systems, and the quality, accuracy, and importance of what they show can play a role in determining what steps are taken or what strategies are adopted. A poorly specified measure may lead a health stakeholder to make changes where none are needed or to overlook a significant problem that may not have been captured quantitatively. The management dictum “what gets measured gets done” captures this critical role of measurement in directing productive action.

Similarly, a common set of measures allows variations—whether among different geographic regions, clinicians, or treatments—to be identified and leveraged. For example, a common measurement framework in cardiac surgery allowed one organization to identify variations in clinical outcomes among different providers and then to share the best practices from high performers throughout the organization (IOM, 2013). Others have found that public reporting of performance measures can help organizations identify areas that need improvement and track improvement over time.

Engaging Broadly

Core measures may rally the support and involvement of diverse coalitions of stakeholder groups seeking to improve health and health care, as well as encourage and empower engagement at different levels within an organization, from leadership to facilities and operations. The different partners involved, which might include county-based health departments, health care delivery organizations, community-based organizations, and employers, will have different ways of collecting and storing data and different perspectives on the most pressing areas for improvement. Core measure sets can help these diverse groups work together by defining a common target for improvement and identifying the areas in which data need to be collected and shared. Core measures can also highlight areas of greatest urgency for the health system as well as compelling opportunities for change. In this way, core measures can effect broader alignment at the local, state, and national levels for improving health and care.

Understanding Impact

In addition to engagement, core measures can enable a deeper understanding of the forces at play in America’s health. For example, core

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

measures can allow for improved health monitoring and tracking over time. Many health care organizations today find themselves contending with the need to adjust frequently to new reporting requirements from multiple sources such that data are not necessarily comparable from one cycle or year to the next or from one organization to the other. This issue is especially problematic when one is considering health outcome measures, as effects may be seen only years or even decades after an intervention. A well-specified and -maintained core measure set can bring relative permanence and consistency to monitoring of the health system, such that meaningful comparisons can be made not only among regions and systems but also across time. This functionality can allow for stronger, more meaningful analysis of which approaches and initiatives are making a difference, as well as enable the health system to broadly recognize high performance and, in turn, replicate the most successful programs and policies.

Systems Approaches and Composite Measures

Composite measures and scores represent a potentially powerful tool for managing complexity in assessing health and health care performance. The 2006 IOM report Performance Measurement: Accelerating Improvement discussed composite measures as an approach to integrating performance monitoring across multiple dimensions and, by extension, improving the quality of the information gleaned from performance measurement (IOM, 2006). The prioritization of efficiency in the collection and use of information is also reflected in the principles of systems theory and lean management systems. These management approaches, which were initially cultivated in the manufacturing sector, have since been incorporated and applied in a wide range of industries, including health care.

Lean management, as its name suggests, emphasizes reducing waste and streamlining processes. A critical component of this streamlining is the prioritization of process points that contribute the most value to the final product, which, in the context of the health system, is better health (IHI, 2005). As such, identifying those measures that convey the most meaning and drive the most improvement in performance is both a key element of applying systems thinking to health and health care and a potential role for core measures.

While the number and diversity of health measures is reflective of the complexity of patient needs and characteristics, not all measures contribute equally to improving health. An analysis of the net health benefit of 13 different Agency for Healthcare Research and Quality (AHRQ) quality indicators found that 7 of these measures accounted for 93 percent of total benefits, while the remaining 6 measures accounted for only 7 percent of total benefits (Meltzer and Chung, 2014). Identifying those measures with

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

the largest value-add for health will require significant research and analysis, and the measures needed are likely to evolve over time with changes in health and health care. The use of composites that combine multiple elements with varied weights could enable reporting and performance measurement activities to be more responsive to these changes through adjusting individual elements of composites rather than continually adding new measures to existing activities.

A variety of composites or “scores” have been proposed as potential alternatives to the trend of continually adding new measures and new complexity that may not result in improved information. For example, one recent proposal called for a measurement system that would present a whole-person view of health, while remaining adaptable and flexible for different medical specialties, different patients, and different care settings. An individualized care quality score, in this approach, could be derived from three components: (1) an inventory of patient care needs, (2) a tool for matching those needs with evidence-based care approaches, and (3) patient preferences and health goals (McGlynn et al., 2014). In this way, a single composite score could be used to provide information about multiple facets of care quality and patient experience. This approach to integrating a variety of elements into a single measure or score is also seen in a variety of health and health care reporting activities, such as The Commonwealth Fund’s State Scorecards and the County Health Rankings (McCarthy et al., 2009; RWJF and UWPHI, 2013).

Motivating Innovation

Finally, core measures can encourage broader thinking about ways to impact the forces and elements that underlie health, potentially leading to innovation in approaches and interventions that can improve outcomes. Core measures have a symbiotic relationship with data sources: while data sources are used to calculate core measures, core measures can be used to guide the creation of a robust, rational digital infrastructure. A core set of measures can be used to identify the necessary data elements that a data system should capture as part of routine operations. For example, the Vermont Blueprint for Health used core measure sets to identify the necessary data elements that its electronic health record systems should capture during routine care. In this case, the core set of measures served as the basis for a data dictionary around which the electronic health record system was designed. The resulting system was then able to collect and export these key elements, populate the core measures in a dynamic fashion, and ensure transmission and exchange of the key data elements. Similar principles can

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

apply to other data systems, from multi-payer claims databases to health surveillance systems.

CRITERIA FOR A CORE MEASURE

In preparation for identifying criteria for a core measure set, the Committee discussed what key characteristics would be most critical to its usability and impact. This discussion included a review of criteria used by other groups to assess and compare health measures. These criteria include the importance for health of the issue addressed by a measure, the strength of the measure’s linkage to progress on that issue, the understandability of the measure, the technical integrity of the measure as an indicator of the targeted issue, the potential for broader system impact, and the measure’s utility at multiple levels of focus. Criteria for core measures and for a core measure set (discussed in the next section) are presented in Box 3-1.

Importance for Health

The foundational factor that the Committee considered in its vision for a core measure set was that the issues addressed by the measures should represent the highest-priority issues for improving the nation’s health at every level—from the individual to the overall population. Therefore, the Committee sought to craft a core measure set that would accurately reflect the state of the nation’s health and its health system, highlighting its strengths and, of greatest value, its weaknesses. In this respect, emphasis was given to those issues associated with the greatest health-related societal burden and the component elements of those issues with the most direct potential to make a difference. Focusing measurement on what matters most is a critical prerequisite for progress.

BOX 3-1
Criteria for Core Measure Development

Criteria for core measures

  • Importance for health
  • Strength of linkage to progress
  • Understandability of the measure
  • Technical integrity
  • Potential for broader system impact
  • Utility at multiple levels

Criteria for the set

  • Systemic reach
  • Outcomes-oriented
  • Person meaningful
  • Parsimonious
  • Representative
  • Utility at multiple levels
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

Strength of Linkage to Progress

The Committee envisions core measures as a tool for driving progress toward better health, better care, lower costs, and engaged patients and communities. Accordingly, another critical feature of a core measure set is a strong linkage to progress. Not only should the measures selected reflect the most critical issues at present for the health of the public; they should also be able to show progress over time toward key aims, such that any improvement in the results of core measures should indicate as clearly and directly as possible a real, meaningful advance in the performance and quality of the health system and, more broadly, the health of the public. For some measures, for example, current performance may already be at a high level, such that additional investment in monitoring and improving may be of limited value. An outcomes-based approach allows the measurer to remain agnostic to the strategy or type of intervention used for improvement and engagement and to focus instead on whether results are achieved. But whether a core measure is oriented to a process or an outcome a strong linkage between processes and outcomes and between measures and progress in health is a key requirement.

Understandability of the Measure

The Committee concluded that if a core measure set is to be relevant and meaningful to the full range of health system stakeholders, the content, language, and presentation of the measures must be accessible to a general audience. Thus, the Committee envisioned a core measure set that would be easily understood such that the meaning behind the numbers would be immediately apparent for all stakeholders, from statisticians and measure developers to students, patients, and other individuals. For example, HbA1C is a common metric for diabetes care, but its meaning is not readily apparent to a nonexpert audience. Understanding and relating to such measures as self-reported health status and satisfaction with patient–clinician communication does not require significant background or expertise.

Technical Integrity

Basic to any measurement activity is a measure’s technical integrity—that is, the evidence in support of its reliability as a true reflection of the state of the targeted issue, the robustness of the validation process in its support, the practical ease and likely consistency of its application, and its requirements for statistical power under anticipated use. Distortions on any of these dimensions can negate the measure’s utility or even introduce adverse and unintended consequences. Technical integrity of the measure

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

chosen is its validity, construct, applicability, and statistical power in practical use. As a core measure set for broad-scale use, the development, testing, and application of candidate measures is critical to ensure their technical integrity.

Potential for Broader System Impact

Selecting a small number of measures to represent health at large requires that each measure selected have the capacity to demonstrate and promote progress and change across a range of issues, perspectives, and stakeholder groups. By targeting high-level health outcomes important to a broad range of stakeholders, measures can catalyze improvement across the nation through the alignment of critical stakeholders, from clinicians, to patients, to payers, to employers, to government officials at many levels. While a clinical process measure can bring a care team together around a shared goal, an outcome measure can bring a community or a state together to tackle a complex problem with numerous potential approaches and leverage points. For example, a health care system can measure body mass index (BMI) among its patient population, but making progress toward reducing overweight and obesity calls for the active involvement of communities, schools, employers, and other key stakeholders that play a role in healthy behaviors.

Utility at Multiple Levels of Focus

Any measure selected for a core set should have meaning and relevance at multiple levels. Thus, it should be possible to readily translate a national core set of measures to a state, regional, local, or institutional core set that, while translated to local circumstances, measures progress toward goals measured by the national set. This feature of usability at multiple levels is critical for advancing the ultimate development of a fully interoperable, scalable set of core measures. For example, a measure such as self-reported health status can be implemented for populations at multiple levels, from a small community to the nation as a whole, and the concept of wellness represented by this measure is highly relevant for stakeholders both within and external to the health care system.

CRITERIA FOR THE CORE SET

Building on lessons learned from previous initiatives to select core measures, the Committee developed criteria for the core measure set to guide the selection process. It is important to note that these criteria are intended to apply to the set of measures as a whole, not to the individual measures

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

within it. Additional considerations are needed to construct a high-quality set of measures. Because few organizations have proposed characteristics for a core set, the report focuses its work on criteria for a core set of measures by identifying key attributes that a set as a whole should possess in order to achieve its aims. The core set taken as a whole needs to reflect as much as possible what health care providers, policy makers, business owners, patients, and members of the public view as their overarching goals for health and health care. These criteria, also listed in Box 3-1, are described below.

Systemic Reach

A core measure set needs to capture not only progress on the specific measures it includes but also progress on overarching, meaningful priorities for health across the health system, touching on the full range of actors and stakeholders involved and driving improvement throughout. Further, the core set should be specified such that, taken as a whole, it can capture improvement in performance that indicates meaningful change occurring in the health system and in communities. For example, a core measure set could focus on a particular population, such as Medicare or people with chronic conditions. However, the scope of this core set would be limited, as would its relevance and interest for many stakeholder groups.

Outcomes-Oriented

The Committee concluded that a well-constructed core measure set would focus on outcomes of good health rather than on the processes that might lead to those outcomes. Thus the core set should be agnostic to the route or strategy taken to achieve improvement, encouraging innovation in addressing the highest-priority health problems. Further, a core measure set orientation to outcomes, while importantly incorporating selected process elements, is likely to be a more direct measurement of what a strategy for improvement is intended to achieve. For example, “aspirin at arrival” for acute myocardial infarction is often used as a hospital care quality measure, as it assesses whether clinical standards are being followed in care for a relatively common admission. However, this measure addresses only one element of the broader picture of cardiac care, emergency care, or cardiovascular risk factors; by contrast, outcome measures focused on mortality, readmissions, or management of chronic diseases and risk factors provide a broader view that does not emphasize a particular clinical action or care setting.

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

Person Meaningful

An ideal core measure set will be readily comprehensible and meaningful to a wide range of stakeholders, most critically to lay individuals, including patients and families. This criterion represents a challenge both for the content of the core measure set and its expression and communication strategy. The intent of each measure should be readily apparent to a nonexpert audience, and the core set as a whole should make a clear statement about the health system’s priorities and current performance. For example, a standardized infection ratio for a hospital-acquired infection provides meaningful information about patient safety but is not well understood by the general population; therefore, it does not meet the criterion of being person meaningful.

Parsimonious

A core measure set should comprise the minimum number of measures needed to assess health and health care. Meeting this criterion requires balancing the goals of efficiency and comprehensiveness. Thus, while there is no “right” number of core measures in a set, the Committee worked to identify the smallest number of measures possible and assessed the set as a whole based on the extent to which it balanced the need for comprehensive coverage of the most important health issues and efficiency of expression. The Committee also set basic benchmarks for parsimony, concluding that a set of 50 or 100 measures would be too large to be accessible and meaningful, while a set of fewer than 5 would be too limited to provide a comprehensive view of the health system. Balance, synergy, and representativeness (below) are key to the impact.

Representative

Just as critical as the number of measures is the extent to which they represent the most critical issues and priorities of the American health system. As such, the Committee evaluated the core set using the criterion of “representativeness,” or the extent to which the core set reflected health realities. For example, while care for rare diseases is an important area for improvement in the health system, it does not meet the criterion of representativeness because it represents only a small population of both patients and providers and has limited implications for the elements of health that lie outside of the care system.

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

Utility at Multiple Levels

The measure set as a whole should be useful and relevant at multiple levels of aggregation, from the individual to the national level. The importance of this criterion was discussed above in the section on criteria for individual measures. It is also important to consider how measures interact with each other in a set and how the full set represents or excludes different subpopulations. For instance, a high-quality set could be constructed that assessed care for diabetes and heart disease, yet that set would exclude many people in the population and many parts of the broader health system. The challenge is to construct a set that captures progress toward improving health and health care for the widest possible range of people and throughout the health system.

RELATED EXPERIENCE WITH SENTINEL MEASURES

Core measures serve the purpose of sentinel measures because they capture the ability of the health system to meet critical societal goals and to produce highly valued outputs system-wide. Improving performance on core measures will have far-reaching implications for system and societal health care performance. Some sentinel measures are identified as the best indicator of progress in a particular disease or treatment domain, for example, the reduction of teen pregnancy is an indicator of progress in reproductive health. Improvement on other sentinel measures—that is, measures including core measures that are intended to drive improvement—reflects broader systemic changes: for example, progress against maternal mortality in the early 20th century was associated with the overall improvement of public health capacity and led to the coining of the term “sentinel indicators” (Rutstein et al., 1983). The Committee considered a wide range of sentinel measurement initiatives throughout its deliberations, and drew on lessons learned from these experiences. Box 3-2 lists the sentinel measurement initiatives that the Committee considered closely in its review, and these measure sets are also reproduced in full in Appendix D. Figure 3-2 illustrates the heterogeneity of measurement areas and topics covered by these sentinel measurement initiatives.

Because experience with sentinel measures is relevant to the potential impact of a core measure set, the Committee assessed several efforts to develop such measures. In particular, the Committee identified areas of commonality in these efforts as well as differences among them both in the content of the measures and in implementation and dissemination. Appendix D presents a catalog of prominent core measurement initiatives, illustrating areas of convergence and divergence. While neither a census nor a representative sample of current core measurement-related activities,

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

BOX 3-2
Sentinel Measurement Activities Considered

  • ASPE Health System Measurement Project
  • Blue Cross Blue Shield of Massachusetts: Alternative Quality Contract
  • Buying Value Coalition: Buying Value Ambulatory Core Set
  • Canadian Institute for Health Information: Canadian Health System Performance Measurement
  • CDC Surveys (e.g., NHANES, NHCS, NHIS, NVSS)
  • CMS: Health Homes Core Measures
  • CMS: Medicaid Adult Health Care Quality Core Set
  • CMS: Medicaid/CHIP Children’s Health Care Quality Measures (2013 Set)
  • CMS: Medicare Advantage Rating Measures
  • CMS: NQF Evolving Core Measure Set for Dual Eligible Beneficiaries
  • CMS: Shared Savings Program (ACOs)
  • The Commonwealth Fund: Why Not the Best?
  • Consumer Reports Health: Hospital Quality Measures
  • CQO Roundtable: Illustrative Set of Quality, Outcome, and Cost Measures
  • DOD: Military Health Service Strategic Imperatives Scorecard
  • HHS: Leading Health Indicators for Healthy People 2020
  • HHS: National Quality Strategy Measures
  • HRSA: Core Clinical Measures
  • IHA: P4P California Core Measure Set
  • IHI: Measures for Triple Aim Communities
  • Joint Commission: Accountability Measures
  • Joint Commission Example: Acute Myocardial Infarction Core Measure Set
  • Leapfrog: Hospital Safety Score Methodology
  • NCQA: HEDIS Measures (Health Plans, 2013)
  • ONC: Meaningful Use Clinical Quality Measures for Eligible Hospitals (2014)
  • ONC: Meaningful Use Clinical Quality Measures for Providers (2014)
  • Oregon Health Authority: Coordinated Care Organization Core Measures
  • Patient-Centered Medical Home Evaluators Collaborative
  • Premier: QUEST Measures
  • State of California: Let’s Get Healthy California
  • State of Massachusetts: Standard Quality Measure Set
  • State of Minnesota: Statewide Quality Reporting and Measurement System
  • State of the USA Health Indicators
  • State of Vermont: ACO Core Measure Set
  • UnitedHealth Foundation: America’s Health Rankings
  • University of Wisconsin: County Health Rankings
  • Veterans Health Administration: ASPIRE Measure Set
  • World Health Organization Millennium Development Goal Scorecard

——————

NOTE: Selected measure sets are not intended to provide a complete list or a representative sample. These measure sets are reproduced in full in Appendix D.

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

images

FIGURE 3-2 Number of sentinel measure initiatives on topics in five key areas.
NOTE: EHR = electronic health record; HC = health care.

it does illustrate the range and heterogeneity of sentinel measurement efforts already under way. Although not all of the examples may reflect the selection of measures that are truly sentinel, Box 3-2 presents a number of core measure initiatives that identified a limited set of measures from a larger pool. The initiatives displayed represent a variety of areas, from diabetes to cost and utilization, and they also reflect significant variation in the number of measures included in each set, ranging from as few as 10 to more than 100. Appendix D also provides further detail on the types of measures included in these measurement initiatives, including their focus and the concepts assessed, in the form of a table identifying the relevant foci of different initiatives.

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

Example Cases

In the Appendixes and related material, significant activity is reflected ongoing in the field to improve the quality, reliability, usefulness, and transparency of health measurement. This includes not only efforts to align and prioritize measures, as discussed above, but also efforts to develop and implement better measures and to achieve meaningful results through targeted measurement activities. Four examples are described below.

Using Measurement of Total Cost of Care to Reduce Overall Costs

The Network for Regional Healthcare Improvement (NRHI) is coordinating a project that illustrates the type of measure development that, in the Committee’s view, is needed to ensure that measures in use reflect a broad range of factors in and influences on health and provide a high-level view of the state of different aspects of health. The aim of this project is to identify the drivers of regional health care costs and develop strategies for reducing spending at the community level. The results of this work have the potential to inform future efforts in regional and national cost reduction. They also should help future Regional Healthcare Improvement Collaboratives (RHICs) create similar reporting systems for total costs of care and resource use that could be used in their communities to create a business case for payment reform, value-based benefit design, and changes in the organization and delivery of health care. This project, funded by the Robert Wood Johnson Foundation, will be conducted over an 18-month period and will explore a common measurement standard for costs and resource use across the participating regions. The partnering RHICs will create a benchmark to permit comparison of commercial costs and resource use both within communities and across regions, and they will engage in multi-stakeholder dialogue to further understand the results and devise with ways of using this information to reduce costs. Focused efforts with physician partners will lead to the creation of a curriculum for teaching physicians how to leverage the results to develop strategies for reducing costs. The project is developing a physician leadership curriculum to train and support physician champions to lead the movement toward cost transparency. The project will culminate in a national summit that will review the results of this research and its national implications.

The project will work to implement the measure set for total cost of care and resource use developed by HealthPartners. This measure set was chosen because it is a public set for which substantial documentation is available on the HealthPartners website, and it has been endorsed by the National Quality Forum (NQF).

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

In the initial planning phase, the five participating partner organizations of NRHI were brought together to identify the issues for which standardization is most important. One issue identified early on was risk adjustment, as the method used for risk adjustment determines the development of benchmarks and hence the extent of comparability of measures across communities. Because some communities had already selected specific risk adjustment methods for use in public reporting, significant effort and buy-in were required across the collaborative stakeholders. After 2 months of discussion, the collaborators agreed on the use of the risk adjustment method included in the NQF endorsement. However, those communities with existing risk adjusters will continue to use them for practice-level measurement and reporting initiatives.

Because this is a pilot, it is an opportunity to assess variation, try new ideas, and understand the impact of standardization. This process highlights some of the challenges of standardization. The sites were selected to participate because of the adequacy and availability of their data and strong alignment of local and project goals. All but one of the partner sites operates an aggregated multi- or all-payer claims database. The data included in each database vary—for example, in the number of International Classification of Diseases (ICD)-9 codes available—reflecting local policies. Another difference is the access to substance abuse and behavioral health data, as this type of data is highly sensitive and requires substantial data security. Yet these data are used in the risk adjustment software, and for comparability, either all collaboratives or none must use them. Another technical issue was whether to include incentive payments (such as with pay-for-performance contracts) in the total cost of care measure. The collaboratives had to resolve 22 key questions to ensure comparability, sometimes addressing a very detailed level of individual codes.

Another important consideration is the use of the data. The goal generally is to identify trends and large-scale variations, which the communities can use to identify opportunities for improvement and learn from high performers. The results can open up a conversation among the stakeholders and lead to change, with some regional employers planning to use the results for payment and benefit redesign. The project has already demonstrated that significant resources are required to reach agreement on standardization in such areas as risk adjustment and data quality.

CollaboRATE: Involving Patients in the Development of a Shared Decision-Making Measure

CollaboRATE illustrates measure development initiatives addressing patient experience and engagement and provides an example of how patients and families can be directly involved in the measure development

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

process. A team from the Dartmouth Institute for Health Policy and Clinical Practice working on health and health care measurement recognized the critical role of shared decision making in health, and accordingly they began developing a new type of measure targeting the patient’s role in clinical decision making, called CollaboRATE. Frustrated by the absence of a patient-reported measure of shared decision making that was psycho-metrically sound, sufficiently generic to suit any health care encounter, and scalable, the researchers proceed to develop this new measure through active partnership with end users.

The team interviewed 27 men and women in a rural hospital setting in two phases of iterative development and refinement (Elwyn et al., 2013). During this process, three core shared decision-making tasks were identified—provision of information, elicitation of patient preferences, and integration of patient preferences in decision making—and three corresponding items were constructed to form the CollaboRATE measure. Brief pilot testing with another 30 men and women demonstrated that CollaboRATE was easily understood by users and could be completed in less than 1 minute on exit from the clinical encounter.

Subsequently, the researchers assessed the psychometric properties of CollaboRATE experimentally in an online study of a representative sample of 1,341 adults in the United States (Barr et al., 2014). Study participants were randomly allocated to view one of several animated doctor–patient encounters featuring different levels of shared decision making. They were instructed to imagine themselves as the patient in the encounter and to complete CollaboRATE and two other measures of shared decision making. A subsample was resurveyed 1-2 weeks later, when they again viewed an animated encounter and completed CollaboRATE. Under these controlled conditions, CollaboRATE demonstrated discriminative validity, concurrent validity, sensitivity to change, and test-retest reliability.

The researchers have since completed a pilot implementation of CollaboRATE among a diverse network of clinical teams in the United Kingdom, during which the measure was administered to more than 5,000 patients via a paper survey upon exit from their clinical encounter. The team also has begun a large trial to rigorously assess the psychometric properties of CollaboRATE in real-world clinical settings in the United States. Overall, the development and testing of CollaboRATE in partnership with end users demonstrates the feasibility and utility of a collaborative approach to the development of patient-reported measures and the importance of using patient-reported measures that have been demonstrated to be comprehensible to the target audience (Thompson, 2014).

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

California Hospital Assessment and Reporting Taskforce (CHART)

A variety of projects nationwide are developing score cards, report cards, or ranking systems to provide information about health system performance, both to inform consumers and to enable assessing and monitoring progress over time. CHART, a project of the California HealthCare Foundation, has produced a standardized statewide online report card on hospital performance and quality. Developed through the collaborative work of a broad group of stakeholders that includes hospitals, government, health plans, employers, labor unions, and consumers, the CHART report card consists of 50 hospital performance measures aligned around system-wide goals. Hospitals are rated on a five-point scale—superior, above average, average, below average, and poor—for each measure. Although the program is voluntary, it has been adopted by 240 hospitals throughout California.

One reason why the CHART report card was able to achieve this level of adoption was that it requires less administrative effort than other reporting programs. Its adoption also benefited from active community efforts driven by consumers advocating for transparency in hospital performance data. The primary barriers to the report card’s implementation were the resource requirements of data collection, the selection of measures acceptable to all, and opportunity costs. Officials involved in implementation found that hospitals were most amenable to adoption when the report card was presented as an opportunity to take a proactive measurement approach in preparation for the likelihood that performance measurement would become obligatory. There is currently concern about how to align this program with new national requirements for health care performance measures to ensure that it remains effective.

Bailit Buying Value Initiative

The Bailit Buying Value Initiative, under the auspices of NQF, supported a landscape study of value and measurement in 48 states, designed to identify critical challenges and implementation efforts under way. The original goal of the project was for Bailit to develop a core measure set for use in value purchasing; however, it was decided that knowledge of whether current sets do or do not align was first necessary (BHP, 2013). Bailit found that the most critical barrier to standardization and efficiency appears to be misalignment of measure sets across states. Large numbers of measure sets were identified, and despite being drawn from similar national sources, the measure sets of individual states are either measuring different data or tailoring measures to meet state-dependent demands. The Buying Value Report is intended to describe the scope of the problem and to provide

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

recommendations for creating more alignment among measure sets across states and regions. Notably, the only consistently aligned measures are derived from Medicaid practices, likely because these programs primarily adopt Healthcare Effectiveness Data and Information Set (HEDIS) measures. Also noteworthy is that California appears to have better-aligned measure sets (perhaps because of the CHART program described above) compared with Massachusetts (only these two states were compared side by side). Another unique finding was that in Minnesota, legislative mandates for the implementation of recommendations from the State Quality Reporting System incentivized more alignment.

This landscape study also identified many “innovative measures,” or new measures created in lieu of the adoption of measures from existing programs. Many states develop such measures in an effort to measure in a way that is tailored precisely to their needs, priorities, resources, and populations. The study found that roughly 40 percent of states were creating such measures, and most of these were an attempt to fill gaps in measurement (e.g., care coordination, patient self-management, care management) (Bazinsky, 2014).

REFERENCES

Baker, L. C. 1999. Association of managed care market share and health expenditures for fee-for-service Medicare patients. Journal of the American Medical Association 281(5):432-437.

Barr, P. J., R. Thompson, T. Walsh, S. W. Grande, E. M. Ozanne, and G. Elwyn. 2014. The psychometric properties of collaborate: A fast and frugal patient-reported measure of the shared decision-making process. Journal of Medical Internet Research 16(1):e2.

Bazinsky, K. R. 2014. Staff interview with Kate Bazinsky, Bailit Health Purchasing. In Bailit Health Purchasing, edited by S. Johnson. Unpublished.

BHP (Bailit Health Purchasing). 2013. Bailit Health Purchasing website. http://www.bailit-health.com (accessed November 12, 2013).

Elwyn, G., P. J. Barr, S. W. Grande, R. Thompson, T. Walsh, and E. M. Ozanne. 2013. Developing collaboRATE: A fast and frugal patient-reported measure of shared decision making in clinical encounters. Patient Education and Counseling 93(1):102-107.

Glied, S., and J. G. Zivin. 2002. How do doctors behave when some (but not all) of their patients are in managed care? Journal of Health Economics 21(2):337-353.

IHI (Institute for Healthcare Improvement). 2005. Going lean in health care. Cambridge, MA: IHI.

IOM (Institute of Medicine). 2006. Performance measurement: Accelerating improvement. Washington, DC: The National Academies Press.

IOM. 2013. Core measurement needs for better care, better health, and lower costs: Counting what counts: Workshop summary. Washington, DC: The National Academies Press.

Lee, P. V., R. A. Berenson, and J. Tooker. 2010. Payment reform—the need to harmonize approaches in Medicare and the private sector. New England Journal of Medicine 362(1):3-5.

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×

McCarthy, D., S. How, C. Schoen, J. Cantor, and D. Belloff. 2009. Aiming higher: Results from a state scorecard on health system performance. New York: The Commonwealth Fund Commission on a High Performance Health System.

McGlynn, E. A., E. C. Schneider, and E. A. Kerr. 2014. Reimagining quality measurement. New England Journal of Medicine 371(23):2150-2153.

McGinnis, J. M. 1985. Recent history of federal initiatives in prevention policy. American Psychologist 40(2):205-212.

McGinnis, J. M., J. A. Harrell, L. M. Artz, A. A. Files, and D. R. Maiese. 1997. Objectives-based strategies for disease prevention. In Oxford textbook of public health, third edition, edited by R. Detels, W. Holland, J. McEwen, and G. Omenn. Oxford: Oxford University Press.

McGinnis, J. M., P. Williams-Russo, and J. R. Knickman. 2002. The case for more active policy attention to health promotion. Health Affairs 21(2):78-93.

Meltzer, D. O., and J. W. Chung. 2014. The population value of quality indicator reporting: A framework for prioritizing health care performance measures. Health Affairs (Millwood) 33(1):132-139.

Rutstein, D. D., R. J. Mullan, T. M. Frazier, W. E. Halperin, J. M. Melius, and J. P. Sestito. 1983. Sentinel health events (occupational): A basis for physician recognition and public health surveillance. American Journal of Public Health 73(9):1054-1062.

RWJF (Robert Wood Johnson Foundation) and UWPHI (University of Wisconsin Population Health Institute). 2013. County health rankings & roadmap. http://www.countyhealthrankings.org/app/home (accessed April 9, 2013).

Thompson, R. 2014. Involving patients in the development of a brief yet valid, reliable measure of share decision making: A case study of the development of collaboRATE. Unpublished case study developed in consultation with Committee staff.

Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 101
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 102
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 103
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 104
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 105
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 106
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 107
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 108
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 109
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 110
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 111
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 112
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 113
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 114
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 115
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 116
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 117
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 118
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 119
Suggested Citation:"3 Improving the Impact of Measurement." Institute of Medicine. 2015. Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. doi: 10.17226/19402.
×
Page 120
Next: 4 The Core Measure Set »
Vital Signs: Core Metrics for Health and Health Care Progress Get This Book
×
Buy Paperback | $60.00 Buy Ebook | $48.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Thousands of measures are in use today to assess health and health care in the United States. Although many of these measures provide useful information, their usefulness in either gauging or guiding performance improvement in health and health care is seriously limited by their sheer number, as well as their lack of consistency, compatibility, reliability, focus, and organization. To achieve better health at lower cost, all stakeholders - including health professionals, payers, policy makers, and members of the public - must be alert to what matters most. What are the core measures that will yield the clearest understanding and focus on better health and well-being for Americans?

Vital Signs explores the most important issues - healthier people, better quality care, affordable care, and engaged individuals and communities - and specifies a streamlined set of 15 core measures. These measures, if standardized and applied at national, state, local, and institutional levels across the country, will transform the effectiveness, efficiency, and burden of health measurement and help accelerate focus and progress on our highest health priorities. Vital Signs also describes the leadership and activities necessary to refine, apply, maintain, and revise the measures over time, as well as how they can improve the focus and utility of measures outside the core set.

If health care is to become more effective and more efficient, sharper attention is required on the elements most important to health and health care. Vital Signs lays the groundwork for the adoption of core measures that, if systematically applied, will yield better health at a lower cost for all Americans.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!