5

Barriers and Opportunities for
Using Models to Inform Population
Health Interventions and Policies

The workshop’s next panel session featured three presentations looking at some of the circumstances in which modeling is used and the kinds of barriers and opportunities that arise when people start using these tools to help make public health policies. Michael Weisberg, an associate professor and the graduate chair of philosophy at the University of Pennsylvania, spoke about the nature of evidence with regard to modeling and how models are validated for use as decision-making tools. The team of Karen Minyard, the director of the Georgia Health Policy Center at Georgia State University’s Andrew Young School of Policy Studies, and Representative Sharon Cooper, the chair of Health and Human Services for the Georgia House of Representatives, then addressed the need to improve communication about the usefulness of models to policy makers. (Box 5-1 offers highlights from these presentations.) An open discussion moderated by Bobby Milstein followed the three presentations.

MODEL VALIDATION AND DECISION MAKING¹

As previous speakers had stated, Michael Weisberg said, all models are approximate and idealized, but nonetheless the goal is to use these less-than-perfect models to generate truer theories or to produce more

___________________

¹This section is based on the presentation by Michael Weisberg, associate professor and graduate chair of philosophy at the University of Pennsylvania, and the statements are not endorsed or verified by the Institute of Medicine.

accurate information about the world. Weisberg said that he thinks of the practice of modeling as a form of surrogate reasoning, or an indirect representational analysis of some real-world target using a model. To illustrate how a computational model works, he discussed one of the earliest examples in which a model was used to learn something about a community—in this case, Philadelphia.

The population of Philadelphia is quite diverse, being composed of approximately 40 percent African Americans, 40 percent Caucasians, 10 percent Asians, and 10 percent Latinos. However, when looked at by individual census tract, Weisberg said, Philadelphia is not diverse at all and is actually quite segregated, as are most Northeastern cities. A demographic diagram known as an exposure index shows that in Philadelphia a Caucasian has a 75 percent chance of having Caucasian neighbors, while an African American has a 75 percent chance of having African American neighbors. A simple model called the Schelling’s model of segregation can reproduce this pattern, and it was one of the first agent-based models.

One explanation of this demographic pattern is that racism is to blame, but Schelling imagined a different reason, Weisberg said. Schelling started with the assumption that people are either happy or unhappy, and if they are happy they stay put, but if they are unhappy they move. Happiness in this case is determined entirely on the basis of who a person’s neighbors

are and, in particular, by whether a person’s neighbor is like that person or not. Furthermore, there is some threshold—some percentage of neighbors who are like that person—that determines whether a person is happy or not and whether that person will move or stay put. Weisberg illustrated this with a grid representing Philadelphia with yellow and blue figures distributed randomly across the grid. With a threshold value set at 30 percent, the model soon produces a gird with patches of blue and patches of yellow that represent a level of segregation of 70 to 75 percent, reproducing the data from Philadelphia. If the threshold is set at 70 percent, the model produces nearly 100 percent segregation, Weisberg said.

The Schelling model was very influential in that it guided the way people think about segregation, Weisberg said, even while acknowledging its simplicity. The key question, though, is whether this model actually reveals anything true about the social dynamics of Philadelphia or any other city. In order to determine this, one has to show that a model has been validated and that it is robust. Validity, Weisberg said, comes from establishing a fit between the model and the target, while reliability relates to how believable a model is. Validation, he explained, involves understanding how well the structures that make up a model represent the system being modeled. “We want the model to truthfully describe the target,” Weisberg said, working through a series of equations characterizing the validity of a model based on which features of the real system are included in the model and how they are weighted in importance in the model compared to the real system. The weights, he explained, are related to what the experimentalist or the consumer of the model most wants to understand.

To determine reliability, Weisberg said, one can employ a process called robustness analysis, which essentially consists of looking at multiple models and seeing if they produce similar results. If models, despite their different assumptions, lead to similar conclusions, the result is a robust theorem that is relatively free of the details of the model. “We’re trying to find the truth at the intersection of independent lies,” is the way that Weisberg explained robustness analysis. In a sense, this is sensitivity analysis taken to a new level. In sensitivity analysis, the model’s parameters are changed to see how much the model’s output changes. Robustness analysis involves changing the basic structural assumptions in a model—essentially creating a new model—to see how the results change. “What we’re trying to figure out is what we can get away without knowing and what do we really need to know,” Weisberg said.

In summary, he said, “Put your faith in the robust results of well-validated models where you’ve had input into the way that the fidelity criteria are set.” It is important, he added, to think not only about the question to be answered, but also about the standards of fidelity that

are required. It is important to not simply accept the results of one set of simulations on a model, but instead to poke and prod a model to see where it fails.

IMPROVING COMMUNICATION WITH POLICY MAKERS ON THE USE AND USEFULNESS OF MODELS²

Karen Minyard, the next presenter, began by explaining that the Georgia Health Policy Center at the Andrew Young School of Policy Studies at Georgia State University, where she works, provides evidence-based research, program development, and policy guidance on local, statewide, and national levels. What drew her and her colleagues to this work, she explained, was the recognition that legislators are not faced just with routine technical problems but also with issues that are hard to define. “There are no clear solutions, and there are no experts that can say exactly what to do,” she said, “so our goal is to think about how to help legislators think about the whole picture, rather than the tip of the iceberg.” From interviews that the Health Policy Center staff conducted with state legislators, it became clear that the legislators have a variety of learning needs, ranging from those who need to learn about such basic information as the difference between Medicare and Medicaid to those who want to know everything about health policy.

To meet the needs of this latter group, the center developed a Health Policy Certificate Program, which uses system dynamics and system thinking to encourage broader and more systemic approaches to solving health policy challenges. Minyard said that when the program started, the expectation was that between 10 and 30 people would want this certificate, but so far more than 130 Georgia legislators and staff have taken part in this course, and 96 have received certificates. The course has been offered four times, with a fifth scheduled for late spring 2015. The course includes core sessions about thinking systematically, health status, financing, and coverage and access. These concepts are then applied to topics that each class chooses for itself, and these topics have included financing, behavioral health, and obesity. Minyard and her colleagues hypothesized that a combination of health policy content and a system dynamics–based approach to education could begin to change the way that legislators frame issues, ask questions, build understanding, and develop and weigh solutions to complex health care issues.

___________________

²This section is based on the presentation by Karen Minyard, director of the Georgia Health Policy Center at Georgia State University’s Andrew Young School of Policy Studies, and Sharon Cooper, chair of health and human services for the Georgia House of Representatives, and the statements are not endorsed or verified by the Institute of Medicine.

As part of the course, Minyard and her team created a six-question framework that they put on pocket cards and which they encourage legislators to ask every time someone brings a bill to them to consider supporting. Minyard said that lobbyists now ask her for the six questions so that they can be prepared before meeting with legislators. The six questions are

1. What is the important (perhaps troublesome) trend related to health in Georgia? What is the shape of this trend over the past several years?
2. Who are the stakeholders concerned about the trend?
3. Why this trend (what’s the cause, what is responsible)?
4. Where is there leverage (some policy) to address the underlying cause of the trend?
5. How will it work? How will it play out over time? How might unintended consequences occur? How might the policy positively or negatively impact . . .
   1. Health status?
   2. State health spending?
   3. Health care system?
   4. Health equity?
6. When would the policy create an impact on health status? When would you see an improvement in some other indicators (i.e., spending, services)?

Another concept that the course stresses is how behavior has changed over time, leading up to the current moment, and how behavior is expected to change over time, which is intended to give legislators a better perspective on an a given issue. The use of stock-and-flow maps is another important component of the course. One of the results of using these tools with legislators, Minyard said, is that they have a much stronger appreciation for the influence of prevention programs. The first session of the course also has the participants build, test, and revise a system dynamics model of childhood obesity, which contributed to legislators passing a bill on this issue. Building the model required having enough relevant data for both building and testing the model. Policy options in the model included ensuring safe routes to school, improving school food options, improving school physical education, improving nutrition and physical activity education in preschool programs, improving nutrition and physical activity education in after-school programs, and reimbursing medical nutrition therapy for obese children insured by Medicaid.

In addition to offering this course, Minyard and her colleagues have been involved in a wide range of policy modeling projects, including

one conducted using ReThink Health’s model. That project, the Atlanta Regional Collaborative for Health Improvement, went in 1 day from having no strategy to having a 28-year plan and included components on teenage pregnancy and injury prevention.

Representative Sharon Cooper then spoke about the value of the certificate program from her perspective as a state legislator and chair of the Georgia Legislature’s health committee. As background, she said that the Georgia Legislature has a 40-day session, and of the 236 members in the state house and senate, only eight have any hands-on experience in health care. She also said that the state leadership has backed the collaboration between the legislature and the Health Policy Center and that she is quite persistent in getting members of the legislature to take the certificate course. Participation has been bipartisan, and the 4-day course is offered when the legislature is out of session, which, she said, is a plus, given that legislators no longer have as many opportunities to get together with members of the opposing party outside of the legislative session. “This workshop gives us a chance to talk about things that we believe in on different issues in a bipartisan way,” she said.

When working with state legislators, Cooper said, the key is to start with something simple. One example that she spoke about involved modeling school-based exercise programs aimed at combating obesity. In a previous session of the legislature, a bill that would have changed the way state school boards ran school physical education programs failed to pass. During the subsequent course, the participants modeled obesity prevention, and the model showed that one of the most effective was to decrease obesity in children was to have good physical education programs in schools. Out of that modeling exercise came a legislative effort to learn why current programs were not working, and the finding was that the state education department and public health department were operating in separate silos. The legislature took action, restructuring the departments to eliminate the silos, creating new school-based physical education programs, and requiring all students in Georgia’s school to have their body mass index determined. This information is provided to parents along with information on obesity and the need for physical exercise. Aggregated data are provided to the schools and to the Governor, and the schools are ranked. The result is that 530 schools in Georgia have now enacted a specific, no-cost physical education program, called Power Up 30, which covers some 300,000 students. Power Up 30 was developed by one of the schools, which had accumulated 4 years of data showing that 30 minutes of activity at the beginning of a school day increased scores in math and science and decreased behavior problems.

DISCUSSION

Milstein started the discussion by saying that one message he heard from the presentations was that a real-world challenge is how to get busy people—legislators, for example—into a place where they have the space to think about a problem and perhaps discover through modeling that there is a bigger area for agreement on policy issues than might have been presumed, given today’s political realities. He then asked Minyard if she could speak more about this aspect of her program. Minyard replied that building the capacity to have conversations has been one of her favorite parts of the program. As an example she told of a time when the course ran through the stock-and-flow diagram exercise around moving from being at risk to being healthy. The participants developed a list that included exercise, she said, and one of the legislators said that he was morally opposed to the government paying for weight-loss programs. One of his colleagues pointed out that the state can either pay now or pay much more later in the form of higher health care costs. That remark became the theme for the course and the subsequent legislative session, which resulted in the bill that Cooper discussed being passed. Cooper agreed that the communications piece is critically important and noted that the Health Policy Center now offers a special course for the legislators on how to communicate and listen better to their colleagues.

Pamela Russo asked Minyard how the obesity model was developed, given the time constraints of the course. The model itself was built over the 4 months preceding the course, Minyard explained, based on input from legislators obtained at a biennial gathering of legislators held at the University of Georgia prior to the legislative session. Modelers, subject experts, legislators, staff, and translators all worked on and tested the model during that 4-month period to have it ready for the course. Cooper added that the legislature was shown parts of the model and was told how it was developed, which seemed to have a positive impact on the legislature as a whole.

Rachel Ferencik, who oversaw the model’s development at the Georgia Health Policy Center, noted that the six-question framework that Minyard discussed, combined with stock-and-flow maps and other information the legislators were given, allowed the legislators in the course to use the model in an efficient and productive manner.

George Isham asked Minyard if this program could be generalized to other state legislatures, and she replied that the Health Policy Center convened a meeting at which representatives from eight states came to talk to her and her colleagues about how this type of program could be implemented in their states. “We began to understand that you might design this a little differently depending on how much staff there is, what the situation is with term limits, and what the balance of power

between the administrative and the legislative body is in a particular state,” Minyard said. South Carolina has since implemented a similar program, Colorado has taken a component of the program and adapted it to its needs, and Kansas has added the program to one that it already had. Minyard noted that the Health Policy Center has developed additional, abbreviated certificate programs for the Governor ’s Office of Budget and Policy and the state’s Medicaid office. Finally, Cooper commented that legislators are hungry for knowledge and that they are only slowly discovering the resources that are available at state-funded universities.