Innovations in Design and
Utilization of Measurement Systems
to Promote Children’s Cognitive,
Affective, and Behavioral Health
WORKSHOP SUMMARY
Steve Olson and Noam I. Keren, Rapporteurs
Forum on Promoting Children’s Cognitive, Affective, and Behavioral Health
Board on Children, Youth, and Families
INSTITUTE OF MEDICINE AND
NATIONAL RESEARCH COUNCIL
OF THE NATIONAL ACADEMIES
THE NATIONAL ACADEMIES PRESS
Washington, D.C.
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
NOTICE: The workshop that is the subject of this workshop summary was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine.
This activity was supported by contracts between the National Academy of Sciences and the American Academy of Pediatrics (Unnumbered Award); the American Board of Pediatrics (Unnumbered Award); the Annie E. Casey Foundation (213.0427); Autism Speaks (Unnumbered Award); the Centers for Disease Control and Prevention (200-2011-38807, TO #16); the Department of Justice Office of Juvenile Justice and Delinquency Prevention (2013-MU-MU-0002); the National Institutes of Health (HHSN26300035); the Robert Wood Johnson Foundation (71071); the Substance Abuse and Mental Health Services Administration (HHSP233201300244P); and the William T. Grant Foundation (182528). Additional support came from the American Orthopsychiatric Association, the American Psychological Association, the Hogg Foundation for Mental Health, the Society for Child and Family Policy and Practice, the Society of Clinical Child and Adolescent Psychology, and the Society of Pediatric Psychology. The views presented in this publication do not necessarily reflect the views of the organizations or agencies that provided support for the activity.
International Standard Book Number-13: 978-0-309-36748-6
International Standard Book Number-10: 0-309-36748-4
Additional copies of this workshop summary are available for sale from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
For more information about the Institute of Medicine, visit the IOM home page at: www.iom.edu.
Copyright 2015 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
The serpent has been a symbol of long life, healing, and knowledge among almost all cultures and religions since the beginning of recorded history. The serpent adopted as a logotype by the Institute of Medicine is a relief carving from ancient Greece, now held by the Staatliche Museen in Berlin.
Suggested citation: IOM (Institute of Medicine) and NRC (National Research Council). 2015. Innovations in design and utilization of measurement systems to promote children’s cognitive, affective, and behavioral health: Workshop summary. Washington, DC: The National Academies Press.
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. C. D. Mote, Jr., is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Victor J. Dzau is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C. D. Mote, Jr., are chair and vice chair, respectively, of the National Research Council.
This page intentionally left blank.
PLANNING COMMITTEE FOR THE WORKSHOP ON
INNOVATIONS IN DESIGN AND UTILIZATION OF
MEASUREMENT SYSTEMS TO PROMOTE CHILDREN’S
COGNITIVE, AFFECTIVE, AND BEHAVIORAL HEALTH1
C. HENDRICKS BROWN (Chair), Professor, Departments of Psychiatry, Behavioral Sciences, and Preventive Medicine, Northwestern University
WILLIAM R. BEARDSLEE, Director, Baer Prevention Initiatives; Chairman Emeritus, Department of Psychiatry, Boston Children’s Hospital; and Gardner/Monks Professor of Child Psychiatry, Harvard Medical School
DARA BLACHMAN-DEMNER, Social Science Analyst, Crime, Violence, and Victimization Research Division, National Institute of Justice
DAVID A. BRENT, Academic Chief, Child and Adolescent Psychiatry, Western Psychiatric Institute and Clinic and Professor of Psychiatry, Pediatrics, and Epidemiology, School of Medicine, University of Pittsburgh
KIMBERLY E. HOAGWOOD, Society of Clinical Child and Adolescent Psychology, American Psychological Association and Professor and Vice Chair for Research, Department of Child and Adolescent Psychiatry, New York University School of Medicine
JENNIFER KAMINSKI, Acting Team Lead, Child Development Studies Team, Child Development and Disabilities Branch, Centers for Disease Control and Prevention
LAUREL K. LESLIE, Board of Directors, American Board of Pediatrics; Professor of Medicine and Pediatrics, Tufts University School of Medicine; and Director, Center for Aligning Researchers and Communities for Health, Tufts Clinical and Translational Science Institute
RUTH PEROU, Acting Mental Health Coordinator, Program Performance and Evaluation Office, Centers for Disease Control and Prevention
EVE E. REIDER, Health Scientist Administrator, Prevention Research Branch, National Institute on Drug Abuse
ANDY SHIH, Senior Vice President of Scientific Affairs, Autism Speaks
BELINDA E. SIMS, Health Scientist Administrator, Prevention Research Branch, National Institute on Drug Abuse
___________
1 Institute of Medicine planning committees are solely responsible for organizing the workshop, identifying topics, and choosing speakers. The responsibility for the published workshop summary rests with the workshop rapporteurs and the institution.
This page intentionally left blank.
FORUM ON PROMOTING CHILDREN’S COGNITIVE,
AFFECTIVE, AND BEHAVIORAL HEALTH1
WILLIAM R. BEARDSLEE (Co-Chair), Director, Baer Prevention Initiatives; Chairman Emeritus, Department of Psychiatry, Boston Children’s Hospital; and Gardner/Monks Professor of Child Psychiatry, Harvard Medical School
C. HENDRICKS BROWN (Co-Chair), Professor, Departments of Psychiatry, Behavioral Sciences, and Preventive Medicine, Northwestern University
DARA BLACHMAN-DEMNER, Social Science Analyst, Crime, Violence, and Victimization Research Division, National Institute of Justice
THOMAS F. BOAT, Professor of Pediatrics and Dean Emeritus, University of Cincinnati College of Medicine, Cincinnati Children’s Hospital Medical Center
FELESIA R. BOWEN, Assistant Professor and Director, Center for Urban Youth, Rutgers College of Nursing
DAVID A. BRENT, Academic Chief, Child and Adolescent Psychiatry, Western Psychiatric Institute and Clinic and Professor of Psychiatry, Pediatrics, and Epidemiology, School of Medicine, University of Pittsburgh
LAUREN CALDWELL, Director, Children, Youth, and Families Office and Public Interest Directorate, American Psychological Association
DAVID A. CHAMBERS, Associate Director, Dissemination and Implementation Research and Chief, Services Research and Clinical Epidemiology Branch, Division of Services and Intervention Research, National Institute of Mental Health
WILMA PETERMAN CROSS, Senior Public Health Advisor, Office of Disease Prevention, National Institutes of Health
COSTELLA GREEN, Branch Chief, Division of Community Programs, Center for Substance Abuse Prevention, Substance Abuse and Mental Health Services Administration
J. DAVID HAWKINS, Social Work Endowed Professor of Prevention, University of Washington School of Social Work
KIMBERLY E. HOAGWOOD, Society of Clinical Child and Adolescent Psychology, American Psychological Association and Professor and Vice Chair for Research, Department of Child and Adolescent Psychiatry, New York University School of Medicine
___________________
1 Institute of Medicine forums and roundtables do not issue, review, or approve individual documents. The responsibility for the published workshop summary rests with the workshop rapporteurs and the institution.
COLLEEN HORTON, Policy Program Officer, Hogg Foundation for Mental Health, University of Texas, Austin
UMA KOTAGAL, Senior Vice President for Quality, Safety and Transformation, and Executive Director, James M. Anderson Center for Health Systems Excellence, Cincinnati Children’s Hospital Medical Center
LAUREL K. LESLIE, Board of Directors, American Board of Pediatrics; Professor of Medicine and Pediatrics, Tufts University School of Medicine; and Director, Center for Aligning Researchers and Communities for Health, Tufts Clinical and Translational Science Institute
MARY ANN McCABE, Society for Child and Family Policy and Practice, Society of Pediatric Psychology, Associate Clinical Professor of Pediatrics, George Washington University School of Medicine and Affiliate Faculty in Psychology, George Mason University
JUSTIN MILNER, Senior Research Associate, Urban Institute
WILLIAM MODZELESKI, Senior Consultant, SIGMA Threat Management Associates
JENNIFER NG’ANDU, Program Officer, Robert Wood Johnson Foundation
JENNIFER OPPENHEIM, Public Health Advisor and Director, Project LAUNCH, Center for Mental Health Services, Substance Abuse and Mental Health Services Administration
LAWRENCE A. PALINKAS, Albert G. and Frances Lomas Feldman Professor of Social Policy and Health, and Director, Behavior, Health, and Society Research Cluster, University of Southern California School of Social Work
RUTH PEROU, Acting Mental Health Coordinator, Program Performance and Evaluation Office, Centers for Disease Control and Prevention
EVE E. REIDER, Health Scientist Administrator, Prevention Research Branch, National Institute on Drug Abuse
MARY JANE ROTHERAM-BORUS, Bat-Yaacov Professor of Child Psychiatry and Biobehavioral Sciences, Director, Global Center for Children and Families; and Director, Center for HIV Identification, Prevention & Treatment Services, Department of Psychiatry, University of California, Los Angeles
PAT SHEA, Deputy Director, Technical Assistance and Prevention, National Association of State Mental Health Program Directors
ANDY SHIH, Senior Vice President of Scientific Affairs, Autism Speaks
JOSÉ SZAPOCZNIK, Professor and Chair, Department of Public Health Sciences and Director, Miami Clinical Translational Science Institute and Center for Family Studies, University of Miami
VERA FRANCES TAIT, Associate Executive Director, and Director, Department of Child Health and Wellness, American Academy of Pediatrics
JENNIFER TYSON, Social Science Analyst, Office of Juvenile Justice and Delinquency Prevention, Department of Justice
DONALD WERTLIEB, Professor Emeritus, Eliot-Pearson Department of Child Development, Tufts University
Project Staff
MORGAN A. FORD, Senior Program Officer
WENDY KEENAN, Program Associate
TARA MAINERO, Associate Program Officer
NOAM I. KEREN, Research Associate
AMANDA PASCAVIS, Senior Program Assistant
ANTHONY JANIFER, Program Assistant
Board on Children, Youth, and Families Staff
PAMELLA ATAYI, Administrative Assistant
FAYE HILLMAN, Financial Associate
KIMBER BOGARD, Board Director
This page intentionally left blank.
Reviewers
This workshop summary has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published workshop summary as sound as possible and to ensure that the workshop summary meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the process. We wish to thank the following individuals for their review of this workshop summary:
Denise Dougherty, Agency for Healthcare Research and Quality
Vetisha L. McClair, Centers for Medicare & Medicaid Services
Sarah Hudson Scholle, National Committee for Quality Assurance
Bradley D. Stein, RAND Corporation
Although the reviewers listed above have provided many constructive comments and suggestions, they did not see the final draft of the workshop summary before its release. The review of this workshop summary was overseen by Hugh H. Tilson. Appointed by the National Research Council and the Institute of Medicine, he was responsible for making certain that an independent examination of this workshop summary was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this workshop summary rests entirely with the rapporteurs and the institution.
This page intentionally left blank.
Contents
2 MAXIMIZING THE VALUE OF NATIONAL, STATE, AND LOCAL MEASUREMENT SYSTEMS
Data on Children’s Health and Well-Being in the State of Maryland
Health Care Expenditures for Children with Disruptive Behavior Disorders in Medicaid/CHIP
Multiple Service Systems Use Among Illinois Families
Using Data to Improve Patient-Centered Outcomes: A Perspective from the Medicaid Medical Directors
3 MEASUREMENT SYSTEMS TO ASSESS INDIVIDUAL- AND POPULATION-LEVEL CHANGE
Evaluating Substance Abuse Prevention on a Large Scale Across State Populations
How Data Repositories Are Opening Access to Research Data on Autism and Mental Health Illnesses
Using Data to Inform Decision Making in Maryland Public Schools
Measuring Population-Level Progress in the Fight for Drug-Free Communities
4 USING QUALITY MEASURES TO FACILITATE SYSTEM CHANGE
Ensuring High Quality in Measures of Child Care and Preschool: A Cautionary Tale
The Juvenile Justice Reform and Reinvestment Initiative
Can Child Mental Health Cross the Quality Chasm?
Quality Measures and Improvement in Health Care
5 TOWARD EFFICIENT AND SUSTAINABLE DELIVERY OF INTERVENTIONS
Measuring Implementation of Evidence-Based Prevention: Lessons from Communities That Care
The Stages of Implementation Completion
Implementing Programs with Fidelity
Measuring Implementation Fidelity Using Computational Methods
Measurement at the Individual Level
Measurement at the Community Level
Figures and Table
FIGURES
2-1 The state of Maryland’s indicators and results framework
2-2 The Integrated Database on Child and Family Programs in Illinois
2-3 Costs of services for multi-system families (MSFs)
2-4 Types of problems experienced by families
3-1 Suicide rates among persons aged 15 to 19 years
3-2 Emergency department self-inflicted injuries by age and sex
3-3 Suicidal ideation and behavior among high school students
3-4 The U.S. Department of Education’s School Climate Model
3-5 Framework for community change of the National Coalition Institute
4-1 Matching risks in an evidence-based operating platform for juvenile justice (JJ)
4-2 All youth 12-month recidivism by matrix adherence level
4-3 The “six P” conceptual framework
5-1 The Communities That Care timeline
| ACA | Affordable Care Act |
| ADHD | attention deficit hyperactivity disorder |
| CADCA | Community Anti-Drug Coalitions of America |
| CDC | Centers for Disease Control and Prevention |
| CHIP | Children’s Health Insurance Program |
| CLASS | Classroom Assessment Scoring System |
| CMMI | Center for Medicare & Medicaid Intervention |
| CMS | Centers for Medicare & Medicaid Services |
| CTC | Communities That Care |
| DBD | disruptive behavior disorder |
| DFC | Drug-Free Communities |
| ECERS | Early Childhood Environment Rating Scale |
| ED-STARS | Emergency Department Screen for Teens at Risk for Suicide |
| FERPA | Family Educational Rights and Privacy Act |
| FIMP | fidelity of implementation |
| GLS | Garrett Lee Smith |
| GUID | global unique identifier |
| HIPAA | Health Insurance Portability and Accountability Act |
| IOM | Institute of Medicine |
| JJRRI | Juvenile Justice Reform and Reinvestment Initiative |
| LGBTQ | lesbian, gay, bisexual, transgender, and queer |
| MAP | Measures Applications Partnership |
| MAX | Medicaid Analytic eXtract |
| MDS3 | Maryland Safe and Supportive Schools |
| MSF | multi-system family |
| MTFC | Multidimensional Treatment Foster Care |
| NAASP | National Action Alliance for Suicide Prevention |
| NCHS | National Center for Health Statistics |
| NCQA | National Committee for Quality Assurance |
| NDAR | National Database for Autism Research |
| NIH | National Institutes of Health |
| NIMH | National Institute of Mental Health |
| NQF | National Quality Forum |
| NSF | National Science Foundation |
| NVDRS | National Violent Death Reporting System |
| PBIS | Positive Behavioral Interventions and Supports |
| PCORI | Patient-Centered Outcomes Research Institute |
| PMTO | Parent Management Training Oregon |
| SAMHSA | Substance Abuse and Mental Health Services Administration |
| SIC | Stages of Implementation Completion |
| SPEP | Standardized Program Evaluation Protocol |
| SPF-SIG | Strategic Prevention Framework State Incentive Grant |
| WISQARS | Web-Based Injury Statistics Query and Reporting System |
