Assessing Accomplished Teaching Advanced-Level Certification Programs (2008) / Chapter Skim
Currently Skimming:
2 The Evaluation Framework and Collection of Data
2 The Evaluation Framework and Collection of Data
Pages 20-37
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 20... ...
The final section focuses on the evidence, discussing the evidence available from existing studies and the information we collected ourselves. conducting PROGRAM EVALUATIONs The committee's charge was to conduct an evaluation of the NBPTS program.
|
From page 21... ...
For example, summative evaluations might study such issues as the program's accomplishment of its intended objectives, impacts beyond those that were intended, how effectively resources have been used, the benefits of the program and what it costs to produce these benefits, and alternative interventions that might produce similar benefits. Summative program evaluations usually focus on the effects of a program on outcomes for a client population and consider the extent to which the program changes the outcomes for participants.
|
From page 22... ...
Evaluating Credentialing Tests The national board's program consists primarily of a certification assessment, and several sets of standards exist for guiding evaluations of assessment programs. The most well known are Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 1999)
|
From page 23... ...
Credentialing Tests and Validity Evidence Although sensitive to the common misunderstanding that there are different "types" of validity, psychometricians have defined several different kinds of validity evidence that can be used to contribute to the question of whether inferences based on the scores from a given test are valid. Content validity evidence examines the extent to which the test covers the intended domain of content and skills.
|
From page 24... ...
It is rarely possible to provide convincing criterion-related validity evidence for credentialing tests because of the difficulty in obtaining external measures that themselves satisfactorily assess performance across all practice settings. This is reflected in the established standards for the measurement field (e.g., American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 1999, p.
|
From page 25... ...
Because impact evidence is key to the broader program evaluation process and it was explicitly part of our charge, we defined our framework broadly and to encompass the notion of criterion-related validity. the evaluation framework We developed the evaluation framework by first theorizing about the ways that an advanced-level certification program for teachers might affect teaching practices and the teaching profession.
|
From page 26... ...
• Ultimately, all of these changes to the teaching field will help to improve teacher quality and, in turn, improve student learning. The committee grouped these assumptions into eight primary questions that form the basis for the evaluation framework.
|
From page 27... ...
The theory on which our eight primary questions are based can be understood as a model for thinking about the potential impact of a certification program for accomplished teachers on teacher quality and student learning, as shown in Figure 2-1. In this figure, rectangular boxes indicate aspects of the model included in our evaluation framework, and the numbers in parentheses indicate the specific framework question.
|
From page 28... ...
What evidence is available that board-certified teachers actually practice in ways that are con sistent with the knowledge, skills, dispositions, and judgments they demon strate through the assessment process? Do knowledgeable observers find them to be better teachers than individuals who failed when they attempted to earn board certification?
|
From page 29... ...
How do these actions influence teachers' attitudes toward certification and participation in the process? Question 4: To what extent does the advanced-level certification program identify teachers who are effective at producing positive student outcomes, such as learning, motivation, school engagement, breadth of achievement, educational attainment, attendance rates, and grade promotion?
|
From page 30... ...
Has the program been a useful tool for increasing the numbers of accomplished teachers in high-needs schools? Question 8: To what extent does the advanced-level certification program accomplish its objectives in a cost-effective manner, relative to other ap proaches intended to improve teacher quality?
|
From page 31... ...
FIGURE 2-1 Hypothesized impacts of an advanced-level certification program for teachers. 31 2-1 Broadside
|
From page 32... ...
During the conference, NBPTS staff members identified the areas in which they sought investigation, including the impact of board-certified teachers on student achievement, the impact on low-performing schools, leadership activities of board-certified teachers, standards-based professional development, adverse impact associated with the assessment, the NBPTS digital edge program, and psychometric/technical issues. Subsequent to the conference, potential researchers submitted proposals.
|
From page 33... ...
and in the Standards for Reporting on Empirical Social Science Research in AERA Publications (American Educational Research Association, 2006)
|
From page 34... ...
Information Collected by the Committee In all, the committee held six meetings, of which four included time for presentations intended to focus on specific aspects of the evaluation framework. We also arranged for a number of meetings outside committee meetings and additional analyses reported in four papers (Ladd, Sass, and Harris, 2007; McCaffrey and Rivkin, 2007; Perda, 2007; Russell, Putka, and Waters, 2007)
|
From page 35... ...
• Emerson Elliott, National Council for Accreditation of Teacher Education • Mary Futrell, George Washington University; former president of the National Education Association and member of the original board of directors • Kathy Madigan, formerly with ABCTE NBPTS researchers and consultants: • Lloyd Bond, Carnegie Foundation; former co-director of the Tech nical Analysis Group • Lee Shulman, Carnegie Foundation; former director of the Teacher Assessment Project at Stanford • Gary Sykes, Michigan State University; former doctoral student of Lee Shulman and consultant to the NBPTS board of directors • Suzanne Wilson, Michigan State University; former doctoral student of Lee Shulman and consultant to the NBPTS board of directors
|
From page 36... ...
was also devoted to hearing firsthand accounts from teachers and teacher educators. The goal of this panel was to learn more about the ways in which the NBPTS has influenced teacher education and professional development.
|
From page 37... ...
McCaffrey and Rivkin assisted the committee by identifying additional analyses that would help clarify the results, and Harris, Sass, and Ladd conducted these studies. We sought assistance in evaluating the psychometric qualities of the NBPTS assessment, and Teresa Russell helped with this aspect of our work.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.