National Academies Press: OpenBook

Assessing Accomplished Teaching: Advanced-Level Certification Programs (2008)

Chapter: Appendix A: Reviews of Studies That Provided Evidence for the Evaluation

« Previous: References
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 275
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 276
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 277
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 278
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 279
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 280
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 281
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 282
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 283
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 284
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 285
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 286
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 287
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 288
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 289
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 290
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 291
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 292
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 293
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 294
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 295
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 296
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 297
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 298
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 299
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 300
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 301
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 302
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 303
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 304
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 305
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 306
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 307
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 308
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 309
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 310
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 311
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 312
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 313
Suggested Citation:"Appendix A: Reviews of Studies That Provided Evidence for the Evaluation." National Research Council. 2008. Assessing Accomplished Teaching: Advanced-Level Certification Programs. Washington, DC: The National Academies Press. doi: 10.17226/12224.
×
Page 314

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Appendix A Reviews of Studies That Provided Evidence for the Evaluation This appendix provides additional details about the studies that pro- vided the bulk of the evidence for our evaluation. Box A-1 presents the primary questions on our evaluation framework, and for each study, we describe the specific relevance to the framework as well as the general pur- pose, the participants, and the findings. Also, for each study we provide a comment section that highlights the primary contributions of the study as well as any concerns the committee had about the methodology that af- fected the weight we placed on the findings. We hope that these comments will assist researchers with future investigations intended to build on this body of research. Barfield, S.C., and McEnany, J. (2004). Montana’s national board-certified teachers’ views of the certification process. Unpublished article, Montana State University-Billings. Relevance to evaluation framework: Question 3 (Chapter 6). Purpose: The authors sought to determine why more teachers in Mon- tana had not pursued board certification. Subjects/participants: National board-certified teachers (NBCTs) in Montana. Methodology and findings: Barfield and McEnany queried NBCTs in Montana about their certification experience. In spring 2003, the authors distributed surveys to the 31 NBCTs in the state and received responses from 22 (71 percent response rate). The survey instrument was adopted 275

276 ASSESSING ACCOMPLISHED TEACHING BOX A-1 The Committee’s Evaluation Framework Question 1: To what extent does the certification program for accomplished teach- ers clearly and accurately specify advanced teaching practices and the character- istics of teachers (the knowledge, skills, dispositions, and judgments) that enable them to carry out advanced practice? Does it do so in a manner that supports the development of a well-aligned test? Question 2: To what extent do the assessments associated with the certification program for accomplished teachers reliably measure the specified knowledge, skills, dispositions, and judgments of certification candidates, and support valid interpretations of the results? To what extent are the performance standards for the assessments and the process for setting them justifiable and reasonable? Question 3: To what extent do teachers participate in the program? Question 4: To what extent does the advanced-level certification program iden- tify teachers who are effective at producing positive student outcomes, such as learning, motivation, school engagement, breadth of achievement, educational attainment, attendance rates, and grade promotion? Question 5: To what extent do teachers improve their practices and the out- comes of their students by virtue of going through the advanced-level certification process? Question 6: To what extent and in what ways are the career paths of both success- ful and unsuccessful candidates affected by their participation in the program? Question 7: Beyond its effects on candidates, to what extent and in what ways does the certification program have an impact on the field of teaching, the educa- tion system, or both? Question 8: To what extent does the advanced-level certification program ac- complish its objectives in a cost-effective manner, relative to other approaches intended to improve teacher quality? from one used by the National Board for Professional Teaching Standards (NBPTS) in their survey of candidates in fall 2001 (National Board for Professional Teaching Standards, 2001e). The authors sought to determine why more teachers in Montana had not pursued board certification and thus added questions to the instrument to ask the respondents if they knew other teachers who were interested in earning board certification but had not yet done so.

APPENDIX A 277 Reasons for not pursuing board certification included the time com- mitment, the cost, the support required to complete the process (e.g., the videotaping and portfolio assembly), and lack of administrator support. The respondents also said teachers were concerned about the consequences. For example, they said some teachers thought it was very public and too risky, some were fearful of not being successful, and some had observed harassment of teachers who do become certified. Comments: This study is one of the few that address nonparticipation in the program, and the findings are useful in that regard. However, the sample for this study was very small, and the participants may not have been the most appropriate to query about the questions in which the in- vestigators were most interested. That is, it would have been better to ask nonparticipants why they had not pursued board certification instead of asking NBCTs to speculate about their nonparticipating colleagues. This provides a first step in learning about reasons for not participating, and the findings could serve as a basis for future studies with nonparticipants. Belden, N. (2002). California teachers’ perceptions of national board certi- fication. Santa Cruz, CA: Center for the Future of Teaching and Learning. Relevance to evaluation framework: Question 3 (Chapter 6); Question 6 (Chapter 8); Question 7 (Chapter 10). Purpose: The author sought to gather information about teachers’ rea- sons for pursuing board certification. Subjects/participants: NBCTs in California. Methodology and findings: eldon surveyed all NBCTs in California B (n = 785) in summer 2001 and received responses from 519 (68 percent response rate). A focus group discussion was also held in June 2001 in Sacramento; comments from this activity are incorporated into the paper. The survey asked NBCTs about their motivations for pursuing certification and the effects of the process on them and their teaching. It also gathered information about the type of school in which the NBCTs work. Most respondents said that they pursued certification because it was a personal challenge (84 percent) and provided an opportunity to strengthen their teaching (79 percent). Between 54 and 59 percent reported that they pursued certification to receive the state’s monetary compensations. The opportunity for career advancement was also important to more than half (53 percent), as was the prospect of receiving recognition of one’s teaching qualities (50 percent). Comments: Strengths of this study are its large sample size and response rate, relative to other studies of this nature, as well as the use of focus groups to follow up the survey results. An issue that should be considered in in- terpreting the findings is that all survey questions are all worded positively;

278 ASSESSING ACCOMPLISHED TEACHING there is no opportunity for a respondent to say the process did not have an effect on his or her teaching. The only exception is the question: “Do you feel that the certification process made you a much better teacher, somewhat better, or did not impact your practice?” The inclusion of both positively and negatively worded questions would have increased the objectivity of the survey and would have helped the researchers to detect problems with response sets (i.e., the tendency for respondents to respond in a given way or to select what he or she regards as an acceptable response). Bond, L., Smith, T., Baker, W.K., and Hattie, J.A. (2000, September). The certification system of the National Board for Professional Teaching Stan- dards: A construct and consequential validity study. Greensboro: University of North Carolina, Center for Educational Research and Evaluation. Relevance to evaluation framework: Questions 1 and 2 (Chapter 5). Purpose: This study was a validity investigation that sought to evalu- ate the extent to which teachers who achieved board certification exhibited the assessed knowledge, dispositions, skills, and judgments as part of their actual classroom practices. Subjects/participants: Participants were first-time candidates who had attempted certification in one of two areas: early adolescence English lan- guage arts and middle childhood generalist. The sample included 65 teach- ers working in Delaware, Maryland, North Carolina, Ohio, or Virginia, 31 NBCTs and 34 non-NBCTs. Methodology and findings: The authors used a two-pronged approach to investigate the validity of the NBPTS. They first attempted to validate the qualities assessed on the NBPTS assessments through a literature review. They reviewed the literature and tried to identify the dimensions of accom- plished teaching. Then they observed teachers who had participated in the NBPTS assessment and rated them on the identified dimensions. Through their literature review, they identified 15 dimensions: (1) use of knowledge; (2) identifying essential representations: deep representations; (3) identifying essential representations: problem solving; (4) setting goals for di- verse learners: improvisation; (5) setting goals for diverse learners: challenge of objectives; (6) guiding learning through classroom interactions: classroom climate; (7) guiding learning through classroom interactions: multidimen- sional perception; (8) guiding learning through classroom interactions: sen- sitivity to context; (9) monitoring learning and providing feedback; (10) monitoring learning and providing feedback: test hypotheses; (11) respect for students; (12) passion for teaching and learning; (13) motivation and self-ef- ficacy; (14) outcomes of lessons: surface and deep; and (15) outcomes of les- sons: achievement. They then developed protocols for evaluating each of the dimensions. Dimensions 1 through 13 were evaluated by observing teachers

APPENDIX A 279 in their classrooms; Dimensions 14 and 15 involved review of student work and achievement. To identify the sample, the authors used performance results for teach- ers who had taken the NBPTS assessments. Using the NBPTS score data, they grouped teachers as follows: (1) total score at least 1.25 standard deviations (SDs) below the cut score; (2) total score between 0.25 and 0.75 SDs below the cut score; (3) total score between 0.25 and 0.75 SDs above the cut score; and (4) total score at least 1.25 SDs above the cut score. They used this strategy to maximize the possibility of detecting differences among the groups. The list of teachers who fell into each group was randomized, and teachers were recruited until a sufficient number was obtained for the particular group. Ultimately, between 15 and 17 candidates were recruited for each group. Teachers’ performance with regard to these dimensions was evalu- ated through a variety of mechanisms, including classroom observations, reviews of teacher assignments and students’ work, interviews with stu- dents, student questionnaires that asked about classroom environment and climate and evaluated students’ motivation and self-efficacy, and students’ performance on a writing assessment. With regard to Dimensions 1 through 13, the results from classroom observations revealed that NBCTs scored higher on all of these dimen- sions than did the non-NBCTs; the differences were statistically significant (p < .05) on 11 of the 13 dimensions. Analyses of student work indicated that 74 percent of the work sam- ples of students taught by NBCTs reflected deep understanding, while 29 percent of the work samples of non-NBCTs were judged to reflect deep understanding. On the writing tasks, the mean was slightly higher for stu- dents taught by NBCTs than by non-NBCTs, but the differences were not significant (p > .05). Differences between NBCTs and non-NBCTs were negligible with re- gard to student motivation and self-efficacy levels. The authors also com- pared teachers on their participation in professional activities, including (1) collaborative activities with other professionals to improve the effectiveness of the school and (2) to engage parents and others in the community in the education of young people. Again, differences between NBCTs and non- NBCTs were negligible. Comments: This is a comprehensive study that examines construct- based validity evidence for the assessments for two NBPTS certificates. It draws from the literature on effective teaching to develop protocols for evaluating teachers and then compares teachers’ ratings on these protocols with their performance on the NBPTS. Studies of this nature are challenging to carry out. One issue with this study is that the authors’ description of the sampling procedures is somewhat vague. The report indicates that they

280 ASSESSING ACCOMPLISHED TEACHING recruited teachers via phone calls, and they provide the verbatim protocol for recruiting teachers, but they do not provide any details about this pro- cess. They do not specify how many calls were required in order to obtain the necessary numbers of participants for each of the four NBPTS score groups or whether it was more difficult to fill any of the groups. It would have been useful to know how representative the participating samples of teachers were of the full set of individuals identified for each score group. Cantrell, S., Fullerton, J., Kane, T.J., and Staiger, D.O. (2007, April 16). National board certification and teacher effectiveness: Evidence from a random assignment experiment. Unpublished paper. A paper developed under a grant from the Spencer Foundation and the U.S. Department of Education. Available: http://harrisschool.uchicago.edu/programs/beyond/ workshops/ppepapers/fall07-kane.pdf [accessed May 2008]. Relevance to evaluation framework: Question 4 (Chapter 7). Purpose: The authors examined the relationships between board certi- fication and student achievement. Subjects/participants: NBPTS applicants and non-NBPTS applicants teaching grades 3-5 in the Los Angeles Unified School District during the 2003-2004 and 2004-2005 school years. Methodology and findings: In this study, the authors were able to implement random assignment of classrooms to teachers. To accomplish this, NBPTS provided them with a list of applicants for board certification. Each applicant in the sample was matched with a nonapplicant comparison teacher in the same school and grade (comparison teachers had at least three years experience); 99 pairs of teachers participated. Classrooms were randomly assigned to teachers in this “experimental sample” (although students were not randomly assigned to classrooms). Another “nonexperi- mental sample” of NBPTS applicants and nonapplicants was also identified to allow the researchers to study the effects of random and nonrandom assignment. Analyses used the covariate model (lagged achievement test score) and the gain score model, as well as a set of student characteristics, classroom peer characteristics, and fixed effects for school by grade by administrative track by year. The NBPTS certification status variables included passed, failed, or withdrawn. The researchers obtained assessment results for the teacher applicants, which included both the pass/fail score and the numeric scores on each of the components of the assessment. They examined the extent to which different weightings of the component scores altered the relationship between certification status and students’ test score gains. The authors found that teachers who applied for board certification but were unsuccessful were less effective than nonapplicant teachers. The

APPENDIX A 281 coefficients for the unsuccessful group were always negative—specifically, with the covariate model, –.17 for math and –.13 for reading; and with the gain score model, –.36 for math and –.21 for reading. Comparison of the coefficients for teachers who achieved board certification and teach- ers who were unsuccessful revealed that the differences were statistically significant (or approached significance with p = .05). They summarize this finding saying that the board-certified teachers outperformed the unsuccess- ful applicants by 0.2 standard deviations in math and language arts. No statistically significant differences were found in comparisons of NBCTs and nonapplicants. The results for the nonexperimental group demonstrated the same pat- terns, but the effect sizes were smaller. Comments: This study makes a significant contribution in its use of random assignment of students to teachers, which helps to control for preexisting differences among the groups of students assigned to board- certified and nonboard-certified teachers. By randomly assigning students to teachers, the study removed many of the potential threats to the valid- ity of inferences about the effectiveness of NBCTs. In addition, the study restricted the comparison teachers to those with at least three years of teaching experience to make them more like the NBCTs. However, it was not able to match more closely in terms of experience and it did not include years of experience as a control variable in the analyses. So it is possible the NBCTs and their matched pair teachers might differ in terms of experience. The authors also report some student switches after assignment (less than 15 percent) that might also have affected the results. Cavalluzzo, L.C. (2004, November). Is national board certification an effective signal of teacher quality? Alexandria, VA: CNA Corporation. Available: http://www.nbpts.org/resources/research/browse_studies?ID=11 [accessed November 20, 2007]. Relevance to evaluation framework: Question 4 (Chapter 7). Purpose: The author examined the relationships between board certifi- cation and student achievement. Subjects/participants: Students of NBCTs and non-NBCTs teaching 9th and 10th grade mathematics in Miami–Dade County, Florida, during the 2000-2001 to 2002-2003 school years. The sample included 107,997 students and 2,137 teacher-years. It includes all the NBCTs teaching the selected grades during the chosen school years. Student scores on the state’s end of grade accountability test (the Florida Comprehensive Assessment Test, Sunshine Standards Tests) provide the measure of student outcomes. Methodology and findings: The author used education production function methods to study differences between the outcomes of students

282 ASSESSING ACCOMPLISHED TEACHING taught by NBCTs, current NBPTS applicants, teachers who applied for but did not receive NBPTS certification, and other teachers. The covariates used in the models are detailed student background variables, including grade level, age, gender, race/ethnicity, English language proficiency, participation in free or reduced price meal programs, grade retention, gifted status, spe- cial education status, school suspensions, days absent, grade point average, math effort and conduct, and whether the students’ math class was above, below, or at grade level. The models also included teacher background variables: whether or not the teacher is teaching in the subject area of certification, the salary step (as a measure of years of experience), certification status, whether or not the teacher has a graduate degree, and the selectivity of the teacher’s undergraduate college or university. The production function used a linear model that included these vari- ables and the student’s prior year math score to predict current scores. Some models also included variables measuring school attributes, and others included school fixed effects—that is, indicator variables for each school that equal 1 if the student attended the school and 0 otherwise. Scores from 9th and 10th grade students were combined into a single data set and fit to a single model. The author found that after adjusting for all the variables mentioned above, the test scores of students whose teachers were NBCTs were statisti- cally significantly higher than students whose teachers had no participation with NBPTS. Similarly, students whose teachers were currently NBCT ap- plicants scored statistically significantly higher than students whose teach- ers had no participation with NBPTS; students whose teachers applied for NBPTS certification but failed to be certified scored statistically significantly lower than students whose teachers had no participation with NBPTS. For the author’s preferred model, the effect sizes were about 0.07 (standard deviations) for NBCTs, 0.02 for current applicants, and –0.02 for teachers who failed to receive certification. These results were relatively insensitive to variations in the model, including the use of student fixed effects instead of using prior year scores as a covariate. Comments: This study is one of the few that focus on high school stu- dents and teachers. The study used a large sample of students and explored several different models, which helps to evaluate the consistency of findings across models. One concern with this study is that the analysis does not account for the fact that student test scores are nested within classes, within schools. Given that the effect sizes are small, it is very likely that many would not be statistically significant if this clustering was accounted for. A second concern is that the model does not account for the course content, and NBCTs might not be teaching courses with the same content as other teachers. This could result in the confounding of content and NBCT

APPENDIX A 283 effects. There is no discussion of course content, so the possible extent of bias cannot be assessed. Course content is particularly important with high school mathematics students because the content is highly differentiated across courses but the tests are not course specific. Clotfelter, C.T., Ladd, H.F., and Vigdor, J.L. (2007, March). How and why do teacher credentials matter for achievement? Working paper 2. National Center for Analysis of Longitudinal Data in Education Research. Avail- able: http://www.caldercenter.org/PDF/1001058_Teacher_Credentials.pdf [accessed November 27, 2007]. Relevance to evaluation framework: Question 4 (Chapter 7); Question 5 (Chapter 8). Purpose: The study examines the relationships between student achieve- ment and board certification status. Subjects/participants: North Carolina students in grades 3, 4, and 5 for the 1994-1995 to the 2003-2004 school years and their teachers. Methodology and findings: In this study, the authors model the rela- tionship between a variety of teacher characteristics and student achieve- ment test scores. The study used a production function approach and a series of alternative specifications for the model. To motivate their model, the authors first introduce a simplified model for student achievement with the assumption that the effects of teacher quality on student achievement were the same at every grade level and were constant across all years of the study. In addition, they assumed that these effects decay at a constant rate every year. This yields a structural model for current-year test scores as an additive linear function of the prior achievement score and current-year teacher inputs. The authors used this model to motivate five more com- plex models that they then fit to the data to estimate the effects of various teacher attributes on student achievement. The first model was a simple value-added model with current-year score as the outcome or the dependent variable, and the explanatory variables in the model included prior-year score and time-invariant and time-varying teacher, classroom, and student characteristics. The authors extended this model by adding school fixed effects, so that the effects of teacher charac- teristics were measured by variation within schools and differences in the student populations across schools were not confounded with the estimates of the effects of teacher characteristics. The third model used student gain scores as the dependent variable, rather than using level score as the depen- dent variable. This model did not include prior-year score as a covariate. The fourth model returned to using current-year achievement level as the dependent variable but replaced student prior-year test score and student time-invariant variables with student fixed effects. The fifth model used

284 ASSESSING ACCOMPLISHED TEACHING student fixed effects with gain scores. All models were fit separately for mathematics and reading using all available student data. The primary model specification included an indicator variable for whether or not a teacher is currently an NBCT. For mathematics, the coef- ficient is statistically significantly positive for every model. The coefficients range from 0.018 to 0.028, but most estimates are close to 0.02. Using the models with student fixed effects, Models 4 and 5, the authors compared teachers’ effectiveness across years. They considered teachers two years prior to certification, one year prior to certification, the year of certification, and one or more years after initial certification. They found that for mathematics using Model 4, that the effects were largest two years prior to certification and postcertification, with a dip in effects the year before certification (the application year) and the first year of certifica- tion. However, with Model 5 the effects were largest for teachers prior to certification and smallest in the two years postcertification. For reading, Model 4 suggests that for teachers who are certified some- time during the study, their students scored highest relative to other stu- dents when the teachers were two years prior to certification. The effects get smaller with every year of certification staging, so that the effects for NBCTs postcertification were less than half as large as the effects two years prior to certification. This pattern did not repeat with Model 5. With Model 5, students of certified teachers did best when the teacher was two years prior to certification and during the year of certification. Thus, for both reading and mathematics, the results of this secondary analysis were highly sensitive to model specification and inconsistent with the simpler model for- mulation that included a single indicator for current NBCTs. These analyses thus yield unstable estimates that need further investigation. Comments: This is a comprehensive study that evaluates the relation- ship between board certification and student achievement for three elemen- tary grade levels across nine years. The researchers examine the results for different models, providing information about the robustness of the findings to model specification. The consistency of effects for NBCTs across multiple models for both mathematics and reading provides compelling evidence that the cohort of NBCTs in North Carolina between 1995 and 2004 raised achievement test scores more than other teachers. One shortcoming of the paper is the fact that the authors do not use longitudinal data on the teachers to study how the same teacher’s students score as the teacher’s NBPTS status changes. This could provide more in- terpretable measures of NBCT effects than the comparisons that compare teachers prior to certification with other teachers.

APPENDIX A 285 Cohen, C.E., and Rice, J.K. (2005, August). National board certification as professional development: Design and cost. Arlington, VA: National Board for Professional Teaching Standards. Relevance to evaluation framework: Question 8 (Chapter 11). Purpose: The authors evaluate the costs associated with support pro- grams to prepare teachers for the NBPTS assessments. They compare costs with other mechanisms for providing professional growth to teachers. Subjects/participants: Eight sites that provide a preparatory program for teachers going through the NBPTS assessment process. Methodology and findings: The authors examine how the certifica- tion process and candidate support programs provide opportunities for teacher learning and how this model of professional development relates to principles of high-quality professional development found in the litera- ture. They looked at these issues in relation to the costs of the certification process and support programs and who bears these costs. They focused specifically on eight sites that offer support programs for teachers preparing for national board certification: Cincinnati, Miami–Dade County, Missis- sippi Gulf Coast, North Carolina A&T, San Antonio, San Diego County, Stanford, and Winston-Salem. The report provides detailed information on the costs associated with four of these sites. The authors estimate that the program-related costs per participant for these four sites ranged from $1,000 (for a program with 60 participants) to $11,200 (for a program with nine participants). They indicated that some of the variability in costs is explained by the economy of scale realized by the larger programs. They compare the costs of NBPTS support programs with the costs of obtaining a master’s degree and the costs of several state- or district-level professional development programs. Comments: This extensive study is useful for states and localities that are considering implementing a support program for teachers pursuing board certification. The authors give detailed cost estimates for four pro- grams that provide various levels and kinds of supports. For our purposes, this study was relevant to one part of our cost-effectiveness analyses, and we drew from the authors’ cost estimates for our analyses. The study might have been extended to provide information about the effectiveness of the programs. For example, it would have been useful to know the pass rate for candidates who went through each program. This would have helped states and localities in making design choices about such programs. Darling-Hammond, L., and Atkin, J.M. (2007, March). Influences of na- tional board certification on teachers’ classroom assessment practices. Un- published paper, Stanford University.

286 ASSESSING ACCOMPLISHED TEACHING Relevance to evaluation framework: Question 5 (Chapter 8). Purpose: This study examined the impact of the national board- c ­ ertification process on mathematics and science teachers’ classroom as- sessment practices. Subjects/participants: Middle and high school teachers (n = 16) in the area of Stanford University who planned on pursuing board certification. Methodology and findings: Through the National Board Resource Cen- ter at Stanford University, the authors recruited a group of middle and high school teachers interested in becoming board certified and randomly split them into two groups. The NBPTS group went through the certification process during the time period of the study, while the comparison group postponed their application for board certification. Teachers in the com- parison group were compensated for delaying their application by having their application fee paid when they ultimately applied. The researchers initially identified 102 participants; from this group, 60 attended the initial orientation session. All expressed interest, but once they learned what was involved, many dropped out. The researchers in- dicated that their goal was to have 20 teachers per group, but there was considerable attrition during the course of the study. In the end there were only 16 participants: nine in the national board group and seven in the control group. Participating teachers were followed for three years, and data collection was timed according to the stages of the application process: one year prior to pursuing board certification, the year of candidacy, and the year after candidacy. Data were collected from both groups at the same times. The data collected included twice yearly videotapes of lessons, written responses to questions about the videotaped lessons, student work samples from the unit that was videotaped, twice yearly interviews with the teach- ers about their practices and assessment approaches, surveys of students and teachers, and final reflective interviews with teachers about perceived changes in practices. The focus of the data collections was on (1) the ways that teachers use assessment in their classrooms; (2) the quality, range, and coherence of assessment methods; (3) the clarity and appropriateness of goals and expectations for learning; (4) opportunities for self-assessment; (5) modifications to teaching based on assessment information; and (5) quality and appropriateness of feedback to students. Results indicated that the NBPTS group began with mean assessment practice scores that were lower than the scores of teachers in the control group. During the certification year, the assessment practice scores of the NBPTS group rose and surpassed those of the control group. In the post- certification year, the scores were stable for the NBPTS group. The researchers found that teachers in the NBPTS group improved their formative assessment practices while engaging in the certification process

APPENDIX A 287 and largely maintained these practices in the following year. This group appeared to be using a wider range of assessment methods and question- ing strategies in class discussions that elicited more complete explanations from students. They were better able to integrate their assessments with ongoing instruction. The authors report that the difference between the NBPTS group and the comparison group were, for the most part, statisti- cally significant (p < .05). Comments: This is the only available study that examined the effects of the certification process on teachers’ classroom practices. The study was comprehensive in the types of information collected and the kinds of prac- tices examined. The rate of attrition is a serious problem that affected the validity of the findings, however. The authors provide some explanation for why teachers dropped out, but they do not discuss how the attrition might have affected the results. Some of the teachers dropped out after the first year of the study, but no data are reported for these teachers and there are no analyses of the potential biases introduced by this loss of participants. The resulting sample size is low. Nonetheless, the methods are novel, and with an appropriately sized sample of participants (and a lower rate of attri- tion), likely to yield useful information about the impact of the certification process on teachers. Goldhaber, D., and Anthony, E. (2007). Can teacher quality be effectively assessed? National board certification as a signal of effective teaching. Re- view of Economics and Statistics, 89(1), 134-150. Relevance to evaluation framework: Question 4 (Chapter 7) and Ques- tion 5 (Chapter 8). Purpose: Examine the relationships between board certification and student achievement. Subjects/participants: North Carolina students in grades 3 to 5 for the 1996-1997 to the 1998-1999 school years and their teachers. Methodology and findings: The study used education production func- tion methods to estimate the differences between teachers with differing involvement with NBPTS. The production functions included the student, teacher, school, and school district characteristics. The dependent variable was student gain scores on achievement tests in mathematics and reading. The authors considered four specifications for their models. Models 1 and 2 used a covariate adjustment approach. Model 3 replaced all the school and district variables with school fixed effects. Model 4 replaced all the student covariates with student fixed effects. The model with student fixed effects did not include school fixed effects. The authors included three variations of Model 2 (the model with all the covariates and no fixed ef- fects). The first included an indicator variable for whether or not a teacher

288 ASSESSING ACCOMPLISHED TEACHING was currently an NBCT and a separate indicator variable for whether or not the teacher would be an NBCT in the future but was not currently an NBCT. The second variation to Model 2 separated current NBCTs into those in their first year of certification and those certified more than a year ago. This variation also separated future NBCTs into current applicants and other future NBCTs. The third variation to Model 2 allowed for the study of the application process by including variables for future applicants, current applicants, and past applicants. Past applicants included current NBCTs and NBCT applicants who were not certified. Separate models were fit for reading and mathematics. The authors found that, compared with students of teachers who never applied to NBPTS or applied and did not receive certification, the students of NBCTs made significantly higher gains in reading (p < .05) but not in mathematics. They also found that the students of teachers who would be certified in the future made consistently higher gains than the students of other teachers. This result held for both reading and mathematics and regardless of the model’s specification of the comparison group of teachers. In addition, the study found that current applicants’ students made lower gains on average than students of other teachers, and this result held for all the mathematics models and nearly all reading models. Students of NBCTs in their first year of certification had larger gains than similar students in other teachers’ classes. This result held for all math- ematics models and all reading models except for the model with student fixed effects. However, the results for teachers who had been certified for more than a year were very inconsistent. In general the differences were positive for reading but not statistically significant (p > .05), and they were negative for mathematics but statistically significant (p < .05) only in the model with student fixed effects. In general, the results were relatively insensitive to the inclusion of school fixed effects as opposed to school-level covariates. The models were much more sensitive to the inclusion of student fixed effects, which may have been due to changes in the sample size. Comments: This was one of the first studies to evaluate the differ- ences in achievement test performance for students of board-certified and nonboard-certified teachers (the original version of the report was released by the Urban Institute in 2004). As such, it provided the first information about the relationship between board certification and student achieve- ment. The comparison of the results from different models was useful in examining their robustness to model specification. The findings generally indicated that board certification provides a signal of teacher effectiveness but that the process does not improve their effectiveness, a result generally confirmed by other studies. One limitation is that the authors do not account for the clustering of

APPENDIX A 289 students in classes, which means that tests of statistical significance may overstate the significance of the effects. The concern about bias in the statis- tical tests is exacerbated because many of the effects for NBPTS applicants and NBCTs are very small. Another concern relates to the examination of the effects of NBCTs at different times in the application process. Ideally such results would use longitudinal data on teachers to determine differences in performance during the application process, so that differences could be attributed to the process not the teacher sample. However, the authors were unable to use longitudinal data on teachers for their analyses. Thus, the sample of t ­ eachers who were certified for one year did not contain the same ­teachers as the sample of teachers who had been certified more than one year, and the sample of teachers prior to certification may not have included all teachers in the certified group. Differences in student outcomes among the groups of teachers at different stages of the certification process could have resulted from sampling error among the teachers. Goldhaber, D., and Hansen, M. (2007). National board certification and teacher career paths: Does NBPTS certification influence how long ­teachers remain in the profession and where they teach? Arlington, VA: National Board for Professional Teaching Standards. Available: http://www.nbpts.org/ resources/research/browse_studies?ID=184 [accessed November 27, 2007]. Relevance to evaluation framework: Question 6 (Chapter 9). Purpose: The authors investigated the impact of board certification on the job transitions and career paths of teachers employed in North Carolina. Subjects/participants: The primary sample included most of those who taught in the state public schools between the 1996-1997 and 1999-2000 school years. Methodology and findings: The researchers studied a sample of teach- ers who taught in North Carolina public schools between 1996 and 2000 and tracked their job transitions over an eight-year period from 1997 through 2003. The researchers obtained data on the following types of job transitions: (1) moving to another teaching position at a different public school within the same district, (2) moving to another teaching position in another public school district within the state, (3) leaving the North Caro- lina public school system. For the above three job transitions included in the database, the analy- ses made several comparisons: (1) those who obtained board certification versus those who had not; (2) among the latter group, the analysis com- pared those teachers who had never applied for board certification with those who had applied; and (3) among the latter group, the analysis com-

290 ASSESSING ACCOMPLISHED TEACHING pared successful applicants with unsuccessful applicants. Because those who apply for board certification may be different than those who never apply, the latter comparison among successful and unsuccessful applicants helps to mitigate selection bias. In addition, the analyses were broken out by teachers’ experience level and race, because the data showed differences among these groups in the likelihood of passing and in the impact of obtaining board certification. As a result of these many groupings—by type of transition, by board- c ­ ertification status, by experience, and by race—there were many permuta- tions of possible comparisons in the results. The analyses used competing-risks models to estimate the hazard (like- lihood) of an individual experiencing each of the types of job transitions, after controlling for a series of teacher and school characteristics. To further mitigate selection bias, the analyses also used a regression discontinuity method (a quasi-experimental method) in the comparisons of the successful and unsuccessful applicants. This method was used to estimate the effect of successfully passing the NBPTS assessment on two outcomes: the likelihood of experiencing one of the above three job transitions and the characteris- tics of the new schools to which they moved. The results of the analyses showed that those who obtained board certification, overall, had more mobility than those who were not board certified. However, the analyses also showed that these differences lay not so much with those who had never applied for board certification (the ma- jority of teachers in the state), but rather, the differences in mobility were primarily found among those who had applied—between the successful applicants and unsuccessful applicants. A more nuanced picture emerged when comparing the latter two groups. Although the coefficients were not always statistically significant across the different teacher experience levels, the direction of the signs was consistent. Those who passed the assessment and obtained the certification were more likely to move between schools and districts and more likely to leave the North Carolina public school system than were those who did not pass the assessment. A different picture also emerges depending on the race of the teacher. For African American applicants, the results indicate that board certification has little impact on career mobility. The researchers also studied the extent to which teachers who change jobs move to (or away from) schools with high-needs students. Their analy- sis compared the nature of the school moves made by those who earned board certification compared to teachers who were unsuccessful applicants. The school characteristics the authors examined included the percentage of enrolled students in poverty, the percentage of minority students, per- pupil expenditures, and median housing values in the district. The authors reported the results separately for white and African American teachers.

APPENDIX A 291 For white teachers, the results were generally weak. There was some consistency in the sign of the coefficients; that is, white board-certified teachers tended to move to schools with fewer students in poverty and fewer minority students than did unsuccessful applicants. However, most of the coefficients were neither statistically nor substantively significant. For example, there was generally less than a 1 percent difference between successful and unsuccessful applicants in terms of the percentages of stu- dents in poverty at the schools to which they moved, and differences in the range of 1 to 2.5 percent in the percentages of minority students at their new schools. Comments: This is the first study to examine the relationships between teachers’ career paths and their board-certification status. The methods allow the researchers to examine mobility subsequent to obtaining board certification and to compare this for teachers who had passed and failed the NBPTS assessments. This is a significant contribution. However, we high- light two issues with this study. First, as with other state databases, the data used in this study had no information on whether those who left the North Carolina public schools had moved to a public school job out of state, had moved to a private school job in or out of the state, or had left teaching entirely. This is an important limitation because it means the study could not specifically isolate the influence of board certification on attrition from the teaching fields. While the researchers note this in their discussion, we re- emphasize this point as a limitation in interpreting the findings. In addition, there is a tendency in the paper to overstate the findings when discussing the types of schools to which white teachers move. Some of the differences in the characteristics between the old and the new schools for white teachers were on the order of 1 to 2 percent and generally not significant, but they are discussed as if they were statistically significant. Harris, D.N., and Sass, T.R. (2006, August 22). The effects of NBPTS- certified teachers on student achievement. Arlington, VA: National Board for Professional Teaching Standards. Available: http://www.caldercenter. org/PDF/1001059_Teacher_Training.pdf [accessed November 27, 2007]. Relevance to evaluation framework: Question 4 (Chapter 7), Question 5 (Chapter 8). Purpose: Examine the relationships between board certification and student achievement. Subjects/participants: Florida students in grades 3 to 10 for the 1999- 2000 to the 2003-2004 school years and their teachers. Methodology and findings: This study is based on data from students in grades 3 to 10 on both mathematics and reading for both the state’s Florida Comprehensive Assessment Test (FCAT) norm-referenced test

292 ASSESSING ACCOMPLISHED TEACHING (FCAT-NRT), and the state’s criterion-referenced FCAT Sunshine State Standards Test (FCAT-SSS). The researchers used a production function approach. The study used gain scores in achievement (current-year score less prior year score) as the outcome. One set of models used FCAT-NRT scores as the outcome and another set used the FCAT-SSS as the outcome. The models included stu- dents and classroom variables as covariate variables as well as student fixed effects and school fixed effects. The authors report that, to avoid computational problems, they in- cluded student by school or “spell effects” for each period that a student is in a different school, rather than including separate fixed effects for each school and separate fixed effects for each student. The authors fit two pri- mary models. Model 1 included a single indicator variable for whether or not a teacher was ever board certified during the span of the data. Model 2 estimated separate effects for the ever-board-certified group during three periods of the certification process: the years prior to application, the year of application, and the years following certification. The two tests (FCAT-NRT and FCAT-SSS) yielded somewhat different stories about the relationship between board certification status and student achievement. For Model 1, the effects associated with board certification for the FCAT-NRT were negative for both reading and mathematics but positive for both subjects for the FCAT-SSS. For Model 2, gains on the FCAT-NRT mathematics test were statis- tically significantly greater for students whose teachers would someday achieve NBPTS certification. However, the gains on these tests were negative for students whose teachers were current applicants for NBPTS certifica- tion (and would be awarded certification) and for students whose teachers were currently certified. For the mathematics FCAT-SSS, students in each group made greater gains than students whose teachers would never be board certified within the span of the data, but none of the differences was significant. In reading, students whose teachers were not current NBCTs but would someday be awarded NBPTS certification and students whose teachers were current NBCTs made greater gains on the FCAT-NRT than students whose teachers would never be board certified during the span of the study; however, neither difference was statistically significant. The gains on the FCAT-NRT reading tests were negative, but not significant, for stu- dents whose teachers were current applicants for NBPTS certification (and would be awarded certification). The FCAT-SSS reading gains were signifi- cantly higher for students whose teachers would someday apply for and be awarded NBPTS certification and for students who teachers were current NBCTs than for students who teachers would not be awarded certification during the span of the study. The FCAT-SSS reading gains were negative but not significant during the application year of future NBCTs.

APPENDIX A 293 Comments: This is an extensive study that includes five years of data for students from elementary through high school. The comparison of re- sults for two different outcome measures is an important contribution. The researchers made no adjustment for the clustering of students within teach- ers, which could result in biased significance tests; however, many of the effect sizes were quite small. The complex adjustments of including student by school or spell fixed effects resulted in many students and teachers being excluded from the analysis. For example, students must be in a school for a minimum of two years to contribute to the estimation of teacher effects. Similarly, some teachers were excluded from the analysis; for example, teachers who teach only during the last year and teach third, sixth, or tenth grade were excluded. Moreover, these adjustments yield consistent estimates only under many assumptions, including the assumption that students’ achievement scores are growing at student-specific rates. Thus, it is possible that restrictions to the sample and estimation error that results from using many fixed effects lead to the inconsistency of the results. Helding, K.A., and Fraser, B.J. (2005, April). Effectiveness of national board certified teachers in terms of attitudes, classroom environments and achievement among secondary science students. Paper presented at the an- nual meeting of the American Educational Research Association, Montreal, Canada. Relevance to evaluation framework: Question 4 (Chapter 7). Purpose: The authors compared NBCTs and non-NBCTs in terms of classroom environment, student attitudes, and student achievement. Subjects/participants: Eighth and tenth grade students in Miami–Dade County taught by NBCTs and non-NBCTs. Methodology and findings: The researchers recruited a sample of NBCTs to participate in this study and to gather information about their students’ attitudes. The researchers used two instruments to measure stu- dents’ perceptions of their science classes and their attitudes about sci- ence. One instrument is a questionnaire called “What Is Happening in This Class?” (WIHIC). The WIHIC questionnaire presents students with a series of statements about practices in the class and asks them to rate the frequency with which each occurs, using a five-point scale ranging from “almost never” to “almost always.” The questionnaire measures students’ perceptions with regard to seven factors: student cohesiveness (how well students work together in the class), teacher supportiveness, the student’s involvement in class activities, the student’s investigation practices, the student’s level of task orientation, the student’s level of cooperation, and the student’s perception of equity in the class (e.g., that teachers treat all students equally).

294 ASSESSING ACCOMPLISHED TEACHING The second instrument is an attitude scale derived from the Test of Science-Related Attitudes (TOSRA). The TOSRA presents students with 10 statements about science classes (e.g., Science lessons are fun; I look forward to science lessons; I want to find out more about the world in which we live) and asks them to indicate if they agree, disagree, or are not sure about the statement. Students were also compared with regard to their FCAT scores in science, which assess higher order cognitive skills in physical and chemi- cal science, earth and space science, life and environmental science, and scientific thinking. To recruit the study samples, the researchers first contacted the princi- pal of each NBCT in Miami–Dade County. After obtaining the principal’s consent, the researchers contacted each NBCT. Each NBCT who agreed to participate was asked to recruit a non-NBCT who taught the same subject (presumably, though not stated, in the same school). Of the NBCTs who were contacted, 42 percent (n = 16) agreed to participate; 14 non-NBCTs agreed to participate. Each teacher was asked to provide one or two sci- ence classes, and a total of 38 classes (n = 927 students) participated in the study. The NBCT group included 443 students in 21 classes taught by 16 teachers, and the non-NBCT group included 484 students in 17 classes taught by 14 teachers. The authors compared mean scores for students taught by NBCTs and non-NBCTs with regard to the seven factors on the WIHIC, the overall score on the TOSRA, and FCAT scores in science. In all cases, students taught by NBCTs scored slightly higher than those taught by non-NBCTs. The authors tested the differences for statistical significance, and reported that six of the nine comparisons were significant at p < .05. Differences in achievement were not statistically significant. Comments: This is the only study currently available that focuses on the impact of board-certified teachers on student outcomes other than achievement test scores. As such, it provides an example of ways to extend the array of student outcomes considered in this type of research. However, there were several important methodological problems that affected the validity of the findings. First, the methods for obtaining the non-NBCT sample are problematic and are likely to have introduced selection bias into the comparison sample. The findings could be attributed simply to the way in which the sample was obtained. Second, the authors did not adjust for the fact that students were clustered in classrooms, some having the same teacher and some not (i.e., there were 21 classes taught by 16 NBCTs and 17 classes taught by 14 non- NBCTs); thus, the significance tests are likely to overstate the significance of the differences. Finally, there is no consideration of (or controls imple- mented for) preexisting differences between students assigned to NBCTs

APPENDIX A 295 and non-NBCTs. It may be that the students assigned to the NBCTs had more positive attitudes from the start. Indiana Professional Standards Board. (2002, Spring). Status of national board-certified teachers in Indiana. Indianapolis: Author. Available: http:// www.nbpts.org/resources/research/browse_studies?ID=26 [accessed No- vember 27, 2007]. Relevance to evaluation framework: Question 3 (Chapter 6). Purpose: To gather information about why teachers in Indiana decide to pursue board certification. Subjects/participants: NBCTs in Indiana. Methodology and findings: This study involved a survey of the 71 NBCTs in Indiana. At the time of the study, only 71 of roughly 75,000 teachers in the state were board certified. The survey was sent to all the 71 NBCTs in the state, and 32 (48 percent) responded. A focus group was also convened to further explore the survey questions. The survey consisted of 20 mostly open-ended questions designed to gather information about (1) the characteristics of board applicants, (2) how they were informed about the process, (3) their perceptions of the difficulty of the certification process, (4) the current support provided to candidates in the state, and (5) the sup- port needed for future state candidates. The respondents were generally favorable about their experience of going through the certification process. The survey asked why teachers pursue board certification. Generally, the respondents indicated that teach- ers pursue board certification because they have a desire to improve their effectiveness as a teacher, like challenges, are lifelong learners, are intrinsi- cally motivated, and consider that board certification serves to validate their practices. Most of the respondents did not cite monetary reasons as their motive for pursuing board certification; however, at the time, only limited financial support was offered to candidates. Some respondents felt supported by their colleagues and principals, and some did not. Some noted that their colleagues questioned why they would want to go through such a difficult process or “scorned the idea,” believing that the teacher was “showboating.” Most respondents reported that certification brought them new opportunities, including leadership roles, invited speaking opportunities, and serving as members of Disney American Teacher review committees. Most of the respondents felt that the process affected their teaching, about half reporting that they became more reflective. Comments: This study provides additional insight into the reasons why teachers decided to pursue board certification. The use of a focus group to follow up the survey was an important addition to the study. The sample

296 ASSESSING ACCOMPLISHED TEACHING size is very small, however, and the response rate was below 50 percent. There is no analysis of the respondents to examine the extent to which the respondents were representative of the full group that was surveyed. Koppich, J.E., Humphrey, D.C., and Hough, H.J. (2006). Making use of what teachers know and can do: Policy, practice, and national board certi- fication. Education Policy Analysis Archives, 15(7), 1-30. Relevance to evaluation framework: Question 3 (Chapter 6), Question 5 (Chapter 8), Question 7 (Chapter 10). Purpose: Collect data on the impact NBCTs are having at their schools. Subjects/participants: Samples of NBCTs in California, Florida, Mis- sissippi, North Carolina, Ohio, and South Carolina; colleagues of NBCTs in case-study schools. Methodology and findings: The authors built on their earlier work studying the impact of board certification in six states. Data collection methods included a mail survey to NBCTs in the six states, focus groups, and site visits to selected case study schools. The participants in the mail survey were selected via stratified random sampling methods, and independent samples were selected in each of the six states. In each state, the schools were assigned to one of four strata: elementary low-performing, elementary nonlow-performing, secondary low-performing, or secondary nonlow-performing. Within each stratum, a random sample of NBCTs was selected. The survey was distributed to 1,136 in the 6 states, and 854 responded (75 percent). Case studies were conducted at three schools each in California, North Carolina, and Ohio. Case study schools were those in which at least 9 percent of teachers were board certified (or 9 percent of a single depart- ment of secondary schools). Six focus groups were convened in California, North Carolina, and Ohio to supplement the information gathered from the case studies. Focus groups were held in Los Angeles and San Francisco; in Chapel Hill and Durham; and in Cincinnati and Cleveland. Participating teachers were those from schools not represented by the case-study sites in each of the respective states. The authors examined what NBCTs do after becoming certified and what it takes for them to make a difference in a school. Overall, they found little evidence of schools using NBCTs to serve as mentors or in leader- ship positions. They found that many of the NBCTs were in unsupportive situations. They noted that there was little evidence that NBCTs sought or were given the opportunity to move beyond the conventional obligations of classroom teaching. For example, while roughly 60 percent of the sur- veyed NBCTs said their principals view board certification very favorably,

APPENDIX A 297 49 percent said that the administration was not supportive of roles outside the classroom that NBCTs might be interested in pursuing. The authors report that principals are generally not adept at making use of the skills that NBCTs offer. Some principals are not familiar with what board certification represents, and others are reluctant to make use of NBCTs, not wanting to show favoritism of some teachers over others. The researchers report that more than 90 percent of NBCTs say they are no more influential than other teachers on such matters as selecting cur- riculum and materials, advising on professional development programs, teacher hiring or evaluation, advising on budget, or determining the focus of school reform efforts. Interviews with NBCTs and their colleagues led the authors to conclude that there is a culture of “individualism and egalitarianism that remains alive in the profession.” NBCTs report that they are often given “the cold shoulder” by non-NBCTs and nearly 43 percent said “my school culture is not welcoming of teachers stepping into leadership positions.” The authors found that NBCTs actually go to considerable lengths to downplay any distinction between themselves and their non-NBCT colleagues, sometimes even concealing the fact that they are board certified. The authors did find an exception in a single elementary school in North Carolina, and they provide a very lengthy description of how NBCTs are supported and used in this school—key to this is the fact that the principal and assistant principal are both board certified. An excerpt from their description of the environment at this school appears below (pp. 19-20): At Adam Elementary [a fictitious name], decision-making was organized around learning teams. All teachers participated in the teams that met weekly for an hour and focused on improving the schools’ literacy in- struction. NBCTs led many of the learning teams, although accomplished teachers who had not made that choice also filled formal leadership roles. The activities of the teams were consistent with the kinds of reflection and problem solving that are part of National Board Certification. National Board “language” was used throughout the school so that even teachers who had not pursued certification became familiar with the language, the standards, and the approach to teaching. The researchers observed one of the team meetings during which seven experienced teachers (four NBCTs) and one inexperienced teacher were discussing a videotape of strategies for teaching a vocabulary lesson. The researchers were struck by the level of conversation about the instruc- tional strategies and the extent of support that the experienced teachers offered to the inexperienced teacher. The researchers commented about the amount of professional conversation about teaching and learning that took place during the meeting and that occurred every day at the school.

298 ASSESSING ACCOMPLISHED TEACHING They recounted a recent debate at the school over curriculum policy. A small group of NBCTs led a larger cadre of their colleagues in a presenta- tion to the local school board, arguing against the acceptance of a $1.2 million federally funded Reading First grant. For them the Reading First program came with “too many strings” and would force them to teach reading uniformly. The teachers pointed to the success they were having with their current approaches and argued for adapting instruction and materials to serve all students’ literacy needs. The superintendent publicly backed the teachers, and the school board voted to turn down the grant. The school culture at Adam was enabled by state policies that encouraged teachers to earn board certification. In addition, district’s policies and support programs for candidates, along with the community awareness and support for NBCTs, were well aligned with the efforts underway at the school. The fact that the principal and the assistant principal had both earned board certification was crucial to their understanding of the board processes and standards. It was their ability to infuse the National Board standards and the practices that paralleled the certification process into the school’s professional development and improvement strategy that made the difference in the school’s teaching and learning culture. Comments: This is an extensive study that provides the first informa- tion about the effects of contextual issues on NBCTs. The use of multiple data collection strategies (surveys, focus groups, interview, and case studies) is a strong point of this study. The one area that the researchers might have also explored is the issue of failing the assessment and any impacts that might have on teachers. At the time that the study was released, the finding that NBCTs often face unsupportive environments had not been previously reported. It would be worthwhile to investigate the extent to which these findings are evident in states besides the six included in this study. Lustick, D., and Sykes, G. (2006). National board certification as profes- sional development: What are teachers learning? Education Policy Analy- sis Archives, 14(5). Available: http://epaa.asu.edu/epaa/v14n5 [accessed March 1, 2006]. Relevance to evaluation framework: Question 5 (Chapter 8). Purpose: To investigate what teachers learn by going through the cer- tification process. Subjects/participants: Teachers pursuing board certification in the area of adolescent and young adult science. Methodology and findings: This study involved a simulation of the NBPTS portfolio exercises conducted independently of the actual assess- ment. The simulation involved sending a packet to the study participants, conducting a phone interview with them, and then rating their responses us-

APPENDIX A 299 ing the actual NBPTS scoring rubric and procedures. The packet contained a sealed six-minute video clip of a whole class discussion in science, along with student artifacts from the lesson. The exercises consisted of five assignments. Two of them asked teachers to describe their own experiences, much like what is required on the actual assessment. One requested that the teacher describe a recent successful les- son she or he had taught, and the other asked the teacher to recount the types of professional activities in which she or he had participated. Three assignments involved reviewing materials that the researchers sent. One consisted of a student’s written response to an assessment ques- tion before and after a lesson on the kinetic theory of matter. The researcher then asked the teacher to discuss the students’ strengths and weaknesses using evidence from the response and to describe the feedback that should be given to the student. Another assignment involved a written scenario of a lesson about scientific inquiry. The scenario described the lesson, the teacher’s objectives, and an interaction among four students. The researcher asked the teacher about her or his appraisal of the students’ skills from this interaction, what they appeared to understand and misunderstand, and what instructional steps should be taken next. The final assignment involved viewing a videotaped class discussion about ecosystems. The researcher queried the teacher about the videotaped interactions—the extent to which the students were engaged in the discus- sion and understood the lesson, the effectiveness with which the teacher facilitated the discussion, the quality of the interactions, and the advice she or he would give to the teacher as ways to improve the instruction. The sample was recruited with the assistance of the NBPTS, and par- ticipants were drawn from teachers who registered for the assessment between 2001-2002 and 2003-2004. Approximately 450-650 candidates register for this assessment each year, and about half of each year’s cohort was randomly selected and invited to participate. The researchers set a goal of recruiting 40 teachers from each cohort, and participants were the first 40 from each cohort who agreed to participate. A total of 118 teachers participated in the study. Participants were randomly assigned to one of four groups. Two groups participated in the study prior to undergoing the actual NBPTS assessment (the pretest groups), and two groups participated after completing the ac- tual assessment (the posttest groups) but before receiving the results. One of the pretest groups also participated in the posttest, solely for the purpose of evaluating the effects of taking the pretest—the posttest results were not used in the final analysis. Results showed that scores on the posttest were statistically signifi- cantly higher than scores on the pretest (p = .009), with a moderately large effect size of .473. Additional analyses focused on the sources of the differ-

300 ASSESSING ACCOMPLISHED TEACHING ences. Performance on the simulation tasks had been scored according to the various NBPTS science standards (e.g., a score for each standard), and comparisons were made to see if the pretest-posttest differences were largely attributable to performance on one or more standards. Results showed that performance with regard to “advancing student learning” (p = .008) and “supporting teaching and student learning” (p = .005) were significantly different, with effect sizes of .482 and .524, respectively. Based on phone interviews with the participants, the authors character- ized the type of learning teachers undergo as a consequence of the process. They found that roughly half of the teachers fell into the “dynamic learn- ing category,” meaning self-reports of immediate, meaningful change in a teacher’s beliefs, understandings, and actions in the classroom. About one- quarter fell into the category of “technical learning,” which the authors characterize as an emphasis on acquiring techniques useful in obtaining certification but that don’t necessarily carry over into teaching itself (e.g., learning how to be better candidates for national board certification but not how to be better teachers). The other quarter of teachers fell into a category the researchers defined as “deferred learning,” in which the things that teachers learn from the process are deferred to a time when they have more opportunity to reflect and to consider how to use them. The authors have since accumulated information on the assessment results for the participants. As is typical for the general pool of all NBPTS applicants, about half of each pretest and posttest group passed. Analyses indicate that both teachers who passed and teachers who failed showed gains on the simulated exercises, suggesting that even teachers who are not successful learn something from the process. Comments: This study examined the extent to which teachers learned from the board certification process and is one of only two that study this issue. It represents a first step in examining the effects of the process on teachers, but it does not evaluate the extent to which teachers incorporate what they learn into their classroom practices. The authors present the results of an analysis of covariance to evaluate the extent of the variance in performance attributable to gender, years of experience, class size, stu- dent type (with regard to general ability/motivation level), school context, and geographic region. The results suggest that the findings are explained by the covariates. In discussions with the first author, we learned that the analyses of covariance were not entirely correct, in part because of prob- lematic coding of the student type variable. The authors have since rerun these analyses, which show that the differences between pretest and posttest scores remain after controlling for these background variables. McColskey, W., Stronge, J.H., Ward, T.J., Tucker, P.D., Howard, B., Lewis, K., and Hindman, J.L. (2005, June). Teacher effectiveness, student achieve-

APPENDIX A 301 ment, and national board certified teachers. Arlington, VA: National Board for Professional Teaching Standards. Available: http://www.nbpts.org/ UserFiles/File/Teacher_Effectiveness_Student_Achievement_and_National_ Board_Certified_Teachers_D_-_McColskey.pdf [accessed June 2008]. Relevance to evaluation framework: Question 2 (Chapter 5). Purpose: This study compares NBCTs and non-NBCTs in terms of both their classroom practices and their students’ achievement test performance. Subjects/participants: Teachers from four school districts in North Carolina, separated into groups of NBCTs, highly effective non-NBCTs, and least effective non-NBCTs. Methodology and findings: The study involved two phases. Phase 1 involved comparison of achievement test data for NBCTs and non-NBCTs. Phase 2 involved collection of a complex set of information on the teachers, including analyses of lesson plans, classroom observations, student work, and questionnaire data. For Phase I, two years of student test scores in reading and math from 307 5th grade teachers were used; 25 were board certified and 282 were not. Students’ predicted test scores were determined by regressing gender, ethnicity, free or reduced-price lunch status, and English language proficiency on achievement test scores. For each student, the residual was determined as the difference between his or her actual test score and the predicted test score. Students were linked to their teacher, and the residual averaged across students to form a residual score for the teacher (referred to as a Teacher Achievement Index, or TAI). Average student residuals were then compared for non-NBCTs and NBCTs for reading and for math. Mean residuals were not statistically significantly different for the two groups although the variances were found to differ (students of NBCTs were more homogenous). Quartiles were de- termined for the residuals, and the percentages of teachers in each quartile compared for the two groups of teachers. For non-NBCTs, the percentages were roughly equal across the quartiles. In math, NBCTs were more con- centrated in the middle two quartiles (66 percent); in reading, they were more concentrated in the top two quartiles (61 percent). For Phase II, the teachers were classified into three groups based on the achievement test results: NBCTs, highly effective non-NBCTs, and least effective non-NBCTs. The highly effective teachers were those with average residuals in the top quartile; those with average residuals in the bottom quartile were considered least effective. Lists were made of teachers who fell in the two quartile ranges and who were eligible for board certification. Teachers on the lists were contacted and asked to participate in Phase II of the study. All NBCTs were also contacted.

302 ASSESSING ACCOMPLISHED TEACHING The authors had significant trouble recruiting teachers to participate in Phase II. The authors do not specify the number of non-NBCTs invited to participate, but, based on the percentages, it appears that 70 least effective and 70 highly effective teachers were invited, along with the 25 NBCTs. A total of 51 teachers agreed to participate (21 NBCTs, 16 highly effective, and 14 least effective). Nearly all NBCTs agreed to participate, but only about a quarter of teachers in the other two groups agreed. Classroom observations and artifacts of teaching were gathered and compared by group. Attributes of teachers or teaching considered in this part of the study include planning and assessment practices; quality of assignments; teacher beliefs about their instructional strategies, student engagement, and classroom management; teacher questioning activity; stu- dent questioning; student time on task; management strategies and the nature of interventions; and observations that rated teachers on specified dimensions. The results indicated that NBCTs had statistically significantly higher ratings in the cognitive challenge of typical reading comprehension assign- ments than did both groups of non-NBCTs. NBCTs also had the highest mean ratings on teachers’ planning practices. No statistically significant group differences were found in the cog- nitive demands of the questions asked by teachers or by their respective students. In addition, no statistically significant group differences were found in classroom management (number of disruptions or students visibly disengaged), although higher numbers of students of least effective teach- ers were visibly disengaged. No statistically significant group differences were found in terms of teacher interventions used to address disruptions or disengagement. Statistically significant group differences were found on four of 15 dimen­sions of teacher effectiveness (based on classroom observations). In all four cases, highly effective non-NBCTs scored highest on the dimension. Comments: This study is useful in the focus on actual classroom prac- tices and types of information collected from the teachers. The comparison across the three groups of teachers and the methods for assigning teachers to groups were novel. However, the difficulties that the authors experienced in recruiting non-NBCTs to participate are likely to have resulted in a biased sample. Of particular concern are the “least effective” non-NBCTs who agreed to participate. The teachers in this group who agreed to par- ticipate may have been quite selective and different from those who did not agree—in part characterized by their willingness to open up their class- rooms for observations and to talk about their teaching. It is not known how this potential selection bias affected the results.

APPENDIX A 303 Moore, J.W. (2002, December). Perceived barriers to the National Board for Professional Teaching Standards certification. Unpublished dissertation, East Tennessee State University. Relevance to evaluation framework: Question 3 (Chapter 6). Purpose: The author investigated reasons why teachers choose not to pursue board certification. Subjects/participants: Teachers in Cocke and Sevier Counties in Tennessee. Methodology and findings: s of 2002, there were only 40 NBCTs in A Tennessee, and the study focused on uncovering reasons why more teachers had not participated. This researcher administered a survey to the partici- pants, which presented respondents with a list of 38 statements that used a Likert 5-point response scale (strongly agree to strongly disagree). Cluster sampling methods were used to identify the sample. That is, six schools from Cocke County and eight schools from Sevier County were randomly selected from the list of all schools in the county. The survey was distributed to all teachers in the selected schools who were eligible for board certification but had not attained it. There were 1,200 teachers in the two counties who were eligible (300 in Cocke and 900 in Sevier), and surveys were distributed to 700 of the eligible teachers. Usable responses were received from 448 teachers (64 percent response rate). Of the 448, 57 percent (n = 253) said they would not attempt board certification, 38 percent were unsure (n = 171), and 5 percent (n = 24) said they would in the future. The respondents were generally quite negative about board certification. Most (68 percent) said they were poorly informed about the program, and almost two-thirds (62 percent) had a negative over- all opinion about the program. Only 38 percent had a positive opinion. Overall, the respondents indicated that personal reasons tended to present the biggest obstacles to their participation, including the extent of paperwork involved and the time commitment required. They also felt that the effort was not worth the benefits, generally agreeing with statements that achieving board certification represented “professional certification without a professional salary” and “more work without more pay.” There was also some skepticism among respondents about the qualifi- cations of teachers who achieve board certification. Respondents generally agreed with statements suggesting that the NBPTS does not necessarily identify or recognize better teachers. Some also thought that board certifica- tion tended to ostracize certain teachers. Comments: This study is one of only two that investigate reasons why teachers do not participate in the NBPTS, and, unlike the other study, it focuses on teachers who have had no involvement with the NBPTS (the other surveyed NBCTs about their nonparticipating colleagues). As such,

304 ASSESSING ACCOMPLISHED TEACHING it contributes to understanding nonparticipation and yields information that could be used to increase teachers’ involvement in the program. One issue that likely bears on the findings is that there were no NBCTs in the two counties at the time of the study. Only one teacher had attempted the process but had not been successful. (This was learned through personal conversation with the first author.) As the respondents noted, most were minimally familiar with the certification process, and it is quite likely that they may never have met a board-certified teacher. Thus, their responses patterns may identify issues that could be pursued in educating teachers about the NBPTS and the certification process. National Board for Professional Teaching Standards. (2001a, November). I am a better teacher. Arlington, VA: Author. Available: http://www.­nbpts. org/resources/research/browse_studies?ID=23 [accessed November 20, 2007]. Relevance to evaluation framework: Question 5 (Chapter 8). Purpose: To evaluate the impact of the certification process on teachers who have gone through it. Subjects/participants: National sample of teachers who had completed the certification process, both successful and unsuccessful candidates. Methodology and findings: This is a report of a survey conducted by the NBPTS. Surveys were sent to 10,700 candidates who had recently completed the assessment process. The survey contained 27 questions. Within four weeks, 5,641 responses (53 percent) were received, and find- ings are based on these responses. The results in the report are based on 10 ­questions, which were grouped in a section of the survey titled “Ben- efits of the Process for You.” The questions are worded as statements that participants are asked to agree/disagree with (on a five-point scale from strongly disagree to strongly agree). All questions are worded positively (e.g., “participating in the NBPTS process helped me develop stronger cur- ricula”; “participating in the NBPTS process helped me develop improved ways to evaluate student learning”; “as a result of participating in the national board-certification process, I believe I am a better teacher”; “I found that the National Board’s assessment process enhanced the quality of my interactions with my students”). No negatively worded statements are included. The report cites the following as findings: • 92 percent of the candidates surveyed said that they believed the national board certification process made them better teachers. • 96 percent of respondents rated the national board certification

APPENDIX A 305 process as a(n) “excellent,” “very good,” or “good” professional development experience. • Participation in the national board certification process equips teachers to create stronger curricula (89 percent), improves their abilities to evaluate student learning (89 percent), and helps them to develop a framework in which they can use state content stan- dards to improve teaching (80 percent). • Participation in the national board certification process enhances teacher interaction with students (82 percent) and parents and guardians (82 percent), and helps to improve collaborations with other teachers (80 percent). • There is now a high level of awareness in schools (68 percent) and school districts (81 percent) of teachers who are candidates for national board certification and of those who have achieved certification. • Candidates for national board certification are receiving high levels of support from their teaching colleagues (86 percent), principals (80 percent), and district administrators (63 percent). Comments: This survey collected important information about the ex- periences and attitudes of NBPTS participants. However, the report of the findings is written as an advocacy piece, not a research report. There is no information on nonrespondents and no evaluation of the extent to which respondents are representative of the test-takers. The survey had the poten- tial to yield information about teachers who passed and who failed, but the results are not reported separately by group. In addition, all of the survey questions are worded positively. Inclusion of both positively and negatively worded questions would have increased the objectivity of the survey and would have allowed the researchers to examine the presence of response sets (individuals who tend to always select the same response or who tend to provide what they perceive to be an acceptable response). National Board for Professional Teaching Standards. (2001d). The impact of national board certification on teachers: A survey of national board- certified teachers and assessors. An NBPTS research report. Arlington, VA: National Board for Professional Teaching Standards. Relevance to evaluation framework: Question 5 (Chapter 8). Purpose: To evaluate the impact of the certification process on teachers. Subjects/participants: Random sample of all NBCTs. Methodology and findings: his is a report of a survey conducted by T the NBPTS. Surveys were sent to a random sample of 600 of the 4,804

306 ASSESSING ACCOMPLISHED TEACHING teachers who achieved national board certification from 1994 through 1999. Competed surveys were received from 235 respondents (41 percent). The main research questions investigated were (1) How do NBCTs and as- sessors rate the certification process as a professional development experi- ence? and (2) What effect does the certification experience have on NBCTs and assessors and other stakeholders? The authors cite the following findings: (1) the national board certifica- tion process is an excellent professional development experience; (2) NBCTs indicate that the certification experience has had a strong effect on their teaching practices; and (3) the certification process has had a positive effect on students and has led to positive interaction with teachers, administra- tors, and communities. Comments: This survey collected important information about the expe- riences of NBPTS participants. However, the report of the findings is written as an advocacy piece, not a research report. The report provides only an over- view of selected results from the study. There is not enough detail provided to make independent judgments about the validity of the results. Sanders, W.L., Ashton, J.J., and Wright, S.P. (2005, March). Comparison of the effects of NBPTS-certified teachers with other teachers on the rate of student academic progress. Arlington, VA: National Board for Professional Teaching Standards. Available: http://www.nbpts.org/resources/research/ browse_studies?ID=15 [accessed November 27, 2007]. Relevance to evaluation framework: Question 4 (Chapter 7). Purpose: The study examines the relationships between student achieve- ment and board-certification status. The authors also evaluate the impact of model specification on the findings. Subjects/participants: Students in 3rd through 8th grade for the 1999- 2000 through 2002-2003 school years in Charlotte–Mecklenburg and Wake County school districts in North Carolina and their teachers. Methodology and findings: or this study, the authors separated teach- F ers into four groups: NBCTs, future NBPTS candidates, NBCT applicants who failed to be certified, and teachers with no NBPTS involvement. They then compared student achievement in mathematics and reading for teach- ers in the four groups. The authors fit four models for each subject area. In two models they used the current-year score on the state’s end-of-year test as the outcome or dependent variable, and in the other two models they used the gain score (prior-year score less the previous-year score) as the outcome variable. For both subject areas and each outcome (level score or gain score), one model included random effects for teachers and the other did not. Including

APPENDIX A 307 teacher random effects accounted for the nesting of students within classes and should provide substantially more accurate standard errors than the models that ignore this nesting. The two models without teacher random effects were designed to be similar to those used in the studies by Cavaluzzo (2005) and by Goldhaber and Anthony (2007). The results for these models were compared with results for models that included teacher random effects, so as to examine the effect of model specification on the results. Models with the current score as the outcome included prior-year math- ematics and reading scores as covariates. All models also controlled for students’ gender and race/ethnicity, and teacher’s years of experience. The authors fit models separately by grade and subject area, comparing NBCTs with each of the other NBPTS groups (failed applicants, future applicants, and nonapplicants). The authors reported statistically significant differences between the student outcomes for NBCTs and nonapplicants for grades 5, 6, 7, and 8 in both mathematics and reading for at least one outcome specification. For mathematics, none of the differences was statistically significant in models that include random teacher effects. For reading, differences were statistically significant in models that do and do not include teacher random effects, with the exception of grade 8, in which the effects were significant only in models that included teacher random effects. In addition, the au- thors reported that the models with random effects indicated that there was typically more variance within group than across groups. That is, there was more variance among teachers with board certification than between NBCTs and each of the other groups. Comments: The major contribution of this study is the illustration of the sensitivity of results to model specification. Specifically, the models that accounted for the nesting of students in classrooms (the models that included random effects) generally resulted in substantially larger standard errors, and thus the effects were less likely to be statistically significant. This provides evidence that analyses that do not account for such nesting produce downwardly biased standard errors, which raise the probability of reporting statistically significant effects in error. One limitation of this study is that it gives up power to detect differ- ences by analyzing the grades separately. This is particularly problematic given the likely small numbers of teachers in grades 6, 7, and 8 (sample sizes are not reported). However, the differences between NBCTs and nonappli- cants vary considerably across grades, with some of the largest differences between grades 4 and 5, in which the sample sizes were largest. Thus, even if the data were pooled across grades, it is unlikely that a strong and sig- nificant difference would exist for NBCTs.

308 ASSESSING ACCOMPLISHED TEACHING Smith, T.W., Gordon, B., Colby, S.A., and Wang, J. (2005). An examina- tion of the relationship between depth of student learning and national board-certification status. Arlington, VA: National Board for Profes- sional Teaching Standards. Available: http://www.nbpts.org/UserFiles/File/­ Applachian_State_study_D_-_Smith.pdf [accessed June 2008]. Relevance to evaluation framework: Question 2 (Chapter 5). Purpose: To compare work samples for students of NBCTs and of teachers who failed to earn board certification. Subjects/participants: Teachers in 17 states who had attempted to earn board certification; roughly half of the sample consisted of NBCTs, and half were teachers who had failed the assessment. Methodology and findings: This study builds on prior work by Bond et al. (2000) and uses some of the same methodologies. The study involved comparison of instructional practices and students’ work for 64 teachers from 17 states. The sample included NBCTs and teachers who had at- tempted to become board certified but were unsuccessful. The teachers were randomly selected from information provided by the NBPTS. Initial contact with potential participants was via a mail survey. Recruitment proceeded by telephone for teachers who returned their surveys expressing interest in participating. A letter of invitation was mailed to 705 teachers (280 NBCTs, 425 unsuccessful candidates) who had pursued board certification in one of four areas: (1) middle childhood generalist, (2) early adolescence English lan- guage arts, (3) adolescence/young adulthood science, and (4) adolescence/ young adulthood social studies–history. The authors had some difficulty with recruitment, originally trying for 200 participants, 50 in each of the four certificate areas. They were not able to recruit that many participants. Initially, 202 teachers verbally agreed, but there was considerable attrition at various stages of the recruitment process. The final sample included 35 NBCTs and 29 teachers who had failed the assessment. No intermediate details are provided about the sampling methodology. Data evaluated for each teacher included (1) the teacher’s description of a unit of lessons; (2) student work samples for 6 randomly selected students from each teacher’s classroom; and (3) for the generalist and language arts teachers, students’ responses to a writing task. The teachers’ instructional materials and the students’ work samples were evaluated for deep versus surface features using a taxonomy developed by Hattie and described in Bond et al. (2000). Analysis of student work samples showed that students in classrooms of NBCTs demonstrated deeper responses more often than students in classrooms with teachers who had failed the assessment, although the dif- ferences were not statistically significant. The authors noted that sometimes

APPENDIX A 309 the assignment was an issue, not the students’ level of understanding, in that the assignment was not designed to elicit “deep” thinking. For the writing assessment, 18 teachers (nine NBCT, nine non-NBCT) submitted 377 writing assessment responses. The writing samples were given a holistic score as well as six analytic scores based on specific ­writing/ composition features (controlling idea, organization, elaboration of ideas, voice, sentence formation). Discriminant function analysis was used to determine the relationships between certification status and writing perfor- mance. The results were statistically significant (p < .05), indicating differ- ences in the writing performance of students taught by NBCTs and teachers who had failed the assessment. Analysis of teachers’ assignments to students showed that the major- ity of the teachers (64 percent) aimed instruction and assignments toward surface learning. However, NBCTs were more than twice as likely to aim in- struction at deeper learning than teachers who had failed the assessment. Comments: This study adds to the construct-based validity evidence provided by Bond et al. (2000). It expands the sample to teachers in 17 states (the sample in Bond et al. was drawn from five states) and used a dif- ferent sampling strategy. The findings generally concur with those reported by Bond et al. One issue with this study was that the authors experienced significant difficulties recruiting participants, and as a result, the sample sizes are small. Most of the attrition seemed to have occurred at the recruitment stage, rather than during the course of the study. The exception is for the writing assessment piece; only 18 of the 64 participating teachers submitted materials to be evaluated. No details are provided to compare the char- acteristics of the final sample with the initial group of recruits. This may be particularly important when considering the representativeness of the study participants who had failed the NBPTS. The teachers in this group who agreed to participate may have been quite selective and different from those who did not agree. It is not known how this potential selection bias affected the results. Sykes, G., Anagnostopoulos, D., Cannata, M., Chard, L., Frank, K., M ­ cCrory, R., and Wolfe, R. (2006). National board-certified teachers as organizational resource. Arlington, VA: National Board for Professional Teaching Standards. Available: http://www.nbpts.org/resources/research/ browse_studies?ID=174 [accessed November 27, 2007]. Relevance to evaluation framework: Question 3 (Chapter 6), Question 6 (Chapter 9), Question 7 (Chapter 10). Purpose: To evaluate the impact of board certification on teachers’ experiences in their schools.

310 ASSESSING ACCOMPLISHED TEACHING Subjects/participants: NBCTs and non-NBCTs in South Carolina and Ohio. Methodology and findings: This report uses the results from three data collections: a school-level survey, a state-level survey, and a four-school field study that focus on samples of teachers in South Carolina and Ohio, two states with high concentrations of NBCTs. The state-level survey was distributed to random samples of teachers in the two states. The authors used stratified random sampling methods, with school level (elementary, middle, and high school) and school location (ur- ban, suburban, and rural) serving as the strata. Surveys were distributed to the 1,500 teachers for whom both a mailing address and an e-mail address were available. Usable responses were obtained from 1,153 (77 percent), roughly half from each of the two states (566 from South Carolina and 587 from Ohio). The researchers designed some of the questions so that the responses could be compared with those from the School and Staffing Survey (SASS), which is based on a national sample of teachers. Using the SASS data, they make comparisons between NBCTs and all teachers in Ohio and in South Carolina. The schools participating in the school-level survey were selected in a multistage process. First, two urban districts were identified in each state based on the district policies about board certification. Then districts neigh- boring the urban districts were selected (one neighboring district for each urban district in South Carolina; multiple neighboring districts for each urban district in Ohio due to smaller numbers of NBCTs). Schools in each district were grouped by the density of NBCTs. Six schools were selected in each urban district: a case-study school, a school with no NBCTs, and four additional schools with varying numbers of NBCTs. Six schools in each neighboring district were also selected: one with no NBCTs and five with varying numbers. This resulted in a final sample of 47 schools (one declined after data collection began), all elementary schools. The school-level survey was administered in person to the entire faculty in each of the 47 schools. A total of 1,583 surveys were completed with an average school response rate of 84 percent. The field study involved interviews with faculty and staff at four schools in the sample of 47, two in Ohio and two in South Carolina. The schools were selected from the urban districts and on the basis of the percentages of NBCTs relative to the district average, with a goal of identifying schools with a “critical mass” of NBCTs. Survey Results. Based on the state-level survey, the authors found that in both states, NBCTs tended to perceive that they have more influence over schoolwide policies than do all teachers (based on comparisons with the SASS). Difference between NBCTs and all teachers were statistically

APPENDIX A 311 significantly with respect to their perceptions of influence on curriculum, the content of inservice professional development, evaluation of teachers, and hiring new full-time teachers. Comparisons between NBCTs and non-NBCTs with regard to the same issue again shows that NBCTs perceive higher levels of influence on school- wide policies, but differences were statistically significant only in the areas of establishing curriculum and evaluating teachers. The state-level survey queried NBCTs about the activities in which they have participated since becoming certified. Over half indicated that they mentor other teachers, serve as team leaders, develop or select curriculum materials, support other national board candidates, and provide profes- sional development to teachers at their schools. Some NBCTs also reported participating in district-level or state-level activities, and this seemed to vary with the number of years they had held their board certification. For example, over half of the teachers who held their certification for seven or more years, reported serving as mentors, serving as team leaders, providing professional development, supporting national board candidates, and de- veloping curriculum materials at the district level. The authors did not ask respondents if they were involved in these activities prior to receiving board certification and point out that it may be that teachers who participate in leadership activities self-select into the national board process. Field Studies. This portion of the study focuses on two elementary schools in each of the studied states. At all four schools, roughly one-fifth of the teachers were board certified. The team of researchers spent roughly a week in each school and conducted interviews with all of the teachers. They generally found that teachers did not interact with each other about their instructional practices and not too much was made of board certification. The teachers who had obtained board certification were generally positive about the experience, although they were reluctant to state that board certification signaled a level of competence that set them apart from their colleagues. The non-NBCTs tended to think there was no difference be- tween the NBCTs and themselves, sometimes citing stories of well-qualified teachers who tried and did not pass or less qualified teachers who passed. Principals also noted that they were careful about how they meted out as- signments, not wanting to seem to overly favor the NBCTs or to engender envy or resentment from the non-NBCTs. The authors provide descriptions of the ways in which NBCTs are viewed at each of the four schools and the type of leadership activities in which they are involved. In one school, “Stevenson,” the district facilitated NBCT leadership by enabling NBCTs to qualify for grade-level team leader positions within the school. The teams served as important opportunities for collaboration and the sharing of technical expertise. The team leader was seen as a potentially highly influential position. District policy allowed

312 ASSESSING ACCOMPLISHED TEACHING NBCTs to leave their classrooms to assist other schools in instructional improvement. In this school, over half of the non-NBCTs identified the leadership roles that NBCTs held within the school, although they still said that NBCTs were “no different” than other teachers in terms of their com- mitment to schoolwide issues, initiatives, and concerns. In the other schools, less experienced teachers tended to seek advice and support from the schools’ NBCTs; however, often this was by informal mechanisms. Formal mechanisms in which NBCTs could provide advice or leadership were not present, and there was evidence that principals down- played board certification and that NBCTs themselves “concealed” the fact that they were board certified. Comments: This is a comprehensive study that corroborates findings reported in Koppich et al. (2006) with regard to the unsupportive environ- ments faced by NBCTs. This study and the Koppich study drew samples of teachers from some of the same states (both studied teachers in Ohio and South Carolina), so it would be useful to investigate the extent to which the reported conditions exist in other states. The use of multiple data col- lection strategies (school-level survey, state-level survey, and case studies) is a strong feature of this study. The design of survey questions so that comparisons can be made with SASS results is also very useful. The one area that the researchers might have also explored is the issue of failing the NBPTS assessment and any impacts that might have on teachers. Yankelovich Partners. (2001, April). Accomplished teachers taking on new leadership roles in schools: Survey reveals growing participation in efforts to improve teaching and learning. Arlington, VA: National Board for Pro- fessional Teaching Standards. Available: http://www.nbpts.org/resources/­ research/browse_studies?ID=22 [accessed November 27, 2007]. Relevance to evaluation framework: Question 5 (Chapter 8), Question 7 (Chapter 10). Purpose: To evaluate the extent to which NBCTs assume new leader- ship roles. Subjects/participants: National sample of NBCTs. Methodology and findings: This study, sponsored by the NBPTS, in- volved a survey of teachers who had received board certification in 1999 or earlier and focused on their participation in leadership roles. Surveys were sent to all NBCTs (n = roughly 4,800) in November 2000 and accepted until mid-January 2001. The report summarizes findings based on the 2,186 who responded as of this date (46 percent response rate). Nearly all respondents (99 percent) said that they had a very favorable or somewhat favorable regard for national board certification.

APPENDIX A 313 Over half of the respondents indicated that they have engaged in the following behaviors since obtaining board certification: • mentoring other teachers pursuing board certification (90 percent); • mentoring struggling teachers (83 percent); • developing or selecting materials to support student learning (80 percent); • involvement in school or district leadership activities (68 percent); • developing instructional strategies or curricula (62 percent); • developing teacher professional development programs or activities (58 percent); • speaking publicly about national board certification (57 percent); • highlighted as experts by the school, press, or community (53 percent); • seeking grants to support teaching and learning (53 percent); and • working with teacher preparation programs at colleges (51 percent). A separate question asked if the respondent had been involved in the activity prior to becoming board certified, and participation is reported as a percentage of those who indicated they are currently involved in the activ- ity. For each of the activities listed above, more than half of the respondents indicated that they had been involved in these leadership activities prior to obtaining board certification. Another question asked about the impact of certification on obtaining or keeping these leadership roles. The leadership roles that appear to be most affected by obtaining board certification all involve NBPTS in some way. For example, over half say that obtaining board certification had an impact on participation in a network of NBCTs, mentoring NBPTS can- didates, advocating for board certification, speaking publicly about board certification, and helping the NBPTS to offer board certification. Very few of the respondents indicate that board certification had an impact on their engaging in other leadership roles. The majority of respondents agreed (strongly or somewhat) with state- ments about the positive effects of leadership activities. For example, they agreed that participation in leadership activities enhanced career satisfac- tion, made them feel more significant in the profession, increased effective- ness as an educator, increased desire to remain in the profession, make them feel that the profession has a lot to offer. These statements all represent positive aspects of such participation. The report does not include any negative statements, such as leadership activities are time-consuming or it is difficult to make time for leadership activities.

314 ASSESSING ACCOMPLISHED TEACHING Comments: There are two versions of this report. One is an advocacy piece called “Leading from the Classroom.” The other is in the form of a memo (from Andrew Kennelly, with Yankelovich Partners, to Mary Buday, with the NBPTS). Neither provides a complete documentation of the meth- odology and findings. The memo simply provides a copy of the survey with the percentages of candidates who selected each response option, which we have summarized above. There is also no information provided to document the extent to which the characteristics of respondents represent those of the group who received surveys (only a statement that asserts the respondents were representative). The lack of details about the methodol- ogy makes it difficult to evaluate the robustness of the findings.

Next: Appendix B: Biographical Sketches of Committee Members and Staff »
Assessing Accomplished Teaching: Advanced-Level Certification Programs Get This Book
×
Buy Paperback | $61.00 Buy Ebook | $48.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The mission of the National Board for Professional Teaching Standards (NBPTS) is to establish "high and rigorous standards for what teachers should know and be able to do, to certify teachers who meet those standards, and to advance other education reforms for the purpose of improving student learning in American schools." In response to a request from the U.S. Congress, the National Research Council developed a framework for evaluating programs that award advanced-level teacher certification and applied that framework in an evaluation of the impacts of the NBPTS. Specifically, this book addresses the impacts on students, teachers, and the educational system in this country. Assessing Accomplished Teaching finds that teachers who earn board certification are more effective at improving their students' achievement than other teachers, but school systems vary greatly in the extent to which they recognize and make use of board-certified teachers. Many of the questions on the evaluation framework could not be answered because the data have not been collected, and the report makes recommendations for the kinds of research that are needed to fully evaluate the impacts of board certification by the NBPTS.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!