A Plan for Evaluating the District of Columbia's Public Schools From Impressions to Evidence (2011) / Chapter Skim
Currently Skimming:
7 From Impressions to Evidence: A Program for Evaluation
7 From Impressions to Evidence: A Program for Evaluation
Pages 129-160
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 129... ...
The indicator system should provide long-term trend data to track how well programs and structures of the city's public schools are working, the quality and implementation of key strategies to improve education, the conditions for student learning, and the capacity of the system to attain valued outcomes. The in-depth studies should build on indicator data.
|
From page 130... ...
It is designed to be adaptable to changing priorities and circumstances as well as to the varying availability of resources to support evaluation, in DC and in any school district. Figure 7-1 depicts our proposed evaluation framework, which begins with the goals the District has set for itself, as shown in the horizontal box that appears at the top of the figure.
|
From page 131... ...
and students? Better Conditions etter Conditions New Strategies e w S t r a t e g ie s Better Outcomes etter Outcomes New Structures and R l St t nd Roles Mayoral Control Reduced absenteeism evaluation, retention, Increased student learning • • Personnel recruitment, • Effective teachers, principals support Reduced achievement gaps New roles: teaching and learning • • Rigorous, differentiated Improved graduation rates Chancellor • curriculum, assessments Increased participation in adequate facilities • • Rigorous standards, • Clean and safe schools; postsecondary education • • State Superintendent and facilities supplies and services • State Board of Ed.
|
From page 132... ...
The framework simplifies the realities of urban school districts in order to help evaluators and the entire community make sense of a complex reality and to ensure that a full range of data are collected to answer the most important questions. As noted above, it is designed to accommodate the city's changing priorities and concerns over time.
|
From page 133... ...
For strategies to improve student outcomes, such as academic achievement or high school graduation rates, or to reduce gaps in achievement among student subgroups, another step is needed -- the new structures and
|
From page 134... ...
Improved conditions for learning are the critical means by which reforms influence student outcomes, and monitoring them is an integral aspect of the proposed evaluation program. The monitoring should be done in individual classrooms and schools, but it will also be important to look at distributions of conditions across the entire system and how they vary by neighborhood, type of school, and subgroups of students.
|
From page 135... ...
In education, for example, school districts typically collect average scores on a standardized reading assessment for each grade to monitor how well students are meeting basic benchmarks as they progress in reading. Other commonly used indicators include high school graduation rates, rates of truancy, ratios of teachers to students, and
|
From page 136... ...
The first steps in the committee's proposed evaluation program will be to examine thoroughly the data already collected regularly and to assess the quality of the measures and whether they yield the information needed for evaluation purposes. In-Depth Studies Ongoing indicators provide a general picture of a system and identify patterns that warrant further investigation, but in-depth studies are needed to provide finer resolution.
|
From page 137... ...
Some studies seek answers to explanatory questions, such as how particular student outcomes occurred. A third type of question aims to attribute an outcome to a particular policy or practice, asking, "Did this particular policy or practice cause student outcomes to improve?
|
From page 138... ...
This reporting would also be supplemented by the reports generated by the evaluators. AN EXAMPLE OF INTEGRATING EVALUATION ACTIVITIES: IMPROVING TEACHER QUALITY The committee's proposed evaluation framework and the discussion above provide an overview of the primary elements of evaluation and the kinds of anlyses it would include, but they do not indicate in detail how these pieces would be integrated and how priorities for topics and studies will be established.
|
From page 139... ...
Improving the quality of DCPS's teachers was a key element of the strategy of former chancellor Michelle Rhee: see Box 7-1. As we discuss in Chapter 6, the research on teacher quality suggests that it is the product of many district and school strategies, including efforts to • recruit and retain effective teachers and ensure that they are equi tably distributed across schools; • evaluate teachers' effectiveness; • provide professional support and development to all teachers, as well as targeted support for teachers who need it; and • foster working conditions that support trust and collaboration among teachers.
|
From page 140... ...
. This knowledge base provides the basis for thinking about ongoing indicators that can be used to monitor the District's strategies for fostering teacher quality and in-depth studies that would supplement the indicators.
|
From page 141... ...
. The purpose of seeking high teacher quality is to improve student outcomes.
|
From page 142... ...
We also note that indicators of teacher effectiveness that rely on student achievement or teacher observations will need to be reconsidered as knowledge about the characteristics of the measures improves. Indeed, in-depth studies should include internal analyses of the validity of measures of teacher quality, and those measures should be improved as needed.3 Ongoing indicators related to teacher quality that can produce the kinds of information the District needs fall into two categories: those that measure characteristics of the teachers themselves and those that measure teacher recruitment, retention, and support for teachers.
|
From page 143... ...
Recruitment, Retention, and Professional Support Similarly, a range of measures would be valuable as indicators of teacher recruitment, retention, and professional support: • number and percentage of high-quality (by district definition) teachers retained, on a districtwide basis and by school character istics, such as the socioeconomic status of the students; • timely and efficient recruitment process, as measured by the per centage of offers to prospective teachers and principals before the end of the school year; • percentage of novice teachers who receive mentoring or other induction supports such as reduced teaching load, common plan ning time, orientation seminars, or release time to observe other teachers; • percentage of teachers who participate in high-quality professional learning opportunities, that is, those that are sustained, content focused, and involve participation with colleagues; • percentage of teachers who participate in high-quality professional learning opportunities for high-need subjects and populations; and • percentage of teachers who report positive working conditions and professional learning environment in their schools.
|
From page 144... ...
4 Many organizations are currently supporting research on teacher quality, including the Bill & Melinda Gates Foundation (see http://www.gatesfoundation.org/highschools/Documents/ met-framing-paper.pdf [accessed January 2011]
|
From page 145... ...
. A combination of ongoing indicators and in-depth studies can also be used to assess the evidence base for specific approaches the DCPS is using to attract and keep effective teachers in schools in the highest poverty areas -- which would include assessing the measures of quality on which they relied; whether the numbers of highly effective teacher in those schools did increase; and whether educational experiences improved in any measurable way in those schools.
|
From page 146... ...
These two mandates can be compared to the warp and woof in a piece of fabric, as shown in Figure 7-2: the elements of reform and the broad evaluation questions pertaining to them are depicted in the vertical boxes; the substantive areas of responsibility are depicted with shaded horizontal bands. A comprehensive evaluation program would include indicators and studies to address critical evaluation questions about each of the elements of reform (depicted in Figure 7-1, above)
|
From page 147... ...
Outcomes: " Established and Are strategies Are conditions for Are valued outcomes working as intended? evidence-informed, of student learning being attained overall improving overall and and across diverse Reform Elements of quality, and across diverse schools schools and students?
|
From page 148... ...
? Classroom Teaching Do professional Do teachers who Do the students of and Learning: development plans have received teachers who have Alignment Between and activities professional received professional Curriculum and address the learning development development Professional goals that are demonstrate demonstrate greater Development articulated in the increased use of the mastery of curricular curriculum?
|
From page 149... ...
Does parents report greater custodians, and the use vary across involvement and other community neighborhoods, satisfaction with their residents to voice grade levels, etc.? children's education their concerns, over time?
|
From page 150... ...
For DCPS, a primary reform strategy has been upgrading the qualifications and effectiveness of human resources and strategies to improve teacher quality, so this is one obvious evaluation priority. But other strategies should also be included in the evaluation program.
|
From page 151... ...
ESTABLISHING LONG-TERM EVALUATION CAPACITY Building and maintaining a set of high-quality indicators, designing indepth studies that address pressing issues, and organizing the presentation and dissemination of findings so that all stakeholders can use them will require deliberate and skillful management. We have argued that periodic attempts to evaluate the effects of PERAA and the status of the public schools will provide neither the breadth nor depth of information needed.
|
From page 152... ...
performs many of the research functions that are important for school districts, and it is an early example of a structure for providing independent information to support a district's efforts to improve. This consortium, which includes researchers from the University of Chicago, the school district, and other organizations, was formed in 1990 to study reform efforts in the city's public schools, following legislation that decentralized their governance.7 CCSR maintains a data archive that includes student test scores, administrative records, and grade and transcript files, as well as other data such as census and crime information and qualitative information from annual 6Formore information, see http://www.dataqualitycampaign.org/ [accessed November 2010]
|
From page 153... ...
BOX 7-2 Baltimore Education Research Consortium Founded in 2007, the Baltimore Education Research Consortium (BERC) is a partnership between Baltimore City Public Schools (BCPS)
|
From page 154... ...
The Central Office Review for Results and Equity, housed at the Annenberg Institute, also conducts reviews of individual districts, using data to assess how well they meet its criteria for effective district functioning: • communicating big ideas, • service orientation, • data orientation, • increasing capacity, • brokering partnerships, • advocating for and supporting underserved students, and • addressing inequities. Another model is the Strategic Education Research Partnership (SERP)
|
From page 155... ...
SERP is funded by a number of private foundations, individual donations, and federal agencies. use of student test scores to monitor districts' progress with broad reform goals, including limitations to the inferences that can be drawn from tests in a few subject domains, methodological concerns about how to include demographic data in analysis, and the difficulty of establishing causal links between specific practices or reform and information about student outcomes.
|
From page 156... ...
Just as there is no one model that can readily be adopted in the District, it is unlikely that the specific program adopted by DC will be instantly transferable to other school districts. Our hope, though, is for an evaluation program that is independent of school and city leaders, while remaining responsive to the needs and challenges of all stakeholders, that can generate research that meets the highest standards for technical quality.
|
From page 157... ...
. Identifying Effective Teachers Using Per formance on the Job.
|
From page 158... ...
. What makes a secondary school science and/or mathematics teacher highly qualified?
|
From page 159... ...
. Using data to support human capital development in school districts: Measures to track the effectiveness of teachers and principals.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.