An Evaluation of the Public Schools of the District of Columbia: Reform in a Changing Landscape (2015)

Appendix B

Sample Interview Protocol

The committee and its contractors conducted a total of 44 interviews with a wide range of people who are involved with the school system in D.C., including officials and other employees in city education agencies and community members who have been active in public education-related work (see Chapter 1). They were asked a range of questions about their experiences and views of the Public Education Reform Amendment Act (PERAA), with somewhat different interview protocols, depending on the interviewees’ roles.

This appendix reproduces an example of the interview protocols used in the interviews conducted by the committee and staff with city leaders: see Chapter 1 for a discussion of the interviews.

INTERVIEW TOPICS

As you know, the National Research Council is conducting an evaluation of how D.C.’s schools are faring since PERAA was passed that draws on a variety of information. To understand how PERAA’s reforms are operating on the ground and to help us in interpreting the information we’ve been collecting, we are interviewing a sample of those most directly involved in implementing reforms. Your responses will be kept confidential: nothing you say will be attributed to you personally or to your office.

Although we have some familiarity with the responsibilities of your office, in our interview, we would like to get a better sense of the details of your work.

1. Of all the functions for which your office is responsible, which ones have been the most central to the smooth implementation of PERAA?
2. [Ask as appropriate, given responses to the first question.]

You’ve mentioned your office’s most important functions, now we would like to ask you in greater detail about some of those activities [and also about a few others that have come up in our research.]

a. Could you briefly describe the process by which the IMPACT system was developed, and the most important factors that influenced its design?

• how IMPACT was validated and piloted, including who was asked to provide feedback on its initial results

b. To what extent has IMPACT lived up to your expectations as a strategy for ensuring effective teaching?

c. Have any aspects of IMPACT been significantly modified since its initial implementation?

If yes: — what was the impetus for the modifications?

d. IMPACT seems to be one of the most controversial of the PERAA-related reforms, including being subject to considerable media scrutiny. Why do you think that has been the case?

To what extent are criticisms of either the technical quality of IMPACT or assertions that it is unfair to teachers valid?

3. a. What have been the main strategies in DCPS’ efforts to strengthen the quality of school leadership?

b. What would you describe as the main differences between the characteristics of school principals prior to PERAA and those currently leading DCPS schools?

c. Some commentators have questioned the relatively high level of turnover in DCPS principalships since PERAA’s enactment. How do you respond to that concern?

4. a. In considering all the changes that have been implemented in D.C. since PERAA, which ones do you think have been especially effective in accomplishing their intended purpose?

b. Are there others that have fallen short of expectations or that have been particularly difficult to implement?

c. In deciding whether or not PERAA-related initiatives are effective, what criteria or yardsticks do you use?

5. Finally, what are the biggest lessons that you take away from your work in DCPS thus far?