Rebuilding the Research Capacity at HUD (2008) / Chapter Skim
Currently Skimming:
3 Evaluation of External Research
3 Evaluation of External Research
Pages 33-64
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 33... ...
Following a brief overview of the processes used to select and supervise external research organizations, the chapter delineates three broad categories of research -- large-scale, high-impact research studies; intermediate-scale policy and program studies; and small-scale exploratory studies -- and defines criteria for evaluating studies in each category. The chapter then addresses each category in turn, first evaluating individual studies in the category and then assessing the overall portfolio of research projects in the category.
|
From page 34... ...
Types of External Research and criteria for evaluation Three basic types of research studies form part of a comprehensive, policy research agenda: large-scale, high-impact studies; intermediate-scale n some cases, PD&R has awarded "indefinite quantity contracts" to several research I organizations (selected competitively) , which are then tapped for specific, quick-turnaround research projects.
|
From page 35... ...
These studies, conducted in 1977, 1989, and 2000, pioneered the use of the "paired-testing" methodology. Other important examples of PD&R-funded research in this category include the Moving to Opportunity for Fair Housing (MTO)
|
From page 36... ...
Two of these competitions related to the mortgage purchase activities of Fannie Mae and Freddie Mac, and a third explored the topic of socioeconomic change in cities. In 2003 PD&R instituted a different method for securing smallerscale scholarly research on specific topics relevant to HUD assisted housing programs: the "research cadre." In this program, PD&R authorized a private contractor to perform all tasks necessary to select and fund a cadre of as many as 20 social science researchers capable of conducting policy research and analysis using HUD's program administrative data as well as data from other sources.
|
From page 37... ...
Quality of research Products should include complete Products should include complete Products should include a complete products documentation of data and methods; documentation of data and documentation of data and methods; comprehensive reporting of results; methods; comprehensive reporting comprehensive reporting of results; and understandable assessment of of results; and understandable and understandable assessment of the the implications. Quality can also be assessment of the implications.
|
From page 38... ...
The 1977 Housing Market Practices Study provided the first solid estimates of the prevalence of discrimination against African American home seekers (Wienk et al., 1979) and helped build the case for strengthening the enforcement of federal fair housing protections in the 1988 Fair Housing Act Amendments.
|
From page 39... ...
This team included staff and consultants who had been involved in previous paired testing studies and had extensive expertise in fair housing issues, the paired testing methodology, sampling methodologies, and management of large-scale field data collection. Each of the study's three phases involved selection of a representative sample of metropolitan areas in which testing was conducted, selection of representative samples of advertised housing units in these metropolitan areas, highly standardized paired testing protocols, and rigorous statistical analysis.
|
From page 40... ...
to manage the demonstration operations, including baseline data collection, random assignment, monitoring counseling operations, and tracking household outcomes. The Abt Associates team was well qualified for this assignment, consisting of sampling and survey specialists, experts in experimental design demonstrations, and staff with extensive experience in the operations of public housing agencies and the voucher program.
|
From page 41... ...
Housing Vouchers for Welfare Families In 1999 Congress passed a special appropriation of housing vouchers for a demonstration initiative targeted specifically to families making the transition from welfare to work. Public housing agencies were competitively selected to participate in this demonstration, based on locally designed strategies for coordinating housing assistance with welfare reform and welfare-to-work initiatives.
|
From page 42... ...
Together, the three organizations developed the basic framework for the Jobs-Plus demonsration, which was ultimately implemented in randomly selected public housing developments in five cities, with randomly selected comparison developments in each city to allow for rigorous estimates of the impact of saturation services and incentives. Jobs-Plus was designed to test the impact of a saturation intervention that included work incentives, employment services, and community supports for work on employment and earnings among pub aunched in 1992, the HOPE VI Revitalization of Severely Distressed Public Housing L Program replaces severely distressed projects with redesigned mixed-income developments and provides housing vouchers to enable the original residents to rent apartments in the private market.
|
From page 43... ...
The project staff included well-qualified statistical, sampling, and survey methodologists and experts in administrative data assembly and analysis, as well as personnel with extensive experience in the implementation of demonstration initiatives, and the evaluation design for the demonstration implemented creative methods for using administrative data on residents of both treatment and control sites from before and after the intervention to rigorously measure the impact of "saturating" public housing developments with services and incentives. Over roughly a decade, MDRC published 11 formal reports on JobsPlus design, implementation, and results, culminating in a final report on the demonstration's impacts on employment, earnings, and neighborhood health (Bloom et al., 2005)
|
From page 44... ...
demonstration, which allowed a small number of public housing agencies unprecedented flexibility by granting waivers of federal statutes and rules related to the Public Housing and Housing Choice Voucher programs (P.L.
|
From page 45... ...
In addition, while rigorous research documents the persistence of racial and ethnic discrimination in urban housing markets, little is known about how families search for housing and how barriers such as discrimination or unaffordable prices and rents affect their search and decision processes and their ultimate housing outcomes. Similarly, too little is known about the operation of the supply side of today's city and suburban housing markets, including the filtering process (the extent to which particular zoning and land-use regulations restrict the total volume of production)
|
From page 46... ...
a national survey of public attitudes about federal fair housing protections; (3) a pilot paired testing study of discrimination in home insurance; (4)
|
From page 47... ...
Public Attitudes Toward Fair Housing Protections HUD's Office of Fair Housing and Equal Opportunity is responsible not only for enforcing federal fair housing laws, but also for administering programs that expand public awareness of and support for fair housing protections. In order to assess levels and trends in both awareness and support, PD&R commissioned two surveys of public awareness, the first in 2000-2001 and the second in 2005.
|
From page 48... ...
In addition, supplemental samples of four targeted populations -- African Americans, Hispanics, families with children, and people with disabilities -- were interviewed in order to assess the extent to which their awareness or attitudes might differ from the national average. Analysis of the follow-up survey found little change in public awareness of federal fair housing protections, but significant increases in public support.
|
From page 49... ...
The research was conducted by a team with extensive prior experience with paired testing techniques in both real estate and labor market transactions and that had expertise in both testing field operations and statistical methodology and analysis of test data. The final report provided an overview of the home insurance industry and outstanding research questions related to its behavior toward minorityoccupied neighborhoods.
|
From page 50... ...
One such set of studies used an adjusted interrupted time-series econometric model to assess the neighborhood property value effects of various types of assisted housing (Galster et al., 1999, 2000; Johnson and Bednarz, 2002)
|
From page 51... ...
falls into the set of intermediate-scale studies reviewed by the committee. Like the third evaluation, it was conducted by Abt Associates.
|
From page 52... ...
Although the purpose of this study was directly to inform policy makers and program managers, an article reporting the results of lowering the premium for refinances was subsequently published in the peer-reviewed journal Real Estate Economics (Rodda, Lam, and Youn, 2004)
|
From page 53... ...
2001 Abt Associates Study of Homebuyer Activity Through the HOME Investment Partnerships Program (December 2003) 2002 Sociometrics How Do Prospective First-Time Corp.
|
From page 54... ...
2002 Abt Associates Implications of Project Size in Section 811 and Section 202 Assisted Projects for Persons with Disabilities (March 2004) 2003 ESI Abt Associates The State of Affordable Housing in the U.S. -- 2000 (November 2004)
|
From page 55... ...
, and it is easily misused. PD&R could sometimes obtain better information at lower cost by limiting its customary addition of a small number of site visits and interviews to quantitative analysis projects and making better use of administrative data.
|
From page 56... ...
A number of recent exploratory studies have drawn on HUD program data, typically as part of the research cadre. Feins and Patterson (2005)
|
From page 57... ...
in the amount of funding that PD&R has allocated to support emerging housing and urban scholars in the form of dissertation and postdoctoral grants (see Table 3-4)
|
From page 58... ...
SOURCE: Unpublished data from HUD, Office of Policy Development and Research. THE AGENDA-SETTING PROCESS AND OVERALL AGENDA The processes used to develop HUD's funded research agenda limit input from outside the department and constrain PD&R's access to creative and innovative thinking about both research issues and methodologies.
|
From page 59... ...
If PD&R had a policy of encouraging, financially supporting, and perhaps assigning staff to attend selected conferences on a regular basis, it could help PD&R staff stay up to date on evolving research and methods, find out about promising scholars, gain insight on emerging policy questions, and generate fresh ideas about potential research that HUD should be supporting. PD&R's shrinking budget constrains staff from conceptualizing a more ambitious, high-impact research agenda.
|
From page 60... ...
For example, the only rigorous evaluation of HUD's supportive housing programs addressed the effects of supportive housing developments on neighborhood property values and crime in Denver. The only program for which rigorous evaluations have systematically (and repeatedly)
|
From page 61... ...
Investment Partnerships Program Empowerment Zone and Enterprise Community Program Fair housing grant programs Low-Income Housing Tax Statistically controlled estimates of LIHTC project Credits (LIHTC) impacts on property values (2002)
|
From page 62... ...
Also, too few have produced rigorous estimates of impacts or cost-effectiveness for HUD's major programs. As a consequence, PD&R has missed opportunities to inform HUD, Congress, and the public about emerging housing and urban development challenges or about the impacts and cost-effectiveness of alternative strategies for addressing these problems.
|
From page 63... ...
Recommendation 3-2: Congress should allocate a small fraction of HUD program appropriations to support rigorous evaluations designed and conducted by PD&R. Recommendation 3-3: PD&R should design and fund more ambitious, large-scale studies that answer fundamental questions about housing and mortgage markets and about the impact and effectiveness of alternative programs and strategies.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.