Rebuilding the Research Capacity at HUD (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

5 Evaluation of In-House Research Like most of their federal counterparts, PD&R staff periodically under- take in-house research and analysis to supplement the division’s external program of funded contract research and evaluation activities. This chapter describes and assesses PD&R’s in-house research and the purposes it serves, which range from descriptions of program activities to sophisticated model- ing and hypothesis testing on a par with academic and scholarly writing in highly regarded peer-reviewed publications. In-House Research: When and Why According to senior PD&R officials, many factors enter into a decision as to whether a given piece of research will be contracted out or conducted by in-house staff. These factors include, among others, staff capabilities and available resources; whether a project has a long planning cycle or a high-priority need for data and analysis that arises suddenly; how quickly the information is needed and by whom; and the sensitivity of the informa- tion being sought and its translation into useful policy guidance. Though there have been some exceptions, most multisite, heavily data-intensive field studies, and those employing large-scale household surveys, are conducted externally. PD&R simply does not have a big enough staff or sufficient travel and other administrative resources for these types of projects. Indeed, as mentioned elsewhere in this report, due to staff reductions PD&R no longer has a policy demonstration division with the capability to carry out even limited field pilot programs or experiments. However, in-house research is usually the preferred approach when access to administrative 79 

80 REBUILDING THE RESEARCH CAPACITY AT HUD records is needed, especially if the questions are policy sensitive or if the time frame for planning and completion is short. Notwithstanding what at first seem to be clear-cut distinctions between in-house and contract research, staff interviews and a review of actual PD&R research products reflects more of a continuum than a clear division of labor. The effective management of complex and state- of-the-art research programs requires active participation in an agency of people who are themselves current on research methods and results. Consequently, for most large-scale external studies, PD&R staff play a significant role in developing the research questions, research design, sampling plan, and at times the data collection instruments. At the back end, staff review and comment upon draft reports, and oversee a rigor- ous report review process. Similarly, some projects using administrative data that are generally undertaken in-house are sometimes contracted out to PD&R’s “research cadre,” a cohort of independent scholars with strong statistical and analytical capabilities who may be specially quali- fied to undertake particular kinds of statistical analysis. Still other ­studies involve more explicit collaboration between PD&R staff and outside contractors. Much of PD&R’s research relating to government sponsored enterprise (GSE) and rule making has involved con­tractor assistance with data analysis. A Typology and Criteria for Evaluation The overwhelming share of PD&R’s in-house research studies serve one of four reasonably distinct purposes: to support (1) policy development, (2) program administration, or (3) regulation and oversight of GSEs, or to provide (4) confidential advice and recommendations to the secretary and the White House. This last body of demand-driven work is largely unpublished and serves the policy needs of the secretary and the president. Though not available to the committee for independent review and there- fore not included in this assessment, examples of these products include, among others, a paper summarizing the federal government’s efforts to sup- port the production of affordable housing and examining ways to increase production over the next several years and a policy paper on how HUD programs and initiatives support an ownership society. The committee identified three reasonably distinct types of work that might fairly encompass the range of in-house research products. Research may: (1) be primarily descriptive; (2) consist largely of a literature review; or (3) involve in-depth data analysis and formal modeling. Together, these two dimensions of the in-house research program—purpose of research and type of work—provide a framework for conceptually classifying every in-house research product. 

EVALUATION OF IN-HOUSE RESEARCH 81 It is useful to apply the same evaluative criteria when assessing PD&R’s external and in-house research because it is the totality of these activities that is aimed at strengthening programs and policies, and informing the sec- retary, Congress, and other constituencies about the department’s responsi- bilities, activities, accomplishments, and needs. These criteria are presented in Table 5-1. While Chapter 3 interpreted these criteria slightly differently for each of the three categories of external studies (large-scale/high impact; intermediate-scale; and small-scale, exploratory studies), here the commit- tee applied the same criteria across all three types of in-house activities. Sample Selection In-house publication records were taken from HUD USER for 1997- 1999 and from division-by-division lists provided to the committee by PD&R for the post-2000 period. Initially, a total of 117 documents were identified (72 in-house documents between 2000 and mid-2007; 45 docu- ments from 1997-1999). The 10-year list of in-house PD&R publications was stratified pre- and post-2000 in order to ensure that the committee sampled from more than one political administration. Prior to drawing a sample, the titles of all documents were reviewed. In-house reports that did TABLE 5-1  Criteria for Evaluating In-House Research Evaluation Criteria Explanation Relevance and importance of Individual studies should be directly related to HUD’s topic current agenda, or otherwise of high department priority. Rigor and appropriateness of Methodology should be appropriate to address the study’s methodology basic objectives and should yield useful, defensible results, though not necessarily statistically generalizable, appropriate to the topic and time frame. Timeliness Study should produce results in a time frame consistent with the need for the information. Qualifications of research team PD&R staff assigned to the study should possess the technical and/or policy qualifications appropriate for the project. Quality of research products Products should include sufficiently complete documentation of data and methods suitable for third- party understanding of study’s strengths and limitations; comprehensive and understandable assessment of implications. A minority of studies may eventually be published in peer-reviewed journals. 

82 REBUILDING THE RESEARCH CAPACITY AT HUD not have an explicit research orientation—for example, data set documen- tation, strategic plans, or other similar products—were removed from the sample. Ultimately, the committee reviewed 29 documents, out of the 60 deemed eligible (see Table 5-2). Net of disqualified reports, the final over- all sampling fraction was roughly one out of every two in-house research reports (48 percent). The sample included 18 of 43 selected reports for the 2000-2007 period (42 percent) and 11 of 17 pre-1999 reports (65 percent). TABLE 5-2  Sample of In-House Research, by Purpose and Type of Work To Support Policy Development Descriptive Studies Waiting in Vain: An Update on America’s Rental Housing Crisis (U.S. Department of Housing and Urban Development, 1999a) Housing Our Elders: A Report Card on the Housing Conditions and Needs of Older Americans (U.S. Department of Housing and Urban Development, 1999b) Rental Housing Assistance—The Crisis Continues: The 1997 Report to Congress on Worst Case Housing Needs (U.S. Department of Housing and Urban Development, 1998a) Rental Housing Assistance—The Worsening Crisis: A Report to Congress on Worst Case Housing Needs (U.S. Department of Housing and Urban Development, 2000b) A Report on Worst Case Housing Needs in 1999: New Opportunities Amid Continuing Challenges (U.S. Department of Housing and Urban Development, 2001) Affordable Housing Needs: A Report to Congress on the Significant Need for Housing—Annual Compilation of a Worst Case Housing Needs Survey (U.S. Department of Housing and Urban Development, 2007a) Literature Review Does Housing Assistance Perversely Affect Self-Sufficiency? A Review Essay (Shroder, 2002) In-Depth Analysis Welfare Reform Impacts on Public Housing Program: A Preliminary Forecast (U.S. Department of Housing and Urban Development, 1998c) Vouchers Versus Production Revisited (Shroder and Reiger, 2000) Unequal Burden: Income and Racial Disparities in Subprime Lending in America (U.S. Department of Housing and Urban Development, 2000d) Can Housing Assistance Support Welfare Reform? (Khadduri, Shroder, and Steffen, 2003) The Impacts of Welfare Reform on Recipients of Housing Assistance (Lee, Beecroft, and Shroder, 2005) The Flexible Voucher Program: Why a New Approach to Housing Subsidy Is Needed (U.S. Department of Housing and Urban Development, 2004b) continued 

EVALUATION OF IN-HOUSE RESEARCH 83 TABLE 5-2  Continued To Support Program Administration Descriptive Studies In the Crossfire: The Impact of Gun Violence on Public Housing Communities (U.S. Department of Housing and Urban Development, 2000a) Current Housing Unit Damage Estimates from Hurricanes Katrina, Rita, and Wilma (U.S. Federal Emergency Management Agency, U.S. Department of Housing and Urban Development, and U.S. Small Business Administration, 2006) The Uses of Discretionary Authority in the Public Housing Program: A Baseline Inventory of Issues, Policy, and Practice (Devine, Rubin, and Gray, 1999) The Number of Federally Assisted Units Under Lease and the Costs of Leased Units to the Department of Housing and Urban Development (U.S. Department of Housing and Urban Development, 2007c) Literature Review Section 8 Tenant-Based Housing Assistance: A Look Back After 30 Years (U.S. Department of Housing and Urban Development, 2000c) In-Depth Analysis Redistribution Effect of Introducing Census 2000 Data into the CDBG Formula (Richardson, Meehan, and Kelly, 2003) CDBG Formula Targeting to Community Development Need (Richardson, 2005) To Support Regulation and Oversight of GSEs Descriptive Studies The GSEs’ Funding of Affordable Loans: A 1996 Update (Bunce and Scheesele, 1998) Characteristics of Mortgages Purchased by Fannie Mae and Freddie Mac, 1993-1995 (Manchester, Neal, and Bunce, 1998) The GSEs’ Funding of Affordable Loans: A 2000 Update (Bunce, 2002) An Analysis of Mortgage Refinancing, 2001-2003 (U.S. Department of Housing and Urban Development, 2004a) Literature Review Understanding Consumer Credit and Mortgage Scoring: A Work in Progress at HUD (Bunce, Reeder, and Scheesele, 1999) In-Depth Analysis The Multifamily Secondary Mortgage Market: The Role of Government Sponsored Enterprises (Segal and Szymanoski, 1997) HMDA Coverage of the Mortgage Market (Scheesele, 1998a) The GSEs’ Purchase of Single-Family Rental Property Mortgages (DiVenti, 1998) An Analysis of GSE Purchases of Mortgages for African-American Borrowers and Their Neighborhoods (Bunce, 2000) 

84 REBUILDING THE RESEARCH CAPACITY AT HUD Operating as it does in an extreme resource-constrained environment, it is important that PD&R focus intently on research that supports the department’s broad strategic goals. The committee was pleased to find that all of the 29 research products it evaluated met that criterion, with some supporting more than one goal. Eight studies addressed HUD’s goal of increasing home ownership opportunities; 20 spoke to the promotion of affordable housing; 5 focused on expanding fair housing and equal oppor- tunity; and 11 were about strengthening communities. Regardless of research purpose, the sampled studies were evenly split between those that consisted primarily of descriptive analysis and those that featured more complex statistical analysis and modeling. Describing a study as descriptive is not meant pejoratively. Certainly, it would be a mis- take to conclude that almost half of PD&R’s internally produced research consisted of pro forma displays of routine data. Nothing could be further from the truth. As the following discussion of the various research types suggests, whether in support of policy development, program administra- tion, or within the context of GSE regulation, most descriptive studies were quite complex, nuanced, and highly informative. And, frequently, this important information could not have been obtained without the creative and complex merging of administrative data from more than one data base. RESEARCH IN SUPPORT OF POLICY DEVELOPMENT As suggested above, some research reports in support of policy devel- opment are purely informational, taking the form of annual reports on particular program activities, sometimes directed to Congress, sometimes to the public. Generally, while this type of work does not require sophis- ticated research methodologies, it may involve the extraction of relevant information from one or more administrative data systems requiring com- plex computer programs and sophisticated data manipulation skills. Not infrequently, a single descriptive report may require the need to draw compatible data from multiple data bases or to merge program data from different systems. Though in-house reports of this sort rarely involve formal assessments of program impacts or involve comparison groups, as is the case with much external research, they may involve quite sophisticated statistical analysis. More often than not, they provide various HUD clients and constituents important descriptive metrics and benchmarks over time. Because of their program and policy importance, their methodology, accuracy of reporting and interpretation are paramount. Prominent examples of this type of in- house work include PD&R’s periodic reports on worst case housing needs 

EVALUATION OF IN-HOUSE RESEARCH 85 for rental assistance, descriptive papers on the location patterns of hous- ing choice voucher recipients, and analysis of the length of stay in assisted housing. Some examples of recent work of this type are given below. Evaluating the Effect of Changes in Discretionary Authority in Public Housing Programs In 1999 PD&R issued The Uses of Discretionary Authority in the P ­ ublic Housing Program: A Baseline Inventory of Issues, Policy, and ­ ractice (Devine, Rubin, and Gray, 1999). The passage of the Quality P Housing and Work Responsibility Act (QHWRA) of 1998 was a pivotal time in the history of the public housing program because it granted the nation’s public housing agencies unprecedented flexibility and latitude in such areas as tenant selection, the use of income incentives, and the set- ting of minimum and ceiling rents. Because QHWRA had the potential to fundamentally change the nature of public housing, establishing a baseline for how Public Housing Authorities were exercising discretionary authority prior to QHWRA was essential in laying the groundwork for PD&R to be able to assess the long-term impacts of public housing reform in the future in a systematic way. The Impact of Gun Violence on Public Housing Authorities In 2000 PD&R issued In the Crossfire: The Impact of Gun Violence on Public Housing Communities (U.S. Department of Housing and Urban Development, 2000a), a report that used both HUD and Bureau of Justice statistics to examine the scope and magnitude of gun-related violence in and around public housing. The study used the National Crime Victimiza- tion Survey to identify respondents residing in public housing and to report their exposure to crime. Among the report’s key findings was that people in public housing are over twice as likely to suffer from firearm-related victimization as people living elsewhere. The Cost of Leased Units In 2007 PD&R issued The Number of Federally Assisted Units Under Lease and the Costs of Leased Units to the Department of Housing and Urban Development (U.S. Department of Housing and Urban Develop-   ouseholds H with worst case needs are renters who do not receive housing assistance from federal, state, or local government programs; have incomes below 50 percent of their local area median family income, as determined by HUD; and pay more than one-half of their income for rent and utilities or live in severely substandard housing. 

86 REBUILDING THE RESEARCH CAPACITY AT HUD ment, 2007c). Though purely descriptive, this brief report is an exemplar of PD&R expertise in the use of administrative data to create policy-important program data otherwise unavailable from any source. Knowing the size and federal cost of supporting the project-based assisted housing stock is important to policy makers, appropriators, and housing advocates. Esti- mates were derived using administrative data from several data bases; the work included procedures for avoiding duplication of records and double counting or under reporting, and created policy-relevant metrics otherwise unavailable. The report exemplifies the sophisticated data base manage- ment and analysis skills necessary for PD&R to be able to report seemingly straightforward and highly useful program information. Monitoring Affordable Housing Needs In the 1980s, HUD began reporting to Congress what are defined to be worst case housing needs, but the reports became increasingly informal and irregular. In 1990 the Senate Appropriations Committee directed HUD to “resume the annual compilation of a worst case needs survey of the United States.” Consequently, an important in-house research product of PD&R is the regular report to Congress on worst case housing needs, and several versions of this report fell into the committee’s sample. The principal source of national housing data for the worst case needs report has been the biennial American Housing Survey (AHS). However, in various versions of the report, PD&R staff have supplemented AHS data with data from administrative data bases and other federal studies to enrich the analysis and policy importance of these studies. Over time, PD&R staff have also improved and refined the methodology used to prepare the report. For example, while the first worst case needs report, published in 1991, only used data from the 1989 national AHS, the second report, published a year later, augmented the 1989 AHS data with information from the metro- politan surveys conducted during 1987-1990. Two years later, in 1994, the report was based on the 1991 AHS supplemented with data from the 1990 decennial census. In 1996, the report incorporated HUD administrative data for the first time in order to report on the characteristics of households participating in public housing and Section 8 programs. In 1998, PD&R brought supply considerations into the analysis, noting that “the stock of housing affordable to the lowest income families is shrinking” (U.S. Depart- ment of Housing and Urban Development, 1998a, p. i). With extreme rent burdens accounting for the vast majority of hous- ing need, more recent worst case needs reports have used data from HUD administrative datasets such as the Multifamily Tenant Characteristics Sys- tem and the Tenant Rental Assistance Certification System, and from the federal Survey of Income and Program Participation (SIPP) to measure 

EVALUATION OF IN-HOUSE RESEARCH 87 changes in severe rent burdens for individual households over time. This information could not be obtained from the AHS national panel because AHS follows the same housing units over time, not households; SIPP is a household-based survey. Thus, the newest report is not only able to deter- mine by how much worst case needs have grown, but also to report on the stability of high rent burdens over time. Housing for the Elderly In 1999 PD&R issued Housing Our Elders: A Report Card on the Housing Conditions of Older Americans (U.S. Department of Housing and Urban Development, 1999b). The study was based on a special supple- ment to the 1995 AHS national panel—on home accessibility needs and modifications—which was used to develop a baseline of information on elderly housing conditions, needs, and strategies. This supplement laid the groundwork for greater policy concentration on the housing conditions and maintenance needs of low-income seniors. RESEARCH AND Analysis IN Support oF PROGRAM ADMINISTRATION Many federal programs use complex formulas to distribute large flows of program resources to units of government, housing authorities, and other populations. The development and fine tuning of program formulas require a sophisticated understanding of legislative and regulatory program require- ments and the methodological ability to develop alternative formulas and test their sensitivity. Prominent examples of this important work include the fine tuning and consideration of alternative formulas for HUD’s ­largest community development program, the Community Development Block Grant (CDBG) and the development of formulas for one-time use to dis- tribute disaster relief in accord with statutory intent or regulatory require- ments. Examples of particularly useful recent in-house work are described in the following sections. Refining the Formula for Community Development Block Grants The CDBG formula has undergone five major assessments since its introduction in 1974. CDBG Formula Targeting to Community Develop- ment Need (Richardson, 2005) assessed how well the CDBG formula tar- gets areas most in need after the release and introduction into the formula of data from the 2000 census. The report shows that while the formula generally continues to target those areas in most need, targeting toward community development need has declined substantially over that period. 

88 REBUILDING THE RESEARCH CAPACITY AT HUD Furthermore, the amount of funds going to the neediest grantees on a per capita basis has decreased, while the amount of funds going to the least needy grantees on a per capita basis has increased. The report offered four alternative formulas that would substantially improve targeting to commu- nity development need. Among the most policy-relevant findings from the sophisticated factor analysis conducted for this study was that “two new patterns of variance arose in 2000 … [that] were not evident in 1970, 1980, or 1990: (1) a factor representing fiscal stress associated with immigrant growth; and (2) a factor reflecting low-density places with high poverty concentrations but declining poverty rates” (Richardson, 2005, p. ix). Estimating Housing Unit Damage from Hurricanes Katrina, Rita, and Wilma In late December 2005, the President approved a supplementary appro- priation that included $11.5 billion for the CDBG program to provide “disaster relief, long-term recovery, and restoration of the infrastructure in the most impacted and distressed areas” of the five states affected by H ­ urricanes Katrina, Rita, and Wilma (P.L. 109-148). HUD was charged with dividing the funds among Alabama, Florida, Louisiana, Mississippi, and Texas, with the caveat that no individual state could receive more than 54 percent of the total. PD&R staff developed a formula, briefed senior staff, and provided background on the allocation methodology when the formula was announced on January 25, 2006. PD&R’s report, Current Housing Unit Damage Estimates: Hurricanes Katrina, Rita, and Wilma (U.S. Federal Emergency Management Agency, U.S. Department of Housing and Urban Development, and U.S. Small Business Administration, 2006), provided Congress with detailed tables on the extent and type of hurricane-caused damage for individual housing units, by tenure, insurance status, and housing type for properties in the five states. Inspections were carried out by staff of the Federal Emergency Management Agency and the Small Business Administration. PD&R staff developed the templates for data presentation and methodology for mini- mizing duplicate reporting and undercounting damaged properties. The committee notes that this report was not only important for policy; it also reflects the confidence of the administration in the agility and creativity of HUD and PD&R in high-sensitivity formula-related analysis. RESEARCH IN SUPPORT OF REGULATING AND OVERSEEING GSEs Up until the recent passage of the Housing and Economic Recovery Act of 2008, HUD has been responsible for regulating and overseeing the 

EVALUATION OF IN-HOUSE RESEARCH 89 affordable housing goals of two GSEs, Fannie Mae and Freddie Mac. It has carried out these activities primarily through in-house staff analysis, some- times supplemented by external research. Because this work has served as the federal government’s primary basis for setting goals at particular levels, it has been essential that the quality, accuracy, and rigor of the underlying research and analysis of this work be at the highest possible level and pass the judgment of outside scholars, industry experts, and analysts. Several of the in-house reports in the committee’s sample were part of a large body of work that has been undertaken by PD&R staff over the years on housing finance. In 1992, Congress expressed concern about the GSEs’ funding of affordable loans for low-income families, particularly those living in inner- city neighborhoods that had been “redlined” by prime lenders. Because of this concern, Congress called for HUD to establish three affordable housing goals that the GSEs must meet: (1) a low and moderate income goal, which targets borrowers with incomes no larger than the area median income; (2) a special affordable goal, which targets very low-income ­borrowers and low-income borrowers living in low-income census tracts; and (3) a geo- graphically targeted or underserved areas goal, which targets low-income and high-minority neighborhoods. This mandate resulted in PD&R’s under- taking a number of thoughtful and sophisticated statistical analyzes to determine whether Fannie Mae and Freddie Mac individually and collec- tively lead or lag the conventional primary mortgage market. In July 2008 the President signed into law the Housing and Economic Recovery Act of 2008 (P.L. 110-289), the most comprehensive housing legislation in decades. The bill contains a large number of provisions, including the establishment of a new independent regulatory agency, the Federal Housing Finance Agency (FHFA), designed to improve the safety and soundness supervision of Fannie Mae, Freddie Mac, and the Federal Home Loan Banks. The bill empowers FHFA with broad supervisory and regulatory powers over the operations, activities, corporate governance, safety and soundness, and mission of the GSEs, and provides new and more flexible authority to establish minimum and risk-based capital require- ments. The bill also increases the authority of the U.S. Treasury to buy stock or debt in the GSEs, if necessary, to stabilize markets, prevent disruptions in mortgage availability, and protect taxpayers. Although PD&R, and indeed HUD, have not been responsible for GSE safety and soundness since the housing legislation in 1992, the establishment of the new regulatory agency means that such functions as the setting of GSE affordable housing goals will be transferred from PD&R to the new regulator. It is worth noting that some of the staff for this new agency will come from PD&R’s already understaffed Office of Economic Affairs, which will still be responsible for 

90 REBUILDING THE RESEARCH CAPACITY AT HUD fair market rents, income limits, analysis under the Real Estate Settlement Procedures Act, and, generally, housing finance and public finance issues. CONCLUSIONS AND RECOMMENDATIONS With some reservations noted below, the committee gives generally high marks to PD&R’s in-house research program. Across the products the committee sampled, PD&R researchers displayed deftness in the selec- tion, merging and manipulation, and analysis of a wide range of program- s ­ pecific administrative data bases and public-use surveys required to answer important research and policy questions. Generally speaking, individual researchers and research teams were well suited to their assignments, with no evidence of a substantive lack of expertise or a mismatch between research design and the capabilities of staff to complete high-quality work. The sampled studies had clearly stated purposes and adequate explana- tions of the methodology and limitations of the data used. In a few cases, discussions of methodology accounted for a disproportionate share of the total research product, rather than the presentation and analysis of data, but this is appropriate when the sources and assumptions underlying criti- cal estimates of program costs, activities, or other important metrics are the focus of the project and are necessary to secure public confidence in their reliability. Though mostly descriptive and stopping far short of estimating pro- gram impact or effectiveness, the majority of in-house studies are never­ theless highly analytical and policy relevant. Taken as a whole, they are testaments to PD&R staff program and policy expertise and to their exper- tise in drawing data from one or more administrative data systems to create important program indicators. Virtually all studies filled important information gaps, and a substantial fraction addressed new or previously unexplored or underexplored research questions important to policy or program administration. Perhaps the most sophisticated studies the committee reviewed were products of PD&R’s Office of Economic Affairs, which as noted previ- ously until the introduction of the Housing and Economic Recovery Act of 2008 involved HUD’s oversight of Fannie Mae and Freddie Mac. This office’s working paper series on housing finance has consistently been of a ­ quality, sophistication, and statistical rigor on a par with high-level economic research. Because these studies have guided HUD regulations of the GSEs, the underlying research has often been highly contested, and indeed challenged through formal administrative procedures; almost without exception, the analysis has withstood high-level peer review and adversarial scrutiny. Another notable area of high-quality in-house research are studies 

EVALUATION OF IN-HOUSE RESEARCH 91 dealing with the nexus of housing assistance and welfare reform; several staff papers and studies on this topic have been published in respected peer- reviewed journals. What is notable about these publications is that they are mostly inferential, asking and answering pointed program and policy questions, and thereby sharing more in common with externally produced impact assessments than with other in-house research activities. A complicated by-product of outside publication is that the conclusions of the authors—HUD employees—may not reflect the official opinion or policy of the department, and this is always clearly stated by the authors. Despite this complication, the committee believes that in an organization with serious staff constraints, the ability to publish is testament to the intellectual quality and drive of individuals rather than to the demands of the position. The committee also believes that encouraging staff to create publishable studies by providing access to data sources not generally avail- able to the outside research community, and through other inexpensive incentives, is important to staff development and retention. In many cases, timeliness and usefulness go hand-in-hand. If answers to specific questions are required by a time certain for budget making or possible inclusion in a legislative agenda, or other priority need and they are not forthcoming, PD&R would not be doing its job. However, the commit- tee had a much harder time judging the timeliness of internally produced research results than evaluating the quality of the work. Nevertheless, the issue of timeliness and the basis for PD&R’s in-house research agenda- s ­ etting is an important one. The committee is concerned that recent budget cuts will result in staff having increasingly less time to conduct internal research as time set aside for internal research is increasingly eroded, in the course of dealing with other matters. Major Recommendation 3: PD&R should treat the development of the in- house research agenda more systematically and on a par with the external research agenda. Recommendation 5-1: PD&R should develop a formal process for setting the in-house research agenda with clear priorities and timelines for project delivery. As priorities shift during the year, changes in delivery dates should be formally noted.   Recommendation 5-2: PD&R should develop a more explicit relationship between the in-house and external research agendas. Not following up internally conducted baseline studies with formal external studies of the systematic impacts of policy change risks wasting internal resources. 

92 REBUILDING THE RESEARCH CAPACITY AT HUD Recommendation 5-3: PD&R should encourage and assign staff to attend selected conferences on a regular basis, to help staff stay up to date on evolving research and methods, find out about promising scholars, gain insight on emerging policy questions, and generate fresh ideas about poten- tial research that HUD should be conducting. Recommendation 5-4: The assistant secretary of PD&R should provide incentives to professional research staff to publish their work.