Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
3 Defining the Population of Interest A significant portion of the workshop discussion focused on identify- ing a probabilistic econometric model for identifying the population of interest consistent with HUDâs objectives. Participants suggested that be- fore assessing whether the current model is appropriate for the parameters of interest, some important questions need to be addressed: ⢠What is the model for the outcomes? ⢠What is actually being sampled? ⢠Given the sample, what is the role of sampling weights? How does the analysis move from paired events to a universe of populations? ⢠What is the population to which one can generalize from such events? ⢠How does that universe relate to the housing market? TARGET POPULATION During the workshop, the sponsors defined the target population as the housing market in the test sites. Participants noted that the target population differs from the population suggested by the sampling frame, and posed the question, What is the correct definition for the population given the sampling frame? The researchers responded that the actual popu- lation is the housing stock served by advertisements appearing in a siteâs major metropolitan newspaper on Sunday. 15
16 MEASURING HOUSING DISCRIMINATION IN A NATIONAL STUDY Participants suggested an alternative sampling frame that would re- semble the target population more closely. This sample would include all known newspapers with housing advertisements circulating in the test site. With this method, researchers would lose a priori knowledge of the selec- tion probability for a given advertisement. However, this limited knowl- edge may not be necessary since the true probabilities can be computed and weighted appropriately. This alternative sampling frame would yield un- equal probabilities of selection, but participants did not view this as a ma- jor limitation. Another alternative, raised by Tom Louis of the RAND Corporation, would be to abandon the sample survey goal in favor of an alternative unbiased selection procedure. The objective would be to select neighbor- hoods or communities in which advertising is prevalent without the goal of selecting a random sample. This method would ignore the sampling proce- dure and would not be concerned with sample survey weights or statistical comparisons within samples. Louis also noted that the reliance on adver- tisements produces selection bias since part of the market is missed (e.g., availability that is âadvertisedâ through word of mouth). The researchers responded that pilot tests in the current study design include nonadvertisement sampling that begins to address this issue by identifying areas that are either protected (e.g., gated communities) or neglected (e.g., small communities that do not typically advertise available housing units in newspapers). Field reconnaissance in these areas provides researchers with available housing stock from which a secondary sample is drawn. Participants discussed several ways to make the sampling frame more realistic. For example, one could look at the income and asset distribution of the minority population in individual metropolitan areas and sample housing units on the basis of that distribution. Urban Institute researchers pointed out that this sampling frame is quite different from one in which the sample of units is based on the distribution of where minorities cur- rently liveâan alternative approach offered during the discussion of audit- ing in underserved communities (see Chapter 6). The researchers expressed concern about this latter modification because it would institutionalize out- comes that may be the result of discrimination. Participants were asked to discuss ways of making the sample more representative of the target populations, but in a neutral manner with re- spect to potential variants existing in the housing market. There was sub- stantial agreement that this would be extremely difficult to do unless HUD were able to clarify the population to which it wants to generalize. The
DEFINING THE POPULATION OF INTEREST 17 alternatives to the sampling frame discussed by participants included some that represent an attempt to overcome the limitations of the paired-testing methodology and allow alternative forms of generalization. Participants agreed that the current methodology poses many difficulties when applied to underserved communities, in which the paradigm does not appear to work well given differences in housing market structures. For example, it is highly unlikely a white individual would seek housing in some small, pre- dominantly minority community. POTENTIAL BIAS IN TEST SITE SELECTION Workshop participants noted that many larger metropolitan areas have newspapers targeted to particular ethnic communities. The housing search patterns for people in some communities, especially minority groups, may not include advertisements in the major Sunday newspaper. In such cases, the initial sampling frame would not encompass advertisements more likely to be read by the subpopulation of interest, a limitation that would have implications for the credibility of the point of entry into the housing mar- ket that the researchers were trying to establish. Furthermore, omitting certain newspapers from the sampling frame could compromise the ability to draw inferences about the population the auditors represent. The researchers noted that the sampling frame will be modified during the pilot phase of the study to include alternative advertisement sources, such as small neighborhood papers. For five specific sites, two types of enhancements to the usual newspaper advertisement sample are being used. The first type of enhancement involves exploring the overall distribu- tion of newspaper advertisements independently for rentals and sales, with the audit researchers looking at the relative distribution of those advertise- ments across neighborhoods within an audit site. Researchers will assess the multisource enhancements used in the pilot studies and develop a sam- pling frame for selecting advertisements in Phase II of the study. In Phase II, researchers will draw from a multiframe sampling source, using multiple neighborhood or community newspapers. However, sampling issues arise from multiple newspaper advertisement sources. In particular, the spon- sors asked the workshop participants to discuss the implications for the sampling frame of the potential overlap in housing unitsâunits that are sampled more than once because they are advertised in several papers. Some participants questioned why multiple newspaper sources were not included in the Phase I study. The researchers responded that they
18 MEASURING HOUSING DISCRIMINATION IN A NATIONAL STUDY wished to preserve known probabilities of selection for the advertisements. Devising a method to calculate the sampling probabilities would have been daunting given multiple newspapers and the fact that on a given day, adver- tisements must be drawn from 22 sites. In addition, the researchers wanted to mirror the 1989 sampling procedure. They acknowledged the limita- tions of this method. Researchers noted further that the Phase I sampling frame was chosen to be comparable with the 1989 HDS. The newspaper methodology used for the 1989 and 2000 HDS to select advertised housing units does not reach the entire housing market. In fact, many minorities often do not seek housing opportunities in areas that do not receive a large amount of news- paper coverage. In addition, there are issues within minority neighbor- hoods and other types of areas regarding obtaining access to the housing stock that is available for both rental and sales. In some instances, it is unlikely a majority individual would seek housing in certain minority neighborhoods. In other instances, housing units may not be advertised in mainstream or community neighborhoods. Participants also discussed is- sues involved in testing in smaller metropolitan areas, some of which are joint rural-type counties. The second enhancement of the sample involves obtaining census es- timates for the proportion of available rental and sales housing for each test site. During the initial visit, auditors will ask the housing agent for the addresses of similar units. The resulting information will yield an auxiliary set of addresses to be used in identifying underrepresented areas that never reach the major metropolitan newspapers. Census housing market data will be matched with the auxiliary addresses. The researchers will then sample from these underrepresented areasâareas with fewer advertise- ments than available housing unitsâwhich are likely to be in minority communities. DRAWING INFERENCES TO THE POPULATION OF INTEREST In assigning tester characteristics, researchers guarantee that auditors are qualified for the housing units. Workshop participants suggested that by allowing for individuals who are on the margin financially, analysts may miss the discrimination of interest. Specifically, some participants believe that pairs of clearly qualified and clearly unqualified home seekers of differ- ing races would receive similar treatment (either acceptance or rejection), but that on the margin, housing agents would make more subjective judg-
DEFINING THE POPULATION OF INTEREST 19 ments and exercise greater discretion by offering additional assistance or compensation to the nonminority tester. By representing only eminently qualified testers, therefore, the audit results may understate the real degree of housing discrimination in the market. The sponsors responded that they believe discrimination may occur for both marginal and overqualified individuals and that both forms of discrimination are important to mea- sure. The strategy of the study, however, is to identify clear and convincing disparate treatment when less discretion on the part of housing agents is allowed. Participants discussed whether the population from which the auditors are drawn is a fair representation of the people who are seeking housing, in other words, whether the audit findings can be used to generalize about the discrimination a typical couple would face in the housing market. Partici- pants suggested that unobservable characteristics of the testers may repre- sent differences from the population of home seekers and that these charac- teristics may not be eliminated through training or protocols. However, as Stephen Ross noted, audit pairs conduct more than one audit. Since many of the pairs are fixed in terms of the people who make them up, their experiences in different settings allow researchers to test for whether unob- servable tester characteristics matter. This issue is discussed further in Chap- ter 5. Ross noted that the researchers have a difficult problem in dealing with very small sample sizes. HUD wants not only a national measure of hous- ing discrimination and changes in the national incidence of discrimination, but also individual metropolitan measures. The audit methodology will produce only 70 samples. Accordingly, the research team will explore dif- ferent options for obtaining exact estimates, such as permutation-type esti- mators or estimators that are corrected for small sample sizes. Such meth- ods were not discussed during the workshop, but Ross (2000) notes that because permutation tests rely on frequency data, weights cannot be used. WEIGHTING ADVERTISED UNITS Participants noted it may be impractical to sample every newspaper in a large metropolitan area, but, as noted earlier, researchers are exploring the use of multiple papers and other advertisement resources in smaller areas, where a multiframe sampling source is more practical. The Urban Institute can use data gathered on the relationship between the sample and the popu-
20 MEASURING HOUSING DISCRIMINATION IN A NATIONAL STUDY lation for these smaller areas to provide information about potential differ- ences in the incidence of discrimination across advertisement sources. Some participants were unclear on the appropriate weight that should be applied to the results, but noted that it should reflect the universe of available housing stock. Weighting audit results by housing unit size may not be particularly relevant, but weights should incorporate the various sources of information on housing availability, including major metropoli- tan newspapers, community newspapers, Internet-related databases, the Realtors® database (or MultiList), and other sources. Participants discussed the implications of accessing a fairly comprehensive database for available sales housing, such as the MultiList available to real estate agents. Perhaps such a list would serve as a base against which researchers could sample units and create approximate weights. Although participants did not know of the feasibility of this option, one stated it is more appropriate to obtain approximate weights for the right population than precise equal probability weights for the wrong population. Each specification has associated trade- offs that should be considered before results are reported.
