Review of U.S. Department of Transportation (USDOT) Study on Implementation of Changes to the Section 4(f) Process: September 15, 2011, Letter Report (2011)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

September 15, 2011 The Honorable Ray LaHood Secretary of Transportation U.S. Department of Transportation 1200 New Jersey Avenue, SE Washington, D.C. 20590 Dear Secretary LaHood: We are pleased to transmit to you this third and final letter report of the Committee for a Review of U.S. Department of Transportation (USDOT) Study on Implementation of Changes to the Section 4(f) Process. Section 6009 of the Safe, Accountable, Flexible, Efficient Transportation Equity Act: A Legacy for Users (SAFETEA-LU) called for USDOT to implement regulations to streamline the process of evaluating impacts of transportation projects on 4(f) resources. The legislation also required USDOT to study the consequences of those changes and the Transportation Research Board (TRB) of the National Academies to comment on the evaluation design and the draft reports. The Phase II draft the committee reviewed at its June 16, 2011, meeting reports positive results from the new de minimis determinations mandated by SAFETEA-LU and from regulatory clarification of what constitutes a feasible and prudent alternative to having an impact on a 4(f) resource. The committee, however, finds that the objectives outlined in SAFETEA-LU Section 6009 have not been adequately met by the USDOT study as presented in the draft report. The study relies on an incomplete conceptual framework for evaluating the outcomes reported and inadequately represents the perspectives of nonfederal stakeholders in 4(f) resources because of low response rates from these groups. After providing a brief overview of the changes instituted by SAFETEA-LU and summarizing the findings from the Phase II draft, the committee provides its review of the draft and offers recommendations for improvement. INTRODUCTION 1 Established in the U.S. Department of Transportation Act of 1966, Section 4(f) was designed to protect publicly owned parks, recreational areas, wildlife and waterfowl refuges, and public and private historical sites [all of which are referred to in this report as ―4(f) resources‖] from use by transportation projects unless USDOT determines that there is no ―feasible and prudent‖ 1 Most of the material presented in this introduction is background information reprinted from the first and second letter reports.

2 avoidance alternative and that ―all possible planning to minimize harm‖ has occurred. Consideration of 4(f) resources is part of the project planning process and is typically conducted concurrently with the National Environmental Policy Act (NEPA) process and the Section 106 process, both of which contribute to the Section 4(f) process. [However, Section 4(f) considerations have their own legislative and regulatory authority, which predates NEPA.] Section 4(f) originated during the peak period of Interstate highway construction. Its goal was the preservation of urban parks and historical sites that were in jeopardy of being destroyed by the rapid expansion of urban transportation corridors, especially freeways. The U.S. Supreme Court articulated a high standard for compliance with Section 4(f) in its 1971 Overton Park decision (e.g., http://supreme.justia.com/us/401/402/). However, in the years that followed, federal courts applied the Overton Park ruling differently in similar situations, reaching diverse conclusions about the extent to which certain mitigating factors may be considered in determining whether an avoidance alternative is feasible and prudent. Congress amended Section 4(f) in Section 6009 of SAFETEA-LU in August 2005, leading to two important changes intended to streamline the Section 4(f) process. First, USDOT was granted authority to approve a project that results in a use of parkland or historic property that is so minor that it does not ―adversely affect the activities, features, and attributes‖ of the Section 4(f) resource, referred to as a de minimis impact. When USDOT determines that an impact is de minimis and the responsible officials with jurisdiction (OWJs) over the resource agree, compliance with Section 4(f) is complete. 2 Unlike a traditional Section 4(f) review, de minimis approvals require public review but not the analysis of avoidance alternatives. Second, USDOT was directed to clarify the factors to consider and the standard to apply for determining the feasibility and prudence of alternatives that avoid the use of Section 4(f) property. USDOT accomplished the second charge through rulemaking and the publication of a final rule on March 12, 2008. The final rule took effect on April 11, 2008. Many transportation agency officials expressed the opinion that the Section 4(f) streamlining provisions of SAFETEA-LU would reduce costs and save time in cases where alternatives are not available or when the use of a 4(f) resource is minor. Although historic preservation and environmental protection groups generally agreed that some modification of Section 4(f) was acceptable in principle, development of the regulations for implementing changes to evaluate feasible and prudent avoidance alternatives proved contentious, and many of these groups remain cautious about how the regulations will be applied. (Implementation of the de minimis rulings did not require regulations.) Partly in anticipation of these concerns, SAFETEA-LU specified that USDOT conduct a study of the effectiveness of implementing the streamlining regulations. Specifically, USDOT was asked to report on the following: Efficiencies that may result from implementation of Section 6009, 2 The committee notes that the language of Section 6009 of SAFETEA-LU lends itself to differing interpretations of what the Federal Highway Administration (FHWA) is required to do. It appears from the text of Section 6009 that the statutory standards for de minimis impact determinations affect historic properties and parks (and other nonhistoric properties) differently. These differences are not necessarily reflected in the short summary provided in the introduction. While distinctions of de minimis impact determinations exist and may be important, sorting out how provisions of Section 6009 are implemented or administered by FHWA is beyond the committee‘s statement of task.

3 The postconstruction effectiveness of impact mitigation and avoidance commitments, and The quantity of projects with impacts that are considered de minimis. SAFETEA-LU also specified that USDOT commission an independent review of the study plan and methodology and any associated conclusions by TRB. Specifically, the TRB committee was tasked with reviewing the design of and the final report for the USDOT study of the implementation of the revised Section 4(f) process. The committee was not tasked with evaluating whether regulatory decisions made by USDOT with regard to 4(f) projects under the new law are appropriate; instead, its role was limited to providing (a) advice on the study design concerning the appropriateness of the methodology and data collection for evaluating impacts and (b) a review of the final report of the study to determine whether the report‘s findings and associated conclusions are justified by the data collected and the methods applied in interpreting the data. The pertinent technical questions for the committee involved items such as what data should be collected, how results should be measured, which parties should be interviewed, and how cases should be selected for consideration. This is the third letter report produced by the TRB committee that was established to carry out the independent review. The report reviews the draft Phase II report of the study to determine whether its findings and associated conclusions are justified by the data collected and methods applied in interpreting the data before the final Phase II report is released. PROCESS FOR THIS ACTIVITY As provided for in Section 6009 of SAFETEA-LU, USDOT is conducting the mandated study in two phases. Phase I examines how the de minimis provision has been applied since it was enacted in August 2005. It also describes the feasible and prudent avoidance alternative standard and reviews the process used to develop the final rule. The Phase I draft was completed in mid- December 2008. Phase II of the study focused on evaluating implementation of the feasible and prudent avoidance alternative standard and on updating and extending the evaluation of the de minimis impact provision. In accordance with USDOT‘s two-phase study, TRB is carrying out its independent review in three stages. In the first stage, the TRB committee reviewed USDOT‘s Phase I draft study plans and provided a letter report in June 2008 documenting the committee‘s evaluation. In the second stage, documented in the second letter report provided in April 2009, the TRB committee reviewed the draft Phase I study report and evaluated the draft study plan for Phase II. This third letter report reviews the draft Phase II report and determines whether the findings and conclusions are supported by evidence presented in the report. USDOT and its designated contractors at the Volpe National Transportation Systems Center shared the draft Phase II report with the committee and discussed it at the committee‘s meeting on June 16, 2011. In addition, representatives of the following relevant stakeholder organizations participated in the meeting and shared their views on the draft report: Advisory Council on Historic Preservation,

4 National Trust for Historic Preservation, Rails-to-Trails Conservancy, and American Association of State Highway and Transportation Officials. Other stakeholders were invited but were unable to attend the meeting (see Enclosure B). DRAFT PHASE II REPORT Phase II Evaluation Methodology and Survey Instrument The study team used ―a two-section survey to collect information on‖ (a) ―the de minimis impact provision and‖ (b) ―the revised feasible and prudent avoidance alternative standards.‖ ―The survey included close-ended, structured questions‖ that ―included multiple choice questions and statements asking respondents to specify their levels of agreement. . . .‖ ―Additionally, the survey provided respondents with the opportunity to provide qualitative information to describe the reason for their answers to the close-ended questions‖ (p. 8).3 The study team identified the population of interest as including representatives from four stakeholder groups: 1. Federal transportation agencies, 2. State and local transportation agencies and transportation authorities (project sponsors), 3. Officials who have jurisdiction over Section 4(f) properties, and 4. Citizen and advocacy groups. The study team included stakeholders in the survey ―who had been involved in (1) project(s) where a de minimis impact determination was made and construction of the portion of the project related to the Section 4(f) resource was at least 75 percent complete, and/or (2) project(s) that required a Section 4(f) evaluation (either draft or final) since April 11, 2008‖ (p. 8). The study team reports that the online survey was initially distributed only to federal transportation contacts with the hope that they would forward the survey to other staff. Eighty- five transportation individuals responded. The 85 ―Federal survey respondents were asked to identify appropriate points of contact for the three additional stakeholder groups for each project with which they were involved‖ (p. 9). According to the study team, the newly identified contacts were then sent surveys and asked to identify contacts for each project with which they were involved. In total, 140 individuals from nonfederal agencies were identified (69 project sponsors, 59 OWJs, and 12 citizen and advocacy groups). The draft report indicates that ―every individual identified within the population of interest‖ received an e-mail that ―included a web link to the online survey and a hard copy of the survey‖ (p. 9) and that each survey recipient was 3 Unless specified otherwise, page numbers refer to those of USDOT‘s draft May 2011 report, SAFETEA-LU Section 6009 Implementation Study, Draft Phase II Report.

5 given 3 weeks to complete the survey and was sent a reminder e-mail after 2 weeks. Incorrect or undeliverable e-mails were redirected to other individuals within the agency or organization. Individuals who did not complete the survey after the original due date were sent a third e-mail and given a 2-week extension to complete the survey. During this period, the study team also made phone calls to remind citizen and advocacy groups who had not completed the survey. Individuals who had not completed surveys after 5 weeks were considered nonrespondents. Self-Reported Data Quality and Research Limitations On pp. 10–11 of the draft Phase II study, the study team indicates several limitations: In some de minimis impact determinations, ―data are incomplete and/or are reported to varying degrees of accuracy and detail.‖ The potential population of interest was affected by the nonresponse of federal transportation contacts [e.g., four of 52 Federal Highway Administration (FHWA) division offices 4 and three of 10 Federal Transit Administration (FTA) regional offices], since potential project sponsors, OWJs, and citizen and advocacy groups in those states or regions were not included in the survey. Response rates from OWJs and citizen and advocacy groups were low (15 of 59 and two of 12, respectively; see Table 5, p. 19, which is reproduced at the end of this section). Respondents who indicated no experience with either de minimis projects or 4(f) evaluations were excluded from that part of the survey. However, respondents indicating experience with either de minimis projects or 4(f) evaluations, but whose project was later determined not to involve a de minimis determination or updated feasible and prudent standards, were still included in the survey results. A limited number of projects have completed a final Section 4(f) evaluation using the revised feasible and prudent standards; results cannot be generalized to the overall population. Some Phase II survey questions concerning efficiencies and effectiveness associated with the de minimis impact provision differed from the questions in Phase I, thereby limiting comparisons across the two surveys. Inventories According to data collected from USDOT, FHWA, FTA, and Federal Railroad Administration (FRA) offices, between 2005 and the end of 2010, ―Section 4(f) de minimis impact determinations had been made for 822 projects‖ (p. 12). Of the 822 projects, the study team reports that 635 had a single de minimis impact determination, 109 projects had two de minimis impact determinations, and 78 projects had more than two determinations (see Figure 2, p. 13). Most determinations involve historic sites only (57 percent). Twenty-five percent of the determinations are for parks only, 9 percent are for recreation only, and 2 percent are for wildlife refuges only. The remaining impact determinations (6 percent) involve some combination of Section 4(f) resources. 4 FHWA has a division office in each state. According to Figure 1 of the draft Phase II report (p. 12), three states— Arizona, Louisiana, and Nevada—and Puerto Rico had no de minimis impact determinations, and two states— Missouri and North Dakota—had only one.

6 According to the study team, a database with all individual Section 4(f) evaluations reviewed by the U.S. Department of the Interior (DOI) was provided to USDOT and contained data from April 2008, when the feasible and prudent standard took effect, to December 2010. During this period, DOI received 198 individual Section 4(f) evaluations: 159 highway projects, 38 transit projects, and one joint FHWA–FTA project (see pp. 16–17). Survey Response Rates The table below is duplicated from the draft Phase II report (p. 19) and provides survey response rates by stakeholder group.

7 SUMMARY RESULTS OF DRAFT PHASE II STUDY REPORT With respect to the findings about efficiencies relating to de minimis impact provisions, the draft Phase II report Executive Summary (see pp. 1–2) states the following: U.S. DOT concludes the following: • The Phase II survey results support the efficiency findings reported in Phase I. The majority of transportation officials (80 percent of Federal transportation agency respondents and 86 percent of project sponsor respondents) responding to the Phase II survey share the perception that the de minimis impact provision has improved the timeliness for completing transportation projects. Several transportation officials reported that the process associated with making a de minimis impact determination often generates similar outcomes as would be reached with alternative processing options; yet, the de minimis impact provision often provides for a more streamlined process for achieving those outcomes. Similarly, the majority of transportation officials (56 percent of Federal transportation agency respondents and 62 percent of project sponsor respondents) reported that the de minimis impact provision has reduced the costs associated with completing a project. The majority of officials with jurisdiction did not have information on which to base an answer on the time and cost efficiencies (50 percent and 80 percent, respectively) associated with the de minimis impact provision. The one citizen/advocacy group responding to the survey reported that the de minimis impact provision does not result in time or cost savings. • The majority of transportation professionals (67 percent of Federal transportation agency respondents and 56 percent of project sponsor respondents) responding to the Phase II survey reported that the de minimis impact provision at least maintains the protection of Section 4(f) resources as compared to other Section 4(f) processing options. Overall, the officials with jurisdiction perceive that the de minimis impact provision at least maintains the protection of Section 4(f) resources as compared to other Section 4(f) processing options. However, given the small number of officials with jurisdiction responding to the survey, it is not possible to generalize the feedback collected from these individuals to the larger population. The draft Phase II report Executive Summary (see p. 2) states the following with regard to the feasible and prudent avoidance alternative standards: • The Phase II survey results suggest that the new feasible and prudent standard protects Section 4(f) resources at least to the same degree they were protected before the feasible and prudent avoidance alternative definition was revised. Generally, Federal transportation agency and project sponsor respondents agreed that the new standard was helpful in articulating what constitutes a feasible and prudent alternative but that it had not had any substantial effect, positive or negative, on the Section 4(f) resources. • A third of Federal transportation agency and project sponsor respondents indicated that the new feasible and prudent standard had no effect on the post-construction effectiveness of impact mitigation and avoidance commitments. Half of the officials with jurisdiction respondents indicated that the effectiveness of post-construction impact mitigation and avoidance commitments had decreased. However, it is not possible to generalize the

8 feedback collected from these individuals to the larger population due to the small number of officials with jurisdiction that responded to the question. • Ten assessment criteria intended to help clarify the determination of whether an avoidance alternative is feasible and prudent were developed as a part of the Final Rule. According to survey results, most respondents from all stakeholder groups believed that each assessment criterion had clarified the determination of whether an avoidance alternative is feasible and prudent. • The Final Rule includes a ―least overall harm‖ determination that balances seven factors, which are to be used when all alternatives result in the use of Section 4(f) property and there is no feasible and prudent alternative that avoids a Section 4(f) use. Survey respondents from all stakeholder groups generally agreed that the opportunity to weigh the factors has had a very positive effect on determining which alternative has the least overall harm. OVERALL FINDINGS OF THE COMMITTEE The committee was charged with reviewing the draft final report and determining whether its findings and associated conclusions are justified by the data collected and methods applied in interpreting the data. Before the committee can address this question, it is important to consider whether the findings and conclusions presented in the draft final report address the objectives of the study as outlined in Section 6009 of SAFETEA-LU. This section presents the committee‘s overall determination as to whether the findings and conclusions address these legislative objectives. The following subsections indicate concerns relating to the study‘s shortcomings and offer strategies for addressing the shortcomings. The committee finds that the objectives outlined in SAFETEA-LU Section 6009(c)(2) have not been adequately met by the USDOT study as presented in the draft final report: Section 6009(c)(2)(A): o Requirement: ―[The Secretary shall evaluate] the processes developed under this section and the amendments made by this section and the efficiencies that may result. . . .‖ o Summary of committee findings: The committee finds that this requirement has not been fully addressed by the study, either with respect to the de minimis impact provision or the feasible and prudent avoidance alternative standards. Study design and implementation choices, especially in terms of a narrow definition of ―efficiency,‖ have produced a limited representation of perspectives, mainly those of federal and state transportation officials. Although the study has provided information about ―process efficiencies‖ and has partially addressed study objectives, the committee believes that these efforts are incomplete and that the study should take a broader view of ―efficiency‖ that looks beyond process and considers outcomes.

9 Section 6009(c)(2)(B): o Requirement: ―[The Secretary shall evaluate] the post-construction effectiveness of impact mitigation and avoidance commitments adopted as part of projects conducted under this section and the amendments made by this section. . . .‖ o Summary of committee findings: The committee finds that the study does not substantively address the postconstruction effectiveness of impact mitigation alternatives, either with respect to the de minimis provisions or the feasible and prudent avoidance alternative standards. The study relies on an incomplete conceptual framework for evaluating these outcomes and inadequately represents the perspectives of stakeholders in 4(f) resources. Section 6009(c)(2)(C): o Requirement: ―[The Secretary shall evaluate] the quantity of projects with impacts that are considered de minimis under this section and the amendments made by this section, including information on the location, size, and cost of the projects. . . .‖ o Summary of committee findings: The committee finds that the study has addressed this requirement of SAFETEA-LU. While the committee finds that the study has provided an inventory of projects with impacts that are considered de minimis and has produced information to support narrow conclusions concerning the cost and time savings associated with the SAFETEA-LU amendments, the findings and conclusions presented in the draft final report do not adequately address the legislative objectives of the study. It is unclear what value can be derived from the committee‘s review of whether these limited findings and conclusions are supported by the data collected and methods applied. However, the committee offers its review in the spirit of providing constructive feedback to help USDOT gain as much insight as possible from existing information and identify the steps necessary for finalizing the study. The next section presents the committee‘s findings with regard to the study methodology as applied. Methodology The committee recognizes that the study team faced significant challenges in the implementation of the study. Among them were (a) identifying participants to be included in the study, other than federal transportation officials, who had adequate firsthand knowledge of specific 4(f) projects and (b) characterizing the postconstruction effectiveness of mitigation and avoidance commitments given the relatively small universe of projects that have involved the amended provisions and that have been completed (or are near completion). However, the committee believes that information adequate for addressing the objectives of the study was available. As of December 2010, more than 1,400 determinations of de minimis impact had been made in relation to 822 transportation projects in 47 states. In addition, 198 individual Section 4(f) evaluations tracked by DOI had been completed in relation to 194 projects in 39 states between April 2008 and December 2010. By definition, these determinations and evaluations involved or affected hundreds of OWJs, including most state historic preservation officers (SHPOs), and thousands of users of the affected 4(f) resources. Nonetheless, the final report attempts to draw conclusions on the basis of input from only 10 OWJs and one representative of a citizen advocacy group.

10 The committee has emphasized from the start of the review process in 2007 that perspectives of nontransportation officials are necessary in addressing the objectives of the study. The underrepresentation of these perspectives undermines the study‘s validity. The committee acknowledges the difficulties associated with obtaining information from local OWJs and users of 4(f) resources but believes that the onus was on the study team to design the study to engage nontransportation officials‘ perspectives and, when difficulties arose, to adapt the design to address the study objectives. The committee believes that alternative approaches could have been used. We acknowledge the challenges associated with midcourse corrections to an evaluation, including challenges associated with budget, time, and compliance with the Paperwork Reduction Act. However, the committee believes that the study team placed unnecessary constraints on itself when it was faced with the need to adapt the study in light of difficulties encountered while attempting to obtain broader perspectives. Greater attention to the formulation of an evaluation framework that operationalized key concepts in terms of the questions posed by SAFETEA-LU and that recognized data limitations and opportunities would have provided a more solid foundation for adapting the evaluation to the challenges encountered. Absent such attention, the study methodology, as currently implemented, raises several important issues, which are outlined below. The study does not appear to have been informed by the committee‘s first letter report. That report recommended use of a coherent evaluation design built on a theory of change, operationalized in terms of hypotheses related to the study objectives, and based on a weighing of alternative approaches. 5 The committee believes that during the implementation of the Phase II study, the study team should have reacted to the low contact and response rates by revisiting the original design foundation and identifying feasible alternatives for hypothesis testing. This practice is common in program evaluation and can be carried out effectively if it is built on a coherent framework. While the committee supports the study team‘s decision to begin with interviews and develop a survey instrument and sampling plan on the basis of the interview findings, a completed survey will often leave some questions unanswered, with a need for follow-up to help interpret the results. The study team conducted the survey but did not use some of the other qualitative methods identified in the draft Phase II study plan, on the basis of the committee‘s suggestion that such methods would be of limited value. The committee believes that the team misinterpreted its advice. In fact, methods such as follow-up interviews and focus groups could have been used to delve deeper into and better understand the survey findings. The study does not set forth clear hypotheses linking the amendments introduced in Section 6009(c) of SAFETEA-LU to the efficiency and postconstruction effectiveness outcomes that 5 In the context of this sentence, a ―theory of change‖ is a theory describing how a policy intervention (such as the SAFETEA-LU amendments) is expected to affect outcomes [such as the quality of a transportation project or the attributes of a 4(f) resource]. In evaluation practice, the theory of change is usually expressed in terms of a series of linked direct, intermediate, and long-term (or end) outcomes. ―Operationalized‖—in the context of this sentence—is the process of expressing the conceptual theory of change in terms of testable hypotheses (and, by association, measurable variables).

11 are the subject of the study‘s evaluation. The report should make the hypotheses behind the study‘s evaluation explicit and explain how the de minimis provisions and the feasible and prudent standards are expected to affect efficiencies and postconstruction effectiveness. Without this link, it is difficult to understand how the survey data address the objectives of the study, to what extent the objectives of the study have been met, whether the methodology was appropriate, and, most important, whether the conclusions are supported by the data. The study relies on an incomplete definition of the concept of ―efficiency.‖ In its first letter report, the committee suggested that the study team adopt an efficiency evaluation approach that considers not only cost and time but also impact of the amendments in terms of 4(f) resource outcomes. The study team chose to define efficiency solely in terms of cost and time savings based on stakeholder input, which disproportionately represented the process orientation of transportation officials. Such an approach might have been valid had there been sufficient evidence indicating that the quality of the actual outcomes with regard to 4(f) resources has not and will not vary under the new provisions and standards; however, the Phase I data suggested that this has not been the case (e.g., see Figure 7 of the Phase I report, p. 24). Because the study team chose to use a definition of efficiency (―process efficiency‖) that is incomplete, the study fails to recognize and communicate the extent to which the lack of diversity in perspectives undermines its validity in addressing SAFETEA-LU Section 6009(c)(2)(A). There are issues associated with inconsistent units of analysis. The study team selected the ―person‖ as the unit of analysis rather than the ―project,‖ and this choice had several implications for the study. These implications are discussed below. 1. Response frame: The response frame of an individual participant in the study refers to the capacity of that individual to provide useful and valid information, where capacity is a function of experience and knowledge of factual information. Response frames of transportation officials and local OWJs or users will be different, and the implications of the choice of unit of analysis will differ for each group of respondents. a. For transportation officials, a project-specific design would have encouraged responses based on their experiences and on the facts surrounding a specific project or case [e.g., Was it a categorical exclusion? Was it uncontroversial? Could it have been done under a programmatic 4(f)?]. Project-focused questions would have encouraged respondents to rely on factual information in speculating about the counterfactual (―what-if‖) outcome. A person-based approach does not encourage this type of response and may actually discourage it. For example, if a respondent has worked on several projects where the facts suggest conflicting outcomes—less time versus no change in time—the transportation official might find it easier to ignore the facts about some cases and generalize. b. Local stakeholders most likely have a limited response frame and are unlikely to be in a position to address broad process questions. A project-focused design would have aligned much better with this response frame, in that it would have asked local stakeholders to reflect on their knowledge of a specific resource and the effects of a specific transportation project on that resource, rather than on more abstract concepts associated with process improvements.

12 2. Survey design choices: If the unit of analysis had been clearly defined as the project, the survey questions might have looked much different. a. Project-oriented questions would have asked transportation officials to draw on their experience and would have encouraged follow-up questions that could have drawn on how the specific circumstances of the project led to the observed outcomes. b. A project-oriented approach would have made the requirement that different stakeholders be asked different survey questions more obvious. 3. Unclear or inconsistent units of analysis can lead to issues in interpreting the data. a. Using the person as the unit of analysis can aggregate information associated with projects and can lead to problems in interpreting the data. (1) Respondents have different experiences and will use different ―internal decision-making processes‖ to reflect on these experiences across multiple projects. Understanding or controlling for these multiple internal processes is difficult. (2) When respondents are asked to reflect on their experiences across multiple projects, all of the variation among the projects will be lost in terms of circumstances and outcomes. (3) In addition, when responses are tracked by respondent and not by project, multiple respondents may provide information on a single project, creating a chance of natural hierarchies 6 in the data and leading to statistically invalid conclusions if not controlled. Given that the online survey was distributed to federal transportation contacts, who then forwarded it to other staff and stakeholder groups with whom they worked, such a distribution pattern would only heighten the possibility of overlap based on shared projects and networking patterns. b. The ability to conduct hypothesis testing and multivariate analyses consistent with the study objectives was lost when the study team operationalized the study in terms of the person and not in terms of the project or resource. The questions posed by the legislative objectives for the study focus on outcomes occurring at the project or resource level (i.e., time saved, cost saved, impacts on resources, etc.). The hypothesis being tested is that the new provisions and standards affected outcomes associated with specific projects and resources, where the dependent variable is the impact on project outcomes or resources, the independent variable is process change, and other project attributes describing the context (e.g., type of resource, size, and types of project) are control variables. Data were not collected at the resolution consistent with this framework. 4. Although the choice of person as the unit of analysis has critical implications, as discussed above, trade-offs do exist, and the researcher might explicitly choose this approach in lieu of a project-based approach. A ―person-based‖ approach may provide the researcher two benefits. a. An ability to analyze response frame asymmetries. Because different types of respondents may answer questions differently, responses may not reflect actual 6 In general, a natural hierarchy is an ordered sequence of variables used for analysis. The hierarchy is composed of variables, and each variable has a member property of the variable below it (for example, day > month > quarter > year is a natural hierarchy).

13 differences in the outcomes that they are describing. Data at the person level could be analyzed to evaluate this possibility, because response frame is an attribute of the variable being measured—the person. However, this type of analysis can be done with project as the unit of analysis by including respondent attributes in the survey to control for these differences. b. An ability to account for natural hierarchies in respondents‘ perspectives. As described above, respondents with experience across multiple projects may aggregate information (perspectives) about all projects into a single response. For example, their experience may suggest that de minimis provisions usually save time, and they may provide this response for each question about each project despite the fact that their experience may vary from project to project. This can create natural hierarchies in the data. Controlling for these hierarchies after the fact is not easy; the researcher can prevent them by avoiding general questions about a respondent‘s perceptions and by focusing the questions on the facts concerning each specific project. The process for identifying survey respondents introduces potential bias. As the study team explained in the draft Phase II report, the survey was sent initially only to potential federal transportation respondents. Respondents were asked to identify contacts for the three other stakeholder groups linked to each project with which they were involved. Such a screening process could invite sources of selection bias, since transportation officials could inadvertently (or advertently) identify some, but not all, points of contact for the other three stakeholder groups. In addition, the screening criteria were used inconsistently, introducing further bias. Federal respondents were chosen on the basis of both their specific project knowledge and their general knowledge of the subject matter, while nontransportation respondents were selected only on their knowledge of a specific project. The committee acknowledges the difficulties associated with obtaining information from stakeholders in this study. However, the draft report‘s underrepresentation of nontransportation officials and users of 4(f) resources undermines its findings and validity. In both of the committee‘s previous letter reports, an emphasis was placed on engaging these groups and seeking their perspectives for an outcome evaluation. These nontransportation perspectives are necessary for the evaluation of both efficiencies and postconstruction effectiveness, especially the latter because transportation officials may be less able than nontransportation officials to evaluate the ―post-construction effectiveness of impact mitigation and avoidance commitments.‖ The study also suffers from an inadequate sample size of nontransportation officials. The sample size issue is especially noticeable for project sponsors and for nontransportation stakeholders, such as OWJs and citizen and advocacy groups. Of these three stakeholder groups, 26 of 69 potential project sponsors, 15 of 59 potential OWJs, and two of 12 potential citizen or advocacy groups responded to the survey (see Table 5 of draft Phase II report, p. 19). Although the study team acknowledges that a small number of these groups participated in the survey (only 43 of 140), the lack of representation of and responses from these groups limits the inferences that can be drawn. Given the small sample size for nontransportation stakeholders, the committee recommends that the study team seek perspectives from the underrepresented

14 stakeholder groups so that the views from others in these communities are reflected in the conclusions. The team could use this opportunity to collect information from tribal historic preservation officers. Questions asked of nontransportation officials should be focused on outcomes (rather than process) and should be designed on the basis of an understanding of the different respondents‘ response frames.7 Because the sample frame of federal transportation officials was not defined and the sampling method was nonrandom, the study‘s approach raises questions as to how representative of the population these perspectives are. 8 An unknown proportion of the entire population was sampled, so the study team is limited in the inferences it can draw with regard to the population of federal transportation officials. If the number sampled represents a sufficiently large share of the total population, random sampling is less important, but the study provides no information about the population of federal transportation officials. The committee believes that while some overlap in survey questions is useful, different survey questions should have been asked of different stakeholder groups. This approach was recommended in the committee‘s second letter report dated April 2009 (see pp. 13–14). 1. Survey questions asking about the Section 4(f) process, such as those dealing with time and cost, are more likely to be better understood by transportation officials. OWJs and citizen and advocacy groups are more concerned with resource protection than with time and cost efficiencies, so questions to these groups could have been formulated in terms of the impacts on resources associated with specific transportation projects. 2. OWJs understand Section 106 of the National Historic Preservation Act of 1966 but may have less experience with the Section 4(f) process, particularly SHPOs who do not view Section 4(f) as their process. SHPOs would not have understood some of the questions, and the phrasing of the questions may not have elicited the responses that the study team was seeking. Some of the Phase II survey questions were worded in a way that led to difficulties in interpreting responses, as acknowledged by the study team. For example, Question 3 (Appendix A-4) asked respondents to rate whether ―the ‗activities, features, and attributes‘ of the Section 4(f) resource changed as a result of the transportation project.‖ The feedback from survey respondents indicated that a positive response to this question (i.e., a scale value of 4 or 5) could reflect either a positive or a negative change. In addition, Questions 4 and 5 (Appendix A-5) of the survey did not allow for the possibility of increased cost or time, respectively, as a result of the de minimis provisions. Phase I results suggested that these possibilities exist, at least in terms of increased time to comply (see Figure 4, p. 17 of the Phase I Report, December 2008). More background information about the questions may have helped nontransportation stakeholders understand the questions and their importance. Finally, the addition of Section 6009(a) in 2005, and the possibility of a de minimis determination, has changed the Section 4(f) evaluation process, but the changed process is not examined. Section 6009(a) allows a de minimis designation but also requires concurrence from the OWJ. How does the need for OWJ concurrence influence efficiencies 7 See p. 11 of the current letter report for a more detailed discussion of respondents‘ response frames. 8 The committee notes that true random sampling of these groups would be difficult to achieve.

15 and postconstruction effectiveness? A possible assessment avenue would be to explore the role and influence that de minimis sign-off gives OWJs. As detailed above, the committee has raised several issues associated with the study methodology of the draft Phase II report. The shortcomings include (a) failure to make the hypotheses behind the evaluation explicit, (b) failure to operationalize key concepts such as ―efficiency‖ and ―postconstruction effectiveness,‖ (c) choice of units of analysis inconsistent with the objectives of the study, (d) potential selection bias of the population of interest, (e) underrepresentation and small sample size of key stakeholder groups, and (f) nonrandom sampling of federal transportation officials. Given these shortcomings, the committee believes that the study team should revisit the study and collect additional information with valid methods. Analysis The analysis in the draft report is plagued by methodological issues that limit the extent to which the analysis can be improved. However, the committee wants to discuss several issues further: (a) an analysis of Phase I and Phase II data to explain negative responses; (b) attempts to cross- tabulate data, though this is limited by the unit of analysis; (c) categorical analysis; and (d) respondent demographics. These issues are discussed below. The committee believes that the analysis in the final report should integrate information collected during Phase I as well as during Phase II. Phase I provided information in terms of possible determinants of variation in the outcomes, and drawing on this information to present hypotheses and to explain the results when the Phase II data are reviewed would be useful. For example, this approach could be helpful in exploring negative responses. In addition, by conducting such an integrated analysis, the study team may be able to identify gaps in understanding the survey results. Such a suggestion (to use all of the information collected to formulate findings) is a basic standard of practice. While the study team may have done this, it is not evident from the current draft report. As an example, the Phase II survey asked respondents to approximate the time savings associated with a de minimis impact determination (Appendix A-5, Question 4), and the results are presented in Figure 8 (p. 21) of the Phase II report. The Phase I report more systematically evaluated factors associated with the de minimis provisions resulting in reduced or increased time. By integrating these findings from the Phase I report into the discussion of the Phase II findings, the current draft report would provide the study team a much more comprehensive and balanced picture. The study team‘s selection of the person as the unit of analysis, as well as the lack of variation in the officials about whom data were collected, limits what might be extracted from more sophisticated statistical analysis. However, the team may be able to find some proxies in the data. For example, the study team could associate responses from those who deal only with historic properties with a ―historic‖ category and responses from those who deal with parks and refuges with an ―environmental‖ category. The team could then cross-tabulate and attempt categorical analyses of the outcomes to explore the idea of variation in outcomes depending on context. Unfortunately, this type of analysis would still suffer from many of the issues that the committee has identified elsewhere—issues such as a lack of balance or variation in perspective and aggregated data.

16 The total cost for highway projects considered for the study ranges from $27,000 to $2,560,000,000, while the de minimis use size ranges from 0.0001 acres to 51 acres for highway projects (see Table 2, p. 15). Such a wide range in the data set can distort measures of central tendency. To address such potential distortions, the team could combine responses by size of project [possibly by using ranges for dollar amounts and for acreage (e.g., ranges such as small, medium, and large and the number of de minimis projects associated with each)]. More important, the wide range suggests that the outcomes, both in terms of streamlining and resources, should have been explored relative to this variation. The committee believes that this was not possible because the study team disregarded earlier comments concerning the unit of analysis. The committee recommends that the study team provide more demographic and geographic data on both transportation and nontransportation respondents. As the data are currently presented, the reader is given little guidance in judging their representativeness. A section in the report discussing the limitations and uncertainties in the quantitative analysis of the data or explaining the representativeness might be helpful. For example, the report should explain why there are no responses from groups such as the Federal Aviation Administration and the tribal historic preservation offices. The report does not state whether these groups declined to participate or were excluded from the study. The committee believes that the study team should address such concerns directly so that the reader is not left guessing. Although the current draft is hampered by methodological problems, the committee believes that the final report could benefit from the types of additional analysis outlined above. Interpretation The committee recommends that the study team use objective language in presenting and interpreting data in the draft report. The draft report does not fully connect the presented data with the objectives of the evaluation (as outlined in SAFETEA-LU). In addition, the study team should discuss threats to the report‘s validity and their implications for the study‘s conclusions. The findings may have a bias against respondents with negative statements. Negative statements tend to have caveats (see conclusions on p. 33). The study team appears to downplay negative statements, yet positive statements tend to be grouped and do not receive the same caveats (see paragraph before Figure 9 on p. 22). In addition, more rigor should be exercised with regard to which qualitative comments from the survey are included in the text. By providing a more detailed reporting of qualitative comments, the authors would allow the reader to make decisions about their validity. Negative responses may need more analysis to explain what is occurring. For example, Figure 11 shows user judgment concerning whether the Section 4(f) resources have been or will be maintained. Fourteen percent of transportation officials do not agree that a de minimis determination is protective. This negative response deserves exploration and explanation. Even if the study team has confidence in the results from the transportation community with regard to how the process is working, at the very least the team should go back to people who said the process is not working and ask why.

17 OWJs may not understand (or be familiar with) the new changes to Section 4(f). However, the study team should not conclude that the response rate for OWJs is low because OWJs did not have an opinion or did not believe the problem worthy of investing the time to respond (see p. 34). 9 The methodological issues outlined above, particularly those related to recognizing the response frame of survey respondents and designing questions to elicit valid data based on those response frames, provide a more immediate explanation for low response rates and the frequency of ―no opinion‖ or ―unknown‖ responses. The data for project sponsors, OWJs, and citizen and advocacy groups—in terms of sample size and diversity of respondents—are too limited to permit inferences about resource protection and postconstruction effectiveness. There may be justifiable reasons why further assessment activities were not possible (such as budget and time restrictions), but the authors should be more cautious about data-based conclusions. Because the sample size is too small, the study team should not draw inferences about OWJs (pp. 33–34) unless it expands the survey, as recommended above. Figures 14 and 15 on the updated feasible and prudent standard are important in another way. An examination of the percentages indicates that the differences in the answers are considerable; however, the numbers of responding project sponsors and OWJs are small. Two of four OWJs (50 percent) believe that the feasible and prudent standard has decreased protection of the Section 4(f) resource, compared with one of 14 project sponsors (7 percent) and no federal transportation officials. Because of the smallness of the sample, the study team should use care in presenting the results; as currently portrayed, the results can be misleading. As discussed above, because of the limited diversity and sample size for some types of respondents, the study team should be much more circumspect about what conclusions can be drawn from the data, especially with regard to resource protection and postconstruction effectiveness. An incomplete definition of efficiency and an underrepresentation of nontransportation officials undermine the draft report‘s validity and findings, and the committee recommends that the study team acknowledge these shortcomings and discuss efficiencies from a federal transportation perspective only. Given that the OWJs are less familiar with the 4(f) determination process, the study team could retain the discussion of the OWJ responses in its conclusions but only if it includes an overt acknowledgment of the data limitations and the inability to draw inferences about process impacts and the protection of resource outcomes from these isolated responses (see p. 34). The study team should be cautious in its presentation of percentages based on a small number of respondents, because the results as currently portrayed can be misleading. The study team should use more objective language in presenting qualitative comments, both positive and negative. More important, the committee questions whether the findings and conclusions as presented in the draft final report adequately address the objectives outlined in SAFETEA-LU Section 6009(c)(2). 9 Because the results of the draft study report suggest that some stakeholders may not understand or be familiar with the new changes to Section 4(f), the committee recognizes the need for more and better training among stakeholders. Anticipating this need, FHWA is updating the content of the courses that cover Section 4(f) regulations and the content of its Section 4(f) Policy Paper.

18 RECOMMENDATIONS AND SUGGESTIONS Draft Phase II Study On the basis of the limitations of the draft Phase II study described above, the study team should revisit the study and collect additional information with valid methods in order to meet the objectives outlined in SAFETEA-LU. The team should expand the population of survey respondents to include OWJs and citizen groups and consider refinements to the survey questions that are designed to elicit valid data from different types of respondents. The committee believes that the current interpretation of outcomes of de minimis designations is incomplete without the response of these groups. The team should consider additional qualitative activities to help in understanding and interpreting the survey results and in addressing the study objectives. Should the study team choose not to follow the committee‘s recommendation, but instead to finalize the current draft report on the basis of data already collected, the committee does not believe it would fully satisfy the requirements of Congress in calling for a study. However, the committee believes that the current draft report would provide insights, as perceived by transportation officials, about increased efficiency in the 4(f) regulatory process. If the study team does choose to pursue this approach, the committee suggests that the study team, at a minimum, take the actions listed below. 1. Provide more demographic and geographic data on both transportation and nontransportation respondents. 2. Separate responses from transportation and nontransportation respondents. 3. Make the hypotheses behind the evaluation explicit. 4. Integrate the findings from the Phase I study into the interpretation of Phase II results. 5. Discuss efficiencies only from a federal transportation perspective. 6. Retain the discussion of the OWJ responses in the conclusions, but include an acknowledgment of the data limitations. 7. Be much more circumspect about inferences concerning project outcomes. Even if the steps recommended above for a more balanced reporting of survey responses by transportation officials are taken, the lack of random selection will remain a threat to the validity of the results. The report should be forthright in acknowledging this limitation. Suggestion for Future Research During the review of the draft Phase II report, the committee uncovered a related issue that is not directly within the committee‘s task statement. Several discussions emerged about how Section 106 and Section 4(f) interact: the former is a broader project assessment, while the latter is a property-by-property assessment. The de minimis provision currently requires transportation officials to apply the criteria of adverse effect of a project to each property, whereas SHPOs are accustomed to evaluating whether an entire project has an adverse effect on any historic property. The two sets of regulations apparently do not work well in tandem—one or both would benefit from being revised to work in coordination. Research could explore whether and

19 why these processes do not work well together. Research could also identify ways of enhancing coordination of the two processes. In this letter report, the committee has offered constructive criticism of the Phase II draft, which it hopes is helpful to USDOT in finalizing its report to Congress. If we may be of further assistance, please do not hesitate to ask. Sincerely, Michael D. Meyer, Committee Chair and Professor of Civil and Environmental Engineering, Georgia Institute of Technology