14
Frameworks for Balancing and Improved Integration

The New York City Department of Environmental Protection (NYC DEP) asked the National Academies to “review and evaluate the NYC DEP’s watershed protection program—with the goal of determining whether the current suite of individual programs is appropriate and adequate to comply with the Surface Water Treatment Rule into the future.” In essence, the goal of this study was to conduct an overall program evaluation and assess the appropriateness of the portfolio of watershed protection programs. This chapter presents a framework for systematically evaluating and balancing New York City’s (NYC’s) multi-faceted Watershed Protection Program and its diverse component subprograms, which support a larger multi-barrier approach to providing safe drinking water for New York City. In doing so, the Committee recommends programs that merit more or less emphasis, both as they exist today and with the improvements detailed in earlier chapters. This is followed by a discussion on coordination between programs. Although each Watershed Protection Program component makes unique contributions, links and comparisons across subprograms can provide efficiencies and help increase program and component effectiveness.

In previous chapters, the individual components of the Watershed Protection Program (listed in Table 1-1) were assessed, including strengths and weakness of each subprogram and their contributions to meeting water quality and community vitality goals. However, those chapters are not a comprehensive, overall evaluation of the Watershed Protection Program because much of the data and information needed for more formal evaluation are currently unavailable or inconsistent across programs. This chapter provides an overall qualitative assessment of balance among the subprograms and makes recommendations on how NYC DEP can better support ongoing program evaluations and comparisons for the components of the Watershed Protection Program, beginning with a general overview of program evaluation.

Although this is the final chapter, it is not a conclusions chapter (which is provided in the Summary), but rather builds on the findings of earlier chapters.

PROGRAM EVALUATION

In writing Chapters 5 through 12, which assess the strengths and weaknesses of the subprograms within the Watershed Protection Program, it was apparent that although NYC DEP tracks implementation, it does not necessarily track performance towards water quality objectives or conduct systematic program evaluation. Program evaluation (hereafter, evaluation) involves collection and analysis of information to assess the design, implementation, and impacts of programs in terms of predefined explicit objectives and standards (Rossi et al., 2004). Evaluation can help in improving the contributions of each component within the Watershed Protection Program. Evaluation also can aid in balancing investments across the various components of the Watershed Protection Program. The text below outlines major steps in evaluation, highlighting key aspects that could be strengthened within NYC DEP’s existing analyses.

Major Steps in Program Evaluation

Step 1 – Identify Program and Subprogram Goals. NYC’s Watershed Protection Program has two overarching goals outlined in the 1997 Memorandum of Agreement (MOA): protecting drinking water quality and supporting community vitality. Within each of these overarching goals, the Watershed Protection Program has more specific goals. For example, in protecting drinking water quality, NYC DEP is interested in reducing and preventing loads of turbidity, nutrients, and pathogens to its water supply reservoirs. Systematic program evaluation for each subprogram in the Watershed Protection Program will require articulating specific goals for each subprogram. While articulation of these goals is currently straightforward for some subprograms (e.g., septic system repairs and Nutrient Management Plans), for other subprograms (e.g., land acquisition) specific goals are not necessarily related to the MOA goals. For most subprograms, goals for community vitality are not well defined.

Step 2 – Define Program Evaluation Criteria. Evaluation criteria are a means for assessing performance in achieving goals. Criteria could include, for example, (1) effectiveness—that is, how well a program achieves program goals; (2) efficiency—that is, the resources or costs expended to achieve performance; (3) equity—that is, how program costs and benefits are distributed; and (4) sustainability—that is, the probability that desired program results will continue into the future, which often is related to the certainty of a program being effective. Cost is a central criterion—fitting in to (2) above—as NYC DEP seeks to implement programs that achieve its goals at the least cost to its customers. As such, NYC DEP may want to consider the incremental improvements in performance with cost. Programs often have diminishing incremental returns, especially when implementation nears saturation. Better articulation of evaluation criteria and how NYC DEP will prioritize across criteria will help NYC DEP make decisions about balancing resources among subprograms.

Step 3 - Identify Causal Paths for Program Success. Understanding how a program is expected to achieve objectives can help in diagnosing causes of success and failure, and in improving programs (Ferraro, 2009). Commonly, a causal conceptual model or sequences of events are used to explain how program actions are expected to achieve desired objectives. While broad conceptual causal models exist for each subprogram within the Watershed Protection Program, and some subprograms have very clear causal models (e.g., the waterfowl and scat clean up around Kensico and Hillview reservoirs), detailed causal or mechanistic models for each subprogram that consider potential goals and side effects will provide NYC DEP with a clearer understanding of points of leverage within the system and potential unintended effects.

Various methods can be used to develop detailed causal models (Mayne, 2015; McLaughlin and Jordan, 2004; Rogers, 2008). Most important is to disentangle, stepwise, expected relationships between program implementation and achieving objectives and, at each step along the causal path, to assess conditions needed for success, and recognize underlying assumptions of causality and effects. Depicting causal paths also facilitates assessing where and how to monitor and evaluate performance.

For example, an objective of the Watershed Agricultural Program might be to reduce average annual loads of phosphorus to a reservoir. To do so, the program may work with farmers to develop and implement nutrient management plans. A rough diagram of the causal path for how this part of the program reduces nutrient inputs appears in Figure 14-1. This causal path identifies assumptions embedded in this part of the program (summarized in Table 14-1). In evaluation, these assumptions can each be explored to test their validity or implications if the assumptions do not hold.

Step 4- Collect and Analyze Monitoring Data and Information. NYC DEP collects extensive amounts of data in the watershed. This information is fundamental for program evaluation, yet, as described in Chapters 5 through 12, additional information and much more data analysis are needed to better evaluate most programs and subprograms.

Evaluation requires monitoring and analyzing data across the causal paths from inputs to impacts. Human, financial, and other resources for a program produce outputs – i.e., the specific activities or products of

the program. Outcomes are intended or achieved short- or medium-term effects or changes resulting from outputs. Impacts include broader program effects over the long-term, such as achievement of the primary objectives of the program, any spillover effects, and any transfers of welfare from one group to another (both intended and unintended) (Cameron, 2011). Table 14-2 gives an example of these four types of metrics (inputs, outputs, outcomes, and impacts) across the causal path for two subprograms: the Watershed Agricultural Program and the Stream Management Program.

Although NYC DEP tracks inputs and outputs fairly well, outcomes and impacts are not well tracked for many subprograms. For example, while the number of acres of land purchased, septic systems replaced, or best management practices (BMPs) implemented are tracked, NYC DEP does not assess or estimate how each subprogram contributes to reducing pollutant loads or broader water quality objectives. Tracking of outcomes and impacts may be limited by time, human, and financial resources as well as technical challenges (particularly for preventive programs—see Table 1-1); nevertheless, strategic improvements could be made.

Deterministic or mechanistic modeling, statistical analysis, GIS, aerial and satellite imagery, and remote sensing can help identify current patterns and assess potential contaminant sources and paths. Additional tools for evaluating outcomes can include use of leading indicators;¹ use of past results as a proxy for future results; analytical assessment based on conceptual, statistical, or mechanistic models; or scenario assessment, among others.

___________________

¹ Leading indicators predict future outcomes, whereas lagging indicators measure changes from past actions.

TABLE 14-1 Assumptions in the Conceptual Causal Model of How Nutrient Management Plans Reduce Stream Nutrient Loads

Step 5 – Compare Performance Data on Evaluation Criteria and Interpret Results: The final evaluation step uses monitoring and performance estimates to evaluate and improve the program using the identified evaluation criteria. By analyzing data across the causal chain, this analysis also helps diagnose if the program performed as expected, in terms of its assumptions and metrics for success. The evaluation also should identify and discuss evident unexpected effects. Unexpected or surprising results are opportunities to thoughtfully examine and improve assumptions, understanding, and ultimately improve actual performance effectiveness. Various quantitative evaluation methods exist (Khandker et al., 2009), yet qualitative analysis of information is also important (Anderson, 2003; Rossi et al., 2004). Formal evaluation results can be used to improve program design and operation, but more strategic uses include resource management and program replacement. Modeling and statistical analysis can be especially useful for integrating such evaluations into adaptive management (Holling, 1978).

Applying Formal Evaluation to the NYC DEP Watershed Protection Program

The evaluation approach described above can be applied to the Watershed Protection Program as a whole, to individual subprograms, and to components within each subprogram. The Statement of Task (Box 1-1) asks, “can the various watershed protection components (e.g., operational controls, regulatory programs and their enforcement, voluntary programs, and partnership programs) be better balanced to be more effective and sustainable?” Balancing may entail more or less emphasis on a given subprogram (e.g., Stream Management Program vs. Watershed Agricultural Program) and more or less emphasis on particular components of

TABLE 14-2 Examples of Monitoring Metrics Across All Stages of Project Implementation

each subprogram (stream management plans, stream projects, buffer program, local flood mitigation program, agricultural BMPs, etc.). Balancing also may entail emphasis on specific components or programs in different locations.

The next section presents some concepts and approaches for addressing this complex balancing more formally. This conceptual guidance would need to be developed at appropriate programmatic and geographic scales of subprogram’s objectives. For example, when balancing components, it is more useful to have more precise criteria for each program’s contribution to “water quality” related to pathogens, nutrients, turbidity, or other water quality characteristics at important locations. Inevitably, there will be tradeoffs among objectives. For example, what may be most effective for water quality might have undesirable community vitality impacts. Methods from fields of “structured decision-making” and “multi-attribute decision analysis” can help guide and organize how NYC DEP approaches this complexity (Dodgson et al., 2009; Gregory et al., 2012; Runge et al., 2013).

Balancing and Re-balancing of Watershed Program Investments

The efficient support of public health, communities, and other management objectives (including minimizing costs) requires an appropriate balance among multiple barriers and among watershed management activities. These balancing problems are complex and have important differences and uncertainties. Although our ability to understand, model, and manage these problems is limited, some analytical principles and methods exist to help managers and policy makers objectively allocate resources efficiently among diverse activities to improve performance.

This section applies several ideas from economics and engineering on how to balance investments within and among pollution prevention programs to achieve overall objectives cost-effectively. This section also highlights and discusses trade-offs among sometimes conflicting watershed management objectives.

Cost-Effective Balancing Across Programs

Watershed protection and management activities should be developed and applied to yield the greatest achievement of program objectives for the effort invested. For most programs, initial efforts typically provide the greatest incremental benefits, with incremental effectiveness diminishing with increasing investment, as illustrated in Figure 14-2. For example, relatively inexpensive primary treatment of wastewater might remove 60 percent of biological contamination, more expensive secondary treatment might remove an additional 30 percent, and still more expensive tertiary treatment might remove an additional 5 to 9 percent of contamination—illustrating the well-known principle of diminishing marginal returns. In some cases, overinvestment in an activity can result in no additional benefit, such as distillation of water already treated by reverse osmosis. It may even lead to negative benefits, in an economic and operational sense, such as where excessive removal of minerals from a drinking water system increases the acidity of water and hence its tendency to leach metals from distribution system pipes.

For the usual case of diminishing incremental benefits, it is best to balance investments across activities so the incremental contributions of each activity to the overall objective are equal. This condition of equal marginal contributions to benefits means that shifting a small amount of resources from one activity to another produces no net improvement in overall benefits. When incremental contributions to an overall benefit are not equal, this implies that shifting resources from an activity with a smaller incremental benefit to an activity with a greater incremental benefit can increase the overall benefit. This common economic optimization principle is derived and expressed mathematically in Box 14-1.

Example: Balancing Two Watershed Protection Components to Minimize One Contaminant

New York City’s many watershed management programs often have an additive effect for reducing particular contaminant loads from entering the City’s water supply system. For minimizing a single contaminant, it does not matter if a reduction to contaminant loading at Kensico Reservoir comes from one program or another, so the City’s investment objective is likely to be the greatest load reductions for some limited budgeted cost across all contaminant-reduction watershed programs. This is a special case of the derivation in Box 14-1, for only one contaminant, so the result is the simple rule that the incremental reduction in the single contaminant for each dollar investment should be the same for each program activity intended to remove that contaminant.

As a simplified hypothetical example, consider a case of trying to balance septic system retrofit investments between east-of-Hudson (EOH) and west-of-Hudson (WOH) areas. The least effective septic systems in each region are likely to be prioritized for retrofit, and among equally ineffective septic systems, those closest to NYC water intakes are likely to be prioritized. If each of the many septic systems in each watershed were evaluated in terms of estimated upgrade cost and likely reduction in reservoir phosphorus loading, they could be sequenced and prioritized in terms of cost-effectiveness.

Because of the lack of data, an illustrative analysis was developed using hypothetical removal functions for each basin. These results are depicted in Figure 14-3, where hypothetical removal functions for phosphorous are shown as dashed lines for each basin, with their corresponding incremental removals shown as thin solid lines. The highest total phosphorus removal occurs where incremental removal rates are equal (the intersection of the incremental removal curves), allocating about 86 percent of the budget to EOH septic improvements (in this hypothetical example). Increasing the share of a limited budget spent in one basin (with less invested in the other) would optimally allocate more septic system funding to EOH, because potential removal decreases more rapidly than for WOH phosphorus removal in this illustrative example.

However, for this example, also note that a fairly wide range of allocations provide very similar total phosphorus removal, allowing room for other objectives and considerations to drive resource allocations within this range. Most components of the Watershed Protection Program currently lack sufficient data or analysis to estimate such removal curves.

The practical management lesson from this result is, again, that investments in diverse watershed management programs should be made so that the incremental contribution of each program (and components within each program) should be similar (ideally, equal or equivalent) for each incremental dollar cost to the entire watershed management budget.

Unless justified by relative effectiveness with other contaminants or program objectives, it makes little sense to spend $1,000/kg on incremental phosphorous removal in one program, when another program could provide similar benefits for $5/kg. Overall phosphorous removal, in this example, could be increased by shifting funds from the expensive program to any other program with more than a kilogram of phosphorus removal for $1,000.

These examples are a simple illustration of the basic equi-incremental principle of optimal balancing of investments across programs or program components. The data and analytical needs for doing this precisely are usually unrealistic, but, as discussed later, even approximate implementation of this approach will likely eliminate the greatest inefficiencies across a watershed management program. When the incremental benefits of investments are even approximately equivalent, most efficiency gains will have been realized. When incremental benefits differ substantially, there are potential gains from shifts in investments, and the decision as to whether shifts should be made will require deliberation regarding how investments affect other program objectives.

A similar concept of balancing investments should apply for all program objectives, not just attainment of high water quality. Ideally, programs that support watershed community vitality also should have equal or nearly equal incremental effectiveness for these purposes.

Balancing Investments Across Watershed Management Objectives

As alluded to above, different watershed management investments have different potential effects on the City’s public health and water quality objectives and the watershed region’s community vitality objectives. Some investments support two or three sets of objectives (which one can equate with the benefits Bj in Box 14-1). In many cases there are trade-offs among these objectives. Even where investments support only one objective, there is a trade-off as this investment makes funds unavailable for other objectives.

Often the primary objectives of protecting water quality and enhancing community vitality go hand in hand. This is because a potential byproduct of most watershed program investments is expenditures in the regional economy of the watershed, increasing employment, income, and tax revenues. In addition, many watershed program elements, such as investments in sewers, septic systems, streambank stabilization, and flood systems, provide some direct practical support for residents in the region. However, some watershed program actions are thought to suppress community vitality in the watershed. These include NYC DEP’s land purchases, which remove some land from economic activity, and some flood, stormwater management, and other programs that may influence costs and flexibility of some activities, relative to similar areas outside of the watershed (e.g., the southern half of Delaware County). Although these watershed effects are of less direct concern for NYC residents, they can be a primary concern of their watershed partners. Data and analysis of the effects of watershed investments on watershed communities are largely absent (see Chapter 13) and have not been systematically examined, from a scientific perspective, for purposes of balancing and program evaluation.

Over time, the incremental effectiveness (i.e., the benefit derived from the next dollar invested) of water quality investments in the watershed should tend to converge and diminish. This can increase flexibility to select watershed management actions that also support watershed community vitality with little or no adverse influence on water quality improvement.

Qualitative Balancing of Current NYC Watershed Management Programs

Because of the lack of performance monitoring data and information (e.g., the amount of phosphorus or other contaminant abatement per dollar invested in each program), the mathematical framework described above for effectively balancing investments across multiple activities for one or more objectives cannot be completely implemented for the Watershed Protection Program in this report. However, qualitative analysis of incremental effectiveness can be applied to provide initial insights as well as to illustrate how a more systematic evaluation program might work. In the future, when cost and performance information from more systematic data collection become available, this management concept could be applied more explicitly and accurately.

As a coarse approximation of applying the idea that competing programs should have similar incremental cost-effectiveness, the incremental effectiveness of the major program areas shown in Table 1-1 were roughly classified for meeting the objectives of enhancing drinking water quality and community vitality, the two main objectives articulated in the MOA. In this, the Committee judged how much additional benefit would be likely from an additional $1 million investment in each program, using the classification shown in Figure 14-2. That is, would the programs accrue large, medium, small, or no benefits from an additional increment of funding? The results appear in Table 14-3.

Qualitative assessments are made for incremental improvements in drinking water quality for the “current” programs, meaning the programs as they are implemented today, as well as for “improved” programs, in which the program would be improved along the lines suggested by recommendations in earlier chapters. Similar, but much more tentative, qualitative assessments are made for the objective of improving community vitality.

Improvements are not always tied to new funding, but also may arise from improvements to the structure and delivery of program activities. In some cases, improving a program requires both some reorganization and additional funding. A systematic performance data and assessment effort, as recommended later, could change the results of the qualitative assessment presented here.

Table 14-3 merits considerable discussion. As a mostly mature set of watershed management programs, many of the most cost-effective program activities have been completed. Hence, those programs, as currently implemented, are unlikely to produce large incremental water quality benefits from additional investments. Only a few current programs (e.g., the Stream Management Program and the Watershed Forestry Program) seem likely to produce medium additional water quality benefits from further additional investment. These conclusions—that additional investment in many programs would produce only small or very small improvements in water quality—are expected for mature, well-managed and well-resourced programs.

Incremental changes in water quality might differ greatly for incremental reductions in program investments. For example, reducing funding for operations and maintenance of existing wastewater treatment plants or the Waterfowl Management Program could lead to large degradations in water quality, even though increases in these programs might cause only very small water quality improvements. Hence, reducing the budget for a successful, appropriately funded program could be harmful, while increasing the budget would produce little or no benefit.

For protecting drinking water quality, most of the programs are well developed, such that the previous chapters’ suggested improvements to program management would not lead to large incremental improvements in cost-effectiveness. A notable exception is the Watershed Agricultural Program, for which programmatic improvements could lead to substantially greater incremental cost-effectiveness, as discussed in Chapter 5. Lesser but still substantial improvements in cost effectiveness could be made with the suggested improvements to the Septic System Program discussed in Chapter 8 and the Land Acquisition Program in Chapter 7.

Enhancing community vitality is a secondary objective for most watershed protection subprograms. As discussed in Chapter 13, almost no data and information are available to assess the extent to which programs are meeting this objective, so the Committee’s judgments here are much more tentative. Incremental improvements in community vitality from existing programs are mostly small and medium, with a concern that community vitality will decrease rather than improve with additional investment under the current Land Acquisition Program. If improved as suggested in preceding chapters, some programs could substantially enhance community vitality with additional investment, particularly in the land acquisition, agriculture, stormwater, septic system, and forestry programs.

Overall, the suite of all Watershed Protection Programs provides substantial drinking water quality and community vitality benefits. Their differences in incremental benefits point toward some additional implications for policy and management. Table 14-4 reorganizes Table 14-3 using the qualitative categories in Figure 14-2 for incremental benefits for drinking water quality of additional investment in each program. Incremental benefits for community vitality were sufficiently tentative and uncertain to limit further discussion. Despite their subjective nature, several insights arise from Table 14-4.

All programs listed in Table 14-4 are effective in protecting water quality, and most programs have only small or very small potential to further improve water quality with greater investment. The wastewater treatment plant (WWTP) programs, pathogen monitoring, waterborne disease surveillance, waterfowl management, and recreation programs have small or very small potential to further improve drinking water quality,

and there are limited prospects for greatly improving the effectiveness of these already fairly effective programs with additional funding.

The Watershed Forestry Program, as currently conducted, might provide greater water quality benefits from additional investments if program uptake could be increased. Modest increases in staffing for the Watershed Forestry Program have potential to extend the influence of this program to a much larger proportion of private forest land (i.e., from 40 to 70 or 80 percent). At a time when the rate and areal extent of timber harvesting is increasing, more field foresters could substantially increase implementation of soil erosion and sediment control BMPs. This is particularly important since New York State does not have forest cutting practices law but relies on voluntary compliance with recommended BMPs. Collaborative water quality protection efforts and direct financial reimbursements to logging contractors for BMP implementation are already in place.

Similarly, the Stream Management Program has good potential to improve water quality with additional investment if expanded to new areas. The program managers could accelerate outreach and recruitment efforts with private landowners for the Catskill Stream Buffer Initiative. Although there is uncertainty about the effectiveness of the Stream Management Program, this program is collecting data that should better establish the program’s long-term effectiveness.

Substantial improvements to the Watershed Agricultural Program and Septic System Program have potential to increase their incremental cost-effectiveness (e.g., from small to large for the Watershed Agricultural Program). Improvements to the Septic System Program discussed in Chapter 8, particularly adoption of best available control technologies for septic systems, could lead to substantial improvements in water quality. Improvements to the Watershed Agricultural Program (see Chapter 5) could be even more substantial in term of water quality protection because of the larger pollutant loads from this source. For instance, conducting more rigorous data analysis, investing in manure conversion technologies, expanding the Precision Feed Management Program, and targeting BMP implementation (particularly for stream fencing, manure storage, and better nutrient management on soils with a high runoff potential) would require substantial new or reallocated funding. Yet, adopting these recommendations would result in large incremental improvements in program effectiveness.

Reorienting some larger programs could take them from being potentially overinvested in to becoming good candidates for increased investments. Unless reoriented, the Land Acquisition Program is a good candidate for shifting investments to other programs with higher incremental water quality benefits, such as the Stream Management Program and the Watershed Agricultural Program. This is especially true considering the negative community vitality aspects of the Land Acquisition Program (see Table 14-4). Yet, if the Land Acquisition Program adopted different prioritization strategies, such as a narrower focus on riparian lands or lands identified by models as high risk for mobilization of pollutants (see Chapter 7), it could improve its water quality effectiveness. Shifting money from programs with large operations and maintenance budgets supporting long-term ongoing contaminant removal, such as the WWTP programs, is unlikely to be feasible or desirable.

Over time, changing conditions in watershed development, climate, and contaminants of concern could change this assessment, and there will be opportunities to improve and adjust the components of the Watershed Protection Program. So a more formal comparative evaluation and rebalancing of component programs is recommended every few years.

Building and Managing Programs to Achieve Balance

Better integration of data, statistical analysis, and modeling could help the Watershed Protection Program more explicitly balance and adapt its diverse efforts for maintaining drinking water quality and other objectives. This would require an overall effort across major elements of the Watershed Protection Program to standardize estimates of effectiveness and program costs, so that overall effectiveness and costs could be more readily compared. Effectiveness estimates for various water quality criteria would be the most important, but effectiveness estimation for community vitality and any other objectives also would be useful. Developing and implementing some common performance effectiveness estimates will require some effort, and may not be

practical for some smaller programs, but should be reasonable for larger programs and should usefully inform resource managers, decision makers, and regulators.

Integrating water quality modeling into individual watershed management programs should be an intermediate-term objective. Having common water quality modeling across programs would provide a basis to better inform resource allocations within and among programs. It might be most useful to have two watershed models, one a mechanistic watershed quality model (such as the Soil and Water Assessment Tool, SWAT) and one a more statistically based model (such as the Spatially Reference Regressions on Watershed Attributes model, or SPARROW). Seeing where these different models agree and disagree can provide valuable insights and cause NYC DEP and collaborating researchers to further explore and improve modeling representations and assumptions. Having two models of a complex problem, each taking different approaches, enhances opportunities to test modeling inputs and outputs and refine and improve both models as well as overall scientific and technical understanding of actual processes and likely program effectiveness, including quantification of uncertainties. This modeling-based approach to the City’s diverse watershed management program would extend and integrate with the success that NYC DEP has had using the Operations Support Tool as a unifying modeling framework for system operations and planning. A modeling working group should be created with modelers assigned to work with and directly support each subprogram of the Watershed Protection Program, as well as to carry out common technical model development, documentation, and analysis.

A focused effort on data management and analysis would be needed for integrated statistical and mechanistic modeling to better understand watershed water quality problems and the likely effectiveness of solutions. Improved data management and analysis are important not only for water quality data, but also for data on land use/land cover, BMP implementation, point sources, livestock type and distribution, the extent of application of chemical fertilizer and manure, distribution of legacy phosphorus in soils, and many other descriptions of the natural and human characteristics of the landscape. In many respects, data on these drivers of water quality are as important as the observed water quality data from the monitoring network. Assembling and curating these input data sets is a major enterprise needed to improve the ability to predict how future actions will change water quality.

A focused effort to improve data analysis also could be very valuable in informing all types of stakeholders on the issues. An effort to improve the Annual Water Quality Report (discussed in Chapter 12) should emphasize graphical analyses that illuminate progress toward (or away from) water quality objectives. Quantifying geographic and temporal variations in conditions is crucial for helping system managers, watershed residents, customers, and other interested stakeholders to focus on high-priority improvements going forward. Small investments in additional staffing focused on these modeling, analysis, and data management needs could yield substantial benefits across all elements of the Watershed Protection Program.

COORDINATION ACROSS THE WATERSHED PROTECTION PROGRAM

The Watershed Protection Program consists of actions by multiple programs operating in differing spatial, jurisdictional, and topical areas. It is an effective partnership between NYC DEP and other organizations involved in maintaining the filtration avoidance determination, including the Watershed Agricultural Council, the Catskill Watershed Corporation, county soil and water conservation districts, Cornell Cooperative Extension of Ulster County and the Delaware County Planning Department; along with the Coalition of Watershed Towns and the individual towns and municipalities within the watershed. Other entities participating in this partnership include the U.S. Geological Survey, New York State Department of Environmental Conservation, and the U.S. Forest Service, among others. Other NYC agencies also have key roles, particularly the Office of Management and Budget, NYC Law Department, Mayor’s Office of Contracts, Comptroller’s Office, and the Department of Citywide Administrative Services. Lastly, NYC DEP itself is not a monolithic unit, but has various programs and divisions.

This diverse institutional structure has advantages and challenges. Each organization or program within an organization brings strengths, knowledge, and capacities. However, inefficiencies occur from overlapping

or countervailing actions, communication delays, and transaction costs for navigating multiple administrative structures (Lubell, 2015; McGinnis, 2005). Further, tensions arise from differing perspectives and priorities across organizations and programs (Andersson and Ostrom, 2008; Black, 2008; McGinnis, 2016; Nicholson-Crotty, 2005). This section makes some preliminary suggestions for improving coordination among and within entities involved in the Watershed Protection Program, based on presentations to the Committee and several site visits in the WOH watersheds.

The Watershed Protection Program has evolved over the last two decades. Programs, policies and relationships have changed over time, and while many elements remain the same, the number and scope of activities have grown. Periodic review of the set of activities and partners of the Watershed Protection Program, with an eye toward improving coordination, could improve program effectiveness and efficiency. Such reviews should consider how watershed protection actions by one program or organization influence processes, decisions, and outcomes by others. Increased coordination would ensure consistency and coherence among objectives, among policies and projects, and in implementation, monitoring, and enforcement (Meijers and Stead, 2004). Several specific areas for better coordination are identified below.

Coordination Across Land-Management-Related Efforts

Many potential synergies exist among programs that seek to contribute to watershed protection through land management, yet in many respects each program operates independently. Improved communication between the Stream Management Program and the Land Acquisition Program, particularly in identifying and prioritizing land for protection, could improve outcomes for both programs. Opportunities exist to improve communications among localized flood assessments, town hazard mitigation plans, and the streamside and land acquisition programs, as well as between the Land Acquisition Program and municipalities. In several cases, the Land Acquisition Program’s individualized relationship with landowners has inadvertently constrained opportunities for other NYC DEP watershed programs and for local municipalities. For example, the Hamlet of New Kingston could not implement a 30-house community septic system because the only viable site for a large leach field (20 acres) was a 374-acre property with an agricultural conservation easement. While the owner would have sold the property, the cost of repaying the Watershed Agricultural Council for the easement on the 20 acres needed for the leach field was too high. Similarly, a proposed community septic system in DeLancy could not be constructed due to easements on suitable sites, so a Septic Maintenance District was created instead (which provides some water quality benefit, but likely less than a properly sited community septic system would have) (A. Rosa, Catskill Watershed Corporation, personal communication, 2019). Better coordination between entities implementing projects to support the filtration avoidance determination and local communities, along with an ability to revise easements to achieve similar or greater protection, could increase overall water quality protection.

Coordination of Subprograms with Watershed Monitoring and Modeling

Each of the Watershed Protection Program’s many subprograms collects and manages a vast array of data, much of which has potential to better inform managers, regulators, and the public. Increasing communication and coordination from the NYC DEP monitoring and modeling efforts could substantially improve the effectiveness and efficiency of all programs. For example, the Watershed Agricultural Council has information on the location of farms, fields, and BMPs; the potential sources of pollution on farms (crops, fertilizers, pesticides, and animals); information on the watershed protection measures adopted on farms (BMPs, nutrient management plans, precision feeding, etc.); and detailed monitoring data on soil phosphorus concentrations. Currently these data are unavailable to NYC DEP’s monitoring and modeling efforts, or, when made available, are provided in aggregate formats with insufficient georeferencing. The result has been that watershed modeling to date (e.g., Hoang et al., 2019) must make assumptions, rather than use empirical data or professional judgment that may be available in other programs. Adding finer-scale data to watershed modeling efforts would further the understanding of pollution sources and transport within the watershed and also could enable more

accurate and precise evaluation of the effectiveness of watershed protection activities. Further, watershed modeling could, in return, provide timely and useful information on how to better prioritize actions within the Watershed Agricultural Program.

As another example, the Stream Management Program has focused data collection to specific locations, particularly on turbidity and suspended sediments. Furthermore, many watershed towns and communities have developed HEC-RAS hydraulic models for their local flood analyses. Many other subprograms also have georeferenced information, including the Catskill Watershed Corporation data on septic system replacements and repairs, Land Acquisition Program data on conservation easements, and information on forest management practices. These data have potential to be useful in SWAT, SPARROW, and other model development.

Each of the Watershed Protection Program’s subprograms and each partnering agency independently track their activities and monitor outcomes. Examination of the range of information regularly collected across all of the subprograms, and analysis of what information could jointly benefit a range of subprograms, could readily identify synergies in information needs and opportunities to more efficiently and effectively collect, manage, analyze, and disseminate data. Having all of these monitoring efforts contribute to an enhanced Annual Water Quality Report by NYC DEP would help improve the consistency and utility of this information (as recommended in Chapter 12).

Coordination with NYC Office of Management and Budget

Many programs and organizations routinely experience delays or roadblocks in contracting, budgeting, and approvals from City agencies outside NYC DEP. These barriers are particularly problematic for watershed protection activities that can only be conducted seasonally and for protection activities that rely on voluntary participation of property owners or farmers. In each case, delays can lead to inefficiencies, lost opportunities, and sometimes ill will. As discussed in Chapters 8 and 9, these issues are particularly acute for the septic system and stormwater programs.

WATERSHED PROTECTION FOR THE LONG TERM

A safe and dependable water supply is fundamental for all people, communities, and societies. Typically, a combination of source protection, water treatment, and water quality and disease surveillance monitoring is needed to reliably meet this vital goal. In a few exceptional cases (e.g., New York City, San Francisco, and Boston’s water supply systems) it has been possible and economical to rely on robust disinfection treatments, source protection, operational expertise, strong compliance monitoring, and stakeholder efforts and cooperation to consistently meet state and federal regulatory requirements without water filtration. Filtration avoidance has been feasible for NYC’s water system primarily because high-quality watershed conditions. Given that 75 to 90 percent of the landscape in the individual watersheds is forested, the potential for pollution from human activities (residential, agricultural, or industrial) is quite limited. Further, the large size and operational capabilities of the City’s diverse reservoir system, with advanced decision support and real-time monitoring (the Operations Support Tool; NASEM, 2018), and the experience and professional judgment of NYC DEP water supply managers, provides flexibility in delivering the cleanest water from these forested watersheds. Finally, although the City’s water is unfiltered, it is not untreated. The robust use of two different types of disinfection (ultraviolet light and chlorination), supplemented with occasional alum treatment of source water entering Kensico Reservoir, provides a high reliability of safe and abundant drinking water. Nevertheless, these components would not be sufficient to meet the daunting and necessary public health requirements of the filtration avoidance determination without the cooperation and participation of watershed residents and communities following the 1997 MOA and sustained investments by all stakeholders.

Influenced by a long and colorful history (summarized in Chapters 2 and 3) and forged by intense conflict and negotiation from 1991 to 1997, under the MOA the Watershed Protection Program has evolved over two decades into a successful model that some doubted would be sustained. This success is built on the shared

responsibility of the MOA signatories to maintain or enhance water quality while sustaining the community vitality of watershed communities. The MOA strives to allocate program benefits and costs in an equitable, ethical, and sustainable way across NYC water consumers, watershed residents and communities, and New York State.

NYC DEP’s investment in the Watershed Protection Program over the past 20 years has exceeded $2 billion. Thousands of stakeholders, regulators, and others also have been actively involved. In contrast, facility costs for filtering the WOH system are estimated between $8 billion to $10 billion plus annual operation and maintenance costs of about $365 million.² At roughly $100 million/year (see Table 1-1), the Watershed Protection Program offers a more economical means to protect drinking water quality for NYC water customers. These water supply protection efforts also serve community vitality and potential ecosystem objectives, making the Watershed Protection Program compatible with many broadly supported values.

Watershed protection and management are inherently complex, especially at the scale of NYC’s water supply system. In any system—large or small—watershed protection requires both steadfastness and constant adaptation to balance the needs of different groups with the dynamic characteristics of biophysical and socioeconomic systems over long periods (Brooks et al., 2013). The uncertainty of global climate change and inherent tensions, tradeoffs, and conflicts amplify challenges of maintaining a safe and dependable water supply in the coming decades.

During the two-year course of this study, “filtration” was discussed as (1) a tacit admission of failure, (2) necessary and long overdue, (3) inevitable, or, as (4) “the nuclear option” implying that with a filtration plant, NYC would abandon most or all commitments in the MOA and investments in the Watershed Protection Program. If filtration is added in the future, it would lead to a rebalancing of programs, investments, costs, and benefits. However, adding filtration probably would not lead to abandonment of source water protection, because of its continued (though diminished) value for drinking water quality and long-standing state and federal laws, regulations, and agreements.

The 50-year trend in water supply and public health laws and regulations—with strong support of the majority of the electorate—is to add safeguards, rules, and compliance standards. Yet, even if a revision of laws and regulations required NYC to add water filtration tomorrow, it would take at least 20 years to design, permit, finance, construct, and test a massive plant before its first day of reliable operation. So, a long-term Watershed Protection Program (1) based on adaptive management, (2) consistent with the MOA, and (3) supported by two decades of cumulative experience and professional judgment continues to be a necessary and prudent investment now and into the future.

CONCLUSIONS AND RECOMMENDATIONS

The 1997 MOA and Watershed Protection Program have largely succeeded in maintaining or enhancing water quality for the NYC water supply system and providing sustained investments to enhance the economic vitality of watershed communities. Active and evolving partnerships with the Catskill Watershed Corporation, Watershed Agricultural Council, and many other organizations and agencies show the potential—and tradeoffs—of balancing water quality protection with community vitality. The following conclusions and recommendations are made to improve the overall effectiveness of the Watershed Protection Program. Supporting program-specific conclusions and recommendations are found in Chapters 5 through 12.

The Watershed Protection Program overall appears to have admirably supported watershed water quality sufficient for compliance with the Surface Water Treatment Rule, with strong indications that it will remain effective into the future. However, in the last 25 years many of the most effective watershed management actions have already been implemented. Additional efforts will likely be more costly and less effective per incremental investment for achieving water quality objectives. Given the decreasing incremental effectiveness and increasing incremental costs of most program elements, as well as uncertainties in estimating costs and effectiveness, many near-optimal combinations of program activities could provide similar overall

__________________

²https://www.dec.ny.gov/lands/58524.html

performance. This means that there is increased flexibility to select watershed management actions that also support community vitality with little or no adverse influence on water quality. However, this wide range of near-optimal solutions will make it more controversial for managers to identify a best mix of watershed management activities and will make consistent management analysis all the more important.

The component programs within the Watershed Protection Program are generally well-balanced, with a few exceptions. The New York City Department of Environmental Protection should reduce expenditures in the Land Acquisition Program to fund other programs that will lead to more direct improvements in water quality. Programs with greater incremental value include an improved Watershed Agricultural Program, an improved Septic System Program, and the Watershed Forestry Program. This reallocation of funds is based on the seemingly small incremental contributions of the Land Acquisition Program to drinking water quality and its negative effects on community vitality, compared with the likely improvements to water quality from additional resources provided to these other programs.

Enhanced communication and coordination across the subprograms of the Watershed Protection Program, partner organizations, and watershed counties and towns (as envisioned in the Memorandum of Agreement) could reduce transaction costs, further improve working relationships, and foster cooperation in ways that benefit, admittedly to varying degrees, all stakeholders. Local planning and development often could be better coordinated with Watershed Protection Program actions, and with other NYC departments for mutual benefits. Improved provision of information from common statistical and modeling analyses and annual reporting of data should support this process.

More systematic evaluation of the individual components of the Watershed Protection Program would help managers and regulators prioritize investments and actions to better achieve water quality and community vitality objectives. Such evaluations will require improving performance monitoring, analysis, and modeling to connect program activities with outcomes and impacts. Water quality performance estimation would often be enhanced by the development and use of common water quality statistical analyses and modeling across major programs (also discussed in Chapter 12). This initiative would extend the success of NYC’s modeling approach, illustrated by the Operations Support Tool, from operational integration to its Watershed Protection Program, with additional benefits for technical and policy integration across programs. Performance assessment of programs that affect community vitality should be developed within this effort. Improved annual reporting of performance would provide a common and accessible basis for using such information in discussions involving the Watershed Protection Program’s diverse range of managers, regulators, and stakeholders.

REFERENCES

Dodgson, J. S., M. Spackman, A. Pearman, and L. D. Phillips. 2009. Multi-criteria analysis: A manual. London: Department for Communities and Local Government. https://www.researchgate.net/publication/30529921_Multi-Criteria_Analysis_A_Manual.

Ferraro, P. J. 2009. Counterfactual thinking and impact evaluation in environmental policy. In: Environmental program and policy evaluation: Addressing methodological challenges. M. Birnbaum and P. Mickwitz (eds.). New Directions for Evaluation 122:75–84.

Gregory, R., L. Failing, M. Harstone, G. Long, T. McDaniels, and D. Ohlson. 2012. Structured decision making: A practical guide to environmental management choices. John Wiley & Sons.

Hoang, L., R. Mukundan, K. E. Moore, E. M. Owens, and T. S. Steenhuis. 2019. Phosphorus reduction in the New York City water supply system: A water quality success story confirmed with data and modeling. Ecological Engineering 135:75-88.

Holling, C. S. 1978. Adaptive Environmental Assessment and Management. Chichester, UK: John Wiley and Sons.

Khandker, S., G. Koolwal, and H. Samad. 2009. Handbook on impact evaluation: Quantitative methods and practices. Washington, DC: The World Bank. http://documents.worldbank.org/curated/en/650951468335456749/pdf/520990PUB0EPI1101Official0Use0Only1.pdf.

Lubell, M. 2015. Collaborative partnerships in complex institutional systems. Current Opinion in Environmental Sustainability 12:41-47.

Mayne, J. 2015. Useful theory of change models. Canadian Journal of Program Evaluation 30(2):119-152.

McGinnis, M. D. 2005. Costs and challenges of polycentric governance. In: Workshop on analyzing problems of polycentric governance in the growing EU. Berlin: Humboldt University.

McGinnis, M. D. 2016. Polycentric Governance and Development: Dimensional Aspiration and Practical Limitations. In: Vincent and Elinor Ostrom Workshop in Political Theory and Policy Analysis. Bloomington, Indiana.

McLaughlin, J. A., and G. B. Jordan. 2004. Using logic models. Pp. 7-32 In: Handbook of Practical Program Evaluation, 2nd ed., J. S. Wholey, H. P. Hatry, K. E Newcomer (eds.). San Francisco: Jossey-Bass.

Meijers, E., and D. Stead. 2004. Policy integration: What does it mean and how can it be achieved? A multi-disciplinary review. In: Berlin Conference on the Human Dimensions of Global Environmental Change: Greening of Policies-Interlinkages and Policy Integration. Berlin. http://userpage.fu-berlin.de/ffu/akumwelt/bc2004/download/meijers_stead_f.pdf

NASEM. 2018. Review of the New York City Department of Environmental Protection Operations Support Tool for Water Supply. Washington, DC: The National Academies Press.

Nicholson-Crotty, S. 2005. Bureaucratic competition in the policy process. Policy Studies Journal 33(3):341-361.

Rogers, P. J. 2008. Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation 14(1):29-48.

Rossi, R., M. Lipsey, and H. Freeman. 2004. Evaluation A Systematic Approach. Thousand Oaks, London, New Delhi: Sage Publications.

Runge, M., J. Grand, and M. Mitchell. 2013. Structured Decision-Making. Pp. 51-72 In: Wildlife Management and Conservation: Contemporary Principles and Practices. P. Krausman and J. Cain III (eds.). Baltimore, MD: The Johns Hopkins University Press.