State Product Evaluation Programs (2004)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

18 CHAPTER FOUR DISCUSSION OF EFFECTIVE PROGRAMS Measuring the performance of research and technology programs is a difficult and subjective task. This is well- documented and was recently studied in NCHRP Synthesis of Highway Practice 300: Performance Measures for Re- search, Development, and Technology Programs (12). New product evaluation programs fall well within the scope of development and technology issues. The FHWA addressed this in the 1990 Engineers’ Guide to Program and Product Evaluation (13). Given that evaluation programs exist to provide a formal process for incorporating new and inno- vative products into practice so that improvements may be made in transportation operations and management, it is important to identify and recognize their respective value to the DOTs. This chapter will look at various examples of how exist- ing programs are defined and how their success and effec- tiveness is measured. The relevant benefits of introducing a new product into practice as a result of these programs will be discussed as addressed from the standpoint of the re- spective agency. In addition, any feedback from the DOTs regarding claims of bias from the vendors, manufacturers, or applicants is also reported. MEASURING FEEDBACK FROM OUTSIDE SOURCES Of the 34 DOTs that responded to the survey and con- firmed that they have formal programs for new product evaluations, 8 have experienced claims of bias from out- side sources. These claims ranged in substance from a sup- plier or vendor believing that an agency used an improper test to verbal claims of bias that were never pursued legally. Typi- cally, formal, legal actions are not often taken to pursue a claim of this nature. The California DOT (Caltrans), how- ever, has experienced claims and has a mechanism for fil- ing appeals designed into their evaluation process. Only two states reported being asked for compensation as the result of bias. One state resolved the claim by retest- ing and the other claim was still pending at the time this report was being compiled. It appears to be relatively common for agencies to retest a product to resolve differ- ences and avoid formal claims. It is apparent that the clearer the process for product evaluation is spelled out, the fewer the number of claims of bias will be presented. Establishing specific and easy-to- follow guidelines as many states have requires careful planning, but may be well worth the effort to avoid claims of bias. DOTs wishing to develop guidelines for filing claims can learn from those examples that have been de- veloped and tested by others, such as in California. It ap- pears that whatever mechanisms DOTs have used to deal with product requests from outside sources, substantiated claims of unfairness are rare. This synthesis did not include contact with outside sources or vendors who have submitted requests for prod- uct approvals. If the issue of claims becomes a significant factor in how DOTs design or revise their evaluation pro- grams, some research or surveying of vendors could be of value to the effort. Seeking input from the private-sector community that uses the various programs could be useful in identifying any potential pitfalls, areas of confusion, or shortcomings of existing programs. IDENTIFYING EVALUATION PROGRAM SUCCESSES The range of products that are evaluated and submitted for evaluation is broad (see Table 2). They can be of a techni- cal or nontechnical nature. The products may be market tested and readily available to purchase and deploy or they may be untested in actual practice. They may be totally new to an agency or provide a new perspective of a famil- iar subject. What works in one jurisdiction may not work or apply in another. Some evaluations can be accomplished in days, whereas others may take years. Measuring the success of programs and the benefits of various product applications is subjective and difficult for all of these reasons. Nevertheless, there are many examples of program experiences and successes that may be helpful to share, particularly among agencies with common problems. Examples are provided here to illustrate the variety of issues associated with evaluation and testing programs. These ex- amples also illustrate important criteria for measuring effec- tiveness and success and, to that end, provide a framework for discussion on how evaluation programs add value and sup- port the broader missions of transportation agencies (14). Following are several examples that demonstrate the opportunities for innovation that formal programs create. Some are cases in programs with fairly rigid, well- established guidelines, whereas others are from programs that are still developing and more flexible depending on the nature of the product.

19 Utah Experiences Typically, the Utah DOT (UDOT) always strives to meas- ure and evaluate a product against a current UDOT stan- dard specification. If the vendor or supplier can demon- strate that a product meets or exceeds the minimum requirements of that specification, the product is accepted as part of an Accepted Product Listing (APL). In instances where there is no applicable standard, the Utah program relies on current industry standards for product evaluation. When a product is found to meet one of these standards, it is cited in a Performance Data Prod- uct Listing (PDPL). When there is no UDOT or national standard that covers the product, the need for that product is assessed and, if a need is identified, a demonstration, often an experimental feature in a test section, is conducted. If the test is success- ful, the product is included in the PDPL for use under pro- visional conditions. As more information is gathered after subsequent use of the product, a new specification or modification to an existing specification may be recom- mended and the product may be moved to the APL. This general approach to product evaluation is fairly common among agencies. The UDOT provides perspective vendors or suppliers with a flow chart that illustrates the agency acceptance criteria and procedures. Specific exam- ples of the UDOT evaluation procedure are described here. • Several years ago, a manufacturer proposed some temporary pavement markings. The state had no specification to cover such markings, but a need was identified. An evaluation was conducted and the product was field tested as a temporary roadway marking providing guidance to construction traffic and striping crews. The product was very successful and resulted in the creation of a new standard specifi- cation on temporary pavement markings. Shortly thereafter, other DOTs adopted similar products. • Corrosive soils in the west Utah desert have been slowly eroding standard pipe culverts. A new product was presented to the department consisting of an ex- pansive polyvinyl chloride (PVC) liner, which when properly installed conforms to the shape and size of the pipe culvert. This liner system reduces corrosion and repairs damaged pipes without their removal and reconstruction. The product is still being tested, but appears to be very successful and has been placed on the state’s PDPL, as no standard specification covers its installation and performance. • Much discussion has taken place on the subject of concrete sealers, including its different formulations and performance claims. UDOT performed a series of concrete sealer tests under laboratory and field conditions. The findings resulted in the creation of a new standard specification on the subject and a list of approved sealers has been placed on the APL from which state maintenance employees and contractors can choose. Iowa Experiences The Iowa DOT approves several products for use each year, with the majority approved as alternates to an exist- ing standard product already used by the agency. They view the value of this effort not so much in direct cost or time savings, but in the added product options it gives their staff and the increased competition within the marketplace to provide better products. This increased competition is noteworthy because there is considerable volatility in the market with both the sup- pliers and manufacturers. The shelf life of products can be short, particularly without orders. A formal, active program facilitates the opportunity for suppliers and manufacturers to get to the market in a reasonable time. Several of the products recently approved are briefly described here. • Iowa noted that geocomposite pavement drains, which are prefabricated drains, placed vertically ad- jacent to pavements to intercept and remove water from a subbase and subgrade, are an example of a successful evaluation that provides a variety of op- erational options and applications. These drains are currently used when specific bedrock, soil, or other design conditions exist that preclude the use of longi- tudinal subdrains. • Plastic guardrail blockouts have recently been evalu- ated and approved for use. Specifications are cur- rently being revised to allow for their standard use. • A procedure for early concrete sawing was recently evaluated and approved by the Iowa evaluation pro- gram. This procedure consists of sawing transverse joints in portland cement concrete pavement while the concrete is still green and not fully hardened. The procedure is less expensive than traditional concrete sawing owing to the lighter saw equipment required, the fewer saw blades consumed, and the less power required. There are significant environmental advan- tages because there is minimal dust caused by the procedure. Nevada Experiences The Nevada DOT’s (NDOT) new product evaluation pro- gram is considered the key to the department’s construc- tion operation. By providing a process for the prequalifica- tion of highway products and materials the program is

20 instrumental to an effective and successful construction program. Nevada’s program includes formal, written procedures that are shared throughout the transportation community. This communicates to all parties, including manufacturers and vendors, that the process treats everyone equally and fairly. The evaluation program is managed from the research office and coordinated with the other appropriate depart- ments to ensure that all facets of the process work to improve the quality of the products and materials used by the depart- ment. The process involves high-level managers from the ma- jor operating divisions of the agency to provide the broadest perspective to the effort. Many standard specifications for various products and materials have been developed in conjunction with the product evaluation process. • Several standard specifications have been created in response to safety issues. As a result of the evaluation program, NDOT has developed specifications for fluorescent retroreflective sheeting that helps im- prove traffic, worker, and motorist safety. This speci- fication has recently been modified for application at school and pedestrian crossings. • There has also been considerable advancement with hydraulic-related products and materials as a result of the program. The state has developed several hydrau- lic-related QPLs based on new acceptance criteria and standard specifications that did not previously exist. These products include cellular erosion control mats, stormwater treatment systems, and trench drains. • Another product of the program is a general QPL, acceptance criteria, and quality control/quality assur- ance criteria for nonmasonry, nonconcrete, or uniquely constructed soundwall systems (15). Ven- dors are provided with an application package that al- lows them to show that their product complies with the NDOT criteria for placing their system on the general QPL for soundwall systems. Actual systems are selected from the general QPL based on project- specific requirements. The Nevada DOT’s process evaluates all products on the basis of need, performance, cost-effectiveness, and compliance with recognized specifications and standards. New York State Experiences In 2002, the New York State DOT (NYSDOT) adopted a new evaluation procedure. This program has been designed to reduce the time it takes to evaluate, approve, and im- plement a new product. The new program was the result of a response to an upper management request that the NYSDOT be more responsive to innovative ideas from outside the department. The program focuses on products that are not covered by a standard specification or the 300- plus-page departmental-approved products list. NYSDOT’s program evaluates products that are truly unique to the ex- isting practices of the department. These are typically products that are proprietary, limited in use, and out of the conventional or traditional mainstream. A product evaluation committee with voices from each of the department’s 11 divisions was formed to deal with product requests on a monthly basis. This committee makes the process more effective by focusing and central- izing approvals for the entire department. Instead of a product being assigned to one individual to evaluate with no formal follow-up, the committee has elevated the prior- ity of the process and has shortened the time it takes to get resolution of an evaluation request. The committee has also improved the communication of evaluation results throughout the state by using various tools. One such tool is the Engineering Bulletin. As soon as an action is taken on a product and either approved or re- jected, a one-page Engineering Bulletin is issued. This no- tice is distributed among manufacturers, the DOT offices, local governments, regions and agencies, surveyors, con- sultants, and contractors as appropriate. The bulletins serve to communicate the results of an evaluation and are effec- tive immediately. This can reduce the time in getting the product into practice by not requiring the development of standard specifications or special provisions before use. The bulletins expire 1 year after issuance unless replaced sooner. More formal Engineering Instructions are issued when an approved new product is considered appropriate for permanent change and a new standard specification. It should be noted that the New York State program and the new product evaluation committee only consider prod- ucts for evaluation that show a potential benefit to the de- partment and for which there is no existing department specification. This allows the department to look at a broad range of innovative ideas that would otherwise not get evaluated. Since the program went into effect in 2002, the NYSDOT has seen a sharp increase in the number of prod- ucts being submitted for evaluation. In turn, they have be- come more responsive to requests and they better commu- nicate the results of evaluations. Although by their own admission they are still experiencing some growing pains, they are better supporting upper management by encourag- ing innovation in the department. California Experiences The California DOT (Caltrans) has created an effective, formal program and application process for introducing

21 new products into practice. This is particularly significant and beneficial in such a large, diverse state. Instead of ven- dors and manufacturers presenting the same request for evaluation to all of the department’s 12 district and multi- division offices, Caltrans uses a centralized program that involves top level managers who have statewide responsi- bilities. The time and resources to act on requests have been minimized and optimized. In California, a centralized process and standardized protocols for product approval have eliminated the main variables that create problems. These processes are well communicated to the customers and access to the program is included on the Caltrans web- site. The Caltrans staff responsible for the new product evaluation program has compared themselves to the greet- ers at WalMart, in that they work extensively with their customers, vendors, and manufacturers to ensure that the process is customer friendly and fair to all. The evaluation staff relies on the technical experts within the agency to ac- tually conduct the evaluations. Although the staff acknowledged that because of budget constraints and recent downsizing, they cannot change a specification or develop a new standard special provision (SSP) every time they have a request, they do respond to the vast majority of requests. The agency also pointed out that although a sole source approval is a major issue for some states it has not been a major issue for Caltrans. Al- though some states may shy away from anything that re- sembles a sole source approval, Caltrans has determined that once the vending/manufacturing community learns of their interest in a specific product or technology there is no shortage of suppliers, vendors, or manufacturers. Examples of successful product evaluations are provided here. • One example of product evaluation that has been successfully deployed in California was their devel- opment of plastic wood standards. Because of envi- ronmental and other concerns, use of creosote-treated wood and lumber in marine environments is no longer allowed in the state. Caltrans has developed an SSP for all future projects that require plastic wood applications. • Remote deicing systems have also been evaluated by the Caltrans program. They are being considered in areas where black ice is a concern. These systems are essentially sprinkler heads charged with deicing products that are sensor controlled by mini-weather stations. When the conditions read a threat of black ice, the sprinklers discharge a few seconds of the de- icing material. These systems have applications on both roadways and structures. HITEC is currently working with Caltrans on a full-scale field test of a roadway section in southern California, and the department is undertaking another full-scale field test on a bridge in the Sierra Mountains in northern Cali- fornia. Although there is no SSP yet developed for this technology, it has shown great promise. • Other product evaluations that have produced SSPs include graffiti removers, pavement markers, and re- flective signing materials. One additional point that the Caltrans staff added to the discussion concerns the affect that political pressure can have regarding the evaluation and acceptance of new prod- ucts. There have been cases where vendors have gone to their respective state legislators in an attempt to bring some pressure on Caltrans to evaluate or to adopt a specific product. There have also been cases where legislators have questioned Caltrans on their procedures as a follow-up to these vendor contacts. This is something that can happen and for which the evaluation staff must be prepared. It has not necessarily been a negative process and there has never been an example of a legislator putting pressure on the agency to accept or to test a particular product. Caltrans is comfortable that their program is clearly defined and func- tions as fairly as possible to all applicants within the bounds of their budget and resources. Maryland Experiences For many years, the MDSHA has conducted an effective new product evaluation program and it has been assessing its current status and future optimum position within the department. The Maryland program has resulted in many operational successes for the agency and those successes have been shared with others in a variety of ways. In 2002, the MDSHA sponsored and conducted a survey among state transportation agencies designed to lend insight to their ongoing organizational assessment. That survey, given the 65% response rate, has proven to be a great com- plement to the electronic survey conducted as part of this synthesis, which coincidentally had a very similar response rate. Although the purpose of the Maryland survey was to help their agency assess their own program, the results of that survey reinforced several findings of the synthesis survey and provided additional pertinent information to this study. (The Maryland survey and survey responses can be found in Appendices C and D.) One area of particular note in the Maryland program is their ongoing effort to improve communications within the evaluation and testing community. Their survey findings on product information indicated that most respondents are using, or are moving toward using, databases to track new product applications and to publish evaluation results. The majority of these databases are available on-line. The Maryland survey also indicated that whereas many states use information on the AASHTO APEL, most do not enter data on APEL. It is a goal of the Maryland program to

22 make evaluation and product information, records, and re- sults available through an on-line database that will elec- tronically up-link to APEL. As this resource develops it could serve as a national model for respective states to document and share their evaluation information. Addi- tional examples of evaluations are described here. • One operational area that has been well served by the Maryland product evaluation program is winter maintenance. Several products have been success- fully implemented after thorough evaluations and field tests, including deicing and anti-icing technolo- gies and products that have spun off from the suc- cessful testing of anti-icing chemicals and treatments. The MDSHA has developed a new saddle tank de- sign tied to their spreader boxes to assist in the effec- tive application of the winter maintenance materials. This design has been installed on more than 90% of agency dump trucks. • Alternatives to conventional silt fence and stormwa- ter management are currently being studied by the MDSHA and will result in new or revised specifica- tions. These materials show real promise in reducing time and labor. • Erosion control matting and barriers are other prod- ucts that have shown great promise. With the assis- tance of certified erosion control laboratory testing and NCHRP Report 350 (16) crash testing, the evaluation process for these types of products has been considerably reduced. Florida Experiences The Florida DOT (FDOT) has a well-established and con- tinuously evolving product evaluation program. It is man- aged within the state specification office located in the de- partment’s design office. The program has four full-time staff members available to facilitate product evaluations with the appropriate technical staff within the department. FDOT uses a QPL as a list of pre-approved products that have been evaluated against adopted and implemented specifications or standards. Typically, these products are common and used in practice by FDOT and by others in the transportation industry. The goal of this approach is to create a reliable list of products for construction personnel that meets the standards of the department and is equitable to all parties. When a QPL has been established for a par- ticular product type, FDOT specifications limit the contrac- tor’s choice of products to those on the QPL. New products that are not common, that have not been used in practice, and for which no adopted and imple- mented specification or standard exists are directed to a re- cently established product evaluation oversight committee. The role of this committee is to assess the product’s poten- tial for use on the state highway system in Florida. By ac- tion of the committee, the product is either • Determined to be a product of marginal or no interest to the agency, • Determined to be a product that may be of interest to the department but needs some further technical in- vestigation of potential use by a technical champion assigned by the committee, or • Determined to be a product that could have some po- tential use but is not considered a good candidate for inclusion in statewide specifications or standards. These approved products typically have limited use and application in practice. They could be incorpo- rated as a feature of the design of a particular project, but the QPL would not be used in the process. Florida also uses two programs to keep the QPL up- dated and reliable. The first addresses and investigates questions of alleged deficiencies in products included on the QPL, which can result in the removal of a product for failure to perform. The second is a requalification process that establishes the time cycles, certification requirements, and technical resubmittal criteria required from the manu- facturer or vendor for the product to remain on the QPL. The Florida product evaluation program, established as a result of a value engineering program in the early 1980s, has historically provided outreach to local governments for product evaluations. The state is now re-evaluating their approach to local government outreach because of the re- sources necessary to support the demands from locals. Of- ten the product requests from local jurisdictions are of lim- ited interest to, or are no longer used by, the state. Maintaining an effective and prudent outreach program for Florida’s local jurisdictions is a challenge facing the DOT. Other Experiences There were several other common experiences that DOTs shared regarding their respective programs. There have al- ways been many materials products submitted for evaluation, but there are an increasing number of main- tenance products and systems being submitted for re- view and approval. One reason for this may be the na- tional emphasis on maintaining and preserving the existing infrastructure as opposed to new construction (17). Evaluations and innovation for safety, traffic, environ- mental, winter maintenance, and constructability products are also frequently requested (for example see ref. 18). In addition, there are many products of new product evaluation programs that help DOTs manage their quests for innovation. These include the various APLs and QPLs that result from successful product testing and evaluation.

23 One important benefit of successful state product evaluation programs that became apparent is the benefit provided to cities, counties, and other local jurisdictions. Many local governmental agencies do not have the re- sources, laboratories, personnel, or expertise to appropri- ately evaluate new products. Often their respective specifi- cations and performance standards originated as a state specification or special provision. These local governments depend on the states to set standards and acceptance crite- ria on new products. Through programs such as the Local Technical Assistance Program, many cities and counties have gained the knowledge of and access to new products. It is also not uncommon for a state to receive a request from a local jurisdiction to test a specific product. There are also examples of local jurisdictions that have their own evaluation programs. One such example came from Ottawa, Ontario, Canada. The city of Ottawa has es- tablished evaluation procedures that resulted in APLs for their sewer and water distribution systems. The city antici- pated having similar APLs and evaluation procedures in place for their road and sidewalk products by mid-2003. As can be seen by these very diverse examples of prod- uct evaluation results, the respective success of a program can be judged on a number of factors. Cost and time sav- ings are always benchmarks for success, but often are dif- ficult to identify and may not yield a true picture of a product’s value. Less tangible factors, such as providing greater flexibility in problem solving, can be equally im- portant. Establishing good communications and rapport with various industry and commercial entities is also im- portant and can be critical in bringing new, innovative products to market. New product evaluation programs af- fect and support the operations of their respective agencies in many ways.