National Academies Press: OpenBook

State DOT Product Evaluation Processes (2024)

Chapter: Chapter 5 - Summary of Findings

« Previous: Chapter 4 - Case Examples
Page 77
Suggested Citation:"Chapter 5 - Summary of Findings." National Academies of Sciences, Engineering, and Medicine. 2024. State DOT Product Evaluation Processes. Washington, DC: The National Academies Press. doi: 10.17226/27809.
×
Page 77
Page 78
Suggested Citation:"Chapter 5 - Summary of Findings." National Academies of Sciences, Engineering, and Medicine. 2024. State DOT Product Evaluation Processes. Washington, DC: The National Academies Press. doi: 10.17226/27809.
×
Page 78
Page 79
Suggested Citation:"Chapter 5 - Summary of Findings." National Academies of Sciences, Engineering, and Medicine. 2024. State DOT Product Evaluation Processes. Washington, DC: The National Academies Press. doi: 10.17226/27809.
×
Page 79
Page 80
Suggested Citation:"Chapter 5 - Summary of Findings." National Academies of Sciences, Engineering, and Medicine. 2024. State DOT Product Evaluation Processes. Washington, DC: The National Academies Press. doi: 10.17226/27809.
×
Page 80
Page 81
Suggested Citation:"Chapter 5 - Summary of Findings." National Academies of Sciences, Engineering, and Medicine. 2024. State DOT Product Evaluation Processes. Washington, DC: The National Academies Press. doi: 10.17226/27809.
×
Page 81
Page 82
Suggested Citation:"Chapter 5 - Summary of Findings." National Academies of Sciences, Engineering, and Medicine. 2024. State DOT Product Evaluation Processes. Washington, DC: The National Academies Press. doi: 10.17226/27809.
×
Page 82

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

77   The intent of state DOT PEPs is to provide a consistent approach to evaluating the potential applications and performances of products, materials, engineered systems, and innovative tech- nologies to be used in construction projects and maintenance operations. The variation in the products and materials used requires a wide range of specifications, testing methods, payment approaches, and subject-matter expertise. A common practice among state DOTs is to use PEPs to functionally organize and categorize what can be a complex system of evaluation criteria, methods, and so forth. There is no standard approach to product evaluation, and this synthesis found state DOT PEPs that differ in methods and procedures, policies, staffing, support funding, and use of A/QPLs, among other variations. Product evaluation is seen as an exhaustive process at state DOTs. Time- lines from product application to approval were also widely ranging, not only according to a specific material or product, but also across state DOTs. The objective of this synthesis was to document the current state of practices, funding, poli- cies, management techniques, tools, and workflows of state DOT PEPs. The information col- lected included the following: • Policies and practices regarding product evaluation; • Structure of DOT A/QPLs; • Application requirements and rules (e.g., requirements on who may submit or evaluate and whether third parties can submit or evaluate); • Tools and workflows used in the product evaluation process (e.g., technical worksheets, typical submittal information, field or laboratory evaluations); • Categorizations of products submitted for evaluation (e.g., initial decision trees to determine whether a product fits in an existing A/QPL, is new or innovative, or is a material); • Process management requirements; • Typical evaluation timelines; • Relevant procurement policies; • Follow-up processes (e.g., periodic re-evaluation or performance issues); • Use of other states’ A/QPLs or subunits of government use of DOTs’ A/QPLs; and • DOT usage of the AASHTO NTPEP. Product evaluation information was collected through a literature review, a national survey questionnaire sent to state DOTs, and interviews with selected agencies for the development of case examples. Within the literature review, few academic resources were available regarding research on state DOT PEPs. The reference material available was more aligned with practitioner guidance, policies, and procedure manuals from state DOTs, NTPEP, and AASHTO resources. These resources, where available, proved useful in defining the general structure and workflows of state DOT PEPs and associated A/QPLs. The findings in the literature review aided in defining C H A P T E R 5 Summary of Findings

78 State DOT Product Evaluation Processes PEP and A/QPL processes and procedures, terminology understanding, and highlighted areas for investigation in developing and conducting the national survey. The national survey was conducted to capture the breadth of the state of practice regarding the use of state DOT PEPs. The survey collected 42 state DOT responses from its distribution to all 50 state DOTs and the Washington, DC, DOT. The resulting response rate was 82.3%. Of note from the survey, the main benefits reported in implementing PEPs at state DOTs were: • Adding value to agency’s operation (34 DOTs, 81%), • Enhancing consistency in evaluating products (33 DOTs, 79%), • Providing a more efficient process to evaluate products (28 DOTs, 67%), and • Reducing the time needed to evaluate and approve products (24 DOTs, 57%). These benefits are notable because the survey indicated that only 64% (27) of the respondents had formal state DOT PEPs in place, while the remaining respondents acknowledged using an informal PEP. Other information captured from the survey is summarized in the following subsections and combined with findings from the literature and case examples. The survey infor- mation highlighted several state DOTs as having practices of interest. These state DOTs were contacted for case example interviews to collect additional details regarding their PEPs. The case example interviews provided detailed insight into the DOT PEPs of the states selected for the case examples (Arizona, California, Florida, Michigan, New Hampshire, Ohio, and Tennessee). These case examples highlighted that enhanced efficiency from state DOT PEPs can be gained with well-organized computer-based systems, where workflows and notifications can be automated. The case examples also pointed to understanding the varied nature of state DOT PEPs. While many of the interviewees noted possible increases in uniformity, they also noted that variations in climate, traffic volume, project conditions, and other factors necessitate variations in approval criteria, performance criteria, testing, demonstrations, and so forth. 5.1 General Structure and Workflow of State DOT PEPs Regarding the structure and workflow of state DOT PEPs, a central element is the categories that group the products being evaluated. There is variation not only in the name and number of these categories, but also in how they are defined functionally at state DOTs. According to the survey results, 37 state DOTs (88%) stated that the categories they used were based on material types and specifications (e.g., asphalt, concrete, and sealants), and 24 state DOTs (57%) stated that the categorization of the products submitted for evaluation was based on the suitability of being grouped with existing approved or qualified products. The case example interviews noted that an important aspect of the categories used is their alignment with the specifications. Aligning material categories to construction or other specifications allows for easy referencing to field inspection and testing requirements as well as performance measurement and payment approaches. These categories additionally help to group subject-matter expertise and functional business units with potential interest in products and provide the structure for A/QPLs. While the categories seemingly vary from state to state, more than half of the state DOT survey respon- dents use the categories listed in Table 21. In terms of how products are submitted for evaluation, most of the state DOT survey respon- dents noted that product manufacturers and representatives are allowed to submit an application for product evaluation. The feedback from the case example interviews noted that the state DOT needs to see the value the product can provide to the DOT before evaluating it. Once an evalua- tion has begun, it is often assigned to an SME group or individual. According to the survey results, more than half of the respondents use designated SMEs within their agency, materials engineers, and state DOT PEP staff to evaluate products. The case example interviews and literature review provided the following general understanding of the workflow of a state DOT PEP. For products

Summary of Findings 79   that are for previously defined uses and are likely part of the A/QPL and align with an existing specification, state DOTs likely have existing testing and performance measures that must be met for approval. Additional information may also be reciprocated with the manufacturer through the evaluation process. Once an evaluation is complete, state DOTs choose from a range of values for approval status, including rejected, approved, conditionally approved, and more. Oregon DOT uses a formal product status approach that designates evaluated products among 10 different status levels. 5.2 Evaluating New and Innovative Products The evaluation process is more difficult and time-consuming for products received by state DOTs for evaluation that do not align with an existing specification. When a product was considered innovative or experimental, state DOTs noted there was less structure to follow for the evaluation and identification of an SME, which could be more of a challenge. Typically, these products took considerably more time to evaluate due to the additional time it took to determine the evaluation process and testing criteria for an innovative product. Also, if an innovative product is determined to provide value to the state DOT and would be beneficial to use, new or revised specifications have to be created before the product can be listed on the A/QPL and used for construction and maintenance operations. Development of evaluation criteria and specifications for a new or innovative product can take significant time to complete. However, the UP3 program exists through the AASHTO NTPEP, which assists DOTs in evaluating inno- vative and proprietary transportation products. 5.3 Product Evaluation Timelines Beyond structure and workflow, differences in state DOT PEPs were noted in evaluation time- lines, staffing, and funding support. Sixty-nine percent (35) of the survey respondents indicated that the timelines of product evaluations vary depending on the product. Collectively, the survey respondents noted that typical evaluation timelines range between 30 days (e.g., signage) and several years (e.g., corrosion inhibitors) depending on the product. Survey respondents and case example interviewees noted the assistance of systems that automate notifications, which aids Category Responding State DOTs (n = 42 ) Sealants 38 (91%) Concrete materials 38 (91%) Erosion control 37 (88%) Painting/coatings 35 (83%) Waterproofing 34 (81%) Asphalt materials 32 (76%) Drainage 31 (74%) Adhesives 31 (74%) Roadway safety 29 (69%) Traffic control 27 (64%) Work zone safety 27 (64%) Signage 27 (64%) Piping 26 (62%) Soils/geotechnical 25 (60%) Pedestrian safety 25 (60%) Maintenance 23 (55%) Steel 21 (50%) Table 21. Common categories of approved products for state DOT PEPs.

80 State DOT Product Evaluation Processes in reducing timelines. Regarding the state DOT PEP procedure manuals reviewed, a specific number of days is to be stated, such as receiving requested information from a manufacturer or supplier within 30 days, or the evaluation is terminated. The case examples also pointed to initial reviews having much shorter timelines (typically 30 to 90 days for initial review) as well as products that align with existing evaluation methods (i.e., products similar to those existing on A/QPLs). A growing number of state DOTs have begun streamlining processes, especially for those products that match up with an existing specification, and initial successes have included reduced evaluation time. 5.4 State DOT PEP Staffing and Funding With regard to staffing, one-third of the survey respondents (17) noted they rely on SMEs, materials engineers, and offices of materials and testing staff to oversee their PEPs. Sixteen state DOTs reported having dedicated full-time staff to operate and manage their state DOT PEP and associated A/QPLs. Fourteen of these 16 DOTs have one to five dedicated full-time staff members for their PEP, while the Pennsylvania DOT indicated it had 11–15 full-time staff members to operate and manage its PEP and associated A/QPL. However, Ohio DOT noted in its case example that one person, the NPE, facilitates the entire state DOT PEP, manages the QPL, and works with other internal staff and SMEs to complete evaluations. The case example interviewees noted that dedicated staff members aid in PEP efficiency; however, subject-matter expertise is required from other business units of the DOT to complete evaluations because limited PEP staffing at state DOTs does not allow for evaluations to be conducted by PEP staff exclusively. Only nine of the state DOT survey respondents had allocated annual funding to their PEP. The amount of annual funding reported ranged from $25,000 to $80,000, with one state allocating 1,000 labor hours as a resource. According to discussions in the case examples, other state DOTs reported using general funds and project funds to support product evaluations. Also, some state DOTs require the applying manufacturer to obtain and pay for the laboratory and field testing of a product. The case example interviewees also noted that successful state DOT PEPs are depen- dent on the support of the agency and assistance from internal staff. 5.5 Use of A/QPLs The literature review and case examples indicated that using A/QPLs is an effective practice in building an efficient state DOT PEP. In fact, the literature review and state DOT survey respon- dents noted that all state DOTs have an A/QPL in place. The use of A/QPLs provides readily available information regarding products approved for construction and maintenance use at a state DOT. This information is often of interest to manufacturers, contractors, and internal DOT staff. In addition, the case examples showed that state DOTs use an APL or QPL, but typically not both. However, Caltrans noted in its case example interview that it has a separate process in place for approved products and innovative products, and each has its own list. Furthermore, the structure of A/QPLs varied among the state DOT survey respondents and case examples interviewees. However, the A/QPL structure was typically based on categories that align with state DOT specifications and associated established testing and evaluation criteria. 5.6 Terms of Product Approval and Re-evaluation Once a product is approved or qualified, it is added to the A/QPL. State DOTs have varying timelines regarding the maintenance of the approval status. The survey responses indicated that 60% (25) keep approved products on their A/QPL until they are removed. Removal can

Summary of Findings 81   be due to noticeable performance issues in the field, a product not being used, or the changing of state DOT specifications. The case example interviews highlighted that product removal remains in place until a manufacturer makes changes to their products and resubmits the product for re-evaluation. The remaining survey respondents indicated a re-evaluation process must occur 1 to 3 years after approval (19% or eight) or 4 to 6 years after approval (21% or nine), as these state DOTs use a finite time limit for approved products before they need to be re-evaluated. Ohio DOT mentioned in its case example interview that recertification is required and due by January 1 every year. The case example interviewees also mentioned that re-evaluation would also occur if the manufacturer made changes to the product or if the state DOT modified an existing specification. Regarding the re-evaluation process, 24 of the state DOT survey respon- dents noted that re-evaluation follows the same process as a new product evaluation. 5.7 Electronic Systems and Tools to Support State DOT PEPs Various items are reviewed by state DOTs during product evaluations. The survey results indicated that 40 state DOTs review technical worksheets and submittal information for their product evaluations, 36 use laboratory testing, 32 use field testing, and 23 use pilot testing. Approximately three-fourths of the state DOT survey respondents have policies in place to evaluate products. State DOTs use their PEPs to increase efficiency in evaluating products and to implement various tools to achieve objectives. Electronic systems in the form of databases, cloud-based systems, and the like provide opportunities for efficiency in state DOT PEPs. Florida DOT uses a sophisticated PEP program called PATH for all product evaluations and automates portions of the evaluation, such as automatically adding an approved product to the APL. Systems with automated workflows and notifications are especially helpful but take significant initial effort to set up. This fact was noted by Florida DOT, who also noted that its PATH system is available to other state DOTs. Conversely, Ohio DOT uses an Excel spread- sheet to track and evaluate products in a less automated approach. 5.8 Use of NTPEP, DataMine, and UP3 As previously noted, assisting state DOT PEP efficiency is also the objective of NTPEP. The survey results showed that most state DOTs use the AASHTO NTPEP in some capacity. For product evaluation resources, typical elements of NTPEP that state DOTs use include attending the NTPEP Annual Meeting, evaluation processes for products, NTPEP product data on DataMine, and facility auditing of quality management systems and on-site inspec- tions of manufacturers or plants. The main benefits of using NTPEP found in the survey were the following: • Increases efficiency in product evaluation (34 DOTs, 81%), • Prevents duplication of evaluation effort (32 DOTs, 76%), • Produces lower evaluation costs through shared information (31 DOTs, 74%), and • Reduces the time needed to evaluate and approve a product for use (29 DOTs, 69%). The survey results showed that 21 state DOT respondents used other AASHTO NTPEP resources, such as UP3. The top three used components of UP3 were links to other state product lists, procedures, or resources; other state-approved proprietary products; and other state- patented products. These AASHTO resources were noted to aid in streamlining evaluations and providing efficiency in evaluation procedures.

82 State DOT Product Evaluation Processes 5.9 Use of Other State DOT PEPs and A/QPLs State DOTs also noted using the A/QPLs of other states. The case examples found that practice can be implemented but with care. Florida DOT noted that its climate is drastically different from other states, and that limits the information it is willing to use from other states’ A/QPLs. Ohio DOT mentioned that it rarely uses other state agencies’ A/QPLs and has only done so once in the past 5 years. 5.10 Identified Knowledge Practice Gaps While state DOT PEP challenges were not specifically collected in the survey, this synthesis highlighted the varied nature of state DOT PEPs, and formal state DOT PEPs were not imple- mented by all state DOTs. A gap exists for possible research and improvement in building con- sistency among state DOTs’ use of agency PEPs. While it is noted that climate, traffic needs, and many other factors may necessitate some level of variation between states, areas for improved standardization exist. As mentioned in the surveys and case examples, the AASHTO NTPEP is a helpful tool, and enhanced guidance from NTPEP on implementing a formal state DOT PEP would be beneficial. Additionally, NTPEP does not set the testing limit, nor does it approve products; the state DOT is responsible for both actions. Further guidance may be needed to determine the appropriate tolerances for testing products that can be used by state DOTs to evaluate and approve products for use. Another finding of the synthesis pointed to potential guidance needs in the implementation of electronic systems supporting state DOT PEPs. Florida DOT demonstrates a very robust PEP system (e.g., PATH), and findings from its case example interview could be a starting point from which to build guidance for other state DOTs seeking to develop a similar system. It is therefore recommended that research be proposed to develop guidance for increasing uniformity among state DOT PEP use that is inclusive of well-appointed electronic systems from which to operate and automate the state DOT PEP and associated A/QPLs. In many cases, when a product that fits with an existing specification is submitted for evalua- tion, the state DOT has a process in place to evaluate those products. However, state DOTs noted the difficulties they experience when a product that does not fit with an existing specification is submitted for evaluation, as these products are typically considered new or innovative products. Products that do not fit with a specification typically do not have any associated evaluation criteria, and this leaves the state DOT with the tasks of creating a reactive process to try to deter- mine a robust and consistent approach to evaluate a new or innovative product and developing a new specification to use the product in construction and maintenance operations. Therefore, guidance is needed to assist state DOTs in the evaluation of new and innovative products that do not align with existing specifications. With Buy America/Build America and the advent of environmental product declarations, state DOTs have more requirements for the types of products that can be used in transportation construction and maintenance. While a few state DOTs did acknowledge that they incorporate environmental reviews and Buy America/Build America eligibility requirements in the evalu- ation process, many transportation agencies are unsure how to handle these types of products. Therefore, guidance is needed on how to evaluate, approve, and designate products that include environmental product declarations and meet Buy America/Build America requirements.

Next: References »
State DOT Product Evaluation Processes Get This Book
×
 State DOT Product Evaluation Processes
Buy Paperback | $91.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The construction, maintenance, and operation of transportation infrastructure requires immense amounts of products and materials. New products, materials, engineered systems, and innovative technologies are presented to state departments of transportation (DOTs) by suppliers and manufacturers, as well as sometimes by contractors and internal DOT staff.

NCHRP Synthesis 616: State DOT Product Evaluation Processes, from TRB's National Cooperative Highway Research Program, documents current state DOT practices, funding, policies, management techniques, tools, and workflows of product evaluation processes.

Supplemental to the report is a dataset of various administrative documents from different state DOTs.

READ FREE ONLINE

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!