SECTION 2

Analysis Approach

This section describes the data collection, analysis, and modeling methods used in the analysis as they are linked to the evaluation objectives listed in the previous section. The first two objectives focused on evaluating the process by which the ABQT technology was conceived and funded under the FHWA R&T Program, as well as the execution of research and commercialization efforts. The process evaluation was primarily a qualitative assessment based on interviews with FHWA staff and industry SMEs.

In addition to industry SMEs, the evaluation team interviewed knowledgeable individuals at FHWA and LTI. The bulk of the physical research and prototype development was conducted under FHWA’s CRADA with its partner LTI. The evaluation team investigated the timeline of investment and decision-making to understand how the decision to fund ABQT research was made and what criteria were used in the decision process.

To evaluate the business case for and barriers to adoption of the ABQT (Objective 3), SMEs were asked qualitative questions about how they see the ABQT fitting into their business or the industry at large. This feedback provided important insights for the potential commercialization of the ABQT.

The fourth objective was to quantify the prospective benefits and costs of ABQT development and adoption. Research addressing the fourth objective drew information from a range of sources. Using input from SMEs, the analysis estimated potential net benefits (i.e., benefits minus costs) in terms of changes in road maintenance and changes in testing costs resulting from adopting the ABQT. The final objective was to develop an overall assessment and evaluation of the ABQT program built on all the data collection, analysis, and modeling.

2.1 Interviews

The evaluation team used a combination of desk research and expert recommendations to identify SMEs capable of providing relevant insights into the impact of the ABQT on the highway and asphalt supply chain. The team conducted 35 interviews with experts from organizations that fall into six types of stakeholder groups in the ecosystem of potential users and beneficiaries of the ABQT device. Appendix B contains the final interview guide used to conduct the interviews. Table 2-1 provides a summary of the number of interviews by stakeholder group.

Table 2-1. Number of SME interviews by stakeholder group.

2.2 Adoption Scenarios for Benefit Analysis

Because of uncertainty related to which parts of the binder supply chain will likely adopt the ABQT device, the analysis modeled three adoption scenarios:

• Scenario 1: DOT testing labs adopt the ABQT as a screening tool. The level of full comprehensive testing is not changed, but samples are prioritized for full testing.
• Scenario 2: In addition to being adopted by DOTs, the ABQT is adopted by asphalt binder suppliers as a quick screening tool to be used at terminals to screen binders before shipping.
• Scenario 3: Mix plants also adopt the ABQT (in addition to DOTs and suppliers) as a quick screening tool to verify binder specifications upon delivery.

These three scenarios modeled the adoption incentives and business cases across the three major players in the binder quality supply chain. The scenarios were modeled as additive (stacked) with increasing adoption of the ABQT throughout the supply chain. Based on the interviews, DOT adoption (Scenario 1) is viewed as the most likely scenario. The probability of ABQT adoption by asphalt suppliers in addition to DOTs (Scenario 2) and adoption by DOTs, suppliers, and mix plants (Scenario 3) was considered less likely. In general, the private sector would not be likely to proceed with significant adoption of the ABQT without state DOT testing labs adopting it first.

The analysis described in Section 6 presents additional detail on the adoption scenarios.

2.3 Probabilistic Model Used to Quantify Benefits

A probabilistic model was used to estimate the potential impact that adoption of the ABQT device could have on maintenance and replacement costs. Reduced road maintenance and replacement costs are the core of the monetized benefits calculated in the analysis. The model calculated how using the ABQT device as a screening tool can increase the probability of identifying out-of-specification binder and, hence, reduce the probability of using a subpar binder, which leads to decreased maintenance and replacement costs (this is described in more detail in Section 6).

SMEs confirmed that this probabilistic modeling approach was frequently employed by owner agencies to assess the frequency and type of maintenance activities they conduct. However, no owner agency was aware that this approach had ever been used to monetize cost savings from different testing or inspection activities.

2.4 Follow-Up Survey to Finalize Key Parameters of the Model

After the interviews, a subset of SMEs took an online survey to review the values of the key parameters used in the model. This semi-Delphi process shared back with state DOTs the average values for parameters obtained from the interviews, such as

• Share of submitted samples tested and not tested,
• The average out-of-specification failure rate for the samples tested, and
• The average delay/lag between the time a sample is received and it is tested.

Appendix C contains the survey questions. SME respondents were asked to comment on whether they thought the average/typical values identified during the interviews were representative of the industry as a whole (too high, too low, about right). This gave SMEs who did not feel comfortable citing numerical values in the interviews a chance to provide input to the parameter values used in the final analysis.

The electronic survey was sent to the 13 state DOT SMEs interviewed. Five completed the survey. All of the surveys received were supportive of the key parameter values that were used in the model. In addition, the key parameter values were vetted with the TFPE-01 project panel, which reviewed interim report documents. Based on the panel’s comments, minor adjustments were made. Hence, the evaluation team determined that these values are reasonably applicable for most DOTs.