Precision Estimates for AASHTO Test Method T308 and the Test Methods for Performance-Graded Asphalt Binder in AASHTO Specification M320 (2005)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

4 CHAPTER 2: ANALYSIS TECHNIQUE 2.1 TECHNIQUE OVERVIEW The analysis method used to determine precision estimates for this study is designed to determine robust estimates of precision representative, as much as possible, of testing performed in accordance with the test standards. The desire is to obtain estimates that will compare favorably to those that might be obtained from a strictly controlled inter-laboratory study. A literature survey was conducted to investigate methods applicable to the AMRL PSP data. Where applicable, sources used for the development of the analysis technique will be referenced in the following sections. The method is designed to extract the core of the data from the data sets and then to analyze that core to determine repeatability and reproducibility precision estimates. It is these data that stand the best chance of representing testing performed in conformance with each of the test methods. The AMRL Proficiency Sample Program is based on the testing of two samples of the same material having nearly identical, but not necessarily exactly identical, test properties. This type of program is described by Arni, Crandall and Blaine, and Youden [7,8,9]. One test is performed on each of the samples. This type of program provides two independent test results from each laboratory and allows for the evaluation of both within-laboratory and between-laboratory performance and for determining corresponding estimates of precision. The within-laboratory data are obtained under repeatability conditions by specifying the test method and by having testing in each laboratory performed by a single operator using the same equipment in a short period of time. The between-laboratory data are obtained under reproducibility conditions with different operators in different laboratories using different equipment. The number of participants in the AMRL program is sufficiently large enough to ensure a statistically sound basis for determination of estimates of precision for standard test methods among laboratories using various types of equipment [10]. For most of the standards under consideration of this study, the number of participants is on the order of several hundred. Even for those tests for which the populations are smaller, the number of participants is sufficiently large (in the range of thirty to fifty) for a sound inter-laboratory study [10]. Due to the relatively large number of participants in the PSP it is expected that the original data obtained during a round of testing contains a significant number of test results submitted from laboratories whose testing procedures may not be in conformance to the test standards or whose equipment may not meet the requirements specified in the test methods. The analysis technique is an attempt to identify and eliminate those test results prior to determining the precision estimates. The analysis method used in this study employs procedures to identify invalid data and outlying data by extrapolating to cutoff points in the extremes of a data set based on the spread of the most reliable data near the center (or median) of the data set. Precision estimates are then determined from the “core” of reliable data that remains after invalid data and outlying data are removed.

5 As shown in Figure 1, the analysis technique employs a four step process. First, null responses and unpaired data (i.e. where laboratories did not submit results for both samples, x and y) are removed (Section 2.2.1). Second, invalid data are removed (Section 2.2.2). Third, outliers are removed (Section 2.2.3). Forth and finally, traditional standard deviation-type analyses are performed on the remaining core data to obtain estimates of repeatability and reproducibility precision (Section 2.2.4). The first three steps are applied to the between-laboratory results for each of the two samples and also to the within-laboratory results. The criteria in the first three steps used for the elimination process help to assure that the results for each of the test samples contain data representative of testing performed in conformance with the test method. The within-laboratory, or repeatability, data to which the criteria are applied are numerically equal to the difference between the two results submitted, one for each of the two test samples, by each laboratory. The difference between the two results is adjusted for any difference between the median values for each of the two samples according to the following equation [9]: Repeatability data point: ( ) ( )medmediii yxyxr −−−= for i = 1 to n (Equation 1) Where: =n number of laboratories =ix result from laboratory ‘i’ on sample ‘x’, =iy result from laboratory ‘i’ on sample ‘y’, =medx median of test results from all laboratories on sample x, =medy median of test results from all laboratories on sample y. 2.2 STEPS OF ANALYSIS 2.2.1 REMOVE UNPAIRED AND NULL DATA (STEP 1) The analysis technique will not work for null and unpaired data. As a result, all null and unpaired data from the x and y data sets are removed prior to being analyzed. Unpaired data result from participating laboratories that submit results for only one of the two samples. Null responses occur from laboratories that receive the PSP samples but do not submit any testing results.

6 ORIGINAL PROFICIENCY DATA PAIRED DATA r paired data set x paired data set y paired data set x valid data set y valid data set VALID DATA CORE DATA x core data set y core data set r valid data set r core data set SRx Sr SRy Unpaired Data Unpaired Data Invalid Data Unpaired Data Invalid Data Outliers Unpaired and null data are identified. Laboratories submitting unpaired or null data are removed. Outliers are identified from valid data sets x, y, and r. Laboratories submitting outlying data are removed from further analysis. Using the data from core data sets x and y, reproducibility estimates sRx and sRy are determined. Repeatability estimate sr is determined using the data from core data set r. Invalid data are identified from paired data sets x, y, and r. Laboratories submitting invalid data are removed from further analysis. Step 1 Step 2 Step 3 Step 4 Figure 1 – Visual Representation of Analysis Technique

7 2.2.2 DETERMINE INVALID DATA (STEP 2) Invalid data are defined as data falling above and below the values IU and IL, respectively; using Equations 2 and 3 based on Hoaglin et al [11,12]. See Appendix A for a more detailed description. ( )( )7575 555.1 RIRII UU += = upper limit for invalid data (Equation 2) ( )( )7575 555.1 RIRII LL += = lower limit for invalid data (Equation 3) Where: RI75 = RI75U – RI75L = the range of the inner 75% of data RI75U = 87.5th percentile point of data (upper extent of the range of the inner 75 percent of all paired data) RI75L = 12.5th percentile point of data (lower extent of the range of the inner 75 percent of all paired data) Data determined to be invalid (i.e. falling beyond IU and IL) are beyond the equivalent of 4.725 standard deviations from the median value [11,12]. Even though this robust technique is applicable to Gaussian and non-Gaussian data [13], for normally distributed data, the probability is approximately 0.0000024 that data lying beyond IU and IL should be included in the population of results [11]. Any laboratory submitting invalid data is eliminated from further analysis. Figure 2 below gives a graphical representation of the location of the upper and lower limits for invalid data. Appendix B gives a step-by-step example of how the equations are used to identify invalid data. INNER 75% RI75 DATA INVALID INVALID DATA 1.555 x RI75 1.555 x RI75 Using the Inner 75% to Identify Invalid Data RI75L RI75UIL IU Figure 2 – Graphical Representation of Using the Inner 75% of Data to Determine Invalid Data

8 2.2.3 DETERMINE OUTLIERS (STEP 3) Outliers are defined as data falling above and below the values OU and OL, respectively; using Equations 4 and 5 based on Hoaglin et al [11,12]. See Appendix A for a more detailed description. OU = RI75U *+ (0.674(RI75*)) = upper limit for outlying data (Equation 4) OL = RI75L* – (0.674(RI75*)) = lower limit for outlying data (Equation 5) Where: RI75* = RI75U*– RI75L* = the range of the inner 75% of data without invalid data RI75U* = revised 87.5th percentile point of valid data (i.e. upper extent of the inner 75 percent of data remaining after the removal of invalid data) RI75L* = revised 12.5th percentile point of valid data (i.e. lower extent of the inner 75 percent of data remaining after the removal of invalid data) Using the method described above, outliers fall beyond the equivalent of 2.7 standard deviations from the median value [11,12]. Similar to the method for determining invalid data, this technique is also applicable to Gaussian and non-Gaussian types of distributions [13]. However, the probability is approximately 0.007 [11] that data lying beyond the designated limits, OU and OL, should be included in the population of results for normally distributed data. Any laboratory submitting outlying results is eliminated from further analysis. Figure 3 below gives a graphical representation of the location of the upper and lower limits for outliers. Appendix B gives a step-by-step example of how the equations are used to identify outlying data. RI75* INNER 75% 0.674 x RI75* 0.674 x RI75* Using the Revised Inner 75% to Identify Outlying Data OUTLYING DATADATA OUTLYING RI75L* RI75U*OL OU Figure 3 – Graphical Representation of Using the Inner 75% of Data to Identify Outliers

9 2.2.4 ANALYSIS OF CORE DATA (STEP 4) Once laboratories submitting either invalid or outlying data are eliminated, traditional standard deviation-type analyses are performed on the remaining data to determine repeatability and reproducibility precision estimates. Since the two samples comprising a pair of AMRL proficiency samples are not identical in many cases, sr (repeatability) estimates are obtained in the manner described by Youden [9] by applying the following equation to the paired data: ( ) ( )[ ] ( )12 2 − −−−= ∑ n yxyx s iir (Equation 6) Where: rs = repeatability estimate ix = laboratory test result from the odd number sample of a pair iy = laboratory test result from the even number sample of a pair x = average of all xi y = average of all yi n = number of laboratories This equation removes any actual differences in the samples and allows the paired test results to be treated as replicates. Reproducibility estimates, sRx and sRy, are obtained independently for each of the two samples by applying the following equations for determining the sample standard deviations [3]. ( )     − −∑= 1 2 n xxs iRx (Equation 7) ( )     − −∑= 1 2 n yys iRy (Equation 8) Where: Rxs = reproducibility estimate for odd number sample pair Rys = reproducibility estimate for even number sample pair ix = laboratory test result from the odd number sample of a pair iy = laboratory test result from the even number sample of a pair x = average of all xi y = average of all yi n = number of laboratories

10 2.3 Check for Normality According to ASTM E 177, the multiplier for determining the difference two-sigma (d2s) limits assumes an underlying normal distribution. To ensure the assumption of normality is a correct assumption, a comparison was made of the average 95% limits, for the differences between two results, by count to the pooled d2s limits for each of the 12 data groupings. The summary tables comparing the average 95% limits by count and the pooled d2s limits can be found in Appendix D. The Coefficient of Correlation from normal probability plotting can also be found in Appendix D.