Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Appendix B CMF User Preferences and Practices B-1
Table of Contents 1. Summary ............................................................................................................................................... B-4 1.1 Selecting and Applying CMFs ........................................................................................................ B-4 1.2 CMF Quality ................................................................................................................................... B-4 1.3 Presentation of CMFs ..................................................................................................................... B-5 1.4 Second Edition of HSM and Part D ................................................................................................ B-5 2. Introduction ........................................................................................................................................... B-6 3. Questionnaire ........................................................................................................................................ B-6 3.1 Questionnaire Participant Demographics........................................................................................ B-7 4. Participant Responses to Questionnaire: Content Questions ................................................................ B-9 4.1 Question #4 ..................................................................................................................................... B-9 4.2 Question #5 ................................................................................................................................... B-13 4.3 Question #6 ................................................................................................................................... B-17 4.4 Question #7 ................................................................................................................................... B-19 4.5 Question #8. .................................................................................................................................. B-23 4.6 Question #9. .................................................................................................................................. B-26 4.7 Question #10 ................................................................................................................................. B-29 4.8 Question #11 ................................................................................................................................. B-35 5. Participant Responses to Questionnaire: CMF Scenarios ................................................................... B-38 5.1 Question #12 ................................................................................................................................. B-38 5.2 Question #13 ................................................................................................................................. B-41 6. Focus Group ........................................................................................................................................ B-44 6.1 Responses to Questions on Using CMFs ...................................................................................... B-45 6.2 Responses to Questions on Quality/Reliability of CMFs ............................................................. B-47 6.3 Responses to Questions on the Presentation of CMFs .................................................................. B-48 6.4 Responses to Questions on HSM Part D ....................................................................................... B-49 6.5 Responses to Other Topics ............................................................................................................ B-50 Table 1. Questionnaire Participants by Position ....................................................................................... B-7 Table 2. Questionnaire Participants by Agency ........................................................................................ B-8 Figure 1. Responses to Question 4 (entire group) ................................................................................... B-10 Figure 2. Responses to Question 4 by Position Type.............................................................................. B-12 Figure 3. Responses to Question #5 (entire group) ................................................................................. B-14 B-2
Figure 4. Responses to Question #5 by Position Type............................................................................ B-16 Figure 5. Question 6 Responses (entire group) ....................................................................................... B-17 Figure 6. Question 6 Responses by Position Type.................................................................................. B-18 Figure 7. Responses to Question 7 (entire group) ................................................................................... B-20 Figure 8. Responses to Question 7 by Position Type.............................................................................. B-22 Figure 9. Responses to Question 8 (entire group) ................................................................................... B-23 Figure 10. Responses to Question #8 by Position Type.......................................................................... B-25 Figure 11. Responses to Question #9 (entire group) ............................................................................... B-27 Figure 12. Responses to Question #9 by Position Type.......................................................................... B-28 Figure 13. Responses to Question 10 (entire group) ............................................................................... B-30 Figure 14. Reponses to Question 10 by Position Type ........................................................................... B-34 Figure 15. Responses to Question #11 (entire group) ............................................................................. B-35 Figure 16. Responses to Question #11 by Position Type........................................................................ B-38 Figure 17. Responses to Question #12 (entire group) ............................................................................. B-39 Figure 18. Responses to Question #12 by Position Type........................................................................ B-41 Figure 19. Responses to Question #13 (entire group) ............................................................................. B-42 Figure 20. Responses to Question #13 by Position Type........................................................................ B-43 B-3
1. SUMMARY This report provides the results from Task 3: Determine User Preferences and Practices. This task involved a questionnaire that was distributed in 2015 and a focus group of experienced users that was conducted by online meeting in February 2016. This section provides the summary on the findings. Further details are provided in the rest of the report. 1.1 Selecting and Applying CMFs Focus group participants were asked how they selected CMFs when presented with multiple options. Many said that they prioritized CMFs that had high star ratings on the CMF Clearinghouse. They also said that they prioritized CMFs that were developed in the same geographic region or that matched closely to the characteristics of the site of interest. There was also a concern expressed that some agencies (i.e., DOT districts) may often select a CMF that shows the greatest benefit even if it does not match site conditions or is highly rated. Questionnaire responses showed that the majority of people use both aggregate (total crash) and disaggregate (crash type and crash severity) CMFs. They typically apply these CMFs to modify historic crash values (the past 3 to 5 years of data) at the site of interest. This was true for all subgroups of individual position types, except researcher/teachers, where most people applied CMFs to expected crashes produced from safety performance functions. When presented with quiz-type scenarios, almost all questionnaire participants selected the correct specific CMF to use in the given scenario. However, when presented with a scenario in which they had to apply a given CMF correctly to a given set of crash types, 81% applied it correctly. Most of the incorrect responses were due to the application of the specific crash type CMF to the total crashes of the site. Several focus group participants responded that CMF availability is a factor in project implementation, since they strive for a data-driven approach and they desire to have a defensible backing for a decision to implement a safety treatment. If there is only a low star-rated CMF available for a given treatment on the CMF Clearinghouse, they may still use it, but they may not base high level projects on it. Most focus group participants indicated that they rarely or never used CMFunctions. They said this is due to the lack of CMFunctions that address the countermeasure of interest or match their site conditions. It is also due to the fact that CMFs (point factors) are simpler, easier to communicate, and easier to use at a planning level. 1.2 CMF Quality Questionnaire participants indicated that they use information on CMF quality most of the time or always. The majority of them indicated that they use this information to prioritize CMFs when selecting from multiple options. This agrees with the findings from the focus group, where many people mentioned the CMF star rating as one of their prime prioritization measures. Most focus group participants referred to CMF quality by star rating (from the CMF Clearinghouse) and said that they did not use the standard error value, though one person indicated that they would use standard errors to compute a range of potential CMF values. No one referred to using the HSM adjusted standard error as an indication of quality, with several even admitting that they really didnât understand it. Focus group participants generally considered three stars to be a minimum acceptable level for selecting a CMF, but several said that they would use a lower star-rated CMF if it was the only one B-4
available for the countermeasure of interest. They ideally preferred to see multiple high-rated CMFs in agreement (close to the same value) before using a CMF estimate. Regarding the criteria used to rate the quality of a CMF, focus group participants said that a CMF should be given a lower rating if it does not have much accompanying detail (such as the crash type to which it applies) or if it is a general CMF (for total crashes only). 1.3 Presentation of CMFs The majority of questionnaire participants responded that they would want to see all available CMFs presented for each specific condition (as opposed to only a single CMF). Written responses on the questionnaire generally indicated a preference for some kind of curated directory of CMFs, where there would be an indication of a recommended or practice-ready CMF while still listing all other available CMFs. Focus group participants were generally in favor of presenting all CMFs available (online), but a few indicated a desire for an HSM-designated CMF for each countermeasure. They also mentioned showing the CMF as a range of values to indicate how reliable it was. Focus group participants said that it was very important to be able to understand what specific kind of crash type was used in developing the CMF. Often the details or even the source document do not adequately explain how the researcher defined the crash type. They also said that it was important to have CMFs that addressed specific injury levels (i.e., KABCO). 1.4 Second Edition of HSM and Part D On a question about the future of HSM Part D, the highest percentage of questionnaire participants (50%) preferred that the HSM not include CMFs in Part D but provide a âstamp of approvalâ to CMFs in the CMF Clearinghouse that meet the HSM reliability requirements. This was followed by 33% who preferred to see the HSM continue the same format for Part D as the first edition (include CMFs that meet the HSM reliability requirements). However, the answers by individual position type were divided, with traffic safety engineers, traffic engineers, and researchers/teachers largely preferring that the HSM Part D not contain CMFs, and highway designers, transportation planners, and agency administrators preferring that the HSM continue in the same format as the first edition by including CMFs in Part D. Focus group participants were generally in favor of presenting CMFs online in the CMF Clearinghouse. They reasoned that the website was more accessible to the CMF user community, it is more up-to-date, and it provides the opportunity to dig deeper to see more details about any CMF. However, some also expressed a value in having selected CMFs appear in the published HSM, with references to the Clearinghouse for the list of all available CMFs. Regarding guidance in the HSM, questionnaire participants almost unanimously showed a desire for Part D to include guidance on how to apply CMFs in safety practice. About half of participants also wanted Part D to include guidance on how to develop CMFs in research studies. One focus group participant also mentioned a desire for guidance on selecting CMFs. B-5
2. INTRODUCTION A key consideration for the issue of CMFs and the next edition of the HSM is the preferences and practices of the CMF user community. CMF users include safety and design engineers, planners, researchers, academics, and other positions at national, state, and local levels. Since the product developed in this project will serve these people, it is critical to consider how they currently use CMFs and how best to present CMF information in the 2nd edition of the HSM. Under Task 3 of NCHRP Project 17-72, the research team conducted a broad online questionnaire and a focus group via online meeting. These efforts were intended to address topics such as how users access CMF information, how CMFs and related information should be presented, what information is important to users, and how users identify the best estimate of the effect when there are multiple CMFs from which they can choose. 3. QUESTIONNAIRE The team first developed a questionnaire to be issued broadly across the nation. The questions were held to a minimum and the answers were generally constrained to a multiple choice format to provide consistency and allow for easier compilation. The objective of the questionnaire was to obtain a national opinion on several key questions related to CMFs and the next edition of the HSM. The questionnaire was developed as a Google form and distributed electronically. The advertisement was done through emails to various committees and listservs. The following text was used to advertise the questionnaire: The project team for NCHRP 17-72, Update of Crash Modification Factors for the Highway Safety Manual, is conducting a survey of road safety professionals. The goal of NCHRP 17-72 is to identify which CMFs should be eligible for the Highway Safety Manual and how they should be presented. Your responses on this survey will provide valuable information about the ways in which road safety professionals are selecting and using CMFs in their daily work. Your input will affect which CMFs are selected for the HSM and how they will be presented. This survey contains 13 questions and should take about 10-12 minutes to complete. To take the survey, please go to: http://goo.gl/forms/3QAccT6seY The closing date for this survey is Friday, December 18, 2015 at 5:00pm Eastern. Please direct any questions about this survey to Daniel Carter (daniel_carter@unc.edu). Those interested in providing input on other aspects of the Highway Safety Manual should be aware that a closely related project is NCHRP 17-71, Proposed AASHTO Highway Safety Manual, Second Edition. In the near future, the project team for 17-71 will issue a survey that will pose questions on the usability, accessibility, and technical coverage of the next edition of the HSM. The following groups were asked to circulate this text and the links to their membership: B-6
⢠TRB Committee ANB25 (Highway Safety Performance) ⢠TRB Committee ANB20 (Safety Data Analysis and Evaluation) ⢠TRB Committee AHB65 (Operational Effects of Geometrics) ⢠TRB Committee AFB10 (Geometric Design) ⢠TRB Committee ANB10 Transportation Safety Management ⢠State Safety Engineer Listserv (NSEL) ⢠CMF Clearinghouse subscriber list ⢠National Associate of County Engineers ⢠Local Technical Assistance Program 3.1 Questionnaire Participant Demographics A total of 141 people responded to the questionnaire. The following sections describe the demographics of the questionnaire participants, broken down by job position type, agency type, and state of origin. Position The questionnaire participants were predominantly from a traffic safety engineering or traffic engineering position (Table 1). Researchers and faculty members were also represented significantly while all other positions were only represented by a few participants. Table 1. Questionnaire Participants by Position Position Type Number of Responses Percent of Responses Traffic safety engineering 51 36% Traffic engineering 37 26% Research and/or teaching 22 16% Highway design 10 7% Transportation planning 6 4% Other 6 4% Other engineering 5 4% Agency administration 4 3% Agency State departments of transportation and private consulting firms accounted for more than half the participants (Table 2). Universities, municipalities, and federal agencies accounted for most of the rest of the participants. B-7
Table 2. Questionnaire Participants by Agency Agency Type Number of Responses Percent of Responses State DOT 48 34% Private consulting firm 36 26% University 21 15% Municipality 15 11% Federal agency 13 9% Other 5 4% MPO 2 1% County 1 1% State Geographic diversity was well accomplished, with participants responding from 37 states (including Washington D.C.). Eleven responses were also received from participants outside the United States (specific country not specified). The states represented in the questionnaire responses are listed below: ⢠AL ⢠MD ⢠OR ⢠AR ⢠ME ⢠PA ⢠AZ ⢠MI ⢠SD ⢠CO ⢠MN ⢠TN ⢠CT ⢠MO ⢠TX ⢠FL ⢠MS ⢠UT ⢠GA ⢠MT ⢠VA ⢠IA ⢠NC ⢠VT ⢠IL ⢠NH ⢠WA ⢠IN ⢠NJ ⢠Washington, D.C. ⢠KS ⢠NY ⢠WI ⢠KY ⢠OH ⢠MA ⢠OK B-8
4. PARTICIPANT RESPONSES TO QUESTIONNAIRE: CONTENT QUESTIONS Following the demographic questions were eight questions about the participantâs opinion related to future directions for CMFs. The responses to these questions are summarized below. 4.1 Question #4 What kind of Crash Modification Factors (CMF) do you typically use? The objective of question #4 was to determine how many CMF users were working solely on the basis of aggregate (total crash) CMFs and how many used disaggregate (specific crash type or severity) CMFs regularly. The intention was to inform the priority of aggregate vs. disaggregate CMFs in the selection and inclusion process for the HSM 2nd edition. It also provides insight for CMF user training and guidance. The answer choices were as follows. Participants were allowed to choose more than one answer or write in their own response. ⢠CMFs for total crashes ⢠CMFs for specific crash types ⢠CMFs for specific crash severities ⢠I don't use CMFs When responses are viewed as a group for the entire audience, the majority of participants (60%) indicated that they commonly used both aggregate and disaggregate CMFs (Figure 1). The next highest percentage of users were those who used only aggregate (total crash) CMFs, but the percentage was much lower (12%). Between the two general disaggregate types of CMFs presented on the questionnaire (crash type and crash severity), crash type was shown to have greater use than crash severity, but not by much (8% vs. 3%). B-9
Figure 1. Responses to Question 4 (entire group) When responses are viewed individually for each type of job position, the same is generally true as for the entire group (Figure 2). The highest percentage of responses for most position types was for those selecting all three options (CMFs for total crashes, CMFs for specific crash types, and CMFs for specific crash severities). B-10
Traffic Safety Engineering Traffic Engineering Research and/or Teaching Highway Design B-11
Agency Administration Transportation Planning Other Engineering Other Figure 2. Responses to Question 4 by Position Type B-12
Takeaway point from Question 4 (what kind of CMFs do you typically use?): The majority of participants use both aggregate (total crash) and disaggregate (crash type and crash severity) CMFs. 4.2 Question #5 When addressing a specific segment or intersection, what crash value do you typically modify with the selected CMF? The answer choices were as follows. Participants were allowed to choose one answer or write in their own response. ⢠Historic crash data for the site (e.g., the past 3 to 5 years of crashes) ⢠Expected number of crashes estimated using safety performance functions (SPFs) ⢠Expected number of crashes estimated using SPFs and empirical Bayes adjustment In retrospect, this question should have been formatted as a âmulti selectâ answer, where a participant could select all answers that were true for him/her. The single select format forced participants to select only one answer. Presumably the participants selected the answer that was the most typical way they applied CMFs. A few of the responses that were written in indicated that some people would have selected more than one response if allowed. When viewed as an entire group as shown in Figure 3, the majority of participants (58%) said that they applied CMFs to modify the historic crash data for the site (e.g., the past 3 to 5 years of crashes). A much smaller percentage applied CMFs to the expected number of crashes estimated using SPFs and empirical Bayes adjustments (21%) or the expected number of crashes estimated using only SPFs (13%). Eight percent of people wrote in their own answers (listed below). B-13
Figure 3. Responses to Question #5 (entire group) Those who answered âOtherâ and wrote in their response provided the following descriptions of what crash value they modify with CMFs. It was clear that several people applied CMFs to combinations of the answers provided. ⢠Typically historic crash data (3 years) but testing using of SPFs since we just had state-specific SPFs for intersections developed. ⢠CMFs used as a measure with regard to how effectively a treatment addresses a collision type, we do not use the numerical reduction or societal value calculations that would often come next ⢠All three. The first answer is the most common but we use CMF in all three scenarios. ⢠Have done both A and C [Historic crash data for the site and expected number of crashes at the site estimated using safety performance functions] ⢠Roundabout research on intersections ⢠Any/all of the above ⢠The following are answered as if I would be using CMF's. ⢠Use the CMF to refer relatively to expected crash reduction. Eg. A "x"% reduction expected. ⢠Depends on available data; have encountered all 3 scenarios ⢠Both historic crash data & SPFs ⢠Mixture of the three The responses from most individual position types mirrored that of the entire group â that people were applying CMFs to modify the historic crash data (Figure 4). A minority in each category applied CMFs to some kind of expected number of crashes from safety performance functions. The exception was the âResearch/teachingâ position, where largest percentage of participants applied CMFs to expected number of crashes estimated using SPFs and empirical Bayes adjustment (36%), followed by those who applied CMFs to expected number of crashes estimated using only SPFs (32%). Only 27% of research/teaching participants applied CMFs to historic crash data. B-14
Traffic Safety Engineering Traffic Engineering Research and/or Teaching Highway Design B-15
Agency Administration Transportation Planning Other Engineering Other Figure 4. Responses to Question #5 by Position Type Takeaway point from Question 5 (what crash value do you typically modify with the selected CMF?): The majority of participants apply CMFs to modify historic crash values (the past 3 to 5 years of data). This was generally true for each category by position type, except researcher/teachers, B-16
where most people applied CMFs to expected crashes estimated using safety performance functions. 4.3 Question #6 When you use CMFs, how often do you use information about a CMFâs quality or reliability? The objective of question #6 was to determine how much priority is being placed by CMF users on the quality rating given by the HSM or CMF Clearinghouse. This question was asked in a general sense; participants were not asked to indicate which type of quality they used (HSM adjusted standard error, study-reported standard error, or CMF Clearinghouse star quality rating). The answer choices were as follows. Participants were allowed to choose one answer or write in their own response. ⢠Always ⢠Most of the time ⢠Some of the time ⢠Never As a combined group, the participants predominantly replied that they used quality or reliability information always (47%) or most of the time (28%) as shown in Figure 5. Figure 5. Question 6 Responses (entire group) The responses by individual position type showed that traffic safety engineers, traffic engineers, researchers/teachers, and highway designs predominantly also used CMF quality indications either always or most of the time (Figure 6). Agency administrators, transportation planners, and other engineering positions were less likely to use indications of CMF quality. Traffic Safety Engineering Traffic Engineering B-17
Research and/or Teaching Highway Design Agency Administration Transportation Planning Other Engineering Other Figure 6. Question 6 Responses by Position Type Takeaway point from Question 6 (how often do you use information about a CMFâs quality or reliability?): B-18
The majority of participants use CMF quality or reliability information always or most of the time. This was true for the combined group of all participants and for most of the individual position types. 4.4 Question #7 How do you use information about the quality or reliability of a CMF? The objective of question #7 was to determine whether there were any prevalent ways in which participants used information about CMF quality. This was intended to inform the decisions related to how best to provide and communicate information about CMF quality in the HSM. The answer choices were as follows. Participants were allowed to choose one answer or write in their own response. ⢠I will not use CMFs below a certain quality threshold ⢠I use it to prioritize CMFs if there are multiple CMFs to select from ⢠I donât use information about CMF quality/reliability In retrospect, this question should have been formatted as a âmulti selectâ answer, where a participant could select all answers that were true for him/her. The single select format forced participants to select only one answer. Presumably the participants selected the answer that was the dominant reason for them to use CMF quality information. A few of the responses that were written in indicated that some people would have selected more than one response if allowed. When viewed as an entire group, the majority of participants (62%) said they used information on CMF quality to prioritize CMFs when selecting from multiple options (Figure 7). Only 21% said that they used information on CMF quality as a threshold, below which they would not use a CMF. Nine percent wrote in their own answer (answers listed below), and 8% of participants said that they did not use information about CMF quality. B-19
Figure 7. Responses to Question 7 (entire group) Those who answered âOtherâ and wrote in their response provided the following descriptions of how they use information about CMF quality: ⢠Recommend engineering judgement ⢠It depends on what I am doing with the results of my estimate (e.g., design, ballpark est., etc.) ⢠We haven't started using CMFs yet. ⢠I check the data (samples, states, year), method, and standard error ⢠Generally distrustful of the factors ⢠I look to see if the CMF is applicable to my situation (e.g., looking at the background reports if available) ⢠Most of the time we will use CMF with higher star ratings. However, if studies with higher star ratings are not available, we use studies with lower star ratings or studies with no star ratings. ⢠I use it as a starting point to understand the quality of the statistics, I then look further at the study behind the CMF to determine whether the study adequately considered engineering/physics of the treatment to determine if the CMF/study is valid. ⢠Want at least 3 stars, if multiple over 3 stars, then prioritize ⢠I use it for both of the first two options ⢠Currently, we use only 5- and 4-star rated CMFs. ⢠I will evaluate CMFs based on their origin by geography and agency responsible for development Responses by type of position are shown below in Figure 8. For each type of position except Other and Other Engineering, the top reason for using CMF quality information was to prioritize CMFs. Traffic B-20
engineers, highway designers, and agency administrators each had a small portion of responses that said they did not use information about CMF quality. Traffic Safety Engineering Traffic Engineering Research and/or Teaching Highway Design B-21
Agency Administration Transportation Planning Other Engineering Other Figure 8. Responses to Question 7 by Position Type Takeaway point from Question 7 (How do you use information about the quality or reliability of a CMF?): The majority of participants said they used information on CMF quality to prioritize CMFs when selecting from multiple options. For most of the individual position types, this was also the top answer. B-22
4.5 Question #8. When determining the quality of a CMF, do you think there should be different judging criteria for CMFs for rare crash types (pedestrian, bicycle, head on, etc.)? The HSM and CMF Clearinghouse both employ methods of rating the quality of a CMF that incorporate a measure of statistical reliability. A CMF is judged to be less reliable if it is based on a small sample size or has a high standard error (more variability). However, many CMFs that address rare crash types, such as pedestrian, bicycle, or special target crash types, will be based on a small sample due to the nature of the crash type. The intention of question #8 was to determine whether CMF users would consider it a negative thing to have quality criteria that are different for these CMFs that address rare crash types. The answer choices were as follows. Participants were allowed to choose one answer. ⢠Yes ⢠No ⢠No opinion The majority of participants (58%) replied that there should be different judging criteria for CMFs for rare crash types (Figure 9). Many participants (24%) had no opinion on the matter. Seventeen percent said that there should not be different judging criteria. Figure 9. Responses to Question 8 (entire group) B-23
Within each position type category, the majority of responses (at least 50% in each one) indicated that there should be different judging criteria (Figure 10). Some groups had noticeable percentages of those who had no opinion on the matter (traffic engineering, highway design, other engineering). Traffic Safety Engineering Traffic Engineering Research and/or Teaching Highway Design B-24
Agency Administration Transportation Planning Other Engineering Other Figure 10. Responses to Question #8 by Position Type Takeaway point from Question 8 (Do you think there should be different judging criteria for CMFs for rare crash types?): The majority of participants said there should be different judging criteria for CMFs for rare crash types. This was true for the combined group of all participants and for all of the individual position types. B-25
4.6 Question #9. When seeking a CMF for a particular countermeasure, how do you prefer to see the CMFs presented to you? The first edition of the HSM presented only a single CMF for each countermeasure, according to a specific condition (crash type, severity, and site characteristics). By contrast, the design of the CMF Clearinghouse is such that the user can be presented with many CMFs for the same countermeasure under a specific condition (crash type, severity, and site characteristics). The objective of this question was to determine which format was preferred among CMF users. This was intended to direct the development/revision of the CMF inclusion criteria for the second edition of the HSM. The answer choices were as follows. Participants were allowed to choose one answer or write in their own response. ⢠Only one CMF presented for each specific condition (crash type, crash severity, site characteristics) ⢠All available (published) CMFs for each specific condition (crash type, crash severity, site characteristics) ⢠No opinion The majority of participants (68%) responded that they would want to see all available CMFs for each specific condition. Only 18% indicated that they would want to see only one CMF presented for each condition. Eight percent had no opinion, and 6% selected âotherâ. Those who selected âotherâ were given the opportunity to write in their own responses. Those responses are listed below. These written responses generally indicated a preference for some kind of curated directory of CMFs, where there would be an indication of a recommended or practice ready CMF while still listing all other available CMFs. ⢠One value should be published in the HSM and all available should be available on a website or other source. ⢠Would like to see harmonization of existing CMF's done by a statistician. ⢠All available above a set quality/reliability ⢠One general factor for an intersection by mode ⢠All good CMFs. Ones determined poor should be deleted when better ones are found. ⢠All available, with 1 recommended ⢠The highest rated CMF for each specific condition should be presented first followed by CMF for similar conditions in order of quality rating ⢠I'd like to see an added field that speaks to the "practice readiness" of the CMF B-26
Figure 11. Responses to Question #9 (entire group) All subgroups by individual position type indicated that the majority of participants would want to see all available CMFs for each condition. Those from traffic safety engineering, traffic engineering, and agency administration had the highest percentages of respondents that wanted to see only one CMF presented; at 22%, 24%, and 25%, respectively. Traffic Safety Engineering Traffic Engineering B-27
Research and/or Teaching Highway Design Agency Administration Transportation Planning Other Engineering Other Figure 12. Responses to Question #9 by Position Type B-28
Takeaway point from Question 9 (How do you prefer to see the CMFs presented to you?): The majority of participants responded that they would want to see all available CMFs for each specific condition. This was true for the combined group of all participants and for all of the individual position types. Written responses generally indicated a preference for some kind of curated directory of CMFs, where there would be an indication of a recommended or practice ready CMF while still listing all other available CMFs. 4.7 Question #10 How should the upcoming 2nd edition of the Highway Safety Manual address CMFs in Part D? Since 2010, there have been two major resources for CMFs. The 1st edition of the Highway Safety Manual provides CMFs in Part D of the printed manual and the CMF Clearinghouse provides CMFs online. At the time of this questionnaire, there had not been a decision reached about the format, content, or structure of Part D for the 2nd edition of the HSM. One of the major undecided points was whether the HSM should include CMFs in the published manual or simply rely on the CMF Clearinghouse to serve as the repository for CMFs. The objective of this question was to gauge the feeling of the CMF user community on this question. The answer choices were as follows. Participants were allowed to choose one answer. ⢠The HSM should continue to include CMFs in Part D that meet the HSM reliability requirements (same as in the 1st edition of the HSM). ⢠The HSM should not include CMFs in Part D but should provide an HSM âstamp of approvalâ to CMFs in the CMF Clearinghouse that meet the HSM reliability requirements. ⢠The HSM should not include CMFs in Part D nor provide an HSM âstamp of approvalâ to CMFs in the CMF Clearinghouse. ⢠No opinion As shown in Figure 13, half of the participants responded that they would prefer the HSM not to include CMFs in Part D but to provide a âstamp of approvalâ to CMFs in the CMF Clearinghouse that meet the HSM reliability requirements. A smaller percentage (33%) responded that the HSM should continue to include CMFs in Part D that meet the HSM reliability requirements (same as in the 1st edition of the HSM). B-29
Figure 13. Responses to Question 10 (entire group) The responses by individual position type showed a strong difference between two groups. Traffic safety engineers, traffic engineers, and researchers/teachers largely preferred that the HSM did not contain CMFs but provided the âstamp of approvalâ to CMFs listed on the CMF Clearinghouse. However, highway designers, transportation planners, and agency administrators preferred that the HSM continue in the same format as the first edition by including CMFs in Part D. B-30
Traffic Safety Engineering Traffic Engineering B-31
Research and/or Teaching Highway Design B-32
Agency Administration Transportation Planning B-33
Other Engineering Other Figure 14. Reponses to Question 10 by Position Type Takeaway point from Question 10 (How should the upcoming 2nd edition of the Highway Safety Manual address CMFs in Part D?): As an entire group, the highest percentage of participants (50%) preferred that the HSM not include CMFs in Part D but provide a âstamp of approvalâ to CMFs in the CMF Clearinghouse that meet the HSM reliability requirements. This was followed by 33% who preferred to see the HSM continue the same format for Part D as the first edition (include CMFs that meet the HSM reliability requirements). However, the answers by individual position type were divided, with traffic safety engineers, traffic engineers, and researchers/teachers largely preferring that the HSM did not contain CMFs, and highway designers, transportation planners, and agency administrators preferring that the HSM continue in the same format as the first edition by including CMFs in Part D. B-34
4.8 Question #11 What kind of guidance related to CMFs should be provided in Part D of the upcoming 2nd edition of the Highway Safety Manual? Continuing with the topic raised in Question #10 concerning the future of Part D in the HSM, this question focused on content that could be included in Part D apart from actual CMFs. It has been recognized in discussions among TRB and AASHTO groups that Part D could have guidance related to applying CMFs appropriately (for a practitioner audience), developing CMFs (for a researcher audience), or both. The objective of this question was to determine whether there is any clear opinion from the CMF users on this issue. The answer choices were as follows. Participants were allowed to choose multiple answers. ⢠Guidance on how to apply CMFs in safety practice ⢠Guidance on how to develop CMFs in research studies ⢠No guidance of this sort should be provided in Part D ⢠No opinion As shown in Figure 15, almost all participants selected an answer that included âguidance on how to apply CMFs in safety practice.â About half of these also selected âguidance on how to develop CMFs in research studies.â Aside from this, there were very small percentages of participants who selected only âguidance on how to develop CMFs in research studiesâ (3%), âno guidance of this sort should be provided in Part Dâ (1%), or âno opinionâ (1%). Figure 15. Responses to Question #11 (entire group) B-35
Figure 16 shows the responses by individual position type. Participants from all position types selected âguidance on how to apply CMFsâ. Most position types also had at least 50% of respondents that selected âguidance on how to develop CMFsâ as well. The exceptions were transportation planning, other engineering, and other, to whom there was less priority on developing CMFs. Traffic Safety Engineering Traffic Engineering B-36
Research and/or Teaching Highway Design Agency Administration Transportation Planning B-37
Other Engineering Other Figure 16. Responses to Question #11 by Position Type Takeaway point from Question 11 (What kind of guidance related to CMFs should be provided in Part D of the upcoming 2nd edition of the Highway Safety Manual?): The responses almost unanimously showed a desire for Part D to include guidance on how to apply CMFs in safety practice. About half of participants also wanted Part D to include guidance on how to develop CMFs in research studies. 5. PARTICIPANT RESPONSES TO QUESTIONNAIRE: CMF SCENARIOS One concern that is commonly raised in some discussions among TRB and AASHTO committees is whether those who use CMF resources (either the HSM or CMF Clearinghouse) are selecting and applying CMFs correctly. This questionnaire provided an opportunity to gain some understanding (albeit brief and simple) regarding the level of knowledge among CMF users. 5.1 Question #12 CMF Application Question The objective of this question was to determine if the participant knew how to apply a CMF correctly. The question presented a scenario in which the participant was asked to apply a CMF to the given crash values, as follows: An engineer is considering installing a countermeasure at an intersection. The countermeasure has a CMF of 0.70 for angle crashes. Without the countermeasure, the B-38
intersection has an expected annual crash frequency of 10 rear end crashes, 3 angle crashes, and 1 sideswipe crash, for a total of 14 expected total crashes per year. What is the expected total crashes per year after installing the countermeasure? Choose the best answer given the information provided. The answer choices were as follows: ⢠9.8 ⢠11.0 ⢠13.1 ⢠20.0 The correct answer was 13.1. That is, the CMF of 0.7 should be applied only to the expected 3 angle crashes, resulting in an expected number of angle crashes of 2.1 (i.e., 3 x 0.7 = 2.1). The other crashes would not be affected, so the final expected crash frequency would be 10 rear end crashes, 2.1 angle crashes, and 1 sideswipe crash for a total of 13.1 crashes per year. The majority of participants (81%) answered correctly (Figure 17). However, 14% of participants answered 9.8, showing that they applied the CMF of 0.7 to the entire crash total at the example site (14 crashes per year). This could be due to either a misreading of the question or, an incorrect understanding that a CMF for a specific crash type should only modify crashes of that crash type. Figure 17. Responses to Question #12 (entire group) Responses by individual position type (Figure 18) did not reveal any trends that differed significantly from the overall group response breakdown. B-39
Traffic Safety Engineering Traffic Engineering Research and/or Teaching Highway Design Agency Administration Transportation Planning Other Engineering Other B-40
Figure 18. Responses to Question #12 by Position Type Takeaway point from Question 12 (CMF application question): The majority of participants answered correctly. However, about a fifth of participants answered incorrectly. Most of the incorrect responses were due to the application of the specific crash type CMF to the total crashes of the site. 5.2 Question #13 CMF Selection Question The objective of this question was to determine if the participant knew how to select a CMF correctly. The question presented a scenario in which the participant was asked to select the appropriate CMF, as follows: A highway designer is asked to estimate the safety impact of installing a certain countermeasure on a rural, two-lane road where the safety impact of interest is all run-off-road crashes. Given the CMFs provided in the table below, which CMF should the designer use to estimate the safety impact as requested? CMF Crash Type Crash Severity Area Type 0.83 Run-off-road Fatal and Injury Rural 0.73 Run-off-road All Rural 0.99 All All Urban 0.93 All Fatal and Injury Rural The answer choices were as follows: ⢠0.83 ⢠0.73 ⢠0.99 ⢠0.93 The correct answer was 0.73. Almost all participants (95%) selected this answer. The next most frequently selected answer was 0.83, selected by 4% of participants. This reflects the few respondents who did not notice the distinction in the question about âallâ run-off-road crashes. B-41
Figure 19. Responses to Question #13 (entire group) Responses by individual position type (Figure 18) did not reveal any trends that differed significantly from the overall group response breakdown. Traffic Safety Engineering Traffic Engineering Research and/or Teaching Highway Design B-42
Agency Administration Transportation Planning Other Engineering Other Figure 20. Responses to Question #13 by Position Type Takeaway point from Question 13 (CMF selection question): Almost all participants selected the correct CMF that should be used in the scenario. B-43
6. FOCUS GROUP Determining how people select and use CMFs was a topic that needed more detail than could be obtained on a broad questionnaire that was generally restricted to multiple choice answers. For this reason, the research team convened a focus group consisting of traffic safety practitioners who were experienced CMF users. The focus group allowed the opportunity for group discussion and more in-depth responses to issues related to CMFs and the 2nd edition of the HSM. Given the need for a broad selection of participants across the country, the focus group was conducted via online meeting. The focus group was assembled by requesting participation from individuals known to be experienced in traffic safety. These were generally state traffic safety engineers or those in similar positions. The initial request was issued to 60 such people via email. The following text was issued to request participation in the focus group. Hi, The research team for NCHRP 17-72, Update of Crash Modification Factors for the Highway Safety Manual, would like to ask for your help. You may have recently taken a questionnaire that we issued, where we asked for input about how you select and use CMFs in your daily work. To follow up to that questionnaire, we would like to convene a focus group of âexperienced CMF usersâ. Experience users would be defined as those who regularly use CMFs in their work and would make use CMF resources such as the Highway Safety Manual or the FHWA CMF Clearinghouse. Your input would affect decisions being made for CMFs in the 2nd edition of the Highway Safety Manual. We would like to ask this focus group to discuss and give feedback to more detailed CMF-related questions than we could ask on the broadly issued questionnaire. This focus group would consist of 4 to 6 people and last no more than one hour. It would be conducted via webmeeting and take place on a day during the period of February 8 to February 26. Exact day and time will be determined via poll once the group has been identified and finalized. If you are interested in participating, please reply to this email or contact Daniel Carter at daniel_carter@unc.edu or (919-962-8720). Thank you. From the responses received, the team selected representatives from eight different states. The final selection was designed to reflect a geographic representation across the nation. Representatives from the following state agencies participated in the focus group: ⢠Mississippi DOT ⢠Minnesota DOT ⢠Louisiana DOT ⢠Kentucky DOT ⢠Utah DOT ⢠Georgia DOT ⢠Washington DOT ⢠Oklahoma DOT The focus group discussion addressed the following questions. The exact discussion on each question is provided in the following pages. B-44
⢠Questions on Using CMFs o If there are multiple CMFs available for a countermeasure, how do you select the CMF(s) that you are going to use? I.e., what factors direct your decision? o Is the availability of a CMF a factor in your decision to implement a new or innovative treatment? o Have you ever (or how often) used a Crash Modification Function? If not, is there a particular reason? ⢠Questions on Quality/Reliability of CMFs o How do you use indications of quality/reliability? What is your minimum acceptable quality of a CMF? o What criteria do you think should be used in judging the quality/reliability of a CMF? o Is standard error something you use (or know how to use)? If yes, do you understand what the HSM adjusted standard error is? ⢠Questions on the Presentation of CMFs o What kind of information (details) do you want to see accompanying a CMF? Which are the most useful to you? o How would you prefer to select CMFs - by general roadway classification or type or by individual roadway characteristics (e.g., area type, number of lanes, traffic volume)? ⢠Questions on HSM Part D o CMFs that qualify for inclusion in Part D could be presented in different ways. They could be included in the published Part D (as they were in the 1st edition) or, they could be presented only in the CMF Clearinghouse with some indication (i.e., âstamp of approvalâ) to show that they have met the HSM qualifications for inclusion, or both. From your experience in CMFs, which would you prefer and why? What pros and cons do you see for each presentation method? o What kind of CMF guidance would be useful to include in the HSM? o One of the options is to have the CMFs on the Clearinghouse but some of them will meet criteria for the HSM second edition. If thatâs the way things go, weâll have to decide how to indicate which ones met HSM approval. Do you have any thoughts about how to indicate that? 6.1 Responses to Questions on Using CMFs If there are multiple CMFs available for a countermeasure, how do you select the CMF(s) that you are going to use? I.e., what factors direct your decision? State 11: We use the CMF Clearinghouse especially for newer ideas that we donât have a lot of experience with in our state, so weâll read through those. When we search the site, we often have to read the reports or gather a lot of information about the research that was used to decide if the conditions were similar. State 3: To us the background research into the CMF is really important, assuming weâre not familiar with it. Sometimes that information is available, but sometimes it links to a resource that you have to go buy, which is a problem. State 7: Agreed that (above comment about free availability of resources) is a barrier. 1 The states are included in random order (e.g., State 1 does not necessarily refer to Mississippi DOT) B-45
Unknown: I like the ones with the higher stars⦠the ones that are vetted. When I pick something I try to find one that is locally researched versus national. When we do HSIP applications, they tend to pick the number that works best for them. There are a couple different ways it gets chosen. State 5: We do something similar (to the comment about choosing CMFs with higher stars, above). We go off the star ratings, we typically only use CMFs with a star rating with 3 or above and then try to find something close (in terms of site conditions) to the application weâre using it for. State 7: We do the same, I like them to have three stars or above. Beyond that we look at the application details (site conditions). But often thereâs pressure to use the lowest number. State 6: We try to focus on the 4 or 5 star. Depending on the availability we will consider going below that but we may try not to base gigantic projects on that because we donât have the faith in those lower rated ones. We follow a lot of the information that is there on the background info. We have gone as far as go down into the reports. We have more faith in the empirical based ones. Someone brought up that they look at the location. For some reason this comes to mind -- we were looking a couple weeks back and we shied away from things that are very focused on a particular route in a city or in a particular city. State 4: We usually pick higher star rated CMF among multiple. Beyond that, similar conditions or locally developed (as opposed to national level CMF). Our divisions pick the CMF that is most pleasing to project. Is the availability of a CMF a factor in your decision to implement a new or innovative treatment? State 7: The availability of a CMF does factor into our decisions and itâs not just having confidence that the treatment will be a benefit, but also having some basis for defending that decision. Thatâs more usually a funding concern. Occasionally we get arguments from people and itâs nice to have some solid research to bring out. State 3: Given that one of our requirements is that we take a data driven approach, we want to have research that shows a CMF has a safety benefit or we do it under an experimental waiver. Have you ever (or how often) used a Crash Modification Function? If not, is there a particular reason? State 7: We really havenât used them in Oklahoma just because of the lack of them. State 4: We havenât used many of them in Minnesota; itâs probably too complex State 6: In Mississippi we only really use the CMF. State 8: Functions are great, but itâs nice to have a CMF also for planning level. But it can make a big difference. Also, the CMF Clearinghouse should have more information or a better presentation of CMFunctions, something more similar to the HSM. If a function for particular countermeasure, Part D has a graph with the function. Need more of that in the Clearinghouse. State 3: Weâll use functions if we can find a good one. Meaning, one that applies to the situation at hand (has similar site conditions). B-46
6.2 Responses to Questions on Quality/Reliability of CMFs How do you use indications of quality/reliability? What is your minimum acceptable quality of a CMF? State 7: We prefer the higher star ratings but even the four and five star ones I may take that with a grain of salt if the result doesnât seem to be reasonable. For example, there was an edge line that showed a 50 percent decrease in crashes. I think it has since been removed. State 4: We donât look just at one 5 star report. One of the best things we can do if there is a slew of 4 or 5 stars is we can use as an aggregate. If they are all agreeing, thatâs a pretty good sign. State 6: I agree with what Derick just said (about looking for multiple agreeing CMFs). The repeatability just gives more credence. It would be nice if there was some objective process to combine CMFs to have a standard result. State 1: When you explain your choices to others, this helps with explaining how valid they may be. State 5: When thereâs a fairly new application without much study done, I remember finding only one CMF and it was only a two star rating, we did go ahead and use it. What criteria do you think should be used in judging the quality/reliability of a CMF? Unknown: I would suggest that there may be a minimum amount of info that may be required to receive star rating, such as standard error, crash type it applies to, etc. Unknown: Maybe CMFs that donât have all the info (like severity types) should just be down rated a few stars. Iâve just seen several cases where someone made a report and they used a CMF that didnât use any detail about crash type and they used it as a blanket specific for a highly severe crash type. State 7: Maybe CMFs that are only aggregate (total) should be rated lower. Iâve seen a case where someone took a total crash CMF and applied it to only specific severe crash type. Is standard error something you use (or know how to use)? If yes, do you understand what the HSM adjusted standard error is? State 7: We donât actually use standard error and there are a lot of studies where itâs not given in the Clearinghouse and there are cases where weâd want to modify the standard error because the conditions of the CMF donât match. Thereâs not a standard for adjusting the standard error. State 2: I never really knew the difference between standard error and adjusted standard error. State 6: We donât use adjusted standard error. State 3: We never really understood what it meant either. Will use standard error quantitatively. Often theyâre missing. There are other key fundamental things that are often missing, like what facility types it applies to. The star rating doesnât give a large enough latitude to really vet which ones are the good ones and which ones arenât. I think the conversation is going to turn to how we qualify how we determine what goes into the HSM. B-47
State 8: We look for countermeasures and if thereâs low stars we run a sensitivity analysis and if itâs within range weâre fairly comfortable using it. Context is the most important to us. Weâll produce a confidence range to evaluate the potential benefit/cost result. 6.3 Responses to Questions on the Presentation of CMFs What kind of information (details) do you want to see accompanying a CMF? Which are the most useful to you? State 4: The biggest thing for me is does the CMF youâre using have the same criteria as the site that is trying to get funded? Is it matching the environment? For example, we wouldnât use a CMF for continuous freeway lighting if weâre dealing with lighting an intersection. State 7: Basically the same, how well the study matches the conditions and the target crash type. Thatâs often missing â a good definition of the type of crashes that were targeted. If it listed say angle crashes, it doesnât tell us what that means for that particular study. It might be different than how we describe in Oklahoma. State 5: Thatâs something we run into [unclear crash type]. Because even the police officers donât always know what something is when they report it. There are issues from the source data and differences in the way they report a type of a crash. State 6: We donât have certain characteristics were looking for. Weâre a small group so well work together and ask questions about best approach and best CMF to use. Nothing set in stone. State 2: The only other issue that I can think of is the injury â doesnât define what injury code an injury is. That comes into play with our SHSP. State 7: Weâve had the same problem in Oklahoma. Unknown: CMFs could be broken down by severity level. How would you prefer to select CMFs - by general roadway classification or type or by individual roadway characteristics (e.g., area type, number of lanes, traffic volume)? Unknown: We would like to specify what the intersection looks like and it crunches out what you should use. Rather than looking at it from our perspective, look at it the other way. We would like the HSM or CMF Clearinghouse to serve as idea generation for what safety treatments could be done. B-48
6.4 Responses to Questions on HSM Part D CMFs that qualify for inclusion in Part D could be presented in different ways. They could be included in the published Part D (as they were in the 1st edition) or, they could be presented only in the CMF Clearinghouse with some indication (i.e., âstamp of approvalâ) to show that they have met the HSM qualifications for inclusion, or both. From your experience in CMFs, which would you prefer and why? What pros and cons do you see for each presentation method? State 5: I would like to see one particular one in the HSM and let the other ones be on the Clearinghouse website. The HSM will have to be definitive, if you want something for various different situations you can look on the website for those particular situations. State 7: That would be useful to us as long as the HSM ones are reasonable good quality numbers. But thereâd have to be some sort of weighted average. State 4: I wouldnât have any opposition to it being printed again, but I do like the website better. Itâs more fluid and up to date [than the HSM]. All HSIP funding applications almost always have CMFs that come from the CMF Clearinghouse. A manual that very few people have, they donât go digging around in there. State 1: I really like the idea of having a parking spot for those, but I also like the idea of having them on the website. For folks in the field, a design engineer is just going to skim and not really understand. If someoneâs not trained properly, you think of a cookbook. On a website you can really dig in and get more information. Doesnât need a direct relationship, but make it known that these are not the only CMFs just because theyâre printed in a book. State 7: Maybe it would be appropriate for the book to have a confidence interval in the range to make it clear that thereâs a potential variable there. Need to make it clear that the CMF is an estimate, not exact value. State 4: Explain that CMF is the middle of the range. If you start explaining confidence interval, you lose people. State 2: Part D has a risk of becoming outdated if you have it in there. Maybe include the methodology to use it and guidance, but put the actual CMFs themselves online. State 3: Iâd like to second that. I would speculate that 99 percent of practitioners would go to the website, not the manual. State 8: We do use the CMF Clearinghouse more because itâs more up to date. State 7: CMF Clearinghouse is more transparent and more in it than could be in printed manual. B-49
6.5 Responses to Other Topics What kind of CMF guidance would be useful to include in the HSM? State 6: I think some of that would be good to have in the HSM or just out there, some general guidance on how you should be picking CMFs. I struggle a lot when I try to explain to other counties on how to do that. Sometimes it feels like more of an art than a science. One of the options is to have the CMFs on the Clearinghouse but some of them will meet criteria for the HSM second edition. If thatâs the way things go, weâll have to decide how to indicate which ones met HSM approval. Do you have any thoughts about how to indicate that? State 7: An up and down flag is perfectly sufficient B-50
