Improving Safety Culture in Public Transportation (2015)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

44 C H A P T E R 5 Introduction Assessing the state of safety culture in an organization is as important as measuring on-time performance or equipment reliability. In order to improve the state of safety culture in an organization, it is essential to establish a baseline and then periodically measure progress. It is therefore the research team’s opinion that transit agencies need to have the expertise neces- sary to assess and reassess their safety cultures. This chapter will discuss: • Methods for assessing safety culture, • Assessment planning, and • Survey design. The chapter will then: • Provide an example of the development of a survey, • Discuss the testing of that survey at two transit agencies, • Provide examples of how such results can be used, and • Explain the process for survey validation and reliability testing. Finally, the chapter will present the team’s conclusions about safety culture assessment. Methods for Assessing Safety Culture Various individual methods may be used to assess an orga- nization’s safety culture. Among the most common and fre- quently used are direct observation, interviews, surveys, focus groups, and performance indicator tracking. Safety culture assessments require a significant investment of resources to gather information and, sometimes, a far greater investment to remedy problems found. Such full assessments are therefore most often launched by a transit agency board and/or a CEO committed to the improvement of the state of safety culture in the organization. After making the decision to proceed, the first and most important step is to engage manage- ment and union leadership teams to explain the purpose and mechanics of the assessment. Supervisors and hourly workers need to be similarly engaged and involved. Direct Observation Direct observations of workplace behavior may provide objective information regarding the aspects of safety culture, including effectiveness of training, management, account- ability, and behavior expectations. Direct observation of employees at work can provide valuable information on employee involvement, attitude, and willingness to confront perceived unsafe behavior. The observer can watch the culture at work and can confirm results obtained from interviews and surveys. Observations can provide new information on cultural phenomena. However, observations—even if scored on a checklist—cannot be precisely quantified, and there is always the risk of overgeneralization from too few observations (EFCOG/DOE, 2009). Conducting sufficient observations to produce an accurate assessment of the state of safety culture in an organization of any size is necessarily time-consuming and expensive. Interviews Interviews can also be used to develop information on the state of safety culture in an organization. One advantage of interviews is that the respondents are not limited by the word- ing or structure of a written survey. The greater flexibility inher- ent in an interview allows the interviewer to drill down until an issue or problem is fully clarified and all ambiguity resolved. As with direct observation, generalization of findings to the entire organization is risky if the interviews are limited in num- ber and if interviewees do not accurately represent the overall Methods/Tools for Assessing Safety Culture

45 a response mandatory; offering completion incentives, such as payment of overtime to operators taking the survey; and following up with potential respondents. Perhaps the most influential step that can be taken, however, is to make taking the survey as easy and convenient as possible for all employees. Maintenance employees can be assembled in a lunchroom or a conference room prior to the start of work, operators can complete a web-based survey on a bank of computers in the dispatcher’s office, and so on. Performance Indicator Tracking Performance indicator tracking can also be used to provide information on the state of an organization’s safety culture. While there appears to be no single indicator sufficient for accurate measurement of the overall state of safety culture in an organization, monitoring trends in leading performance indicators as a function of time may provide insight into the strengths and weaknesses of a particular safety culture and may show the direction in which the state of safety culture is going. For example, if the number of signal violations sud- denly doubles in an organization, that could constitute a red flag with respect to the state of safety culture. Assessment Planning Assessments must be carefully planned and organized. Essential elements of such a plan include who is to be inter- viewed and how many focus groups covering which topics need to be organized. Based on the input from the interviews and focus groups, a customized survey could then be con- structed to focus on the problems and issues specific to a par- ticular transit agency. The survey can then be administered, the results tabulated, and perceived problems and issues either be confirmed or not. A series of interviews and focus groups could then be conducted in order to provide more specific information on the issues identified. Management can then formulate a plan of action to deal with the highest-priority problems and issues. The specifics of this plan and periodic updates on progress need to be distributed to all employees. An abbreviated approach involves the use of a standard survey to provide general information followed by a series of interviews and focus groups to develop specific and in- depth information on issues emerging from the survey. This combination has the advantage of using the survey to identify the issues as perceived by employees at all levels of the orga- nization in the most economical manner possible and then concentrating the interviews and focus groups on obtaining detailed information on those issues. A plan of action could be drawn up based on the results. This is a more efficient pro- cess. However, as standard surveys are proprietary, using them can still be fairly expensive, although not nearly as expensive workforce. As with direct observation, interviews can be time- consuming and expensive. Interviews can also be used to pro- vide input to the design of a safety culture survey or to explore safety culture issues in greater depth that have emerged from a survey. Focus Groups Focus groups are more efficient but less flexible than indi- vidual interviews in assessing safety culture. One interviewer can elicit the views of multiple employees in a single focus group. Valid focus groups are conducted by skilled, experi- enced facilitators who bring all participants into the discus- sion. Well-designed focus groups can provide great sources of insight beyond surveys. Flexibility is somewhat reduced because the facilitator generally uses a set of prepared ques- tions to bring basic organization and direction to the discus- sion. A significant downside to focus groups is that, without a skilled moderator, a minority of participants can dominate a discussion and provide input that might differ significantly from the results from individual interviews of each member of the group. Ideally, focus groups allow the facilitator to see a microcosm of the organization’s culture in real time as partici- pants talk among themselves, influence one another’s responses and insights, and compare ideas. Surveys Surveys are useful and effective tools to assess safety cul- ture. They have the advantage of greater efficiency over other assessment methods. The views of large numbers of employ- ees can be obtained with a lower expenditure of resources than required for direct observation, interviews, and focus groups. Individuals may also feel more comfortable taking a survey because their individual responses are anonymous and treated as confidential. The use of cross-tabulation of survey results also offers insights into similarities and differences among subcultures—operators, mechanics, managers, and so forth— within a transit agency. A significant limitation is that surveys are somewhat inflexible: the only information that can be elic- ited is the direct response to each specific question posed. The elicitation of subtle distinctions is difficult to obtain from a survey. Also, the outcomes can be affected by poor or slanted construction of the questions. A frequent problem that occurs with surveys is a low response rate. This is particularly true for bus and train oper- ators, whose work schedules can span 24 hours per day and 7 days per week. If the response rate is very low, the results will not be representative of the survey population. The prob- lem with low response rates is often exacerbated if the survey is both voluntary and confidential. Increasing response rates is possible by such methods as keeping surveys short; making

46 tion and analysis and reduce the cost required to conduct the survey. The research team reduced the effort required to com- plete the survey by eliminating fill-in-the-blank responses and streamlining all Likert scales to four points. Ideally, if all survey respondents at a transit agency had access to the Inter- net, there would be no need for manual input or analysis out- side the capabilities of SurveyGizmo. Unfortunately, at most transit authorities not all operating employees have Internet access at work. This is particularly true of vehicle operators. In order to arrive at reasonably representative response rates, the use of paper surveys for employees who do not have Inter- net access is required. Results from completed paper surveys were manually entered. The research team’s next step was to send the survey to management at two of the nine transit agencies for which mini–case studies were done in Chapter 2 (Transit Agencies A and I). These two transit agencies were asked to review the wording of the questions and proposed procedures. Com- ments were received from both transit agencies, and modifi- cations were made. The revised survey was then administered at both transit agencies. Appendix D contains the safety culture survey developed by the research team using the Reason taxonomy as the source of the dimensions and the results of that survey. Please note that the research team was unable to validate this survey instrument for the U.S. transit industry, as discussed in the next section, and it is therefore not suitable for general use in the transit industry. The survey response rate for Transit Agency A was 28%; the response rate for Transit Agency I was 35%. Scores were calculated by assigning weights to responses, ranging from 4 to 1, with a weight of 4 assigned to both “strongly agree” for positively phrased questions and “strongly disagree” for nega- tively phrased questions. The overall Likert-based score was then calculated by averaging the individual scores for each question. The results suggest the following: • The overall Likert average score for Transit Agency A was 3.06, and for Transit Agency I it was 3.10. This means that the average response to Likert scale questions for both sur- veys was better than “agree” for positively phrased ques- tions and stronger than “disagree” for negatively phrased questions. Having ratings that average between the highest possible rating and the next highest, particularly for a sur- vey measuring something as difficult and complicated as the state of safety culture, likely equates to high marks for both transit agencies. • The survey results for each transit agency highlight pos- sible areas of strength and possible areas for improvement. A few examples are as follows: – Transit Agency A’s highest score is 3.74, for the question “It is important to me that there is a continuing empha- as designing a customized survey. A standard public trans- portation instrument maintained by APTA or the FTA and available to the industry at no cost could help, especially as the survey process should be repeated to measure progress every 2 or 3 years or so. Survey Design If the decision is made to use a survey customized for a spe- cific organization, the following process needs to be followed. Transit agencies should thoroughly understand this process whether the design is accomplished with in-house resources or outside professional assistance. Safety Culture Components or Dimensions to Be Measured The first decision to make is what components of safety culture are to be measured. In the draft safety culture sur- vey, the researchers used Reason’s taxonomy (“informed cul- ture, reporting culture, learning culture, flexible culture, and just culture”) augmented by the Reason dimension entitled “underlying perceptions, attitudes, and behaviors.” Sources of Possible Survey Questions Possible sources for questions abound in published sur- veys. In aviation, examples are “Controlled Flight into Terrain: A Study of Pilot Perspectives in Alaska” (Bailey et al., 2000); “Development and Initial Validation of a Safety Culture Sur- vey for Commercial Aviation” (Wiegmann et al., 2003); and “Toward Measuring Safety Culture in Aviation Mainte- nance: The Structure of Trust and Professionalism” (Taylor and Thomas, 2003). A significant amount of surveying has also been done for offshore petroleum operations, with a good example contained in “Assessing Safety Culture in Offshore Environments” (Cox and Cheyne, 2000). In nuclear opera- tions, “Assessment of Safety Culture at a Nuclear Reprocessing Plant” (Lee, 1998) is a good source. In mining and construc- tion, Michael O’Toole demonstrated the use of surveys in “The Relationship Between Employees’ Perceptions of Safety and Organizational Culture” (O’Toole, 2002). Questions from previous surveys used by members of the research team in New York and Los Angeles were also considered. The next step is to choose proposed questions for each of the proposed dimensions to be measured. Pilot Test The survey can now be put into various commercially available formats so that it can be taken online by individual employees anonymously. The major motivation for using Internet-based survey instruments is to simplify administra-

47 Survey Validation and Reliability Testing Of the thousands of surveys administered every week by reputable organizations, few are subjected to validation and reliability testing. However, the additional effort required to conduct such testing would be justified for a survey that is going to be used to compare and contrast the state of safety culture throughout the public transportation industry. Survey validation and reliability testing normally begins with analyzing content validity, a priori scale reliability, and factor analysis. Achieving content validity requires determining that the questions or items included in the survey are appropriate to the components that are being measured; a priori scale reliabil- ity determines internal consistency. As explained by Morrow and Barnes (2012), this determines whether items in a factor or component “are consistently measuring the same underlying construct. For example, if a respondent expresses agreement with items in a measure such as ‘I like running’ and ‘I have enjoyed running in the past,’ and disagreement with the item ‘I dislike running,’ then a factor created from these items would be said to demonstrate good internal consistency.” The most widely used statistical test of internal consistency is Cronbach’s coefficient alpha. Factor analysis determines which of the components intended to be measured are actually being measured by the survey. The factor analysis technique employed in this research is called principal components analysis (PCA). PCA is per- formed by computing the correlations between all of the items and sorting the items into factors such that: 1. Items within each factor have the highest correlations with each other, 2. Each factor accounts for as much variance in the data as possible, and 3. The factors are maximally distinct from each other. Those factors that account for a significant amount of vari- ance and consist of items that appear to represent an inter- pretable theme in the data are retained in the final factor solution (Morrow and Barnes, 2012). The research team provided the survey results to an out- side consulting firm to perform validation and reliability testing. Content Validity An outside firm determined that the survey as written had acceptable content validity—that is, based on their transit experience, they determined that the questions included in the survey were appropriate to the components being measured. sis on safety.” Its lowest score is 2.19, for the question “How much of a factor is ‘Too few workers to get the work done’ in potentially affecting whether all of the safety rules are followed all of the time in your immedi- ate workplace?” The latter may indicate a staffing prob- lem that needs further examination. – Transit Agency I’s highest score is 3.65, for the ques- tion “Do you personally closely follow your immedi- ate workplace safety rules and procedures?” Its lowest score is 2.32, for the question “Do you agree that man- agement takes a no-blame approach if workers report ‘near misses’?” The latter suggests that Transit Agency I should review its near-miss disciplinary policy. Comparisons between transit agencies can also be instruc- tive. For example: – In response to the question “How would you rate the overall quality of the initial safety training you received in the first few months on the job?” Transit Agency A received a score of 3.48; Transit Agency I scored a 3.04. This result implies that Transit Agency I might have room for improvement in terms of initial safety training. The fact that Transit Agency A is rated high in quality of safety training confirms findings from the mini–case study in Chapter 2. – In response to the question “Do your coworkers closely follow your immediate workplace safety rules and pro- cedures?” Transit Agency I scored a 3.45; Transit Agency A scored a 3.18. This result implies that Transit Agency A should devote additional effort to rule compliance. Cross-tabulations of survey results can be used to sharpen the focus of the analysis by determining if a strength or weakness is spread evenly across a transit agency or is pri- marily concentrated in one part of the organization, in a par- ticular group of employees, or in longer- or shorter-service employees. These brief examples suggest the potential value that a public transportation database from an industry-wide stan- dardized safety culture survey would have. Transit agencies could assess their overall state of safety culture compared to their peers, the state of safety culture within each of their major functions, and areas of comparative strength and weakness in the various aspects of their operations that affect safety culture, as well as the personnel who affect it. Such a survey and associated database could become the equivalent of DuPont’s safety perception survey, which is described in the literature review. And, if DuPont can effectively use such a database to compare and contrast its diverse customer base, then certainly the public transportation industry could use a similar database made solely up of public transportation survey results.

48 Compared to the Reason taxonomy, these components are separate and distinct from each other to a greater extent and therefore involve less overlap. Overlap among components creates a situation in which different survey respondents may associate specific items with different components, thereby corrupting the factor analysis. Conclusions The research team believes that a fully validated safety cul- ture survey and associated confidential database would be a major contribution to the public transportation industry’s pursuit of improved safety culture. Other assessment possibilities include: • Using the matrix developed in the article “A Framework for Understanding the Development of Organizational Safety Culture” (Parker et al., 2006). As noted in Appendix A, this matrix can be used to rank how each organization rates in incident/accident reporting, causes of accidents, purpose of procedures, and so forth, locating it on the scale between pathological and generative in each of 18 categories. The average of the results for all of the categories can be used to determine where on the scale, from pathological to generative, the organization being evaluated rests. • Employing checklists—for instance, the Transport Can- ada safety culture checklist (http://www.tc.gc.ca/media/ documents/railsafety/sms_checklist.pdf) or the Reason checklist (http://www.tc.gc.ca/eng/civilaviation/publications/ tp13844-menu-275.htm). Such checklists may be helpful but generally are much more subjective than surveys. (The Reason checklist, however, has on occasion been adminis- tered as a survey.) The Parker matrix and such checklists can be helpful in a qualitative review of a transit agency’s safety practices, pro- grams, involvement, and so forth. It is possible that the prac- tices an organization uses are ultimately more enlightening than quantitative scores, so a comparison—even if qualitative and subjective—may have great value. Quantitative scores could only show inter-organizational comparisons, which to become useful would have to be analyzed for how particular scores were generated, which then might be determined by the presence or absence of actual practices. Scale Reliability Analysis Standard practice is to judge scales with Cronbach alpha coefficients of 0.70 or greater as reliable (internally consis- tent). For the pilot survey, two of the scales (informed cul- ture and underlying perceptions, attitudes, and behaviors) achieved a reliability of 0.70 or greater, which indicated inter- nal consistency. Three of the scales (reporting culture, learn- ing culture, and flexible culture) had alpha coefficients of 0.67 to 0.68, which indicated marginal internal consistency. Only one scale (just culture), which had an alpha coefficient of 0.45, demonstrated inconsistent results. Factor Analysis Principal components factor analysis was performed on all questions using a four-point Likert response scale. The results revealed a seven-factor solution. The seven factors, however, did not conform to the Reason taxonomy (informed culture, reporting culture, learning culture, flexible culture, just cul- ture, and underlying perceptions, attitudes, and behaviors). To summarize, the Reason taxonomy was acceptable with respect to content validity and marginally acceptable with respect to a priori scale reliability analysis but unacceptable with respect to factor analysis. It therefore would be necessary to reconstruct the survey using a component framework that is much more specific than the Reason taxonomy. A good example would be: • Leadership commitment; • Adequacy of resources; • Development and communication of safety information; • The reporting system (reporting and visible action taken on reports); • Accountability (recognition and discipline); • Organizational learning; • Training and education; • Organizational flexibility (openness to opposing views and willingness to adjust to changing circumstances); • Degree of employee involvement and ownership; • The appropriateness of safety policies, procedures, and rules; and • The degree of mutual trust.