National Academies Press: OpenBook

Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach (2013)

Chapter: Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices

« Previous: Module 1 - Tailor Effective Strategies into Workforce Practices
Page 59
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 59
Page 60
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 60
Page 61
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 61
Page 62
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 62
Page 63
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 63
Page 64
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 64
Page 65
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 65
Page 66
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 66
Page 67
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 67
Page 68
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 68
Page 69
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 69
Page 70
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 70
Page 71
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 71
Page 72
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 72
Page 73
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 73
Page 74
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 74
Page 75
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 75
Page 76
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 76
Page 77
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 77
Page 78
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 78
Page 79
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 79
Page 80
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 80
Page 81
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 81
Page 82
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 82
Page 83
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 83
Page 84
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 84
Page 85
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 85
Page 86
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 86
Page 87
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 87
Page 88
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 88
Page 89
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 89
Page 90
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 90
Page 91
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 91
Page 92
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 92
Page 93
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 93
Page 94
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 94
Page 95
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 95
Page 96
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 96
Page 97
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 97
Page 98
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 98
Page 99
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 99
Page 100
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 100
Page 101
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 101
Page 102
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 102
Page 103
Suggested Citation:"Module 2 - Use Metrics to Evaluate the Impact of Workforce Practices ." National Academies of Sciences, Engineering, and Medicine. 2013. Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach. Washington, DC: The National Academies Press. doi: 10.17226/22489.
×
Page 103

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Module Use Metrics to E the Impact of Wo Practices 2. valuate rkforce Building a Susta Workforce in the Transportation In A Systems App inable Public dustry— roach

2-1 Module 2 Contents Module 2: Use Metrics to Evaluate the Impact of Workforce Practices Overview of Metrics Scorecards How to Use the Metrics Scorecards Adjusting or Creating Metrics Scorecard Anchors 2.1 Global Metrics 2-5 2.1.1 Defining the Metrics 2-5 2.2 Recruitment Overview and Scorecards 2-8 2.2.1 Background 2-8 2.2.2 Defining the Scorecard 2-9 2.2.3 Interpretation of Results 2-13 2.3 Retention Overview and Scorecards 2-15 2.3.1 Background 2-15 2.3.2 Defining the Scorecard 2-16 2.3.3 Interpretation of Results 2-22 2.4 Training and Development Overview and Scorecards 2-24 2.4.1 Background 2-24 2.4.2 Defining the Scorecard 2-25 2.4.3 Interpretation of Results 2-30 2.5 Professional Capacity-Building Overview and Scorecard 2-33 2.5.1 Background 2-33 2.5.2 Defining the Scorecard 2-34 2.5.3 Interpretation of Results 2-39 2.6 Applying the Scorecards 2-40 This module includes information that can be used to evaluate workforce practices that fall under the four organizational processes discussed in Module 1: recruitment, retention, training and development, and professional capacity building. To evaluate workforce practices within each of these organizational processes, separate metrics scorecards have been developed. M o d u l e 2 Use Metrics to Evaluate the Impact of Workforce Practices

2-2 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach Table 2-1 provides an overview of all of the metrics included in this module, including the type of organizational process for which each should be used to evaluate, the type of position the metric is appropriate for, and page numbers for both a description of the metric and the location of the metric on a scorecard. The scorecards developed for each organizational process include information on metrics that can be used to evaluate workforce practices. Metrics are defined as performance measures that Table 2-1. Overview of metrics used in Module 2. Metrics Type of Position Pa ge N um be r o f D es cr ip tio n Pa ge N um be r o f Sc o re ca rd (s) M a n a ge m en t/A dv iso ry / A dm in ist ra tiv e/ Te ch n ic a l Fr o n tli n e GLOBAL Stakeholder buy-in 2-5 all Time to implement 2-6 Cost to implement 2-6 Full return on investment 2-7 Sustainability 2-7 RECRUITMENT Time to fill position 2-9 New-hire turnover within the first year 2-10 Offer-to-acceptance ratio 2-10 New-hire turnover during initial training 2-11 New-hire performance ratings 2-12 Recruiting cost ratio 2-12 RETENTION Employee voluntary turnover rate 2-17 Cost to fill open positions 2-18 Diversity turnover 2-19 Employee engagement 2-19 Impact of turnover on employee knowledge 2-21 TRAINING AND DEVELOPMENT Pre- and post-training knowledge/skill testing 2-25 Employee performance post-training 2-26 Percent of employees rating training as job-relevant 2-27 Impact of training on system operations, safety, and customer service 2-28 Percent of trainees satisfied with training 2-28 Recency of training materials 2-30 PROFESSIONAL CAPACITY BUILDING Timeliness of task completion 2-34 2-41 Timeliness of scheduled activities 2-34 Percentage of employees who participate in employee development opportunities 2-35 Percentage of jobs filled internally 2-36 Percentage of operational supervisory positions filled by frontline personnel 2-36 Employee involvement in organization’s decision making 2-37 Percentage of positions with an up-to-date job description 2-38 2-14– 2-15 2-23 2-32

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-3 allow for assessment or projections of the impact of the practice. For each metric, four anchors are provided to help determine ratings. These anchors provide descriptions that demonstrate the varying levels along the performance metric (from low to high) and often indicate the level of effectiveness. The anchors are example statements of what a rating of a certain level could mean. Given that the workforce processes are interrelated as described in the introduction to this guidebook, they do not function alone, and thus, performance in one workforce process may lead to similar performance outcomes in another workforce process. For example, if effective recruitment strategies are used, and a public transportation organization recruits well, it is likely that the retention of these employees will also be favorable. Alternatively, there are some situa- tions in which what works well for one workforce process will conflict with what would lead to effectiveness for another process. For example, one of the metrics used to evaluate professional capacity building is promoting from within, with good performance on this metric being that many employees from within the organization are promoted to supervisory or high-level posi- tions. However, while a strategy to promote from within fares well in terms of professional capac- ity building, it may impede recruitment diversity initiatives if internal recruitment is overused as a source for talent. Thus, it is important to maintain a balance of new and existing talent to lever- age the internal expertise for the purpose of knowledge management while capitalizing on the diversity of new talent available in the labor market. The gist of this discussion is that workforce processes do not function in silos—they are interrelated and can affect one another. When evalu- ating workforce practices, it is important to be aware of and understand this interrelatedness. Overview of Metrics Scorecards This module is organized with a section for each of the previously described organizational processes. Within each section there is detailed information to describe the organizational pro- cess and how workforce practices within the organizational process can be evaluated. Specifically, each section contains: • Background information on the organizational process; • Definition of the metrics included in the metrics scorecard, including what the metric is, why it is of value to public transportation, and how the metric can be measured or implemented; • How to interpret the results of the metrics scorecard, including a sample fictitious transit case study; and • Two metrics scorecards: one for management, advisory, administrative, and technical posi- tions and one for frontline positions. A separate metrics scorecard is presented for management/advisory/administrative/technical positions and frontline positions since some metrics that are valuable for understanding workforce practices implemented for these different types of positions can differ greatly. For the purposes of these metrics scorecards, frontline positions are defined as bus operator/driver, rail operator, and other operational positions, as appropriate. It is up to individual public transportation organiza- tions to determine in which of these two categories individual positions will be considered. How to Use the Metrics Scorecards The metrics scorecards should be used to evaluate existing workforce programs or practices that are under consideration for implementation. The purpose of these scorecards is not to compare one organization to another or create an evaluation of an organization that will be shared publically. Rather, these scorecards can be used internally to compare various practices to determine the most effective way to use resources. Further, while many of these metrics require data collection at an individual employee level, the scorecards are not designed to be used as a performance evaluation for any one employee. Instead, by aggregating data across employees

2-4 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach and across metrics, the public transportation organization can evaluate or estimate the impact of particular initiatives. All 10 of the metrics (global and specific) provided in the appropriate scorecard should be used together to evaluate a single workforce practice such that the total score on the scorecard is for a single practice, and then a new scorecard can be completed for a second practice if com- parison is desired. To do this evaluation, ratings for each metric should be selected based on the anchors provided in the scorecard. To further explain how to use the metrics scorecards, consider the example metric in Exhibit 2-1. For example, if post-training surveys for a specific training and development program show that 77% of trainees are satisfied with the training program, this program may be given a rating of 52 based on extrapolation from the range of anchors in Exhibit 2-1. If post-training surveys for another program show that 85% of trainees are satisfied with the program, this second pro- gram may be given a rating of 65. The choice in the absolute score given on each metric is not as important as consistency is to the functioning of the scorecard. For example, if a transit leader gave the second program described a 67 instead of 65, this would be acceptable as long the same logic used to generate that score is applied when comparing another training program of interest along the same metric. Once ratings are assigned for each of the metrics contained in a scorecard, a total score should be computed by summing each of the individual metric ratings. This number can help provide an overall picture of a practice that can then be compared internally to other practices. When using these metrics to evaluate workforce practices, performance on one metric should not be the sole determinant of whether a practice is effective. While metrics related to each of the orga- nizational processes were identified, there could be other external factors or processes that affect the performance on that metric. Therefore, performance on multiple metrics should be consid- ered when evaluating workforce practices. Adjusting or Creating Metrics Scorecard Anchors While the anchors included in the scorecards were developed to be applicable to a wide range of public transportation organizations, individual organizations may find that the anchors do not meet their specific needs. In this case, the anchors can be adjusted slightly so that they are specific to and helpful for the individual public transportation organization using the scorecards. An example of how the anchors can be adjusted is presented in Exhibit 2-2 and Exhibit 2-3; notice that the same metric is used in Exhibits 2-2 and 2-3, while the anchors (i.e., descriptors under the scale points) have been changed in Exhibit 2-3. Metrics Rating Scales 0–25 26–50 51–75 76–100 1. Percent of trainees satisfied with training Less than 50% satisfied 50% up to 75% satisfied 76% up to 90% satisfied Greater than 90% satisfied Exhibit 2-1. Example of training and development metric and anchors. Metrics Rating Scales 0–25 26–50 51–75 76–100 1. Time to fill position Over 4 months Greater than 2 up to 4 months Greater than 1 up to 2 months Less than 1 month Exhibit 2-2. Example of recruitment metric and anchors.

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-5 When trying to evaluate recruitment programs using this metric and its associated anchors, a public transportation organization may find that all of its bus driver positions are filled in less than 2 months. In this case, it would be necessary to adjust the anchors to better differentiate between various recruitment programs that target bus drivers. When revising any of the anchors, it is important to make sure that the revised anchors follow these guidelines: • The anchors should form a continuous scale (i.e., there is no break between categories so all possible values are covered), • Each anchor should be mutually exclusive (i.e., there should be no overlap in scale points), and • The same anchors should be used when evaluating programs for comparison with one another. Using these guidelines, the anchors could be revised as shown in Exhibit 2-3 to better meet the needs of the public transportation organization just described. The remainder of this module presents the different types of metrics: global or organizational- process–specific metrics. The module also presents definitions for each organizational process (i.e., recruitment, retention, training and development, and professional capacity building) to lead into the metrics that evaluate practices within each of those process areas. This information is then followed by background explanations that speak to the importance of the metric, sug- gestions for how to gather the necessary data to evaluate the metric, and metrics scorecards for each organization process. 2.1 Global Metrics This section provides an overview of the global metrics that are included in the metrics score- cards developed for all four organizational processes. 2.1.1 Defining the Metrics Each of the metrics scorecards contains the same five global metrics to assist agencies in assess- ing the extent to which workforce practices are effectively implemented within public trans- portation organizations. These metrics are intentionally general in nature given that they are designed to be applicable across all of the organizational processes. The value of these global metrics is that they allow a public transportation organization to compare two different types of practices, say a training program and a recruitment initiative, which is particularly valuable if human resource funding is limited and priorities need to be determined. A brief definition and overview of each of the global metrics are provided in the following. Metric 1: Stakeholder buy-in • What this metric is: Stakeholder buy-in refers to the support available for a specific practice from important stake- holders (e.g., management, employees, unions). This metric considers the amount of under- standing that stakeholders have of a practice and their willingness to sponsor or promote the implementation of that practice. Metrics Rating Scales 0–25 26–50 51–75 76–100 1. Time to fill position Over 2 months Greater than 1 up to 2 months Greater than 2 weeks up to 1 month Less than 2 weeks Exhibit 2-3. Example of recruitment metric and revised anchors.

2-6 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach • Why this metric is of value to public transportation: Stakeholders provide critical support for implemented practices and champion these efforts to others. Without stakeholder buy-in, stakeholders will not have a clear understanding of a specific practice; thus they will be unlikely to promote the practice or help make it successful. • How this metric can be measured/implemented: To make use of the stakeholder buy-in metric, appropriate stakeholders for an organization should be identified. These can be individuals or groups who have a vested interested in public transportation, the organization, or the program that is being implemented. Once stakehold- ers are identified, their level of awareness of and support for the targeted practice should be evaluated. This could be done by interviewing stakeholders about the practice and asking that they support it. Alternatively, appropriate stakeholders could be asked to respond to a survey that is developed to gather information about their perceptions of the practice. Metric 2: Time to implement • What this metric is: The time to implement metric measures the amount of time required to fully put a new prac- tice into action. This metric includes time for developing and piloting the practice plus the time that is necessary to fully implement the practice. • Why this metric is of value to public transportation: Time to implement is an important metric due to its relationship with the cost of developing and implementing a practice. The longer the development and implementation process, the greater the expense is to a public transportation organization, and the longer it will be before the expected improvements can be realized. • How this metric can be measured/implemented: Time to implement is calculated as the number of months from the time the practice was first identified through to full implementation. Accurate and up-to-date records of all workforce practices should be kept, including when the ideas for practices are identified, when work is done to develop the practices, and when implementation is complete and the practice is functional. Metric 3: Cost to implement • What this metric is: Cost to implement measures the total amount of money that would be needed to begin and/ or maintain a specified practice. This metric is designed to compare the cost of implementing a practice with the budget that was projected for its implementation. While it is important to factor in all costs when determining the value of a practice, this metric excludes the cost for design and development since these metrics presume the practice has already been articulated by a competitor (and identified via benchmarking). Further, those costs should be isolated so that similar costs can be compared to one another (e.g., compare communication dissemi- nation costs of one practice to costs associated with communications for another practice). Resource requirements (e.g., personnel) can be captured in dollars and may also be factored into the implementation cost as long as they are considered across all the practices being compared. • Why this metric is of value to public transportation: Cost to implement is an important metric because budgetary restrictions or constraints can often hamper the implementation of a practice. If the practice cannot be implemented within

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-7 these budgetary constraints, monies may need to be allocated from elsewhere in the budget, which may negatively affect other organizational programs. • How this metric can be measured/implemented: To use this metric, a budget for implementing or maintaining the specified practice needs to be developed. Having this budget in place allows for a comparison of the actual expense of implementing the practice to the budgeted amount and determination of a rating for cost to implement. All costs for each step involved in implementing and maintaining the practice should be tracked and documented, and these records should be kept in a central location for ease of access by all parties with a need to know. Metric 4: Return on investment (ROI) • What this metric is: Return on investment assesses the amount of time necessary to realize the desired outcomes after a practice has been fully implemented. This metric can be used to compare practices to determine which are anticipated to have the greatest impact in the desired direction in the least amount of time. • Why this metric is of value to public transportation: ROI provides information about the gains experienced from implementing a specific practice. The sooner benefits can be realized after a practice is implemented, the less time there will be for the public transportation organization to question the value of the practice. • How this metric can be measured/implemented: How ROI is measured will depend on the practice being evaluated because the ROI has to do with the valued outcomes occurring after a practice is implemented. For example, a recruit- ment program may be implemented with a goal of recruiting diverse applicants to the public transportation organization. The desired outcome of this program would be to have a more diverse workforce. ROI for this practice could be measured as the amount of time it takes to attract new diverse talent to the organization. As another example, if implementing a train- ing program for bus mechanics, the desired outcome may be that all bus mechanics that participate in the training program be able to change a tire within a specified amount of time. Measuring ROI could include information such as the use of a practice or the perceived value of the practice to employees. Metric 5: Sustainability • What this metric is: As a metric, sustainability evaluates the ability to maintain a program at its intended level after its initial implementation. It is the ability to maintain a specific program over time and in varying situations. Sustainability here refers to the level of maintenance required to keep a program up-to-date. It includes updates needed to keep the program current with available technology, workplace procedures and policies, or other aspects that affect the applicability of the program. • Why this metric is of value to public transportation: Sustainability is an important metric because programs that can be maintained without sig- nificant resource demands are usually more cost-effective and provide the greatest ROI to the organization. If a program can only be used for one type of job or for one particular situa- tion, it will not be as valuable as a program that can be used many times without updates or revisions.

2-8 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach • How this metric can be measured/implemented: Sustainability can be evaluated by determining whether the practice being evaluated would work in different situations or at different periods of time. The ability of a practice to be effectively used in multiple situations without revisions or updates indicates sustainability. If changes must be made to implement the practice in a new situation, the practice would be less sustainable. 2.2 Recruitment Overview and Scorecards This section provides an overview of recruitment as it relates to public transportation organizations. In addition, recruitment performance metrics are introduced, and the recruitment scorecards are described. 2.2.1 Background One of the major challenges facing the public transporta- tion industry today is the retirement of baby boomers, with an estimated 50% of the transportation workforce becoming eligible to retire in the near future (Warne, 2005; TRB, 2003). As such, personnel losses experienced in public transporta- tion organizations will likely be a significant problem in the near future. Additionally, NCHRP Report 693 states that when these experienced employees retire, there can be skill gaps in multiple critical task areas (Cronin et al., 2012). Further- more, there has been an increasing level of demand for ser- vice placed on transportation over the past few decades. The potential losses in personnel coupled with the increasing demand for service highlights the importance of recruitment and the need for transportation agencies to recruit highly qualified individuals. Compounding the need for effective recruitment is the realization that public transportation is increasingly less attractive as a career choice in part due to the industry’s image, work culture, and compensation. Because of this, tra- ditional sources of public transportation recruitment are no longer as useful as they were (TRB, 2003). This decrease in the utility of traditional recruitment sources highlights the importance of evaluating recruitment practices to determine their effectiveness. NCHRP Synthesis 323 states that recruiting qualified employ- ees is important to creating an effective workforce, and the fail- ure to attract the right kind of individuals with the necessary skills can render an organization ineffective (Warne, 2003). However, all recruitment practices will not have the same impact at all public transportation organizations due to dif- fering geographical location, size, and operating characteristics Recruitment Metric Highlights ¾ Public Transportation Workforce Target • Bringing in employees with key knowledge and skills. • Supporting training and development, professional capacity-building efforts. Professional Capacity Building Recruit- ment Processes Training and Development Retention Processes ¾ Importance to Public Transportation • Increasing demand on public transportation orga- nizations in addition to large numbers of retiring employees contributes to critical need of recruiting highly qualified employees. ¾ Key Public Transportation Challenges • Large numbers of baby boomers retiring. • Public transportation becoming a less attractive career choice. • Increasing demands on public transportation systems. ¾ Targeted Public Transportation Metrics • Time to fill position. • New hire turnover within the first year. • Offer-to-acceptance ratio. • New hire turnover during initial training. • New hire performance ratings. • Recruiting cost ratio.

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-9 of transit organizations. Evaluating recruitment practices with regard to one’s own organizational characteristics will increase the likelihood that the practices will be effective in recruiting the best employees for the available public transportation jobs. 2.2.2 Defining the Scorecard Each Recruitment Practices Scorecard presents five targeted metrics that are relevant to the job type indicated. These targeted metrics are provided to assist public transportation organiza- tions in assessing the extent to which recruitment initiatives are successful and effective for the organization, suggesting that the most qualified employees are being hired into open positions. The recruitment metrics are defined in the following. Metric 1: Time to fill position • What this metric is: Time to fill position is the average time required to recruit and hire applicants to fill open posi- tions. A consistent time period should be used when comparing two practices along the same metric. The amount of time to fill position is quantified as the number of days between the time approval to hire for that position is granted (prior to any marketing of the open position) until the formal acceptance of a job offer by a candidate. • Why this metric is of value to public transportation: Time to fill position is an important metric because the cost of a job vacancy can be signifi- cantly abridged by reducing the length of time a position stays open (Employer’s Association Development Group, 2010). Lengthy position vacancies can cause significant financial losses for public transportation organizations due to recruitment costs, resource deficiencies, or the cost of unmet service demands. − Frontline positions: Vacant frontline positions can lead to operational inefficiencies such that remaining operators may not be able to meet both their own schedule and rider load demands and those of the vacant position. Further, the organization may be responsible for paying overtime to workers who are nonexempt under the Fair Labor Standards Act. Public transportation organizations must also consider the costs and potential constraints associated with negotiations that may occur for workers under a collective bargaining agreement who are asked to take on the added responsibilities of the vacant position. − Management/advisory/administrative/technical positions: Recruitment costs are higher for these positions given the sophisticated channels required to seek skilled labor with specific technical or managerial expertise. Given that the length of a vacancy is corre- lated with recruitment costs, the financial impact can be great for management/advisory/ administrative/technical positions left unfilled. Further, the lack of technical expertise or supervisory direction can leave frontline workers without the guidance needed, thus result- ing in operational inefficiencies for the public transportation organization. • How this metric can be measured/implemented: Public transportation organizations should maintain records by job to track the time it takes to fill job openings within a calendar year. In addition to using these data to measure the metric, these records can be valuable for identifying which jobs are most challenging to fill. Records may also pinpoint the seasons during which recruiting is most difficult. Using these data, recruitment initiatives can then be directed toward resolving challenges. Focus group members who participated in TCRP Project F-16A agreed that there are varying time-to-fill targets and costs associated with filling different positions. For instance, a frontline position

2-10 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach might need to be filled immediately so that the public transportation organization can con- tinue to provide the real-time service that defines its business, whereas for a newly developed position, it may be more cost-effective to take extra time to define job requirements and recruit and screen applicants. Due to the disparity between frontline positions and management/ advisory/administrative/technical positions, the rating anchors presented for this metric vary for these two types of positions. Metric 2: New-hire turnover within the first year • What this metric is: This metric indicates the number of new hires who leave the public transportation organiza- tion (either voluntarily or involuntarily) during their first year of employment. New hires may leave their job if they realize that they do not fit with the organization, or they may be termi- nated if the organization realizes it made a mistake in hiring (Selden and Orenstein, 2011). • Why this metric is of value to public transportation: New-hire turnover is important to measure because new hires have to be recruited, inter- viewed, and trained, all at the expense of the public transportation organization. A public transportation organization will obviously spend less on recruiting if new-hire turnover is low. This new-hire turnover metric helps to determine if the people with the appropriate skill sets are being hired and whether the organization is giving sufficient focus to integration and development for new hires. • How this metric can be measured/implemented: These data can be collected from organizational records. Whenever an employee leaves the organization, the separation date as well as the reason for leaving (e.g., termination, resigna- tion) should be documented. This metric can then be calculated by dividing the number of new hires who turn over during their first year by the total number of new hires brought on during that year. In some cases, turnover may not be fully reflective of the recruitment prac- tice; for example, some public transportation organizations heavily recruit students who are more transient than nonstudents and can be expected to have variable schedules that only allow for temporary employment. In this situation, a high turnover rate could be a normal function of the organization and not a red flag. Public transportation organizations should therefore develop their own appropriate anchors for this metric. In addition, turnover for supervisory and technical positions will differ from that for frontline positions, so anchors for this metric should reflect that. Frontline positions fill more quickly and easily than supervi- sory and technical positions, so a high turnover for these would be less detrimental to a public transportation organization than high turnover for supervisory and technical positions. With this understanding of new-hire turnover, public transportation organizations can tailor their recruitment practices to screen for people who are less likely to contribute to turnover within the first year of employment. Metric 3a: Offer-to-acceptance ratio (management/advisory/ administrative/technical positions only) • What this metric is: Offer-to-acceptance ratio measures the ratio of the number of job offers accepted to the num- ber of job offers extended. A low offer-to-acceptance ratio would indicate that many of the individuals to whom job offers were given did not choose to accept employment with the public transportation organization, whereas a high offer-to-acceptance ratio would indicate that many of the individuals who received job offers accepted them.

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-11 • Why this metric is of value to public transportation: The offer-to-acceptance ratio can help organizations understand the value of offers given to candidates, which is important for hiring high-quality candidates (Australian Public Service Commission, 2009). In other words, this metric looks at how much candidates value the job offers given by the public transportation organization. When time and money are spent recruiting high-quality candidates, it is desirable for these candidates to accept the job offers that are given to them. Focus group participants noted that recruitment prac- tices for supervisory or technical positions can be meaningfully evaluated using this met- ric. Candidates recruited for these positions are more likely to be carefully selected based on skills and organizational fit, so a low offer-to-acceptance ratio speaks to the effective- ness of the employer’s recruitment practice. This metric should be applied to supervisory and technical positions only; focus group members stated that offer-to-acceptance ratio is not a valuable measure of recruitment practice success for frontline positions since most frontline offers are accepted. • How this metric can be measured/implemented: This metric is measured as a direct ratio of the number of offers accepted to the number of offers extended. Public transportation organization records should track this information. They can then compare recruitment practices and choose to implement those that recruit candidates in a way that will lead to a high number of job offer acceptances. Metric 3b: New-hire turnover during initial training (frontline positions only) • What this metric is: This metric indicates the number of new hires who leave the public transportation organiza- tion (either voluntarily or involuntarily) during their initial training period. New hires may be terminated or leave their job if they realize that they do not fit with the organization. The initial training period is when many of the new hires will be first asked to learn and perform the necessary tasks for frontline operations. Turnover during this training period will identify those candidates that were unable or unwilling to perform these tasks or for some reason were inappropriately selected for the job. • Why this metric is of value to public transportation: The new-hire turnover during initial training metric is only applicable to frontline employees due to the specific skill training associated with these positions. Unlike supervisory or techni- cal positions, frontline positions are marked by intensive initial training periods during which employees learn and develop the necessary skills for the job. Focus group members from this project who represent small transit systems indicated that a probationary training period is an important time during which they can determine whether employees are a good fit. As with other positions, frontline employees have to be recruited, interviewed, and trained all at the expense of the organization. A public transportation organization will obviously spend less on recruiting if new-hire turnover is low. This new-hire turnover during initial training metric helps to determine if the people with the appropriate motivation and skill sets to complete training are being hired and whether the organization is giving sufficient focus to integration and development for new hires. • How this metric can be measured/implemented: These data can be collected from organizational records. As with new-hire turnover during the first year, turnover that occurs during initial employee training should be documented in pub- lic transportation organization records. This documentation should include the date of and

2-12 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach reason for separation from the transit system during training. This metric is then computed as a percentage: the number of employees who turn over during training divided by the total number of employees participating in the initial training program. With these percentages in hand, public transportation organizations can compare across different recruitment practices and employ those practices that are more apt to bring in employees who are likely to remain with the organization after the initial training period. Metric 4: New-hire performance ratings • What this metric is: New-hire performance ratings assess the quality of hired employees. Performance evaluations assess whether new hires are meeting the public transportation organization’s standards and expectations. For frontline positions, according to representatives of large and small public transportation organizations, employees are either performing satisfactorily or unsatisfacto- rily, whereas management and technical staff are more likely to be evaluated through complete performance ratings. • Why this metric is of value to public transportation: The quality of new hires has been identified as an important recruitment metric (Wilkins, 2007). If new employees are high performers and are doing well on the job, this is an indication that the correct individuals for positions are being recruited and hired. Conversely, lower ratings suggest that the public transportation organization may not be recruiting the right people. • How this metric can be measured/implemented: Data for comparisons should be gathered from existing performance reviews of employees at the same job level. Focus group participants agreed that frontline employees are evaluated dif- ferently than supervisory and technical employees. Behind-the-wheel evaluations were effec- tive in determining whether frontline employees were performing adequately. Evaluations for management and technical staff, however, can include assessments of different knowledge, skills, or abilities appropriate for a particular role to determine how well the employee is performing on the job as a whole. With an understanding of new hires’ performance, a public transportation organization can adjust its recruitment practice and screen for role-specific skills and abilities in candidates. Metric 5: Recruiting cost ratio • What this metric is: Recruiting cost ratio evaluates the costs incurred to recruit employees. Recruitment practices that are able to bring in talent at a lesser cost per job to the public transportation organiza- tion are considered to be more effective than practices that are more expensive to implement or operate. Costs to consider when determining the recruiting cost ratio include things like advertising for the position and travel and lodging for candidates (Employer’s Association Development Group, 2010). • Why this metric is of value to public transportation: An awareness of the recruiting cost ratio for a particular recruitment practice allows the public transportation organization to determine whether a method is cost-effective. With an understanding of the costs associated with particular recruitment practices, public trans- portation organizations can choose practices that prove to bring in the best talent at the lowest cost.

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-13 • How this metric can be measured/implemented: Recruiting cost ratio is a summation of all recruiting costs [e.g., advertising, placement fees, employee referrals, travel and lodging, relocation, reference checking, or any other recruitment/hiring costs (Employer’s Association Development Group, 2010)] divided by the total compensation of positions that are being filled. To calculate this metric, all costs involved in the recruitment of employees should be tracked and documented. This ratio will allow public transportation organizations to compare the relative costs of different recruit- ment practices. 2.2.3 Interpretation of Results When using the scorecard to evaluate only a single practice, the recruitment practice assessed with this scorecard should have positive results for each of the six metrics. When using the scorecard to compare two or more types of practices, the overall summative score for each practice will indicate the practice likely to have the greatest overall return. For example, a public transportation organization might have a recruitment practice of using social media outlets to reach new groups of applications. To recruit a younger and more diverse set of employees, it might promote the positive aspects of its employment on Facebook to increase visibility and positive perceptions of the organization. Using data gathered during recruit- ment or projections for the defined metrics, it could better determine if this initiative is the best use of recruitment resources or if another recruitment practice would provide a more positive outcome. Sample transit case study – applying recruitment metrics. A large urban multimodal transit agency located on the West Coast had been experiencing problems in filling manage- rial and technical positions in the organization. These problems were manifested in two criti- cal areas. First, management feels it was taking an extraordinary length of time to fill many of these professional positions, causing financial problems in terms of excessive recruitment costs and resource deficiencies. The time to fill vacant positions also had the potential to affect mission achievement by creating unmet service demands. Second, once applicants for managerial and technical positions were brought on board, a significant number were leaving the organization within the first year of employment, thus compounding transit leadership concerns. Based on these issues, the executive director met with the human resource director in an attempt to identify the agency’s existing recruitment practices that needed to be evalu- ated and possibly re-engineered to improve the situation. As a result of their discussion, the agency decided to use the recruitment metrics presented in this module. Practices identified to be measured included how and where managerial and professional vacancies were advertised and the viability of the existing tools used to screen and interview candidates for managerial and technical positions. The HR director then initiated an evaluation process of the organization’s existing recruit- ment activities. This evaluation process was based on a review of vacancy advertising strategies presently used by the agency, human resource records on the number of applicants the strategy identified, and the time period between initial vacancy announcements and the onboarding of new managerial or technical employees. Parallel with this evaluation process, the HR direc- tor also examined the agency’s existing screening and interview process using the Recruitment Practices Scorecard, in conjunction with identifying the percentage of managerial or techni- cal employees that left the organization either voluntarily or involuntarily in their first year of employment. The results of these parallel evaluations confirmed that the existing recruitment and screening processes for managerial and technical staff were, in fact, ineffective and caus- ing organizational damage. At this point, the executive director and HR director determined

2-14 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach it would be necessary to experiment with and evaluate the effectiveness of different and more proactive recruitment strategies, including outreach to university students about to graduate from programs related to critical professional skill areas, advertising in transit trade association magazines, and incorporating the use of social media into recruitment efforts. Simultaneous with the exploration of these new recruitment strategies, the HR director adapted its candidate screening process for managerial and technical positions by expanding the employment applica- tion to include questions that were more directly relevant to potential success in those positions and to incorporate a structured interview process. These new recruitment and screening strategies were put into practice and measured for effectiveness again using the Recruitment Practices Scorecard. The results of these new prac- tices were then compared to the result of the previously employed recruitment and screening practices. These comparisons confirmed both the ineffectiveness of previous practices and the enhanced effectiveness of the experimental new practices. Based on this assessment, within 1 year of initiating the new recruitment and screening process, the transit agency institutional- ized its new approaches to recruitment and screening for managerial and technical positions. The overall results of this effort were a dramatic reduction in time to fill professional positions and an accompanying reduction in professional employees leaving the transit agency within their first year of employment. Please see Exhibit 2-4 and Exhibit 2-5 for the Recruitment Practices Scorecards for management/advisory/administrative/technical positions and front- line positions. Title of Practice: [Enter here] Metrics Recruitment-Specific Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 1. Time to fill position Over 8 months Greater than 5 up to 8 months Greater than 2 up to 5 months Less than 2 months 2. New-hire turnover within the first year More than 40% of new hires turn over Over 20% up to 40% of new hires turn over 10–20% of new hires turn over Less than 10% of new hires turn over 3. Offer-to-acceptance ratio Less than 40% of offers accepted More than 40% up to 60% of offers accepted More than 60% up to 80% of offers accepted More than 80% of offers accepted 4. New-hire performance rating Bottom 25% of employees Slightly lower than average employees Slightly higher than average employees Top 25% of employees 5. Recruiting cost ratio More than budget target Right at budget target Slightly under budget target Significantly under budget target Sub-score: Metrics Global Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–5 76–100 6. Stakeholder buy-in Stakeholders unaware of program Stakeholders know of program but have a lot of questions. Stakeholders understand program and only have a few questions. Stakeholders already on board 7. Time to implement Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 8. Cost to implement More than budget target Right at budget target Slightly under budget target Significantly under budget target 9. Full return on investment Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 10. Sustainability One-time program Program information must be continually updated to remain current. Program can be updated annually and reused. Program can be continually used with minimal maintenance. Sub-score: Notes: Likelihood of Success Going Fwd Total Metrics Score Success Very Unlikely 0–250 Success Unlikely 251–500 Success Likely 501–750 Success Very Likely 751–1000 Exhibit 2-4. Recruitment Practices Scorecard – management/advisory/administrative/technical positions.

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-15 2.3 Retention Overview and Scorecards This section provides an overview of retention as it relates to public transportation agencies. In addition, retention performance metrics are introduced, and the retention scorecards are described. 2.3.1 Background Retention is a very important topic in today’s transportation industry. There are many factors specific to the industry that contribute to the need to retain highly qualified employees. Specifi- cally, as noted in TCRP Report 77, there is an increasing demand on the industry to hire new talent from a shallow pipeline, not to mention the strong competition from other industries for employees in the fields relevant to transportation careers (McGlothlin Davis, 2002). Addition- ally, public transportation organizations are experiencing reduced quality and quantity of people who are able to provide services (McGlothlin Davis, 2002). Although focus group participants reported that low turnover can be a mixed blessing in a union environment given the challenge of terminating poor performers, retaining top performers can be a challenge even in a weak economy. Furthermore, new retention practices must be used to address the changing public trans- portation workforce. With the retirement of the baby boomers, the newer workforce com- prises younger and more diverse employees. Younger, more diverse employees value different Title of Practice: [Enter here] Metrics Recruitment-Specific Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 1. Time to fill position Over 4 months Greater than 2 up to 4 months Greater than 1 up to 2 months Less than 1 month 2. New-hire turnover within the first year More than 40% of new hires turn over Over 20% up to 40% of new hires turn over 10–20% of new hires turn over Less than 10% of new hires turn over 3. New-hire turnover during initial training More than 30% of new hires turn over Over 20% up to 30% of new hires turn over 10–20% of new hires turn over Less than 10% of new hires turn over 4. New-hire performance rating Performance deemed unacceptable Significant coaching required Some coaching required No coaching required 5. Recruiting cost ratio More than budget target Right at budget target Slightly under budget target Significantly under budget target Sub-score: Metrics Global Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 1. Stakeholder buy-in Stakeholders unaware of program Stakeholders know of program but have a lot of questions. Stakeholders understand program and only have a few questions. Stakeholders already on board 2. Time to implement Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 3. Cost to implement More than budget target Right at budget target Slightly under budget target Significantly under budget target 4. Full return on investment Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 5. Sustainability One-time program Program information must be continually updated to remain current. Program can be updated annually and reused. Program can be continually used with minimal maintenance . Sub-score: Notes: Likelihood of Success Going Fwd Total Metrics Score Success Very Unlikely 0–250 Success Unlikely 251–500 Success Likely 501–750 Success Very Likely 751–1000 Exhibit 2-5. Recruitment Practices Scorecard – frontline positions.

2-16 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach things from those employees who are retiring from the work- force, necessitating a wider range of practices to retain top talent. For example, retention of women may be improved by offering family-supportive polices (e.g., flexible hours, child care, maternity leave), whereas this retention practice may not have been as important in the past (Cronin et al., 2011). By focusing on the needs of diverse employees and imple- menting targeted retention practices, public transportation organizations can maintain a highly successful workforce. While the need to implement effective retention practices spans all public transportation organizations, the specific practices that would be the most successful will vary from one organization to another. An individualized evaluation of exist- ing or potential retention practices can help identify the most effective ways in which an organization can work to retain its workforce. However, focus group participants agreed that developing ways to measure the impact of retention practices is frequently a challenge. These scorecards provide a foundation for public transportation organizations to evaluate the impact of retention practices using proven metrics. An important distinction when evaluating retention prac- tices is the difference between voluntary and involuntary turn- over. Voluntary turnover occurs when an employee makes his or her own decision to leave the organization (i.e., will- ingly resigning), while involuntary turnover occurs when the employee has no choice but to leave the organization (i.e., employer-initiated termination). Although practices exist to prevent both types of turnover, most practices that specifically target retention are more effective at mitigating voluntary turn- over. Such practices rely on increasing the organizational com- mitment among employees the organization wants to retain so that they are less likely to decide to leave. However, invol- untary turnover is more often related to performance issues or downsizing, both of which are not easily addressed through such practices. Moreover, since union contracts often specify the terms under which employees can be terminated, it is often easier to prevent performance issues through more effective recruitment, selection, and orientation of new employees than it is to address these issues once they occur (Cronin et al., 2011). Trends in involuntary turnover can suggest problems related to recruitment, selection, training, or management and supervi- sory issues, but may not be an indication of effectiveness for a particular retention practice. Thus, while the focus of this section will be on evaluating practices to prevent voluntary turnover, public transportation organizations may also find involuntary turnover to be an effective indicator for a range of human resource and management practices. 2.3.2 Defining the Scorecard The Retention Practices Scorecards provide five targeted metrics to assist public transporta- tion organizations in assessing the extent to which retention initiatives are successful, both in Retention Metric Highlights ¾ Public Transportation Workforce Target • Retaining key knowledge and skills. • Supporting training and development, professional capacity building efforts. Professional Capacity Building Recruitment Processes Training and Development Retention Processes ¾ Importance to Public Transportation • Increasing demand on public transportation orga- nizations in addition to strong competition for the same skills contributes to critical need of retaining highly qualified employees. ¾ Key Public Transportation Challenges • Reaching younger, diverse employees. • Offering new or different practices. • Reduced quality and quantity of potential employees. ¾ Targeted Public Transportation Metrics • Employee voluntary turnover rate. • Cost to fill open positions. • Diversity turnover (race/gender). • Employee engagement. • Impact of turnover on employee knowledge.

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-17 regard to retaining employees and to the effect of turnover on the organization. The first four metrics are relevant to all positions. However, there are slight differences in the way the fifth metric would be used based on the type of position, either frontline or management/advisory/ administrative/technical, so a distinction is made between the two. Each of the retention metrics is defined in the following. Metric 1: Employee voluntary turnover rate • What this metric is: Employee voluntary turnover rate identifies the number of employees who leave the public transportation organization by choice (i.e., voluntarily resigning) as a proportion of the aver- age number of employees working at the organization in a given period. It is important to distinguish this from involuntary turnover (i.e., forced termination) and overall turnover, which encompasses both voluntary and involuntary turnover. • Why this metric is of value to public transportation: The employee voluntary turnover rate is suitable for measurement because it is often prevent- able and can be reduced through targeted retention practices (Cascio and Boudreau, 2008). Since voluntary turnover results from a decision made by the employee, having effective reten- tion practices may help to keep these employees within the organization. When employees leave the organization, recruiting and training costs rise. Additionally, high turnover rates could affect safety due to a lack of experienced drivers. This metric can also affect recruit- ment efforts since potential employees could see the turnover rate as an indicator of how past employees perceived the work environment. Focus group participants reported measuring both voluntary and involuntary turnover in order to gain more detail on the nature of any retention challenges. Voluntary and involuntary turnover can be measured in a similar fashion, but most retention practices are designed to reduce the number of employees who choose to leave an organization, so voluntary turnover may be a more effective measure of such practices. • How this metric can be measured/implemented: These data can be gathered through turnover records, and the metric can be calculated as the number of voluntary turnover incidents (e.g., voluntary resignations) per period divided by the average workforce size. A simple way to calculate average workforce size is to add the total number of employees at the beginning and end of a period (e.g., fiscal calendar) and divide by two (though if workforce size has fluctuated significantly during this period, two counts closer together may be required). Dividing the number of turnover incidents by this figure yields a percentage that indicates turnover performance (lower is generally better). Because turnover rates can be affected by a number of factors out of the organization’s control, the length of period of analysis should be long enough to allow short-term fluctuations to even out. At the same time, it should be short enough that any changes can be reasonably attributed to implementation of a particular retention practice. One focus group participant also suggested collecting turnover data on individual job fami- lies or positions since that can indicate retention problems within specific segments of the organization. Exit interviews are an effective way to record the reason employees leave the organization and can help to determine any patterns that might require specific practices to improve retention. It is important to recognize that turnover is a fact of organizational life, and some turn- over is good since it brings new skills and fresh perspectives into an organization and allows employees who may not be a good fit to seek opportunities elsewhere. Exactly what is

2-18 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach considered high turnover varies based on the type of job and the nature of the labor mar- ket. Frontline positions typically have higher voluntary turnover since selection processes for these roles may be less rigorous than for management/advisory/administrative/technical roles, and frontline positions may offer less opportunity for growth. The anchors listed in the recruitment scorecards account for this fact, but public transportation organizations should account for the unique characteristics of their operating environment when evaluating per- formance on this metric. Metric 2: Cost to fill open positions • What this metric is: Cost to fill open positions specifies the cost to fill positions that become available because of employee turnover as a percentage of the annual wages for those positions. A higher value indicates that employee turnover is placing a greater financial burden on the organization. The focus of this metric is on positions that are available due to an employee leaving the orga- nization, rather than those that are newly created due to increased demand or other factors. This ensures that performance on the metric is more tightly linked to a public transportation organization’s retention practices. • Why this metric is of value to public transportation: The cost to fill open positions metric allows the organization to quantify the financial and operational costs of turnover. Turnover can be a substantial expense to organizations due to the operational challenges it creates and the cost of subsequent recruitment, orientation, training, and productivity losses (Cascio and Boudreau, 2008). While a public transportation organization may track recruitment and training costs sepa- rately, only by linking these costs to an employee leaving the organization can the cost of an organization’s turnover performance be accurately calculated. Once the financial cost is esti- mated, it can be used to evaluate existing retention practices, demonstrate the return on invest- ment in new retention practices, and compare the financial impact of various approaches. • How this metric can be measured/implemented: Data can be gathered through existing records and typically include expenses such as separa- tion costs, replacement costs, new-employee training and orientation costs, and lost produc- tivity (Employer’s Association Development Group, 2010; Cascio and Boudreau, 2008). The easiest approach to capture these values would be to record the total values for each cost com- ponent across the organization and then apply a percentage based on the number of openings that were related to turnover (as opposed to new positions) for a given period. This provides a dollar value that can also be divided by the average wage for the positions experiencing the turnover to yield a percentage. The percentage allows for comparison across positions or organizations with different labor costs. For a more detailed approach, each of the expenses listed previously could also be linked to the position experiencing the turnover. This provides valuable detail since costs can vary sig- nificantly depending on an employee’s compensation and level of responsibility in the orga- nization. For example, separation costs may include a standard cost for exit processing as well as variable costs for separation pay or unemployment taxes that correlate with an employee’s pay rate. Replacement costs can include posting the announcement, time spent on interviews, testing, and orientation, which can be estimated by recording total costs and taking a per- employee average. Training costs include material development, equipment, instructor labor, and employee labor. The impact of lost productivity is difficult to measure directly, but it can be estimated by calculating the wages (including overtime) of employees working the shifts of the vacated position while a replacement is recruited and trained.

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-19 Metric 3: Diversity turnover (race/sex) • What this metric is: Diversity turnover is used to describe the subset of the overall voluntary turnover described in retention metric 1 but related to any protected group (e.g., race, gender). Many of the same principles regarding turnover covered in retention metric 1 apply here. To compare turnover rates for each diversity category such as race and gender, public transportation organizations can calculate the number of employees in each group who voluntarily leave the organization and divide this by the total number of employees in that category. These rates can then be com- pared to organization-wide employee turnover rates or those for other demographic groups. • Why this metric is of value to public transportation: A comparison of turnover rates among demographic groups has been identified as an impor- tant retention metric for retaining a diverse workforce. Research demonstrates that organiza- tions with programs and policies to manage diversity are likely to have stronger organizational performance as rated by employees. Operating a highly diverse organization without effective practices to address the challenges of maintaining a diverse workplace can lead to negative per- ceptions among employees (Choi and Rainey, 2010). As a result, organizations with effective practices to support and retain people of different demographic backgrounds may be more successful at retaining both minority and nonminority employees. Conversely, low diversity turnover numbers may indicate the availability of opportunities for development, mentor- ing, and other supportive programs that help to encourage individuals of all demographics to remain and grow in the organization. • How this metric can be measured/implemented: Data can be collected on turnover rates (see retention metric 1) by grade level and race, national origin, sex, and disability, and compared to overall turnover rates. This can be accomplished by counting each employee that voluntarily left the organization and cross-referencing the list with the demographic information each employee provided when hired (if provided) to arrive at separate rates for each demographic category. If the data reveal that some groups have higher- than-average turnover rates, this may indicate that the organization could be doing more to support and retain members of that group. However, before taking action, public transporta- tion organizations should consider whether there are other factors that might contribute to one demographic group having higher turnover rates than another. For example, it may be that a minority group tends to be in frontline positions, which may have more turnover than techni- cal roles. Likewise, just because a group does not have a higher-than-average turnover rate does not mean that the organization is free of diversity issues that could affect retention. To obtain a clearer picture of the impact of diversity on long-term retention, organizations can also collect data on rates of selections for promotions, training opportunities, and perfor- mance incentives by race, national origin, sex, and disability since these factors indirectly affect turnover. Success over time can be measured by the percent reduction in turnover of minority individuals (U.S. Department of the Interior, 2005). One challenge that organizations may face in calculating this metric is that employee reporting on ethnicity is voluntary, so data may be unavailable in some cases, and making presumptions about ethnicity or other demographic factors can lead to faulty conclusions. Thus, it should be recognized that the estimates made are no more than estimates. Metric 4: Employee engagement • What this metric is: Employee engagement measures the level of emotional involvement, commitment, and sat- isfaction that employees feel with respect to their jobs and the organization. There are

2-20 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach multiple scales commonly used to measure employee engagement. It is typically studied through asking employees to respond to surveys that assess the extent to which they are energized by and engrossed in their work. Engagement surveys also assess the extent to which employees believe their efforts and opinions matter and that they are rewarded for their performance, and assess their level of commitment to succeeding and growing with the organization. • Why this metric is of value to public transportation: Employee engagement has been shown to be related to employee turnover as well as sev- eral other elements of organizational performance, including customer satisfaction, profit- ability, productivity, and safety (Harter, Schmidt, and Hayes, 2002; Harter, Schmidt, and Killham, 2003). It has been demonstrated that highly engaged employees perform 20% better and are 87% less likely to leave an organization than those employees who are not engaged (Lockwood, 2007). Whether an employee is a bus operator or a senior executive, it is important that top per- forming employees feel that they are making a difference and that their supervisor values their contributions to the organization. Given budget constraints at many public transportation organizations, it is difficult to promote retention through pay increases alone, but the rela- tionship of employee engagement to predicting turnover and other performance indicators suggests that practices that affect engagement could be a more cost-effective way to promote retention. Furthermore, practices that might boost short-term retention but do nothing to improve employee engagement may have limited long-term impact. • How this metric can be measured/implemented: Engagement surveys can assess the extent to which employees consistently say positive things about working at an organization, want to stay with that organization, and strive to achieve beyond minimum expectations. Each survey question could include a scale that asks employ- ees how much they agree or disagree with a statement. By averaging the scores across items and employees, public transportation organizations can estimate overall employee engagement. There are typically a number of antecedents (i.e., predictors) of employee engagement that are measured on the survey such as supervisor and organizational support, communications, resources, and goal/value alignment. Then, engagement itself is measured as an attitude with three general facets: vigor, dedication, and absorption in work. An example of each type of survey item follows: − Vigor: I tend to persevere in my work, even when things are not going well. − Dedication: I am inspired to go above and beyond what is normally expected to help my organization succeed. − Absorption: I am immersed in my work at this organization. Some mixed-size focus group participants felt that employee engagement could be difficult to accurately measure for frontline employees since disgruntled employees may be inclined to blame management for their lack of engagement. However, this feedback reiterates how the effects of employee engagement can permeate all levels of the organization. Thus, employee engagement is a valuable indicator of retention at all levels, and many of the same survey items can be used for both frontline and management/advisory/administrative/technical positions. Given that employee engagement is a very specific type of employee workplace attitude and one that requires candor to effectively measure, public transportation organizations may find it valuable to contract with a neutral, third-party consultant that specializes in the measure- ment of employee engagement. Working with a third-party data collector can also help to bolster a sense of confidentiality for survey participants.

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-21 Metric 5: Impact of turnover on employee knowledge • What this metric is: − Frontline positions: This metric indicates the potentially negative impact on system opera- tions, safety, and customer service related to knowledge loss when frontline employees leave the organization. For frontline positions, most of the knowledge lost when an employee leaves is maintained by other employees, documentation, and training materials, but per- formance could suffer if a number of employees leave in a short span since it will take time to recruit and train new employees. − Management/advisory/administrative/technical positions: This metric indicates the amount of institutional knowledge lost when these employees leave the organization. Insti- tutional knowledge consists of the accumulated training, work experience, and general know-how of employees at an organization. • Why this metric is of value to public transportation: − Frontline positions: Turnover of frontline employees can be detrimental to effective sys- tem operations, employee and passenger safety, and the delivery of high-quality customer service. If turnover is high, new employees may not be able to be recruited and trained in time to ensure operations, safety, and service are not affected. For example, a focus group participant reported that a pension reform expected to occur later this year will result in many long-term employees leaving the organization around the same time, which could create significant operational challenges. This metric could be used to evaluate various practices to mitigate that risk. − Management/advisory/administrative/technical positions: Turnover of supervisory and technical personnel is especially detrimental if critical institutional knowledge and intel- lectual capital are lost after employees leave (Rodgers, 2002). For supervisory and technical roles, turnover can result in a loss of specific information (explicit knowledge) that only the departing employee knows as well as general know-how (implicit knowledge) that may take replacement employees a long time to acquire on their own. This metric can be used to evaluate practices that may not necessarily affect the number of employees who leave an organization but that encourage documentation and sharing of knowledge among employ- ees to reduce institutional knowledge lost as a result of turnover. • How this metric can be measured/implemented: − Frontline positions: To determine the impact of frontline turnover on knowledge loss, data can be collected at regular intervals through service delivery records, on-time performance, accident and incident records, customer complaints, and customer surveys both before and after implementation of practices expected to affect retention. By comparing performance over a period of time or before and after a specific event, the organization can estimate the impact that knowledge loss has had on organizational performance. Alternately, when comparing various practices, the organization can estimate the impact that each practice will have on operational performance due to the loss of experienced employees. This will require the organization to project both the number of employees expected to be lost and the impact that employees in various roles have on performance. − Management/advisory/administrative/technical positions: To determine the impact of future knowledge loss, data can be collected through employee knowledge management assessments prior to anticipated turnover to identify both the tacit/implicit knowledge (i.e., knowledge that is often unspoken and difficult to transfer) and explicit knowl- edge (i.e., specific facts, information, and processes that can be easily documented) that employees hold. When an employee leaves, these data can then be used to determine the percent of loss relative to retained knowledge held by remaining staff. If the percentage of knowledge loss is greater for a particular practice, it would have a correspondingly lower score on this metric.

2-22 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach 2.3.3 Interpretation of Results The targeted retention practice being assessed with this scorecard should be one that results in positive scores on each of these metrics. For example, an agency might implement flexible schedules that allow eligible employees to work outside of the typical 9 a.m. to 5 p.m. workday. This can aid in retention by helping employees reduce their work-life conflict by allowing time to address nonwork obligations that can be difficult to complete after 5 p.m. Using data or pro- jections around the defined metrics, the organization could better determine if this initiative is the best use of retention resources. The organization might also be able to conclude whether this particular retention practice should be refined or replaced with an alternate practice, such as implementing a mentoring program to help retain high potential employees. Sample transit case study – applying retention metrics. A small urban bus transit agency located in the Great Plains had been experiencing problems retaining skilled mechanics. This problem was affecting the transit agency in three critical areas. First, the lack of a fully staffed and effective maintenance team made it difficult for the transit agency to meet the vehicle pre- ventive maintenance schedule, which had the potential to create the need for more corrective maintenance and difficulties in making vehicle pullout. Second, the cost of continually filling open maintenance positions was beginning to strain the agency’s recruitment budget. Lastly, mechanic turnover was negatively affecting the level of technical skill and knowledge available on the shop floor. Based on these issues, the general manager met with the personnel officer to begin to objectively measure the effect of the maintenance retention problems; if these results indicated a need, they would explore and measure new strategies to improve mechanic retention. A decision was made to use the retention metrics in this module. The personnel officer then initiated an evaluation process of mechanic retention. This process was based on personnel records that documented when mechanics were hired, how long they stayed with the agency, and the percentage of employees that voluntarily left the agency within given time periods. The results of this evaluation confirmed that the present mechanic retention level was in fact low and had a serious possibility of ultimately creating catastrophic maintenance failure. The general manager and the personnel officer decided it would be necessary to experi- ment with new organizational approaches that had the potential to improve mechanic retention by making the transit system a more attractive place to stay employed. To determine what was causing mechanics to leave the agency, the personnel officer formally surveyed the existing main- tenance team on their feelings about the organization and why they felt mechanics were choosing to leave. The personnel officer also began an exit interview process whereby feedback was gath- ered from mechanics that were leaving the organization as to why they were leaving. Based on this input, the benefits package available to mechanics was modified to allow for more individual tailoring, external maintenance skill development offerings through a local technical school were offered to mechanics, and maintenance shift hours and days were re-engineered to provide each mechanic with some time off during the desirable periods of the week, including weekends. These new maintenance retention strategies were initiated and then measured for effectiveness using the Retention Practices Scorecard presented in this module. The measured results of the new retention practices for mechanics were then compared to the retention measurements prior to instituting the new practice. These comparisons confirmed the enhanced effectiveness of the new practices designed to improve mechanic retention. Based on this assessment, the agency decided to continue to use these new approaches on an ongoing basis to maximize mechanic retention. This new practice resulted in a reduction in mechanic turnover and led to improved adherence to preventive maintenance schedules, reduced costs for mechanic recruitment, and an increased maintenance skill level across all shifts on the shop floor. Please see Exhibit 2-6 and Exhibit 2-7 for the Retention Practices Scorecards.

Title of Practice: [Enter here] Metrics Retention-Specific Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 1. Employee voluntary turnover rate Greater than 20% turnover Greater than 10% up to 20% turnover Greater than 5% up to 10% turnover Less than 5% turnover 2. Cost to fill open positions More than budget target Right at budget target Slightly under budget target Significantly under budget target 3. Diversity turnover More than 50% minority employees turn over in 1st year of hire. Turnover of minorities significantly greater than nonminorities within 1st year of hire Turnover of minorities slightly greater than nonminorities within 1st year of hire. Turnover of minorities equal to nonminorities within 1st year of hire 4. Employee engagement Less than 25% of employees indicate high engagement. 25% up to 50% of employees indicate high engagement. Greater than 50% up to 75% of employees indicate high engagement . Greater than 75% of employees indicate high engagement. 5. Impact of turnover on employee knowledge Greater than 50% decline in employee knowledge over 5 years Greater than 25% up to 50% decline in employee knowledge over 5 years Greater than 0% up to 25% decline in employee knowledge over 5 years No negative impact on employee knowledge Sub-score: Metrics Global Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 1. Stakeholder buy-in Stakeholders unaware of program Stakeholders know of program but have a lot of questions. Stakeholders understand program and only have a few questions. Stakeholders already on board 2. Time to implement Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 3. Cost to implement More than budget target Right at budget target Slightly under budget target Significantly under budget target 4. Full return on investment Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 5. Sustainability One-time program Program information must be continually updated to remain current. Program can be updated annually and reused. Program can be continually used with minimal maintenance. Sub-score: Notes: Likelihood of Success Going Fwd Total Metrics Score Success Very Unlikely 0–250 Success Unlikely 251–500 Success Likely 501–750 Success Very Likely 751–1000 Title of Practice: [Enter here] Metrics Retention-Specific Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 1. Employee voluntary turnover rate Greater than 10% turnover Greater than 5% up to 10% turnover Greater than 2% up to 5% turnover Less than 2% turnover 2. Cost to fill open positions More than budget target Right at budget target Slightly under budget target Significantly under budget target 3. Diversity turnover More than 50% minority employees turn over in 1st year of hire. Turnover of minorities significantly greater than nonminorities within 1st year of hire Turnover of minorities slightly greater than nonminorities within 1st year of hire Turnover of minorities equal to nonminorities within 1st year of hire 4. Employee engagement Less than 25% of employees indicate high engagement. 25% up to 50% of employees indicate high engagement. Greater than 50% up to 75% of employees indicate high engagement. Greater than 75% of employees indicate high engagement. 5. Impact of turnover on employee knowledge Major impact on employee knowledge and related outcomes Significant impact on employee knowledge and related outcomes Limited impact on employee knowledge and related outcomes No negative impact on employee knowledge and related outcomes Sub-score: Metrics Global Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 6. Stakeholder buy-in Stakeholders unaware of program Stakeholders know of program but have a lot of questions. Stakeholders understand program and only have a few questions. Stakeholders already on board 7. Time to implement Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 8. Cost to implement More than budget target Right at budget target Slightly under budget target Significantly under budget target 9. Full return on investment Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 10. Sustainability One-time program Program information must be continually updated to remain current Program can be updated annually and reused. Program can be continually used with minimal maintenance. Sub-score: Notes: Likelihood of Success Going Fwd Total Metrics Score Success Very Unlikely 0–250 Success Unlikely 251–500 Success Likely 501–750 Success Very Likely 751–1000 Exhibit 2-6. Retention Practices Scorecard – management/advisory/administrative/technical positions. Exhibit 2-7. Retention Practices Scorecard – frontline positions.

2-24 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach 2.4 Training and Development Overview and Scorecards This section provides an overview of training and development as they relate to public trans- portation organizations. In addition, training and development performance metrics are intro- duced, and the respective scorecards are described. 2.4.1 Background One of the most significant findings in the workforce chal- lenge study in TRB Special Report 275 (TRB, 2003) is that there is a significant deficiency in training among public transpor- tation organizations; there is therefore a great need to allo- cate additional resources to this area to prepare personnel to meet their job requirements. Studies of other industries have revealed that most private and public agencies spend approxi- mately 2% of salaries on training, whereas transit organiza- tions on average only spend about 0.5% of salaries on training (TRB, 2003). Training for new technologies is an area of par- ticular concern for the industry, given that public transporta- tion organizations are using new technology to help improve service and lower costs (Lave and Mathias, 2000). It is impera- tive that public transportation employees be trained to use new technologies required to effectively perform their work. As noted in TCRP Report 139, in the public transportation industry, during times of financial difficulty, training and development programs are typically among the first victims of budget cuts (Anderson et al., 2010). These cost-cutting mea- sures are often in direct conflict with the needs of organiza- tions to better train their employees on important job skills, provide them with new and useful certifications, and equip them to use new technology. Public transportation organiza- tions are also challenged with determining the right amount and variety of training courses. At times, the training budget may not be sufficient to provide a needed training course. In addition, organizations often struggle to find effective trainers to lead programs (Anderson et al., 2010). A prevailing thought throughout the industry is that using current employees as training instructors, as opposed to using external training consultants, could be a more effective and cost-saving method of training employees. However, finding incumbents with good communications skills can often be a challenge (Cronin et al., 2011). As stated in TCRP Report 139, “Training is typically deliv- ered using a structured seminar, course, or online instruc- tion that occurs over a finite period of time to teach a specific skill, skill set, or competency. Development refers to a broader approach to helping employees grow professionally” (Ander- son et al., 2010, p. 34). The decision of whether to train an employee or encourage the employee to participate in some type of developmental activity should be based on the desired Training and Development Metric Highlights ¾ Public Transportation Workforce Target • Teaching required knowledge and skills. • Bridging recruitment and retention processes and supporting professional capacity-building efforts. Professional Capacity Building Recruitment Processes Training and Develop- ment Retention Processes ¾ Importance to Public Transportation • Changing technologies in addition to budgetary cuts that threaten training and development pro- grams contribute to critical need to train employees to be effective. ¾ Key Public Transportation Challenges • Budgetary cuts. • Changing technologies. • Lack of existing training programs. ¾ Targeted Public Transportation Metrics • Pre- and post-training knowledge/skill testing. • Employee performance post-training. • Percent of employees rating training as job-relevant. • Impact of training on system operations, safety, and customer service. • Percent of trainees satisfied with training. • Recency of training materials.

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-25 outcome and need of the employee. As previously mentioned, this is not always the case in public transportation organizations, with the cost of training employees often becoming the deciding factor in whether to provide access to training. However, the failure to adequately train and fur- ther develop employees can lead to poor morale and reduced organizational commitment, and ultimately can result in retention issues. The costs of not training and developing employees can significantly outweigh the savings of cutting training and development initiatives (Anderson et al., 2010). Therefore, it is critical that public transportation organizations not only imple- ment training and development practices, but also have ways to gauge the effectiveness of these practices. 2.4.2 Defining the Scorecard The Training and Development Practices Scorecards provide five targeted metrics to assist organizations in assessing the extent to which training and development programs are prepar- ing public transportation employees for upcoming changes and challenges to their work. The metrics are defined in the following. Metric 1: Pre- and post-training knowledge/skill testing • What this metric is: Pre- and post-training knowledge/skill testing is a measure of the knowledge or skills acquired during a training program. Specifically, this metric assesses whether attendance in a training program helped trainees gain knowledge and skills that are necessary to perform their jobs. For this metric, both pre- and post-tests are used so that any learning can be attributed to the training program that trainees complete. Pre- and post-training knowledge/skill testing uses assessments that target the learning objectives of a training program to ensure that partici- pants are achieving the intended learning goals. • Why this metric is of value to public transportation: Pre- and post-training knowledge/skill testing can help determine the actual amount of learn- ing that occurs during a training program (Kirkpatrick, 1996). When both a pre- and post-test are used, trainees’ change in knowledge or skill over the course of the training program can be assessed, and the effectiveness of the training program understood. If only a post-test were conducted, it would be difficult to know if trainees came into the program with the necessary knowledge and skills or if they were successfully taught during the training program. • How this metric can be measured/implemented: Pre- and post-training knowledge/skill testing can be measured through assessment of knowl- edge and skills using a test both before and after the training program. The test should be directly tied to the training content to ensure that the learning goals of the training pro- gram are being met. Additionally, the same test or a test with the same content but different questions should be used for the pre- and post-test so that learning can be directly assessed. When measuring pre- and post-training knowledge/skill testing, it is necessary to make sure that all trainees complete the testing so that there are complete data about the learning that occurred across trainees during a given training program. Because training for frontline and management/advisory/administrative/technical positions may have different foci, ways to implement the measuring of pre- and post-training knowledge/skill testing for these two types of positions are described separately: − Frontline positions: It may often be the case that training programs designed for front- line positions will be based on skills these employees need to do their jobs. For example, a training program may be designed to teach bus operators how to assist passengers in

2-26 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach wheelchairs with getting onto the bus. In a training program such as this, there is a specific skill that is being taught and can be observed. For these skill-based training programs, a performance test should be used to measure skills gained during training. This should not be on-the-job performance that is measured, but rather a test that is set up in association with the training to measure skills gained before the trainees are sent to actually perform the job. Alternatively, if a training program for frontline employees is knowledge-based, the implementation process described for management/advisory/administrative/technical positions should be followed. − Management/advisory/administrative/technical positions: The training objectives for management/advisory/administrative/technical positions will likely be knowledge-based, meaning that there is not a physical skill that can be observed to show that learning has occurred. For example, a training program may be designed to help transit managers learn about the bus procurement process and to teach about procurement requirements and reg- ulations. In this case, a paper and pencil or computer-based test should be used to measure knowledge gained during the training (Kirkpatrick, 1996). The test should be developed to align with the specific training objectives of the course. Developers of these tests should also make sure that all questions on the test are covered during the course of the training program. Metric 2: Employee performance post-training • What this metric is: Employee performance post-training is a metric designed to measure the on-the-job perfor- mance of employees who have completed a training program. This metric is used to determine if the training program has the intended effect on the actual job performance of employees. While the way in which employee performance is measured may differ based on the employ- ee’s position, employee performance post-training should be measured in a consistent way across employees within the same positions. For example, the same performance evaluation form should be used for all train operators; however, this evaluation form may differ from that used to evaluate bus mechanics. • Why this metric is of value to public transportation: Employee performance post-training identifies whether the training program being evalu- ated is having an actual impact on the work conducted. There is a difference between learning knowledge or skills in a training program and actually applying that information on the job (Kirkpatrick, 1996); while training participants may learn all of the necessary knowledge or skills during a training program, if they do not apply the knowledge and skills once they are on the job, the training was not necessarily effective. Measuring employee performance following a training program is important for both frontline and management/advisory/administrative/ technical positions since the goal of all training is to improve performance or give employees the knowledge and skills necessary to effectively perform their jobs. • How this metric can be measured/implemented: Data can be collected through employee performance assessments that occur approximately 3 months after the training and target the competencies addressed by the learning objectives of the training. This gives trainees the opportunity to put what they have learned during train- ing into practice (Kirkpatrick, 1996). Additional performance assessments can be conducted at later times to ensure that performance on the job continues to be acceptable. When gather- ing data regarding employee performance post-training, appraisals of performance should be gathered from one or more sources. These sources can include supervisors, subordinates, peers, customers, or other individuals who are familiar with the employees’ performance on

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-27 the job (Kirkpatrick, 1996). For example, one focus group representative noted that his orga- nization uses customer input to help determine if remedial training is necessary for frontline employees or if training has been successful. − Frontline positions: Employee performance post-training can be successfully used to eval- uate the effectiveness of training programs developed for frontline employees. For example, a focus group participant described that training effectiveness of a bus operator training program is measured by looking at accidents, complaints, and operating violations that are reported. These measures show the performance of the bus operators once they are on the job and are related to the training received. In addition to these measurements of bus operator performance, supervisory and onboard instructor evaluations are completed. Including these evaluations with the measurements is an example of how evaluations from multiple sources can be incorporated. − Management/advisory/administrative/technical positions: Employee performance post- training can also be used to evaluate the effectiveness of training programs developed for management/advisory/administrative/technical positions. For example, in a training pro- gram designed to teach transit supervisors about their role in safety, trainees may learn about their responsibilities for conducting safety training, documenting safety incidents, and supporting a drug and alcohol policy for all employees. Their performance could then be measured by checking their documentation of safety incidents to ensure completeness and accuracy as well as asking their subordinates if required safety training was conducted and how it went. Metric 3a: Percent of employees rating training as job-relevant (management/advisory/ administrative/technical positions only) • What this metric is: Percent of employees rating training as job-relevant indicates the percentage of management/ advisory/administrative/technical employees that participate in a training program and sub- sequently perceive that it is related to their job. This metric applies to job-specific training and not policy training such as workplace conduct, company policy, or sexual harassment training. This caveat is included because while some employees may not see this type of policy training as relevant to daily job requirements, it is still important for all employees to be aware of any policies and codes of conduct that are in place. This metric, percent of employees rating training as job-relevant, applies to management/ advisory/administrative/technical positions because job-related training is likely to be seen by employees in these positions as relevant to the work that they do. On the other hand, focus group participants noted that trainees in frontline positions too easily believe that training is not relevant to them based on surface-level information. For example, one focus group par- ticipant stated that if a bus operator drives a large bus but the training bus is small, they will quickly decide that because of the size of the bus, the training is not relevant to them. • Why this metric is of value to public transportation: If trainees perceive skill-related training to be irrelevant to their jobs in that the learning objectives do not directly align with the competencies required on the job or the content of the training does not include realistic scenarios, then it is unlikely that the training program will be effective. Focus group participants noted that this is an important metric. Trainees tend to tune out quickly and no longer pay attention to the information given if they perceive the training to be at all irrelevant to their jobs. If this occurs, employees will not gain the informa- tion necessary to do their jobs well, which can hurt their performance as well as the overall effectiveness of the organization.

2-28 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach • How this metric can be measured/implemented: Data to evaluate the percent of employees rating training as job-relevant can be collected using an employee self-report survey assessing training relevance. This survey should be given to employees after they complete the training program. If the training program is designed to introduce new employees to their jobs, the survey should not be given until they have been able to spend time in their new position so that they know what that position actually entails. Examples of statements that could be included in a survey to determine if trainees feel that the training is job-relevant include the following: − This training program helped me to develop knowledge or skills that I use in my job. − The content of this training program was relevant to my job. − This training program was relevant to our overall organization. Metric 3b: Impact of training on system operations, safety, and customer service (frontline positions only) • What this metric is: This metric indicates the potentially positive impact on system operations, safety, and cus- tomer service that can occur when frontline employees have completed training. The impact of training on system operations, safety, and customer service metric provides information about important outcomes that are often affected by frontline employee training. While this metric is valuable to measure for frontline positions, it is not relevant for management/advisory/administrative/technical positions. Typically, supervisory and techni- cal positions, while they may have a secondary influence, do not have a direct impact on the daily operations, safety, and customer service of public transportation organizations. For this reason, when evaluating the effectiveness of training programs, it is not helpful to consider this metric for training developed for management/advisory/administrative/technical positions. • Why this metric is of value to public transportation: Ineffective training of frontline employees can be detrimental to system operations, employee and passenger safety, and the delivery of high-quality customer service. To ensure that train- ing is effective and public transportation organizations are being successful in these areas, it is necessary to measure these outcomes and be aware of how they are affected by training programs in which frontline employees participate. • How this metric can be measured/implemented: To determine the impact of training on system operations, safety, and customer service, data can be collected through on-the-job evaluations, service delivery records, on-time perfor- mance, accident and incident records, customer complaints, and customer surveys. These measures provide valuable information about the work completed by frontline employees. When measuring these outcomes, it is important to make sure that the topics covered in training as well as the training objectives are related to the measures being used to evaluate the impact of training on system operations, safety, and customer service. Metric 4: Percent of trainees satisfied with training • What this metric is: The percent of trainees satisfied with training metric identifies the percent of trainees who are happy with a training program they have completed. This metric is measured by calculat- ing trainee satisfaction with training based on responses to questions or a survey following the training program—typically immediately after the program is completed. The percent of

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-29 trainees satisfied with training can include satisfaction with multiple facets of training, includ- ing the training program itself, the trainer/facilitator, the training topic, the training schedule, and perceived impact of the training on one’s own performance. • Why this metric is of value to public transportation: The percent of trainees satisfied with training can affect the extent to which the trainee trans- fers skills or competencies taught in training to his or her job. If participants do not like a training program, there is little chance that they will put forth the effort to fully engage in and learn from it (Kirkpatrick, 1996). In other words, participants must like a training program in order to be able to get the most out of it. This metric is also important in that the responses gathered from trainees regarding the various aspects of a training program can be used to help improve the training program for future participants. For example, if frontline bus operators indicated that they were not satisfied with the trainer because he or she spoke too fast during the training session, and they thus could not comprehend everything that was discussed, this feedback could be shared with the trainer, who would then know to speak more slowly. The trainer in this scenario could also learn to ask the participants if they understand the informa- tion being covered when leading future training sessions. • How this metric can be measured/implemented: Data regarding the percent of trainees satisfied with training can be collected using post- training attitude surveys of training participants. These surveys should be designed to gather information about multiple aspects of the training program and be provided to all partici- pants following the program. A focus group participant noted that her transit system conducts satisfaction surveys following each training program to capture participant attitudes to the course and that the system finds this information useful. When designing a survey to gather data regarding trainees satisfied with training, all questions should be designed such that the responses can be combined and quantified (Kirkpatrick, 1996). Results will be easier to understand and actionable if responses across training participants are easily combined into one set of ratings. Ensuring that responses can be combined and quantified can be accomplished by asking questions with set, numeric response scales. For example, ask training participants to respond to each item on the survey using the following scale: 1 = Strongly disagree 2 = Disagree 3 = Neither agree nor disagree 4 = Agree 5 = Strongly agree Using this scale, responses from all training participants can be summed to better understand reactions to the training program. Additionally, the percent of trainees satisfied with training can be assessed by determining the percentage of trainees who provide a response of either 4 or 5 on this scale. Some examples of statements that could be used on a survey to measure trainee satisfaction with training are: 1. I enjoyed this training course. 2. Concepts in the course were clearly and fully explained. 3. The course was interesting and easy to follow. 4. This course was an effective use of my time. 5. Course materials, handouts, or videos helped me to understand the training content. 6. The training facilitator was easy to understand and follow. 7. The training facilitator fully answered questions raised during the training.

2-30 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach 8. Overall, the training facilitator was a good teacher. 9. Overall, this was a good training course. 10. I would recommend this course to colleagues. While this is not an exhaustive list of items that can be used, items such as these provide information regarding the percent of trainees satisfied with training. When evaluating two training programs and comparing them to one another, the same questions should be asked to participants in both training programs. Metric 5: Recency of training materials • What this metric is: The recency of training materials metric specifies the amount of time elapsed since training materials were last updated. The assumption associated with this metric is that the more recent the materials are, the more relevant the training will be. With ever-changing technolo- gies, practices, and regulations, training programs can very easily become outdated. If training programs do not match the current environment of the public transportation organization, employees may not be trained in the correct areas or ways to accomplish job tasks. • Why this metric is of value to public transportation: The public transportation environment and demands for transit services are constantly changing. Consequently, training must be regularly refined in order for it to remain relevant to advancements in equipment and technology, evolving service delivery requirements, and current staff development needs. Focus group participants supported the notion that training materials should be regularly updated to ensure that they are relevant. For example, training participants may key into things such as the style of dress in a training video, and some- thing this small can still make an impact on their perceptions of the training. If trainees believe that training materials are not recent and therefore not relevant, they will likely get less out of the training and be less likely to transfer what they learn to their performance on the job. • How this metric can be measured/implemented: Data regarding the recency of training materials can be collected using internal monitor- ing and audits of training materials. A system should be set up that documents all training materials used by the public transportation organization. This system could be as simple as a Microsoft Excel worksheet that includes space to track information such as: − Training program name, − When the training program was created, − When the training program and training materials were last updated, − When this training program is required, − Intended participants, and − Training objectives. The information regarding the date the training program was created and when the program and its materials were last updated can then be used to determine the recency of training materials. This documenting system could also be used to make sure that no training pro- grams are too outdated to be effective. 2.4.3 Interpretation of Results The target training and development practice being assessed with this scorecard should be moving an organization’s standing on each of these metrics in a positive direction. For

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-31 example, an organization might implement a computerized job simulation to help light rail drivers improve their driving skills. Aspects of this practice could include ways to improve the punctuality of light rail and training on the exact timing involved to stop the train at a light rail station. Using data or projections around the defined metrics, the organization could better determine if this initiative is the best use of training and development resources. The organization might also be able to conclude whether this particular training and development initiative should be refined or replaced with an alternate practice, such as a more hands-on training experience. Sample transit case study – applying training and development metrics. A rural bus tran- sit agency located in New England had been experiencing a significant increase in preventable bus accidents. This problem had begun affecting the transit agency in a number of areas, includ- ing increased lawsuits and higher insurance premiums; the need to pull an unacceptable number of vehicles out of service after accidents, which affected the ability to provide required services; and an increase in bus driver vacancies and recruitment efforts due to driver terminations based on preventable accident records. The transit manager met with the individual responsible for driver training to discuss the need to evaluate training effectiveness. Once that evaluation was completed, the organization would decide whether the new-hire and refresher driver training programs would need to be significantly modified. A decision was made to use the training and development metrics in this module. The transit manager and trainer then initiated an evaluation process of the new-hire and refresher driver training programs. This process was based on an examination of post-driver training behind-the-wheel evaluations, supervisory performance evaluations immediately after training and annually, and transit agency accident records over the past 2 years. The results of this evaluation confirmed that the existing driver new-hire and refresher training programs required dramatic revision in the area of defensive driving skills. The training manager and trainer then decided it would be necessary to experiment with new training curricula and training deliv- ery approaches, both for new-hire and refresher training, to improve driver performance and enhance defensive driving skills. New-hire driver training was revamped to extend the train- ing period by 1 week, update old training materials, incorporate the National Safety Council defensive driving training module, and formalize driver post-training/knowledge skill testing. The approach to driver refresher training was revised to require 1 day of refresher defensive driving training for each transit driver every 3 years using the basic elements of the National Safety Council defensive driving training module. An additional component incorporated into refresher training was a mandatory supervisory behind-the-wheel evaluation of each driver to be conducted the day after the refresher training experience. This evaluation included coaching as might be necessary and was mandated to occur before that experienced driver was allowed to return to revenue service. These new training and development strategies used for drivers were initiated and then eval- uated for effectiveness using the Training and Development Practice Scorecard. The measured results of the revised new-hire and refresher driver training programs were then compared to the driver training measurements prior to instituting the new program. These comparisons confirmed the effectiveness of the revamped driver training program, both for new and experi- enced drivers. Based on this assessment, the transit agency institutionalized the redesigned driver training program for new hires and experienced drivers. Within 2 years, the transit agency realized a significant reduction in serious accidents, experienced 75% fewer lawsuits, received a reduction in insurance premiums, and found that driver turnover due to terminations was cut in half. Please see Exhibit 2-8 and Exhibit 2-9 for the Training and Development Practices Scorecards.

Title of Practice: [Enter here] Metrics Training and Development-Specific Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 1. Post-training knowledge/skill testing Testing indicates less than 60% post-training effectiveness Testing indicates 60% up to 79% post-training effectiveness Testing indicates 80% up to 89% post-training effectiveness Testing indicates 90% or greater post-training effectiveness 2. Employee performance post- training Performance decreases post-training Performance remains the same post-training. Performance improved somewhat post-training. Performance is greatly improved post-training. 3. Percent of employees rating training as job-relevant Less than 50% rate as job- relevant 50% up to 75% rate as job- relevant 76% up to 90% rate as job- relevant Greater than 90% rate as job-relevant 4. Percent of trainees satisfied with training Less than 50% satisfied 50% up to 75% satisfied 76% up to 90% satisfied Greater than 90% satisfied 5. Recency of training materials Greater than 10 years old Greater than 5 years up to 10 years old 1 year up to 5 years old Less than 1 year old Sub-score: Metrics Global Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 6. Stakeholder buy-in Stakeholders unaware of program Stakeholders know of program but have a lot of questions. Stakeholders understand program and only have a few questions. Stakeholders already on board 7. Time to implement Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 8. Cost to implement More than budget target Right at budget target Slightly under budget target Significantly under budget target 9. Full return on investment Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 10. Sustainability One-time program Program information must be continually updated to remain current. Program can be updated annually and reused. Program can be continually used with minimal maintenance. Sub-score: Notes: Likelihood of Success Going Fwd Total Metrics Score Success Very Unlikely 0–250 Success Unlikely 251–500 Success Likely 501–750 Success Very Likely 751–1000 Title of Practice: [Enter here] Metrics Training and Development-Specific Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 1. Post-training knowledge/skill testing Testing indicates less than 60% post-training effectiveness Testing indicates 60% up to 79% post-training effectiveness Testing indicates 80% up to 89% post-training effectiveness Testing indicates 90% or greater post-training effectiveness 2. Employee performance post- training Performance decreases post-training Performance remains the same post-training. Performance improves somewhat post-training. Performance is greatly improved post-training. 3. Impact of training on system operations, safety, and customer service Negative impact on operational safety and effectiveness No impact on operational safety and effectiveness Some positive impact on operational safety and effectiveness Significantly positive impact on operational safety and effectiveness 4. Percent of trainees satisfied with training Less than 50% satisfied 50% up to 75% satisfied 76% up to 90% satisfied Greater than 90% satisfied 5. Recency of training materials Greater than 10 years old Greater than 5 years up to 10 years old 1 year up to 5 years old Less than 1 year old Sub-score: Metrics Global Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 6. Stakeholder buy-in Stakeholders unaware of program Stakeholders know of program but have a lot of questions. Stakeholders understand program and only have few questions. Stakeholders already on board 7. Time to implement Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 8. Cost to implement More than budget target Right at budget target Slightly under budget target Significantly under budget target 9. Full return on investment Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 10. Sustainability One-time program Program information must be continually updated to remain current. Program can be updated annually and reused. Program can be continually used with minimal maintenance. Sub-score: Notes: Likelihood of Success Going Fwd Total Metrics Score Success Very Unlikely 0–250 Success Unlikely 251–500 Success Likely 501–750 Success Very Likely 751–1000 Exhibit 2-8. Training and Development Practices Scorecard – management/advisory/administrative/ technical positions. Exhibit 2-9. Training and Development Practices Scorecard – frontline positions.

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-33 2.5 Professional Capacity-Building Overview and Scorecard This section provides an overview of professional capacity building as it relates to public transportation organizations. The purpose of using the term “professional” is simply to sug- gest capacity within the workplace as opposed to personal capacity. In this section, professional capacity-building performance metrics are also introduced, and the relevant scorecards are described. 2.5.1 Background The retirement of the baby boomer generation has been cited as one of the greatest challenges facing transportation agencies, as described in TCRP Synthesis 40 (Moffat, Ashton, and Blackburn, 2001). Studies indicated that 50% of the transit workforce would become eligible to retire by 2009 (e.g., New Mexico State Highway and Transportation Department, 1999). Given the state of the economy in 2009, many retirement- eligible staff did not retire. Thus, in addition to these employees who are ready and waiting to retire, more workers are becom- ing eligible to retire with each year that passes. This creates a bubble of workers who will likely leave as soon as the econ- omy improves and their retirement plans begin to rebound. In many cases, those eligible to retire are the only ones who possess specialized knowledge and unique experience that are critical for efficient operation of the organization (Rothwell and Poduch, 2004). These challenges have led to a greater emphasis on profes- sional capacity building in public transportation organiza- tions. Once employees who are near retirement decide to leave the organization, mechanisms and processes are needed that allow employees to rise up through the organization to assume the roles held by these recently departed employees. These professional capacity-building mechanisms include a variety of succession planning tools, methods that allow employees to gain knowledge regarding other relevant jobs in their organiza- tions, and mechanisms that allow employees to perform better at their current jobs. Measurement of these aspects is crucial to determining whether an organization is effectively develop- ing its current workforce while simultaneously preparing for potential future workforce challenges. Most organizations cannot function successfully after los- ing knowledgeable employees if newer employees do not pos- sess the institutional knowledge needed to perform effectively. Due to difficulties that are often involved in recruiting external employees, it is valuable to appropriately develop employees from within for future leadership roles in public transpor- tation organizations. These internal development practices could involve the promotion of high-performing front- line employees to supervisory or administrative jobs within Professional Capacity-Building Metric Highlights ¾ Public Transportation Workforce Target • Developing employees and building organizational knowledge. Professional Capacity Building Recruitment Processes Training and Development Retention Processes ¾ Importance to Public Transportation • Institutional knowledge may be lost when employ- ees leave the organization, and it is necessary to ensure that employees who possess key knowledge and skills are able to assume key positions. ¾ Key Public Transportation Challenges • Large numbers of retiring baby boomers. • Retirees are the only people who possess important knowledge. ¾ Targeted Public Transportation Metrics • Timeliness metrics. • Percentage of employees who participate in employee development opportunities. • Internal employee advancement metrics. • Employee involvement in organization’s decision making. • Percentage of jobs with up-to-date job descriptions.

2-34 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach a public transportation organization. This practice ensures that the higher levels of a public transportation organization contain employees with both frontline and management/advisory/ administrative/technical perspectives. When public transportation organizations promote prac- tices designed to increase their professional capacity, they will run more effectively and ulti- mately reduce future workforce planning challenges. 2.5.2 Defining the Scorecard The Professional Capacity-Building Practices Scorecard provides five targeted metrics to assist public transportation organizations in assessing the extent to which professional capacity- building initiatives are helping address present and future workforce challenges. The metrics are defined in the following. Metric 1: Timeliness metrics Metric 1a: Timeliness of task completion (management/advisory/administrative/ technical positions only) Metric 1b: Timeliness of scheduled activities (frontline positions only) • What this metric is: Timeliness metrics assess the punctuality, by workers, in completing existing management/ advisory/administrative/technical and frontline tasks. This is also referred to as the degree to which tasks are successfully completed and prioritized in an appropriate and beneficial man- ner, although prioritization is more of an implied measure for the timeliness of task comple- tion metric. The time period in question for the timeliness metrics should be consistent when comparing two practices across this metric. The degree to which an organization’s practices are effective in building professional capacity will be exhibited by the timely and successful completion, or lack thereof, of important employee tasks. − Management/advisory/administrative/technical positions: Management/advisory/ administrative/technical employees may need to complete a variety of tasks, such as a sub- ordinate’s performance review, a portion of their organization’s 10-year strategic plan, and review and revision of policies for employees (e.g., revising bus driver training) in a 1-week time period. The employee’s completion of the required tasks in the required 1-week time period indicates the timeliness of task completion. − Frontline positions: A frontline employee may need to operate a bus or a train on a certain fixed schedule. The employee’s performance on the timeliness of scheduled activities metric indicates how successful he or she is in contributing to the public transportation organiza- tion’s overall punctuality. • Why this metric is of value to public transportation: These metrics show the degree to which capacity-building programs are effective in help- ing employees complete work on time. Employees in organizations with ineffective capacity- building programs would likely have difficulties completing the majority of their tasks on time because they may not have all the required knowledge, skills, and abilities to complete the tasks, or they may be frequently completing tasks designated for other employees who are not timely in their task completion. − Management/advisory/administrative/technical positions: When management/advisory/ administrative/technical employees consistently complete their work in a timely manner, it serves as confirmation that an organization possesses sufficient professional capacity and that its employees are well trained and attentive to a task’s time constraints. If these

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-35 employees consistently complete tasks late, it reflects poorly on the entire organization and often leads to untimely task completion by lower-level employees. − Frontline positions: When frontline employees such as drivers consistently deliver timely service to customers (e.g., train and passengers), it reflects well on the entire organization. Conversely, untimely service will likely result in lower revenues and reduced ridership and could increase the public transportation organization’s operating costs. • How this metric can be measured/implemented: Public transportation organizations should maintain some type of employee time-monitoring system, and it is recommended that this be electronic. Each employee will record the time it takes to complete a specific project into the time-monitoring system. Data tracking the timeliness metrics can be gathered using these employee time-monitoring systems that track time spent per employee from task beginning to final task completion. For example, if it is anticipated that a task should take about 20 hours to complete, and employees are consistently reporting an average of 40 hours to complete it, it is possible that a public transportation orga- nization will need to better focus on capacity-building initiatives that will remove impedi- ments to employee performance and result in a more responsive workforce. Data tracking for the timeliness metrics can also be gathered from reviewing specific dead- lines for both frontline and supervisory/technical employees, then determining the percentage of time, within a given time period, that these deadlines were met. For example, if a train is scheduled to arrive at 8:00 a.m. at a certain station, and average arrival time at that station during a given week is 8:11 a.m., then the transit system needs to better focus on building the capacity of train conductors. As mentioned, all employee data must be aggregated to form an organization-level assess- ment before completing this metric on the scorecard. Further, in some cases, the timeliness metrics will be influenced by other factors, such as road construction, traffic, unexpected personal events, and sicknesses. This reiterates the importance of examining scores on this metric in conjunction with scores on other metrics. Metric 2: Percentage of employees who participate in employee development opportunities • What this metric is: Percentage of employees who participate in employee development opportunities indicates the percentage of employees who take advantage of both formal and informal opportunities to enhance their professional skills. Focus group participants indicated that this metric and its anchors differ between management/advisory/administrative/technical employees and frontline employees. Management/advisory/administrative/technical employees have more formalized development programs than frontline employees. − Management/advisory/administrative/technical positions: The percentage of employ- ees who participate in employee development opportunities measures the percentage of employees who, after their new-hire training, participate in any additional opportunity designed to enhance professional skills and provide a broader understanding of the context of their organization over a given time period. These types of opportunities are usually formalized programs, are varied, and include job rotational programs, mentoring/coaching programs, and leadership skills training. To calculate this metric, a public transportation organization must have records of employee participation in these programs. In defining this metric, it should be noted that required new-hire training does not count as career development. − Frontline positions: The percentage of employees who participate in employee develop- ment opportunities measures the percentage of frontline employees who take advantage of

2-36 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach employee development opportunities presented to them over a given time period. These types of opportunities are more informal than for management/advisory/administrative/ technical employees and could include such things as the opportunity to mentor a more junior bus driver, or a driver’s involvement in maintenance on a vehicle he or she drives regularly. In defining this metric, it should be noted that required new-hire training does not count as career development. • Why this metric is of value to public transportation: A higher percentage of employee participation in employee development opportunities results in improved skills and greater knowledge about an organization. This helps to ensure continued success for the organization by developing employees’ skills from within. Addi- tionally, these data show the current state of overall employee development within a public transportation organization and provide a gauge of internal talent for future job vacancies. For example, public transportation organizations scoring high on this metric will possess more talented employees, will likely retain more skilled employees, and will have the ability to promote employees from within. Thus, these public transportation organizations may have fewer vacant positions and lower recruiting expenses. • How this metric can be measured/implemented: Public transportation organizations can examine the percentage of employee participation in employee development opportunities for all employees by asking relevant questions on employee surveys. However, differences also exist in assessing this percentage by employee level. − Management/advisory/administrative/technical positions: When public transporta- tion organizations administer any formal employee development programs, they should maintain attendance records. In many cases, electronic employee development tracking systems can be used to track employee attendance in programs such as coaching, mentor- ing, job rotations, and progress monitoring. The number of employees participating in any development program should be summed and divided by the total number of employees within the group being considered. For example, if 900 of the 1,100 management/advisory/ administrative/technical employees in a large public transportation organization partici- pated in at least one development program during a given year, then the percentage of employee participation in employee development opportunities would equal 81.8%. − Frontline positions: The percentage of employee participation in employee development opportunities can be measured for frontline positions by asking employee engagement questions on system-wide surveys. For example, frontline employees answering “yes” to questions such as “I actively mentor newer employees in my field” or “I take advantage of development opportunities my organization provides me” most likely have participated in informal or formal types of development programs. The number of employees who indi- cate participation in activities such as these would then be divided by the total number of frontline employees in an organization. Metric 3: Internal employee advancement metrics Metric 3a: Percentage of jobs filled internally (management/advisory/administrative/ technical positions only) Metric 3b: Percentage of operational supervisory positions filled by frontline personnel (frontline positions only) • What this metric is: The internal employee advancement metrics are designed to indicate the percentage of open positions that are filled by internal hires in a public transportation organization. When jobs

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-37 are filled with internal candidates rather than with external candidates, public transportation organizations save money on recruitment and show that they have effective programs to build their current employees’ professional capabilities. − Management/advisory/administrative/technical positions: Percentage of jobs filled inter- nally indicates the percentage of open management/advisory/administrative/technical job positions in an organization that are filled with internal candidates, via promotion or other means of internal job replacement, rather than by external hires. For example, a current administrative employee in a public transportation organization may replace a retiring supervisor in the same organization. − Frontline positions: Percentage of operational supervisory positions filled by frontline per- sonnel indicates the percentage of open operational supervisory job positions in an orga- nization that are filled with internal frontline candidates via promotion or other means of internal job replacement rather than by external hires. Focus group participants indicated that the internal employee advancement metrics differ between frontline and management/ advisory/administrative/technical positions since a frontline employee’s initial job (e.g., bus driver, maintenance worker) is not generally filled by an internal candidate. An expe- rienced bus driver being promoted into an operational supervisory position is an example of this concept in practice. • Why this metric is of value to public transportation: This metric indicates how well a public transportation organization develops its employees to move up into internal job positions expected to be vacated and thereby promotes succession planning. An advantage of filling jobs from within is that internal employees already under- stand the organization, how it functions, and what is important to succeed in the organization. Filling job vacancies from within also means that employees develop a greater understanding of the organization by experiencing work in multiple positions. In ideal conditions, the most qualified employees to fill a job vacancy would be from within. Filling more job vacancies from within will save an organization time and money in recruiting external candidates. • How this metric can be measured/implemented: Public transportation organizations should keep accurate employment records, including information regarding which job vacancies are filled by internal hires and which jobs are filled by external hires. Data for the internal employee advancement metrics can then be gathered from these employment records. Percentage of jobs filled internally can be calculated by deter- mining the number of internal management/advisory/administrative/technical employees who are hired to vacant management/advisory/administrative/technical positions within an organization through various means, divided by the total number of management/advisory/ administrative/technical jobs filled over a given time period. Percentage of operational super- visory positions filled by frontline personnel can be calculated by determining the number of frontline employees promoted to vacant operational supervisory positions, divided by the total number of operational supervisory jobs filled over a given time period. For example, if 25 out of 30 operational supervisory positions are filled with internal frontline employees, then this percentage equals 83.3%. Metric 4: Employee involvement in organization’s decision making • What this metric is: The amount of employee involvement in organization’s decision making identifies the percent- age of decisions rated as important in an organization for which management solicits input from employees or groups of employees (i.e., centralized decision making versus decentralized decision making). Management and other relevant stakeholders should define what consti- tutes an important decision; generally this will be a decision that affects the vast majority of

2-38 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach a public transportation organization’s employees. The logic behind employee involvement in an organization’s decision making is that when a decision must be made that affects most or all employees, all levels of employees should have input to help decide what changes may need to be made in their daily work lives. This method is preferred over having a small group of managers make all the decisions for all employees. • Why this metric is of value to public transportation: The degree to which managers form meaningful partnerships with the employees who per- form the work will determine how effectively future capacity is built and process improve- ments are derived. When employees are able to participate in decision making, they learn about the organization and can better understand why specific decisions are made. Addition- ally, when employees feel that they have input in decisions that affect them in the workplace, they are more likely to be satisfied and to feel valued by the organization. Finally, in cases where management’s decisions affect the day-to-day life of a group of employees, that group of employees can offer management vital information that will help them make an informed and fair decision. • How this metric can be measured/implemented: Public transportation organizations may, formally or informally, possess records regard- ing employees who provided input for important organizational decisions. Data measuring employee involvement in an organization’s decision making can be calculated by examining these records of decisions that have a large impact on employees. Management must internally define what is considered “a large impact” on employees. Information on employee involve- ment in an organization’s decision making can also be gathered through antecedent ques- tions frequently presented on employee engagement surveys. Examples of items on employee engagement surveys that measure this are “Management asks me for input on decisions” and “I have a say in what happens in my organization.” These statements would be rated by employees on a numeric scale, and employee responses would be combined to determine the overall level of employee involvement in decision making. Metric 5: Percentage of jobs with an up-to-date job description • What this metric is: The term job is deliberately used in this metric instead of position because position refers to the work conducted by a specific person, whereas job refers to the requirements regard- less of the person. In the case of this metric, it is important that the job description not be person-specific. This metric measures the recency and relevance of job descriptions within an organiza- tion. Organizations that have better capacity-building practices will know the most current knowledge, skills, abilities, and duties required for a certain job. For example, a train main- tenance mechanic job description that is 5 years old will likely not list the most up-to-date knowledge of tools and technology needed to succeed in that job. A job description describes the responsibilities and duties of a specified job, and is up-to-date if it is relevant to the job as it currently occurs. The relevancy of a job description can be determined through job analysis or interviews/surveys with job incumbents. • Why this metric is of value to public transportation: Relevant job descriptions help ensure that incumbents and future workers understand their specific job requirements, subsequently streamlining how work is performed and reduc- ing the extent to which efforts are duplicated. A high percentage of jobs with an up-to-date

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-39 job description is desirable to provide present and future employees in an organization with insight into the job and the most current factors that determine success in a particular job. In addition, public transportation organizations with out-of-date job descriptions often will not be able to effectively assess candidates when needed to replace incumbents in a particular job. • How this metric can be measured/implemented: It is recommended that public transportation organizations perform regular audits of materi- als important to the organization; this includes audits of job descriptions. Data for the percent- age of jobs with an up-to-date job description can be calculated and collected by performing an audit of all job descriptions within an organization through employee input, determining the ages and current relevance of existing job descriptions, and dividing the number of job descriptions deemed up-to-date by the total number of jobs in the organization. Further, if the organization chooses, job descriptions for mission-critical jobs may be weighted more heavily when a number is computed than those for non–mission-critical jobs. Employee input should be solicited using interviews and surveys. A qualified human resources representative should perform this data collection when possible. Some public transportation organizations may need to hire external consultants to perform an audit’s essential tasks (i.e., data collection and determining current relevance of job descriptions). 2.5.3 Interpretation of Results The target professional capacity-building practice being assessed with this scorecard should be moving an organization’s standing on each of these metrics in a positive direction. For example, a public transportation organization might have a workforce development program that includes strategic planning reviews as an element to help build its professional capacity (e.g., Capital District Transportation Authority, 2009). These reviews evaluate the purpose, goals, and objec- tives of the practice and indicate where adjustments to it are needed. Using data or projections around the defined metrics, the public transportation organization could better determine if this practice, or an individual portion of it, is the best way to build professional capacity. The organization may conclude that one element of its workforce development practice is very effec- tive at addressing professional capacity building, whereas another element of the practice needs drastic improvement. Sample transit case study – applying professional capacity-building metrics. A mid-sized urban bus transit agency located in the southeastern region of the country had begun experienc- ing problems due to the retirement of a large number of baby boomer senior transit managers. This problem was affecting the transit agency because of losing the retirees’ institutional knowl- edge and because adequately skilled individuals were not available to replace the retirees. Ulti- mately, there was serious concern at the top of the organization that these two issues would soon begin to dramatically affect the quality of senior level decision making and leadership and lead to the inability of the transit agency to effectively carry out its mission. Based on this concern, the chairman of the transit board of directors, the transit chief executive officer, and the transit assis- tant executive officer for human resources conducted a series of meetings. The three agreed that they needed to objectively measure the impact of these retirements and their current approach to succession planning. A decision was made to use the professional capacity-building metrics presented in this module. The assistant executive officer for human resources then initiated an evaluation process of professional capacity building within the agency. This process was based on a review of junior and senior management performance evaluations, particularly in reference to timeliness of task completion; training records of junior and senior managers, including the percentage that participated in employee development activities over the previous year; personnel records to determine the percentage of junior or senior management positions filled internally;

2-40 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach and junior and senior manager job descriptions. The results of the Professional Capacity-Building Scorecard indicated that the lack of professional capacity-building opportunities and nonstruc- tured approach to succession planning within the transit agency were significant concerns for the board and senior management. These concerns, combined with the ongoing schedule of baby boomer retirements, had the potential to greatly reduce organizational effectiveness. Senior management then initiated a process to explore and experiment with new profes- sional capacity-building practices. To develop these new practices, the transit agency revised and updated management job descriptions emphasizing detailed job qualifications. A survey of junior and senior management was administered to identify opportunities managers felt would improve their upward mobility and organizational culture changes that would increase manage- rial engagement and involvement in decision making. Training and development records and programs were examined to identify existing managerial training and development opportuni- ties. Based on these activities, the board and senior management identified new training and development programs, both internal and external, that would be offered to junior managers; instituted a senior-to-junior manager mentoring program; increased junior manager autonomy on specific projects; and began holding monthly staff meetings to improve organizational com- munication, team building, decision making, and empowerment. Further, the transit agency incorporated all of this information to create a formalized approach to succession planning that incorporated the use of the updated job information, a systematic process of identifying high potentials among junior staff, and the identification of training and development programs relevant to the mission-critical jobs of most concern. These new professional capacity-building strategies were initiated, and the process of mea- suring for effectiveness began, again using the Professional Capacity-Building Practices Score- card. The measured results of the new practices were then compared to the measurements taken prior to instituting the new practices. After 1 year of experimentation, these comparisons con- firmed that the new professional capacity-building practices and approach to succession plan- ning had the potential to prepare junior managers to effectively replace senior manager retirees. Based on this assessment, the experimental practices were incorporated into the organizational culture and continue to be implemented on an ongoing basis. By providing more opportunities for skill development and upwardly mobility for junior managers, many positions vacated by retirees could be effectively filled internally, reducing the difficult, costly, and often unsuccessful task of recruiting senior managers from the outside. Please see Exhibit 2-10 and Exhibit 2-11 for the Professional Capacity-Building Practices Scorecards. 2.6 Applying the Scorecards Exhibit 2-12 provides a scorecard with tips on how an organization can best use the scorecards. Exhibit 2-13 and Exhibit 2-14 are real-world examples of how two different public transporta- tion organizations actually applied these scorecards to practices used by their organizations. The use of two different scorecards is present in these real-world examples: an application of the Recruitment Practices Scorecard and an application of the Training and Development Practices Scorecard. For each of these examples, a brief description of the practice being rated is provided. The real-world examples were gathered from online surveys the researchers conducted as part of this project to test the scorecards. The public transportation organizations were first asked to provide a rating and then provide an explanation of why that rating was selected. In Exhibit 2-13 and Exhibit 2-14, the rating category each chose is circled, the actual score that was selected can be seen in the last column of the exhibit, and the explanations for each of the ratings are provided in the boxes surrounding the scorecards.

Title of Practice: [Enter here] Metrics Professional Capacity-Building–Specific Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 1. Timeliness of task completion Less than 50% of tasks completed on time Greater than 50% up to 75% of tasks completed on time Greater than 75% up to 90% of tasks completed on time Over 90% of tasks completed on time 2. Percentage of employees who participate in employee development opportunities 20% or less of employees; system does not have opportunities Greater than 20% up to 40% of employees Greater than 40% up to 60% of employees Over 60% of employees 3. Percentage of jobs filled internally 25% or less of jobs filled with internal candidates Over 25% up to 50% of jobs filled with internal candidates Over 50% up to 75% of jobs filled with internal candidates Over 75% of jobs filled with internal candidates 4. Employee involvement in organization’s decision making Employees involved 10% or less of time Employees involved more than 10% up to 30% of time Employees involved more than 30% up to 50% of time Employees involved over 50% of time 5. Percentage of jobs with an up-to- date job description Less than 25% of jobs More than 25% up to 50% of jobs More than 50% up to 75% of jobs More than 75% of jobs Sub-score: Metrics Global Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 6. Stakeholder buy-in Stakeholders unaware of program Stakeholders know of program but have a lot of questions. Stakeholders understand program and only have a few questions. Stakeholders already on board 7. Time to implement Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 8. Cost to implement More than budget target Right at budget target Slightly under budget target Significantly under budget target 9. Full return on investment Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 10. Sustainability One-time program Program information must be continually updated to remain current. Program can be updated annually and reused. Program can be continually used with minimal maintenance. Sub-score: Notes: Likelihood of Success Going Fwd Total Metrics Score Success Very Unlikely 0–250 Success Unlikely 251–500 Success Likely 501–750 Success Very Likely 751–1000 Title of Practice: [Enter here] Metrics Professional Capacity-Building–Specific Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 1. Timeliness of scheduled activities Less than 70% of activities completed on time Greater than 70% up to 80% of activities completed on time Greater than 80% up to 90% of activities completed on time Over 90% of activities completed on time 2. Percentage of employees who participate in employee development opportunities 20% or less of employees; system does not have opportunities Greater than 20% up to 40% of employees Greater than 40% up to 60% of employees Over 60% of employees 3. Percentage of operational supervisory positions filled by frontline personnel Less than 25% of supervisory positions filled Over 25% up to 50% of supervisory positions filled Over 50% up to 75% of supervisory positions filled Over 75% of supervisory positions filled 4. Employee involvement in organization’s decision making Employees involved 10% or less of time Employees involved more than 10% up to 30% of time Employees involved more than 30% up to 50% of time Employees involved over 50% of time 5. Percentage of jobs with an up-to- date job description Less than 25% of jobs More than 25% up to 50% of jobs More than 50% up to 75% of jobs More than 75% of jobs Sub-score: Metrics Global Rating Scales (Based on Experience or Projections) Score 0–25 26–50 51–75 76–100 6. Stakeholder buy-in Stakeholders unaware of program Stakeholders know of program but have a lot of questions. Stakeholders understand program and only have a few questions. Stakeholders already on board 7. Time to implement Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 8. Cost to implement More than budget target Right at budget target Slightly under budget target Significantly under budget target 9. Full return on investment Over 1 year Over 6 months up to 1 year 3 up to 6 months Less than 3 months 10. Sustainability One-time program Program information must be continually updated to remain current. Program can be updated annually and reused. Program can be continually used with minimal maintenance. Sub-score: Notes: Likelihood of Success Going Fwd Total Metrics Score Success Very Unlikely 0–250 Success Unlikely 251–500 Success Likely 501–750 Success Very Likely 751–1000 Exhibit 2-10. Professional Capacity-Building Practices Scorecard – management/advisory/administrative/ technical positions. Exhibit 2-11. Professional Capacity-Building Practices Scorecard – frontline positions.

2-42 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach Professional Capacity Building Practices Scorecard – Management/Advisory/Administrative/Technical Positions Title of Practice: (Enter here) Metrics Professional Capacity Building-Specific Rating Scales (Based on experience or projections) Score 0-25 26-50 51-75 76-100 1. Timeliness of task completion • Less than 50% of tasks completed on time • Greater than 50% up to 75% of tasks completed on time • Greater than 75% up to 90% of tasks completed on time • Over 90% of tasks completed on time 2. Percentage of employees who participate in employee development opportunities • 20% or less of employees; system does not have opportunities • Greater than 20% up to 40% of employees • Greater than 40% up to 60% of employees • Over 60% of employees 3. Percentage of jobs filled internally • 25% or less of jobs filled with internal candidates • Over 25% up to 50% of jobs filled with internal candidates • Over 50% up to 75% of jobs filled with internal candidates • Over 75% of jobs filled with internal candidates 4. Employee involvement in organization’s decision-making • Employees involved 10% or less of time • Employees involved more than 10% up to 30% of time • Employees involved more than 30% up to 50% of time • Employees involved over 50% of time 5. Percentage of jobs with an up-to- date Job Description • Less than 25% of jobs • More than 25% up to 50% of jobs • More than 50% up to 75% of jobs • More than 75% of jobs Subscore: Metrics Global Rating Scales (Based on experience or projections) Score 0-25 26-50 51-75 76-100 6. Stakeholder Buy-In • Stakeholders unaware of program • Stakeholders know of program but have a lot of questions • Stakeholders understand program and only have few questions • Stakeholders already onboard 7. Time to implement • Over 1 year • Over 6 months up to 1 year • 3 up to 6 months • Less than 3 months 8. Cost to implement • More than budget target • Right at budget target • Slightly under budget target • Significantly under budget target 9. Full Return on investment • Over 1 year • Over 6 months up to 1 year • 3 up to 6 months • Less than 3 months 10. Sustainability • One time program • Program information must be continually updated to remain current • Program can be updated annually and reused • Program can be continually used with minimal maintenance Subscore: Notes: Likelihood of Success Going Fwd Total Metrics Score Success Very Unlikely 0-250 Success Unlikely 251-500 Success Likely 501-750 Success Very Likely 751-1000 Anchor points can be adjusted to meet the needs or specific situation of the organization using the scorecards. See page 2-4 in this module for instruction on revising metrics Performance on just one metric should not be used to determine if a metric is effective. Performance on multiple metrics should be considered when determining the effectiveness of a practice. Ratings can be anywhere within the range specified for the anchor that matches the organization’s situation. The purpose of the scorecards is not to compare results with other organizations, but rather to compare internal practices or make decisions on where to invest resources. The global metrics are the same across all organizational processes and can be used to compare different types of strategies. Identifies the types of positions the scorecard is designed for use with. Exhibit 2-12. Using scorecards. This exhibit includes tips on how to use the scorecard, including various pieces of information that were presented in the previous sections of this module. These examples serve as a quick reference in effectively implementing the scorecards to rate organizational practices.

Module 2: use Metrics to evaluate the Impact of Workforce Practices 2-43 Recruitment Practices Scorecard – Supervisory/Technical Positions Title of Practice: (Enter here) Metrics Recruitment-Specific Rating Scales (Based on experience or projections) Score 0-25 26-50 51-75 76-100 1. Time to fill position • Over 8 months • Greater than 5 up to 8 months • Greater than 2 up to 5 months • Less than 2 months 70 2. New hire turnover within the first year • More than 40% of new hires turnover • Over 20% up to 40% of new hires turnover • 10-20% of new hires turnover • Less than 10% of new hires turnover 25 3. New hire performance rating • Bottom 25% of employees • Slightly lower than average employees • Slightly higher than average employees • Top 25% of employees 75 4. Recruiting cost ratio • More than budget target • Right at budget target • Slightly under budget target • Significantly under budget target 26 5. Offer to acceptance ratio • Less than 40% of offers accepted • More than 40% up to 60% of offers accepted • More than 60% up to 80% of offers accepted • More than 80% of offers accepted 80 Subscore: 276 Metrics Global Rating Scales (Based on experience or projections) Score 0-25 26-50 51-75 76-100 6. Stakeholder Buy-In • Stakeholders unaware of program • Stakeholders know of program but have a lot of questions • Stakeholders understand program and only have few questions • Stakeholders already onboard 0 7. Time to implement • Over 1 year • Over 6 months up to 1 year • 3 up to 6 months • Less than 3 months 60 8. Cost to implement • More than budget target • Right at budget target • Slightly under budget target • Significantly under budget target 50 9. Full Return on investment • Over 1 year • Over 6 months up to 1 year • 3 up to 6 months • Less than 3 months 50 10. Sustainability • One time program • Program information must be continually updated to remain current • Program can be updated annually and reused • Program can be continually used with minimal maintenance 80 Subscore: 240 Notes: Likelihood of Success Going Fwd Total Metrics Score Success Very Unlikely 0-250 Success Unlikely 251-500 Success Likely 501-750 Success Very Likely 751-1000 “It might take 3 or 4 months to fill a staff position.” “Many are unaware of this practice." “It does not take long to develop a consistent interviewing strategy, which includes developing interview questions, scorecards, etc., that can be used for all positions.” “There is very low turnover within the first year. Most would occur within the probationary period or for misconduct after 1 year.” “Most of the staff employees are rated ‘consistently exceeds’ from year to year.” “Most of candidates are local, and we pay for local advertising.” “It is rare to receive a declination of an offer.” “I developed the basic interview guide and trained interview committee members. Hiring managers and interview committee members generally make few changes.” “New hires come up to speed rather quickly and start contributing within the first 6 months.” “Interview questions can be tweaked as the job changes, but the basic process is followed.” Exhibit 2-13. Real-world application of the Recruitment Practices Scorecard. This is an example of evaluating the use of the structured interview. Quotations are provided to explain why each rating was given. With an overall score of 516, this recruitment practice is likely to be successful and effective.

2-44 Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach Training and Development Practices Scorecard – Frontline Positions Title of Practice: (Enter here) Metrics Training and Development-Specific Rating Scales (Based on experience or projections) Score 0-25 26-50 51-75 76-100 1. Post-training knowledge/skill testing • Testing indicates less than 60% post-training effectiveness • Testing indicates 60% up to 79% post-training effectiveness • Testing indicates 80% up to 89% post-training effectiveness • Testing indicates 90% or greater post-training effectiveness 75 2. Employee performance post- training • Performance decreases post-training • Performance remains the same post-training • Performance improved somewhat post-training • Performance is greatly improved post-training 60 3. Impact of training on system operations, safety, and customer service • Negative impact on operational safety and effectiveness • No impact on operational safety and effectiveness • Some positive impact on operational safety and effectiveness • Significantly positive impact on operational safety and effectiveness 60 4. Percent of trainees satisfied with training • Less than 50% satisfied • 50% up to 75% satisfied • 76% up to 90% satisfied • Greater than 90% satisfied 90 5. Recency of training materials - • Greater than 10 years old • Greater than 5 years up to 10 years old • 1 year up to 5 years old • Less than 1 year old 95 Subscore: 380 Metrics Global Rating Scales (Based on experience or projections) Score 0-25 26-50 51-75 76-100 6. Stakeholder Buy-In • Stakeholders unaware of program • Stakeholders know of program but have a lot of questions • Stakeholders understand program and only have few questions • Stakeholders already onboard 90 7. Time to implement • Over 1 year • Over 6 months up to 1 year • 3 up to 6 months • Less than 3 months 85 8. Cost to implement • More than budget target • Right at budget target • Slightly under budget target • Significantly under budget target 50 9. Full Return on investment • Over 1 year • Over 6 months up to 1 year • 3 up to 6 months • Less than 3 months 25 10. Sustainability • One time program • Program information must be continually updated to remain current • Program can be updated annually and reused • Program can be continually used with minimal maintenance 95 Subscore: 345 Notes: Likelihood of Success Going Fwd Total Metrics Score Success Very Unlikely 0-250 Success Unlikely 251-500 Success Likely 501-750 Success Very Likely 751-1000 “We have successfully retrained nearly 70% of the workforce.” “Still too early to get good measurement, but appears to be creating improvement.” “It is too early say the full impact but safety has improved.” “Have received good feedback from drivers.” “The training program is only 1 year old.” “Right on budget.” “Get good feedback for the need for this training program from the union.” “We do not tend to wait long to make changes.” “Takes a year to fully retrain everyone and realize the benefits of the training program.” “This program trains mostly basic skills that need little updating.” Exhibit 2-14. Real-world application of the Training and Development Practices Scorecard. This is an example of evaluating the training and development practice of working with a consortium of transit systems to develop and use online training. Quotations explain why each rating was given. With an overall score of 725, this practice is likely to be a successful training program.

Next: Module 3 - Improve Image Management to Become an Employer of Choice »
Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

TRB’s Transit Cooperative Research Program (TCRP) Report 162: Building a Sustainable Workforce in the Public Transportation Industry—A Systems Approach provides a guidebook that addresses contemporary issues in workforce development, retention, and attraction, and public transportation image management.

The guidebook provides practical tools to transit agencies on a variety of workforce issues including workforce strategies that enhance organizational processes, performance metrics to evaluate the impact of workforce strategies, image management techniques that improve perceptions of the public transportation industry, and benchmarking processes that allow for continuous organizational improvement.

The guidebook is separated into modules that may be used independently or together in the form of the fully integrated guidebook.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!