Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
110 A P P E N D I X I Level 4 Training Evaluation Tool The Kirkpatrick/Philips training evaluation model was developed by Donald L. Kirkpatrick and Jack Phillips. The model consists of five levels of training evaluation: Level 1: The reaction of participants Level 2: The level of learning achieved Level 3: Changes in learner behavior Level 4: Business results derived from training Level 5: Return on investment (ROI) that the training delivers This tool is intended to assist transit agencies in conducting Level 4 training evaluations. Level 4 evaluations are highly beneficial in that they enable organizations to assess the impact of training on the organizationâs overall performance. In other words, Level 4 evaluations tie training results to business results. How to Conduct a Level 4 Training Evaluation The following section provides step-by-step guidance for conducting a Level 4 evaluation. This is followed by two tools that can be used to conduct the evaluation. Step 1: Identify and define metrics When conducting a Level 4 evaluation, it is necessary to identify and define the Level 4 metrics beforehand. These metrics should be tied directly to business outcomes and will be used to measure the trainingâs effectiveness in improving the organizationâs performance. Therefore, metrics should be focused at the organization-wide level, as opposed to the individual employee level. By identifying metrics beforehand, an organization can better envision what a successful training will look like and select a training option aligned with the vision. For example, outcomes may include increased on-time performance, increased mean distance between service interruptions, and increased system reliability. Step 2: Establish anchors for each metric Next, anchors must be established for each metric. While the numerical rating scale used to score each metric will remain consistent across the evaluation, definitions for the rating scale scores will depend on the anchors established. Anchors will vary based on the metric. When establishing anchors, it is important to follow these guidelines:
Level 4 Training Evaluation Tool 111 Metrics Rating Scales 25 50 75 100 1. Increased on-time performance Services are on time less than 50% of the time Services are on time 50% to 75% of the time Services are on time 76% to 90% of the time Services are on time over 90% of the time Step 3: Identify data collection methods. Before conducting the evaluation, it is necessary to also identify how to collect the data needed. Based on the metric being evaluated, determine if quantitative data (e.g., performance data gathered from an internal database) or qualitative data (e.g., information shared by a department manager in an interview) will be most effective. Additionally, determine the appropriate timing for the data collection. For example, consider a transit agency measuring the impact of training on service interruptions. It will be most effective for leaders conducting the evaluation to extract this information from an existing database. When conducting the post- training evaluation, it may be beneficial for the leaders to wait 2â3 months before extracting and evaluating the data as this will give the training participants time to transfer and apply the new knowledge to their jobs. Step 4: Conduct pre- and post-training evaluations After establishing metrics for the Level 4 evaluation, conduct the pre-evaluation prior to the training. These results can then be compared to the post-training evaluation to determine how the organizationâs performance has changed. It is also important to note factors other than training that may contribute to changes in performance. For example, new technologies or processes implemented during the same timeframe as the training may impact performance. The following pages contain two tools that can be used in conducting Level 4 training evaluations. The first tool is a pre-evaluation planning tool, which can be used prior to conducting the evaluation to identify and define metrics and data collection methods. The second tool is a Level 4 Evaluation Scorecard that can be used to conduct the evaluation. For further details, please see the instructions accompanying each tool. Metric Anchor In the following example, the metric is increased on-time performance. The anchors are the descriptions listed under the numerical ratings. In this case, the anchors are the percent of time that services are on time. Based upon the frequency of on-time services, the metric will be given a score of 25, 50, 75, or 100. For example, if 80 out of 100 buses are on time, the individual completing the evaluation would report a score of 75. Each anchor should be mutually exclusive (i.e., there should be no overlap in scale points), and The same anchors should be used when comparing the impact of two or more training courses/programs. The anchors should form a continuous scale (i.e., there is no break between categories so all possible values are covered),
112 Transit Technical Training, Volume 2: Guide to Overcoming Barriers to Implementing Best and Innovative Training Pre-Evaluation Planning Tool Instructions: Complete the following table prior to conducting a Level 4 evaluation. First, list and define each metric that will be used to evaluate the impact of the training. Next, identify what data will be necessary to collect for each metric, as well as the source of the data. Finally, determine when the data will be collected. Consider both the pre- and post-training evaluations when determining the data collection timing. Metric Definition Potential Data Source/Method Potential Data Collection Timing Example: Increased on-time performance The extent to which services are either on time or delayed based on schedules. Internal database Immediately before training; 3 months post-training Metric 1 Metric 2 Metric 3
Level 4 Evaluation Scorecard Instructions: Complete the following scorecard before and after a training, to evaluate the impact of the training on the transit agencyâs performance. First, list metrics that will be used to evaluate the training. Next, list anchors for the rating scale aligned with each metric. Ratings for each metric should be selected based on the anchors. Report a rating in the far right-hand column accordingly. Once ratings are assigned for each of the metrics contained in the scorecard, a total score should be computed by summing each of the individual metric ratings. The total score can then provide an overall picture of whether or not the training course has positively impacted the transit agencyâs performance. If there are additional factors aside from training that may have impacted performance during this timeframe (e.g., new technology), list this information in the Notes section of the scorecard. Metrics Rating Scales Score 25 50 75 100 1. Example: Increased on-time performance Services are on time less than 50% of the time Services are on time 50% to 75% of the time Services are on time 76% to 90% of the time Services are on time over 90% of the time 75 2. Example: Increased mean distance between failures (MDBF) MDBF is less than 10 miles MDBF is 11-20 miles MDBF is 21-30 miles MDBF is over 30 miles 50 3. 4. 5. Sub-score: Notes: Degree of Training Impact Total Score Low Impact 0-250 Moderate Impact 275-500 Significant Impact 525-750 Very Significant Impact 775-1000