Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
114 A P P E N D I X J Level 5 Training Evaluation Tool The Kirkpatrick/Philips training evaluation model was developed by Donald L. Kirkpatrick and Jack Phillips. The model consists of five levels of training evaluation: Level 1: The reaction of participants Level 2: The level of learning achieved Level 3: Changes in learner behavior Level 4: Business results derived from training Level 5: Return on investment (ROI) that the training delivers This tool is intended to assist transit agencies in conducting Level 5 training evaluations. Level 5 evaluations enable organizations to determine the monetary impacts resulting from a training. Purpose of Measuring ROI What is ROI? Return on Investment (ROI) assesses the effectiveness of training by measuring the monetary gains an organization experiences as a result of the training. For example, a transit agency can evaluate the difference in revenue before and after a training or the difference in maintenance costs of a particular type of equipment. This can also involve assessing the quantity of a type of equipment that is often repaired or replaced. For example, a transit agency can assess if the quantity of repairs and/or replacements has increased or decreased post-training. This can then be tied to monetary values by assessing the costs of the repairs and/or replacements, the difference in resulting revenue, and more. Why is it valuable to assess ROI? By measuring ROI, transit agencies can determine if a specific training used is effective, by assessing outcomes and benefits achieved financially. Transit agencies can also compare different training conducted on the same or similar topics to determine which one leads to the most positive financial outcomes in the least amount of time. Ultimately, this can help agencies select the most cost-effective training options.
Level 5 Training Evaluation Tool 115 Step 2: Identify associated financial outcomes Then identify the potential costs and revenues that are associated with each outcome. For example, if employees become more proficient at repairing a given vehicle component, this may lead to reduced replacement costs and higher revenues. Additional outcomes to consider may include the following: Number and cost of equipment Miles traveled and revenue per mile Time and cost spent on repairs Step 3: Compare pre-training and post-training results In order to effectively assess ROI, outcomes must be measured both before and after training. This enables management to review and compare the results and, thereby, whether or not the agency has experienced a return on the training investment. The following pages contain tools that will assist transit agencies in performing ROI assessments and identifying the cost-effectiveness of a training. How to Measure ROI The following section provides step-by-step instructions to measure ROI, followed by two tools to help plan for and conduct the evaluation. Step 1: Determine organizational outcomes achieved through training Because Level 5 training evaluations focus at a higher level than individual employee performance, outcomes should be at the department or agency-wide level. The easiest way to do this is to refer to the training objectives and identify the primary purpose of the training. For example, the overall purpose of a training may be to ensure employees know how to operate, install, and/or repair a specific type of equipment, technology, or vehicle.
116 Transit Technical Training, Volume 2: Guide to Overcoming Barriers to Implementing Best and Innovative Training Pre-Evaluation Planning Tool Instructions: Complete the following table prior to conducting a Level 5 evaluation. First, determine how ROI can be assessed, based on the training objectives. Then list each metric that can be used to assess and evaluate the ROI (e.g., cost of equipment, number of equipment, revenue). Next, identify the source of the data. Finally, determine when the data will be collected. Consider both the pre- and post-training evaluations when determining the data collection timing. ROI Assessment Item Metric Potential Data Source/Method Potential Data Collection Timing Example: Cost of new batteries installed over 6 months Number of new batteries installed Maintenance database/log 6 months before training; 6 months post- training Cost per battery Procurement database 6 months before training; 6 months post- training Assessment Item 2 Assessment Item 3 Assessment Item 4
Level 5 Evaluation Tool Instructions: Complete the following table after a training to evaluate the ROI. If comparing more than one training program or course, complete a separate table for each and review the comparisons at the bottom of the table to determine which training provides the highest ROI. List the ROI assessment item at the top of the table. Then, list each metric being used to measure ROI for the assessment item. Calculate the total cost or revenue, based on the metrics being used. To determine the ROI, determine the difference in total values pre-training and post-training, and enter the results in the Comparison row. ROI Assessment Item 1: Cost of new batteries installed over 6 months Pre-Training Example: # of Batteries Example: Cost Per Battery Total Cost Post-Training Example: # of Batteries Example: Cost Per Battery Total Cost Comparison (i.e., Difference) ROI Assessment Item 2 Pre-Training Metric Metric Total Cost/Revenue Post-Training Metric Metric Total Cost/Revenue Comparison (i.e., Difference) ROI Assessment Item 3 Pre-Training Metric Metric Total Cost/Revenue Post-Training Metric Metric Total Cost/Revenue Comparison (i.e., Difference)
