National Academies Press: OpenBook
« Previous: Appendix H - Level 3 Training Evaluation Tool
Page 110
Suggested Citation:"Appendix I - Level 4 Training Evaluation Tool." National Academies of Sciences, Engineering, and Medicine. 2018. Transit Technical Training, Volume 2: Guide to Overcoming Barriers to Implementing Best and Innovative Training. Washington, DC: The National Academies Press. doi: 10.17226/25158.
×
Page 110
Page 111
Suggested Citation:"Appendix I - Level 4 Training Evaluation Tool." National Academies of Sciences, Engineering, and Medicine. 2018. Transit Technical Training, Volume 2: Guide to Overcoming Barriers to Implementing Best and Innovative Training. Washington, DC: The National Academies Press. doi: 10.17226/25158.
×
Page 111
Page 112
Suggested Citation:"Appendix I - Level 4 Training Evaluation Tool." National Academies of Sciences, Engineering, and Medicine. 2018. Transit Technical Training, Volume 2: Guide to Overcoming Barriers to Implementing Best and Innovative Training. Washington, DC: The National Academies Press. doi: 10.17226/25158.
×
Page 112
Page 113
Suggested Citation:"Appendix I - Level 4 Training Evaluation Tool." National Academies of Sciences, Engineering, and Medicine. 2018. Transit Technical Training, Volume 2: Guide to Overcoming Barriers to Implementing Best and Innovative Training. Washington, DC: The National Academies Press. doi: 10.17226/25158.
×
Page 113

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

110 A P P E N D I X I Level 4 Training Evaluation Tool The Kirkpatrick/Philips training evaluation model was developed by Donald L. Kirkpatrick and Jack Phillips. The model consists of five levels of training evaluation: Level 1: The reaction of participants Level 2: The level of learning achieved Level 3: Changes in learner behavior Level 4: Business results derived from training Level 5: Return on investment (ROI) that the training delivers This tool is intended to assist transit agencies in conducting Level 4 training evaluations. Level 4 evaluations are highly beneficial in that they enable organizations to assess the impact of training on the organization’s overall performance. In other words, Level 4 evaluations tie training results to business results. How to Conduct a Level 4 Training Evaluation The following section provides step-by-step guidance for conducting a Level 4 evaluation. This is followed by two tools that can be used to conduct the evaluation. Step 1: Identify and define metrics When conducting a Level 4 evaluation, it is necessary to identify and define the Level 4 metrics beforehand. These metrics should be tied directly to business outcomes and will be used to measure the training’s effectiveness in improving the organization’s performance. Therefore, metrics should be focused at the organization-wide level, as opposed to the individual employee level. By identifying metrics beforehand, an organization can better envision what a successful training will look like and select a training option aligned with the vision. For example, outcomes may include increased on-time performance, increased mean distance between service interruptions, and increased system reliability. Step 2: Establish anchors for each metric Next, anchors must be established for each metric. While the numerical rating scale used to score each metric will remain consistent across the evaluation, definitions for the rating scale scores will depend on the anchors established. Anchors will vary based on the metric. When establishing anchors, it is important to follow these guidelines:

Level 4 Training Evaluation Tool 111 Metrics Rating Scales 25 50 75 100 1. Increased on-time performance Services are on time less than 50% of the time Services are on time 50% to 75% of the time Services are on time 76% to 90% of the time Services are on time over 90% of the time Step 3: Identify data collection methods. Before conducting the evaluation, it is necessary to also identify how to collect the data needed. Based on the metric being evaluated, determine if quantitative data (e.g., performance data gathered from an internal database) or qualitative data (e.g., information shared by a department manager in an interview) will be most effective. Additionally, determine the appropriate timing for the data collection. For example, consider a transit agency measuring the impact of training on service interruptions. It will be most effective for leaders conducting the evaluation to extract this information from an existing database. When conducting the post- training evaluation, it may be beneficial for the leaders to wait 2–3 months before extracting and evaluating the data as this will give the training participants time to transfer and apply the new knowledge to their jobs. Step 4: Conduct pre- and post-training evaluations After establishing metrics for the Level 4 evaluation, conduct the pre-evaluation prior to the training. These results can then be compared to the post-training evaluation to determine how the organization’s performance has changed. It is also important to note factors other than training that may contribute to changes in performance. For example, new technologies or processes implemented during the same timeframe as the training may impact performance. The following pages contain two tools that can be used in conducting Level 4 training evaluations. The first tool is a pre-evaluation planning tool, which can be used prior to conducting the evaluation to identify and define metrics and data collection methods. The second tool is a Level 4 Evaluation Scorecard that can be used to conduct the evaluation. For further details, please see the instructions accompanying each tool. Metric Anchor In the following example, the metric is increased on-time performance. The anchors are the descriptions listed under the numerical ratings. In this case, the anchors are the percent of time that services are on time. Based upon the frequency of on-time services, the metric will be given a score of 25, 50, 75, or 100. For example, if 80 out of 100 buses are on time, the individual completing the evaluation would report a score of 75. Each anchor should be mutually exclusive (i.e., there should be no overlap in scale points), and The same anchors should be used when comparing the impact of two or more training courses/programs. The anchors should form a continuous scale (i.e., there is no break between categories so all possible values are covered),

112 Transit Technical Training, Volume 2: Guide to Overcoming Barriers to Implementing Best and Innovative Training Pre-Evaluation Planning Tool Instructions: Complete the following table prior to conducting a Level 4 evaluation. First, list and define each metric that will be used to evaluate the impact of the training. Next, identify what data will be necessary to collect for each metric, as well as the source of the data. Finally, determine when the data will be collected. Consider both the pre- and post-training evaluations when determining the data collection timing. Metric Definition Potential Data Source/Method Potential Data Collection Timing Example: Increased on-time performance The extent to which services are either on time or delayed based on schedules. Internal database Immediately before training; 3 months post-training Metric 1 Metric 2 Metric 3

Level 4 Evaluation Scorecard Instructions: Complete the following scorecard before and after a training, to evaluate the impact of the training on the transit agency’s performance. First, list metrics that will be used to evaluate the training. Next, list anchors for the rating scale aligned with each metric. Ratings for each metric should be selected based on the anchors. Report a rating in the far right-hand column accordingly. Once ratings are assigned for each of the metrics contained in the scorecard, a total score should be computed by summing each of the individual metric ratings. The total score can then provide an overall picture of whether or not the training course has positively impacted the transit agency’s performance. If there are additional factors aside from training that may have impacted performance during this timeframe (e.g., new technology), list this information in the Notes section of the scorecard. Metrics Rating Scales Score 25 50 75 100 1. Example: Increased on-time performance Services are on time less than 50% of the time Services are on time 50% to 75% of the time Services are on time 76% to 90% of the time Services are on time over 90% of the time 75 2. Example: Increased mean distance between failures (MDBF) MDBF is less than 10 miles MDBF is 11-20 miles MDBF is 21-30 miles MDBF is over 30 miles 50 3. 4. 5. Sub-score: Notes: Degree of Training Impact Total Score Low Impact 0-250 Moderate Impact 275-500 Significant Impact 525-750 Very Significant Impact 775-1000

Next: Appendix J - Level 5 Training Evaluation Tool »
Transit Technical Training, Volume 2: Guide to Overcoming Barriers to Implementing Best and Innovative Training Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

TRB's Transit Cooperative Research Program (TCRP) Research Report 199: Transit Technical Training, Volume 2: Guide to Overcoming Barriers to Implementing Best and Innovative Training provides public transportation agencies with best practices, strategies, and resources to assist with the implementation of effective and innovative training programs and techniques for frontline employees.

TCRP Research Report 199: Transit Technical Training is a two-volume set that presents guidance on technical training programs and the implementation of those for transportation agencies. The report's first volume, Guide to Applying Best Practices and Sharing Resources, documents the best models of technical training programs serving U.S. and international transportation agencies and related industries.

A product of this research also includes a training resource catalog to help transit agencies provide technical training for their employees. Training course information listed includes course descriptions, objectives, target audience, length, cost, training standards, and directions on how to access the course. The training resource catalog is available at https://ntrb.enotrans.org/.

Disclaimer - This software is offered as is, without warranty or promise of support of any kind either expressed or implied. Under no circumstance will the National Academy of Sciences or the Transportation Research Board (collectively "TRB") be liable for any loss or damage caused by the installation or operation of this product. TRB makes no representation or warranty of any kind, expressed or implied, in fact or in law, including without limitation, the warranty of merchantability or the warranty of fitness for a particular purpose, and shall not in any case be liable for any consequential or special damages.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!