Simulated Voyages Using Simulation Technology to Train and License Mariners (1996) / Chapter Skim
Currently Skimming:
5 Performance Evaluation and Licensing Assessment
5 Performance Evaluation and Licensing Assessment
Pages 120-142
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 120... ...
Yet few training courses described for the committee rigorously addressed measuring, evaluating, and assessing individual mariner performance. Measuring Human Performance Measuring human performance, either simulator-based or real-performancebased, is often difficult.
|
From page 121... ...
In this context, evaluation is an element of the instructional design process. The evaluation can be informal or formal, subjective or objective, or both (see "Forms of Evaluation and Assessment" below)
|
From page 122... ...
Generally, formal records are not retained, although there are exceptions, such as when grades need to be assigned to meet baccalaureate requirements. Private, Informal Evaluations Sometimes, albeit infrequently, training sponsors or pilot associations have requested that simulator facilities conduct a private evaluation of a specific
|
From page 123... ...
The committee did, however, find instances where formal performance evaluations were conducted on a simulator. These cases included cadet evaluation using bridge watchkeeping courses at maritime education institutions, an offshore towing deck officer training program, an active watchstander course, and a leadership course (discontinued)
|
From page 124... ...
To be effective, formal evaluations or assessments must have standardized and structured monitoring and must include a critique of individual performance in a range of exercises appropriate to the instructional or licensing objectives or criteria. Formal evaluations or assessments must be consistent in method, timing, and responsibility from class to class or test to test, so that results can be compared and contrasted with a high degree of reliability.
|
From page 125... ...
Use of subjective evaluations can be effective and, in the absence of scientific performance-based criteria, is currently the primary means for ascertaining whether an individual can effectively apply knowledge in conducting actual operations. For example, a subjective evaluation may be used in determining whether an individual can collect, correlate, and interpret considerable information from multiple sources, make appropriate decisions based on this information and his or her nautical knowledge, and perform multiple tasks in the correct time sequence under the routine pressures of actual operating conditions.
|
From page 126... ...
TRAINING AND EVALUATION WITH SIMULATORS Although there are notable exceptions, evaluation is not systematically applied nor is the methodology for conducting performance evaluation well developed. Most performance evaluation methodologies that have been attempted rely on adherence to prescribed procedures (e.g., operating certain equipment or using correct radio procedures)
|
From page 127... ...
As noted earlier, concerns in applying instructional design include limited usefulness and age of detailed task analyses available within the industry, 1Normative-referenced testing is applied to a particular group of people at a particular time. The scores of the individuals are ranked, and the results permit comparison of performance among individuals or across a group of individuals assumed to be similar in makeup and knowledge.
|
From page 128... ...
. There is also concern that formal training performance evaluation records, including specific descriptions of course conduct and documentation of individual performance, could become evidence in accident investigations or disciplinary proceedings.
|
From page 129... ...
The Panama Canal Commission program, described in Box 5-1, includes formal pilot training using both onboard and simulator-based evaluation by senior pilots. Boxes 5-2 and 5-3 include comments on use of simulators for testing pilots and a summary of a simulator-based checkride, respectively.
|
From page 130... ...
Similarly, a designated examiner used to sign off practical factors application of other license or certificates may prove desirable." Among the report's recommendations was "Place significant increased emphasis on approved courses, and other, more formalized methods of training and de-emphasize ‘seatime,' un-verifiable for quality or quantity, as the principal guarantor of competency." The committee, however, disagrees. It believes that before simulation can be effectively applied to the licensing process, there are a number of critical issues in both training performance evaluation and licensing assessment that should be
|
From page 131... ...
Before a large-scale program is undertaken, these issues should be addressed. The Need for a Systematic Approach The recent decision of the USCG to add a simulator-based training evaluation to the master's licensing process was not systematic.
|
From page 132... ...
Robbins Pilot Training Officer Panama Canal Commission
|
From page 133... ...
Elements of instructional design that might be integrated include: • characterizing populations for which marine licensing is required and specifying competency requirements based on specific task and subtask analyses; • developing marine licensing goals and objectives; • developing standard performance criteria to measure whether licensing goals and objectives are met; • determining the knowledge, skills, and abilities (or ranges for each) re quired to meet standard performance criteria; • determining requirements for practical experience needed to develop knowledge, skills, and abilities; • developing examining and assessing methodologies, including matching assessment media to assessment objectives; • identifying resource requirements and testing media and validating and correlating them with marine licensing objectives, including a detailed inventory of simulators available by type and an estimate of percent time potentially available for simulator-based licensing assessment; • matching specific assessment techniques to licensing requirements; • establishing assessor qualification, selection, training, and certification requirements necessary to ensure the quality of the marine licensing pro cess relative to established objectives; 4In Licensing 2000 and Beyond, the term third party is someone who trains or teaches, and fourth party is someone, other that the USCG or a third-party trainer, who administers a test or makes subjective judgments about the competency of an individual applicant.
|
From page 134... ...
, which are basic to the application of instructional design to the marine industry: • An important element in applying instructional design to marine licens ing is specification of competency requirements based on task and sub task analyses. As discussed earlier, although the maritime industry has conducted several task analyses over recent decades, the results of stud ies have not been widely accepted as accurate depictions of the skills, knowledge, or abilities of various license levels (third mate, second mate, chief mate, or master)
|
From page 135... ...
Not all licenses require the high face or apparent validity possible with a full-mission ship-bridge simulator. Before approving a simulator for formal training performance evaluations or licensing assessments, it is important, as part of the validation process, to have credible experts subjectively assess whether or to what degree a simulation consistently results in behavior that would be expected under identical or similar real-world conditions.
|
From page 136... ...
If training (with or without formal evaluation of performance) were to be required as part of marine licensing, it is likely that company and union management would become more involved in observation and assessments.
|
From page 137... ...
Until cross-platform and cross-student (or candidate) studies have been performed, all simulator-based-training performance evaluation and licensing assessment must be recognized as being based on a particular simulator or simulator-evaluator/assessor combination.
|
From page 138... ...
Implications of Combining Training with Formal Licensing Assessment In addition to the concerns discussed above about combining the instructor/ evaluator function with that of the assessor in marine licensing, consideration must be given to the development of adequate security measures at the operating and authorizing levels to ensure fairness, accuracy, and reliability. Familiarization with the Simulator No matter how high the fidelity and accuracy of a ship-bridge simulator, it is not a real ship.
|
From page 139... ...
The degree and level of familiarization that should be conducted depends on whether the familiarization is used as training in advance of a performance evaluation or is in advance of an assessment required for marine licensing. Familiarizing an individual with the simulator in conjunction with training is appropriate insofar as that which is being measured is performance outcome resulting from training.
|
From page 140... ...
Before the agency undertakes more extensive use of simulation in marine licensing, program framework that includes consideration of the following issues should be developed: • validation of the scenarios used in performance evaluation and licensing assessment; • determination of the appropriate level of simulation for each license level; • qualification and certification of the evaluators and assessors; • availability of simulators; and • source of funding for the cost of simulator use, especially in licensing assessment.
|
From page 141... ...
Research Needs In the course of its investigation of the uses of simulation for training performance evaluation and licensing assessment, the committee identified a number of areas where existing research and analysis did not provide sufficient information to extend its analysis. The "Research Needs" section of Chapter 3 discusses the need to expand and update specific task and subtask analyses, including data on dimensions related to behavioral elements and specific steps needed to execute each subtask.
|
From page 142... ...
To judge which platform to use for which licensing assessment, it is necessary to understand the advantages and limitations of each type of platform, as well as the necessary levels of validity and fidelity for the different levels of license. The committee could not find any research that would support or discredit the combining of simulator training and simulator-based assessment in a single course.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.