Simulation Training Evaluation

How do you evaluate performance?
How do you determine performance metrics?
How do I measure training effectiveness? What is a TEE (Transfer Effectiveness Evaluation)? What is a transfer effectiveness evaluation?
What is reverse transfer of training? What is negative transfer of training?
How do I measure skill retention?
What is the cost effectiveness of simulator training and how is it measured?
What is the cost effectiveness of distributed mission training and how is it measured?

How do you evaluate performance?

Performance enhancements are a well-known advantage of simulator training. Researchers have shown that flight simulators effectively improve pilot performance related to landing skills (Hays, Jacobs, Prince, & Salas, 1992) and instrument and flight control abilities (Pfeiffer, Horey, & Butrimas, 1991). Training using virtual environments has also been demonstrated to improve user performance. For example, Sebrechts (2000) found that wayfinding ability (ability to navigate through a space) was greater in users who were allowed to explore a virtual environment in comparison to the actual building. To successfully evaluate performance in their own programs, trainers should proceed through four levels of training evaluation including reaction, learning, behavior, and results. In addition, trainers who wish to measure performance must begin with valid and reliable performance metrics. (Image courtesy of Moves Institute)

How do you determine performance metrics?

Performance measurement involves the measurement of task performance and operator skill in addition to the evaluation of task related knowledge and user attitudes (Morrison & Hammon, 2000). Performance measurement does not pertain to the performance of the operational equipment (i.e. engineering of the system). Instead, performance measures should involve carefully defined metrics related to the task for which training is being conducted. How well the user performs the task should be examined, including measures of behavioral processes and task outcomes built into the assessment. The general steps in establishing performance metrics include identifying specific measurements, devising a measurement plan, using valid and reliable measures, imposing experimental control on the research situation, and using analytical models throughout (Morrison & Hammon, 2000). When task performance cannot be assessed directly using performance metrics, more subjective estimates of the effectiveness of the task are needed such as surveys and subject matter expert opinion. In addition, the attitudes and impressions of trainers and trainees should also be measured and included in every evaluation.

How do I measure training effectiveness? What is a TEE (Transfer Effectiveness Evaluation)? What is a Transfer Effectiveness Ratio?

It is possible to evaluate training effectiveness based on the notion of transfer of training. Transfer of training is the extent to which the simulation system prepares individuals or collections of individuals for real world performance (Morrison & Hammon, 2000). A general transfer of training paradigm is the one commonly used in flight simulation, where a trainee is instructed in a flight simulator for a predetermined number of sessions, and then allowed to operate an actual aircraft under the guidance of an experienced pilot. The experienced pilot assesses the performance of the trainee and determines a positive or negative transfer of training (Martin, 1981).

Evaluating the extent to which training with a simulator transfers to the actual task is often measured using a Transfer Effectiveness Evaluation (TEE). Most TEE's involve both a learning experiment and a transfer experiment. Learning experiments are evaluations that quantify the amount of learning resulting from simulation training. The simplest way to evaluate the amount of learning that has taken place is to measure performance prior to training and compare it with performance measures after training has taken place (Morrison & Hammon, 2000). However, a TEE is not complete without the addition of a transfer experiment, which seeks to measure how effectively the training is transferred to the operational equipment. A transfer percentage is then calculated. While learning experiments only evaluate the training effectiveness in the experimental conditions in which they are conducted, transfer experiments attempt to measure performance in real world conditions. (Morrison & Hammon, 2000)

In general, transfer experiments provide more convincing evidence that the simulation system has prepared individuals for a real world scenario. However, some researchers believe that a TEE alone is not sufficient to measure whether transfer of training has actually occurred. Some believe that a transfer percentage does not measure the efficiency for which skills are learned. Therefore, a Cumulative Transfer of Effectiveness Ratio (CTER) was developed to measure the average number of trials needed to reach standard proficiency on the operational equipment. A CTER of 1.0 indicates that the simulator is just as efficient as training on actual operational equipment, while values above and below 1.0 indicates that the simulator is more or less efficient than the equipment (Morrison & Hammon, 2000).

See also Reverse Transfer of Training

What is reverse transfer of training? What is negative transfer of training?

Transfer of training has three possible outcomes. Positive transfer of training occurs when practice on "A" improves performance on "B". For example, if a pilot's performance is enhanced on an actual flight after training in a simulator, positive transfer has occurred. Negative transfer occurs when practice on "A" interferes with practice on "B". For example, an individual trained in a virtual environment may learn to tilt his head in a direction to obtain a better field of view. However, in a real world task this movement may interfere with his performance. The third outcome is that practice on "A" has no effect on the performance of "B". On rare occasions, reverse transfer of training occurs when practice in " B" actually improves performance on "A". For example, an expert pilot may perform much better on a simulator because he or she is so accustomed to flying a real aircraft. A simulator provides an environment free of danger for the pilot, so his performance on the simulator may be better then in actual flight. This outcome is not desirable, for it defeats the purpose of training to enhance performance in real world conditions (Martin, 1981). (Image courtesy of Space Daily)

See also Training Effectiveness

How do I measure skill retention?

Skill retention is a difficult concept to measure. Skills are considered forgotten when task performance drops below the desired level of proficiency. There are well-established theories of forgetting which help to explain why this phenomenon occurs. Unfortunately, trainers have to rely on reported or observed decrements in performance before they implement refresher training to remedy the problem (Rose, Radtke, Shettel, & Hagman, 1985). As criterion for measurement, trainers often use post-training performance evaluation to understand the degree to which a skill has been lost. In addition, trainers examine how the same task is performed under slightly different conditions, allowing them to gage whether decay has occurred based on the original skill learned. Furthermore, theory indicates that the likelihood for decay of skill depends greatly on the degree to which the skill was actually learned (Loftus, 1985). In other words, skills that were learned in a high acquisition environment (e.g. extensive training in a simulator) are retained longer then skills learned in a low acquisition environment (e.g. one-day seminar). Therefore, when measuring skill retention, it is just as important to measure skill acquisition (degree to which the skill has been effectively learned), as it is to measure post-training performance (Healy & Sinclair, 1994).

See also Modeling Skill Retention

What is the cost-effectiveness of simulator training and how is it measured?

Aviation training using simulators offers a cost effective and safe alternative to training during actual flight. Actual aircraft training requires the time and expertise of an experienced pilot as well as the necessary equipment (e.g. airplane, runway, etc.). In the military, it is often necessary to train pilots for emergencies and other dangerous real world experiences. Training in simulators offers a virtually risk-free environment while still providing a realistic experience for the user. Simulator training is often self-paced, allowing users to train when they are ready rather then when the pilot and equipment are available. In terms of cost, simulators offer a tremendous advantage over real world training. Researchers have found that simulator training costs one-tenth of the investment of actual field training exercises (Orlansky, Taylor, Levine, & Honig, 1997). Although guidelines for measuring the cost effectiveness of simulators are generally vague, it is believed that the only way to determine whether a simulator is worth the cost is to compare the initial investment to the value that might be gained (Sullivan, 2002). When dealing with potentially life-threatening situations, the cost of simulator training is well worth the investment. (Image courtesy of the Texas Higher Education Coordinating Board)

What is the cost effectiveness of distributed mission training and how is it measured?

Distributed Mission Training (DMT) offers a unique alternative to simulator training, providing a linked network of virtual simulations that can be accessed by a collective group of individuals. Virtual simulation permits the interaction of users and personnel by networking a group of simulators together, and allows their interactions to be real time without the logistical constraints and heavy financial resources associated with employing planes, ships, and weapons (Simpson, West, & Gleisner, 1995). The training effectiveness of distributed mission training is still being evaluated, primarily because it is time-consuming and expensive to conduct tests on large numbers of military personnel. In addition, although evidence concerning the cost-effectiveness of distributed mission training is limited, the anticipated results are very encouraging. Cost effectiveness is measured by computing the costs avoided, usually related to logistical matters. For example, research shows that over eighty percent of Army training cost is related transportation to and from training sites, and distributed mission training virtually eliminates these costs (Simpson et al, 1995). The DoD expects that the cost savings already resulting from SIMNET will be enough to fund the program's continued operations and maintenance through the year 2005 (Simpson et al, 1995).