Qualified to Fail
How police firearms qualifications miss the mark
Publication
Orr, R. M., Canetti, E. F. D., & Schram, B. (2024). A comparison of two law enforcement marksmanship assessments. WORK, 80(3), 1370-1379. https://doi.org/10.1177/10519815241290347
What was the question?
Orr and colleagues asked whether the structure of a marksmanship assessment affects both qualification rates and officer perceptions (comfort, confidence, safety) in using their sidearms. Specifically, they compared an agency’s current Traditional Pistol Assessment (Traditional) with a proposed Alternate Pistol Assessment (Alternative) designed to more closely mirror real-world policing conditions.
How did they look at it?
The study used a within-subjects, randomized, repeated measures design involving 14 Australian law enforcement officers. Each officer completed both assessments in counterbalanced order. Data collected included; qualification outcomes on both tests; self-reported comfort, confidence, and safety; and physiological measures (heart rate, breathing rate, skin temperature) during each assessment.
Assessment Details
Traditional Pistol Assessment (Traditional): This was the agency's standard qualification test, consisting of seven set phases designed to evaluate marksmanship and firearm handling. These involved static firing positions from 3 to 12 meters, single and two-handed firing, and tasks such as clearing a stoppage or moving into a kneeling position. Officers needed to hit all rounds on target in most phases to pass. The test emphasized static accuracy and procedural consistency.
Alternate Pistol Assessment (Alternative): The Alternative added operational realism. It included six firing situations requiring officers to issue verbal commands, move backward while firing, engage targets from behind cover, perform stoppage drills, and respond to simulated dynamic threats. Unlike the Traditional, the Alternative emphasized task-relevant actions like movement, timing, and use of cover - reflecting tactical complexities officers might face in real situations.
A variety of statistical tests were performed to compare participants across conditions.
What did they find?
Qualification Rates: More officers passed the Alternative (50%) than the Traditional (29%), though the difference was not statistically significant.
Perceptions: Officers reported significantly higher confidence, comfort, and safety after completing the Alternative versus the Traditional. These improvements held regardless of whether they passed. The Alternative seemed to promote a stronger sense of readiness and realism. Officers felt more capable of applying their skills under realistic constraints. In contrast, the Traditional's static drills may have fostered a sense of disconnection between training and operational demands, limiting its impact on officers' perceived preparedness.
Physiological Load: There were no significant differences in heart rate, breathing, or skin temperature between the two assessments. Both tests produced an equal level of stress in the participants.
Assessment Order: Performing the Alternative first boosted Traditional performance.
Phase-level Findings: Most failures in the Traditional occurred in the final phase involving multiple rounds from a draw. In the Alternative, the highest failure rate was in a station involving movement to cover - a skill not taught in the Traditional training.
So what?
According to the authors, the nature of the assessment itself plays a role in both qualification outcomes and officer perceptions. The Traditional, being static and less contextually demanding, may fail to reflect the realities officers face in the field and therefore may not build or reinforce operational confidence. In contrast, the Alternative, which incorporates elements such as movement, verbal commands, and use of cover, better reflects real-world scenarios. The authors observed that these features likely contributed to improved perceptions of comfort, confidence, and safety among officers.
Despite the added complexity, the Alternative assessment did not increase physiological stress compared to the Traditional. Moreover, the authors suggest that the gains in self-efficacy - defined as officers’ belief in their capabilities - may be linked to the more operationally relevant nature of the Alternative, which allows officers to experience and succeed in more realistic engagements.
In summary, the findings support the idea that incorporating operational relevance into assessments can positively influence both performance and perception, thereby better preparing officers for the demands of policing.
My $.02
Here again we have a study where training in a more realistic (Alternative) environment improved performance in the less realistic (Traditional) environment, but not vice versa (I know, I know, a lot of you trainers are saying that qualifying is not training. You are correct, but as this study and many others show, the level of training for many police is so low that even doing a qualification task can improve performance). As I noted in this post, this general finding has also been found in a few other firearms studies and a lot of sports related studies.
This should make you stop and think hard about how we have been traditionally doing training. Practicing isolated movements and then adding variation once the trainee masters the “fundamentals” does not seem to work the way that many trainers assumes it does. Ecological Dynamics (EcoD) and the Constraints-Led Approach (CLA) offer several takes on why this may be, but perhaps we have simply reversed how fundamentals work. Rather than learning the fundamentals and then trying to push them into the game or performance situation, maybe we learn the game and the fundamentals emerge as effective solutions based upon the constraints of the situation? This view aligns with EcoD and the CLA, which both emphasize that skills develop through interaction with the environment and task demands, not through isolated repetition. By engaging with scenarios that contain the important elements of the real thing from the start, learners pick up on critical information that guides the emergence of effective actions. This is where the fundamentals had to come from in the first place after all. Early martial arts techniques or battlefield tactics weren't developed in a sterile practice hall. They emerged from real combat. Similarly, sports fundamentals originated from playing the actual game. Coaches then created fundamentals drills to try and shortcut the process and make beginners get better faster, but ironically these shortcuts may actually be slowing new players’ progress.
While the Alternative approach used in this study represents a significant improvement over the Traditional in terms of context and relevance, it is still uncertain how well the skills practiced in the Alternative will transfer to the full complexity of real-world performance. One limitation is that even though the Alternative introduces movement, verbal commands, and use of cover, the officers are notionalizing the reasons for these actions. In the field, officers rely heavily on rich and dynamic perceptual information - such as body language, tone of voice, environmental cues, and unfolding patterns of behavior - to guide their actions. The Alternative assessment lacks much of this perceptual complexity. Given these limitations, the extent to which the Alternative transfers to real-world performance remains uncertain. To strengthen the connection between training and performance, future versions could incorporate more realistic perceptual cues based on what officers actually face in the field.
The authors also found that the Alternative qualification produced more confidence (also called self-efficacy) than the Traditional, and suggested that this may be part of the reason participants performed better on the Traditional test when they completed the Alternative first. This is plausible given the large body of literature on self-efficacy showing that if people don't believe they can do something well, they generally don't. Self-efficacy influences not only whether individuals initiate an action, but also how much effort they invest, and how resilient they are in the face of setbacks.
This finding underscores the importance of designing training that challenges participants to improve while simultaneously leaving them with the belief that they can succeed. Successfully tackling challenging but manageable tasks builds confidence that transfers to other situations. Without this belief, even technically proficient individuals may falter under pressure. Effective training must therefore strike a balance: it should push participants to extend their capabilities but also ensure that they leave each session with a reinforced sense of "I can do this." This balance is not just about skill development - it's directly tied to officer safety and sound decision-making under pressure, where confidence and competence must go hand in hand.