All police tests are a prediction that the person “scoring” the highest on a pre-determined scale ranging from 0-100 can do the job better than someone scoring lower. If the person creating the examination process knows how to do valid job task analysis (which is rare) then the knowledge, skills, abilities, and personnel traits (KSAPs) a person must possess (or be capable of obtaining through job related training and experience) have been identified and ranked from “critical” to “least important.”
If I were to provide this document to the average company or human resource person who creates and administers police entry level and promotional examinations they would make several assumptions. First, that the JTA indicates “which” of the knowledge, skills, abilities and personal traits (KSAPs) the examination process must test for and that some test variables are more important to test for than others (weighting of testing dimensions). Second, that the way to determine if one candidate has the KSAPs) “better” than another is to administer a multiple choice examination. If the written examination can’t test for a critical variable (oral communication for example) let’s give the candidates an oral examination by a three person panel of “experts” from their own profession.
Further, let’s take the JTA and use it to interpret the testing variables and further “weight” the examination process by counting the written exam 60 percent of a candidate’s final score and the average of the three panel member’s scores a total of 40 percent of a candidate’s final score. Candidates are provided with a reading list, the two components of the testing process-written and oral exam-are administered and a “certified” list is provided to the city of town. Everyone’s happy-right? The only problem is that the testing process didn’t do what it was supposed to which is identify the best candidate(s) for the job. That’s why when the list comes out we collectively shake our heads and make comments like, “Joe is a nice guy and is very smart (master’s degree and all), but couldn’t lead himself or anyone else out of a room.”
Policing is a “doing” profession. The problem with using knowledge-based tests followed by a 30-minute snapshot oral exam (which often is just the written exam given again in an oral format) is that it is simply not enough to “know about” police work — which is what knowledge-based testing will determine — we must be able to actually do it.
All of us know this. This is why a police officer is required to demonstrate proficiency with whichever firearm is required by actually shooting the gun. No amount of classroom training or paper and pencil test can replace the officer’s ability to actually shoot the weapon with a pre-determined standard of proficiency. We now even try to recreate combat conditions knowing that paper targets don’t shoot back!
When I was commander of the Hartford, Ct police academy back before most readers of this article were born we began experimenting with “practical exams” sometimes called “stations days.” We wanted to see if police recruits could take academic instruction and apply what they learned to real life situations. So, after a week of instruction in felony stops I invited a police officer from another town to assist us in “assessing” to what degree police recruits- who scored high in a written exam on felony stops-could apply what they had been taught. The recruit class consisted of 35 males and five females.
My “role player” was an especially beautiful young woman who I asked to wear a short red dress. We put her in a MG and provided the recruits with both a description of the vehicle and of the suspect and told them she was wanted for murder. We also gave the recruits a firearm loaded with blanks. One by one the recruits were given a police cruiser and we had the role player drive by them and waited to see if the recruit did what we had trained him/her to do. Here’s what happened! All the male recruits were shot to death by the role payer who also had a gun loaded with blanks. All the female recruits survived. Can you guess why? The male recruits saw the short red skirt and a pretty smile and it would have cost them their lives in real life. We videotaped each felony stop and it made for some interesting “discussion” when we returned to the classroom.
High test marks on a paper and pencil test are a poor predictor of future work related success.
Testing Knowledge Versus Testing Ability
Are all practical exercises good indicators of future successful performance? Of course not! The design of the practical exercise (test) must be able to meet strict standards of test validity and reliability. Another short story might illustrate what I mean by poorly designed practical activity. When I was a young officer it quickly became apparent to me that I’d better learn how to defend myself. We didn’t have TASERs or pepper spray back them — just a gun, a blackjack, a wooden nightstick, and our hands. You walked a beat without a portable radio — just a call box key.
So, I began what was to become a lifelong journey in the martial arts. I started with White Lotus Kung Fu and then switched to Tae Kwon Do. We spent many hours shadow boxing and throwing a variety of kicks while yelling at imaginary opponents. Great physical fitness, but it doesn’t work very well on the street. Eventually an old time cop steered me to a guy who taught Hapkido. He had a “dojo” at the end of a filthy alley in the worst part of town. I went there and asked what I thought at the time was an old man behind the counter smoking a big cigar if I could see the “sensei.” He looked me up and down with a smile — really a smirk — on his face and said, “What the fuck to you want?”
I was a little irritated by this point so I said “I have a black belt, but I want to refine some of my techniques.” The old man reached into a draw and handed me a small rubber object and said, “Go into the toilet and wash this off in hot water and stick it in your mouth” It was a mouthpiece.
When I came back the old man yelled to another old man, “Frank, see what this kid’s got!”
Frank waved me over and we climbed up into the ring. He handed his cigarette to another guy and said, “Lesson one. Protect yourself.” I said, “We aren’t wearing gloves and I don’t want to hurt you, so I’ll go a little slow.” He laughed and said, “We don’t wear gloves.”
I bowed to Frank in the martial arts tradition and when I looked up he stuck a finger in both of my eyes. When I instinctively reached my hands up to my eyes to protect them he threw a short left hook and broke two of my ribs. After I got taped up at the hospital I went back and got more of the same. They taught me how to defend myself. These guys were all ex-military and retired cops who were using mixed martial arts way before its widespread popularity. By the way, they didn’t believe in belts of any color — only results.
So, what does this tell us?
the test or series of tests have to be designed to assess specific abilities, not general knowledge. Every police job task analysis I complete rates oral communication skills as a critical ability to be tested for — regardless of the rank. This skill is often defined as “effective verbal expression when presenting ideas to an individual or group” or the “transfer of information from one person to another including non-verbal expression.”
One of the primary tasks of an “oral board” is to test for this ability. If the oral examination is for the rank of Sergeant, we invite three lieutenants from another town to ask candidates a series of questions designed to — among other things-assess the candidate’s ability based on one of the above definitions. So what’s the problem? The problem is that both of these definitions are flawed when applied to the practical world of policing. What the panel and thus the test never get to is whether the candidate can effectively “communicate” with a street person who is half drunk and holding a bottle of ripple? Can he/she communicate with the homeless, the deranged, and the addicted, in emergency situations or with the mayor?
So, in my opinion, the test is flawed because it does not test for what it’s supposed to.
Benefits of Using an Assessment Center
An assessment center is not a place — it is a method of testing. It has been my experience that assessment center testing methodology has much higher test validity and reliability that traditional testing. The process tests for actual, observable, work-related behavior, not just knowledge. The job task analysis is used to identify specific criteria (test dimensions) for which candidates are tested. A series of job related activities-mini tests- are designed to recreate what a Sergeant-for example- would be expected to be able to do in real life. So, rather than merely asking a candidate in an oral board about the “discipline process” the candidate is provided with information relative to an officer under his/her supervision (sexual harassment for example) and then has to call the officer into a real office and handle the matter while “assessors” grade the candidate. Since there may be several correct ways of handling the situation the assessors must be carefully selected and trained.
The scenarios replicate actual working conditions which separates an assessment center from traditional testing.
I recently created an entry-level police officer testing process for a city in Connecticut. We did give candidates a multiple choice test and took the top twenty to move forward to an oral examination. Most candidates did well on the oral test. I then took the top ten candidates and put them through s short assessment center test. I brought the candidates outside, one by one, to the city hall parking lot. We showed them a sign that said, “Handicapped Parking Only.” I provided the candidate with a parking ticket book and instructions to not allow anyone without a valid handicapped parking permit to park in the handicapped parking spaces. I also told them to use “good judgment.”
I had hired an elderly woman as a role player. She pulled into the handicapped parking space, but her vehicle did not have a permit. She slowly got out of her car and used a walker to assist her in moving toward city hall. It would have been obvious to anyone the woman was indeed handicapped. Almost all of the candidates who had scored very high on the traditional oral exam blocked the woman’s path and ordered her back in the car. When she mildly protested pointing out she had difficulty walking some even began yelling at the woman. The two candidates who had scored lower on the traditional test for “oral communication” helped the woman into the building and mentioned the process to obtain a handicapped parking permit. This little exercise was graded by the oral panel who were amazed at the difference between candidates in a sterile setting and when they had to relate with an actual person.
There is a host of information readily available about assessments center testing. I strongly recommend you advocate for it in your next entry level or promotional process.
Larry the Jet
