How to evaluate the flying skills of police drone pilots

By Alan Frazier

Your agency recently purchased a small unmanned aircraft system (sUAS), commonly referred to as a drone. The chief assigned Officer Wright as the pilot because he professed to have “Hundreds of hours of model aircraft experience.” Wright possesses a Federal Aviation Administration Remote Pilot Certificate but has never completed any type of flying skills evaluation.

All is going well until Wright responds to the termination of a stolen vehicle pursuit. The suspect has fled into a cornfield after having lost control of the stolen vehicle and striking a light standard on Interstate 29. The winds are a bit high: 15 mph gusting to 20 mph. Not impossible for an sUAS, but challenging.

Wright is an enthusiastic young officer and wants to see the suspect apprehended. He launches his sUAS in the hope of finding the suspect. Five minutes into the flight, Wright receives a message on his sUAS hand controller indicating that the Global Positioning System (GPS) receiver in his aircraft has failed. Without GPS, Ofc. Wright must “hand fly” the aircraft without the assistance of an autopilot.

Wright struggles with maintaining control of the aircraft ultimately losing control and striking the windshield of a vehicle traveling northbound on I-29 at 75 mph. The driver, blinded by the opaqueness of his suddenly shattered windshield, loses control of the vehicle, travels across the center median, and strikes a school bus traveling southbound. The final tragic toll is 14 fatalities (including both drivers and 12 students) and 26 injured students. Could this accident have been prevented? If so, how?

How drones grew exponentially in public safety

The use of drones by U.S. law enforcement agencies has grown exponentially over the last six years.

In 2014 less than 40 law enforcement agencies had sUAS. Today, an estimated 1,150 law enforcement agencies and 550 fire departments and search and rescue teams are using sUAS.

Many factors led to this growth including the cost-effectiveness of the technology when compared to traditional manned aircraft; a realization that drone use in law enforcement is not inherently an invasion of people’s privacy; and the Federal Aviation Administration’s enactment in August 2016 of 14 CFR Part 107.

Part 107 was eight years in the making as the regulations started with recommendations made by the Small Unmanned Aircraft Systems (sUAS) Aviation Rulemaking Committee (ARC) that was formed in 2008. Part 107 addresses a variety of sUAS regulatory issues including maximum altitude, maximum velocity, cloud clearances, and, most applicable to this discussion, remote pilot certification.

Part 107 created a new FAA certificate: Remote Pilot. Applicants for a Remote Pilot Certificate must:

• Pass an FAA examination consisting of 60 questions covering 12 aeronautical subject matter areas;
• Apply for the certificate in-person at an FAA Flight Standards District Office (FSDO) or use the FAA’s online Integrated Airman Certificate and Rating Application (IACRA);
• Be vetted by the Transportation Security Administration.

The FAA Remote Pilot Examination covers a broad spectrum of topics including airspace, aeronautical charts, meteorology, aeronautical decision-making and the specifics of Part 107 regulations. Part 61 manned aircraft pilots (other than student pilots) who meet FAR 61.56 (biennial flight review) requirements, may use an abbreviated process for obtaining a Remote Pilot Certificate. Interestingly, an individual can obtain an FAA Remote Pilot Certificate without ever having flown a drone!

Standardizing test methods

Flying safely in our national air space requires knowledge and skill. While the FAA’s Remote Pilot Examination is a good evaluation of remote pilot knowledge, the lack of a practical examination leaves a void that potentially increases the liability exposure of individuals and agencies using sUAS. There is a clear need for an accompanying skills evaluation to ensure the safety of the remote pilot, bystanders, property and manned aircraft in the area.

“The first step toward credentialing remote pilot skills is to get everybody onto the same measuring stick,” said Adam Jacoff, a project manager at the National Institute of Standards and Technology (NIST). “That’s where standard test methods can play a key role. Especially across public safety, industrial, commercial and even recreational pilots. All need to demonstrate essential maneuvers to maintain positive aircraft control while performing whatever payload functionality is necessary to successfully perform the intended tasks.”

Jacoff is leading an international effort to develop standard test methods for small unmanned aircraft systems.

The initial suites for maneuvering and payload functionality can be used to quantitatively evaluate various system capabilities and remote pilot proficiency. They are being standardized through the ASTM International Standards Committee on Homeland Security Applications; Response Robots (ASTM E54.09). They are also referenced as Job Performance Requirements in the National Fire Protection Association Standard for Small Unmanned Aircraft Systems Used For Public Safety Operations (NFPA 2400) and the ASTM Standard Guide for Training for Remote Pilot in Command of Unmanned Aircraft Systems Endorsement (ASTM F38.03 F3266-18). The U.S. Department of Homeland Security, Science and Technology Directorate has been supporting the development of these tests, as well as many more tests currently being validated.

Basic maneuvering and payload functionality testing

The test methods for basic maneuvering and payload functionality are being replicated across the country and internationally to focus training with quantitative measures of remote pilot proficiency.

They are low cost and easy to fabricate so everyone can measure their own progress over time and compare their proficiency to regional or national averages on similar systems. The results are comparable no matter when or where the tests are conducted.

“Suites of standard test methods provide common measures with quantitative scores. They can be conducted individually, in sequences, or embedded into operational training scenarios to provide reproducible scores that augment typically qualitative assessments,” said Jacoff. “Organizations using these tests set their own thresholds of acceptable system and pilot performance to align with their airspace, environment, and mission complexities. But those decisions are easier to make and trust when they are based on quantitative performance data.”

Participants in NIST’s test method validation exercises learn how to fabricate apparatuses, conduct trials, and embed them into their own training and credentialing programs.

While participating in one of NIST’s exercises I first realized the potential of these test methods. I was greatly impressed by the potential for agencies that wish to internally credential sUAS pilots or serve as a credentialing resource for others. The NIST test methods are already being used as the basis for state-wide credentialing of emergency responders in Colorado and Texas. Many other state and local emergency response organizations are also adopting the test methods. Canada is moving quickly to implement these tests as the basis for credentialing their emergency responders nationwide. Others will certainly follow. The Airborne Public Safety Accreditation Commission (APSAC) is strongly considering their adoption, as is the Civil Air Patrol, an auxiliary of the U.S. Force, as they seek to standardize their pilot credentialing across 52 wings consisting of over 1200 sUAS pilots.

Ben Miller, Director of the Colorado Center of Excellence for Advanced Technology Aerial Firefighting, has followed NIST’s sUAS Standard Test Methods project from the inception. “NIST was one of the very first evaluation groups to show interest during the early days of UAS in public safety. The rigor that today’s Standard Test Methods show is a direct result of their years of work into the project. The applicability of the method supports acquisition decisions as well as employment considerations. The NIST sUAS Standard Test Methods produce data that can be used to answer the questions of what system do I buy and what system do I use for which mission?”

Miller is convinced that the NIST sUAS Standard Test Methods have great applicability to public safety sUAS operations. He has backed up that opinion by being an early adopter of the NIST sUAS Standard Test Methods.

“The Colorado Department of Public Safety has adopted the NIST sUAS Standard Test Methods within our UAS certification process,” Miller said. “Managed by The Center of Excellence for Advanced Technology Aerial Firefighting (CoE) within the Division of Fire Prevention and Control (DFPC), the CoE provides this certification process to stakeholder public safety agencies within the State of Colorado. To date, 16 agencies and 42 UAS operators have gone through the process.”

The NIST sUAS Test Methods include four different “test lanes”:

• Basic Proficiency Evaluation for Remote Pilots (Part 107 qualification);
• Open Test Lane;
• Obstructed Test Lane;
• Confined Test Lane.

These test methods can all be used to evaluate sUAS capabilities and sensor systems, or remote pilot proficiency for credentialing. The tests are easy to conduct alone or in groups, and inexpensive enough to set up multiple concurrent lanes. They are quick to perform, typically less than 30 minutes to conduct all the tests in a given lane, so they can support flying practice for remote pilots at the beginning of every training session. This is a good way to track improvement in pilot proficiency.

NIST has done an excellent job of creating a comprehensive user guide, scoring forms and apparatus targets you can be printed and placed in the buckets. It’s that simple. But the results are more than the sum of their parts. Fly the lane once and you immediately see how easy it is to evaluate precise station-keeping (with or without GPS, downward image flow, windy conditions, etc.). There are various repeatable maneuvering flight paths, and tests for zoom lenses and any additional sensors. The flight paths get incrementally harder but all use the same essential bucket alignment tasks so you can evaluate yourself, know your range to various targets, then compare your results over time or against others.

Basic Proficiency Evaluation for Remote Pilots

The Basic Proficiency Evaluation for Remote Pilots (BPERP) is the entry-level test method. It is designed to complement the Federal Aviation Administration’s Part 107 Remote Pilot Certificate by providing an inexpensive, easily duplicatable, mechanism for assessing remote pilot flying skills.

The BPERP can be administered in 10 minutes using 3 omni bucket stands, a 50’ tape measure and a stopwatch.

The BPERP requires a compact test area of 50’x 20’ so can easily be administered indoors or outdoors.

The BPERP requires the remote pilot to conduct 3 takeoffs and landings from a 12” radius circle, climb to specified altitudes of 10’ and 20’ AGL, conduct yawing turns, and conduct forward, reverse, and transverse flight maneuvers.

The goal is to capture still images of 36 targets that are placed within 2-gallon buckets that are fastened to three omni bucket test stands that are constructed from 2”x 4” and 4”x 4” lumber. The bucket stands are easy to assemble and can be transported in a couple of nylon golf club bags or simply stacked and placed in a vehicle.

The test consists of one maneuvering phase and two transverse flight phases. Pilots earn one point for each accurately captured target image, 2 points for an accurate first landing, and one point each for accurate second and third landings. Scoring sheets are available from NIST. Agencies set their own benchmarks scores for passing the test.

I administered the Basic Proficiency Evaluation to novice and experienced members of the Grand Forks Northeast Regional sUAS Team (hosted by the Grand Forks County Sheriff’s Office) and remote pilots assigned to the Civil Air Patrol (CAP) North Dakota Wing. The test methods were unanimously endorsed by every pilot that I have run through the course.

A measuring stick system

If I sound enthusiastic about the NIST sUAS Test Methods, it is because I believe they represent an excellent way for organizations to “raise the bar” on remote pilot credentialing with more rigorous and comparable evaluations. But also use them to get more informed about what different sUAS equipment can reliably do.

The combination of pilot skills and equipment capabilities, with tracked scores over time, provide an essential measure of “readiness” for any given mission. Each set of tests, either conducted in a standard test lane or embedded into an operational training scenario, enables each pilot and organization to evaluate their readiness more rigorously while practicing their procedures, data collection and logging. Maybe your equipment is lacking, or maybe it’s your training. Either way, quantitative scores captured in standard test methods can provide the rationale for changes that need to be made. By establishing minimum thresholds of remote pilot proficiency, agencies will further insulate themselves from potential civil liability by demonstrating due diligence in vetting their sUAS pilots. The NIST sUAS Test Methods can provide that missing element in every organization’s training program, an easy to implement “measuring stick” for systems and pilots.

Conclusion

While almost nothing is certain, it is likely that the sUAS accident described in the preamble could have been prevented if Officer Wright had been subjected to a rigorous sUAS flying skills evaluation.

Such an evaluation could have made him aware of shortcomings in manual flight skills and encouraged him to improve those skills.

Secondarily, implementation of a flying skills evaluation, such as the NIST sUAS Standard Test Methods, would be further evidence that Officer Wright’s agency had been diligent in their pilot screening and training. Such evidence is extremely valuable in the defense of a civil liability suit.

For more information, click here and see below.

NIST SUAS Test Methods - Introduction by epraetorian on Scribd

NEXT: How to buy police drones

About the author

Alan Frazier currently serves as a senior fellow at Georgetown University. He is working with the National Institute of Standards and Technology (NIST) to validate and implement these and other test methods with any interested public safety organization. He hosts “train the trainer” workshops and exercises to help such organizations get started. He is also involved in developing and validating new test methods for sUAS. Alan previously served as an Associate Professor of Aviation at the University of North Dakota. He is a 40-year law enforcement professional having served as a sworn officer and supervisor with city, county, state and federal law enforcement agencies. He founded and served 10 years as the officer-in-charge of, the Northeast Regional sUAS Team in Grand Forks, North Dakota. He is an FAA Airline Transport Pilot rated to fly single and multi-engine airplanes, helicopters, gliders, and small unmanned aircraft systems.