Features
Evaluating drone pilot skills

It is estimated that several hundred Canadian public safety agencies have acquired remotely piloted aircraft systems (RPAS). Approximately one third of those agencies are fire service oriented. Canadian fire departments that have acquired RPAS and are using them with great success gain additional situational awareness at the scenes of structure and wildland fires, as well as natural and man-made disasters. RPAS tasks include providing real-time video to fire scene management, penetrating hot zones such as HAZMAT spill areas and potential BLEVE evacuation zones, post disaster damage assessment, and post-response assessment and evaluation for training purposes. The relatively low cost of RPAS make the technology affordable for most fire service agencies.

Transport Canada has done an excellent job in crafting regulations, including pilot certification levels, related to RPAS. Two main categories of RPAS operations exist: basic and advanced. Each of the two categories have a different set of rules that RPAS pilots must follow. Basic operations must be conducted in uncontrolled airspace, more than 30 metres from bystanders and never overflying bystanders. If all three of these conditions cannot be met the flight is considered an advanced operation.

For basic operations, the RPAS pilot candidate must score at least 65 per cent on a 35 question online “small basic examination” covering eight knowledge areas including air law, air traffic rules and procedures, theory of flight, and meteorology. The exam has a 90 minute time limit. Immediately after successfully completing the examination, the candidate can download their pilot certificate-basic operations.

For advanced operations, the RPAS pilot candidate must score at least 80 per cent on a 50 question online “small advanced examination” which assesses the candidate’s knowledge of the same eight areas found on the basic examination (albeit at a higher level of understanding). The advanced exam has a 60 minute time limit. The candidate has an additional requirement to complete a flight review conducted by a Transport Canada authorized RPAS flight reviewer. Upon successful completion of the flight review, the candidate is eligible to apply for the pilot certificate-advanced operations.

Some Canadian Fire Agency RPAS operations would likely require their pilots to conduct advanced operations due to the need to fly in controlled airspace and/or fly within 30 metres of bystanders. However, many flights, especially in rural areas, can be conducted as basic operations requiring only a pilot certificate-basic operations.

In both cases, especially when exercising a pilot certificate-basic operations certificate that does not require a flight review, agencies can benefit from implementing internal standards for pilot flight skills evaluation. Benefits of such standards include: establishment of benchmark flight proficiency levels to which agency pilots can aspire to reach; enhanced safety through initial and periodic evaluation of agency pilot flight skills; and increased insulation from liability by demonstrating that the agency has adopted a robust internal RPAS pilot credentialing mechanism in addition to that provided by Transport Canada. Once an agency has decided to implement internal pilot credentialing standards, the devil is in the details…What tasks should the standards assess? How do you assess the selected tasks? What should the evaluation course consist of?

Enter NIST, the National Institute of Standards and Technology. NIST has developed a comprehensive suite of sUAS (i.e small RPAS) standard test methods. NIST is a non-regulatory agency within the U.S. Department of Commerce. NIST’s mission is to promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve our quality of life. While this might sound like NIST only assists U.S. entities, the reality is quite different. NIST has been very active in Canada as well as Japan. NIST employs approximately 3,400 talented scientists, engineers, technicians and administrative personnel at two campuses: Gaithersburg, Md., and Boulder, Col. Fifteen years ago, Adam Jacoff, an engineer and project manager at NIST, was conducting a validation of NIST terrestrial and submersible robot standards (think bomb robots and exploring the Titanic submersibles). A participant in the validation happened to have a drone. Jacoff offered the participant the opportunity to show what the drone could do. That was all it took for Jacoff to identify a technology in great need of standardized test methods. He went full speed ahead in establishing a talented team to begin working on standard test methods for drones. According to Jacoff, “The first step toward credentialing remote pilot skills is to get everybody onto the same measuring stick. That’s where standard test methods can play a key role. Especially across public safety, industrial, commercial, and even recreational pilots. All need to demonstrate essential maneuvers to maintain positive aircraft control while performing whatever payload functionality is necessary to successfully perform the intended tasks.”

Jacoff is leading an international effort to develop standard test methods for small unmanned aircraft systems. The initial suites for Maneuvering and Payload Functionality can be used to quantitatively evaluate various system capabilities and remote pilot proficiency. They are being standardized through the ASTM International Standards Committee on Homeland Security Applications; Response Robots (ASTM E54.09).They are also referenced as job performance requirements in the National Fire Protection Association Standard for Small Unmanned Aircraft Systems Used For Public Safety Operations (NFPA 2400) and the ASTM Standard Guide for Training for Remote Pilot in Command of Unmanned Aircraft Systems Endorsement (ASTM F38.03 F3266-18). The U.S. Department of Homeland Security, Science and Technology Directorate has been supporting the development of these tests.

Position Test Graphic

Traverse Test Graphic

NIST has developed a turnkey set of sUAS standard test methods that are available, free of charge, for adoption and utilization by agencies that wish to either internally credential sUAS pilots or serve as a national credentialing resource. The Airborne Public Safety Accreditation Commission (APSAC) is strongly considering adoption of the NIST RPAS Standard Test Methods as a portion of their sUAS Unit Accreditation Program as is the Civil Air Patrol, an auxiliary of the United States Air Force, who currently has over 1200 volunteer remote pilots operating over 1700 RPAS nationwide.

The NIST sUAS Standard Test Methods encompass four different “test lane” protocols: Basic Proficiency Evaluation for Remote Pilots (Part 107 qualification); Open Test Lane; Obstructed Test Lane; and Confined Test Lane. The test methods may be utilized to evaluate sUAS airframes and sensor systems as well as remote pilot evaluation and credentialing. They are easily performed utilizing test apparatus that can be assembled inexpensively with materials readily available at any large hardware store. NIST has done an excellent job of creating a comprehensive user guide, scoring sheets, and apparatus target printable documents.

The Basic Proficiency Evaluation for Remote Pilots (BPERP) is the entry level test method. It is designed to complement the Federal Aviation Administration’s Part 107 Remote Pilot Certificate by providing an inexpensive, easily duplicatable mechanism for assessing remote pilot flying skills. The BPERP can be administered in 10 minutes utilizing three omni bucket stands, a 50’ tape measure, and a stop watch. The BPERP requires a compact test area of 50’x 20’ so can easily be administered indoors or outdoors. The BPERP requires the remote pilot to conduct three takeoffs and landings from a 12” radius circle, climb to specified altitudes of 10’ and 20’ AGL, conduct yawing turns, and conduct forward, reverse, and transverse flight maneuvers. The goal is to capture still images of 36 targets that are placed within two gallon buckets that are fastened to three omni bucket test stands that are constructed from 2”x 4” and 4”x 4” lumber. The bucket stands are easy to assemble and can be transported in a couple of nylon golf club bags or simply stacked and placed in a vehicle. The test consists of one maneuvering phase and two transverse flight phases. Pilots earn one point for each accurately captured target image, two points for an accurate first landing, and one point each for accurate second and third landings. Scoring sheets are available from NIST. Agencies set their own benchmarks scores for passing the test.

Ben Miller, director of the Colorado Center of Excellence for Advanced Technology Aerial Firefighting, has followed NIST’s sUAS Standard Test Methods project from the inception.

“NIST was one of the very first evaluation groups to show interest during the early days of UAS in public safety. The rigor that today’s Standard Test Methods show is a direct result of their years of work into the project. The applicability of the method supports acquisition decisions as well as employment considerations. The NIST sUAS Standard Test Methods produce data that can be used to answer the questions of what system do I buy and what system do I use for which mission?”

Miller is convinced that the NIST sUAS Standard Test Methods have great applicability to public safety sUAS operations. He has backed up that opinion by being an early adopter of the NIST sUAS Standard Test Methods. “The Colorado Department of Public Safety has adopted the NIST sUAS Standard Test Methods within our UAS certification process. Managed by The Center of Excellence for Advanced Technology Aerial Firefighting (CoE) within the Division of Fire Prevention and Control (DFPC), the CoE provides this certification process to stakeholder public safety agencies within the State of Colorado. To date, 16 agencies and 42 UAS operators have gone through the process.”

Adoption of NIST sUAS Standard Test Methods provides public safety agencies with a mechanism for evaluating sUAS airframes and sensor systems as well as raising the bar on remote pilot credentialing. The combination of requiring remote pilots to obtain a Transport Canada RPAS Pilot Certificate and achieve a minimum passing score on the NIST sUAS BPERP will provide fire agencies with a solid foundation and show due diligence in their efforts to ensure safety of their sUAS operations.

Throughout the next year the Airborne Public Safety Association (APSA) will be presenting several NIST sUAS Standard Test Methods Train-the-Trainer Workshops throughout the United States, and hopefully within Canada. These three-day courses are appropriate for experienced sUAS pilots who serve as trainers, supervisors and managers within sUAS Units. Check the APSA website, publicsafetyaviation.org, for dates and locations of upcoming workshops. Agencies wishing to co-host a workshop should contact the author at alan.frazier@georgetown.edu.

Alan Frazier currently serves as a senior fellow at Georgetown University. He is assigned full-time to the National Institute of Standards and Technology where he works on development of sUAS standard test methods. Alan previously served as an Associate Professor of Aviation at the University of North Dakota. He is a 40 year law enforcement professional having served as a sworn officer and supervisor with city, county, state and federal law enforcement agencies. He founded, and served 10 years as the officer-in-charge of, the Northeast Regional sUAS Team in Grand Forks, North Dakota. He is an FAA Airline Transport Pilot rated to fly single and multi-engine airplanes, helicopters, gliders, and small unmanned aircraft systems.