Test and Evaluation of Autonomy for Air Platforms

August 1, 2021

Air Force Test Center, Edwards Air Force Base, California

The 2018 National Defense Strategy emphasizes that the effective implementation of autonomy is essential for future engagements. Key to this implementation is the ability to test and evaluate systems that perform autonomous tasks. The purpose of this research is to equip testers with tools, approaches, and insights to confidently approach the testing of autonomy on air platforms. The air domain is chosen specifically for its applicability to the Air Force mission and to help scope the focus of this research. The intent is not to be an exhaustive reference for testing and evaluating autonomy; rather, the goal is to provide a launching point for greater investigation.

Testing autonomy on airborne platforms brings unique challenges and, therefore, to be successful, testing should leverage the following five principles: Early User Involvement, Continuous and Cumulative Feedback, Streamlined Process and Products, Pilot Training Approach, and Human-Machine Interaction Consideration. The overarching test approach is split into two paradigms: three phases of test and Agile Development and Operations (DevOps).

First, the current flight test paradigm of plan, execute, and analyze (three phases) is reexamined through the lens of testing autonomy. Of the three phases, executing a test of autonomy is currently the biggest challenge within the test community. The goal of any test, and of autonomy in particular, is that it is safe, secure, effective, and efficient. Therefore, the optimal test execution of an autonomous task or mission should include some type of run time assurance, live-virtual-constructive, open systems architecture, and a surrogate platform.

To support the execution of autonomy testing that is responsive and timely, a close pairing of the planning and analysis phases is recommended. Different approaches, like design of experiments, human factors considerations, hardware-in-the-loop, and systems theoretic process analysis, are provided as tools for facilitating the needed planning and analysis capabilities for autonomy test. Within this paradigm, the goal is to adapt existing practices and modify to facilitate a spiral test and evaluation strategy of autonomy.

The second paradigm features a more dramatic shift from the current test practices to one that incorporates the principles of Agile and DevOps. The Agile philosophy emphasizes individuals and interactions over processes and tools, working products over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. It has been instrumental in modern software development and will certainly play a role going forward with the development of autonomy.

Additionally, DevOps is the revolutionary concept that integrates the development and operation functions of an organization. This process relies on continuous integration, continuous delivery, continuous monitoring and logging, microservices, and collaborative, cross-functional teams to implement high-quality products, quickly. As the Air Force begins to adopt more of the Agile and DevOps principles, it will be able to realize the DoD’s goals of “delivering at the speed of relevance,” “organize for innovation,” and “streamline rapid, iterative approaches from development to fielding.”

This work was done by Riley A. Liver-more and Avery W. Leonard for the Air Force Test Center. For more information, download the Technical Support Package (free white paper) below. AFRL-0305

---

---

This Brief includes a Technical Support Package (TSP).

Test and Evaluation of Autonomy for Air Platforms

(reference AFRL-0305) is currently available for download from the TSP library.

LOGIN TO DOWNLOAD

Don't have an account? Sign up here.

---