Understanding the Limits of Artificial Intelligence for Predictive Maintenance

The U.S. Air Force (USAF) deploys flying units with readiness spares packages (RSPs) to try to ensure that the units are stocked with enough parts to be self-sufficient for 30 days. This report is the third in a five-volume series addressing how AI could be employed to assist warfighters in four distinct areas: cybersecurity, predictive maintenance, wargames, and mission planning, with predictive maintenance in focus.

Predicting which parts are likely to fail — and, therefore, which parts should be included in the RSPs — is important because overstocking can be expensive and understocking can threaten mission readiness. This report presents a discussion of whether and when artificial intelligence (AI) methods could be used to improve parts failure analysis, which currently uses a model that assumes a probability distribution. To do this, several machine learning (ML) models were developed and tested on historical data to compare their performance with the optimization and prediction software currently employed by the USAF, using A-10C aircraft data as a test case.

The A-10C Thunderbolt, pictured here is the first Air Force aircraft specially designed for close air support of ground forces. To evaluate the limitations of artificial intelligence for predictive maintenance applications, the researchers developed several machine learning models and tested them on historical data to compare their performance with the optimization and prediction software currently employed by the USAF, using A-10C aircraft data as a test case. (Image: U.S. Air Force)

Using A-10C aircraft data as a test case, we addressed the following research questions:

  • How does the current RSP failure analysis approach perform in a retrospective analysis against historical data?

  • How can AI help inform the failure analysis process, and what are its limitations?

  • What other potential improvements can augment the existing approach?

The USAF currently uses the aircraft sustainability model (ASM) to calculate RSPs to maintain aircraft fleet availability and to manage maintenance budgets. ASM outputs a shopping list of parts for part managers to purchase for their RSPs. ASM was created by the Logistics Management Institute in the 1970s to calculate the composition of war reserve spares kits (WRSKs), the predecessor to RSPs. ASM calculates the composition of RSPs to maximize aircraft availability across the fleet for a given budget. A summary of this process is shown in Figure 2.1.

ASM predicts part failures (and thus demand for spare parts) for a given base within a 30-day period. Using this prediction and the amount of on-shelf inventory, ASM estimates the shortfall (number of backorders) in parts. Each backordered part, across all bases, results in not-mission- capable aircraft and decreases fleet-wide availability. Finally, ASM performs marginal analysis by iteratively identifying the least expensive spare part that yields the greatest increase to fleet-wide availability. The output is a shopping list of spare parts that decisionmakers can use to calculate RSP compositions. With marginal analysis, ASM helps decisionmakers answer two kinds of questions: (1) What availability can be achieved with a given budget and (2) what is the cost to achieve this target availability?

A historical comparison of part failure rates for the U.S. Air Force’s fleet of A-10C aircraft. (Image: RAND Corporation)

Failure analysis is the core of ASM’s functionality and the foundation for the entire RSP predictive maintenance analysis. ASM predicts part failures solely on the basis of the anticipated flying hours for contingency operations, as dictated by the War and Mobilization Plan (WMP). ASM assumes that part failures are characterized by a stationary Poisson distribution driven by the mean hourly demand. This demand, in addition to on-shelf base inventory, is used to calculate the number of expected backorders, which powers the subsequent availability and marginal analysis calculations. As a result, the Poisson distribution assumption is crucial to ASM’s optimal RSP composition calculation.

ASM is currently used to calculate RSPs, and it tackles a highly complex problem. Managing failure analysis across myriad parts across different aircraft, multiple bases, and depots is a difficult logistics and operations research problem that is not currently suited for AI. However, there could be potential for improvement in the narrow component of failure analysis. In the previous assessment, we identified the challenges of failure analysis, especially by extrapolating peacetime flying data. In the absence of contingency data, the best option is to learn from peacetime data and quickly update that data as time goes on.

Key Findings

AI can improve failure analysis for RSPs on a case-by-case basis. Compared with the static Poisson process that the USAF currently uses, we show that AI can generate better predictions and save money. Specifically, in our narrowly scoped analysis, the AI predictions would cost $25.1 million per month less than the overpredictions of the Poisson process. Although we only showed this for a single platform (A-10C), the results are likely generalizable because our statistical tests showed that part failures are not quite governed by the Poisson process. However, we did not consider the larger issue of allocation from depots to bases.

Current Poisson distributions do not seem to be rooted in empirical data; we demonstrated that updating these distributions with data can achieve an upper limit of performance that is close to the AI predictions. Comparisons of performance reveal regimes in which AI does better (parts with more frequent failures) and in which AI does worse (parts with rare failures).

A complex and labor-intensive data operations pipeline to USAF maintenance databases is necessary before any AI application can occur. Pulling LIMS-EV data is a manual process that involves scripting, drop-down lists, and nested menus. It is practical only for proof-of-concept models of the kinds described in this research. Moreover, considerable data cleaning is necessary to unlock historical data (e.g., linking variants prior to platform upgrades) and other potential predictors of failure. It might be unrealistic to conduct this analysis on multiple platforms by attempting to make these LIMS-EV data pulls manually. Automation might be necessary for these data pulls before AI can be applied.

How aircraft sustainability models calculate readiness spare packages. (Image: RAND Corporation)

AI cannot alleviate the scarcity of wartime data. Additional assumptions and policy considerations will be needed to account for this scarcity. As mentioned, one of the main limitations for the application of AI in this use case is its inability to estimate truly rare events. Certain events in war could be considered rare because of their unpredictable nature. As a result, different approaches to modeling AI might be required to deal with these changing circumstances. However, a regular retraining and updating policy, which is possible with an AI model, can ensure adaptability during war.

Recommendations for Air Force Materiel Command

Work with USAF Logistics to build a data operations pipeline to conduct retrospective analysis of aircraft maintenance and RSP efficiency. Aircraft maintenance programs and databases function effectively for the purposes for which they were designed, but they were not designed for retrospective analysis or to train AI models. Unless the data can be properly conditioned and pulled for this analysis, none of the following recommendations can be implemented.

Experiment with AI to improve failure analysis for RSPs. Extend the proof-of-concept models to all aircraft. This extension will likely have to be done on a part-by-part, platform-by-platform basis. Automated or partially automated data extraction will likely be necessary if AI is used to conduct these analyses. For RSP parts with hard-to-predict rare failures, we can modify the AI model’s cost function to prefer overpredictions or rely on overpredictions via a Poisson distribution or the problem can be modeled as survival analysis (predict time to failure).

Limit AI to failure analysis within the RSP process. The ASM software tackles a large and complex operations research problem of selecting which parts to send from which depot to which base. Current AI capabilities are data hungry and better suited to solving narrowly scoped problems. Splitting parts failures across multiple depots and bases will fragment the data too much for algorithms to learn anything useful.

This research was performed by Li Ang Zhang, Yusuf Ashpari, Anthony Jacques for RAND Corporation (Santa Monica, CA). The article features a summary of their results. For more information, visit here .