Supervisory Control State Diagrams to Depict Autonomous Activity

This method will enable a human operator to monitor, inspect, and manipulate activities of multiple UAVs, including situation assessment, decision-making, planning, and actions.

The military seeks to enable agile and adaptive mission management and control for a team comprised of unmanned aerial vehicles (UAVs), unattended ground sensors (UGS), dismounted war-fighters with mobile control stations, and an operator located in a central control station. With UAVs equipped and authorized to re-plan and act without human input, the challenge is developing methods for a human operator to sufficiently monitor, inspect, and manipulate the UAVs’ activities, which include goal-directed task selection, situation assessment, decision-making, planning, and actions.

The L-PRISM and its three components: A) the State Diagram, B) the Timeline, and C) the Payload Viewer.

In addition to providing more detailed information concerning the information processing and behavior of an autonomous system, there is a challenge to present the information in a manner that affords quick and accurate assessment of the multi-vehicle system. The goal of this work was to design concepts that can support the use of symbols and patterns in an attempt to support “at-a-glance” recognition of complex activities.

Hierarchical pattern-oriented state diagram concepts are being developed to represent the autonomous activities. Layered finite state machine diagrams combined with a control timeline and payload inspection display, collectively referred to as Layered Pattern Recognizable Interfaces for State Machines (LPRISM), will be integrated with the multi-UAV control station’s tactical situation map and system status information to assist the operator to not only be aware of the vehicles’ locations and planned routes, but also their mission goals, associated tasks, and states to achieve the mission goals.

The L-PRISM concept is composed of three display components: A) the State Diagram, B) the Timeline, and C) the Payload Viewer (see figure). The State Diagram uses the conventions of finite state machine diagrams to represent the autonomous system activity to the operator in real time. The State Diagram depicts the autonomy through the state nodes and transition arcs of the diagram. State nodes are display elements that represent autonomous tasks, showing what the vehicle is doing. The transition arcs represent the specific criteria for changing tasks, showing why the vehicle may move to a new task.

Multiple vehicles can be displayed in one diagram to accommodate multi-UAV monitoring and control. The State Diagram simultaneously shows the state of each UAV in the mission through vehicle icons. Vehicle icons depict the vehicle type, identify the vehicle by its call sign and unique color, and provide a time-on-task clock. The autonomous system can only make task and sub-task changes that follow the transition arcs. For tasks that have no transition arcs, task changes can only be made by operators. In general, vehicle control within L-PRISM has the flexibility to support different levels of automation, as long as there is adequate communication with the particular vehicle(s).

L-PRISM uses a layered arrangement of nested state diagrams to provide representation of autonomous tasks at varying levels of abstraction. These levels of abstraction are expected to enhance understanding and management of autonomous activities in part by displaying connections between actions, plans, and goals.

The Timeline presents information on significant mission events and provides controls to review that information across the control station. An event is represented by a colored tile containing a letter or icon that depicts a vehicle task change or payload delivery. Colors match the vehicle color that is used throughout the control station, and icons show the type of payloads. Operators can proceed forward or backward in time to investigate the tasks and actions of the vehicles and rationale behind those actions.

The Payload Viewer displays a sortable list of mission-relevant data referred to as payloads. Typical payloads are images or videos from UAV sensors, but could be operator-generated images or videos as well. Additionally, all operators (i.e., dismounted warfighters with mobile control stations and the stationary central control station) can create and send text messages or “voice note” audio recordings as payloads.

L-PRISM is an evolving supervisory control display concept that shows promise for providing many of the desired attributes and features for displaying information on a highly autonomous multi-vehicle system.

This work was done by Michael Patzek, George Bearden, and Allen Rowe of the Air Force Research Laboratory; Clayton Rothwell of InfoSciTex Corporation; and Benjamin Ausdenmoore of Ball Aerospace. AFRL-0227