Dynamically Managing Task Allocation Between Humans and Machines in Surveillance Operations

Constructing an Autonomous Manager (AM) for use as an integral component of distributing multiple tasks between humans and autonomous agents, particularly in Intelligence, Surveillance, and Reconnaissance (ISR) applications.

Increasingly sophisticated technology must be leveraged in surveillance environments to enable eventually achieving the goal of allowing analysts to increase throughput by managing multiple simultaneous feeds. Maintaining this increased tasking will likely introduce additional workload and fatigue. Fortunately, analysts can currently offload some of these tasks to automation and will, in the future, be able to offload additional tasking to streamline the intelligence analysis process.

An illustration of a possible multi-tasking configuration consisting of multiple visual search surveillance tasks (T1, T2, T3, T4), involving either standard image/video feeds or infrared with a simplified representation of the AM's decision logic below.

Currently, various speech-to-text and text-to-speech programs can be used to convert spoken information into chat and automation can be used to copy text to multiple needed locations simultaneously. Automation has aided in the transmission of information between analysts and organizations. Tools are also being developed to augment the detection of important visual features within surveillance scenes. However, the degree of assistance autonomous systems can provide is still somewhat limited for cognitively complex tasks, but progress is being made incrementally toward viable assistive tools. Balancing analyst workload while maintaining multiple tasks will require intelligent and dynamic distribution of tasks between humans and autonomy.

To address this challenge of maintaining performance in a Human-Machine Team (HMT), a program was developed to dynamically distribute tasks between a human and automation. By re-conceptualizing the team dynamic within the surveillance working environment, it was possible to develop a supervisory role called the Autonomous Manager (AM). The AM dynamically reallocates tasks based on task performance and physiological indicators of a human analyst's workload.

The AM's decision logic was tested across multiple scenarios using simulation, allowing the examination of the benefits and limitations of the AM more thoroughly than would be feasible with a series of empirical studies using human subjects. The benefits of the AM were tested based on performance improvement across tasks, improvement with highly variable mean performance, and specific use-case scenarios. The simulated AM can be leveraged to answer a variety of real-world questions without the expense of physical implementation, prior to full-scale development or empirical testing.

In most work environments, accuracy and efficiency are both crucial. To continue growing productivity and throughput, work organizations must increasingly rely on sophisticated automation to complete complex tasks. However, many cognitively taxing tasks cannot be performed by automation to a level of proficiency that matches an expert human at full arousal (i.e., an analyst who is not heavily fatigued or overworked). Integration of automation into real-world workplaces demands effective HMT to compensate for the relative strengths and weaknesses of each human and machine agent. Prior autonomy research has attempted to establish an appropriate balance of tasks from a range of theoretical perspectives. The perspective taken for development of this AM tool was from previous work in managing multiple tasks with humans and machines with twofold assumptions:

  • Tasking can be distributed according to a spectrum, ranging from automation/machine having full control to human having full control. There are variable degrees of split tasking between humans and automation/machines that can include parsing complete or partial tasks between agents.

  • Dynamic implementation of automation provides a greater benefit than simple substitution of a single or multiple humans. There are times where human control of tasks may be more appropriate and other times where automation control of tasking is more appropriate, based on features such as workload or fatigue.

This work was done by Mary Frame and Alan Boydstun of Wright State Research Institute for the Air Force Research Laboratory. For more information, download the Technical Support Package (free white paper) from the link below. AFRL-0307



This Brief includes a Technical Support Package (TSP).
Document cover
Dynamically Managing Task Allocation Between Humans and Machines in Surveillance Operations

(reference AFRL-0307) is currently available for download from the TSP library.

Don't have an account? Sign up here.