Distributed, Collaborative Human- Robotic Networks for Search, Identify, and Track

February 1, 2012

Air Force Office of Scientific Research, Arlington, Virginia

A human-robotic system is under development that can map an unknown environment, as well as discover, track, and neutralize several static and dynamic objects of interest. In addition, the robots can coordinate their individual tasks with one another without overly burdening a human operator. The testbed utilizes the Segway RMP platform, with lidar, vision, inertial, and GPS sensors. The software draws from autonomous systems research, specifically in the areas of pose estimation, target detection and tracking, motion and behavioral planning, and human robot interaction.

Environmental sensing includes SICK and Hokuyo laser range finders, and three EO cameras. Localization sensing is provided by an inertial measurement unit (IMU), wheel odometry, and a dual GPS system. A distributed Real-time Data Distribution Network was developed, allowing for accurate time synchronization of data for key algorithms such as sensor fusion. The pose (localization and attitude) estimator uses a custom Bayesian filter, supplemented with lidar-based SLAM methods in order to ensure robust localization in complex terrain (e.g. during wheel slippage) and during indoor operations.

An estimate of the world state, defined as both the static environment and dynamic targets, is maintained locally as well as globally (collaborative). Static terrain maps are developed using probabilistic grid fusion methods, extended to a collaborative, decentralized system. An important characteristic of the static mapping is to maintain local and global representations of the maps for different functions such as path and mission planning.

The problem of autonomous target identification is addressed using computer vision detection methods fused with lidar in a formal Bayesian tracking and classification estimator. The expert human operator validates the targets and removes false positives. Collaborative behaviors are managed at the global level, with the connections to the Human-Machine Interface (HMI). A flexible interface has been designed that is capable of both Playbook and tabletstyle interaction.

The robot is designed to mount various sensors, electrical components, two mini-ITX computers, and two LiPo batteries. Each robot is equipped with one SICK LMS-291 and one Hokuyo URG- 04LX laser range finder, three PGR Firefly MV Firewire Cameras, one MicroStrain Inertia-Link Inertial Measurement Unit (IMU), and two Septentrio AsteRX1 GPS receivers. In order to provide a more accurate timestamp for all of the sensors on the robot, Cornell Skynet’s real-time data network technology is leveraged. Each sensor measurement packet obtains its timestamp from a microcontroller.

Each computer is equipped with a Core 2 Duo Mobile CPU, 2GB RAM, a wireless network PCI card, and an IEEE 1394 Firewire port. It is also equipped with a solid-state hard drive to prevent hard-drive failure from vibration as the robot maneuvers over rough terrain.

The purpose of the Localization component, termed Pose, is to form the best estimate of the robot position, attitude, and internal states possible given all of the data collected by the robot’s sensor systems, subject to computational limits. Maintaining a smooth trajectory estimate is required for robot tasks such as obstacle avoidance and local target tracking. Without an accurate history of its own position and orientation, the robot would be unable to properly merge data sets taken at different points in time, without which operation would be extremely limited if not impossible.

In order to globally localize themselves, robots rely on two 4Hz Septentrio GPS receivers with antennas mounted fore and aft of the robot center. These receivers are connected via wireless link to a ground station providing differential corrections.

The human-machine interface (HMI) serves three main purposes: to maintain operator situation awareness (SA), to facilitate natural and efficient interaction, and to support system flexibility and scalability. The current HMI solution consists of a double-monitor display and the hardware E-STOP. One screen displays a global view of the environment in which the robots are operating, including all robot poses and current statuses, detected obstacles, and dangerous areas such as Object of Interest (OOI) blast zones. The other screen facilitates more specialized interaction, such as viewing robot camera feeds, examining the local environment around an individual robot, teleoperating robots, and OOI confirmation and neutralization.

Streaming camera images from all robots are displayed at the top of the screen; the operator can click a robot’s camera view to select it as the Focus Robot. The Focus Robot’s camera stream is enlarged and displayed in the center of the screen, as well as a local scrolling map centered at the robot’s location. This allows the operator to gain a better understanding of the environment around a particular robot. In rare cases when the operator must control a robot directly, a teleoperation panel also provides joystick-like control of the Focus Robot. Teleoperation is most helpful when the robot is “stuck” or can otherwise not be safely controlled by the onboard planner.

This work was done by Daniel Lee, Mark McClelland, Joseph Schneider, Tsung-Lin Yang, Dan Gallagher, John Wang, Danelle Shah, Nisar Ahmed, Pete Moran, Brandon Jones, Tung-Sing Leung, Aaron Nathan, Hadas Kress-Gazit, and Mark Campbella of Cornell University for the Air Force Office of Scientific Research. AFRL-0196

---