Teleoperations Will Keep Humans in the AV Loop

As part of the bridge between today’s piloted vehicles and full autonomy, teleoperations will permit the remote operation of autonomous vehicles.

October 1, 2019

Bradley Berman

In June 2019, a Starsky Robotics truck used a combination of onboard self-driving technology and teleoperations to remotely pilot a tractor trailer along nine miles of the Florida Turnpike. (Starsky)

In March 2017, Leanid Tsurankou began working as a self-driving engineer for Uber’s Advanced Technologies Group in San Francisco. Those were heady days, when a wave of euphoria about autonomous vehicles swept across the automotive industry. But Tsurankou saw a different reality with Uber’s vehicles on Arizona roads, where he spent nearly half his time.

Leanid Tsurankou, chief executive officer of Qibus, believes teleoperations can be offered as a subscription service for mainstream vehicles. (Qibus)

“One day I came from one of our remote locations and thought this is a great technology,” Tsurankou recounted. “But the AI of a self-driving car is not like a human. And in the near future, it’s still not going to be as good as a human.” That’s when Tsurankou became convinced that keeping a human in the loop was the right way to deal with complex situations facing an autonomous vehicle (AV).

Tsurankou is now the founder and chief executive of Qibus, a San Francisco-based teleoperations start-up formed in February 2019. Qibus consists of approximately a half-dozen employees split between San Francisco and Tsurankou’s native town of Minsk, Belarus. The company is building teleoperations software that Tsurankou says will one day allow millions of drivers to push a button on the dashboard and turn driving responsibility over to a teleoperator. “Our idea is to provide Uber-like rides to people in their own cars,” he said.

Cruise, Nissan, Uber, Waymo and Zoox are among those working on various teleoperations strategies. They are joined by technology providers such as Designated Driver and Phantom Auto. “In the last two years, there’s been a lot more work on teleops,” said Praveen Penmetsa, chief executive of Motivo, a contract design and engineering firm with facilities in Gardena and Fremont, California.

Motivo has worked on numerous mobility and robotics projects for AV companies. Penmetsa makes a distinction between two types of teleoperations. In the first case, a remote operator is an integral part of the safety protocol and monitors the vehicle for extended periods. The second approach activates the teleoperator only when an autonomous vehicle is stuck.

Based on complexity

Starsky Robotics, a three-year-old company headquartered in San Francisco, garnered headlines in June 2019 when it used a combination of onboard self-driving tech and teleoperations to pilot a driverless truck for more than nine miles along the Florida Turnpike. Kartik Tiwari, Starsky’s chief technology officer and co-founder, is convinced that long-haul trucking is an ideal use for teleoperations.

“It’s getting verified not just by us, but by the broader industry that autonomy is a big problem to solve,” Tiwari said. “A human in the loop is the only viable solution.” Tiwari believes that teleoperation is not a whole-cloth replacement for self-driving, but rather an ongoing component for autonomy. “There will never be a time where we have a binary step of driving with a safety driver one day, and then suddenly tomorrow there is no involvement of any person ever,” said Tiwari.

For Starsky, the extent to which a teleoperator becomes involved is based on the complexity of an operating environment. Onboard autonomous systems will do the job for a long, straight stretch of highway. However, an off-ramp in a small town might require assistance from teleoperations.

A safety driver in a Starsky Robotics truck allows the vehicle to steer itself. (Starsky)

Kartik Tiwari, chief technology officer for Starsky Robotics, believes long-haul trucking is the ideal use case for teleoperations. (Starsky)

A Starsky teleoperator uses images from about eight cameras with different focal length. The stitched images provide nearly a 360-degree view around the vehicle. (Starsky)

Tiwari explained how a driving scenario could quickly change, for example, if an autonomous truck identifies an unanticipated construction site in the roadway ahead. Should the truck wait for a few minutes? Should it proceed slowly in the same lane? Or should it change lanes at its current pace? “A human in the loop is actually best suited to make that call,” he said. “All they would do is look at the scene, press a button, and a lane change happens.”

The range of use cases for teleoperations spans the gamut from robotaxis and shuttles to warehouse logistics and sidewalk delivery robots. Motivo’s Penmetsa believes that specific, customized teleoperations strategies will be designed based on the relative strengths and weakness of each company’s technology.

Not a panacea

There is a consensus that latency times are already acceptable in the hundreds of milliseconds and will be further reduced when 5G arrives at scale. Besides, Tiwari believes that latency is mostly a human rather than technological problem. “Even when we are manually driving a car, there is latency,” he explained, applying brakes before we come to a traffic light or beginning to turn the wheel before approaching a curve. “It only requires training for the teleops station. There is no physical limitation.” Tiwari compared the situation to an air-traffic controller that advises a plane about a change of direction 10 or 20 minutes before the instructions are implemented.

It will likely take one or more product cycles for teleoperations to be adequately integrated with new autonomous vehicles. One of the biggest challenges will be defining whether or not a teleoperator constitutes having a human being in the car. Some jurisdictions require a human safety driver in an autonomous vehicle, but they don’t specify if a remote human being qualifies for this role.

SAE is already considering how teleoperations will affect its definition of the five levels of autonomy. Under current definitions, humans can intervene at levels two, three, and four – but the J3016 standard does not yet specify if the intervention needs to happen from inside the vehicle. “It’s tricky,” said Tiwari. “We can try to shoehorn teleoperations as a Level-Two system. And we can also try to shoehorn it as a Level-Four system.”

Regardless of the SAE designation, teleoperations will require its own set of safety guidelines, including training, connectivity protocols and strategic geographic placement of teleoperations centers based on latency and the desired speed of vehicles. “Teleops gets us closer to an autonomous future,” said Penmetsa. “But it also creates a host of new problems that need to get resolved.”