Automotive HMIs Aren’t Just for Driving Any More

As autonomous driving inches into the marketplace, human-machine interfaces will undergo significant changes to make it simple to switch between driving and work/entertainment modes.

July 1, 2015

Terry Costlow

Linking all the systems related to autonomous driving with human-machine interfaces is challenging for OEMs like Jaguar Land Rover.

Most autonomous driving discussions focus on the sensors that inform vehicle controls and the techniques those controls will use to make decisions and take action. But as autonomous driving inches into the marketplace, human-machine interfaces (HMIs) will undergo a significant evolutionary change.

HMIs will have to engage drivers when the vehicle’s doing the driving, letting them work or entertain themselves without being too distractive. User interfaces also will have to make it simple to switch between driving and work/entertainment modes. It’s doubtful that humans will completely give up driving for quite some time.

Visteon includes simulators in the tools it’s using to determine how to address the nontraditional challenges that come with user interfaces for vehicles with various levels of autonomy.

“When vehicles become fully autonomous, our research has shown that people will probably continue to drive themselves somewhere around a third of the time, typically for shorter trips,” said TC Wingrove, Senior Manager, Global Electronics Innovation, Visteon. “For the other two-thirds of the time, they will want the car to assume driving duties. They will likely let the car drive for longer or more mundane trips.”

Figuring out when drivers need to be alerted about potential issues involving other vehicles and pedestrians is no simple task. In semi-autonomous vehicles, systems will have to determine when and how to alert drivers. For full autonomy, the HMI may warn people of automated braking or steering so they aren’t startled. In the complex world of automotive electronics, getting all these pieces to work together will take a lot of cooperation throughout the supply chain.

“An automotive message broker could provide a way to abstract all messages on the car’s network into a common format, maybe HTML 5,” said Matt Jones, Head of Future Infotainment at Jaguar Land Rover. “That would make it easy to share information from the speedometer, temperature gauges, and other devices. It would require all the OEMs to come on board and create a single methodology.”

Many modes

HMIs may evolve from today’s music-related displays to include video, said Nuance’s Pavan Mathew.

User interfaces are going to operate differently depending on conditions. Sometimes, people will have to drive, particularly in busy urban environments. Other times, they will want to.

“Drivers may want to switch from a driving mode to a business mode where they can dictate email or have the car read email for them,” said Ingo Krueger, Business Unit Director at IAV. “People may want to work close to the way they work on a plane or train.”

It’s going to be challenging to let drivers reduce boredom or do something productive while safety is maintained. That’s likely to make HMIs even more of a product differentiator.

“Connectivity and infotainment capabilities are becoming major factors in a vehicle purchase,” said Pavan Mathew, Senior Director at Nuance Automotive. “As autonomous driving technologies deploy, we may even see additional services (video streaming) becoming part of the HMI experience. Usability and safety will remain paramount, and industry players will need to ensure that infotainment-related services are not a driving distraction.”

Activities now classified as dangerous distractions will eventually become the norm for automated vehicles. But it’s going to be a long time before cars can safely navigate non-standard driving patterns. Humans who aren’t paying attention will have to take charge when something unusual happens, so HMIs will have to bring their concentration back to driving.

“For vehicles that employ some version of partially automated driving, a number of nontraditional challenges in system design need to be addressed,” said Michael Tschirhart, Human Factors Technical Fellow at Visteon. “One such challenge involves managing the transfer-of-control, both from car-to-driver and driver-to-car. In the former case, it is important for the vehicle to determine the readiness of the driver to assume control of the vehicle. Similarly, in the latter case, the driver needs to assess the readiness of the vehicle to assume control.”

People and peculiarities

Audi used LEDs to tell drivers when they could relax and when they needed to take over driving.

Autonomous controls can already handle routine tasks like lane keeping and adjusting speed to safely match the levels of surrounding vehicles. But when unusual events call for subjective actions, humans are still required.

When it’s time for humans to drive, HMIs will be tasked with refocusing the driver’s eyes on the road. One challenge will be to determine whether the driver is watching the road or looking elsewhere.

“Critical decisions for vehicle operation, such as driving, steering, speed, etc., will be based on the state of the driver,” said Stephan Tarnutzer, Vice President, Electronics, at FEV North America. “Therefore, driver monitoring will be a critical safety function of an autonomous vehicle, becoming a big part of functional safety.”

Product developers have to balance the goal of directing drivers’ focus early enough for them to respond with avoiding false alarms. Alerts will probably be different when humans are engaged with driving than when drivers are reading or are otherwise distracted.

“By augmenting semi-autonomous systems with a driver-monitoring system, the vehicle can give the driver more time to react to a potential threat and prepare them to take over control of the vehicle,” said Brian Daugherty, Director of Global Intellectual Property at Visteon. “If the system senses that a driver is distracted, a warning can be issued based on lower confidence level information than if the driver is attentive and appears to be monitoring the potential threat.”

Bright lights

HMIs will evolve from providing input for driving to more infotainment and alerts that tell when to take control instead of relying on autonomous systems.

When an Audi pilot vehicle drove itself from San Francisco to the CES International show in Las Vegas early this year, the hands-off concept was a novelty for drivers used to watching the road. A simple light bar worked to tell them that they needed to put their hands on the wheel.

“One of our main goals is to implement a scientifically proven HMI with no weird buttons or new technologies,” said Daniel Lipinski, Project Lead at Volkswagen Group of America’s Electronic Research Lab. “We added an LED bar. Green is good, yellow means prepare to take over. When the driver takes over, they have to push two buttons. We don’t want anyone to accidentally de-activate the control system.”

Simple lights work well when drivers are relatively engaged with driving. But as drivers get more used to autonomous control, they’re more likely to become engrossed in work or entertainment. That poses challenges, especially for cameras that watch drivers.

“It is not enough to check only one item like the eyes,” IAV’s Krueger said. “You may have to check several things to really get the status of a driver and how fast he is able to take control again.”