Next-Gen HMI Crucial to Increasing ADAS Efficacy

September 10, 2021

Markus Wallmyr

Next-generation HMI (human-machine interface) technology platforms allow off-highway OEMs and system designers to support productivity tools like computer vision – including object detection, advanced driver-assistance systems (ADAS) and semi-automatic equipment control – as well as traditional HMI features such as engine monitoring, navigation and instrumentation clusters. For example, CrossControl’s CCpilot V700, V1000 and V1200 displays are equipped with a powerful quad core SoC (system on a chip) that delivers triple the graphical performance of former industrial ARM design to the construction, agriculture, mining and material-handling sectors.

More computational power enables systems to provide more advanced functionality – aka “soft functionality” – to aid the operator, but it also can increase the operator’s mental stress. CrossControl invests in research into Interaction and UX design and performed two recent studies in conjunction with Mälardalen University that shed some insight into best practices for system deployment.

The research was carried out through both field and laboratory studies. The field-work study of actual work situations performed by forestry and construction equipment took place in Sweden. The operator’s attention and visual focus was analyzed using eye-tracking glasses. The benefit of glasses compared to, for example, display-fitted eye-trackers, is that they also capture everything the operator sees.

Optimizing display placement

Studying the operators gives some interesting insights into HMI design – specifically, cabin design and display placement. The forestry operators, engaged in thinning (removing unwanted trees to encourage more growth), spent nearly 95% of their attention through the front windscreen. One reason for the high focus straight-ahead is because the whole cabin rotates with the crane; this way the operator is always facing the area of operation.

The setup of the forestry vehicle featured two screens arranged in the lower section of the front windscreen. While the display provided constantly updated information, it primarily was used to adjust settings and, through quick glances, obtain system information when operation was not behaving as expected. This trend was similar for most vehicles and even more evident for vehicles with the display placed to the lower side of the cabin; the operators only looked at the system’s default displays for an occasional glance, and some had no glances all. This might result in the operator not benefitting fully from the system or even missing critical information being presented.

The attention paid to support or add-on displays, usually placed higher up and to the side of the operator than the default display, was noticeably different. Their use naturally related to the task being performed – for example, high precision excavator work – but the higher placement also received additional glances. But even here there are limitations. In most cabins the displays are mounted on the right side, but as the operator interacts with the surrounding world and coworkers, attention gets drawn straight-ahead or to the left. Thus, we are again in a situation where information is presented in a place not benefitting the operator.

Then there is specific operator support functionality. One such example is the use of reversing cameras, beneficial for higher awareness, safety and a better ergonomic climate for the operator. For the systems with displays closer to the natural line of sight, the usage was higher; the reversing cameras using dash displays weren’t used at all. The most-used reversing camera situation was for a display mounted close to the rearview mirrors, which meant very little extra effort to use in the natural attention process and task execution.

As the eye-tracking study clearly shows, line of sight is important. Placing displays within the operator’s field of view provides readily accessible information that is consumed with less effort throughout the task. The results of this study suggest most situations could benefit from a high A-pillar display allowing the driver to easily observe key information and quickly switch focus between the areas of interest and information.

This solution is now attainable with relatively high-performance computers due to low-energy-consuming central processing units (CPUs) and lightweight materials. For example, the new Vision Line display series from CrossControl has a Valox housing that provides ample protection (rates to IP65, 66 and 67 and meets the latest electromagnetic compatibility [EMC] requirements) at a lighter weight than all-metal enclosures. The CCpilot V700 weighs just 650 grams (1.43 lb), allowing easy pillar, post dash, console or roof installation.

Head-up displays twice as helpful

To take the eye-tracking study to the next level, a mixed-reality excavator rig was built to evaluate the use of HMI solutions in different deployments with “operators” engaged in performing the same set of tasks, including both vehicle navigation and operation of the boom. In a direct comparison between two of the display setups, the traditional head-down display placement – low and to the left of the operator – was universally deemed to be the least helpful and to provide the greatest risk of missing crucial information or safety warnings. By contrast, a head-up display (HUD) placed close to the operator’s natural line of site was more than twice as helpful and had half the associated risk of missing key information.

The subjects using this simulator setup also rated the workload level of using the HUD as being 20% lower than the associated workload of the head-down display, even though the task was identical. This could lead not only to operators missing information, but also to operators becoming fatigued faster, increasing the risk of accidents. Subjects also recorded a much higher level of frustration, nearly 33% more, when using the head-down display compared to the HUD. The head-down display also placed a greater physical toll on the users, nearly 40% more, due to operators needing to contort their bodies to see the contents of the display placed low and to the side away from their natural eye line and where they had to focus for the task.

While head-up displays still face some obstacles to heavy-vehicle adoption, it is important for OEMs to be able to freely choose a display size that suits a vehicle, one that fits the cabin and is large enough to provide the necessary information. Larger, more-complex machines often need to provide more guidance and information, especially when aiding less-experienced operators to meet productivity goals. Technology platforms that support a split concept display solution could help in these situations. Having multiple screens – with, for example, full information on the right and key notifications on the left – makes it possible to present key information regardless of where the operator is paying attention.

The latest displays such as the Vision Line series, which are based on the iMX8 family of application processors from NXP, allow the visual user experience to be modernized and more appealing for the operator to use, and they enable the same software base to be used regardless of display size. Having the current and next generation of wired and wireless interfaces such as CAN, Ethernet and USB allows for connection to both traditional machine systems and trending new peripherals, including Ethernet cameras and lidar.

One advantage with Ethernet cameras is that you can connect many cameras on one network and thus show several camera streams simultaneously on the same screen, reducing the number of displays needed in the vehicle to regain freedom of placement and visibility of surroundings. Showing multiple video streams is one thing – the real power of next-generation HMI solutions is that they support the emerging trend of image processing like 360° surround view (bird’s-eye view) and real-time analysis of the video stream, allowing for the implementation of object detection to increase awareness and operator productivity.

With next-generation computing technology and solutions for different equipment types, all supported by a common platform, designers can choose a display placement and size that suits the needs of the end user. For sectors requiring diverse tasks, such as construction and agriculture, designers can tailor the machine system to meet the wide range of application requirements rather than cumbersomely installing a heavy display either out of sight, out of reach or worst of all, out of mind. Powerful, graphical displays enable OEMs and system integrators to supply industrial machines with the next step in machine intelligence.

Dr. Markus Wallmyr, head of UX at CrossControl and affiliated researcher with Mälardalen University in Sweden, wrote this article for SAE Media.

---