Low-Latency Cameras, Onboard Computing Combine to Advance Operator Assistance

Complex imaging systems integrated with onboard computing will revolutionize the approach to everyday and complex tasks in the next generation of machines.

Computer algorithms can be trained to recognize specific images, or aspects of a video feed, to trigger events. (Stoneridge-Orlaco)

Modern industrial machines have been using cameras for years to improve visibility and eliminate blind spots. With the introduction of Ethernet cameras, an accepted standard for networked computing, more complex and cost-effective camera configurations can be implemented into on- and off-highway vehicles.

The devices can offer base machine controls like monitoring of engine and hydraulics system, control system settings and logging and visualization of production data or more advanced operator support and guidance system, and even functionality for semi-automation. (CrossControl)

One such configuration is the EMOS camera from Stoneridge-Orlaco, which fulfills requirements for use in most vehicles with its EMC (electromagnetic compatibility) conformity and resistance to shock, vibration and water. The camera boasts a low system latency thanks to the use of the RTP protocol over UDP, meaning the system is not slowed down by an internal browser, unlike some conventional IP cameras. As communication with the camera uses this independent protocol, developers have a choice of peripheral equipment and do not need to rely on a browser interface. The flexibility of the camera also offers the opportunity for advanced image processing and additional sensor fusion like lidar.

Once a camera system has been integrated into a machine it introduces a range of advantages to the operator. Some cabs incorporate dedicated camera monitors to share video streams from one or more feeds while others integrate the video stream with existing display platforms, like those available from CrossControl. At a base level HMI displays can mirror the video streams and give the operator additional information related to their task by showing new viewing angles or displaying active production processes.

The Stoneridge-Orlaco EMOS camera can be positioned discreetly on any work vehicle with an existing Ethernet network (either standard or automotive). (Stoneridge-Orlaco)

The latest generation of displays and onboard computing solutions feature powerful SoCs (systems on chip), which offer hardware support for rapidly emerging technologies like multiple digital camera streams, stream stitching to create panoramic views, object detection and classification as well as speech recognition. The devices can offer base machine controls like monitoring of engine and hydraulics system, control system settings and logging and visualization of production data or more advanced operator support and guidance system, and even functionality for semi-automation.

By combining low-latency cameras and powerful computing platforms, it opens industrial vehicles to new smart systems for enhanced productivity and safety. Experts from CrossControl (booth S84043) and Stoneridge-Orlaco (booth S65319) will be discussing the possibilities in the next generation of machines at the ConExpo-Con/Agg tradeshow in Las Vegas.

Upping computing power

Cameras are popular in agriculture with many combines and tractors already featuring preinstalled camera systems, which allow farmers and their teams to increase visibility and monitor key processes like harvesting. With more powerful computing tools, the cameras themselves can be used to recognize important aspects of tasks. Computer algorithms can be trained to recognize specific images, or aspects of a video feed, to trigger events.

Additionally, pre-trained modules, produced in centers of excellence, can be integrated into a software package to increase its functionality and operator support. With these increased software tools, the system can be used for a variety of new tasks. For example, this can be used as part of a sprayer’s control system activated by a multiple camera system that identified plants and activated the sprayer nozzles when needed.

In material handling, camera systems have been installed on the machine for additional visibility. If paired with a more powerful onboard computing solution, rather than a camera monitor, the system can have increased functionality and work to make the machine smarter, safer and more productive. One of the most common tasks for a forklift or reach stacker is finding and approaching the next cargo; with smart systems, the machine can be trained to identify the next load, either in a sequence, as ordered by the user or by a predefined priority list, and then plot a path along the optimal route for the operator to follow to the identified load.

Cameras play a key role on industrial machines for security, productivity and increasingly safety. Advanced systems utilizing the latest ARM cores, like the iMX8X, can incorporate modules for obstacle detection into their software framework. This allows the system to identify obstacles and hazards, including people, and highlight them to the operator or even automatically shut down the machine to avoid accidents.

Just a few years ago such a system was too slow to be implemented on machinery, but now with the latest advancements any machine can incorporate this technology to ensure the safest possible working environments. With the power to work in low light situations, thanks to the sensor’s high degree of light sensitivity (0.05 Lux), the EMOS camera also performs in dusk-to-dark environments (moonlight is sufficient). This allows systems to be operational and effective throughout the whole day.

With the correct implementation and support a system can be trained to measure distance and move tools into position. This can be a key productivity aid in many industrial environments like forestry where operators are tasked with repeatedly moving complex machinery into position to fell trees and cut timber to length. Having these tasks assisted or automated can be a huge boost in productivity and reduce the amount of training and experience needed for successful operators. The imaging system can incorporate pre-trained modules to identify the right location to attach tools and begin production without the need for additional operator input, leaving them free to maneuver the vehicle or handle other tasks.

Ken Lindfors, head of software development at CrossControl, and Thieme Wels, content marketeer at Stoneridge-Orlaco, jointly produced this article for SAE Truck & Off-Highway Engineering.