AV and ADAS Testing from out of the Box

A new tool from Foretellix guides engineers through virtual systems testing to minimize the risk of getting dangerous pre-deployment bugs.

The new ADAS and Highway Solution package is designed to assist engineers developing SAE Level-4 commercial vehicles. (Foretellix)

For self-driving vehicle systems to gain the full confidence of the public, the mobility industry and government regulators, OEMs have to prove that AVs (and those equipped with enhanced SAE Level-2 automated-driving systems) are safer being driven by software than by humans. But a recent string of unfortunate events in 2020 involving automatic emergency braking (AEB) systems is causing concerns.

For self-driving vehicle systems to gain the full confidence of the public, the mobility industry and government regulators, OEMs have to prove that AVs and those equipped with enhanced SAE Level 2 automated-driving systems are safer being driven by software than by humans. (Foretellix)

Early in the year, Volvo recalled 750,000 cars from 2019 and 2020 production over a fault in its AEB. Engineers discovered a software glitch that prohibited the system from activating under certain temperatures. Then in June, in Taiwan, a Tesla Model 3 operating in Autopilot mode slammed into an overturned tractor-trailer rig at 70 mph, reportedly due to software issues related to the car’s forward-facing sensor array and an AEB that failed to react. This incident was followed in August when the American Automobile Assoc. (AAA) released results of a blockbuster study  that painted a worrisome picture of the real-world performance of active driver-assistance systems.

“We can work with all the popular testing tools used by OEMs because we are ‘simulator-agnostic,’” noted Foretellix’s Roy Fridman. (Foretellix)

“There is the tragic impact on human lives, and a huge cost to the manufacturer in these scenarios,” noted Roy Fridman, VP Business Development at Foretellix, the Tel Aviv-based developer of automation and analytic tools for AV-systems testing and development. In an interview with SAE International, he argued that both the Volvo and Tesla incidents could have been avoided with improved testing, using Foretellix’s new ADAS and Highway Solution package, part of the company’s Foretify platform.

Fridman describes ADAS and Highway Solution as “an out-of-the-box tool that is able to tell you what you need to test in order to get a real, thorough verification for your ADAS and highway (AV) system – from [SAE] Level 2 through the fully autonomous Level-4 systems now in use in highway trucks.” It guides engineers on how they need to test, while providing “a kind of benchmark noting when enough testing has been done, so you can minimize the risk of your systems getting these pre-deployment kind of bugs,” he said.

The new tool contains a library of 40 scenario categories. Once you know what to look for, the “engine of Foretify generates millions of meaningful tests and edge cases – things that we as humans would not think about before deploying a system,” Fridman explained. Upon completion of a test run, the data is aggregated into a dashboard that suggests more specific investigations and test scenarios.

For example, ADAS and Highway solution will report, “You have not done enough right turns in the rain, and enough left turns into the snow – but you have completed enough merges into the highway when you have two vehicles on your right and on your left,” he asserted. “It guides you to where you need to direct your next effort for further verification.”

Fridman said he believes that had Volvo and Tesla used the Foretellix product, it is likely that they would have discovered their AEB-related problems and fixed them. He added that soon after the Tesla Taiwan crash Foretellix engineers replicated the incident and had tested more than 100,000 variables “within a couple weeks.”

Founded in 2018, Foretellix’s “mission” is to enable measurable safety of AVs by transitioning from the traditional industry standard of measuring the number of miles/kilometers the vehicle has driven, to automatically testing and covering billions of possible scenarios before the vehicle enters road use. The company’s innovative algorithm for describing scenarios is Measurable Scenario Description Language (M-SDL) – an open, human-readable language that allows simplifying the capture, reuse and sharing of driving scenarios.

M-SDL is capable of specifying any matrix of scenarios and operating conditions to identify previously unknown hazardous core and edge cases, the company claims. It has been adopted for development of the OpenSCENARIO 2.0 industry standard. “We can work with all the popular testing tools used by OEMs because we are ‘simulator-agnostic,’” Fridman said. “We’re easily integrated into OEM systems, and easy to operate.”

Foretellix’s ADAS and Highway Solution package was developed with inputs from customers, data from past accidents, technology limitations, upcoming regulations, safety standards, and disengagement reports. The company claims it delivers the industry’s most effective solution for exposing bugs and verifying the quality and reliability of ADAS and AV systems.