10 Best Practices for ADAS and AV Testing
A veteran tester who has datalogged many thousands of miles in the U.S. and Japan offers suggestions for rapidly acquiring good test and validation data.
Testing for automated driver-assist systems (ADAS) has required a completely new approach to testing. The most obvious reason for this is the sheer number of sensors and actuators involved in any given feature. Whereas engineers used to test single sensors at a time, today there are upward of 20 sensors to specify. These sensors – short- and long-range radar, mono/stereo cameras, sonar and lidar – need to act in concert. This means that several layers of sensor fusion must be implemented.
For example, when it comes to the actuators for the braking system, there are up to seven distinct systems able to apply the brakes. As a result, a failure is not easily tracked to a single root cause but can be a complex combination of several factors. Imagine expanding this to ADAS and autonomous-vehicle (AV) functionalities such as automated cruise control (ACC), lane-keeping assist (LKA), automated emergency braking (AEB) and valet parking, as well as convenience features like blind-spot monitoring, night vision and steering beams, to name just a few.
We have a task of gargantuan proportions in front of us. With such complexity, best practices are mandatory. The following is a “Top-10” checklist for testing today’s increasingly complex ADAS and AV systems. It is based on my company’s decades of experience testing systems for leading Tier 1s and automakers, as well as my own recent experience logging thousands of miles in support of a new cloud-based testing and validation solution, EB Assist Test Lab, across the U.S. and around Tokyo.
- Filter or annotate your data as soon as and wherever you can, as part of a multi-step approach to annotation
Doing so will ensure that each step takes advantage of the actual capabilities of your system. This is the single best piece of advice I can share with anyone embarking on a test project. By tagging a recording in real- or near real-time around a new, unique or unusual use-case, you can dramatically reduce the amount of data you ultimately need. This logic should be applied throughout the entire process of data acquisition.
- Make your driver and/or passenger responsible for manual annotation
Traditionally the driver drives and that’s it. Introducing ways for the driver to easily tag data (by pushing a button on the steering wheel or voice control, for example) as it comes in makes it possible to quickly know which part of the recording might contain something relevant.
- Run algorithms in the vehicle data logger to identify the most relevant data
Many checks can be done inside the car data logger, saving time and frustration later on. These can be as simple as checking that the recording is consistent and not corrupted. Another example is if the values of signals exceed pre-defined thresholds – say, for acceleration – and therefore must be tagged. Doing so will result in a dramatic reduction of the data which needs to be uploaded.
- Ensure your upload station takes advantage of the pre-annotated data and can properly prioritize uploads to the cloud or your on-premise data center
It will invariably take a lot of time to upload all the recorded data, whether it’s to on-premise storage, a cloud provider or some hybrid solution. You’ve already annotated your data, so you should take one extra step and determine which data shall go where. Data that is not tagged or annotated should not be uploaded to a high-cost storage location as it will invariably be deleted or archived later. Delete it now and save money!
This best practice especially relevant if your teams are located in the same country or region and for urgent/priority projects, as your teams can start working on the relevant data as soon as it’s uploaded. Also, be sure that the upload station provides updates on its current status to the relevant individuals.
- Build a custom testing “platform” that meets your needs
Establishing a best-of-breed partner ecosystem is the only way to guarantee you can truly benefit from the different offerings on the market. Such an ecosystem should include a fast-upload datacenter, hyperscalers and cloud providers to handle all potential use cases, as well as software and hardware partners.
Partnering with companies that are experienced and understand your needs will ensure your resources are well utilized. There is no one-size-fits-all solution, and each company has different needs – oftentimes needs differ even inside a single corporation. All organizations, however, have a few common goals when it comes to their testing approaches, and these include flexibility, cost effectiveness and speed. Depending on your use-case each of these will eventually play a role.
The approach used by some OEMs and Tier 1s is to try to build it all themselves, but this isn’t efficient or cost effective. These companies should instead focus on improving their products and bringing them to market more quickly and efficiently. In the end, this is what will differentiate them from their competitors.
- Create a direct pipeline from your logger to the cloud that will allow the driver to share a snippet of a recording if something unusual is happening
This direct pipeline approach ensures that an engineer can investigate the issue ASAP and provide a fix or insights as well as determine whether the test drive is still relevant. This is critical to avoid dealing with unnecessary data. Examples range from a wrong version of software running on the ECU, to the updated software behaving unusually under certain conditions, such as low light.
- More data is not better – the diversity of your dataset will drive your success
Being able to identify how much coverage you have and being critical about your training dataset will ensure the best results. You must clearly define metrics to rank and evaluate your data.
Don’t underestimate the value of tools that can be used to explore your recordings and datasets with different criteria such as a heatmap of the position of traffic signs, showing where they were collected and at what time of day. Several companies collect and archive as much data as they can for Hardware-in-the-Loop (HIL) tests. They make sure they don’t miss a thing.
However, after analyzing the archived data from one vendor to determine its true value, I found that a tiny fraction – less than 10% – was relevant. The rest of it was duplicate information and therefore resulted in a very expensive, time-consuming process. Another important aspect is the quality of the recorded data. You must record your data with high accuracy (25 ns) so that you can later replay it in HIL farms with a microsecond accuracy.
- Test drive and simulation data should be treated equally
Being able to address them equally is key. By this I mean that your data-management system should be able to search regardless of the source of the data. I strongly recommend securing relevant data from third-party companies. While the sensor setup won’t be the same, the extracted scenario will be valuable.
- Identify unusual real-world scenarios to open up a new world of possibilities
A few examples of scenarios I’ve personally encountered include life-size imagery on side of a large semi-truck; chrome that acts like a mirror; a mannequin that has been ejected from the back of pickup truck; a plow with a trailer that is drifting sideways; a ladder opened on the highway; and a police car reducing the speed of traffic by continuously changing lanes with its sirens on.
These examples can later be reproduced in simulation and diversified in thousands of variances. The challenge is to only create plausible scenarios that can be enforced by the simulation engine and its rules and/or by the description language of a scenario used.
- Future-proof your toolchain using industry standards
To ensure the toolchain you’re creating is usable by others and future-proofed, look to industry standards such as Open Drive, Open Scenario and Open Label. Doing so will also ensure it is interoperable with other systems you may wish to use later. Assuming you’re achieving your current objectives by using clean interfaces and standards, these will also enable you to quickly onboard a new partner at any time.
The goals: Rapid feedback, good data
If there is one key “lesson learned” from my time as a road warrior, it’s that the driver can simplify the process for all involved by playing a more active role. While often overlooked, the simplicity and ease of use of the system in the hands of the driver will positively impact the quality of the collected data.
As the driver, you’re the person responsible for the entire chain of events. From personal experience, I can tell you that I want to find something I’ve never seen before. As a driver you also want to get quick feedback on what shall be collected and understanding the why enables you to obtain better data.
Jeremy Dahan is head of technology research at Elektrobit (EB), where he manages business development for the company’s efforts and partnerships in the Silicon Valley. He has logged thousands of miles driving across the U.S. and around Japan in EB’s test vehicle in 2019 and 2020.