Making the Strategic Move into AV Testing

Having built its global engineering reputation in the combustion-engine era, FEV makes a critical leap into automated-vehicle testing.

Knowledge of and experience in cloud computing, mechatronics, data science – and a systems mindset – are some skillsets FEV values in its AV test engineers. (FEV)

Traditional powertrain testing can be full of challenges. But it doesn’t compare to the often-bizarre edge cases that confront those testing automated-vehicle systems. Just ask Tom Tasky. He cut his engineering teeth in the powertrain-test space before transitioning into his current role as director, Intelligent Mobility, at FEV North America.

An autonomous shuttle developed by FEV and under test in Europe. (FEV)

“Tackling the edge cases in ADAS and AV development is sometimes overwhelming,” Tasky told SAE’s Autonomous Vehicle Engineering. “Lots of crazy scenarios that you could never envision. To cite just one example, our guys were road-testing an ADAS system in Europe when they noticed, in the distance, a car parked on the side of the road. It appeared to be stranded. Then a flatbed truck appeared, parked in front of the car. The truck, possibly a recovery vehicle, was covering the lane line.”

The truck had reflective white stripes painted on the back of its cab. And it was those white stripes that the fast-approaching FEV test car’s sensor array locked on to, “thinking” it was a new lane. An impending collision was averted by the quick-reacting FEV engineers, Tasky noted. “It was just another point of learning for them and for our algorithms,” he said.

More than 40 years after its founding, the FEV Group continues to expand its engineering-services portfolio beyond its traditional powertrain focus, to keep pace with the industry’s shifts into electrification, alternative fuels and connectivity. Most recently, the company turned its attention to automated--driving systems. “Over the past few years, we’ve had an increasing amount of customer activity in the ADAS, AV, and data-collection space,” Tasky said. “This prompted our creation of a new division, Intelligent Mobility, which I head in North America. With this new group we broke away, in a sense, from the ‘engine mindset.’”

FEV management heard opportunity knocking hard. There was particularly strong customer demand to extend its hallmark benchmarking competencies into sensor systems. The learning curve involved with all things related to ADAS and AV was admittedly steep; FEV had no previous expertise in the new domain. The move required new internal processes focused on serving everyone from sensor startups to incumbent Tier 1s and OEMs.

“It was a giant step,” Tasky explained. “We made a critical investment in the platforms, including Ford Fusion demonstrator/test vehicles, to get our ADAS and AV testing services out on the road. We’ve invested in people with the right skillsets. We found experts and engaged with some universities. We needed to get up to speed quickly, because a lot of our customers were already working in this space and had invested heavily to roll out their systems. They expect us to be in sync with what they’re doing.”

An exponential transition

Driver reaction times and human performance is an area FEV is studying and will likely expand into further, according to Tom Tasky. (FEV)

Tasky asserts that his group has rapidly developed, within a short period, a high level of maturity and quality that enables them to benefit a diverse customer base. “It’s a specialty area of engineering,” he said. “We keep adding people. But you can’t just take an engine-test or calibration engineer and throw them into AVs. It’s a different mindset for creating test cases and scenarios – rounding a curve without lane markers with a truck coming in the opposite direction, for example. Developing a scenario where you can encounter thousands and in some cases millions of events, and testing for repeatability within that scenario. Even then, however, you must factor in all the potential weather conditions. And manage the mountain of data as the testing proceeds.”

Compare AV testing to traditional development of an ECU, for example, that includes a hardware-in-the loop (HIL) setup and a degree of road testing. With ADAS and AV systems, the level of systems complexity, the number of potential driving scenarios, and the scope of the working environment and weather conditions requires a greater range of frameworks for simulation, track testing, data analysis and validation. “It’s an exponential transition,” Tasky said. “The cost and time needed are enormous. And you can only come up with so many test cases, because you’ve only got so much time. So we focus on the ‘edge cases’—the unique areas that are so hard to discover.”

One key area of focus for Tasky’s FEV team is helping the customer in SOTIF (Safety of the Intended Functionality) covering all five SAE Levels of Driving Automation. A phalanx of challenges facing them includes determining how ADAS sensors operate with degraded functionality, or in extreme environmental conditions.

“The only way to tackle it is to break it down into manageable chunks and tasks, and into simple pieces. We have teams looking at the edge-case scenarios on the sensors level and on the systems level. We evaluate each sensor individually then bring them together at the system level and do more validation.” He said this is where more simulation frameworks using “unique tools and unique approaches to their use” are required for systems validation.

Despite the increased fidelity of the latest “sim” tools, FEV believes up-front simulation and physical road testing are equally needed at this stage of the automated-vehicle era. “That’s because of the complexity of the AV safety challenge,” Tasky noted. “You can’t do it all in the physical world, and you can’t do it all in simulation. People who have tested vehicles for years know that it also takes years to develop a robust simulation platform that can cover it all. If indeed that is possible.”

He said FEV North America is increasingly leveraging its recently commissioned Vehicle Development Center (VDC) – where it conducts emissions and EV range testing – for AV testing. “We have some innovative ideas on how to optimize a Vehicle in the Loop, on a dyno setup, to test the ADAS while performing other tests,” Tasky said.

AV test standards lacking

While powertrain testing and validation are performed within a robust set of established standards, the same is not yet true for AV testing. “The lack of standards in this area is, overall, a big problem,” he stated. “I think there is a basis for some type of baseline for Level 2 and Level 3 features, following the example of NCAP testing, for example. Although it’s nice to have a bit of freedom without them right now—and we need a balance—it’s a bit like the Wild West. You may have to self-certify. A lot of companies that are new to this tend to struggle,” he said.

Tasky noted that weather conditions and combinational events are a significant, and unique, factor in ADAS and AV testing. “And just because you tested to and passed a standard, does not mean that companies should be totally comfortable. That’s where we step in.” The jump from SAE Level 2 to Level 4, with the still-controversial Level 3 between them, is “tricky” from a testing perspective, Tasky affirmed. He said the technology choice is up to the OEMs, but driver alertness and reaction time remain critical for L3. FEV has developed its own Driver Monitoring System to help engineers understand operational capabilities.

“It’s another point of functional safety. You need to understand reaction times, controllability – the expectations of which are different between passenger cars and commercial vehicles with trained drivers,” he explained. Driver reaction times and human performance is an area FEV is studying and will likely expand into further. Tasky noted that engineers are not ideal for the studies because they understand system design and the concept of reaction times. “So, you need average people for these studies,” he noted.

FEV has an AV test facility at its Auburn Hills, Michigan location, where it conducts sensor component and system tests and benchmarking. “Our setup is simple and repeatable for what we’re trying to capture,” Tasky said. “Our test vehicles are configured to be flexible platforms, easily swapping out sensors and equipment. We can keep the data confidential, or we can sanitize it in a scatterband so customers can see where their product fits in a plotted curve of other data. Some companies want to promote their data, so we’ll put their name on a slide to show where they fit in. We’ll also pull in experts on optics and other specialty areas from within our company to work with the sensor makers to improve their technologies.”

He cited the example of one (San Francisco) Bay Area sensor maker that needed guidance on functional safety before their review meeting with an OEM. “We were very familiar with the OEM and their expectations, so we helped the customer get their hardware and software ready for that meeting,” he recalled. That included providing a guidance path for integrating their sensor into a vehicle to meet the ASIL (Automotive Safety Integrity Level) for a certain ADAS feature.

Bolstering its AV testing competency means FEV is constantly seeking to add top-quality talent. Combining new engineering hires, including those versed in domain-control architectures and cloud computing, with its experienced people “is the structure we’ve created at FEV,” Tasky explained. Mechatronics and data science are important degrees for young engineers, he said, as is software. And “a systems engineering mindset is critical because we just don’t have ‘the chassis person’ or ‘the battery person’ any more in vehicle development. They have to work together. “Complexity in AV testing is not going to slow down; it’s going to accelerate,” he asserted.