Toyota’s Advanced R&D Puts Humans First

April 1, 2023

Lindsay Brooke

Gill Pratt has a gift for explaining complex topics in simple terms. And as Toyota Motor Co.’s chief scientist and CEO of the Toyota Research Institute (TRI), he also speaks frankly about the promises and potential pitfalls of new technologies. Addressing a rare group of visitors — tech reporters and analysts, including SAE Media — recently at TRI’s Silicon Valley headquarters, Pratt noted the heightened public discourse around artificial intelligence, a core area of focus for many of TRI’s 200 scientists and engineers.

“Everybody is worried ChatGPT is going to be writing term papers for college students,” Pratt said half-jokingly about the controversial “chat bot” introduced in late 2022 by OpenAI. “But even our humor reflects the anxiety we have about this technology and its dual nature of good and evil.” Society, he observed, has become “much less naive about technology, and much more skeptical.” Much of that comes from the AI industry’s early hype about self-driving vehicles, which TRI did not share.

“The first CES talk we did seven or eight years ago was different than that of most companies,” Pratt said. “We spent most of our time explaining why this problem [autonomous driving] is really, really hard—and why we didn’t know how long it was going to take until we had [SAE] Level 4 automated cars in substantial operational design domains. And why no one knew how to do Level 5.”

It’s a measured and honest stance by a research organization founded in 2015 on an initial $1 billion investment by Toyota. TRI’s annual budget comprises “a very small part” of the roughly $10 billion Toyota spends yearly on R&D, Pratt said. Its research pillars—machine learning, robotics, energy and materials research, and human-centered AI—aim to ultimately help solve the challenges of climate change, aging society, and human understanding. And in doing so, to fundamentally create a pathway to individual happiness and societal harmony. That’s not as lofty a goal as it sounds, asserted the directors of TRI’s key divisions during the open house event. They believe it’s rooted in the jidoka principle—“automation with a human touch”—Toyota has followed since its founding as a maker of automatic looms in 1926.

Technology should amplify the work of human beings, rather than displace them, thus “upholding the dignity of human work,” Pratt said. “We believe jidoka can be applied in many ways.” TRI’s role as an enabler, then, is to do “the highest-risk, highest-reward research,” he explained.

“Highest risk means we actually expect to fail a good fraction of the time. While we don’t want to fail, I know if our projects aren’t failing around a third of the time, then we’re going after projects that are too easy. They’re not taking the high risk to get the high reward.”

Outgoing Toyota CEO Akio Toyoda set Pratt’s job expectation as simply, “Surprise me.” It continues under incoming CEO Koji Sato. “That’s our job—to make that leap; take those risks. It’s okay if you fail a fraction of the time, but some of the time you should succeed.” In doing so TRI will help keep Toyota at the frontier of innovation and leadership in IP generation.

Humans and AI in the loop

Within the vehicle-development community, AI is increasingly being accepted as integral to vehicle and systems development, by helping humans to solve complex engineering problems. The benefits are reduced test and validation time, faster time to market and lower cost. AI also is a vital layer in machine learning, as well as in the design of the human-machine interface (HMI) in vehicles and its functionality.

“The history of AI is the history of hype,” Adrien Galdon, director of Machine Learning (ML) at TRI and an adjunct professor of computer science at Stanford University, asserted. “What I’m hopeful for is, we’re seeing tools on the Web that are becoming more and more useful, particularly if they have human oversights and have humans and machines working in tandem.”

Galdon stressed the importance of human amplification in AI’s development and applications. “You’ve seen how far it is to go from the web to the physical world for AI, because data is so much more expensive, slow, partial, and dangerous to collect. There are always going to be more blind spots in your data. I think AGI [artificial general intelligence] is the same because it assumes infinite compute, infinite energy, infinite money, while the real world is much richer, data-wise, than the web.”

He explains Foundation models, a new generation of ML models that are trained on as wide and general a dataset as possible. Like the foundation of a house, you can build any type of house on it, he said. And through in-context learning, the system becomes personalized to you through repeated interaction. It’s part of a transition in AI where systems that execute specific tasks in a single domain are yielding to broad AI, with more general learning, that works across domains and problems.

“Train one big model once to capture, for example, common sense, and program it to adapt to you through, say, natural language,” offered Max Bajracharya, senior VP of Robotics at TRI. He said machine learning is both about generalization—the collective ‘we’—and adapting to the user. His 50-person robotics team is focused on the challenge of helping aging society—by 2050 half of Japan’s population will be at least 65 years old, he said. The diversity of robotic applications and use cases is “a huge challenge,” Bajracharya said.

Building off of what Galdon said about having humans in the loop, Charlene Wu, TRI’s director of Human-Centered AI, affirmed that how the technology is used is as important as the tech itself. She shared an example from human-centered AI and her team’s future product innovation work.

“We’ve achieved a nice human-machine interchange by leveraging some of the recent generative-AI tools,” Wu explained. “A product designer might have a concept in their mind, something they can visualize. They can use the AI tools and input a prompt that describes what they might like in the concept. There are two ways in which the interaction with AI can be really productive. First, it allows designers to explore all kinds of visualizations of just a single input. They can explore that space and maybe even tap into some ideas that they had yet to encounter.”

Once the designer has seen a concept or idea that they want to pursue further, they can then use AI to help refine and fine-tune the initial concept. “This is really powerful,” Wu noted, “because it allows people to be much more creative and also more efficient. That’s because you can access so many more manifestations of a particular idea in such a short amount of time. This is how we’re choosing productive ways to use the technology in ways in which humans and AI can interact more.“

Amplifying the driver

“In human interactive driving, we’re trying to create a sense of great experience for people when they’re driving—and have the car be able to amplify that,” Avinash Balachandran, director of TRI’s Human Interactive Driving (HID) division, explained during a visit to TRI’s engineering garage. Circled around him are three development platforms: a Lexus LC loaded with data-acquisition gear for gathering driver inputs and real-world road data; an open tube-framed, electric mule vehicle with wheel-hub motors, by-wire controls and four-wheel steering; and a race-kitted Supra set up for autonomous drifting and data acquisition. All three platforms were built by Toyota Racing Developments (TRD). A few floors above him is a multi-axis driving simulator and control room, co-developed with TRD.

Balachandran frames his team’s mission: “Imagine yourself skiing or riding a bike or any enjoyable activity. It’s the liking and happiness you derive from it that make you get better and better at it. It’s what we’re trying to engender in our relationship between the AI amplifying the person as well.”

Before joining TRI, Balachandran was an engineer on Uber’s early self-driving vehicle program and was part of the team that developed Uber’s first (2016) autonomous service in Pittsburgh. His division works at the human-vehicle nexus, which begins with understanding human behavior. The team’s work in the near-term, he explained, is to make ADAS in the SAE L2/L3 space “seamlessly reassuring and intuitive” without noticeable tech intrusions. It also adheres to an Akio Toyoda fundamental: Never forget that driving should be fun.

“We’ve learned that if you want to build a safety system or a vehicle that has the ability to train you to have mastery in the skills of driving—to really have that enjoyment—then fun-to-drive is super important,” he noted. “That’s where the happiness comes in.” While Toyota is developing AV solutions (TRI co-led development of the automaker’s Guardian and Chauffeur systems), the current focus is on advanced driver assistance technologies, Balachandran said.

TRI co-founder Pratt sees technology that can adapt to each person as one of the ultimate deliverables of his organization’s work. He added that it’s important for technologists to remain humble. “We shouldn’t have the hubris to think that we have all the answers—that we should have the power to decide for society, on behalf of society, just because we’re good at technological stuff. That’s just deeply, deeply wrong.”

---