Reproducing the Driver’s Senses by Engineering—DENSO’s Challenge to Develop Automated Driving Systems

Hiroshi Ino: Today, I would like to concisely explain our technology development efforts based on the theme of “Automated Driving System That Understands Human Characteristics—Technology Development Based on Human Characteristics.”

First, let me briefly introduce our company. This is the Long-term Policy 2030.

Over the next decade, DENSO aims to continuously generate value to enrich mobility that achieves sustainability, happiness and peace of mind for everyone.

Green technologies and technologies that give peace of mind play a key role. DENSO will focus on these technologies to open up the future.

This slide shows the extensive lineup of products offered by DENSO.

DENSO operates five major businesses: thermal systems, mobility systems, powertrain systems, electrification systems, and electronic systems. In fact, our components are incorporated in almost all vehicles on the road.

We call this “DENSO Inside.” Many of our products are used in your cars without you realizing. This is an overview of our business.

These are our conventional products. DENSO will focus on four main fields to shape its future:
electrification, connected cars, advanced safety and automated driving, and non-automotive business.

I will simply explain our efforts in advanced safety and automated driving.

Let me show you a video that introduces our development initiatives.

Although we are a component manufacturer, we develop automated driving systems and conduct field tests on public roads. This test was conducted in Hokkaido. We have gained extensive knowledge in the system development process. We use appropriate components and sensors, which we produce, to make right and left turns at traffic signals.

DENSO Lv4 ADV FOT in Japan, Tokyo & Hokkaido

Such a system requires large-scale integration of many sensor groups. We conduct the development process by changing the product combinations.

This shows a right turn at an intersection. The vehicle allows oncoming vehicles to pass before making a right turn. This is a very difficult situation to cope with.

We also ensure the sensing performance to move a vehicle at an intersection without traffic signals. Sensors must be fully utilized despite their limitations in terms of field of view (FOV) and sensing range. Our sensing technologies play a crucial role.

It is also necessary to achieve automated driving in more crowded urban areas. We are working to ensure recognition of sidewalks and pedestrians crossing a street.

Another challenge is cooperative automated driving between vehicle and infrastructure. In addition to using simple in-vehicle sensors, the technology uses communication devices to enable cooperation with infrastructure. These are only part of our work. As shown in the video, the technology automates most of the driving.

It should be noted that automated driving technology covers many fields, from recognition, to forecasting, planning, vehicle dynamics control, and actuator operation. Our challenge is to integrate a large number of components.

I will go into this topic in some detail.

To attain marketability, we must achieve comfortable automated driving that gives peace of mind. The question is how to attain that goal.

The details, including recognition, will be discussed by General Manager Naoki Ito, who is responsible for research on AI. I will only give an overview.

The point is what you sense during driving. We must clarify the specific senses and reproduce them by engineering in order to improve the system by integrating components of our five major businesses. This will be an important challenge in developing the next-generation system.

In addition to simply using machine learning technology or conventional optimization methods, we try to understand human characteristics.

(While pointing to the video screen) This footage was recorded to show the driving behavior. During driving, we do not directly perceive objects such as pedestrian bridges. Instead, we recognize the driving scenery.

Visual Sensation/Optical Flow

In the field of computer vision, this is called “optical flow,” which is also discussed in cognitive psychology. The driver recognizes the approximate vehicle speed based on the optical flow. Rotation can be sensed based on deformation or distortion in the optical flow. A poor driver is not good at controlling the optical flow.

Efforts have been made for many years to use the vision information sensed by humans in automated driving systems.

This slide shows an optical flow experiment by the U.S. Air Force. In the 1940s, many Air Force pilots crashed due to lack of kinesthesia (e.g., sense of distance) during landing.

To solve the problem, the U.S. Air Force analyzed human characteristics to identify what the pilots sensed and to determine characteristics that could be factors for improving maneuverability.

Research in aviation is far ahead of that in the auto industry, and part of such research has been spun off for automotive applications. For example, research on visual motion perception has been found to be highly effective in preventing car sickness.

Another interesting aspect of our research is the contact time. Adaptive cruise control (ACC) is a common feature of advanced driver assistance systems (ADAS). ACC controls the vehicle speed depending on the distance from the preceding vehicle. We focus on the internal contact time, which is equivalent to the “time to contact” in biopsychology and cognitive psychology.

We visually perceive an oncoming object based on two factors. One is the size of the image of the object on the retina, and the other is the way the image spreads on the retina. We then estimate how many seconds it will take until the object hits us.

In other words, the numerical value that we use when observing an oncoming object is the internal contact time instead of distance or speed. The characteristics of the eyes are also utilized in developing ACC for ADAS.

Motion perception also involves the sense of gravity. Those who like cars may be familiar with the friction circle, which represents a tire’s grip.

The friction circle is shaped like a ball. Deviation from the friction circle results in drift-out or spin-out.

The concept is to ensure control within the capability of a tire. The controlled sense of gravity that is comfortable for the occupants is shown in the rightmost figure. Two models are presented.

The previous model is designed to avoid exceeding the limitation of vehicle dynamics. Namely, when the status is at the end of this ball, it is forcibly returned to the range within the ball. Meanwhile, in the so-called “ball-in-ball” concept, control is ensured efficiently within the limitation of vehicle dynamics. G-Bowl showed that the trajectory of the virtual G ball is controlled in a somewhat circular shape within the vehicle’s limitation. Research has gradually revealed that such control is very human-like.

Automakers and suppliers, including DENSO, have developed various systems by analyzing these characteristics from the viewpoint of automotive engineering, defining the sense of gravity, and reproducing the human senses by engineering.

Such systems can be applied to mountainous roads with many curves. This is our test course, which is very difficult to drive on. Our system can ensure control on this course within the vehicle’s limitation and thus give peace of mind to the occupants.

Result of automated driving on mountain rode

The system was also used to drive on dry roads and snow-covered roads. This shows the results.

Result of automated driving on mountain rode on different pavement condition

Automated driving on snow-covered roads would be particularly convenient, but such roads are very difficult to cope with due to slip. Stable control can be attained by properly reproducing human characteristics.

An understanding of human characteristics helps to develop technologies that ensure safety, comfort, and peace of mind. This shows quantitative results. Even if the tires slip, the vehicle can be controlled properly by countersteering.

We are working on such technology development initiatives even though DENSO sells sensor components, not automated driving systems.

To improve the sensing technology, it is necessary to determine the marginal performance of automated driving or to understand how automated driving gives peace of mind in a driver-friendly manner, as you saw in the video. Namely, we must understand how OEMs, service companies, and enablers want to move the vehicles. Without such understanding, we cannot improve sensor technologies.

Our voluntary R&D efforts and field tests will help develop future mobility systems that ensure safety and peace of mind, as I introduced at the outset.