D-tote: a data acquisition platform that solves CASE-era problems

― I have heard that the D-tote platform developed by DENSO is being used to optimize the acquisition of proof of concept (PoC) data, which helps to solve in-house problems.

Takuya Kondo：D-tote has indeed helped us to solve problems in-house, but that is not what it was originally designed for. Let me explain the role of our division, the Cloud Services R&D Div., within DENSO prior to D-tote’s creation, as this will make everything else more clear.

DENSO is now pursuing a “software first” policy corporation-wide. Although our value originally came from providing physical products—namely products for cars—to our customers, society today demands that we also provide less tangible things, such as systems as services.

Software is at the core of these, and we are using software as the foundation of our further growth.
To head up these efforts, the Digital Innovation Dept. was founded in 2017. After some organization changes, it is now the Cloud Services R&D Div., which consists of the Digital Innovation Dept. for software development and the Business Innovation Dept. for business models development.

In addition to solving in-house problems, this new division is tasked with identifying needs both inside and outside of DENSO, creating software in-house in collaboration with customers, and then offering the resulting products to wider society.

― What needs and demands was the D-tote development project based on?

Yusuke Takagi：Many people wanted to keep using their own data acquisition systems once they had created them.

Kondo：The Thermal Business Group, for example, was working to create a system which used sensors to acquire information on vehicle-cabin air quality, automatically upload sensor data to cloud storage, and then make use of that information. To determine the specifications of this system, they needed to carry out PoC testing.

One problem we recognized at that time was the haphazard use of multiple, similar-looking data acquisition systems throughout DENSO. Hardware-development engineers in particular, who create sensors and similar products, often find it hard to work with software- and cloud-related technologies, and thus they have no choice but to outsource the development of data acquisition systems.

Against this background, conventional data acquisition systems were designed specifically for individual projects, and they could not be used for other types of project. Therefore, D-tote was designed as a general-purpose, common-use system that is compatible with various in-house projects.

― How did you decide on the specifications of D-tote?

Y. Takagi：At first we were going to create the data acquisition system to be suitable only for in-house PoC operations, but afterwards we tried to create the system to meet both internal and external needs so that it could be used to develop new business for DENSO.

So, we proceeded by trial and error and ultimately settled upon several system requirements: users can start the measurements easily; users also can select and customize which sensors to use for data acquisition based on their specific needs; and it must be possible to adjust data acquisition settings remotely for multiple devices simultaneously.

To meet these requirements, the system needs to acquire data from sensors in actual vehicles and process the data on a cloud network at the same time. Minami and I had developed cloud services, but never developed Internet of Things (IoT) services, so we were exploring ways of developing the system.

Minami：After trying various approaches, we could somehow upload data from an in-vehicle device to cloud storage, and create a functional prototype.

Y. Takagi：Once we achieved this prototype, we were able to set specific goals based on customer needs. At this point, we were finally able to move forward with agile development.

― Considering convenience for acquiring data, I think that many customers expected you to develop a smartphone app for this system.

Y. Takagi：That’s right. One of our customers asked us to release the app soon, but we were too busy to do so. But after we discussed the issue with the team many times, we realized that what we needed was to create a system to conduct questionnaires with test drivers.

Therefore, we didn’t have to develop an app; rather, we only needed to use Google Forms to create the questionnaire. So we did that quickly, and the customer was satisfied with it. If we had instead outsourced the app creation, it would have cost us a lot of time and money.

Customers don’t necessarily want something fancy that costs a lot of money to make. As long as they can get the value they’re seeking, then something simple (like the questionnaire this time) may satisfy them. That’s what we learned through our experience.

― How is the system structure of D-tote different from that of a conventional data acquisition platform?

Y. Takagi：First, in the conventional data acquisition system, in-vehicle devices acquire data through their sensors and upload them to the cloud. Message brokers of the data storage function in the cloud receive and save the data, and the application layer for data analysis acquires the data from the data storage function and utilizes it. This is the typical process of the conventional system, and customers develop systems for each PoC to acquire data.

We integrated common functions required for all types of PoC operations into D-tote, and PoC managers develop other functions depending on the project, which include data acquisition from sensors and application layers.

In-vehicle devices are operated by Linux-based computers, and the Docker container organizes the D-tote functions such as conversion of data acquired by sensors and uploading of data to the cloud.

*3 The Docker container is the package of software resources (relevant libraries, middleware and so on) necessary to execute applications. It enables them to run without complex settings even in an environment different from that of development.

D-tote’s cloud is located on Amazon Web Services (AWS). D-tote focuses on the data acquisition function, and thus does not have application layer functions such as data analysis. Data is analyzed with the application layer functions owned by users.

― The diagram shows that D-tote converts data acquired from sensors. What format is the data converted into?

Y. Takagi：Sensor data is saved as text data in JSON format.*4 The fields within the JSON file use a simple format comprising the data-acquisition timestamp and the data value. Each assigned filename is made up of the sensor’s signal code and data timestamp, which makes it easy to identify each unique signal at a glance based on the filename. All sensor data is compressed and saved using the same JSON format, so it does not take up much storage space and can be transmitted at low cost during uploads.

*4 JavaScript Object Notation. Originally it is a data notation format used in the JavaScript programming language. It is a simple text-based format that separates keys and values using commas, and is used in various programming languages today.

From a data-handling perspective, D-tote is organized like a pipeline: any data that is acquired with a sensor can be uploaded to the cloud and then processed with the application layer. The system is compatible with sensors for Controller Area Networks (CANs), temperature and humidity, GPS and others, and can be modified to work with additional sensor types as needed.

For JSON data management, we chose to use Apache Hive,*5 which facilitates extraction of required data using the AWS Athena service.

*5 One approach to organizing data files by folder. Hive creates folders based on key and value data in JSON files for the purpose of data-file management.

― Even with this approach, wasn’t it challenging for engineers who specialize in cloud services to work with IoT hardware?

Kondo：I joined the Cloud Services R&D Div. later than other members, after it had already started. Looking back and reading documents on early D-tote development revealed much trial and error. I came across descriptions such as, “We purchased equipment A and wired it to equipment B. The signal went through fine, and the data was successfully received.” In other words, they tackled each issue one by one.

Y. Takagi：Embedded software development and cloud service development have very different cultures although both of them are software development, and I was very confused by the difference. During development, I should have been able to receive specific signals when I sent them to the sensor under development, but it didn’t work no matter how hard I tried. In the end, the failure was due to the wrong version of sensor.

― What is the general process flow of using the D-tote system?

Kondo：If users have necessary in-vehicle devices and sensors for data measurement, they can start using cloud-based services in just one day at earliest.

― It is impressive that D-tote has been developed by engineers with a wide array of backgrounds.

Y. Takagi：I used to develop cloud services at an Internet-based services company before joining DENSO.

Minami：I was in charge of developing gasoline engine systems. Although I sometimes wrote code, I led the entire development of the systems, in which I decided which parts should be mounted in the engines.

Shingo Takagi：Until a year ago, I was involved in mechanical design for automotive air-conditioning systems, but I used the career-change program to start my new career as a software engineer.

The Cloud Services R&D Div. uses various agile development approaches; personally, I feel that the “extreme programming” approach is particularly effective. In this approach, some members make one group, and do pair programming on one computer screen. Thanks to this working style and the advice and support by more experienced coworkers of mine, I was able to adjust to the team’s way of doing things and fit in.

The current D-tote development project involves everything from communicating with hardware to cloud-based processing, and by building up knowledge in both the hardware and software fields I want to be able to propose a wider range of products and solutions to customers.

Kondo：DENSO is proactively training new software engineers through a career-change program entailing recurrent education to help people make the switch from hardware to software engineer, and is recruiting associates of DENSO through in-house calls for new software engineers to create opportunities for anyone to join software-related projects. Of course, we proactively recruit persons with the skills we require from outside DENSO, too.

The Cloud Services R&D Div. has many diverse members with many useful skills.

― What is the future of development of D-tote?

Kondo：D-tote can be used to develop factory facilities and equipment systems as well as vehicle-related products and services.

We expect DENSO Group companies to use D-tote for developing products and services, so we would like to meet the needs of external users as well.

Furthermore, DENSO is currently working on “D-toteCar”, which makes it easy to acquire data for various projects using company cars. Mobility-field projects require driving tests, but they often suffer from a lack of manpower for acquiring a lot of data. We will have DENSO employees use company cars equipped with the necessary sensors and in-vehicle devices for their business trips, and thus acquire the required data on behalf of the project team.

IT companies internally use the phrase “eating our own dog food,” which means using their own software in their organizations in order to improve their services.

DENSO has focused on users’ needs and improving its products in an agile manner. D-tote embodies this software-first approach of DENSO.