Feb 10, 2023

D-tote: a data acquisition platform that solves CASE-era problems

Supporting the Hassle of Data Collection, the Future Opened Up by Data Utilization and DENSO's New Challenge

“D-tote” is a vehicle-data acquisition platform developed by DENSO Corporation’s Cloud Services R&D Div. The auto industry is undergoing a once-in-a-century transformation, as seen in the CASE (connected, autonomous, shared and services, and electric) paradigm shift and other such changes, and DENSO is rising to the new challenges presented by this transformation. DENSO’s efforts in this area have been described in previous articles,*1,2 one of which (see below) described a recurrent training program which helps software engineers develop their career paths. As measures gradually spread and take hold throughout DENSO, teams of engineers with diverse backgrounds have come together to deliver specific results and create new products. D-tote is one of such examples.

*1 Software innovation by DENSO in the era of CASE

*2The story behind auto component manufacturer DENSO’s pursuit of software innovation, and a look at the engineers who support it

  • Cloud Services R&D Div.TAKUYA KONDO

    Takuya Kondo joined DENSO in 2009 where he worked on developing a new refrigerant system. With the goal of helping to develop new business for DENSO, he joined the software recurrent training program, and subsequently requested a transfer to the Cloud Services R&D Div. After working on D-tote development for a year, he switched to his current position as product owner.

  • Cloud Services R&D Div.Yusuke Takagi

    After accumulating experience in software services at a systems integrator, and then at an Internet-based company, Yusuke Takagi entered the auto industry with the goal of contributing to transforming society. Working on agile development in DENSO, a manufacturing company, he strives to promote software-first approaches and data utilization in order to switch DENSO from being a company that makes tangible products to one that offers intangible ones such as systems and services.

  • Cloud Services R&D Div.KEITARO MINAMI

    Keitaro Minami joined DENSO in 2009, starting out in system development for internal combustion engines. He was reassigned to the Digital Innovation Dept. in 2019, where he worked on vehicle telematics services before moving on to D-tote development. Inspired by the shift from tangible to intangible products, he decided to change career direction. He is currently using his experience in engine and vehicle evaluations while learning skills that enable him to combine these with IoT technologies to play a more active role in the IoT field.

  • Cloud Services R&D Div.SHINGO TAKAGI

    Upon joining DENSO in 2018, Shingo Takagi started out in design and development for automotive air-conditioner compressors. Impressed by his experience in 2020 in a future technologies planning project, he pursued a new career as a software engineer with the aim of working on development from the planning stage. After completing the software recurrent education program in 2021, Takagi transferred to the Cloud Services R&D Div. where he engages in D-tote development while striving to disseminate data-driven development more widely throughout DENSO and society.

Contents of this article

    Determining which services DENSO’s customers really need

    ― I have heard that the D-tote platform developed by DENSO is being used to optimize the acquisition of proof of concept (PoC) data, which helps to solve in-house problems.

    Takuya Kondo:D-tote has indeed helped us to solve problems in-house, but that is not what it was originally designed for. Let me explain the role of our division, the Cloud Services R&D Div., within DENSO prior to D-tote’s creation, as this will make everything else more clear.

    DENSO is now pursuing a “software first” policy corporation-wide. Although our value originally came from providing physical products—namely products for cars—to our customers, society today demands that we also provide less tangible things, such as systems as services.

    Software is at the core of these, and we are using software as the foundation of our further growth.
    To head up these efforts, the Digital Innovation Dept. was founded in 2017. After some organization changes, it is now the Cloud Services R&D Div., which consists of the Digital Innovation Dept. for software development and the Business Innovation Dept. for business models development.

    In addition to solving in-house problems, this new division is tasked with identifying needs both inside and outside of DENSO, creating software in-house in collaboration with customers, and then offering the resulting products to wider society.

    ― What needs and demands was the D-tote development project based on?

    Yusuke Takagi:Many people wanted to keep using their own data acquisition systems once they had created them.

    Kondo:The Thermal Business Group, for example, was working to create a system which used sensors to acquire information on vehicle-cabin air quality, automatically upload sensor data to cloud storage, and then make use of that information. To determine the specifications of this system, they needed to carry out PoC testing.

    One problem we recognized at that time was the haphazard use of multiple, similar-looking data acquisition systems throughout DENSO. Hardware-development engineers in particular, who create sensors and similar products, often find it hard to work with software- and cloud-related technologies, and thus they have no choice but to outsource the development of data acquisition systems.

    Against this background, conventional data acquisition systems were designed specifically for individual projects, and they could not be used for other types of project. Therefore, D-tote was designed as a general-purpose, common-use system that is compatible with various in-house projects.

    ― How did you decide on the specifications of D-tote?

    Y. Takagi:At first we were going to create the data acquisition system to be suitable only for in-house PoC operations, but afterwards we tried to create the system to meet both internal and external needs so that it could be used to develop new business for DENSO.

    So, we proceeded by trial and error and ultimately settled upon several system requirements: users can start the measurements easily; users also can select and customize which sensors to use for data acquisition based on their specific needs; and it must be possible to adjust data acquisition settings remotely for multiple devices simultaneously.

    To meet these requirements, the system needs to acquire data from sensors in actual vehicles and process the data on a cloud network at the same time. Minami and I had developed cloud services, but never developed Internet of Things (IoT) services, so we were exploring ways of developing the system.

    Minami:After trying various approaches, we could somehow upload data from an in-vehicle device to cloud storage, and create a functional prototype.

    Y. Takagi:Once we achieved this prototype, we were able to set specific goals based on customer needs. At this point, we were finally able to move forward with agile development.

    ― Considering convenience for acquiring data, I think that many customers expected you to develop a smartphone app for this system.

    Y. Takagi:That’s right. One of our customers asked us to release the app soon, but we were too busy to do so. But after we discussed the issue with the team many times, we realized that what we needed was to create a system to conduct questionnaires with test drivers.

    Therefore, we didn’t have to develop an app; rather, we only needed to use Google Forms to create the questionnaire. So we did that quickly, and the customer was satisfied with it. If we had instead outsourced the app creation, it would have cost us a lot of time and money.

    Customers don’t necessarily want something fancy that costs a lot of money to make. As long as they can get the value they’re seeking, then something simple (like the questionnaire this time) may satisfy them. That’s what we learned through our experience.

    Building a data acquisition platform through trial and error

    ― How is the system structure of D-tote different from that of a conventional data acquisition platform?

    Y. Takagi:First, in the conventional data acquisition system, in-vehicle devices acquire data through their sensors and upload them to the cloud. Message brokers of the data storage function in the cloud receive and save the data, and the application layer for data analysis acquires the data from the data storage function and utilizes it. This is the typical process of the conventional system, and customers develop systems for each PoC to acquire data.

    Conventional data acquisition systems

    We integrated common functions required for all types of PoC operations into D-tote, and PoC managers develop other functions depending on the project, which include data acquisition from sensors and application layers.

    In-vehicle devices are operated by Linux-based computers, and the Docker container organizes the D-tote functions such as conversion of data acquired by sensors and uploading of data to the cloud.

    *3 The Docker container is the package of software resources (relevant libraries, middleware and so on) necessary to execute applications. It enables them to run without complex settings even in an environment different from that of development.

    D-tote’s cloud is located on Amazon Web Services (AWS). D-tote focuses on the data acquisition function, and thus does not have application layer functions such as data analysis. Data is analyzed with the application layer functions owned by users.

    An overview of the D-tote data acquisition system developed by DENSO. Common sections used for all project types are shown in pink and were developed as part of D-tote; the blue sections must be prepared by each project operator.

    ― The diagram shows that D-tote converts data acquired from sensors. What format is the data converted into?

    Y. Takagi:Sensor data is saved as text data in JSON format.*4 The fields within the JSON file use a simple format comprising the data-acquisition timestamp and the data value. Each assigned filename is made up of the sensor’s signal code and data timestamp, which makes it easy to identify each unique signal at a glance based on the filename. All sensor data is compressed and saved using the same JSON format, so it does not take up much storage space and can be transmitted at low cost during uploads.

    *4 JavaScript Object Notation. Originally it is a data notation format used in the JavaScript programming language. It is a simple text-based format that separates keys and values using commas, and is used in various programming languages today.

    From a data-handling perspective, D-tote is organized like a pipeline: any data that is acquired with a sensor can be uploaded to the cloud and then processed with the application layer. The system is compatible with sensors for Controller Area Networks (CANs), temperature and humidity, GPS and others, and can be modified to work with additional sensor types as needed.

    For JSON data management, we chose to use Apache Hive,*5 which facilitates extraction of required data using the AWS Athena service.

    *5 One approach to organizing data files by folder. Hive creates folders based on key and value data in JSON files for the purpose of data-file management.

    ― Even with this approach, wasn’t it challenging for engineers who specialize in cloud services to work with IoT hardware?

    Kondo:I joined the Cloud Services R&D Div. later than other members, after it had already started. Looking back and reading documents on early D-tote development revealed much trial and error. I came across descriptions such as, “We purchased equipment A and wired it to equipment B. The signal went through fine, and the data was successfully received.” In other words, they tackled each issue one by one.

    Y. Takagi:Embedded software development and cloud service development have very different cultures although both of them are software development, and I was very confused by the difference. During development, I should have been able to receive specific signals when I sent them to the sensor under development, but it didn’t work no matter how hard I tried. In the end, the failure was due to the wrong version of sensor.

    The power of in-house software created by diverse members

    ― What is the general process flow of using the D-tote system?

    Kondo:If users have necessary in-vehicle devices and sensors for data measurement, they can start using cloud-based services in just one day at earliest.

    ― It is impressive that D-tote has been developed by engineers with a wide array of backgrounds.

    Y. Takagi:I used to develop cloud services at an Internet-based services company before joining DENSO.

    Minami:I was in charge of developing gasoline engine systems. Although I sometimes wrote code, I led the entire development of the systems, in which I decided which parts should be mounted in the engines.

    Shingo Takagi:Until a year ago, I was involved in mechanical design for automotive air-conditioning systems, but I used the career-change program to start my new career as a software engineer.

    The Cloud Services R&D Div. uses various agile development approaches; personally, I feel that the “extreme programming” approach is particularly effective. In this approach, some members make one group, and do pair programming on one computer screen. Thanks to this working style and the advice and support by more experienced coworkers of mine, I was able to adjust to the team’s way of doing things and fit in.

    The current D-tote development project involves everything from communicating with hardware to cloud-based processing, and by building up knowledge in both the hardware and software fields I want to be able to propose a wider range of products and solutions to customers.

    Kondo:DENSO is proactively training new software engineers through a career-change program entailing recurrent education to help people make the switch from hardware to software engineer, and is recruiting associates of DENSO through in-house calls for new software engineers to create opportunities for anyone to join software-related projects. Of course, we proactively recruit persons with the skills we require from outside DENSO, too.

    The Cloud Services R&D Div. has many diverse members with many useful skills.

    ― What is the future of development of D-tote?

    Kondo:D-tote can be used to develop factory facilities and equipment systems as well as vehicle-related products and services.

    We expect DENSO Group companies to use D-tote for developing products and services, so we would like to meet the needs of external users as well.

    Furthermore, DENSO is currently working on “D-toteCar”, which makes it easy to acquire data for various projects using company cars. Mobility-field projects require driving tests, but they often suffer from a lack of manpower for acquiring a lot of data. We will have DENSO employees use company cars equipped with the necessary sensors and in-vehicle devices for their business trips, and thus acquire the required data on behalf of the project team.

    IT companies internally use the phrase “eating our own dog food,” which means using their own software in their organizations in order to improve their services.

    DENSO has focused on users’ needs and improving its products in an agile manner. D-tote embodies this software-first approach of DENSO.

    Send your reaction

    • Like

    • Exciting

    • Wow

    Did you find this article helpful for what you want to achieve, learn, or to expand your possibilities? Share your feelings with our editorial team.