Anguished Responses from Farmers: “There is no one to take over my farm.” “I’m too old to continue farming.” FARO, an automated tomato harvesting robot, uses technology to solve social issues .
DENSO Tech Links Tokyo #15 was held on the theme of DENSO’s vision for “manufacturing plants” for agriculture. It introduced “Profarm T-cube,” a next-generation agricultural greenhouse which was developed to make agriculture accessible to anyone, anywhere; “AgriD,” a large-scale agricultural greenhouse; and “FARO,” an automated tomato harvesting robot. Takaomi Hasegawa of the Food Value Chain Business Development Div., DENSO Corporation, gave a presentation on the development of FARO, an automated tomato harvesting robot.
【Speaker】
Takaomi Hasegawa
Food Value Chain Business Development Div.,
DENSO Corporation
In Charge of AI Image Recognition Technology for an Automated Tomato Harvesting Robot
The theme of my presentation is “Development of FARO, an Automated Tomato Harvesting Robot.”
First let me introduce myself briefly. I joined DENSO in 2006 and started out developing hardware for car navigation systems. My job was to design circuits and substrates. I was then transferred via the internal job posting system to develop software for an automated tomato harvesting robot. I’m in charge of AI image recognition technology.
FARO, an Automated Tomato Harvesting Robot Developed to Cope with the Aging of Farmers
Nakazato showed a video of FARO in operation. I’ll now outline the background of its development. As Bessho explained, Japan has a serious labor shortage due to the aging population and declining birthrate. The farming population in Japan has shrunk by 50% in the past 15 years, losing one million farmers in just 10 years.
In Japan, more than 70% of farmers are elderly, aged 60 or older. When starting the development, I had a valuable opportunity to interview farmers across Japan, but it was tough to hear their anguished comments such as: “There is a serious labor shortage.” “We recruit workers from China but still struggle.” “I must close my farm due to my age.” “There is no one to take over my farm.”
Against this backdrop, DENSO established AgriD with Asai Nursery, which is involved in advanced agriculture. Demonstrations and evaluations have been conducted for years using robots at its huge agricultural greenhouse in Inabe City, Mie Prefecture, which you saw in the video. We are automating various tasks to integrate agriculture and industry.
Farming involves various tasks. We’re focusing on automating the harvesting, which accounts for a large proportion of the work, and that’s why we developed FARO, an automated tomato harvesting robot.
System Configuration of FARO
This slide shows a schematic of the system. A robot arm manufactured by DENSO WAVE is installed on the mobile base unit, which is equipped with an industrial PC. Some of you may recognize the VS-060 robot arm: it is a six-axis robot arm also used for the Denoutekun shogi-playing robot.
A special harvesting hand developed by DENSO is attached to the end of the arm, and is designed to cut and hold tomatoes. One camera is mounted on the arm. FARO also has two fixed cameras, one each for the left and right vision. These cameras and AI image recognition technology can accurately recognize tomatoes.
Key Point 1 of the Robot Development: AI-based Image Recognition Technology
We focused mainly on three points when developing the robot: the special harvesting hand in terms of hardware, AI image recognition technology in terms of software, and cultivation management in terms of the environment to make harvesting easy for the robot. Due to time constraints, I’ll explain AI image recognition and cultivation management.
Regarding AI image recognition, we use deep learning technology, which is also used for automated driving, to improve the performance.
Image recognition for harvesting is done by six AI models as shown in the figure on the left: object detection to detect clusters and fruits, cluster direction determination to make harvesting easy for the robot, ripeness determination to check whether tomatoes are ready for picking, obstacle detection to check the risk of being caught by obstacles during harvesting, and accurate detection of peduncles, which are cut for harvesting, in pixels.
This video visualizes how the AI enables FARO to recognize the image. As you can see, FARO quickly and accurately identifies tomatoes that are ready for picking.
The table on the right shows that the results far exceeded the initial target thanks to the AI. Unfortunately we cannot give specific numbers regarding the quantitative performance.
During the daytime, the cameras could not get depth information for the thin peduncle due to strong sunlight. However, we solved this problem by developing our own AI-based compensation technology together with measurement experts from the Measurement & Instrumentation Eng. Div. of DENSO.
Software Development Environment
Next, I’ll explain the software development environment. The table on the right lists the technologies used for the robot software. If you are familiar with software development, you have probably seen most of them. Specifically, we use Linux and Windows for the OS and C++ and Python for the language. Regarding the library, we use Open CV, which is used for image processing, and TensorFlow and PyTorch, which are AI frameworks.
Regarding the infrastructure, we use GitHub, AWS, and GCP to manage the software version. We use both open software and DENSO’s proprietary robot development environment. Specifically, we use WinCAPS III as the integrated development environment for the robot and VRC as the robot emulator, and combine these environments for development. The figure on the left shows what can be done through this combination.
This shows a simulation of the farm on a PC. We can perform tests to see whether the developed software works properly on a PC without going to the farm by combining the data obtained on the farm with the development environment created by DENSO.
Key Point 2 of the Robot Development: Cultivation Management
Next, I’ll explain the environment in which the robot works. We went to the farm and showed the robot to many of the staff there. In the early phase of development, many of our members experienced farming operations to understand the difficulties and learned about cultivation management to make harvesting easy for the robot.
To test the robot, we brought it to the farm while its performance was still low and showed it to many farm staff to get their ideas on how to improve the performance and make it easy for the robot to work at the farm. The development process revealed that it was necessary to increase cooperation with farm staff to enable the robots to perform optimally.
Product Derived from Our Efforts and Future Outlook
These are the products (vegetables and fruits) picked by FARO, which can now harvest hundreds of kilograms of tomatoes each month. In the initial phase of development, we could not sell tomatoes in the market, so development members brought them home and ate them.
I took these photos at home. When the testing was at its climax, I ate tomato dishes prepared by my wife every day. Many development members also continued to eat tomatoes.
Finally, let me explain the future outlook. We will expand the scope of automation. At present, we focus on harvesting, but we will automate many more operations using FARO. We will also increase the crop species that can be harvested. At present, FARO can harvest only cherry tomatoes, which grow in clusters. We will expand the potential of FARO so that it can harvest ordinary tomatoes as well as other fruits and vegetables.
This concludes my presentation. Thank you for your attention.
*FARO is registered trademark of DENSO corporation in Japan.
FARO is being developed by Food Value Chain Biz Development Division in Japan.
