The soft fruit industry is facing unprecedented challenges due to its reliance of manual labour. We are presenting a newly launched robotics initiative which will help to address the issues faced by the industry and enable automation of the main processes involved in soft fruit production. The RASberry project (Robotics and Autonomous Systems for Berry Production) aims to develop autonomous fleets of robots for horticultural industry. To achieve this goal, the project will bridge several current technological gaps including the development of a mobile platform suitable for the strawberry fields, software components for fleet management, in-field navigation and mapping, long-term operation, and safe human-robot collaboration.
In this paper, we provide a general overview of the project, describe the main system components, highlight interesting challenges from a control point of view and then present three specific applications of the robotic fleets in soft fruit production. The applications demonstrate how robotic fleets can benefit the soft fruit industry by significantly decreasing production costs, addressing labour shortages and being the first step towards fully autonomous robotic systems for agriculture.
The soft fruit industry is very important globally: soft fruit is an important part of a healthy diet and the sector contributes significantly to economies of many individual countries. For example, the UK soft fruit production sector is worth £244 m to primary producers and employs 35,000 people to pick fruit each day. The fruit production is labor intensive and therefore facing many socio-economical challenges. In many developed countries such as the UK, the industry relies mainly on migrant labor which is unsustainable in the long term due to sociopolitical pressures (e.g. Brexit), labor inflation (higher minimum wages) or changing aspirations of migrant workers. Countries that traditionally do not rely on a migrant workforce face labor shortages due to other problems such as aging population in Japan or after-effects of one-child policy in China.
Soft fruit production involves many operations such as planting, plant care, fruit picking and transportation of the crop to storage facilities. So far, these processes have been very difficult to automate using the current technology due to their complex character. For example, picking strawberries requires advanced cognitive abilities to locate and assess the fruit ripeness and fine manipulation skills allowing to reach and carefully pick a specific fruit. And yet, there are many opportunities for optimization of the entire soft fruit production that could improve productivity, cut cost and address labor shortages. For example, an automated solution for in-field transportation on its own could save around 20% labor cost and 10% land usage.
To unleash the potential for automation in agriculture and to address the issues faced by the soft fruit industry, a new robotics initiative was recently launched. The RASberry project (Robotics and Autonomous Systems for Berry Production) aims to develop autonomous fleets of robots for the horticultural industry. In particular, the project considers strawberry production both in a traditional open ground fashion and in polytunnels. The first major objective is to support in-field transportation operations to aid and complement human fruit pickers, followed by other objectives on applications such as plant treatment, yield forecasting and fruit picking. To achieve this goal, the project will bridge several current technological gaps including the development of a mobile platform suitable for strawberry fields, software components for fleet management, in-field navigation and mapping, long-term operation, and safe human-robot collaboration. RASberry is a collaboration between academia, the robotics industry, and strawberry growers to cover multidisciplinary competencies required for successful delivery of the project.
In this article, we provide a general overview of the project, describe the main system components, highlight interesting challenges from a control point of view and then present three specific applications of the robotic fleets in soft fruit production. The proposed system involves low-level motor control for robot actuators, but also intelligent control for safe navigation of individual robots and finally the coordination of the entire fleet. Many of the control aspects involve not only direct information provided by sensors, but also the learned representations of the environment and intelligent interpretation of its state illustrating the complex requirements faced by control systems deployed in the agricultural domain.
The RASberry system consists of a coordinated fleet of dedicated mobile platforms, which navigate safely around the strawberry environment and interact intelligently with human workers to achieve a given task (e.g., transporting a tray of picked fruit). The mobile platform and its control software are customized for the tasks such that no changes to the environment are necessary. The individual robots are able to interact and support human workers in an intuitive way. All available tasks are shared between multiple robots, which are orchestrated by a central coordination system to assure the best utilization of the entire fleet. The key innovation here is the development of autonomous systems that can work safely and over long periods of time in a complex farm environment.
The robotic platform employed in this project is Thorvald, a modular, general purpose, lightweight outdoor mobile platform designed for autonomous operation in the agricultural domain. The robot design is based on a selection of modules, with modules serving various purposes, like propulsion, battery storage, suspension, etc. Modules can be combined to create a wide array of robots with various properties. The robot’s software is designed to support the modularity found in hardware, and can control robots with any number of arbitrarily-placed modules. This means that differential-drive robots, Ackermann steering robots and all-wheel drive, all-wheel steering robots all run the same software. Configuration-specific parameters are read from one single configuration file at startup.
The modular design allows for easy re-configuration for a specific application and environment. To support strawberry production, we use a 1.0 m wide robot fitted with four-wheel drive and four-wheel steering. This particular robot configuration can carry up to two batteries. However, to limit the unit cost of the robot, only one battery is installed. This single battery pack provides roughly 10 hours of autonomy without payload. The robot is controlled through a built-in computer running Linux OS and Robot Operating System (ROS). The final configuration of the robot will also include on-board multi-modal sensing to support safe and autonomous navigation of the robots and their collaboration with humans.
The presented platform is generic but each specific application will involve additional customizations: for example, a dedicated apparatus for UV-light treatment or a picking tray storage for in-field transportation.
The core ability of each robot in the fleet is its autonomous operation including safe navigation, automated charging and fault detection. For precise positioning, typical GPS-based systems used in modern agricultural machinery are not suitable in the polytunnels where severe signal losses and reliability problems are quite common. Therefore, we will use a combination of GPS, inertial, and LIDAR (Light Detection and Ranging Device) sensors to achieve centimeter precision and robustness. LIDAR and time-of-flight 3D cameras will be used to build geometric maps of the environment and sense the presence of obstacles and humans so that the path of the robot can be adjusted accordingly. The system will use the topological representation of the field with all its important features utilizing the existing navigation and simulation solutions developed by the RASberry team (see Figure 2). To ensure continuous operation of the robot, an automated fault detection system and remote reporting will need to be devised and the infrastructure will also need to provide auto-charging functionality.
Safe Human-Robot Interaction
Safe and effective interaction of the robotic platform with human workers is of paramount importance in the agricultural domain. Therefore, the project will develop appropriate techniques for the detection of humans and the characterisation of their behavior such that the use of appropriate interactive strategies can be employed. Human-aware navigation will ensure that the humans are treated not just as obstacles, but as interactive agents with intentionality. This will improve the general safety of operation. Human-appropriate means of coordination and communication with the robot will ensure that the intentions of the robot are clear to the human (such as intent to pick up crates or just passing by), and that the human can communicate their requirements to the robot. The information about humans will be provided by a dedicated, multimodal people detection and tracking system utilizing sensors such as 3D LIDAR, RGB-D, thermal cameras but also RFID antennas.
In this project, a fleet of robots is to be coordinated and its utility to be maximized. Hence, individual robots need to be scheduled to arrive where and when they are needed. A central coordination unit will allocate tasks to robots and coordinate their behavior, by activating and monitoring operational modes (e.g., a robot is ready to receive picked trays). In order to optimize the robotic fleet, the fleet coordination component will learn to anticipate human worker needs and optimize the spatial distribution of robots, i.e., making robots available for a picker, anticipating their needs, to further optimize workflows. One aspect considered here is that, while the fleet is scheduling itself autonomously, anticipating tasks, it will still have to be responsive to on-demand tasks, requested by workers in the environment. Furthermore, resource constraints, such as battery charge, will have to be factored in as anticipated tasks.
About 20-30% of the labor cost is spent walking trays of picked fruit and empties back and forth from the crop to the ends of field/ greenhouse and on farm logistics hubs. In addition, around 10% of the field area is designated for transportation needs (access for tractors, trucks, etc.). Our fleet for in-field transportation will eliminate the need for workers to carry picked crop and replenish them with empty trays. In addition, the transportation infrastructure can be significantly reduced since the robots do not require special arrangements. The robot will be equipped with a dedicated picking tray storage and weight sensors for basic quality assurance. This functionality will enable more precise traceability of the produce and more precise yield estimation. The in-field transportation is an universal problem for production of various crops and hence the results of the project will be directly transferable to other domains.
We have conducted an initial study, in which tele-operated robots were working with the expert human pickers, to evaluate how an autonomous robot-based in-field transportation system would work for them, what constraints are imposed by such a solution, and also what benefits this may confer . The results indicate that after the experience with the robots, they were generally viewed positively, with the behavior viewed as appropriate and safe. This was despite the participants having mixed views on robots in general, although they seemed less worried about the prospect of their work being directly replaced by robots, which is in contrast to an overall relatively negative view of robots in terms of job replacement reported in other areas of industry.
It is well-known that UV light has a positive effect on plants. Our experiments show that if UV light is applied to plants during the nighttime, even with a very small dose, this will practically eliminate outbreaks of fungus. Tests have shown that this works extremely well on mildew, and we are now testing for greymold. There are also other likely positive effects such as on bacteria, and even on the plant and the fruit itself. Even though the effects of UV on plants have been known for a long time, there has not been an efficient way to apply this treatment to the plants without exposing humans to UV’s harmful effects. We have therefore mounted UV lamps on the Thorvald platform. The robot has been working autonomously several times every week throughout the season on a strawberry farm in Norway with very good results. We have also designed a UV rig for use in greenhouses with cucumber and tomato, where we are currently doing similar experiments.
A robotic fruit picker has also been developed for the Thorvald system . The system consists of a specially-designed gripper for strawberry harvesting, which contains a container that encapsulates the strawberry, and then once the strawberry is identified to be inside the container, the gripper cuts the stem and stores the berry in the container. We are currently able to pick strawberries at a speed of 7.5 s per berry. The system is relatively fast because we can store the strawberries locally in the gripper and pick up to 500 grams of berries before we return them to the base. This reduces the time needed to pick each strawberry. Furthermore, the system is gentle on the berries as we do not touch the fruit, but only cut the stem. Currently the system struggles with berries that are localized in clusters, but a new gripper design is currently being tested to deal with these problems.
In this article, we have presented an overview of the RASberry project aiming to develop fully autonomous fleets of agricultural robots to support the soft fruit industry. We have presented the individual system components and their main characteristics, demonstrated some initial design efforts and presented three application examples of the robotic fleet for in-field transportation, plant treatment and fruit picking. We have discussed how such solutions can revolutionize the soft fruit industry but also lead to widespread adoption of robotic systems in agriculture in general.
The project is still in its early stages and over the next two years we will be developing our fleet’s autonomy, efficiency and long-term robustness so that it is ready for adoption by the industry. A solution for autonomous fleets of agricultural robots will significantly decrease soft fruit production costs, address labor shortages and be the first step towards fully autonomous robotic systems for berry production.
About the Authors
Pål Johan From is Professor of Agri-robotics at the Norwegian University of Life Sciences and University of Lincoln, UK. He has a PhD from the Norwegian University of Science and Technology in modeling and control of complex robotic systems. From is head of the Robotics Group at the Norwegian University of Life Sciences, which has designed and built the Thorvald agricultural robot. From was a visiting researcher at Hong Kong University of Science and Technology from 2007-2008 and at UC Berkeley from 2008-2010.
Lars Grimstad holds an MSc in Mechanical engineering and is currently a PhD student in the robotics group at the Norwegian University of Life Sciences. His research focuses on modular and reconfigurable mobile robots for the agricultural domain and their applications. His main contribution in this field is his work related to the development of the Thorvald robots.
Marc Hanheide is a Professor of Intelligent Robotics & Interactive Systems at the University of Lincoln, UK. He received the PhD degree (Dr.-Ing.) from Bielefeld University in 2006. From 2006 to 2009 he held a position as a senior researcher at Bielefeld University. From 2009 to 2011 he was a research fellow at the University of Birmingham, UK. Marc Hanheide currently serves as a PI and Co-PI in various robotics projects funded by H2020, EPSRC, and InnovateUK, investigating long-term robotic behavior and artificial intelligence in general, and human-robot interaction in particular, with applications in care, logistics, security, and agriculture.
Simon Pearson is Professor of Agri-Food Technology and Founding Director of the Lincoln Institute of Agri Food Technology (LIAT) at the University of Lincoln. His research interests focus on the application of technology, in particular robotics and digital systems, within the agri-food supply chain from farm through to consumer. All LIAT's on-going research is cross disciplinary with both national and international partners. Prior to joining Lincoln, Simon spent 15 years in industry running significant farming estates and with a large international retailer.
Dr. Grzegorz Cielniak is a Senior Lecturer at the School of Computer Science, University of Lincoln. His research interests include mobile robotics, artificial intelligence, real-time computer vision systems and multi-sensor fusion. In particular, he is interested in robotic applications in agriculture. Grzegorz holds a number of funded grants for collaborative projects with the agri-food industry and others totalling more than £1.5M. He has published widely in the international scientific community, has served as a program committee member for international conferences and as a reviewer for several international journals, conferences and European funding proposals.
RASberry is funded by its collaborators including Norwegian University of Life Sciences, University of Lincoln, Saga Robotics, CBS Ltd., Berry Gardens Growers and Ekeberg Myhrene. Further funding is provided by RCN Forny, Innovate UK, Innovation Norway and Innovate UK Grant IUK grant 103339.