Research Articles in Indexed Journals

 Download Pdf

Fruit detection in an apple orchard using a mobile terrestrial laser scanner

Fruit_detection 3D point cloud models obtained for trees. https://ars.els-cdn.com/content/image/1-s2.0-S1537511019308128-gr2_lrg.jpg

Authors: Gené-Mola, J.; Gregorio, E., Guevara, J.; Auat, F.; Sanz-Cortiella, R.; Escolà, A.; Llorens, J.; Morros, J.R.; Ruiz-Hidalgo, J.; Vilaplana, V.; Rosell-Polo, J.R., 2019. Fruit detection in an apple orchard using a mobile terrestrial laser scanner. Biosystems Engineering 187, 171-184. https://doi.org/10.1016/j.biosystemseng.2019.08.017

Abstract: The development of reliable fruit detection and localization systems provides an opportunity to improve the crop value and management by limiting fruit spoilage and optimised harvesting practices. Most proposed systems for fruit detection are based on RGB cameras and thus are affected by intrinsic constraints, such as variable lighting conditions. This work presents a new technique that uses a mobile terrestrial laser scanner (MTLS) to detect and localise Fuji apples. An experimental test focused on Fuji apple trees (Malus domestica Borkh. cv. Fuji) was carried out. A 3D point cloud of the scene was generated using an MTLS composed of a Velodyne VLP-16 LiDAR sensor synchronised with an RTK-GNSS satellite navigation receiver. A reflectance analysis of tree elements was performed, obtaining mean apparent reflectance values of 28.9%, 29.1%, and 44.3% for leaves, branches and trunks, and apples, respectively. These results suggest that the apparent reflectance parameter (at 905 nm wavelength) can be useful to detect apples. For that purpose, a four-step fruit detection algorithm was developed. By applying this algorithm, a localization success of 87.5%, an identification success of 82.4%, and an F1-score of 0.858 were obtained in relation to the total amount of fruits. These detection rates are similar to those obtained by RGB-based systems, but with the additional advantages of providing direct 3D fruit location information, which is not affected by sunlight variations. From the experimental results, it can be concluded that LiDAR-based technology and, particularly, its reflectance information, has potential for remote apple detection and 3D location.

Grant RTI2018-094222-B-I00 (PAgFRUIT project) by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union.

Fruit detection, yield prediction and canopy geometric characterization using LiDAR with forced air flow

LID

Authors: Gené-Mola, J.; Gregorio, E., Auat, F.; Guevara, J.; Llorens, J.; Sanz-Cortiella, R.; Escolà, A.; Rosell-Polo, J.R., 2020. Fruit detection, yield prediction and canopy geometric characterization using LiDAR with forced air flow. Computers and Electronics in Agriculture 168, 105121. https://doi.org/10.1016/j.compag.2019.105121

Abstract: Yield monitoring and geometric characterization of crops provide information about orchard variability and vigor, enabling the farmer to make faster and better decisions in tasks such as irrigation, fertilization, pruning, among others. When using LiDAR technology for fruit detection, fruit occlusions are likely to occur leading to an underestimation of the yield. This work is focused on reducing the fruit occlusions for LiDAR-based approaches, tackling the problem from two different approaches: applying forced air flow by means of an air-assisted sprayer, and using multi-view sensing. These approaches are evaluated in fruit detection, yield prediction and geometric crop characterization. Experimental tests were carried out in a commercial Fuji apple (Malus domestica Borkh. cv. Fuji) orchard. The system was able to detect and localize more than 80% of the visible fruits, predict the yield with a root mean square error lower than 6% and characterize canopy height, width, cross-section area and leaf area. The forced air flow and multi-view approaches helped to reduce the number of fruit occlusions, locating 6.7% and 6.5% more fruits, respectively. Therefore, the proposed system can potentially monitor the yield and characterize the geometry in apple trees. Additionally, combining trials with and without forced air flow and multi-view sensing presented significant advantages for fruit detection as they helped to reduce the number of fruit occlusions.

Grant RTI2018-094222-B-I00 (PAgFRUIT project) by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union.

Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry

Photg1

Authors: Gené-Mola, J.; Sanz-Cortiella, R., Rosell-Polo, J.R., Morros, J.R, Ruiz-Hidalgo, J., Vilaplana, V., Gregorio, E., 2020. Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry. Computers and Electronics in Agriculture 169, 105165. https://doi.org/10.1016/j.compag.2019.105165

Abstract: The development of remote fruit detection systems able to identify and 3D locate fruits provides opportunities to improve the efficiency of agriculture management. Most of the current fruit detection systems are based on 2D image analysis. Although the use of 3D sensors is emerging, precise 3D fruit location is still a pending issue. This work presents a new methodology for fruit detection and 3D location consisting of: (1) 2D fruit detection and segmentation using Mask R-CNN instance segmentation neural network; (2) 3D point cloud generation of detected apples using structure-from-motion (SfM) photogrammetry; (3) projection of 2D image detections onto 3D space; (4) false positives removal using a trained support vector machine. This methodology was tested on 11 Fuji apple trees containing a total of 1455 apples. Results showed that, by combining instance segmentation with SfM the system performance increased from an F1-score of 0.816 (2D fruit detection) to 0.881 (3D fruit detection and location) with respect to the total amount of fruits. The main advantages of this methodology are the reduced number of false positives and the higher detection rate, while the main disadvantage is the high processing time required for SfM, which makes it presently unsuitable for real-time work. From these results, it can be concluded that the combination of instance segmentation and SfM provides high performance fruit detection with high 3D data precision. The dataset has been made publicly available and an interactive visualization of fruit detection results is accessible at http://www.grap.udl.cat/documents/photogrammetry_fruit_detection.html.

Grant RTI2018-094222-B-I00 (PAgFRUIT project) by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union.

Analyzing and overcoming the effects of GNSS error on LiDAR based orchard parameters estimation

Lidar_data2

Authors: Guevara, J., Auat, F., Guevara, J., Gené-Mola, J., Rosell-Polo, J.R., Gregorio, E., 2020. Analyzing and overcoming the effects of GNSS error on LiDAR based orchard parameters estimation. Computers and Electronics in Agriculture 170, 10525529. https://doi.org/10.1016/j.compag.2020.105255

Abstract: Currently, 3D point clouds are obtained via LiDAR (Light Detection and Ranging) sensors to compute vegetation parameters to enhance agricultural operations. However, such a point cloud is intrinsically dependent on the GNSS (global navigation satellite system) antenna used to have absolute positioning of the sensor within the grove. Therefore, the error associated with the GNSS receiver is propagated to the LiDAR readings and, thus, to the crown or orchard parameters. In this work, we first describe the error propagation of GNSS over the laser scan measurements. Second, we present our proposal to overcome this effect based only on the LiDAR readings. Such a proposal uses a scan matching approach to reduce the error associated with the GNSS receiver. To accomplish such purpose, we fuse the information from the scan matching estimations with the GNSS measurements. In the experiments, we statistically analyze the dependence of the grove parameters extracted from the 3D point cloud -specifically crown surface area, crown volume, and crown porosity- to the localization error. We carried out 150 trials with positioning errors ranging from 0.01 meters (ground truth) to 2 meters. When using only GNSS as a localization system, the results showed that errors associated with the estimation of vegetation parameters increased more than 100 when positioning error was equal or bigger than 1 meter. On the other hand, when our proposal was used as a localization system, the results showed that for the same case of 1 meter, the estimation of orchard parameters improved in 20 overall. However, in lower positioning errors of the GNSS, the estimation of orchard parameters were improved up to 50% overall. These results suggest that our work could lead to better decisions in agricultural operations, which are based on foliar parameter measurements, without the use of external hardware.

Assessing the Performance of RGB-D Sensors for 3D Fruit Crop Canopy Characterization under Different Operating and Lighting Conditions

Authors: Gené-Mola, J., Llorens, J., Rosell-Polo, J.R., Gregorio, E., Arnó, J., Solanelles, F., Martínez-Casasnovas, J.A., Escolà, A., 2020. Assessing the Performance of RGB-D Sensors for 3D Fruit Crop Canopy Characterization under Different Operating and Lighting Conditions. Sensors 20(24), 7072; https://doi.org/10.3390/s20247072

Abstract: The use of 3D sensors combined with appropriate data processing and analysis has provided tools to optimise agricultural management through the application of precision agriculture. The recent development of low-cost RGB-Depth cameras has presented an opportunity to introduce 3D sensors into the agricultural community. However, due to the sensitivity of these sensors to highly illuminated environments, it is necessary to know under which conditions RGB-D sensors are capable of operating. This work presents a methodology to evaluate the performance of RGB-D sensors under different lighting and distance conditions, considering both geometrical and spectral (colour and NIR) features. The methodology was applied to evaluate the performance of the Microsoft Kinect v2 sensor in an apple orchard. The results show that sensor resolution and precision decreased significantly under middle to high ambient illuminance (>2000 lx). However, this effect was minimised when measurements were conducted closer to the target. In contrast, illuminance levels below 50 lx affected the quality of colour data and may require the use of artificial lighting. The methodology was useful for characterizing sensor performance throughout the full range of ambient conditions in commercial orchards. Although Kinect v2 was originally developed for indoor conditions, it performed well under a range of outdoor conditions.

Grant RTI2018-094222-B-I00 (PAgFRUIT project) by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union.

In-field apple size estimation using photogrammetry-derived 3D point clouds: Comparison of 4 different methods considering fruit occlusions

Authors: Gené-Mola, J., Sanz-Cortiella, R., Rosell-Polo, J.R., Escolà, A., Gregorio, E., 2021. In-field apple size estimation using photogrammetry-derived 3D point clouds: Comparison of 4 different methods considering fruit occlusions. Computers and Electronics in Agriculture 188, 106343. https://doi.org/10.1016/j.compag.2021.106343

Abstract: In-field fruit monitoring at different growth stages provides important information for farmers. Recent advances have focused on the detection and location of fruits, although the development of accurate fruit size estimation systems is still a challenge that requires further attention. This work proposes a novel methodology for automatic in-field apple size estimation which is based on four main steps: 1) fruit detection; 2) point cloud generation using structure-from-motion (SfM) and multi-view stereo (MVS); 3) fruit size estimation; and 4) fruit visibility estimation. Four techniques were evaluated in the fruit size estimation step. The first consisted of obtaining the fruit diameter by measuring the two most distant points of an apple detection (largest segment technique). The second and third techniques were based on fitting a sphere to apple points using least squares (LS) and M−estimator sample consensus (MSAC) algorithms, respectively. Finally, template matching (TM) was applied for fitting an apple 3D model to apple points. The best results were obtained with the LS, MSAC and TM techniques, which showed mean absolute errors of 4.5 mm, 3.7 mm and 4.2 mm, and coefficients of determination () of 0.88, 0.91 and 0.88, respectively. Besides fruit size, the proposed method also estimated the visibility percentage of apples detected. This step showed an of 0.92 with respect to the ground truth visibility. This allowed automatic identification and discrimination of the measurements of highly occluded apples. The main disadvantage of the method is the high processing time required (in this work 2760 s for 3D modelling of 6 trees), which limits its direct application in large agricultural areas. The code and the dataset have been made publicly available and a 3D visualization of results is accessible at http://www.grap.udl.cat/en/publications/apple_size_estimation_SfM.

Grant RTI2018-094222-B-I00 (PAgFRUIT project) by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union.

 

A photogrammetry-based methodology to obtain accurate digital ground-truth of leafless fruit trees

Authors: Lavaquiol, B., Sanz, R., Llorens, J. Arnó, J., Escolà, A. 2021.A photogrammetry-based methodology to obtain accurate digital ground-truth of leafless fruit trees. Computers and Electronics in Agriculture 191, 06553. https://doi.org/10.1016/j.compag.2021.106553

In recent decades, a considerable number of sensors have been developed to obtain 3D point clouds that have great potential in optimizing management in agriculture through the application of precision agriculture techniques. In order to use the data provided by these sensors, it is essential to know their measurement error. In this paper, a methodology is presented for obtaining a 3D point cloud of a central axis training system defoliated fruit tree (Malus domestica Bork.) obtained from stereophotogrammetry techniques based on structure-from-motion (SfM) and multi-view stereo-photogrammetry (MVS). The point cloud was made from a set of 288 photographs of the scene including the ground truth tree which was used to generate the digital 3D model. The resulting point cloud was validated and proven to faithfully represent reality. The bias of the resulting model is −0.15 mm and 0.05 mm, for diameters and lengths, respectively. In addition, the presented methodology allows small changes in the ground truth actual tree to be detected as a consequence of the wood dehydration process. Having an actual and a digital ground-truth is the basis for validating other sensing systems for 3D vegetation characterization which can be used to obtain data to make more informed management decisions.

Grant RTI2018-094222-B-I00 (PAgFRUIT project) by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union.

3D Spectral Graph Wavelet Point Signatures in Pre-Processing Stage for Mobile Laser Scanning Point Cloud Registration in Unstructured Orchard Environments

Authors: Guevara, D.J., Gené-Mola, J., Greogrio, E., Auat Cheein, F. 2021. 3D Spectral Graph Wavelet Point Signatures in Pre-Processing Stage for Mobile Laser Scanning Point Cloud Registration in Unstructured Orchard EnvironmentsIEEE Sensors Journal. https://doi.org/10.1109/JSEN.2021.3129340

The use of three-dimensional registration techniques is an important component for sensor-based localization and mapping. Several approaches have been proposed to align three-dimensional data, obtaining meaningful results in structured scenarios. However, the increased use of high-frame-rate 3D sensors has lead to more challenging application scenarios where the performance of registration techniques may degrade significantly. In order to improve the accuracy of the procedure, different works have considered a representative subset of points while preserving application-dependent features for registration. In this work, we tackle such a problem, considering the use of a general feature-extraction operator in the spectral domain as a prior step to the registration. The proposed spectral strategies use three wavelet transforms that are evaluated along with four well-known registration techniques. The methodology was experimentally validated in a dense orchard environment. The results show that the probability of failure in registration can be reduced up to 12.04% for the evaluated approaches, leading to a significant increase in the localization accuracy. Those results validate the effectiveness and efficiency of the spectral-assisted registration algorithms in an agricultural setting and motivate their usage for a wider range of applications.

Grant RTI2018-094222-B-I00 (PAgFRUIT project) by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union.

Comparison of 3D scan matching techniques for autonomous robot navigation in urban and agricultural environments

Authors: Guevara, D.J., Gené-Mola, J., Greogrio, E., Torres-Torriti, M., Reina, G., Auat Cheein, F. 2021. Comparison of 3D scan matching techniques for autonomous robot navigation in urban and agricultural environments. Journal of Applied Remote Sensing 16(2), 024508. https://doi.org/10.1117/1.JRS.15.024508 http://hdl.handle.net/10459.1/71527

Global navigation satellite system (GNSS) is the standard solution for solving the localization problem in outdoor environments, but its signal might be lost when driving in dense urban areas or in the presence of heavy vegetation or overhanging canopies. Hence, there is a need for alternative or complementary localization methods for autonomous driving. In recent years, exteroceptive sensors have gained much attention due to significant improvements in accuracy and cost-effectiveness, especially for 3D range sensors. By registering two successive 3D scans, known as scan matching, it is possible to estimate the pose of a vehicle. This work aims to provide in-depth analysis and comparison of the state-of-the-art 3D scan matching approaches as a solution to the localization problem of autonomous vehicles. Eight techniques (deterministic and probabilistic) are investigated: iterative closest point (with three different embodiments), normal distribution transform, coherent point drift, Gaussian mixture model, support vector-parametrized Gaussian mixture and the particle filter implementation. They are demonstrated in long path trials in both urban and agricultural environments and compared in terms of accuracy and consistency. On the one hand, most of the techniques can be successfully used in urban scenarios with the probabilistic approaches that show the best accuracy. On the other hand, agricultural settings have proved to be more challenging with significant errors even in short distance trials due to the presence of featureless natural objects. The results and discussion of this work will provide a guide for selecting the most suitable method and will encourage building of improvements on the identified limitations.

Grant RTI2018-094222-B-I00 (PAgFRUIT project) by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union.

3D characterization of a Boston Ivy double-skin green building facade using a LiDAR system

Authors: Pérez, G., Escolà, A, Rosell-Polo, J.R., Coma, J., Arasanz, R., Marrero, B., Cabeza, L.F., Gregorio, E. 2021. 3D characterization of a Boston Ivy double-skin green building facade using a LiDAR system. Building and Environment 206 (2021), 108320. DOI: https://doi.org/10.1016/j.buildenv.2021.108320

On the way to more sustainable and resilient urban environments, the incorporation of urban green infrastructure (UGI) systems, such as green roofs and vertical greening systems, must be encouraged. Unfortunately, given their variable nature, these nature-based systems are difficult to geometrically characterize, and therefore there is a lack of 3D objects that adequately reflect their geometry and analytical properties to be used in design processes based on Building Information Modelling (BIM) technologies. This fact can be a disadvantage, during the building's design phase, of UGIs over traditional grey solutions. Areas of knowledge such as precision agriculture, have developed technologies and methodologies that characterize the geometry of vegetation using point cloud capture. The main aim of this research was to create the 3D characterization of an experimental double-skin green facade, using LiDAR technologies.

From the results it could be confirmed that the methodology used was precise and robust, enabling the 3D reconstruction of the green facade's outer envelope. Detailed results were that foliage volume differences in height were linked to plant growth, whereas differences in the horizontal distribution of greenery were related to the influence of the local microclimate and specific plant diseases on the south orientation. From this research, along with complementary previous research, it could be concluded that, generally speaking, with vegetation volumes of 0.2 m3/m2, using Boston Ivy (Parthenocissus Tricuspidata) under Mediterranean climate, reductions in external building surface temperatures of around 13 °C can be obtained and used as analytic parameter in a future 3D-BIM-object.

Grant RTI2018-094222-B-I00 (PAgFRUIT project) by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union.

Delineation of management zones in super-intensive almond orchards based on vegetation indices from UAV images validated by LiDAR-derived canopy parameters

Authors: Martínez-Casasnovas, J.A., Sandonís-Pozo, L., Escolà, A., Arnó, J., Llorens, J., 2022. Delineation of management zones in super-intensive almond orchards  based on vegetation indices from UAV images validated by LiDAR-derived canopy  parameters. Agronomy 12(1), 102. https://doi.org/10.3390/agronomy12010102

One of the challenges in orchard management, in particular of hedgerow tree plantations, is the delineation of management zones on the bases of high-precision data. Along this line, the present study analyses the applicability of vegetation indices derived from UAV images to estimate the key structural and geometric canopy parameters of an almond orchard. In addition, the classes created on the basis of the vegetation indices were assessed to delineate potential management zones. The structural and geometric orchard parameters (width, height, cross-sectional area and porosity) were characterized by means of a LiDAR sensor, and the vegetation indices were derived from a UAV-acquired multispectral image. Both datasets summarized every 0.5 m along the almond tree rows and were used to interpolate continuous representations of the variables by means of geostatistical analysis. Linear and canonical correlation analyses were carried out to select the best performing vegetation index to estimate the structural and geometric orchard parameters in each cross-section of the tree rows. The results showed that NDVI averaged in each cross-section and normalized by its projected area achieved the highest correlations and served to define potential management zones. These findings expand the possibilities of using multispectral images in orchard management, particularly in hedgerow plantations.

Grant RTI2018-094222-B-I00 (PAgFRUIT project) by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union.

Satellite multispectral indices to estimate canopy parameters and within‑field management zones in super‑intensive almond orchards

Authors: Sandonís-Pozo, L., Llorens, J., Escolà, A., Arnó, J., Pascual, M., Martínez-Casasnovas, J.A., 2022. Satellite multispectral indices to estimate canopy parameters and within‑field management zones in super‑intensive almond orchards. Precision Agriculture, accepted. https://doi.org/10.1007/s11119-022-09936-w

Continuous canopy status monitoring is an essential factor to support and precisely apply orchard management actions such as pruning, pesticide and foliar treatment applications, or fertirrigation, among others. For that, this work proposes the use of multispectral vegetation indices to estimate geometric and structural orchard parameters from remote sensing images (high temporal and spatial resolution) as an alternative to more time-consuming processing techniques, such as LiDAR surveys or UAV photogrammetry. A super-intensive almond (Prunus dulcis) orchard was scanned using a mobile terrestrial laser (LiDAR) in two different vegetative stages (after spring pruning and before harvesting). From the LiDAR point cloud, canopy orchard parameters, including maximum height and width, cross-sectional area and porosity, were summarized every 0.5 m along the rows and interpolated using block kriging to the pixel centroids of PlanetScope (3 × 3 m) and Sentinel-2 (10 × 10 m) image grids. To study the association between the LiDAR-derived parameters and 4 different vegetation indices. A canonical correlation analysis was carried out, showing the normalized difference vegetation index (NDVI) and the green normalized difference vegetation index (GNDVI) to have the best correlations. A cluster analysis was also performed. Results can be considered optimistic both for PlanetScope and Sentinel-2 images to delimit within-field management zones, being supported by significant differences in LiDAR-derived canopy parameters.

Grant RTI2018-094222-B-I00 (PAgFRUIT project) by MCIN/AEI/10.13039/501100011033 and by “ERDF, a way of making Europe”, by the European Union.

   Last modification: