Cereal Crop Ear Counting in Field Conditions Using Zenithal RGB Images

Summary

We present a protocol for counting durum wheat and barley ears, using natural color (RGB) digital photographs taken in natural sunlight under field conditions. With minimal adjustments for camera parameters and some environmental condition limitations, the technique provides precise and consistent results across a range of growth stages.

Abstract

Ear density, or the number of ears per square meter (ears/m²), is a central focus in many cereal crop breeding programs, such as wheat and barley, representing an important agronomic yield component for estimating grain yield. Therefore, a quick, efficient, and standardized technique for assessing ear density would aid in improving agricultural management, providing improvements in preharvest yield predictions, or could even be used as a tool for crop breeding when it has been defined as a trait of importance. Not only are the current techniques for manual ear density assessments laborious and time-consuming, but they are also without any official standardized protocol, whether by linear meter, area quadrant, or an extrapolation based on plant ear density and plant counts postharvest. An automatic ear counting algorithm is presented in detail for estimating ear density with only sunlight illumination in field conditions based on zenithal (nadir) natural color (red, green, and blue [RGB]) digital images, allowing for high-throughput standardized measurements. Different field trials of durum wheat and barley distributed geographically across Spain during the 2014/2015 and 2015/2016 crop seasons in irrigated and rainfed trials were used to provide representative results. The three-phase protocol includes crop growth stage and field condition planning, image capture guidelines, and a computer algorithm of three steps: (i) a Laplacian frequency filter to remove low- and high-frequency artifacts, (ii) a median filter to reduce high noise, and (iii) segmentation and counting using local maxima peaks for the final count. Minor adjustments to the algorithm code must be made corresponding to the camera resolution, focal length, and distance between the camera and the crop canopy. The results demonstrate a high success rate (higher than 90%) and R² values (of 0.62-0.75) between the algorithm counts and the manual image-based ear counts for both durum wheat and barley.

Introduction

The world cereal utilization in 2017/2018 is reported expand by 1% from the previous year¹. Based on the latest predictions for cereal production and population utilization, world cereal stocks need to increase yields at a faster rate in order to meet growing demands, while also adapting to increasing effects of climate change². Therefore, there is an important focus on yield improvement in cereal crops through improved crop breeding techniques. Two the most important and harvested cereals in the Mediterranean region are selected as examples for this study, namely, durum wheat (Triticum aestivum L. ssp. durum [Desf.]) and barley (Hordeum vulgare L.). Durum wheat is, by extension, the most cultivated cereal in the south and east margins of the Mediterranean Basin and is the 10th most important crop worldwide, owing to its annual production of 37 million tons annually³, while barley is the fourth global grain in terms of production, with a global production at 144.6 million tons annually⁴.

Remote sensing and proximal image analysis techniques are increasingly key tools in the advancement of field high-throughput plant phenotyping (HTPP) as they not only provide more agile but also, often, more precise and consistent retrievals of target crop biophysiological traits, such as assessments of photosynthetic activity and biomass, preharvest yield estimates, and even improvements in trait heritability, such as efficiency in resource use and uptake^5,6,7,8,9. Remote sensing has traditionally focused on multispectral, hyperspectral, and thermal imaging sensors from aerial platforms for precision agriculture at the field scale or for plant phenotyping studies at the microplot scale¹⁰. Common, commercially available digital cameras that measure only visible reflected light were often overlooked, despite their very high spatial resolution, but have recently become popular as new innovative image-processing algorithms are increasingly able to take advantage of the detailed color and spatial information that they provide. Many of the newest innovations in advanced agricultural image analyses increasingly rely on the interpretation of data provided by very high-resolution (VHR) RGB images (for their measurement of red, green, and blue visible light reflectance), including crop monitoring (vigor, phenology, disease assessments, and identification), segmentation and quantification (emergence, ear density, flower and fruit counts), and even full 3D reconstructions based on a new structure from motion workflows¹¹.

One of the most essential points for improvement in cereal productivity is a more efficient assessment of yield, which is determined by three major components: ear density or the number of ears per square meter (ears/m²), the number of the grains per ear, and the thousand-kernel weight. Ear density can be obtained manually in the field, but this method is laborious, time-consuming, and lacking in a single standardized protocol, which together may result in a significant source of error. Incorporating the automatic counting of ears is a challenging task because of the complex crop structure, close plant spacing, high extent of overlap, background elements, and the presence of awns. Recent work has advanced in this direction by using a black background structure supported by a tripod in order to acquire suitable crop images, demonstrating fairly good results in ear counting¹². In this way, excessive sunlight and shadow effects were avoided, but such a structure would be cumbersome and a major limitation in an application to field conditions. Another example is an automatic ear counting algorithm developed using a fully automated phenotyping system with a rigid motorized gantry, which was used with good accuracy for counting ear density in a panel composed of five awnless bread wheat (Triticum aestivum L.) varieties growing under different nitrogen conditions¹³. Recent work by Fernandez-Gallego¹⁴ has optimized this process for quicker and easier data capture, using VHR RGB color images followed by more advanced, yet still fully automated, image analyses. The efficient and high-quality data collection in field conditions emphasizes a simplified standardized protocol for consistency and high data capture throughput, while the image-processing algorithm employs the novel use of Laplacian and frequency domain filters to remove undesired image components before applying a segmentation for counting based on finding local maxima (as opposed to full delineation as in other previous studies, which may result in more errors with overlapping ears).

This work proposes a simple system for the automatic quantification of ear density in field conditions, using images acquired from commercially available digital cameras. This system takes advantage of natural light in field conditions and, therefore, requires consideration of some related environmental factors, such as time of day and cloud cover, but remains, in effect, simple to implement. The system has been demonstrated on examples for durum wheat and barley but should be extendable in application to bread wheat, which, besides exhibiting ears with similar morphology, are frequently awnless, but further experiments would be required in order to confirm this. In the data capture protocol presented here, zenithal images are taken by simply holding the camera by hand or using a monopod for positioning the digital camera above the crop. Validation data can be acquired by counting the ears manually for subplots in the field or during postprocessing, by counting ears in the image itself. The image-processing algorithm is composed of three processes that, first, effectively remove unwanted components of the image in a manner that, then, allows for the subsequent segmentation and counting of the individual wheat ears in the acquired images. First, a Laplacian frequency filter is used in order to detect changes in the different spatial directions of the image using the default ImageJ filter settings without window kernel size adjustments (Find Maxima segmentation technique determines the local peaks after the median spatial filter step, at which stage the pixels related with ears have higher pixel values than soil or leaves. Therefore, Find Maxima is used to segment the high values in the image, and those regions are labeled as ears, which identifies ears while also reducing overlapping ear errors. Analyze Particles is then used on the binary images to count and/or measure parameters from the regions created by the contrast between the white and black surface created by the Find Maxima step. The result is then processed to create a binary image segmentation by analyzing the nearest neighbor pixel variance around each local maximum to identify the wheat ear shapes in the filtered image. Finally, the ear density is counted using Analyze Particles, as implemented in Fiji¹⁵. Both Find Maxima and Analyze Particles are standalone functions and available as plugins in Fiji (https://imagej.nih.gov/ij/plugins/index.html). Though not presented specifically in the protocol here, preliminary results presented as supplementary material suggest that this technique may be adaptable to conducting ear count surveys from unmanned aerial vehicles (UAVs), providing that the resolution remains sufficiently high¹⁴.

Protocol

1. Prefield crop growth stage and environmental conditions

1. Make sure that the crop growth stage is approximately between grain filling and near crop maturity, with ears that are still green even if the leaves are senescent (which corresponds in the case of wheat to the range 60-87 of Zadoks' scale¹⁶). Some yellowing of the leaves is acceptable but not necessary.
2. Prepare a sampling plan for image capture with various replicates (pictures per plot) in order to capture plot/area variability; the image-processing algorithm will count the number of ears in the image and convert that to ears per square meter (ears/m²) based on the camera specifications.
3. Plan the field excursions to capture the images within two hours of solar noon or, alternatively, on an overcast day in diffuse lighting conditions in order to avoid the negative effects of ear shadowing on the ear counting algorithm.
4. Once in the field, check the top of the crop canopy to make sure that it is dry in order to avoid specular light reflection from moisture.
   NOTE: In considering the objectives of this protocol, it is important to first consider whether the growth stage of the crop is suitable for applying ear counts. Capturing images outside of the recommended growth stage will either result in suboptimal or in meaningless results (if ears are not present or fully emerged). Image quality also has a considerable impact on processing the results, including resolution and sensor size, and some environmental conditions, such as time of day and cloud cover, need to be carefully considered before proceeding with image capture.

2. Image capture in field conditions with natural light

1. Prepare a "phenopole" as shown in Figure 1 or a similar acquisition system (even handheld) to capture images quickly and yet in a standardized and consistent manner at each plot or target location.

Figure 1: Ear counting system. Ear counting system using the "phenopole" shown in the field on the left, with a remotely controlled natural color (RGB) large sensor and high-resolution digital camera system with camera tilt and height, indicating the necessary parameters for adjusting the image-processing algorithm. The sensor and image resolution are detected automatically by the image properties, while the user should input the specifics for the lens focal length and the distance from the canopy. These are necessary to adjust the algorithm for the estimated number of pixels per ear and also the conversion of the image-based total ear count to ear density (ears/m²). For that reason, it is recommended to use the same camera and lens focal length for all field images. Please click here to view a larger version of this figure.

2. Position the camera on a suitable monopod or "selfie" pole so that it may be maintained level, either using level bubbles or an in-camera stabilization system, to obtain zenithal images.
3. Use a mobile phone, tablet, or another device to connect the camera for both remote control image capture and image visualization for the best results with correctly focused images. Program the camera for autofocus in order to reduce any errors in case the user is not familiar enough with their camera or photography techniques to set a correct manual focus, as demonstrated by the examples of zenithal images with correct focus and exposure in Figure 2.

Figure 2: Zenithal crop images. Durum wheat and barley ear zenithal images for ear counting data set examples with an acceptable stage of growth and senescence from approximately 61 to 87 according to Zadoks' scale. (Left) Durum wheat zenithal image data set example. (Right) Barley zenithal image data set example. Please click here to view a larger version of this figure.

4. Take note of the image number prior to image capture in order to match the images correctly with the field plots. Record one image of the general field area at the start and one image of the ground/field between blocks for postprocessing controls.
5. Position the camera at approximately 80 cm above the top of the crop canopy, using a ruler or measurement string to periodically check the camera height above the canopy. Ensure that the camera is level and capture the image. This technique may require 1-2 researcher(s).
6. If additional field ear count validation is desired apart from a manual image count validation, install an extension arm to the frame (e.g., a small circle) and position it in the middle of the image in order to conduct manual field counts of a precise image subset; this technique may require 2-3 people to implement.
   NOTE: Three major considerations in selecting a camera, therefore, include: (1) camera specifications; in this case, the sensor's physical size; (2) focal length of the image lens; (3) distance between the canopy and the camera: smaller distances or greater zoom lenses will capture a smaller area while images captured from a greater distance will capture a bigger crop area. See Figure 1 for the details on the relevant camera specifications.

3. Algorithm implementation and adjustments

NOTE: Here we present algorithm implementation and adjustments for different camera specifications (sensor size, megapixels, focal length, distance to crop) and crop (durum wheat or barley) for automatic ear counting. An overview of the algorithm is presented graphically in Figure 3.

Figure 3: Image-processing pipeline for two-row barley ear counting. Image-processing pipeline for two-row barley ear counting as implemented using specific computer code or using the CerealScanner software, both of which operate within Fiji (ImageJ). Panel 1 shows the original image. Panel 2 shows the results of the applications of the Laplacian filter. Panel 3 shows the application of the median filter, and Panel 4 shows the results of the final Find Maxima and segmentation for producing the final ear count. Then, the calculations are made to convert the image count to ear density, as shown in Figure 1. These images are an example taken from the Arazuri field site (Northeastern Spain, 42°48'33.9"N 1°43'37.9"W) in diffuse light conditions. Please click here to view a larger version of this figure.

1. Download and install Fiji, Java 8, and the processing code or the University of Barcelona’s proprietary CerealScanner plugin (https://fiji.sc/, https://www.java.com/en/download/, and https://integrativecropecophysiology.com/software-development/cerealscanner/ [information] or https://gitlab/sckefauver/CerealScanner [code repository]); contact the corresponding authors for access permissions. The plugin is installed within Fiji by simply copying it into the plugins folder.
2. Open the plugin from the top menu through Plugins > CerealScanner > Open Cereal Scanner.
   NOTE: Apart from the work presented here, the CerealScanner plugin includes several different RGB-based vegetation indices related to crop vigor, stress, or chlorophyll^17,18. The specific CerealScanner portion includes specific algorithms for Early Vigor (Fernandez-Gallego, In review), Ear Counting¹⁴, and Crop Senescence¹⁹, as shown in Figure 4.
3. Enter the adjustments of the camera specifications and image capture details if they are different from the default values (see Figure 1 and Figure 4 for details).
   1. Adjust the algorithm parameter for the camera focal length.
   2. Adjust the algorithm parameter for the distance from the crop canopy.

Figure 4: The CerealScanner 2.12 Beta central tab on both levels, marking the Ear Counting function within the CerealScanner algorithm collection. The user must select the … button to the right of Batch Inputs to select the folder where the image files are stored, change the default values of the H Distance (distance from the camera to the top of the crop canopy) and Focal Length, if different from the default values, and then select the … button to the right of Results File to choose the name and location of the final results file. The other tabs of the CerealScanner provide algorithms for trait-based phenotyping for Early Vigor and onset of Maturity as part of the CerealScanner code suite. Under the Biomass tab, there are several algorithms for estimations of more general crop vigor and biomass calculations, also for RGB digital images. The example refers to two-row barley, as it was demonstrated in detail in Figure 3.Please click here to view a larger version of this figure.

Figure 5: Algorithm adjustments. Adjustments required in the image-processing pipeline in order to successfully count both wheat and barley ears using the same algorithm are managed automatically as part of the camera-specific adjustments of H Distance (distance between the camera and the crop canopy) and Focal Length and serve to ensure that the number of pixels per ear remains more or less constant between different applications. Please click here to view a larger version of this figure.

4. Select the center tab CerealScanner and the subsequent central tab Ear Counting in order to calculate the number of ears in each image of a field data set.
   1. Under Options, enter, in Batch Inputs, the location of the photos to analyze.
   2. In the Results Files, select where to save the results file. The results file will include two columns with the image file name and the ear counting results.
   3. Finally, click on Process, and the results file with the ear density in square meters (ears/m²), using a simple ratio using the camera settings and the distance between canopy and camera to convert the image area to an actual canopy area in square meters following Figure 1, will be automatically produced in a few minutes, depending on the computer speed.
5. Conduct a post-processing validation after the data collection by manually counting the number of wheat or barley ears in the image and then converting this to the number of ears per square meters (ears/m²), as is shown in Figure 1, to compare the results with those based on the algorithm values.
   1. Use the simple point placement tool built within Fiji, which provides easy support for this process, and the Fiji Analyze Particles function for producing the counts automatically; this is shown graphically in Figure 6.
   2. Optionally, conduct a validation using a small area circle during field data acquisition as described in step 2.6; manual counts in the field and manual image counts in the laboratory can, then, be used for algorithm validation as shown in Figure 7.

Figure 6: Algorithm validation using manual in-image ear counts. Manual in-image ear counts for (left) durum wheat and (right) barley. The small dots were created using the Fiji Point Tool and then counted using the Analyze Particles Function with a 0.90-1.00 Circularity constraint after applying a Color Threshold from the Hue Saturation Intensity color space for the color specified by the Point Tool. This method ensures more accurate image-based manual ear counts. Please click here to view a larger version of this figure.

Figure 7: Algorithm validation using manual counts in the field and manual in-image ear counts of wheat and barley, using a circle. (Left) Wheat image count validation example using a circle. (Right) Barley image count validation example using a circle. The subset counts of the ears within the white circle were counted using the same technique as described in Figure 6 with the Point Tool, Color Threshold, and then, Analyze Particles Function with Circularity constraints and color selection using Hue. Please click here to view a larger version of this figure.

Representative Results

In Figure 8, the results show the determination coefficient between the ear density (number of ears per square meters) using manual counting and the ear counting algorithm for wheat and barley at three different crop growth stages. The first one is durum wheat with a Zadoks' scale between 61 and 65 (R² = 0.62). The second one is two-row barley with a Zadoks' scale between 71 and 77 (R² = 0.75), and the last one is durum wheat with a Zadoks' scale between 81 and 87 (R² = 0.75).

Figure 8: The coefficient of determination between ear density (number of ears/m²) using manual image-based counting and the image algorithm for ear counting of durum wheat and two-row barley at different acceptable crop growth stages (at Zadoks' scale 61-87). Both axes show calculations, including conversions to ear density, rather than image-based results only. The representative results are presented here for two different crops over three different growth stages, as well as under different light conditions, namely direct sunlight images of durum wheat at Zadoks' scale 61-65 on the top (R² = 0.62, n = 72), diffuse light images of barley at Zadoks' scale 71-77 in the middle (R² = 0.75, n = 30), and diffuse light conditions for durum wheat at Zadoks' scale 81-87 at the bottom (R² = 0.75, n = 24). An example image of each is also shown as an inset in the bottom right corner of each graph. Please click here to view a larger version of this figure.

Discussion

Increased agility, consistency, and precision are key to developing useful new phenotyping tools to assist the crop-breeding community in their efforts to increase grain yield despite negative pressures related to global climate change. Efficient and accurate assessments of cereal ear density, as a major agronomic component of yield of important staple crops, will help provide the tools needed for feeding future generations. Focusing on the improvement and support of crop-breeding efforts in field conditions helps keep this research and the techniques presented here more closely tied to real-world climate change scenarios and the needs of the breeding community but also presents technical difficulties. As such, it is important to pay close adherence not only to the data capture and image processing in this protocol but also to the recommendations for optimal environmental conditions and crop growth stages for its successful implementation¹¹. As a major agronomical component of yield, ear density is considered one of the most important target traits in the push for increasing cereal crop yields (see the article by Slafer et al.²⁰ and references therein). The focus in this protocol toward an optimally cost-efficient, agile, and straightforward field phenotyping technique considers that these aspects of the protocol are key to its adoption and implementation by the breeding community at large. In contrast, previous related studies with similar aims at assessing ear density or other yield component quantification have used more involved data capture and environmental control structures, such as enclosing structures and fixed camera supports or even artificial light inputs, that effectively impede practical application under field conditions and implementation in actual breeding programs^12,20,22.

Thus, we have presented here a detailed protocol that is the result of testing various different techniques in an iterative optimization process, resulting in a simple but effective image data capture method requiring only a commercially available, moderately high-resolution RGB digital camera and a rudimentary "phenopole" for holding the camera above the crop canopy. Other image-filtering attempts based on RGB color or alternative color spaces, such as hue-saturation-intensity or CIELAB, were not as effective or consistent as the use of the Laplacian and median frequency domain filters in removing unwanted image elements, especially the awns. We have designed the image capture pipeline system with different elements, some of which can be easily adjusted with minor changes in the image-processing algorithm implementation. In the case studies presented here, we have used two different compact cameras with relatively large sensors and 20.1 megapixels (MP) and 16.0 MP resolutions for capturing images with wide-angle lenses of 16-20 mm from a distance of 80 cm from the crop canopy. This has proven more than sufficient to produce detailed canopy barley and wheat information, with simulations demonstrating that the technique maintains high levels of precision down to approximately 8 MP¹⁴ (with similar lenses and distances from the canopy).

Although the consistency and precision of the presented image-processing techniques depend on the environmental conditions and phenological stage at the time of image capture, the algorithms show promise in providing a robust performance in their application to different small-grain cereals, including durum and bread wheat and two-row and six-row barley varieties. While this algorithm has yet to be fully tested, the image capture would the same, with perhaps some minor adjustments with regard to the relative size and position of the ear in the images in order to provide for optimal ear counting results. In the presented protocol, the image-based ear density estimates achieved maximal accuracy and correlated best with manual image counts and grain yield compared to images captured at later growth stages, when crop senescence resulted in a loss of color and illumination contrast between the rest of the crop canopy and the ears. This may be a result of higher temperatures or lower water availability at the later parts of grain filling, which are especially common in typical Mediterranean conditions, that can cause the leaves and culm to senesce before the ears do²³; this contrast is essential for the effectiveness of the separation between the ears, leaves, and soil. In overly mature or senescent canopies with the ears already yellow, the contrast between the different image elements is not adequate for ear counting. As such, in other climates, the optimal timing may be slightly different if there is no water stress during senescence onset.

Even though the data collection in field conditions requires close attention to such environmental conditions as sunlight intensity and sunlight illumination angles, the robust image analysis algorithm presented here provides some leeway in the image capture window by using spatial techniques that ignore image albedo effects, given that the correct image exposure was used for the particular light conditions at the moment of image capture; automatic settings have worked well in that respect. In previous work, a fuller range of lighting effects was tested, indicating that the only major source of error with regard to light effects is the production of strong shadows in the image when capturing images in direct sunlight either early or late in the day, due to the angle of the sun¹⁴. The first two image filter applications help to minimize any apparent effects from excess illumination (although not via camera overexposure) while also reducing any background components of the image; at the same time, these filters also contribute to smoothing and noise reduction, both of which aide in the subsequent Find Maxima process^23,24. Therefore, while the natural illumination factors must be accounted for, such as the angle of the sun when the images are taken in direct rather than in diffuse light conditions, this is mainly in order to reduce errors related to shadow artifacts.

Furthermore, correlations between algorithm ear counts from the presented protocol and grain yield were greater and more significant than manual (field-based) ear counts of the same experiment¹¹, which supports the claim that this protocol is not only more precise but also more consistent as a standardized protocol for the assessment of ear density. While not presented here specifically, similar data capture and processing techniques appears to be feasible using mobile phone, aerial or other automated platforms as they performs quite well under simulated reduced image resolutions. Additional tests for greyscale reduction (for faster image processing) and reduced resolution image simulations (from the application of other cameras or UAVs) were conducted by applying the image conversions before the first filter¹⁴ and suggested that, in optimal conditions, processing times may be reduced in these ways without any loss of accuracy. As for possible future directions, the image-processing algorithms presented here only take advantage of the VHR RGB color data as it is captured by the camera (similar to the human eye), but other potential improvements may result from the conversion to hybrid color spaces, such as hue-saturation-intensity, or through data fusion in combination with other more advanced scientific sensors, such as multispectral or thermal, which both have become more affordable recently and offer the potential for improvement in ear density estimations, although perhaps at different growth stages or in different field conditions.

In summary, the critical steps for the implementation of this protocol include first and foremost proper planning for the time of year and environmental conditions of the crop, which are optimal when the crop is in growth stages 60-87 of Zadoks' scale and either at solar noon or in diffuse light conditions. Furthermore, the acquisition of digital images should be conducted in a controlled manner accounting for camera angle, distance from the canopy, and camera focus for each image. Finally, optimized computer-processing options are presented in detail for reproducing the processing code pipeline, or contact the authors for either the original code or the code integrated as a graphical user interface (GUI) in a plug-in for Fiji, namely, the CerealScanner.

Materials

Name	Company	Catalog Number	Comments
ILCE-QX1 Camera	Sony	WW024382	Compact large sensor digital camera with 23.2 x 15.4 mm sensor size.
E-M10 Camera	Olympus	E-M10	Compact large sensor digital camera with 17.3 x 13.0 mm sensor size.
Multipod Monpod	Sony	VCT MP1	"Phenopole" in the JoVE article
Computer	Any PC/Mac/Linux	--	Data and image analysis
ImageJ/FIJI (FIJI is just Image J)	NIH	http://fiji.sc	Plug-in and algorithms for data and image analysis
Circle/Metal Ring	Generic	Generic	Metal ring for in-field validation
Crab Pliers Clip	Newer	90087340	Circle support and extension arm