A Two-Stage Self-Supervised Learning Framework for Winter Crop-Weed Image Classification

Summary

This work assesses the application of a two-stage deep learning pipeline for the self-supervised pretraining and supervised fine tuning of winter crop and weed image classification. The experiments on the WinterCropWeedDB dataset are conducted using a single internal split, with Grad-CAM visualizations included.

Abstract

Precision agriculture requires accurate discrimination between winter crops and weeds, but there is a lack of annotated image data for winter cropping systems. This paper investigates a two-stage deep learning approach that integrates self-supervised feature learning with supervised fine-tuning for winter crop and weed image classification. A new winter crop and weed image dataset, WinterCropWeedDB, is proposed and used in this paper, which contains 1,136 high-resolution images of six winter crop species and four weed species collected from agricultural fields in central India. In the first stage of self-supervised learning, an EfficientNet-B3 model is pre-trained using a SimCLR-style self-supervised learning approach with an InfoNCE loss function (temperature τ = 0.5) on the images. The average contrastive loss value reduces from 2.0712 in the first iteration to 1.6835 at the end of pretraining. In the second stage of supervised fine-tuning, the pre-trained EfficientNet-B3 model is fine-tuned with a supervised classifier head on the images and tested on a single internal validation split (30%) of the dataset. The fine-tuned model reaches a maximum validation accuracy of 98.27%, with a macro-averaged F1-score of 0.98. Gradient-weighted class activation mapping (Grad-CAM) and Grad-CAM++ are used on the fine-tuned model to provide a qualitative visualization of the image regions that contribute to class predictions. The experiment outcomes demonstrate the viability of using self-supervised pretraining and supervised fine-tuning for the classification of winter crop and weed images on a region-specific dataset, while also emphasizing the importance of additional testing on independent test sets.

Introduction

Precision agriculture has seen the increasing use of deep learning-based computer vision methods for automated crop and weed classification¹. Convolutional neural networks have been shown to be effective in plant recognition tasks; however, their performance is usually dependent on the availability of large datasets that are annotated in a precise manner². In most agricultural settings, especially in underrepresented cropping systems like winter crops, the process of obtaining extensive labeled image data is time-consuming, laborious, and expensive^3,4. This is a hindrance to the development of decision support systems that are data-driven for winter agroecosystems. Self-supervised learning (SSL) is a paradigm that has recently shown promise for representation learning, where it uses a large number of images to learn features that are informative even though they are not labelled⁵. Through the design of pretext tasks, such as contrastive learning, SSL allows neural networks to learn structural and semantic patterns in image data, which can then be transferred to supervised learning tasks⁶. Prior studies in agricultural imaging have shown that SSL-based pretraining can improve downstream performance when labeled data are limited, motivating its exploration for crop and weed recognition problems^1,7.

Concurrently, semi-supervised learning and contrastive learning methods have also been explored. Semi-supervised learning methods that leverage both labeled and unlabeled data, like teacher-student learning and pseudo-labeling, have been explored for weed classification and detection^2,4. More recently, contrastive learning methods with class-aware objectives have shown the potential to learn transferable representations from agricultural images, especially in scenarios with class imbalance or few annotations^1,3. Although these studies indicate the potential benefits of semi-supervised learning and related techniques over fully supervised learning, the majority of the existing literature has been limited to summer cropping systems, object detection, or large-scale datasets.

Apart from the predictive accuracy, the interpretability of the internal decision-making mechanisms of deep neural networks is still a relevant concern for practical agricultural applications. Explainable artificial intelligence (XAI) techniques are designed to offer interpretable explanations of model predictions, thus attempting to fill the gap between model predictions and human expertise⁸. Gradient-based visualization algorithms, such as Gradient-weighted class activation mapping (Grad-CAM) and its extension Grad-CAM++^9,10, produce class-discriminative heatmaps that point out the regions of interest in an image that are most relevant to a model's predictions. These algorithms are commonly employed for the qualitative analysis of learned representations in agricultural image analysis tasks, but do not represent a formal validation of interpretability.

The primary aim of this research work is to investigate the potential of a two-stage learning framework for the classification of winter crop and weed images using self-supervised representation learning and supervised fine-tuning, along with qualitative visualization. In this research, the application of a two-step learning pipeline for winter crop and weed image classification, integrating self-supervised pretraining with SimCLR and supervised fine-tuning with EfficientNet-B3 was investigated. The experiments are performed on the WinterCropWeedDB dataset, a region-specific dataset of winter crop and weed images collected from agricultural fields in central India¹¹. The results are analyzed based on a single internal split, and Grad-CAM and Grad-CAM++ are used for visualizing the attention of the model. The aim of this research is to investigate the possibility of combining self-supervised representation learning and visualization methods for winter crop and weed image classification, while also being aware of the limitations of the experimental setup. This workflow is meant for exploratory assessment on region-specific crop-weed datasets with limited labeled data and is not intended as a validated benchmark for generalization or large-scale use.

Protocol

Dataset description

The WinterCropWeedDB dataset consists of 1,136 high-resolution (red, green, blue) RGB images of six winter crop species (wheat, chickpea, pea, lentil, mustard, and grass pea) and four weed species (common vetch, lesser canary grass, goosefoot (Chenopodium album), and Euphorbia clementei) taken from winter agricultural fields in the state of Chhattisgarh, India (Figure 1). The images were taken under natural conditions, including varying illumination, growth stages, and background complexity. All images were initially unannotated and used only for self-supervised pretraining. For supervised fine-tuning, the images were manually annotated and split using stratified sampling into training (70%) and validation (30%) sets. The validation set was used only for internal model evaluation and was not augmented. To address class imbalance and robustness issues during training, data augmentation was performed only on the training set. The augmentation techniques involved random rotations (+/-20°), horizontal flipping, scaling, translation, brightness changes, and mild affine transformations. The training samples were augmented to a target of 150 images per class, resulting in a total of 1,500 training images, while the validation set consisted of 347 original images. No augmented or synthetic images were included in the validation set to prevent evaluation bias. Besides the hold-out evaluation, a stratified five-fold cross-validation was also performed on the original labeled dataset as a secondary analysis. In each fold, data augmentation was done only on the training sets, and the validation sets were kept unchanged to maintain consistency with the main evaluation scheme.

Computational workflow for self-supervised and supervised model training

A two-step computational pipeline was used to investigate the viability of integrating self-supervised representation learning and supervised fine-tuning for winter crop vs. weed image classification (Figure 2). This pipeline involves: (i) self-supervised pretraining with unlabeled images, and (ii) supervised fine-tuning with a labeled sample of the images. All images were resized to 300 × 300 pixels and normalized before training. Evaluations of the models were conducted only on one internal split, without using an external test set.

Stage 1 - self-supervised pretraining

In the initial stage, the EfficientNet-B3 architecture was employed as the backbone network in a self-supervised learning setup inspired by SimCLR. The two-layer projection head reduced the backbone representation to a 128-dimensional embedding space. For self-supervised learning, each image was converted into two views using random resized cropping, horizontal flipping, color transformation, Gaussian blurring, and converting to grayscale. The two views were processed together to jointly optimize the contrastive loss function. The self-supervised learning process was performed for 30 epochs, where one epoch denotes a complete pass of the entire training dataset through the model during optimization, with a batch size of 16 images using the Adam optimizer (an adaptive gradient-based optimization algorithm that estimates first- and second-order moments of the gradients to adjust learning rates during training). Gradient clipping with a maximum norm of 1.0 was used to ensure stable training. Additionally, model checkpoints were saved after every epoch to facilitate reproducibility. The temperature value of 0.5 was used while computing the InfoNCE loss during self-supervised learning.

Stage 2 - supervised fine-tuning

After self-supervised pretraining, the projection head was removed, and the pretrained backbone weights were used for supervised fine-tuning. A fully connected classifier head was added to the backbone for multi-class classification. Fine-tuning was done using only the labeled images from the training set. Supervised training used cross-entropy loss with class weighting and label smoothing to handle class imbalance. The Adam optimizer was used with different learning rates for the backbone (1 × 10⁻⁵) and classifier (1 × 10⁻⁴). A learning rate scheduler was used to decrease the learning rate when the validation accuracy plateaued. Training was done for 20 epochs, and the model checkpoint with the highest validation accuracy was saved for evaluation.

In addition to the main hold-out evaluation, a stratified five-fold cross-validation was carried out as an additional analysis on the labeled set. In each fold, the augmentation was only done on the training sets, and the validation sets were left unchanged. The results of the cross-validation were presented separately and were not used for model selection or optimization of the checkpoints. The performance metrics presented in this study are based solely on the internal validation split and represent the results of the observed performance under the current experimental setup.

Grad-CAM and Grad-CAM++ visualization

For the qualitative analysis of attention in the model, gradient-based class activation mapping methods (Grad-CAM and Grad-CAM++) were used on the fine-tuned model. The feature maps and the gradients were obtained from the final convolutional layer of the base network, and class-specific heatmaps were obtained for chosen validation images. These were used only for the purpose of qualitative analysis of the model and were not validated. The Table of materials lists all the hardware, software libraries, datasets, and custom training scripts used in this workflow.

Results

Overview of prior approaches and workflow positioning

A summary of representative learning approaches applied previously for agricultural crop-weed recognition is given in Table 1. The table presents an overview of supervised, semi-supervised, and self-supervised learning strategies, datasets used, reported results, and the limitations presented. Most earlier works focus on summer cropping systems or object detection tasks and are evaluated on relatively large...

Discussion

This work examined the use of a two-step deep learning approach consisting of self-supervised pretraining using the SimCLR idea and supervised fine-tuning for winter crop and weed image classification on the WinterCropWeedDB dataset. The results show that the proposed approach can be trained reliably on a region-specific winter agriculture image dataset and tested on one split of the internal validation set. This section presents an analysis of the results, implications, and limitations of this work.

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
CUDA Toolkit	NVIDIA	12.8
cuDNN	NVIDIA	9.1
Graphics processing unit (GPU)	NVIDIA	GeForce RTX 5050 Laptop GPU
Matplotlib	Matplotlib Developers	3.9.2
NumPy	NumPy Developers	1.26.0
Operating system	Microsoft	Linux (WSL2), kernel 6.6.87
Python	Python Software Foundation	3.12.7
PyTorch	PyTorch Foundation	2.1.0 (development build)
scikit-learn	scikit-learn Developers	1.5.1
timm	GitHub repository	1.0.24
Torchvision	PyTorch Foundation	0.25.0 (development build)
WinterCropWeedDB	Mendeley Data, DOI: 10.17632/m4h6zdsh79.1	Version 1