Real-time Production Line Safety Monitoring Using Deep Learning-based Object Detection and Feature Enhancement

Summary

This paper proposes a production line safety monitoring method integrating You Only Look Once version 11 (YOLOv11) with convolutional neural networks. It achieved mean average precision at 0.5 intersection-over-union threshold (mAP@0.5) of 0.91, mAP@0.5:0.95 of 0.82, and 120 frames per second on a graphics processing unit, outperforming evaluated baseline models.

Abstract

With the deepening of Industry 4.0, the automation and intelligence levels of production lines have significantly improved, placing higher demands on the real-time performance, accuracy, and safety of monitoring. Traditional monitoring systems, relying on manual inspections or simple threshold-based decisions, generally suffer from slow response times, high false alarm rates, and limited intelligence. Therefore, this paper proposes a production-line safety monitoring system that integrates You Only Look Once version 11 (YOLOv11) with a convolutional neural network (CNN). First, the acquired images were preprocessed. Then, YOLOv11 was used to identify workers, equipment, and potential hazards in real time. Next, an enhanced CNN network with multi-scale feature fusion and attention mechanisms was introduced to improve the feature extraction capabilities for small and occluded targets. Finally, the detection results were fused with the CNN-enhanced features to assess safety status. Experiments were conducted using a self-built production-line safety dataset, employing the stochastic gradient descent (SGD) optimizer with momentum 0.9, an initial learning rate of 0.01, weight decay of 0.0005, cosine-annealed learning rate adjustment, a batch size of 32, and training for 200 epochs. mAP@0.5 and detection speed in frames per second (FPS) were used as evaluation metrics for comparison with the evaluated baseline algorithms. The results show that the proposed system achieved an mAP@0.5 of 0.91 and a detection speed of 120 FPS. It also demonstrated robust performance under complex conditions such as shading and varying lighting, supporting its effectiveness and practical application potential in production line safety monitoring.

Introduction

In modern industrial production, safety is the cornerstone of ensuring production efficiency and personnel well-being. The production line environment typically involves high-speed mechanical equipment, complex technological processes, and collaborative operations that involve multiple tasks. Any small potential safety hazard can lead to serious production accidents, resulting in significant economic losses and even fatalities. Therefore, it is crucial to establish an efficient, intelligent, and real-time safety monitoring system to enhance safety in industrial production^1,2.

Traditional safety monitoring methods mainly rely on manual video surveillance and various sensors (such as infrared, smoke, and vibration sensors)³. Manual monitoring is not only inefficient but also prone to missed detections due to monitoring personnel fatigue, and it cannot provide continuous and comprehensive coverage. Although sensors can provide a certain early warning function, their detection range is limited, and they are sensitive to environmental interference, resulting in prominent problems of false alarms⁴. Intelligent monitoring systems have become a growing focus of research. Real-time analysis of a video stream using a deep learning algorithm can automatically identify abnormal events, unsafe behaviors, and potential hazards, thereby greatly improving the monitoring system's intelligence^4,5.

Object detection is a core technology in intelligent monitoring. As representatives of single-stage object detectors, the You Only Look Once (YOLO) series algorithms demonstrate significant advantages. From YOLOv1 to YOLOv8, this collection of algorithms has undergone ongoing development, with enhancements to the network architecture, loss function, and training methodology, among other aspects^6,7. YOLOv11, in particular, has achieved higher detection accuracy while preserving a high frame rate, particularly when confronted with complex backgrounds and densely packed targets⁸.

However, despite its excellent performance on general object detection tasks, YOLOv11 still faces challenges in specific production-line safety-monitoring scenarios⁹. For example, YOLOv11 detection accuracy may decrease when workers are partially obscured by equipment or when safety signs are small and closely resemble the background color. This requires the model to have stronger feature extraction and semantic understanding capabilities¹⁰.

Convolutional neural networks (CNNs) have powerful capabilities in image feature extraction¹¹. By designing deeper and more complex network architectures, CNNs can learn richer, more abstract image features. Combining a CNN with YOLOv11 leverages YOLOv11's fast detection capabilities and the CNN's deep feature extraction, thereby building a more robust and intelligent safety monitoring system.

Based on this, this paper proposes a production-line safety monitoring system using YOLOv11 and a CNN. The innovative aspects of this study include: (1) proposing a deep fusion architecture of YOLOv11 and an improved CNN; (2) introducing multi-scale feature fusion and a composite attention mechanism to enhance the recognition of key safety features; and (3) verifying the performance advantages of the model on a self-built complex scene dataset.

Development of YOLO series algorithms
Since its initial introduction by Redmon et al. in 2015¹², the You Only Look Once (YOLO) algorithm has garnered considerable interest. Unlike two-stage detectors that rely on region proposals, YOLO treats object detection as a regression task. By directly predicting the classes and positions of objects in an image, YOLO markedly boosts detection speed, making it suitable for real-time use¹³.

As the pioneering work in this series, YOLOv1 achieves end-to-end target detection for the first time¹⁴. YOLOv1 involves partitioning the input image into a grid structure, where each grid cell, typically arranged in an S × S format, is tasked with detecting objects that fall within its boundaries.

Specifically, YOLOv1's output is a tensor. Among the S × S × (B × 5 + C), the prediction of each bounding box contains 5 elements: center point coordinate (x,y), width w, height h, and confidence c. The calculation process is illustrated in Equation (1):
   (1)

Among them, Pr(Object) represents the probability that there is a target in the grid, and IOU represents the ratio of the intersection over the union between the predicted bounding box and the ground-truth box. Each grid also predicts a set of class probabilities, Pr representing the probability that the target belongs to the i-th class in the presence of a target. The loss function of YOLOv1 is composed of three main components: the loss for coordinate prediction, the loss for confidence estimation, and the loss for category prediction¹⁵.

YOLOv2 introduces Batch Normalization, a high-resolution classifier, and a multi-scale training strategy, which improve stability and accuracy¹⁶. YOLOv3 uses Darknet-53 as its backbone network and draws inspiration from the Feature Pyramid Network (FPN) concept to enhance target detection¹⁷.

YOLOv4 has made numerous optimizations based on YOLOv3, introducing technologies such as Cross Stage Partial Network (CSPNet)¹⁸, Path Aggregation Network (PANet)¹⁹, and Mosaic data enhancement to further improve the model's performance. YOLOv5 has made significant contributions to engineering, offering an easier-to-use and more efficient implementation²⁰.

In recent years, the YOLO series has continued to evolve, with versions such as YOLOv6, YOLOv7, and YOLOv8 being released one after another. By introducing the Extended Efficient Layer Aggregation Network (E-ELAN) structure and model re-parameterization techniques, YOLOv7 achieves new breakthroughs in both speed and accuracy. These enhancements not only improve the efficiency of feature learning but also optimize the network's inference performance, enabling YOLOv7 to surpass previous versions and many mainstream detectors across various real-time object detection tasks. YOLOv8 further optimizes the network structure, adopts an anchor-free design, simplifies the model, and improves its generalization ability²¹.

Building on the advantages of previous versions, YOLOv11, used in this study, has been optimized for complex industrial scenarios. Its core improvements include a feature-extraction network, a more efficient feature-fusion module, and a more robust loss-function design. These improvements enable YOLOv11 to perform better when handling occlusions, small targets, and complex backgrounds in production environments. YOLOv11 is a version of the YOLO series released by Ultralytics. Compared with YOLOv8, it improves the C3K2 module and the feature fusion path, and introduces an adaptive anchor box strategy, thereby providing stronger detection robustness in industrial scenarios.

Convolutional Neural Network (CNN) foundation and progress
The convolutional neural network (CNN) is a pivotal model that has profoundly influenced the success of deep learning within computer vision. It operates by extracting local image features via convolutional operations and subsequently reducing feature dimensionality via pooling operations, thereby achieving hierarchical image abstraction²².

A typical CNN is composed of multiple convolutional, pooling, and fully connected layers. The convolutional layer scans the input image with a convolution kernel, computes dot products over local regions, and produces a feature map. The calculation process is shown in Equation (2):

   (2)

Where x is the input image, w^k and b^k are the weight and bias of the convolution kernel, respectively, and are the values of the output feature map at the position (i,j). The pooling layer commonly uses Max Pooling or Average Pooling. The fully connected layer is responsible for extracting features and predicting classification probabilities.

Based on this, this paper designs a CNN feature enhancement module for YOLOv11, which consists of two parallel branches: a multi-scale convolution branch that uses 3 × 3 and 5 × 5 convolution kernels to extract multi-scale features; and an attention branch that sequentially connects the channel attention (Squeeze-and-Excitation) and spatial attention modules to enhance the discriminative features of key targets. The outputs of the two branches are fused by element-wise addition, and the corresponding feature enhancement loss function is shown in formula (3):

  (3)

L_coord is the bounding box coordinate regression loss; L_conf is the target confidence loss; L_cls is the category classification loss; L_feat is the feature enhancement loss, used to constrain the discriminative features of small targets and occluded targets extracted by the CNN enhancement module; λ_coord, λ_conf, λ_cls, λ_feat are the weight coefficients of the corresponding loss terms. For this task, λ_feat = 0.05 is set, and the remaining weights are balanced according to the detection task requirements.

Classical CNN structures, such as VGGNet²³ and ResNet²⁴, have achieved great success on image classification tasks. Among these architectures, ResNet effectively mitigates the vanishing gradient problem in deep network training, thereby facilitating the construction of deeper and more complex network structures.

In object detection tasks, CNN is usually used as a feature extractor (backbone). For example, Darknet, cross-stage partial Darknet (CSPDarknet), etc., in the YOLO series of algorithms are all built upon CNN²⁵. To enhance feature extraction capabilities, researchers have proposed numerous improved CNN structures. For example, the Inception module extracts features in parallel through multi-scale convolution kernels. DenseNet enhances feature reuse with dense connections. The Squeeze-and-Excitation Network adaptively adjusts the importance of feature channels through attention mechanisms²⁶.

In this study, we designed an improved CNN module to enhance the feature extraction capabilities of YOLOv11. This module combines multi-scale feature fusion with an attention mechanism to more effectively capture key safety information in production line scenarios.

Application status of deep learning in industrial safety monitoring
Deep learning has gained significant traction in industrial safety monitoring. CNN-based algorithms are employed to determine whether workers are properly wearing safety helmets, detect unauthorized intrusions into hazardous zones, and monitor the condition of malfunctioning equipment²⁷.

In terms of behavior recognition, 3D CNNs and recurrent neural networks are used to analyze workers' operational behavior and identify potentially unsafe actions. For example, by analyzing workers' posture sequences, it can be determined whether they are violating safety operating procedures²⁸.

YOLO and Faster R-CNN are widely used for personnel and equipment detection in real-time surveillance videos. For example, a real-time helmet detection system based on YOLOv5 has been proposed in the literature, effectively enhancing the intelligence of construction-site safety management²⁹.

Although some progress has been made in existing research, several challenges remain. First, industrial scenes typically feature complex illumination, occlusion, and background interference, which impose stringent requirements on the algorithm's robustness. Second, real-time performance is a key requirement, and the algorithm must achieve high-speed inference while maintaining accuracy. Finally, existing research primarily focuses on single-task detection and lacks exploration of multi-source information fusion and comprehensive analysis³⁰.

In this study, the proposed YOLOv11 and CNN fusion model aims to address the challenges above and achieve comprehensive, real-time monitoring of production line safety risks by enhancing feature extraction and fusion capabilities.

Protocol

YOLOv11 and CNN algorithm

System overall architecture
The production line safety monitoring system proposed in this paper adopts a hybrid architecture that combines YOLOv11 with an improved CNN to achieve high-precision, high-speed real-time safety monitoring. As shown in Figure 1, the system mainly consists of an input preprocessing module, a YOLOv11 object detection module, a CNN feature enhancement module, and a fusion decision module. The input image is first normalized and augmented by the preprocessing module to improve the model's generalization ability. Subsequently, features are extracted by the CSPDarknet backbone, and the receptive field is expanded by the spatial pyramid pooling-fast (SPPF) module. Multi-scale features are fused after upsampling, and channel attention and spatial attention are applied sequentially to the fused features to further enhance the discriminative representation of key targets. The CNN feature enhancement module is inserted after the outputs of the C3, C4, and C5 layers of the YOLOv11 backbone network, receiving multi-scale feature maps with sizes of 80 × 80 × 256, 40 × 40 × 512, and 20 × 20 × 1,024, respectively. The CNN feature enhancement module further enhances feature extraction for small and occluded targets. Finally, the fusion decision module combines the detection results of YOLOv11 with the CNN-enhanced features and jointly optimizes them through a designed loss function to output the precise location, category, and confidence score of the target.

YOLOv11 detection framework
YOLOv11 has introduced several critical enhancements that leverage the strengths of the YOLO series' single-stage detection framework. Its architecture is divided into three main components: a backbone network, a feature fusion network, and a detection head. The backbone network utilizes an enhanced CSPDarknet structure. The feature fusion network incorporates cross-stage local connections and depthwise separable convolutions, drawing inspiration from feature pyramid networks and path aggregation networks to effectively merge multi-scale features.

Feature enhancement module
The feature enhancement module aims to improve the model's sensitivity to key safety features. It processes the feature maps produced by each layer of the YOLOv11 backbone network, fusing information across different scales through upsampling and downsampling operations. Specifically, the multi-scale branch captures spatial scale diversity, while the attention branch dynamically enhances key channels and positional responses. These two branches do not act independently but are complementary and fused through residual connections and feature recalibration. The feature map processed by the enhancement module at each scale is adjusted to match the original feature map's number of channels via a 1 × 1 convolution, then fused with the original YOLOv11 feature map via element-wise addition. The fused feature map is then fed into the subsequent Feature Pyramid Network (FPN) for multi-scale fusion, thereby forming a hierarchical safety feature representation. To ensure seamless integration between modules, an end-to-end joint training strategy is adopted, with the loss function consisting of the YOLOv11 detection loss and the CNN module feature enhancement loss.

Figure 1. YOLOv11-CNN algorithm. The feature enhancement module aims to amplify the model's sensitivity to critical safety features. It processes feature maps from various layers of the YOLOv11 backbone, fusing features of different scales through upsampling and downsampling operations. Additionally, it incorporates channel- and spatial-attention mechanisms. To ensure seamless integration between the YOLOv11 and CNN modules, a joint training strategy is employed. During training, the parameters of both modules are optimized end-to-end. The loss function combines the detection loss of YOLOv11 with the feature enhancement loss of the CNN module. Please click here to view a larger version of this figure.

Results

To verify the effectiveness of the proposed YOLOv11 and CNN fusion model in production line safety monitoring, a dataset covering various safety hazard scenarios was constructed. This dataset contains 12,000 production line scene images, divided into training, validation, and test sets in a 7:1.5:1.5 ratio, with a fixed random seed of 42. It covers various working conditions, including standard lighting, low lighting, partial occlusion, heavy occlusion, motion blur, and complex backgrounds. The confidence threshold for i...

Discussion

Experimental results show that the proposed YOLOv11–CNN fusion model improves detection accuracy and real-time performance for production line safety monitoring. Leveraging the efficient single-stage detection of YOLOv11 and the feature enhancement of the CNN module, the system identified workers, equipment, and hazardous areas under complex lighting and occlusion conditions. Multi-scale feature fusion and attention mechanisms enhanced the ability to focus on key safety elements, providing interpretable visual evid...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
COCO API		N/A	Assessment tool used for target detection accuracy evaluation and statistical analysis.
CUDA 11.4	NVIDIA	N/A	Parallel computing platform used with the PyTorch 1.12 framework.
Edge TPU		N/A	Deployment platform used for latency and power consumption testing; reported latency was 18 ms and power consumption was 15 W.
Jetson Xavier edge computing device	NVIDIA		Edge computing device used for throughput testing; reported processing speed was 70 FPS with 99th percentile latency within 50 ms.
NVIDIA Tesla V100 GPU	NVIDIA		GPU used in the training/evaluation server.
PyTorch 1.12		N/A	Deep learning framework used for model training and evaluation.
scikit-learn		N/A	Assessment/statistical analysis tool used for model evaluation.
Self-built production line safety dataset	N/A	N/A	Dataset of 12,000 production line scene images in VOC format, covering standard lighting, low lighting, partial occlusion, heavy occlusion, motion blur, and complex backgrounds; six annotation categories: workers, safety helmets, robotic arms, hazardous areas, tools, and vehicles.
Training server			Server equipped with an NVIDIA Tesla V100 GPU and Ubuntu 20.04 operating system.
Ubuntu 20.04 operating system		N/A	Operating system used on the training/evaluation server.
VOC annotation format	N/A	N/A	Annotation format used for the self-built production line safety dataset.
YOLOv11 object detection model	Ultralytics	N/A	Object detection model used to detect workers, equipment, and potential hazards in real time.