Name: End-To-End Deep Neural Network for Salient Object Detection in Complex Environments (Video) | JoVE Journal
Uploaded: 2023-12-15T13:10:00.000+00:00
Duration: 3 min 31 s
Description: Explore deep neural networks for salient object detection in complex environments using convolutional architectures. This computational method improves accuracy for smart grid insulator inspection applications.

ai_agent_IP

ai_agent

maintenance

instantTrialDurationMinutes

isBiopharma

enableNewMyJoveSectionDesign

educationFeatures

seoMetaApi

ai_copilot_institution_blacklist

ai_chat_institution_blacklist

freeAccessWindow

ssoRoleCapture

scheduledMaintenanceBanner

joveLabsFeatures

disable_autoplay_for_bots

extended_trial_duration

5 minutes

This work is supported by 2024 Henan Provincial Higher Education Institutions Key Scientific Research Project Funding Program Establishment (Project Number:24A520053). This study is also supported by Specialized Creation and Integration Characteristic Demonstration Course Construction in Henan Province.

1. Experimental setup and procedure
<ol>
	<li>Load the pre-trained VGG16 model. 
	NOTE: The first step is to load the pre-trained VGG16 model from the Keras library6.
	<ol>
		<li>To load a pre-trained VGG16 model in Python using popular deep learning libraries like PyTorch (see Table of Materials), follow these general steps:
		<ol>
			<li>Import torch. Import torchvision.models as models.</li>
			<li>Load the pre-trained VGG16 model. vgg16_model = models.vgg16(pretrained=True).</li>
			<li>Ensure that the summary of the VGG16 model is &#34;print(vgg16_model)&#34;.</li>
		</ol>
		</li>
	</ol>
	</li>
	<li>Define the DCL and DEDN models.
	<ol>
		<li>For the pseudo-code of the DCL algorithm, provide Input: Image dataset SOD and Output: Trained DCL model.
		<ol>
			<li>Initialize the DCL model with the VGG16 backbone network.</li>
			<li>Preprocess the image dataset D (e.g., resize, normalization).</li>
			<li>Split the dataset into training and validation sets.</li>
			<li>Define the loss function for training the DCL model (e.g., binary cross-entropy).</li>
			<li>Set the hyperparameters for training: Learning rate (0.0001), Number of training epochs set (50), Batch size is (8), Optimizer (Adam).</li>
			<li>Train the DCL model: for each epoch in the defined number of epochs, do for each batch in the training set. Input the following:
			<ol>
				<li>Forward pass: Feed batch images to the DCL model. Compute the loss using the predicted saliency maps and ground truth maps.</li>
				<li>Backward pass: Update the model parameters using gradient descent end. Compute the validation loss and other evaluation metrics on the validation set end.</li>
			</ol>
			</li>
			<li>Save the trained DCL model.</li>
			<li>Return the trained DCL model.</li>
		</ol>
		</li>
		<li>For pseudo-code for the DEDN algorithm, input: Image dataset (X), Ground truth saliency maps (Y), Number of training iterations (N).
		<ol>
			<li>For the Encoder Network, ensure that the encoder is based on the VGG16 skeleton with modifications (as mentioned below). 
			NOTE: encoder_input = Input(shape=input_shape) 
			encoder_conv1 = Conv2D(64, (3, 3), activation=&#39;relu&#39;, padding=&#39;same&#39;)(encoder_input) 
			encoder_pool1 = MaxPooling2D((2, 2))(encoder_conv1) 
			encoder_conv2 = Conv2D(128, (3, 3), activation=&#39;relu&#39;, padding=&#39;same&#39;)(encoder_pool1) 
			encoder_pool2 = MaxPooling2D((2, 2))(encoder_conv2) 
			encoder_conv3 = Conv2D(256, (3, 3), activation=&#39;relu&#39;, padding=&#39;same&#39;)(encoder_pool2) 
			encoder_pool3 = MaxPooling2D((2, 2))(encoder_conv3)</li>
			<li>For the Decoder Network, ensure that the decoder is based on the VGG16 skeleton with modifications (as mentioned below). 
			NOTE: decoder_conv1 = Conv2D(256, (3, 3), activation=&#39;relu&#39;, padding=&#39;same&#39;)(encoder_pool3) 
			decoder_upsample1 = UpSampling2D((2, 2))(decoder_conv1) 
			decoder_conv2 = Conv2D(128, (3, 3), activation=&#39;relu&#39;, padding=&#39;same&#39;)(decoder_upsample1) 
			decoder_upsample2 = UpSampling2D((2, 2))(decoder_conv2) 
			decoder_conv3 = Conv2D(64, (3, 3), activation=&#39;relu&#39;, padding=&#39;same&#39;)(decoder_upsample2) 
			decoder_upsample3 = UpSampling2D((2, 2))(decoder_conv3) 
			decoder_output = Conv2D(1, (1, 1), activation=&#39;sigmoid&#39;, padding=&#39;same&#39;)(decoder_upsample3)</li>
		</ol>
		</li>
		<li>Define the DEDN model. model = Model (inputs = encoder_input, outputs = decoder_output).</li>
		<li>Compile the model. model.compile (optimizer = adam, loss = binary_crossentropy).</li>
		<li>Select the Training loop. 
		&#8203;NOTE: For iteration in range(N): # Randomly select a batch of images and ground truth maps; batch_X, batch_Y = randomly_select_batch(X, Y, batch_size).
		<ol>
			<li>Train the model on the batch. loss = model.train_on_batch(batch_X, batch_Y). Print the loss for monitoring.</li>
		</ol>
		</li>
		<li>Save the trained model. model.save (&#39;dedn_model.h5&#39;).</li>
	</ol>
	</li>
	<li>Combine.
	<ol>
		<li>Combine the outputs of the DCL and DEDN networks and refine the saliency map using a fully connected conditional random field (CRF) model.</li>
	</ol>
	</li>
</ol>
2. Image processing
<ol>
	<li>Click on run code to bring up the GUI interface (Figure 4).</li>
	<li>Click on open image to select the path and thus the image to be detected.</li>
	<li>Click on the display image to display the image that has been selected for detection.</li>
	<li>Click on start detection to detect the selected image. 
	NOTE: Detection result will appear with the detected image, i.e., the result of the salient object (Figure 5).</li>
	<li>Click on select the save path to save the image results of the salient object detection.</li>
</ol>

Enhanced Precision in Salient Object Detection Using Deep Neural Networks

<ol><li>Wang, W. G., Shen, J. B., Ling, H. B. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((A+deep+network+solution+for+attention+and+aesthetics+aware+photo+cropping%5BTitle%5D)+AND+%22IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%22%5BJournal%5D)">A deep network solution for attention and aesthetics aware photo cropping.</a> IEEE Transactions on Pattern Analysis and Machine Intelligence. 41 (7), 1531-1544 (2018).</li>
<li>Wang, W. G., Sun, G. L., Gool, L. V. Looking beyond single images for weakly supervised semantic segmentation learning. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aWang%2C+WG+%22IEEE+Transactions+on+Pattern+Analysis+and+Machine%22">IEEE Transactions on Pattern Analysis and Machine</a>. , (2022).</li>
<li>Mei, H. L., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Exploring+dense+context+for+salient+object+detection%5BTitle%5D)+AND+%22IEEE+Transactions+on+Circuits+and+Systems+for+Video+Technology%22%5BJournal%5D)">Exploring dense context for salient object detection.</a> IEEE Transactions on Circuits and Systems for Video Technology. 32 (3), 1378-1389 (2021).</li>
<li>Borji, A., Itti, L. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((State-of-the-art+in+visual+attention+modeling%5BTitle%5D)+AND+%22IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%22%5BJournal%5D)">State-of-the-art in visual attention modeling.</a> IEEE Transactions on Pattern Analysis and Machine Intelligence. 35 (1), 185-207 (2012).</li>
<li>Fully convolutional networks for semantic segmentation. Long, J., Shelhamer, E., Darrell, T. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, , 3431-3440 (2015).</li>
<li>Simonyan, K., Zisserman, A. Very deep convolutional networks for large-scale image recognition. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aSimonyan%2C+K+%22arXiv+preprint%22">arXiv preprint</a>. , 1409-1556 (2014).</li>
<li>Deep residual learning for image recognition. He, K., Zhang, X., Ren, S., Sun, J. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, , 770-778 (2016).</li>
<li>Dhsnet: Deep hierarchical saliency network for salient object detection. Liu, N., Han, J. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, , 678-686 (2016).</li>
<li>A bi-directional message passing model for salient object detection. Zhang, L., Dai, J., Lu, H., He, Y., Wang, G. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, , 1741-1750 (2018).</li>
<li>Wu, R., et al. A mutual learning method for salient object detection with intertwined multi-supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, , 8150-8159 (2019).</li>
<li>Deep contrast learning for salient object detection. Li, G., Yu, Y. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, , 478-487 (2019).</li>
<li>Ren, Q., Hu, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Multi-scale+deep+encoder-decoder+network+for+salient+object+detection%5BTitle%5D)+AND+%22Neurocomputing%22%5BJournal%5D)">Multi-scale deep encoder-decoder network for salient object detection.</a> Neurocomputing. 316, 95-104 (2018).</li>
<li>Wang, W. G., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Salient+object+detection+in+the+deep+learning+era%3A+An+in-depth+survey%5BTitle%5D)+AND+%22IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%22%5BJournal%5D)">Salient object detection in the deep learning era: An in-depth survey.</a> IEEE Transactions on Pattern Analysis and Machine Intelligence. 44 (6), 3239-3259 (2021).</li>
<li>Zhou, T., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((RGB-D+salient+object+detection%3A+A+survey%5BTitle%5D)+AND+%22Computational+Visual+Media%22%5BJournal%5D)">RGB-D salient object detection: A survey.</a> Computational Visual Media. 7, 37-69 (2021).</li>
<li>Fu, K., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Siamese+network+for+RGB-D+salient+object+detection+and+beyond%5BTitle%5D)+AND+%22IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%22%5BJournal%5D)">Siamese network for RGB-D salient object detection and beyond.</a> IEEE Transactions on Pattern Analysis and Machine Intelligence. 44 (9), 5541-5559 (2021).</li>
<li>Lai, Q., et al. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Weakly+supervised+visual+saliency+prediction%5BTitle%5D)+AND+%22IEEE+Transactions+on+Image+Processing%22%5BJournal%5D)">Weakly supervised visual saliency prediction.</a> IEEE Transactions on Image Processing. 31, 3111-3124 (2022).</li>
<li>A bi-directional message passing model for salient object detection. Zhang, L., Dai, J., Lu, H., He, Y., Wang, G. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, , 1741-1750 (2018).</li>
<li>A mutual learning method for salient object detection with intertwined multi-supervision. Wu, R. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, , 8150-8159 (2019).</li>
<li>Wang, W., Shen, J., Dong, X., Borji, A., Yang, R. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Inferring+salient+objects+from+human+fixations%5BTitle%5D)+AND+%22IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%22%5BJournal%5D)">Inferring salient objects from human fixations.</a> IEEE Transactions on Pattern Analysis and Machine Intelligence. 42 (8), 1913-1927 (2019).</li>
<li>Design and perceptual validation of performance measures for salient object segmentation. Movahedi, V., Elder, J. H. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops, , 49-56 (2010).</li>
<li>Shi, J., Yan, Q., Xu, L., Jia, J. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Hierarchical+image+saliency+detection+on+extended+CSSD%5BTitle%5D)+AND+%22IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%22%5BJournal%5D)">Hierarchical image saliency detection on extended CSSD.</a> IEEE Transactions on Pattern Analysis and Machine Intelligence. 38 (4), 717-729 (2015).</li>
<li>Frequency-tuned salient region detection. Achanta, R., Hemami, S., Estrada, F., Susstrunk, S. 2009 IEEE Conference on Computer Vision and Pattern Recognition, , 1597-1604 (2009).</li>
<li>Saliency detection via graph-based manifold ranking. Yang, C., Zhang, L., Lu, H., Ruan, X., Yang, M. H. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, , 3166-3173 (2013).</li>
<li>Wei, Y., et al. Geodesic saliency using background priors. <a target="_blank" href="http://scholar.google.com/scholar?hl=en&safe=off&q=author%3aWei%2C+Y+%22Computer+Vision-ECCV+2012%22">Computer Vision-ECCV 2012</a>. , Springer. Berlin Heidelberg. 29-42 (2012).</li>
<li>What makes a patch distinct. Margolin, R., Tal, A., Zelnik-Manor, L. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, , 1139-1146 (2013).</li>
<li>Saliency filters: Contrast based filtering for salient region detection. Perazzi, F., Krähenbühl, P., Pritch, Y., Hornung, A. 2012 IEEE Conference on Computer Vision and Pattern Recognition, , 733-740 (2012).</li>
<li>Hou, X., Harel, J., Koch, C. <a target="_blank" href="http://www.ncbi.nlm.nih.gov/pubmed?term=((Image+signature%3A+Highlighting+sparse+salient+regions%5BTitle%5D)+AND+%22IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%22%5BJournal%5D)">Image signature: Highlighting sparse salient regions.</a> IEEE Transactions on Pattern Analysis and Machine Intelligence. 34 (1), 194-201 (2011).</li>
<li>Salient object detection: A discriminative regional feature integration approach. Jiang, H., et al. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, , 2083-2090 (2013).</li>
<li>Visual saliency based on multiscale deep features. Li, G., Yu, Y. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, , 5455-5463 (2015).</li>
<li>Deep saliency with encoded low level distance map and high-level features. Lee, G., Tai, Y. W., Kim, J. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, , 660-668 (2016).</li>
<li>Dhsnet: Deep hierarchical saliency network for salient object detection. Liu, N., Han, J. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, , 678-686 (2016).</li>
</ol>

The authors have nothing to disclose.

The article introduces an end-to-end deep neural network specifically designed for the detection of salient objects in complex environments. The network is composed of two interconnected components: a pixel-level multiscale fully convolutional network (DCL) and a deep encoder-decoder network (DEDN). These components work synergistically, incorporating contextual semantics to generate visual contrasts within multiscale feature maps. Additionally, they leverage both deep and shallow image features to improve the precision ...

<table><tbody><tr><td>Matlab</td><td>MathWorks</td><td>Matlab R2016a</td><td>MATLAB&#039;s programming interface provides development tools for improving code quality maintainability and maximizing performance.&lt;br/&gt; It provides tools for building applications using custom graphical interfaces.&lt;br/&gt; It provides tools for combining MATLAB-based algorithms with external applications and languages</td></tr><tr><td>Processor&amp;nbsp;</td><td>Intel</td><td>11th Gen Intel(R) Core (TM) i5-1135G7 @ 2.40GHz</td><td>64-bit Win11 processor&amp;nbsp;</td></tr><tr><td>Pycharm</td><td>JetBrains</td><td>PyCharm 3.0</td><td>PyCharm is a Python IDE (Integrated Development Environment)&lt;br/&gt; a list of required python:&lt;br/&gt; modulesmatplotlib&lt;br/&gt; skimage&lt;br/&gt; torch&lt;br/&gt; os&lt;br/&gt; time&lt;br/&gt; pydensecrf&lt;br/&gt; opencv&lt;br/&gt; glob&lt;br/&gt; PIL&lt;br/&gt; torchvision&lt;br/&gt; numpy&lt;br/&gt; tkinter</td></tr><tr><td>PyTorch&amp;nbsp;</td><td>Facebook</td><td>PyTorch 1.4&amp;nbsp;</td><td>PyTorch is an open source Python machine learning library , based on Torch , used for natural language processing and other applications.PyTorch can be viewed both as the addition of GPU support numpy , but also can be viewed as a powerful deep neural network with automatic derivatives .</td></tr></tbody></table>

end-to-end deep neural network for salient object detection in complex environments

End-To-End Deep Neural Network for Salient Object Detection in Complex Environments

Salient object detection has emerged as a burgeoning area of interest within the realm of computer vision. However, prevailing algorithms exhibit diminished precision when tasked with detecting salient objects within intricate and multifaceted environments. In light of this pressing concern, this article presents an end-to-end deep neural network that aims to detect salient objects within complex environments. The study introduces an end-to-end deep neural network that aims to detect salient objects within complex environments. Comprising two interrelated components, namely a pixel-level multiscale full convolutional network and a deep encoder-decoder network, the proposed network integrates contextual semantics to produce visual contrast across multiscale feature maps while employing deep and shallow image features to improve the accuracy of object boundary identification. The integration of a fully connected conditional random field (CRF) model further enhances the spatial coherence and contour delineation of salient maps. The proposed algorithm is extensively evaluated against 10 contemporary algorithms on the SOD and ECSSD databases. The evaluation results demonstrate that the proposed algorithm outperforms other approaches in terms of precision and accuracy, thereby establishing its efficacy in salient object detection within complex environments.

This study introduces an end-to-end deep neural network comprising two complementary networks: a pixel-level multi-scale fully convolutional network and a deep encoder-decoder network. The first network integrates contextual semantics to derive visual contrasts from multi-scale feature maps, addressing the challenge of fixed receptive fields in deep neural networks across different layers. The second network utilizes both deep and shallow image features to mitigate the issue of blurred boundaries in target objects. Final...

The present protocol describes a novel end-to-end salient object detection algorithm. It leverages deep neural networks to enhance the precision of salient object detection within intricate environmental contexts.

Salient object detection has emerged as a burgeoning area of interest within the realm of computer vision. However, prevailing algorithms exhibit ...

end-to-end-deep-neural-network-for-salient-object-detection-complex

Research

JoVE Journal

Engineering

The purpose of this protocol is to utilize pre-built convolutional neural nets to automate behavior tracking and perform detailed behavior analysis. Behavior tracking can be applied to any video data or sequences of images and is generalizable to track any user-defined object.

A Step-by-Step Implementation of DeepBehavior, Deep Learning Toolbox for Automated Behavior Analysis

The goal of the work presented in this article is to develop technology for automated recognition of food and beverage items from images taken by mobile devices. The technology comprises of two different approaches - the first one performs food image recognition while the second one performs food image segmentation.

Deep Neural Networks for Image-Based Dietary Assessment

The goal of this experiment is to determine and control the size, shape and stability of self-assembled discotic amphiphiles in water. For aqueous based supramolecular polymers such level of control is very difficult. We apply a strategy using both repulsive and attractive interactions. The experimental techniques applied to characterize this system are broadly applicable.

Controlling the Size, Shape and Stability of Supramolecular Polymers in Water

We describe the use of pulsed laser deposition (PLD), photolithography and wire-bonding techniques to create micrometer scale complex oxides devices. The PLD is utilized to grow epitaxial thin films. Photolithography and wire-bonding techniques are introduced to create practical devices for measurement purposes.

Fabrication of Spatially Confined Complex Oxides

Because of the importance and extensive use of palladium, gold and cobalt metals in high-technology equipment, their recovery and recycling constitute an important industrial challenge. The metal recovery system described herein is a simple, low-cost means for the effective detection, removal, and recovery of these metals from the urban mine.

Detection and Recovery of Palladium, Gold and Cobalt Metals from the Urban Mine Using Novel Sensors/Adsorbents Designated with Nanoscale Wagon-wheel-shaped Pores

A scanner for imaging magnetic particles in planar samples was developed using the planar frequency mixing magnetic detection technique. The magnetic intermodulation product response from the nonlinear nonhysteretic magnetization of the particles is recorded upon a two-frequency excitation. It can be used to take 2D images of thin biological samples.

Frequency Mixing Magnetic Detection Scanner for Imaging Magnetic Particles in Planar Samples

This paper presents an impedance-based apparatus for evaporation rate detection of solutions. It offers clear advantages over a conventional weight loss approach: a fast response, high-sensitivity detection, a small sample requirement, multiple sample measurements, and easy disassembly for cleaning and reuse purposes.

A High Performance Impedance-based Platform for Evaporation Rate Detection

Here, we present a procedure for the measurement of simultaneous impedance, rheology and neutron scattering from soft matter materials under shear flow.

Dielectric RheoSANS &#8212; Simultaneous Interrogation of Impedance, Rheology and Small Angle Neutron Scattering of Complex Fluids

The goal of the protocol is to create liquid crystalline polymer films that can mechanically oscillate under continuous light irradiation. We describe in great detail the conception of free-standing films, from the liquid crystal alignment method to the photo-actuation. The experimental protocol applied to prepare this material is broadly applicable.

Preparation of Liquid Crystal Networks for Macroscopic Oscillatory Motion Induced by Light

This article presents a protocol for seeding scarce population of cells using pipette-tips to droplet microfluidic devices in order to provide higher encapsulation efficiency of cells in droplets.

A Pipette-Tip Based Method for Seeding Cells to Droplet Microfluidic Platforms

Explore deep neural networks for salient object detection in complex environments using convolutional architectures. This computational method improves accuracy for smart grid insulator inspection applications.

End-To-End Deep Neural Network for Salient Object Detection in Complex Environments

Explore More Videos