GARNN-AE-LSTM: A Multimodal Deep Learning Approach for High-Accuracy Video Summarization

Summary

This study proposes a multimodal deep learning framework for video summarization by integrating audio-visual features using pretrained GARNN models. Leveraging GRUs, AlexNet, and an adversarial LSTM classifier, the system enhances keyframe detection, reduces redundancy, and achieves high summarization accuracy with an average F-score of 0.985.

Abstract

Video summarization focuses on creating concise versions of lengthy videos by preserving essential content. This study reveals a multimodal machine learning strategy that integrates visual and auditory information by using pretrained Gated Recurrent Neural Network architectures referred to as GARNN, combining Gated Recurrent Units (GRUs) and AlexNet, to extract audio, image, and visual features. The keyframe detection is improved by removing redundant frames and applying motion-compensated feature reduction, followed by optional PCA-based dimensionality reduction. An adversarial encoder-based Long Short-Term Memory (AE-LSTM) classifier is employed for temporal modeling by achieving high accuracy in summarization. The results got evaluated by using sensitivity, F-scores, and positive predictive values, and the method attained an average F-score of 0.985. A gated AlexNet is introduced in a multimodal GARNN-AE-LSTM framework, where motion-compensated PCA-based reduction eliminates redundancy, GRUs record temporal progression, and gating fine-tunes spatial feature selection, all of which contribute to a more accurate and efficient video summarization system. The improved F1 score demonstrates the model's effectiveness in generating accuracy in video summaries by creating a meaningful video. This approach highlights the potential of multimodal feature extraction and advanced deep learning techniques for robust video analysis and compression.

Introduction

Multimodal Analysis (MMA) systems have emerged, combining several modalities like audio, video, text, and sensor data to provide insights into how students learn¹. These technologies can aid in gaining a thorough understanding of student behavior and engagement levels by evaluating multimodal data². However, there are a number of obstacles and restrictions when it comes to creating efficient MMA systems³. There are a ton of digital videos due to the growth of the Internet and security cameras. It is imperative that these videos be compiled into databases. A video summary can be helpful in this sit....

Protocol

This study proposes a multimodal supervised video summarizing method that falls into the generic video summarization category, commonly referred to as video skimming. This includes techniques that concentrate on finding key segments of a larger video to create a temporally condensed version of it. This study suggests a supervised technique to analyze the video stream in 1 s sections that include audio and visual representations, as shown in Figure 1. These segments are then categorized as either "uninteresting" or "informative". As opposed to synthetic or simulated data, "real data" now refers to the actual video datasets utilized for exp....

Results

The proposed GARNN-based feature extraction and AGLSTM-based video summarization of lengthy videos were experimented on two datasets, namely VSUMM and SumMe. This section discusses the experimental results and comparison with conventional approaches. For the discriminator LSTM, we employ a two-layer LSTM having 1024 hidden units at every layer. For encoder_LSTM and decoder_LSTM, respectively, we employ two two-layer LSTMs having 2048 hidden units in each layer. A decoder LSTM that seeks to store and synthesize the revers.......

Discussion

In this study, the video summarization of long videos is presented using an efficient multimodal ML and DL models, which use audio, image, and visual modalities of data generated from the input data. The binary classifier is trained to learn the discrimination between the important segments that are the produced summary part and the "non-important" segments that are discarded. The model is trained using datasets such as SumMe and VSUMM, and the scalability of the model is demonstrated in terms of the metrics. Initially, .......

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
AlexNet (pre-trained model)	MATLAB / PyTorch	PyTorch Hub — AlexNet pre-trained model: https://pytorch.org/hub/pytorch_vision_alexnet/	Used for visual feature extraction
FFmpeg	FFmpeg.org	https://ffmpeg.org/	Audio extraction from video
GRNN-based model	Custom implementation	Library “neupy” implements GRNN: http://neupy.com/apidocs/neupy.algorithms.rbfn.grnn.html	Used for multimodal feature fusion
LSTM (Long Short-Term Memory)	PyTorch	https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html	Used in Selector, Encoder, Decoder
Multimodal_movie_analysis lib	GitHub	https://github.com/tyiannak/multimodal_movie_analysis	For visual feature extraction
NumPy	Python Software Foundation	https://pypi.org/project/numpy/	Numerical computation and matrix ops
PCA (Principal Component Analysis)	Scikit-learn	https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html	Dimensionality reduction
PyAudioAnalysis	GitHub	https://github.com/tyiannak/pyaudioanalysis	Audio feature extraction
PyTorch	PyTorch Foundation	https://pytorch.org/	Deep learning framework
SumMe Dataset	Public dataset	https://gyglim.github.io/me/vsum/index.html	Benchmark video summarization dataset
VSUMM Dataset	Public dataset	http://www.vision.ime.usp.br/~creativision/vsumm/	Benchmark video summarization dataset