Solar Power Forecasting Using Hybrid Deep Learning: Performance Enhancement with Random Forest-BiLSTM and Ensemble Modeling

Summary

This study advances concentrated solar power plant performance through comprehensive data analysis and error correction methodologies. By integrating spectrum analysis, thermal efficiency optimization, and hybrid machine learning models, the research provides actionable strategies for enhancing operational efficiency and reliability, thereby supporting the role of solar energy as a sustainable power source.

Abstract

Accurate solar power forecasting is critical for grid integration and operational stability of renewable energy systems. This study presents a hybrid deep learning ensemble approach to predict solar generation by capturing complex temporal dependencies in irradiance data. Five hybrid architectures were evaluated: RF-BiLSTM, CNN-LSTM, CNN-BiLSTM, CNN-GRU, and CNN-Transformer, each combining convolutional or recurrent components to extract spatial and sequential features from historical time series. The RF-BiLSTM model achieved the best individual performance with R² = 0.6568, MAE = 30,728 W, and MSE = 1.81 × 10⁹ W². An ensemble model integrating the top three architectures using inverse MAE-weighted averaging demonstrated superior performance with R² = 0.6933, MAE = 28,809.89 W, and MSE = 1.53 × 10⁹ W², reducing prediction error by 6.2% compared to the best individual model. The proposed ensemble framework effectively balances model strengths, enhances forecast robustness, and provides a scalable, data-driven solution for renewable energy forecasting in smart grid and energy management systems.

Introduction

The accelerating global transition toward renewable energy has positioned solar power as a pivotal source in the sustainable energy mix. As countries increasingly commit to decarbonizing their energy systems, solar photovoltaic (PV) technology has witnessed exponential growth due to its scalability, declining costs, and environmental benefits. However, the widespread integration of solar energy into national and regional power grids presents significant challenges, primarily due to its intermittent and weather-dependent nature. Solar irradiance is influenced by a variety of environmental factors, including cloud cover, atmospheric conditions, seasonal shifts, and diurnal cycles, all of which introduce variability and uncertainty into solar power generation. This inherent variability complicates the task of grid balancing and power system planning. Operators must accurately predict solar power output to ensure optimal resource allocation, reduce reliance on fossil-fuel-based backup systems, prevent overloading or under-utilization of infrastructure, and maintain overall grid stability. As solar energy penetration increases, the need for robust, reliable, and precise forecasting models becomes even more pressing. Accurate short-term and day-ahead solar forecasts are particularly critical for applications such as energy market participation, load dispatch, battery scheduling, and microgrid management¹.

Traditional forecasting methods, such as physical models based on meteorological data and statistical time-series techniques (e.g., ARIMA, exponential smoothing), often fall short in capturing the nonlinear and dynamic behavior of solar generation. These models tend to rely on linear assumptions, handcrafted features, or detailed weather simulations, which limit their scalability and adaptability to changing patterns in solar data². In contrast, deep learning (DL) models have emerged as a transformative approach in time series forecasting. These data-driven methods can automatically learn complex features and temporal dependencies directly from raw input data without requiring explicit feature engineering^3,4.

Among the most widely used architectures are Recurrent Neural Networks (RNNs) and their improved variants, such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) networks. These models are designed to capture sequential dependencies and long-term temporal relationships in time-series data^2,5,6. Meanwhile, Convolutional Neural Networks (CNNs) have demonstrated strong capabilities in spatial feature extraction and have been adapted to process temporal data through 1D convolutions, particularly in hybrid configurations^7,8. Hybrid DL models, which combine the strengths of different architectures such as CNNs and RNNs, have gained traction in solar forecasting due to their ability to extract both local and long-range dependencies from time series data^7,8,9

For instance, CNN-LSTM or CNN-BiLSTM models apply convolutional layers to preprocess and filter the input sequence before feeding it into recurrent layers, resulting in more efficient and accurate learning^9,10. Several studies have demonstrated the superiority of hybrid architectures over standalone models. Research using SSA-RNN-LSTM hybrid models achieved significant reductions in error metrics across multiple PV technologies, showing improvements of 15-23% in RMSE compared to alternative hybrid approaches⁹. Similarly, CNN-LSTM architectures have outperformed both standard machine learning approaches and single deep learning models across multiple evaluation metrics when applied to real-world solar power data¹⁰. The effectiveness of decomposition-based hybrid methods has also been established, where wavelet packet decomposition combined with LSTM networks demonstrated superior performance over individual LSTM, RNN, GRU, and MLP models in hour-ahead PV power prediction². In wind power forecasting, hybrid models combining convolutional layers with GRU networks have achieved notable improvements in very short-term predictions, with validation across multiple locations confirming their robustness and generalizability⁷. Additionally, attention-based mechanisms such as Transformers offer further potential by selectively focusing on relevant input segments across time steps. Recent investigations into CNN-LSTM-Transformer hybrids have achieved exceptionally low error rates, representing pioneering efforts to incorporate Transformer networks into hybrid models for solar power forecasting¹¹.

The success of hybrid models extends beyond architectural combinations to include preprocessing techniques and specialized adaptations for real-world conditions. Signal decomposition techniques have proven valuable in capturing the multiscale characteristics of PV power generation, improving forecasting accuracy through better representation of temporal patterns². For industrial-scale solar plants operating under curtailment conditions, enhanced LSTM-based approaches incorporating specialized preprocessing have achieved significant error reductions by addressing data inconsistencies¹². The impact of input data quality has also been examined, revealing substantial performance differences when using historical versus forecasted weather data, with innovative feature engineering techniques helping to mitigate accuracy losses under imperfect input conditions⁶. Machine learning approaches have further demonstrated effectiveness in grid-connected systems, showing potential for reducing reliance on conventional spinning reserve capacity through accurate forecasting¹³. Earlier foundational work established the viability of artificial neural networks for various solar energy applications, demonstrating their ability to handle noisy and incomplete data while providing rapid predictions once trained^3,4,14. Research on optimal forecasting horizons and minimal-input approaches has provided practical guidance for system design and deployment in data-scarce regions^15,16,17. Hybrid methods combining mechanism modeling with deep learning have also shown promise for complex solar thermal power applications, accurately identifying key meteorological factors and their spatiotemporal relationships¹⁸. Comparative studies have established the advantages of advanced recurrent architectures, particularly bidirectional LSTM networks, which have achieved exceptional performance under challenging environmental conditions such as cloudy weather¹⁹.

Ensemble learning, particularly through weighted averaging, offers a compelling solution. By aggregating the predictions of complementary models, ensemble methods can reduce generalization error, improve robustness, and mitigate the weaknesses of individual models. This study investigates the performance of five advanced hybrid DL models: RF-BiLSTM, CNN-LSTM, CNN-BiLSTM, CNN-GRU, and CNN-Transformer for solar power forecasting. Each model is evaluated using rigorous metrics, including the coefficient of determination (R²), mean absolute error (MAE), and mean squared error (MSE). Based on performance benchmarking, the top three models are selected and combined into an optimized ensemble using a weighted averaging technique. The goal is to develop a DL-only ensemble that enhances forecasting accuracy while maintaining generalization and computational feasibility. This research aims to provide practical, high-performance forecasting solutions for grid operators and renewable energy stakeholders.

Despite considerable advances in renewable energy prediction methodologies, several critical limitations persist in the current body of knowledge. While photovoltaic systems have attracted substantial research focus, forecasting applications specifically tailored for concentrating solar power remain markedly underrepresented, leaving questions about thermal efficiency prediction and operational optimization largely unaddressed^15,16. Current forecasting frameworks typically proceed under the assumption that sensor measurements are inherently accurate, neglecting the implementation of systematic error correction procedures for Direct Normal Irradiance instrumentation, which introduces potential reliability concerns for both retrospective analysis and prospective predictions²⁰. Existing approaches concentrate predominantly on temporal prediction without examining spectral characteristics of solar radiation under varying atmospheric conditions, despite the known influence of spectral distribution on system performance¹⁷. Although hybrid architectures combining convolutional and recurrent networks have proven effective for photovoltaic and wind applications, their adaptation to concentrating solar thermal systems remains largely unexplored, particularly configurations integrating Random Forest feature processing with bidirectional recurrent layers^7,10. The prevalence of hourly forecasting intervals in published studies overlooks the necessity for higher temporal resolution capable of capturing rapid thermal response dynamics essential for real-time system management^18,19. Furthermore, data quality enhancement and predictive modeling exist as disconnected research domains without integrated frameworks demonstrating how measurement rectification translates into forecasting improvements²⁰. Finally, computational efficiency considerations, including training duration, inference speed, and hardware requirements, receive insufficient attention relative to accuracy metrics alone, limiting practical deployment guidance²⁰.

This investigation addresses these deficiencies by establishing a comprehensive methodology that incorporates concentrating solar power-specific analysis with thermal optimization, implements rigorous sensor error correction protocols, conducts spectral distribution examination, introduces a Random Forest-Bidirectional LSTM architecture for thermal power prediction, executes minute-resolution forecasting for enhanced temporal granularity, connects data rectification processes with performance outcomes, and provides systematic computational benchmarking across five hybrid architectures using standardized graphics processing hardware. The key research gaps identified in the existing literature are summarized in Table 1.

Research Gap	Existing Literature	What's Missing	This Study Addresses
Limited CSP-Specific Research	Extensive PV forecasting studies15,16	CSP thermal efficiency data rectification	Comprehensive CSP data analysis with thermal optimization
Inadequate Sensor Error Correction	Studies assume data accuracy17	Zero-error correction protocols for DNI instruments	Implemented zero-error correction for accurate assessment
Absence of DNI Spectral Analysis	Temporal forecasting focus only18	Spectral distribution under atmospheric variations	Spectrum analysis revealing cloud/atmospheric influences
Limited Hybrid Models for CSP	CNN-LSTM for PV10, CNN-GRU for wind7	RF-BiLSTM for CSP applications	Novel RF-BiLSTM achieving R2 = 0.657
Lack of Minute-Wise Analysis	Hourly predictions18,19	High-resolution for thermal dynamics	Minute-wise evaluation for real-time optimization
No Integrated Framework	Separate forecasting and quality studies20	Link between rectification and performance	Integrated data-to-performance improvement framework
Insufficient Computational Analysis	Accuracy comparisons only20	Training efficiency and deployment feasibility	Computational analysis on T4 GPU across 5 models

Table 1: Research gaps addressed in the current study. Summary of existing research limitations, missing elements in current literature, and specific contributions of this study in addressing identified gaps in CSP forecasting and data quality assessment.

Protocol

Dataset collection and description
The dataset (Figure 1) used in this research comprises historical records crucial for solar power forecasting. The dataset comprises daily operational data from a 50 MW concentrated solar thermal plant operated by Megha Engineering and Infrastructures Limited (MEIL), located near Anantapur, Andhra Pradesh, India, utilizing parabolic trough Concentrating Solar Power(CSP) technology that captures Direct Normal Irradiance (DNI) and transfers heat via a Heat Transfer Fluid (HTF) to generate electricity. The dataset was collected from 01 January 2015 to 03 October 2025 and contains seven key attributes that capture temporal information, solar irradiance measurements, and power generation output. The temporal attributes include 'Date', providing the calendar date in standard format, 'Year' indicating the year of data collection, 'Month' representing the month number, 'Day' denoting the day of the month, and 'Julian Day' offering a sequential day numbering system throughout the year for continuous temporal analysis. The primary meteorological input variable is 'DNI SUM' measured in kWh/m², which represents the total Direct Normal Irradiance (DNI), the cumulative solar energy received per square meter of the collector surface, serving as the critical factor influencing CSP plant thermal conversion efficiency. The target variable 'Actual Generation', measured in kWh, captures the electrical power output produced by the CSP plant, reflecting the result of the solar-to-thermal-to-electrical energy conversion process. These attributes collectively enable comprehensive analysis of plant performance, including thermal efficiency determination, DNI-to-power conversion modeling, identification of atmospheric and cloud cover influences through spectral analysis, implementation of zero-error correction protocols for sensor calibration, and development of advanced hybrid machine learning forecasting models for optimizing real-time operational planning and enhancing overall CSP plant efficiency and reliability. Plant details available at: https://solarpaces.nrel.gov/project/megha-solar-plant

Figure 1: Top five rows of the dataset. Sample data showing the initial entries of the solar power generation dataset, displaying input features and target variables used for model training and evaluation. Please click here to view a larger version of this figure.

Data preparation
The study utilizes solar generation time-series data spanning from 01 January 2015 to 10 March 2025. To account for potential data quality issues in early years and focus on more recent patterns, the records were filtered from 01 January 2017 onward. Temporal columns (Date, Year, Day) were removed based on preliminary correlation analysis showing negligible predictive value. Missing values were imputed using a moving average technique to maintain temporal continuity while minimizing distortion of underlying patterns. Three lag features were created using the target variable (Actual Generation (kW/h)) to capture temporal dependencies.

Dataset splitting
To establish balanced and representative training, validation, and test cohorts, the pre-processed dataset was segmented using a stratified sampling method. This approach ensured that 70% of the data (2091 data) were allocated for training, while both validation and test sets each comprised 15% (448 data per set).

Data normalization
Features were standardized using StandardScaler, while target values were normalized via MinMaxScaler to [0,1] range for neural network stability.

Model training
Five hybrid models (Random Forest-BiLSTM, CNN-LSTM, CNN-BiLSTM, CNN-GRU, and CNN-Transformer) were implemented for solar generation forecasting. The input data was restructured into a sequential format reshaped as (samples, timesteps, features) where timesteps = 1 for most models, except CNN-LSTM, which used a sliding window of 15 steps. Training, validation, and test sets were scaled while preserving temporal order to avoid data leakage. All models are trained with 32 batch-size and 30 epochs.

Random Forest-BiLSTM (Figure 2): The proposed hybrid model combines a Bidirectional Long Short-Term Memory (BiLSTM) network with a Random Forest (RF) regressor to improve prediction accuracy. First, the BiLSTM model is trained on the input time-series data to capture temporal patterns and generate initial predictions. After this, the residual errors (differences between actual and predicted values) from the BiLSTM are calculated. A Random Forest model is then trained on the original input features to learn and predict these residuals. To enhance the performance of the RF model, the six most important features are selected based on feature importance scores. Finally, the corrected prediction is obtained by adding the RF-predicted residuals to the BiLSTM outputs. This hybrid approach leverages the sequence modeling ability of BiLSTM and the ensemble learning strength of Random Forest to achieve better generalization and predictive performance.

Let  be the input sequence at time step t.

BiLSTM Prediction:

 ,  

 

 

Residual Computation:  

Residual Learning using Random Forest: Let Z⊂X be the top-k features selected using feature importance.

 

Final Prediction:  

Figure 2: Architecture of Random Forest-Bidirectional Long Short-Term Memory model. Schematic diagram illustrating the RF-BiLSTM hybrid architecture, showing the integration of Random Forest feature processing with bidirectional LSTM layers for temporal sequence learning. Please click here to view a larger version of this figure.

CNN-LSTM (Figure 3): The CNN-LSTM hybrid model begins by processing the input sequence using a 1D Convolutional layer to extract local spatial features, followed by a LeakyReLU activation, batch normalization, and max pooling. The extracted features are then passed through a stack of three LSTM layers to learn temporal dependencies, with layer normalization and dropout applied after the first two LSTMs for regularization. The final LSTM output is passed through fully connected dense layers with activation and dropout and finally mapped to the output using a single neuron.

Let  be the input sequence, where T is the time window and F is the number of features.

CNN operation:  

Max pooling:  

LSTM cell:  

 

Output:  

Figure 3: Architecture of CNN-LSTM model. Structural representation of the Convolutional Neural Network-Long Short-Term Memory hybrid model, demonstrating convolutional feature extraction followed by unidirectional temporal sequence processing. Please click here to view a larger version of this figure.

CNN-BiLSTM (Figure 4): The CNN-BiLSTM hybrid model first extracts spatial features using a 1D convolutional layer with 32 filters, followed by batch normalization and max pooling to reduce dimensionality. The output is then passed through a stack of two Bidirectional LSTM layers to capture long-term temporal dependencies in both forward and backward directions. Regularization is applied via dropout and batch normalization. A dense layer with 128 neurons refines the learned representation before the final output layer maps it to a single predicted value.

CNN operation:  

Max pooling:

Bidirectional LSTM:  ,   

 

Output:  

Figure 4: Architecture of CNN-BiLSTM model. Architecture diagram of the Convolutional Neural Network-Bidirectional Long Short-Term Memory model, highlighting the combination of convolutional layers with bidirectional recurrent processing for enhanced temporal dependency capture. Please click here to view a larger version of this figure.

CNN-GRU (Figure 5): The CNN-GRU hybrid model starts with a Conv1D layer using a kernel size of 1 to extract spatial features from the single timestep. Max pooling reduces spatial dimensions. This is followed by a stack of GRU layers the first returns sequences to capture temporal dependencies, and the second summarizes the sequence to a compact representation. A final dense layer outputs the predicted value. Dropout regularization is applied between GRU layers to prevent overfitting.

CNN operation:  

Max pooling:  

GRU cell: 

Output: 

Figure 5: Architecture of CNN-GRU model. Schematic of the Convolutional Neural Network-Gated Recurrent Unit hybrid model, showing convolutional preprocessing integrated with GRU layers for efficient temporal modeling. Please click here to view a larger version of this figure.

CNN-transformer (Figure 6) The CNN-transformer model starts with a Conv1D layer to extract local features from the input sequence, followed by a max pooling layer. These features are passed through a Transformer encoder block consisting of a multi-head self-attention mechanism, layer normalization, and a feed-forward dense network. Global average pooling is then applied before a final dense layer outputs the prediction. This architecture is designed to capture both spatial patterns (via CNN) and global dependencies (via Transformer attention).

CNN operation: 

Multi-Head Self-Attention:

Where: Q, K, V = XW^Q, XW^K, XW^V and d_k is the dimension of keys.

Feed Forward Network:

Add & Norm Layers:

Output: 

Figure 6: Architecture of CNN-Transformer model. Structural overview of the Convolutional Neural Network-Transformer hybrid model, incorporating convolutional feature extraction with multi-head attention mechanisms for advanced temporal pattern recognition. Please click here to view a larger version of this figure.

Ensemble model development
To enhance forecasting accuracy and model robustness, we implemented a weighted average ensemble approach using predictions from the five hybrid deep learning models: RF-BiLSTM, CNN-LSTM, CNN-BiLSTM, CNN-GRU, and CNN-Transformer. The ensemble was constructed by assigning optimized weights to each model's predictions, with higher weights given to models demonstrating superior individual performance, as measured by their R² scores. This weighting strategy ensures that more accurate models contribute more significantly to the final forecast while still leveraging the complementary strengths of all architectures. The ensemble output was then evaluated using standard performance metrics: R², mean absolute error (MAE), and mean squared error (MSE) to assess its predictive accuracy, consistency, and generalization capability. This deep learning ensemble aims to integrate temporal feature extraction from multiple perspectives, thereby achieving greater accuracy and robustness than any single hybrid model in isolation.

Mathematical Formulation of the ensemble technique:

Let 

represent the set of base models corresponding to CNN-RF-BiLSTM, CNN-LSTM, CNN-BiLSTM, CNN-GRU, and CNN-Transformer.

Each base model M_i produces a prediction: 

The meta-feature matrix for stacking is formed as: 

The Ridge Regression meta-learner estimates the final prediction as: 

where:

-- w_i are the learned stacking weights

-- w₀ is the bias term

To avoid overfitting, Ridge Regression minimizes the following regularized loss function:

where:

-- y_j = true target for the j^th sample

 -- N = total number of samples

-- α = regularization parameter controlling weight shrinkage

The ensemble prediction is obtained as: 

where the weights w_i are automatically learned by minimizing the Ridge loss function.

Results

Individual model performance evaluation
The performance evaluation of five hybrid deep learning (DL) models RF-BiLSTM, CNN-GRU, CNN-BiLSTM, CNN-LSTM, and CNN-transformer was conducted using a comprehensive set of standard regression metrics, including R² (coefficient of determination), mean absolute error (MAE), and mean squared error (MSE), to rigorously assess their capability in forecasting solar power generation under varying meteorological conditions and temporal dependencies.

Discussion

The proposed methodology follows a structured workflow as shown in Figure 12. Initially, the dataset undergoes comprehensive preprocessing, including missing value imputation, normalization, and feature engineering, to ensure data quality and enhance model learning^3,6. The processed dataset is then partitioned into training (70%), validation (15%), and testing (15%) sets to enable robust model development and performance evaluation

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
BiLSTM	TensorFlow/Keras	TensorFlow 2.10.0
CNN layers	TensorFlow/Keras	TensorFlow 2.10.0
Google Colab	Google LLC	Cloud Platform
GRU	TensorFlow/Keras	TensorFlow 2.10.0
Matplotlib	Matplotlib Dev Team	3.7.1
NumPy	NumFOCUS	1.25.2
NVIDIA T4 GPU	NVIDIA Corporation	Tesla T4
Pandas	NumFOCUS	2.0.3
Pyrheliometer for DNI measurement	Kipp & Zonen	CH1-DL
Python	Python Software Foundation	3.10.12
Random Forest	Scikit-learn Developers	1.2.2
Scikit-learn	Scikit-learn Developers	1.2.2
Temperature sensors	Vaisala	HMP155
TensorFlow/Keras	Google	Version 2.10.0
Transformer	TensorFlow/Keras	TensorFlow 2.10.0
Weather station	Davis Instruments	Vantage Pro2