Gold Futures Price Prediction Using Transformer Deep Learning Models with Data Scraped via UiPath

Summary

The study develops a transformer deep learning model to accurately predict gold prices using historical data from 2014 to 2024. Achieving 93% accuracy, the model outperforms traditional methods by capturing complex trends and dependencies. It aids investors and policymakers in decision-making and suggests future improvements using sentiment analysis and global factors.

Abstract

Gold is one of the most valuable and widely traded commodities worldwide, particularly in India, and plays a significant role in economic and financial markets. Historically, gold has been a cornerstone of international trade and economic stability, with central banks maintaining reserves to manage inflation and foreign debt. The price of gold serves as a key economic indicator that influences market trends and investment strategies. However, accurately predicting gold prices is challenging due to the complex and nonlinear nature of financial markets which are influenced by various factors including interest rates, economic recessions, oil price fluctuations, and geopolitical events. The study transformer model was used to predict the daily gold prices which were collected from investing.com through web scraping by using UiPath. It is a Robotic Process Automation (RPA) platform to preserve the integrity of the data and enhance model performance, preprocessing operations such as missing data handling and MinMax scaling were performed. The model was tested and trained on key performance metrics and achieved a Mean Squared Error (MSE) of 0.0224, Root Mean Squared Error (RMSE) of 0.1496, and R-squared of 0.93, with a high prediction accuracy. The study results confirm that the transformer model efficiently detects short-term price movements and long-term market trends offering a more accurate and dependable method than traditional forecasting methods. The study provides valuable guidance to investors, financial analysts and policymakers in making informed decisions in the gold bullion market. Future research can be improved by the inclusion of alternative data sources such as sentiment from news headlines and social media which can potentially offer richer insight into market movements.

Introduction

Gold is one of the most significant commodities in the financial markets, a critical economic resource and a price-controlling tool. Gold constitutes a substantial portion of central bank reserves in many countries¹. Recently, gold has become more well-known as an inflation hedge². It is a key hedging asset and its volatility forecasting improves using Generalized Autoregressive Conditional Heteroskedasticity - Mixed Data Sampling (GARCH-MIDAS) models that account for asymmetry, extremes, and jumps³. India and China are the largest importers and consumers of gold accounting for approximately 60% of global demand. The continuous rise in gold prices driven by market dynamics and commodity relations has positioned gold as a preferred investment for future returns⁴. In India the magnitude of physical gold holdings is substantial with households often treating gold as both a cultural asset and a financial safeguard. Its demand for physical gold directly impacts market trends emphasizing the need to consider its relevance in gold price forecasting^5,6. Investors are constantly seeking investment opportunities that offer the highest returns with the lowest risk. In this regard gold serves as a safe-haven asset providing financial security during economic uncertainties. Among the top investment alternatives in India are stocks, banks, deposits, insurance, gold postal savings, other savings and investment options⁷. Given its stability gold is widely regarded as the best alternative investment option among various investment options and a valuable asset that protects against inflation and financial risks^1,8. Gold price determination is influenced by factors like gold demand, supply, inflation, and exchange rates. A surprising finding indicates that dollar appreciation correlates with increased gold prices potentially due to speculative investment⁹. It is difficult to predict the gold price because of its volatile, unpredictable, nonlinear and uncontrollable price movement¹⁰. Therefore, accurate gold price prediction is crucial for investors, portfolio managers, and policymakers as improved forecasting can lead to significant profit opportunities¹¹. The current price trend of gold which is four times higher than two decades ago and the desire to keep gold attract investors to invest in related assets¹². Therefore, before investing in gold every investor must understand the key factors influencing its price¹³. In the financial industry the capacity to forecast the direction of gold prices is essential even minor advancements in performance prediction can result in profits money is essential to lead a life and sound financial forecasting will support economic expansion¹⁴.

Figure 1 explores the gold price trends in India over a period of time spanning from 2014 to 2024 and how gold prices are increasingly based on global factors like the dollar index, federal funds rate, the consumer price index (CPI), exchange rate, oil prices and the S&P500¹⁵. Other factors include business cycles, nominal interest rates, commodity prices, exchange rates and stock prices¹⁶. It highlights the significant price changes, volatility in the market and a period of sustained expansion in gold prices particularly in recent years. Prices showed little volatility between 2014 and 2018 however, in 2019 a robust increasing trend with more price swings was observed. Gold's position as a safe-haven asset was strengthened by a substantial increase in 2020, most likely brought on by concerns about inflation and economic upheaval. The close price closely tracks the open price but greater volatility particularly after 2020 is shown by broader gaps between high and low prices. Prices have been rising steadily during the 2023-2024 timeframe hitting all-time highs. The chart shows how gold has appreciated throughout time, making it a useful asset for analysts and investors.

Accurate gold price prediction is vital for informed decision-making but it is challenging due to complex factors like economic and geopolitical conditions¹⁷. Traditional models like Autoregressive Integrated Moving Average (ARIMA) often struggle to capture these complexities limiting their predictive accuracy^18,19. In a quest to overcome these limitations the recent advancements in machine learning and deep learning have significantly improved financial forecasting accuracy. Experiments demonstrated that deep learning models like Long Short-Term Memory (LSTM) networks perform significantly better than traditional statistical models by capturing sequential relationships and trends in gold prices²⁰. Hybrid forecasting models are a key consideration in forecasting gold prices in big data computation and economics²¹. Nevertheless, regardless of their performance recurrent models like Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) have natural limitations like disappearing gradient problems, high computational complexity and inability to capture long-range relationships which may impact their efficiency in handling large-scale finance time series data.

In order to bridge these deficits and further enhance the accuracy of prediction transformer models are now an attractive choice in time-series forecasting, compared to LSTM networks transformers leverage self-attention mechanisms, hence making them effectively detect long-term patterns and short-term oscillations in financial data and parallel processing to improve scalability. Transformers have considerable potential to improve gold price prediction accuracy given their effectiveness in financial market forecasting. The present study aims to develop a transformer model for gold price forecasting using past gold prices and major economic data providing a more robust approach for gold prediction.

Earlier studies had predicted the gold price using several techniques like neural networks, algorithms, machine learning, deep learning and artificial intelligence. Some of the studies related to gold price prediction are as follows:

Gaussian process regression (GPR) has emerged as a robust tool for metal price forecasting with applications in steel, copper and precious metals highlighting its effectiveness in handling high-dimensional financial data and optimizing models through Bayesian methods to predict the gold price more accurately RMSE of 0.87²². The study presents a novel hybrid model that combines a Random Convolutional Kernels-based Neural Network (RCK) with Generalized Autoregressive Conditional Heteroskedasticity (GARCH) to predict the volatility of gold, silver and platinum. The accuracy of the suggested method is 53% higher than that of the GARCH-LSTM model²³. The study used ARIMA and Holt-Winters models to forecast the future gold prices and inflation rates in India and the United States by considering the Consumer Price Index (CPI). Gold is a good hedge against inflation for both Indian and United States Investors²⁴. The study compares deep learning and machine learning models to analyse Bitcoin's relationship with global factors such as gold, oil, US Dollar Index (DXY) and interest rates. Temporal Fusion Transformer (TFT) outperforms others highlighting gold and DXY as key influencers amid global events and regulations²⁵. Moreover, the study systematically reviews the factors affecting gold futures prices like inflation rate, interest rate, crude oil price and US dollar index with three main forecasting methods: traditional time series models (ARIMA GARCH, ARDL), hybrid models (MS-MIDAS-CJ, VMD-ICSS-BiGRU) and deep learning methods (SGRU-AM) DBN)²⁶. Transformer architectures have strong potential in long-term time series forecasting by effectively capturing long-range dependencies and outperforming traditional Recurring Neural Network (RNN) based models²⁷. The study explored the relationship between gold prices and key determinants like stock markets, crude oil, exchange rates, inflation and interest rates. It utilizes machine learning algorithms finding gradient boosting regression is more accurate than random forest regression in predicting gold prices¹⁴. The study used Autoregressive Distribution Lag (ARDL) model to forecast the annual gold prices using treasury bill rates, gold demand and lagged gold prices as covariates²⁸. The proposed Convolutional Neural Network-Long Short-Term Memory (CNN-LSTM) model for gold price forecasting demonstrates superior performance compared to traditional methods with lower MAE and RMSE values. Incorporating convolutional layers into the LSTM architecture further enhances its ability to capture both short-term patterns and long-term dependencies in the data²⁹.

Future gold prices are predicted from four commodities: historical data of gold prices, silver prices, crude oil prices and the Standard and Poor's 500 stock index (S&P500) index and foreign exchange rate by using the Extreme Learning Machine (ELM) model with an accuracy of 91.67%⁴. A Multilayer Perceptron (MLP) neural network model was employed to predict stock price fluctuations and gold prices based on data from the Tehran Stock Exchange (TSE). The result indicated that the artificial neural network outperformed traditional forecasting models in terms of prediction accuracy³⁰. The study compared LSTM with Linear Regression models and discovered that Linear Regression achieved a marginally higher accuracy rate of 53.02% in forecasting the daily fluctuations of gold prices. This suggests that even simpler models can be effective in predicting gold price movements highlighting the importance of understanding the relationships between gold prices and broader economic factors³¹. Wavelet Neural Network (WNN) modelling for gold price forecasting study experimental results demonstrated that the enhanced Artificial Bee Colony (ABC) algorithm outperforms the traditional ABC approach in gold price forecasting utilizing wavelet neural networks³².

Several advanced predictive models have been proposed for gold and financial market forecasting. One of the studies developed a gold price prediction model using an Adaptive Neuro-Fuzzy Inference System (ANFIS) with past gold prices as inputs. The model accurately predicted next-day gold prices outperforming ARIMA and CNN models in terms of RMSE. It also exceeded the performance of both Buy and Hold strategies by delivering significantly higher returns on investment demonstrating strong potential for gold price forecasting³³. Another study integrated an ELM with a Fuzzy Inference System (FIS) and used the Chaotic Quantum Crisscross Search Algorithm (CQCSA) optimization technique. This hybrid model exhibits superior performance in predicting the gold and oil prices³⁴. A research employed various machine learning models on financial market prediction during the COVID-19 period where Vanilla Stacked LSTMs surpassed other approaches in time-series forecasting. The findings emphasized the importance of incorporating non-economic variables alongside historical financial data for short-term market predictions³⁵. The study explored ensemble models for predicting the gold and silver stock price movements. These approaches achieved notable accuracy rates of 85% for gold and 79% for silver indicating their capability to capture complex market dynamics³⁶. In another study focusing on deep learning architectures for gold price prediction the Bidirectional Long Short-Term Memory (BiLSTM) model outperformed standard LSTMs by achieving higher R², lower MAE, and RMSE values. By effectively capturing temporal patterns from normalized historical data BiLSTM demonstrated its potential to enhance the accuracy of financial forecasts³⁷.

Accurate prediction of gold prices has long posed a significant challenge for investors, analysts, and policymakers due to the volatile, nonlinear, and multifactorial nature of financial markets¹⁰. Traditional statistical models such as ARIMA and regression-based approaches have shown limitations in capturing the complex and dynamic relationships inherent in gold price movements particularly under the influence of macroeconomic indicators, geopolitical tensions, and global financial shifts¹⁹. While ARIMA and similar models are useful for short-term linear forecasting, they lack the ability to capture nonlinearity, structural breaks, and long-term dependencies^18,38. Although deep learning models like LSTM and GRU have improved prediction performance by learning sequential patterns²⁰. These models still suffer from intrinsic limitations such as vanishing gradients and difficulties in modelling long-term dependencies in time-series data²⁹. Furthermore, while hybrid and ensemble models have been explored, they often struggle with scalability and computational efficiency in large-scale financial datasets²³. Gold, increasingly used as a hedge against inflation and economic uncertainty, especially in countries like India, where it holds substantial cultural and financial significance, the demand for robust, high-accuracy predictive models has become paramount². Recent innovations in deep learning, particularly the transformer architecture known for its self-attention mechanism, have shown promising results in time-series forecasting tasks across financial domains¹⁷. However, existing literature lacks a comprehensive investigation of transformer models specifically for gold price prediction using long-term historical datasets enriched with diverse economic indicators. Most previous studies have either focused on short-term prediction or used limited data inputs, neglecting the full potential of transformer models in capturing both short-term and long-term market trends³¹.

Therefore, the present study addresses the critical research gap by proposing and evaluating a transformer deep learning model trained on a decade's worth of historical gold price data (2014-2024) collected through an automated and reproducible web scraping method using UiPath. The model aims to enhance prediction accuracy outperform existing models like LSTM, CNN-LSTM, and regression. It provides a reliable tool for investors and policymakers in making informed decisions under market uncertainty²².

The unstoppable increase in gold prices has heightened the need for forecasting which is more important than ever for investors, financial analysts and policymakers. Gold is widely valued as a safe-haven asset and plays a key role in investment strategies particularly in countries like India where it holds economic and cultural importance. In an increasingly volatile market and global uncertainty environment reliable predictions are crucial for maximizing returns, managing risks, and making sound financial decisions. The study is designed to develop an accurate and effective gold price forecasting model that aligns with the complexities of today's financial markets.

Protocol

The methodology for this research is a systematic process designed to develop a transformer model for gold price prediction. Data collection is the initial step where historical gold price data and relevant financial indicators are gathered to form a comprehensive dataset. The dataset undergoes data preprocessing, model development, evaluation, performance metrics and finally gold price prediction with a detailed explanation in the subsequent sections.

Data collection and reliability:
Historical gold price data for this study were collected using an automated process developed on the UiPath Robotic Process Automation (RPA) platform. The dataset was obtained from Investing.com a widely utilized and reputable financial platform known for its structured and accurate commodity and market data³⁹. It included daily values for Date, Open, High, Low, Close, Volume and Percentage Change covering the period from 1 January 2014 to 31 December 2024 (Supplementary Table 1). The automation was programmed to navigate to the relevant historical data page specify the defined date range and export the dataset into Excel. To improve stability and execution speed the process avoided the calendar picker interface and incorporated validation routines, time delays and conditional checks to accommodate webpage variations. This approach allowed for efficient, repeatable and accurate data extraction without requiring manual intervention.

To ensure data integrity and reliability, multiple validation steps like UiPath's Element Exists, conditional checks, and delays to control dynamic page loads were included to guarantee data integrity and dependability. Web scraping from external sources can be a valuable method for acquiring large datasets efficiently. However, it may also present challenges such as inconsistent data formats, changes in webpage structure, or variations in data availability over time. The scraping tool has error-handling procedures and has undergone periodic reliability testing to reduce these risks. To confirm correctness the retrieved data was further cross-checked at random with information from other sources, like Yahoo Finance. During preprocessing, forward filling and interpolation were used to manage non-trading days and missing records. This method offers a robust yet effective way to produce a trustworthy dataset that is appropriate for deep learning models.

Data preprocessing:
In the present study the gold price dataset was first loaded into a pandas DataFrame to create a tabular dataset for analysis and examined using methods such as info, describe and unique to assess data types, statistical distributions and the number of unique values in each column. The dataset was then sorted in ascending order based on its index, ensuring chronological arrangement for time series analysis. Missing values were identified and eliminated using dropna and duplicate entries were detected using the duplicated method and subsequently removed with drop_duplicates to maintain data integrity. Column names were standardized for consistency by renaming variables like Price to Close, Volume to Volume, and Change % to Change, thereby facilitating clarity in further analysis. Through these steps, the dataset was cleaned, structured, and prepared to support accurate and reliable subsequent modelling and interpretation.

Model development:
The transformer model relies on self-attention mechanisms rather than recurrence or convolution enabling efficient sequence modelling. In the context of time series forecasting such as gold price prediction this architecture can effectively capture long-range dependencies⁴⁰. Hence, the transformer model in Figure 2 is used for gold price prediction by leveraging advanced deep-learning techniques to analyse historical price trends. After data preprocessing the input data first passes through an embedding layer converting numerical features into dense vector representations making the data more meaningful for analysis. Positional encoding is applied to retain temporal information. Each encoder block contains a multi-head attention mechanism with 4 attention heads followed by a feed-forward neural network. Each attention head uses scaled dot-product attention to focus on relevant historical data points. The model has four encoder layers each having a feed-forward inner-layer size of 512 and a dimensionality (d_model) of 128. Each sub-layer is followed by dropout regularization at a rate of 0.1 to avoid overfitting. Constant use of residual connections and layer normalization stabilizes training and enhances convergence. A key component of the model is the attention mechanism which identifies and prioritizes the most relevant past data points rather than treating all historical values equally. This allows the model to capture important patterns and dependencies in gold price movements. The processed information is then passed through a neural network layer which further refines the patterns and relationships to enhance prediction accuracy. Additionally, the model incorporates multiple stacked transformer layers strengthening its ability to learn complex trends and improve predictive performance. Finally, the model accuracy is evaluated using performance metrics such as Mean Squared Error (MSE), Root Mean Squared Error (RMSE) and R-squared (R²). This transformer approach efficiently captures both short-term and long-term dependencies in financial time-series data making it highly effective for precise gold price forecasting.

To normalize the input features, Min-Max scaling was applied as shown in the following Equation.

Let x be a feature value, x_min and x_max be the minimum and maximum of the feature⁴¹.

x: the original value (today's gold price).
x_min, x_min: the training period's min and max of that feature.
[a, b]; the target range (usually [0, 1]; sometimes [-1, 1]).

The core of the transformer encoder is the scaled dot-product attention mechanism⁴⁰ defined as:

Mean square error (MSE)
The model was trained to minimize the mean square error (MSE) between predicted and actual values⁴².

MSE =

Root mean square error (RMSE)
The performance was evaluated using Root Mean Square Error (RMSE)⁴³, calculated as

RMSE =

Training and validation
The dataset is partitioned into three subsets with 80% allocated for training, 10% for testing and 10% for validation. The model is optimized using the Adam optimizer⁴⁴ with a batch size of 64 and a learning rate of 0.001. To prevent overfitting training is carried out over 50 epochs with early halting based on validation loss. Mean Squared Error (MSE) serves as the loss function. Target outputs are the value at t+30 and sequence inputs are generated with a 30-day lookback window (L = 30). When required padding and masking are applied to preserve constant sequence lengths. TenSorFlow⁴⁵ was used to implement the model guaranteeing its scalability and reproducibility. Reproducibility is supported by these comprehensive specifications which also improve comprehension of the transformer architecture's adaptation for financial forecasting.

Testing
The final evaluation of the transformer model is conducted using the test dataset comprising the remaining 10% of the total data. The predicted gold prices are compared with actual values and performance metrics are calculated to measure prediction accuracy.

Model evaluation
The predictive performance of the model was evaluated using key statistical metrics like the Mean Squared Error (MSE), Root Mean Squared Error (RMSE) and R-squared (R²). Results are presented in the subsequent section.

Results

In the study analysis performance metrics were assessed through historical data these metrics were used to give a comprehensive picture of how well the transformer model predicted the future gold prices.

In Figure 3, the gold price candlestick chart presents a thorough visual representation of price swings, emphasizing significant patterns, market corrections, and stable intervals across time. To make it simpler to follow market movements each candlestick represen...

Discussion

The present study proposed a transformer deep learning model that uses data scraped by UiPath, an automated platform ensuring the timely and organized acquisition of financial data to predict the gold futures prices with improved accuracy and reliability. The model incorporates data on the price of gold with important technical indicators like volume metrics, 30-day and 100-day moving averages, trends, gold price prediction, and candlestick patterns. Outperforming conventional models like LSTM, CNN-LSTM, and linear regre...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Data Source	Investing.com	https://in.investing.com/commodities/gold-historical-data	Investing.com is a financial market platform that provides market quotes, information about stocks, futures, etc.
Transformer Model (Python implemented in Google Colab)	Google	https://colab.research.google.com/	Developed a python source code for data analysis to extract insights from data, and that code has been executed on this Google Colab virtual environment
UiPath Robotic Process Automation	UiPath, Inc.	https://www.uipath.com	To extract the data from the source, and the code has been executed on this platform UiPath Studio.