Optimization of Traditional Stock Market Strategies Using the LSTM Hybrid Approach

Abstract

Investment decision-makers increasingly rely on modern digital technologies to enhance their strategies in today’s rapidly changing and complex market environment. This paper examines the impact of incorporating Long Short-term Memory (LSTM) models into traditional trading strategies. The core investigation revolves around whether strategies enhanced with LSTM technology perform better than traditional methods alone. Traditional trading strategies typically depend on analyzing current closing prices and various technical indicators to take trading action. However, by applying LSTM models, this study aims to forecast closing prices with greater accuracy, thereby improving trading performance. Our findings indicate that trading strategies that utilize LSTM models outperform traditional strategies. This improvement suggests a significant advantage in using LSTM models for market prediction and trading decision making. Acknowledging that no one-size-fits-all strategy works for every market condition or stock is crucial. As such, traders are encouraged to select and tailor their strategies based on thorough testing and analysis to best suit their needs and market conditions. This study contributes to a better understanding of how integrating LSTM models can enhance traditional trading strategies, offering a path toward more effective decision making in the unpredictable stock market.

1. Introduction

In the era of economic globalization and the rapid development of computer technologies, we are witnessing an unprecedented accumulation of financial data. This accumulation is a consequence of technological innovation and reflects the growing interest in stock market trading. To address the limitations of traditional analysis in this evolving landscape, this study introduces an innovative approach by integrating Long Short-term Memory (LSTM) neural networks with traditional trading strategies. Given the continuous growth of the data volume, traditional analysis methods need to be revised [1]. There is an articulated need to develop more advanced techniques that can efficiently process and analyze these data. Financial time series are inherently complex due to the various factors that influence them. Long-term trends, seasonal variations, and non-linear movements shape their dynamics. In addition, stock markets are subject to the influence of various economic, political, and social events [2]. For example, political decisions, changes in interest rates, or unexpected economic news can cause significant changes in the market. The behavior of investors, driven by various factors, further complicates the prediction of market trends. Therefore, financial time series analysis is essential for understanding and predicting market trends.

Machine learning, a part of artificial intelligence, plays a crucial role in financial data analysis, particularly in forecasting stock price movements [3]. Our research adds to this growing area by showing how LSTM models, combined with traditional strategies, significantly improve predictions and provide better trading decisions. We focus on the advantage of LSTM models over traditional analysis, which mainly uses past stock data for predictions and often falls short in forecasting future trends [4]. While technical indicators offer valuable insights into past market trends, relying exclusively on them for trading decisions may only sometimes yield consistent profits. Considering the significant fluctuations and dynamic nature of the stock market, identifying the most opportune moments for trading with reduced risk becomes essential. Machine learning provides tools that enable the analysis and prediction of such complex time series to ensure highly profitable returns. Machine learning algorithms, such as diverse variations of Deep Neural Networks (DNNs), have proven particularly effective in analyzing financial data due to their ability to model complex nonlinear relationships [5].

In particular, Long Short-term Memory (LSTM) networks, a type of recurrent neural network, have become popular in financial forecasting due to their ability to process time series and recognize long-term data dependencies efficiently. By uniquely focusing on the exclusive incorporation of LSTM into traditional trading strategies, this paper gave a deeper understanding of how advanced AI technologies can revolutionize prediction accuracy and trading efficiency in the volatile environment of stock markets. Comparisons of LSTM with traditional models, such as ARIMA, have shown that LSTM often provides more accurate stock price predictions with a smaller root mean square error [1]. In addition, algorithmic trading, which relies on automated algorithms to implement trading strategies, has become ubiquitous in the modern financial sector. These algorithms allow investors to optimize their trading decisions, minimize risk, and maximize profits. Combining machine learning and algorithmic trading opens up new opportunities for developing sophisticated trading strategies that can adapt to the dynamic environment of stock markets [6]. Ultimately, the role of machine learning is becoming more prominent. Through sophisticated algorithms and techniques, machine learning enables investors to make informed decisions based on deep data analysis. This maximizes potential returns and significantly reduces the associated risks, placing machine learning at the center of innovation in the financial industry.

This research aimed to analyze how the performance of traditional trading strategies on the stock market, which are still the basis of many decisions, can be improved by applying machine learning, especially LSTM neural networks. Furthermore, our findings lay a foundation for future investigations into integrating machine learning techniques with financial strategies, inviting further exploration into their long-term impact on market prediction and decision-making processes. The contribution of this paper is manifested in the fact that we analyze and compare the performance of the traditional strategies MACD, TEMA, MOM, and P-MA with traditional ones (previously mentioned) induced with the LSTM approach (hybrid traditional trading strategies). To the best of our knowledge, this is the first attempt to include the LSTM approach exclusively in traditional methods as dominant in predicting the movement of stock prices to increase the effectiveness of traditional trading strategies.

Selected traditional strategies MACD, TEMA, MOM, and P-MA, according to the literature [4,7,8,9], have been tested on particular stock indices and shares, using collected data on value trends. LSTM networks for price prediction have been integrated into the traditional strategies, resulting in hybrid traditional strategies. A backtest was performed on past data [3], comparing the results of traditional strategies (without LSTM integration) with hybridized traditional strategies (with LSTM integration) to see if hybridized strategies are better than simple traditional ones. For an additional comparison of the effectiveness of traditional strategies (without and with LSTM), the “buy and hold” strategy was applied [10].

For clarity and structure of the research, each chapter of the paper provided a deeper understanding of the key components. The chapter on theoretical background has presented the basics of LSTM neural networks, their architecture and operation, and their role in the analysis of financial series: traditional trading strategies considered basic technical analyses and the challenges accompanying their application. The empirical literature overview has focused on implementing machine learning in financial predictions and how trading strategies can be better for trading using different machine learning techniques.

The section on research methodology describes the process, from data collection to backtesting. The research results compared the performance of different trading strategies by analyzing their profitability. In the end, all the important points of this research are brought together for the conclusion, providing recommendations and guidelines for future research.

2. Theoretical Background

2.1. LSTM Neural Networks

Long Short-term Memory (LSTM) networks occupy a special place in deep learning and neural networks due to their ability to process time series and sequential data. While traditional neural networks often encounter challenges in modeling temporal patterns, LSTM networks are designed to solve such problems efficiently. This chapter thoroughly explores the basics, architecture, and applications of LSTM networks, emphasizing their role in time series analysis.

Long Short-term Memory (LSTM) is a neural network architecture that was proposed by Hochreiter and Schmidhuber in 1997 [11]. Its exceptional ability lies in precisely modeling short-term and long-term data dependencies. While traditional feedforward neural networks often face the vanishing gradient problem, LSTM overcomes this challenge by maintaining a constant error across time steps. Its key component is a cell that enables complex internal data processing [12]. This cell, equipped with a special gate, allows the network to selectively keep or forget the information, which is crucial for its ability to model long-term dependencies.

LSTM networks are specially adapted to work with sequential data, emphasizing learning long-term interactions. The selective ability to forget information is achieved through forgetting gates, which allow the network to decide which information should be kept or discarded from its long-term memory [13]. In addition, the inner loop in LSTM allows for the accumulation of information over longer periods.

LSTM networks have proven themselves in various sequential tasks, especially natural language processing. They achieved outstanding results in handwriting recognition, automatic speech recognition, and music composing. Although fewer papers apply LSTM to forecasting real-time series, research has shown that it often outperforms traditional methods such as SVM, ARIMA, and Kalman filters [12,14].

LSTM in Time Series Analysis

In the time series analysis, data are usually available in standardized time patterns where the goal is then to understand or predict future points in that time series. In this context, LSTM networks have become an extremely important tool. Their inherent ability to recognize and model long-term data dependencies makes them particularly suitable for this type of analysis [5,13].

Financial time series present a particular challenge due to their instability and volatility. Factors such as geopolitical events, macroeconomic indicators, corporate reports, and even market sentiments [15] can drastically affect the prices of stocks, bonds, currencies, and other financial instruments. This complexity and unpredictability make financial time series particularly difficult to model [1,13,14,16,17]. Traditional methods often need help to capture the complex patterns in data and rapid changes inherent in financial markets. This is exactly where LSTM networks come into play. Their ability to recognize long-term dependencies allows them to identify patterns often invisible to traditional models.

Similarly, these networks help accurately predict energy consumption in the energy sector, while in meteorology, they provide promising results in short-term weather forecasting. In urban planning, LSTM networks are used to anticipate traffic flows, contributing to optimizing traffic routes and reducing congestion. Given their adaptability and efficiency in capturing complex dependencies, applying LSTM networks in time series analysis will continue to expand and develop.

2.2. Traditional Trading Strategies

While new tools and technologies, such as pre-processed LSTM networks, are increasingly used to develop sophisticated trading strategies, traditional methods and approaches remain the foundation of many trading decisions. It is, therefore, worthwhile to provide insight into the basics of their technical analysis [18], popular strategies used over the years, and challenges faced by approaches that rely exclusively on traditional methods.

2.2.1. Basics of Technical Analysis

Technical analysis is based on studying historical price trends and changes in trading volume to predict future price changes. The basic assumption of this analysis is that price trends are often repeated, which means that if we recognize certain patterns, we can predict future price trends [19]. However, predicting future stock prices takes more work, especially in financial time series. Financial market prices result from a complex interaction between supply and demand, where any change in one of these factors can be reflected in the final market price. In addition, market prices are also subject to various external influences, including economic news, political events, and other unexpected factors.

To understand these complex movements, technical analysts use different tools and methods. This includes different types of charts and technical indicators [19]. In addition, technical analysts often use special jargon to describe the various patterns and trends they observe in the market. One of the key aspects of technical analysis is trend recognition [19]. Trends can be upward, downward, or horizontal, and identifying these trends can help traders make informed trading decisions. In addition, technical analysis also deals with identifying different patterns in the market, such as head and shoulders, double tops and bottoms, and others. Recognizing these patterns can help traders predict future price trends and make informed trading decisions [20].

However, it is important to note that technical analysis is a more complex science. While it can provide useful insights and help traders make informed decisions, there is always the risk that the market will react differently than expected. Therefore, combining technical analysis with other tools and methods is important to achieve as much accuracy as possible in predicting price trends.

2.2.2. Technical Indicators

Technical indicators are key tools in the world of trading that enable traders to analyze market price trends and make informed decisions. Using various mathematical formulas and historical data, these indicators help identify market trends, recognize potential reversals, and assess volatility. While some indicators are simple and provide basic market information, such as moving averages (MA), others are more complex and tailored to specific trading strategies [18]. Despite their value, it is important to note that every technical indicator is infallible. The effectiveness of each indicator can vary depending on market conditions, so traders must combine technical analysis with other tools and information to make the best possible decisions on their trading activities [19].

In the remainder of this chapter, several popular technical indicators that have proven to be key tools in the world of trading are analyzed. These indicators not only help to understand market trends but will be the basis for developing and implementing hybrid trading strategies that are the basis of this paper, providing us with better informed and considered decisions in the market.

Exponential Moving Average (EMA) is one of the most popular technical indicators that provides a smoother version of financial asset prices, filtering out random price fluctuations. It is used to identify trends, taking the average prices over a certain period [21]. The formula for EMA is

$$EM{A}_t=\mathsf{\alpha }\cdot S_t+\left(1-\mathsf{\alpha }\right)\cdot EM{A}_{\mathrm{t}-1}$$

Moving Average Convergence/Divergence (MACD) is an indicator that measures the difference between two EMAs, usually short term and long term. The MACD line represents that difference, while the signal line is the EMA of the MACD line itself [22]. The formulas are

$$MAC{D}_t=EM{A}_t-EM{A}_n$$

including the formula for the signal line:

$$Signa{l}_t=EM{A}_p\cdot MAC{D}_t$$

The momentum rule represents the difference between the current closing price P_t and the closing price n − 1 of the previous period P_t−n+1. Through this difference, the indicator provides insight into the direction and strength of the current market trend. A positive value may indicate an increasing trend, while a negative value may indicate a decreasing trend [19].

$${\mathrm{Indicator}}_{\mathrm{MOM}\left(n\right)}^t={\mathrm{MOM}}_t\left(n\right)=P_t-P_{t-n+1}$$

The moving average (MA) is a basic technical indicator to smooth prices and identify market trends. A moving average is calculated by taking the arithmetic average of a certain number of past prices. The most commonly used type of moving average is SMA (Simple Moving Average), which takes the arithmetic average of prices for a certain number of days [23]. The formula for SMA is

$$SM{A}_t\left(n\right)=\frac{1}{n}{\displaystyle \sum }_{i=0}^{n-1}{P}_{t-i}$$

2.2.3. Trading Strategies

While technical indicators provide insight into market trends and behavior, trading strategies based on them allow traders to turn this information into concrete trading plans. This chapter looks at several prominent trading strategies, such as the MACD and TEMA strategies, which have proven to be key tools of many successful traders [22,23].

The MACD (Moving Average Convergence Divergence) strategy is an abbreviation for Moving Average Convergence/Divergence and is a technical indicator that measures the difference between two exponentially moving average values (EMA). Based on this indicator, a trading strategy can be developed using the difference between the MACD and signal lines to identify potential trading [22].

The strategy for moment t + 1 is defined as follows:

• Buy, if MACD_t crosses over Signal_t;
• Sell, if MACD_t crosses below Signal_t.

This strategy suggests a potential buy signal is generated when the MACD line crosses above the signal. In contrast, a potential sell signal is generated when the MACD line crosses below the signal.

The TEMA (Triple Exponential Moving Average) strategy is used to identify market trends and combat false market signals. This strategy is based on three Exponential Moving Averages (EMAs) representing short, medium, and long-term periods. A short-term EMA (often called a fast EMA) is the first to detect a possible change in trend. When the fast EMA crosses the medium and long-term EMA, it confirms a trend change [24].

Mathematically, if we define three periods as m, n, and p where m < n < p, the strategy for moment t + 1 is defined as follows:

• Buy if EMA_mt is higher than EMA_nt and if EMA_mt is higher than EMA_pt;
• Sell if EMA_mt is lower than EMA_nt and if EMA_mt is lower than EMA_pt.

Therefore, a buy signal is generated when the fast EMA crosses above the medium-term and long-term EMA. In contrast, a sell signal is generated when the fast EMA crosses below one of them (medium-term or long-term EMA).

The Momentum (MOM) strategy is a basic market strategy that compares the current closing price with the closing price of the previous period. This strategy assumes that the current market trend will continue, rising or falling [23].

The strategy for moment t + 1 is defined as follows:

• Buy, if P_t is higher than P_t−n+1;
• Sell, if P_t is lower than P_t−n+1.

This strategy suggests that if the current closing price is higher than the closing price n−1 of the previous period, there is potential for a continuation of the uptrend; therefore, a buy signal is generated. On the other hand, if the closing price is lower, it may indicate the potential to continue the downtrend, thus generating a sell signal.

The P-MA (Price Minus Moving Average) strategy is a trading rule that uses moving averages to identify the direction of a market trend. The basic idea behind this rule is based on the lag of a moving average. When stock prices are uptrend, the moving average is below the price. On the contrary, when stock prices are downtrended, the moving average is above the price [23].

The strategy of moment t is defined as follows:

• Buy if P_t is higher than MA_t(n);
• Sell if P_t is lower than MA_t(n).

This strategy suggests that a potential buy signal is generated when the last closing price is above the moving average. In contrast, a potential sell signal is generated when the last closing price is below the moving average. In practice, traders often use the SMA (Simple Moving Average) as the moving average in this rule.

2.2.4. Challenges of the Traditional Approach

The traditional approach to trading that relies on technical indicators such as moving averages, although widely accepted, faces several challenges in today’s financial environment. Moving average (MA) is a basic technical indicator used for price prediction to identify market trends. While its application in identifying trends is undeniable, its effectiveness as a trading strategy may need to be revised in particular market conditions.

First, moving averages are inherently reactive. Because they are based on past prices, there is a lag in their response to new market information. This delay can result in missed opportunities or false signals in a fast-paced and volatile market. Second, while some moving averages, such as the 200-day SMA, have shown consistent performance across different market cycles, their past performance does not guarantee future results. Several factors can affect the success of moving averages as a trading strategy, including the volatility of market conditions and the behavior of other traders [23].

It is also important to note that moving averages can protect during market downturns and reduce potential returns during favorable market conditions. This is especially prominent in periods when the market is experiencing strong upward trends. Ultimately, while moving averages provide a valuable tool for traders, it is important to approach them with critical thinking and combine them with other tools and strategies to reach an optimal trading decision.

3. Empirical Literature Overview

The paper [24] explores the application of LSTM networks to predict movements in the S&P 500 market from 1992 to 2015. Through a comprehensive analysis, the paper demonstrates the superiority of LSTM networks in extracting significant information from financial time series. It is particularly noteworthy that LSTM networks, compared to other models, provide impressive daily returns of 0.46% before transaction costs, thus confirming their efficiency and potential in financial forecasting. In the following paper [25], the authors present a ModAugNet model, which brings a new data augmentation approach for stock market index prediction using LSTM networks. Through testing on the S&P500 and KOSPI200 indices, ModAugNet showed a significant improvement in forecasting accuracy. For the S&P500, the test means square error (MSE), mean absolute percentage error (MAPE), and mean absolute error (MAE) were reduced to 54.1%, 35.5%, and 32.7%, respectively, compared to the SingleNet model used for comparison. These results confirm the effectiveness of the ModAugNet framework in predicting market movements, highlighting its ability to extract information from financial time series efficiently.

In a recent study [26], the authors focused on predicting stock movement performance using the ensemble method of independent and parallel LSTM neural networks. Applying this model to the OMX30 stock index showed that a portfolio based on LSTM predictions provided better average daily returns and higher cumulative returns over time than a randomly selected portfolio or a portfolio containing all index stocks. In addition, the LSTM portfolio exhibited lower volatility, resulting in higher risk-return ratios. The research suggests that there is room for further improvement of this approach, such as applying a reduced learning rate or more systematically deciding to retrain the model to achieve even better results.

By presenting the Transformers architecture [27] as a “state-of-the-art” approach in the world of machine learning in general, with an emphasis on application in NLP, the application was also studied in the context of predicting financial time series [28]. This model uses an attention mechanism to capture long-term dependencies and interactions among features and perform multiple time series forecasting. The research focuses, in particular, on applying the Transformer model in portfolio management, which is used for trend-following strategies and volatility forecasting in multi-period portfolio optimization [29]. Although the Transformer model shows strong results in forecasting time series, the research also highlights the challenges of applying machine learning in finance, such as maintaining a balance between model complexity, tuning quality, and forecast quality.

The paper [22] presents an LSTM network model for predicting the value of observed shares and stock market indices. This structurally simple model is used to analyze prices over a certain number of previous trading days to optimize profitability instead of exclusive accuracy in predicted prices. The key innovation lies in a trading strategy that does not act solely based on predicting the rise or fall of the price but uses the distribution of predicted returns to make decisions based on the expected profitability of possible trading action. The strategy was tested on major US stock indexes such as the S&P 500, DJIA, NASDAQ, and Russell 2000 over the period of 2010 to 2018 and achieved impressive cumulative returns of 329%, 241%, 468%, and 279%, outperforming the standard “buy-and-hold” strategy and other mentioned approaches. After the publication of the previous paper [30], the same authors conducted new research [11]. They improved the approach by adding additional models that include Random Forests and gradient-boosted trees to provide a more comprehensive analysis and achieve better results. The strategy achieved cumulative returns of 350% for the S&P 500, 403% for the DJIA, 497% for the NASDAQ, and 333% for the Russell 2000, surpassing the results of the previous paper.

In the paper [23], the authors used machine learning techniques: Linear Model (LM), Artificial Neural Network (ANN), Random Forests (RF), and Support Vector Regression (SVR) in combination with traditional trading strategies TEMA and MACD, for testing these hybridized strategies with traditional ones for three major indices—Ibex35 (IBEX), DAX, and Dow Jones Industrial (DJI)—from 2011 to 2019. Proposed hybridized strategies were better than the standard ones, while authors emphasize that more research is necessary to confirm the effectiveness of hybrid strategies. Our paper, therefore, refers to the paper [31], whereas, in our paper, we use only the LSTM approach, taking into account that the LSTM in the literature is the one that has proven to be the best in predicting stock prices [30,31,32]. Our paper is also unique because we use MOM and P-MA traditional strategies and introduce individual stocks into the analysis.

The following paper [33] presents an interesting new trading strategy that uses Deep Neural Networks (DNN) to predict future stock prices. After an exploratory analysis of ten securities, those with the same volatility coefficient were selected to train the DNN network that predicts prices for the next 30 days. The proposed system then determines the most effective technical indicators by applying them to DNN predictions. The results showed that DNN enables the selection of a trading strategy that increases the expected return and some other financial indicators. In conclusion, this approach, which combines DNN predictions with technical indicators, offers traders better trading guidance with the potential for higher returns.

Despite the growing number of papers, making investment decisions on the stock market related to trading strategies and the application of artificial intelligence still needs to be sufficiently researched, so every paper on that topic is of exceptional importance [34].

4. Materials and Methods

Given the complexity and dynamism of financial markets, using advanced techniques and tools to achieve precise and reliable results was crucial. At the heart of the methodology were Tensorflow and the Keras library, which were used to develop LSTM models, while Numpy and Pandas were used for data processing and analysis. The integration of LSTM with traditional trading strategies made it possible to generate precise trading signals. Backtrader, a tool specialized for trading simulation (backtest), was used to simulate and evaluate strategy performance.

4.1. LSTM Network Architecture and Operation

The architecture of Long Short-term Memory (LSTM) neural networks represents a sophisticated structure that differs from traditional feedforward neural networks. The core of this architecture is the LSTM cell, which is equipped with a series of specific gates that enable precise control of the flow of information [35]. These gates and their functionality are key to understanding the LSTM’s ability to model long-term dependencies. In Figure 1, we can see a graphic representation of the architecture of an LSTM cell [36], and below are presented in detail the mathematical formulas that describe the operation and interactions within the LSTM cell, providing a deeper understanding of each architecture component.

The forget gate is the first step in an LSTM cell and is responsible for deciding which information from the cell’s previous state should be kept and which should be discarded. Here, W_f is the weight associated with the forget gate, x_t is the current input vector, h_t−1 is the previous output, and b_f is the bias. The sigmoid function σ ensures that the output is between 0 and 1, meaning that a value of 1 keeps the information, while a value of 0 discards it.

$${\mathrm{f}}_{\mathrm{t}}=\mathsf{\sigma }\left({\mathrm{W}}_{\mathrm{f}}\cdot \left[{\mathrm{h}}_{\mathrm{t}-1},{\mathrm{x}}_{\mathrm{t}}\right]+{\mathrm{b}}_{\mathrm{f}}\right)$$

Input gate and candidate value—Once we decide what to forget, the next step is to decide what new information should be added to the cell state. The input gate i_t decides which values in the cell state should be updated using the weights W_i, the previous output h_t−1, the current input x_t, and the bias b_i. Then, a “candidate value” vector C_t is created, containing the possibly new information to be added, using the weights W_C, the previous output h_t−1, the current input t_x, and the bias b_C.

$${\mathrm{i}}_{\mathrm{t}}=\mathsf{\sigma }\left({\mathrm{W}}_{\mathrm{i}}\cdot \left[{\mathrm{h}}_{\mathrm{t}-1},{\mathrm{x}}_{\mathrm{t}}\right]+{\mathrm{b}}_{\mathrm{i}}\right)$$

$$\widetilde{C_t}=\tanh \left(W_C\cdot \left[h_{t-1},x_t\right]+b_C\right)$$

Cell state update—Now we combine the previous two steps: multiply the forget gate f_t by the previous cell state C_t−1 to discard the information we do not want and add the result of the input gate i_t multiplied by the “candidate value” C_t. The result is our new cell state that combines the previous information with the new one.

$$C_t=f_t\times C_{t-1}+i_t\times \widetilde{C_t}$$

Output gate and final output—The last step in an LSTM cell is to decide what to output. This is carried out by taking the current input x_t and the previous output h_t−1, and passing them through a sigmoid function, which gives the output gate o_t. The weights W_o and the bias b_o are connected with this step. Then, the current cell state C_t is passed through the tanh function (which resumes the values between −1 and 1) and is multiplied by the output gate o_t to obtain the final output of the h_t LSTM cell.

$$o_t=\mathsf{\sigma }\left(W_o\cdot \left[h_{t-1},x_t\right]+b_o\right)$$

$$h_t=o_t\times \tan \mathrm{h}\left(C_t\right)$$

This description of the operation of an LSTM cell explains how the cell decides which information to keep, which to discard, how to update its state, and what to output as final output. Given this detailed analysis, it is clear why LSTM networks can efficiently model long-term sequential data dependencies, thus achieving outstanding results in many deep learning applications, especially finance [7].

4.2. Data Selection and Processing

Data are downloaded through the yfinance library, which downloads data directly from the Yahoo Finance platform. The stock market indices S&P 500 (SPY) and Dow Jones Industrial Average (DIA) were selected for the analysis as relevant indicators of the state of the financial market, and two dominant stocks on the market were additionally included: Apple Inc. (AAPL) and Microsoft Corp. (MSFT), along with TSLA (Tesla), BRK-B (Berkshire Hathaway Inc. Class B), NVDA (Nvidia), JPM (J. P. Morgan), and XOM (ExxonMobil).

The collected data cover the period from 1 January 2015 to 1 January 2020 for model training, while data from 1 January 2020 to 1 October 2023 are used to simulate trading. In addition to basic data such as the closing price, the opening price, and the lowest price, the technical indicators that proved to be the most relevant [37], detailed in

Table 1. Selected technical indicators and their formulas [37].

Name of Technical Indicator	Formula
Simple 10-day moving average	$SMA=\frac{C_t+C_{t-1}+\dots +C_{t-9}}{n}$
Weighted 10-day moving average	$WMA=\frac{\left(10\right)C_t+\left(9\right)C_{t-1}+\dots +C_{t-9}}{n+\left(n-1\right)+\dots +1}$

, were integrated to enrich the input data set and thereby achieve more precise predictions of the future closing price of the security.

Data processing is a key stage in data preparation for use within the model. Transforming data into a form acceptable for use within a machine learning model requires careful application of appropriate techniques. Before the processing, the input variables were defined and contained the technical indicators shown in Table 1 and the past closing price, opening price, and lowest price. The input variables were used to predict the future closing price of the selected security.

A further step in data processing was scaling, which enabled faster execution of machine learning models. All variables were scaled to values ranging from −1 to 1 using an appropriate technique. The preliminary steps also involved dividing the data into a training and a test set [38].

4.3. LSTM Model Specification

The Python programming language with the Tensorflow, Keras development environment was used to create the prediction model. Keras is recognized as a highly abstracted development environment intended for fast and efficient prototyping of Artificial Neural Network models. This selection underscores the importance of utilizing advanced computational tools and libraries to harness the full potential of machine learning in financial analysis.

One of the key steps in creating the model was optimizing the hyperparameters to achieve optimal performance. In this context, hyperparameters refer to the number of neurons in a particular layer of the neural network, the values of a particular optimization technique, and the number of training epochs. To improve the performance of the model, the “bach normalization” techniques were used to speed up training, and the “dropout” technique was used to prevent “overfitting.” The optimization algorithm “Adaptive Moment Estimation” (ADAM) [17] was chosen for this model. Fine-tuning these settings was crucial for ensuring the model accurately predicts market trends.

A random parameter search technique was utilized to identify the optimal hyperparameter values due to limitations in computational resources and the time required for experiment execution. This approach efficiently narrows down the search for the best hyperparameter combinations, in contrast to grid search, which iteratively evaluates every possible combination. We significantly reduced the computational load and time investment by employing random search, ensuring a practical yet effective exploration of the hyperparameter space, as demonstrated in

Table 2. Presentation of the architecture and parameters of the prediction model.

Hyper Parameters	Value
First layer (LSTM cell)	512
First dropout	0.1
Second layer (LSTM cell)	512
Second Dropout	0.1
Third layer (dense celse (Relu))	64
Fourth layer (dense celse (Relu))	1
Adam	0.1
Bach	128
Epoche	300

.

4.4. Integrating LSTM into Trading Strategies

In the research framework, traditional trading strategies were upgraded by integrating the LSTM (Long Short-term Memory) model of neural networks. The integration of LSTM aims to improve the ability to predict trading signals accurately. In basic trading strategies, instead of using the current day’s closing price to generate technical indicators and make trading decisions, an LSTM model is applied to predict the future closing price. Forecasting the closing price is important for making good decisions and increasing the efficiency of adopted trading strategies [39].

Based on that predicted closing price, the technical indicators that are the basis of each used trading strategy were further calculated to make a buy or sell decision. Central to this integration was the model’s ability to recognize and interpret complex patterns in historical data, using them to create more accurate predictions. As a consequence of this integration, the trading strategy’s ability to generate accurate trading signals should be improved, potentially contributing to better market performance when using machine learning methods [24,25]. Construction of the model and estimation process is highly important because it affects model efficiency [40].

4.5. Simulation Process and Evaluation Criteria

We simulated how well different trading strategies worked, comparing those that used the LSTM approach to those that did not. The previously mentioned Backtrader Python library was used to perform simulations and tests. This library enables a detailed analysis of the performance of trading strategies on historical data.

The cumulative return on investment metric expressed in percentages was used to measure the success of a particular strategy. Cumulative return on investment is calculated for the selected period to assess the overall profitability of each strategy. This metric provides a clear view of the total profit or loss generated over some period.

The “buy and hold” strategy was used as a reference point for performance comparison. This passive trading strategy served as a benchmark, comparing active trading strategies and a simple approach to holding stocks in the long term. Each basic trading strategy was tested in two variants: without future price prediction and with future price prediction using the LSTM model. This made it possible to assess the LSTM model’s contribution to improving the trading strategies’ performance.

When evaluating the accuracy of LSTM models, two key metrics are measured: the mean squared error (MSE) and the mean squared absolute error (MAE). These metrics provide information on the difference between actual and predicted values, allowing the model’s accuracy in predicting future prices to be assessed.

5. Research Results

5.1. Overview of LSTM Prediction Model Results

To adequately assess the precision and productivity of the model in predicting future prices, two key metrics were used, as already mentioned: Mean Square Error (MSE) and Mean Absolute Error (MAE). These metrics allow a quantitative assessment of the difference between actual and predicted values, with smaller values indicating higher model accuracy.

Table 3. Overview of LSTM prediction model results.

Simbol	Mean Square Error (MSE)	Mean Absolute Error (MAE)
SPY	0.00040	0.01432
DIA	0.00035	0.01355
AAPL	0.00030	0.01293
MSFT	0.00031	0.01371
TSLA	0.00139	0.02608
BRK-B	0.00044	0.1653
NVDA	0.00034	0.01446
JPM	0.00041	0.01547
XOM	0.00168	0.03267

presents the calculated values of MSE and MAE for nine selected securities: SPY, DIA, AAPL, MSFT, TSLA, BRK-B, NVDA, JPM, and XOM.

The results show that the model consistently predicts future prices for different stocks accurately, evidenced by the low MSE and MAE values for all listed securities.

In addition, a visual comparison of actual and predicted prices for the SPY security can be seen in Figure 2. This figure provides additional insight into the model’s performance, allowing a visual assessment of the accuracy of the predictions versus actual prices.

5.2. Comparison of Performance of Different Strategies

It was assumed that the initial capital for simulation purposes was $10,000. Using the available capital, as many shares as could be covered by that amount were bought at each buy signal. On the other hand, all the held shares were sold on every sell signal, regardless of their number. It is important to note that this simulation did not consider the trading commission. However, it is possible to add that aspect to the simulation to obtain an even more realistic picture of the performance of the strategies.

Table 4. Presentation of the percentage cumulative return according to the strategy.

Symbol	Trading Strategy
	MACD		TEMA		MOM		P-MA		B/H
	STAND.	LSTM	STAND.	LSTM	STAND.	LSTM	STAND.	LSTM	STAND.
SPY	31.74%	31.71%	20.63%	24.36%	26.94%	43.70%	−4.40%	45.52%	31.90%
DIA	15.69%	11.59%	−1.89%	8.00%	24.64%	33.03%	−7.13%	9.03%	16.87%
AAPL	135.84%	107.26%	150.72%	98.26%	25.25%	87.24%	23.26%	86.53%	128.91%
MSFT	−4.21%	46.27%	18.63%	19.13%	−6.46%	91.59%	28.84%	15.49%	97.61%
TSLA	312.74%	461.99%	574.81%	546.44%	128.63%	284.08%	874.81%	408.64%	746.49%
BRK-B	53.59%	30.57%	46.27%	42.08%	−0.51%	36.99%	60.07%	67.51%	53.58%
NVDA	183.22%	166.06%	286.00%	421.89%	36.03%	319.80%	185.26%	375.02%	624.52%
JPM	−22.24%	−20.13%	−27.72%	−28.72%	−26.44%	0.76%	−24.24%	−2.36%	5.26%
XOM	85.27%	110.77%	41.39%	47.28%	25.16%	19.94%	64.67%	77.35%	67.11%

shows the total cumulative returns for the period from 1 January 2020 to 1 October 2023 for different trading strategies, both for traditional (labeled as “STAND”) and traditional ones with integrated LSTM (labeled as “LSTM”). A “buy and hold” (B/H) strategy is also a reference point.

In Figure 3, we visually compare trading actions for the DIA security using the MOM (Momentum) strategy. The left side of the figure shows the trading points for the standard MOM strategy, which achieved a cumulative return of 24.64%.

On the other hand, on the right side of the figure, trading points for the MOM strategy with an integrated LSTM model are shown with a cumulative return of 33.03%. It is visible that the trading points are different between the two strategies, which indicates the differences in trading decisions based on the predictions of the LSTM model compared to the traditional strategy.

Analyzing the results, it can be noted that some trading strategies with an integrated LSTM model achieved better performance than traditional strategies. In contrast, others achieved similar or somewhat weaker results. It is particularly interesting to observe how performance differs between different securities, which indicates the specifics of each stock and its reaction to certain trading signals within trading strategies.

5.3. Interpretation and Discussion of the Results

To better understand the results obtained from this research, it is important to consider the context of the financial market from 2020 to 2023, for which trading strategies were tested. This period was marked by significant market fluctuations, including the consequences of the global COVID-19 pandemic that began at the end of 2019. The pandemic has caused major economic disruptions, resulting in major market swings. To navigate these fluctuations, LSTM models, with their deep learning capabilities, offered a way to predict market movements, showcasing their resilience in the face of economic uncertainties triggered by global events.

In addition, the technology sector, led by companies such as Apple (AAPL) and Microsoft (MSFT), experienced significant growth. In contrast, ETFs following the broader market indices, such as the S&P 500 (SPY) and the Dow Jones Industrial Average (DIA), exhibited greater stability and less pronounced profitability [37,41]. This divergence in market behavior illustrates the versatility of LSTM models in analyzing and predicting trends across different sectors, highlighting their potential for future applications in a wide array of financial markets. The inclusion of a more extensive set of stocks, such as Tesla (TSLA), Berkshire Hathaway (BRK-B), NVIDIA (NVDA), JPMorgan Chase (JPM), and ExxonMobil (XOM), has shed light on varied market responses to the deployed trading strategies. For instance, TSLA’s exceptional growth contrasted with JPM’s downturn, highlighting the necessity for adaptable strategies that reflect the diverse nature of market sectors [39]. The differential performance of strategies, such as MACD, TEMA, MOM, and P-MA, when enhanced with LSTM, underscores the model’s potential to leverage unique market conditions and stock-specific behaviors. ETFs tracking indices like SPY and DIA provide a macroeconomic perspective, whereas the individual stocks’ susceptibility to rapid price shifts offers a robust test of the LSTM model’s predictive strength. The LSTM model’s significant benefits in forecasting the movements of more volatile stocks are attributed to its sophisticated pattern recognition capabilities.

These findings are pivotal in customizing LSTM models to suit various market conditions, bridging the gap between theoretical predictions and practical trading outcomes. As evidenced across different sectors, the LSTM model’s robustness and adaptability affirm its potential beyond tech stocks, providing insights into the overall market dynamics. This comprehensive study underlines the necessity for a differentiated approach to each security, echoing that no singular strategy can be universally effective [39]. Moreover, the LSTM model, tested on data from 2015 to 2020, demonstrated its ability to outperform traditional trading strategies, aligning with the literature that suggests LSTM’s role in reshaping stock market strategies [37]. Despite being tested on historical data, the model’s ability to generate profitable trading signals, particularly for individual stocks, illustrates its resilience and predictive power. This adaptability of LSTM models to various market conditions and their sustained accuracy over time are significant advancements that address current gaps in applying machine learning to financial trading [33], suggesting a paradigm shift towards more dynamic, data-driven decision-making processes in finance.

Integrating LSTM models into trading strategies has notably enhanced performance for most strategies and securities. However, it remains crucial to contextualize these strategies and recognize the distinct nature of different securities. The integration process has also illuminated the importance of ongoing model optimization to maintain relevance and accuracy in a constantly evolving market landscape. While LSTM models provide valuable insights and have the potential to improve trading performance, traders must remain cognizant of the models’ limitations and the inherent risks of predictive-based trading. By acknowledging these challenges and continuously refining the models with new data and market developments, we can further narrow the gap between theoretical modeling and practical trading success, charting a course for more reliable and effective trading strategies in the future.

6. Conclusions

In the modern age of economic globalization and technological progress, financial markets are experiencing an unprecedented accumulation of data. This accumulation, which results from technological innovation and growing interest in trading, challenges traditional analysis methods. In this context, the application of machine learning, especially LSTM neural networks, has proven to be crucial in analyzing and predicting financial market movements. This research focused on integrating LSTM models into traditional trading strategies to improve performance. The results showed that the LSTM model, tested on data from 2015 to 2020, outperformed most traditional strategies, especially for individual stocks like Apple and Microsoft. However, it is important to consider contextual factors, such as the economic consequences of the COVID-19 pandemic and differences in volatility between stock indices and individual stocks.

While ETFs, which follow stock indexes such as the S&P 500 and the Dow Jones Industrial Average, provided a more stable view of market trends, individual stocks were more susceptible to sudden price changes. This difference in volatility could explain the differences in the performance of trading strategies, suggesting that the LSTM model may be better suited to predict more volatile stocks, given the larger relative difference in returns between standard strategies and those that use an additional LSTM approach. Integrating LSTM models into trading strategies has significantly improved performance for most strategies and securities. While an LSTM model can provide valuable insights and improve trading performance, traders should always be aware of the model’s limitations and the potential risks associated with trading based on predictions. Investors should be aware that there is no single optimal strategy for all possible securities but that they should be selected based on the tests conducted. Our hybrid trading strategies improved most traditional trading strategies, and we confirmed the importance of applying artificial intelligence in novel trading decisions.

Moreover, this study underscores the transformative potential of LSTM models within the broader context of financial market analysis, illustrating their capacity to enhance trading outcomes and inform risk management practices. By elucidating the specific conditions under which LSTM models excel, our research contributes valuable insights into the strategic deployment of machine learning techniques in financial decision making. The adaptability and predictive accuracy of LSTM models, as demonstrated in this research, pave the way for their application in other areas of financial analysis, including portfolio management and algorithmic trading.

Future research could significantly enhance the field by integrating Deep Reinforcement Learning (RL) [42] techniques with Generative Pre-trained Transformer (GPT) [43] models, in addition to the existing Long Short-term Memory (LSTM) frameworks for real-time market analysis. Including DRL methods would bring a sophisticated approach to sequential decision making, allowing trading systems to adjust strategies dynamically based on market conditions. Meanwhile, leveraging GPT models for their natural language processing (NLP) capabilities could offer a deeper understanding of textual data that impacts market trends, such as news articles and financial reports. This comprehensive approach would improve the adaptability of trading strategies across different markets and asset classes and pave the way for developing autonomous trading systems capable of understanding and reacting to the complex dynamics of financial markets. By combining these advanced machine learning techniques, future research can lead to the creation of more sophisticated, efficient trading strategies, marking a significant step forward in the application of AI in finance.