Predictive Patterns and Market Efficiency: A Deep Learning Approach to Financial Time Series Forecasting

Abstract

This study explores market efficiency and behavior by integrating key theories such as the Efficient Market Hypothesis (EMH), Adaptive Market Hypothesis (AMH), Informational Efficiency and Random Walk theory. Using LSTM enhanced by optimizers like Stochastic Gradient Descent (SGD), Adam, AdaGrad, and RMSprop, we analyze market inefficiencies in the Standard and Poor’s (SPX) index over a 22-year period. Our results reveal “pockets in time” that challenge EMH predictions, particularly with the AdaGrad optimizer at a size of the hidden layer (HS) of 64. Beyond forecasting, we apply the Dominguez–Lobato (DL) and General Spectral (GS) tests as part of the Martingale Difference Hypothesis to assess statistical inefficiencies and deviations from the Random Walk model. By emphasizing “informational efficiency”, we examine how quickly new information is reflected in stock prices. We argue that market inefficiencies are transient phenomena influenced by structural shifts and information flow, challenging the notion that forecasting alone can refute EMH. Additionally, we compare LSTM with ARIMA with Exponential Smoothing, and LightGBM to highlight the strengths and limitations of these models in financial forecasting. The LSTM model excels at capturing temporal dependencies, while LightGBM demonstrates its effectiveness in detecting non-linear relationships. Our comprehensive approach offers a nuanced understanding of market dynamics and inefficiencies.

1. Introduction

The growing adoption of artificial intelligence (AI) and machine learning (ML) in financial markets has spurred substantial research aimed at testing the validity of the Efficient Market Hypothesis (EMH). First proposed by [1], the EMH asserts that stock prices reflect all available information, making it impossible to consistently achieve excess returns. Scholars such as [2,3] have rigorously tested the boundaries of the EMH, often by developing models designed to predict financial returns and exploit market inefficiencies. Despite the demonstrated predictive power of some of these models in controlled environments, their application in real-world contexts has often failed to deliver consistent profits for investors.

This paradox, as highlighted by [4], underscores a fundamental dilemma in financial research: while forecasting models may achieve high predictive accuracy, markets tend to self-correct too quickly, preserving their efficiency and limiting the opportunities for arbitrage. Much of the prior literature has concentrated on refining forecasting models, particularly those utilizing AI and ML techniques to predict market behavior [5,6,7,8,9].

Recent advances in AI and ML, however, have produced promising developments in addressing some of these challenges. [10] introduced a deep neural network decision model for predicting financial risk in carbon trading, incorporating multiple economic and environmental factors, significantly improving accuracy compared to traditional linear methods. Similarly, [11] developed the RHINE regime-switching model, which captures dynamic market shifts and improves the prediction of financial regimes through non-linear kernel representations, and [12] further extended AI’s potential by introducing a deep reinforcement learning model, enabling agents to adapt to complex and uncertain environments. Moreover, [13] proposed a hybrid model combining Convolutional Neural Networks (CNNs) and Transformers, which outperforms traditional methods such as ARIMA in capturing short-term and long-term dependencies in financial time series. These advancements underscore the growing recognition of AI’s ability to navigate the complexities of non-linear financial data.

However, even with sophisticated techniques such as LSTM models and deep neural networks, the consistent exploitation of market inefficiencies remains elusive [14]. This raises an important question: is forecasting alone enough to challenge the principles of market efficiency, or should broader market dynamics, including informational and behavioral factors, also be considered?

Unlike many studies that focus solely on improving forecasting accuracy, this research takes a broader approach by incorporating several key theories of market efficiency and behavior. These include the Random Walk theory, the Adaptive Market Hypothesis (AMH), and Informational Efficiency. [15,16] provides a dynamic understanding of market behavior, proposing that markets may exhibit temporary inefficiencies driven by environmental and behavioral changes. Contrasting the more rigid assumptions of the EMH, the AMH acknowledges that market efficiency fluctuates over time [17,18,19] based on factors like investor sentiment, market conditions, and external shocks. Deviations from the Random Walk behavior of asset prices create “pockets in time” where predictive models can outperform. Research in market efficiency has gained significant traction in recent years, yet studies specifically addressing the AMH remain fewer in comparison to those centered on the EMH. Scholars have examined various aspects of market efficiency, such as market capitalization, trading volume, and cryptocurrency markets [20,21,22,23]. Many employ the Martingale Difference Hypothesis (MDH) to transition from a static EMH perspective to a more dynamic analysis aligned with the AMH framework. While these studies have advanced the field, our research takes a broader view, offering a more comprehensive evaluation of the EMH through multiple methodological experiments. The complete theoretical background is presented in Supplementary S1.

Building on this theoretical foundation, our study investigates market inefficiencies not solely through the lens of forecasting, but by employing a combination of market efficiency techniques. We integrate insights from the EMH, AMH, Information Efficiency, and Random Walk theories, examining the daily returns of the Standard and Poor’s (SPX) index over a 22-year period. Using LSTM models optimized through Stochastic Gradient Descent (SGD), AdaGrad, Adam, and RMSprop, we simultaneously apply statistical measures of efficiency to assess market behavior. While the EMH serves as a foundational framework in our analysis, we expand its scope by incorporating the AMH, which provides a dynamic view of market behavior. The AMH recognizes that market efficiency fluctuates over time due to environmental, behavioral, and structural factors, which may create temporary inefficiencies [16]. This approach contrasts with the static nature of the EMH and allows us to explore how market conditions evolve in response to investor behavior, external shocks, and informational changes. Moreover, our study goes beyond forecasting models, which dominate much of the existing literature, by integrating market efficiency testing techniques. We utilize the Martingale Difference Hypothesis with the DL and GS tests to assess whether financial return patterns deviate from the Random Walk model, which is a critical indicator of market inefficiency. In this way, our research bridges the gap between predictive accuracy and the underlying structural dynamics of financial markets. We also emphasize “informational efficiency”, analyzing how quickly new information is absorbed into stock prices, offering a holistic view of market efficiency beyond predictive accuracy. By incorporating these multiple perspectives—EMH, AMH, Random Walk theory, and Informational Efficiency—our study offers a broader and more nuanced evaluation of market dynamics.

Our study addresses a critical gap in the literature by combining forecasting models with market efficiency frameworks to explore how predictive models interact with periods of inefficiency. Although recent advancements in AI and ML forecasting models have shown encouraging results [10,11,12,13], our research contributes by offering a broader, multi-faceted framework of market efficiency. We argue that forecasting alone is insufficient to refute the EMH; rather, inefficiencies should be understood as transient phenomena influenced by both market structure and information flow.

By examining multiple optimization techniques and applying them across different LSTM model configurations, our research provides insights into how these models perform under various market conditions. However, the objective is not to determine the optimal forecasting model, but to analyze how LSTM models, particularly with AdaGrad optimization and HS 64, interact with short-term inefficiencies. In this context, we explore “pockets in time”, where temporary market deviations allow forecasters to identify profitable opportunities, as suggested by [4].

This study makes several key contributions. First, it is the first to test the “pockets in time” theory using LSTM models optimized with AdaGrad to challenge the EMH by identifying localized inefficiencies. Our findings align with recent studies [24,25,26], which underscore the potential of machine learning in financial markets. Second, by incorporating statistical inefficiency tests, we bridge the gap between forecasting accuracy and practical market application, as noted by [27,28]. Finally, our research contributes to the understanding of market efficiency as a dynamic, evolving phenomenon, influenced by structural shifts and external factors, in line with the theories of [16,29].

In an additional experiment, we incorporate two more forecasting techniques: ARIMA with Exponential Smoothing and LightGBM. These methods are employed alongside the LSTM model to offer a comprehensive comparison of forecasting approaches, each with distinct strengths in financial time series analysis. The LSTM model is particularly well suited for time series data as it captures long-term dependencies and patterns through its memory cells. By introducing ARIMA with Exponential Smoothing and LightGBM after the LSTM, this study aims to: (a) demonstrate the advantages of machine learning models, such as LightGBM, in detecting non-linear relationships, and (b) provide a comparative analysis with LSTM, which is specifically designed to address the temporal dependencies inherent in financial data. To mitigate the risk of overfitting in the LSTM model, we implemented dropout, a technique that randomly deactivates a percentage of neurons during training. This approach encourages the model to generalize better by preventing it from becoming overly reliant on specific neurons.

2. Methodological Framework

2.1. Data and Samples

The dataset encompasses historical data on SPX daily returns, sourced from the Bloomberg terminal. Notably, the SPX represents eleven distinct market sectors, serving primarily as a barometer for market activity within the United States. From daily SPX prices, we calculate returns. The designated time frame for both the training and testing phases extends from 13 January 2000 to 12 January 2021. This period sums up a total of 7672 observations (see Figure 1).

In pursuit of optimal performance for the forecasting model, this study implements data normalization. This normalization process is uniformly applied across all data categories, effectively scaling them into a range between 0 and 1, as depicted in Figure 2.

Subsequent to the normalization process, the dataset undergoes a division into training and testing subsets. Specifically, 6199 historical data points, spanning from 13 January 2000 to 31 December 2016, are allocated for the training set, while the remaining 1467 data points, covering the period from 1 January 2017 to 12 January 2021, are designated for the testing (validation) set, as illustrated in Figure 3. This study conscientiously circumvents the potential bias of predicting future trends using retrospectively acquired data. Consequently, the automated data splitting feature of the sklearn library’s train_test_split tool is deemed unsuitable. Instead, a manual approach is employed to partition the dataset, adhering to an 80% (training) and 20% (test) sample distribution.

We develop a forecasting model by utilizing the PyTorch API [30], a renowned open-source deep learning library tailored for constructing and training deep learning models. Additionally, the Seaborn Python library is employed for visualizing complex statistical plots derived from the requisite financial time series. Complementing this, the NumPy, Pandas, and Matplotlib libraries are integral in the creation of machine learning models, optimization procedures, and the testing of Random Walk models. For assessing market dynamics, the ‘crypto2’ package within the R programming environment is utilized to calculate the SPX returns, employing a volume-weighted average method across all market returns reported during the dataset period. The methodological framework of this process is systematically outlined in Figure 4.

2.2. Experimental Setup

2.2.1. Forecasting Model Formulation

Equilibrium

This study introduces a local “pockets in time” model, initially conceptualized by [4]. Central to this approach is the premise that investors, each equipped with distinct beliefs and information, opt for a forecasting model that optimizes a particular financial performance metric. In this context, agents remain oblivious to the precise nature of the forecasting model. Consequently, the utility of the mathematical expectations operator, traditionally integral to the definition of market efficiency, diminishes. It becomes more practical, therefore, to evaluate market efficiency on a local temporal scale, guided by the information set ${\varnothing }_t$ and the forecasting model ${\varphi }_{it}({\psi }_t;\widehat{{\chi }_t}$), selected from a range of available models ${\mathsf{\Sigma }}_t$. Underpinning this approach is the following assumption:

$$E\left[f_t(R_{t+1}^{°},{\varphi }_{it}({\psi }_t;\widehat{{\chi }_t}),c_t)\right]=0,$$

(1)

where $\widehat{{\chi }_t}$ is a vector of parameters assessed from the used data up to time t (for only the forecasting techniques available in this period) and ${\psi }_t\in {\varnothing }_t$. There is also an assumption for the absence of transaction costs:

$$E\left[f_t\left(R_{t+1}^{°},c_t\right)|{\mathsf{\Sigma }}_t\right]=0$$

(2)

where $c_t$ is a vector of transaction cost parameters for the $f_t$ set of possible transactions at time t. Considering this assumption, where ${\widehat{R}}_{it+1}^{°}$ is predicted value of $R_{t+1}^{°}$ from the $i_{th}$ forecasting model, efficiency local in time is presented as follows.

$$E\left[R_{t+1}^{°}{\widehat{R}}_{it+1}^{°}\right]=0$$

(3)

According to [4], certain models exhibit predictive capabilities; however, this does not contravene the EMH, as delineated in the preceding equation. This is attributed to the fact that such models do not constitute elements of the set ${\mathsf{\Sigma }}_t$. Nonetheless, the EMH does encounter violations when considering population expectations, as follows:

$$E\left[R_{t+1}^{°}|{\psi }_t\right]=0$$

(4)

where, for all transformations of ${\psi }_t$, variables in${\mathsf{\Sigma }}_t$ for tested market efficiency must be orthogonal to $R_{t+1}^{°}$.

Market Inefficiency

Originating from the hypothesis that contemporary forecasting techniques outperform in the local “packet in time”, it is inferred that the market exhibits inefficiency within these specific local time intervals:

$$\mathsf{{\rm Y}}\left[t_{beg},\dots t_{end}\right],0\le t_{beg}\le t_{end}\le T$$

(5)

where the interval t ∈ (0,1…T) and forecasting model ${\varphi }_{it}{\in \mathsf{\Sigma }}_t$ are identified at $t_{beg-1}$. According to [4], the given forecasting model ϕ_it and a vector of transaction cost parameters $c_t$ for the $f_t$ set of possible transactions at time $t$ are positive.

$$E\left[f_t(R_{t+1}^{°},{\varphi }_{it}({\psi }_t;\widehat{{\chi }_t}),c_t)\right]>0,\mathrm{for}t\in \mathsf{{\rm Y}}$$

(6)

Under the assumption of zero transaction costs, the expected cross-product of actual and predicted returns is positive.

$$E\left[R_{t+1}^{°}{\widehat{R}}_{it+1}^{°}\right]>0,\mathrm{for}t\in \mathsf{{\rm Y}}$$

(7)

However, when predictable patterns within local time intervals are discerned and transition into the realm of public knowledge, the prospect of accruing additional profits via transactions dissipates. This outcome is primarily due to the actions and adjustments of agents within the market.

2.2.2. Forecasting Model Formulation

We employ the LSTM neural network forecasting model due to its proficiency in retaining information over prolonged periods, effectively addressing the issue of long-term dependencies. The LSTM model is distinguished by its chain-like structure, which operates through a series of gates and layers within neural networks [31]. Central to the LSTM architecture is the cell state, initially introduced by [32]. This cell state extends across the entire LSTM, regulated by gates that control the addition or subtraction of information. The significance of information within the LSTM cell state is dynamically altered by these gates, which decide whether data are included or excluded. Additionally, the model includes closed cells, which store information from previous LSTM outputs or outputs of other layers (Figure 5). This design is fundamental to the memory capabilities of the LSTM. The LSTM forecasting process is characterized by three distinct gates within its cells.

• Forget gate (F): The forget gate is responsible for discarding information from the cell state that is considered no longer relevant. It utilizes the sigmoid function to evaluate the necessity of each piece of datum in the output and current cell state. This operation is concisely expressed in Equation (8).

  $$f_t={\sigma }_g\left(W_f\ast x_t+U_f\ast x_{t-1}+b_f\right)$$

  (8)
• Input gate (I): The input gate plays a critical role in integrating pertinent information into the cell state. Its decision-making mechanism for information inclusion operates in two distinct phases. In the first stage, a sigmoid function calculates which values need updating. Following this, a tan function creates a vector of potential new values (represented as c’) that may be added to the state. This dual-stage process is depicted in Equations (9) and (10).

  $$i_t={\sigma }_g\left(W_i\ast x_t+U_i\ast x_{t-1}+b_i\right)$$

  (9)

  $$c_t^{\prime }={\sigma }_c\left(W_c\ast x_t+U_c\ast x_{t-1}+b_c\right)$$

  (10)
• Output gate (O): The output gate is tasked with retrieving relevant information from the current cell state for subsequent presentation. It crafts the output by filtering information, a process governed by a sigmoid function that determines which components of the cell state are influential for the output. The final phase of this gate’s operation involves the multiplication of the tanh function’s output with that of the sigmoid function. This process is methodically delineated in Equations (11)–(13):

  $$o_t={\sigma }_g\left(W_o\ast x_t+U_o\ast x_{t-1}+b_o\right)$$

  (11)

  $$c_t=f_t\ast c_{t-1}+i_t\ast c_t^{\prime }$$

  (12)

  $$h_t=o_t\ast {\sigma }_c\left(c_t\right)$$

  (13)

  where $x_t$represents input vector of the LSTM unit, $f_t$ is forget gate, $i_t$ is input gate, $o_t$ is output gate, $c_t$is cell state, $h_t$is hidden state, ${\sigma }_g$ is sigmoid function, ${\sigma }_c$is tanh f function, $W$ and $U$ are weight matrices to be learned, and $b$ is the bias vector to be learned.

Considering these three gate functions in LSTM, we express our forecasting model ${\varphi }_{it}({\psi }_t;\widehat{{\chi }_t}$) as${\varphi }_{it}(F;I;O;\widehat{{\chi }_t}$), for $\widehat{{\chi }_t}$ vector of parameters assessed from the used data in steps F, I, O, up to time t.

2.2.3. Hyperparameter Tuning

In pursuit of identifying the optimal configuration of hyperparameters that enhance performance, generalization, computational efficiency, and overall accuracy of the LSTM model, a specific set of hyperparameters tuning is employed.

• Learning rate: A value of 0.01 is established through experimentation and tuning. This rate is identified as the most effective for facilitating faster convergence and improved performance of the LSTM model.
• Batch size: Set at 12,719, this size is selected to strike a balance between training speed (where the model processes 12,719 training examples simultaneously before updating internal parameters based on calculated gradients) and generalization performance.
• Look back: Fixed at 5, it indicates the number of previous time steps or input features used to predict the next time step in the LSTM model. This implies the use of the last 5 observed values for predicting the next value, typically arranged in a sliding window manner.
• Sequence length: Each input sequence consists of 10 consecutive time steps. The LSTM model utilizes this sequence to predict the next value in the time series.
• Weight decay: Applied to encourage the model to learn smaller, more generalized weights, helping to prevent overfitting and enhance the generalization of the LSTM results.
• Bidirectional LSTM: Utilized to incorporate both past and future context in sequence prediction.
• Number of training epochs: 20 epochs are deemed optimal to avoid underfitting and overfitting in the LSTM model;
• HS layers: This study progressively increases the HS layers from 4 to 256 across seven iterations during hyperparameter tuning, doubling the HS in each new iteration.

The LSTM model is trained using PyTorch’s RNN model, configured with specific input size (look_back) and HS ranging from 4 to 256 across seven iterations. The RNN model undergoes training for 20 epochs, iterating over the training data and extracting input variables (var_x) and corresponding target variables (var_y). The training involves feeding input variables into the RNN model, predicting outputs, calculating loss, and updating model parameters using a specific optimizer. In the evaluation phase, the model is set to evaluation mode for consistency in behavior. Training and test data are prepared, predictions are made, and outputs are converted to a NumPy array for analysis. Finally, the Matplotlib library is employed to visualize the real data, predicted training data, and predicted test data on a unified graph.

2.2.4. Training and Optimization

The LSTM forecasting process in this study is conducted for selected HS using four distinct optimizers: SGD, AdaGrad, Adam, and RMSprop. This study employs LSTM optimization in a dual capacity, acknowledging the absence of forecasting model ${\varphi }_{it}({\psi }_t;\widehat{{\chi }_t}$), from a set of existing models ${\mathsf{\Sigma }}_t$, which remains unknown to other investors. This is our study limitation, as the closest representation to ${\varphi }_{it}({\psi }_t;\widehat{{\chi }_t}$ is achieved through LSTM optimization. This study initially hypothesizes that optimization contributes to forecasting performance by enhancing accuracy.

This study employs four optimization techniques: SGD is presented as ${\varphi }_{it\left(SGD\right)}({\psi }_t;\widehat{{\chi }_t}$), AdaGrad as ${\varphi }_{it\left(AdG\right)}({\psi }_t;\widehat{{\chi }_t}$), Adam as ${\varphi }_{it\left(Ad\right)}({\psi }_t;\widehat{{\chi }_t}$), and RMSprop ${\varphi }_{it\left(RMS\right)}({\psi }_t;\widehat{{\chi }_t}$), as per expression (1). Building on the expression (6), this study posits that these new forecasting techniques surpass performance in local time packets, implying market inefficiency in these intervals. These models, ${\varphi }_{it\left(SGD\right)}{\in \mathsf{\Sigma }}_t$, ${\varphi }_{it\left(AdG\right)}{\in \mathsf{\Sigma }}_t$,${\varphi }_{it\left(Ad\right)}{\in \mathsf{\Sigma }}_t$, and ${\varphi }_{it\left(RMS\right)}({\psi }_t;\widehat{{\chi }_t}$), are projected to be presented as ${\widehat{R}}_{it\left(opt\right)+1}^{°}$ in the future period t + 1 in according to expression (7).

Each optimization technique is expected to yield different forecasting performances. SGD employs incremental gradient descent to minimize or maximize error through iterations. However, the primary limitation in the assumption of ${\varphi }_{it\left(SGD\right)}({\psi }_t;\widehat{{\chi }_t}$) is when ${\varphi }_{it\left(SGD\right)}$ is not convex or pseudo convex. In such a case, ${\varphi }_{it\left(SGD\right)}$ could converge to a local minimum. To overcome this limitation and increase the speed, robustness, and scalability of ${\varphi }_{it\left(SGD\right)}$, this study employs the AdaGrad of ${\varphi }_{it\left(AdG\right)}$ and of ${\varphi }_{it\left(Ad\right)}$ optimization functions. AdaGrad is more optimal for large-scale neural nets, assuming that, from expression (7), the ${\widehat{R}}_{it\left(AdG\right)+1}^{°}$ market is more efficient than ${\widehat{R}}_{it\left(SGD\right)+1}^{°}$, where ${\varphi }_{it\left(AdG\right)}({\psi }_t;\widehat{{\chi }_t}$) outperforms ${\varphi }_{it\left(SGD\right)}({\psi }_t;\widehat{{\chi }_t}$). Similarly, the Adam algorithm, an extension of SGD, updates the learning rate individually for each network weight, positing that ${\widehat{R}}_{it\left(Ad\right)+1}^{°}$ is more efficient than ${\widehat{R}}_{it\left(SGD\right)+1}^{°}$, where ${\varphi }_{it\left(Ad\right)}({\psi }_t;\widehat{{\chi }_t}$) outperforms ${\varphi }_{it\left(SGD\right)}({\psi }_t;\widehat{{\chi }_t}$) from expression (7).

2.2.5. The Adoptive Market Efficiency

To investigate the dynamics of efficiency and the identification of forecasting patterns, we utilize Martingale difference sequences applied to the daily returns of the SPX. A Martingale is characterized as following a Random Walk, which aligns with the concept of mean independent increments [33]. Given the unpredictable nature of SPX returns, this study adopts a transformation of a Martingale difference sequence in line with the method proposed by [34].

$$E\left[Y_t|,Y_{t-1}\right]=\mu ,\mathrm{a}.\mathrm{s}.,\mu \in R_{t+1}^{°},\mathrm{for}\mathrm{interval}t\in \left(\mathrm{0,1}\dots T\right)\mathrm{and}\mathrm{for}\mathrm{information}\mathrm{set}{\varnothing }_t$$

(14)

We assume that $Y_t$ follows a Martingale difference sequence, denoted as the one period log returns, where the daily returns of SPX are denoted as the natural logarithm in time t, and where $Y_t$represents the one period log returns for the first difference of $Y_t=\widehat{{\chi }_t}$ − $\widehat{{\chi }_{t-1}}$. The parameter μ represents a real number. This study converts the daily returns into a logarithmic function according to the following:

$$R_t^{°}=(L_{n{p}_t}-L_{n{p}_{t-1}})\times 100$$

(15)

where $L_{n{p}_t}$ is the natural logarithm of the daily returns of SPX on day t and $L_{n{p}_{t-1}}$ is the natural logarithm of the daily returns on day t − 1.

Considering the inherently chaotic, dynamic, and non-linear nature of financial returns [35], this study adopts both linear and non-linear measures to assess their dependence. To capture evolving linear and non-linear dependencies in SPX returns, we implement the DL consistent test and the GS test within a rolling window framework. These tests, DL and GS, are particularly valuable in addressing issues related to non-normality, size distortion, stationarity, and conditional heteroscedasticity.

Furthermore, both the DL and GS tests operate under the null hypothesis that an infinite number of autocorrelations are equal to zero, while simultaneously allowing for the presence of dependence beyond second moments [34]. In line with the methodology of [36], this study employs the Cramer–von Mises function to construct the DL test:

$${CvM}_{t.p.}={\displaystyle \frac{1}{{\mu }^2}}\sum _{i=1}^t{\left[\sum _{t=1}^t\left(R_t^{°}-{\widehat{R}}_t^{°}\right)·1\left({\widehat{R}}_{t.p.}^{°}\le {\widehat{R}}_{j.p.}^{°}\right)\right]}^2,$$

(16)

where $\mu$ is any real number in [0, 1], p is a positive number, ${\widehat{R}}_{t.p.}^{°}={\left({R^{°}}_{t-1},..,{R^{°}}_{t-p}\right)}^{,}$, and $1\left({\widehat{R}}_{t.p.}^{°}\le {\widehat{R}}_{j.p.}^{°}\right)$ are indicator functions.

Proposed by [36], we employ the GS test as an extension of the generalized spectral density function. The GS test demonstrates robustness to higher order dependence and effectively captures non-linear dependence in the conditional mean of asset returns. The null hypothesis of the GS test posits that the expected future value of a return remains unchanged, irrespective of the previous realization of the return. As delineated by [36], this null hypothesis is considered refuted when the squared value of the D statistic is comparatively large.

$$D_t^2={\displaystyle \frac{1}{{\lambda }^2{t}^2}}{\displaystyle \frac{T-j}{{\left(j\pi \right)}^2}}{\sum }_{t=j+1}^T{\sum }_{s=j+1}^{T}exp(-0.5{\left({R^{°}}_{t-j}-{R^{°}}_{s-j}\right)}^2)$$

(17)

Further, we employ a bootstrapping procedure to generate p-values for the DL and GS tests. This approach is adopted based on the premise, as noted by [36], that bootstrapping enhances the performance of these tests. Bootstrapping offers a more precise estimation of standard errors, which proves to be a more efficient measure of the variability of estimated parameters than p-values, especially in scenarios characterized by strong non-linear dependence. After the bootstrapping process, the p-values for the tests are derived from the bootstrap distribution. If these p-values fall below a specified significance level (either 1% or 5%), the null hypothesis is rejected. This rejection indicates, with a certain degree of confidence, that the returns of the SPX do not conform to a Martingale difference sequence.

Furthermore, all p-values are graphically represented on a timeline. This visual representation not only demonstrates the dynamics of market efficiency over the observed period, but also illustrates the changing predictability of cryptocurrency returns over time. This approach provides a comprehensive view of market behavior and its temporal evolution.

2.2.6. RW Benchmark Model

The RW model operates under the assumption that all future-relevant information is already embedded in the current data. In this study, the RW model is formulated as follows:

$$R_{t+1}^{°}=R_t^{°}+\mu$$

(18)

where $R_t^{°}$ represents the observation in the present period and $R_{t+1}^{°}$ signifies the observation in the subsequent period. The term $\mu$ is characterized as the noise component, consisting of independent and identically distributed normal variables with zero mean and constant variance.

Our study employs the Autoregressive Integrated Moving Average model (ARIMA model, denoted as p, d, q) to generate forecasts for the test data. This model bases its forecasts on the assumption that they are a sum of a random error and the most recent observation value. The parameters set for the ARIMA model are as follows: the number of Autoregressive terms (p) is 0, the number of non-seasonal differences required for stationarity (d) is 1, and the number of lagged forecast errors (q) is 0.

Additionally, the effectiveness of the RW model is contingent on the properties of the random errors. These errors are assumed to be independent and identically distributed (i.i.d.), as per the model’s assumption. This approach allows for a simplified yet insightful analysis of future market movements based on historical data.

2.2.7. Additional Forecasting Experiment

In this additional experiment, we employ two (recently popular) forecasting techniques: ARIMA with Exponential Smoothing and LightGBM. We first apply Exponential Smoothing to detrend and deseasonalize the data. After forecasting with the Exponential Smoothing model, we extract the residuals by subtracting the forecasted values from the actual data. These residuals represent the remaining structure of the time series, which is then modeled using the ARIMA model. The ARIMA specification chosen is (1, 0, 1), representing one Autoregressive term, no differencing for stationarity, and one lagged forecast error in the prediction equation. The next step involves combining the forecasts produced by Exponential Smoothing and the ARIMA model to obtain the final forecast.

Light Gradient Boosting Machine (LightGBM) is a gradient-boosting framework based on tree-based learning algorithms. The original time series is used as input for LightGBM because decision trees, which form the foundation of LightGBM, are flexible in adapting to sudden shifts in data distribution and capturing non-linear relationships. LightGBM builds trees iteratively, allowing for sequential error correction. Model features are derived from the original time series by constructing 30 lagged variables for each data point.

Extrapolation in time series forecasting refers to making predictions beyond the observed data range. This ability is essential for effective forecasting, especially in cases where trends are expected to evolve in ways not previously reflected in the training data. Tree-based models, such as LightGBM, create decision rules that split the data into different leaves based on input features. However, when tasked with making predictions beyond the range of the training data, these models struggle because they lack a mechanism to generalize beyond observed trends. To address this issue, we implement walk-forward validation in this study.

The source data are divided into training and test sets in an 80:20 ratio (like for the LSTM experiment), maintaining the chronological order to preserve the time sequence. The model is initially trained, and predictions are made step by step. After each prediction, the actual value is revealed to the model, which is then retrained using the new data point. This expanding window approach allows the model to incorporate additional information over time, leading to more accurate forecasts. This method is particularly suitable for the time series under study, as it allows the model to continuously adapt to the drastically changing market conditions and volatility.

3. Empirical Tests

3.1. Forecasting Performance

We report forecasting results in Figure 6. Our study reveals a local market inefficiency, consistent with the findings of [4]. This inefficiency is observed only during specific periods identified in the study. The AdaGrad optimization technique, denoted as ${\varphi }_{it\left(Adg\right)}$, demonstrates superior performance over localized „packets in time”. This empirical evidence corroborates the theoretical model proposed by [4] in Section 2.2.1.

Considering the information set ${\varnothing }_t$and the forecasting model ${\varphi }_{it}({\psi }_t;\widehat{{\chi }_t}$), from a set of available models ${\mathsf{\Sigma }}_t$, AdaGrad optimization outperforms SGD and Adam optimizations for ${\varphi }_{it\left(Adg\right)}({\psi }_t;\widehat{{\chi }_t})$, ${\varphi }_{it\left(SGD\right)}({\psi }_t;\widehat{{\chi }_t})$, ${\varphi }_{it\left(Ad\right)}({\psi }_t;\widehat{{\chi }_t})$. Therefore, we provide the final inefficiency expression $E\left[f_t(R_{t+1}^{°},{\varphi }_{it\left(Ad\right)}({\psi }_t;\widehat{{\chi }_t}),c_t)\right]>0$, for $t\in \mathsf{{\rm Y}}$, considering ${\varphi }_{it\left(Adg\right)}$ as a “new” forecasting technique that outperforms in the local “packet in time”. The market is inefficient for local time intervals (please see brakes in dynamics of efficiency in Figure 7).

• $\mathsf{{\rm Y}}\left[t_{2003},\dots t_{2004}\right]$, $0\le t_{2003}\le t_{2004}\le T$,
• $\mathsf{{\rm Y}}\left[t_{2005},\dots t_{1sthalfof2006}\right]$, $0\le t_{2005}\le t_{1sthalfof2006}\le T$,
• $\mathsf{{\rm Y}}\left[t_{2007},\dots t_{2009}\right]$, $0\le t_{2007}\le t_{2009}\le T$, and
• $\mathsf{{\rm Y}}\left[t_{2020},\dots t_{2021}\right]$, $0\le t_{2020}\le t_{2021}\le T$.

We report a detailed explanation of market inefficiency in Section 3.2.

To prevent overfitting in the LSTM model, we implemented dropout, which randomly “drops” or deactivates a certain percentage of neurons during training. This prevents the LSTM network from becoming overly reliant on specific neurons, thereby encouraging better generalization to unseen data. The dropout process is implemented in three steps. (1) Dropout was added within the LSTM layer using the dropout parameter in the nn.LSTM module. (2) The dropout parameter in the nn.LSTM layer applied dropout to the outputs of each LSTM layer, except the last layer. (3) A dropout layer was added before the final linear layer to further regularize the network. The forecasting results are shown in Figure 6. We present the complete forecasting results, including various HS values, as well as the results before and after applying the dropout procedure, in Supplementary S2.

The LSTM model in our study yields varying forecasting results, contingent upon the chosen optimization technique and the configuration of HS layers. Through the hyperparameter tuning process, the most effective performances are noted for the HS of 32, 64, and 256 layers. Notably, the AdaGrad optimization technique consistently outperforms both SGD and Adam optimizations across each HS. The AdaGrad optimization with an HS of 64 denoted as ${\varphi }_{it\left(Adg\right)}$ emerges as the most accurate in forecasting. This specific configuration aligns closely with the actual market trends throughout the entire observed period. While our findings generally align with those of [35,37,38], our study diverges in the aspect of LSTM model optimization. Contrary to the assertion that the LSTM model excels without optimization, our results demonstrate that different optimizations lead to distinct forecasting performances.

Our observations resonate with [39], who report that AdaGrad ensures a more stable forecasting process and mitigates the risk of overfitting. Similarly, we corroborate the findings of [40], who assert the superiority of AdaGrad over SGD in terms of accuracy. However, our study did not extend to testing other modified SGD-type algorithms, as explored by [40], particularly in relation to forecasting performance, convergence rate, and accuracy. This gap in research presents an opportunity for future investigations.

Table 1. LSTM accuracy metrics: the SGD, AdaGrad, Adam, and RMSprop optimizations.

Metrics	The SGD Optimizer
	HS =4	HS = 8	HS = 16	HS = 32	HS = 64	HS = 128	HS = 256
MAPE	0.6136	0.6173	0.8509	0.7328	0.6073	0.6420	0.6523
MDAPE	0.6135	0.6170	0.8509	0.7326	0.6071	0.6419	0.6522
RMSE	1780.10	1801.01	2449.46	2120.10	1776.76	1870.84	1899.56
MPE	1087.31	1072.79	417.18	748.96	1099.53	1002.80	973.84
R2	−43.56	−29.16	−37.33	−20.59	−29.32	−28.55	−28.07
	AdaGrad Optimizer
	HS =4	HS = 8	HS = 16	HS = 32	HS = 64	HS = 128	HS = 256
MAPE	0.5398	0.3408	0.1014	0.0503	0.0190	0.5075	0.1671
MDAPE	0.5390	0.3387	0.0923	0.0324	0.0105	0.5086	0.1671
RMSE	1589.35	1031.58	384.57	232.01	98.44	1478.47	503.94
MPE	1289.65	1852.85	2535.35	2708.26	2849.79	1388.22	2358.12
R2	−11.79	−1.67	−0.53	0.62	0.82	−3.88	−5.49
	Adam Optimizer
	HS =4	HS = 8	HS = 16	HS = 32	HS = 64	HS = 128	HS = 256
MAPE	0.3899	0.5593	0.4196	0.1931	0.2055	0.2242	0.6316
MDAPE	0.3886	0.5588	0.4188	0.1892	0.2016	0.2258	0.6319
RMSE	1178.35	1639.02	1251.65	631.64	644.32	642.82	1842.61
MPE	1710.75	1236.22	1629.73	2269.85	2239.63	2204.56	1031.86
R2	−8.73	−1.03	0.45	0.35	0.81	0.88	−20.91
	RMSprop Optimizer
	HS =4	HS = 8	HS = 16	HS = 32	HS = 64	HS = 128	HS = 256
MAPE	0.1684	0.4314	0.2865	0.3979	0.5058	0.5248	0.3004
MDAPE	0.1641	0.4287	0.2793	0.3965	0.5057	0.5241	0.29989
RMSE	588.81	1285.97	916.22	1185.39	1497.56	1550.79	893.04
MPE	2335.35	1596.00	1998.06	2919.11	1384.35	1330.91	1975.22
R2	−1.90	−12.81	−6.01	−10.74	−17.73	−19.09	−18.27

Accuracy metrics: Mean absolute percentage error (MAPE) is a commonly used metric for evaluating prediction accuracy: $MAPE={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left|{\displaystyle \frac{a_i-f_i}{a_i}}\right|$. Median absolute percentage error (MDAPE) is the error metric used to measure the performance of machine learning models: $MdAPE=median\left|{\displaystyle \frac{a_i-f_i}{a_i}}\right|\ast 100$. Root mean square error (RMSE) is an approach calculating the accuracy or error of a prediction model used to evaluate how closely the predicted values match the observed values: $RMSE=\sqrt{{\displaystyle \frac{\sum _{i=1}^N\left(f_i-a_i\right)2}{N}}}$. Mean percentage error (MPE) is the computed average of percentage errors by which forecasts of a model differ from actual values of the quantity being forecast: $MPE={\displaystyle \frac{100\mathrm{\%}}{n}}\sum _{i=1}^n{\displaystyle \frac{a_i-f_i}{a_i}}$.

in this study provides a comprehensive comparison of accuracy metrics for the SDG, AdaGrad, and Adam optimizers. The metrics presented include mean absolute percentage error (MAPE), median absolute percentage error (MDAPE), root mean square error (RMSE), and mean percentage error (MPE). The table is structured to highlight the smallest error values in each row in bold, thereby allowing for an easier assessment of the most accurate optimizer under various conditions.

For both the SDG and AdaGrad optimizations, it is observed that, in almost all cases (particularly for MAPE, MDAPE, and RMSE), the forecasting accuracy is superior when the HS is set to 64. This trend aligns with the findings illustrated in Figure 6. Furthermore, AdaGrad, particularly with HS = 64, demonstrates significantly lower forecasting errors compared to both the SDG and Adam optimizations. This underlines the superior performance of the AdaGrad optimizer ${\varphi }_{it\left(Adg\right)}$ in capturing local market inefficiencies. Conversely, while the Adam optimization reports the lowest errors for HS = 32, these error values are substantially higher when compared to the AdaGrad optimization with HS = 64. This comparative analysis indicates that, while Adam optimization may have its merits in certain configurations, it is outperformed by AdaGrad, particularly in the context of HS = 64, in terms of forecasting accuracy and reliability.

In an additional experiment, we present the forecasting results of ARIMA with Exponential Smoothing and LightGBM. Both visual comparisons and error metrics provide empirical evidence that LightGBM is a significantly more effective and accurate model in this forecasting context (Figure 7). Its capacity to capture non-linear relationships and manage complex time series patterns with greater precision is clear when compared to the ARIMA with Exponential Smoothing model, which demonstrates limitations in both accuracy and bias. LightGBM consistently outperforms ARIMA with Exponential Smoothing across all considered metrics (

Table 2. ARIMA with Exponential Smoothing and LightGBM accuracy metrics.

	ARIMA with Exponential Smoothing	LightGBM
MAPE	13%	0.8%
MDAPE	13.2%	0.86%
RMSE	464.04	36.58
MPE	130%	30%
R Squared	−0.638	0.989

).

The substantial differences in error metrics, such as MAPE and RMSE, highlight that LightGBM not only achieves superior accuracy, but also exhibits greater reliability in identifying the underlying patterns within the time series. The weak R-squared and high MPE values for ARIMA indicate a misalignment with the actual data and a tendency toward systematic error, whereas LightGBM demonstrates high precision and minimal bias.

Although LightGBM is effective in many machine learning tasks, its reliability in financial time series forecasting is limited by its inability to model time dependencies. While LightGBM is a powerful tool for detecting non-linear relationships [41], LSTM is often better suited for capturing the sequential nature and volatility of financial markets. Financial time series data frequently exhibit complex temporal dependencies (such as trends, seasonality, and autocorrelation) that the models must capture effectively [42].

As a tree-based model, LightGBM excels at identifying non-linear relationships, but does not inherently account for time-based structures or the sequential ordering of financial data [43]. Tree-based methods split data based on feature values without considering the temporal order or relationships between data points over time [41]. Additionally, like many machine learning models, LightGBM performs well in interpolation, predicting values within the range of the training data [44]. However, it struggles with extrapolation, particularly in financial markets where future data may extend beyond observed ranges. Given the volatility and unpredictability inherent in financial markets, LightGBM faces challenges in generalizing and predicting beyond the patterns it learned during training.

In contrast, LSTM is specifically designed to capture temporal dependencies, understanding how past values influence future ones [32]. While LightGBM can incorporate lagged variables as features, it lacks a built-in mechanism to model sequential relationships over time. Financial data often display momentum effects or mean reversion, and models that do not account for such patterns risk overlooking key predictive signals [42].

For these reasons, we select the LSTM with the Adagrad64 as the most reliable forecasting technique in this context due to its superior ability to model time dependencies and adapt to changing market conditions [32,43].

The Adagrad64 follows the trend of the actual values quite closely. This suggests that the AdaGrad optimizer with an HS = 64 is relatively effective in capturing the trend of the dataset. Yet, the Adagrad64 does not show random behavior, but rather a smoothed trend following the index (like in 2020, where the forecasted values diverge significantly from the actual values and may struggle with sudden changes). The forecasting errors metric is presented in

Table 3. Random Walk accuracy metrics.

Metric	Adagrad64	Index Data
RMSE	0.013879	0.009307
MAPE	0.5258%	0.6657%
MPE	−0.1491%	0.0398%
MdAPE	0.1147%	0.2463%

.

The Adagrad64 model seems to be better at forecasting based on these metrics, with generally lower errors and a tighter distribution of those errors (as suggested by MAPE and MdAPE). The negative sign of MPE Adagrad64 indicates that the forecasts are, on average, slightly below the actual values. The magnitude being very small implies that there is no significant bias toward underestimation or overestimation. The MAPE for Adgrad64 is a very low percentage, indicating that the forecast errors are small relative to the size of the actual values they are trying to predict. Both models show a good fit to the data, as indicated by the low error metrics.

The variation in stationarity across different HS settings suggests that the choice of HS can significantly impact the statistical properties of the time series data (

Table 4. ADF test statistics for different HS settings using the AdaGrad optimizer.

	HS =4	HS = 8	HS = 16	HS = 32	HS = 64	HS = 128	HS = 256
ADF Statistic	−3.60	−1.506	−1.934	−1.816	−1.258	−0.578	−0.942
p-value	0.006	0.531	0.316	0.373	0.648	0.876	0.774
	Critical Values
1%	3.435	3.435	3.435	3.435	3.435	3.435	3.435
5%	−2.864	−2.864	−2.864	−2.864	−2.864	−2.864	−2.864
10%	−2.568	−2.568	−2.568	−2.568	−2.568	−2.568	−2.568

). This, in turn, affects the forecasting performance. For Adagrad64, the choice of HS could be a balancing act between achieving stationarity and maintaining point accuracy. The stationarity of the data, as influenced by the AdaGrad optimizer’s HS settings, appears to impact the forecasting performance of the models. Adagrad64, possibly tuned for a specific HS setting showing strong stationarity, performs better in percentage error terms but lags in point accuracy. The lower MAPE and MdAPE suggest good performance, in relative terms, due to the model being well tuned to the stationarity aspects at certain HS settings.

The ADF test results indicate that all HSs demonstrate p-values well above the significance level (except for HS = 4), failing to reject the null hypothesis and thus providing no statistical evidence against the Random Walk Theory. This means that, for most configurations, the RW theory cannot be statistically rejected, indicating the possibility of Random Walk behavior in the data. The Random Walk theory cannot be universally rejected. It appears that the MEF holds to varying degrees. The degree to which financial time series exhibit Random Walk characteristics is influenced by the underlying market dynamics.

3.2. Adoptive Market Efficiency

Both the DL and the GS tests in our study were conducted using a rolling window approach with each window encompassing 300 observations. The window advanced by 7 days for each subsequent iteration. Given the total sample size of 7671 observations and considering the length of the rolling window along with the 1-week forward step for each iteration, this study conducted a total of 1053 tests.

Figure 8 presents the p-values derived from the DL test applied to the S&P 500 time series. An analysis at the 95% confidence interval indicates that the MDH does not hold for specific periods, notably from mid-2002 to 2004, early 2005, mid-2006, and from 2007 to around mid-2008. The p-values also dip below the 5% significance level during the latter half of 2008 and early 2009. Additionally, at a 10% confidence interval, the MDH does not hold for brief periods at the start of 2014, 2016, and 2017, and throughout 2020 (Figure 8). These findings suggest that the S&P 500 exhibited a degree of predictability during these intervals, challenging the MDH.

The break in efficiency observed in 2003 can be linked to the substantial rise in the S&P 500, following a 3-year downturn in major US stock indexes due to the 2000–2001 recession and the dot-com crisis. The index experienced a steady increase from 2003 until the pronounced decline at the end of 2007 and in 2008, attributed to the financial crisis.

The significant drop in the S&P 500 in 2008 is captured in Figure 9, reflected as a sharp increase in market efficiency. The high predictability in 2007 correlates with the extended period of growth in the S&P 500. In 2020, after considerable returns drop, the S&P 500 rapidly recovered, reaching new record highs. The periods of inefficiency identified could be a result of the accumulated momentum in the S&P 500 returns.

Conversely, according to the GS test results shown in Figure 10, the MDH holds across the entire observation period at the 95% confidence interval, suggesting high efficiency in the S&P 500. This finding contrasts with the DL test results. However, at a 90% confidence interval, the MDH was not upheld at the beginning of 2003, end of 2009, beginning of 2015, or in 2019, indicating some exceptions to the overall trend of market efficiency.

Our findings corroborate the concept that market efficiency is not static but a dynamic phenomenon, subject to influence from various elements such as structural changes and external events. This perspective aligns with the insights of [16,29], who underscore the impact of structural modifications on market dynamics, and [45,46], who highlight the role of external factors. Ref. [16] also posits that, over time, these influences lead to relative optimization in market behavior. In scenarios where the market reaches a state of equilibrium, this study confirms that the efficiency condition can be represented as $E\left[f_t(R_{t+1}^{°},{\varphi }_{it}({\psi }_t;\widehat{{\chi }_t}),c_t)\right]=0$. This indicates a balance where market returns fully reflect all available information, leaving no room for excess returns through arbitrage. However, our study also acknowledges that, in the short term, behavioral biases and market anomalies can create opportunities for arbitrage. Market participants, recognizing these opportunities, can exploit them for potential gains. In such instances, particularly with the application of the AdaGrad optimization technique ${\varphi }_{it\left(Adg\right)}$, the efficiency condition is modified to $E\left[f_t(R_{t+1}^{°},{\varphi }_{it\left(Ad\right)}({\psi }_t;\widehat{{\chi }_t}),c_t)\right]>0$. This signifies a scenario where the market is not fully efficient, and the predictive model ${\varphi }_{it\left(Adg\right)}$ can identify and benefit from these inefficiencies.

3.3. Effects of Statistical Efficiency

To examine the impact of past return distribution, encompassing mean variance information, stationarity, and the correlation matrix, our study focuses on expressing distribution variabilities. The returns for the SPX are central to this analysis. These returns are systematically plotted and presented in Figure 11.

While the index values, specifically the returns of the SPX, were found to be non-stationary, the logarithmic returns met the criterion for stationarity. This conclusion is substantiated by the application of the ADF test. The outcomes of the ADF test are displayed in

Table 5. Stationarity test.

	Intercept	Trend and Intercept	None
logSPX	0.235526 (0.9748)	−2.315184 (0.4250)	1.174996 (0.9388)
RSPX	−97.36762 (0.0001)	−97.39885 (0.0001)	−97.35839 (0.0001)

Note: p-values are presented in parentheses. The ADF test performed on each time series rejects the null hypothesis of the existence of a unit root in the returns, which suggests the stationary nature of the data. This implies that some underlying process that generates data is constant over time.

.

Upon establishing the stationarity of the series, the study proceeds to apply the Autoregressive Moving Average (ARMA) model. Utilizing the Akaike Information Criterion (AIC), it is determined that the returns of the SPX match the ARMA (5,4) model. Conversely, when employing the Schwarz Bayesian Criterion (SBC), this study finds that the SPX returns align with the Autoregressive (AR) model of order 1, or AR (1).

Following the application of the AR (1) model, this study conducts a volatility analysis using the Generalized Autoregressive Conditional Heteroskedasticity (GARCH) model. This step is crucial for understanding the volatility patterns in the SPX returns. The findings and results derived from the GARCH model are concisely summarized and presented in

Table 6. Volatility model.

GARCH = C (2) + C (3) ∗ RESID (−1)2 + C (4) ∗ GARCH (−1)
Variable	Coefficient	Std. Error	z-Statistic	Prob.
C	0.017933	0.003489	5.139736	0.0000
	Variance Equation
C	0.001737	0.000441	3.943345	0.0001
RESID (−1)2	0.070991	0.007921	8.962541	0.0000
GARCH (−1)	0.919645	0.008084	113.7622	0.0000
R-squared	−0.000762	Mean dependent var		0.005458
Adjusted R-squared	−0.000762	S.D. dependent var		0.451858
S.E. of regression	0.452030	Akaike info criterion		0.731540
Sum squared resid.	1567.017	Schwarz criterion		0.735162
Log likelihood	−2801.455	Hannan-Quinn criter.		0.732782
Durbin–Watson stat.	2.209563

Note: Table’s results show that the sum of the estimated ARCH and GARCH parameters is close to 1, so it is safe to say that the SPX index’s volatility has a unit root, or that it follows the IGARCH (1,1) process.

.

Drawing on the conditional variance obtained from the volatility analysis, this study calculates the daily conditional volatility of the SPX returns. Figure 12 displays the fluctuations in the SPX return volatility.

To further validate the robustness of the estimated GARCH model for the SPX, this study employs the Ljung–Box test. Residual autocorrelation could indicate model misspecification or inadequacy in capturing the dynamics of the data. Moreover, this study also considers the results of the Autoregressive Conditional Heteroskedasticity-Lagrange Multiplier (ARCH-LM) test. The outcome of this test reveals the absence of heteroskedasticity, or more specifically, a lack of lagged ARCH effects in the model residuals (

Table 7. Diagnostic of the model.

Ljung–Box Statistics	ARCH-LM (5) Test
Q (36)	F- stat.	Obs ∗ R2
45.273	1.47584	7.37787
(0.138)	(0.1941)	(0.1940)

Note: The p-values are presented in parentheses. The Ljung–Box test evaluates the time series for the presence of autocorrelation. A test with lag values of 1 shows no statistical significance, indicating that the residuals are independently distributed and that there is no presence of autocorrelation in data with 1 lag. ARCH-LM tests the null hypothesis where Autoregressive conditional heteroscedasticity among the lags in the data is not present. The result indicates that the time series of SPX does not exhibit the presence of ARCH-LM effects. The null hypothesis is accepted.

).

Our findings challenge the assertion made by [47] regarding the EMH, which posits that markets reflect all available information in prices and thus enable minimum variance predictions for future price (returns) movements. Contrary to this view, our research indicates that investors are indeed susceptible to future return variability. This is exemplified in scenarios where $E\left[f_t(R_{t+1}^{°},{\varphi }_{it\left(Ad\right)}({\psi }_t;\widehat{{\chi }_t}),c_t)\right]>0$, particularly during the periods $0\le t_{2003}\le t_{2004}\le T$, $0\le t_{2005}\le t_{1sthalfof2006}\le T$, $0\le t_{2007}\le t_{2009}\le T$, and $0\le t_{2020}\le t_{2021}\le T$. Our results align with the observations made by [48,49,50], who acknowledge future return variability and its inherent chaotic and dynamic nature, as also highlighted by [35]. This variability suggests that market returns are not always the best estimators of the true underlying value of an asset, as they are influenced by various factors beyond the available information, leading to periods of inefficiency.

Also, our study identifies the prevalence of market anomalies and opportunities for arbitrage, as noted by [16], particularly in mostly 2-year periods. These anomalies and arbitrage opportunities contribute to market inefficiency, contradicting the EMH assumption of rational, well-informed markets. This observation is further supported by [51]’s statement, which suggests that the effects of a crisis on output and employment have enduring impacts, thereby influencing market behavior and efficiency in the post-crisis period.

4. Challenging the Paradigm and Implications

The EMH posits that stock prices fully incorporate all available information, making it impossible to achieve consistent excess returns. However, our findings suggest temporary deviations from this principle. These “pockets in time” indicate periods of market inefficiency, particularly during times of heightened volatility or structural shifts. This evidence challenges the continuous applicability of the strong form of the EMH, as noted by [4]. Another important consideration is the speed at which efficiency is restored after the occurrence of inefficiencies. [4] argue that market inefficiencies tend to self-correct rapidly, with prices quickly reverting to an efficient state. In contrast, our results show that some inefficiencies, particularly those triggered by structural shifts or external shocks, such as the COVID-19 pandemic, persisted longer than expected.

Malkiel [3] contends that even the most advanced forecasting models fail to consistently deliver excess returns because stock prices efficiently incorporate new information. While our study acknowledges the existence of temporary inefficiencies, our LSTM models exhibited predictive accuracy during specific periods, thus challenging the broader assertion that markets are always efficient. This finding is particularly significant, as it contrasts with conclusions drawn by [52], who argue that financial markets predominantly operate under conditions of Informational Efficiency.

In contrast to the EMH, the AMH offers a more flexible interpretation of market efficiency, positing that inefficiencies can temporarily arise in response to external shocks or changing market conditions. Our findings, particularly the detection of inefficiencies during key periods such as the 2007–2009 financial crisis and the COVID-19 pandemic in 2020, align with the premise of the AMH that markets evolve and adjust over time. These “pockets in time” provide empirical support for the idea that market efficiency is not static but fluctuates based on external factors. This dynamic view supports the notion that investors adapt their strategies to exploit temporary inefficiencies.

While our study supports the AMH by identifying temporary inefficiencies, we observed different behavior compared to [17,18], who suggest that these inefficiencies can be systematically exploited by advanced machine learning models. In contrast, our results indicate that, while LSTM models can capture temporary inefficiencies, these are not as predictable or systematically exploitable as some proponents of the AMH suggest. This highlights a potential overestimation in the literature regarding the consistent profitability of machine learning models under the AMH framework.

Ref. [4] emphasize that, while models may detect inefficiencies, these opportunities are typically short-lived, as markets tend to self-correct. Our empirical analysis aligns with this view: the LSTM models successfully identified “pockets in time” where predictive accuracy was high, indicating temporary inefficiencies. However, as expected, these inefficiencies diminished once recognized and exploited by market participants, restoring market efficiency over time. Although the LSTM model uncovered inefficiencies consistent with [4]’s findings that models can identify exploitable patterns, the market’s self-correction in our study occurred more swiftly than anticipated. This suggests that market inefficiencies are fleeting and difficult to capitalize on, contrasting with the more optimistic view presented in other machine learning studies.

Ref. [25] suggests that machine learning models can provide sustained predictive advantages during periods of inefficiency. However, our findings do not fully support this conclusion. The rapid dissipation of inefficiencies observed in our study underscores the limited practical utility of machine learning forecasts in real-world financial trading. Although market inefficiencies were identified during periods of external shock, these opportunities proved to be fleeting. Successfully capitalizing on them requires sophisticated tools and precise timing. Investors should recognize that, while machine learning models may offer valuable insights into short-term market fluctuations, the self-correcting nature of financial markets makes consistent excess returns difficult to achieve.

For policymakers and regulators, this study emphasizes the importance of closely monitoring market dynamics, especially during periods of economic stress or external disruption. A better understanding of how inefficiencies arise and dissipate can help in designing regulatory frameworks that enhance market stability while enabling adaptive strategies for market participants. This dynamic view of market behavior supports the need for flexible policies that respond to evolving market conditions rather than relying on static assumptions of market efficiency.

5. Concluding Remarks

We provide a comprehensive analysis of market efficiency through the application of advanced forecasting techniques and market efficiency tests, encompassing key theories such as the EMH, AMH, Information Efficiency, and Random Walk theory. Utilizing LSTM models optimized with various techniques AdaGrad, SGD, Adam, and RMSprop, we reveal significant insights into market dynamics and inefficiencies. Our findings challenge the assumptions of the EMH, uncovering “pockets in time” (Timmermann and Granger, 2004) where predictive models detect temporary inefficiencies, particularly during periods of heightened volatility or structural shifts. The LSTM model, especially when optimized with AdaGrad at HS 64, outperformed other models in capturing these inefficiencies, demonstrating its ability to model the temporal dependencies inherent in financial markets. However, the results also indicate that these inefficiencies are fleeting, as markets tend to self-correct, aligning with the predictions of Timmermann and Granger (2004).

We employed two more forecasting techniques in an additional experiment setup: ARIMA with Exponential Smoothing and LightGBM. In contrast, LightGBM exhibited strong performance in detecting non-linear relationships within the data, consistently outperforming ARIMA with Exponential Smoothing across all error metrics, including MAPE and RMSE. While LightGBM demonstrated superior accuracy in short-term forecasts, its inability to model time dependencies, compared to the LSTM, limits its effectiveness in financial time series forecasting. The ARIMA with Exponential Smoothing model, on the other hand, struggled with both accuracy and bias, as evidenced by high error metrics and weak R-squared values, confirming its limitations in handling complex market patterns. Despite the strength of LightGBM in forecasting non-linear relationships, and the LSTM model’s superior handling of temporal dependencies, our results emphasize the transient nature of market inefficiencies. Both machine learning and traditional statistical models face challenges in consistently exploiting these inefficiencies for long-term profitability. Our empirical evidence suggests that, while machine learning models can identify short-term opportunities, these are not systematically exploitable over extended periods, reinforcing the idea that markets are self-correcting.

Furthermore, the application of market efficiency tests, including the DL and GS tests, provided additional validation of our findings. These tests revealed inefficiencies during key periods such as the 2007–2009 financial crisis and the COVID-19 pandemic, supporting the dynamic perspective of market efficiency as proposed by the AMH. Market efficiency fluctuates in response to external shocks and behavioral changes, offering a more nuanced understanding than the static assumptions of the EMH.

Finally, while LSTM models, LightGBM, and ARIMA with Exponential Smoothing provide valuable insights into market inefficiencies, they do not offer a sustainable framework for achieving consistent excess returns. Our research highlights the importance of integrating forecasting models with market efficiency testing to obtain a holistic view of market behavior. Inefficiencies should be viewed as temporary phenomena influenced by structural and informational dynamics, rather than reliable opportunities for arbitrage. Future research should further explore the interaction between forecasting models, market conditions, and external shocks to refine the application of machine learning in financial markets.

Our study acknowledges several limitations. First, the forecasting model ϕit that is undiscovered from other investors, represented by LSTM with HS64, is applied within the “local pocket in time” framework of [4]. While this model is used hypothetically to demonstrate the theoretical approach, in practice, it is already familiar to investors. The purpose here is to illustrate the theoretical concept rather than to suggest it as an undiscovered technique. Second, both LSTM and LightGBM demonstrate strong performance within the range of the training data, but exhibit limited ability when extrapolating beyond the observed data. Financial markets frequently undergo structural shifts that extend beyond historical norms, and these models may struggle to adapt to new regimes. A potential solution to these limitations lies in the implementation of a dynamic regime-switching model [27]. Although our use of walk-forward validation helps mitigate the effects of this limitation, it does not fully resolve the challenges posed by regime switching [53] in financial markets.