Emerging Trends in AI-Based Stock Market Prediction: A Comprehensive and Systematic Review ^†

Abstract

This research paper provides a comprehensive review of the emerging trends in AI-based stock market prediction. The paper highlights the key concepts, approaches, and techniques employed in AI-based stock market prediction and discusses their strengths and limitations. Key topics covered include deep learning, natural language processing, sentiment analysis, and reinforcement learning. This paper also presents case studies and evaluates the performance of different AI-based models in predicting stock market trends. Overall, this research paper provides valuable insights into the latest advancements in AI-based stock market prediction and their potential implications for investors, financial analysts, and policy makers.

1. Introduction

The stock market, a complex and dynamic system vital to the global economy, is influenced by politics, economics, and investor sentiment. This unpredictability has challenged financial experts. Traditionally, fundamental and technical analyses were used for predictions, but they had limitations. Artificial Intelligence (AI) is reshaping the stock market landscape through predictive analytics, trading algorithms, risk management, fraud detection, and portfolio optimization. Artificial Intelligence (AI) employs predictive analytics to analyze extensive financial data, revealing patterns and predicting market movements and empowering traders and investors with well-informed decisions. Real-time trading algorithms driven by AI adjust strategies based on market conditions, bolstering profitability. AI aids in identifying and mitigating risks by analyzing varied data sources like financial news, social media, and economic indicators. AI’s pattern recognition capabilities are instrumental in detecting fraudulent trading and market manipulation activities. Portfolio optimization benefits from AI’s analysis of market trends and risk factors, reshaping decision-making, risk management, and profit realization. The study on Emerging Trends in AI-based Stock Market Prediction aims to illuminate recent breakthroughs and their impact on investors, financial analysts, and policymakers. AI-driven forecasts outshine conventional methods, holding potential for informed investment choices. Acknowledging AI’s strengths and limitations, this research strives to heighten prediction accuracy and reliability, ultimately benefiting the financial sector. In recent times, AI techniques have gained traction for stock market prediction due to their superior performance compared to traditional methods. With ample financial data and advanced machine learning, AI models are revolutionizing the finance industry. Approaches like fundamental analysis, technical analysis, quantitative analysis, machine learning, sentiment analysis, and expert opinion categorize prediction methods based on methodology and data sources, reflecting AI’s evolving impact on stock prediction. This paper delivers valuable insights into recent strides in AI-based stock market prediction and their potential repercussions for investors, financial analysts, and policymakers. Its aim is to foster a comprehensive comprehension of the present AI-based stock market prediction landscape, spotlighting areas warranting further exploration. These findings empower better-informed investment choices and foster more precise and dependable AI models for stock market prediction.

2. Literature Review

The research topic “Emerging Trends in AI-based Stock Market Prediction: A Review” has garnered substantial interest, prompting various studies to explore AI and machine learning’s potential in forecasting stock trends and prices. Presented here is a concise literature overview on this subject. Based on our review study, Figure 1 reflects % use in stock market prediction vs. ML Techniques.

Machine learning models offer investors a potential edge in predicting stock prices and trends by swiftly processing extensive data, though their accuracy is variable due to the intricate and multifaceted nature of the stock market [1]. Unraveling intricate non-linear stock market patterns requires careful technique selection, with feature choice playing a pivotal role in revealing crucial insights and ensuring reliable predictions [2]. Identifying complex stock market patterns requires intricate techniques, with feature selection being pivotal for robust predictions, as features unveil crucial insights from intricate data [3]. In their 2023 research, Vitor Azevedo et al. investigated the prediction of capital market anomalies, using diverse machine learning techniques and models on a massive dataset, with top models showing significant monthly returns of 1.8% to 2.0% [4]. The authors investigate ANN, SVM, and LSTM neural networks, highlighting their distinct characteristics and practical applications, underscoring the transformative role of machine learning in shaping investment strategies [5]. The study highlights data labeling’s significance in trading system creation, introducing N-Period Min-Max (NPMM) labeling to tackle information loss and noise issues and showing its superior effectiveness in stock price trend prediction compared to other methods [6]. A novel algorithm merging deep reinforcement learning with portfolio theory, utilizing a 3D convolutional neural network for feature extraction and Deep Deterministic Policy Gradient (DDPG) for portfolio optimization, consistently demonstrates improved performance [7]. Michele Costola et al. reveals a statistical connection between the stock market and COVID-19 emotions, showcasing how pandemic-related news influenced investor expectations and financial market trends [8]. The study introduces a novel approach using an LSTM predictor and AC-SFLA optimizer for enhanced efficiency and is demonstrated through testing on a real stock market dataset [9]. LSTM effectively processes temporal data by utilizing input, output, and forgetting gates to manage information flow [10]. Yanrui Li et al. (2022) present a comprehensive method that enhances company ranking for investment in China’s A market, yielding a remarkable 9.2% increase in P-return compared to averages and indicating enhanced accuracy and performance [11]. In their 2022 study, VMalti Bansal et al. found that deep learning algorithms outperformed traditional machine learning in predicting stock prices of Indian companies, using various algorithms and data from 2015 to 2021 [12]. Shamima Ahmed et al. investigate AI and machine learning (ML) in finance, examining 348 articles from 2011 to 2021. It reveals a growing trend since 2015, showcasing applications in bankruptcy prediction, stock prices, portfolio management, and more. Leading contributors are the US, China, and the UK [13]. Different graph neural networks like Graph Convolutional Network (GCN), Graph Attention Network (GAT), and Gaussian and Neural Accelerator (GNA) are used, and GCN and GAT are found to be the predominant choices for stock prediction [14]. Abdulhamit Subasi et al.’s 2021 study evaluates seven classification methods for stock market datasets, revealing strong predictive performance by random forest and Bagging (93% accuracy) with leaked data, followed by AdaBoost (82%), while Decision Trees perform less effectively (49% accuracy) [15]. Utilizing ensemble deep learning, the model predicts next-day stock prices with around 85% accuracy, enhanced by various deep learning methods, outperforming other techniques and yielding error rates of 0.0% to 13.3% [16]. Deep learning algorithms, notably LSTM and BI-LSTM, significantly influence modern technology, particularly in time series-based prediction models like stock price prediction, where accurate parameter adjustment is pivotal for enhancing prediction accuracy [17]. Vachhani et al. (2020) provide a succinct overview of machine learning’s impact on stock market analysis, addressing limitations, exploring regression models, decision trees, neural networks, and support vector machines, and emphasizing feature selection and future research directions [18].

AI and ML methods demonstrate the potential of forecasting stock market trends and prices, necessitating thorough assessment and comparison of models for optimal selection in distinct situations.

3. Results

The systematic review of AI-based stock market prediction examines performance metrics, data sources, feature selection, timeframes, and comparison with traditional methods, and challenges. It offers valuable insights into AI’s effectiveness in predicting stock prices and guides future research in the field.

Table 1. Comparative study among selected papers on AI-based stock market predictions.

Year	Objective	Dataset	F*	Techniques	PT*	Metrics	Results	RG*	R*
2023	SM* prediction	NA	NA	ML, ANN, SVM, NN, LSTM	NA	NA	High accuracy	NA*	[1]
2023	Stock price, trend prediction	NA	Price, Trend	BPNN, CNN, GRU, LSTM	NA	Accuracy, Error	High accuracy	NA	[2]
2023	Stock prediction	News	Price	ML, MLP, SVM, LSTM, ANN	M*	Accuracy	High accuracy	Market data and text data can lead to more accuracy	[3]
2023	Enhancing stock market anomalies	NA	Profit	ML	M	Profit margin	Fitness of model	The factor zoo	[4]
2022	ML models’ stock market prediction	NA	NA	ML, ANN, SVM, LSTM	NA	NA	NN working efficiently in depth	NA	[5]
2023	ML trading system for the SM	NASDAQ	Price, Trend	ML, NPMM, XGBoost	A*	Accuracy	Labelling is found productive	NA	[6]
2023	DRL for stock portfolio optimization	NA	Portfolio Optimization	DL, RL, DDPG	A, M, W*	Sharpe ratio	Algorithms outperformed by the suggested methods	Dynamically modifies the weight	[7]
2023	ML sentiment analysis and SM reactions	COVID-19 News, S&P 500	Correlation	ML, NLP, BERT	A	Sentiment scores	Positively correlated and statistically significant	Brief window examining only	[8]
2022	AI-based day-ahead SM forecasting	China Stock Market	Profit	LSTM, SFLA	D	Profit margin	LSTM, AC-SFLA has high efficiency	NA	[9]
2023	Stock prediction and analysis	SSE	Price	LSTM	D*	Error	MAE of 0.029, MAPE of 0.61%, and RMSE of 0.037	Refining the model architecture	[10]
2022	Automatic stock selection like fund managers	China’s A Share Market	Profit	ML, Scoring, Screening Model	NA	Profit margin	P-return is a notable increase	NA	[11]
2022	SM prediction	BSE, NSE	Profit	ML, K-NN, LR, SVR, DTR, LSTM	M	Accuracy	The LSTM is outperforming	In time series data, ML appears to produce less reliable	[12]
2022	GNN in SMP	NA	Price	GNN, GCN, GAT, GNA	NA	NA	GCN and GAT are the most frequently utilized	NA	[14]
2021	SM prediction based on ML Algorithms	NASDAQ, NYSE, NIKKEI, FTSE	Accuracy	DM, RF, SVM, ANN, Bagging, AdaBoost, Decision Trees, K-NN	NA	Accuracy	RF, Bagging with a leaked dataset results in high performance	NA	[15]

PT*: prediction type, R*: reference, D*: daily, M*: monthly, W*: weekly, A*: annually, RG*: research gaps, F*: features, SM*: stock market, and NA*: not available.

provides the comparative studies of selected papers based on various parameters.

Indian broking companies are now following these AI and ML trends. Zerodha: Zerodha may collect and analyze customer data on trading patterns, portfolio performance, and market sentiment [19]. Upstox: Upstox may collect and analyze customer data on investment preferences, portfolio performance, and market trends [20]. ICICI Direct: ICICI Direct may collect and analyze customer data on investment goals, risk tolerance, and portfolio diversification [21]. HDFC Securities: HDFC Securities may collect and analyze customer data on investment preferences, portfolio performance, and market trends [22]. Motilal Oswal: Motilal Oswal may collect and analyze customer data on investment preferences, portfolio performance, and market sentiment [23]. 5Paisa: 5Paisa may collect and analyze customer data on investment preferences, portfolio performance, and market trends [24]. Grow: Grow may collect and analyze customer data on investment goals, risk tolerance, and portfolio performance [25]. The AI-based stock market prediction market is set to reach USD 7.3 billion by 2024, growing at a CAGR of 32.9%, while AI models like deep learning and ensemble learning enhance accuracy. Feature selection methods such as MI-based approaches optimize efficiency, yet challenges include data complexity, model transparency, and overfitting risks [26].

4. Observations and Discussion

The global AI market is projected to reach USD 267 billion by 2027, contributing USD 15.7 trillion to the global economy by 2030. Around 37% of businesses utilize AI, while the rise of AI will generate 97 million new jobs and replace 85 million by 2025. Voice assistant usage is increasing, with over three billion in use and eight billion expected to be in use by 2023. The AI industry is forecasted to earn USD 126 billion annually by 2025. Additionally, 67% of Americans trust self-driving cars, and 25 countries are actively developing autonomous vehicles, which could lead to a USD 600 billion industry within five years [27]. Key observations from the studies include the prevalent use of machine learning techniques like neural networks and support vector machines, diverse data sources such as news articles and historical prices, varying prediction accuracy, and the demand for more research to enhance models by incorporating market sentiment and addressing data imbalance. These findings underscore the increasing interest in machine learning for stock prediction, emphasizing the necessity for refining existing models and exploring alternative data sources to improve prediction accuracy.

5. Limitations, Future Scope, and Challenges

AI-based stock market prediction faces challenges due to the stock market’s uncertainty and reliance on historical data quality, complex models, and overfitting. Future opportunities include improving data quality, making AI models interpretable, integrating with other technologies, and providing personalized recommendations. Challenges involve regulatory issues, human bias, ethical concerns, and high costs for AI implementation.

6. Conclusions

AI- and ML-based methods hold potential for stock market prediction by analyzing historical data, but challenges like market unpredictability, data quality, bias, and overfitting persist. Despite these hurdles, progress is seen in transparent models and the integration of diverse data sources like social media. With ongoing research, improved accuracy could aid investor decisions and enhance market performance.