Predicting the stock market with natural language processing and machine learning
Natural language processing and machine learning techniques have been used to predict stock market movements.
Link to source code at the end of the article. I tried coding this research paper, you can download the code and work further on it.
Introduction
The prediction of stock movement is a central task in computational and quantitative finance, and many researchers have done meticulous research on it. As a way to explain the law of stock market changes, Eugene Fama proposed the efficient market hypothesis. Financial time series analysis uses convolutional neural networks (CNNs), long short-term memory (LSTM) recurrent neural networks, and multitask recurrent neural networks.
Methodologies
Noisy Recurrent State Transition
Heyan Huang and Xiao Liu proposed a novel future event prediction module to factor in natural events consequences when predicting stock movements. ANRES models both noise and events over a recurrent stock value state, so it’s more explainable.
Deep Learning
Using CNN, Jiake Li suggested replacing basic emotional features at the emotional extraction level with in-depth emotional information. Additionally, Agüero and M.M. Salas proposed a method of Multilingual Sentiment Analysis (MSA) to address the sentiment analysis issue through several strategies.
Constrained Learning
Chen, X., Rajan, D., and Quek, H.C. proposed an efficient and interpretable neuro-fuzzy system for stock price prediction using multiple technical indicators, focusing on the interpretability-accuracy trade-off. Real-world problems like stock market environments are well suited to neuro-fuzzy systems.
Graph-Based Prediction
In this paper, Junran Wu and Ke Xu use graphs to solve the problems of long-term dependencies and chaotic properties of time series. Several state-of-the-art benchmarks showed this framework had the best performance. Contextual information within documents can be used in learning-based methods to make better predictions.
Liquidity Prediction for Learning Models
Using recent years from 2011 to 2019, [6] develops machine learning models for liquidity prediction on the Vietnamese stock market. Analysis of 220 companies representing different sectors was done using daily data.
System Architecture and Models
Time Series Model
By learning the law of past changes, a stock model based on time series can predict future stock market changes. In order to make the retained information more effective for model learning, the time series data are divided by finding the key trend points.
Deep Learning Model
Time series data is modeled with a recurrent neural network, and sentiment analysis results are added to build a trend prediction model. There are lots of models that use convolutional neural networks (CNNs) and LSTMs.
Heterogeneous Graphs
[7] predicts future stock reactions based on one or more news documents of a corporation, capturing rich contextual information within financial texts.
Methodology
Dataset Description
Hamidreza Faaljou and Hadi Rezaei used an open-source tool to extract information from Yahoo Finance’s daily financial time series. Using daily stock data from three indices: BSE, CNX Nifty, and S&P 500, we evaluated the proposed system using five fundamental quantities: maximum price, open price, minimum price, close price, and trading volume.
Dataset Pre-Processing
To work with this dataset, you’ll need to preprocess and visualize it so you can see trends, missing values, noise, and outliers. In the financial stock exchange environment, a wavelength transform, a mathematical function, reduces noise.
Equations
To evaluate deep learning models, loss error is usually used, along with root mean square error (RMSE), mean absolute error (MAE), and mean absolute percentage error (MAPE). A linear function was used to predict future stock market fluctuations based on the concatenation of corporation and sentence representations.
Experiments and Evaluation
Evaluation Metrics
The forecasting accuracy was measured by RMSE and MAPE in [2]. ROC, MACD, and CCI were used to measure the trend following the characteristics of market prices. The authors used Duan et al. (2018)’s dataset to train and test their system architecture predictions on S&P 500 new process data in [7]. We used the Scikit-Learn library and the open-source openIE tool to implement logistic regression and Random Forest models. They set multi-head attention heads to 8 and started word nodes with 300-dimension GloVe embeddings.
Standard & Poor’s 500 stock (S&P 500) index and its selected stocks were predicted using public financial news data crawled from Reuters and Bloomberg. To predict the S&P 500 index and selected individual stocks, Matthews Correlation Coefficient was used.
Evaluation of deep learning models was done using Simulation Environment and Data, and index forecasting effect analysis was done using 1219-day data samples for five years from 2015 to 2019. Loughran and McDonald (2011) released sentiment lexicons and methods for sentiment analysis and stock market prediction, along with LSTM, DARNN, MFNN, and CA-SFCN models.
A semantic knowledge acquisition method performed better than a feature-based method at predicting stock prices. Our prediction model uses 1219-day Shanghai Composite Index data samples for 5 years from 2018 to 2022 to predict closing prices.
Results and Discussion
A modern deep learning and natural language processing technique significantly improved stock market prediction accuracy. A CEEMD-CNN-LSTM algorithm was used to extract deep features and time sequences, then applied to one-step-ahead stock price forecasting. This combined approach worked well in practice.
Conclusion
In conclusion, predicting stock market prices remains a challenging task due to the complex and noisy nature of financial data. LSTMs and CNNs, especially deep learning models, have shown significant potential in NLP and machine learning. To improve prediction accuracy, contextual information, noise reduction, and sophisticated models are important.
Future Work
There’s a possibility of using TSK-based fuzzy systems with constrained gradient-based techniques to improve models like ANRES. It’s possible to reduce rule base size while keeping interpretability by using Mamdani-type fuzzy systems. Furthermore, sentiment analysis on social media platforms like Twitter, along with other qualitative indicators like news and policy changes, can help improve prediction models. Stock market trend predictions can also be improved using Elliott waves.
Read the research paper here: https://arxiv.org/pdf/2208.13564
[6]: Khang, P.Q., Hernes, M., Kuziak, K., Rot, A., & Gryncewicz, W. (2020). Liquidity prediction on Vietnamese stock market using deep learning. KES.
[3]: Gyamerah, S., & Awuah, A. (2020). Trend Forecasting in Financial Time Series with Indicator System.
[7]: Xiong, K., Ding, X., Du, L., Liu, T., & Qin, B. (2021). Heterogeneous graph knowledge enhanced stock market prediction. AI Open, 2, 168–174.
